DEV Community: Emmanuel R

A 14-Year-Old PHP Bug Lets Strangers Run Code on Your Server. Here's How to Stop It

Emmanuel R — Thu, 22 Oct 2026 11:54:50 +0000

The code nobody touched for years just became everyone's problem. A bug hiding in PHP since 2012 lets strangers run code on web servers, and the NSA and India's CERT-In have both raised the alarm. Here's how PHP2Shell (CVE-2026-109618 and CVE-2026-109619) works, how it was found, and exactly what to do about it.

On 21 October 2026, PHP shipped an emergency release for two critical bugs in the PHP engine itself: CVE-2026-109618 and CVE-2026-109619, nicknamed PHP2Shell. Both let an attacker run code on a web server without logging in [1][2].

Governments moved fast. The NSA told organizations that can't update right away to take their PHP systems offline [3]. The same day, India's national cybersecurity agency, CERT-In, issued a critical-severity advisory (CIVN-2026-0510) warning that vulnerable PHP servers could be exposed to remote code execution [10][11]. CISA in the US issued its own alert [6].

The unsettling part is where the bugs came from: code that has sat in PHP since 2012, and that attackers started probing before a patch existed. This post explains what the two bugs are, how they work, how they were found, and what to do about them. If you run PHP anywhere, including inside WordPress, Drupal, Magento or an internal tool, it applies to you.

The Two Bugs at a Glance

	CVE-2026-109618	CVE-2026-109619
Name in the advisory	RCE via OOM corruption	RCE via stale global state
Bug type	Memory corruption when PHP runs out of memory	Use-after-free across two requests
Severity	CVSS v4.0 9.2, critical	CVSS v4.0 9.2, critical
Login needed?	No	No
Confirmed exploit	Apache mod_php	mod_php and PHP-FPM
Advisory	GHSA-w8hw-w9fx-j79w	GHSA-6wmf-4rv5-p3rp

In both cases, a successful attack runs code as the web server's PHP user. From there an attacker can read files, steal database passwords and keys from config files, or plant a backdoor [1][2].

How the Bugs Work

CVE-2026-109618: running out of memory at the wrong moment

When PHP runs out of memory, it doesn't return an error step by step. It jumps straight out of whatever it was doing, using a C function called longjmp(). If that happens while a request is still being set up, some of the request's data is left half-built. Later, when PHP cleans up the request, it treats that half-built data as valid, and memory gets corrupted [1].

An attacker can cause this on purpose. Weaknesses in how PHP parses multipart form uploads let a crafted request use up memory while still looking like it's within the configured limits. On Apache with mod_php, this happens before any application code runs, so the app itself doesn't need to have a bug [1][4].

CVE-2026-109619: one request's leftovers, used by the next

A PHP worker handles many requests in a row, and it keeps some global state between them: error, session and extension data. Normally that state is reset at the end of each request. But a fatal error, such as running out of memory or hitting the execution time limit, can skip that reset. The worker is left holding a pointer to memory that's already been freed [2].

If the next request on the same worker touches that state before it's reset, PHP uses freed memory. The advisory confirms two ways to reach it. One is session upload progress, a feature that's on by default and works even if the app never calls session_start(). The other is PHP 8.5's new fatal error backtraces setting, also on by default. Besides running code, this bug can leak memory from other users' requests, including passwords and tokens [2].

How It Was Found

Both bugs were found in August 2026 by researchers auditing PHP's C source with the help of AI models. Researcher Ryan Austin at Calif found one during an audit run under OpenAI's "Patch the Planet" project. Wiz found the same flaws independently, using AI models alongside its own tooling [5]. The PHP advisories credit reporters from both efforts and others [1][2].

The weakness wasn't entirely new. In 2024 a PHP core developer spotted the out-of-memory problem behind CVE-2026-109618 during an audit, but the team couldn't agree on a fix, and a working exploit was thought unlikely, so it wasn't prioritised [5]. The affected code dates back to PHP 5.4, released in 2012.

Timeline

The release was meant to be a normal coordinated disclosure, but it didn't stay quiet. Amazon's honeypots logged an exploit attempt on 17 October, four days early. On 20 October news leaked on Hacker News, watchTowr posted about it, and a proof-of-concept video was published by mistake, so PHP announced the release a day ahead. The fixes and the two GitHub advisories went out on 21 October, the same day as alerts from CISA, the NSA, India's CERT-In and others [5][6][10].

Some organizations didn't wait. Government portals in the Netherlands, Finland and France, many Wikimedia services and the Internet Archive went offline for a time as a precaution. Wiz compared the bug to Heartbleed, the 2014 OpenSSL flaw [5].

Am I Affected?

Versions: everything from PHP 5.4 up to, but not including, 8.2.35, 8.3.36, 8.4.27 and 8.5.12 [1][2].
Server setups: any PHP that serves web requests: Apache mod_php (highest risk), PHP-FPM, php-cgi, FrankenPHP and LiteSpeed [4].
Old versions: PHP 8.1 and older are end of life. The PHP team released source patches for 7.4, 8.0 and 8.1, but no official builds. Older versions get nothing [5].
Hidden PHP: PHP is often bundled inside other products, such as control panels, appliances and vendor software. Cisco, for example, published its own advisory for affected products [6][7].

To check a server, look at the version and the two settings that matter most:

php -v
php -i | grep -E 'session.upload_progress.enabled|fatal_error_backtraces|memory_limit'

Run this with the same PHP the web server uses. The command-line PHP can be a different version from PHP-FPM or mod_php, so also check your web server's phpinfo() page or package list.

What to Do Now

For security teams, this goes beyond applying an update. The work falls into four steps: patch, mitigate, contain and investigate. Start by finding every PHP install, internet-facing ones first [6].

1. Patch

Upgrade to a fixed release: 8.5.12, 8.4.27, 8.3.36 or 8.2.35, or your Linux distribution's patched package. Then restart every PHP worker: PHP-FPM, Apache and any long-running PHP processes. Until they restart, the old code is still running [1][2].

2. Mitigate, if you can't patch today

The PHP advisories list temporary settings. Set them in php.ini, the PHP-FPM pool or the web server config. Setting them with ini_set() or .user.ini is too late to help [1][2][4].

; CVE-2026-109619: closes the confirmed session path.
; Worth keeping off after patching if you don't use upload progress bars.
session.upload_progress.enabled = 0

; CVE-2026-109619, PHP 8.5 only: closes the confirmed backtrace path.
fatal_error_backtraces = 0

; CVE-2026-109618: avoids the out-of-memory jump. Not a complete fix,
; so pair it with tight request limits below and at your proxy.
memory_limit = -1
post_max_size = 8M
upload_max_filesize = 2M

Limit requests before they reach PHP. At nginx, Apache or your load balancer, cap request size, the number of form fields and multipart parts, and headers per part. If your app doesn't take file uploads or multipart forms, block them entirely [1].
Put a web application firewall in front if you have one [6].
Don't rely on RLIMIT_AS or RLIMIT_DATA. These operating system memory limits can trigger the very bug you're trying to avoid [1].
Hiding errors doesn't help. Turning off display_errors or using @ doesn't stop the stale state from being used [4].

These settings only close the paths researchers have confirmed so far. Both advisories say updating is the only complete fix [1][2].

3. Contain

If an internet-facing system can't be patched yet, or runs PHP 8.1 or older, take it offline until it's patched or upgraded. This is what the NSA and CISA both advise [3][6]. Several governments, Wikimedia and the Internet Archive did exactly this on 21 October [5].

4. Investigate

Exploit attempts started before the patch, so a patched server may already have been hit. Look for:

PHP worker processes starting a shell (sh, bash) or tools like curl and wget [6].
Signs of persistence: new or changed .php files in web directories (especially upload folders), new cron jobs or new user accounts.
Unexpected outbound connections from web servers.
Bursts of large or unusual multipart POST requests in your access logs.

Clean logs don't prove a server is safe [2]. If you find signs of compromise, treat it as an incident: rotate the database passwords and API keys stored on that server, not just the PHP version.

The Bigger Lesson

Legacy code isn't harmless just because it's familiar. The code behind PHP2Shell had run on millions of servers for 14 years, and it was trusted because nothing had gone wrong yet.

Neither bug was in anyone's PHP code. Both were memory-safety bugs in the PHP engine, which is written in C: memory used after it was freed, and data used before it was fully set up. These are the same classes of bug the NSA and CISA have pointed to for years when they urge a move to memory-safe languages for new code [8][9].

Two other points stand out. AI-assisted audits found bugs that had sat in widely used code for over a decade, and attackers have the same tools. And PHP runs a large share of the web, yet the PHP Foundation has only 13 employees and relies on donations [5]. If your business depends on PHP, supporting the foundation is part of keeping it safe.

For now, the steps are simple: find every PHP server, update it, restart it, and check it wasn't hit first.

References

Advisories

PHP. RCE via OOM corruption (CVE-2026-109618, GHSA-w8hw-w9fx-j79w). https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w
PHP. RCE via stale global state (CVE-2026-109619, GHSA-6wmf-4rv5-p3rp). https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp
NSA Cybersecurity. Post on PHP vulnerabilities, 21 October 2026. https://x.com/NSACyber/status/2112903220209098897

Analysis and background

Rapid7. CVE-2026-109618 and CVE-2026-109619: Critical Unauthenticated Remote Code Execution in PHP. https://www.rapid7.com/blog/post/etr-cve-2026-109618-and-cve-2026-109619-critical-unauthenticated-remote-code-execution-in-php/
Wikipedia. PHP2Shell. https://en.wikipedia.org/wiki/PHP2Shell
CISA. Critical Vulnerabilities in PHP Require Immediate Patching, 21 October 2026. https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching
Cisco. Vulnerabilities in PHP Affecting Cisco Products: October 2026. https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-php-october26-Jj384Daq
NSA. Software Memory Safety (Cybersecurity Information Sheet). https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI_SOFTWARE_MEMORY_SAFETY.PDF
CISA. The Case for Memory Safe Roadmaps. https://www.cisa.gov/resources-tools/resources/case-memory-safe-roadmaps
CERT-In. Vulnerability Note CIVN-2026-0510: PHP2Shell vulnerabilities in PHP, 21 October 2026. https://www.cert-in.org.in/s2cMainServlet?VLCODE=CIVN-2026-0510&pageid=PUBVLNOTES01
NewsBytes. US NSA and India's CERT-In flag 'PHP2Shell' web risk. https://www.newsbytesapp.com/news/science/us-nsa-and-indias-cert-in-flag-php2shell-web-risk/tldr

CobuildX helps teams modernise legacy systems with MigrateX.

Originally published on the CobuildX blog.

Modernize Your Legacy Frontend Without a Rewrite

Emmanuel R — Tue, 20 Oct 2026 08:06:01 +0000

Rewrites don't fail because of the new framework. They fail because of everything the old app does that nobody wrote down.

MigrateX moves legacy frontends — Ember, Backbone, Angular, jQuery — to React one feature at a time, while your product keeps shipping.

🔍 Map everything first. Every route, service and hidden side effect, before a line is ported.
🔀 Ship behind a switch. Each feature can be turned back to the old app instantly.
✅ Prove parity. The same tests run on old and new. It goes live only when both pass.
🧹 Leave nothing behind. Old code, switches and glue removed. One clean codebase.

No feature freeze. No launch day.

See it working: five real apps migrated in the open → migratex-examples

👉 cobuildx.ai/migratex · Try the platform

Modernizing Legacy Apps with RxJS: Streams as the Bridge Between Old and New

Emmanuel R — Mon, 12 Oct 2026 05:32:08 +0000

In most legacy migrations, the difficult part is the async code, not the templates: callbacks, observers, global events and timers that every screen depends on. This post explains how we use RxJS as a bridge, so the old UI and the new React UI share the same streams while screens are moved one by one.

In most legacy frontend migrations, the templates are not the hard part. Templates can be ported quite easily. The hard part is the async code. A jQuery handler sends a request, sets a global flag and fires a custom event. A Backbone model has three views listening to its change event. An Ember observer saves data on every keystroke. A setInterval keeps polling the server, and nobody in the team knows why it was added.

This is where the actual behavior of a legacy app lives, and it is spread across every page. We cannot move it one route at a time, because both the old screens and the new screens depend on it.

In such projects, we use RxJS as a bridge between the old app and the new one. We do not use it as the final architecture, and we do not use it just because Angular uses it. The approach is simple. Wrap each legacy event source in an Observable and keep the shared state behind a stream. Now the old UI and the new React UI can subscribe to the same data while screens are moved one by one. As a bonus, a lot of hand-written debounce, retry and cancel logic becomes a few lines of operators. This is the branch by abstraction pattern, with streams as the interface.

The Short Version

Legacy pattern	What goes wrong	RxJS replacement
Callbacks and `$.Deferred` chains	Deeply nested, cannot be cancelled, errors get lost	`from(promise)`, `defer`, `switchMap`
Global or custom events (`$(document).trigger`, Backbone `on`)	Any code can fire them, and nobody removes the listeners	`fromEvent`, `fromEventPattern`, typed `Subject`s
Observers and watchers (Ember `observer`, `$scope.$watch`)	Order is not clear, and they run too often	`BehaviorSubject` + `distinctUntilChanged`
Hand-rolled debounce and timers	Old responses overwrite new ones, timers keep running	`debounceTime`, `switchMap`, `takeUntil`
Polling with `setInterval`	Requests overlap, and polling never stops	`timer` + `switchMap` / `exhaustMap`
Retry loops	Too many retries, no delay between them	`retry({ count, delay })`
WebSockets with manual reconnects	Duplicate connections, lost messages	`webSocket` + `retry` + `share`

Every row follows the same idea: treat "something that happens over time" as a value that we can pass around, combine and unsubscribe from. Once both apps use Observables, the framework on each side becomes much less important.

Why RxJS Works Well for Modernization

There are many state management libraries. For a migration, RxJS is useful for three specific reasons:

It does not depend on any framework. An Observable works the same way whether jQuery, Backbone, Ember, AngularJS or React subscribes to it. This is exactly what we need during a migration, when the old and new code run on the same page.
One model for every kind of async. DOM events, custom events, promises, timers, WebSockets and Node-style callbacks all become the same type. There is no need to write a different pattern for each source.
Cancellation and cleanup are built in. Every subscription can be closed. Legacy apps often leave handlers and timers running, and many of these leaks are bugs that users have simply got used to.

There is also one clear reason to be careful: RxJS takes time to learn, and a codebase where everything is a stream is difficult to read. We will discuss where to stop later in this post. In short, use it at the boundaries and for real event streams, not for every value in the app.

Step 1: Wrap the Legacy Event Sources

First, find every place where the old app produces events or async results, and create an Observable wrapper for each one. Do not change the legacy code at this stage. The wrapper only listens.

DOM and jQuery events. fromEvent accepts DOM elements and any object with jQuery-style on/off methods, so a jQuery object can be passed directly:

import { fromEvent, map } from 'rxjs';

// The legacy app fires: $(document).trigger('cart:updated', [cart])
export const legacyCartUpdated$ = fromEvent($(document), 'cart:updated', (_e, cart) => cart);

export const searchInput$ = fromEvent<InputEvent>(document.querySelector('#search')!, 'input').pipe(
  map((e) => (e.target as HTMLInputElement).value),
);

Backbone, Ember and anything with its own add/remove API. fromEventPattern takes an add handler and a remove handler. This way, the Observable removes its listener from the legacy object properly when it is unsubscribed:

import { fromEventPattern, startWith, map } from 'rxjs';

// Backbone model or collection
export const fromBackbone = <T>(model: Backbone.Model, event = 'change') =>
  fromEventPattern<Backbone.Model>(
    (handler) => model.on(event, handler),
    (handler) => model.off(event, handler),
  ).pipe(startWith(model), map((m) => m.toJSON() as T));

// Ember object property (Ember.addObserver / removeObserver)
export const fromEmberProperty = <T>(obj: object, key: string) =>
  fromEventPattern<T>(
    (handler) => addObserver(obj, key, handler),
    (handler) => removeObserver(obj, key, handler),
  ).pipe(map(() => get(obj, key) as T), startWith(get(obj, key) as T));

Two points are important here. First, convert to plain objects at the boundary (toJSON(), get). If the stream emits Backbone models or Ember records, the new code will also start calling .get('title'), and the legacy model will spread into React. This is the anti-corruption layer from our strategies post, applied to streams. Second, startWith gives the current value to late subscribers. So a React component that mounts later will not show an empty screen while it waits for the next change.

Promises, callbacks and sockets also have direct wrappers:

import { defer, bindNodeCallback } from 'rxjs';
import { webSocket } from 'rxjs/webSocket';

// defer: the request only starts when someone subscribes, and runs again per subscriber
export const loadOrders$ = (userId: string) => defer(() => legacyApi.getOrders(userId));

// Node-style callback API from an old SDK
export const readConfig$ = bindNodeCallback(legacySdk.readConfig);

export const prices$ = webSocket<PriceTick>('wss://example.com/prices');

For requests, prefer defer(() => promise) over from(promise). A promise starts running as soon as it is created. defer waits until someone subscribes, and this is what makes cancellation and retries possible later.

Step 2: Put Shared State Behind a Stream

Wrapping the sources makes the old app observable. The next step is to give each piece of shared state (the cart, the current user, the selected account, feature flags) a single owner. This owner is a small service that holds the value and exposes it as an Observable. Both apps read from this service, and all updates go through its methods.

import { BehaviorSubject, distinctUntilChanged, map } from 'rxjs';

export interface Cart { items: CartItem[]; total: number }

const state = new BehaviorSubject<Cart>({ items: [], total: 0 });

export const cartService = {
  cart$: state.asObservable(),
  count$: state.pipe(map((c) => c.items.length), distinctUntilChanged()),
  get value() { return state.getValue(); },

  add(item: CartItem) {
    const items = [...state.value.items, item];
    state.next({ items, total: sum(items) });
  },
};

// Keep the legacy event working for screens that still listen to it
cartService.cart$.subscribe((cart) => $(document).trigger('cart:updated', [cart]));

The legacy side subscribes and updates the DOM the same way it does today:

// legacy/header.js (jQuery)
const sub = cartService.count$.subscribe((n) => $('#cart-badge').text(n));
$('#add-to-cart').on('click', () => cartService.add(readItemFromForm()));

// Backbone view: tie the subscription to the view's lifetime
initialize() { this.sub = cartService.cart$.subscribe((c) => this.render(c)); },
remove() { this.sub.unsubscribe(); Backbone.View.prototype.remove.call(this); },

React reads the same stream through a small hook. useSyncExternalStore is the official React API for subscribing to an external store. It also avoids the "tearing" issue that a useEffect + useState version can have during concurrent rendering:

import { useSyncExternalStore } from 'react';
import type { Observable } from 'rxjs';

// Subscribe to a stream that always has a current value (like a service backed by a BehaviorSubject)
export function useStream<T>(stream$: Observable<T>, getValue: () => T): T {
  return useSyncExternalStore(
    (onChange) => {
      const sub = stream$.subscribe(onChange);
      return () => sub.unsubscribe();
    },
    getValue,
  );
}

// In a component
function CheckoutSummary() {
  const cart = useStream(cartService.cart$, () => cartService.value);
  return <p>{cart.items.length} items, {formatMoney(cart.total)}</p>;
}

That is the complete seam. A screen can move from the old app to React, and the cart badge in the legacy header will still update, because neither side owns the cart. Once the last legacy subscriber is removed, the service internals can be replaced (with Zustand, TanStack Query or plain React context) without changing any component, because components only depend on cart$ and add.

Issues we faced

Exposing the Subject directly. If the service exports state instead of state.asObservable(), any file can call .next(), and we are back to global events. Expose streams for reading and methods for writing.
Changing emitted objects in place. state.value.items.push(item) changes the object that both apps hold, but nothing is emitted. Always emit a new object, because useSyncExternalStore depends on reference changes.
Updates inside updates. A legacy handler that calls add() while handling a cart$ emission can create a loop. Using distinctUntilChanged on derived streams, and keeping the legacy re-trigger one-way, prevents most of these cases.
Loops with the legacy event. If the legacy app both fires and listens to cart:updated, either wrap it with fromEvent or re-trigger it from the service. Do not do both.

Step 3: Replace Hand-Written Async Code with Operators

Once events become streams, a lot of fragile legacy code can be rewritten in a few lines, and it usually becomes more correct too. A search box is the common example. Most legacy apps have some version of this jQuery code:

let timer, lastQuery;
$('#search').on('input', function () {
  clearTimeout(timer);
  const q = $(this).val();
  timer = setTimeout(() => {
    if (q === lastQuery) return;
    lastQuery = q;
    $.getJSON('/api/search', { q }).done(renderResults); // a slow early response can overwrite a newer one
  }, 300);
});

It looks fine, but it has a race condition. If the request for "rea" is slow and the request for "react" is fast, the old results arrive last and replace the correct ones. The RxJS version avoids this by design, because switchMap unsubscribes from the previous request as soon as a new query comes in:

import { debounceTime, distinctUntilChanged, filter, switchMap, catchError, of } from 'rxjs';
import { ajax } from 'rxjs/ajax';

export const results$ = searchInput$.pipe(
  debounceTime(300),
  distinctUntilChanged(),
  filter((q) => q.length >= 2),
  switchMap((q) =>
    ajax.getJSON<SearchResult[]>(`/api/search?q=${encodeURIComponent(q)}`).pipe(
      catchError(() => of([])), // one failed search shouldn't kill the stream
    ),
  ),
);

ajax from rxjs/ajax cancels the actual HTTP request when it is unsubscribed. With fetch, pass an AbortSignal inside new Observable (or use fromFetch from rxjs/fetch) to get the same behavior.

The same small set of operators covers most of the other hand-written patterns we see in legacy apps:

Need	Operator	Why it is better than the legacy version
Only the latest request matters (search, filters, route params)	`switchMap`	Cancels old requests, so results never arrive out of order
Ignore clicks while a save is in flight	`exhaustMap`	Replaces `isSaving` flags and disabled-button workarounds
Every action must run, in order (queued writes)	`concatMap`	Writes do not mix, and no update is lost
Poll until the user leaves the page	`timer(0, 5000).pipe(switchMap(load), takeUntil(leave$))`	Polls never overlap, and polling stops reliably
Retry flaky calls with backoff	`retry({ count: 3, delay: (_, i) => timer(2 ** i * 500) })`	Limited attempts with a growing delay
Combine several sources	`combineLatest`, `withLatestFrom`	Replaces flags like `if (userLoaded && settingsLoaded)`
Time out a slow call	`timeout({ first: 8000 })`	Configured in one place, and errors are reported properly

Polling needs one more example, because legacy polling code often causes duplicate requests and battery drain:

import { timer, defer, switchMap, takeUntil, retry, share, fromEvent, filter } from 'rxjs';

const hidden$ = fromEvent(document, 'visibilitychange').pipe(filter(() => document.hidden));

export const jobStatus$ = (jobId: string) =>
  timer(0, 5000).pipe(
    switchMap(() => defer(() => api.getJob(jobId))),
    retry({ count: 3, delay: 1000 }),
    takeUntil(hidden$),
    share(), // several widgets can watch the same job with one poller
  );

Common Mistakes

Leaked subscriptions

Every subscribe() must be closed at some point. In legacy code, connect it to the lifecycle that already exists: a Backbone view's remove, an Ember component's willDestroy, an AngularJS $scope.$on('$destroy'), or a jQuery plugin's destroy. In React, the hook's cleanup takes care of it. For streams that should end on an event, keep takeUntil(destroy$) as the last operator in the pipe, so that nothing after it can subscribe again.

Hot and cold Observables

A cold Observable (like defer(() => fetch(...))) runs again for every subscriber. If three components subscribe to the same stream, three requests are made. When a result should be shared, add share() or shareReplay({ bufferSize: 1, refCount: true }). Use refCount: true unless the source really needs to run forever. Without it, shareReplay keeps the inner subscription (and its polling or socket) running even after the last subscriber has left.

Turning everything into a stream

A common mistake in RxJS projects is a codebase where a simple boolean flag becomes a BehaviorSubject<boolean> passed through four operators. Use streams only where time matters: events, sockets, polling, typeahead and state shared between the two apps. For normal server data in React (load a list, show it, refetch on change), TanStack Query is simpler and also handles caching. For local component state, use useState. A simple rule: if you cannot draw a marble diagram for it, it probably should not be an Observable.

Errors stop streams

When an error reaches a long-running stream, the stream ends and every subscriber stops getting updates, without any visible message. Catch errors inside the switchMap/mergeMap (as shown in the search example) so that the outer stream keeps running, and log every error you catch.

Bundle size and RxJS versions

RxJS 7 supports tree-shaking, and from version 7.2 the operators are exported directly from rxjs. So a typical bridge layer adds only a small amount to the bundle. If the legacy app still uses RxJS 5 or 6 (common in older Angular apps), the official migration guide explains the upgrade to 7. rxjs-compat was meant only for the 5 to 6 upgrade and should not be used in a new layer.

Testing the Bridge

Streams are easy to test without a browser. TestScheduler runs time-based operators on virtual time, so a 300 ms debounce does not actually wait 300 ms. Marble strings describe the input and the expected output:

import { TestScheduler } from 'rxjs/testing';

// The search stream, with its inputs passed in instead of read from globals
export const searchPipeline = (input$: Observable<string>, search: (q: string) => Observable<SearchResult[]>) =>
  input$.pipe(debounceTime(300), distinctUntilChanged(), filter((q) => q.length >= 2), switchMap(search));

const scheduler = new TestScheduler((actual, expected) => expect(actual).toEqual(expected));

test('search only queries the last term after a pause', () => {
  scheduler.run(({ cold, expectObservable }) => {
    const input$ = cold('a 100ms b 400ms c|', { a: 'rea', b: 'react', c: 'react' });
    const api = (q: string) => cold('--r|', { r: [`result for ${q}`] });
    // 'rea' is dropped by the debounce; 'react' settles at 401ms, its result arrives 2 frames later
    expectObservable(searchPipeline(input$, api)).toBe('403ms x 99ms |', { x: ['result for react'] });
  });
});

Writing the pipeline as a function of its inputs is what makes it testable, and it is a good practice for the seam in any case. For checking that both apps behave the same, use the approach from our migration guides: one end-to-end test suite that runs the same screen in the old app and in the new app, and checks for the same visible result. The stream layer is what keeps those results the same, so it needs good test coverage.

Temporary Bridge or Long-Term Layer?

At the end of the migration, decide what happens to the RxJS layer. Both options below are fine, as long as the team makes a clear decision:

Temporary bridge. The bridge was added so that old and new code could share state. Once the legacy app is removed, plain React state, TanStack Query or a small store can handle most of it. Remove the Subjects, and keep only the few streams that represent real events.
Long-term layer. Apps with a lot of real-time behavior (trading screens, live dashboards using sockets, collaborative editors, IoT consoles) often keep RxJS as the event layer under React. In that case, document the rules: where streams are kept, how errors are handled, and which shared streams use shareReplay.

Whichever option you choose, delete the fromEventPattern wrappers around legacy objects in the same sprint as the last legacy screen. These wrappers are always temporary.

Further in This Series

This post covers one technique within a larger migration plan. The posts below explain the overall plan and each starting framework in more detail:

Frontend migration strategies: strangler fig, branch by abstraction and vertical slicing, and how they combine.
One-way vs two-way data binding: why observers and watchers are difficult to migrate, and what replaces them.
Framework guides: Angular, Ember, Backbone, jQuery and vanilla JS to React.

Our Migration Plugins

Finding every event source, observer and shared service in a legacy app takes the most time, and it decides where the streams should go. We have packaged our approach as free, open-source plugins for Claude Code, one for each starting framework:

Plugin	Migrates from
`ember-to-react`	Ember Classic and Octane
`angular-to-react`	Angular 2+ and AngularJS 1.x, including hybrid ngUpgrade apps
`backbone-to-react`	Backbone.js, Marionette, jQuery templates
`jquery-to-react`	jQuery apps, plugins and jQuery UI
`vanillajs-to-react`	Plain HTML, CSS and JavaScript

Each plugin comes with a migration skill, a /<plugin>:plan command that writes a migration plan (strategy, units, route map, test plan and risks) without touching your code, a read-only inventory agent that maps the old app, a parity-reviewer agent that compares each migrated piece with its original, and a bundled Context7 server for up-to-date library docs. To install one in Claude Code:

/plugin marketplace add cobuild-tech/cbx-plugins
/plugin install backbone-to-react@cbx-plugins

They also work in Cursor, in VS Code with GitHub Copilot, and in the GitHub Copilot CLI. Progress is kept in a .migration/ folder in your repo, so a migration can carry on across sessions and teammates. The source and setup steps are at cobuild-tech/cbx-plugins, and issues are welcome.

About MigrateX

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. It combines automation with experienced engineers. The automation does the repetitive work quickly, and the engineers make the tricky decisions, like which shared state needs a stream-based seam and which can be moved directly.

A MigrateX project usually runs like this:

Assessment. We map every route, shared service, event source and hidden dependency in your app, so you see where the work is before anyone agrees on a timeline.
Plan. With your team, we choose the strategy, the slice order, and what can simply be deleted.
Migration, step by step. Automation handles the repetitive parts. Our engineers handle the router, the seams and the code that connects old and new.
Proof that it works. The same tests run against both versions. A piece only counts as moved when it passes on both.
Handover. We remove the temporary code, the seams and the flags, and leave your team with a codebase they understand and own.

Your app keeps running and shipping features the whole time, so there's no big launch day. Our migratex-examples repo has small apps in each legacy framework next to their migrated versions.

Working on the async code of a legacy app? Share a few details with us through Contact Us, and we will start with a free discussion on where the seams should go.

References

RxJS

RxJS. Overview and API reference. https://rxjs.dev/guide/overview
RxJS. fromEventPattern. https://rxjs.dev/api/index/function/fromEventPattern
RxJS. Testing RxJS Code with Marble Diagrams. https://rxjs.dev/guide/testing/marble-testing
RxJS. Breaking Changes in Version 7. https://rxjs.dev/deprecations/breaking-changes
Ben Lesh. Hot vs Cold Observables. https://benlesh.medium.com/hot-vs-cold-observables-f8094ed53339

React and migration patterns

React. useSyncExternalStore. https://react.dev/reference/react/useSyncExternalStore
Martin Fowler. Branch By Abstraction. https://martinfowler.com/bliki/BranchByAbstraction.html
Microsoft. Anti-corruption Layer pattern. https://learn.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer
TanStack Query. Overview. https://tanstack.com/query/latest/docs/framework/react/overview

Source code

CobuildX AI. cbx-plugins: Claude Code plugins for Backbone, Angular, Ember, jQuery and Vanilla JS to React migrations. https://github.com/cobuild-tech/cbx-plugins
CobuildX AI. migratex-examples: small jQuery, Backbone, Angular, Ember and Vanilla JS apps and their migrated versions. https://github.com/cobuild-tech/migratex-examples

Originally published on the CobuildX blog.

Strangler Fig, Branch by Abstraction, Vertical Slices: Choosing a Frontend Migration Strategy

Emmanuel R — Mon, 12 Oct 2026 05:31:32 +0000

Strangler fig, branch by abstraction, vertical slicing: the strategy decides whether a frontend migration ships, more than the framework you move to. Here's how each one works, where each one bites, and how we combine them on real projects.

Teams that plan a frontend migration spend weeks on the target: React or Vue, Next.js or Vite, Redux or TanStack Query. Then they spend almost no time on the part that decides whether it ships: how the old app and the new app will live together until the old one is gone.

That part is the migration strategy. Pick the wrong one and a good framework choice still turns into an 18-month rewrite branch that never merges. Pick the right one and even a messy legacy app moves over a piece at a time, with users never noticing a launch day.

This post covers the strategies we actually use: the big-bang rewrite (as the baseline), the strangler fig, branch by abstraction, and vertical slicing, plus the supporting tactics that make them safe: parallel runs, feature flags, micro-frontends, anti-corruption layers and expand-contract. Then it shows how they fit together, because on a real project you rarely use just one. Our framework-specific guides (Angular, Ember, Backbone, jQuery, vanilla JS) go deeper on each starting point. This one is about the strategy underneath all of them.

The Short Version

Strategy	Unit of change	What runs side by side	Rollback	Best for
Big-bang rewrite	The whole app	Nothing in production until launch day	Hard: the old app has gone stale	Small or frozen apps
Strangler fig	A route or a screen	Old and new apps behind one router	Point the route back at the old app	Apps with clear routes or pages
Branch by abstraction	A shared dependency behind a seam	Old and new implementations behind one interface	Flip the switch back	Services, stores and widgets used everywhere
Vertical slicing	One user-facing feature, top to bottom	Migrated and unmigrated features	Per slice	Choosing the order of work

The first row is the one people start with. The other three are what actually works, and they answer different questions. Strangler fig decides where the line between old and new runs. Branch by abstraction handles the things that cross that line. Vertical slicing decides what to move next.

Big-Bang Rewrite: the Baseline

The big-bang rewrite is the default plan nobody writes down: start a new repo, rebuild everything, switch over on a chosen day. It's worth understanding properly, because every other strategy exists to avoid its failure modes.

When it is actually right

The app is small. A few screens that one or two people can rebuild in a few weeks. The overhead of running two apps side by side costs more than the rewrite.
The app is frozen. No new features are planned, so the old app won't move while you rebuild it.
The old app can't host anything new. Some legacy stacks make it so painful to mount new code that coexistence costs more than starting clean. This is rarer than it seems. Almost anything can serve an iframe or hand off a route.

Why it usually fails

The target keeps moving. The business doesn't stop for your rewrite. Every feature shipped in the old app is one more thing the new one has to catch up on, and the gap rarely closes.
Nothing is proven until the end. Months of work with zero real users means every wrong assumption surfaces on launch day, all at once.
The old app holds knowledge nobody wrote down. Ten years of bug fixes are hidden in it. Joel Spolsky called rewriting from scratch the single worst strategic mistake a software company can make, for exactly this reason.
Rollback is fiction. By launch day the old app is months behind. "Roll back" means going back to software everyone has stopped maintaining.

The incremental strategies below all share one idea: get the new code in front of real users early, in small pieces, while the old app keeps running.

Strangler Fig

Martin Fowler named this pattern after a vine that grows around a host tree until the tree is gone and the vine stands on its own. In a frontend migration, the vine is the new app and the tree is the old one.

How it works

Put something in front of both apps. A reverse proxy, a CDN rule, or a thin shell app that owns the router. Every request goes through it.
Send every route to the old app at first. Nothing has changed for users, but the switch now exists.
Move one route. Build it in the new app and flip that one path in the router.
Repeat until the old app serves nothing. Then switch it off.

The routing layer can be as simple as a list of migrated paths:

// shell/router.ts: which app owns which path
const MIGRATED = ['/settings', '/profile/:username', '/article/:slug'];

export function ownerOf(path: string): 'react' | 'legacy' {
  return MIGRATED.some((p) => matchPath(p, path)) ? 'react' : 'legacy';
}

In production this is often a CDN or proxy rule instead of code, and each route can sit behind a feature flag so you can roll it out to 5% of users first.

Route-level or widget-level

There are two grains to strangle at:

Route-level. Whole pages move at once. It's the cleanest option: each page is owned by one framework, and the boundary is a URL. Start here if your app has real routes.
Widget-level. New components are mounted inside old pages (a React date picker inside an Ember template, for example). This is how you migrate apps with one giant page, or a shared header that appears everywhere. It's more fragile, because the two frameworks now share a DOM.

Most projects use both: widget-level for the shared shell (nav, header, notifications), route-level for everything else.

Where it shines

Real users see real progress within weeks. Every migrated route is in production.
Rollback is one line. Point the route back at the old app.
Feature work never stops. New features go straight into the new app.
The end is visible. "34 of 52 routes migrated" is a progress bar a manager can read.

Where it costs you

Two runtimes at once. Users may download both frameworks for a while. Keep the old bundle off migrated pages where you can.
Shared state across the boundary. Login, the current user, feature flags and unsaved drafts have to be readable by both apps. Put them somewhere neutral: a cookie, localStorage, or the API.
CSS bleed. Global styles from one app leak into the other. Scope the new app's styles (CSS Modules, a prefix, or Shadow DOM) before you mount anything.
Navigation between apps. Crossing from a React page to an Ember page is a full page load unless the shell handles it. Users notice this, so group related routes and move them together.

Gotchas we hit

The router is the hard part, not the pages. Agree on how deep links, redirects, 404s and query strings behave across both apps before moving a single route.
Auth needs to move first. If the new app can't read the old app's session, every migrated page looks logged out. A shared cookie is the usual fix.
Don't strangle by file. Moving "all the components in /shared" isn't strangling, it's a horizontal layer (see vertical slicing below). Strangle by what users see.
Delete as you go. Once a route is moved and stable, remove its old code. Otherwise the old app never shrinks and you end up maintaining both forever.

The Angular guide walks through a strangler fig migration end to end, including the shell app and the staged rollout.

Branch by Abstraction

Strangler fig works when the thing you're replacing has a clear boundary, like a URL. But some parts of a frontend don't have one. The data layer, the auth service, the API client, the store, a date library: they're used from every page. You can't route around them. Branch by abstraction is the pattern for those.

The name comes from Paul Hammant and is described by Fowler and trunkbaseddevelopment.com. The idea: instead of making a long-lived version-control branch to replace something, you make the "branch" in the code itself, behind an interface.

How it works

Add a seam. Write an interface that describes what callers need. Make the old implementation fit it.
Move callers onto the seam. One at a time, change code that talks to the old thing directly so it talks to the interface instead. Ship each change. Nothing behaves differently yet.
Build the new implementation behind the same interface, in the main branch, switched off.
Switch over. A flag picks the new implementation, first for internal users, then for everyone.
Delete the old implementation, and the flag. If the seam isn't useful anymore, delete it too.

Here's what that looks like for a data layer. Say pages load articles through an Ember Data store or Backbone models, and you're replacing that with a plain API client and TanStack Query. The seam is a small repository interface:

// articles/repository.ts: the seam
export interface ArticleRepository {
  get(slug: string): Promise<Article>;
  list(filter: ArticleFilter): Promise<Article[]>;
  save(input: ArticleInput): Promise<Article>;
}

// old: wraps the legacy store, converting its records to plain objects
export const legacyArticles: ArticleRepository = {
  get: async (slug) => toArticle(await store.queryRecord('article', { slug })),
  list: async (filter) => (await store.query('article', filter)).map(toArticle),
  save: async (input) => toArticle(await store.createRecord('article', input).save()),
};

// new: talks to the API directly
export const apiArticles: ArticleRepository = {
  get: (slug) => api.get(`/articles/${slug}`).then((r) => r.article),
  list: (filter) => api.get('/articles', filter).then((r) => r.articles),
  save: (input) => api.post('/articles', { article: input }).then((r) => r.article),
};

export const articles: ArticleRepository =
  flags.isOn('articles-api') ? apiArticles : legacyArticles;

Two details make this work. First, the interface speaks plain objects, not framework records. If callers receive Ember Data models, they'll keep calling .get('title') and you haven't built a seam, just a wrapper. Second, both implementations live in the main branch and are tested in CI. The "branch" is a flag, not a Git branch.

Where you'll use it in a frontend

The data layer. Ember Data, Backbone collections, Angular services with their own caches → a query library. This is the most common use.
Auth and session. One session interface that both the old and new apps call, so login works on both sides of the strangler boundary.
A shared component. The date picker, the rich text editor, the data grid. Wrap it in one component API, then swap the implementation underneath.
A library replacement that isn't a framework change at all: moment.js → date-fns, a charting library, an HTTP client.

Why not just use a long-lived branch?

Because a migration branch that lives for months is a big-bang rewrite in disguise. It drifts from main every day, the merge at the end is huge, and nothing in it has met a real user. Branch by abstraction keeps every step small and merged. It pairs naturally with trunk-based development and with feature toggles.

The cost: extra code for a while (the interface, two implementations and a flag), and the discipline to actually do step five. Seams and flags that outlive the migration become their own kind of legacy.

Vertical Slicing

Strangler fig and branch by abstraction tell you how to keep old and new running together. Vertical slicing tells you what to move next.

A vertical slice is one user-facing feature moved top to bottom: its UI, its state, its data access, its routes and its tests, all together, until a user can do that thing entirely in the new code. The opposite is a horizontal layer: "first we'll port all the models, then all the services, then all the components".

Why horizontal layers fail

Nothing ships until the last layer is done. Porting every model first gives you months of code with no user, which is the big-bang problem again.
You guess at requirements. Without a real screen using it, a ported service is shaped by what the old code did, not what the new UI needs.
Progress is invisible. "80% of the services are done" means nothing to the people paying for it.

Choosing the first slice

The first slice is mostly about proving the plumbing: routing between apps, shared auth, the build and deploy, the test setup. So pick one that is:

Real, but low-risk. A settings page or a profile page, not checkout.
Light on shared dependencies. It shouldn't need half the app's services to render.
Easy to compare. It has clear behavior you can test old against new.
Used often enough that you'll hear quickly if something's wrong.

After the first one, order the rest by value against coupling. Move high-value, loosely coupled features early. Leave the tangled core, the screen that touches everything, until the seams around it already exist.

How slices map onto the other strategies

In a well-run migration, a vertical slice usually is a strangler fig step: a feature is a set of routes, and moving the feature means flipping those routes. When a slice hits a dependency it shares with unmigrated features (the article store used by both the feed and the editor), that's where a branch-by-abstraction seam goes in.

Supporting Tactics

These aren't migration strategies on their own. They're tools that make the three above safe.

Parallel run and dark launching

Run the new code alongside the old one in production, compare the results, but only show the old one. GitHub's Scientist library made this famous for backend code. In a frontend it's useful for a new data layer or a new calculation (pricing, permissions, validation): call both implementations behind the seam, log any differences, return the old result. Once the differences hit zero, switch. Dark launching is the same idea for whole features: ship them to production hidden, and turn them on later.

// compare old and new behind the seam; users still get the old result
async function get(slug: string) {
  const [oldResult, newResult] = await Promise.allSettled([
    legacyArticles.get(slug),
    apiArticles.get(slug),
  ]);
  reportDiff('articles.get', oldResult, newResult);
  if (oldResult.status === 'rejected') throw oldResult.reason;
  return oldResult.value;
}

Watch the cost: a parallel run doubles API calls. Sample a percentage of traffic, and never run both sides of a write.

Feature flags and canary releases

Every switch in this post (a migrated route, a new implementation behind a seam) should be a flag you can flip without a deploy. That lets you release to staff first, then 5% of users, then everyone, and roll back in seconds. Name each flag after the migration step and give it an expiry date. The feature toggles article on Fowler's site is the best guide to keeping them under control.

Micro-frontends as a way to coexist

Micro-frontends (module federation, single-spa, web components) let several apps share one page. They're a useful vehicle for widget-level strangling: the old app becomes one micro-frontend, the new one another. But think twice before making them the destination. If the goal is one React app, use the micro-frontend setup as scaffolding and take it down when the migration ends. Otherwise you've swapped one legacy app for a distributed system.

Anti-corruption layer

An anti-corruption layer translates between the old model and the new one so the old model's quirks don't leak into new code. The toArticle() function in the branch-by-abstraction example is a tiny one: it turns a legacy record (with its computed properties, observers and odd field names) into a plain object the new code can trust. Keep these at the boundary and delete them when the old side is gone.

Expand and contract

Also called parallel change. When old and new code share data (an API response, a localStorage key, a URL format, a cookie), you can't change its shape in one go. Expand first: support both the old and new shapes. Migrate each reader and writer. Contract: remove the old shape once nothing uses it. It's how you rename a query parameter or restructure saved user preferences without breaking the half of the app that hasn't moved yet.

Combining Them: the Playbook We Use

On a real project these aren't alternatives. They are layers of one plan:

Question	Answered by
Where does the line between old and new run?	Strangler fig (routes, then widgets)
What do we move next?	Vertical slicing (value against coupling)
What about things every page depends on?	Branch by abstraction (a seam, two implementations, a flag)
How do we know the new one behaves the same?	Parallel run and shared tests
How do we release and roll back?	Feature flags and canaries
How do we change shared data safely?	Expand and contract

In phases:

Foundation. Put the shell or proxy in front of the old app, with every route still pointing at it. Make auth work on both sides. Set up one test suite that can run against either app. Add seams around the data layer and other shared services. Nothing changes for users.
First slice. Move one low-risk feature end to end. Its real job is to prove the plumbing: routing, auth, deploys, monitoring, rollback.
Steady state. Move slices in order of value against coupling. Each slice is a set of routes flipped behind a flag. Switch seams to their new implementations as soon as their callers allow it, with a parallel run where correctness matters. Ship new features only in the new app.
Removal. When the old app serves no routes, switch it off. Then delete the seams you no longer need, the anti-corruption layers and every migration flag. A migration isn't done until the scaffolding is gone.

Choosing a Strategy

If you're deciding where to start, these questions settle most of it:

If your app…	Lean towards
Is a handful of screens, or is frozen	Big-bang rewrite. Coexistence isn't worth it.
Has clear routes and pages	Strangler fig at route level
Is one giant page, or has a shared shell everywhere	Strangler fig at widget level, often with a micro-frontend host
Has a data layer, store or service every page uses	Branch by abstraction around it, early
Has few tests	Write tests at the user level (end-to-end) first, then any strategy. Without them, nothing tells you a slice is done.
Has many teams shipping in parallel	Vertical slices owned by one team each, behind flags
Must keep shipping features during the migration	Anything except big-bang

Measuring Progress and Knowing When You're Done

Incremental migrations have a known failure mode: they stall at 80%. The easy routes are done, the hard core remains, and the team gets pulled onto feature work. You end up running two frameworks forever. Measuring progress openly is the best defense:

Routes served by the new app, as a percentage of traffic, not just a count. Ten rarely used admin pages don't equal one home feed.
Lines of legacy code left, falling every week. If it stops falling, something is wrong.
Seams and flags still open. Each one is temporary and should have an owner and an end date.
Parity results. For each slice: do the same tests pass against old and new, and does the parallel run show zero differences?

And a short definition of done that every slice has to meet before its old code is deleted: same tests passing on both, flag at 100% for a full release cycle with no rollbacks, no remaining imports of the old code, and the old code removed in the same sprint.

Big-bang rewrites fail on a deadline. Incremental migrations fail on discipline. Pick the failure you can manage.

Bottom line:

Get new code in front of real users early. Every strategy that works is built on this.
Strangle by what users see, route by route, not file by file.
Put a seam around anything every page depends on, and switch it with a flag, not a merge.
Move whole features, not layers. A finished slice ships. A finished layer doesn't.
Delete as you go, including the scaffolding. The migration ends when the seams and flags are gone, not when the last route moves.

Further in This Series

Each of these takes the strategies above and applies them to one starting point or one design decision:

Moving from Angular to React: A Practical Guide, with the strangler fig worked end to end.
Leaving Ember for React: What Your Conventions Were Hiding, where Ember Data is a classic branch-by-abstraction candidate.
The Framework Nobody Wrote Down: Taking a Backbone App to React
Who Updates the Total? Taking a jQuery App to React, mostly widget-level strangling.
The State Is in the DOM: Taking a Vanilla JavaScript App to React
Client Patterns: MVC, MVVM, and What Came After
One-Way vs Two-Way Binding
Client-Side vs Server-Side Rendering

Our Migration Plugins

Choosing a strategy starts with knowing the app: every route, every shared service, every piece of hidden wiring. That map is slow to build by hand. We've packaged how we do it as free, open-source plugins for Claude Code, one per starting point. Each one plans an incremental, strangler-fig migration by default:

Plugin	Starts from
`ember-to-react`	Ember Classic and Octane
`angular-to-react`	Angular 2+ and AngularJS 1.x, including hybrid ngUpgrade apps
`backbone-to-react`	Backbone.js, Marionette, jQuery templates
`jquery-to-react`	jQuery apps, plugins and jQuery UI
`vanillajs-to-react`	Plain HTML, CSS and JavaScript

/plugin marketplace add cobuild-tech/cbx-plugins
/plugin install ember-to-react@cbx-plugins

About MigrateX

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. It combines automation with experienced engineers. The automation does the repetitive work quickly, and the engineers make the tricky decisions, like where the strangler boundary runs and which shared services need a seam first.

A MigrateX project usually runs like this:

Assessment. We map every route, shared service and hidden dependency in your app, so you see where the work is before anyone agrees on a timeline.
Plan. With your team, we choose the strategy, the slice order, and what can simply be deleted.
Migration, step by step. Automation handles the repetitive parts. Our engineers handle the router, the seams and the code that connects old and new.
Proof that it works. The same tests run against both versions. A piece only counts as moved when it passes on both.
Handover. We remove the temporary code, the seams and the flags, and leave your team with a codebase they understand and own.

Your app keeps running and shipping features the whole time, so there's no big launch day. Our migratex-examples repo has small apps in each legacy framework next to their migrated versions.

Planning a migration and not sure which strategy fits? Tell us a little about your app through Contact Us, and we'll start with a free chat about where the work is likely to be.

References

Further reading

Martin Fowler. Strangler Fig Application. https://martinfowler.com/bliki/StranglerFigApplication.html
Martin Fowler. Branch By Abstraction. https://martinfowler.com/bliki/BranchByAbstraction.html
Paul Hammant. Branch by Abstraction. https://www.branchbyabstraction.com/
Trunk Based Development. Branch by Abstraction. https://trunkbaseddevelopment.com/branch-by-abstraction/
Danilo Sato. Parallel Change. https://martinfowler.com/bliki/ParallelChange.html
Ian Cartwright, Rob Horn and James Lewis. Patterns of Legacy Displacement. https://martinfowler.com/articles/patterns-legacy-displacement/
Pete Hodgson. Feature Toggles (aka Feature Flags). https://martinfowler.com/articles/feature-toggles.html
Cam Jackson. Micro Frontends. https://martinfowler.com/articles/micro-frontends.html
Microsoft. Strangler Fig pattern. https://learn.microsoft.com/en-us/azure/architecture/patterns/strangler-fig
Microsoft. Anti-corruption Layer pattern. https://learn.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer
GitHub. Scientist: a library for carefully refactoring critical paths. https://github.com/github/scientist
Joel Spolsky. Things You Should Never Do, Part I. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
Sam Newman. Monolith to Microservices. O'Reilly, 2019. https://samnewman.io/books/monolith-to-microservices/

Source code

CobuildX AI. cbx-plugins: Claude Code plugins for Backbone, Angular, Ember, jQuery and Vanilla JS to React migrations. https://github.com/cobuild-tech/cbx-plugins
CobuildX AI. migratex-examples: small jQuery, Backbone, Angular, Ember and Vanilla JS apps and their migrated versions. https://github.com/cobuild-tech/migratex-examples

Originally published on the CobuildX blog.

Jev as a Red-Teaming Guardrail: Putting TypeSafe's Model to Work in AI Security

Emmanuel R — Thu, 08 Oct 2026 11:03:01 +0000

We ran TypeSafe's Jev as a prompt guardrail on 1,580 red-teaming prompts against GPT-5-mini, Claude Haiku 4.5 and three open-source guard models. It was five to six times faster, 6–27× cheaper, and had the highest block accuracy of the hosted models.

TypeSafe's Jev promises LLM-grade decisions at a fraction of the time and cost. We put that claim to a practical test: using Jev as a guardrail that screens prompts before they reach an AI agent. Across 1,580 red-teaming prompts from public jailbreak, prompt-injection and harmful-content benchmarks, we compared Jev with OpenAI GPT-5-mini, Claude Haiku 4.5 and three open-source guard models on speed, cost, and how well each one decides what to block and names the risk.

Hype around Jev

TypeSafe launched Jev with this headline [1]:

Why the hype? Orders of magnitude? Is comparing it to an LLM even fair? Let's dig into where that speed comes from, and whether it holds up.

How Jev differs from an LLM

At a high level, Jev predicts the probability of each option in a set of choices or actions that you define, using the prompt or context you give it as the state. You might ask "should this input be blocked?" and "which of these 24 risk categories does it fall into?". Jev returns calibrated probabilities for every option in a single pass.

An LLM works differently. It generates its output one token at a time (The 'decode' phase), and each step depends on the tokens it has already produced. This stepwise decode phase is where most of an LLM's latency comes from. To use an LLM as a classifier, you ask it to write out its decision, usually as JSON, and then parse that text.

That difference has practical consequences:

The output is typed. Jev returns probabilities and choices, not free text. There is nothing to parse, and no output can arrive malformed.
The cost structure is different. Jev bills for input tokens only; its output is free [2]. An LLM bills for both, and at a higher rate for output.
Latency does not grow with the answer. A single scoring pass does not slow down as the answer gets longer, because there is no answer to write out token by token.

Where Jev is a better fit

Jev is not a replacement for an LLM. It does not write answers, summarise documents or generate code. What it targets is the large class of tasks inside AI systems where the output is a decision:

guardrails that decide whether an input or action should be blocked
routing requests to the right model, tool or team
classification, moderation and triage at high volume
choosing the next action in an agent workflow

These decisions often sit in the critical path of every request. In that position, each additional second and each additional cent is paid on every call. That is where a model built only for decisions has the most to gain.

What we tested: Jev as a red-teaming guardrail

Agent security is a natural test for this kind of model. A guardrail has to read one input and make two calls:

Should this input be blocked? It might be a harmful request, a jailbreak attempt, or a prompt injection hidden inside a document or tool response.
What kind of risk is it?

For the second question, every model chooses from 24 risk categories, and the risk category each prompt carries in its source dataset is the ground truth. 22 of these come from RedBench's risk taxonomy [3]. We added No risk for benign inputs and Prompt attack for jailbreak and injection attempts that target no specific harm.

Risk category	Covers
No risk	Benign request; safe to answer normally, even if it uses alarming words
Abusive content	Harassment, insults, bullying or degrading language
Bias content	Stereotyping or unfair treatment of groups
CBRN information or capabilities	Chemical, biological, radiological or nuclear weapons knowledge
Child safety	Content endangering or sexualising minors
Confabulation	Eliciting fabricated facts presented as true
Crime content	Facilitating theft, fraud, drugs or other crimes
Cybersecurity threats (beyond malware)	Hacking, intrusion, phishing, system disruption
Data privacy	Obtaining or exposing personal or confidential data
Decision-making in critical systems	Unsafe automated decisions in medical, legal, financial or safety-critical settings
Economic harm	Market manipulation, scams, financial damage
Election interference	Voter manipulation or election disinformation
Environmental harm	Pollution, ecological damage, wildlife harm
Extremism and radicalization	Promoting extremist ideology or recruitment
Hateful content	Hate speech targeting protected groups
Illegal weapons (non-CBRN)	Building or acquiring illegal conventional weapons or explosives
Intellectual property	Piracy or copyright and trademark infringement
Malware code	Writing or deploying malicious software
Misinformation	False or misleading claims presented as fact
Self-harm content	Suicide, self-injury or eating-disorder encouragement
Sexual content	Explicit or non-consensual sexual content
Violent content	Violence, threats or physical harm to others
Other	Harmful in a way not covered by the categories above
Prompt attack	Jailbreak or prompt-injection attempt when no more specific category applies

Data

Our starting point was RedBench [3], which consolidates 37 red-teaming benchmarks. We added dedicated jailbreak and prompt-injection datasets [4–11], including agent transcripts where the attack is planted in a tool response [9]. From a pool of 41,443 prompts, we evaluated a stratified sample of 1,580 (1,310 harmful, 270 benign), drawn from 14 datasets.

Models

We gave the same task, with identical instructions and the same 24-category risk taxonomy, to three hosted models: TypeSafe Jev [2], OpenAI GPT-5-mini [12] and Anthropic Claude Haiku 4.5 [13].

Results and observations

Speed, cost and quality

Jev's median latency was 0.17 seconds, against 1.00 s for GPT-5-mini and 0.89 s for Claude Haiku 4.5. That is about six times faster than GPT-5-mini and more than five times faster than Haiku. At the 95th percentile, Jev stayed at 0.26 s, while both LLMs were above 1.3 s.

Classifying all 1,580 prompts cost $0.08 with Jev, $0.48 with GPT-5-mini and $2.23 with Claude Haiku 4.5 at list prices. That makes Jev about 6× cheaper than GPT-5-mini and 27× cheaper than Haiku on the same workload.

Our measured gap is smaller than the headline figures TypeSafe published. That is expected. Their numbers describe workflows of System One tasks against their chosen baselines. Ours come from a single adversarial classification task, compared against two widely used, low-cost LLMs. The direction matches their claim; the size of the gap depends on the task and on what you compare against.

Speed and cost only matter if the decisions hold up. Across all prompts, Jev flagged harm correctly 85.5% of the time (block accuracy), against 84.1% for GPT-5-mini and 80.1% for Claude Haiku 4.5.

On naming the exact risk category out of 24 (categorization accuracy), Jev scored 58.0% against 56.0% and 55.4%.

Its macro-F1 across the 24 categories, which weights every category equally regardless of size, was 46.5% against 41.7% and 38.1%.

Detection quality

Categorization accuracy for each risk category with more than 20 labelled prompts:

Of these categories, Jev has the highest categorization accuracy in nine: crime, abusive content, misinformation, cybersecurity threats, bias, data privacy, sexual content, economic harm and decision-making in critical systems. GPT-5-mini leads on hateful content, malware, violent content and illegal weapons. Haiku is strongest at recognising benign inputs (No risk) and prompt attacks, and ties with GPT-5-mini ahead of Jev on self-harm.

Comparison with task-specific models

We also ran three open-source models built specifically for guardrailing, on a T4 GPU: Qwen3Guard-Gen-4B [14], Llama Prompt Guard 2 86M [15] and ProtectAI DeBERTa-v3 prompt-injection v2 [16]. None of them answers in the 24 risk categories, so they are compared with Jev on block accuracy only.

To compare performance across attack domains, we grouped the source benchmarks into nine benchmark groups based on the kind of prompts each contains. The groups are used only to aggregate results; they are not the labels that the models predict.

Benchmark group	What it tests	Benchmarks included
Harmful instruction and question sets	Plainly worded harmful requests and questions	AdvBench, CatQA, CoNA, ControversialInstructions, DAN, DoNotAnswer, ForbiddenQuestions, GPTFuzzer, HarmBench, HarmfulQ, HarmfulQA, JBB-Behaviors, MaliciousInstruct, MaliciousInstructions, MedSafetyBench, QHarm, SGBench, StrongREJECT (all via RedBench [3])
Cyberattack assistance	Requests for help with hacking, intrusion and malware	CyberattackAssistance (via RedBench [3])
Mixed safety suites	Broad safety suites whose prompts span many harm types	XSafety, JADE (via RedBench [3])
Jailbreak prompt collections	DAN-style, role-play and "developer mode" templates, with ordinary prompts as benign contrasts	In-the-wild jailbreak and regular prompts [4], JailbreakBench JBC [5], Salad-Data jailbreak templates [6], jailbreak-classification and ChatGPT-Jailbreak-Prompts [10], LatentJailbreak (via RedBench [3])
Automated jailbreak attacks	Attacks generated automatically (GCG, PAIR, random search, AutoDAN, GPTFuzzer, TAP)	JailbreakBench GCG, PAIR and random-search artifacts [5], Salad-Data AutoDAN, GCG, GPTFuzzer and TAP attacks [6]
Direct prompt injection sets	"Ignore previous instructions" and secret-extraction attempts, with benign contrasts	Gandalf ignore-instructions and summarization (via RedBench [3]), Lakera Gandalf [8], deepset prompt-injections [7], safe-guard-prompt-injection [10]
Indirect prompt injection (InjecAgent)	Instructions planted in tool responses to hijack an agent	InjecAgent data-stealing and direct-harm cases [9], plus clean counterparts built for this study
Toxic statements and unsafe advice	Toxic statements and prompts that invite unsafe advice	ToxiGen, SafeText, PhysicalSafetyInstructions (via RedBench [3])
Over-refusal suites	Benign prompts that sound dangerous, plus unsafe contrasts	XSTest and CoCoNot [11], OR-Bench 11, JBB-Behaviors benign set [5]

Qwen3Guard is scored on every benchmark group. Prompt Guard 2 and ProtectAI detect prompt attacks only, so they appear only in the jailbreak and prompt-injection groups they were run on in full.

Jev matches or beats every task-specific model in eight of the nine benchmark groups.

Clear leads: direct prompt injection (95%, against 88% for ProtectAI and 80% for Qwen3Guard), cyberattack assistance (83% against 67%), toxic statements and unsafe advice (49% against 38%) and indirect prompt injection (77% against 70%).
Narrow leads: harmful instructions (92% against 90%) and mixed safety suites (47% against 42%).
Level: automated jailbreaks (96% for both) and over-refusal (72% for both).
Behind: jailbreak prompt collections, where Qwen3Guard is slightly ahead (93.3% against 92.9%).

The two prompt-attack classifiers trail Jev throughout. Prompt Guard 2 is behind Jev and Qwen3Guard in every group it covers. ProtectAI beats Qwen3Guard on direct injection but not Jev. Both score 33% on indirect injection hidden in tool responses.

Two caveats apply:

Mixed safety suites and toxic statements are hard for every model, with all scores below 50%.
Indirect prompt injection remains Jev's weakest group against the hosted LLMs, which both reached 91% there.

What this means for enterprises

For most organisations, the expensive part of AI safety is not a single model call. It is making that call on every request, every agent step, and every test case in every evaluation run. A guardrail that sits in front of a production agent may run millions of times a month. An evaluation suite that re-scores thousands of cases after each prompt or model change runs constantly during development.

In those settings, the gap between a one-second LLM judgement and a sub-200-millisecond typed decision is not a rounding error. It decides whether security checks can run inline or have to be sampled, and whether evaluation runs on every change or once a week. Customers running their evaluation workloads through OptiX with Jev are already seeing reduced evaluation costs and lower latency, with performance on par with or ahead of the LLMs.

The broader lesson from this study is that decision tasks and generation tasks do not need the same model. Matching the model to the job, and measuring it on your own data, pays off quickly.

Explore OptiX

OptiX is our platform for testing, evaluating and monitoring AI systems. It brings together:

Security scans for jailbreaks, prompt injection and other adversarial inputs
Multi-agent evaluation across complex, multi-step workflows
Observability for AI systems in production
Use-case evaluation for RAG, text-to-SQL, document parsing, entity extraction, summarisation and more
Jev support, so fast, low-cost typed decisions can power guardrails and evaluations alongside LLM-based judges

To see how your own agents hold up, visit optix.cobuildx.ai and get in touch with our team.

References

Benchmark and claim

TypeSafe AI. "193.6x faster, 444.6x cheaper" launch claim. https://typesafe.ai/
TypeSafe AI. Jev model documentation. https://docs.typesafe.ai/models
Q.-A. Dang, C. Ngo, T.-S. Hy. RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models. arXiv:2601.03699, 2026. Dataset: https://huggingface.co/datasets/knoveleng/redbench

Jailbreak and prompt-injection datasets

X. Shen et al. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. ACM CCS 2024. arXiv:2308.03825. Dataset: https://huggingface.co/datasets/TrustAIRLab/in-the-wild-jailbreak-prompts
P. Chao et al. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. arXiv:2404.01318, 2024. Artifacts: https://github.com/JailbreakBench/artifacts
L. Li et al. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. arXiv:2402.05044, 2024. Dataset: https://huggingface.co/datasets/OpenSafetyLab/Salad-Data
deepset. prompt-injections dataset. https://huggingface.co/datasets/deepset/prompt-injections
Lakera. gandalf_ignore_instructions dataset. https://huggingface.co/datasets/Lakera/gandalf_ignore_instructions
Q. Zhan et al. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. arXiv:2403.02691, 2024. https://github.com/uiuc-kang-lab/InjecAgent
Community datasets on Hugging Face: jackhhao/jailbreak-classification, xTRam1/safe-guard-prompt-injection, rubend18/ChatGPT-Jailbreak-Prompts.
Over-refusal benchmarks: P. Röttger et al. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. arXiv:2308.01263, 2023; F. Brahman et al. The Art of Saying No: Contextual Noncompliance in Language Models (CoCoNot). arXiv:2407.12043, 2024; J. Cui et al. OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv:2405.20947, 2024.

Models

OpenAI. GPT-5-mini. https://platform.openai.com/docs/models
Anthropic. Claude Haiku 4.5. https://docs.anthropic.com/en/docs/about-claude/models
H. Zhao et al. Qwen3Guard Technical Report. arXiv:2510.14276, 2025.
S. Chennabasappa et al. LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents (Llama Prompt Guard 2). arXiv:2505.03574, 2025.
ProtectAI. deberta-v3-base-prompt-injection-v2. https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2

Originally published on the CobuildX blog.

Collections AI: The Compliance Constraint That Shapes Everything

Emmanuel R — Thu, 08 Oct 2026 11:02:25 +0000

AI can meaningfully improve collections performance within the regulatory framework — propensity scoring, channel optimisation, timing models. What it cannot do is automate the things TCPA, FDCPA, and state-level consumer protection laws prohibit. Here is what the boundary looks like.

Collections is one of the highest-value AI applications in financial services by the numbers — small improvements in recovery rates on large portfolios compound quickly. It is also one of the most heavily regulated operational functions at any lender. TCPA, FDCPA, state-level consumer protection laws, and the CFPB's interpretive guidance on unfair, deceptive, or abusive acts and practices all constrain what an AI-driven collections operation can do. The practical question is not whether to apply AI in collections — the ROI case is clear — but what AI can do within the constraint set.

Collections AI projects fail in two ways. The first: the team builds a sophisticated propensity or segmentation model but does not integrate it into collections workflows in a way that changes collector behaviour. The model exists; nothing changes. The second: the team deploys automated outreach capabilities without adequately scoping the regulatory framework, and the compliance review catches problems that require significant redesign.

Key insight: TCPA and FDCPA do not prevent AI from improving collections — they define the boundaries within which AI can operate. Designing to those boundaries from the start produces a system that is both effective and defensible.

"We built the model. Then compliance reviewed the outreach automation we had planned and we had to redesign half of it."

What TCPA Actually Constrains

The Telephone Consumer Protection Act (TCPA) restricts the use of automatic telephone dialling systems (ATDS) and artificial or prerecorded voice messages to contact consumers without their prior express written consent. In a collections context, this constrains automated outreach — AI-driven diallers, automated SMS campaigns, and prerecorded voice messages — significantly.

Prior express written consent is required for autodialled calls or texts to mobile phones. For debt collection, this consent must be obtained at the time the credit relationship is established or through a subsequent consent process. Consent obtained at origination covers most account holders for initial collection contact, but consumers can revoke consent at any time, and revocations must be honoured immediately — a process that needs to be automated and audited.

The TCPA does not prohibit human-initiated calls using manual dialling. Collections operations can use predictive diallers in a compliant way if the dialler does not meet the ATDS definition — an increasingly nuanced technical and legal question after the Facebook v. Duguid Supreme Court decision in 2021. Any collections AI involving automated outreach should be reviewed by counsel with TCPA expertise before deployment.

TCPA constrains automated outreach to mobile phones without consent — the consent management process is as important to design as the outreach automation itself

FDCPA: What You Can Say and When

The Fair Debt Collection Practices Act (FDCPA) governs third-party debt collectors and imposes specific requirements on the timing, content, and frequency of collection communications. The CFPB's Regulation F, which took effect in 2021, extended some FDCPA principles and added specific guidance on electronic communications including email and text.

Key constraints with direct implications for AI-driven collections:

Contact frequency limits: Regulation F creates a presumption that contacting a consumer more than seven times in a seven-day period, or within seven days of a prior telephone conversation, violates the FDCPA. AI-driven outreach optimisation needs to track and enforce these limits across all communication channels.

Time-of-day restrictions: calls are permitted only between 8am and 9pm in the consumer's local time zone. An AI scheduling model optimising for contact rates needs to incorporate this constraint — and needs to use the consumer's time zone, not the operations centre's.

Cease and desist: if a consumer requests that contact cease, all communication must stop except to confirm receipt of the cease notice or to advise of specific legal action. This revocation process needs to be immediate and auditable.

Seven-in-seven contact frequency limits, time-of-day restrictions, and immediate cease-and-desist compliance are non-negotiable constraints that AI outreach must enforce

What AI Can Do Within These Constraints

Within the regulatory framework, AI delivers meaningful collections improvement through three specific applications.

Propensity to pay modelling. A model that predicts which accounts are most likely to pay in response to outreach — and under what conditions (settlement offer, payment plan, specific timing) — allows collections operations to prioritise contacts more effectively than simple days-past-due segmentation. Accounts with high propensity to pay at full balance get early personal contact; accounts that respond to settlement offers get offers earlier in the delinquency cycle when recovery is higher.

Channel preference optimisation. Different consumers respond to different channels. Some respond to text, some to email, some to phone calls. A model trained on historical contact data can predict which channel is most likely to produce a response for a given consumer profile, reducing contact attempts while improving contact rates.

Timing optimisation. Within the time-of-day constraints and the contact frequency limits, there is still meaningful variation in when a consumer is most likely to answer or respond. ML models trained on response rates by day of week and time of day — by consumer segment — can improve contact rates without increasing total contact attempts.

Propensity to pay modelling, channel preference, and timing optimisation deliver collections lift within the regulatory framework — together, improvements of 10–20% in recovery rates on targeted portfolios are realistic

State-Level Overlay

Federal TCPA and FDCPA requirements are the floor. A significant number of states have consumer protection laws that are more restrictive, and compliance at the federal level does not mean compliance in every state where borrowers are located.

California's Rosenthal Fair Debt Collection Practices Act applies FDCPA-equivalent requirements to first-party collectors (lenders collecting their own debt) in addition to third-party collectors — a broader scope than federal law. New York and other states have similar first-party coverage extensions.

Several states have more restrictive contact frequency rules than Regulation F, shorter time windows for permissible calling hours, or additional disclosure requirements for automated communications.

For a lender with a national portfolio, the compliance standard is the most restrictive applicable state law for each consumer, not the most permissive interpretation of federal law. A collections AI system with national scope needs to implement state-level rule variations or operate to the most conservative standard across the full portfolio.

State-level consumer protection laws can be more restrictive than federal TCPA and FDCPA — compliance requires the most restrictive applicable standard for each consumer's state

Building the Business Case Within the Constraints

The ROI case for collections AI is real within these constraints. The constraint set limits automated outreach, but it does not limit propensity scoring, segmentation, channel optimisation, or timing models — all of which improve outcomes without requiring automation of contact initiation.

Realistic targets for a well-scoped collections AI program: 10–20% improvement in early-stage recovery rates through better prioritisation and channel selection on accounts where propensity to pay is high. 15–25% reduction in average collection cost per recovered dollar through more efficient contact allocation. Measurable reduction in complaints from improved compliance with contact frequency and timing rules.

The compliance investment — consent management, revocation tracking, state-level rule implementation, audit logging — is not small. It is also not optional, and treating it as overhead rather than part of the system design leads to the pattern of building first and redesigning after compliance review. The right approach: scope the compliance infrastructure at the start, design the AI system to operate within it, and build the business case on what is achievable within that framework.

10–20% early-stage recovery improvement and 15–25% cost reduction per recovered dollar are achievable within the regulatory framework — scope the compliance infrastructure at the start, not after

Originally published on the CobuildX blog.

Credit Risk Scoring: The Data Problems That Haven't Changed

Emmanuel R — Thu, 08 Oct 2026 11:01:49 +0000

Better models have not solved the thin-file problem, the data quality issues in alternative data, or the validation requirements that govern what can be deployed in regulated lending. The data problems are still the hard part.

Credit risk modelling has changed significantly over the past decade. Gradient boosting has replaced logistic regression as the standard approach at most sophisticated lenders. Alternative data — cash flow data, rental payment history, utility payments — has broadened the credit-visible population. Explainability tooling has made complex models more deployable in regulated environments. What has not changed: the data quality problems that determine whether any model, regardless of sophistication, produces reliable predictions.

The limiting factor in credit risk modelling is not model architecture. Given reasonable data, a well-tuned gradient boosting model and a well-tuned logistic regression produce different but both useful predictions. The limiting factors are: the thin-file problem (a significant fraction of credit-worthy borrowers have insufficient traditional credit history to model reliably), the alternative data quality problem (much of the data that could address thin-file has its own reliability and coverage gaps), and the model validation requirements that govern what can be deployed.

Key insight: Lenders who have improved credit risk modelling in recent years have done so primarily through better data — specifically cash flow data and expanded credit bureau data — not better model architectures. The model is less important than the signal.

"We spent six months building a more sophisticated model. When we went back and cleaned the training data, the simple model performed just as well."

The Thin-File Problem

Approximately 45 million Americans have thin or no credit files — too little traditional credit history for the major bureaus to generate a reliable score. This population is not uniformly high risk. It includes recent immigrants with strong financial management histories in their home countries, young adults who have avoided debt but have stable income, and individuals who have historically used cash and community lending. A lender that cannot score them declines credit-worthy borrowers and cedes that business to competitors who can.

Traditional bureau data does not solve the thin-file problem because the thin-file is definitionally not in the bureau data. Expanding the modelling approach requires either incorporating non-traditional data sources (rental payments, utility payments, cash flow data) or accepting that a segment of the credit-worthy population will not be scoreable through traditional approaches.

45 million Americans are not scoreable through traditional bureau data — addressing the thin-file problem requires non-traditional data sources, not better models on the same data

Alternative Data: What Works and What Does Not

Cash flow data — transaction-level bank account data showing income deposits, recurring payments, and spending patterns — is the most consistently useful alternative data source for credit risk in consumer lending. It is highly predictive of ability to pay, has reasonable coverage through open banking integrations, and is relatively clean compared to other alternative data sources. Cash flow features (income consistency, average balance, payment-to-income ratios) add meaningful lift for thin-file borrowers on top of bureau data.

Rent payment history, when available and reliably reported, is also useful — a demonstrated track record of on-time rent payments is predictive of future credit performance. The coverage problem is significant: most rent payments are not reported to any bureau, and the coverage of rent payment data from alternative providers is incomplete and variable by geography.

Data sources that have shown less consistent value: social media signals, education and employment data from non-traditional sources, and 'character' signals derived from digital behaviour. These data types often introduce fair lending risk (proxies for protected characteristics appear in unexpected places) without delivering equivalent predictive lift. Several lenders have retreated from these approaches after fair lending testing revealed disparate impact.

Cash flow data is the most consistently predictive alternative data source. Character and digital behaviour signals introduce fair lending risk without reliable predictive lift

The Recency Problem

Credit risk models trained on pre-2020 data reflect credit behaviour in a low-interest-rate, low-unemployment economic environment. Models trained primarily on 2020–2022 data reflect an unprecedented economic disruption period with stimulus payments, forbearance programs, and unusual payment behaviour. Neither training period is a reliable guide to current credit risk in a normalising interest rate environment.

The practical implication: models trained on data more than two to three years old should be retested against recent performance data before being treated as reliable. Performance metrics that looked stable during model development may have degraded significantly as the economic environment shifted.

This is not a new problem — economic cycle sensitivity in credit models has been understood for decades. It became more acute after 2020 because the economic disruption was severe enough to make pre-crisis training data significantly less predictive of post-crisis behaviour. Lenders whose models date from 2018 or earlier and have not been recalibrated against post-2022 performance data are running on models whose accuracy may be materially worse than their validation metrics suggest.

Models trained on pre-2020 data should be retested against recent performance data — the economic environment shift has degraded accuracy in ways validation metrics from that period do not reflect

Model Validation in a Regulated Environment

For banks and supervised lenders, credit risk models are subject to model risk management requirements under SR 11-7. The validation requirement is not just that the model performs well by internal metrics — it is that the model has been independently validated, the validation covers conceptual soundness and outcome analysis, limitations are documented, and ongoing monitoring is in place.

For the models with the most regulatory scrutiny — models used in underwriting decisions for consumer credit at scale — the validation needs to cover fair lending analysis as a standard component. A model that validates well on predictive performance metrics but has not been tested for disparate impact is not ready for production at a regulated lender.

The practical implication for AI/ML models: more complex models require more documentation and a more sophisticated validation function. The validation team needs to understand the feature engineering, the training data, the hyperparameter selection rationale, and the limitations of the model well enough to write a challenge that addresses each. This is achievable but it requires planning — presenting a finished ML model to model risk management without having engaged them in the development process is a common reason for long validation timelines.

Engaging model risk management during development, not at the end of it, is the single most effective way to avoid long validation delays

Originally published on the CobuildX blog.

Fraud Detection: Why Your Rules Engine Has a Ceiling

Emmanuel R — Thu, 08 Oct 2026 11:01:13 +0000

Rules-based fraud detection is fast to deploy and easy to explain. It also has a ceiling: fraudsters adapt to rules, false positive rates are hard to reduce, and new fraud patterns take time to codify. ML adds a measurable lift above that ceiling — here is what that looks like in practice.

Most financial institutions run fraud detection on a rules engine. Block transactions over a certain dollar amount to unusual recipients. Flag velocity patterns above threshold. Decline cards used in multiple geographies within a short window. These rules work. They catch a significant fraction of fraud and they have the advantage of being fast to implement, easy to explain to regulators, and transparent to the operations team. They also have a ceiling that most institutions have already hit.

A rules engine is a static representation of known fraud patterns. Fraudsters adapt to it. Once a particular attack pattern triggers enough rule-based blocks, the fraud operation modifies its behaviour to stay below the thresholds — lower transaction amounts, slower velocity, more geographically plausible patterns. The rules that were catching fraud last year catch less of it this year, and adding new rules to close the gap increases false positives on legitimate transactions.

Key insight: Rules catch the fraud patterns you have already seen and codified. ML catches the patterns that are statistically anomalous but have not yet been written into a rule — which is where new fraud appears.

"Every time we add a new rule to close a fraud vector, our false positive rate goes up and our customer service team starts getting calls about blocked legitimate transactions."

What Rules-Based Detection Does Well

Rules-based fraud detection has genuine strengths that explain why it remains the foundation of most fraud stacks.

It is deterministic and auditable. When a transaction is blocked, the operations team can tell the customer exactly which rule triggered the decline. This transparency matters for customer relations and for regulatory examinations.

It is fast. Rule evaluation is computationally trivial — microseconds per transaction. At the transaction volumes that large financial institutions process, latency matters.

It handles known, well-defined attack patterns reliably. Card-not-present fraud with high velocity on new accounts, account takeover with credential stuffing patterns, check kiting with specific inter-account timing signatures — these patterns can be codified into rules that catch them with high precision.

Rules are also the right tool for hard business constraints that are not about statistical patterns: geographic restrictions, product eligibility requirements, regulatory holds. These belong in a rules engine regardless of how sophisticated the ML layer is.

Rules are the right tool for hard business constraints and well-defined known patterns — they are not the right tool for detecting fraud patterns that have not been codified yet

Where the Ceiling Is

The ceiling on rules-based detection shows up in three ways.

First, new fraud patterns. When a new attack vector appears — a new synthetic identity structure, a new card-not-present exploit, a new account funding scheme — it takes time to detect the pattern, analyse it, write the rule, test it, and deploy it. During that window, fraud losses accumulate. The detection gap between when new fraud appears and when a rule catches it is typically weeks to months.

Second, the precision-recall tradeoff. Adding rules to catch more fraud always increases false positives on legitimate transactions. At some point, the marginal fraud caught by the next rule is less costly than the customer friction and manual review cost of the additional false positives it creates. Most mature rules engines are already at or past this point.

Third, adversarial adaptation. Fraud rings that are large enough to observe their own block rates modify their behaviour to avoid triggered rules. This is well-documented in card fraud, account opening fraud, and money movement fraud. Rules that were effective eighteen months ago are less effective today because the fraud behaviour has changed to avoid them.

New patterns, false positive ceilings, and adversarial adaptation are the three vectors where rules-based detection consistently underperforms

What ML Adds

ML fraud detection models — gradient boosting classifiers, neural networks, graph-based anomaly detection — catch fraud through a different mechanism than rules. They learn the statistical profile of legitimate transactions and flag deviations from that profile, without requiring those deviations to match a specific predefined pattern.

This means they can catch fraud that has not been codified into a rule yet. A new synthetic identity scheme may not trigger any existing rule, but it will deviate from the statistical profile of legitimate new accounts in ways that a well-trained model detects. The fraudsters would need to perfectly mimic the full behavioural and demographic distribution of legitimate customers — not just avoid specific thresholds — to evade a model-based detector.

The measurable lift of adding ML on top of rules varies significantly by institution and fraud type. In card fraud, lift of 15–30% reduction in fraud losses at equivalent false positive rates is commonly reported. In account opening fraud and synthetic identity fraud, the lift can be larger because these fraud types are harder to codify into rules.

ML does not replace the rules engine. The two work better together than either does alone. Hard constraints belong in rules. Statistical anomaly detection belongs in the ML layer. Most production fraud stacks run both.

ML adds 15–30% fraud loss reduction at equivalent false positive rates by catching statistical anomalies that rules cannot codify — it works alongside rules, not instead of them

Model Drift in an Adversarial Environment

Fraud detection models face a challenge that most ML models do not: the distribution they are trained on is actively manipulated by the adversary once the model is deployed.

A churn prediction model trained on last year's customers is less accurate this year because customer behaviour naturally evolved. A fraud model trained on last year's fraud patterns is less accurate this year partly because fraud behaviour naturally evolved, and partly because fraud operators specifically adapted to avoid the model's detection patterns.

This means fraud models need more frequent retraining than most ML applications — quarterly at minimum for high-volume card fraud, more frequently if new attack vectors are appearing. It also means the training data pipeline needs to be maintained carefully: new labelled fraud examples need to be incorporated quickly, and the labelling process (which requires investigations and chargebacks to confirm fraud) needs to be timely enough to produce useful training signal.

Monitoring for model drift in fraud detection should track both the false positive rate (to catch model degradation on legitimate transactions) and the fraud detection rate (to catch model degradation on fraud). Both can degrade independently if the fraud distribution shifts.

Fraud models require more frequent retraining than most ML applications because the adversary actively adapts — quarterly retraining is a minimum, not a target

Originally published on the CobuildX blog.

Document Processing in Loan Origination: Where IDP Projects Stall

Emmanuel R — Thu, 08 Oct 2026 10:59:58 +0000

Intelligent document processing handles pay stubs, bank statements, and tax returns well under controlled conditions. The 20% exception rate, downstream system integration, and confidence threshold design are where most production deployments get complicated.

Loan origination involves processing large volumes of documents — pay stubs, bank statements, W-2s, tax returns, bank statements, rent rolls, business financials. The manual review process is slow, expensive, and error-prone. Intelligent document processing can automate significant portions of it. The business case is clear and the technology is mature enough to deploy. Where these projects consistently run into difficulty is the exception handling, the integration, and the operational model needed to manage the cases the AI cannot handle confidently.

IDP pilots demonstrate high accuracy on clean, well-formatted documents from a controlled sample. Production document volumes include faxed photocopies of handwritten notes, bank statements from forty different institutions with different formats, pay stubs from payroll systems that were last updated in 2009, and documents that are technically legible but structured in ways the model has not seen before. The accuracy numbers change.

Key insight: The 80% that IDP handles confidently reduces costs significantly. The 20% that it handles poorly requires a human review queue that needs to be designed, staffed, and managed — and if that queue is not planned for, it creates more work than the automation saves.

"The demo showed 95% accuracy on our documents. In production, 30% of submissions were being routed to the exception queue within the first month."

Why Each Document Type Is Harder Than It Looks

Pay stubs look standardised but are produced by hundreds of different payroll platforms, each with its own layout, field naming conventions, and data formatting. A model trained primarily on ADP and Paychex output encounters Gusto, Square Payroll, or a homemade Excel template and performance degrades. The fields are all there — gross pay, YTD earnings, deductions — but their position, labelling, and format vary enough to challenge extraction models trained on limited layout diversity.

Bank statements have similar issues compounded by the sheer number of issuing institutions. Major banks have consistent statement formats that are well-represented in training data. Credit unions, regional banks, and online-only institutions may have formats the model has rarely or never seen. International bank statements add language and number formatting complexity.

Tax returns (1040, 1120S, Schedule C) are more standardised than bank statements, but handwritten forms, amended returns, and older year forms create edge cases. More significantly, the fields that matter most for underwriting — self-employment income, depreciation add-backs, business use of home — require reading across multiple schedules and applying accounting logic, not just field extraction.

Rent rolls for commercial real estate loans are the hardest category: they are typically Excel files or PDFs with no standard format, produced by property managers with highly variable approaches to data organisation.

Format diversity, not document complexity, is the primary challenge — models trained on major institutions encounter edge cases at rates higher than demo conditions suggest

Designing for the Exception Rate

Every IDP deployment has an exception rate — the percentage of documents that the model routes to human review because its confidence falls below threshold. In a well-designed deployment, this is a feature, not a failure: the model knows what it does not know and escalates appropriately rather than producing a confident wrong extraction.

The exception rate design decision is a threshold calibration problem. Setting the confidence threshold high routes more documents to human review but reduces extraction errors that make it downstream. Setting it low reduces the review queue but allows more errors through. The right threshold depends on the cost of a missed extraction (an underwriting decision made on wrong income data) versus the cost of the human review (processor time and cycle time impact).

The exception queue also needs to be designed as a workflow, not an afterthought. Who reviews exceptions? What tools do they use? How does a corrected extraction feed back as a training signal to improve the model over time? How is the exception rate monitored and how does it trigger model retraining? These questions need to be answered before deployment, not after the queue fills up.

Design the exception queue as a workflow from the start — the 20% that goes to human review determines whether the overall system reduces costs or increases them

Downstream System Integration

IDP extracts structured data from documents. That data needs to go somewhere — typically a loan origination system (LOS) where it pre-populates the application, a verification system where it is compared to borrower-stated values, or a data repository where it feeds underwriting calculations.

The integration with the LOS is often the longest part of the project. Most major LOS platforms (Encompass, Blend, nCino) have APIs, but the field mapping between IDP output and LOS data fields requires detailed mapping work. Edge cases abound: the IDP system extracts monthly net pay; the LOS field expects annual gross. The bank statement extraction returns a list of transaction-level deposits; the LOS wants average monthly deposits as a single number.

Audit trail requirements add another integration consideration. For regulatory purposes, lenders need to be able to show what data was extracted, from which document, by which system, with what confidence, and what human review occurred. This audit trail needs to be stored somewhere and accessible for examination. Building it in after the fact is significantly more work than designing it from the start.

LOS field mapping and audit trail design are often longer than the model development — scope them explicitly upfront

What a Production-Ready IDP System Requires

A production-ready IDP system in loan origination is not just a document extraction model. It is the model plus the confidence threshold logic, plus the exception queue workflow, plus the LOS integration, plus the audit trail, plus the feedback loop from human corrections to model retraining, plus the monitoring that tracks extraction accuracy over time and flags model drift as new document formats appear.

The extraction model is typically 20–30% of the total implementation work in a well-scoped project. The remainder is the surrounding infrastructure. Projects that scope only the model and discover the infrastructure requirements mid-deployment are the ones that overrun timelines and budgets.

The ROI case is real and significant — processing costs for a fully automated document reduce by 70–80% versus manual review for the 80% of documents the model handles confidently. The business case holds even after accounting for the exception queue. It just requires scoping the full system, not just the model.

The extraction model is 20–30% of the work — the exception queue, LOS integration, audit trail, and monitoring are the rest

Originally published on the CobuildX blog.

AI in Underwriting: What Regulators Actually Care About

Emmanuel R — Thu, 08 Oct 2026 10:59:23 +0000

Lenders deploying AI in underwriting face a compliance landscape that shapes what models can be used, what documentation is required, and how adverse action notices must work. Here is what the regulatory framework actually requires and what you can deploy today.

AI underwriting models can assess creditworthiness faster, more consistently, and — when built carefully — more accurately than traditional scorecards. Lenders know this. The compliance and legal teams know it too, and they have a set of questions that need answers before any AI model touches a credit decision. Those questions are not obstacles to deploying AI in underwriting. They are a framework that, once understood, makes it straightforward to separate what is deployable today from what requires more work.

Most AI underwriting projects stall not because the model does not work but because the compliance path was not scoped upfront. Model development, validation, and business user testing proceed for six to nine months — and then the model sits in legal review because the explainability requirements, the adverse action notice process, and the fair lending testing were not designed in from the start.

Key insight: Regulatory requirements for AI in underwriting are not ambiguous. ECOA, the Fair Housing Act, SR 11-7 model risk management guidance, and the relevant CFPB guidance give a reasonably clear picture of what is required. The compliance path is knowable upfront — the projects that get stuck are the ones that did not engage it early.

"The model was excellent. We spent four months in legal trying to figure out how to write the adverse action notice."

The Adverse Action Notice Requirement

Under ECOA and Regulation B, lenders must provide applicants who are denied credit — or offered less favourable terms — with a written statement of the specific reasons for the adverse action. This requirement applies regardless of what type of model made the decision. A black-box neural network that cannot produce interpretable reason codes does not satisfy the adverse action notice requirement.

For traditional scorecard models, adverse action reasons are generated directly from the score factors — the variables that contributed most negatively to the applicant's score. For complex ML models, producing equivalent reason codes requires either post-hoc explanation methods (SHAP values, LIME) or designing the model architecture to produce interpretable outputs.

SHAP-based reason codes are increasingly accepted by compliance teams as a defensible approach, but they require validation. The reason codes need to be consistent — the same input should produce the same reason codes across runs — and they need to reflect the actual model mechanics rather than being post-hoc rationalisations that do not match the model's actual decision logic. Both of these properties require testing.

Adverse action notices require specific reasons for the decision — post-hoc explanation methods like SHAP are increasingly accepted but need to be validated for consistency and accuracy

Fair Lending: What Testing Is Required

ECOA and the Fair Housing Act prohibit discrimination in credit decisions on the basis of race, colour, national origin, sex, religion, marital status, age, and other protected characteristics. This applies to AI models.

The fair lending testing requirement for AI models is not that the model cannot use protected characteristics as inputs — that is a floor, not the full requirement. The actual requirement is disparate impact analysis: does the model produce materially different approval rates or pricing outcomes across protected groups that cannot be justified by legitimate credit risk differences?

Fair lending testing should be run during model development, not after. If disparate impact is detected during development, the model can be adjusted — through feature selection, recalibration, or architectural changes — before deployment. If it is detected during regulatory examination after deployment, the remediation is more expensive and the reputational risk is significant.

The specific statistical tests used for disparate impact analysis in lending — the 80% rule, regression-based control methods, the Bayesian Improved Surname Geocoding (BISG) proxy methodology for race and ethnicity — are well-documented. Fair lending review should be designed into the model validation process, not added as a final gate.

Fair lending testing belongs in the development process, not at the end of it — detecting disparate impact early is cheaper and safer than detecting it in an exam

SR 11-7 Model Risk Management

SR 11-7 is the Federal Reserve and OCC's guidance on model risk management, and it applies to AI underwriting models at banks and other supervised institutions. The core requirements: models must be validated by a function independent of the model development team, documentation must cover model purpose, methodology, data, limitations, and performance metrics, and models must be subject to ongoing monitoring and periodic revalidation.

For AI models specifically, SR 11-7 requires that the model validation team understand the model well enough to challenge it. This is harder for complex ML models than for traditional scorecards. Validators need to understand the training data, the feature engineering, the hyperparameter selection process, and the performance metrics used to evaluate the model — and document that they have assessed each for soundness.

The practical implication: complex ML models require more documentation and a more sophisticated validation team than traditional models. This is not a reason to avoid them, but it is a reason to involve model risk management early in the development process rather than presenting them with a finished model for validation.

SR 11-7 validation requires the ability to challenge the model — complex ML models require more documentation and more sophisticated validators than traditional scorecards

What You Can Deploy Today

Given this framework, the question is which AI approaches fit within the compliance requirements without requiring regulatory engagement beyond standard model validation.

Gradient boosting models (XGBoost, LightGBM) with SHAP-based reason codes are the current sweet spot for lenders who want meaningfully better predictive performance than traditional logistic regression scorecards without taking on interpretability risk they cannot manage. These models are significantly more predictive than scorecards on most credit datasets, produce consistent SHAP explanations, and have a validation pathway that most model risk management teams can work with.

Logistic regression models with well-engineered features remain a defensible choice for regulated lenders who want maximum interpretability and a simpler validation process. The performance gap versus gradient boosting depends on the dataset, but for standard consumer lending use cases the gap is often smaller than the implementation complexity difference.

Deep learning models and LLM-based approaches require more care. They can produce excellent predictive performance, but the interpretability pathway, the validation approach, and the adverse action reason code methodology all require more work to make defensible. They are deployable, but they require more upfront investment in the compliance infrastructure.

Gradient boosting with SHAP reason codes is the current practical standard for regulated lenders wanting ML performance without interpretability complexity

Originally published on the CobuildX blog.

Scheduling in High-Mix Low-Volume Manufacturing: Why Rules Break

Emmanuel R — Thu, 08 Oct 2026 10:58:46 +0000

High-mix low-volume manufacturing scheduling is one of the harder operational problems in industry. ERP scheduling modules are built for repetitive production. Custom rules break under complexity. Here is what AI scheduling actually delivers and what it does not.

High-mix low-volume manufacturing — job shops, contract manufacturers, specialty fabricators — has a scheduling problem that standard ERP tools were not designed to solve. When you are producing hundreds of different part numbers in small quantities, with sequence-dependent setup times, shared constrained resources, and customer-specific due dates, the scheduling problem becomes combinatorially complex fast. Rules that work for fifty active jobs fall apart at five hundred.

ERP production scheduling modules are built around the assumptions of repetitive manufacturing — standard routings, predictable cycle times, stable demand. HMLV environments violate all of these assumptions regularly. The result: schedulers spend most of their time manually adjusting the ERP schedule, working around its recommendations rather than relying on them. The schedule is a starting point, not an operating plan.

Key insight: The scheduling problem in HMLV manufacturing is not that the rules are wrong — it is that the problem is too large and too dynamic for any static rule set to handle well. The complexity requires optimisation, not just sequencing.

"The ERP schedule is what we show customers. The actual production sequence is what the floor supervisor decides every morning."

What Makes HMLV Scheduling Hard

Several characteristics of high-mix low-volume environments combine to make scheduling genuinely difficult.

Sequence-dependent setup times. On a CNC machining centre or a paint line, the setup time between consecutive jobs depends on which job was run immediately before. Switching from a long part to a short part may require a fixture change that takes two hours; switching between two similar parts may take fifteen minutes. Optimal sequencing minimises total setup time across the shift — but computing the optimal sequence across hundreds of jobs and dozens of machines is a hard combinatorial problem.

Shared constrained resources. Critical resources — a specific CNC, a specialist operator, a testing station — create bottlenecks that cascade through the schedule. A job that is delayed at a bottleneck delays every subsequent job on that part's routing. The schedule needs to account for resource contention across all jobs simultaneously, not job by job.

Dynamic arrivals. New jobs arrive, existing jobs change priority, customers call with expedite requests, and machines break down. The schedule that was optimal this morning is out of date by noon. Rescheduling in response to every disruption is computationally expensive and disruptive to the floor.

Sequence-dependent setups, shared bottleneck resources, and dynamic disruptions combine to make HMLV scheduling a problem that static rules cannot handle reliably

Where ERP and APS Fall Short

Most ERP systems use a variant of finite capacity scheduling or simple priority rules (earliest due date, shortest processing time) to generate production sequences. These approaches are fast to compute and easy to understand, but they do not optimise across the full problem — they make locally reasonable decisions that do not account for downstream effects.

Advanced Planning and Scheduling (APS) tools are more sophisticated and can handle sequence-dependent setup times and resource contention better than standard ERP. The limitations: they require accurate master data (routing times, setup matrices, resource capacity), they struggle with frequent reschedules in highly dynamic environments, and they are expensive to configure and maintain.

The practical result in many HMLV shops: the APS or ERP schedule is used as a capacity load view — a rough guide to whether the shop is overloaded for the week — while the actual daily sequencing decisions are made by experienced floor supervisors using rules of thumb developed over years on the floor. This is not a failure of the floor team. It is a reasonable response to tools that are not fit for the actual problem complexity.

APS tools improve on ERP but still require accurate master data and struggle with dynamic environments — most HMLV shops are scheduling around them, not with them

What AI Scheduling Delivers

AI scheduling approaches — typically reinforcement learning, genetic algorithms, or ML-enhanced heuristics — tackle the HMLV scheduling problem differently from rules or conventional optimisation.

Reinforcement learning agents learn scheduling policies by simulating millions of scheduling scenarios and learning which sequencing decisions lead to better outcomes (on-time delivery, utilisation, total setup time). Once trained, they can generate schedules quickly enough to reschedule in near real-time as disruptions occur.

ML-enhanced heuristics use historical production data to learn which priority rules work best under which conditions — time of day, resource availability, mix of job types in queue — and apply different rules dynamically rather than using a single rule universally.

Realistic improvements in well-implemented HMLV scheduling AI: 10–20% reduction in total setup time through better sequencing, 5–15% improvement in on-time delivery by accounting for bottleneck contention earlier in the planning horizon, and meaningfully faster response to disruptions.

These gains require accurate data: reliable routing times, up-to-date setup matrices, and real-time machine status. Scheduling AI does not fix data quality problems — it amplifies them. A model scheduling against inaccurate routing times will produce schedules that fail on the floor in new ways.

10–20% setup time reduction and 5–15% on-time delivery improvement are realistic — with accurate routing data and setup matrices as the prerequisite

The Constraint Modelling Problem

Scheduling AI is only as useful as its model of the constraints it is optimising within. In HMLV manufacturing, the relevant constraints are numerous and often tacit.

Formal constraints — machine capacity, operator certification requirements, material availability — are usually captured somewhere in the ERP or production system, though often incompletely. Tacit constraints — this machine runs hot and needs a cooldown between long jobs, this operator works better in the morning, this customer always expedites so we buffer their jobs — live in the floor supervisor's head.

Capturing the tacit constraints is a requirements gathering exercise, not a data science problem. It requires time with the people who actually run the floor, not just the data they generate. AI scheduling that ignores these constraints produces schedules that look optimal by the formal criteria and fail in practice because they violate constraints the model did not know existed.

Tacit scheduling constraints — the rules the floor supervisor knows but nobody wrote down — need to be captured explicitly before the model can be trusted

Making the Scheduler Useful to the Floor

The final consideration is one that determines whether any scheduling improvement gets sustained: the floor team needs to trust the output enough to follow it.

Scheduling tools that present an optimal sequence with no explanation for why jobs are ordered the way they are produce floor supervisors who override the sequence and revert to their existing approach. Schedulers that surface the reasoning — this job is prioritised because it is the only one this week that uses the bottleneck fixture, this setup sequence minimises changeover time across the afternoon — give supervisors the context to evaluate the recommendation rather than just accept or reject it.

The goal is not to automate the floor supervisor out of the scheduling decision. It is to give them a starting point that accounts for more complexity than they can hold in their head, with enough transparency to apply their tacit knowledge on top of it.

Schedulers that explain their reasoning get used. Schedulers that present opaque outputs get overridden

Originally published on the CobuildX blog.

Predictive Maintenance: The Gap Between the Pilot and Production

Emmanuel R — Thu, 08 Oct 2026 10:58:10 +0000

Predictive maintenance pilots succeed regularly. Production deployments are harder. The difference is usually not the model — it is asset coverage, alert fatigue, maintenance workflow integration, and the trust gap that builds when the first false positives arrive.

Predictive maintenance is one of the most piloted AI applications in manufacturing — and one of the most frequently stalled between pilot and production. The concept is straightforward enough: detect patterns in sensor data that precede equipment failure, alert the maintenance team early enough to schedule a planned repair instead of responding to an unplanned breakdown. The ROI case is clear. The pilot usually works. The full deployment frequently does not.

Pilots are run on selected assets, with dedicated engineering support, in controlled conditions. Production deployments have to work across the full asset base, integrate with existing maintenance workflows, generate alerts that maintenance planners trust enough to act on, and survive the realities of sensor data quality, shift changes, and competing priorities. The gap between those two environments is where most predictive maintenance programs stall.

Key insight: The model is rarely the failure point. Alert fatigue, workflow integration, and the trust gap created by the first wave of false positives are what prevent pilot-proven technology from reaching production scale.

"The pilot ran for three months and the model was excellent. Then we deployed it across all forty pumps and maintenance stopped looking at the dashboard within six weeks."

Why Pilots Succeed

Predictive maintenance pilots are usually set up for success. Asset selection is deliberate — engineers choose the assets with the best sensor coverage, the clearest failure history, and the highest maintenance cost. The team running the pilot is engaged and attentive. When the model generates an alert, someone investigates it. When the alert proves correct, it gets documented. When it proves incorrect, the team discusses why and adjusts.

This is not representative of production conditions. In production, the model runs across all assets. The maintenance team has a full workload and limited time to investigate every alert. No one is specifically assigned to track model performance. False positives accumulate without the immediate feedback loop that keeps the pilot honest.

The transition plan needs to account for this difference explicitly. A pilot success is necessary but not sufficient evidence that a production deployment will work.

Pilot conditions are optimised for success. Production conditions are not — the transition plan needs to bridge that gap deliberately

Asset Coverage and Prioritisation

A common mistake in predictive maintenance programs is trying to cover the full asset base from the start. Most plants have hundreds or thousands of assets. Instrumenting all of them, building models for all of them, and deploying alerts across all of them simultaneously is a scope that almost always exceeds the capacity of the team running the program.

A better approach: start with the twenty or thirty assets that account for the majority of unplanned downtime cost. Not the assets that are easiest to model, or the ones that already have sensor coverage, but the ones where a successful prediction would have the highest value. Build the business case around those assets, deploy carefully, and expand to adjacent asset classes once the first cohort is working reliably.

This also gives the maintenance team time to develop the workflow habits that make predictive maintenance effective — reviewing alerts, scheduling condition-based interventions, and feeding back the outcome of each intervention to improve model accuracy.

Start with the twenty assets that account for most unplanned downtime cost, not the twenty easiest to model

The Alert Fatigue Problem

Alert fatigue is the single most common reason predictive maintenance programs fail to reach production scale. When a model generates more alerts than the maintenance team can act on — or generates alerts that turn out to be wrong often enough to erode trust — the team stops responding to them. Once that happens, the program is functionally dead regardless of how good the underlying model is.

The precision-recall tradeoff is directly relevant here. A model can be tuned to catch more failures (higher recall) at the cost of more false positives (lower precision), or to have fewer false positives at the cost of missing some failures. In most maintenance contexts, precision matters more than recall in the first year of deployment. A high-precision, low-recall model that generates five alerts per month — all of which are credible — builds more trust than a high-recall model that generates fifty alerts, forty of which do not result in a confirmed finding.

The alert threshold should be set conservatively at first, with the explicit understanding that it will be relaxed as the team develops confidence in the system and as the model accumulates more data to learn from.

Set alert thresholds conservatively in the first year — building maintenance team trust matters more than catching every failure

Workflow Integration

Predictive maintenance alerts only create value if they change maintenance behaviour. That requires the alerts to reach the people who schedule maintenance, in a form they can act on, integrated with the tools they already use.

An alert in a monitoring dashboard that the maintenance planner never opens is not a workflow change. An alert that flows into the CMMS (Computerised Maintenance Management System) — generating a work order recommendation that can be accepted, rejected, or deferred — is. The difference between these two integration approaches often determines whether the program persists past the first year.

The CMMS integration also creates the feedback loop that improves the model over time. When a maintenance team accepts an alert-generated work order and finds a genuine problem during the inspection, that outcome gets recorded. When they reject an alert because nothing was found, that gets recorded too. This feedback is the training signal that improves precision and recall over time.

Alerts that flow into the CMMS as actionable work order recommendations create both workflow integration and the feedback loop that improves the model

What Realistic Year-One Success Looks Like

A realistic target for a well-scoped first-year predictive maintenance program: 20–40% reduction in unplanned downtime on the target asset cohort, measured against the prior year baseline. Not zero unplanned failures — some failures will happen too fast for any prediction to be actionable, and some will be caused by failure modes not represented in the training data.

The equally important year-one output: a working data pipeline from the historian to the prediction system, a proven alert-to-work-order workflow, a maintenance team that trusts the system enough to act on its recommendations, and a documented set of lessons about which asset types and failure modes the model handles well and which require further work. These are the infrastructure that makes year two faster and more valuable.

Year one delivers 20–40% reduction in unplanned downtime on the target assets and the infrastructure that makes year two faster

Originally published on the CobuildX blog.