DEV Community: Shrikant Mhatre (श्री)

We Broke WordPress for 30 Minutes. Nginx Cache Kept Google From Noticing.

Shrikant Mhatre (श्री) — Fri, 09 Oct 2026 08:16:45 +0000

A WordPress directory migration, a missed Apache vhost config, a pre-warmed Nginx proxy cache, and a Coraza WAF in between. This is the story of how a cache warmer saved a blog from an SEO disaster when the backend silently broke for 30 minutes.

Bagful runs a technical blog at bagful.net. The blog had been live for six weeks, indexed by Google, and generating its first organic impressions. The stack was straightforward: Nginx as the reverse proxy, Coraza WAF for request filtering, Apache serving WordPress on the backend. The site was small, around 190 posts, but the team had spent weeks optimizing content for search visibility. Then, on a routine Tuesday afternoon, an engineer moved the WordPress directory to a more secure location, ran a find-and-replace with sed, missed one config file, and silently broke every page on the site. This is what happened, how the proxy cache absorbed the failure, and what the team learned.

The Architecture: Four Layers Between the Browser and WordPress

Before the incident, it helps to understand the proxy chain. Every request to bagful.net passes through four layers before reaching WordPress.

Layer 1: Nginx (ports 80/443). The public-facing reverse proxy. Handles TLS termination, HTTP/2, static file caching, and proxy caching. Nginx caches full HTML responses from the backend so that repeat visitors get served directly from disk without hitting Apache at all.

Layer 2: Coraza WAF. An open-source web application firewall running as a reverse proxy between Nginx and Apache. Coraza inspects every request against OWASP Core Rule Set (CRS) rules, blocks SQL injection, XSS, and other malicious payloads before they reach the application. It listens on an internal port and forwards clean requests to Apache.

Layer 3: Apache. The application server running mod_php. Apache processes PHP files, executes WordPress code, queries the MySQL database, and returns the rendered HTML response back through the chain.

Layer 4: WordPress. The CMS itself. Themes, plugins, the block editor, the REST API, and WP-CLI all run here. WordPress lives in a directory on the filesystem, and Apache’s DocumentRoot points to it.

The proxy cache sits between Layer 1 and Layer 2. When Nginx has a cached response, it never contacts Coraza or Apache. The request is served entirely from Nginx’s disk cache. This detail is what saved the site.

Free to use, share it in your presentations, blogs, or learning materials.

The four-layer proxy chain: Browser to Nginx (with cache) to Coraza WAF to Apache/WordPress. During the incident, Nginx served cached 200s while the backend returned 404.

Why the Directory Move Was Necessary

WordPress had been installed at the default Apache document root: a path inside /var/www/html/. This is where most tutorials tell you to put it. The problem is that /var/www/html/ is the default document root for every Apache installation. If another virtual host is misconfigured, if a default site is accidentally enabled, or if a new service is added without careful vhost isolation, files under that path could be exposed. Security audits consistently flag this as a risk.

The plan was simple: move WordPress from /var/www/html/cms/ to /var/www/insights/, a dedicated directory outside the default root. The new path is explicit, isolated, and does not share a parent directory with any other web service. The NIST SP 800-123 Guide to General Server Security recommends isolating web application files from default server roots to reduce the attack surface from misconfiguration.

The Migration Plan

The engineer, Ravi, started by investigating every file on the server that referenced the old path. The goal was to identify every config, script, cron job, and binary that would need updating.

Searching for hardcoded path references

$ grep -rn ‘/var/www/html/cms’ /etc/apache2/ /etc/nginx/ /etc/systemd/ /etc/crontab
$ grep -rn ‘/var/www/html/cms’ /root/.bashrc /home/*/.bashrc
$ strings /usr/local/bin/site-backup | grep ‘/var/www’

The search found references in three places: the Apache virtual host config (DocumentRoot and Directory directives), a separate Apache SSL vhost, and a Go binary used for automated backups. Nginx config was clean because it only proxies to Coraza’s internal port and never references the filesystem directly. Crontabs were clean. Shell profiles were clean. WordPress’s own wp-config.php uses relative paths, so it did not need changes.

With the investigation complete, Ravi laid out the steps:

Copy the entire WordPress directory to the new location with permissions preserved
Run sed -i to replace the old path in all Apache config files
Restart Apache
Clear Nginx proxy cache
Run the cache warmer to pre-warm all pages
Verify with curl
Delete the old directory

The Cache Warmer: 190 Pages in 5 Minutes

Before the migration, the team had already deployed a cache warmer. It was a small Go binary that fetched the WordPress sitemap, extracted every URL, and made an HTTP GET request to each page using two concurrent workers. The purpose was to keep Nginx’s proxy cache warm after cache clears, deployments, or server restarts. Every 30 minutes, a cron job ran the warmer to ensure that no visitor ever hit an uncached page.

The warmer’s output looked like this after a typical run:

Cache warmer output after a full run

=== GSL Cache Warmer ===

Fetching sitemap: https://bagful.net/post-sitemap.xml

Found 190 URLs
[W1] 200 | MISS | TTFB: 245ms | OK | /zero-trust-architecture-explained/

[W2] 200 | MISS | TTFB: 189ms | OK | /nginx-proxy-cache-wordpress-setup/

[W1] 200 | MISS | TTFB: 312ms | OK | /coraza-waf-reverse-proxy-guide/

…
=== Summary ===

Total URLs: 190

Cache MISS: 190 (freshly cached)

Cache HIT: 0

Errors: 0

Time elapsed: 5m12s

Every page showed MISS (freshly cached from the backend). This meant Nginx now had a complete, warm cache of every page on the site. For the next 2 hours (the cache TTL), any visitor requesting any page would get a response directly from Nginx’s disk cache without touching Coraza, Apache, or WordPress.

The cache warmer ran successfully at 15:30. The migration started at 15:45.

The Migration: Copy, Sed, Restart

Step one: copy the files.

Copying WordPress to the new secure directory

$ sudo cp -a /var/www/html/cms /var/www/insights
$ sudo chown -R www-data:www-data /var/www/insights
$ ls -la /var/www/insights/wp-config.php
-rw-r—– 1 www-data www-data 3858 Apr 6 13:54 /var/www/insights/wp-config.php

The -a flag preserves ownership, permissions, and timestamps. The copy took about 90 seconds for the 190-post site with its media uploads.

Step two: update Apache configs with sed.

Replacing the old path in Apache configs

$ sudo sed -i ‘s|/var/www/html/cms|/var/www/insights|g’ /etc/apache2/sites-available/blog-ssl.conf
$ sudo sed -i ‘s|/var/www/html/cms|/var/www/insights|g’ /etc/apache2/sites-available/blog.conf
$ grep DocumentRoot /etc/apache2/sites-available/blog-ssl.conf /etc/apache2/sites-available/blog.conf
/etc/apache2/sites-available/blog-ssl.conf: DocumentRoot /var/www/insights

/etc/apache2/sites-available/blog.conf: DocumentRoot /var/www/insights

Ravi ran sed against the two config files he had found during the investigation. Apache config test passed. Apache restarted cleanly.

Config test and restart

$ sudo apache2ctl configtest
Syntax OK

$ sudo systemctl restart apache2

Step three: clear the Nginx cache and test.

Clear Nginx cache and quick test

$ sudo rm -rf /var/cache/nginx/*
$ sudo systemctl reload nginx
$ curl -sk -o /dev/null -w ‘%{http_code}’ https://bagful.net/
200

The homepage returned 200. Ravi tested two more pages. Both returned 200. Everything looked good. He ran the cache warmer again to pre-warm the entire site from the new backend location. All 190 pages came back as 200 MISS (freshly cached).

Step four: check for open files and delete the old directory.

Check for open files and delete old directory

$ sudo lsof +D /var/www/html/cms/ | wc -l
0

$ sudo rm -rf /var/www/html/cms
$ ls -d /var/www/html/cms 2>&1
ls: cannot access ‘/var/www/html/cms’: No such file or directory

Zero open files. Old directory deleted. Migration complete. Or so Ravi thought.

The Config That sed Missed

Twenty minutes later, Ravi ran a routine health check from his laptop.

Health check from external machine

$ curl -sk -o /dev/null -w ‘%{http_code}’ https://bagful.net/
200
$ curl -sk -o /dev/null -w ‘%{http_code}’ https://bagful.net/wp-admin/
404

The homepage returned 200. But wp-admin returned 404. That made no sense. WordPress was running. The files existed at the new path. Apache had restarted without errors.

The homepage was 200 because Nginx was serving it from cache. The wp-admin path was 404 because wp-admin is never cached (the Nginx config skips caching for authenticated paths). So wp-admin hit the full proxy chain: Nginx > Coraza > Apache. And Apache was returning 404.

Ravi tested each layer individually.

Testing each layer of the proxy chain

# Test Apache directly

$ curl -s http://127.0.0.1:8080/ -H ‘Host: bagful.net’ -o /dev/null -w ‘%{http_code}’
404
# Test Coraza (which forwards to Apache)

$ curl -s http://127.0.0.1:8082/ -H ‘Host: bagful.net’ -o /dev/null -w ‘%{http_code}’
404
# Test Nginx (which serves from cache for public pages)

$ curl -sk https://bagful.net/ -o /dev/null -w ‘%{http_code}’
200 (cached)

Apache was returning 404 on every request. The cache was masking the problem for public pages. Ravi checked the Apache vhost configs he had updated with sed.

Checking the Apache configs

$ grep DocumentRoot /etc/apache2/sites-available/blog-ssl.conf
DocumentRoot /var/www/insights ← correct
$ grep DocumentRoot /etc/apache2/sites-available/blog.conf
DocumentRoot /var/www/insights ← correct
$ grep DocumentRoot /etc/apache2/sites-enabled/blog-main.conf
DocumentRoot /var/www/html/cms ← OLD PATH!

There it was. The sites-enabled directory had a third config file: blog-main.conf. This was the vhost listening on the internal port that Coraza forwards to. Ravi’s sed command had updated blog-ssl.conf and blog.conf, but missed blog-main.conf because his investigation had found only two files in sites-available. The third file existed only in sites-enabled as a standalone file, not a symlink.

The proxy chain was: Nginx (443) > Coraza (internal) > Apache (internal port using blog-main.conf). The vhost that actually served requests from the WAF was the one that sed missed. The two vhosts that sed updated were for direct Apache access and legacy SSL, neither of which was in the active request path.

The Fix: One sed Command

Fixing the missed config

$ sudo sed -i ‘s|/var/www/html/cms|/var/www/insights|g’ /etc/apache2/sites-enabled/blog-main.conf
$ sudo systemctl restart apache2$ curl -s http://127.0.0.1:8080/ -H ‘Host: bagful.net’ -o /dev/null -w ‘%{http_code}’
200

One command. Apache now served from the correct directory. The entire proxy chain returned 200.

Free to use, share it in your presentations, blogs, or learning materials.

The investigation traced through six steps: testing public URL (cached 200), testing wp-admin (uncached 404), testing Apache directly (404), testing Coraza (pass-through 404), finding the missed config file, and fixing it with one sed command.

How the Cache Saved SEO: Minute by Minute

Here is the timeline of what happened during the 30-minute window when Apache was broken:

Free to use, share it in your presentations, blogs, or learning materials.

Minute-by-minute timeline: cache warmer at 15:30, migration at 15:45, directory deleted at 15:53 (backend breaks silently), cache absorbs the failure for 27 minutes, 404 discovered at 16:20, fixed at 16:22.

15:30 Cache warmer runs. All 190 pages cached with 200 responses. Cache TTL: 120 minutes
15:45 Migration starts. Files copied. sed runs against two config files. Apache restarts
15:48 Nginx cache cleared. Cache warmer runs again. All 190 pages return 200 from the new backend. Cache is fresh
15:53 Old directory deleted. Apache’s internal vhost (blog-main.conf) still points to the deleted path. Apache starts returning 404 for every uncached request
15:53 to 16:20 The broken window. Every request that hits the backend returns 404. But Nginx serves all public pages from cache (200 HIT). Only wp-admin, wp-cron, and logged-in user requests see the 404
16:20 Ravi discovers the 404 on wp-admin. Traces through the proxy chain. Finds the missed config file
16:22 One sed command fixes the config. Apache restarts. Site fully operational

Ravi immediately checked the Nginx access logs for Google crawler activity during the broken window.

Checking for Googlebot hits during the 404 window

$ sudo grep -i ‘googlebot’ /var/log/nginx/access.log | grep ’15:5[3-9]\|16:[01]’
# (empty: zero Googlebot hits during the window)

Zero Google crawler hits during the entire 30-minute window. Google did not request a single page while the backend was broken. Even if it had, Nginx would have served the cached 200 response. The crawler would have seen a perfectly healthy site.

The Nginx cache had a 120-minute TTL. The broken window was 30 minutes. The cache would have continued serving valid responses for another 90 minutes before expiring. Even without the fix, the site would have appeared healthy to every external visitor until the cache TTL expired at 17:48.

Why the Cache Warmer Was the Real Hero

The proxy cache alone would not have saved the site. Here is why. Nginx only caches a page after someone requests it. If the cache had been cold (empty) when the backend broke, the first visitor to each page would have received a 404. That 404 would have been cached (Nginx was configured with proxy_cache_valid 404 1m) and served to every subsequent visitor for one minute. With 190 pages, it would have taken many visitors hitting 404s before anyone noticed.

The cache warmer eliminated this scenario. By pre-fetching every page before the migration, it ensured that Nginx had a warm, complete cache of every URL on the site. When the backend broke, there was no “first visitor” problem. Every page was already cached with a valid 200 response.

The warmer ran on a 30-minute cron schedule from a separate server:

Cache warmer cron job

# Run cache warmer every 30 minutes

*/30 * * * * /opt/cache-warmer/bin/warmer >> /opt/cache-warmer/log/warmer.log 2>&1

The combination of proxy cache (Nginx) and cache warmer (external cron) created an unintentional safety net. The cache absorbed the backend failure. The warmer ensured the cache was complete. Together, they bought the team 120 minutes of runway to find and fix the problem without any visitor or crawler seeing a broken page.

The Nginx Proxy Cache Configuration

For context, here is the Nginx proxy cache configuration that made this possible. The cache zone is defined in the http block and used in the server block.

PHPNginx proxy cache configuration

 1<br />
 2# In http block: define the cache zone<br />
 3proxy_cache_path /var/cache/nginx/blog levels=1:2 keys_zone=blog_cache:10m<br />
 4 max_size=500m inactive=120m use_temp_path=off;</p>
 5<p># In server block: use the cache<br />
 6location / {<br />
 7 proxy_pass http://127.0.0.1:8082; # Coraza WAF</p>
 8<p> proxy_cache blog_cache;<br />
 9 proxy_cache_valid 200 120m; # Cache 200 responses for 2 hours<br />
10 proxy_cache_valid 301 302 10m; # Cache redirects for 10 minutes<br />
11 proxy_cache_valid 404 1m; # Cache 404s for only 1 minute<br />
12 proxy_cache_bypass $skip_cache;<br />
13 proxy_no_cache $skip_cache;<br />
14 proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;</p>
15<p> add_header X-Cache-Status $upstream_cache_status always;<br />
16}<br />

Two settings were critical during the incident:

proxy_cache_valid 200 120m meant every cached 200 response stayed valid for 2 hours. The backend could be completely down for 2 hours and visitors would still see the site
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504 tells Nginx to serve stale (expired) cache if the backend returns an error. This did not activate during the incident because the cache was still fresh, but it would have provided an additional safety net if the outage had lasted longer than the TTL

The $skip_cache variable bypasses caching for logged-in users, wp-admin, wp-cron, and POST requests. This is why Ravi saw the 404 on wp-admin but not on public pages. wp-admin always hits the backend.

The Coraza WAF in the Middle

The team chose Coraza WAF over ModSecurity for a specific reason: memory footprint. The blog server runs on a modest VPS with limited RAM. ModSecurity 3.x (libmodsecurity) loads as an Apache or Nginx module, sharing the web server’s memory space and adding significant overhead per worker process. Coraza is written in Go and runs as a standalone reverse proxy with its own memory management. In testing, the team measured Coraza using 50MB of RAM with the full OWASP Core Rule Set loaded, compared to 120-180MB per Apache worker process with ModSecurity embedded. On a memory-constrained server running WordPress with multiple Apache prefork workers, that difference is the gap between stable operation and OOM kills.

Coraza is also a modern, actively maintained project. ModSecurity’s development slowed significantly after Trustwave transferred it to OWASP in 2024. Coraza is fully compatible with the OWASP CRS rules, supports the SecLang rule language, and can be deployed without compiling into Nginx or Apache. If you are looking to implement Coraza in your own stack, we have articles that walk through the full setup: Part 1: Architecture and Prerequisites, Part 2: Compiling, Testing, and Findings, and Writing Custom Coraza WAF Rules for PHP and WordPress.

The presence of Coraza in the proxy chain added a layer of complexity that contributed to the missed config. In a simple Nginx > Apache setup, there are only two config files: the Nginx server block and the Apache vhost. With Coraza in between, Apache needs two vhosts: one for the WAF’s internal forwarding port and one for direct/legacy access. Coraza acted as a transparent proxy, forwarding clean requests from Nginx to Apache’s internal vhost.

Ravi’s investigation found the Apache configs in sites-available but missed the one in sites-enabled that was not a symlink. This is a common Apache pattern: most files in sites-enabled are symlinks to sites-available, but standalone files can exist there too. A more thorough investigation would have searched both directories independently.

The investigation command that would have caught it

# Search sites-available AND sites-enabled independently

$ grep -rn ‘/var/www/html/cms’ /etc/apache2/sites-available/ /etc/apache2/sites-enabled/

That single grep against both directories would have revealed the third config file. The original investigation only searched /etc/apache2/ recursively, which should have found it. But the file was found, counted, and then sed was run against only the two files in sites-available, not the standalone file in sites-enabled. A manual oversight in a scripted process.

The Google Search Console Check

The team checked Google Search Console the next morning. The Coverage report showed zero new 404 errors. The crawl stats showed no crawl activity during the 30-minute window. The site’s impressions continued their upward trend without a dip.

This was not just luck. Google’s crawler does not visit small sites continuously. For a blog with 190 posts and moderate authority, Googlebot typically crawls a few pages per hour, not per minute. The probability of a crawler hit during any specific 30-minute window is low. Combined with the proxy cache serving valid 200 responses, the risk of a Google-visible 404 was near zero.

However, if the outage had lasted longer (past the 2-hour cache TTL), the risk profile changes dramatically. Once the cache expires, Nginx would request a fresh copy from the backend, receive the 404, and cache that 404 for 1 minute. Googlebot could then see a 404. Google does not immediately deindex a page that returns 404. It retries over several days. But if the 404 persists, the page is dropped from the index within 1-2 weeks. For a new blog still building authority, losing indexed pages would have been devastating.

Lessons Learned

The team documented five takeaways from the incident.

1. Always test the full proxy chain, not just the endpoint. Ravi tested curl https://bagful.net/ and got 200. That tested Nginx (which served from cache). He should have also tested Apache directly: curl http://127.0.0.1:8080/ -H 'Host: bagful.net'. Testing every hop in the proxy chain would have revealed the 404 immediately.

2. Search all config directories, not just the expected ones. Apache’s sites-enabled can contain standalone files that are not symlinks to sites-available. Always search both. Better yet, search the entire /etc/apache2/ tree and process every match.

3. A cache warmer is not just a performance tool. It is a resilience layer. The warmer ensured that every page was cached before the migration. Without it, the first visitor to each page during the outage would have seen a 404.

4. Set proxy_cache_use_stale aggressively. The error timeout http_500 http_502 http_503 http_504 directive tells Nginx to serve stale cache when the backend fails. Consider adding http_404 to this list for content sites where the URLs are stable. A stale cached page is always better than a 404 for a page that existed 5 minutes ago.

5. Do not delete the old directory until you have verified the backend independently. Ravi deleted the old directory after seeing 200 responses from the public URL. Those responses came from cache, not from the backend. The old directory should have stayed until a direct backend test confirmed the new path was serving correctly.

References

Nginx, ngx_http_proxy_module: proxy_cache documentation
Nginx, proxy_cache_use_stale: serving stale content during errors
Coraza WAF, Introduction and Setup Guide
NIST, SP 800-123: Guide to General Server Security
Google Search Central, HTTP status codes and network errors for Google Search
Apache, Apache Virtual Host documentation
OWASP, Core Rule Set (CRS) for ModSecurity/Coraza

Frequently Asked Questions

Can Nginx proxy cache prevent SEO damage during downtime?

Yes. Nginx proxy cache serves stored HTML responses without contacting the backend. If the backend is down or returning errors, Nginx continues serving cached 200 responses to visitors and crawlers. With proxy_cache_use_stale configured, Nginx can even serve expired cache during errors. Combined with a cache warmer that pre-fetches all pages, the cache acts as a complete safety net. Google crawlers see 200 responses and the site’s search rankings are unaffected.

What is a cache warmer and why does WordPress need one?

A cache warmer is a script or binary that pre-fetches every page on your site so the proxy cache is fully populated. Without it, the first visitor to each page after a cache clear hits the backend directly (cache MISS). For WordPress sites behind Nginx proxy cache, a warmer ensures every page is served from cache instantly. It also creates a resilience layer: if the backend breaks, all pages are already cached with valid responses.

Is it safe to move WordPress to a different directory?

Yes, but you must update every config that references the old path: Apache DocumentRoot and Directory directives, any standalone vhost files in sites-enabled, backup scripts, cron jobs, and compiled binaries with hardcoded paths. Search both sites-available and sites-enabled independently. Test the backend directly (not through a proxy cache) after making changes. Do not delete the old directory until you confirm the backend serves correctly from the new location.

What does proxy_cache_use_stale do in Nginx?

The proxy_cache_use_stale directive tells Nginx to serve expired (stale) cached content when the backend returns specific errors. For example, proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504 serves stale cache when the backend is unreachable or returns server errors. This prevents visitors from seeing error pages during backend outages. For content sites, consider adding http_404 to serve stale cache instead of 404 errors for previously valid pages.

How long before Google deindexes a page that returns 404?

Google does not immediately deindex a page that returns 404. It retries over several days, typically 3-5 crawl attempts spread over 1-2 weeks. If the 404 persists across all retries, the page is dropped from the index. A brief 404 (minutes to hours) followed by recovery has virtually zero impact on indexing. However, for new sites still building authority, even a temporary loss of indexed pages can slow organic growth.

What is Coraza WAF and why put it between Nginx and Apache?

Coraza is an open-source web application firewall compatible with the OWASP Core Rule Set. Placing it between Nginx and Apache creates a defense-in-depth architecture: Nginx handles TLS and caching, Coraza inspects request payloads for attacks (SQL injection, XSS, command injection), and Apache runs WordPress. This separation means each layer can be updated, scaled, and secured independently. Coraza adds 2-5ms of latency per request for rule evaluation.

Why should WordPress not be in the default Apache document root?

The default Apache document root (/var/www/html/) is shared by default configurations and potentially other virtual hosts. If another service or a default Apache config is accidentally enabled, files in that directory could be exposed. Moving WordPress to a dedicated directory (like /var/www/insights/) isolates it from other services. NIST SP 800-123 recommends isolating web application files from default server roots to reduce misconfiguration attack surface.

The post We Broke WordPress for 30 Minutes. Nginx Cache Kept Google From Noticing. appeared first on GetSetLive Blogs.

What AI Actually Does in Bank Fraud Detection (Part 1): The Architecture

Shrikant Mhatre (श्री) — Fri, 09 Oct 2026 07:57:21 +0000

A walk through the real-time fraud detection stack we rebuilt with a US mid-size bank: Kafka ingest, Flink features, a three-model ensemble, the analyst’s alert queue, which model does which job, and why the card decline you hate at the grocery store still happens.

Every fraud vendor who walks into a bank says the same thing: AI will solve your fraud problem. I spent 16 months on a fraud-platform rebuild with a mid-size US bank (4.2 million retail accounts, about 480,000 small-business accounts) whose head of fraud operations had heard that pitch a dozen times and did not believe a word of it. The brief was simple. Build the thing, then write down what AI does and what two humans and a phone still do. This is the first half, the architecture. How a card swipe becomes a fraud score and a decision in under 80 milliseconds, which models sit where, and what holds them together. No magic. Part 2 covers where it broke.

By Shri Director of Technology at GetSetLive and Bagful Cloud Hosting, with 25 years across infrastructure, datacenters, cloud platforms, and modern AI/ML. Day-to-day work focuses on consulting clients to make their existing systems AI-aware and AI-ready, and architecting AI/ML solutions tailored to each customer environment. Every article here comes from direct field exposure, written to help the next batch of professionals design and deploy more sophisticated solutions in their own careers. Read all posts by Shri →
Editor, GetSetLive Technical Blogs17 min read

Key Takeaways

AI does not stop fraud. It cut the bank’s alert haystack from 18,000 a day to about 400, and rules plus analysts decide what happens to those 400.
The stack is an ensemble of three narrow models (XGBoost, an autoencoder, GraphSAGE) behind a rules engine, not one big model; median swipe-to-decision is 72 ms.
Step-up authentication, the “confirm this purchase” push, only exists because the model outputs a probability. It is the single biggest customer-experience win.
The feedback loop from analyst labels back into monthly retraining is the part vendors never demo, and the part that keeps the model alive.

What Is Inside

The sentence in every vendor deck
What the bank ran before the rebuild
What the model watches that rules could not
Anatomy of the real-time pipeline
Which model does which job
One card swipe, end to end
A word on SARs and AML
What comes next

The Sentence in Every Vendor Deck

“AI stops fraud.” I have seen that line on roughly 70% of the fraud vendor slides I have sat through, and it is wrong in a specific, interesting way. AI does not stop fraud. What it did at the bank was shrink the alert haystack from something a human team cannot review (18,000 alerts a day before the rebuild) to something they can (around 400 a day now). Three things together stop the fraud itself: the model’s score, a rule that decides what to do with that score, and, in a small but critical slice of cases, a phone call from an analyst to the customer.

On my first day the head of fraud operations put it to me in a way I have quoted many times since. A model is a filter. It does not decide anything. The decision is a rule: if the score is above 0.83 and the merchant category is one of 14 high-risk ones, block the transaction and send a push notification. The model knows none of that. It returns a number. Everything around the number is policy, and policy is a person figuring out the bank’s risk appetite for the quarter. I have heard a version of that speech from every fraud operations lead I have worked with, and it is always the truest thing they say all day.

What the Bank Ran Before the Rebuild

Until late 2023 the bank ran fraud detection the way most regional banks still do. A rules engine, written in a mix of old SAS code and a newer Drools layer, scored every transaction against roughly 240 rules. The rules were the classics. A transaction above 500 dollars in a country the customer has never transacted in. Three card-not-present transactions inside four minutes. A velocity breach on a specific merchant category code. Each rule produced a yes or no. If any rule fired, the transaction was flagged; if enough fired, it was blocked. The fraud team wrote those rules over a decade, almost always after an incident. Someone got defrauded, a rule went in. Someone complained, a rule got tuned.

The system caught fraud. It also produced around 18,000 alerts per business day, and about 12,000 of those needed an analyst. Fourteen analysts cannot review 12,000 alerts a day, so in practice they reviewed none of them properly. The floor had 14 analysts on two shifts, so each analyst was expected to dispose of roughly 850 alerts per shift. That is 35 an hour, under 90 seconds each. Nobody investigates fraud in 90 seconds. What happened instead, and I watched it during our discovery weeks, is that analysts built reflexes. If it looks like the 300 alerts already closed today that turned out to be nothing, close it. The false positive rate on blocked transactions sat around 68%. Real customers got declined at the grocery store while a script somewhere burned through stolen card numbers unseen.

Below is the pre-AI stack as we mapped it in the first month. The red stops are where volume, accuracy, or plain blindness was costing the bank money and customers.

Free to use, share it in your presentations, blogs, or learning materials.

The pre-AI fraud stack at the bank. Red stops mark where volume, accuracy, or blind spots cost money and customers.

The failures were structural. Better rules would not have fixed them. A rule cannot see beyond a single transaction. There was no memory of “this account has been quiet for 14 months and just lit up”, because that needs a windowed feature across time and the rules engine had no compute for it. There was no view of mule accounts, because mule detection needs a graph and rules do not do graphs. Encrypted mobile banking sessions were a black box. First-party fraud, where the customer is the fraudster, is about 11% of losses and was almost invisible, because every rule assumed the customer was the victim.

And then there were the SARs. A Suspicious Activity Report is a regulatory filing a bank must make when it suspects money laundering. Each one took an analyst two to three hours to write. A senior analyst might finish two in a day. That number came up in every planning meeting we had, and it turned out to be the easiest thing in the whole programme to fix.

What the Model Watches That Rules Could Not

The rebuild took the bank’s data engineering team and us 16 months, four months longer than the plan. Twelve of those months went into data plumbing, not models. On day two I asked the data engineering lead to name the single thing a model does that a rule cannot. No hesitation: context. A rule sees one transaction on its own. A model sees it against the last 10 minutes, the last 30 days, and the links that account has to other accounts. Same data, joined differently, and the joining is the whole game. Rules are a lookup table. Models are a function that takes a vector.

Here is what the model now watches, roughly in order of how much each one moved our numbers:

Windowed transaction features. Count in the last 10 minutes, average amount over 30 days, distinct merchant categories in 24 hours, ratio of the current amount to the 90-day average. Flink computes all of these continuously from the Kafka stream.
Device fingerprint and behavioural biometrics. Typing cadence on mobile logins, swipe pressure, scroll pattern, and the JA4 TLS fingerprint of the app session, through a BioCatch integration. These tell the real account holder apart from someone who merely has the password.
Graph proximity. Is this account within two hops of a known mule? Shared device, shared phone number, shared beneficiary? The graph lives in Neo4j and a PyTorch Geometric GraphSAGE model queries it.
Merchant embedding similarity. Every merchant the customer has used gets a 64-dimensional embedding. A transaction at a merchant cluster unlike anything this customer has touched is a weak signal on its own, and a useful one in combination.
Cross-channel signals. A physical ATM withdrawal in Chicago at 14:32 while the mobile app is active in Atlanta at 14:30 is a red flag even when each event alone looks fine.

None of these are rules. They are features. A rule says “block if X”. A feature is an input to a model that says “the probability this is fraud is 0.73, and here are the six inputs that drove it”. The difference matters because a feature-based system combines weak signals into a strong one. Three weak anomalies on a rules engine are three rules that did not fire. Three weak anomalies on a gradient-boosted tree are a 0.91 fraud score. That, more than any single model choice, is what the 16 months bought.

Anatomy of the Real-Time Pipeline

The architecture drawing on the data engineering team’s wall covers most of the wall. The diagram below is the simplified version. It reads left to right: events in on the far left, a decision out on the far right. Median time from swipe to decision is 72 milliseconds, with a 99th percentile of 118. We cannot afford to be slower, because the card networks give an issuer roughly two seconds for the whole authorisation and Visa and Mastercard use a good chunk of that themselves.

Free to use, share it in your presentations, blogs, or learning materials.

Six lanes, thirteen components. Median card-swipe-to-decision time is 72 milliseconds.

Lane 1: Event Ingest

Everything the model will ever see enters through Apache Kafka. The bank runs a Confluent Cloud cluster with five topics: card-auth (Visa and Mastercard authorisation messages), ach-events (ACH pushes and pulls), wire-events (Fedwire and SWIFT), mobile-sessions (app login and in-app behaviour), and atm-events. Peak is about 14,000 events per second, and Black Friday 2024 pushed it to 19,000 for six hours, which is the day we learned our partition count was two too low. Every event has a schema in Confluent Schema Registry. Each downstream consumer knows exactly which fields to expect, and a producer cannot quietly change one.

Lane 2: Stream Processing and Feature Engineering

Apache Flink is the workhorse. A Flink job enriches every event on Kafka with windowed aggregates computed on the fly: events in the last 10 minutes, card-not-present amount in the last 24 hours, distinct IP addresses in the last 7 days, and about 80 other features held in per-account keyed state. The state lives in RocksDB on the Flink task managers, so lookups are local and fast. When a transaction arrives, Flink computes the features and attaches them to the event. It then writes the enriched record to a Kafka topic for model serving and to the online feature store.

This lane is where our worst early incident lived. In month seven a checkpoint failure during a deploy replayed 40 minutes of the card-auth topic, and for about 25 minutes every “count in last 10 minutes” feature was roughly double. Scores drifted up. The step-up rate tripled. The care queue noticed before our dashboards did. Our fix was idempotent state updates keyed on the authorisation ID, plus an alert on feature distribution shift, which Part 2 comes back to. Nothing about it was a modelling problem.

Lane 3: Feature Store

The bank uses Tecton, the commercial feature store built by people from the Uber Michelangelo team. Its job is the one every feature store has: the features the model learned on must be the exact same features available at serving time. Online lookups hit Redis in under a millisecond. Historical features go to Snowflake for training. Every feature definition is versioned, so a feature can change or retire without breaking the model in silence. This sounds boring. It is boring. It is also the thing that separates a working ML system from a flaky demo, and I would pick it over any model upgrade.

Lane 4: Model Inference

This is where the models live, and there are three of them in the transaction path. The primary is an XGBoost classifier that takes around 240 features and outputs a fraud probability between 0 and 1. Beside it runs an autoencoder for unsupervised anomaly detection, and a GraphSAGE graph neural network that scores how close the transaction sits to known fraud in the account graph. A small logistic meta-model combines the three scores into one number. The ensemble runs inside a FastAPI container per region, with TorchServe handling the GraphSAGE inference and XGBoost serving itself. Median inference latency across all three is 9 milliseconds. Twelve Kubernetes pods sit behind an internal load balancer and scale on request rate.

We argued for a month about whether the autoencoder earned its place. It adds latency and it is hard to explain to a regulator. It stayed because in the shadow period it flagged a card-testing pattern (hundreds of one-dollar authorisations across fresh merchant IDs) two weeks before the supervised model had any labelled examples of it. That is the whole case for an unsupervised model in the ensemble: it sees shapes before anyone has named them.

Lane 5: Decision Engine

The ensemble score, plus a dozen hard rules that still exist (sanctions list hit, card reported stolen, customer on fraud hold), flow into a Drools decision engine. Drools produces one of five outcomes: allow, allow with customer notification, step-up authentication (an OTP challenge or a biometric check in the app), block and notify, or block and lock the card. The fraud operations team sets the thresholds between those outcomes and reviews them monthly; the model has no say in them. When the retail fraud loss ratio drifts above 0.04% of swipe volume, thresholds tighten. When complaints about declines spike, they loosen. It is a permanent trade between fraud loss and customer experience. Nobody solves it; they manage it.

Lane 6: Case Management and Feedback

Alerts that need a human land in Actimize, the case management system. Each case carries the transaction, the features, the top 10 SHAP values from XGBoost, the graph neighbourhood, and a one-paragraph summary written by an LLM. Analysts close the case with one of six labels, including “confirmed fraud” and “confirmed legitimate”. Those labels flow back through Kafka into the training set. The monthly retrain on the last 90 days of labelled data then learns from last month’s analyst decisions. This feedback loop is the most important piece of the whole system. Without it the model goes stale within weeks, because the people on the other side adapt faster than any vendor’s release cycle.

Which Model Does Which Job

Seven distinct models run across the bank’s fraud and AML stack, three of them in the real-time path. Every choice was deliberate. The data engineering lead had to justify each one to the model risk committee in writing, and that exercise killed two models we liked. The diagram shows the mapping; the table underneath is the short version of those justifications.

Free to use, share it in your presentations, blogs, or learning materials.

Seven jobs, seven model families. Each matched to the shape of the problem, not picked for novelty.

Function	Model	Why this one
Primary fraud score at transaction time	XGBoost	Fast tabular inference, handles missing values, SHAP-explainable for regulator review
Unsupervised anomaly (zero-day fraud)	Autoencoder	No labels needed; catches patterns the supervised model has not seen yet
Mule ring and collusion detection	GraphSAGE (GNN)	Tabular models cannot see account-to-account relationships; the GNN traverses the neighbourhood
Behavioural biometrics on mobile	LSTM on keystroke and swipe sequences	Biometrics are time series; the LSTM captures cadence
Device fingerprinting	Random Forest on JA4/TLS features	Hard to evade, interpretable, cheap at scale
Synthetic identity detection	Entity resolution + graph embeddings	Synthetic identities are stitched across sources; the graph exposes the stitching
SAR narrative drafting	Fine-tuned LLM (human-reviewed)	Only model that writes regulator-grade prose; kept out of every automatic decision

Nothing in that table is a foundation model for fraud. There is no GPT-fraud. What there is, is a set of small specialist models, each picked because its assumptions match the shape of one problem. I have now seen the same pattern at a lender, an insurer, and a bank: narrow models do narrow jobs well, and one big model does nothing well.

One Card Swipe, End to End

The best way to feel how the pipeline holds up is to follow one transaction. The example below comes from the bank’s staging replay logs, altered to protect the customer but otherwise real. A 41-year-old debit card holder who lives near the bank’s home city is at an electronics store in Charlotte on a Saturday afternoon, buying a 1,200 dollar laptop. That is unusual for this customer on three counts: never shopped in Charlotte, never at that retailer, and the amount is three standard deviations above his 90-day average.

T+0 ms: The store terminal swipes the card. The authorisation request goes to Mastercard, which forwards it to the bank over ISO 8583.
T+4 ms: The bank’s payment gateway normalises the authorisation into a JSON event and writes it to the card-auth topic.
T+9 ms: Flink picks up the event and pulls the customer’s 80 windowed features from Tecton. Transactions in the last 10 minutes: 0. Thirty-day average amount: 47 dollars. Distinct cities in 30 days: 1. Features enriched and attached.
T+14 ms: The enriched event lands on the inference topic and a FastAPI worker reads it.
T+23 ms: XGBoost returns 0.71 (somewhat suspicious). The autoencoder returns 0.68 (high reconstruction error). GraphSAGE returns 0.12 (no fraud-graph proximity). The meta-model combines them to 0.69.
T+28 ms: A decision rule fires: score between 0.60 and 0.80, plus high amount, plus new merchant city, equals step-up authentication. Response: challenge with a mobile push.
T+32 ms: The bank returns a conditional response to Mastercard: approve with 3-D Secure step-up.
T+600 ms: The customer’s phone buzzes: “Confirm $1,200 at an electronics store in Charlotte?” He taps yes; Face ID confirms.
T+1,840 ms: The bank sends final approval to Mastercard. The terminal beeps. He walks out with the laptop.

The inference took 9 milliseconds and the decision was out 28 milliseconds after the swipe hit the gateway. The remaining 1.8 seconds was a person reacting to a push notification. Under the old rules engine this purchase would have been either fully approved (no rule caught “first time in Charlotte”) or fully blocked (a velocity rule might have fired on the amount). The middle path, where the customer is asked rather than refused, did not exist in the rules world. The model’s probability is what unlocks it.

Step-up authentication is the unsung hero of modern fraud detection. It is invisible to the customer who is real and fatal to the attacker who is not. On the bank’s numbers it now resolves about 62% of ambiguous transactions without an analyst and without a decline, which is the single line on the board deck that made the whole programme worth it.

A Word on SARs and AML

The card pipeline is the fast, real-time side. The anti-money-laundering side is slower and more regulated. In the US a Suspicious Activity Report has to be filed with FinCEN within 30 days of detection. Before the rebuild the bank filed around 3,800 SARs a year. Each one took a senior analyst two to three hours to draft. We added a fine-tuned LLM that reads the case file, the relevant transaction history, and the analyst’s preliminary notes, and produces a first draft of the narrative. An analyst reads it, corrects it, and signs it. That draft saves about 90 minutes per SAR. Today the bank files around 4,600 a year (detection got better, not worse) and the per-SAR effort is down to roughly 45 minutes of senior analyst time.

The LLM is explicitly not in the decision path. It does not decide whether a SAR gets filed. It writes a draft; a human decides. Every SAR is signed by a named compliance officer who carries personal regulatory exposure if it is wrong. That is the pattern FinCEN and the OCC have said they expect, and it is the only pattern our compliance head would sign off on. For the first month the compliance head read every single draft against the source case before trusting a paragraph of it, and I think that was the right call.

What Comes Next

The architecture is the mechanical half. The more interesting half, honestly, is what happens when this stack fails in production, and it does. The bank had three meaningful incidents in the year after go-live, each of which taught the fraud team something about how AI detection breaks at scale. Part 2 covers the named case studies that put real numbers on the scale of the problem (DBS, JPMorgan, HSBC, Danske Bank, Standard Chartered), the brutal arithmetic of false-positive economics on a 0.04% base rate, what SR 11-7 and the EU AI Act mean for a fraud team, and what the senior analyst does on the cases the model refuses to handle. Part 2 is here.

References

Kai Waehner, Fraud Detection with Apache Kafka, KSQL and Apache Flink
NVIDIA, Supercharging Fraud Detection with Graph Neural Networks
Fintech Global, How AI Is Reshaping AML Compliance in Modern Banking, 2026
Conduktor, Real-Time Fraud Detection with Streaming
Ampcome, Agentic AI Examples in Banking: Compliance & Risk, 2026

Frequently Asked Questions

Does AI actually stop bank fraud?

Not on its own. The models produce a fraud probability for each transaction. Rules set by the fraud operations team decide what to do with that score: allow, challenge with step-up authentication, or block. The model narrows the field; rules and humans make the call.

Which machine learning model is used for real-time fraud detection?

Most banks run an ensemble. XGBoost is the primary supervised classifier on tabular features, an autoencoder catches unsupervised anomalies, and a graph neural network such as GraphSAGE catches mule rings. A small meta-model combines the three scores, and the whole ensemble answers in under 15 milliseconds.

What is step-up authentication in fraud detection?

The middle-ground response between approve and decline. When the fraud score is ambiguous (roughly 0.60 to 0.80), the bank asks the customer to confirm the transaction through a mobile push, OTP, or biometric check. It catches fraud without declining legitimate purchases, and it only works because the model produces a probability rather than a yes or no.

How long does an AI fraud decision take?

Model inference runs in 5 to 15 milliseconds. End to end, from card swipe to bank response, the median is 70 to 90 milliseconds. Card networks allow about two seconds for the whole authorisation, so there is headroom, but the pipeline is tuned hard because latency compounds across hops.

What is a feature store and why does a fraud system need one?

A feature store guarantees that the features a model was trained on are computed the same way at serving time. Without one, training and production drift apart silently and the model scores wrong with full confidence. The bank in this article uses Tecton with Redis for online lookups and Snowflake for training data.

Why keep an autoencoder if XGBoost is more accurate?

Because the autoencoder needs no labels. It flags transaction shapes the supervised model has never been trained on, such as a new card-testing pattern, weeks before enough labelled examples exist to retrain XGBoost. It costs a few milliseconds and some explainability, and it earns its place on zero-day fraud.

Can AI replace fraud analysts?

No. AI cut the daily alert volume from a number no team can review to a number that fits in a shift. The analysts who remain handle the hard cases, phone customers, confirm fraud rings, and draft Suspicious Activity Reports. AI does the bulk filtering; humans do the judgement and carry the regulatory exposure.

The post What AI Actually Does in Bank Fraud Detection (Part 1): The Architecture appeared first on GetSetLive Blogs.

DEV Community: Shrikant Mhatre (श्री)

We Broke WordPress for 30 Minutes. Nginx Cache Kept Google From Noticing.

The Architecture: Four Layers Between the Browser and WordPress

Why the Directory Move Was Necessary

The Migration Plan

The Cache Warmer: 190 Pages in 5 Minutes

The Migration: Copy, Sed, Restart

The Config That sed Missed

The Fix: One sed Command

How the Cache Saved SEO: Minute by Minute

Why the Cache Warmer Was the Real Hero

The Nginx Proxy Cache Configuration

The Coraza WAF in the Middle

The Google Search Console Check

Lessons Learned

References

Frequently Asked Questions

What AI Actually Does in Bank Fraud Detection (Part 1): The Architecture

Key Takeaways

What Is Inside

The Sentence in Every Vendor Deck

What the Bank Ran Before the Rebuild

What the Model Watches That Rules Could Not

Anatomy of the Real-Time Pipeline

Lane 1: Event Ingest

Lane 2: Stream Processing and Feature Engineering

Lane 3: Feature Store

Lane 4: Model Inference

Lane 5: Decision Engine

Lane 6: Case Management and Feedback

Which Model Does Which Job

One Card Swipe, End to End

A Word on SARs and AML

What Comes Next

Related Reading

References

Frequently Asked Questions