Blocking AI Scrapers in Spring Boot Without Redis: A Lightweight Edge Filter

Ashutosh Pandey — Thu, 19 Feb 2026 12:09:43 +0000

Bots Found My API Before My Users Did

If you expose a public API, something interesting happens.

Within minutes, bots discover it.

Headless browsers. AI scrapers. Random scripts running from cloud VMs.

They start hammering endpoints, filling logs, consuming CPU, and quietly increasing your cloud bill before your real users even show up.

The usual advice is:

“Just add Redis and implement rate limiting.”

Which works — if you already run Redis.

But for smaller services, side projects, or lean microservices, spinning up Redis just to block a few bots can feel like architectural overkill.

I wanted something simpler.

Something that:

lives inside the application
adds almost zero latency
blocks bots before they hit Spring Security
requires zero external infrastructure

So I built a small experiment called VelocityGate.

Move the Defense to the Earliest Possible Layer

In Spring Boot, a request does not go directly to your controller.

It travels through several layers first:

Tomcat
DispatcherServlet
Spring Security filters
logging filters
interceptors
finally your @RestController

If a bot sends 100 requests per second, letting those requests reach Spring Security is already expensive.

You're allocating objects. Building security contexts. Possibly touching a database.

That’s wasted work.

So instead I added a filter that runs before everything else.

@Bean
public FilterRegistrationBean<BotBouncerFilter> velocityFilter() {
    FilterRegistrationBean<BotBouncerFilter> registrationBean = new FilterRegistrationBean<>();
    registrationBean.setFilter(new BotBouncerFilter());
    registrationBean.addUrlPatterns("/*");

    // Run before everything else
    registrationBean.setOrder(Ordered.HIGHEST_PRECEDENCE);

    return registrationBean;
}

This filter acts like a bouncer at the front door.

If a request looks suspicious, it immediately returns a 403.

Which means:

no controller allocation
no Spring Security processing
no database calls
no business logic execution

The request dies instantly — exactly how it should.

In-Memory Rate Limiting (No Redis)

For rate limiting I needed something:

thread-safe
extremely fast
lock-free
in-memory

So I used a ConcurrentHashMap with AtomicInteger.

private final Map<String, AtomicInteger> requestCounts = new ConcurrentHashMap<>();

public boolean isAllowed(String ipAddress, int limit) {
    AtomicInteger count = requestCounts.computeIfAbsent(ipAddress, k -> new AtomicInteger(0));

    if (count.incrementAndGet() > limit) {
        return false;
    }

    return true;
}

That’s it.

No network calls.
No serialization.
No distributed locks.

Just atomic increments inside the JVM.

A scheduled task clears the map periodically, effectively resetting the rate limit window.

This works well for single-instance deployments or smaller services.

If you run multiple replicas and need global synchronization, Redis or another centralized store is still the right approach.

But for many APIs, this is more than enough.

Layer 2: Headless Browser Detection

Rate limiting alone doesn’t stop slow scrapers.

So VelocityGate also checks for common headless browser fingerprints.

Examples include:

HeadlessChrome in headers
missing or suspicious User-Agent values
automation-related flags
incomplete browser header sets

If you're running default Puppeteer or Playwright setups, many of those requests get blocked before the rate limiter even runs.

So the defense becomes two layers:

Detect obvious automation immediately
Throttle aggressive traffic

Simple. Effective. Fast.

Why I Avoided Redis

Redis is fantastic.

But for this use case it introduces:

network latency
operational overhead
another dependency
more complexity during local development

If you're operating a globally distributed system, Redis absolutely makes sense.

But if you're just trying to stop bots from melting a small API or VPS, an in-memory approach can be surprisingly effective.

Open Source Repo

VelocityGate is open source here:

👉 https://github.com/ashutosh-stark/velocity-gate

Inside the repo you'll find:

the filter implementation
header inspection logic
the in-memory rate limiter
configuration options

It’s packaged as a Spring Boot starter so it can be integrated quickly.

Exploring a Bigger Idea

While working on this, something interesting came up.

Someone asked whether this could work for Node or Bun.

Which made me realize the problem isn't really about Spring Boot.

The real problem is:

Bots and AI scrapers hitting APIs and increasing infrastructure costs.

So I'm exploring whether a language-agnostic gateway might make sense.

Something that sits in front of any backend:

Node.js
Python
Go
Java
PHP

Basically a lightweight developer-focused API shield that blocks scraper traffic before it reaches your backend.

Before investing months into building distributed synchronization and edge deployment, I want to validate whether developers actually need this.

If this sounds interesting, you can join the early access waitlist here:

👉 https://ashutosh-stark.github.io/velocitygate-cloud/

Final Thoughts

You don’t always need:

Redis
Kubernetes
a service mesh
a managed API gateway

Sometimes you just need:

a well-placed filter
an in-memory counter
and the discipline to keep things simple.

I'm curious how other developers are dealing with this problem.

Are bots hitting your APIs?

And if so, how are you handling it?

Redis rate limiting
Cloudflare
API Gateway
something custom

Would love to hear your experience.

DEV Community: Ashutosh Pandey

[Boost]