DEV Community

Cover image for Blocking AI Scrapers in Spring Boot Without Redis: A Lightweight Edge Filter
Ashutosh Pandey
Ashutosh Pandey

Posted on • Edited on

Blocking AI Scrapers in Spring Boot Without Redis: A Lightweight Edge Filter

#webdev #ai #opensource #security

Bots Found My API Before My Users Did

If you expose a public API, something interesting happens.

Within minutes, bots discover it.

Headless browsers. AI scrapers. Random scripts running from cloud VMs.

They start hammering endpoints, filling logs, consuming CPU, and quietly increasing your cloud bill before your real users even show up.

The usual advice is:

“Just add Redis and implement rate limiting.”

Which works — if you already run Redis.

But for smaller services, side projects, or lean microservices, spinning up Redis just to block a few bots can feel like architectural overkill.

I wanted something simpler.

Something that:

  • lives inside the application
  • adds almost zero latency
  • blocks bots before they hit Spring Security
  • requires zero external infrastructure

So I built a small experiment called VelocityGate.

Move the Defense to the Earliest Possible Layer

In Spring Boot, a request does not go directly to your controller.

It travels through several layers first:

  • Tomcat
  • DispatcherServlet
  • Spring Security filters
  • logging filters
  • interceptors
  • finally your @RestController

If a bot sends 100 requests per second, letting those requests reach Spring Security is already expensive.

You're allocating objects. Building security contexts. Possibly touching a database.

That’s wasted work.

So instead I added a filter that runs before everything else.

@Bean
public FilterRegistrationBean<BotBouncerFilter> velocityFilter() {
    FilterRegistrationBean<BotBouncerFilter> registrationBean = new FilterRegistrationBean<>();
    registrationBean.setFilter(new BotBouncerFilter());
    registrationBean.addUrlPatterns("/*");

    // Run before everything else
    registrationBean.setOrder(Ordered.HIGHEST_PRECEDENCE);

    return registrationBean;
}
Enter fullscreen mode Exit fullscreen mode

This filter acts like a bouncer at the front door.

If a request looks suspicious, it immediately returns a 403.

Which means:

  • no controller allocation
  • no Spring Security processing
  • no database calls
  • no business logic execution

The request dies instantly — exactly how it should.

In-Memory Rate Limiting (No Redis)

For rate limiting I needed something:

  • thread-safe
  • extremely fast
  • lock-free
  • in-memory

So I used a ConcurrentHashMap with AtomicInteger.

private final Map<String, AtomicInteger> requestCounts = new ConcurrentHashMap<>();

public boolean isAllowed(String ipAddress, int limit) {
    AtomicInteger count = requestCounts.computeIfAbsent(ipAddress, k -> new AtomicInteger(0));

    if (count.incrementAndGet() > limit) {
        return false;
    }

    return true;
}
Enter fullscreen mode Exit fullscreen mode

That’s it.

No network calls.
No serialization.
No distributed locks.

Just atomic increments inside the JVM.

A scheduled task clears the map periodically, effectively resetting the rate limit window.

This works well for single-instance deployments or smaller services.

If you run multiple replicas and need global synchronization, Redis or another centralized store is still the right approach.

But for many APIs, this is more than enough.

Layer 2: Headless Browser Detection

Rate limiting alone doesn’t stop slow scrapers.

So VelocityGate also checks for common headless browser fingerprints.

Examples include:

  • HeadlessChrome in headers
  • missing or suspicious User-Agent values
  • automation-related flags
  • incomplete browser header sets

If you're running default Puppeteer or Playwright setups, many of those requests get blocked before the rate limiter even runs.

So the defense becomes two layers:

  1. Detect obvious automation immediately
  2. Throttle aggressive traffic

Simple. Effective. Fast.

Why I Avoided Redis

Redis is fantastic.

But for this use case it introduces:

  • network latency
  • operational overhead
  • another dependency
  • more complexity during local development

If you're operating a globally distributed system, Redis absolutely makes sense.

But if you're just trying to stop bots from melting a small API or VPS, an in-memory approach can be surprisingly effective.

Open Source Repo

VelocityGate is open source here:

👉 https://github.com/ashutosh-stark/velocity-gate

Inside the repo you'll find:

  • the filter implementation
  • header inspection logic
  • the in-memory rate limiter
  • configuration options

It’s packaged as a Spring Boot starter so it can be integrated quickly.

Exploring a Bigger Idea

While working on this, something interesting came up.

Someone asked whether this could work for Node or Bun.

Which made me realize the problem isn't really about Spring Boot.

The real problem is:

Bots and AI scrapers hitting APIs and increasing infrastructure costs.

So I'm exploring whether a language-agnostic gateway might make sense.

Something that sits in front of any backend:

  • Node.js
  • Python
  • Go
  • Java
  • PHP

Basically a lightweight developer-focused API shield that blocks scraper traffic before it reaches your backend.

Before investing months into building distributed synchronization and edge deployment, I want to validate whether developers actually need this.

If this sounds interesting, you can join the early access waitlist here:

👉 https://ashutosh-stark.github.io/velocitygate-cloud/

Final Thoughts

You don’t always need:

  • Redis
  • Kubernetes
  • a service mesh
  • a managed API gateway

Sometimes you just need:

  • a well-placed filter
  • an in-memory counter
  • and the discipline to keep things simple.

I'm curious how other developers are dealing with this problem.

Are bots hitting your APIs?

And if so, how are you handling it?

  • Redis rate limiting
  • Cloudflare
  • API Gateway
  • something custom

Would love to hear your experience.

Top comments (1)

Collapse
 
ashutosh_stark profile image
Ashutosh Pandey •

One thing I'm still exploring is whether this approach works well beyond single-instance deployments.

Right now the in-memory limiter works great for smaller services, but for distributed systems Redis or another centralized store probably makes more sense.

Curious what others are using for bot / scraper protection in production APIs.

Are people mostly relying on Cloudflare, API gateways, or custom rate limiting?
Here is a discussion I have started on GitHub

github.com/ashutosh-stark/velocity...