Defeating High-Velocity Scrapers at the Edge: A Spring Boot Drop-in (Without Redis)

If you’ve ever exposed a public API to the internet, you already know what happens next.

Within hours (sometimes minutes), bots find it.

Headless browsers. AI scrapers. Random scripts running from cloud VMs. They start hammering your endpoints, eating CPU, filling logs, and quietly increasing your cloud bill before your real users even show up.

The default advice is usually:

“Just use Redis and add a rate limiter.”

Which is fine… if you already run Redis.

But for small services, side projects, or lean microservices, spinning up Redis just to block a few bots feels like architectural overkill.

I wanted something simpler.

Something that:

• Lives inside the app
• Adds almost zero latency
• Stops bots before they touch Spring Security
• Requires zero external infrastructure

So I built VelocityGate.

---

Move the Defense to the Edge

In Spring Boot, a request doesn’t go straight to your @RestController.

It passes through:

• Tomcat
• DispatcherServlet
• Spring Security filter chain
• Logging filters
• Interceptors
• Then finally your controller

If a bot is hitting your API 100 times per second, letting those requests reach Spring Security is already too late. You're allocating objects. Building security contexts. Possibly touching the database.

That’s wasted work.

So instead, I injected a custom Filter and forced it to run first.

@Bean
public FilterRegistrationBean<BotBouncerFilter> velocityFilter() {
    FilterRegistrationBean<BotBouncerFilter> registrationBean = new FilterRegistrationBean<>();
    registrationBean.setFilter(new BotBouncerFilter());
    registrationBean.addUrlPatterns("/*");

    // Run before everything else
    registrationBean.setOrder(Ordered.HIGHEST_PRECEDENCE);

    return registrationBean;
}

This filter acts like a bouncer at the front door.

If the request looks like a bot, we immediately return a 403 and end it.

That means:

• No controller allocation
• No Spring Security context creation
• No database pool usage
• No business logic execution

The request dies instantly — exactly how it should.

---

In-Memory Fixed Window Rate Limiter (No Redis)

For rate limiting, I needed something:

• Thread-safe
• Fast
• Lock-free
• In-memory

I used a ConcurrentHashMap with AtomicInteger.

private final Map<String, AtomicInteger> requestCounts = new ConcurrentHashMap<>();

public boolean isAllowed(String ipAddress, int limit) {
    AtomicInteger count = requestCounts.computeIfAbsent(ipAddress, k -> new AtomicInteger(0));

    if (count.incrementAndGet() > limit) {
        return false;
    }

    return true;
}

That’s it.

No network hop.
No serialization.
No distributed lock.
No external dependency.

Just atomic increments inside the JVM.

A lightweight scheduled task clears the map at fixed intervals, effectively resetting the window.

This approach works best for single-instance deployments or smaller services. If you're running multiple replicas or need global synchronization, a centralized store like Redis is still the right choice.

For many APIs, though, this is more than enough.

---

Layer 2: Signature Detection

Rate limiting alone doesn’t stop slow scrapers.

So VelocityGate also checks for common headless browser fingerprints.

Things like:

• HeadlessChrome in headers
• Missing or suspicious User-Agent
• Automation-related flags
• Incomplete browser header sets

If you're using default Puppeteer or Playwright configurations, you won’t even reach the rate limiter — the request is rejected immediately.

Now we have two layers:

1. Block obvious automation instantly
2. Throttle aggressive traffic

Simple. Effective. Fast.

---

Why I Avoided Redis

Redis is great.

But for this use case:

• It adds network latency
• It adds operational overhead
• It introduces another failure point
• It complicates local development

If you're running a globally distributed system with multiple replicas, sure — use Redis or a centralized store.

But if you're just trying to stop bots from melting your VPS, you probably don’t need distributed architecture.

---

Open Source Repo

The project is open source here:

👉 velocity Gate

Inside the repo you'll find:

• The full filter implementation
• Header inspection logic
• Fixed window rate limiter
• Configuration options

It’s built as a drop-in Spring Boot starter so you can integrate it quickly.

---

What’s Next

Now I’m exploring something bigger.

Instead of a Spring-only solution, I’m thinking about building a lightweight reverse proxy (probably in Go or Rust) that sits in front of any backend:

• Node.js
• Python
• PHP
• Go
• Java

Basically a developer-first Cloud WAF focused specifically on blocking AI scrapers and bot traffic for APIs.

Before I spend months building distributed synchronization and edge deployment, I want to validate demand.

Would you use something like that? Join Eaily Access waitlist here

---

Final Thoughts

You don’t always need:

• Redis
• Kubernetes
• A service mesh
• A managed API gateway

Sometimes you just need:

A well-placed filter.
An in-memory counter.
And the discipline to keep things simple.

How are you handling bot traffic in your projects right now?

Redis?
Cloudflare?
API Gateway?
Or just hoping for the best?