DEV Community: Sruthi sri G

Building an API Rate Limiter from Scratch with Node.js and Redis

Sruthi sri G — Fri, 30 Oct 2026 07:15:10 +0000

Building an API Rate Limiter from Scratch with Node.js and Redis

When building public APIs, unthrottled endpoints are an invitation for trouble. A rogue client script, a scraping bot, or a distributed denial-of-service (DDoS) spike can exhaust your database connections and crash your services.

While off-the-shelf NPM packages exist, relying on black-box middleware without understanding how it manages race conditions in distributed environments is dangerous.

In this tutorial, we will build a production-ready rate limiter from scratch in TypeScript, Node.js, and Redis using the Token Bucket Algorithm.

Why Redis?

In a multi-instance deployment (e.g., behind a load balancer on AWS or Kubernetes), in-memory counters inside a single Node.js process do not work:

[Client Request] ──► [Load Balancer]
                           ├──────► [Node Instance A (Counter: 1)]
                           └──────► [Node Instance B (Counter: 1)]

If a client sends 10 requests, they get split across instances, allowing the client to bypass the limit. Redis provides a centralized, blazing-fast in-memory store that all app servers query.

How the Token Bucket Algorithm Works

Imagine a bucket that holds a maximum capacity of tokens (e.g., 10 tokens).

Every incoming request consumes 1 token.
If tokens are available, the request is allowed.
If the bucket is empty (0 tokens), the request is rejected with HTTP 429 Too Many Requests.
Over time, tokens refill at a steady rate (e.g., 1 token every second) up to the bucket capacity.

               Tokens refill at a fixed rate (e.g. +2/sec)
                             │
                             ▼
                    ┌─────────────────┐
                    │  Token Bucket   │
                    │  [●] [●] [●]    │ Capacity = 5
                    └────────┬────────┘
                             │
            Request Arrives? │ Consumes 1 Token
                             ▼
              Tokens > 0? ──► YES ──► Allow (200 OK)
                          └──► NO  ──► Block (429 Too Many Requests)

Project Setup

Initialize a new TypeScript Node.js project:

mkdir rate-limiter-redis && cd rate-limiter-redis
npm init -y
npm install express ioredis dotenv
npm install -D typescript @types/node @types/express ts-node
npx tsc --init

Ensure your tsconfig.json contains:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "./dist"
  }
}

Step 1: Initialize the Redis Client

Create src/redis.ts:

// src/redis.ts
import Redis from "ioredis";

const REDIS_URL = process.env.REDIS_URL || "redis://localhost:6379";

export const redis = new Redis(REDIS_URL, {
  maxRetriesPerRequest: 3,
  enableReadyCheck: true,
  lazyConnect: false,
});

redis.on("connect", () => {
  console.log("Connected to Redis successfully.");
});

redis.on("error", (err) => {
  console.error("Redis connection error:", err);
});

Step 2: The Concurrency Problem & Lua Scripting

If we read the current token count and update it with two separate Redis commands (GET followed by SET), concurrent requests from the same user create a race condition:

Request A reads: 1 token remaining.
Request B reads: 1 token remaining.
Both deduct 1 token, allowing 2 requests when only 1 should have passed.

To make the check-and-decrement step completely atomic, we execute the logic inside a Redis Lua script. Redis executes Lua scripts sequentially on a single thread, guaranteeing zero race conditions.

Create src/rateLimiter.ts:

// src/rateLimiter.ts
import { redis } from "./redis";

// Lua script for atomic Token Bucket execution
const TOKEN_BUCKET_LUA = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2]) -- tokens per millisecond
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

-- Retrieve current bucket state
local data = redis.call("HMGET", key, "tokens", "last_updated")
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])

if not tokens then
    -- First request: initialize bucket
    tokens = capacity
    last_updated = now
else
    -- Calculate refilled tokens based on elapsed time
    local elapsed = math.max(0, now - last_updated)
    local refilled = elapsed * refill_rate
    tokens = math.min(capacity, tokens + refilled)
    last_updated = now
end

-- Check if enough tokens exist
if tokens >= requested then
    tokens = tokens - requested
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    -- Expire bucket after 1 hour of inactivity to conserve memory
    redis.call("EXPIRE", key, 3600)
    return {1, math.floor(tokens)} -- Allowed: 1, remaining tokens
else
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    redis.call("EXPIRE", key, 3600)
    return {0, math.floor(tokens)} -- Blocked: 0, remaining tokens
end
`;

export interface RateLimitConfig {
  capacity: number;       // Max tokens burst
  refillPerSecond: number; // Replenish rate
}

export async function checkRateLimit(
  identifier: string,
  config: RateLimitConfig
): Promise<{ allowed: boolean; remaining: number }> {
  const key = `ratelimit:${identifier}`;
  const now = Date.now();
  const refillRatePerMs = config.refillPerSecond / 1000;

  const result = (await redis.eval(
    TOKEN_BUCKET_LUA,
    1,
    key,
    config.capacity,
    refillRatePerMs,
    now,
    1
  )) as [number, number];

  const allowed = result[0] === 1;
  const remaining = result[1];

  return { allowed, remaining };
}

Step 3: Create Express Middleware with Standard Headers

RFC specifications recommend including rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After) so frontend clients can throttle their own requests proactively.

Create src/middleware.ts:

// src/middleware.ts
import { Request, Response, NextFunction } from "express";
import { checkRateLimit, RateLimitConfig } from "./rateLimiter";

export function createRateLimiter(config: RateLimitConfig) {
  return async (req: Request, res: Response, next: NextFunction): Promise<void> => {
    // Identify user by API key, User ID, or fallback to client IP
    const clientIdentifier =
      (req.headers["x-api-key"] as string) ||
      req.ip ||
      req.socket.remoteAddress ||
      "unknown-client";

    try {
      const { allowed, remaining } = await checkRateLimit(clientIdentifier, config);

      // Set standard headers
      res.setHeader("X-RateLimit-Limit", config.capacity);
      res.setHeader("X-RateLimit-Remaining", Math.max(0, remaining));

      if (!allowed) {
        res.setHeader("Retry-After", Math.ceil(1 / config.refillPerSecond));
        res.status(429).json({
          error: "Too Many Requests",
          message: "You have exceeded your request quota. Please wait before retrying.",
        });
        return;
      }

      next();
    } catch (error) {
      console.error("Rate limiter failure:", error);
      // Fail-open: If Redis is down, allow traffic rather than breaking the application
      next();
    }
  };
}

Design Pattern Note: Fail-Open vs Fail-Closed

Notice the catch block calls next(). In high-availability architectures, you often want to fail-open so an infrastructure outage on Redis does not take down your entire public API.

Step 4: Putting It All Together in an Express App

Create src/index.ts:

// src/index.ts
import express from "express";
import { createRateLimiter } from "./middleware";

const app = express();
app.use(express.json());

// Apply limit: 5 requests max burst, refilling 1 token every 2 seconds (0.5 tokens/sec)
const apiLimiter = createRateLimiter({
  capacity: 5,
  refillPerSecond: 0.5,
});

app.get("/api/public-data", (req, res) => {
  res.json({ message: "This is an unthrottled public endpoint." });
});

app.get("/api/sensitive-action", apiLimiter, (req, res) => {
  res.json({
    success: true,
    data: "Access granted to resource.",
    timestamp: new Date().toISOString(),
  });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Step 5: Testing with cURL

Run your server:

npx ts-node src/index.ts

In another terminal, send 6 rapid requests:

for i in {1..6}; do curl -i http://localhost:3000/api/sensitive-action; done

On requests 1 through 5, you will receive HTTP 200 OK with decreasing X-RateLimit-Remaining headers:

HTTP/1.1 200 OK
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 4

On the 6th request, you immediately receive:

HTTP/1.1 429 Too Many Requests
Retry-After: 2
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 0

{"error":"Too Many Requests","message":"You have exceeded your request quota..."}

Wait 4 seconds, send the request again, and tokens will have refilled!

Key Takeaways

Token Bucket gives you the best of both worlds: it handles sudden legitimate traffic bursts while maintaining a strict long-term average rate.
Atomic Lua Scripts are essential in distributed environments to prevent double-spending and race conditions.
Fail-Open Strategy prevents your rate-limiting caching layer from becoming a single point of failure for your entire service.

Next Steps

Want to push this further? Try adding tier-based limits (e.g., 10 req/min for free users, 100 req/min for paid accounts) by looking up user subscription status before invoking the Lua script.

Building a Resilient Web Scraping & Alert Pipeline in Python

Sruthi sri G — Fri, 30 Oct 2026 07:05:55 +0000

Building a Resilient Web Scraping & Alert Pipeline in Python

Most web scraping tutorials show you how to fetch a static page with requests and print a price to the terminal. While fine for toy demos, that approach quickly falls apart in production: servers throttle requests, network connections drop, and website layouts change unexpectedly.

In this tutorial, we will build a production-ready, asynchronous price-monitoring pipeline in Python.

By the end of this guide, you will have a pipeline that:

Fetches multiple target URLs concurrently using aiohttp.
Parses product data reliably with BeautifulSoup4.
Handles retries with exponential backoff.
Dispatches formatted alert notifications to a Discord or Slack webhook when target thresholds are hit.

Architecture Overview

Here is how data flows through our pipeline:

[Target URLs] 
      │
      ▼
[Async HTTP Fetcher] ──(Backoff / Retries)──► [Network Resilience]
      │
      ▼
[Parser & Validator] ──(Extracts price & title safely)
      │
      ▼
[Threshold Engine] ───(Compares current vs. desired price)
      │
      ▼
[Discord Webhook Alert]

Prerequisites

Before starting, ensure you have:

Python 3.9+ installed.
A basic understanding of asynchronous Python (async/await).
A Discord channel with an active Webhook URL (or a dummy URL for testing).

Create and activate a virtual environment, then install the required dependencies:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install aiohttp beautifulsoup4 pydantic

Step 1: Define the Data Models with Pydantic

Strong typing prevents silent failures when parsing scraped HTML. Let's create models.py to define our data structures.

# models.py
from pydantic import BaseModel, HttpUrl, Field
from typing import Optional

class TargetProduct(BaseModel):
    name: str
    url: HttpUrl
    target_price: float = Field(gt=0, description="Alert if price drops below this value")

class ScrapedResult(BaseModel):
    name: str
    url: str
    current_price: Optional[float] = None
    in_stock: bool = False
    error: Optional[str] = None

Using Pydantic guarantees that invalid target configurations fail loudly at startup rather than mid-execution.

Step 2: Build the Asynchronous Fetcher with Retries

Scrapers frequently encounter temporary rate limits (429 Too Many Requests) or connection timeouts. Instead of failing immediately, we implement exponential backoff.

Create fetcher.py:

# fetcher.py
import asyncio
import logging
import aiohttp
from typing import Optional

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")

DEFAULT_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

async def fetch_html_with_retry(
    session: aiohttp.ClientSession,
    url: str,
    max_retries: int = 3,
    base_delay: float = 1.5,
) -> Optional[str]:
    """
    Fetches raw HTML from a URL with exponential backoff on transient errors.
    """
    for attempt in range(1, max_retries + 1):
        try:
            async with session.get(url, headers=DEFAULT_HEADERS, timeout=aiohttp.ClientTimeout(total=10)) as response:
                if response.status == 200:
                    return await response.text()
                elif response.status in (429, 500, 502, 503, 504):
                    delay = base_delay * (2 ** (attempt - 1))
                    logging.warning(f"HTTP {response.status} for {url}. Retrying in {delay:.1f}s (Attempt {attempt}/{max_retries})...")
                    await asyncio.sleep(delay)
                else:
                    logging.error(f"Unrecoverable HTTP {response.status} for {url}")
                    return None
        except (aiohttp.ClientError, asyncio.TimeoutError) as err:
            delay = base_delay * (2 ** (attempt - 1))
            logging.warning(f"Network error on {url}: {err}. Retrying in {delay:.1f}s...")
            await asyncio.sleep(delay)

    logging.error(f"Exceeded max retries for {url}")
    return None

Why this matters:

ClientTimeout(total=10): Prevents requests from hanging indefinitely.
Exponential backoff (2 ** (attempt - 1)): Gives the target server breathing room instead of slamming it with immediate retries.
Custom User-Agent: Emulates a standard browser header to avoid naive bot blockers.

Step 3: Implement Resilient HTML Parsing

Websites change selectors frequently. We isolate our extraction logic inside parser.py and sanitize price strings into clean floating-point numbers.

# parser.py
import re
from typing import Optional
from bs4 import BeautifulSoup
from models import TargetProduct, ScrapedResult

def parse_product_html(html: str, target: TargetProduct) -> ScrapedResult:
    """
    Parses product details safely from HTML using multi-selector fallbacks.
    """
    soup = BeautifulSoup(html, "html.parser")

    # Common selector patterns for prices across modern storefronts
    price_selectors = [
        ".price", "#priceblock_ourprice", ".a-price-whole",
        "[data-testid='product-price']", ".current-price"
    ]

    extracted_price: Optional[float] = None

    for selector in price_selectors:
        element = soup.select_one(selector)
        if element and element.get_text(strip=True):
            raw_text = element.get_text(strip=True)
            # Match numbers like $129.99 or 1,299.00
            cleaned_match = re.search(r"(\d+[\d,]*\.?\d*)", raw_text.replace(",", ""))
            if cleaned_match:
                extracted_price = float(cleaned_match.group(1))
                break

    # Check for Out of Stock indicators
    stock_text = soup.get_text().lower()
    out_of_stock = any(phrase in stock_text for phrase in ["out of stock", "currently unavailable", "sold out"])

    return ScrapedResult(
        name=target.name,
        url=str(target.url),
        current_price=extracted_price,
        in_stock=not out_of_stock,
        error="Price not found" if extracted_price is None else None
    )

Step 4: Dispatching Alerts via Discord Webhook

When a price drops below our target, we notify the user immediately via an embedded message.

Create notifier.py:

# notifier.py
import aiohttp
import logging
from models import ScrapedResult, TargetProduct

async def send_discord_alert(
    session: aiohttp.ClientSession,
    webhook_url: str,
    product: TargetProduct,
    result: ScrapedResult
) -> bool:
    """
    Sends an alert card to a Discord webhook channel.
    """
    payload = {
        "embeds": [
            {
                "title": f"🚨 Price Drop Alert: {product.name}",
                "url": str(product.url),
                "color": 3066993,  # Green
                "fields": [
                    {"name": "Current Price", "value": f"${result.current_price:.2f}", "inline": True},
                    {"name": "Target Price", "value": f"${product.target_price:.2f}", "inline": True},
                    {"name": "Status", "value": "In Stock" if result.in_stock else "Out of Stock", "inline": False},
                ],
                "footer": {"text": "Python Automation Pipeline"},
            }
        ]
    }

    try:
        async with session.post(webhook_url, json=payload) as resp:
            return resp.status in (200, 204)
    except Exception as exc:
        logging.error(f"Failed to deliver webhook alert: {exc}")
        return False

Step 5: Putting It All Together

Create the orchestrator in main.py using asyncio.gather to scrape multiple targets concurrently:

# main.py
import asyncio
import os
import aiohttp
from models import TargetProduct
from fetcher import fetch_html_with_retry
from parser import parse_product_html
from notifier import send_discord_alert

# Replace with your actual Discord Webhook or an environment variable
DISCORD_WEBHOOK_URL = os.getenv("DISCORD_WEBHOOK_URL", "https://discord.com/api/webhooks/YOUR_KEY")

TARGET_WATCHLIST = [
    TargetProduct(
        name="Mechanical Keyboard",
        url="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",  # Safe sandbox url
        target_price=60.00
    ),
    TargetProduct(
        name="Noise Cancelling Headphones",
        url="https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
        target_price=55.00
    ),
]

async def monitor_target(session: aiohttp.ClientSession, target: TargetProduct):
    html = await fetch_html_with_retry(session, str(target.url))
    if not html:
        print(f"[-] Could not retrieve page for: {target.name}")
        return

    result = parse_product_html(html, target)
    print(f"[+] Scraped {result.name}: Current=${result.current_price} | In Stock={result.in_stock}")

    # Trigger alert if criteria is met
    if result.current_price and result.current_price <= target.target_price and result.in_stock:
        print(f"[!] Target hit for {target.name}! Sending alert...")
        await send_discord_alert(session, DISCORD_WEBHOOK_URL, target, result)

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [monitor_target(session, target) for target in TARGET_WATCHLIST]
        await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(main())

Production Best Practices

When deploying this pipeline as a scheduled job (e.g., via GitHub Actions or a cron daemon):

Rotate IP Addresses: For large-scale scrapers (hundreds of URLs), route requests through residential proxies.
Respect robots.txt: Always check the site's scraping policies and avoid hitting internal endpoints too aggressively.
Persist State: Store previous prices in SQLite or Redis to calculate historical price trends.

Summary

In this guide, you built a production-grade scraping pipeline that avoids the common pitfalls of naive scraper scripts. You learned how to:

Enforce strict typing with Pydantic.
Mitigate transient network and rate-limit drops with exponential backoff.
Parse unstructured HTML defensively.
Push real-time asynchronous notifications to chat applications.

You can now expand this pipeline into a full side-project or monetization engine by offering automated competitor tracking to e-commerce merchants.