Your rate limiter is not a security control

Last week a bot pulled ~100% of one passport's data out of my visa API in a few minutes. My rate limiter — 3,000 calls/month on the free tier — never fired once. It didn't need to: a full passport is only ~195 calls.

That's the whole point of this post. Rate limiting protects your infrastructure cost. It does not protect your data. They're two different problems, and conflating them is how solo builders leak their most valuable asset.

Why volume limits fail against extraction

A rate limiter answers one question: is this client costing me too much per unit time? Targeted extraction is specifically designed to stay under that line. My scraper ran at ~25 req/min — trivial load — and would have needed only ~40,000 calls total to copy everything. Spread across a few free accounts, that's invisible to any per-key volume cap.

Volume is the wrong axis. The right axes are identity, coverage, and shape.

Defense 1 — Log the network, not just the key

The first thing that bit me: I could disable the API key, but I hadn't stored the source IP, so re-signup was free. Fix the log first.

ALTER TABLE visa_api_logs
  ADD COLUMN ip inet,
  ADD COLUMN country text;

CREATE INDEX idx_visa_logs_ip ON visa_api_logs (ip, created_at);

Populate ip from cf-connecting-ip (the real client behind Cloudflare — never trust remote_addr at the edge). Now a repeat offender is blockable at the network layer, and you can feed the IP straight to a Cloudflare firewall rule.

Defense 2 — Cap the resource, not the request count

Instead of "N requests/month," cap "N distinct destinations per passport per day." A legitimate user checks a handful of destinations for their own passport. A scraper wants all of them. The cap barely touches real usage and hard-stops a sweep.

Defense 3 — Detect the shape in real time

The sweep has a signature: dense, sequential, exhaustive coverage of a single from. That's detectable per key in a sliding window:

// Sketch: flag a key sweeping one passport across many destinations.
fn is_dump_pattern(hits: &[VisaHit], window: Duration) -> bool {
    let recent: Vec<_> = hits
        .iter()
        .filter(|h| h.ts.elapsed() < window)
        .collect();

    // Group by source passport, count distinct destinations.
    let mut by_passport: HashMap<&str, HashSet<&str>> = HashMap::new();
    for h in &recent {
        by_passport.entry(&h.from).or_default().insert(&h.to);
    }

    // One passport, many destinations, tight cadence = extraction.
    by_passport
        .values()
        .any(|dests| dests.len() > DUMP_THRESHOLD)
        && mean_interval(&recent) < Duration::from_secs(4)
}

Trip it → throttle that key hard (or 402 it) and log the IP. You're now reacting to behavior, not just counting.

The mental model

Split the two jobs explicitly:

• Rate limiter → "don't cost me money." Cheap, coarse, per-key volume.
• Extraction defense → "don't copy my dataset." Identity + coverage + cadence, network-aware.

If you only have the first one, you have a billing guardrail dressed up as security. Ask me how I know.