How I Solved the IP-Bound Cookie Problem in My Web Scraper With a Self-Hosted Proxy

There's a category of airport website that breaks my scraper in a specific, maddening way: they issue session cookies that are pinned to the IP address of the request that created them.

The symptom looks like this: you manually grab a working cookie from your browser, paste it into your server-side scraper, and it immediately returns a 401 or redirects to a login page. The cookie is valid — you just created it. But it was created from your laptop's IP, and your VPS has a different one. The airport's backend checks both the cookie and the origin IP, and when they don't match, it rejects the session.

For MyAirports — a real-time flight data API covering 1,000+ airports — this is a real operational problem. Here's how I solved it.

Why IP-Bound Cookies Exist

Session fixation attacks are the reason. A server that binds cookies to IPs prevents an attacker who steals your session cookie from replaying it from a different location. It's a legitimate security measure, and it's common in systems that handle sensitive operational data — which airport backend systems often do.

For a scraper, it creates a workflow problem. You can:

1. Grab a fresh cookie from a real browser session
2. Replay that cookie from your server

But only if both steps happen from the same IP.

The Manual Cookie Workflow

For airports that require authenticated sessions (particularly those protected by HCaptcha or Incapsula), my architecture uses a grab-cookies.js tool. It launches a real browser, navigates to the airport site, solves challenges if needed, and exports the resulting session cookies.

Those cookies then get passed to the scraper as headers:

// Simplified example from the scraper pipeline
const cookies = await loadCookies('MAD'); // Madrid Barajas
const response = await fetch(endpoint, {
  headers: {
    'Cookie': cookies.join('; '),
    'Referer': 'https://www.aena.es'
  }
});

When grab-cookies.js runs on the VPS itself, everything works — the cookie was issued to the VPS IP, and the scraper makes requests from the same IP. But during development and debugging, running a headless browser on the server just to grab a cookie is slow and inconvenient. You want to do it from your local machine.

That's where the IP mismatch breaks things.

The Solution: Route Your Browser Through the VPS

The fix is simple in principle: make your local browser appear to come from the VPS IP when visiting airport websites. Any traffic that comes from your browser through the proxy will receive cookies bound to the VPS IP — the same IP your production scraper uses.

I used SkipRestriction.uk to set up a self-hosted Shadowsocks proxy on the same VPS where MyAirports runs. The setup is a single Docker command:

docker run -d \
  --name shadowsocks-server \
  --restart always \
  -p 8388:8388/tcp \
  -p 8388:8388/udp \
  shadowsocks/shadowsocks-libev:latest \
  ss-server -s 0.0.0.0 -p 8388 -k YOUR_PASSWORD -m aes-256-gcm

Point your local Shadowsocks client at YOUR_VPS_IP:8388, configure your browser to use it as a SOCKS5 proxy, and now your browser traffic exits from the VPS. Airport websites see your VPS IP. Cookies issued to that session are tied to the VPS IP. When the production scraper replays those cookies from the same VPS, the IP check passes.

Why Shadowsocks and Not a VPN

A traditional VPN would work too, but Shadowsocks has a few practical advantages for this use case:

It's lightweight. Running WireGuard or OpenVPN on the same VPS as a production scraper means managing certificates, key exchange, and routing tables. The Shadowsocks Docker container adds almost nothing to the host.

Traffic looks like HTTPS. Shadowsocks was designed to be indistinguishable from normal encrypted web traffic. Some airport networks filter VPN protocols at the packet level. Shadowsocks is harder to detect and block.

Per-application routing. With a VPN, all traffic from your machine goes through the VPS — including your Slack messages and YouTube tabs. With a SOCKS5 proxy, you configure specific browsers or applications to route through it. You can keep two browser profiles: one going through the VPS for cookie grabbing, one going directly for everything else.

No kernel modules. The Docker container runs entirely in user space. No system-level network reconfiguration needed.

The Full Cookie Workflow

With the proxy in place, the workflow for IP-bound cookies looks like this:

Local browser (via Shadowsocks proxy)
  → VPS IP
  → Airport website
  → Cookie issued to VPS IP

grab-cookies.js export
  → cookies stored server-side

Production scraper (on same VPS)
  → Same VPS IP
  → Airport website
  → Cookie accepted ✓

The roundtrip adds maybe 80-120ms of latency compared to a direct connection, which is irrelevant for a one-time cookie grab operation.

Setting It Up

The SkipRestriction.uk guides cover the full setup for both VPS (cloud server) and Raspberry Pi (local self-hosting). The site also walks through client configuration for every major platform — macOS, Windows, iOS, Android, and Linux.

If you're running a scraper on a VPS and hitting IP-bound session issues, the proxy approach is the cleanest fix. The alternative — running your entire debugging workflow on the server via SSH — is technically viable but slow and error-prone.

Two commands on the server side, one config file on the client side, and the IP mismatch problem disappears.

The proxy runs on port 8388. The MyAirports API runs on port 443. They coexist on the same $5/month VPS without any conflict.

---

MyAirports is a free real-time flight data API covering 1,000+ airports. Try it at myairports.online/developers.

The Shadowsocks setup guide used for this project is at skiprestriction.uk — open source, self-hosted, no accounts required.