DEV Community: José Catalá

Three Ways to Fly: Building Route Options Ranked by Time, CO2 , and Reliability

José Catalá — Mon, 27 Apr 2026 08:14:39 +0000

Three Ways to Fly: Building Route Options Ranked by Time, CO₂, and Reliability

Tags: aviation, webdev, javascript, node

We just shipped the Routes tab on MyAirports. You pick an origin and a destination, and you get up to three connecting itineraries — one optimised for total travel time, one for CO₂, one for reliability. Each card shows the hub sequence, layover breakdown, ticket type, and the per-leg airline strip.

Building it meant solving a set of problems that are more interesting than they look. Here's how the system works.

Routes as views, not entities

The first design decision was the most important: we don't store routes.

The existing pipeline already produces Flight records from scraping, DelayStats from nightly aggregation, and RouteStats / AirlineRouteStats from a weighted recompute. The routes system is built entirely on top of those — it's a graph and a ranker layered over data that already existed.

The alternative was a routes table with its own write path. That would have meant migration risk, dual writes, and a whole new ingestion concern. Instead:

Flight (scraped)
  -> DelayStats (nightly per-airline-per-route-per-DOW)
    -> RouteStats + AirlineRouteStats (nightly recompute)
      -> Graph (24 h in-memory cache)
        -> Candidate BFS (≤ 2 stops, 2× viability filter)
          -> Leg durations + hub layovers
          -> Three-bucket ranking
            -> /routes/indirect response

Every layer already existed. The routes feature cost zero new ingestion work.

The graph

The in-memory graph lives in api/src/graph/build.js. It's a two-level adjacency structure:

// edges: Map<origin, Map<dest, airlineIata[]>>

Built once per process, cached 24 hours. Each airport gets a hubScore — the sum of its incoming and outgoing route counts. That number drives minimum connection time later.

The graph rebuilds nightly after AirlineRouteStats updates, so it never goes stale relative to the underlying data.

Candidate enumeration: the 2× viability filter

The BFS expands up to 2 stops. The question is which of the potentially enormous candidate set to keep.

The filter is a single line:

function isViable(totalKm, directKm, factor) {
  if (directKm <= 0) return true;
  return totalKm <= directKm * factor;  // factor = 2.0
}

Any path whose total great-circle distance exceeds twice the direct distance is dropped immediately. This eliminates the LHR → JFK via NRT class of nonsense before any scoring happens — and it handles ~99% of pathological candidates.

Within each depth level, we keep the top 20 candidates by aggregated flight frequency, then return the shallowest non-empty level. A direct flight will always beat a 1-stop. A 1-stop will always beat a 2-stop.

Leg durations: physical plausibility filtering

For each leg, we compute the median observed flight time from the last 30 days of Flight records. But raw scraped durations are noisy — airport systems sometimes emit 00:00 or obviously wrong values.

The fix is an airspeed window:

const MIN_SPEED_KMH = 150;
const MAX_SPEED_KMH = 1100;

function isPlausible(durationMinutes, distanceKm) {
  const impliedSpeed = (distanceKm / durationMinutes) * 60;
  return impliedSpeed >= MIN_SPEED_KMH && impliedSpeed <= MAX_SPEED_KMH;
}

Anything outside 150–1100 km/h is discarded as a scrape artefact. A flight slower than a fast car or faster than a commercial jet can't be real. This is cleaner than maintaining a per-airline median lookup table.

Hub layovers: matching real arrival/departure pairs

This is the part I'm most pleased with.

The naive approach is to add a fixed layover estimate — 90 minutes, say. The problem is that 90 minutes at Frankfurt is different from 90 minutes at a small regional airport, and neither is the same as an overnight connection in Singapore.

Instead, we match real pairs:

Pull all arrivals at the hub from the origin airport (last 30 days).
Pull all departures from the hub to the next-leg destination (last 30 days).
For each arrival, find the closest departure that respects the minimum connection time. Allow next-day pairings.
Median the gaps. Track overnightPct.

// Minimum connection time is hub-score-driven
function getMCT(hubScore) {
  if (hubScore > 500) return 90;   // major international hub
  if (hubScore > 100) return 60;   // mid-sized hub
  return 45;                        // smaller airport
}

If fewer than 5 pairs match (new route, sparse data), we fall back to MCT + 30 min and mark the result estimated: true. The UI surfaces that flag so users know the layover is a model estimate, not observed data.

The overnight detection falls out for free: if ≥50% of matched pairs cross midnight, the card gets an overnight badge.

Reliability: geometric mean, not arithmetic

Each leg has an on-time percentage from AirlineRouteStats. The combined reliability for a multi-leg itinerary is:

combined = (Π onTimePct_i / 100)^(1/N) * 100

A 95% first leg and a 60% second leg gives 75.5%, not 77.5%.

This is the correct model for passengers. An itinerary is only as reliable as its weakest leg — a late arrival on leg 1 threatens the connection on leg 2. Arithmetic mean treats reliability as independent; geometric mean treats it as compounding. The difference matters most on long hauls with tight connections.

Legs without sufficient sample data are skipped (not zeroed). Zeroing them would unfairly penalise routes with new airlines.

Ticket type detection

For each candidate itinerary, we intersect the airline sets across all legs:

const legAirlines = legs.map(leg => new Set(leg.airlineIatas));
const intersection = legAirlines.reduce((a, b) => new Set([...a].filter(x => b.has(x))));

const ticketType = intersection.size > 0 ? 'sameAirline' : 'selfTransfer';

If the intersection is non-empty, a single-ticket booking is possible. The card shows a "Same airline" badge. This is a meaningful signal for passengers — a single PNR means the airline is responsible for the connection; a self-transfer means you're on your own if leg 1 is late.

The three buckets

The ranker (api/src/graph/ranker.js) produces exactly three outputs:

Fastest — minimise flightMinutes + layoverMinutes
Lowest CO₂ — minimise totalDistanceKm (CO₂ is derived from distance per passenger, per ICAO methodology)
Most reliable — maximise combinedOnTimePct

The same itinerary can win multiple buckets. In practice that's rare — the fastest path is usually not the shortest, and the most reliable is usually neither.

There's one tiebreaker: within 10% of the bucket's winning metric, prefer sameAirline: true over self-transfer. This only fires when two options are essentially equivalent — it never overrides a meaningfully better route.

CO₂: distance is a reasonable proxy

CO₂ per passenger varies by aircraft type, load factor, cabin class, and whether you count radiative forcing. We don't have that granularity — we have distance and passenger counts.

The proxy we use is total route distance (sum of all legs). This correlates strongly with actual emissions for the comparison use case: a 4,000 km itinerary via one hub emits less than a 7,000 km itinerary via two. The absolute number is an estimate; the relative ranking is reliable.

Future improvement: when we have aircraft type data per route, we can use the ICAO carbon calculator's per-type coefficients. For now, shorter is greener.

What surprised me

The overnight detection rate is higher than expected. On intra-Asian routes and certain long-haul pairings, a meaningful fraction of practical connections require an overnight stay at the hub. The flag matters — nobody wants to book what looks like a 6-hour trip and discover it's actually 18 hours with a hotel stop.

The viability filter is doing more work than anticipated. Without it, BFS over a 1,300-airport graph at 2 stops produces thousands of candidates per query. With the 2× cap, it drops to tens. The entire indirect query runs under 50ms even with the layover matching.

And the geometric mean genuinely changes rankings. There are routes where the "obvious" hub (higher on-time per leg) loses to a less-used hub because one of its legs has a 55% on-time rate that drags the geometric mean below what an arithmetic average would show.

What's next

The direct route page (per-route delay stats + airline comparison) is live but Pro-tier only. The indirect routes explorer is public. The obvious next steps are:

Aircraft type integration for more accurate CO₂ estimates
Full status timeline (not just on-time%, but delay distribution) for the reliability bucket
Price integration — the fourth bucket nobody asked for but everyone wants

The Routes API is available at myairports.online/developers. GET /routes/indirect?from=LHR&to=BKK will give you a sense of what the output looks like. Free tier: 100 requests/day, no card required.

What 1,000 Airports Reveal About the World's Airlines

José Catalá — Tue, 21 Apr 2026 14:26:24 +0000

We just shipped the Airlines tab on MyAirports. You pick any airport, click Airlines, and you get every carrier that operates there: name, IATA and ICAO codes, and hub airports.

Building it meant aggregating airline profile data across more than 1,000 airports. And like every other time I've done large-scale aggregation on this dataset, the results were not what I expected.

Here's what the data actually looks like.

643 airlines. Most of them you've never heard of.

Across 1,000+ airports, the MyAirports dataset contains 643 distinct airlines (normalized to IATA codes).

The top 10 appear at 200+ airports each. Ryanair leads in Europe, showing up at 230 airports. American, Delta, and United between them cover nearly every U.S. airport in the dataset. Emirates appears at 89 airports across 6 continents.

Below the top 50, the curve drops fast.

Airlines by airport coverage (unique airports served):

200+  airports:   10 airlines   (~1.5%)
50–199 airports:  43 airlines   (~6.7%)
10–49  airports:  128 airlines  (~20%)
2–9    airports:  241 airlines  (~37%)
1      airport:   221 airlines  (~34%)

More than a third of airlines in the dataset serve exactly one airport. These are ultra-regional carriers, charter operators, and cargo airlines whose entire operation lives at a single hub. They have real IATA codes. They have real flights. Most people who've never missed a connection in Ulaanbaatar have never encountered them.

The long tail is long.

Hub concentration is more extreme than you'd think

For any given airport, I was curious: how concentrated is traffic around the dominant carrier?

I computed a simple metric — the share of tracked airlines at each airport that belong to the single largest carrier group (including its regional subsidiaries and codeshare partners). The results:

Median airport: top carrier accounts for 42% of airlines serving it
Small regional airports (< 5 airlines): top carrier accounts for 78% on average
Major hubs (20+ airlines): top carrier accounts for 31% on average

The extreme end is striking. There are 89 airports in the dataset where a single airline accounts for 100% of observed carriers — monopoly airports where one carrier operates every tracked flight. Some are unsurprising (remote island airports with a single national carrier), but several are medium-sized regional airports in competitive markets that happen to be completely dominated by one low-cost carrier.

The other end is also interesting. The most diversified airport in the dataset — by carrier count — is Frankfurt (FRA) with 113 distinct airlines. London Heathrow (LHR) is second at 108. But even at FRA, the top 5 carriers account for 51% of all flights. Diversity of presence doesn't mean diversity of volume.

The IATA/ICAO code problem is worse than I thought

Every airline has (or should have) two codes: a 2-letter IATA code (the one on your boarding pass) and a 3-letter ICAO code (used in ATC and flight plans). In practice, the data is messier.

From the 643 airlines in the dataset:

71% have both a valid IATA and ICAO code
18% have ICAO only — they appear in airport board data encoded in ICAO format, but have no IATA mapping in any public reference
8% have IATA only — common for charter and wet-lease carriers that never filed ICAO designators
3% have neither — codes that appear in scraped data but don't match any known airline in any reference database

That last bucket — 19 airlines with unrecognized codes — is the interesting one. Some are historical: airlines that merged or went bankrupt but whose codes still appear in airport systems referencing old bookings. Some are data errors in the source airport systems (a transposed character, a regional variant). And a small number are genuinely mysterious — codes used consistently across multiple airports that don't map to any known carrier.

The normalization pipeline handles this with a fallback chain:

// airline-resolver.js

async function resolveAirline(rawCode) {
  // 1. Try IATA direct lookup
  const byIata = await db.airline.findFirst({ where: { iataCode: rawCode.toUpperCase() } });
  if (byIata) return { ...byIata, matchType: 'iata' };

  // 2. Try ICAO lookup, return normalized IATA
  const byIcao = await db.airline.findFirst({ where: { icaoCode: rawCode.toUpperCase() } });
  if (byIcao) return { ...byIcao, matchType: 'icao_normalized' };

  // 3. Try fuzzy name match (for display-name variants like "RYANAIR DAC")
  const byName = await db.airline.findFirst({
    where: { name: { contains: rawCode, mode: 'insensitive' } }
  });
  if (byName) return { ...byName, matchType: 'name_fuzzy' };

  // 4. Unknown — flag for review
  await db.unknownCode.upsert({
    where: { code: rawCode },
    update: { seenCount: { increment: 1 } },
    create: { code: rawCode, seenCount: 1, firstSeen: new Date() }
  });

  return null;
}

The unknownCode table is how I discovered the 19 mystery airlines — they kept accumulating count increments from multiple airports, which meant they weren't random noise.

Low-cost vs legacy: different geographies, different patterns

One thing that stands out when you plot airline profiles by region: the low-cost carrier (LCC) story is almost entirely a European and Southeast Asian phenomenon in the data.

In Europe, LCCs (Ryanair, easyJet, Wizz Air, Vueling) appear at a combined 680+ airports. Legacy carriers (Lufthansa, British Airways, Air France, KLM) appear at fewer than 200 combined — concentrated at major hubs and secondary capitals.

In North America, the pattern inverts. The three legacy mega-carriers (American, Delta, United) appear at more airports than all LCCs combined. Southwest is the exception — it appears at 97 airports, more than any other North American carrier — but it's almost the only LCC with that kind of geographic reach on the continent.

Asia-Pacific is the most fragmented. There are 143 distinct carriers across the region's airports in the dataset, with no single carrier appearing at more than 60. The region has both the densest LCC ecosystem (AirAsia alone spawned 9 separate national subsidiaries) and the most state-owned legacy carriers that haven't meaningfully faced low-cost competition.

What this means for the Airlines tab: the "typical" airport looks very different depending on geography. A European regional airport's airline list looks like a Ryanair schedule. A U.S. regional airport's looks like a United Express feed. An Asian secondary city airport might have 4 carriers you've never heard of in the West.

The subsidiary problem

One thing I had to decide early: do code-share and regional subsidiaries count as separate airlines?

My answer: yes, always. Lufthansa and Lufthansa CityLine are different IATA codes (LH and CL), different fleets, different crew contracts, and separate delay profiles. Merging them would make the airline-level stats meaningless.

But this creates a presentation problem. When you look at the Airlines tab for Frankfurt Airport, you see:

Lufthansa (LH)
Lufthansa CityLine (CL)
Lufthansa Cargo (LH — same code, different entity)
Air Dolomiti (EN — Lufthansa Group)
Swiss (LX — Lufthansa Group)
Austrian (OS — Lufthansa Group)
Brussels Airlines (SN — Lufthansa Group)

Seven distinct carriers with seven distinct operational profiles, all presenting as "Lufthansa" to most passengers. The Airlines tab surfaces them as separate entries because that's what the data is. But the group field on the airline object lets callers roll them up if they want:

{
  "iataCode": "CL",
  "name": "Lufthansa CityLine",
  "icaoCode": "CLH",
  "group": "Lufthansa Group",
  "hubAirports": ["MUC", "FRA"],
  "type": "regional"
}

If you call the airlines endpoint and group by group, you get the commercial story. If you don't, you get the operational truth. Both are useful depending on what you're building.

What's actually in the Airlines tab

Each airline profile in the tab (and in the API) includes:

iataCode / icaoCode: normalized identifiers
name: canonical name (English)
type: full_service, low_cost, regional, charter, or cargo
group: parent group name if applicable (e.g., "IAG", "Lufthansa Group", "Air France-KLM")
hubAirports: array of IATA codes for the carrier's primary hub(s)
country: ISO country code of the carrier's home country

The airlines endpoint is available at:

GET https://myairports.online/api/v1/airports/{iata}/airlines

It returns all carriers currently operating at that airport based on recent flight board data — not a static schedule, but live-derived from actual observed flights over the past 30 days.

One thing I didn't expect

The single most surprising finding from building this: hub airports are not where most airlines are based.

Of the 643 airlines in the dataset, only 38 list a major international hub (top 50 airports globally) as their sole hub. The majority are based at smaller airports — often the only major carrier at a regional airport — and operate point-to-point or spoke-to-hub routes.

The implication: when you're looking at a big airport's airline list, you're mostly seeing visitors — carriers whose operational home is somewhere else, routing passengers through. The airport's "own" airline, if it has one at all, is often the one with the smallest presence there.

Frankfurt is a Lufthansa hub, but Lufthansa Group carriers account for 38% of the Airlines tab entries. The other 62% are carriers for whom FRA is a stopover, a codeshare point, or a cargo hub.

The big airport is everyone's guest house. The airline data makes that visible.

The Airlines tab is live at myairports.online — search any airport and click Airlines. The underlying data is available via the API at myairports.online/developers. Free tier: 100 requests/day, no card required.

How I Built a Delay Risk Score for 628,000 Flights

José Catalá — Mon, 20 Apr 2026 15:22:40 +0000

Every flight in the MyAirports API includes a delayRisk object. It looks like this:

{
  "flightNumber": "VY1234",
  "scheduledTime": "2026-04-21T07:15:00Z",
  "status": "scheduled",
  "insights": {
    "delayRisk": {
      "onTimePct": 61,
      "avgDelayMinutes": 28,
      "cancelledPct": 3.2,
      "sample": 89,
      "context": "VY MAD→BCN, Mon morning"
    }
  }
}

Three numbers. Looks simple. It's not.

Getting from raw scraped flight data to a reliable, per-flight delay prediction involved building a statistics pipeline I didn't know I needed — and making a series of judgment calls that turned out to matter a lot.

Here's how it works.

The core insight: route × time-of-day × day-of-week

A delay risk score is only meaningful if it compares like with like.

A Monday 07:00 Vueling flight from Madrid to Barcelona has a very different delay profile than a Friday 19:00 Vueling flight on the same route. Aggregating them together produces a number that's wrong for both.

The bucketing key I landed on is:

route (origin → dest) × airline × departure_hour_bucket × day_of_week

Where departure_hour_bucket is a 3-hour window: 00–03, 03–06, 06–09, 09–12, 12–15, 15–18, 18–21, 21–24.

This gives enough granularity to capture meaningful patterns without fragmenting the sample so much that each bucket has too few flights to be reliable.

The delay_stats table stores one row per bucket:

CREATE TABLE delay_stats (
  id           SERIAL PRIMARY KEY,
  origin_iata  CHAR(3)  NOT NULL,
  dest_iata    CHAR(3)  NOT NULL,
  airline_code VARCHAR(3) NOT NULL,
  hour_bucket  SMALLINT NOT NULL,  -- 0, 3, 6, 9, 12, 15, 18, 21
  day_of_week  SMALLINT NOT NULL,  -- 0=Sun, 6=Sat
  sample_size  INTEGER  NOT NULL,
  on_time_pct  NUMERIC(5,2),
  avg_delay_min NUMERIC(6,1),
  cancelled_pct NUMERIC(5,2),
  updated_at   TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE (origin_iata, dest_iata, airline_code, hour_bucket, day_of_week)
);

What counts as "on time"

Before computing percentages, I had to define what "on time" actually means. The industry standard is within 15 minutes of scheduled departure. That's what I use.

// insights/analyzers/delay-stats.js

const ON_TIME_THRESHOLD_MINUTES = 15;

function classifyFlight(flight) {
  if (flight.status === 'cancelled') return 'cancelled';

  if (!flight.actualDepartureTime || !flight.scheduledTime) return null; // insufficient data

  const delayMinutes = (
    new Date(flight.actualDepartureTime) - new Date(flight.scheduledTime)
  ) / (1000 * 60);

  return {
    isOnTime: delayMinutes <= ON_TIME_THRESHOLD_MINUTES,
    delayMinutes: Math.max(0, delayMinutes) // negative = early, treat as 0
  };
}

A flight that departs 5 minutes early counts as on time (delay = 0). A flight that departs 16 minutes late is delayed. Cancelled flights go into their own bucket.

One thing that tripped me up early: I was computing delay as actual - scheduled and sometimes getting negative numbers because flights occasionally depart ahead of schedule. Treating those as 0-minute delays (rather than negative delays) keeps the average meaningful.

The pipeline: from raw flights to stats

The stats computation runs nightly as a batch job. It scans all completed flights from the previous 90 days and recomputes each bucket from scratch.

// insights/run-stats.js

async function computeDelayStats() {
  const cutoff = new Date(Date.now() - 90 * 24 * 60 * 60 * 1000);

  // Pull all completed flights with actual departure times
  const flights = await prisma.flight.findMany({
    where: {
      status: { in: ['departed', 'arrived', 'cancelled'] },
      scheduledTime: { gte: cutoff },
    },
    select: {
      originIata: true,
      destIata: true,
      airlineCode: true,
      scheduledTime: true,
      actualDepartureTime: true,
      status: true,
    }
  });

  // Group into buckets
  const buckets = new Map();

  for (const flight of flights) {
    const key = getBucketKey(flight);
    if (!key) continue;

    if (!buckets.has(key)) buckets.set(key, []);
    buckets.get(key).push(flight);
  }

  // Compute stats per bucket and upsert
  for (const [key, bucketFlights] of buckets) {
    const stats = computeBucketStats(bucketFlights);
    if (!stats) continue; // insufficient sample

    const { origin, dest, airline, hourBucket, dayOfWeek } = parseKey(key);

    await prisma.delayStat.upsert({
      where: { origin_dest_airline_hour_day: { 
        originIata: origin, destIata: dest, 
        airlineCode: airline, hourBucket, dayOfWeek 
      }},
      update: { ...stats, updatedAt: new Date() },
      create: { originIata: origin, destIata: dest, 
                airlineCode: airline, hourBucket, dayOfWeek, ...stats }
    });
  }
}

function getBucketKey(flight) {
  if (!flight.originIata || !flight.destIata || !flight.airlineCode) return null;

  const d = new Date(flight.scheduledTime);
  const hour = d.getUTCHours();
  const hourBucket = Math.floor(hour / 3) * 3; // round down to 0,3,6,9...
  const dayOfWeek = d.getUTCDay();

  return `${flight.originIata}:${flight.destIata}:${flight.airlineCode}:${hourBucket}:${dayOfWeek}`;
}

Computing the stats for a bucket

Once flights are grouped, computing the three metrics is straightforward:

const MIN_SAMPLE = 30; // minimum flights required for a reliable stat

function computeBucketStats(flights) {
  if (flights.length < MIN_SAMPLE) return null;

  const cancelled = flights.filter(f => f.status === 'cancelled');
  const departed = flights.filter(f => f.status !== 'cancelled' && f.actualDepartureTime);

  // cancelledPct includes all flights in the bucket denominator
  const cancelledPct = (cancelled.length / flights.length) * 100;

  if (departed.length === 0) {
    return { sampleSize: flights.length, onTimePct: null, avgDelayMin: null, cancelledPct };
  }

  const delays = departed.map(f => {
    const delayMs = new Date(f.actualDepartureTime) - new Date(f.scheduledTime);
    return Math.max(0, delayMs / (1000 * 60));
  });

  const onTimeCount = delays.filter(d => d <= ON_TIME_THRESHOLD_MINUTES).length;
  const onTimePct = (onTimeCount / departed.length) * 100;
  const avgDelayMin = delays.reduce((sum, d) => sum + d, 0) / delays.length;

  return {
    sampleSize: flights.length,
    onTimePct: Math.round(onTimePct * 100) / 100,
    avgDelayMin: Math.round(avgDelayMin * 10) / 10,
    cancelledPct: Math.round(cancelledPct * 100) / 100,
  };
}

The MIN_SAMPLE = 30 threshold is important. Below 30 flights, the percentages are too noisy to be useful — a single bad week skews everything. I return null for under-sampled buckets, and the API reflects that with a lower-confidence indicator.

The minimum sample problem

Thirty flights sounds easy, but it's not for thin routes. A regional airline operating 3 flights per week on a niche route will accumulate 30 samples in about 10 weeks — and if we're only keeping 90 days of history, we have a rolling window of roughly 36 observations at best.

For these routes I use a fallback hierarchy:

async function getDelayStats(origin, dest, airline, hourBucket, dayOfWeek) {
  // 1. Exact match: specific airline on this route/time
  let stats = await findStats({ origin, dest, airline, hourBucket, dayOfWeek });
  if (stats && stats.sampleSize >= MIN_SAMPLE) return { ...stats, confidence: 'high' };

  // 2. Any airline on this route/time
  stats = await findStats({ origin, dest, airline: null, hourBucket, dayOfWeek });
  if (stats && stats.sampleSize >= MIN_SAMPLE) return { ...stats, confidence: 'medium' };

  // 3. Any airline on this route, any time in this day bucket (morning/afternoon/evening)
  const dayPart = hourBucket < 12 ? 'morning' : hourBucket < 18 ? 'afternoon' : 'evening';
  stats = await findStats({ origin, dest, airline: null, dayPart, dayOfWeek });
  if (stats && stats.sampleSize >= MIN_SAMPLE) return { ...stats, confidence: 'low' };

  // 4. No data
  return null;
}

The confidence field surfaces in the API response so callers can decide how much weight to give the number. A high confidence onTimePct: 61 means something; a low confidence one is a rough prior at best.

Handling airline codes: the IATA/ICAO mess

Flight data comes from airport boards in different formats. Some airports use IATA airline codes (2-letter: IB, VY, FR), some use ICAO (3-letter: IBE, VLG, RYR), and some mix them depending on the flight.

The stats pipeline normalizes everything to IATA codes before bucketing. If a flight arrives with ICAO code IBE, it gets mapped to IB before the bucket key is computed:

// airline-normalizer.js
const ICAO_TO_IATA = {
  'IBE': 'IB',  // Iberia
  'VLG': 'VY',  // Vueling
  'RYR': 'FR',  // Ryanair
  'BAW': 'BA',  // British Airways
  'DLH': 'LH',  // Lufthansa
  // ... ~400 more
};

function normalizeAirlineCode(code) {
  if (!code) return null;
  if (code.length === 2) return code.toUpperCase(); // already IATA
  return ICAO_TO_IATA[code.toUpperCase()] ?? null;  // map or drop
}

Unknown airline codes get dropped (returned as null), which means those flights still feed into the route-level stats (fallback level 2) but not the airline-specific stats (level 1).

What the numbers actually look like

After running the pipeline across all accumulated data, some patterns stand out.

The morning variance is real. The 06–09 bucket has the highest avgDelayMin of any time window — roughly 1.4x the all-day average — but also the second-highest onTimePct. What's happening: most early morning flights are fine, but when they go wrong, they go badly wrong. The distribution is bimodal: short delays almost never, then a long tail of cascaded delays from overnight issues.

Fridays are not the worst day. Sunday evenings are. Flights completing weekend itineraries face the worst on-time performance across almost every route in the dataset. Monday mornings are second worst.

Low-cost carrier routes have lower onTimePct but lower avgDelayMin. Budget airlines cancel less (they have fewer connectivity options, so they push through more delays) but are more frequently slightly late. Legacy carriers cancel more but have tighter average delay windows when they do depart.

The 90-day rolling window matters more than I expected. Seasonal patterns mean that data from January is largely irrelevant for predicting August performance. I may eventually shrink the window to 60 days and weight recent flights more heavily.

Surfacing it in the API

When a flight is requested, the stats lookup runs against the delay_stats table:

// api/flights.js
async function enrichWithInsights(flight) {
  const d = new Date(flight.scheduledTime);
  const hourBucket = Math.floor(d.getUTCHours() / 3) * 3;
  const dayOfWeek = d.getUTCDay();
  const airline = normalizeAirlineCode(flight.airlineCode);

  const stats = await getDelayStats(
    flight.originIata,
    flight.destIata,
    airline,
    hourBucket,
    dayOfWeek
  );

  if (!stats) return { delayRisk: null };

  return {
    delayRisk: {
      onTimePct: stats.onTimePct !== null ? Math.round(stats.onTimePct) : null,
      avgDelayMinutes: stats.avgDelayMin !== null ? Math.round(stats.avgDelayMin) : null,
      cancelledPct: stats.cancelledPct !== null ? Math.round(stats.cancelledPct * 10) / 10 : null,
      sample: stats.sampleSize,
      confidence: stats.confidence,
      context: buildContextString(flight, dayOfWeek, hourBucket),
    }
  };
}

function buildContextString(flight, dayOfWeek, hourBucket) {
  const days = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'];
  const timeLabel = hourBucket < 12 ? 'morning' : hourBucket < 18 ? 'afternoon' : 'evening';
  return `${flight.airlineCode} ${flight.originIata}→${flight.destIata}, ${days[dayOfWeek]} ${timeLabel}`;
}

The context string makes the number interpretable at a glance — you see immediately that onTimePct: 61 applies to "VY MAD→BCN, Mon morning" rather than to Vueling in general.

What I'd do differently

Weight by recency. All flights in the 90-day window currently have equal weight. A flight from 85 days ago contributes equally to the stats as one from yesterday. Exponential decay weighting — where more recent flights count more — would make the score respond faster to airline operational changes.

Track the delay distribution, not just the average. avgDelayMinutes: 28 tells you the mean, but it hides a lot. A route with a 28-minute average could be "almost always 25–30 minutes late" or "usually on time but occasionally 4 hours late." The latter is much worse for connecting passengers. Storing the 90th percentile delay alongside the average would be far more useful.

Separate departure and arrival delay. Right now everything is departure-delay. But passengers care about arrival delay. A flight that departs 30 minutes late and lands 10 minutes late (because of a fast cruise and short taxi) is very different from one that lands 35 minutes late. Airport boards rarely surface arrival delay in real time, but when they do, it's worth tracking separately.

Per-aircraft-type stats. Narrow-body short-haul routes flown by a mix of A320s and B737s have meaningfully different delay profiles depending on the day's specific aircraft assignment. I don't have aircraft type data yet, but it's on the roadmap.

The delay risk score started as a single percentage. It's now a layered lookup across four levels of fallback, a batch pipeline that runs every night, and the most-queried field in the API. Getting it right was mostly a series of small decisions — the 15-minute threshold, the 3-hour buckets, the minimum sample of 30 — each of which turned out to matter more than I expected.

The full API, including delayRisk on every flight, is at myairports.online/developers. Free tier: 100 requests/day, no card required.

What 628,000 Flights Taught Me About Cancellations (And Why They're Harder to Track Than You Think)

José Catalá — Sat, 18 Apr 2026 19:26:45 +0000

Flight cancellations look simple from the outside: a flight was scheduled, it didn't happen, it shows "Cancelled." Done.

After processing 628,000+ flights across 1,300 airports, I can tell you it's not that simple. Cancellations are one of the messiest problems in airport data — and understanding them properly forced me to build detection infrastructure I didn't know I needed.

Here's what I learned.

The three types of cancellation (that most people treat as one)

Not all cancellations are the same, and the differences matter for both the traveller and the data pipeline.

Early cancellation: The airline cancels the flight hours or days before departure and updates the schedule. The flight board never shows it. From a scraping perspective, this flight simply doesn't appear. You can't detect it by watching the board — you'd need to compare against the original schedule, which you may not have.

Late cancellation: The flight is on the board with status "Scheduled" or "On time" and then, sometimes within 30 minutes of departure, flips to "Cancelled." This is the one passengers hate most, and it's the one your data pipeline needs to actively detect.

Ghost cancellation: The flight disappears from the board entirely without ever showing "Cancelled." The row was there in the previous scrape, it's gone now, and there's no explicit status change. This is the hardest one.

MyAirports encounters all three, and each needs a different detection strategy.

Ghost flights: the cancellation nobody tells you about

This was the most surprising thing I learned. A non-trivial number of airport websites simply remove a cancelled flight from their board rather than marking it as cancelled. It just vanishes.

If your scraper only looks at the current board state, you'll never see these cancellations. They happened, but the evidence is gone.

The fix is snapshot comparison:

// cancellation-detector.js
async function detectGhostCancellations(iata, direction) {
  const now = new Date();
  const windowMinutes = 90;

  // Get what we saw in the last scrape
  const prevScrape = await db.flight.findMany({
    where: {
      airportIata: iata,
      direction,
      lastSeen: { lt: new Date(now - 15 * 60 * 1000) }, // older than 15 min
      scheduledTime: {
        gte: now,
        lt: new Date(now.getTime() + windowMinutes * 60 * 1000)
      },
      status: { not: 'arrived' }
    }
  });

  // Get what's on the board now
  const currentBoard = await scrapeAirport(iata, direction);
  const currentFlightNumbers = new Set(currentBoard.map(f => f.flightNumber));

  // Anything that was there and is now gone
  const disappeared = prevScrape.filter(f => 
    !currentFlightNumbers.has(f.flightNumber) &&
    f.status !== 'cancelled' // not already marked
  );

  return disappeared.map(f => ({
    ...f,
    status: 'cancelled',
    cancellationReason: 'ghost', // disappeared without notice
    detectedAt: now
  }));
}

The window matters. A flight that departed won't show on the board either — you only want to flag flights that disappeared before their scheduled time. The 15-minute "last seen" buffer prevents false positives from temporary scrape failures.

Normalizing "Cancelled" across 30+ languages

Article 3 in this series covered status normalization in general. Cancellations specifically deserve attention because the variation is extreme.

A cancelled flight might show as:

Language	Status string
English	`Cancelled`, `Canceled`, `CNCL`, `CNX`, `CXL`
Spanish	`Cancelado`, `Anulado`, `CANCELADO`
German	`Gestrichen`, `Annulliert`, `Abgesagt`
French	`Annulé`, `Supprimé`
Italian	`Cancellato`, `Soppresso`
Portuguese	`Cancelado`, `Suprimido`
Russian	`Отменён`, `Отменен`
Arabic	`ملغي`, `ملغاة`
Turkish	`İptal`, `İptal Edildi`
Norwegian	`Kansellert`, `Innstilt`

And then there are airline-specific codes with no natural language at all: CNX, CXL, CNCL, XLD, CX. Some airports emit only the code without any label.

The normalizer handles this in layers:

// status-normalizer.js

const CANCELLED_EXACT = new Set([
  // English
  'cancelled', 'canceled', 'cancellation', 'cncl', 'cnx', 'cxl', 'xld', 'cx',
  // Spanish
  'cancelado', 'anulado',
  // German
  'gestrichen', 'annulliert', 'abgesagt',
  // French
  'annulé', 'annule', 'supprimé', 'supprime',
  // Italian
  'cancellato', 'soppresso',
  // Portuguese
  'cancelado', 'suprimido',
  // Dutch
  'geannuleerd', 'geannuleerd vlucht',
  // Norwegian/Swedish/Danish
  'kansellert', 'innstilt', 'inställd', 'aflyst',
  // Finnish
  'peruutettu',
  // Russian
  'отменён', 'отменен', 'отменен рейс',
  // Turkish
  'iptal', 'iptal edildi',
  // Arabic (normalized, RTL stripped)
  'ملغي', 'ملغاة',
  // Chinese
  '取消', '已取消',
]);

const CANCELLED_CONTAINS = [
  'cancel', 'annul', 'gestr', 'suprim', 'peruut', 'innstil',
];

function normalizeToCancelled(raw) {
  const s = raw.toLowerCase().trim().replace(/[^\p{L}\p{N}\s]/gu, '');

  if (CANCELLED_EXACT.has(s)) return 'cancelled';
  if (CANCELLED_CONTAINS.some(pat => s.includes(pat))) return 'cancelled';

  return null; // not a cancellation
}

The replace(/[^\p{L}\p{N}\s]/gu, '') strips punctuation and special characters using Unicode property escapes — critical for Arabic and Chinese status strings that sometimes arrive with RTL markers or extra punctuation.

Tracking cancellations over time: the `flights` table

Every scraped flight goes into the flights table. When a cancellation is detected — either from an explicit status change or a ghost detection — the flight record gets updated:

// persistence.js
async function saveFlights(iata, direction, flights) {
  for (const flight of flights) {
    await prisma.flight.upsert({
      where: { flightNumber_scheduledTime_airportIata: {
        flightNumber: flight.flightNumber,
        scheduledTime: flight.scheduledTime,
        airportIata: iata,
      }},
      update: {
        status: flight.status,
        lastSeen: new Date(),
        ...(flight.status === 'cancelled' && {
          cancelledAt: new Date(),
          cancellationReason: flight.cancellationReason || 'explicit',
        }),
      },
      create: {
        ...flight,
        firstSeen: new Date(),
        lastSeen: new Date(),
      }
    });
  }
}

The cancelledAt timestamp and cancellationReason fields are what make the historical analysis possible later.

Mass cancellation detection: weather, strikes, and cascades

Individual cancellations are noise. Mass cancellations are a signal worth surfacing.

When an airport has 10+ cancellations in a short window, something structural is happening: weather, strike action, airport closure, ground stop, or an airline operational meltdown. The data pattern is distinctive:

// anomaly-detector.js
async function detectMassCancellations(iata, windowHours = 2) {
  const since = new Date(Date.now() - windowHours * 60 * 60 * 1000);

  const recentCancellations = await prisma.flight.count({
    where: {
      airportIata: iata,
      status: 'cancelled',
      cancelledAt: { gte: since }
    }
  });

  const baseline = await getBaselineCancellationRate(iata, windowHours);

  if (recentCancellations > baseline * 3) {
    return {
      type: 'mass_cancellation',
      airport: iata,
      count: recentCancellations,
      baseline,
      deviationFactor: recentCancellations / baseline,
      detectedAt: new Date()
    };
  }

  return null;
}

async function getBaselineCancellationRate(iata, windowHours) {
  const dayOfWeek = new Date().getDay();

  const result = await prisma.$queryRaw`
    SELECT AVG(daily_count) as avg_count
    FROM (
      SELECT DATE(cancelled_at) as day, COUNT(*) as daily_count
      FROM flights
      WHERE airport_iata = ${iata}
        AND status = 'cancelled'
        AND EXTRACT(DOW FROM cancelled_at) = ${dayOfWeek}
        AND cancelled_at > NOW() - INTERVAL '30 days'
      GROUP BY DATE(cancelled_at)
    ) daily
  `;

  return result[0]?.avg_count ?? 1;
}

The 3x deviation threshold is empirical — below that and you get false positives from normal day-to-day variation. Above 3x is consistently meaningful.

What the data actually shows

After accumulating 628,000+ flights, the cancellation patterns are clear:

Cancellation rate by time of day — Early morning flights (05:00–07:00) cancel at roughly 1.2x the rate of afternoon flights. Not because airlines plan worse in the morning, but because cascades: a late-arriving aircraft the night before causes the early morning departure to cancel, and the delay propagates forward through the day.

Cancellation rate by day of week — Mondays and Fridays have the highest cancellation rates, roughly 1.4x midweek. This matches the business travel calendar: high demand creates less buffer for aircraft rotations.

The "cancellation cascade" pattern — When we detect a cancellation on a route (e.g., MAD→LHR at 06:30), the probability of a cancellation on the return (LHR→MAD, any time that day) jumps by roughly 40%. Same aircraft type, same crew. The cascade is real and detectable from the data alone.

Ghost cancellations by region — Ghost cancellations (flights that vanish without notice) are most common in South American and Southeast Asian airports — roughly 3x the rate of European and North American airports. The likely cause is older CMS software that just deletes cancelled entries rather than updating their status.

Surfacing this in the API

The delay_stats table in the insights engine stores cancelled percentage per route per time bucket:

// insights/analyzers/delay-stats.js (relevant section)
const cancelledPct = bucket.totalFlights > 0
  ? (bucket.cancelledCount / bucket.totalFlights) * 100
  : 0;

await prisma.delayStat.upsert({
  where: { /* composite key */ },
  update: { cancelledPct, /* other fields */ },
  create: { cancelledPct, /* other fields */ },
});

This feeds directly into the delayRisk field on every flight in the API response:

{
  "flightNumber": "IB3163",
  "scheduledTime": "2026-04-18T06:30:00Z",
  "status": "scheduled",
  "insights": {
    "delayRisk": {
      "onTimePct": 68,
      "avgDelayMinutes": 22,
      "cancelledPct": 4.7,
      "sample": 143,
      "context": "IB to LHR, Fri morning"
    }
  }
}

A cancelledPct of 4.7% on a Friday morning Iberia flight to London isn't alarming in isolation — but if you're booking travel insurance or a tight connection, it's exactly the number you want.

The EU261 connection

Under EU Regulation 261/2004, a cancellation notified fewer than 14 days before departure entitles the passenger to compensation: €250–€600 depending on route distance. The detection pipeline feeds directly into this.

When a flight cancels in our system with cancelledAt set to less than 14 days before scheduledTime, the EU261 eligibility flag gets set:

// eu261.js
function checkEu261Eligibility(flight) {
  if (flight.status !== 'cancelled') return null;

  const noticeHours = (flight.scheduledTime - flight.cancelledAt) / (1000 * 60 * 60);
  const noticeDays = noticeHours / 24;

  if (noticeDays >= 14) return { eligible: false, reason: 'advance_notice' };

  const distanceKm = getRouteDistance(flight.originIata, flight.destIata);

  return {
    eligible: true,
    compensationEur: distanceKm < 1500 ? 250 : distanceKm < 3500 ? 400 : 600,
    noticeDays: Math.round(noticeDays),
    distanceKm
  };
}

The catch: cancelledAt is when we detected the cancellation, not necessarily when the airline officially notified passengers. For ghost cancellations — the ones that disappeared without notice — we mark cancellationReason: 'ghost' and flag the EU261 check as needing manual verification.

What I'd do differently

Track the full status timeline. Right now, if a flight goes Scheduled → Delayed → Cancelled, I only have the final state and the timestamps. I should log every status transition. A flight that was delayed 4 times before finally cancelling tells a different story than one that cancelled cleanly.

Capture the board timestamp. Each scrape runs against live airport data, but the data is often cached on the airport side. I should record the "data as of" timestamp from the airport's API response, not just my scrape time.

Watch for reinstatements. Cancelled flights occasionally get reinstated — a replacement aircraft is found, the weather clears faster than expected. Right now, a reinstatement would just update the status from "cancelled" to "scheduled," but I don't surface this event explicitly.

Flight cancellations are a solved problem from the traveller's perspective — the airline's app tells you. But from the data perspective, they're a distributed, multilingual, inconsistent mess. The ghost cancellations alone took me three months to discover were happening.

The full cancellation API is available at myairports.online/developers. Free tier: 100 requests/day, no card required.

How I Automated Onboarding 1,300 Airport Websites Without Looking at a Single One

José Catalá — Thu, 16 Apr 2026 20:38:55 +0000

The original version of MyAirports required a developer (me) to open DevTools, find the network request carrying flight data, copy the URL, figure out the headers, and paste everything into a config file. One airport at a time.

That's fine for ten airports. It doesn't scale to a thousand.

So I built a pipeline that does it automatically: give it an IATA code, and it finds the airport's website, discovers the flight data API, detects the scraping strategy, and writes the config code. No DevTools. No manual inspection. Here's how it works.

The problem with airport websites

Every airport runs its own website. Some are operated by independent airport authorities. Some are run by national operators — Spain's AENA runs 46 airports on the same platform; Finland's Finavia runs 20. Some outsource to the same flight display software vendor.

This creates a few patterns worth detecting:

Operator groups: airports that share the same backend API. Find one, and you've implicitly found all of them.
Standalone airports: unique websites with unique APIs.
Vendor software: airports using products like AirportInfo or similar. Same HTML structure, same endpoints, different domain.
DOM-only: no discoverable API. Flight data is rendered server-side, so you scrape HTML.

The discovery pipeline needs to detect which pattern applies and generate the right config for each.

Step 1: Find the right URL

The first obstacle is simple: given an IATA code like ATH (Athens), what's the official airport website URL?

This sounds trivial but isn't. Search for "Athens airport" and you get aggregators — Skyscanner, TripAdvisor, Wikipedia — before you get the official site. The flight data lives on the official site, not on aggregators.

The solution is a purpose-built search query:

// searcher.js
async function findAirportUrl(iata, airportName, country) {
  const query = `"${airportName}" official airport website flight departures arrivals -skyscanner -tripadvisor -rome2rio`;

  const results = await serperSearch(query);
  return filterAggregators(results);
}

function filterAggregators(results) {
  const blocklist = [
    'skyscanner', 'tripadvisor', 'rome2rio', 'kayak', 'expedia',
    'flightaware', 'flightradar24', 'wikipedia', 'wikitravel',
    'airportia', 'flightsfrom', 'airports-worldwide',
    // 30+ more
  ];

  return results.filter(r => 
    !blocklist.some(blocked => r.url.includes(blocked))
  );
}

The blocklist has grown to 30+ domains over time — whenever a false positive slips through and breaks a discovery, the domain gets added. The negative search terms in the query help, but they're not reliable enough on their own.

For airports where Serper finds nothing useful, the pipeline falls back to in-browser DuckDuckGo search with the same filtering logic.

Step 2: Intercept network traffic

Once the URL is found, the pipeline launches a stealth Playwright browser and loads the page. Rather than looking at the HTML, it watches the network:

// interceptor.js
async function discoverApi(page) {
  const candidates = [];

  page.on('response', async (response) => {
    const url = response.url();
    const contentType = response.headers()['content-type'] || '';

    if (!contentType.includes('application/json')) return;

    try {
      const body = await response.json();
      const score = scoreResponse(body, url);
      if (score > 0) {
        candidates.push({ url, headers: response.request().headers(), score });
      }
    } catch (_) {}
  });

  await page.goto(targetUrl, { waitUntil: 'networkidle' });
  return candidates.sort((a, b) => b.score - a.score);
}

function scoreResponse(body, url) {
  let score = 0;
  const str = JSON.stringify(body).toLowerCase();

  if (str.includes('flight') || str.includes('flightnumber')) score += 3;
  if (str.includes('departure') || str.includes('arrival')) score += 3;
  if (str.includes('iata') || str.includes('icao')) score += 2;
  if (str.includes('status') || str.includes('gate')) score += 2;
  if (str.includes('airline') || str.includes('carrier')) score += 2;
  if (Array.isArray(body) && body.length > 0) score += 1;
  if (body.flights || body.departures || body.arrivals) score += 3;

  // URL indicators  
  if (url.includes('flight') || url.includes('fids')) score += 2;
  if (url.includes('departure') || url.includes('arrival')) score += 2;

  return score;
}

The highest-scoring response is the flight data API. The request headers that triggered it — including any session cookies, CSRF tokens, or API keys — get captured at the same time. One browser session is all it takes; from then on, the system calls the API directly over HTTP without a browser.

This is the core idea behind MyAirports: use a browser once to discover, then never again.

Step 3: WAF detection and FlareSolverr fallback

Before the browser even hits the target page, the pipeline checks for WAF protection:

// waf-detector.js
async function detectWaf(page) {
  const title = await page.title();
  const content = await page.content();

  if (title.includes('Just a moment') || content.includes('cf-browser-verification')) {
    return 'cloudflare';
  }
  if (content.includes('incapsula') || content.includes('visid_incap')) {
    return 'incapsula';
  }

  return null;
}

If Cloudflare is detected, the pipeline hands off to FlareSolverr — a Docker sidecar that runs a real browser, solves the JS challenge, and returns the cookies:

// flaresolverr.js
async function solveCloudflare(url) {
  const response = await fetch('http://localhost:8191/v1', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      cmd: 'request.get',
      url,
      maxTimeout: 60000
    })
  });

  const data = await response.json();
  return {
    cookies: data.solution.cookies,
    userAgent: data.solution.userAgent
  };
}

The solved cookies get stored in a Session table and reused until they expire, so the challenge-solving cost is paid once per airport, not on every scrape.

Step 4: Detect operator groups

Some airports share infrastructure. Spain's AENA airports all use the same API at aena.es/es/flight-search. Once you've discovered one AENA airport, you've got the pattern for all 46.

The group detector looks for these shared patterns:

// group-detector.js
const KNOWN_OPERATORS = [
  { name: 'AENA', urlPattern: /aena\.es/, airports: ['MAD', 'BCN', 'VLC', /* ... */] },
  { name: 'Finavia', urlPattern: /finavia\.fi/, airports: ['HEL', 'OUL', /* ... */] },
  { name: 'ADP', urlPattern: /parisaeroport\.fr/, airports: ['CDG', 'ORY'] },
];

function detectOperatorGroup(discoveredUrl) {
  for (const op of KNOWN_OPERATORS) {
    if (op.urlPattern.test(discoveredUrl)) return op;
  }
  return null;
}

When a group match is found, the code generator writes configs for all airports in the group from a single template.

Step 5: AI pattern learning for DOM-only airports

Some airports don't have a JSON API. Flight data is rendered in HTML tables or card layouts, with no network request to intercept.

The AI learner handles these:

// ai-learn.js
async function learnPattern(url) {
  const page = await launchStealthBrowser();
  await page.goto(url, { waitUntil: 'networkidle', timeout: 18000 });

  const html = await page.content();
  const trimmed = trimHtml(html, 15000); // fits in context window

  const prompt = `
    This is an airport flight board HTML page. 
    Identify CSS selectors for each flight row, flight number, 
    destination/origin, scheduled time, gate, and status.
    Also list any status text values and what they mean.
    Return JSON: { rowSelector, columns: { flightNumber, destination, time, gate, status }, statusMappings }
  `;

  const result = await deepseekChat(prompt, trimmed);
  return JSON.parse(result);
}

Successfully AI-learned airports so far:

Airport	Row selector	Layout
GYD (Baku)	`a.flt_row`	Card with `.fl_col.*` columns
KTW (Katowice)	`div.timetable__row.flight-board__row`	Table-like rows
TLL (Tallinn)	`li.flights-list__item`	List card
DUS (Düsseldorf)	(table class detection)	Standard table

Each of these would have taken 20–30 minutes of manual DevTools work. The AI learner does it in about 30 seconds.

Step 6: Auto-generate config code

Once the pattern is known, the code generator writes the actual config:

// code-generator.js
function generateAirportEntry(iata, name, url, strategy) {
  if (strategy.type === 'api') {
    return `
  ${iata}: {
    name: '${name.replace(/'/g, "\\'")}',
    url: '${url}',
    apiEndpoint: '${strategy.endpoint}',
    headers: ${JSON.stringify(strategy.headers, null, 6)},
  },`;
  }
}

For operator groups it generates a shared template and references:

function generateOperatorEntry(operator, airports) {
  return `
const ${operator.name.toLowerCase()}Config = {
  apiBase: '${operator.apiBase}',
  headers: ${JSON.stringify(operator.headers, null, 2)},
};

${airports.map(a => `  ${a}: { ...${operator.name.toLowerCase()}Config, iata: '${a}' },`).join('\n')}
`;
}

Running at scale: 1,300 airports

npm run auto-discover                    # process all airports
npm run auto-discover -- --country DE   # just one country
npm run auto-discover -- --skip 400     # resume from checkpoint

The tool reads 6,000+ entries from the OpenFlights world airports CSV, processes in batches of 10 with a 60-second timeout per airport, and writes configs directly to disk.

Results after one full run across 1,300 airports:

276 new airports discovered — APIs found, configs generated, immediately usable
~600 DOM-only airports — HTML scraping with multilingual status normalisation
~400 failures — heavy WAF, no official website, or page timeouts
Operator groups detected: AENA (46), Finavia (20), ADP (3), and several smaller groups

Total processing time: ~4 hours on a single VPS. Compared to 20–30 minutes per airport manually, that's roughly 400 hours of DevTools work replaced by a shell command.

What still breaks

The pipeline isn't perfect. Common failure modes:

Heavy bot protection: Akamai Bot Manager and Imperva with device fingerprinting go beyond what FlareSolverr can handle. These remain manual.

Dynamic API tokens: Some airports generate short-lived tokens that expire between scrapes. Discovery captures them, but they're invalid an hour later. These need re-discovery on every request, which is expensive.

iFrame-embedded boards: Some airports embed a third-party flight board in an iframe from a different domain. Network interception doesn't always capture those requests correctly.

No-English flight data: A few airports only publish in their local language (Cyrillic, Arabic, CJK). The status normaliser handles most of it, but field mapping still fails occasionally.

The auto-discovery pipeline turned a manual, one-by-one chore into a batch job. The 276 airports it found in one run would have taken months to add by hand. The failures are the interesting edge cases — and those are where the AI pattern learner earns its keep.

The API is free to try at myairports.online/developers. Free tier: 100 requests/day, no card required.

Two Auth Problems: How MyAirports Handles Identity on Both Sides of the API

José Catalá — Tue, 14 Apr 2026 12:42:50 +0000

Every API has an auth problem. Usually it's one: how do you verify that the person calling your API is allowed to?

Building MyAirports — a real-time flight data API covering 1,000+ airports — gave me two auth problems. The first is the standard one: verifying developers who use my API. The second is messier: authenticating against 1,000+ different airport APIs, each of which expects something different.

Here's how I solved both.

The outward-facing problem: clean developer auth

When a developer hits the MyAirports API, I want the experience to be familiar. You sign up, you get a key, you put it in a header, it works.

API key format

Keys follow this pattern:

ma_live_<32 random hex characters>

The ma_live_ prefix makes keys identifiable at a glance — in log files, in config dumps, in error messages. If you accidentally paste your key into a public repo, a secret scanning tool can spot it. This is the same pattern Stripe, Resend, and most modern API providers use, and for good reason.

Storage: never store the plaintext key

The key is generated once, shown to the user once, and then discarded. What gets stored in the database is the SHA-256 hash of the key:

// generation
const rawKey = `ma_live_${crypto.randomBytes(16).toString('hex')}`;
const hashedKey = crypto.createHash('sha256').update(rawKey).digest('hex');

// store only hashedKey in ApiKey table
// return rawKey to the user (once)

// lookup
const incomingHash = crypto.createHash('sha256').update(incomingKey).digest('hex');
const record = await prisma.apiKey.findFirst({ where: { hashedKey: incomingHash } });

This means a database breach doesn't expose live keys. The attacker gets hashes that are useless without the original plaintext.

Auth middleware: two paths into the system

The middleware checks in priority order:

// api/src/auth/middleware.js
export async function authMiddleware(req, res, next) {
  // Path 1: API key (developer access)
  const apiKey = req.headers['x-api-key'];
  if (apiKey) {
    const hashed = sha256(apiKey);
    const record = await prisma.apiKey.findFirst({ where: { hashedKey: hashed } });
    if (record) {
      req.user = await prisma.user.findUnique({ where: { id: record.userId } });
      return next();
    }
  }

  // Path 2: JWT cookie (dashboard/browser access)
  const token = req.cookies.accessToken;
  if (token) {
    try {
      const payload = jwt.verify(token, process.env.JWT_SECRET);
      req.user = payload;
      return next();
    } catch (_) {}
  }

  // Unauthenticated — public endpoints proceed, protected endpoints return 401
  next();
}

API keys handle programmatic access. JWTs handle the browser dashboard. The same middleware serves both, and there's no ambiguity about which wins — API key always takes precedence.

The JWT setup is standard: 1-hour access tokens, 30-day refresh tokens, both in httpOnly cookies. HS256, signed with JWT_SECRET. The short access token TTL limits blast radius if a token leaks; the refresh token means users don't re-authenticate constantly.

Rate limiting: in-memory counters, not DB hits

Rate limiting on every request is a performance concern. Hitting PostgreSQL on every API call to check a counter would add latency to every response. Instead:

// In-memory store — fast lookup
const counters = new Map(); // userId -> { count, date }

export function checkRateLimit(userId, plan) {
  const today = new Date().toISOString().split('T')[0];
  const entry = counters.get(userId) || { count: 0, date: today };

  if (entry.date !== today) {
    entry.count = 0;
    entry.date = today;
  }

  const limit = plan === 'pro' ? 1000 : 100;
  if (entry.count >= limit) {
    return { allowed: false, remaining: 0, limit };
  }

  entry.count++;
  counters.set(userId, entry);
  return { allowed: true, remaining: limit - entry.count, limit };
}

The in-memory counters are flushed to the ApiUsage PostgreSQL table every hour. This gives you persistence across restarts and a usage history for the dashboard, without a DB round-trip on every request.

Rate limit state is communicated back to callers via standard headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1712275200

And when the limit is exceeded:

HTTP 429 Too Many Requests
{
  "error": "Rate limit exceeded",
  "limit": 100,
  "remaining": 0,
  "resetAt": "2026-04-15T00:00:00Z"
}

The inward-facing problem: 1,000 different auth schemes

Now for the harder problem.

MyAirports scrapes flight data from airport websites to serve its API. Each airport website is different. And "different" doesn't just mean different data formats — it means different auth requirements too.

Here's a sample of what the scraper encounters:

No auth required — most airports. The flight data API is unauthenticated or only requires session cookies the browser already has.
Cloudflare JS challenge — the page won't load without executing a JavaScript fingerprinting challenge first. No challenge, no data.
Incapsula WAF — similar JS challenge, different vendor.
CSRF tokens — some airport APIs require a valid CSRF token extracted from the page before accepting data requests.
Leaked third-party API keys — during browser discovery, the scraper sometimes intercepts network requests that include API keys in headers. Airports that license flight data software sometimes include the vendor's API key in client-side requests.

The last one deserves its own Prisma model:

model LeakedKey {
  id          Int      @id @default(autoincrement())
  airportIata String
  keyName     String   // e.g. "X-API-Key", "Authorization"
  keyValue    String   // the actual leaked key value
  source      String   // URL it was found in
  tested      Boolean  @default(false)
  working     Boolean  @default(false)
  createdAt   DateTime @default(now())
}

When the interceptor finds an API key embedded in a request header during discovery, it logs it here. Some of these keys work for direct calls to the upstream flight data vendor — an interesting optimization, though using them raises its own questions.

WAF session persistence

Solving a Cloudflare or Incapsula challenge is expensive: it requires a full browser session, human-like behavior simulation, and sometimes a FlareSolverr sidecar to handle the cryptographic JS challenge:

docker run -d \
  --name flaresolverr \
  -p 8191:8191 \
  ghcr.io/flaresolverr/flaresolverr:latest

Paying that cost on every scrape is wasteful. Instead, solved cookies are stored in the Session table and reused:

model Session {
  id          Int      @id @default(autoincrement())
  airportIata String   @unique
  cookies     Json     // solved WAF cookies
  userAgent   String   // must match the browser that solved the challenge
  expiresAt   DateTime
  createdAt   DateTime @default(now())
}

The session store checks for a valid unexpired session before launching a browser. A session that's still valid skips the challenge entirely — the scraper just presents the saved cookies.

The anti-bot evasion stack

Beyond WAF cookies, the browser sessions themselves need to look human. The stealth layer manages six distinct browser profiles — each with a coherent set of navigator properties, WebGL fingerprints, HTTP headers, and behavioral patterns.

The profiles are designed to look like real returning users rather than fresh bot fingerprints. Persistent Chromium user-data directories reinforce this — the browser accumulates cookies and browsing history across scrapes, which is a signal many WAFs check.

Stale session eviction

Sessions expire. WAF cookies typically last 24-48 hours. The system evicts sessions as they expire and re-solves challenges when needed.

There's a parallel mechanism for discovered API endpoints: the ApiCache table tracks success and failure counts per endpoint. An endpoint that returns zero flights three times in a row gets flagged as stale:

model ApiCache {
  id           Int      @id @default(autoincrement())
  airportIata  String   @unique
  endpoint     String
  headers      Json
  successCount Int      @default(0)
  failCount    Int      @default(0)
  lastUsed     DateTime
  stale        Boolean  @default(false)
}

A stale endpoint triggers re-discovery — the browser fires up again, re-intercepts the network traffic, and hopefully finds the new API URL.

The architecture of two auth systems

Stepping back: the MyAirports API is a thin, clean auth layer sitting on top of a chaotic, unpredictable auth environment.

Outward-facing: developers see a consistent, well-documented interface. API keys in headers, rate limits in response headers, 429s when limits are hit. Predictable.

Inward-facing: the scraper deals with a different kind of problem every hour. No auth, WAF challenges, leaked keys, CSRF tokens, expired sessions. Unpredictable by design.

The key insight is that these two problems need to be separated architecturally. The developer-facing API layer doesn't know or care about WAF sessions — that complexity is fully contained in the scraping layer. The scraping layer doesn't know anything about ma_live_* keys — that's the API layer's concern.

The split keeps each layer simple. The outward-facing auth is straightforward standard API security. The inward-facing auth is a collection of adapters for a messy world.

The API is free to try at myairports.online/developers. Free tier: 100 requests/day, no card required.

How Airport Websites Try to Stop You Scraping Them (And What I Did About It)

José Catalá — Mon, 13 Apr 2026 09:48:31 +0000

I scrape 200+ airport departure boards every few minutes. Here's every defence I've had to break through — and the ones that still slow me down.

Why Airport Websites Are a Scraper's Nightmare

Airport websites are an unusual target. They're not e-commerce sites trying to protect pricing data, and they're not news sites worried about plagiarism. Most of them just want to display departure boards to travellers. So you'd think they'd be easy to scrape.

They're not.

The problem is traffic patterns. An airport website gets hammered by bots — flight trackers, travel aggregators, airline apps, insurance tools, and journalists — all hitting the same handful of endpoints constantly. To survive, airports have layered in the same enterprise anti-bot stacks that protect banks and ticketing sites. They often don't even know exactly what's protecting them; it came bundled with their CDN contract.

Building MyAirports — a real-time flight data API covering 200+ airports — meant going through every layer. Here's what I found, roughly in order of how annoying each one is.

Layer 1: IP Rate Limiting

The most basic defence. Too many requests from one IP in a short window → 429 Too Many Requests or a silent block.

What it looks like: Your first few requests succeed. Then you start getting 429s, or worse, 200 responses with no flight data (the site is serving a decoy empty page rather than tipping you off with an error code).

What I did: Distribute requests across time. Each airport is on a polling interval calibrated to how often its data actually changes — a regional airport with 12 daily departures doesn't need to be hit every 30 seconds. Spreading the intervals out naturally reduces per-IP request rates.

For airports where even polite rates get blocked, I route through a residential proxy pool. It's not free, but for the ~15% of airports where it's necessary, there's no other option.

The nuance: Silent 200s with empty data are worse than 429s. I added a validation step that checks response plausibility — if a departure board suddenly returns zero flights for a major hub at 09:00 on a Tuesday, that's not real data. The scraper logs it as a suspected block rather than storing empty results.

Layer 2: User-Agent and Header Fingerprinting

Servers look at your request headers. A bare curl or Node fetch has obvious tells:

User-Agent: node-fetch/1.0
Accept: */*
Accept-Language: (missing)
Accept-Encoding: gzip, deflate

Real browsers send 15–20 headers in a specific order, with specific values that match the stated browser version.

What it looks like: Immediate 403, or redirect to a CAPTCHA page that your headless browser can't easily solve.

What I did: Use a real browser engine (Playwright with Chromium) rather than raw HTTP requests. This gives you the genuine browser fingerprint for free — correct header order, correct values, correct navigator properties, correct TLS handshake cipher suite order.

The cipher suite thing matters more than people realise. Node.js and browsers negotiate TLS with different cipher preference orders. Some DDoS protection vendors fingerprint the TLS handshake before the HTTP request even arrives. If your cipher order looks like OpenSSL rather than Chrome, you're flagged at the network layer before you've sent a single byte of HTTP.

// Using Playwright with a realistic browser context
const browser = await chromium.launch({
  headless: true,
  args: ['--no-sandbox', '--disable-blink-features=AutomationControlled']
});

const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...',
  locale: 'en-GB',
  timezoneId: 'Europe/London',
  viewport: { width: 1366, height: 768 }
});

The --disable-blink-features=AutomationControlled flag is important — without it, navigator.webdriver returns true in the page context, which is a trivial bot signal that JavaScript on the page can read.

Layer 3: Cloudflare Bot Management

This is where things get genuinely hard. Cloudflare's bot management suite runs a JavaScript challenge in the browser that collects dozens of signals:

Canvas fingerprint (GPU-specific rendering of a test image)
Audio context fingerprint
Font enumeration
WebGL renderer string
Screen dimensions vs. window dimensions
Mouse movement entropy (does the mouse move like a human?)
Timing of JavaScript execution
Whether window.chrome exists and has the right shape
Dozens more

The challenge runs silently in the background. Pass it and you get a cf_clearance cookie that's valid for a few hours. Fail and you get a 403 or an interactive CAPTCHA.

What it looks like: Page loads fine, but all API calls to the flight data endpoint return 403. Or: you get the page, the page loads some JS, and then the JS makes a secondary XHR that gets blocked.

What I did: Playwright with the playwright-extra stealth plugin, which patches the most common automation detection vectors:

import { chromium } from 'playwright-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

chromium.use(StealthPlugin());

The stealth plugin patches things like:

navigator.webdriver → undefined
navigator.plugins → realistic plugin list
window.chrome → full Chrome object shape
Canvas fingerprint → randomised slightly per session
permissions.query for notifications → returns denied (real browsers do; headless ones used to throw)

This handles maybe 70% of Cloudflare deployments. The other 30% use more aggressive challenges that require the cf_clearance cookie to be renewed more frequently, or require solving an actual interactive challenge. For those, I use a CAPTCHA solving service — 2captcha or AntiCaptcha — which has human solvers available for the rare cases that get through the stealth layer.

Layer 4: Akamai Bot Manager

Akamai is different from Cloudflare. Instead of a one-time challenge, Akamai's bot manager injects a sensor script into every page that runs continuously and phones home with behavioural data. It's looking for things like:

Are mouse movements suspiciously perfect? (Bots often move in straight lines)
Are keystrokes happening at humanly possible intervals?
Is the page being scrolled at all?
Does the _abck cookie (Akamai's session cookie) contain a valid HMAC signature computed from real browser data?

The _abck cookie is the key. It's a JWT-like token containing sensor data, and it's validated server-side on each request. If the cookie is missing or invalid, API requests are rejected — even if you've got a valid session.

What it looks like: You can load the page fine. But when the page makes XHR requests to /api/flights or wherever the data lives, they come back 403 with no body. The failure is invisible to users (the JS handles the retry), but your scraper just gets nothing.

What I did: Two approaches depending on the airport.

For airports where the data is available in the initial HTML (the departure board is server-rendered), I just parse the HTML directly. Akamai only blocks the XHR calls; the first page load often goes through fine and contains all the data I need.

For airports where the data is loaded client-side via XHR, I use Playwright and let the full page JavaScript run. Playwright with stealth generates valid-enough sensor data that the _abck cookie gets accepted. I then extract the cookie and replay it on subsequent requests — though it expires and needs refreshing every 30–60 minutes.

// Wait for the XHR to complete rather than trying to replicate it
await page.waitForResponse(
  response => response.url().includes('/api/flights') && response.status() === 200
);
const data = await page.evaluate(() => window.__FLIGHT_DATA__);

Layer 5: Dynamic Endpoint Obfuscation

Some airport sites don't have a stable API endpoint. The URL for the flight data changes with each deployment — it might be hashed, versioned, or entirely randomised. Hardcoding /api/v1/flights breaks the moment they redeploy.

What it looks like: Your scraper works for weeks, then silently breaks when the airport updates their site. The URL 404s and you don't find out until someone notices the data hasn't updated.

What I did: Intercept network requests rather than targeting a specific URL. Playwright's page.on('response', ...) captures every response the page receives. I filter for responses that look like flight data — JSON with an array of objects containing recognisable fields like flight numbers, times, or status strings.

page.on('response', async (response) => {
  const url = response.url();
  if (!url.includes('flight') && !url.includes('departure') && !url.includes('arrival')) {
    return;
  }

  try {
    const body = await response.json();
    if (isFlightData(body)) {
      await processFlights(body);
    }
  } catch {
    // Not JSON, skip
  }
});

function isFlightData(data) {
  const arr = Array.isArray(data) ? data : data.flights || data.data || data.departures;
  if (!Array.isArray(arr) || arr.length === 0) return false;
  const sample = arr[0];
  return sample.flightNumber || sample.flight_no || sample.ident || sample.designator;
}

This is more resilient to site updates because I'm not coupled to a specific URL structure — I'm coupled to the shape of the data, which changes far less often.

Layer 6: Session Fixation and IP-Bound Cookies

This is the one that took me longest to diagnose. Some airports use session cookies that are bound to the IP address of the first request. The cookie is issued by the server and contains an HMAC of the client IP. If you grab the cookie from one IP (say, your local machine while debugging) and try to use it from another IP (your VPS), the server rejects it.

What it looks like: Everything works locally. On the VPS, you get 403s or empty data even though you're sending the right cookies. The difference is invisible in the request — you're sending a valid-looking cookie, it's just bound to the wrong IP.

What I did: I wrote about this in detail in a separate post, but the short version: I built SkipRestriction — a self-hosted Shadowsocks proxy running on the same VPS as the scraper. When I need to grab cookies for an airport with IP-bound sessions, I browse through the proxy so the cookie is issued to the VPS's IP. The scraper, also running on the VPS, then uses those cookies from the same IP.

Layer 7: JavaScript-Rendered Data with Anti-Automation Timing

Some sites render flight data using JavaScript that deliberately runs on a slight delay, or loads data in chunks, or only renders the full table after a scroll event. These are heuristics designed to detect headless browsers, which typically don't scroll and don't trigger layout-based events.

What it looks like: Playwright loads the page, you query the DOM for flight rows, and you get back 3 rows when there should be 45. The rest haven't rendered yet.

What I did: Inject realistic user behaviour before extracting data:

// Simulate a human reading the page before scraping
await page.waitForLoadState('networkidle');
await page.evaluate(() => window.scrollBy(0, 300));
await page.waitForTimeout(800 + Math.random() * 400);
await page.evaluate(() => window.scrollBy(0, 500));
await page.waitForTimeout(600 + Math.random() * 300);

// Now query for data
const flights = await page.$$eval('.flight-row', rows =>
  rows.map(row => ({
    flight: row.querySelector('.flight-number')?.textContent.trim(),
    status: row.querySelector('.status')?.textContent.trim(),
    time: row.querySelector('.scheduled-time')?.textContent.trim(),
  }))
);

The random delays matter — consistent 800ms waits are a bot signal. Human reading speed has natural variance.

The Layer I Haven't Beaten: Real CAPTCHA with Proof-of-Work

A handful of airports (3–4 in my coverage set) use a challenge mechanism that requires solving a proof-of-work puzzle — essentially, computing a hash with certain properties before the session is established. It's not a visual CAPTCHA; it's a compute challenge that's expensive for bots to complete at scale and trivially fast for a single human browser session.

I currently handle these with a hybrid approach: a real browser session is established manually once (or triggered via a webhook), the resulting cookies are stored, and the scraper uses those cookies until they expire. For airports with long cookie TTLs (24h+), this is workable. For airports that require fresh challenges every hour, it's not scalable and those airports currently have lower update frequency in MyAirports.

If you've cracked this one cleanly, I'd genuinely like to hear how.

What Works, What Doesn't, and What Changes Constantly

The honest summary: Playwright + stealth gets you through 80% of sites. IP management (residential proxies or VPS-matched cookies) handles another 10%. The remaining 10% requires per-site custom solutions, ongoing maintenance, or accepting reduced update frequency.

What changes most often: Cloudflare updates its challenge algorithms roughly every 6–8 weeks. When it does, stealth plugins need updating. I follow the puppeteer-extra-plugin-stealth repo closely and have a monitoring alert that fires when a Cloudflare-protected site's data goes stale — that's usually the signal that a new challenge version is live.

What's surprisingly stable: The actual data extraction logic. Airport websites update their visual design sometimes, but the underlying data structure in the HTML or JSON is remarkably consistent over time. The bot protection changes; the data format rarely does.

The Scraper Stack (For Reference)

Playwright (Chromium) for JS-heavy sites
playwright-extra + stealth plugin for Cloudflare
got (Node HTTP client) for simple HTML-only sites
Residential proxy pool for IP-rate-limited airports
Shadowsocks self-hosted proxy for IP-bound cookie issues
2captcha for the small number of visual CAPTCHAs
PostgreSQL for storage, with a scrape_log table tracking per-airport success/failure rates
Node-cron for scheduling, with dynamic intervals per airport based on data freshness requirements

The full architecture, including the self-learning caching layer, is covered in a previous post.

The flight data is live at myairports.online — if you want to use it rather than scrape airport sites yourself, the API handles all of this so you don't have to.

Have a layer I haven't covered? Drop it in the comments — I'm genuinely curious what else is out there.

Your Flight Was Cancelled. Here's How I Built a Tool That Finds Every EU261 Claim at Any Airport

José Catalá — Fri, 10 Apr 2026 07:59:21 +0000

Most passengers never claim the compensation they're legally owed. I built a feature that changes that — here's how it works under the hood.

The €250 Nobody Asks For

When Lufthansa cancelled flight LH954 from Frankfurt to Birmingham this morning, 25 flights at that airport alone became eligible for compensation under EU261 in the last 7 days.

Most of those passengers won't claim a cent.

Not because they're ineligible — under EU Regulation 261/2004, a cancelled flight on a route under 1,500 km entitles each passenger to €250 in cash compensation, regardless of whether the airline refunds the ticket. It's not a voucher. It's not store credit. It's money.

The problem isn't eligibility. It's awareness. Most travellers don't know the regulation exists, and those who do often assume claiming is too complicated to bother with.

I built MyAirports — a real-time airport data API — to solve the scraping and data normalisation problem. But once I had clean, reliable flight status data for hundreds of airports, a second problem became obvious: all this data about delays and cancellations was sitting there unused. I was already tracking which flights were cancelled, their distances, their routes. The EU261 calculation is basically a lookup table. So I built the Claims tab.

Here's how it works.

EU261 in Two Minutes

EU Regulation 261/2004 applies when:

The flight departs from an EU airport, or arrives at an EU airport on an EU/UK carrier
The flight is cancelled or delayed by more than 3 hours at arrival
The disruption is not caused by "extraordinary circumstances" (bad weather, air traffic control strikes, etc.)

Compensation tiers by route distance:

Distance	Compensation
Under 1,500 km	€250
1,500–3,500 km	€400
Over 3,500 km	€600

UK passengers still have equivalent rights post-Brexit under UK261.

That's it. The airline cannot refuse if the criteria are met. They can offer a voucher — but you're entitled to cash.

The Data Problem I Already Had Solved

MyAirports scrapes live flight data from airport websites — the actual departure boards, not third-party aggregators. Every flight gets:

IATA flight code
Origin and destination airport (with IATA codes)
Scheduled and actual departure/arrival times
Real-time status: on_time, delayed, cancelled, diverted, landed, etc.
Airline name

The status normalisation work was significant — airport websites display cancellations in seven different languages with wildly inconsistent terminology. I wrote about that in a previous post. By the time a flight hits the database, it has a clean, normalised status field.

For the Claims feature, cancelled is the key trigger. But not all cancellations qualify — the regulation only covers flights that departed from or arrived at an EU/UK airport. Since I'm already storing IATA codes, I can check country eligibility instantly against a static lookup of EU member state airport codes.

Calculating Eligibility Automatically

The core logic is straightforward once the data is clean:

function getEU261Compensation(flight) {
  // Must be cancelled or delayed 3+ hours
  if (flight.status !== 'cancelled' && flight.delayMinutes < 180) {
    return null;
  }

  // Route must touch EU/UK
  const fromEU = isEUAirport(flight.origin);
  const toEU = isEUAirport(flight.destination);
  const isEUCarrier = isEUAirline(flight.iataCode.slice(0, 2));

  if (!fromEU && !(toEU && isEUCarrier)) {
    return null;
  }

  // Distance-based compensation
  const distance = getGreatCircleDistance(flight.origin, flight.destination);

  if (distance < 1500) return 250;
  if (distance < 3500) return 400;
  return 600;
}

getGreatCircleDistance uses stored lat/lon for each IATA airport code — a ~7,000-entry lookup table I pre-compiled from public data. The great-circle calculation is just the haversine formula:

function haversine(lat1, lon1, lat2, lon2) {
  const R = 6371; // Earth radius in km
  const dLat = toRad(lat2 - lat1);
  const dLon = toRad(lon2 - lon1);
  const a =
    Math.sin(dLat / 2) ** 2 +
    Math.cos(toRad(lat1)) * Math.cos(toRad(lat2)) * Math.sin(dLon / 2) ** 2;
  return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
}

This runs at query time — no pre-computation needed. The distance between any two airports is deterministic, so it could be cached, but it's cheap enough that it doesn't matter.

What the Claims Tab Shows

When you open any airport page on myairports.online and click Claims, you see every flight in the last 7 days that passes the eligibility check — ranked by date, with:

Flight number and airline
Route (origin → destination)
Status (Cancelled / Delayed)
Distance (so you can verify the tier yourself)
Compensation amount (€250 / €400 / €600)
Regulation (EU261 or UK261)
Claim action — either a direct claim link to the airline's own portal, or a one-click claim via an integrated claims service

Birmingham Airport (BHX) had 25 flights eligible for compensation in the last 7 days when I last checked. That's not unusual. At a mid-sized European hub, 20–30 eligible flights per week is typical in normal operating conditions. During bad weather events or industrial action, that number can spike into the hundreds.

Two Claim Paths

Not every airline makes it easy to claim directly. Some (like British Airways and Lufthansa) have functional online claim forms — they comply, slowly. Others have forms that are buried, broken, or deliberately tedious.

The Claims tab surfaces both options:

Direct airline claim — for carriers with reliable self-service portals (EasyJet, Air France, Ryanair). We link directly to their claim page with the flight details pre-filled where possible.

Assisted claim for €2.99 — for cases where direct claiming is harder, or where passengers just don't want to deal with it. A claims management service handles the entire process: paperwork, follow-up, escalation to alternative dispute resolution if the airline refuses. They charge a flat €2.99 rather than the typical 25–35% cut that most "no win no fee" services take.

The decision of which path to offer is driven by a per-airline lookup table I maintain — essentially a reliability score for each carrier's own claims process. If the airline consistently pays out promptly on direct claims, we show the direct link. If they're known to stall, the assisted option is surfaced first.

The Unexpected Use Case

I built the Claims tab primarily as a user-facing feature. But it turns out developers using the API have found it useful too.

The /airports/{iata}/claims endpoint returns the same structured data:

{
  "airport": "BHX",
  "period_days": 7,
  "eligible_flights": [
    {
      "flight": "LH954",
      "airline": "Lufthansa",
      "date": "2026-04-10",
      "origin": "FRA",
      "destination": "BHX",
      "status": "cancelled",
      "distance_km": 766,
      "compensation_eur": 250,
      "regulation": "EU261",
      "claim_url": "https://www.lufthansa.com/gb/en/flight-irregularity-claim"
    }
  ],
  "total_eligible": 25
}

A few travel agencies and corporate travel tools have integrated this to automatically flag claims for their customers after disrupted trips. The value proposition is simple: most business travellers don't track their own delays. If a tool can surface "your employee on LH954 is owed €250" after the fact, that's real money recovered with zero effort.

What I Learned

Three things stood out building this:

Cancellation data is noisy. Airport websites sometimes mark a flight as cancelled and then re-list it under a new flight number (a "rescue flight"). If you don't de-duplicate carefully, you'll show a claim for a flight that technically departed. I added a reconciliation step that checks if the same route, same date, and similar departure time appears with a landed status anywhere in the dataset.

Extraordinary circumstances are unverifiable at query time. The regulation exempts cancellations caused by things outside the airline's control — storms, ATC strikes, bird strikes. I don't attempt to classify this automatically. The Claims tab shows all cancellations that pass the route/distance check and notes that eligibility is subject to circumstances. The claims process itself resolves extraordinary-circumstance disputes.

Distance matters more than people think. A Dublin→Birmingham cancellation (322 km) yields €250. A Paris CDG→Birmingham cancellation (488 km) also yields €250. But a longer intra-EU route like Lisbon→Helsinki (2,800 km) would yield €400. Passengers often assume cancelled = €250 flat, then feel cheated when their long-haul claim is assessed differently. Making the distance explicit in the UI has reduced those support queries significantly.

Try It

Every airport on myairports.online has a Claims tab. If you flew into or out of a European airport in the last 7 days and your flight was cancelled, check the Claims tab for your airport — there's a reasonable chance your flight is listed.

If you're a developer, the /airports/{iata}/claims endpoint is part of the MyAirports API. The free tier gives you 100 requests/day; the paid tier removes the cap and adds historical claims data going back 90 days.

EU261 was designed to make airlines accountable. The data to enforce it has always existed — it just needed surfacing.

MyAirports is a real-time airport data API covering 200+ airports across Europe, the Americas, and Asia-Pacific. The Claims feature is live at myairports.online.

How I Solved the IP-Bound Cookie Problem in My Web Scraper With a Self-Hosted Proxy

José Catalá — Tue, 07 Apr 2026 11:22:19 +0000

There's a category of airport website that breaks my scraper in a specific, maddening way: they issue session cookies that are pinned to the IP address of the request that created them.

The symptom looks like this: you manually grab a working cookie from your browser, paste it into your server-side scraper, and it immediately returns a 401 or redirects to a login page. The cookie is valid — you just created it. But it was created from your laptop's IP, and your VPS has a different one. The airport's backend checks both the cookie and the origin IP, and when they don't match, it rejects the session.

For MyAirports — a real-time flight data API covering 1,000+ airports — this is a real operational problem. Here's how I solved it.

Why IP-Bound Cookies Exist

Session fixation attacks are the reason. A server that binds cookies to IPs prevents an attacker who steals your session cookie from replaying it from a different location. It's a legitimate security measure, and it's common in systems that handle sensitive operational data — which airport backend systems often do.

For a scraper, it creates a workflow problem. You can:

Grab a fresh cookie from a real browser session
Replay that cookie from your server

But only if both steps happen from the same IP.

The Manual Cookie Workflow

For airports that require authenticated sessions (particularly those protected by HCaptcha or Incapsula), my architecture uses a grab-cookies.js tool. It launches a real browser, navigates to the airport site, solves challenges if needed, and exports the resulting session cookies.

Those cookies then get passed to the scraper as headers:

// Simplified example from the scraper pipeline
const cookies = await loadCookies('MAD'); // Madrid Barajas
const response = await fetch(endpoint, {
  headers: {
    'Cookie': cookies.join('; '),
    'Referer': 'https://www.aena.es'
  }
});

When grab-cookies.js runs on the VPS itself, everything works — the cookie was issued to the VPS IP, and the scraper makes requests from the same IP. But during development and debugging, running a headless browser on the server just to grab a cookie is slow and inconvenient. You want to do it from your local machine.

That's where the IP mismatch breaks things.

The Solution: Route Your Browser Through the VPS

The fix is simple in principle: make your local browser appear to come from the VPS IP when visiting airport websites. Any traffic that comes from your browser through the proxy will receive cookies bound to the VPS IP — the same IP your production scraper uses.

I used SkipRestriction.uk to set up a self-hosted Shadowsocks proxy on the same VPS where MyAirports runs. The setup is a single Docker command:

docker run -d \
  --name shadowsocks-server \
  --restart always \
  -p 8388:8388/tcp \
  -p 8388:8388/udp \
  shadowsocks/shadowsocks-libev:latest \
  ss-server -s 0.0.0.0 -p 8388 -k YOUR_PASSWORD -m aes-256-gcm

Point your local Shadowsocks client at YOUR_VPS_IP:8388, configure your browser to use it as a SOCKS5 proxy, and now your browser traffic exits from the VPS. Airport websites see your VPS IP. Cookies issued to that session are tied to the VPS IP. When the production scraper replays those cookies from the same VPS, the IP check passes.

Why Shadowsocks and Not a VPN

A traditional VPN would work too, but Shadowsocks has a few practical advantages for this use case:

It's lightweight. Running WireGuard or OpenVPN on the same VPS as a production scraper means managing certificates, key exchange, and routing tables. The Shadowsocks Docker container adds almost nothing to the host.

Traffic looks like HTTPS. Shadowsocks was designed to be indistinguishable from normal encrypted web traffic. Some airport networks filter VPN protocols at the packet level. Shadowsocks is harder to detect and block.

Per-application routing. With a VPN, all traffic from your machine goes through the VPS — including your Slack messages and YouTube tabs. With a SOCKS5 proxy, you configure specific browsers or applications to route through it. You can keep two browser profiles: one going through the VPS for cookie grabbing, one going directly for everything else.

No kernel modules. The Docker container runs entirely in user space. No system-level network reconfiguration needed.

The Full Cookie Workflow

With the proxy in place, the workflow for IP-bound cookies looks like this:

Local browser (via Shadowsocks proxy)
  → VPS IP
  → Airport website
  → Cookie issued to VPS IP

grab-cookies.js export
  → cookies stored server-side

Production scraper (on same VPS)
  → Same VPS IP
  → Airport website
  → Cookie accepted ✓

The roundtrip adds maybe 80-120ms of latency compared to a direct connection, which is irrelevant for a one-time cookie grab operation.

Setting It Up

The SkipRestriction.uk guides cover the full setup for both VPS (cloud server) and Raspberry Pi (local self-hosting). The site also walks through client configuration for every major platform — macOS, Windows, iOS, Android, and Linux.

If you're running a scraper on a VPS and hitting IP-bound session issues, the proxy approach is the cleanest fix. The alternative — running your entire debugging workflow on the server via SSH — is technically viable but slow and error-prone.

Two commands on the server side, one config file on the client side, and the IP mismatch problem disappears.

The proxy runs on port 8388. The MyAirports API runs on port 443. They coexist on the same $5/month VPS without any conflict.

MyAirports is a free real-time flight data API covering 1,000+ airports. Try it at myairports.online/developers.

The Shadowsocks setup guide used for this project is at skiprestriction.uk — open source, self-hosted, no accounts required.

628,000 Flights Later: Building a Flight Analytics Engine on Accumulated Scrape Data

José Catalá — Mon, 06 Apr 2026 16:40:24 +0000

Every time MyAirports scrapes a flight board, the flights are stored in PostgreSQL. What started as a caching mechanism turned into something more interesting: a dataset of real flight behaviour that can answer questions no static API can answer.

After accumulating 628,000+ flight records across 1,000+ airports, the data supports four categories of analytics:

Airline reliability rankings per airport (on-time rate, average delay, cancellation rate)
Delay pattern breakdowns by airline, destination, day of week, and time of day
Airport busyness heatmaps (7×24 grid: average flight volume by hour and day)
Route reliability stats with per-airline breakdown (LHR→JFK: which airline is most reliable?)

Here's how the analytics layer works.

The raw data

Each Flight record in PostgreSQL has:

flightNumber, airline (IATA), type (arrival/departure)
airport (IATA), origin, destination
scheduled, estimated, actual (timestamps, local timezone)
status (scheduled/boarding/departed/arrived/delayed/cancelled/unknown)
confidence, strategy, scrapedAt

The actual timestamp and status field are what make analytics possible. When a flight's status is arrived and actual is set, we know when it actually landed. The delay is simply actual - scheduled in minutes.

Two compute strategies: incremental and nightly

Running full analytics queries over 628,000 rows on every API request would be too slow. The system uses a two-track approach.

Incremental counters — updated after each scrape:

When a new batch of flights arrives, the system immediately updates lightweight counters: per-airline flight counts at that airport, per-terminal/gate flight counts. These are cheap to maintain and power fast lookups.

Nightly full recompute — runs at 2 AM UTC:

Once per day, the system runs a full pass over all historical data to recompute the richer analytics tables. This is where the delay stats, airline scorecards, and busyness grids are built. It's more expensive, but the data doesn't need to be real-time — yesterday's delay patterns are as useful as today's.

Airline reliability rankings

The query is straightforward in concept, but requires minimum thresholds to be useful:

SELECT
  airline_iata,
  COUNT(*) as total_flights,
  COUNT(*) FILTER (WHERE delay_minutes <= 15) * 100.0 / COUNT(*) as on_time_pct,
  AVG(delay_minutes) FILTER (WHERE delay_minutes > 0) as avg_delay_minutes,
  COUNT(*) FILTER (WHERE status = 'cancelled') * 100.0 / COUNT(*) as cancelled_pct
FROM flights
WHERE airport_iata = 'LHR'
  AND actual IS NOT NULL
  AND status IN ('arrived', 'departed')
GROUP BY airline_iata
HAVING COUNT(*) >= 10
ORDER BY on_time_pct DESC;

The HAVING COUNT(*) >= 10 is the important part. Without a minimum flight threshold, airlines that had one flight and it happened to be on time appear at the top with 100% on-time performance. Ten flights is a low bar, but it filters out pure noise.

The result for LHR looks like this:

{
  "rank": 1,
  "airlineIata": "LX",
  "airlineName": "Swiss International Air Lines",
  "onTimePct": 88.2,
  "avgDelayMinutes": 7.4,
  "cancelledPct": 0.3,
  "totalFlights": 2340
}

Delay patterns: finding when and why delays happen

The delay breakdown goes a level deeper than overall stats. It slices delay data by:

Day of week — Monday departures are often worse than Wednesday ones
Time bucket — morning (06-12), afternoon (12-18), evening (18-24), night (00-06)
Destination — some routes are systematically late
Airline — some carriers at a specific airport are consistently worse than their global average

The time bucket breakdown is particularly useful. At large hub airports, delays compound through the day: a slot disruption in the morning propagates through afternoon flights. The heatmap data makes this visible.

Airport busyness heatmap

The busyness grid answers "when is this airport busy?" — useful for scheduling, understanding delay risk by time, and just general situational awareness:

SELECT
  EXTRACT(DOW FROM scheduled) as day_of_week,  -- 0=Sun, 6=Sat
  EXTRACT(HOUR FROM scheduled) as hour,
  AVG(flight_count) as avg_flights
FROM (
  SELECT
    DATE_TRUNC('hour', scheduled) as bucket,
    EXTRACT(DOW FROM scheduled) as day_of_week,
    EXTRACT(HOUR FROM scheduled) as hour,
    COUNT(*) as flight_count
  FROM flights
  WHERE airport_iata = $1
  GROUP BY 1, 2, 3
) hourly
GROUP BY day_of_week, hour
ORDER BY day_of_week, hour;

The result is a 7×24 matrix of average hourly flight volumes, suitable for heatmap rendering in any charting library.

Gate and terminal predictions

The most interesting use of accumulated data is predicting which gate or terminal a flight will use before the airport announces it. Airlines tend to use the same gates consistently at their hub airports.

After each scrape, the system updates per-airline terminal and gate counters:

// After normalizing a batch of flights
for (const flight of flights) {
  if (flight.terminal) {
    incrementCounter(`${airport}:${airline}:terminal:${flight.terminal}`);
  }
  if (flight.gate) {
    incrementCounter(`${airport}:${airline}:gate:${flight.gate}`);
  }
}

When serving a flight that doesn't have a gate assigned yet, the API enriches it with the most common gate for that airline at that airport:

if (!flight.gate && insights?.terminal?.predicted) {
  flight.insights = {
    terminal: { predicted: insights.terminal.value, confidence: insights.terminal.confidence }
  };
}

The confidence value is the fraction of historical flights that used the predicted terminal. For BA at LHR, it's nearly always Terminal 5 — confidence approaches 1.0. For a code-share flight operated by a third party, confidence might be 0.3.

The route reliability endpoint

One API endpoint that required no special query design: route-level stats. Given an origin and destination, return reliability data for each airline that operates the route:

GET /api/routes/LHR/JFK/insights

{
  "origin": "LHR",
  "dest": "JFK",
  "route": {
    "totalFlights": 1240,
    "onTimePct": 78.3,
    "avgDelayMinutes": 16.5,
    "cancelledPct": 1.8,
    "topAirline": "BA",
    "topAirlineOnTimePct": 84.2
  },
  "airlines": [
    { "airlineIata": "BA", "onTimePct": 84.2, "totalFlights": 890 },
    { "airlineIata": "VS", "onTimePct": 71.4, "totalFlights": 350 }
  ]
}

This is data that's genuinely hard to find anywhere else. The big flight data providers have it, but it's locked behind enterprise pricing. The MyAirports version is built from first-party airport data, accumulated one scrape at a time.

What the data still can't do

A few honest caveats about 628,000 flights:

It's not uniform. Airports with many active users have more scrape history than airports discovered recently. LHR has thousands of records; a newly added regional airport might have dozens.

It's biased toward requested airports. The system scrapes when someone requests data. Popular airports are over-represented relative to their actual traffic volumes.

Delay calculations require actual timestamps. Many airport APIs only return scheduled times, not actual arrival/departure times. Those flights contribute to volume counts but not delay calculations.

The data is directionally useful — the patterns it reveals are real — but it's not a controlled dataset. It's accumulated opportunistically.

All three endpoints are available in the MyAirports API:

GET /api/airports/:iata/insights/airlines
GET /api/airports/:iata/insights/delays
GET /api/airports/:iata/insights/busyness
GET /api/routes/:origin/:dest/insights

Free tier at myairports.online/developers. No credit card required.

Normalizing Flight Statuses Across 7 Languages: What I Learned Building a Global Airport API

José Catalá — Mon, 06 Apr 2026 16:39:30 +0000

When you scrape flight data from airport websites across 85+ countries, you quickly discover that nobody agrees on anything. Not the data format, not the field names, not the status strings.

"Departed" is SAL, FLY, CER, AIR, DEP, DEPARTED, Departed, Отправлен, Väljunud, DESPEGADO, gestartet, or partito, depending on which airport website you're looking at.

Building MyAirports meant writing a status normalizer that handles all of these — and maps them to a single, predictable enum. Here's how it works and what surprised me along the way.

The target schema

Every flight the API returns has a status field from this set:

scheduled | boarding | departed | arrived | delayed | cancelled | unknown

Seven values. Simple, predictable, usable in any frontend without additional mapping. The hard part is getting there from the real world.

Layer 1: Exact string matches

The first layer is the most straightforward — a lookup table of known status strings to their canonical values:

const EXACT = {
  // English
  'scheduled': 'scheduled',
  'on time': 'scheduled',
  'departed': 'departed',
  'in flight': 'departed',
  'arrived': 'arrived',
  'landed': 'arrived',
  'delayed': 'delayed',
  'cancelled': 'cancelled',
  'boarding': 'boarding',
  'gate open': 'boarding',

  // Spanish (AENA codes)
  'SAL': 'departed',  // salida
  'AIR': 'departed',  // en aire
  'FLY': 'departed',
  'CER': 'departed',  // cerrado
  'LND': 'arrived',   // aterrizó
  'FNL': 'arrived',   // final
  'IBK': 'arrived',   // in-block
  'BOR': 'boarding',  // embarcando
  'EMB': 'boarding',
  'ULL': 'boarding',
  'OPE': 'boarding',
  'EST': 'scheduled', // estimado
  'PRG': 'scheduled', // programado
  'RET': 'delayed',   // retraso
  'CNL': 'cancelled',

  // Russian
  'Отправлен': 'departed',
  'Прилетел': 'arrived',
  'Посадка': 'boarding',
  'Задержан': 'delayed',
  'Отменён': 'cancelled',

  // Estonian
  'Väljunud': 'departed',
  'Saabunud': 'arrived',
  'Pardal': 'boarding',

  // Finnish (Finavia)
  'Lähtenyt': 'departed',
  'Saapunut': 'arrived',
  'Nousussa': 'boarding',
};

This handles probably 60-70% of real-world cases. The remainder need more work.

Layer 2: Substring patterns

Some status strings embed dynamic information — actual times, gate numbers, gate closures. You can't match them exactly, but you can match them by pattern:

const SUBSTRING_PATTERNS = [
  // "Delayed 00:45"
  [/delay/i, 'delayed'],
  // "Cancel" in any language starting with 'cancel'
  [/cancel/i, 'cancelled'],
  // AENA-style: "CERRADO" (closed), "PROGRAMADO" (scheduled)
  [/^cerr/i, 'departed'],
  [/^progr/i, 'scheduled'],
  // "Check-in open" → scheduled (not boarding yet)
  [/check.?in/i, 'scheduled'],
];

Layer 3: Regex patterns for dynamic content

A handful of airports embed actual timestamps or qualifiers directly in the status string. Phoenix Sky Harbor is a good example — their status field for delayed departures looks like "Now 14:22" (the actual departure time). That's not a status, that's a timestamp, and it means the flight has departed.

const REGEX_PATTERNS = [
  // Phoenix: "Now 12:21 PM" → departed
  [/^now\s+\d/i, 'departed'],
  // Finavia: "Expected 14:35" → still scheduled
  [/^expected\s+\d/i, 'scheduled'],
  // Generic: "14:35" alone means actual time → departed/arrived
  [/^\d{1,2}:\d{2}$/, 'arrived'],
];

Layer 4: Russian stem matching

Russian is morphologically rich — words inflect heavily based on grammatical case, number, and gender. "Посадка" (boarding) appears as "посадке", "посадкой", "посадку" in different sentence positions. Exact matching misses these.

The solution isn't a full NLP library (overkill for six words) — it's stem prefixes. The first four or five characters of a word are usually enough to identify it:

const RUSSIAN_STEMS = {
  'посад': 'boarding',   // posad- (посадка, посадке...)
  'отпра': 'departed',   // otpra- (отправлен, отправился...)
  'прил':  'arrived',    // pril- (прилетел, прилетела...)
  'задер': 'delayed',    // zader- (задержан, задержка...)
  'отмен': 'cancelled',  // otmen- (отменён, отменённый...)
};

This catches all the common inflected forms without false positives.

The object guard

One thing I didn't expect: some airport APIs return status as an object rather than a string:

{ "status": { "code": "DEP", "description": "Departed", "color": "#ff0000" } }

The normalizer checks for this and extracts code or description before attempting any matching. Without this guard, typeof status === 'object' crashes the string matching silently.

AENA's 30+ status codes

AENA (the Spanish airport operator, covering all 22 Spanish airports) has the most elaborate status system I encountered. They use two to three character codes from their SIGER ground handling system:

FNL, LND, IBK → arrived
BOR, EMB, ULL, OPE, APE → boarding
SAL, AIR, FLY, CER, APA → departed
EST, PRG, INI, SCH, HOR → scheduled
RET, DEM, REM → delayed
CNL, DES, CAN → cancelled

Each code has a specific operational meaning (IBK = in-block, ULL = final boarding call, DEM = ground delay) that the normalizer collapses to the six canonical values. The operational detail is lost, but that's the right tradeoff for a public API.

What "unknown" means in practice

The unknown bucket catches everything the normalizer can't confidently classify. In production, it appears for:

Empty strings or null values (some airports just don't return status for on-time flights)
Airport-specific codes that haven't been mapped yet
Status strings in languages I haven't configured (Arabic, Japanese, Chinese)

Returning unknown is far better than returning a wrong value. Frontends can show "—" for unknown statuses and users understand it means data isn't available.

Running the normalizer

The normalizer runs at the end of every scrape, after the field mapper has identified which JSON field contains the status. It's a pure function:

normalizeStatus(raw) // → 'scheduled' | 'boarding' | 'departed' | 'arrived' | 'delayed' | 'cancelled' | 'unknown'

Input is always a string (after the object guard). Output is always one of the seven values. No exceptions are thrown — unrecognized inputs return 'unknown'.

The broader lesson

I went into this project expecting the hard part to be scraping. The really hard part turned out to be normalization — taking data from wildly inconsistent sources and making it uniformly useful.

Status normalization is one example. The same problem appears with timestamps (airports use their local timezone, not UTC, and the formats vary wildly), location fields (some put "MILAN, MALPENSA" in the IATA field), and airline codes (some use IATA codes, some use ICAO, some use full names).

The principle that worked: design a strict output schema first, then build the normalization layer to enforce it, and treat every edge case as an addition to the normalizer rather than a special case in the consumer.

The API is live and free to try at myairports.online/developers. Arrivals and departures for 1,000+ airports, statuses normalized to the seven values above.

Next in this series: how 628,000 accumulated flight records are turned into airline delay rankings, gate predictions, and airport busyness heatmaps.

How I Built a Web Scraper That Gets Faster and Cheaper With Every Request

José Catalá — Mon, 06 Apr 2026 16:38:33 +0000

Most web scrapers are dumb: they do the same work every time. Open a browser, load a page, parse the DOM, extract data, close the browser. Every single request.

Building MyAirports — a real-time flight data API covering 1,000+ airports — forced me to think about scraping differently. At scale, running a headless browser for every request is expensive and slow. The system I ended up building instead gets faster and cheaper the more it runs, because it learns.

Here's how it works.

The core insight: most airport SPAs load data from internal JSON APIs

Go to any modern airport website. Open DevTools. Click on the Network tab. Filter for XHR/Fetch requests. What you'll see almost immediately is a call to something like:

GET https://api.heathrow.com/pihub/flights/arrivals
GET https://aena.es/sites/Satellite?pagename=AENA_ConsultarVuelos&airport=MAD
GET https://cph.dk/api/FlightInformation/GetFlightInfoTable?direction=A

The webpage is a React or Angular SPA, and the actual flight data comes from an internal API. That API is often unauthenticated, or authenticated only by session cookies the browser already has. If you can capture that API call, you can make it directly — no browser needed for subsequent requests.

This is the foundation of the whole system.

The 5-layer pipeline

Every flight data request flows through five layers:

Layer 1: Operator registry
For ~500 airports I've already reverse-engineered the API, the system goes directly there. No discovery needed. A curated registry maps IATA codes to endpoints, required headers, and a transform function for normalizing the response. This handles roughly half of all requests, takes 1-2 seconds, and costs nothing beyond the API call.

Layer 2: API cache (discovered endpoints)
The system maintains a database of APIs it has discovered during previous browser sessions. Before firing up a browser, it checks: have we already seen a good API for this airport? If yes, call it directly. This is the "memory" layer — it's how the system gets cheaper over time.

Layer 3: Browser discovery
Only if both previous layers miss does the system launch a real browser (Playwright, stealth mode). But even here it's not scraping the DOM — it's intercepting network traffic. The browser loads the airport's flight page, and the interceptor scores every JSON response it sees:

function scoreResponse(url, body) {
  let score = 0;
  if (url.includes('flight') || url.includes('arrival') || url.includes('departure')) score += 3;
  if (Array.isArray(body) && body.length > 5) score += 2;
  if (body[0]?.flightNumber || body[0]?.flight_no) score += 3;
  if (body[0]?.scheduled || body[0]?.scheduledTime) score += 2;
  return score;
}

Any response with a score above 5 is a candidate. The best candidate is saved to the ApiCache table — the discovered endpoint, the required headers, the response shape. Future requests for that airport hit Layer 2 instead.

Layer 4: Normalizer
Regardless of which layer found the data, every response goes through a normalizer that converts it to a unified flight schema. Airport APIs have wildly different shapes — nested objects, flat arrays, GraphQL responses, tables with inconsistent column names. The normalizer handles all of it.

Layer 5: Persistence and stale-while-revalidate cache
Results are stored in PostgreSQL and kept in an in-memory cache with a stale-while-revalidate strategy. A stale response is returned immediately while a fresh scrape runs in the background.

The result: the system pays for its own speed

When the system first encounters a new airport, it's slow: a full browser session takes 10-15 seconds. But the next time that airport is requested, the discovered API is available in the cache — and the call takes 1-2 seconds.

Over time, the API cache fills up. More and more airports become fast. The "discovery" cost is paid exactly once per airport. After that, it's a cheap HTTP call.

There's also an eviction mechanism: if a cached API endpoint starts returning 0 flights three times in a row, it gets flagged as stale and the system re-discovers it. Airport websites do change their APIs occasionally.

The auto-discovery tool

To seed the API cache without waiting for organic traffic, I built a batch auto-discovery CLI that processes airports from the OpenFlights CSV database:

npm run auto-discover -- --country IT
npm run auto-discover -- --min-size large

It runs the full browser discovery pipeline for each airport, then writes the discovered endpoints to the config files automatically. Processing 1,300+ airports this way found 276 new ones — airports whose APIs I hadn't previously known existed.

AI pattern learning as a last resort

For airports where network interception fails (because the page loads data from a server-side render rather than a client-side API call), there's a DeepSeek-powered fallback: send the page's HTML to the AI and ask it to identify CSS selectors for the flight table.

GYD (Baku) — a.flt_row card layout with .fl_col.* columns
KTW (Katowice) — div.timetable__row.flight-board__row with .flight-board__col--*
TLL (Tallinn) — li.flights-list__item card layout

The AI doesn't always get it right on the first try, but for airports with non-standard layouts it's significantly faster than manually inspecting the DOM.

What I learned

The most important architectural decision was treating discovery as a one-time investment rather than an ongoing cost. Every browser session is expensive — it's a browser, running JavaScript, waiting for a page to load. If you pay that cost once and cache the result, you amortize it across hundreds of future requests.

The second insight was scoring rather than parsing: instead of trying to understand an airport's specific API format upfront, intercept everything and look for responses that smell like flight data. The scoring heuristic is simple but effective, and it works across wildly different API shapes.

The third was building eviction in from the start. APIs change. Caching without a sensible eviction strategy means you end up with a database full of dead endpoints.

The result is a system that today handles most requests in 1-2 seconds, covers 1,000+ airports, and gets incrementally better with every new airport it encounters.

The API is free to try at myairports.online/developers.

If you found this interesting, I write about building MyAirports — a real-time flight data API. The next post covers how the system normalizes flight status strings from 7 different languages.

DEV Community: José Catalá

Three Ways to Fly: Building Route Options Ranked by Time, CO2 , and Reliability

Three Ways to Fly: Building Route Options Ranked by Time, CO₂, and Reliability

Routes as views, not entities

The graph

Candidate enumeration: the 2× viability filter

Leg durations: physical plausibility filtering

Hub layovers: matching real arrival/departure pairs

Reliability: geometric mean, not arithmetic

Ticket type detection

The three buckets

CO₂: distance is a reasonable proxy

What surprised me

What's next

What 1,000 Airports Reveal About the World's Airlines

643 airlines. Most of them you've never heard of.

Hub concentration is more extreme than you'd think

The IATA/ICAO code problem is worse than I thought

Low-cost vs legacy: different geographies, different patterns

The subsidiary problem

What's actually in the Airlines tab

One thing I didn't expect

How I Built a Delay Risk Score for 628,000 Flights

The core insight: route × time-of-day × day-of-week

What counts as "on time"

The pipeline: from raw flights to stats

Computing the stats for a bucket

The minimum sample problem

Handling airline codes: the IATA/ICAO mess

What the numbers actually look like

Surfacing it in the API

What I'd do differently

What 628,000 Flights Taught Me About Cancellations (And Why They're Harder to Track Than You Think)

The three types of cancellation (that most people treat as one)

Ghost flights: the cancellation nobody tells you about

Normalizing "Cancelled" across 30+ languages

Tracking cancellations over time: the flights table

Mass cancellation detection: weather, strikes, and cascades

What the data actually shows

Surfacing this in the API

The EU261 connection

What I'd do differently

How I Automated Onboarding 1,300 Airport Websites Without Looking at a Single One

The problem with airport websites

Step 1: Find the right URL

Step 2: Intercept network traffic

Step 3: WAF detection and FlareSolverr fallback

Step 4: Detect operator groups

Step 5: AI pattern learning for DOM-only airports

Step 6: Auto-generate config code

Running at scale: 1,300 airports

What still breaks

Two Auth Problems: How MyAirports Handles Identity on Both Sides of the API

The outward-facing problem: clean developer auth

API key format

Storage: never store the plaintext key

Auth middleware: two paths into the system

Rate limiting: in-memory counters, not DB hits

The inward-facing problem: 1,000 different auth schemes

WAF session persistence

The anti-bot evasion stack

Stale session eviction

The architecture of two auth systems

How Airport Websites Try to Stop You Scraping Them (And What I Did About It)

Why Airport Websites Are a Scraper's Nightmare

Layer 1: IP Rate Limiting

Layer 2: User-Agent and Header Fingerprinting

Layer 3: Cloudflare Bot Management

Layer 4: Akamai Bot Manager

Layer 5: Dynamic Endpoint Obfuscation

Layer 6: Session Fixation and IP-Bound Cookies

Layer 7: JavaScript-Rendered Data with Anti-Automation Timing

The Layer I Haven't Beaten: Real CAPTCHA with Proof-of-Work

What Works, What Doesn't, and What Changes Constantly

The Scraper Stack (For Reference)

Your Flight Was Cancelled. Here's How I Built a Tool That Finds Every EU261 Claim at Any Airport

The €250 Nobody Asks For

EU261 in Two Minutes

The Data Problem I Already Had Solved

Calculating Eligibility Automatically

Tracking cancellations over time: the `flights` table