How to Reverse-Engineer Transport APIs for a Unified Trip Search

Imagine trying to book a train from Paris to Berlin, only to find you must check three separate websites—SNCF, Deutsche Bahn, and FlixBus—because no single platform aggregates them all. This fragmentation is the daily reality for travelers and developers alike. Official APIs are often unavailable, prohibitively expensive, or locked behind restrictive agreements, especially for smaller providers. Yet the data is out there, exposed by web apps and mobile clients that must talk to backend servers. Reverse-engineering these transport APIs offers a practical, cost-effective path to building a unified trip search. This article is a hands-on tutorial that teaches you how to dissect undocumented APIs from rail and bus operators, extract authentication tokens, parse response formats, and combine results into a single search engine. Using a real-world example—merging European rail and bus APIs into one endpoint—you’ll learn concrete techniques for monitoring network traffic, handling rate limits, normalizing data, and creating a reliable unified search. Whether you’re a seasoned developer or a curious builder, this guide equips you with the skills to scrape transport data and turn fragmented systems into a cohesive solution.

Why Reverse-Engineer APIs Instead of Using Official Integrations

Official transport APIs sound ideal, but they often fall short. Many providers offer no public API at all, or only for a narrow set of routes. Even when an API exists, it may be prohibitively expensive, limit request volume, or require lengthy approval processes. Worse, official APIs can be deprecated without notice, breaking your integration overnight. These limitations make reverse-engineering a practical alternative.

Common limitations of official transport APIs:

• No unified coverage: Each provider exposes its own endpoints, data formats, and authentication methods. Combining them into one search is a complex integration task, not a single API call.
• High costs: Many APIs charge per request or require a monthly subscription. For a small project or startup, these costs can be unsustainable.
• Deprecation risks: Providers change or remove endpoints without warning, leaving your application with broken functionality.

Why reverse-engineering adds value:

• Full control: You choose which endpoints to call, how often, and how to parse the response. There is no reliance on a third-party's roadmap.
• Work around rate limits: Reverse-engineered APIs often have softer limits or no documented limits at all. With careful rate management, you can extract data more aggressively than official counterparts allow.
• Access to undocumented data: Internal APIs frequently expose richer data (e.g., real-time delays, seat availability) that official APIs hide or charge extra for.

Decision matrix: When to use official vs. reverse-engineered integration

Scenario	Recommended Approach	Reason
You need data for a single, well-documented provider with a generous free tier	Official API	Lower maintenance, reliable documentation
You need to combine data from 3+ providers, many without public APIs	Reverse-engineered	Only way to get unified coverage
Your project is a prototype or MVP with limited budget	Reverse-engineered	Avoid high costs; refactor later if needed
You rely on real-time data that official APIs restrict	Reverse-engineered	Access to richer data stream
You need legal compliance and official support	Official API	Avoid legal risks; use when required

Ultimately, the choice depends on your project's stage and constraints. For a unified trip search across multiple transport providers, reverse-engineering is often the only feasible path. It gives you the flexibility to build a comprehensive search engine without waiting for official integrations that may never come.

---

Note: Always respect the provider's terms of service. Reverse-engineering for personal or educational purposes is generally acceptable, but avoid aggressive scraping that could harm the service or violate legal agreements.

Step 1: Mapping the Target API Landscape

Start by opening your target provider’s web interface — say FlixBus or Omio — in Chrome or Firefox. Launch DevTools (F12), click the Network tab, and check "Preserve log" so requests stay visible across page navigations. Perform a real trip search and watch the XHR/Fetch calls populate. Filter by "XHR" or "Fetch" to isolate API traffic from images and CSS. You’ll typically see a POST or GET request to an endpoint like /api/v2/search with URL-encoded or JSON body parameters for origin, destination, and date.

Click any request and inspect the Headers tab for authentication clues. Look for Authorization, api-key, or X-Session-Token values. If those keys are plaintext, you can often reuse them; if they’re generated dynamically, you’ll need to trace the logic.

To uncover hidden endpoints, search within the page’s JavaScript bundles. In the Sources tab, press Ctrl+Shift+F (or Cmd+Option+F on Mac) and search for strings like /graphql, /api/, endpoint, or rate. Many modern transport sites use GraphQL; to find all available queries and mutations, open the Network tab, find a GraphQL request, right-click it, and select "Copy as cURL". Then, send an introspection query ({"query": "{ __schema { types { name fields { name args { name } } } } }"}) to the same endpoint — this often reveals undocumented operations.

Once you have a set of endpoints, document each one in a row of a spreadsheet or a Markdown table: URL, HTTP method, required headers, query/body parameters, and sample responses. Pay special attention to encoding (URL-encoded vs. JSON) and date/time formats (ISO 8601 vs. Unix timestamps). This map becomes the foundation for the authentication-handling and parsing steps that follow.

Step 2: Handling Authentication and Session Management

Once you’ve mapped the endpoints, the next hurdle is authentication. Transport APIs often protect their endpoints with API keys, session tokens, or custom headers. Here’s how to extract and manage them.

Extracting API Keys from Client-Side Code

Many web apps embed API keys directly in JavaScript source. Open the browser’s DevTools, go to the Sources panel, and search for patterns like apiKey, API_KEY, or authorization. For example, a minified bundle might contain:

var config={apiKey:"abc123xyz",baseUrl:"https://api.transport.com/v1"};

If the key is obfuscated, look for it in hidden form fields or XHR request headers. Use the Network tab to capture a request and inspect the x-api-key header or Authorization field. Sometimes the key is encoded in base64 – decode it with a quick Python script.

Session Token Refresh and Rotation

Some APIs use short-lived session tokens obtained via a login or initial handshake. To handle this programmatically, maintain a session object (e.g., requests.Session() in Python) that stores cookies and headers. For example:

import requests

session = requests.Session()
# Perform initial handshake to get token
response = session.post('https://api.transport.com/auth', data={'grant_type': 'client_credentials'})
token = response.json()['access_token']
session.headers.update({'Authorization': f'Bearer {token}'})

Now, every request uses this token. Monitor response status codes – a 401 or 403 indicates the token has expired. Implement a refresh mechanism: if you get a 401, silently re-authenticate and retry the request. Rotate credentials by using multiple API keys or user agents to avoid detection if you’re scraping at scale. Store tokens in a dictionary with expiry times and refresh before they expire.

Handling Custom Headers and Cookies

Some transport APIs require custom headers like X-Requested-With, User-Agent, or Origin to mimic a browser. Copy these from the network tab and include them in your session. For cookie-based auth, let your session object handle cookie persistence automatically. For example:

session = requests.Session()
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'X-Requested-With': 'XMLHttpRequest'
})
# Make first request to set cookies
session.get('https://api.transport.com/home')

After that, subsequent requests include the cookies automatically. If a session cookie is tied to a specific IP or user agent, ensure your scraper respects these constraints.

Extracting from Hidden Form Fields

Some APIs embed authentication tokens in hidden inputs of HTML pages. Parse the page with BeautifulSoup and extract the token:

from bs4 import BeautifulSoup
resp = session.get('https://transport.com/search')
soup = BeautifulSoup(resp.text, 'html.parser')
token = soup.find('input', {'name': 'csrf_token'})['value']
session.headers.update({'X-CSRF-Token': token})

This technique works well for APIs that require CSRF tokens. By systematically extracting and rotating credentials, you keep your unified search pipeline alive and avoid blocks.

Step 3: Decoding Request and Response Formats

Once you have captured the API endpoints and authentication tokens, the next challenge is interpreting the data. Undocumented APIs often return nested JSON, XML, or even binary formats like Protocol Buffers. Start by pasting raw responses into a JSON formatter or schema inference tool (e.g., JSON Hero, Paste JSON as Code) to visualize the structure interactively. This reveals nested arrays for trips, stations, or fares. For XML responses, convert to JSON first using libraries like xmltodict for easier manipulation.

Pay special attention to encoded payloads. Some transport APIs base64-encode fields like seat maps or compressed price tables. Look for strings ending in == and decode them with Python’s base64.b64decode(). Others use gzip compression on response bodies; always check the Content-Encoding header. Non-standard date/time formats are common: you might see 2024-09-01T12:30:00Z (ISO 8601) or a Unix timestamp in milliseconds. Normalize all timestamps to UTC ISO 8601 as early as possible in your pipeline.

Building a normalized data model for trips is essential for the unified search. Define a dictionary or Pydantic model with fields: departure (datetime), arrival (datetime), duration (timedelta), price (float, in a single currency like EUR), transfers (integer), operator (string), and raw_data (original JSON for debugging). Map each source’s response to this model. For example, one API might call departure time as "depTime", another as "start_time"; use a mapping function in Python to translate. This abstraction lets you sort, filter, and merge results without worrying about source-specific quirks.

When dealing with Protocol Buffers (rare but present in some high-volume APIs), download the .proto schema if exposed, or use a hex dump tool like protoc --decode_raw to infer field numbers and types. Alternatively, capture the request payload as well – sometimes the request format hints at the response structure. Always log unexpected fields to adapt to future API changes without breaking the integration.

Step 4: Bypassing and Respecting Rate Limits

When consuming reverse-engineered APIs, rate limiting is inevitable. Transport providers like FlixBus or Deutsche Bahn enforce limits to protect their services. Your goal is to stay within those limits while maximizing throughput, not to bypass them maliciously. Start by detecting rate limits. The most common signal is an HTTP 429 status code, often accompanied by a Retry-After header indicating seconds to wait. Some APIs return a 200 with a JSON error like {"error":"rate_limit_exceeded"}. Always log the response headers and body.

Once you detect a limit, implement exponential backoff. For example, upon first 429, wait 1 second, then 2, 4, 8 seconds, up to a maximum. Use a Python requests Session with a custom retry adapter. For concurrent requests, leverage asyncio with aiohttp to throttle using a semaphore or a rate limiter like aiolimiter. This is especially useful when querying multiple endpoints simultaneously (e.g., rail and bus APIs in parallel for the same origin-destination pair).

import asyncio
import aiohttp
from aiolimiter import AsyncLimiter

limiter = AsyncLimiter(5, 1)  # 5 requests per second

async def fetch(session, url):
    async with limiter:
        async with session.get(url) as resp:
            if resp.status == 429:
                retry_after = int(resp.headers.get('Retry-After', 1))
                await asyncio.sleep(retry_after)
                return await fetch(session, url)
            return await resp.json()

Proxy rotation and user-agent randomization are last resorts. If you must, rotate a small pool of residential proxies (e.g., from Bright Data) and vary user-agent strings from a curated list. Be aware that aggressive rotation can trigger stricter blocks. Always respect robots.txt and terms of service where possible. For a production unified search, cache responses aggressively to reduce repeated calls. The lesson: treat the API as a finite resource—use it wisely to keep your integration running smoothly.

Step 5: Building the Unified Search Pipeline

With authentication handled and individual API quirks decoded, the next challenge is combining responses from multiple transport providers into a single, coherent trip search. This requires building a pipeline that normalizes data, eliminates duplicates, and returns ranked results.

Start by defining a common trip schema. A minimal version might include fields like departure_time, arrival_time, price, currency, provider, duration_minutes, and vehicle_type. Map each provider’s raw response to this schema. For example, FlixBus might return {{ "departure": "2024-05-01T08:00:00", "price": 19.99, "currency": "EUR" }} while a regional rail API sends {{ "start": 1714550400, "fare": "€19.99", "duration": 120 }}. Normalize timestamps to ISO 8601 and prices to a float with a three-letter currency code.

Next, implement a caching layer to avoid hitting APIs for identical queries. In-memory caches like Redis work well, using a key based on normalized search parameters (origin, destination, date, time window) with a Time-To-Live of 5–15 minutes. Before making any API call, check the cache. If a result set is found, return it directly. This dramatically reduces load on both your service and the reverse-engineered endpoints.

After collecting results from all sources, you need to merge and rank them. Deduplicate first: if two providers offer the same departure time and route, keep the cheaper or shorter option. Then sort results by a user-selectable criterion such as price (ascending) or duration (ascending). A simple Python function can iterate over the normalized list and sort using sorted(trips, key=lambda t: t['price']).

Finally, consider error resilience. If one provider’s API returns a 503 or rate-limit response, fall back to cached results from the last successful query for that route, or omit that provider gracefully from the merged output. Log all failures for later analysis to detect when an API changes its schema or authentication method.

This pipeline transforms raw, inconsistent data from reverse-engineered APIs into a clean, responsive unified search experience — exactly the kind of integration that platforms like Paradane help you package into a production-ready product.

Turning Your Integration into a Real Project

You’ve now built a functional unified trip search pipeline that reverse-engineers multiple transport APIs. The next step is turning this prototype into a deployable project. Start by containerizing your service with Docker and setting up a CI/CD pipeline so you can push updates safely. Add error monitoring—for example, log every failed API call with the provider and endpoint, then send alerts if failure rates spike. Design fallback logic: if one provider’s API changes or goes down, your search should gracefully switch to remaining sources and show a notice. APIs evolve without warning, so schedule periodic tests that run your scraper against each endpoint and flag responses that no longer match your expected schema. Set up a simple health-check endpoint that reports the status of each integration.

To make the project real, apply it to a concrete use case: build a Slack bot that accepts a natural language trip query and returns unified results, or create a lightweight public search page for a specific corridor (e.g., London–Paris on rail and coach). Publish the normalized code as an open‑source Python package so others can reuse the data models and caching layer. If you need to accelerate the product side—APIs, authentication handling, scaling the pipeline—Paradane’s team has deep experience building integrations like this into production systems. Visit https://paradane.com to see how we help teams turn reverse‑engineered integrations into reliable, maintainable products.