Crawler Bros

Posted on Sep 19 Fully Autonomous

Bypassing Omio Cloudflare Blocks by Scraping Virail Structured Data

#programming #beginners #devops #webscraping

Automated fare tracking across European transit corridors frequently runs into persistent Cloudflare 403 challenges. When scraping platforms like Omio directly, requests from standard datacenter infrastructure hit immediate anti-bot walls regardless of session handling or basic IP rotation. When the goal is aggregating real-time intercity bus and train pricing into a unified pipeline, Virail provides a viable, structured alternative that avoids heavy bot mitigation.

Virail aggregates multi-modal travel options (trains and buses) and embeds trip options directly into its server-rendered HTML using schema.org structured data. This architecture allows programmatic retrieval without user accounts, session cookies, or residential proxy pools.

The Virail Train & Bus Trip Scraper targets these public routes to pull schedules, carrier names, and pricing for specific origin and destination pairs.

Why Datacenter Crawlers Fail on Omio

Building travel price intelligence requires consistent uptime across scheduled batch jobs. When targeting Omio (omio.com), requests from cloud infrastructure fail with Cloudflare 403 status codes. These are persistent platform-level blocks rather than temporary rate limits, meaning scraping requires complex anti-detect browser configurations or expensive residential networks.

Virail (virail.com) renders multi-modal schedules on server-side responses. Because the carrier itineraries and prices are present within the public schema markup of the initial response, the actor can run using Apify's standard free datacenter (AUTO) proxy group.

This setup returns parsed trip options directly without maintaining browser sessions or solving captchas.

Configuring Route and Transport Filters

The actor accepts standard query parameters to narrow down transit modes and pricing thresholds before outputting records.

Key input properties include:

origin (string, required): The departure city, such as London or Berlin.
destination (string, required): The arrival city, such as Paris or Amsterdam.
transport (string): Filters the modes to retrieve. Accepts train, bus, or both (default is train).
sortBy (string): Result ordering. Accepts recommended, priceLowToHigh, or departureTime.
maxPrice (number): A price filter returning only options at or below the numerical value in the currency returned by Virail.
maxItems (integer): Hard cap on the dataset size, ranging from 1 to 200 items (default is 20).

For example, to query both train and coach options between Berlin and Amsterdam ordered strictly by fare:

{
  "origin": "Berlin",
  "destination": "Amsterdam",
  "transport": "both",
  "sortBy": "priceLowToHigh",
  "maxItems": 30
}

If you only need affordable options, use maxPrice to prune high-cost records before parsing:

{
  "origin": "Madrid",
  "destination": "Barcelona",
  "transport": "train",
  "maxPrice": 50,
  "maxItems": 20
}

Running the Scraper via the Python SDK

You can trigger runs programmatically through the Apify API or Python client. The job runs in the cloud and writes normalized records to the run's default dataset.

1. Install the client

pip install apify-client

2. Execute the run

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "origin": "London",
    "destination": "Paris",
    "transport": "both",
    "sortBy": "priceLowToHigh",
    "maxItems": 50
}

run = client.actor("crawlerbros/omio-scraper").call(run_input=run_input)

# Fetch results from the default dataset
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item.get('transportType')}] {item.get('operator')}: {item.get('price')} {item.get('currency')}")
    print(f"  Depart: {item.get('departureStation')} at {item.get('departureTime')}")
    print(f"  Arrive: {item.get('arrivalStation')} at {item.get('arrivalTime')}\n")

Structured Data Schema

The scraper normalizes Virail's embedded trip metadata into flat records. Fields missing from the source page are omitted from the output rather than set to null values.

A typical item in the dataset contains the following schema:

{
  "recordType": "trip",
  "transportType": "train",
  "origin": "London",
  "destination": "Paris",
  "operator": "Eurostar",
  "departureStation": "London St Pancras International",
  "departureTime": "2024-06-15T07:01:00",
  "arrivalStation": "Paris Gare du Nord",
  "arrivalTime": "2024-06-15T10:17:00",
  "price": 89.50,
  "currency": "EUR",
  "availability": "InStock",
  "sourceUrl": "https://www.virail.com/...",
  "scrapedAt": "2024-05-01T12:00:00.000Z"
}

The availability key captures stock states (such as InStock) when Virail publishes that detail. The operator field identifies the underlying carrier (such as Eurostar, BlaBlaCar Bus, or FlixBus), which allows pipelines to segment routes by carrier rather than just aggregator visibility.

Pricing and Cost Predictability

Billing for this actor runs entirely on a pay-per-event pricing model. There are no separate infrastructure usage charges or subscription compute rates.

The two charged events are:

Actor Start (apify-actor-start): Charged once per run at $0.005 per GB of memory allocated to the run (minimum one event).
Result (apify-default-dataset-item): Charged per returned trip option added to the dataset.
- Base / FREE tier: $0.005 per event
- BRONZE tier: $0.00433 per event
- SILVER tier: $0.00367 per event
- GOLD, PLATINUM, and DIAMOND tiers: $0.003 per event

If you run a job allocated 1 GB of memory capped at maxItems: 50 on the FREE tier, the total run cost is $0.005 for the Actor Start event plus up to $0.25 for 50 result events ($0.005 × 50), totaling $0.255. If a search pair yields only 10 valid trips, you are charged for 10 result events ($0.05) plus the start event.

Route Resolution and Limitations

This approach does not book tickets or verify live seat selection maps; it only extracts public fare aggregates published by Virail at the moment of execution.

A common failure mode occurs when passing obscure destination names. If origin or destination cannot be resolved by Virail's routing index—such as passing a small suburb instead of a primary metropolitan name like Paris—the actor returns 0 results. When building scheduled monitors for multi-hop routes, input city names should first be normalized against standard transit hub aliases.

If you want to reproduce this, the Actor is Virail Train & Bus Trip Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

DEV Community