Building a Real-Time eSIM Price Index with AI

When we started building eSIMDB AI, one of the first hard questions was: how do you maintain accurate, current pricing across 120+ eSIM providers when each has its own pricing structure, API format, and update frequency?

This post covers the architecture we settled on, the problems we ran into, and what we'd do differently.

The Problem Space

Travel eSIM pricing is surprisingly volatile. Providers run flash sales, change regional plan structures, add or remove country coverage, and update pricing with little notice. A comparison that was accurate yesterday may show stale prices today — and sending users to buy a plan that's changed price (or no longer exists) destroys trust fast.

We also needed to handle 120+ providers, each with a different API structure (REST, GraphQL, scraping where no API existed), authentication method, rate limit policy, and data schema for plans, countries, prices, and features.

Architecture Overview

Our pipeline has three main layers.

1. Data Ingestion Layer

Each provider gets an adapter — a normalized interface that handles the specifics of that provider's API or data source. The adapter outputs a standardized plan object:

{
  "provider": "airalo",
  "plan_id": "airalo_eu_5gb_30d",
  "countries": ["FR", "DE", "ES"],
  "data_gb": 5,
  "validity_days": 30,
  "price_usd": 13.50,
  "price_eur": 12.40,
  "hotspot_allowed": true,
  "speed_5g": false,
  "activation_type": "qr",
  "last_updated": "2026-06-01T14:32:00Z"
}

For providers without APIs, we built scrapers — though we prefer official APIs or partnerships wherever possible.

2. Normalization & Scoring Layer

Raw plan data needs normalization before it's useful for comparison: country codes standardized to ISO 3166-1 alpha-2, currency conversion (we store all prices in USD + original currency), data amounts normalized to GB, and feature flags standardized.

Once normalized, each plan gets a composite value score:

def score_plan(plan, query):
    price_score = score_price(plan, query)          # 25%
    reliability_score = provider_reliability[plan.provider]  # 20%
    coverage_score = score_coverage(plan, query)    # 20%
    flexibility_score = score_flexibility(plan)     # 15%
    features_score = score_features(plan, query)    # 10%
    activation_score = score_activation(plan)       # 10%

    return weighted_sum([
        (price_score, 0.25),
        (reliability_score, 0.20),
        (coverage_score, 0.20),
        (flexibility_score, 0.15),
        (features_score, 0.10),
        (activation_score, 0.10),
    ])

Provider reliability scores come from a rolling average of user reviews and activation success rates, updated weekly.

3. Query & Retrieval Layer

When a user types "Paris 10 days 5GB under €20", the AI layer:

1. Extracts intent (destination: France, duration: 10 days, data: ~5GB, budget: €20)
2. Fetches candidate plans (country match + regional plans covering France)
3. Applies scoring with the user's specific context
4. Returns top 3–5 plans with honest tradeoff explanations

We use a fine-tuned small LLM for query understanding rather than regex — it handles ambiguous queries ("Paris and then maybe Amsterdam") much better.

Refresh Strategy

Running full refreshes on 120+ providers constantly is expensive and rate-limit-unfriendly. Our approach:

• High-priority providers (top 20 by volume): refresh every 6 hours
• Mid-tier providers: refresh every 12 hours
• Long-tail providers: refresh every 24 hours
• Event-triggered refresh: when our price anomaly detector flags a >15% price change, force-refresh that provider immediately

Challenges We Didn't Anticipate

Multi-country plan complexity. A "Europe" plan from one provider might cover 26 countries while another covers 42. We now explicitly show which specific countries each regional plan covers.

Currency and pricing volatility. USD/EUR shifts affect comparison accuracy. We refresh FX rates every hour and display prices in the user's inferred currency.

Provider reliability signal. A cheap plan from a provider with a 15% activation failure rate is worse value than a slightly more expensive plan from a reliable provider. Building this signal required aggregating review data from multiple sources.

What We'd Do Differently

1. Build provider adapters as plugins from day one. Early on we had provider-specific code scattered everywhere. Refactoring to a clean adapter pattern took longer than if we'd started there.

2. Invest in anomaly detection earlier. We had periods where stale pricing caused user complaints that better monitoring would have caught.

3. Separate the AI query layer from the comparison engine. They were tightly coupled initially, which made testing and iteration slower.

Current State

The index now covers 15,000+ plans across 195 countries, refreshes every 6–24 hours depending on provider tier, and serves comparisons in under 2 seconds. Try it at esimdb.ai — free, no sign-up.

Happy to discuss any of the architecture decisions in the comments.

---

Built with: Python, PostgreSQL, Redis, and a small fine-tuned LLM for query parsing.