From Wikipedia Tables to a Deterministic API: Engineering a Visa Lookup Actor with Apify

I started this project because I was looking for a visa API.

The requirement sounded simple: given a traveler's passport nationality and a destination country, return the visa requirements. But I couldn't find a reliable API that provided the information in the structured form I needed.

The information itself wasn't difficult to find. Wikipedia has detailed visa requirement pages for different nationalities. The problem was that the data was designed for people to read, not for applications to consume.

A developer shouldn't have to parse a long HTML table just to answer a question like:

"Can an Indian passport holder travel to Qatar without a visa?"

An application needs structured fields. An AI agent needs predictable output. An automation workflow needs data it can process without understanding the layout of a Wikipedia page.

So I decided to build an Apify Actor that would sit between the source and those consumers.

The initial idea was straightforward: take a nationality and destination, find the corresponding information on Wikipedia, and return it as structured data.

The first real challenge appeared once I started looking at multiple nationality pages.

The tables weren't as consistent as I expected.

Column names could differ. Visa terminology varied. Stay durations appeared in different formats, such as 30 days, 6 weeks , 1 year , or 90 days within 180 days . A parser that depended too heavily on one table layout could work for one nationality and fail on another.

That changed the focus of the project.

The difficult part wasn't downloading the Wikipedia page. It was building a parser that could tolerate changes in the source while still producing predictable output.

I eventually designed the Actor around that problem. It identifies the relevant visa table, extracts the destination row, normalizes common visa categories and stay durations, preserves the original wording when normalization isn't safe, and optionally adds localized fields.

The result is a deterministic interface over semi-structured public data.

In this article, I'll walk through how I built it, the engineering decisions that shaped the implementation, the problems I encountered while working with changing Wikipedia tables, and what I learned about building reliable data products from sources that weren't designed to be APIs.

What You'll Learn

By the end of this article, you'll have a practical example of how to:

• Build an Apify Actor around a public, semi-structured data source.
• Design a parser that can tolerate changes in HTML table structure.
• Normalize inconsistent human-readable data into a deterministic schema.
• Preserve source information when it can't be safely normalized.
• Separate crawling, parsing, localization, and output generation.
• Design outputs for applications, datasets, and AI-agent workflows.
• Test a scraper against different inputs and real-world edge cases.
• Think about reliability and maintainability when the source isn't under your control.

More importantly, you'll see the engineering decisions behind those choices—not just the final implementation.

The Moment I Decided to Build This

The idea started with a fairly ordinary developer problem: I needed a visa API.

I wasn't looking to build a scraping project for its own sake. I wanted a simple interface where I could provide a passport nationality and a destination and get back structured visa information.

I couldn't find a reliable API that gave me the information in the form I needed.

That left me with two choices: keep looking for a suitable data source, or build the missing layer myself.

Wikipedia was the obvious starting point because the information was already available and organized by nationality. The challenge was turning those human-readable pages into something software could consume reliably.

So I decided to build an Apify Actor around the data.

My initial mental model was simple:

That first version was enough to prove the idea.

But once I started testing different nationality pages, I realized that the real problem wasn't retrieving the page. It was dealing with the fact that the source could change underneath the parser.

That became the central engineering problem for the project:

How do you build a deterministic interface when the data source itself isn't deterministic?

The rest of the implementation followed from that question.

Defining Success Before Writing Code

Once I decided to build the Actor, I had to decide what a useful result actually looked like.

I didn't want to simply return whatever text happened to be present in the Wikipedia table. That would move the parsing problem from me to every developer using the Actor.

I also didn't want to make the parser overly aggressive. Visa requirements can contain conditions and exceptions that don't fit neatly into a fixed schema. Guessing in those cases could produce a cleaner-looking response but a less trustworthy one.

So I settled on a few principles that shaped the implementation.

Make the output predictable

Given a nationality and destination, the Actor should return a consistent structure.

Fields such as visaType , maxStayDays , stayPolicy , notes , and found give consumers predictable fields to work with instead of forcing them to parse the source HTML themselves.

Normalize where it is safe

If two pages express the same concept differently, the Actor should normalize those common variations.

For example, stay durations such as 30 days and 6 weeks can be represented numerically when the conversion is unambiguous.

But normalization shouldn't come at the cost of losing meaning.

Preserve information that doesn't fit the schema

When a rule contains conditions that can't be represented safely through a small set of structured fields, the original wording should remain available.

That's why the output keeps fields such as visaTypeRaw and allowedStayText alongside normalized values.

The goal isn't to force every piece of information into a structured field. The goal is to make the useful parts structured without throwing away context.

Keep localization additive

I wanted localization to improve the result without changing the underlying representation.

English remains the source of truth, while translated fields are added alongside it when localization is requested.

That means a consumer can use the English fields for stable programmatic logic while presenting localized values to users.

Keep the source complexity inside the Actor

The final principle was perhaps the most important one.

A developer using the Actor shouldn't need to know how Wikipedia organizes its tables.

They shouldn't need to understand different column names, visa terminology, or stay-duration formats.

That complexity belongs inside the Actor.

The public interface should expose the result, not the problems involved in obtaining it.

These principles gave me a useful boundary for the implementation: the source can be messy, but the interface should be predictable.

System architecture

Once I had a clear idea of what the Actor needed to guarantee, I could break the problem into smaller responsibilities.

The important distinction for me was between retrieving the page and understanding the page.

The crawler only needs to get the relevant Wikipedia document. The parsing layer figures out where the visa information is. The normalization layer turns that information into a predictable structure. Localization and output generation happen after that structured result exists.

That gives the Actor a pipeline like this:

The separation was intentional.

If the Wikipedia table changes, I should be able to adjust the extraction logic without rewriting the localization or report-generation code. If I change how a report is presented, I shouldn't have to touch the crawler.

The normalized result acts as the boundary between those responsibilities.

Before normalization, the data is still tied to the source. After normalization, the rest of the Actor can work with fields such as:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "visaType": "Visa-free",
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed",
  "notes": "...",
  "found": true
}

That boundary turned out to be important for another reason: it allowed me to add different outputs without duplicating the scraping logic.

The default dataset can contain the structured visa result. The additional-info dataset can contain related travel requirement links. The Key-Value Store can expose the complete structured result under RESULT, while the HTML report can present the same information for a human reader.

Localization follows the same boundary. The Actor first determines what the source says and how that information should be represented. Only then are user-facing fields translated.

The architecture is therefore less about having many components and more about keeping source-specific complexity in one place.

That separation gave me a useful mental model for the rest of the project:

The closer a component is to Wikipedia, the more it should understand about Wikipedia. The farther it is from the source, the less it should care about the source's structure.

With that architecture in place, the next question was much more concrete: What was the simplest way to retrieve and parse the page?

Challenge 1 — Choosing the simplest scraping strategy

Once I had the architecture in place, the first implementation decision was how to retrieve the Wikipedia page.

The Actor uses CheerioCrawler with Cheerio for HTML parsing. That choice came from the actual requirements of the source: the visa information I needed was available in the page's HTML, so I didn't need a browser to execute JavaScript or interact with the page.

That kept the crawling layer small.

The crawler's job is essentially to retrieve the document and hand it to the parsing logic. It doesn't need to understand visa rules or decide how a stay duration should be represented.

That separation matters because it keeps source retrieval independent from data interpretation.

A simplified view of the responsibility is:

The important part is what happens after the crawler receives the document. That's where the implementation has to deal with Wikipedia's changing table structures.

Using a lightweight HTML parser also fits the Actor's lookup-oriented design. The input describes a single nationality and destination, and the Actor is configured with maxRequestsPerCrawl to control the number of requests made during a run.

I didn't need browser automation simply because browser automation was available. The source and the task didn't require it.

That was the broader lesson from this decision:

Choose the smallest tool that solves the actual problem, not the largest tool that could solve it.

In this project, the difficult engineering work wasn't rendering a webpage. It was interpreting the data after the page had been retrieved.

Challenge 2 — Finding the right data reliably

The hardest part of the crawler wasn't downloading Wikipedia. It was deciding which part of the page I could trust.

A nationality page can contain multiple tables. I didn't want the parser to depend on a fixed table position because that would tightly couple the Actor to the current layout of the page.

Instead, I made the parser inspect the tables and identify the one that actually looks like a visa requirements table.

The implementation filters wikitable elements by their headers:

const table = $('table.wikitable').filter((_, el) => {
    const headers = $(el).find('th').map((_, th) =>
        $(th).text().trim().toLowerCase()
    ).get();

    const hasCountryColumn = headers.some(h =>
        h.includes('country') || h.includes('region')
    );

    const hasVisaColumn = headers.some(h =>
        h.includes('visa')
    );

    return hasCountryColumn && hasVisaColumn;
}).first();

The important detail here is that I'm not asking:

Is this the first table?

I'm asking:

"Does this table have the structure I expect from a visa table?"

That small distinction makes the scraper less dependent on the page's current ordering.

If no matching table exists, the Actor logs the problem and stops processing instead of trying to parse an arbitrary table:

if (!table.length) {
    log.error('Visa table not found');
    return;
}

Once I have the right table, I still can't assume that the columns are always in the same positions.

The parser therefore inspects the header row and records the indexes of the fields it needs:

let countryColIndex = -1;
let notesColIndex = -1;
let allowedStayColIndex = -1;
let visaColIndex = -1;

table.find('tr').first().find('th').each((index, th) => {
    const headerText = $(th).text().toLowerCase().trim();

    if (headerText.includes('country') ||
        headerText.includes('region')) {
        countryColIndex = index;
    }

    if (headerText.includes('allowed stay')) {
        allowedStayColIndex = index;
    }

    if (headerText.includes('visa')) {
        visaColIndex = index;
    }

    if (headerText.includes('notes')) {
        notesColIndex = index;
    }
});

This gives the parser a useful separation between table discovery and row parsing.

If the visa column moves, the parser doesn't need to know its new numeric position. It discovers the position from the header.

I also added explicit checks for required columns:

if (countryColIndex === -1) {
    log.error('Country column not found');
    return;
}

if (notesColIndex === -1) {
    log.error('Notes column not found');
    return;
}

This is important because a scraper can fail in two very different ways.

It can fail loudly because the expected structure is missing, or it can continue with the wrong structure and produce incorrect data.

I prefer the first.

Once the column indexes are known, the parser walks through the rows and compares the destination against the requested input:

const country = $(cols[countryColIndex]).text().trim();

if (
    normalizeForCompare(country) ===
    normalizeForCompare(destinationInput)
) {
    // Parse and normalize the matching row
}

The comparison is deliberately normalized:

const normalizeForCompare = (value: string) =>
    value.trim().toLowerCase().replace(/[^a-z]/g, '');

This means differences in capitalization, whitespace, and non-letter characters don't unnecessarily prevent a match.

The key design decision here is that I made the parser structure-aware rather than position-dependent.

That doesn't make the scraper immune to changes. Wikipedia can still change its headers or table structure in ways the parser doesn't recognize. But it gives the Actor a much better starting point than hardcoding assumptions about where the data happens to appear today.

And that led directly to the next problem.

Finding the correct cell was only half the job. The contents of those cells were inconsistent too.

Challenge 3 — Turning human text into deterministic data

Finding the right row solved only half of the problem.

The values inside that row were still written for humans.

Visa descriptions could contain different terminology for the same concept, while allowed-stay values could appear as days, weeks, months, years, or qualitative rules such as "Unlimited".

My first instinct was to keep the scraped text as-is. That would have been easy, but it would also push the hardest part of the problem onto every consumer of the Actor.

Instead, I added a normalization layer.

Normalizing visa types

The Actor maps common phrases into a small set of predictable categories:

type VisaType =
    | 'Visa-free'
    | 'Visa on arrival'
    | 'eVisa'
    | 'Visa required'
    | 'Freedom of movement'
    | 'Other';

The parser first cleans the Wikipedia text, converts it to lowercase for matching, and then checks known patterns.

For example:

const raw = cleanWikipediaText(text);
const t = raw.toLowerCase();

if (t.includes('freedom of movement')) {
    return { visaType: 'Freedom of movement', visaTypeRaw: raw };
}

if (t.includes('visa not required') || t.includes('visa-free')) {
    return { visaType: 'Visa-free', visaTypeRaw: raw };
}

I intentionally kept the original cleaned value in visaTypeRaw.

That decision is important because normalization is an interpretation. I don't want to lose the source value just because I created a more convenient representation for applications.

The parser also handles combinations. If the source mentions both an online visa and visa on arrival, the result records the primary category as Visa on arrival and preserves the available options:

if (t.includes('online visa') && t.includes('visa on arrival')) {
    return {
        visaType: 'Visa on arrival',
        visaTypeRaw: raw,
        visaOptions: ['eVisa', 'Visa on arrival'],
    };
}

If none of the known patterns match, the parser doesn't guess. It returns Other and keeps the original text.

That fallback became an important rule throughout the project:

An unknown value is better than an incorrect classification.

Parsing allowed stay

Stay duration required a similar approach.

I wanted applications to have a numeric value when the source made that possible, but I didn't want to throw away the human-readable version.

The resulting structure contains three related fields:

{
    maxStayDays,
    allowedStayText,
    stayPolicy
}

The parser handles common units explicitly.

const yearMatch = t.match(/(\d+)\s*year/);

if (yearMatch) {
    return {
        maxStayDays: Number(yearMatch[1]) * 365,
        allowedStayText: raw,
        stayPolicy: 'fixed',
    };
}

The same approach is used for months, weeks, and days. Months are converted using 30 days, years using 365 days, and weeks using 7 days.

For Unlimited, there is no meaningful numeric maximum, so maxStayDays remains null while stayPolicy captures the meaning:

{
    maxStayDays: null,
    allowedStayText: "Unlimited",
    stayPolicy: "unlimited"
}

The parser also distinguishes between values that describe a limited period, conditional rules, and values it doesn't understand. When a value cannot be normalized safely, the original cleaned text remains available and stayPolicy can fall back to unknown.

This is where I found the value of separating normalized data from source text

A developer can filter or compare maxStayDays when that value is meaningful. A human can still inspect allowedStayText to understand what the source actually said.

I don't have to choose between machine-readable data and source fidelity.

I can provide both.

Why I didn't use a more sophisticated parser

The goal wasn't to build a natural-language understanding system for visa policies.

I needed predictable behavior for common patterns.

Regular expressions and explicit rules were enough for that layer because they make the transformation visible and easy to reason about. More importantly, when a rule doesn't match, the parser has a clear fallback instead of inventing a result.

That trade-off is intentional.

The normalization layer handles the cases it understands and preserves the cases it doesn't.

That makes the system easier to debug when Wikipedia introduces a new format.

And that is the real purpose of this layer: not to eliminate ambiguity, but to keep ambiguity from leaking into every application that consumes the Actor.

Challenge 4 — Designing an API instead of exposing HTML

Once the parser could extract the right row and normalize its contents, I had another design question:

What exactly should the Actor return?

The easiest option would have been to return the scraped table data and let consumers interpret it themselves.

I didn't want that.

If I returned the Wikipedia structure directly, every developer using the Actor would still need to understand the source's terminology, column layout, and inconsistent stay descriptions. I would have moved the scraping problem rather than solved it.

Instead, I designed the output around what a consumer actually needs.

A visa result contains fields such as:

interface VisaResult {
    nationality: string;
    destination: string;
    visaType: VisaType;
    visaTypeRaw?: string;
    visaOptions?: string[];
    maxStayDays: number | null;
    allowedStayText: string;
    stayPolicy: StayPolicy;
    notes: string;
    found: boolean;
}

Each field has a different responsibility.

visaType gives applications a normalized value they can work with.

visaTypeRaw preserves the source-derived wording when the normalized category doesn't tell the whole story.

maxStayDays provides a numeric representation when one can be derived.

allowedStayText keeps the human-readable description.

stayPolicy tells consumers how to interpret the duration.

And found makes the "no matching visa information" case explicit instead of forcing consumers to infer it from missing fields.

Why keep both normalized and raw values?

This was one of the most important schema decisions.

Suppose the source contains a detailed visa condition. A normalized field might tell an application that the traveler is Visa-free for 30 days, but that doesn't necessarily capture every condition attached to that rule.

If I only returned the normalized value, I would be throwing away information during parsing.

By keeping the source-derived value as well, consumers can choose the level of abstraction they need.

For example:

{
    "visaType": "Visa-free",
    "visaTypeRaw": "Visa waiver for 30 days, subject to conditions",
    "maxStayDays": 30,
    "allowedStayText": "30 days",
    "stayPolicy": "fixed"
}

An automated workflow can use visaType and maxStayDays.

A human-facing application can display allowedStayText and notes.

A developer debugging an unexpected result can inspect the raw value.

The same record serves all three use cases.

Making "not found" explicit

Another small but important choice was the found field.

A destination might not have a matching entry, or the expected table might not be available on the source page.

I didn't want consumers to have to distinguish between:

{}

and

{
    "found": false
}

The second representation communicates intent much more clearly.

It also means the Actor can return a structured result even when there isn't a visa rule to report.

That makes the API easier to consume because callers can handle success and absence through the same response shape.

The schema becomes the boundary

At this point, the normalized result became the most important boundary in the system.

Before it, the implementation is concerned with Wikipedia:

After it, the implementation doesn't need to know how Wikipedia stores the information:

That separation gave me freedom to change the source-specific parsing logic without forcing every consumer to change with it.

It also made the Actor easier to extend. New output formats can consume the same normalized result instead of implementing their own interpretation of the source.

This was the point where I stopped thinking of the Actor as a scraper with an output attached to it.

The normalized schema was becoming the product interface

The scraper was simply one implementation detail behind it.

Challenge 5 — One source, multiple consumers

Once I had a normalized VisaResult, I didn't want to build a separate representation for every consumer.

The same lookup might be used by a developer through an API, exported for analysis, displayed to a human, or passed into an AI workflow. Creating separate parsing paths for each use case would have made the Actor harder to maintain and created opportunities for the outputs to disagree.

Instead, I kept one normalized result and built different outputs around it.

The default dataset

The default dataset is the primary structured output.

Each run produces the visa result as a dataset item, including fields such as the nationality, destination, normalized visa type, stay information, notes, language, and timestamp.

This is useful when the result needs to be exported or consumed as structured data.

The important part is that the dataset doesn't expose the Wikipedia table itself. It exposes the normalized contract we designed earlier.

The Key-Value Store

For a single lookup, a dataset isn't always the most convenient representation.

The Actor also stores the complete structured result under the RESULT key in the Key-Value Store.

That gives consumers another way to retrieve the same lookup without having to interpret a collection of dataset items.

The two outputs serve different access patterns, but they originate from the same normalized result.

The additional-info dataset

The visa table isn't the only useful information on a nationality page.

The Actor also extracts related travel-information links, such as passport rules, vaccination information, biometrics, or other nationality-specific requirements.

I kept these in a separate additional-info dataset rather than mixing them into the primary visa record.

That keeps the main result focused on the question the Actor was built to answer while still making related information available to consumers that need it.

The HTML report

Not every consumer needs raw JSON.

The Actor also generates an HTML report from the result. This gives someone running the Actor through the Apify interface a readable representation without requiring them to inspect a dataset or parse JSON manually.

The important architectural decision is that the report is generated after normalization.

It doesn't scrape Wikipedia independently.

That means the HTML report and the structured outputs are different representations of the same underlying result.

Localization without replacing the source

Localization introduced another design consideration.

I wanted users to be able to request output in languages such as Hindi, French, Spanish, or German, but I didn't want translation to change the canonical data.

The Actor therefore keeps English as the source of truth and adds localized fields when a non-English language is requested.

For example:

The English fields remain available for programmatic consumers, while the localized fields can be used directly in user-facing applications.

This distinction matters because translation is a presentation concern. It shouldn't change the meaning of the normalized data.

The localization layer also has a fallback: if translation fails, the Actor can continue returning the English result rather than making localization a dependency for the underlying visa lookup.

One normalized result, several interfaces

Looking at the outputs together, the architecture becomes simple:

The benefit isn't just convenience.

It means that if the parsing logic changes, I don't have to update four independent implementations. If I add another output format later, that format can consume the same normalized result.

This became an important maintainability rule for the project:

Parse once, normalize once, then adapt the result to the consumer.

That kept the source-specific complexity contained inside the Actor and prevented each output format from becoming its own mini-scraper.

Engineering for Production

Once the parser was producing the data I wanted, the next question was whether the Actor would behave predictably when things went wrong.

A scraper doesn't control its source. Wikipedia can change. A requested destination might not exist on the page. A table might be missing. A user might provide input that doesn't map cleanly to the source.

I decided early that these cases should produce understandable outcomes rather than obscure failures.

Validate before crawling

The Actor requires both a nationality and a destination.

That validation happens before the crawler starts doing useful work:

if (!nationality || !destination) {
    throw new Error(
        'Input must include both "nationality" and "destination".'
    );
}

This is deliberately simple.

There's no reason to make a network request when the request itself is incomplete.

It also means an invalid invocation fails close to the point where the problem was introduced, rather than producing a confusing parsing error later.

Treat missing source data as a normal case

One of the situations I wanted to handle explicitly was a missing visa table.

The parser doesn't assume that the expected table will always be present. If it can't identify the table, it logs the problem and stops processing that lookup.

The same principle applies when the destination cannot be found.

Rather than pretending that missing information means "visa required" or another default category, the Actor can return a structured result with found: false.

That distinction is important.

No data found is not the same thing as a visa requirement.

The Actor should report what it knows rather than invent an answer.

Keep localization out of the critical path

Localization is useful, but it shouldn't determine whether the underlying visa lookup succeeds.

The Actor first extracts and normalizes the English data. Translation is then applied to user-facing fields.

If localization fails, the Actor falls back to English.

That makes translation an enhancement rather than a dependency of the core scraping pipeline.

It also makes debugging easier: if a run produces the wrong visa result, I can investigate the source parsing independently of the translation layer.

Limit the crawl

The Actor exposes maxRequestsPerCrawl as an input parameter, with a default of 1.

For this lookup-oriented Actor, the default reflects the expected workload: a single nationality-to-destination lookup should not require an uncontrolled crawl.

Keeping the request budget explicit also gives the Actor a simple boundary around network activity.

{
    "nationality": "Indian",
    "destination": "Qatar",
    "maxRequestsPerCrawl": 1
}

The important point isn't the number itself. It's that network work is treated as a resource rather than something the parser can perform without limits.

Make failures observable

Logging became especially useful while working with changing Wikipedia tables.

The Actor logs meaningful stages of the lookup, including situations where the expected table or columns cannot be found.

For example:

log.error('Visa table not found');

A scraper that silently returns an empty result can be difficult to diagnose. A scraper that tells you which structural assumption failed gives you a starting point for fixing the parser.

This matters particularly for public sources because a parser can be perfectly correct against yesterday's HTML and still fail after a source-side change.

Keep stored data fresh

The additional-info dataset is cleared on each run before new additional travel information is stored.

That decision is deliberate.

The Actor represents a current lookup, so retaining stale named-dataset records from previous runs could make the output confusing.

The same principle applies to the generated result: each run should represent the data retrieved during that run, with scrapedAt recording when the lookup occurred.

Deployment is part of the engineering process

My deployment workflow remained intentionally straightforward:

I didn't treat deployment as the end of development.

After deploying, I verified the actual Actor run and checked the outputs rather than assuming that a successful build meant the complete pipeline was working.

That distinction matters for Actors because the final behavior depends on more than TypeScript compiling successfully. The crawler, source website, datasets, Key-Value Store, localization, and report generation all need to work together.

What I would add for a larger workload

The current Actor is designed around a focused nationality-to-destination lookup rather than a large-scale crawl.

If the workload grew substantially, I would revisit areas such as caching, more explicit retry policies, concurrency controls, and stronger monitoring.

Those aren't claims about the current implementation. They're the next engineering questions I'd consider once the workload justified the additional complexity.

For this project, I preferred to keep the implementation proportional to the problem.

That became another useful lesson:

Production quality doesn't mean adding every reliability mechanism you can think of. It means making the failure modes you actually have predictable, observable, and maintainable.

Using the Actor

Once the parsing and normalization layers were working, I wanted the Actor to be straightforward to consume.

The basic interface is intentionally small:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "language": "hi"
}

The Actor takes a passport nationality and destination country, performs the lookup, and returns structured visa information.

The optional language parameter adds localized fields without replacing the canonical English values.

cURL

For a quick test or a shell-based workflow, the Actor can be invoked through the Apify API.

curl -X POST \
  "https://api.apify.com/v2/acts/YOUR_ACTOR_ID/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "nationality": "Indian",
    "destination": "Qatar",
    "language": "en"
  }'

The important part of this request is the input rather than the client. The same Actor contract is used regardless of whether the run is started from the Apify UI, an SDK, or an HTTP request.

For automation, cURL is useful when the Actor needs to be called from a shell script, CI job, or another service that doesn't require a full SDK.

JavaScript

A JavaScript application can use the Apify client to start the Actor and retrieve the resulting dataset.

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: process.env.APIFY_TOKEN,
});

const run = await client
    .actor('YOUR_ACTOR_ID')
    .call({
        nationality: 'Indian',
        destination: 'Qatar',
        language: 'en',
    });

const { items } = await client
    .dataset(run.defaultDatasetId)
    .listItems();

console.log(items[0]);

The useful part of this integration is that the application doesn't need to know anything about Wikipedia.

It doesn't need to construct a Wikipedia URL, locate a table, identify columns, or parse visa terminology.

It only needs to provide the lookup parameters and consume the normalized result.

Python

The same workflow works from Python using the Apify client:

from apify_client import ApifyClient

client = ApifyClient(
    token="YOUR_APIFY_TOKEN"
)

run = client.actor("YOUR_ACTOR_ID").call(
    run_input={
        "nationality": "Indian",
        "destination": "Qatar",
        "language": "en",
    }
)

items = client.dataset(
    run["defaultDatasetId"]
).list_items().items

print(items[0])

This is useful for data-processing scripts, notebooks, reporting pipelines, and applications that already use Python.

Again, the important abstraction is the same: Python consumes the Actor's structured result instead of implementing its own Wikipedia parser.

Working with the dataset

The default dataset contains the primary visa result.

A typical record contains fields such as:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "visaType": "Visa-free",
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed",
  "notes": "...",
  "language": "en",
  "found": true,
  "scrapedAt": "2025-12-26T08:24:19.686Z"
}

Because the result is stored as structured dataset data, it can also be exported through Apify rather than requiring another transformation step.

This is one reason I chose a dataset as a primary output: it works equally well for an application consuming JSON and for someone who wants to export the result for analysis.

Retrieving the complete result

For consumers that want the complete structured response as a single object, the Actor also writes the result to the Key-Value Store under the RESULT key.

That provides a second access pattern:

The dataset and Key-Value Store therefore aren't competing representations. They solve slightly different consumption patterns while sharing the same underlying result.

Localization

If a language other than English is provided, localized fields are added to the response.

For example:

{
  "visaType": "Visa-free",
  "visaType_localized": "वीज़ा-मुक्त",
  "allowedStayText": "30 days",
  "allowedStayText_localized": "30 दिन"
}

The English fields remain available.

This is important for applications that need both machine-stable values and user-facing translations. An application can use visaType for its internal logic while displaying visaType_localized to the traveler.

What the client doesn't need to know

The most important part of this API isn't the invocation syntax.

It's everything the caller doesn't have to implement.

A consumer doesn't need to:

• scrape Wikipedia,
• identify the correct table,
• detect changing columns,
• normalize visa terminology,
• parse stay durations,
• extract additional travel links,
• or implement localization fallback.

Those responsibilities remain inside the Actor.

The client gets the result through a stable interface.

That was the original reason I wanted an API in the first place. I wasn't trying to make developers better at scraping Wikipedia. I wanted them to stop having to scrape Wikipedia at all.

Proving the Parser Works

A parser that works for one Wikipedia page isn't necessarily a reliable parser.

The source was the part of the system I didn't control, so testing had to cover variation rather than just the happy path.

I tested the Actor against different passport nationalities and destination combinations, including different visa policies and different ways of expressing allowed stay.

The goal was to verify two things at the same time:

1. the Actor could find the correct information;
2. the normalization layer could represent that information without losing important context.

Testing different visa policies

I tested destinations covering several common cases:

• Visa-free
• Visa on arrival
• eVisa
• Visa required

These cases exercise different branches of the visa-type parser.

The important part wasn't simply getting a non-empty response. I checked that the normalized visaType matched the intended category while the source-derived information remained available where necessary.

That distinction matters because a scraper can successfully return data while still interpreting it incorrectly.

Testing stay-duration formats

Allowed-stay parsing received particular attention because the source uses several different representations.

I tested values such as:

30 days
6 weeks
1 year
Unlimited
90 days within 180 days

These aren't equivalent from a parsing perspective.

Some can be converted directly into a numeric duration. Others contain additional conditions that need to remain visible in the original text.

That testing helped shape the current representation:

{
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed"
}

The numeric field is useful for software, while the text field preserves what a traveler actually needs to read.

Testing source variation

I also tested the parser against multiple Wikipedia nationality pages.

This was important because the parser wasn't supposed to work only for the page I happened to use during development.

Different pages can expose different table structures and terminology, so testing multiple nationality pages helped verify that the implementation was discovering columns rather than depending entirely on fixed positions.

This is one of the reasons I consider header-based table detection an important part of the implementation.

Testing invalid and missing data

I also tested cases where the requested nationality or destination didn't produce a usable visa result.

The Actor should not turn missing information into a guessed visa requirement.

Instead, the result can explicitly indicate that the lookup wasn't found:

{
  "found": false
}

That gives downstream applications a state they can handle deliberately.

The same principle applies to missing notes or unfamiliar visa terminology. Missing information should remain missing rather than being replaced with invented content.

Testing localization

Localization was tested independently from the core visa parsing.

I verified localized output in multiple supported languages and checked that the English fields remained available alongside the translated fields.

This separation was important because localization should not change the underlying result.

I also tested the fallback behavior. If localization isn't available, the Actor should still be able to return the English result.

That gives the pipeline a useful failure boundary:

Testing the complete output pipeline

The final step was checking the outputs generated from each run.

I verified:

• the default dataset,
• the additional-info dataset,
• the RESULT Key-Value Store record,
• the HTML report,
• and localized fields where requested.

This matters because successful parsing doesn't guarantee successful output generation.

The Actor is a pipeline, and I wanted to verify the pipeline rather than only one function inside it.

What testing changed

Testing wasn't just a final verification step.

It influenced the design.

The more variations I encountered in visa terminology and stay durations, the more important it became to preserve raw values alongside normalized ones.

The more nationality pages I checked, the more important dynamic table and column detection became.

And the more output formats I added, the more useful it became to have one normalized result as the source for all of them.

That was probably the biggest lesson from testing:

Testing a scraper means testing your assumptions about the source.

The code can be correct according to its own logic and still be wrong about the world it is scraping.

For this Actor, the most valuable tests were therefore the ones that challenged those assumptions.

## Lessons Learned

Building this Actor changed how I think about scraping.

I started with a fairly simple goal: find visa information and expose it through an API. I expected the main challenge to be retrieving the data.

It turned out that retrieval was the easy part.

The difficult part was deciding what to do with information that was inconsistent, incomplete, or likely to change.

A few lessons stood out.

Scraping is only the first step

The crawler gets the data into the system. It doesn't make that data useful.

The real work started after the HTML had been downloaded: identifying the right table, finding the right row, interpreting visa terminology, parsing stay durations, and deciding which information could safely be normalized.

That changed my mental model of scraping.

I no longer see the crawler as the main part of the system. It's the entry point into a data-processing pipeline.

Normalization is where most of the value is created

The more Wikipedia pages I tested, the clearer this became.

Extracting:

"30 days"
is easy.

Deciding whether that should become:

{
  "maxStayDays": 30,
  "stayPolicy": "fixed"
}

while still preserving the original text is the more interesting engineering problem.

The normalized representation is what makes the data useful to software.

Without it, every consumer would have to implement its own interpretation of the source.

Preserving the source prevents over-normalization

I initially thought mostly in terms of structured fields.

But visa requirements contain conditions and exceptions that don't always fit cleanly into a predefined schema.

That made visaTypeRaw, allowedStayText, and notes important parts of the design.

I don't need to choose between structured data and source fidelity.

I can provide both.

That also gives me a safer fallback when Wikipedia introduces terminology that the parser doesn't recognize yet.

Unknown is better than wrong

This became one of the strongest principles in the project.

A parser can always be made to produce some category. That doesn't mean the category will be correct.

When the Actor encounters an unfamiliar visa description, preserving the original information and returning Other is safer than guessing.

The same principle applies to stay durations.

If I can't safely derive a numeric value, I would rather return null and preserve the original text than manufacture a number that looks precise but isn't.

For data that may eventually be consumed by applications or AI systems, that distinction matters.

Source changes should be expected

Wikipedia isn't an API that I control.

Its content and table structures can evolve independently of my Actor.

That means the parser shouldn't be designed around one exact snapshot of the HTML.

The header-based table detection and dynamic column discovery were direct responses to that problem.

They don't guarantee that the parser will survive every future change, but they reduce unnecessary coupling to the current layout.

More importantly, they make the assumptions in the parser visible and easier to update.

Architecture becomes more valuable as features accumulate

The project started with a relatively small core: retrieve the page, parse the table, and return the result.

Then the requirements expanded.

Localization was added.

Additional travel information was extracted.

A separate dataset was introduced.

The Key-Value Store became another output.

An HTML report was added.

At that point, keeping everything in one place would have made each new feature harder to implement.

Separating crawling, parsing, normalization, localization, reporting, storage, and utilities gave each part a clearer responsibility.

The refactor didn't make the Actor more impressive on the surface. It made the code easier to change.

That's the kind of improvement that becomes valuable only after a project starts evolving.

Design around consumers, not the source

The source determines what information is available.

It shouldn't determine what the API looks like.

Wikipedia's table structure is useful for Wikipedia.

It isn't necessarily a useful contract for a JavaScript application, a Python data pipeline, or an AI agent.

That distinction influenced the entire output design.

The Actor exposes normalized data, localized user-facing fields, datasets, a Key-Value Store result, and an HTML report because those are useful consumption patterns—not because Wikipedia happens to provide them in that form.

The broader lesson

Looking back, I don't think the most important thing I built was the scraper.

The more valuable part was the abstraction around it.

The Actor takes information written for humans, applies a set of explicit rules to interpret it, preserves uncertainty where necessary, and exposes the result through a predictable interface.

That's a pattern I can reuse far beyond visa requirements.

Whenever an application depends on a public website that wasn't designed to be an API, the same engineering problem appears:

How do you turn an unpredictable source into an interface that your software can trust?

That became the real lesson of this project.

What Happened After I Shipped It

After testing the Actor locally, I deployed it to Apify and started iterating on it as a real project rather than treating the first working run as the finish line.

The project was eventually selected for Best Newcomer at the Apify challenge, an award for a developer who joined Apify during the challenge and made an immediate impact.

That recognition was meaningful to me because the project started from a relatively simple problem: I was looking for a visa API and couldn't find a reliable one.

The award wasn't the reason I built the Actor, but it was useful validation that the project was solving a problem in a way that was valuable to the Apify developer community.

More importantly, the process reinforced something I had learned while building it: a working scraper is only the beginning. The parts that make the project useful to other developers—predictable output, resilient parsing, multiple consumption formats, localization, and maintainable architecture—are what turn a one-off scraper into something people can actually build on.

Conclusion

I started this project because I wanted a visa API.

What I ended up building was more than a way to scrape visa tables. The interesting part was creating a reliable boundary between a changing public source and the applications that depend on its data.

Wikipedia provides information for people to read. The Actor turns that information into structured data that software can consume.

The crawler retrieves the page. The parser identifies the relevant table and row. The normalization layer turns inconsistent visa descriptions and stay durations into predictable fields while preserving the original wording. From there, the same normalized result can be exposed through datasets, the Key-Value Store, an HTML report, and optional localized fields.

The most important lesson for me was that scraping is only the beginning.

The real engineering work starts when you ask what the data should look like after it leaves the website.

Public sources will change. New terminology will appear. Some rules won't fit neatly into a predefined schema. A reliable system shouldn't hide those problems—it should make its assumptions explicit, preserve information it can't safely interpret, and give consumers a stable interface.

That's the approach I took with this Actor.

If I were starting another project that depended on a public website, I'd ask a different question now. Instead of starting with:

"How do I scrape this page?”

I'd start with:

"What interface should the software consuming this data see?"

That shift in perspective is probably the most valuable thing this project taught me.