DEV Community: Raj Gupta

From Wikipedia Tables to a Deterministic API: Engineering a Visa Lookup Actor with Apify

Raj Gupta — Wed, 12 Aug 2026 20:42:30 +0000

I started this project because I was looking for a visa API.

The requirement sounded simple: given a traveler's passport nationality and a destination country, return the visa requirements. But I couldn't find a reliable API that provided the information in the structured form I needed.

The information itself wasn't difficult to find. Wikipedia has detailed visa requirement pages for different nationalities. The problem was that the data was designed for people to read, not for applications to consume.

A developer shouldn't have to parse a long HTML table just to answer a question like:

"Can an Indian passport holder travel to Qatar without a visa?"

An application needs structured fields. An AI agent needs predictable output. An automation workflow needs data it can process without understanding the layout of a Wikipedia page.

So I decided to build an Apify Actor that would sit between the source and those consumers.

The initial idea was straightforward: take a nationality and destination, find the corresponding information on Wikipedia, and return it as structured data.

The first real challenge appeared once I started looking at multiple nationality pages.

The tables weren't as consistent as I expected.

Column names could differ. Visa terminology varied. Stay durations appeared in different formats, such as 30 days, 6 weeks , 1 year , or 90 days within 180 days . A parser that depended too heavily on one table layout could work for one nationality and fail on another.

That changed the focus of the project.

The difficult part wasn't downloading the Wikipedia page. It was building a parser that could tolerate changes in the source while still producing predictable output.

I eventually designed the Actor around that problem. It identifies the relevant visa table, extracts the destination row, normalizes common visa categories and stay durations, preserves the original wording when normalization isn't safe, and optionally adds localized fields.

The result is a deterministic interface over semi-structured public data.

In this article, I'll walk through how I built it, the engineering decisions that shaped the implementation, the problems I encountered while working with changing Wikipedia tables, and what I learned about building reliable data products from sources that weren't designed to be APIs.

What You'll Learn

By the end of this article, you'll have a practical example of how to:

Build an Apify Actor around a public, semi-structured data source.
Design a parser that can tolerate changes in HTML table structure.
Normalize inconsistent human-readable data into a deterministic schema.
Preserve source information when it can't be safely normalized.
Separate crawling, parsing, localization, and output generation.
Design outputs for applications, datasets, and AI-agent workflows.
Test a scraper against different inputs and real-world edge cases.
Think about reliability and maintainability when the source isn't under your control.

More importantly, you'll see the engineering decisions behind those choices—not just the final implementation.

The Moment I Decided to Build This

The idea started with a fairly ordinary developer problem: I needed a visa API.

I wasn't looking to build a scraping project for its own sake. I wanted a simple interface where I could provide a passport nationality and a destination and get back structured visa information.

I couldn't find a reliable API that gave me the information in the form I needed.

That left me with two choices: keep looking for a suitable data source, or build the missing layer myself.

Wikipedia was the obvious starting point because the information was already available and organized by nationality. The challenge was turning those human-readable pages into something software could consume reliably.

So I decided to build an Apify Actor around the data.

My initial mental model was simple:

That first version was enough to prove the idea.

But once I started testing different nationality pages, I realized that the real problem wasn't retrieving the page. It was dealing with the fact that the source could change underneath the parser.

That became the central engineering problem for the project:

How do you build a deterministic interface when the data source itself isn't deterministic?

The rest of the implementation followed from that question.

Defining Success Before Writing Code

Once I decided to build the Actor, I had to decide what a useful result actually looked like.

I didn't want to simply return whatever text happened to be present in the Wikipedia table. That would move the parsing problem from me to every developer using the Actor.

I also didn't want to make the parser overly aggressive. Visa requirements can contain conditions and exceptions that don't fit neatly into a fixed schema. Guessing in those cases could produce a cleaner-looking response but a less trustworthy one.

So I settled on a few principles that shaped the implementation.

Make the output predictable

Given a nationality and destination, the Actor should return a consistent structure.

Fields such as visaType , maxStayDays , stayPolicy , notes , and found give consumers predictable fields to work with instead of forcing them to parse the source HTML themselves.

Normalize where it is safe

If two pages express the same concept differently, the Actor should normalize those common variations.

For example, stay durations such as 30 days and 6 weeks can be represented numerically when the conversion is unambiguous.

But normalization shouldn't come at the cost of losing meaning.

Preserve information that doesn't fit the schema

When a rule contains conditions that can't be represented safely through a small set of structured fields, the original wording should remain available.

That's why the output keeps fields such as visaTypeRaw and allowedStayText alongside normalized values.

The goal isn't to force every piece of information into a structured field. The goal is to make the useful parts structured without throwing away context.

Keep localization additive

I wanted localization to improve the result without changing the underlying representation.

English remains the source of truth, while translated fields are added alongside it when localization is requested.

That means a consumer can use the English fields for stable programmatic logic while presenting localized values to users.

Keep the source complexity inside the Actor

The final principle was perhaps the most important one.

A developer using the Actor shouldn't need to know how Wikipedia organizes its tables.

They shouldn't need to understand different column names, visa terminology, or stay-duration formats.

That complexity belongs inside the Actor.

The public interface should expose the result, not the problems involved in obtaining it.

These principles gave me a useful boundary for the implementation: the source can be messy, but the interface should be predictable.

System architecture

Once I had a clear idea of what the Actor needed to guarantee, I could break the problem into smaller responsibilities.

The important distinction for me was between retrieving the page and understanding the page.

The crawler only needs to get the relevant Wikipedia document. The parsing layer figures out where the visa information is. The normalization layer turns that information into a predictable structure. Localization and output generation happen after that structured result exists.

That gives the Actor a pipeline like this:

The separation was intentional.

If the Wikipedia table changes, I should be able to adjust the extraction logic without rewriting the localization or report-generation code. If I change how a report is presented, I shouldn't have to touch the crawler.

The normalized result acts as the boundary between those responsibilities.

Before normalization, the data is still tied to the source. After normalization, the rest of the Actor can work with fields such as:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "visaType": "Visa-free",
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed",
  "notes": "...",
  "found": true
}

That boundary turned out to be important for another reason: it allowed me to add different outputs without duplicating the scraping logic.

The default dataset can contain the structured visa result. The additional-info dataset can contain related travel requirement links. The Key-Value Store can expose the complete structured result under RESULT, while the HTML report can present the same information for a human reader.

Localization follows the same boundary. The Actor first determines what the source says and how that information should be represented. Only then are user-facing fields translated.

The architecture is therefore less about having many components and more about keeping source-specific complexity in one place.

That separation gave me a useful mental model for the rest of the project:

The closer a component is to Wikipedia, the more it should understand about Wikipedia. The farther it is from the source, the less it should care about the source's structure.

With that architecture in place, the next question was much more concrete: What was the simplest way to retrieve and parse the page?

Challenge 1 — Choosing the simplest scraping strategy

Once I had the architecture in place, the first implementation decision was how to retrieve the Wikipedia page.

The Actor uses CheerioCrawler with Cheerio for HTML parsing. That choice came from the actual requirements of the source: the visa information I needed was available in the page's HTML, so I didn't need a browser to execute JavaScript or interact with the page.

That kept the crawling layer small.

The crawler's job is essentially to retrieve the document and hand it to the parsing logic. It doesn't need to understand visa rules or decide how a stay duration should be represented.

That separation matters because it keeps source retrieval independent from data interpretation.

A simplified view of the responsibility is:

The important part is what happens after the crawler receives the document. That's where the implementation has to deal with Wikipedia's changing table structures.

Using a lightweight HTML parser also fits the Actor's lookup-oriented design. The input describes a single nationality and destination, and the Actor is configured with maxRequestsPerCrawl to control the number of requests made during a run.

I didn't need browser automation simply because browser automation was available. The source and the task didn't require it.

That was the broader lesson from this decision:

Choose the smallest tool that solves the actual problem, not the largest tool that could solve it.

In this project, the difficult engineering work wasn't rendering a webpage. It was interpreting the data after the page had been retrieved.

Challenge 2 — Finding the right data reliably

The hardest part of the crawler wasn't downloading Wikipedia. It was deciding which part of the page I could trust.

A nationality page can contain multiple tables. I didn't want the parser to depend on a fixed table position because that would tightly couple the Actor to the current layout of the page.

Instead, I made the parser inspect the tables and identify the one that actually looks like a visa requirements table.

The implementation filters wikitable elements by their headers:

const table = $('table.wikitable').filter((_, el) => {
    const headers = $(el).find('th').map((_, th) =>
        $(th).text().trim().toLowerCase()
    ).get();

    const hasCountryColumn = headers.some(h =>
        h.includes('country') || h.includes('region')
    );

    const hasVisaColumn = headers.some(h =>
        h.includes('visa')
    );

    return hasCountryColumn && hasVisaColumn;
}).first();

The important detail here is that I'm not asking:

Is this the first table?

I'm asking:

"Does this table have the structure I expect from a visa table?"

That small distinction makes the scraper less dependent on the page's current ordering.

If no matching table exists, the Actor logs the problem and stops processing instead of trying to parse an arbitrary table:

if (!table.length) {
    log.error('Visa table not found');
    return;
}

Once I have the right table, I still can't assume that the columns are always in the same positions.

The parser therefore inspects the header row and records the indexes of the fields it needs:

let countryColIndex = -1;
let notesColIndex = -1;
let allowedStayColIndex = -1;
let visaColIndex = -1;

table.find('tr').first().find('th').each((index, th) => {
    const headerText = $(th).text().toLowerCase().trim();

    if (headerText.includes('country') ||
        headerText.includes('region')) {
        countryColIndex = index;
    }

    if (headerText.includes('allowed stay')) {
        allowedStayColIndex = index;
    }

    if (headerText.includes('visa')) {
        visaColIndex = index;
    }

    if (headerText.includes('notes')) {
        notesColIndex = index;
    }
});

This gives the parser a useful separation between table discovery and row parsing.

If the visa column moves, the parser doesn't need to know its new numeric position. It discovers the position from the header.

I also added explicit checks for required columns:

if (countryColIndex === -1) {
    log.error('Country column not found');
    return;
}

if (notesColIndex === -1) {
    log.error('Notes column not found');
    return;
}

This is important because a scraper can fail in two very different ways.

It can fail loudly because the expected structure is missing, or it can continue with the wrong structure and produce incorrect data.

I prefer the first.

Once the column indexes are known, the parser walks through the rows and compares the destination against the requested input:

const country = $(cols[countryColIndex]).text().trim();

if (
    normalizeForCompare(country) ===
    normalizeForCompare(destinationInput)
) {
    // Parse and normalize the matching row
}

The comparison is deliberately normalized:

const normalizeForCompare = (value: string) =>
    value.trim().toLowerCase().replace(/[^a-z]/g, '');

This means differences in capitalization, whitespace, and non-letter characters don't unnecessarily prevent a match.

The key design decision here is that I made the parser structure-aware rather than position-dependent.

That doesn't make the scraper immune to changes. Wikipedia can still change its headers or table structure in ways the parser doesn't recognize. But it gives the Actor a much better starting point than hardcoding assumptions about where the data happens to appear today.

And that led directly to the next problem.

Finding the correct cell was only half the job. The contents of those cells were inconsistent too.

Challenge 3 — Turning human text into deterministic data

Finding the right row solved only half of the problem.

The values inside that row were still written for humans.

Visa descriptions could contain different terminology for the same concept, while allowed-stay values could appear as days, weeks, months, years, or qualitative rules such as "Unlimited".

My first instinct was to keep the scraped text as-is. That would have been easy, but it would also push the hardest part of the problem onto every consumer of the Actor.

Instead, I added a normalization layer.

Normalizing visa types

The Actor maps common phrases into a small set of predictable categories:

type VisaType =
    | 'Visa-free'
    | 'Visa on arrival'
    | 'eVisa'
    | 'Visa required'
    | 'Freedom of movement'
    | 'Other';

The parser first cleans the Wikipedia text, converts it to lowercase for matching, and then checks known patterns.

For example:

const raw = cleanWikipediaText(text);
const t = raw.toLowerCase();

if (t.includes('freedom of movement')) {
    return { visaType: 'Freedom of movement', visaTypeRaw: raw };
}

if (t.includes('visa not required') || t.includes('visa-free')) {
    return { visaType: 'Visa-free', visaTypeRaw: raw };
}

I intentionally kept the original cleaned value in visaTypeRaw.

That decision is important because normalization is an interpretation. I don't want to lose the source value just because I created a more convenient representation for applications.

The parser also handles combinations. If the source mentions both an online visa and visa on arrival, the result records the primary category as Visa on arrival and preserves the available options:

if (t.includes('online visa') && t.includes('visa on arrival')) {
    return {
        visaType: 'Visa on arrival',
        visaTypeRaw: raw,
        visaOptions: ['eVisa', 'Visa on arrival'],
    };
}

If none of the known patterns match, the parser doesn't guess. It returns Other and keeps the original text.

That fallback became an important rule throughout the project:

An unknown value is better than an incorrect classification.

Parsing allowed stay

Stay duration required a similar approach.

I wanted applications to have a numeric value when the source made that possible, but I didn't want to throw away the human-readable version.

The resulting structure contains three related fields:

{
    maxStayDays,
    allowedStayText,
    stayPolicy
}

The parser handles common units explicitly.

const yearMatch = t.match(/(\d+)\s*year/);

if (yearMatch) {
    return {
        maxStayDays: Number(yearMatch[1]) * 365,
        allowedStayText: raw,
        stayPolicy: 'fixed',
    };
}

The same approach is used for months, weeks, and days. Months are converted using 30 days, years using 365 days, and weeks using 7 days.

For Unlimited, there is no meaningful numeric maximum, so maxStayDays remains null while stayPolicy captures the meaning:

{
    maxStayDays: null,
    allowedStayText: "Unlimited",
    stayPolicy: "unlimited"
}

The parser also distinguishes between values that describe a limited period, conditional rules, and values it doesn't understand. When a value cannot be normalized safely, the original cleaned text remains available and stayPolicy can fall back to unknown.

This is where I found the value of separating normalized data from source text

A developer can filter or compare maxStayDays when that value is meaningful. A human can still inspect allowedStayText to understand what the source actually said.

I don't have to choose between machine-readable data and source fidelity.

I can provide both.

Why I didn't use a more sophisticated parser

The goal wasn't to build a natural-language understanding system for visa policies.

I needed predictable behavior for common patterns.

Regular expressions and explicit rules were enough for that layer because they make the transformation visible and easy to reason about. More importantly, when a rule doesn't match, the parser has a clear fallback instead of inventing a result.

That trade-off is intentional.

The normalization layer handles the cases it understands and preserves the cases it doesn't.

That makes the system easier to debug when Wikipedia introduces a new format.

And that is the real purpose of this layer: not to eliminate ambiguity, but to keep ambiguity from leaking into every application that consumes the Actor.

Challenge 4 — Designing an API instead of exposing HTML

Once the parser could extract the right row and normalize its contents, I had another design question:

What exactly should the Actor return?

The easiest option would have been to return the scraped table data and let consumers interpret it themselves.

I didn't want that.

If I returned the Wikipedia structure directly, every developer using the Actor would still need to understand the source's terminology, column layout, and inconsistent stay descriptions. I would have moved the scraping problem rather than solved it.

Instead, I designed the output around what a consumer actually needs.

A visa result contains fields such as:

interface VisaResult {
    nationality: string;
    destination: string;
    visaType: VisaType;
    visaTypeRaw?: string;
    visaOptions?: string[];
    maxStayDays: number | null;
    allowedStayText: string;
    stayPolicy: StayPolicy;
    notes: string;
    found: boolean;
}

Each field has a different responsibility.

visaType gives applications a normalized value they can work with.

visaTypeRaw preserves the source-derived wording when the normalized category doesn't tell the whole story.

maxStayDays provides a numeric representation when one can be derived.

allowedStayText keeps the human-readable description.

stayPolicy tells consumers how to interpret the duration.

And found makes the "no matching visa information" case explicit instead of forcing consumers to infer it from missing fields.

Why keep both normalized and raw values?

This was one of the most important schema decisions.

Suppose the source contains a detailed visa condition. A normalized field might tell an application that the traveler is Visa-free for 30 days, but that doesn't necessarily capture every condition attached to that rule.

If I only returned the normalized value, I would be throwing away information during parsing.

By keeping the source-derived value as well, consumers can choose the level of abstraction they need.

For example:

{
    "visaType": "Visa-free",
    "visaTypeRaw": "Visa waiver for 30 days, subject to conditions",
    "maxStayDays": 30,
    "allowedStayText": "30 days",
    "stayPolicy": "fixed"
}

An automated workflow can use visaType and maxStayDays.

A human-facing application can display allowedStayText and notes.

A developer debugging an unexpected result can inspect the raw value.

The same record serves all three use cases.

Making "not found" explicit

Another small but important choice was the found field.

A destination might not have a matching entry, or the expected table might not be available on the source page.

I didn't want consumers to have to distinguish between:

{}

and

{
    "found": false
}

The second representation communicates intent much more clearly.

It also means the Actor can return a structured result even when there isn't a visa rule to report.

That makes the API easier to consume because callers can handle success and absence through the same response shape.

The schema becomes the boundary

At this point, the normalized result became the most important boundary in the system.

Before it, the implementation is concerned with Wikipedia:

After it, the implementation doesn't need to know how Wikipedia stores the information:

That separation gave me freedom to change the source-specific parsing logic without forcing every consumer to change with it.

It also made the Actor easier to extend. New output formats can consume the same normalized result instead of implementing their own interpretation of the source.

This was the point where I stopped thinking of the Actor as a scraper with an output attached to it.

The normalized schema was becoming the product interface

The scraper was simply one implementation detail behind it.

Challenge 5 — One source, multiple consumers

Once I had a normalized VisaResult, I didn't want to build a separate representation for every consumer.

The same lookup might be used by a developer through an API, exported for analysis, displayed to a human, or passed into an AI workflow. Creating separate parsing paths for each use case would have made the Actor harder to maintain and created opportunities for the outputs to disagree.

Instead, I kept one normalized result and built different outputs around it.

The default dataset

The default dataset is the primary structured output.

Each run produces the visa result as a dataset item, including fields such as the nationality, destination, normalized visa type, stay information, notes, language, and timestamp.

This is useful when the result needs to be exported or consumed as structured data.

The important part is that the dataset doesn't expose the Wikipedia table itself. It exposes the normalized contract we designed earlier.

The Key-Value Store

For a single lookup, a dataset isn't always the most convenient representation.

The Actor also stores the complete structured result under the RESULT key in the Key-Value Store.

That gives consumers another way to retrieve the same lookup without having to interpret a collection of dataset items.

The two outputs serve different access patterns, but they originate from the same normalized result.

The additional-info dataset

The visa table isn't the only useful information on a nationality page.

The Actor also extracts related travel-information links, such as passport rules, vaccination information, biometrics, or other nationality-specific requirements.

I kept these in a separate additional-info dataset rather than mixing them into the primary visa record.

That keeps the main result focused on the question the Actor was built to answer while still making related information available to consumers that need it.

The HTML report

Not every consumer needs raw JSON.

The Actor also generates an HTML report from the result. This gives someone running the Actor through the Apify interface a readable representation without requiring them to inspect a dataset or parse JSON manually.

The important architectural decision is that the report is generated after normalization.

It doesn't scrape Wikipedia independently.

That means the HTML report and the structured outputs are different representations of the same underlying result.

Localization without replacing the source

Localization introduced another design consideration.

I wanted users to be able to request output in languages such as Hindi, French, Spanish, or German, but I didn't want translation to change the canonical data.

The Actor therefore keeps English as the source of truth and adds localized fields when a non-English language is requested.

For example:

The English fields remain available for programmatic consumers, while the localized fields can be used directly in user-facing applications.

This distinction matters because translation is a presentation concern. It shouldn't change the meaning of the normalized data.

The localization layer also has a fallback: if translation fails, the Actor can continue returning the English result rather than making localization a dependency for the underlying visa lookup.

One normalized result, several interfaces

Looking at the outputs together, the architecture becomes simple:

The benefit isn't just convenience.

It means that if the parsing logic changes, I don't have to update four independent implementations. If I add another output format later, that format can consume the same normalized result.

This became an important maintainability rule for the project:

Parse once, normalize once, then adapt the result to the consumer.

That kept the source-specific complexity contained inside the Actor and prevented each output format from becoming its own mini-scraper.

Engineering for Production

Once the parser was producing the data I wanted, the next question was whether the Actor would behave predictably when things went wrong.

A scraper doesn't control its source. Wikipedia can change. A requested destination might not exist on the page. A table might be missing. A user might provide input that doesn't map cleanly to the source.

I decided early that these cases should produce understandable outcomes rather than obscure failures.

Validate before crawling

The Actor requires both a nationality and a destination.

That validation happens before the crawler starts doing useful work:

if (!nationality || !destination) {
    throw new Error(
        'Input must include both "nationality" and "destination".'
    );
}

This is deliberately simple.

There's no reason to make a network request when the request itself is incomplete.

It also means an invalid invocation fails close to the point where the problem was introduced, rather than producing a confusing parsing error later.

Treat missing source data as a normal case

One of the situations I wanted to handle explicitly was a missing visa table.

The parser doesn't assume that the expected table will always be present. If it can't identify the table, it logs the problem and stops processing that lookup.

The same principle applies when the destination cannot be found.

Rather than pretending that missing information means "visa required" or another default category, the Actor can return a structured result with found: false.

That distinction is important.

No data found is not the same thing as a visa requirement.

The Actor should report what it knows rather than invent an answer.

Keep localization out of the critical path

Localization is useful, but it shouldn't determine whether the underlying visa lookup succeeds.

The Actor first extracts and normalizes the English data. Translation is then applied to user-facing fields.

If localization fails, the Actor falls back to English.

That makes translation an enhancement rather than a dependency of the core scraping pipeline.

It also makes debugging easier: if a run produces the wrong visa result, I can investigate the source parsing independently of the translation layer.

Limit the crawl

The Actor exposes maxRequestsPerCrawl as an input parameter, with a default of 1.

For this lookup-oriented Actor, the default reflects the expected workload: a single nationality-to-destination lookup should not require an uncontrolled crawl.

Keeping the request budget explicit also gives the Actor a simple boundary around network activity.

{
    "nationality": "Indian",
    "destination": "Qatar",
    "maxRequestsPerCrawl": 1
}

The important point isn't the number itself. It's that network work is treated as a resource rather than something the parser can perform without limits.

Make failures observable

Logging became especially useful while working with changing Wikipedia tables.

The Actor logs meaningful stages of the lookup, including situations where the expected table or columns cannot be found.

For example:

log.error('Visa table not found');

A scraper that silently returns an empty result can be difficult to diagnose. A scraper that tells you which structural assumption failed gives you a starting point for fixing the parser.

This matters particularly for public sources because a parser can be perfectly correct against yesterday's HTML and still fail after a source-side change.

Keep stored data fresh

The additional-info dataset is cleared on each run before new additional travel information is stored.

That decision is deliberate.

The Actor represents a current lookup, so retaining stale named-dataset records from previous runs could make the output confusing.

The same principle applies to the generated result: each run should represent the data retrieved during that run, with scrapedAt recording when the lookup occurred.

Deployment is part of the engineering process

My deployment workflow remained intentionally straightforward:

I didn't treat deployment as the end of development.

After deploying, I verified the actual Actor run and checked the outputs rather than assuming that a successful build meant the complete pipeline was working.

That distinction matters for Actors because the final behavior depends on more than TypeScript compiling successfully. The crawler, source website, datasets, Key-Value Store, localization, and report generation all need to work together.

What I would add for a larger workload

The current Actor is designed around a focused nationality-to-destination lookup rather than a large-scale crawl.

If the workload grew substantially, I would revisit areas such as caching, more explicit retry policies, concurrency controls, and stronger monitoring.

Those aren't claims about the current implementation. They're the next engineering questions I'd consider once the workload justified the additional complexity.

For this project, I preferred to keep the implementation proportional to the problem.

That became another useful lesson:

Production quality doesn't mean adding every reliability mechanism you can think of. It means making the failure modes you actually have predictable, observable, and maintainable.

Using the Actor

Once the parsing and normalization layers were working, I wanted the Actor to be straightforward to consume.

The basic interface is intentionally small:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "language": "hi"
}

The Actor takes a passport nationality and destination country, performs the lookup, and returns structured visa information.

The optional language parameter adds localized fields without replacing the canonical English values.

cURL

For a quick test or a shell-based workflow, the Actor can be invoked through the Apify API.

curl -X POST \
  "https://api.apify.com/v2/acts/YOUR_ACTOR_ID/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "nationality": "Indian",
    "destination": "Qatar",
    "language": "en"
  }'

The important part of this request is the input rather than the client. The same Actor contract is used regardless of whether the run is started from the Apify UI, an SDK, or an HTTP request.

For automation, cURL is useful when the Actor needs to be called from a shell script, CI job, or another service that doesn't require a full SDK.

JavaScript

A JavaScript application can use the Apify client to start the Actor and retrieve the resulting dataset.

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: process.env.APIFY_TOKEN,
});

const run = await client
    .actor('YOUR_ACTOR_ID')
    .call({
        nationality: 'Indian',
        destination: 'Qatar',
        language: 'en',
    });

const { items } = await client
    .dataset(run.defaultDatasetId)
    .listItems();

console.log(items[0]);

The useful part of this integration is that the application doesn't need to know anything about Wikipedia.

It doesn't need to construct a Wikipedia URL, locate a table, identify columns, or parse visa terminology.

It only needs to provide the lookup parameters and consume the normalized result.

Python

The same workflow works from Python using the Apify client:

from apify_client import ApifyClient

client = ApifyClient(
    token="YOUR_APIFY_TOKEN"
)

run = client.actor("YOUR_ACTOR_ID").call(
    run_input={
        "nationality": "Indian",
        "destination": "Qatar",
        "language": "en",
    }
)

items = client.dataset(
    run["defaultDatasetId"]
).list_items().items

print(items[0])

This is useful for data-processing scripts, notebooks, reporting pipelines, and applications that already use Python.

Again, the important abstraction is the same: Python consumes the Actor's structured result instead of implementing its own Wikipedia parser.

Working with the dataset

The default dataset contains the primary visa result.

A typical record contains fields such as:

{
  "nationality": "Indian",
  "destination": "Qatar",
  "visaType": "Visa-free",
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed",
  "notes": "...",
  "language": "en",
  "found": true,
  "scrapedAt": "2025-12-26T08:24:19.686Z"
}

Because the result is stored as structured dataset data, it can also be exported through Apify rather than requiring another transformation step.

This is one reason I chose a dataset as a primary output: it works equally well for an application consuming JSON and for someone who wants to export the result for analysis.

Retrieving the complete result

For consumers that want the complete structured response as a single object, the Actor also writes the result to the Key-Value Store under the RESULT key.

That provides a second access pattern:

The dataset and Key-Value Store therefore aren't competing representations. They solve slightly different consumption patterns while sharing the same underlying result.

Localization

If a language other than English is provided, localized fields are added to the response.

For example:

{
  "visaType": "Visa-free",
  "visaType_localized": "वीज़ा-मुक्त",
  "allowedStayText": "30 days",
  "allowedStayText_localized": "30 दिन"
}

The English fields remain available.

This is important for applications that need both machine-stable values and user-facing translations. An application can use visaType for its internal logic while displaying visaType_localized to the traveler.

What the client doesn't need to know

The most important part of this API isn't the invocation syntax.

It's everything the caller doesn't have to implement.

A consumer doesn't need to:

scrape Wikipedia,
identify the correct table,
detect changing columns,
normalize visa terminology,
parse stay durations,
extract additional travel links,
or implement localization fallback.

Those responsibilities remain inside the Actor.

The client gets the result through a stable interface.

That was the original reason I wanted an API in the first place. I wasn't trying to make developers better at scraping Wikipedia. I wanted them to stop having to scrape Wikipedia at all.

Proving the Parser Works

A parser that works for one Wikipedia page isn't necessarily a reliable parser.

The source was the part of the system I didn't control, so testing had to cover variation rather than just the happy path.

I tested the Actor against different passport nationalities and destination combinations, including different visa policies and different ways of expressing allowed stay.

The goal was to verify two things at the same time:

the Actor could find the correct information;
the normalization layer could represent that information without losing important context.

Testing different visa policies

I tested destinations covering several common cases:

Visa-free
Visa on arrival
eVisa
Visa required

These cases exercise different branches of the visa-type parser.

The important part wasn't simply getting a non-empty response. I checked that the normalized visaType matched the intended category while the source-derived information remained available where necessary.

That distinction matters because a scraper can successfully return data while still interpreting it incorrectly.

Testing stay-duration formats

Allowed-stay parsing received particular attention because the source uses several different representations.

I tested values such as:

30 days
6 weeks
1 year
Unlimited
90 days within 180 days

These aren't equivalent from a parsing perspective.

Some can be converted directly into a numeric duration. Others contain additional conditions that need to remain visible in the original text.

That testing helped shape the current representation:

{
  "maxStayDays": 30,
  "allowedStayText": "30 days",
  "stayPolicy": "fixed"
}

The numeric field is useful for software, while the text field preserves what a traveler actually needs to read.

Testing source variation

I also tested the parser against multiple Wikipedia nationality pages.

This was important because the parser wasn't supposed to work only for the page I happened to use during development.

Different pages can expose different table structures and terminology, so testing multiple nationality pages helped verify that the implementation was discovering columns rather than depending entirely on fixed positions.

This is one of the reasons I consider header-based table detection an important part of the implementation.

Testing invalid and missing data

I also tested cases where the requested nationality or destination didn't produce a usable visa result.

The Actor should not turn missing information into a guessed visa requirement.

Instead, the result can explicitly indicate that the lookup wasn't found:

{
  "found": false
}

That gives downstream applications a state they can handle deliberately.

The same principle applies to missing notes or unfamiliar visa terminology. Missing information should remain missing rather than being replaced with invented content.

Testing localization

Localization was tested independently from the core visa parsing.

I verified localized output in multiple supported languages and checked that the English fields remained available alongside the translated fields.

This separation was important because localization should not change the underlying result.

I also tested the fallback behavior. If localization isn't available, the Actor should still be able to return the English result.

That gives the pipeline a useful failure boundary:

Testing the complete output pipeline

The final step was checking the outputs generated from each run.

I verified:

the default dataset,
the additional-info dataset,
the RESULT Key-Value Store record,
the HTML report,
and localized fields where requested.

This matters because successful parsing doesn't guarantee successful output generation.

The Actor is a pipeline, and I wanted to verify the pipeline rather than only one function inside it.

What testing changed

Testing wasn't just a final verification step.

It influenced the design.

The more variations I encountered in visa terminology and stay durations, the more important it became to preserve raw values alongside normalized ones.

The more nationality pages I checked, the more important dynamic table and column detection became.

And the more output formats I added, the more useful it became to have one normalized result as the source for all of them.

That was probably the biggest lesson from testing:

Testing a scraper means testing your assumptions about the source.

The code can be correct according to its own logic and still be wrong about the world it is scraping.

For this Actor, the most valuable tests were therefore the ones that challenged those assumptions.

## Lessons Learned

Building this Actor changed how I think about scraping.

I started with a fairly simple goal: find visa information and expose it through an API. I expected the main challenge to be retrieving the data.

It turned out that retrieval was the easy part.

The difficult part was deciding what to do with information that was inconsistent, incomplete, or likely to change.

A few lessons stood out.

Scraping is only the first step

The crawler gets the data into the system. It doesn't make that data useful.

The real work started after the HTML had been downloaded: identifying the right table, finding the right row, interpreting visa terminology, parsing stay durations, and deciding which information could safely be normalized.

That changed my mental model of scraping.

I no longer see the crawler as the main part of the system. It's the entry point into a data-processing pipeline.

Normalization is where most of the value is created

The more Wikipedia pages I tested, the clearer this became.

Extracting:

"30 days"
is easy.

Deciding whether that should become:

{
  "maxStayDays": 30,
  "stayPolicy": "fixed"
}

while still preserving the original text is the more interesting engineering problem.

The normalized representation is what makes the data useful to software.

Without it, every consumer would have to implement its own interpretation of the source.

Preserving the source prevents over-normalization

I initially thought mostly in terms of structured fields.

But visa requirements contain conditions and exceptions that don't always fit cleanly into a predefined schema.

That made visaTypeRaw, allowedStayText, and notes important parts of the design.

I don't need to choose between structured data and source fidelity.

I can provide both.

That also gives me a safer fallback when Wikipedia introduces terminology that the parser doesn't recognize yet.

Unknown is better than wrong

This became one of the strongest principles in the project.

A parser can always be made to produce some category. That doesn't mean the category will be correct.

When the Actor encounters an unfamiliar visa description, preserving the original information and returning Other is safer than guessing.

The same principle applies to stay durations.

If I can't safely derive a numeric value, I would rather return null and preserve the original text than manufacture a number that looks precise but isn't.

For data that may eventually be consumed by applications or AI systems, that distinction matters.

Source changes should be expected

Wikipedia isn't an API that I control.

Its content and table structures can evolve independently of my Actor.

That means the parser shouldn't be designed around one exact snapshot of the HTML.

The header-based table detection and dynamic column discovery were direct responses to that problem.

They don't guarantee that the parser will survive every future change, but they reduce unnecessary coupling to the current layout.

More importantly, they make the assumptions in the parser visible and easier to update.

Architecture becomes more valuable as features accumulate

The project started with a relatively small core: retrieve the page, parse the table, and return the result.

Then the requirements expanded.

Localization was added.

Additional travel information was extracted.

A separate dataset was introduced.

The Key-Value Store became another output.

An HTML report was added.

At that point, keeping everything in one place would have made each new feature harder to implement.

Separating crawling, parsing, normalization, localization, reporting, storage, and utilities gave each part a clearer responsibility.

The refactor didn't make the Actor more impressive on the surface. It made the code easier to change.

That's the kind of improvement that becomes valuable only after a project starts evolving.

Design around consumers, not the source

The source determines what information is available.

It shouldn't determine what the API looks like.

Wikipedia's table structure is useful for Wikipedia.

It isn't necessarily a useful contract for a JavaScript application, a Python data pipeline, or an AI agent.

That distinction influenced the entire output design.

The Actor exposes normalized data, localized user-facing fields, datasets, a Key-Value Store result, and an HTML report because those are useful consumption patterns—not because Wikipedia happens to provide them in that form.

The broader lesson

Looking back, I don't think the most important thing I built was the scraper.

The more valuable part was the abstraction around it.

The Actor takes information written for humans, applies a set of explicit rules to interpret it, preserves uncertainty where necessary, and exposes the result through a predictable interface.

That's a pattern I can reuse far beyond visa requirements.

Whenever an application depends on a public website that wasn't designed to be an API, the same engineering problem appears:

How do you turn an unpredictable source into an interface that your software can trust?

That became the real lesson of this project.

What Happened After I Shipped It

After testing the Actor locally, I deployed it to Apify and started iterating on it as a real project rather than treating the first working run as the finish line.

The project was eventually selected for Best Newcomer at the Apify challenge, an award for a developer who joined Apify during the challenge and made an immediate impact.

That recognition was meaningful to me because the project started from a relatively simple problem: I was looking for a visa API and couldn't find a reliable one.

The award wasn't the reason I built the Actor, but it was useful validation that the project was solving a problem in a way that was valuable to the Apify developer community.

More importantly, the process reinforced something I had learned while building it: a working scraper is only the beginning. The parts that make the project useful to other developers—predictable output, resilient parsing, multiple consumption formats, localization, and maintainable architecture—are what turn a one-off scraper into something people can actually build on.

Conclusion

I started this project because I wanted a visa API.

What I ended up building was more than a way to scrape visa tables. The interesting part was creating a reliable boundary between a changing public source and the applications that depend on its data.

Wikipedia provides information for people to read. The Actor turns that information into structured data that software can consume.

The crawler retrieves the page. The parser identifies the relevant table and row. The normalization layer turns inconsistent visa descriptions and stay durations into predictable fields while preserving the original wording. From there, the same normalized result can be exposed through datasets, the Key-Value Store, an HTML report, and optional localized fields.

The most important lesson for me was that scraping is only the beginning.

The real engineering work starts when you ask what the data should look like after it leaves the website.

Public sources will change. New terminology will appear. Some rules won't fit neatly into a predefined schema. A reliable system shouldn't hide those problems—it should make its assumptions explicit, preserve information it can't safely interpret, and give consumers a stable interface.

That's the approach I took with this Actor.

If I were starting another project that depended on a public website, I'd ask a different question now. Instead of starting with:

"How do I scrape this page?”

I'd start with:

"What interface should the software consuming this data see?"

That shift in perspective is probably the most valuable thing this project taught me.

Building a Real-Time Multimodal AI Communication Coach

Raj Gupta — Sun, 01 Mar 2026 10:40:35 +0000

Most AI tools on the market today are fundamentally text-based. Even when they "listen" to audio, they act on static transcripts after the fact. But human communication doesn't happen after the fact. It happens in the moment—in our tone of voice, our pacing, our posture, and our eye contact.

“Vision agents don’t just see pixels — they interpret intent, context, and meaning in motion.”

I wanted to build an AI that could actually coach people on how they communicate, rather than just summarizing what they said. I wanted an AI that could see if you were slouching, hear if you were speaking too fast, and interrupt you politely with actionable advice.

The result is Visions, an open-source real-time AI Communication Coach.

AI Multimodal Communication Coach

"Words are only part of the message. Your posture, your pace, your pauses — they say everything the words don't."

In this post, I want to pull back the curtain on the architecture behind Visions. We'll look at how we orchestrated Gemini Realtime API, GetStream, Deepgram STT, ElevenLabs TTS, and Ultralytics YOLO into a seamless, unified real-time loop.

The Architecture Overview

Building a multimodal agent that operates on a live video call requires a robust, low-latency event loop. The system needs to juggle four concurrent responsibilities:

The Edge (WebRTC Video/Audio streaming)
The Ears (Speech-to-Text inference)
The Eyes (Computer Vision pose estimation)
The Brain (The LLM reasoning engine)
The Voice (Text-to-Speech response generation)

To manage this, the core orchestrator (located in performance_agent.py) relies on an async event loop that subscribes to an incoming WebRTC media track.

1. The Edge: GetStream WebRTC

We use GetStream as our WebRTC interface. When the Python agent spins up, it generates a unique call_id and opens a staging URL in the user's browser. Once the user clicks "Join", GetStream establishes the peer-to-peer connection and begins piping real-time audio and video frames directly into our Python backend.

2. The Ears & Voice: Deepgram + ElevenLabs

Low latency is non-negotiable for a natural conversation. For the "ears," we use Deepgram STT configured for eager turn detection. As the user speaks, Deepgram streams STTPartialTranscriptEvent objects into our SpeechProcessor.

The SpeechProcessor doesn't just store text; it actively tracks:

WPM (Words Per Minute)
Filler Word Frequency ("um", "uh", "like")
Hesitation Index (derived from average pause durations)

For the "Voice," we use ElevenLabs TTS (specifically the Multilingual V2 model) because it offers some of the most expressive, human-like cadence available.

3. The Eyes: Real-Time Pose Estimation with YOLO

This is where Visions goes beyond a standard voice-bot. As GetStream pipes video frames into the event loop, we route them through our CommunicationVisionProcessor.

We use Ultralytics YOLO (yolo11n-pose.pt specifically) for lightweight pose estimation. By running at a deliberate 5 FPS to conserve CPU, YOLO identifies key landmarks on the speaker's body.

From these landmarks, the Vision Processor calculates:

Posture ("upright", "slouched", "leaning")
Eye Contact (tracking face orientation relative to the camera center)
Hand Activity (measuring how wildly the wrists are moving relative to the torso)

Suddenly, the AI doesn't just know what you're saying; it knows how you look while saying it.

4. The Brain: The 30-Second Coaching Loop

Wiring Deepgram, YOLO, and ElevenLabs into Gemini is powerful, but it can be overwhelming for the user if the AI is constantly interrupting.

To solve this, we implemented a custom SessionManager that runs a strict 30-second evaluation cadence.
Every 30 seconds, the manager pauses and queries the pipelines. It grabs:

The speech_state (WPM, Hesitation Index)
The vision_state (Posture, Eye Contact, Engagement)
The language_scores (Clarity, Structure)

It bundles these metrics into a structured "coaching context" prompt and fires it to the Gemini Realtime API.

Gemini evaluates the context against its system instructions (which dictate the current mode, such as Interview, Public Speaking, or Debate). By instructing Gemini to be "positive-first and actionable," it generates a maximum 3-sentence spoken response.

For example, Gemini might see that the user's WPM is perfect, but their posture is poor. It will interrupt politely: "Great pacing on that last point. But I notice you're slouching—sit up straight to project more confidence before your next sentence."

To visualize how all these separated folders actually move data in real-time within the orchestrator, here is a diagram of the call flow:

Flowchart

Graceful Degradation and Reporting
Because live calls are chaotic (tabs get closed, internet connections drop), the performance_agent.py script wraps the session in a massive try/except block. Even if the session ends abruptly, the agent gathers all the 30-second snapshots it collected and generates a comprehensive Markdown and JSON Session Performance Report.

How the Code Actually Flows

To understand the value of this architecture in real-time, let's walk through exactly what happens when you start speaking.

If we look inside performance_agent.py, the core orchestration doesn’t rely on a single, monolithic LLM text generation. It relies on a multi-stage event loop.

Here is the exact timeline of a single 30-second window in the life of the AI Coach:

Step 1: Ingesting the Audio & Video Frames (Seconds 0–29)

As you talk into your camera, GetStream routes your audio to Deepgram STT and your video frames to YOLO.

Our agent listens for words. Every time Deepgram processes a word, it triggers an event that we've subscribed to. We pipe that transcript directly into our custom SpeechProcessor:

# ── Wire Deepgram transcripts → SpeechProcessor ─
@agent.events.subscribe
async def on_transcript(event: STTTranscriptEvent):
    speech_proc.on_transcript(event.text or "", is_final=True)

@agent.events.subscribe
async def on_partial_transcript(event: STTPartialTranscriptEvent):
    speech_proc.on_transcript(event.text or "", is_final=False)

Why we wrote this: By handling partial transcripts (is_final=False), the system doesn't have to wait for you to finish your sentence to start calculating your Words Per Minute or counting your "ums" and "uhs".

Step 2: The Coaching Trigger (Second 30)

Exactly every 30 seconds, the SessionManager interrupts the background collection and fires the run_coaching_cycle function.

This is where the magic happens. The orchestrator asks all the processors to hand over their current state simultaneously:

async def run_coaching_cycle(agent, speech_proc, lang_intel, vision_proc, session):
    # 1. Grab raw statistics
    speech_state = speech_proc.get_state()   # Gets WPM, Filler Count, Hesitation
    vision_state = vision_proc.get_state()   # Gets Posture, Eye Contact

    # 2. Build the LLM Prompt
    lang_scores = lang_intel.get_last_scores()
    coaching_context = lang_intel.build_coaching_context(lang_scores, speech_state)

Why we wrote this: If we just fed the raw transcript to the LLM and asked "how did they do?", the LLM would hallucinate or give vague feedback. Instead, we manually give the LLM hard data. The coaching_context string basically tells Gemini: "The user spoke at 110 WPM, used 3 filler words, and their posture is slouched. GIve them a 3-sentence coaching tip."

Step 3: Delivering the Voice (Second 31)

Once the coaching_context prompt is built, we hand it off to the Gemini Realtime API, which generates the text response.

But we don't just print text to a screen. We use the <agent.simple_response()> method provided by the vision_agents framework to automatically route Gemini's output through ElevenLabs TTS:

    # Send coaching trigger to Gemini Realtime (it will speak via ElevenLabs TTS)
    await agent.simple_response(coaching_context)

    # Store snapshot after coaching
    session.record_snapshot(
        speech_state=speech_state,
        language_scores=lang_scores,
        vision_state=vision_state,
        coaching_message=coaching_context[:200], # Save to Session Report
    )

    # Rotate the window to start tracking the NEXT 30 seconds
    speech_proc.reset_window()

Why we wrote this: Calling agent.simple_response() seamlessly generates the highly-emotive ElevenLabs voice (which we defined earlier in the file to use the voice ID "VR6AewLTigWG4xSOukaG") and streams the AI's audio back into the WebRTC call.

Finally, we clear the speech_proc window so the AI doesn't evaluate the same "ums" and "uhs" during the next 30-second loop.

The Value of This Flow

By breaking the architecture into discrete processors (Speech, Vision, Language), we ensure the LLM is acting as a true "Coach"—giving feedback based on quantitative data, rather than just acting as a conversational chatbot.

The Project Structure: Why Architecture Dictates Latency

When building an AI agent that listens, watches, evaluates, and speaks all at the same time, you cannot shove everything into one massive app.py script. The latency will skyrocket, and the AI will constantly interrupt itself.

Decoupling is the secret to a fast multimodal coach.

Here is the folder structure we designed for Visions, and why each piece is strictly isolated:

c:\Visions
├── performance_agent.py        # The Main Event Loop & Orchestrator
├── performance_agent.md        # The Gemini Instruct Prompt & Coaching Rules
├── pipelines/                  # The Real-Time Analytical Engines
│   ├── speech_processor.py     # Tracks WPM, Pauses, and Fillers
│   ├── vision_processor.py     # Runs YOLO for Body Language
│   └── language_intelligence.py# Evaluates STAR structure & Grammar
├── session/                    # Data Persistence & Reporting
│   ├── session_manager.py      # The 30-Second Metronome
│   └── performance_report.py   # Generates the final Markdown Summaries
└── reports/                    # Where local `<session_id>.md` reports save

How the Folders Interact During an Active Call

When you run python performance_agent.py, you start an asynchronous event loop that relies directly on the separation of these folders.

Here is why this structure is so important during the running state:

The Orchestrator (performance_agent.py)

This file is intentionally "dumb." It does not calculate words per minute. It does not look at pixels. It acts purely like an air-traffic controller, establishing the GetStream WebRTC WebSockets to EleventhLabs, Deepgram, and Gemini.

Because we offload the heavy data tracking to the /pipelines folder, the main orchestrator's thread is completely free to handle streaming the AI's spoken audio back to the user without stuttering.

The Analytical Engines (/pipelines)

The files inside /pipelines are isolated state machines.

vision_processor.py: This script subscribes independently to GetStream's video track. It swallows frames without halting the audio loops, calculating the bounding boxes and angles of your shoulders and eyes using YOLO.

"""
Vision Processing Pipeline
Extends YOLOPoseProcessor to compute higher-level body language metrics:
- Face orientation, eye contact, posture, hand movement
- Generates confidence index, engagement score, body language rating
"""
import logging
import math
import time
from typing import Any, Optional

from vision_agents.plugins.ultralytics import YOLOPoseProcessor

logger = logging.getLogger(__name__)

# COCO keypoint indices
KP_NOSE = 0
KP_LEFT_EYE = 1
KP_RIGHT_EYE = 2
KP_LEFT_EAR = 3
KP_RIGHT_EAR = 4
KP_LEFT_SHOULDER = 5
KP_RIGHT_SHOULDER = 6
KP_LEFT_HIP = 11
KP_RIGHT_HIP = 12
KP_LEFT_WRIST = 9
KP_RIGHT_WRIST = 10
KP_LEFT_ELBOW = 7
KP_RIGHT_ELBOW = 8


class CommunicationVisionProcessor(YOLOPoseProcessor):
    """
    Extended YOLO Pose Processor for communication coaching.

    Adds body language interpretation on top of raw pose detection:
    - Eye contact estimation (camera-facing heuristic from nose/ear alignment)
    - Posture scoring (shoulder height symmetry and vertical alignment)
    - Hand movement tracking (wrist velocity)
    - Confidence and engagement composite scores

    get_state() returns a structured dict injected into the LLM each turn.
    """

    name = "communication_vision"

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

        # Wrist velocity tracking
        self._prev_wrist_positions: dict[str, tuple[float, float]] = {}
        self._prev_wrist_time: float = time.time()
        self._wrist_velocity_history: list[float] = []  # last N frames

        # Rolling score history (for smoothing)
        self._eye_contact_history: list[float] = []
        self._posture_history: list[float] = []
        self._confidence_history: list[float] = []
        self._engagement_history: list[float] = []

        self._history_len = 30  # ~1 second at 30fps

        # Latest state for get_state()
        self._latest_state: dict = {
            "confidence_index": 70.0,
            "engagement_score": 70.0,
            "body_language_rating": "good",
            "posture": "upright",
            "eye_contact": "strong",
            "hand_activity": "natural",
            "persons_detected": 0,
        }

    def _safe_keypoint(
        self, kpts: list, index: int, conf_threshold: float = 0.3
    ) -> Optional[tuple[float, float]]:
        """Return (x, y) if keypoint confidence exceeds threshold, else None."""
        if index >= len(kpts):
            return None
        x, y, c = kpts[index]
        return (float(x), float(y)) if float(c) > conf_threshold else None

    def _score_eye_contact(self, kpts: list) -> float:
        """
        Estimate eye contact (0-100) using nose/ear symmetry.
        When looking at camera: nose is centered, ears are equidistant.
        When looking away: nose shifts toward one side, ear asymmetry increases.
        """
        nose = self._safe_keypoint(kpts, KP_NOSE)
        left_ear = self._safe_keypoint(kpts, KP_LEFT_EAR)
        right_ear = self._safe_keypoint(kpts, KP_RIGHT_EAR)

        if not nose or not left_ear or not right_ear:
            return 60.0  # neutral when keypoints unavailable

        nose_x = nose[0]
        left_ear_x = left_ear[0]
        right_ear_x = right_ear[0]

        ear_width = abs(right_ear_x - left_ear_x)
        if ear_width < 1e-3:
            return 60.0

        # Ratio of nose position within ear span (0.5 = perfectly centered)
        nose_ratio = (nose_x - left_ear_x) / ear_width
        center_deviation = abs(nose_ratio - 0.5)  # 0 = perfect center

        # Convert to 0-100 score (0 deviation = 100, 0.5 deviation = 0)
        score = max(0.0, 100.0 - (center_deviation / 0.5) * 100.0)
        return round(score, 1)

    def _score_posture(self, kpts: list) -> tuple[float, str]:
        """
        Score posture (0-100) and classify it.
        Uses shoulder height symmetry and shoulder-to-hip vertical alignment.
        Returns (score, label).
        """
        left_shoulder = self._safe_keypoint(kpts, KP_LEFT_SHOULDER)
        right_shoulder = self._safe_keypoint(kpts, KP_RIGHT_SHOULDER)
        left_hip = self._safe_keypoint(kpts, KP_LEFT_HIP)
        right_hip = self._safe_keypoint(kpts, KP_RIGHT_HIP)

        if not left_shoulder or not right_shoulder:
            return 65.0, "upright"

        # Shoulder symmetry (y coordinates should be similar)
        shoulder_height_diff = abs(left_shoulder[1] - right_shoulder[1])
        shoulder_width = abs(left_shoulder[0] - right_shoulder[0])
        if shoulder_width < 1e-3:
            return 65.0, "upright"

        symmetry_score = max(0.0, 100.0 - (shoulder_height_diff / shoulder_width) * 200)

        # Shoulder vertical position (higher on screen = better posture for seated person)
        # We use shoulder midpoint y as proxy for upright-ness
        shoulder_mid_y = (left_shoulder[1] + right_shoulder[1]) / 2.0
        posture_score = symmetry_score  # Simplified; extend with hip alignment if available

        if left_hip and right_hip:
            hip_mid_y = (left_hip[1] + right_hip[1]) / 2.0
            # Torso height ratio: larger = more upright
            torso_height = abs(hip_mid_y - shoulder_mid_y)
            shoulder_to_hip_ratio = torso_height / max(1, shoulder_width)
            # Ideal ratio ~1.3-1.8 for upright seated posture
            ratio_score = min(100.0, (shoulder_to_hip_ratio / 1.5) * 100.0)
            posture_score = (symmetry_score * 0.5) + (ratio_score * 0.5)

        posture_score = round(max(0.0, min(100.0, posture_score)), 1)

        if posture_score >= 75:
            label = "upright"
        elif posture_score >= 50:
            label = "leaning"
        else:
            label = "slouched"

        return posture_score, label

    def _score_hand_activity(self, kpts: list) -> tuple[float, str]:
        """
        Score hand movement and classify as natural/excessive/minimal.
        Uses wrist velocity relative to body width.
        """
        now = time.time()
        dt = now - self._prev_wrist_time
        self._prev_wrist_time = now

        if dt < 1e-6:
            return 50.0, "natural"

        velocities: list[float] = []
        shoulder_width = 1.0

        left_shoulder = self._safe_keypoint(kpts, KP_LEFT_SHOULDER)
        right_shoulder = self._safe_keypoint(kpts, KP_RIGHT_SHOULDER)
        if left_shoulder and right_shoulder:
            shoulder_width = max(
                1.0, abs(right_shoulder[0] - left_shoulder[0])
            )

        for side, kp_idx in [("left", KP_LEFT_WRIST), ("right", KP_RIGHT_WRIST)]:
            wrist = self._safe_keypoint(kpts, kp_idx)
            if wrist:
                prev = self._prev_wrist_positions.get(side)
                if prev:
                    dx = wrist[0] - prev[0]
                    dy = wrist[1] - prev[1]
                    velocity = math.sqrt(dx**2 + dy**2) / (shoulder_width * dt)
                    velocities.append(velocity)
                self._prev_wrist_positions[side] = wrist

        if not velocities:
            return 50.0, "natural"

        avg_velocity = sum(velocities) / len(velocities)
        self._wrist_velocity_history.append(avg_velocity)
        if len(self._wrist_velocity_history) > self._history_len:
            self._wrist_velocity_history.pop(0)

        smoothed = sum(self._wrist_velocity_history) / len(self._wrist_velocity_history)

        # Classify velocity ranges (normalized to shoulder width per second)
        if smoothed < 0.3:
            return smoothed * 100, "minimal"
        elif smoothed > 2.5:
            return min(100.0, smoothed * 20), "excessive"
        else:
            # Map 0.3-2.5 to "natural" range, score 60-90
            score = 60.0 + ((smoothed - 0.3) / 2.2) * 30.0
            return round(score, 1), "natural"

    def _smooth_score(self, history: list[float], new_val: float) -> float:
        """Exponential smoothing for score stability."""
        history.append(new_val)
        if len(history) > self._history_len:
            history.pop(0)
        if not history:
            return new_val
        # Weighted average (recent values weighted more)
        n = len(history)
        weights = [i + 1 for i in range(n)]
        return sum(v * w for v, w in zip(history, weights)) / sum(weights)

    def _process_person_keypoints(self, kpts: list) -> dict[str, Any]:
        """Compute all body language metrics for one detected person."""
        eye_contact_raw = self._score_eye_contact(kpts)
        posture_raw, posture_label = self._score_posture(kpts)
        hand_score, hand_label = self._score_hand_activity(kpts)

        eye_contact = self._smooth_score(self._eye_contact_history, eye_contact_raw)
        posture_score = self._smooth_score(self._posture_history, posture_raw)

        # Composite confidence index
        confidence_raw = (eye_contact * 0.4) + (posture_score * 0.4) + (hand_score * 0.2)
        confidence = self._smooth_score(self._confidence_history, confidence_raw)

        # Engagement score (eye contact + energy signals)
        activity_bonus = 10.0 if hand_label == "natural" else 0.0
        engagement_raw = (eye_contact * 0.6) + (posture_score * 0.3) + activity_bonus
        engagement = self._smooth_score(self._engagement_history, engagement_raw)

        # Body language rating
        avg_score = (confidence + engagement) / 2.0
        if avg_score >= 80:
            rating = "excellent"
        elif avg_score >= 60:
            rating = "good"
        elif avg_score >= 40:
            rating = "fair"
        else:
            rating = "poor"

        # Eye contact label
        if eye_contact >= 70:
            eye_label = "strong"
        elif eye_contact >= 45:
            eye_label = "intermittent"
        else:
            eye_label = "weak"

        return {
            "confidence_index": round(confidence, 1),
            "engagement_score": round(engagement, 1),
            "body_language_rating": rating,
            "posture": posture_label,
            "eye_contact": eye_label,
            "hand_activity": hand_label,
            "eye_contact_score": round(eye_contact, 1),
            "posture_score": round(posture_score, 1),
            "hand_activity_score": round(hand_score, 1),
        }

    def get_state(self) -> dict:
        """
        Returns body language state dict injected into LLM context each turn.
        Falls back to cached state if no new pose data available.
        """
        return self._latest_state.copy()

    def _process_pose_sync(self, frame_array) -> tuple:
        """
        Override parent to intercept pose_data and compute body language metrics.
        """
        annotated_frame, pose_data = super()._process_pose_sync(frame_array)

        persons = pose_data.get("persons", [])
        if persons:
            # Analyze first detected person (primary speaker)
            kpts = persons[0].get("keypoints", [])
            if kpts:
                body_language = self._process_person_keypoints(kpts)
                body_language["persons_detected"] = len(persons)
                self._latest_state = body_language
                logger.debug(f"Body language: {body_language}")
        else:
            self._latest_state["persons_detected"] = 0

        return annotated_frame, pose_data

speech_processor.py: A lightweight state object that continuously accepts STTPartialTranscriptEvent strings from Deepgram and computes running metrics (like the hesitation_index based on gaps between words).

"""
Speech Processing Pipeline
Measures WPM, filler words, pause duration, and hesitation index from Deepgram transcripts.
"""
import re
import time
import logging
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, Optional

logger = logging.getLogger(__name__)

# Filler words to detect
FILLER_PATTERNS = re.compile(
    r"\b(um+|uh+|er+|ah+|like|you know|you know what i mean|"
    r"basically|literally|actually|honestly|right\?|so|anyway|"
    r"i mean|kind of|sort of|i guess)\b",
    re.IGNORECASE,
)

# Ideal WPM range for most communication contexts
IDEAL_WPM_MIN = 120
IDEAL_WPM_MAX = 160


@dataclass
class TranscriptWindow:
    """Stores a rolling 30-second window of transcript data."""
    text: str = ""
    word_count: int = 0
    filler_count: int = 0
    start_time: float = field(default_factory=time.time)
    end_time: Optional[float] = None
    pause_durations_ms: list[float] = field(default_factory=list)


class SpeechProcessor:
    """
    Measures real-time speech quality metrics from streaming transcripts.

    Designed to be used alongside the Deepgram STT plugin. Subscribe to
    STT transcript events and call `on_transcript` for each utterance.

    Exposes `get_state()` for the vision-agent processor state injection.
    """

    def __init__(self, window_seconds: int = 30):
        self.window_seconds = window_seconds
        self._current_window = TranscriptWindow()
        self._completed_windows: Deque[TranscriptWindow] = deque(maxlen=100)

        # Track all-session cumulative stats
        self._session_start = time.time()
        self._total_words = 0
        self._total_fillers = 0
        self._last_speech_time: Optional[float] = None
        self._pause_threshold_ms = 500  # pauses > 500ms are "notable"

    def on_transcript(self, text: str, is_final: bool = True) -> None:
        """
        Process a new transcript segment from Deepgram.

        Args:
            text: The transcript text for this utterance.
            is_final: Whether this is a final (vs partial) transcript.
        """
        if not text or not text.strip():
            return

        now = time.time()

        # Calculate pause since last speech
        if self._last_speech_time is not None:
            pause_ms = (now - self._last_speech_time) * 1000
            if pause_ms > self._pause_threshold_ms:
                self._current_window.pause_durations_ms.append(pause_ms)

        self._last_speech_time = now

        # Only count final transcripts for window metrics
        if not is_final:
            return

        words = text.strip().split()
        word_count = len(words)
        fillers = FILLER_PATTERNS.findall(text)
        filler_count = len(fillers)

        # Update current window
        self._current_window.text += " " + text
        self._current_window.word_count += word_count
        self._current_window.filler_count += filler_count

        # Update cumulative session stats
        self._total_words += word_count
        self._total_fillers += filler_count

        logger.debug(
            f"SpeechProcessor: +{word_count} words, {filler_count} fillers detected"
        )

        # Rotate window if expired
        if (now - self._current_window.start_time) >= self.window_seconds:
            self._rotate_window(now)

    def _rotate_window(self, now: float) -> None:
        """Archive the current window and start a fresh one."""
        self._current_window.end_time = now
        self._completed_windows.append(self._current_window)
        self._current_window = TranscriptWindow(start_time=now)
        logger.info("SpeechProcessor: 30s window rotated.")

    def get_current_wpm(self) -> float:
        """Calculate WPM for the active window."""
        elapsed = time.time() - self._current_window.start_time
        if elapsed < 1.0 and self._current_window.word_count == 0 and self._completed_windows:
            # Fallback to the last completed window if we just rotated
            last_window = self._completed_windows[-1]
            if last_window.end_time:
                last_elapsed = last_window.end_time - last_window.start_time
                if last_elapsed >= 1.0:
                    return (last_window.word_count / last_elapsed) * 60.0

        if elapsed < 1.0:
            return 0.0
        return (self._current_window.word_count / elapsed) * 60.0

    def get_session_wpm(self) -> float:
        """Calculate overall session WPM."""
        elapsed = time.time() - self._session_start
        if elapsed < 1.0:
            return 0.0
        return (self._total_words / elapsed) * 60.0

    def get_avg_pause_duration_ms(self) -> float:
        """Average pause duration in current window (ms)."""
        pauses = self._current_window.pause_durations_ms
        if not pauses and self._completed_windows and (time.time() - self._current_window.start_time) < 1.0:
             pauses = self._completed_windows[-1].pause_durations_ms

        if not pauses:
            return 0.0
        return sum(pauses) / len(pauses)

    def get_hesitation_index(self) -> float:
        """
        Hesitation index (0-100). Combines filler rate and pause duration.
        Higher = more hesitation.
        """
        window = self._current_window
        actual_elapsed = time.time() - window.start_time
        filler_count = window.filler_count

        # If the window just rotated, add the last completed window for calculations
        elapsed = actual_elapsed
        if actual_elapsed < 1.0 and self._completed_windows:
            last_window = self._completed_windows[-1]
            if last_window.end_time:
                elapsed += max(1.0, last_window.end_time - last_window.start_time)
                filler_count += last_window.filler_count

        elapsed = max(1.0, elapsed)

        # Filler rate component: normalize fillers per minute against 0-10 range
        fillers_per_min = (filler_count / elapsed) * 60.0
        filler_component = min(fillers_per_min / 10.0, 1.0) * 50.0  # 0-50 points

        # Pause component: normalize avg pause 0-3000ms range
        avg_pause = self.get_avg_pause_duration_ms()
        pause_component = min(avg_pause / 3000.0, 1.0) * 50.0  # 0-50 points

        return round(filler_component + pause_component, 1)

    def get_state(self) -> dict:
        """
        Returns the current speech metrics dict for injection into LLM context.
        Called automatically by the vision-agents framework each turn.
        """
        wpm = self.get_current_wpm()
        filler_count = self._current_window.filler_count
        avg_pause = self.get_avg_pause_duration_ms()
        hesitation = self.get_hesitation_index()
        elapsed = round(time.time() - self._current_window.start_time, 1)
        window_text = self._current_window.text.strip()

        return {
            "wpm": round(wpm, 1),
            "filler_count": filler_count,
            "avg_pause_duration_ms": round(avg_pause, 1),
            "hesitation_index": hesitation,
            "window_elapsed_seconds": elapsed,
            "window_text": window_text[:500] if window_text else "",
            "session_wpm": round(self.get_session_wpm(), 1),
            "session_total_words": self._total_words,
            "session_total_fillers": self._total_fillers,
            "wpm_status": (
                "too_slow" if wpm < IDEAL_WPM_MIN and wpm > 0
                else "too_fast" if wpm > IDEAL_WPM_MAX
                else "ideal"
            ),
        }

    def get_window_text(self) -> str:
        """Get the transcript text from the current 30s window."""
        return self._current_window.text.strip()

    def reset_window(self) -> None:
        """Manually reset the current window (e.g., after analysis triggers)."""
        now = time.time()
        self._rotate_window(now)

    def get_session_summary(self) -> dict:
        """Full session-level speech summary for the performance report."""
        return {
            "session_duration_seconds": round(time.time() - self._session_start, 1),
            "total_words": self._total_words,
            "total_fillers": self._total_fillers,
            "average_wpm": round(self.get_session_wpm(), 1),
            "filler_rate_per_minute": round(
                (self._total_fillers / max(1, time.time() - self._session_start)) * 60, 2
            ),
            "windows_completed": len(self._completed_windows),
        }

Why this matters: The Vision Processor and the Speech Processor never talk directly to each other. They mind their own business, continuously updating their own internal speech_state and vision_state dictionaries, completely unaware of what the other is doing.

"You can have brilliant ideas, but if you can't communicate them, your ideas won't get you anywhere." — Lee Iacocca

The Metronome and the Memory (/session)

If the processors are the senses, the /session folder is the brain's working memory.

session_manager.py acts as the system's metronome. Every 30 seconds, it steps in, interrupts the orchestrator, and says, "Give me the current state."

"""
Session Manager
Maintains per-call state, tracks 30-second coaching snapshots, and aggregates scores.
"""
import asyncio
import json
import logging
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
from uuid import uuid4

logger = logging.getLogger(__name__)

REPORTS_DIR = Path("reports")


@dataclass
class CoachingSnapshot:
    """A single 30-second analysis snapshot."""
    timestamp: float
    elapsed_seconds: float
    speech_state: dict
    language_scores: dict
    vision_state: dict
    coaching_message: str = ""

    def overall_score(self) -> float:
        """Compute weighted overall communication score."""
        clarity = self.language_scores.get("clarity_score", 70)
        structure = self.language_scores.get("structure_score", 70)
        confidence_lang = self.language_scores.get("confidence_score", 70)
        confidence_body = self.vision_state.get("confidence_index", 70)
        engagement = self.vision_state.get("engagement_score", 70)

        speech_penalty = min(30.0, self.speech_state.get("hesitation_index", 0) * 0.3)

        raw = (
            clarity * 0.20
            + structure * 0.20
            + confidence_lang * 0.15
            + confidence_body * 0.20
            + engagement * 0.25
        ) - speech_penalty

        return round(max(0.0, min(100.0, raw)), 1)

    def to_dict(self) -> dict:
        return {
            "timestamp": self.timestamp,
            "elapsed_seconds": self.elapsed_seconds,
            "overall_score": self.overall_score(),
            "speech": self.speech_state,
            "language": self.language_scores,
            "vision": self.vision_state,
            "coaching_message": self.coaching_message,
        }


class SessionManager:
    """
    Manages the coaching session lifecycle:
    - Stores timestamped snapshots every 30 seconds
    - Exposes aggregated score trends for the dashboard
    - Triggers session report generation on call end
    """

    def __init__(self, session_id: Optional[str] = None, mode: str = "interview"):
        self.session_id = session_id or str(uuid4())
        self.mode = mode
        self.start_time = time.time()
        self._snapshots: list[CoachingSnapshot] = []
        self._analysis_interval = 30.0  # seconds
        self._analysis_task: Optional[asyncio.Task] = None
        self._on_analysis_callbacks: list = []

        logger.info(f"SessionManager created: session_id={self.session_id}, mode={mode}")

    def register_analysis_callback(self, callback) -> None:
        """
        Register a coroutine to call every 30 seconds.
        Callback signature: async def callback() -> None
        """
        self._on_analysis_callbacks.append(callback)

    def start_analysis_loop(self) -> None:
        """Start the periodic 30-second analysis cycle."""
        self._analysis_task = asyncio.create_task(self._analysis_loop())
        logger.info("Analysis loop started (30s intervals)")

    async def _analysis_loop(self) -> None:
        """Fires registered callbacks every 30 seconds."""
        try:
            while True:
                await asyncio.sleep(self._analysis_interval)
                logger.info(
                    f"[{self.session_id}] 30s analysis cycle firing "
                    f"(elapsed: {self.elapsed_seconds():.0f}s)"
                )
                for callback in self._on_analysis_callbacks:
                    try:
                        await callback()
                    except Exception as e:
                        logger.error(f"Analysis callback error: {e}")
        except asyncio.CancelledError:
            logger.info("Analysis loop cancelled")

    def record_snapshot(
        self,
        speech_state: dict,
        language_scores: dict,
        vision_state: dict,
        coaching_message: str = "",
    ) -> CoachingSnapshot:
        """
        Store a coaching snapshot for the current window.
        Returns the snapshot.
        """
        snapshot = CoachingSnapshot(
            timestamp=time.time(),
            elapsed_seconds=self.elapsed_seconds(),
            speech_state=speech_state.copy(),
            language_scores=language_scores.copy(),
            vision_state=vision_state.copy(),
            coaching_message=coaching_message,
        )
        self._snapshots.append(snapshot)
        logger.info(
            f"Snapshot #{len(self._snapshots)} recorded: "
            f"overall={snapshot.overall_score()}"
        )
        return snapshot

    def elapsed_seconds(self) -> float:
        return round(time.time() - self.start_time, 1)

    def get_score_trend(self) -> list[dict]:
        """Return list of {elapsed, overall_score} for trend charts."""
        return [
            {"elapsed": s.elapsed_seconds, "score": s.overall_score()}
            for s in self._snapshots
        ]

    def get_latest_scores(self) -> dict:
        """Return the most recent snapshot's scores, or defaults."""
        if not self._snapshots:
            return {
                "overall_score": 70.0,
                "speech_score": 70.0,
                "body_language_score": 70.0,
                "structure_score": 70.0,
                "confidence_score": 70.0,
            }
        s = self._snapshots[-1]
        return {
            "overall_score": s.overall_score(),
            "speech_score": max(0, 100 - s.speech_state.get("hesitation_index", 0)),
            "body_language_score": s.vision_state.get("confidence_index", 70),
            "structure_score": s.language_scores.get("structure_score", 70),
            "confidence_score": s.language_scores.get("confidence_score", 70),
            "clarity_score": s.language_scores.get("clarity_score", 70),
            "engagement_score": s.vision_state.get("engagement_score", 70),
        }

    def get_leaderboard_entry(self) -> dict:
        """Compact entry for the GetStream leaderboard."""
        scores = self.get_latest_scores()
        return {
            "session_id": self.session_id,
            "mode": self.mode,
            "elapsed_seconds": self.elapsed_seconds(),
            "overall_score": scores["overall_score"],
            "snapshot_count": len(self._snapshots),
        }

    async def stop(self) -> None:
        """Stop the analysis loop and return session data."""
        if self._analysis_task:
            self._analysis_task.cancel()
            try:
                await self._analysis_task
            except asyncio.CancelledError:
                pass

    def get_all_snapshots(self) -> list[dict]:
        return [s.to_dict() for s in self._snapshots]

It reaches into both /pipelines/speech_processor and /pipelines/vision_processor, pulls their isolated data dictionaries together, and uses that combined state to build the contextual prompt that triggers Gemini's voice via ElevenLabs.

Then, the Session Manager takes a "Snapshot" of that data and saves it. Once you hang up the video call, performance_report.py loops over all those 30-second snapshot files and generates the beautifully formatted Markdown summary that opens in your browser.

By strictly enforcing a separation of concerns—where the orchestrator only handles connections, the pipelines only handle raw math inference, and the session manager handles timing and data storage—we ensure that the YOLO object detection never blocks the WebRTC audio streaming.

What Happens When the Internet Drops?

If you've ever been on a Zoom call, you know that WebRTC connections drop. Audio gets garbled, video frames freeze, and packets are lost.

If our Python agent threw a fatal error and crashed every time the user's Wi-Fi hiccuped, the AI Coach would be completely useless in the real world.

To solve this, Visions is built around the concept of Graceful Degradation.

1. Resilient Analytical Engines
Because the speech_processor.py and vision_processor.py scripts are completely decoupled, they are inherently fault-tolerant.

If the user's video feed freezes due to bad bandwidth, YOLO in our Vision Processor simply stops receiving new frames. It doesn't crash the server. It just holds onto the last known vision_state (e.g., "posture: upright").

Meanwhile, if the audio connection is still surviving, deepgram STT keeps transcribing text, and the Speech Processor keeps counting words. When the 30-second loop triggers, the AI Coach can still provide spoken feedback based on the audio, even if the video feed is temporarily dead.

2. The Unbreakable "Try/Except" Session Wrapper
But what happens if the connection completely drops and the user's browser tab crashes? Do they lose their entire 30-minute coaching session data?

No. If you look at the bottom of the main event loop in performance_agent.py, the entire live call is wrapped inside a massive try/except block:

        try:
            # Wait for the call to end (user hangs up or closes tab)
            await agent.finish()

            # ── Session ended cleanly: generate report ──────────────────
            await _generate_and_deliver_report()

        except BaseException as e:
            # ── Session ended abruptly: still save the report ──────────
            logger.warning(f"Session ended unexpectedly. Saving report anyway...")
            try:
                await _generate_and_deliver_report()
            except Exception as report_err:
                logger.error(f"Failed to generate report after unexpected session end: {report_err}")
            raise

If GetStream WebRTC throws an unexpected disconnection error (BaseException), the orchestration immediately catches it.

Instead of letting all the collected data vanish into memory, the SessionManager forces the system to trigger _generate_and_deliver_report().
The Session Manager grabs every 30-second snapshot it successfully saved before the internet dropped, compiles them, and still generates the final Markdown Session Performance Report (<session_id>.md) in the /reports folder.

The Result: Even if your laptop battery dies in the middle of a mock interview, the AI coach will gracefully save everything you've done up to that exact second, format your scores, and have your feedback waiting on your hard drive when you reboot.

"True communication is a whole-body act. It took a multimodal AI to finally measure it that way."

What's Next?

Building a real-time multimodal stack used to require a massive engineering team. Today, with tools like Gemini Realtime, YOLO edge inference, and GetStream, a single developer can build an agent that genuinely "coaches" humans.

In the future, I plan to expand the Vision architecture to include micro-expression tracking for deeper emotional resonance, and fine-tune exactly how the language intelligence maps out rhetorical structures on the fly.

🌍 Lingo Canvas

Raj Gupta — Mon, 23 Feb 2026 12:13:39 +0000

Building Lingo Canvas: From Local Project to Global Thinking

🚀Live Demo
💻GitHub Repository

Most of us build applications in English. Not because everyone speaks English. But because it’s convenient. Framework documentation is in English. Error messages are in English. Most open-source ecosystems default to English. So naturally, our apps do too.

But here’s the uncomfortable truth:

🌐 The internet is global.
🧩 Most applications are not.

That realization slowly shaped the foundation of my project : Lingo Canvas.

🧠 What Is Lingo Canvas?

Lingo Canvas is an AI-powered canvas application built with:

⚡ Next.js
🖌️ tldraw
🌍 Lingo.dev

It allows users to:

🎨 Generate dynamic content visually on a canvas
🔁 Switch between multiple languages instantly
🌐 Experience both UI and generated content adapt to the selected language

At first glance, it might look like a simple AI canvas tool. But underneath, it became something deeper. It became an experiment in designing software that does not assume a single language.

🎯 The Motivation Behind the Project

Originally, I wanted to build something interactive and AI-driven. Something visual.

Something dynamic.

But while designing it, I asked myself:

What if someone who doesn’t speak English opens this app?

Will they feel included?

Or will they feel like they’re visiting someone else’s space?

That question changed everything.

Because it made me realize:

Localization is not just about translating words.

It’s about shaping user experience.

🔄 The Shift in Thinking

Instead of building everything first and worrying about translations later, I decided to treat multilingual support as part of the architecture from the beginning.

That meant:

🧱 Structuring text intentionally
🔁 Planning for runtime language switching
↔️ Considering right-to-left layout behavior
🧩 Separating static UI from dynamic AI-generated content

Lingo Canvas stopped being just a canvas experiment.

It became a study in building global-ready applications.

📚 What This Series Will Cover

In the upcoming parts, I will break down:

❌ Why traditional localization approaches fail in dynamic apps
🏗️ The architectural decisions behind Lingo Canvas
🔀 How I separated deterministic UI localization from AI content regeneration
⚙️ The role of lingo.dev in creating a structured localization workflow
📈 Lessons learned while building a multilingual AI product

This is not just a project showcase.

It is a journey from:

"Let’s add translations later"

"Localization is infrastructure."

⚠️ Why Traditional React Localization Breaks at Scale

When I first thought about making Lingo Canvas multilingual, I assumed the solution was simple:

📦 Install an i18n library
📁 Create translation JSON files
🔤 Wrap text in a t() function
✅ Done

That approach works. Until it doesn’t.

🪤 The “It Works for Two Languages” Trap

For small projects, traditional localization looks fine.

You create:

// en.json
{
  "welcome": "Welcome",
  "generate": "Generate"
}
// de.json
{
  "welcome": "Willkommen",
  "generate": "Generieren"
}

Then you switch language state and everything updates.

Clean. Simple. Efficient.

But here’s what starts happening as your app grows:

📈 Translation keys multiply
🏷️ Naming becomes inconsistent
❗ Some keys are missing in certain languages
🔨 Refactoring UI breaks translations silently
😬 Adding a new language feels risky

And suddenly, localization becomes technical debt.

🔍 The Hidden Problem: Localization Is Tightly Coupled to Components

In many React apps, text is defined directly inside components.

Example:


<button>{t("generate")}</button>

At first glance, this is fine.

But over time:

🧱 Components grow
🎨 UI changes
🔁 Keys get reused incorrectly
🌀 Context-specific translations get mixed

The result?
You stop trusting your translation system.

🤖 Now Add Dynamic Content

Static UI is one problem. Dynamic content is another level. Lingo Canvas generates content inside a canvas dynamically.

So I had to think:

Should I translate generated content word-by-word?
Or regenerate it contextually in the selected language?

Traditional i18n systems are designed for deterministic UI. They are not designed for AI-generated experiences. That’s where things start breaking conceptually.

🏗️ Designing a Hybrid Localization Architecture

At this point, I knew one thing clearly:

❌ Traditional localization alone wasn’t enough.
🖌️ Lingo Canvas wasn’t just a static dashboard.
🤖 It was an interactive AI-powered canvas.

That meant I was dealing with two very different types of text:

📌 Predictable interface text (buttons, labels, menus)
✨ Unpredictable, AI-generated content inside the canvas

Treating both the same would be a mistake. So instead of forcing everything into one system, I designed a hybrid model.

🧩 Separate Static UI from Dynamic Content

This was the turning point. I divided localization into two layers:

Layer 1 : Deterministic UI

This includes:

🧭 Navigation labels
🔘 Buttons
🛠️ Toolbars
⚙️ Settings
🏷️ Static headings

This text is predictable. It doesn’t change contextually. It just needs to exist in multiple languages reliably. This layer is where structured localization belongs.

Layer 2 : Dynamic AI Content

This includes:

📝 Generated canvas text
🧠 AI-created explanations
🔄 Context-aware content blocks

This text is not predictable.

It changes based on:

👤 User input
🧾 Prompts
📌 Context
🔁 Interaction flow

Trying to pre-translate this would be impossible. Instead of translating it…
I regenerate it in the selected language. That distinction changed everything.

🔤 Translation vs Regeneration

This became the core philosophy of the project.

Translation

🔁 Word substitution
📚 Deterministic
📖 Dictionary-based
✅ Works well for static UI

Regeneration

🧠 Context-aware
🎯 Meaning-driven
🤖 AI-based
✨ Ideal for dynamic content

If I tried to translate AI-generated English text into German, I would lose nuance. Instead, I generate directly in German when the language is switched. That preserves intent, not just vocabulary.

Why This Hybrid Model Works❓

By separating responsibilities:

UI remains stable and structured
AI content remains contextual and natural
Adding a new language doesn’t break the app
Language switching feels intentional

Instead of forcing localization into one bucket, I allowed the system to reflect reality.

Some text is static. Some text is alive. And they deserve different strategies.

🧠 The Architectural Mindset Shift

Before this project, I thought:

“Localization is about translation files.”

After building this hybrid system, I realized:

Localization is about system boundaries.

It’s about deciding:

📌 What should be deterministic?
🔄 What should be contextual?
📚 What belongs in dictionaries?
🤖 What belongs in generation logic?

That architectural clarity made everything simpler.

Not easier , but cleaner.

🧱 Structuring the UI Layer with a Proper Localization Workflow

Once I separated static UI from dynamic content, the next question was obvious:

How do I make the UI localization reliable and scalable?

Because here’s the truth:

Most localization problems are not caused by translation.
They are caused by poor structure.

⚠️ The Mistake Most Projects Make

In many React apps, localization starts like this:

📁 Create a locales folder
➕ Add en.json
➕ Add de.json
🔤 Use a t() function
🚀 Ship it

That works for small projects.

But when the app grows:

🏷️ Keys become inconsistent
🗂️ Files become cluttered
🔨 Refactoring breaks translations silently
😬 Adding new languages feels risky

Localization slowly turns into hidden technical debt. I didn’t want that. So instead of thinking about translation files…

I thought about workflow.

🏗️ Designing Localization as a System

For the deterministic UI layer, I needed:

🧩 Structured translation keys
🏷️ Clear naming conventions
🔁 Predictable fallback behavior
🌍 A clean way to manage multiple locales
🛡️ Confidence when adding new languages

This is where a structured localization tool becomes important. Not because it translates automatically. But because it enforces discipline.

🔁 Making Language Switching Feel Instant (Without Breaking the Canvas)

Adding multiple languages is one thing. Making language switching feel smooth is another. In Lingo Canvas, users can change language at runtime.

⚡ No refresh.
🚫 No reload.
🧠 No disruption.

But making that work required careful thinking.

🏁 The Core Challenge

When a user switches language, three things must happen:

⚡ The UI must update immediately.
↔️ The layout must adapt (especially for RTL languages).
🖌️ The canvas content must remain stable.

If even one of these fails, the experience feels broken. So I had to design the system carefully.

🧭 Step 1: Centralized Language State

Language selection is managed at a global level. Instead of passing language props everywhere, I used a centralized state approach.

When the language changes:

🔁 The localization provider updates
🔄 UI components re-render with new translations
↔️ Layout direction adjusts if needed

This ensures consistency across the entire application. No component is “out of sync.”

↔️ Step 2: Handling RTL Layouts

Supporting languages like Arabic is not just about translating text.

It affects:

📝 Text alignment
🔀 Flex direction
📐 Spacing
🧭 UI flow

So when the language switches to an RTL language:

The dir attribute updates to rtl
Layout adjusts automatically
Components remain readable and aligned

This small detail dramatically improves user comfort.

🛡️ Step 3: Protecting the Canvas State

Here’s where things get tricky.

The canvas contains:

🔷 Shapes
📍 Positions
🖱️ User interactions
🤖 AI-generated content

When language changes, I don’t want:

Shapes to reset
User work to disappear
The canvas to reinitialize

So I separated:

UI rendering logic
Canvas state logic

Language switching triggers UI re-rendering, but the canvas state remains intact. That separation preserves user work. Which is critical.

🔄 Step 4: Regenerating Dynamic Content (When Needed)

For AI-generated content:

If the user switches language, I don’t simply translate existing English text.

Instead, I regenerate it in the selected language.

This ensures:

🎯 Context stays accurate
🗣️ Tone feels natural
💡 Meaning remains intact

The UI updates instantly. The content regenerates intentionally. That distinction makes the experience feel thoughtful rather than mechanical.

🎯 The UX Goal

Language switching should feel:

⚡ Immediate
🌿 Natural
📏 Predictable
🛡️ Non-destructive

If users hesitate before switching languages, something is wrong.

The goal is confidence.

And that confidence comes from architectural separation, not hacks.

📚 Resources

If you're building a multilingual app and want a structured way to manage UI localization, you can explore:

🌍 Lingo.dev — https://lingo.dev

It provides tooling and workflow support for managing translations in modern web applications.

🧠 What Building a Multilingual AI App Taught Me About Architecture

When I started Lingo Canvas, I thought I was building a canvas tool. By the end of it, I realized I was redesigning how I think about software architecture. Localization was never supposed to be the “main challenge.” But it quietly became the most educational part of the project.

🏗️ Lesson 1: Localization Is Not a Feature

It’s easy to treat multilingual support as something you “add later.”

But once your app grows:

📝 Text lives everywhere
🧩 Components depend on language
↔️ Layout reacts to direction
🤖 Dynamic content behaves differently

Localization touches rendering, state, UX, and even performance. That makes it infrastructure. And infrastructure decisions must be intentional.

🔍 Lesson 2: Not All Text Is Equal

Some text is stable and predictable. Some text is contextual and alive. Trying to force both into one system leads to compromise.

Separating:

🧱 Deterministic UI localization
🔄 Dynamic AI content regeneration

made the system cleaner and more scalable.

Architecture improves when responsibilities are clearly divided.

🔁 Lesson 3: Language Switching Is a UX Signal

When a user switches language, they are telling you something:

“I want this experience in my space.”

If switching language:

🐢 Feels slow
💥 Breaks layout
🔄 Resets state
⚠️ Feels incomplete

The product loses trust instantly. Language switching should feel seamless. That only happens when localization is designed, not patched.

🌍 Lesson 4: Global Thinking Changes Design Decisions

Once I started thinking globally, I began asking different questions:

🏷️ Is this key structured clearly?
➕ What happens when we add five more languages?
↔️ Does this layout break in RTL?
🔤 Should this text be translated or regenerated?

Those questions improved more than just localization. They improved the entire architecture.

📌 Final Thoughts

Lingo Canvas started as an experiment in AI + canvas interaction. It ended up being a lesson in multilingual product design. If I had built everything first and “added translations later,” the app would work. But it wouldn’t feel intentional. Designing for multiple languages from the beginning forced clarity. And clarity always improves software.

If you're building something today, ask yourself:

Are you building for your language?
Or for the internet? 🌍

Because the difference between those two mindsets changes everything.