The same visa question gave 8 documents one day and 19 the next. Here's how I fixed it

I run a small tool that researches visa requirements: you give your passport, the country you live in and your destination, and an AI reads the official embassy and visa-centre pages and returns a PDF with where to apply, the documents, the fees and the steps.

It worked fine in demos. Then I ran the same case twice (Indian passport, living in India, tourist visa for France) and got two different reports. One listed 8 documents, the other 19.

For something people use to prepare a real visa application, that isn't a small bug. If it can't give the same answer twice, nobody should trust either answer.

Here's what caused it and what I changed.

1. The model was free to pick its own sources

The first version told the model to use "official sources". So it did, sometimes. Other times it used an agency blog that looked official, or a third-party summary that was two years old.

Fix: decide which sources count in code, not in the prompt. A source is accepted only if it's:

• on one of the destination country's own domains (worked out from the country's ISO code, not a hand-written list of sites),
• an EU institution page (europa.eu) for Schengen cases,
• a visa-centre chain page carrying the destination's code, or
• a visa centre whose own official page names its domain.

Anything else is thrown away before the model sees it. The prompt still says "official only", but nothing depends on the prompt for that anymore.

2. "List the documents" is an open question

When you ask a model for a list, it makes a judgment call about how long the list should be, and that judgment changes between runs.

Fix: turn the open question into a fixed checklist. Next to the general parts of the report, the model now has to answer 15 specific document questions every time: passport rules, photos, application form, travel insurance, proof of accommodation, bank statements, and so on.

It can skip a question, but only by giving a reason, a short quote and the official URL that shows the document isn't needed. A handful of questions (passport, form, photos and a few others) can never be skipped.

Now a short report and a long report have to be explained by the sources, not by chance.

3. Retries made it worse

Earlier versions had a "completion pass": if something looked missing, ask again. In practice each extra call was another chance to drift.

Fix: at most two model calls per report, and no extra completion pass after a fallback. Fewer calls meant fewer different versions of the truth.

4. The same question should get the same report

Even with all of the above, two runs a few minutes apart can come out a little differently.

Fix: save the report for the exact inputs for 3 days. If someone asks the same passport / residence / destination / visa type again, they get the saved report. Its links are checked again first and the PDF is dated the day it's delivered.

5. Small things that broke trust

• Links that "worked" but were empty. One official EU register returned HTTP 200 with a blank body. Now a 200 with an empty body counts as a dead link.
• The model refusing to quote. Long quotes from official pages sometimes tripped the provider's copyright filter and the call failed. Quotes are now capped at 12 words, and a retry rephrases in its own words with even shorter quotes.

The result

The France case now comes back complete in one pass, every fact confirmed on an official page (the sample report on the site shows 36 of 36), and the same documents on a re-run. Two other live tests (Egypt passport living in Saudi Arabia → Austria, India → UK) also came back complete in one pass.

What I'd tell anyone building an "AI researcher"

1. Enforce trust in code. "Use official sources" in a prompt is a wish, not a rule.
2. Replace open questions with fixed ones. Consistency comes from structure, not from asking nicely.
3. Make skipping expensive. Require a quote and a source for every "not needed".
4. Fewer calls, not more. Every retry is a new roll of the dice.
5. Cache by exact input. The same question should get the same answer.

If you want to see the output, there's a free visa check and a real sample report.

Disclosure: I built this tool. It's not a government website, and the embassy makes every visa decision.