Never Trust an AI That Says “I’ve Checked”

Jichitai Watch — Tue, 27 Oct 2026 12:05:01 +0000

A solo builder in Japan on designing for the day updates stop: static hosting, honest missing-data labels, and two AIs researching independently

1. Treating “the day I stop” as a starting assumption

Have you ever tried to compare two towns’ finances before an election? In Japan, local government data is not easy for citizens to read. Every municipality structures its website differently, and much of what is published comes as PDFs or tables saved as images. Before an election, I wanted to look up my own town’s policies and finances. As someone with a full-time job, though, I found it hard even to find a basis for comparison.

Others have tackled this problem before. “Where Does My Money Go?”, a project born in the UK, had a Japanese edition, spending.jp. It started with Yokohama in 2013 and spread to other cities through volunteers. Later, the people involved looked back and wrote that activity had died down once the boom passed, and that keeping it going would need people to support it.

What I took from this was simple: money and motivation run out.

So I set out to build Jichitai Watch on my own. It is a free website that brings together, for ordinary residents, a picture of Japan’s local governments: their towns, finances, councils, and the local systems that shape daily life. Before starting, I set myself one rule:

“Don’t make continuing the goal. Treat the day it stops as a design assumption. Even if I stop working on it, aim for a site that doesn’t keep presenting old information as if it were current.”

Jichitai Watch: https://jichitaiwatch.jp / English: https://jichitaiwatch.jp/en

2. Almost building a worse copy of the government’s dashboard

I had experience in IT infrastructure and project management (PMO), but I was an amateur at web development and design. Still, watching generative AI improve, I wondered whether one person, by directing AI, could organise local government data. So I started building.

Then, just as my first prototype was finished and I was about to publish it, the project ran aground. Japan’s Digital Agency and the Ministry of Internal Affairs and Communications had already released a dashboard of municipal finances. Comparisons with similar municipalities, per-capita figures: the features I had been building were already there.

“If I publish this as it is, it’ll just be a worse version of the government’s site.”

I scrapped the screens I had built and refocused on information the finance dashboard doesn’t carry.

My starting point was a public survey: the Association for Promoting Fair Elections’ 20th Nationwide Survey on the Unified Local Elections (2023). Asked which policy issues they had considered when voting, respondents most often named:

healthcare and elderly care (52.6%)
the economy (44.7%)
measures for an ageing population (42.3%)
childcare and education (40.5%)

Administrative and fiscal reform came in at 8.8%.

The finance dashboard shows year-end accounts in detail. But it doesn’t cover the differences in the systems each municipality has chosen. Examples are the age up to which children’s medical costs are subsidised, and the standard premium for long-term care insurance. These are set locally in Japan, so they can differ even between neighbouring towns.

So I moved the finance data down to a deeper layer and rebuilt the site around four entry points: the town, the administration, the council, and daily life. From each one, residents can compare systems and data with neighbouring municipalities on the same basis.

3. Four design choices so a one-person site won’t mislead when it stops

① Keep fixed costs low: a static site with no database

To keep the cost of continuing low, there is no database. Python turns the source data into JSON, Next.js turns that into static HTML, and Cloudflare’s free hosting serves it.

② Never fill gaps by guessing, and say why data is missing

Municipal data is often unpublished, or sits in PDFs whose layout breaks so machines can’t read them. If you fill gaps by guessing, or keep searching in pursuit of perfection, the project stalls.

So values are used exactly as written in the source documents, and gaps are never filled by guessing. When something can’t be found or read, the result of the machine search and the reason are recorded internally, and the same check isn’t repeated.

On screen, missing data is shown in three separate ways:

Could not be confirmed: we looked, but couldn’t confirm it from published information. It does not mean the information is not public.
Not stated in the published material: the source itself does not give the figure.
In preparation: Jichitai Watch hasn’t prepared this part yet.

The last label was chosen deliberately. It reads as the state of our work, not as the municipality lacking data, so a failure on our side is never mistaken for “this municipality has no data”.

③ Show old data as old

Each indicator has an expiry date based on its update cycle. When the site is built, cards for indicators past their expiry show a warning. Once an indicator is well past its expiry, comparisons with national and prefectural averages and the trend charts are switched off, leaving only the value itself and its source.

The catch: this check runs only at build time. If I stop updating and rebuilding, the pages stay exactly as they were at the last build. Fully keeping the promise “even if I stop” is still an open problem.

④ Neutral colours and words: no traffic-light colours

Colouring municipalities with heavy debt red (“danger”) and those with little debt green (“safe”) pushes the maker’s value judgement onto readers. So the rules are:

never assign red or green to higher or lower values of the same indicator
never use evaluative words such as “good” or “bad”

Colours that simply tell series apart, such as men and women or spending categories, are used freely.

A script scans every file for possible rule violations, and a person makes the final call by looking at the screens. A machine can’t judge neutrality itself, so I made a checklist and review each screen against it.

4. Not trusting an AI that says “I’ve checked”

I had no web development experience, yet I published the first region in about three months: Gifu Prefecture and its 42 municipalities, 349 pages in the Japanese version. I think that was possible because I split the work across several AIs.

It wasn’t set up this way from the start. Not knowing which AI suited which task, I experimented, and their strengths and weaknesses gradually showed. On top of that, I was using every AI on its cheapest plan, so I kept hitting usage limits and work would stall. This is the division of labour I settled on:

Claude: the coordinator. Design, managing specifications, writing code, and reconciling and ruling on research results
ChatGPT: screen mock-ups (HTML)
Grok and Composer (AIs available in the coding tool Cursor): researching each municipality’s website, such as council chairs, council groups, and whether minutes are published
Gemini: communications and outreach strategy

The biggest lesson: an AI saying “I’ve checked”, or several AIs agreeing, is not proof of correctness. Once, Claude reported that the implementation was “consistent” with ChatGPT’s mock-up, when the mock-up hadn’t been reflected at all. Another time, four AIs unanimously recommended a setting that isn’t available on GitHub’s free plan.

So for research on each municipality, the process works like this:

Grok and Composer investigate the same municipality separately, without seeing each other’s results.
Claude compares the two sets of results.
Where they disagree, Claude opens the municipality’s official pages and documents and rules on the answer.
Some items where the two agree are also checked against official sources, but not all of them.

If both AIs read the same wrong page, they can agree on the same mistake. This method reduces errors; it doesn’t eliminate them. That’s why the site has a channel for correction requests, and a set procedure: check each report against the source, then fix it.

5. Three months the first time, about two days after that

Publishing the first region took about three months, including two rebuilds. That time also covered planning and design.

After that, I turned the work into shared, reusable steps, from collecting data and checking for personal information to extracting what can be published. With those steps, I prototyped the next two prefectures (83 municipalities) in about two days. The three after that (69 municipalities) also took about two days. The prototypes aren’t public yet.

Because this work built on an existing foundation, it isn’t a like-for-like comparison with the first round. Still, it feels like a big reduction.

For the English version, I built a glossary grounded in two sources: the official English names used by Japanese government bodies, and the UK government’s GOV.UK style guide. The AI looked terms up in that glossary, and the English version took about four days.

How much effort updates take after publication is something I’m still measuring as the work continues. Building quickly and being able to keep going are, I think, two different problems.

The code is on GitHub under the MIT License. Alongside it are the data specifications, the neutrality checklist, the neutrality-check script, and notes on where I stumbled. The data itself remains subject to each provider’s terms of use.

These documents, too, are written by AI, to cut down on manual work. I don’t think civic tech can last if it depends on one volunteer pulling all-nighters. What’s needed is to use AI under fixed rules, and to build imperfection and the risk of updates stopping into the design from the start.

I hope the design of this small personal project from Japan gives a useful idea to someone, somewhere in the world, who is working to make public data visible.

Please take a look at the site (available in English) and the design documents on GitHub. Feedback from civic tech practitioners and developers around the world is very welcome, as are Stars and discussion on GitHub.

Jichitai Watch: https://jichitaiwatch.jp

English: https://jichitaiwatch.jp/en

GitHub (code under the MIT License):https://github.com/jichitaiwatch/jichitai-watch

DEV Community: Jichitai Watch