DEV Community: Jarno S

An Entity Factsheet for AI Search: A Practical Site-Wide QA Checklist

Jarno S — Wed, 16 Sep 2026 07:56:22 +0000

When an organization has a website, social profiles, and articles, its identity can still be surprisingly hard to piece together. Names vary. Old descriptions remain online. Service pages make claims that the About page never explains.

A simple entity factsheet can help. It is not a magic AI-search ranking factor. It is an internal source of truth that makes public information more consistent and easier to verify.

Start with the canonical facts

Create a short document containing the exact public name, primary domain, location at the appropriate level, what the organization does, who is responsible for the content, and the official profile URLs. Add a date and an owner for each fact.

For a solo expert, keep the person's identity and the project or company identity distinct. A founder's professional experience is not automatically a client outcome for the company.

Audit every public touchpoint

Compare the factsheet with the homepage, About page, service pages, contact page, author pages, social profiles, directory listings, and guest articles.

Look for contradictions that a reader would notice:

Is the company name spelled the same way?
Does the website link point to the preferred HTTPS domain?
Are old offerings still described as current?
Is the same person named as author where appropriate?
Are location and contact details accurate?
Do profiles imply credentials, clients, or results that cannot be substantiated?

Fixing inconsistencies is more useful than creating more empty profiles.

Give each page a clear job

The homepage should explain what the organization does and who it serves. An About page should establish identity and responsibility. A service page should explain scope, process, deliverables, and limitations. An article should answer a specific question with enough context to stand alone.

Do not place every keyword on every page. Instead, connect relevant pages with descriptive internal links so a reader can follow the evidence.

Use structured data carefully

Organization, Person, Article, and other schema types can describe visible facts. Validate the markup and make sure it matches the page. Structured data should not claim reviews, ratings, awards, or services that are absent from the visible content.

Think of schema as a machine-readable description of a clear page, not a substitute for one.

Maintain a change log

If the name, domain, services, or author details change, update the factsheet first. Then review the most important pages and profiles. A quarterly check can catch stale information before it spreads into more places.

For AI-answer monitoring, note the change date. If a system later describes the organization incorrectly, you can compare its answer with the current canonical facts and investigate which source may be outdated. That is a better diagnosis than simply calling the answer a hallucination.

A ten-minute check

Pick one public question: "What does this organization do?" Try to answer it from the homepage, About page, a service page, and one external profile. If you get four materially different answers, the next task is consistency—not more content volume.

I use this evidence-first approach in my work on AI-search visibility at AEOvara. Clear entity information supports readers and crawlers, but it does not guarantee inclusion in an AI-generated answer.

Missing Is Not Zero: A Data Quality Checklist for AI Visibility Tests

Jarno S — Fri, 04 Sep 2026 13:38:18 +0000

Your dashboard says a brand appeared in 25% of AI answers. Before interpreting that number, ask a less exciting question: what counted as an answer?

If a test failed, timed out or never reached the intended question, recording a zero turns a collection problem into an apparent visibility problem.

Here is a practical way to keep those outcomes separate.

1. Separate absence from missing information

A completed answer that does not mention the brand is a valid non-mention.

An interrupted session with no assessable answer is missing data.

Store them differently:

false: the outcome was assessed and absent.
true: the outcome was assessed and present.
null: the outcome could not be assessed.

This distinction should apply separately to brand mentions, owned-domain citations and explicit recommendations. An answer might be scorable for mentions while its citations remain inaccessible.

2. Define completeness before seeing the result

Write down what makes an observation usable before collecting answers.

For example: the system answered the intended question, and the relevant output was preserved for review.

Record refusals separately. Whether they belong in a particular denominator depends on what the metric is intended to measure. Preserve them and disclose the rule.

Do not decide that an answer is “incomplete” simply because the brand is missing.

3. Report coverage beside performance

Consider this hypothetical batch—not an actual client result:

Eight attempts were planned.
Six produced complete, scorable answers.
Two of those six mentioned the brand.
Two attempts failed.

The mention rate among complete answers is 2/6, or 33.3%.

Collection coverage is 6/8, or 75%.

Reporting only 2/8 treats failures as known non-mentions. Reporting only 33.3% hides the missing observations.

Show both. Also explain if failures cluster around particular questions, because the available answers may not represent the planned test set.

4. Preserve reruns

A rerun should create another observation, not erase the first attempt.

Set a rule in advance: for example, retry a technical failure once, but do not rerun a valid answer simply because it omitted the brand.

Selecting the most flattering answer from several attempts measures something different from following a fixed test procedure.

5. Test the scoring rules

Check these edge cases before trusting your dashboard:

A third-party source mentions the brand, but the brand’s website is not cited.
The brand appears in a negative comparison.
A similarly named business appears.
A domain is printed as plain text without being used as a supporting source.
An interrupted run produces no assessable answer.

A mention, citation and recommendation should not automatically receive the same label.

Make the score auditable

Preserve the raw response, prompt version, timestamp, visible model label, interface and retrieval settings when observable. Record “unknown” instead of guessing.

The useful output is not just a percentage. It is a percentage another reviewer can reconstruct.

I work on AEOvara, a Finnish AEO and AI-search visibility project. This proposed checklist extends my earlier repeatable benchmark guide with data-quality checks. It does not establish that any content change causes more AI recommendations.

What edge case would you add to the scoring tests?

How to Build a Repeatable AI Search Visibility Benchmark

Jarno S — Tue, 18 Aug 2026 17:56:17 +0000

Checking whether an AI assistant mentions your company once is interesting, but it is not a measurement system. Answers vary between models, sessions, dates, and prompt wording. A useful benchmark needs a stable set of questions, a consistent scoring method, and a record of what changed.

This article describes a lightweight approach that a small team can run without an enterprise monitoring platform.

1. Start with decisions, not keywords

Traditional rank tracking starts with search terms. AI-search monitoring should start with the decisions a potential customer asks an assistant to help make.

Build prompts from four groups:

Category discovery: “What are the best ways to solve X?”
Provider discovery: “Which companies help with X in Finland?”
Comparison: “What is the difference between approach A and approach B?”
Trust and validation: “How can I evaluate whether a provider is credible?”

Keep the set small enough to run consistently. Twenty carefully selected prompts are more useful than two hundred prompts that change every month.

2. Freeze a golden prompt set

A golden prompt set is a versioned list of prompts that stays stable between measurement rounds. Each row should include at least:

prompt_id, journey_stage, prompt_text, language, market, expected_entities

Do not silently rewrite prompts after a run. If a prompt needs to change, create a new version. Otherwise, an apparent visibility improvement may only be the result of easier wording.

3. Capture more than mentions

A binary “mentioned / not mentioned” score misses most of the useful information. For each response, record:

run_date
model
prompt_id
brand_mentioned
mention_position
recommendation_strength
linked_or_cited
cited_domains
answer_summary

Recommendation strength can use a simple scale:

0: not mentioned
1: mentioned incidentally
2: included as a relevant option
3: clearly recommended or used as a primary example

Keep the raw answer as evidence. Scores alone are difficult to audit later.

4. Separate visibility from citation

These are related but different outcomes:

Visibility: the brand or entity appears in the answer.
Citation: the assistant links to or names a source that supports the answer.

A company can be visible without its own website being cited. A website can also be cited while the brand is not recommended. Track both rates separately.

For a prompt set with N prompts:

mention_rate = prompts_with_brand_mention / N
citation_rate = prompts_citing_owned_domain / N
recommendation_rate = prompts_with_strength_2_or_3 / N

These simple metrics are easy to explain and hard to manipulate.

5. Control the test conditions

Perfect repeatability is not possible, but avoid unnecessary variation:

Run the same prompts in the same order.
Record the model and date.
Use a fresh conversation for each prompt.
Keep language and market context consistent.
Avoid follow-up questions in the benchmark run.
Run on a regular schedule, such as once a month.

The goal is not laboratory certainty. The goal is enough consistency to distinguish a durable trend from a lucky answer.

6. Review the evidence behind changes

When a score moves, inspect the responses before drawing conclusions.

Ask:

Did the assistant discover a new page or third-party source?
Did a competitor disappear rather than our brand improve?
Did the answer cite stronger evidence than last month?
Did the model misunderstand the category?
Is the change visible across several prompts or only one?

This review turns monitoring into an action list. For example, repeated category confusion suggests clearer entity and service descriptions. Missing evidence may suggest publishing a primary source, case study, methodology, or data page.

7. Report trends, not isolated wins

A useful monthly report can fit on one page:

Mention rate
Citation rate
Strong recommendation rate
Most frequently cited domains
Prompts with the largest positive and negative changes
Three evidence-backed actions for the next month

Avoid presenting a single flattering answer as proof of authority. A stronger signal is repeated appearance across relevant prompts, models, and measurement rounds.

The practical takeaway

AI-search visibility is noisy, but it is measurable. The key is to freeze the questions, preserve the raw evidence, score consistently, and investigate why the answers changed.

I use this type of repeatable measurement thinking while developing AEOvara, a Finnish AI-search visibility and AEO project. The method itself is deliberately tool-independent: a spreadsheet and disciplined process are enough to begin.