DEV Community: Ben Kemp | AI Consultant

Measuring AI Visibility: A Practical Framework for Reliable Results

Ben Kemp | AI Consultant — Thu, 15 Oct 2026 13:13:22 +0000

As AI-powered search and assistants become part of how people discover brands, products, and information, developers and analysts face a new measurement challenge.

How do you reliably measure whether a brand appears in AI-generated answers?

At first glance, the task seems simple: submit queries, collect responses, and count mentions. But reliable measurement requires more than string matching and a dashboard. The results depend on how queries are sampled, how observations are classified, how metrics are calculated, and how variability is handled.

This article outlines a practical framework for measuring AI visibility with greater consistency and transparency.

1. Define the observable signals

Start by separating different outcomes rather than treating every appearance as the same event.

For example:

Brand mention: The brand is named in a response.
Citation: A source is explicitly referenced or linked according to a defined rule.
Recommendation: The brand or product is presented as an option for the user's stated need.
Brand accuracy: The factual information presented about the brand is assessed against verifiable information.

These signals can overlap, but none should automatically be used as a substitute for another.

A response may mention a brand without recommending it. A citation may support a factual statement without endorsing the cited organization. A recommendation may appear without a direct citation to the brand's website.

A reliable measurement pipeline should record these outcomes separately.

2. Build a defined query set

A metric is only meaningful relative to the queries used to produce it.

A test set might include queries grouped by:

User intent
Product or service category
Use case
Geographic market
Comparison or recommendation scenario

For example, a software company might test queries about project management, team collaboration, pricing comparisons, and solutions for small businesses.

The objective is not to create the largest possible query set. It is to create a relevant, documented set that reflects the questions the evaluation is intended to investigate.

Record the query text and its category. If the test set changes between reporting periods, document the change so that shifts in results are not mistaken for changes in visibility.

3. Choose the unit of analysis

One of the most important design decisions is deciding what counts as an observation.

Possible units include:

Individual responses
Unique queries
Repeated runs of the same query
Query categories
Sessions or interaction sequences

These units produce different metrics.

Suppose a brand is recommended in 30 out of 100 collected responses. Its response-level recommendation rate is 30%.

Now suppose those 100 responses came from 20 queries, with five responses collected for each query. If the brand was recommended at least once for 12 queries, its query-level recommendation coverage is 60%.

Both results can be valid. They measure different things.

Always state the unit of analysis alongside the metric.

4. Define metrics explicitly

A useful metric definition should include the numerator, denominator, inclusion criteria, and unit of analysis.

For example:

Response-level brand mention rate

[
\text{Mention Rate} =
\frac{\text{Responses containing a brand mention}}
{\text{Valid responses evaluated}} \times 100
]

Query-level recommendation coverage

[
\text{Recommendation Coverage} =
\frac{\text{Queries with at least one qualifying recommendation}}
{\text{Valid queries evaluated}} \times 100
]

These definitions depend on explicit classification rules. A recommendation coverage calculation, for instance, must define what qualifies as a recommendation and how repeated responses for the same query are combined.

Document how empty responses, failed requests, duplicate observations, and ambiguous classifications are handled. Otherwise, two implementations may produce different results while claiming to calculate the same metric.

5. Preserve raw observations

Do not store only aggregated percentages.

For each observation, retain appropriate evaluation metadata, such as:

Field	Purpose
Query ID	Identifies the tested query
Query text	Records the input used
System	Identifies the AI product evaluated
Timestamp	Establishes when the observation occurred
Response text	Preserves the output being analyzed
Mention detected	Records the classification result
Citation detected	Records whether a qualifying citation appeared
Recommendation detected	Records the recommendation classification
Classification method	Documents whether the result was automated, manually reviewed, or both

The exact schema will vary by project. The principle is more general: preserve enough evidence to inspect how an aggregate result was produced.

Respect applicable privacy requirements and platform terms when collecting and storing responses.

6. Treat classification as a measurement problem

Some signals are relatively straightforward to detect. A brand mention may be identified through entity matching, aliases, and contextual checks.

Other signals require more interpretation.

A recommendation, for example, may be explicit:

“For your use case, Product A is a suitable option.”

Or it may be ambiguous:

“Products A, B, and C are commonly used in this category.”

The second statement names several products but does not necessarily recommend any of them.

If classification is automated, document the rules or model used and evaluate its errors against a manually reviewed sample. Track false positives and false negatives where feasible.

For ambiguous cases, consider recording a confidence score or a separate “uncertain” category rather than forcing every response into a binary label.

The aim is not to eliminate all judgment. It is to make judgment visible and consistent.

7. Account for variability

AI-generated responses may vary across runs, time periods, configurations, and products. A single observation should not automatically be interpreted as a stable property of a brand.

A more robust evaluation may repeat selected queries under documented conditions.

For repeated tests, record:

Number of runs per query
Timing and relevant configuration details
Whether the same query wording was used
How repeated observations were aggregated
How missing or failed observations were treated

When the sample size and sampling design support it, uncertainty intervals can help communicate how much confidence to place in an estimate.

Repeated tests do not guarantee that results represent all real-world user experiences. They provide additional evidence about variability under the conditions actually tested.

8. Separate observations from causal claims

A measurement system can establish that a brand appeared in a response under certain conditions. It generally cannot establish, from that response alone, exactly why the system produced it.

For example, finding that a page was cited does not prove that a particular content change caused the citation. The result may be consistent with a hypothesis about relevance or source selection, but causal attribution requires additional evidence and an appropriate study design.

A useful reporting format separates:

Observation: What appeared in the collected output?
Metric: How was the observation aggregated?
Interpretation: What might the result suggest?
Limitation: What cannot be established from the evidence?

This structure helps prevent correlation, speculation, and internal-process assumptions from being presented as verified explanations.

9. Make comparisons reproducible

Comparing brands or time periods requires consistent conditions.

At minimum, document the query set, classification rules, unit of analysis, evaluation window, and system or configuration details where available.

If one report counts every response and another counts each query only once, their percentages are not directly comparable. Likewise, a change in the query set may change the result even if the underlying response patterns remain stable.

A good report should make it possible for another analyst to understand the calculation and identify which methodological differences could affect the comparison.

A practical checklist

Before publishing an AI visibility report, verify the following:

[ ] The metric has an explicit definition.
[ ] The numerator and denominator are documented.
[ ] The query set and sampling method are described.
[ ] Mentions, citations, and recommendations are tracked separately.
[ ] Ambiguous classifications have documented rules.
[ ] Raw observations are retained where appropriate.
[ ] Repeated tests and failed observations are handled consistently.
[ ] Comparisons use equivalent conditions or disclose differences.
[ ] Conclusions distinguish observed outcomes from inferred causes.
[ ] Limitations are stated alongside the results.

Conclusion

Reliable AI visibility measurement is not simply a matter of collecting more responses or producing more charts. It requires explicit definitions, consistent sampling, transparent classification, and a clear separation between evidence and interpretation.

These practices make results easier to audit, compare, and improve over time.

The terminology itself matters, too. When different teams use the same term to describe different calculations, even technically correct reports can be difficult to compare.

AI Visibility Glossary is building a neutral, vendor-agnostic reference for the terminology, metrics, and methodologies used to describe AI visibility.

Explore the glossary: https://aivisibilityglossary.com/

Discussion: If you were designing an AI visibility measurement pipeline, which would you standardize first: query sampling, recommendation classification, or metric definitions?

AI Visibility Measurement: Why Definitions Matter More Than Dashboards

Ben Kemp | AI Consultant — Wed, 14 Oct 2026 10:06:04 +0000

nt system needs a documented methodology.

At a minimum, that means defining the queries being tested, identifying the platforms included, recording when responses are collected, and establishing consistent rules for classifying mentions, citations, and recommendations.

Repeatability matters, too. AI-generated responses can vary across runs, so a single observation may not represent a stable pattern. Sample size, collection frequency, and classification rules all affect how results should be interpreted.

There is also an important technical boundary: observing an answer does not necessarily reveal the internal process that produced it. Measurement should distinguish verifiable outputs from assumptions about model behavior.

Toward a common vocabulary

The industry doesn't necessarily need one universal metric for every use case. It does need terminology that makes differences between metrics explicit.

A shared vocabulary helps teams document their methods, compare findings more carefully, and communicate results without implying more certainty than the evidence supports.

AI Visibility Glossary is building a neutral, vendor-agnostic reference for AI visibility terminology, measurement, and methodology.

For developers and analytics practitioners working with AI search, the underlying principle is simple: before optimizing a metric, establish what it measures.

Explore the glossary: AI Visibility Glossary

Discussion question: If you were designing an AI visibility measurement pipeline, which signal would you prioritize first: mentions, citations, recommendations, or something else?

Sparse Neural Networks in Python — From Pruning to Dynamic Rewiring

Ben Kemp | AI Consultant — Sat, 31 Jan 2026 10:24:25 +0000

Deep learning has followed a predictable pattern for years:

Add more layers. Add more parameters. Add more GPUs.

Dense scaling works — but it’s expensive, wasteful, and increasingly impractical outside hyperscale environments.

Sparse neural networks offer a different direction:

Keep the capacity. Reduce the computation.

And you don’t need trillion-parameter models to understand how.

In this series, I implemented sparse neural networks step-by-step in PyTorch — starting from scratch and moving toward dynamic sparse training.

Here’s what sparse actually means in practice.

What Is a Sparse Neural Network?

A neural network is sparse when:

Many weights are exactly zero

Or only a fraction of neurons activate per input

Or only parts of the network are used conditionally

Instead of computing everything, you compute only what matters.

That changes the scaling equation.

Dense layer compute: FLOPs ≈ input_dim × output_dim
Sparse layer compute: FLOPs ≈ (1 − sparsity) × input_dim × output_dim

At 80% sparsity, you keep 20% of the compute.

That’s not compression — that’s architectural efficiency.

The Python-First Sparse Series

This isn’t theory-heavy.

Each article builds sparse models directly in PyTorch.

1️⃣ Dense vs Sparse (Masking)

We start with a normal MLP and introduce a binary weight mask:

sparse_weight = weight * mask

That’s it.

You immediately control structural sparsity.The Python-First Sparse Series

This isn’t theory-heavy.

Each article builds sparse models directly in PyTorch.

1️⃣ Dense vs Sparse (Masking)

We start with a normal MLP and introduce a binary weight mask:

sparse_weight = weight * mask

That’s it.

You immediately control structural sparsity.

2️⃣ Magnitude-Based Pruning

Train dense → remove smallest weights:

threshold = torch.quantile(weights.abs(), pruning_ratio)
mask = weight.abs() > threshold

You can often prune 80–90% of weights with surprisingly small degradation.

This is the simplest form of structural sparsity.

3️⃣ Activation Sparsity (k-WTA)

Instead of removing weights, restrict which neurons fire:

topk_vals, topk_idx = torch.topk(x, k, dim=1)
mask.scatter_(1, topk_idx, 1.0)

Now only k neurons activate per sample.

Compute drops. Structure stays intact.

4️⃣ Sparse Training From Scratch

Why train dense at all?

Initialize sparse and train only active connections.

Weights that are masked never receive gradient updates.

You eliminate wasted early compute.

5️⃣ Dynamic Sparse Training

Static masks can be limiting.

So we rewire during training:

Prune weak connections

Regrow new ones

Keep total sparsity constant

Now the network doesn’t just optimize weights.

It optimizes connectivity.

This is conceptually close to modern sparse research (RigL-style approaches).

Why Developers Should Care

Sparse networks aren’t just research experiments.

They matter because:

Compute is expensive

Edge devices need efficiency

Model size ≠ model cost

Modern MoE architectures are sparse

Conditional execution is becoming standard

If you’re building models beyond toy datasets, efficiency becomes real very quickly.

Dense Scaling vs Sparse Scaling

Dense scaling: More parameters → more compute

Sparse scaling: More capacity → controlled compute

That shift changes architecture design decisions.

Where This Leads

The next logical step is:

Sparse attention

Mixture of Experts

Conditional token routing

Fair dense vs sparse benchmarking

Because sparsity isn’t about shrinking models.

It’s about scaling smarter.

Final Thought

If you want to understand sparse neural networks, don’t start with theory.

Start with code.

Once you see how much you can remove — and still learn — you’ll realize dense is just one point in the design space.

Sparse networks open the rest of it.

The Neural Network Lexicon: Understand Neural Networks Without the Black Box

Ben Kemp | AI Consultant — Fri, 30 Jan 2026 14:40:17 +0000

Neural networks power modern AI — but for many developers, they still feel like magic.

Not because the math is impossible, but because most explanations are either:

too theoretical, or

hidden behind high-level libraries.

I built the Neural Network Lexicon to fix that.

What Is the Neural Network Lexicon?

It’s a concept-by-concept reference for neural networks, explained from first principles.

One concept per page.
Clear definitions.
No framework lock-in.

Each entry answers:

What is this concept?
Why does it matter?
How does it work conceptually?
What usually goes wrong?

And yes — every concept includes a minimal Python example to make the computation visible.

Why Python (and Why Minimal)?

The Python snippets are intentionally small.

Not to build full models — but to show that:

neural networks are just computations.

Seeing a neuron as a weighted sum or a loss function as a number you can print changes how you think about ML.

Runnable Examples on GitHub

To keep the lexicon readable, full runnable examples live in GitHub:

One idea per file
No frameworks
Edit → run → observe

Read the concept, run the code, tweak a value, and learn faster.

What Does It Cover?

The lexicon is complete, not just introductory:

Core foundations (neurons, activations, loss)
Training & optimization
CNNs, RNNs, Transformers
Generalization & robustness
Explainability, uncertainty, fairness
Deployment & model lifecycle

In total: 100 structured entries.

Who Is This For?

Developers using ML libraries who want real understanding
Students overwhelmed by fragmented explanations
Engineers who want to debug models, not just train them

If you believe understanding comes before optimization, this is for you.

📘 Neural Network Lexicon (GitHub Wiki)
Built as part of SolveWithPython — learning by understanding, not memorizing.

Neural networks aren’t magic.
Once you understand what they compute, everything else follows.