DEV Community: Jesper Deng

I Built an AI Search Monitoring Platform Because “Are We in ChatGPT?” Shouldn’t Be a Guess

Jesper Deng — Wed, 07 Oct 2026 09:16:16 +0000

AI search monitoring is a repeatable way to observe whether a brand appears, is mentioned, and is cited in answers to a stable set of buyer questions.

It is an observation loop, not an official ranking.

I built AI Search Vitals because traditional SEO reports were not answering a question I kept hearing:

When a potential customer asks an AI system for a recommendation, what does it actually say about our brand?

The Problem No One Talks About

Traditional search gives you familiar signals: impressions, clicks, rankings, and backlinks.

AI answers are different.

The same buyer question can produce different results depending on the model, prompt wording, market, language, retrieval mode, source set, and time of day. A screenshot may show what happened once, but it does not explain whether the result is a pattern.

There are four problems I wanted to solve:

A brand can be relevant to a category but never appear in the answer.
A brand can be mentioned without being cited as a source.
A competitor can be recommended while your better explanation is ignored.
A single visibility score cannot explain what changed or what to fix next.

The key insight was simple:

The hard part is not generating another score. The hard part is preserving enough evidence to understand the answer behind the score.

What I Decided to Build

AI Search Vitals is an AI search visibility monitoring platform for brands, marketing teams, SEO and GEO practitioners, agencies, and founders.

The workflow starts with a project:

Add a brand, website, market, language, and competitors.
Save the buyer questions you want to monitor.
Select the AI channels enabled for the workspace.
Run an initial baseline.
Review mentions, recommendations, competitors, and citations.
Schedule daily or weekly observations.
Use the results to decide which page, prompt, or content gap to improve.

The product keeps the evidence behind every successful observation:

The original buyer prompt
The actual model and provider
Market and language
Search mode and run timestamp
Raw AI answer
Brand and competitor mentions
Ordered recommendation position when available
Citation URLs, domains, titles, and snippets
Token usage and request metadata

This makes it possible to ask better questions than “Did our score go up?”

For example:

Did the brand move from absent to mentioned?
Was the mention accurate or misleading?
Did the model cite our website or a competitor?
Which page became the source?
Did the change happen across a prompt group or only once?

AI Search Vitals also includes competitor share of voice, citation exploration, prompt research, monitoring history, CSV export, and GEO audit tools for crawlability, content readiness, and AI Query Expansion.

How It Works Under the Hood

The application runs on Next.js 15, React, and TypeScript, with OpenNext and Cloudflare Workers handling the production runtime.

Workspace, project, prompt, run, observation, mention, and citation data are stored in Cloudflare D1 through Drizzle. OpenRouter adapters keep provider metadata, usage information, citations, and raw response context instead of reducing every response to a plain string.

Scheduled monitoring uses a server-controlled workflow:

Cloudflare Cron finds prompts that are due.
Each prompt and model target becomes a queue job.
The worker executes the observation.
The parser extracts mentions, positions, and citations.
The observation is stored with an idempotency key.
Retryable provider failures can be inspected without duplicating the result.

The goal is not to make AI search look deterministic.

The goal is to make changes comparable.

Why This Is Harder Than It Sounds

1. Prompt stability matters

If the question changes every time, the result is difficult to compare.

That is why AI Search Vitals stores prompt snapshots and encourages teams to separate discovery, evaluation, comparison, and task-based questions. A stable prompt set creates a baseline that can be reviewed over time.

2. Mentions need context

Counting a brand name is not enough.

The system needs to distinguish between a useful recommendation, a passing reference, an incorrect description, and a competitor comparison. A mention without context can lead to the wrong content decision.

3. A mention is not a citation

A model can name a brand without linking to its website.

It can also cite a page for one narrow fact without recommending the whole brand. These are different signals, so they need to be measured separately.

4. Citation formats are inconsistent

Different providers can return citations through annotations, markdown links, metadata, or no structured source list at all.

The parser therefore checks provider annotations first and then uses deterministic URL extraction as a fallback. The result is not perfect, but it is inspectable and tied to the actual answer.

What I Learned Building This

Evidence beats an opaque score

A score can tell you that something changed. Raw answer evidence helps explain why.

The useful unit is not “visibility increased by 8%.” It is:

For this prompt and channel, the answer changed, the brand became more specific, and this page became the cited source.

Measurement boundaries build trust

AI Search Vitals does not claim to reproduce the first-party consumer experience of ChatGPT, Google AI Overviews, or any other platform.

The current product records provider and API proxy observations. These observations are useful for repeatable analysis, but they are not official or universal rankings.

Google AI Overviews still require a manual check in the target market. Estimated intent is directional, not official search volume. A citation is evidence, not a guarantee of future visibility.

GEO and monitoring should be connected

Monitoring tells you what the answer system did.

A GEO audit helps investigate why:

Can the page be fetched?
Are the headings and metadata clear?
Is the entity defined consistently?
Does the page contain enough evidence?
Are there related questions the content should answer?

That creates a loop:

Observe → inspect evidence → improve the page → observe again.

Try It

Start with a small prompt set instead of monitoring everything at once.

Use five to twelve buyer questions across:

Category discovery
Product comparison
Recommendation
Problem solving
Brand-specific queries

Read the raw answers before reacting to the metrics. Then compare the same prompts, channel, market, and language after a content change.

At the time of writing, AI Search Vitals includes a seven-day trial without a credit card, so the easiest way to start is with one project and a focused baseline.

You can try AI Search Vitals, read the AI Search Monitoring Guide, or review the AI Citation Tracking Guide.

I am especially interested in hearing how other teams measure AI visibility without turning a changing answer into a fake ranking report.

I Built an AI Mock Trial Platform Because Practicing Law Shouldn't Require a Full Cast

Jesper Deng — Wed, 20 May 2026 05:20:41 +0000

The Problem No One Talks About

If you're a law student preparing for mock trial, or a lawyer rehearsing for court, you face a frustrating reality: you can't practice alone.

A real trial involves a judge, opposing counsel, witnesses, and jurors. To run even a basic practice session, you need to coordinate 3-5 people's schedules. Most of the time, that just doesn't happen.

So what do people actually do?

Law students rehearse opening statements in front of a mirror
Mock trial teams only get 1-2 full run-throughs before competition
Young lawyers go into their first trial with almost no live courtroom experience
Solo practitioners have zero way to simulate cross-examination or hostile witnesses

The fundamental bottleneck isn't skill — it's access to practice partners.

What I Decided to Build

I asked a simple question: what if AI could play every other role in the courtroom?

Not a chatbot that answers legal questions. Not a document tool. A full courtroom simulation where:

You play the attorney (plaintiff or defense)
AI plays the judge — ruling on objections, managing procedure
AI plays opposing counsel — making arguments against you, objecting to your questions
AI plays witnesses — responding to direct and cross-examination with realistic personalities
The trial follows real procedure: voir dire → opening → witness examination → closing → verdict

At the end, you get scored on your performance.

Why This Is Harder Than It Sounds

The challenge isn't just "make AI talk like a lawyer." It's:

1. Multi-role coherence

The judge, opposing counsel, and witness are all AI — but they need to behave as separate people with different goals. The judge is neutral. Opposing counsel is adversarial. The witness has a backstory and may be unreliable. One model, multiple conflicting personas, in the same conversation.

2. Stage management

A trial isn't a free-form chat. It has strict procedural stages. You can't cross-examine during opening statements. The AI needs to enforce courtroom rules while still feeling natural.

3. Reactive complexity

If you raise an objection, the judge must rule. If sustained, opposing counsel must rephrase. If you introduce surprise evidence, the witness must react consistently with their backstory. Every action cascades.

4. Solo practice must feel real

If it feels like talking to a chatbot, lawyers won't use it. The responses need enough unpredictability and pushback to create genuine practice pressure.

What I Learned Building This

Start with the state machine, not the prompts. I spent too long tweaking AI personalities before realizing the real problem was managing trial flow. Once I built a proper stage system (8 stages, with rules for what's allowed in each), the AI behavior fell into place.

Scoring creates motivation. Early testers would quit mid-trial. Adding a verdict with performance scoring changed everything — people now complete full trials because they want to see their score.

Credits > subscriptions for this audience. Law students are broke. A generous free tier with credits lets them actually use the tool. Power users (practicing attorneys) will pay when they see the value.

Try It

If you're curious:

Live platform: mocktrialonline.com
Demo video: YouTube walkthrough

It's free to sign up and run several full trials without paying. I'm a solo developer building this actively — feedback from anyone (devs, lawyers, or just people curious about legal AI) is genuinely welcome.

How I Made AI Behave Differently Based on Conversation Context (Multi-Role Prompt Engineering)

Jesper Deng — Sat, 09 May 2026 08:35:09 +0000

I've been working on a project that requires multiple AI "characters" to behave differently in the same conversation — think of it like NPCs in a game, except each one needs to respond based on their role, personality, and the current situation.

Here's what I learned about making this work reliably.

The problem

If you just tell an AI "you are character A" and "you are character B" in separate prompts, they all end up sounding the same. Generic. Helpful. Boring. You need them to have distinct behaviors — one should be cooperative, another defensive, another authoritative.

What actually works

1. Behavioral constraints > personality descriptions

Bad:

You are a friendly witness who is helpful.

Better:

You are a witness being questioned. Rules:
- Only answer what is directly asked
- If the question is vague, ask for clarification
- Never volunteer extra information
- If pressed on a contradiction, become defensive

Constraints produce more consistent behavior than adjectives.

2. Context-dependent behavior switching

The same character might need to behave differently depending on who's talking to them. I handle this by passing a mode parameter in the system prompt:

function buildPrompt(character: Character, mode: 'friendly' | 'hostile') {
  const base = `You are ${character.name}. Background: ${character.bio}`;

  if (mode === 'friendly') {
    return `${base}\n\nBehavior: Be cooperative. Give detailed answers. Expand on your responses when appropriate.`;
  } else {
    return `${base}\n\nBehavior: Be defensive. Give minimal answers. Only confirm what you cannot deny. Redirect when possible.`;
  }
}

This tiny switch makes a huge difference in how natural the responses feel.

3. State management across turns

The hardest part: making characters remember what happened earlier and adjust. I maintain a simplified state object:

interface ConversationState {
  currentSpeaker: string;
  previousStatements: string[];
  contradictions: string[];
  mood: 'neutral' | 'defensive' | 'confident';
}

Before each AI call, I inject a summary of what's happened so far. This keeps responses contextually aware without blowing up the token count.

4. The "don't be helpful" problem

LLMs are trained to be helpful. When you need a character to be evasive or unhelpful, you have to fight against this training. What works:

Explicitly say "you do NOT want to help the questioner"
Give the character a motivation for being difficult
Add examples of deflection in the prompt

You are being questioned about something you want to hide.
Your goal is to answer without revealing [specific fact].
Techniques you use: giving technically true but misleading answers,
answering a different question than what was asked, saying "I don't recall."

5. Temperature matters more than you think

For authoritative characters (judges, experts), use lower temperature (0.3-0.5). They should be consistent and decisive.

For emotional or unpredictable characters, bump it up (0.7-0.9). The randomness makes them feel more human.

Key takeaway

Multi-role AI isn't about writing better character descriptions. It's about defining behavioral rules, injecting context, and fighting the model's default "helpful assistant" mode. Once I figured that out, everything clicked.

Would love to hear if anyone else is doing multi-agent stuff — what patterns are you using?