DEV Community

Algis
Algis

Posted on

Building LLM-Powered Audience Testing with AI Agents

#ai #llm #ux

🗒️ Summary

Large Language Models (LLMs) can act as “people spirits”—stochastic simulations of real users[1]. By pairing them with Model Context Protocol (MCP) browser automation, we can already run realistic A/B tests and spot issues before shipping code.

1. The Core Concept: LLMs as People Spirits

Andrej Karpathy calls LLMs “stochastic simulations of people” powered by an autoregressive Transformer[1]. Because they are trained on human text, they develop an emergent, human-like psychology—perfect for audience testing.

2. Research Foundation: LLM-as-Judge Accuracy

Studies find LLM evaluations correlate up to 80 % with human judgment[2][3], though the best models still trail behind inter-human agreement[4]. Stanford’s generative-agent work even showed 85 % self-agreement on survey answers two weeks apart[5].

Bottom line: today’s top models are “good enough” to guide product decisions at scale.

3. System Prompts = Instant Personas

A single system prompt can turn one model into many audiences:

You are a 25-year-old gamer from Berlin who values speed and dark themes.
Enter fullscreen mode Exit fullscreen mode

Combine demographic, psychographic, and cultural cues to create diverse personas. AgentA/B research confirms that LLM personas can navigate real webpages and mimic user behavior[6][7].

4. Wiring an AI-Driven A/B Test

Step What to Do Why It Matters
1 Control vs. Variations – draft baseline and experimental prompts Sets up classic A/B structure
2 MCP Browser Automation – let agents click, scroll, fill forms[9][10] Generates realistic interaction data
3 Log & Score – capture impressions, task success, sentiment Quantifies user experience
4 Analyze – compare KPIs across personas Reveals which version wins and why

5. Business Wins

  • Product Development: Test features with Gen Z gamers, Millennial execs, or rural seniors—overnight.
  • Marketing Copy: Iterate headlines until every persona clicks.
  • UX Audits: Detect accessibility or cultural friction long before launch.

6. Tech Stack: Ready Today

  • LLMs: GPT-4-level or better for high alignment[11].
  • MCP: Standard bridge that lets agents control browsers and other tools[9][12].
  • Automation Servers: Browser MCP or Playwright MCP for GUI tasks[10].

7. Roll-Out Plan

  1. Proof of Concept – spin up 3–5 key personas and test one flow.
  2. Integrate – pipe results into existing A/B dashboards.
  3. Scale – auto-generate new personas, add prompt-tuning loops, build reporting widgets.
  4. Advance – predict reactions to unreleased features, run global localization checks, model competitor responses.

LLMs already let us see through our users’ eyes. Pair them with MCP automation, and you can iterate faster than ever—no waiting for live traffic.

References

[1] Karpathy A. “Software Is Changing (Again)” – YC AI Startup School

[2] Jung J. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

[3] Jung J. Trust or Escalate (companion study)

[4] Thakur A. S. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

[5] Park J. S. Simulating Human Behavior with AI Agents – Stanford HAI

[6] Park J. S. AgentA/B: Automated and Scalable Web A/B Testing with Interactive LLM Agents

[7] Park J. S. AgentA/B (v2)

[9] Anthropic. Introducing the Model Context Protocol

[10] Browser MCP. Automate your browser with AI

[11] MCP.so. Browser Automation MCP Server

[12] Microsoft. Model Context Protocol (MCP): Integrating Azure OpenAI

Top comments (0)