How I'm Building Reproducible Agentic Tests for AI Girlfriend Apps

Sam — Sat, 15 Aug 2026 07:15:23 +0000

AI companion and AI girlfriend apps are a useful stress test for a problem that affects many stateful AI products: the claims users care about are easy to market and surprisingly hard to verify.

"Long-term memory" can mean remembering a name ten messages later, retaining a fact across sessions, or keeping dozens of facts straight after the conversation has accumulated noise. "Consistent characters" can mean tone, biography, appearance, or all three. A single successful image says almost nothing about the next ten generations.

Yet many review pages turn those claims into precise-looking scores without showing the paid plan, test date, prompt sequence, transcript, retries, or failed generations behind the number.

I wanted a test another person could actually rerun.

The manual baseline

I started NoFilterReview by buying the products myself. Five paid hands-on tests are live today.

For each product I record:

the plan and price on the test date
what the free tier actually allows
cancellation steps
privacy and account controls
refusal and moderation behavior
dated screenshots and transcripts
media cost and the failed generations, not just the best output

That already catches a lot of bad review data. Prices change. Models get replaced. A feature shown on a landing page may not exist in the tested plan. A reviewer can easily mistake one lucky generation for normal quality.

But manual testing alone has a serious weakness: I am part of the instrument. My phrasing, timing, patience, and expectations can change the result.

The next step is an agentic benchmark that runs the same scenario against every product.

What the runner needs to control

The benchmark is still in build. I am not presenting the following system as finished.

The runner needs to control five things.

1. Facts and distractors

A memory test should plant facts at known points, then surround them with realistic conversational noise.

A draft scenario might include:

{
  "fact_id": "F07",
  "fact": "My sister's dog is named Pepper",
  "introduced_in_session": 1,
  "recall_check_session": 4,
  "acceptable_answers": ["Pepper"],
  "distractors": [
    "My coworker's dog is named Milo",
    "I used to have a cat named Luna"
  ]
}

The important part is not the JSON. It is the separation between what was introduced, when it was introduced, and what counts as a correct answer.

2. Multi-session behavior

A ten-message chat is not a long-term memory test.

The runner needs separate sessions, controlled gaps, topic changes, conflicting details, and delayed recall checks. It also needs to distinguish:

immediate recall
recall later in the same session
recall after a new session
recall after distractors
false memories that were never introduced
answers that dodge the question without actually recalling the fact

3. Character consistency

Memory and character consistency are related but not identical.

A character can remember a fact while drifting into a different personality. The evaluator should track changes in:

tone and speaking style
stated biography
relationship framing
boundaries and refusals
first-person identity claims

A useful benchmark should not reward a model for parroting facts while the character itself falls apart.

4. Image and video consistency

Media tests need repeated requests, not one cherry-picked result.

For every app, the runner should request comparable scenes and record:

generation cost
latency
visible identity drift
clothing and background adherence
anatomy and artifact failures
moderation or refusal
whether the product silently changes the prompt
all attempts, including failures

The hard part is scoring identity consistency without hiding the judgment inside another black box.

A vision model can help produce candidate scores, but the benchmark still needs a published rubric, calibration examples, and periodic human review. Otherwise the evaluator simply replaces an opaque product claim with an opaque benchmark claim.

5. Cost and failure handling

These products use different currencies, retry rules, queues, and subscription limits. Normalizing the marketing price alone is not enough.

I want to report both the advertised price and the observed cost of completing a fixed test scenario. Failed generations, retries, and timeouts stay in the denominator.

Separate the runner from the grader

The testing agent should not grade its own work.

My current design separates:

a scenario runner that sends prompts and captures raw artifacts
a grader that receives the rubric and those artifacts
an audit layer that preserves prompts, timestamps, screenshots, model/version notes, and scoring explanations

That separation does not eliminate bias, but it makes errors easier to inspect. If the score looks wrong, a reader can trace it back to a particular artifact and rubric rule.

What should be public

A benchmark is not reproducible if readers only see a leaderboard.

For each published result, I plan to expose:

scenario version
test date
product and paid plan
prompt sequence
transcript excerpts with privacy-safe redactions
screenshots and media samples
number of runs
success and failure counts
cost and latency
scoring rubric
known limitations

The goal is not to produce a magical objective score. The goal is to make every important judgment inspectable.

The two questions I have not solved

First, how long should a memory test run before "long-term" means anything? Four sessions may be practical, but practical is not the same as valid.

Second, what is the least misleading way to score identity consistency across images and video? Embedding similarity is tempting, but it can miss the exact visual drift a human notices immediately.

If you have built evaluation systems for stateful agents, multimodal models, or long-running conversations, I would value specific criticism of the protocol.

The current manual methodology and paid-test standard are here: https://nofilterreview.com/how-we-test/

Disclosure: I used an AI writing tool to help organize this article. The paid tests, product observations, methodology decisions, and final edits are mine.

I vibe coded an LM Arena for AI companion apps with Claude Code.

Sam — Sat, 25 Jul 2026 17:52:04 +0000

I can't write code. I've never written a line of it by hand.

I still wanted something that didn't exist: a leaderboard for AI companion sites that shows what was tested, when, and how many people voted. Most "Top 10 AI girlfriend apps" pages are affiliate lists with a number stuck on top. LMSYS Chatbot Arena does it properly, so I described that to Claude Code and we built it: seven leaderboards with community voting and dated monthly snapshots. Astro, Cloudflare Pages Functions, D1. No accounts, no login, no client framework. Voting is a static page and one POST.

Two things I learned from specifying all of it and writing none of it.

Every site on the board wants to stuff the vote

They all pay me affiliate commissions, so they all have a reason to send fake votes.

Ballots over the rate limit don't get rejected. They get written like any other ballot, they just never reach the table the score reads from. If I rejected them I'd have no record, and I couldn't tell you later how many got filtered or why. This way each board can show votes received and votes counted.

Ratings work the same way. Every submission is kept, and the current state is a derived table with one composite key:

ON CONFLICT(voter_id, app_id, category_slug)
DO UPDATE SET rating_event_id = excluded.rating_event_id, updated_at = excluded.updated_at

Vote again next month and the event is appended and the pointer moves, so nothing counts twice. If I find a cheating pattern later, the raw events are still there to rebuild from.

The bug that nearly took voting down

The code works. What almost broke it was a link between two things that look unrelated.

Each rating dimension sent a key to the API, and that key came from slugifying the label on screen. "Filter strength" turned into filter-strength, which matched. Then I renamed the label for a reason that had nothing to do with the API: a 5 meant "never blocks you", so high was good, but "filter strength" sounds like high means more filtering. It read backwards, so I changed the words.

Every ballot in that category would have failed with a 400, because spicy isn't a key the API knows, and nothing would have told me why.

If you're shipping code you can't read, that's the thing to watch for. Not code that's broken when it lands. Code that works and is wired to a word you'll change later without thinking about it.

Next is an agent harness that runs the same script against every site and scores the transcripts on memory, character consistency and image consistency, publishing the prompts and the transcripts with each run.

The boards are live at nofilterreview.com/rankings. Adult AI sites, so consider that your content warning. Try to break the vote system, I want to know where it gives.

DEV Community: Sam