AI companion and AI girlfriend apps are a useful stress test for a problem that affects many stateful AI products: the claims users care about are easy to market and surprisingly hard to verify.
"Long-term memory" can mean remembering a name ten messages later, retaining a fact across sessions, or keeping dozens of facts straight after the conversation has accumulated noise. "Consistent characters" can mean tone, biography, appearance, or all three. A single successful image says almost nothing about the next ten generations.
Yet many review pages turn those claims into precise-looking scores without showing the paid plan, test date, prompt sequence, transcript, retries, or failed generations behind the number.
I wanted a test another person could actually rerun.
The manual baseline
I started NoFilterReview by buying the products myself. Five paid hands-on tests are live today.
For each product I record:
- the plan and price on the test date
- what the free tier actually allows
- cancellation steps
- privacy and account controls
- refusal and moderation behavior
- dated screenshots and transcripts
- media cost and the failed generations, not just the best output
That already catches a lot of bad review data. Prices change. Models get replaced. A feature shown on a landing page may not exist in the tested plan. A reviewer can easily mistake one lucky generation for normal quality.
But manual testing alone has a serious weakness: I am part of the instrument. My phrasing, timing, patience, and expectations can change the result.
The next step is an agentic benchmark that runs the same scenario against every product.
What the runner needs to control
The benchmark is still in build. I am not presenting the following system as finished.
The runner needs to control five things.
1. Facts and distractors
A memory test should plant facts at known points, then surround them with realistic conversational noise.
A draft scenario might include:
{
"fact_id": "F07",
"fact": "My sister's dog is named Pepper",
"introduced_in_session": 1,
"recall_check_session": 4,
"acceptable_answers": ["Pepper"],
"distractors": [
"My coworker's dog is named Milo",
"I used to have a cat named Luna"
]
}
The important part is not the JSON. It is the separation between what was introduced, when it was introduced, and what counts as a correct answer.
2. Multi-session behavior
A ten-message chat is not a long-term memory test.
The runner needs separate sessions, controlled gaps, topic changes, conflicting details, and delayed recall checks. It also needs to distinguish:
- immediate recall
- recall later in the same session
- recall after a new session
- recall after distractors
- false memories that were never introduced
- answers that dodge the question without actually recalling the fact
3. Character consistency
Memory and character consistency are related but not identical.
A character can remember a fact while drifting into a different personality. The evaluator should track changes in:
- tone and speaking style
- stated biography
- relationship framing
- boundaries and refusals
- first-person identity claims
A useful benchmark should not reward a model for parroting facts while the character itself falls apart.
4. Image and video consistency
Media tests need repeated requests, not one cherry-picked result.
For every app, the runner should request comparable scenes and record:
- generation cost
- latency
- visible identity drift
- clothing and background adherence
- anatomy and artifact failures
- moderation or refusal
- whether the product silently changes the prompt
- all attempts, including failures
The hard part is scoring identity consistency without hiding the judgment inside another black box.
A vision model can help produce candidate scores, but the benchmark still needs a published rubric, calibration examples, and periodic human review. Otherwise the evaluator simply replaces an opaque product claim with an opaque benchmark claim.
5. Cost and failure handling
These products use different currencies, retry rules, queues, and subscription limits. Normalizing the marketing price alone is not enough.
I want to report both the advertised price and the observed cost of completing a fixed test scenario. Failed generations, retries, and timeouts stay in the denominator.
Separate the runner from the grader
The testing agent should not grade its own work.
My current design separates:
- a scenario runner that sends prompts and captures raw artifacts
- a grader that receives the rubric and those artifacts
- an audit layer that preserves prompts, timestamps, screenshots, model/version notes, and scoring explanations
That separation does not eliminate bias, but it makes errors easier to inspect. If the score looks wrong, a reader can trace it back to a particular artifact and rubric rule.
What should be public
A benchmark is not reproducible if readers only see a leaderboard.
For each published result, I plan to expose:
- scenario version
- test date
- product and paid plan
- prompt sequence
- transcript excerpts with privacy-safe redactions
- screenshots and media samples
- number of runs
- success and failure counts
- cost and latency
- scoring rubric
- known limitations
The goal is not to produce a magical objective score. The goal is to make every important judgment inspectable.
The two questions I have not solved
First, how long should a memory test run before "long-term" means anything? Four sessions may be practical, but practical is not the same as valid.
Second, what is the least misleading way to score identity consistency across images and video? Embedding similarity is tempting, but it can miss the exact visual drift a human notices immediately.
If you have built evaluation systems for stateful agents, multimodal models, or long-running conversations, I would value specific criticism of the protocol.
The current manual methodology and paid-test standard are here: https://nofilterreview.com/how-we-test/
Disclosure: I used an AI writing tool to help organize this article. The paid tests, product observations, methodology decisions, and final edits are mine.
Top comments (0)