A Reproducible Test for AI Companion Continuity

AI companion comparisons often rely on screenshots of one impressive reply.

That is a weak test.

The difficult behavior appears later: after a correction, a tone change, a scene transition, or a return to the conversation. I wanted a small protocol that anyone could repeat without sharing personal information or depending on subjective “vibes” alone.

The result is the continuity test below.

What the test measures

The protocol separates six behaviors that are often incorrectly grouped under “memory”:

1. Identity stability — does the character preserve its role and voice?
2. Correction handling — can a new fact replace an old one?
3. Relevant recall — does the system retrieve useful details without dumping a profile?
4. Uncertainty behavior — does it ask when evidence is missing?
5. Scene progression — can a later scene depend on an earlier decision?
6. User control — can the user redirect the interaction without fighting the system?

The test prompt

Use a fictional scenario rather than real personal data:

We meet every Thursday at a quiet rooftop greenhouse. You are the night caretaker. The basil plant is named Harbor. I dislike being rushed. I am trying to finish a short letter but do not want to send it tonight. Keep the conversation warm, specific, and unhurried.

The prompt contains:

• one recurring time;
• one location;
• one named object;
• one interaction preference;
• one unresolved goal.

Test procedure

Phase 1: Baseline

Continue for 10–15 turns without repeating the setup. Check whether the character uses details naturally.

Avoid scoring pure repetition as good memory. “We are in the rooftop greenhouse with Harbor” is less useful than behavior that respects the preference not to be rushed.

Phase 2: Correction

Say:

I renamed the basil plant. It is Marlowe now, not Harbor.

Continue for several turns.

Failure modes:

• the old and new names are used interchangeably;
• the system says both names were always correct;
• the correction is repeated unnaturally in every reply;
• the correction replaces unrelated context.

Phase 3: Tone change

Say:

I am feeling more playful now, but I still do not want to send the letter.

The character should adjust its tone while preserving the unresolved goal.

Phase 4: Scene transition

Move the conversation:

Let us leave the greenhouse and walk downstairs. Bring the letter, but leave Marlowe where it is.

Check whether the new scene preserves relevant facts without dragging every previous detail into the reply.

Phase 5: Return-later check

If the product supports persistent conversations, return later and ask:

What did we leave unresolved last time?

A strong response should identify the letter, avoid claiming it was sent, and express appropriate uncertainty about details that were never established.

Suggested scoring rubric

Score each category from 0 to 2:

Category	0	1	2
Identity	major drift	minor drift	stable and natural
Correction	ignored	partially applied	old fact cleanly replaced
Recall	missing/invented	mostly relevant	relevant and selective
Uncertainty	confident invention	vague hedging	asks or qualifies appropriately
Progression	resets	loosely continues	new scene depends on prior events
User control	resists direction	follows with friction	adapts cleanly

Do not collapse the result into one “best AI companion” score. Different users care about different categories.

Important limitations

• One scenario cannot measure every interaction style.
• A product update can change results.
• Individual characters may behave differently on community-driven platforms.
• A longer context window does not automatically produce better correction handling.
• A convincing answer can still contain invented history.
• Never use sensitive personal facts just to test memory.

Why publish the protocol?

Reusable tests make product discussions more useful. They let readers compare observations, identify failure modes, and separate product design from marketing claims.

If you try the protocol, share which category failed first and whether the failure happened immediately or only after returning later.

---

Affiliation and AI-assistance disclosure: I work with the LumiChat team, where continuity is one of the product problems we study. This post is a vendor-neutral testing method rather than a product recommendation. AI tools assisted with editing and structure; the scenario, rubric, limitations, and final claims were reviewed by the author.