A step by step way to test whether a simulated user acts like a real one, based on the UserArena benchmark of 64,729 real shopping sessions.
A note on how this was written: an AI writing assistant drafted it from the sources listed at the end, and every claim was checked against those sources before it went out.
At a glance
- Simulated users are used to test shopping agents, but a simulator can look right on one step and drift badly over a long session.
- UserArena checks a simulator at five levels, from valid output up to real outcomes, and compares it with a real production A/B test.
- Below is how I'd use those five levels as a checklist for any simulator, including my own.
The question I kept asking
One line in a new paper stopped me: it checks simulated shoppers against the result of a real, controlled production A/B test. This one asks whether the whole session looks right, and whether the simulator would have picked the same winner as the real test.
That matters if you build or buy a shopping agent or a recommender. A simulated user is cheap to run. If it acts like real people, you can try many ideas before you put any of them in front of a real customer. If it doesn't, you can tune your agent to please a fake person and learn nothing true.
What UserArena is, in plain words
UserArena is a benchmark posted on Hugging Face Papers on Oct 22, and it had 46 upvotes when I read it. It is built from real e-commerce data: 64,729 shopping sessions from 11,010 users, covering 566,102 items and 13 kinds of actions.
It has three parts. First, the real sessions, long and step by step. Second, an interactive environment, so a simulated user can be dropped into a shopping session and asked what it would do next. Third, a way to check the simulator offline against the results of a controlled A/B test that ran in production.
The paper also offers a training method called On-Policy Trajectory Distillation, or OPTD. In short: the simulator writes its own sessions, and a teacher model that knows the user's goal for the whole session gives it detailed feedback on each part of what it wrote. In the paper's tests, OPTD scored the best overall fidelity of the simulators it compared, and came closest to the effect seen in production.
I care less about OPTD here than about the checking. The five levels work as a checklist for any simulated user, not only theirs.
The five levels as a step by step check
Step 1: Is the output even valid? Before anything else, check that the simulator answers in a form the environment can use: a real action, a real item, nothing made up. A bad result looks like actions outside the allowed set (UserArena has 13 action types), or items that don't exist in the catalog. If this step fails, nothing after it means much.
Step 2: Are single decisions sensible? Give the simulator one moment from a real session and ask for its next move. Compare it with what the real person did. A bad result is a simulator that clicks, adds to cart or buys at rates that look nothing like real people at the same moment. This is where most checks stop. I think it is the easy part.
Step 3: Does the session hold together over time? Let the simulator run a long session and compare the whole path with real paths. A bad result is drift: a shopper who starts looking for shoes and wanders off with no reason, or who never stops browsing, or who buys at once every time. A simulator can pass Step 2 and still fail here, because small errors pile up over many steps.
Step 4: Does it act like this person, not like an average person? Real users differ. Give the simulator two different shoppers' histories and see whether it acts differently for each. A bad result is a simulator that gives everyone the same taste. That kind of simulator will hide the very gains that personal recommendations are meant to bring.
Step 5: Does it get the outcome right? Look at what happens at the end of sessions: what was bought, or whether anything was. A bad result is outcomes far from real ones, even when the steps along the way looked fine.
Step 6: Would it have picked the same winner? This is the part that stood out to me. Take two policies that were already compared in a real A/B test. Run both against the simulator. Did the simulator rank them the same way, and with a similar gap? A bad result is a simulator that calls the losing version the winner. If that happens, every decision you make with it is at risk.
The mistakes I'd watch for
Checking only the next step. A high score on single decisions feels good, but sessions are long. The paper's whole point is that you need to look at the full path too.
Judging the simulator with itself. If the same model writes the sessions and grades them, it can agree with its own mistakes. Real sessions, or a real test result, are the thing to compare against.
Skipping Step 6 because you have no A/B test. I get it: not everyone has one. But one past test with a known result is worth a lot here. Without it, you know the simulator looks real, not that it leads you to the right choice.
Treating one benchmark as the truth about your users. Your shoppers may act differently. I'd use it to learn how to check, then run the same checks on my own data.
Check it yourself
Pull a sample of real sessions from your own logs, full sessions from start to end, not single clicks.
Write down your list of allowed actions and your item list. Run your simulator and count how many outputs fall outside them. Anything above zero needs a fix first.
For a set of real moments, compare the simulator's next action with the real one. Keep the score, but don't stop there.
Let the simulator run whole sessions. Put a few next to real ones and read them side by side. Look for drift, endless browsing or instant buying.
Pick two users with very different histories. Check that the simulator treats them differently.
If you have a past A/B test, replay both versions with the simulator. Write down which one it picks and by how much. Compare with the real result.
If you want to see how the paper sets this up, the code and data are on GitHub (link below).
What I'm unsure about
The paper reports that OPTD did best among the simulators it tested, under its own evaluation rules. I'd want to see how much the result depends on the data coming from one platform, and how many A/B tests the offline check covers. The summary talks about a controlled production test, and one test is a thin base for a strong claim. That doesn't make the idea weak. It means I'd treat Step 6 as a first sign, not proof.
Still, the five levels are the most useful part for me. They turn a vague question (does this fake user act real?) into steps you can run and fail.
Your turn
If you test agents with simulated users, which level do you check today, and have you ever compared your simulator's pick with a real A/B test result?
Sources
Top comments (0)