Simulating 50,000 e-commerce sessions with multi-agent RL: what worked, what didn't, and why I shipped the failure

Techmerch — Sat, 25 Jul 2026 10:02:42 +0000

I wanted to see if a learned policy — not scripted behavior — could produce realistic e-commerce shopper sessions: browsing, getting tempted, abandoning carts, and influencing other shoppers through a social graph. This is FunnelForge: 50 RL agents trained on real review data, simulating a shopping population end to end.

The problem

Most shopper simulations are scripted or rule-based. I wanted agents that learned to behave like real shoppers, including the realistic parts — like abandoning carts most of the time, not just converting on demand.

Architecture

Two planes, deliberately separated:

Data plane (produces behavior):

A min-heap event scheduler instead of fixed timesteps — agents act on their own continuous-time schedule rather than lockstep ticks. More realistic, more complex than a tick loop.
An Erdős–Rényi social graph with BFS influence propagation (1/2^depth decay) — one agent's purchase can nudge its neighbors, decaying with graph distance.
A topological funnel gate — agents move through browse → product_detail → cart → checkout, gated so they can't skip steps.
All three compiled to C++ via pybind11, used by default, with an automatic pure-Python fallback if the compiled modules aren't present.

Control plane (moves + observes):

Every action publishes to Redis and Kafka, streamed over a FastAPI WebSocket to a live dashboard.

Training

Behavioral cloning on real e-commerce review data first, then PPO fine-tuning on top. Pure RL from scratch was impractical here — sparse reward over a large discrete action space (which product, out of thousands) doesn't converge well without a good prior. BC gives the policy a realistic starting point; PPO refines it.

Validation — and the honest part

The simulation is stochastic over ~50 agents, so I validate against a seeded, reproducible batch run (FF_SEED=42), not the live dashboard feed:

Metric	Simulated	Reference	Verdict
Session length (items)	6.8 ± 6.1	real 8.16 ± 7.08	✅ PASS
Conversion rate	10%	2–20% (industry)	✅ PASS
Cart abandonment	90%	60–95% (industry)	✅ PASS
Social influence (top-5)	0.039	0.20–0.40	❌ FAIL

Three of four metrics land inside real industry ranges. The fourth — social-influence concentration — doesn't, and I want to talk about that one specifically, because I think how you handle a failing metric matters more than getting everything to pass.

I traced the cause: a fallback path in the product catalog was thinning the distribution the metric depends on, weakening the measured contagion effect. Instead of tuning parameters until the number looked right, I documented the diagnosis and left the actual fix as an open problem — with one constraint: fixing it can't regress the other three passing metrics. That turned out to be a much more interesting problem than the number itself; the easy "fix" (crank up the social-boost parameter) breaks conversion and abandonment in the process.

What I'd check earlier next time

Two things surfaced late that I'd catch sooner now:

A compiled component silently unused. The C++ pybind11 modules existed and compiled fine — but the live simulation was importing the Python implementation instead, the whole time. Nothing crashed, nothing warned. Only caught it while packaging, by actually testing the runtime path end to end instead of trusting that "it builds" meant "it runs correctly."
Two code paths, two different numbers. I had a live-dashboard path and a separate batch-validation path, and they'd drifted apart — different numbers, and the validator was quietly broken against an API change elsewhere in the codebase. Took real effort to find one canonical, reproducible number and make sure every doc cited the same one.

Both are the same underlying lesson: verify the actual runtime behavior, not just that the code compiles or the demo looks right.

Stack

Python, PyTorch (BC + PPO), C++17/pybind11, Redis, Kafka, FastAPI, React/D3.

Try it

Code + full README: github.com/ojas4414/FunnelForge

Free to run and study — source-available under PolyForm Perimeter. docker compose up gets you the whole stack, live dashboard included. There's also a paid documentation bundle (architecture deep-dive, guided walkthrough, graded exercises, interview prep) for anyone who wants a more structured path through it — linked in the README.

Curious what people think of the min-heap-over-fixed-timesteps call, or whether Erdős–Rényi was the wrong social graph model to start with. Happy to be told either was a bad call.

DEV Community: Techmerch