A Deep Q-Network That Plays Roulette: What Happens When You Train RL on Pure Chance
Why train a DQN on a mathematically hostile environment
European roulette carries a house edge of 2.7% on every bet. No amount of pattern recognition, trend following, or "system" playing changes the fact that the expected value of every spin is negative. It is, in the most literal sense, a stochastic process designed to drain capital over time.
So why build a deep Q-network to play it?
Because this is exactly what makes it a revealing reinforcement learning problem. When you train an agent in an environment where the optimal long-term strategy is simply "don't play," you get to watch a learning algorithm that desperately wants to find signal confront a data-generating process that contains none. The training dynamics reveal how DQNs behave at the boundary of detectability. That has implications far beyond the casino.
This article walks through FAIRS (Fabulous Automated Intelligent Roulette System), an open-source research application built to explore that tension. The full implementation is MIT-licensed on GitHub.
The problem: RL in a zero-signal environment
The classic DQN breakthrough (Mnih et al., 2015) demonstrated human-level performance on 49 Atari 2600 games. Those environments share a critical property: they contain genuine signal. Pixels correlate with game state, actions produce predictable outcomes, and a consistent policy can achieve superhuman scores.
Roulette shares none of these properties. Each spin is independent and identically distributed. The wheel has no memory. Past outcomes do not inform future ones. The state transition function is:
$$P(s_{t+1} \mid s_t, a_t) = P(s_{t+1})$$
That is, the next state is completely independent of both the current state and the agent's action. The Q-function, which in standard RL represents the expected return of taking action $a$ in state $s$, degenerates to its immediate-reward term plus an action-independent constant:
$$Q(s, a) = \mathbb{E}[R \mid a] + C$$
where $C$ is the discounted expected future reward, identical for every action because future spins are independent of both the current state and the current action. The constant does not change how actions rank, so $\arg\max_a Q(s, a) = \arg\max_a \mathbb{E}[R \mid a]$. The state $s$ still carries no predictive information; the only thing that distinguishes actions is the immediate expected value, which for any roulette bet on a European wheel is $-0.027$ per unit wagered.
This makes roulette a useful null-case benchmark. If an RL algorithm learns something, that something is either an artifact of variance, overfitting to noise, or a bug. Observing how it fails tells us about the algorithm's inductive biases.
The DQN algorithm: a refresher
The DQN (Deep Q-Network) algorithm approximates the optimal action-value function $Q^*(s, a)$ using a neural network. The standard temporal-difference update is:
$$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$$
In practice, the network is trained to minimize the mean squared Bellman error:
$$\mathcal{L}(\theta) = \mathbb{E}{(s, a, r, s') \sim \mathcal{D}} \left[ \left( r + \gamma \max{a'} Q(s', a'; \theta^-) - Q(s, a; \theta) \right)^2 \right]$$
Where $\theta$ are the online network parameters and $\theta^-$ are target network parameters that are periodically copied from $\theta$ to stabilise training. The transitions $(s, a, r, s')$ are sampled uniformly from a replay buffer $\mathcal{D}$, breaking temporal correlations (Mnih et al., 2015).
FAIRS extends this with the Double DQN modification (Van Hasselt et al., 2016), which decouples action selection from action evaluation to reduce overestimation bias. The target becomes:
$$y = r + \gamma Q(s', \arg\max_{a'} Q(s', a'; \theta); \theta^-)$$
This is visible in the source code at app/server/learning/training/agents.py:
next_action_selection = model.predict(...)
best_next_actions = np.argmax(next_action_selection, axis=1)
q_futures_target = target_model.predict(...)
q_future_selected = q_futures_target[np.arange(batch_size), best_next_actions]
The online model selects the best action; the target model evaluates it.
FAIRS architecture: a dual-input Q-network
The FAIRS implementation diverges from the vanilla DQN in several design choices worth spelling out.
47-action space
Unlike the common 11-action mapping found in simplified roulette simulations, FAIRS exposes 47 actions: 37 straight number bets (0-36), 9 outside bets (Red, Black, Odd, Even, Low, High, and three Dozens), and a Pass action, so 37 + 9 + 1 = 47. This is a deliberately high-dimensional action space, and most of those 47 are strictly dominated by even-money bets. Watching which actions the agent settles on is therefore more informative than it would be with a small action set.
Dual-input network with learned embeddings
The Q-network (app/server/learning/models/qnet.py) takes two inputs:
-
Timeseries: the last 64 roulette outcomes as integer tokens, fed through a
RouletteEmbeddinglayer that learns distributed representations of each number - Gain: the current capital-to-initial-capital ratio as a scalar context signal
These pathways are merged via an AddNorm layer (a residual connection with layer normalisation), then passed through a QScoreNet head that produces Q-values for all 47 actions. The network uses BatchNormDense layers with ReLU activations, 30% dropout for regularisation, and the AdamW optimiser (Loshchilov & Hutter, 2019).
The FAIRS training workspace. Dataset management on the left, checkpoint panel on the right, and the live training monitor charting rewards and loss in real time.
Reward scaling
Raw roulette rewards range from -10 (loss) to +350 (straight-up win). The environment scales these to $[-1, 1]$ using an asymmetric transform:
negative rewards: (reward + max_bet) / max_bet - 1 → maps to [-1, 0)
positive rewards: reward / (max_bet * 35) → maps to (0, 1]
This prevents the rare +350 reward from dominating gradient updates while preserving the relative ranking of outcomes.
Experience replay with periodic target updates
The agent stores transitions in a deque buffer of configurable size (default 10,000). Training begins once the buffer exceeds the replay batch size (default 1,000). Every update_frequency steps (default 10), the target network weights are synced to the online network:
target_model.set_weights(model.get_weights())
Training dynamics: what the agent actually learns
Phase 1: Uniform exploration
At the start of training, epsilon is 0.75. The agent selects actions nearly at random, exploring all 47 actions roughly uniformly. The replay buffer fills with transitions from all bet types. Loss values are high and unstable because the network has no predictive structure to latch onto.
Phase 2: Convergence to even-money bets
As epsilon decays (default rate: 0.995 per step), the agent begins to follow its learned policy. The key finding: the Q-network converges to a strong preference for even-money bets (Red, Black, Odd, Even) regardless of the state input. This is purely statistical. The network has discovered that these actions minimise variance.
The action distribution shift is the clearest signal that learning has occurred. Early in training, the 47 actions are roughly equiprobable. By the time epsilon reaches 0.1, the even-money bets account for 60-70% of action selections, while straight number bets are nearly abandoned.
This is not the network "understanding" roulette. It is the network learning that the expected reward of a straight bet is -0.027 per unit but with catastrophic variance, while even-money bets deliver the same expected value with much smaller variance. The Q-network has independently discovered the variance-minimising policy within the constraints of its function approximation.
Phase 3: The loss never converges
Unlike supervised learning, where loss typically decreases toward zero, DQN loss in a stochastic environment oscillates persistently. The root-mean-square error hovers around 0.5-1.0 without trending downward.
This is the "moving target" problem in a purely stochastic setting. The target Q-values depend on the network's own predictions. As the network updates, the targets shift. In a non-stationary environment where the optimal Q-values are themselves noisy estimates of a negative expectation, the network never settles.
Persistent non-convergence of the TD error, together with well-behaved action distributions, is a diagnostic signature. It tells you the environment has no exploitable structure. Any apparent convergence in loss on a real-world RL problem should be examined critically: are you learning genuine structure, or just memorising variance?
Capital evolution
Over repeated episodes, the agent's capital follows a random walk with negative drift. Even with the "optimal" policy of exclusively playing even-money bets, expected loss is approximately 2.7% of total wager per spin. The variance can mask this edge for hundreds or even thousands of spins, creating the illusion of a winning strategy before the mathematics reasserts itself.
Here is the core point: in a zero-signal environment, the best any agent can do is minimise the rate at which it loses money. The agent that converges to even-money betting has "solved" roulette in the only way that is possible for it, by losing the least.
Implementation nuances: where the code diverges from the textbook
A few design choices in the FAIRS codebase illustrate the gap between textbook RL and a working implementation.
Gamma = 0.5 (myopic discounting). Standard DQN implementations use Gamma = 0.95–0.99, allowing the agent to reason about long-term consequences. FAIRS uses 0.5 by default, which gives the standard effective horizon of $1/(1-\gamma)=2$ steps (the discount weight $\gamma^t$ only falls below 1% after about 7 steps, a more generous and nonstandard reading). For roulette, where the data-generating process is i.i.d., this is defensible; there are no long-term consequences to reason about because each spin is independent. A higher gamma would simply accumulate more noise. But it also means the agent cannot learn temporal patterns even if they existed, which is a deliberate constraint that simplifies interpretation.
Reward scaling design. The asymmetric scaling function maps losses to $[-1, 0)$ and wins to $(0, 1]$, with straight-up wins (35:1) scaled to 1.0 and even-money wins scaled near 0.03. This creates an inherent bias: the network sees straight-up bets as having higher scaled expected value than their true mathematics warrants, because the rare +350 reward is compressed into the same $[0,1]$ range as smaller wins. This is a deliberate tradeoff to prevent the extreme variance of straight-up bets from dominating gradient updates, but it subtly warps the action-value landscape.
Frequent target network updates (every 10 steps). Vanilla DQN synchronises the target network every 5,000–10,000 steps (Mnih et al., 2015). FAIRS syncs every 10 steps. This is so frequent that the stabilising benefit of a separate target network is largely negated. The motivation is practical: the agent reaches its training horizon in ~2,000 steps per episode, so a 5,000-step sync interval would never fire during a single training run. The net effect is that FAIRS behaves more like standard Q-learning with a slowly-following target than true Double DQN.
Hierarchical dual-agent architecture. The dynamic betting mode introduces a second DQN agent (the StrategyAgent) that selects between 5 betting strategies (Martingale, Reverse Martingale, D'Alembert, Fibonacci, Keep). Both agents receive the same reward signal simultaneously, creating a non-stationary multi-agent learning problem: each agent's policy shifts the reward landscape the other observes. This is an advanced design pattern that goes beyond most DQN implementations, but its effectiveness in a zero-signal environment is questionable. Both agents are chasing noise.
These observations are not criticisms of FAIRS. It is a research tool, not a production trading system. They are documented here because understanding how an implementation diverges from the textbook is often where the real learning happens.
Implications for applied RL
The FAIRS experiment produces takeaways that transfer to real-world RL applications:
1. Low-signal environments require careful validation. If your agent's reward curve looks like the FAIRS training trace, with high variance, no trend, and action distributions converging to something that looks reasonable, you may be learning noise. Run a permutation test: shuffle the transitions and retrain. If the "policy" survives, it is an artifact.
2. Action distribution analysis is more informative than reward curves. In FAIRS, the reward curve is nearly unreadable. It bounces between positive and negative values for hundreds of episodes. The action distribution, by contrast, tells a clear story. When evaluating RL agents, watch what they choose, not just what they earn.
3. The Double DQN modification matters in stochastic environments. Van Hasselt et al. (2016) showed that DQN overestimates Q-values, particularly in noisy environments. FAIRS uses action selection from the online network with evaluation from the target network, which mitigates this. The difference is modest in a purely random environment (overestimation is bounded by the action space size) but becomes critical in environments with a real signal buried in noise.
Running the experiment yourself
FAIRS is designed to be accessible. The project runs as a local web application with a FastAPI backend and React frontend. The launcher script handles dependency installation:
.\start_on_windows.ps1
The training page lets you upload or generate roulette datasets, configure hyperparameters (episodes, learning rate, epsilon schedule, network architecture), and watch the training metrics live. The inference workspace lets you load trained checkpoints and evaluate policies.
What FAIRS is and is not
FAIRS is a research tool. It provides a clean, instrumented environment for studying how deep Q-networks behave in purely stochastic settings. The training monitors, checkpoint system, and inference workspace are designed to make it easy to run experiments, vary hyperparameters, and observe the results.
FAIRS is not a winning roulette system. The house edge is real. No deep learning architecture, no amount of training data, and no exploration strategy can overcome the mathematics of a negative-expectation game. The project's value lies in what the training process reveals about the algorithm.
FAIRS is open source (MIT). The entire codebase is at github.com/CTCycle/FAIRS-Roulette-Player. You can fork it, extend it with new agent architectures (PPO, SAC, curiosity-driven exploration), add roulette variants, or adapt the environment to other stochastic dynamics.
Closing
The roulette wheel is a useful adversary for RL research precisely because it is not trying to be beaten. It offers no pattern, adapts no strategy, and reveals no structure. Training a DQN against it is a reminder that not every problem is solvable by adding more layers. Recognising the absence of signal is itself a form of learning.
The full source code, including unit tests, end-to-end tests, and a pre-configured development environment, is available on GitHub. Contributions, experiments, and open issues are welcome.
References
- Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.
- Van Hasselt, H. (2010). Double Q-learning. Advances in Neural Information Processing Systems 23 (NeurIPS), 2613-2621. The original Double Q-learning paper, which used a roulette game as an evaluation environment.
- Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).
- Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. International Conference on Learning Representations (ICLR).
- FAIRS source code:
app/server/learning/(MIT License).
Built by CTCycle. ML researcher and data scientist working on clinical AI, RL agents, and scientific computing tools. Find me at github.com/CTCycle and on Dev.to at @ctcycle.
Top comments (1)
Roulette is a clean null benchmark because the policy can improve its variance profile while expected value stays pinned. That is a useful sanity check for RL papers too. If the learned action mix survives shuffled transitions, the model probably learned your reward scaling more than the world.