DEV Community: Thomas Virdis

Benchmarking Tokenizers Without the Guesswork

Thomas Virdis — Mon, 31 Aug 2026 12:41:05 +0000

Benchmarking Tokenizers Without the Guesswork

What TKBEN measures, why tokenizer evaluation is deceptively hard, and how reproducible evidence beats a one-off script.

The quiet bottleneck under every model

Every language model has a tokenizer bolted to its front door. Before the model sees a single word, the tokenizer chops the text into units, maps them to numbers, and decides, in effect, what the model is allowed to think about. That choice is not cosmetic. Two tokenizers over the same corpus can differ in vocabulary size, in how many pieces a word gets split into, in how fast they run, in how much memory they use, and in whether arbitrary text can even be reconstructed at the end of the round trip. Each of those differences changes real-world behavior: inference cost, latency, how gracefully a model handles a language it was never trained on, and how trustworthy the decoded output is.

Yet tokenizers are usually the least-examined part of a model pipeline. Most benchmarks of language models treat the tokenizer as a fixed input, something to be accepted rather than evaluated, and most teams pick one the way they pick a default: it came with the model, so it will do.

TKBEN (TKBEN-tokenizers-benchmarker) is built on the opposite assumption. A tokenizer is a choice, and it should be examined the way you would examine any performance-critical component: on repeatable, inspectable data, not on a gut feeling.

Why the naive benchmark is not enough

The obvious way to compare tokenizers is to run a quick script: load one, feed it some text, print a number, move on. That approach has a weakness that shows up only later. The number is gone. The person who ran it cannot reopen the evidence, cannot re-run the comparison against a different dataset, cannot show how the figure was produced, and cannot tell you whether it reflects the test set that actually matters.

Tokenizers are sensitive to what you feed them. A tokenizer that looks efficient on English news text may fragment a multilingual dataset badly. A vocabulary that seems rich can still leave you with a high unknown-token rate on a domain you cared about. A fast encode time tells you nothing about latency at the tail, or memory under load. None of these come through in a single printed stat.

TKBEN treats this properly. It is a local web application that keeps downloaded tokenizer assets, validation reports, and benchmark results in its own workspace, so a later session can reopen and inspect the same evidence. A benchmark becomes a saved report with provenance, re-openable weeks later, rather than a printed number that is gone the moment the script exits.

The concept: validation before comparison

The heart of TKBEN is a separation that most ad-hoc benchmarking collapses: you cannot compare tokenizers until you first understand the text you are going to run over them.

Dataset validation comes first. You load a Hugging Face dataset, a custom identifier, or a local CSV, XLS, or XLSX file, and run it through the validation pipeline. The result is a saved report covering corpus scale and structure (document counts, lengths, and how spread out they are), lexical diversity and frequency (type-token ratio, moving-average TTR, word distributions), word and character signals, document quality signals, structural regularity, and compression and redundancy. In plain terms, TKBEN tells you what kind of text you are about to benchmark, so you are not comparing tokenizers on data you do not actually understand.

Then, and only then, do you bring in the tokenizers. You scan or add tokenizer identifiers, download the assets you need, upload a custom tokenizer.json, and inspect each report's vocabulary statistics and token preview before trusting it in a comparison. Once you have a validated dataset and a set of tokenizers you have actually looked at, you can build a cross-benchmark run.

What the comparison actually measures

The benchmark result is a dashboard of widgets, each one a separate view of a tokenizer's behavior on the validated dataset. The metrics fall into six groups, and each answers a different question:

Metadata. Vocabulary size, added-token count, special-token share. The basics of what a tokenizer carries with it.
Efficiency. Tokens per second, characters per second, bytes per second, and the time split across encoding, dataset streaming, post-processing, and the complete run from start to finish. This is the throughput story.
Latency. Median, p95, and p99 encode latency, plus the raw latency distribution. The tail, where interactive models actually feel slow, is not hidden behind an average.
Fidelity. Whether text survives the trip. Exact token-ID round trip, normalized text round trip, unknown-token rate, byte-fallback rate, and how much character coverage the vocabulary provides. A fast tokenizer that cannot faithfully round-trip its input is not fast, it is lossy.
Fragmentation. Tokens per character, characters per token, pieces per word. A measure of how aggressively text is split, and how that shifts with word length.
Resources. Peak RSS and memory delta. What the tokenizer costs in resident memory while it works.

You can switch between bar, horizontal-bar, box-plot, histogram, dot-whisker, and grouped-bar views, reorder the widgets to match how you think about the problem, and export the whole comparison as a PDF. The dashboard below compares a GPT-2 style tokenizer against RoBERTa-base over the same C4 sample, side by side, with the numbers that actually differ made visible.

The value of a dashboard like this is not the top-of-the-fold number. It is that a memory-hungry tokenizer, a slow tail, and a fracture-prone vocabulary can all live in the same view, so a team sees the whole trade-off instead of a single ranking.

A report you can trust a month later

TKBEN's core design decision is that a benchmark is a saved report, not an event. Each validation and each cross-benchmark run persists in the workspace, with a page to reopen it. Combined with the PDF export, that turns a fragile one-off script into an auditable workflow: you can show exactly which dataset, which tokenizer versions, and which metrics produced a given comparison, and you can reopen that evidence days or weeks later without re-running anything.

That matters whether you are choosing a tokenizer for a new model, auditing the efficiency of an existing pipeline, or documenting a decision for a team. The report becomes the artifact you argue from, and the artifact still exists the next time somebody asks how the decision was made.

Tokenizer assets you can really inspect

Under the hood, a tokenizer report shows what the download actually gave you: model metadata, a vocabulary count, a token-length distribution, and a paginated preview of the tokens themselves. This is a small thing that saves real confusion. Reported and actual vocabulary sizes can drift, especially with added tokens, and downloading a model you think is one thing and inspecting another is how wrong comparisons start. TKBEN surfaces the actual token list and lets you confirm that what was downloaded is what you intended to evaluate.

Local, and honest about it

Like the rest of the CTCycle portfolio, TKBEN runs locally. The browser is the interface; a local FastAPI service does the processing and stores the reports. Config stays in a machine-specific, never-committed settings file, and the embedded SQLite database is managed through explicit migrations. This keeps your datasets and analysis on your machine, which matters when the data you are benchmarking is proprietary or simply not something you want to send to a cloud service. The local setting is a completeness of control, not a limitation.

The evidence stays

Most tooling treats a tokenizer as a fixed input and a benchmark as a number you print and forget. TKBEN does the opposite. It validates the text before it benchmarks, measures the trade-offs that actually differ between tokenizers, and persists the whole thing as evidence you can reopen later.

If a team wants to pick a tokenizer with something better than intuition, TKBEN is the workspace to do it in. It runs on Windows through a single launcher, or on macOS and Linux with a short manual setup, and the source is open under an MIT licence. The repository is the place to start: github.com/CTCycle/TKBEN-tokenizers-benchmarker.

Part of the CTCycle open-source portfolio. See the full open-source portfolio at ctcycle.github.io/CTCycle.

XREPORT: A Local-First Lab for Draft Radiology Reports

Thomas Virdis — Mon, 31 Aug 2026 12:41:03 +0000

XREPORT: A Local-First Lab for Draft Radiology Reports

What it is, why it exists, and the design choices that make it a cautious tool instead of an ambitious one.

The job that hides inside the scan

If you have ever watched a radiologist work a stack of chest X-rays, the first thing you notice is how much of the job is not diagnosis. A lot of it is dictation. The lungs, the cardiac silhouette, the mediastinum, the bones. Normal, unremarkable, no acute findings. The kind of report that makes up most of a working day and takes very little of a radiologist's brain to produce correctly, but still has to be written, in the right format, with the right words, every single time.

That is the problem XREPORT is aimed at, and the honest way to state it is also the narrow way. Doing radiology reporting well is exhausting, and fatigue is the last thing you want anywhere near a diagnostic decision. Automation cannot fix that fatigue, but it can carry some of the routine load so the human review happens on a less tired brain.

This is not a tool that reads a scan and tells you what is wrong. It is a tool that produces a structured first draft of the report, which a qualified clinician then reads, corrects, and signs. The difference between those two jobs is the whole design. XREPORT is built to be a laboratory for making drafts, with the review step kept firmly human.

Local first, because the data decides

The most important sentence in the whole project is in the repository's opening note: generated drafts are not clinically approved and require qualified independent review. The second most important is the privacy boundary. XREPORT is local-first, and for medical imaging that is not a packaging preference, it is the entire point.

X-ray images are patient data under legal and ethical constraints. A tool that sends chest scans to a cloud service is a non-starter for most hospitals regardless of how accurate the model is, because the institution cannot control where the data goes or who can look at it. XREPORT never phones home. Models are downloaded once, cached in a project-local resource directory, verified, and reused offline. Training, validation, and generation all run on the machine the user controls. For clinical settings this is not a feature that needs to be argued for; it is the price of entry.

That single decision shapes everything downstream. Because there is no cloud backend, there is also no silent model update, no surprise behavior change between Tuesday and Wednesday. What the user validated last month is what runs next month, unless they choose to change it.

Not one report engine, but a choice with provenance

XREPORT avoids locking the user into a single black box. It ships with a fixed catalogue of five pinned public Hugging Face report-generation models, and it also accepts locally trained checkpoints through its own training workflow. The pinned models are chosen because each is a known, verifiable starting point. The custom checkpoints exist because a research lab or a hospital with its own data should not be forced onto a generic model when it has spent months building one on a domain-specific set.

Every generated report carries its provenance with it. The output comes back as editable Findings and Impression fields with the model, provider, and revision metadata attached, so anyone reading the draft can see where the text came from. That traceability matters in a domain where the reader needs to know whether the suggestion was produced by the model the team validated or by something else. It is the difference between a reproducible artifact and an output you cannot explain a month later.

What the custom model actually is

The in-house training path treats report generation as an image-to-text problem, the same framing you would use for image captioning, but with medical language and a strict output shape. It is not a chat model asked nicely to describe an X-ray. It is a purpose-built encoder-decoder that maps an image to a token-by-token sequence of report language.

The image side starts with a BEiT image encoder, a vision transformer pretrained on image patches, which turns a 224 by 224 chest X-ray into a set of visual feature vectors. Those features are projected into the shared embedding space. On the text side, a positional embedding encodes the report tokens as they are produced, and a stack of transformer encoder and decoder layers learns to attend from the image to that sequence. The default shape is four encoder layers and four decoder layers with eight attention heads and a 256-dimensional embedding, though the sizes are configuration options rather than hard constants. A softmax classifier over the vocabulary produces the next token at each step.

Training uses a masked cross-entropy loss that only scores the report tokens, with a masked-accuracy metric tracking how often the model predicts the right next token at the right position. The optimizer is AdamW with an optional warmup learning-rate schedule. The important practical detail is the checkpoint lifecycle. A run can be paused, resumed from any saved state, and evaluated against held-out examples, so a team is never forced to finish a model in one sitting or trust a single untested run. The evaluation view shows loss and accuracy for a saved checkpoint before that model is allowed anywhere near real generation.

The five public models, and what they trade off

Because not everyone wants to train their own model, XREPORT also ships a pinned catalogue of five public chest imaging models, each with its capabilities spelled out in the config rather than assumed. The catalogue is deliberately small and explicitly pinned to exact revisions, so a model that works today still works six months from now.

At the lightweight end is CXRMate Multi TF, a 0.1B parameter model that is the recommended default. It is designed for multi-view chest X-rays and its output is structured, with separate Findings and Impression sections. On modest hardware it is usable locally, which makes it the sensible first choice for most runs. Its sibling CXRMate-ED is a similar small model that additionally accepts a clinical indication or history from the user and maps it into the model's context input. Same family, one richer input.

Further up the scale sit two larger, more demanding models. CheXOne, a 4B parameter vision-language model from Stanford, is capable not only of drafting reports but of reasoning and of output that can be localized back to regions of the image, so it is the entry point if grounding matters. CXRMate-2 is a 3B parameter specialist with structured output and clinical-context support, at the cost of much heavier local storage and memory. Both are far more capable than the lightweight pair, and both are far more demanding on hardware, which is exactly the trade-off a local-first tool has to surface rather than hide.

Rounding out the set is MedGemma 1.5 4B, a broader medical-imaging model that, unlike the chest specialists, is not validated only on radiographs. It produces a single raw report rather than separate structured sections, and it is the one model that also supports prior images as context. It is also gated: downloading it requires accepting Google's model terms and configuring a local token, which XREPORT notes up front rather than pretending the download is automatic.

The picture that emerges is not "here is the one best model." It is a ladder of choices from a small fast default up to heavyweight research models, with their parameter sizes, licence restrictions, hardware demands, and structured-versus-raw output all visible in one place. A small radiology group on a workstation and a research lab with a GPU server look at the same screen and pick different entries on the same list.

The workflow is the concept

XREPORT is a loop, not a single inference button. The loop is the point:

Dataset preparation. Import an image folder plus CSV or XLSX report metadata, review how images and reports match, inspect paired records, then clean, tokenize, split, and build training-ready datasets. MIMIC-CXR is supported as a starting validation dataset, alongside custom image-report pair formats. The review step matters, because a training set built without checking that each scan is paired with the right report is quietly teaching the model wrong associations.

Training. Configure CPU or GPU training, watch live metrics and logs, save checkpoints, resume, and evaluate. Model readiness and provenance warnings stay visible throughout.
Generation. Add up to sixteen study images, pick a generation profile, and submit a background job against the chosen model or checkpoint.
Review. The result returns as editable Findings and Impression text with its metadata attached, ready to be copied or exported for qualified review. The screen is built around the edit, not the acceptance. Nothing is published straight from the model; the draft is handed to a person.

Two parts of this deserve emphasis. One is that the user can generate from up to sixteen images in a single case, which matters because a real radiological study is rarely a single view. The other is that validation is a first-class step, not an afterthought. A tool that shows a confident report with no indication of whether the model behind it was ever validated is a tool that has quietly handed the judgment to the machine. XREPORT keeps the validation state in front of the user instead of hiding it.

Why the tool deliberately does less

The temptation in a project like this is to keep pushing toward the interesting part: the idea that a model might one day make a clinical call. XREPORT is explicitly built to stop well short of that line. The value it claims is a coherent first draft that flags what a reader should look for, which is most useful on the routine cases where the findings are unremarkable. For rare or critical findings, an automated draft is only as useful as the clinician's judgment in catching what the model missed. That is why the human-in-the-loop is not a concession but the designed center of the system, and why the output is labeled a research-use draft rather than a diagnosis.

Getting from local promise to usable software took real work on the delivery side too. On Windows the app runs through a single launcher that manages the portable runtimes and dependencies. A Tauri 2 desktop shell packages it as CPU and CUDA installers, so a user who needs the version built for their graphics setup has a normal installer rather than a pile of manual steps. The binaries are currently unsigned, and MSI installs are per-machine, so this is a tool that assumes a technically literate operator, which is honest about the audience.

The restless part under the surface

Underneath, the application is a FastAPI backend with an Angular 22 frontend, with settings kept local and machine-specific, never committed, and the SQLite database managed through explicit migrations. None of that is the story a reader needs to remember. The reason to mention the stack at all is that it was chosen to stay boring and auditable: a report you can trace, a schema you can reason about, dependencies you can pin. In a tool that produces medical text, boring and auditable is a feature.

The design stance

Most report-generation tools are pitched on how much they can automate. XREPORT is more honest than that, and I think that is its best quality. Its automation effort is aimed squarely at the routine load, it keeps the data local, it attaches provenance to every output, and it never pretends to own the diagnostic step. Whether you train a checkpoint or pick a pinned public model, you can see what produced the draft and how validated it was, and the final sign-off stays human.

If a radiologist wants to spend less of their day on unremarkable dictations and more on the cases that actually need them, that is the version of XREPORT that exists. The first run is a research-use tool, not a clinical product, and the source is open under an MIT licence, so any team that disagrees with a design decision can take the code and change it.

If that sounds like your problem, the repository is the place to start: github.com/CTCycle/XREPORT-radiological-reports-generator.

Part of the CTCycle open-source portfolio. Models and generated drafts are not clinically approved and require qualified independent review.

A Deep Q-Network That Plays Roulette: What Happens When You Train RL on Pure Chance

Thomas Virdis — Wed, 26 Aug 2026 15:43:52 +0000

A Deep Q-Network That Plays Roulette: What Happens When You Train RL on Pure Chance

Why train a DQN on a mathematically hostile environment

European roulette carries a house edge of 2.7% on every bet. No amount of pattern recognition, trend following, or "system" playing changes the fact that the expected value of every spin is negative. It is, in the most literal sense, a stochastic process designed to drain capital over time.

So why build a deep Q-network to play it?

Because this is exactly what makes it a revealing reinforcement learning problem. When you train an agent in an environment where the optimal long-term strategy is simply "don't play," you get to watch a learning algorithm that desperately wants to find signal confront a data-generating process that contains none. The training dynamics reveal how DQNs behave at the boundary of detectability. That has implications far beyond the casino.

This article walks through FAIRS (Fabulous Automated Intelligent Roulette System), an open-source research application built to explore that tension. The full implementation is MIT-licensed on GitHub.

The problem: RL in a zero-signal environment

The classic DQN breakthrough (Mnih et al., 2015) demonstrated human-level performance on 49 Atari 2600 games. Those environments share a critical property: they contain genuine signal. Pixels correlate with game state, actions produce predictable outcomes, and a consistent policy can achieve superhuman scores.

Roulette shares none of these properties. Each spin is independent and identically distributed. The wheel has no memory. Past outcomes do not inform future ones. The state transition function is:

$$P(s_{t+1} \mid s_t, a_t) = P(s_{t+1})$$

That is, the next state is completely independent of both the current state and the agent's action. The Q-function, which in standard RL represents the expected return of taking action $a$ in state $s$, degenerates to its immediate-reward term plus an action-independent constant:

$$Q(s, a) = \mathbb{E}[R \mid a] + C$$

where $C$ is the discounted expected future reward, identical for every action because future spins are independent of both the current state and the current action. The constant does not change how actions rank, so $\arg\max_a Q(s, a) = \arg\max_a \mathbb{E}[R \mid a]$. The state $s$ still carries no predictive information; the only thing that distinguishes actions is the immediate expected value, which for any roulette bet on a European wheel is $-0.027$ per unit wagered.

This makes roulette a useful null-case benchmark. If an RL algorithm learns something, that something is either an artifact of variance, overfitting to noise, or a bug. Observing how it fails tells us about the algorithm's inductive biases.

The DQN algorithm: a refresher

The DQN (Deep Q-Network) algorithm approximates the optimal action-value function $Q^*(s, a)$ using a neural network. The standard temporal-difference update is:

$$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$$

In practice, the network is trained to minimize the mean squared Bellman error:

$$\mathcal{L}(\theta) = \mathbb{E}{(s, a, r, s') \sim \mathcal{D}} \left[ \left( r + \gamma \max{a'} Q(s', a'; \theta^-) - Q(s, a; \theta) \right)^2 \right]$$

Where $\theta$ are the online network parameters and $\theta^-$ are target network parameters that are periodically copied from $\theta$ to stabilise training. The transitions $(s, a, r, s')$ are sampled uniformly from a replay buffer $\mathcal{D}$, breaking temporal correlations (Mnih et al., 2015).

FAIRS extends this with the Double DQN modification (Van Hasselt et al., 2016), which decouples action selection from action evaluation to reduce overestimation bias. The target becomes:

$$y = r + \gamma Q(s', \arg\max_{a'} Q(s', a'; \theta); \theta^-)$$

This is visible in the source code at app/server/learning/training/agents.py:

next_action_selection = model.predict(...)
best_next_actions = np.argmax(next_action_selection, axis=1)
q_futures_target = target_model.predict(...)
q_future_selected = q_futures_target[np.arange(batch_size), best_next_actions]

The online model selects the best action; the target model evaluates it.

FAIRS architecture: a dual-input Q-network

The FAIRS implementation diverges from the vanilla DQN in several design choices worth spelling out.

47-action space

Unlike the common 11-action mapping found in simplified roulette simulations, FAIRS exposes 47 actions: 37 straight number bets (0-36), 9 outside bets (Red, Black, Odd, Even, Low, High, and three Dozens), and a Pass action, so 37 + 9 + 1 = 47. This is a deliberately high-dimensional action space, and most of those 47 are strictly dominated by even-money bets. Watching which actions the agent settles on is therefore more informative than it would be with a small action set.

Dual-input network with learned embeddings

The Q-network (app/server/learning/models/qnet.py) takes two inputs:

Timeseries: the last 64 roulette outcomes as integer tokens, fed through a RouletteEmbedding layer that learns distributed representations of each number
Gain: the current capital-to-initial-capital ratio as a scalar context signal

These pathways are merged via an AddNorm layer (a residual connection with layer normalisation), then passed through a QScoreNet head that produces Q-values for all 47 actions. The network uses BatchNormDense layers with ReLU activations, 30% dropout for regularisation, and the AdamW optimiser (Loshchilov & Hutter, 2019).

The FAIRS training workspace. Dataset management on the left, checkpoint panel on the right, and the live training monitor charting rewards and loss in real time.

Reward scaling

Raw roulette rewards range from -10 (loss) to +350 (straight-up win). The environment scales these to $[-1, 1]$ using an asymmetric transform:

negative rewards: (reward + max_bet) / max_bet - 1   → maps to [-1, 0)
positive rewards: reward / (max_bet * 35)             → maps to (0, 1]

This prevents the rare +350 reward from dominating gradient updates while preserving the relative ranking of outcomes.

Experience replay with periodic target updates

The agent stores transitions in a deque buffer of configurable size (default 10,000). Training begins once the buffer exceeds the replay batch size (default 1,000). Every update_frequency steps (default 10), the target network weights are synced to the online network:

target_model.set_weights(model.get_weights())

Training dynamics: what the agent actually learns

Phase 1: Uniform exploration

At the start of training, epsilon is 0.75. The agent selects actions nearly at random, exploring all 47 actions roughly uniformly. The replay buffer fills with transitions from all bet types. Loss values are high and unstable because the network has no predictive structure to latch onto.

Phase 2: Convergence to even-money bets

As epsilon decays (default rate: 0.995 per step), the agent begins to follow its learned policy. The key finding: the Q-network converges to a strong preference for even-money bets (Red, Black, Odd, Even) regardless of the state input. This is purely statistical. The network has discovered that these actions minimise variance.

The action distribution shift is the clearest signal that learning has occurred. Early in training, the 47 actions are roughly equiprobable. By the time epsilon reaches 0.1, the even-money bets account for 60-70% of action selections, while straight number bets are nearly abandoned.

This is not the network "understanding" roulette. It is the network learning that the expected reward of a straight bet is -0.027 per unit but with catastrophic variance, while even-money bets deliver the same expected value with much smaller variance. The Q-network has independently discovered the variance-minimising policy within the constraints of its function approximation.

Phase 3: The loss never converges

Unlike supervised learning, where loss typically decreases toward zero, DQN loss in a stochastic environment oscillates persistently. The root-mean-square error hovers around 0.5-1.0 without trending downward.

This is the "moving target" problem in a purely stochastic setting. The target Q-values depend on the network's own predictions. As the network updates, the targets shift. In a non-stationary environment where the optimal Q-values are themselves noisy estimates of a negative expectation, the network never settles.

Persistent non-convergence of the TD error, together with well-behaved action distributions, is a diagnostic signature. It tells you the environment has no exploitable structure. Any apparent convergence in loss on a real-world RL problem should be examined critically: are you learning genuine structure, or just memorising variance?

Capital evolution

Over repeated episodes, the agent's capital follows a random walk with negative drift. Even with the "optimal" policy of exclusively playing even-money bets, expected loss is approximately 2.7% of total wager per spin. The variance can mask this edge for hundreds or even thousands of spins, creating the illusion of a winning strategy before the mathematics reasserts itself.

Here is the core point: in a zero-signal environment, the best any agent can do is minimise the rate at which it loses money. The agent that converges to even-money betting has "solved" roulette in the only way that is possible for it, by losing the least.

Implementation nuances: where the code diverges from the textbook

A few design choices in the FAIRS codebase illustrate the gap between textbook RL and a working implementation.

Gamma = 0.5 (myopic discounting). Standard DQN implementations use Gamma = 0.95–0.99, allowing the agent to reason about long-term consequences. FAIRS uses 0.5 by default, which gives the standard effective horizon of $1/(1-\gamma)=2$ steps (the discount weight $\gamma^t$ only falls below 1% after about 7 steps, a more generous and nonstandard reading). For roulette, where the data-generating process is i.i.d., this is defensible; there are no long-term consequences to reason about because each spin is independent. A higher gamma would simply accumulate more noise. But it also means the agent cannot learn temporal patterns even if they existed, which is a deliberate constraint that simplifies interpretation.

Reward scaling design. The asymmetric scaling function maps losses to $[-1, 0)$ and wins to $(0, 1]$, with straight-up wins (35:1) scaled to 1.0 and even-money wins scaled near 0.03. This creates an inherent bias: the network sees straight-up bets as having higher scaled expected value than their true mathematics warrants, because the rare +350 reward is compressed into the same $[0,1]$ range as smaller wins. This is a deliberate tradeoff to prevent the extreme variance of straight-up bets from dominating gradient updates, but it subtly warps the action-value landscape.

Frequent target network updates (every 10 steps). Vanilla DQN synchronises the target network every 5,000–10,000 steps (Mnih et al., 2015). FAIRS syncs every 10 steps. This is so frequent that the stabilising benefit of a separate target network is largely negated. The motivation is practical: the agent reaches its training horizon in ~2,000 steps per episode, so a 5,000-step sync interval would never fire during a single training run. The net effect is that FAIRS behaves more like standard Q-learning with a slowly-following target than true Double DQN.

Hierarchical dual-agent architecture. The dynamic betting mode introduces a second DQN agent (the StrategyAgent) that selects between 5 betting strategies (Martingale, Reverse Martingale, D'Alembert, Fibonacci, Keep). Both agents receive the same reward signal simultaneously, creating a non-stationary multi-agent learning problem: each agent's policy shifts the reward landscape the other observes. This is an advanced design pattern that goes beyond most DQN implementations, but its effectiveness in a zero-signal environment is questionable. Both agents are chasing noise.

These observations are not criticisms of FAIRS. It is a research tool, not a production trading system. They are documented here because understanding how an implementation diverges from the textbook is often where the real learning happens.

Implications for applied RL

The FAIRS experiment produces takeaways that transfer to real-world RL applications:

1. Low-signal environments require careful validation. If your agent's reward curve looks like the FAIRS training trace, with high variance, no trend, and action distributions converging to something that looks reasonable, you may be learning noise. Run a permutation test: shuffle the transitions and retrain. If the "policy" survives, it is an artifact.

2. Action distribution analysis is more informative than reward curves. In FAIRS, the reward curve is nearly unreadable. It bounces between positive and negative values for hundreds of episodes. The action distribution, by contrast, tells a clear story. When evaluating RL agents, watch what they choose, not just what they earn.

3. The Double DQN modification matters in stochastic environments. Van Hasselt et al. (2016) showed that DQN overestimates Q-values, particularly in noisy environments. FAIRS uses action selection from the online network with evaluation from the target network, which mitigates this. The difference is modest in a purely random environment (overestimation is bounded by the action space size) but becomes critical in environments with a real signal buried in noise.

Running the experiment yourself

FAIRS is designed to be accessible. The project runs as a local web application with a FastAPI backend and React frontend. The launcher script handles dependency installation:

.\start_on_windows.ps1

The training page lets you upload or generate roulette datasets, configure hyperparameters (episodes, learning rate, epsilon schedule, network architecture), and watch the training metrics live. The inference workspace lets you load trained checkpoints and evaluate policies.

What FAIRS is and is not

FAIRS is a research tool. It provides a clean, instrumented environment for studying how deep Q-networks behave in purely stochastic settings. The training monitors, checkpoint system, and inference workspace are designed to make it easy to run experiments, vary hyperparameters, and observe the results.

FAIRS is not a winning roulette system. The house edge is real. No deep learning architecture, no amount of training data, and no exploration strategy can overcome the mathematics of a negative-expectation game. The project's value lies in what the training process reveals about the algorithm.

FAIRS is open source (MIT). The entire codebase is at github.com/CTCycle/FAIRS-Roulette-Player. You can fork it, extend it with new agent architectures (PPO, SAC, curiosity-driven exploration), add roulette variants, or adapt the environment to other stochastic dynamics.

Closing

The roulette wheel is a useful adversary for RL research precisely because it is not trying to be beaten. It offers no pattern, adapts no strategy, and reveals no structure. Training a DQN against it is a reminder that not every problem is solvable by adding more layers. Recognising the absence of signal is itself a form of learning.

The full source code, including unit tests, end-to-end tests, and a pre-configured development environment, is available on GitHub. Contributions, experiments, and open issues are welcome.

References

Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.
Van Hasselt, H. (2010). Double Q-learning. Advances in Neural Information Processing Systems 23 (NeurIPS), 2613-2621. The original Double Q-learning paper, which used a roulette game as an evaluation environment.
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. International Conference on Learning Representations (ICLR).
FAIRS source code: app/server/learning/ (MIT License).

Built by CTCycle. ML researcher and data scientist working on clinical AI, RL agents, and scientific computing tools. Find me at github.com/CTCycle and on Dev.to at @ctcycle.