DEV Community: Vlad Navitski

Taking apart a result that (never) worked

Vlad Navitski — Thu, 10 Sep 2026 23:17:14 +0000

Sometimes, being restless about small things is a good trait. It helps to dismantle a beautiful facade that hides broken machinery inside, and as such clears the ground for building something that does actual work. That's exactly what happened with my RL experiment, and this is my story about it.

The thing that worked

The setup: a known EGFR inhibitor (a molecule that blocks EGFR, a receptor whose overactivity drives several cancers), fifteen steps to improve it, a composite score combining predicted activity with drug-likeness properties. Five policies - Random, ε-greedy, triggered ε-greedy, UCB, Thompson Sampling - differing in how they trade off exploiting what they've learned / exploring what they haven't. Tabular Q-learning, linear value function, three random seeds.

The results were clean. Learned policies crossed the success threshold more often than random modification. Some cleared conventional significance. The weight matrices showed distinct, interpretable functional-group preferences - the policies had clearly learned something, and it looked like the something was useful.

Bothersome snags

However, there were some irregularities that showed up quite early. The scene was set when I decided to try different hosts just to see if there would be any speedup, as the experiment took a considerable time to run. I didn't expect much as I didn't use any CUDA-like code, so GPUs and TPUs shouldn't have had much to chew on. It transpired exactly as expected - while there was a speed increase, it was moderate and was explained by faster CPUs on those hosts. What I didn't expect was that the performance of RL policies would differ between hosts. While RL policies still outperformed baseline and random modifications, they showed different numbers of hits and different statistical significance, and as a result, different rankings. What kept gnawing at me in the back of my mind was the fact that I couldn't pinpoint where the differences came from. Changing machines might change some arithmetics, but it shouldn't have affected the rankings and numbers of hits.

Why it stayed just an 'irritation' and didn't lead to the project overhaul immediately was the fact that its results essentially stayed the same. Difference in numbers didn't look like much when the significance boundary was crossed confidently. My thought process at that time was that I just had a stochastic process. It didn't challenge results as it still had a stable conclusion underneath.

Ironically, I used the same reasoning to wave away another concern - I had doubts that single matrix could be a good retaining mechanism for the policies in my experiment. The initial molecules that policies are called to improve on are chemically different compounds, they would require different modifications to increase their useful traits. Single matrix retaining preferences for all possible groups of initial scaffolds seemed like a naive implementation. "Yet it worked" - I kept telling myself, "The stats don't lie".

However, they might hint at something. For example, they might hint that the great results I was witnessing were nothing more than random success. Irritating differences among hosts were exactly that - early signs that something was seriously wrong. Tracing those differences back to their roots and learning invaluable lessons along the way turned out to be the most important part of the project, at least this iteration of it.

Pulling the thread

Where the whole fabric started to disintegrate was where it should have been the project's ultimate triumph - increasing the number of experiments (or 'seeds', as each experiment was controlled by a randomizer's seed for reproducibility purposes). As I've mentioned before, the script was quite slow, so I kept the number of experiment runs to a minimum I was comfortable with - just three of them for each policy. When I thought I was ready to share my results with a bigger world, I increased the number of seeds to ten per policy.

It did turn out to be a better sampling, but a better sampling that showed exactly what my initial results were - merely, the lucky chance. With ten seeds the performance of RL policies degraded dramatically, if not catastrophically. Many of them began to trail behind Random, to the point where it might be just one policy showing a significant improvement over random modifications.

The catastrophe was not that RL didn't work - in any case, I already had doubts that my setup was enough for RL agents to learn properly. The real issue was something else - namely, unstable results that in different runs would give you every answer possible, however contradictory. I didn't know what, if anything, worked in my script and what needed to be done to fix what didn't work.

Adding it all together - single matrix doubts, variability across hosts, collapsing with increased number of experiments - I realized that all 'irregularities' need to be accounted for properly if I wanted the project to survive.

First patch

Fixing the script started somewhere unexpected: one line of data preparation.

The pipeline selects a middle band of compounds (EGFR molecules from the curated data set) by sorting on measured score and taking a positional slice. But those score values are mostly ties - 5,400 of 6,000 rows share a value with some other row - and a sort with ties leaves the tied rows in an order nobody defined. NumPy picks its sort routine based on the CPU it's running on. Machines with the AVX-512 instruction set ordered the ties one way; machines without it ordered them another. Different molecules entered the pool, and everything downstream moved with them.

No arithmetic differed. Not one number was computed differently. The ordering of equal values changed, and a positional slice turned that into a different dataset. I've written that part up separately.

Fixing it was one extra column in the sort key. The dataset became reproducible across every environment I've tried since: a CPU vendor change, an AVX-512 boundary, NumPy and Python version bumps.

That should have been the end of it. Instead it just started a chain reaction.

The patch won't stop untangling

Same data on every machine, same seeds - and the policies still diverged. Now my concern reached the point where I put everything else aside and concentrated on tracing it back to its exact sources.

After several debugging experiments I realized that variations come from last-bit differences in floating point results that might differ between machines.

The mechanism is simple to explain. A policy computes action values by multiplying a state vector by a weight matrix, then applies the argmax. That arithmetic is deterministic on any given machine - repeat a run and you get identical bits - but not across machines. Different processors execute it along different instruction paths and can land on a different final bit. If two action values differ only there, argmax returns a different action on one machine than on another. That action changes the molecule, which changes the reward, which updates the weights. From that point the two runs aren't just 'a bit different'. They are completely separate and might lead to different outcomes.

My first instinct was to treat this as a numerical problem and go looking for a sturdier selection rule. That instinct was wrong, and understanding why is the moment the whole project turned over.

A one-bit gap between two action values means the policy considers those actions equally good. Choosing between them arbitrarily is just an exploration phasein RL policy, there's nothing wrong with that. It doesn't matter whether exploration is triggered by a quadrillionth value or by a tenth after rounding it up. What actually is wrong is that a policy keeps deciding at the last bit throughout the whole experiment.

If a trained policy is still deciding at the last bit, it means it never separated the signal from the noise. A policy that had learned something would have gaps between best and second-best action many orders of magnitude larger than machine precision, and a rounding difference couldn't reach them. The sensitivity I'd been treating as a numerical nuisance was a measurement, and what it measured was that nothing had been learned.

It was quite a harsh landing, so naturally I asked myself how it was possible to be in the dark about it for so long.

What was actually stitching it all together

The environment. Specifically, a rule I'd written without thinking hard about it.

Each episode takes a fifteen-step walk and reports the best molecule seen anywhere along the way. That's Monte Carlo maximisation: draw many samples, keep the maximum, and the maximum improves with the number of draws. It's the retention that produces the good results, not the search.

You can see it in what beats what. At the extreme - each method's best fifty molecules - every method clears the screening baseline decisively, Random included. Across everything the methods retained, Random and the screening baseline are statistically indistinguishable, and the learned policies sit slightly below both. The environment's tail is good. Its typical output is ordinary.

Random comes out ahead of every learned policy, which is the detail I find most instructive. Same walks, same retention rule, same budget - and the policy that imposes no preference gets the best tail. A systematic preference that can't tell contexts apart costs you sampling variety and buys nothing back. Consistency without correctness is strictly worse than no consistency at all when you're keeping the best of many samples.

Why there was nothing to learn

My initial doubt was a key to it all. A single linear weight matrix maps molecular fingerprints to action values across every structural family in the pool at once. When different scaffolds reward different edits - hydroxyl here, methyl there - one matrix cannot hold both, so instead it averages them.

That isn't a failure to detect a signal, rather the model had nowhere to put one. You can see it in the learned weights: every action value ends up negative, and the policy is choosing the least-bad option rather than a good one. And it's the same fact the argmax margins were reporting from another direction. Values that can't separate contexts can't separate actions either.

The policy also controls less than it appears to. It picks which functional group to attach. It doesn't pick the starting molecule - random draw. It doesn't pick where the group lands - also random. The chemical context in which its one decision gets made is entirely outside its control.

What I'd take from it

I ended with no result and a much better understanding of what a result would require: a total sort key so the data is stable, host discipline so trajectories are comparable, enough seeds that a verdict isn't a coin flip, and a representation rich enough for the policy's one decision to mean anything.

That's a worse outcome than "Thompson Sampling wins" and a more useful one. The winning version would have been a leaderboard entry, and - given how the ranking moved under every perturbation I eventually tried - an unreproducible one.

The lesson I'd actually hand to myself and to whoever'd listen:

Every inconsistency in your results needs an account, not a label. "It's
stochastic" is a label. An account names what varies, says why, and predicts
the variation well enough that you can reproduce it on demand. If you can't
reproduce your own noise, you don't understand it - and until you understand
it, you cannot tell it apart from the thing you're trying to measure.

The harder part isn't just knowing that. It's that nothing external will make you do something about it. Nothing was challenging my results. They were good, consistent, and the one loose thread had a plausible-sounding explanation attached. The only thing that made me pull on it was that the explanation never quite satisfied me, and I've come to think that low-grade dissatisfaction is worth more than it feels like at the time. A result nothing is questioning is exactly the one that has to be questioned by you.

Code, data, and full write-up:
github.com/farabhi/rl-lead-optimization

Same code, same seed, different answer

Vlad Navitski — Sat, 05 Sep 2026 14:14:42 +0000

The same code, with the same fixed seed and the same input file, gave different answers on different machines. I noticed that early, but waved it away. The differences looked cosmetic, less successful runs didn't disprove my findings, so I lulled myself with a comfortable story about a stochastic process.

That ended when I tried to increase the sample size. Unexpectedly, the result I had been building on collapsed, and with it my reason for not caring. The findings were not supported by my experiment anymore, so I naturally suspected the machine-related variances were early warning signs.

The two problems turned out to be unrelated in cause, but connected in how one obscured another. The machine-dependent noise made the early signs of real failure look like more of the same. Had I fixed the architecture problem when I first saw it, the genuine one would have stood out sooner.

The setup, stripped to its bones

The pipeline starts by selecting a "middle band" of samples from a dataset - discard the extremes, keep the moderate ones. The code did what looks entirely reasonable:

df_sorted = df.sort_values('value')
middle = df_sorted.iloc[1500:4500]   # drop the tails, keep the middle

Nothing special, right? Sort rows by a column, select rows by position. Anyone doing data processing had done it many times. Yet this is exactly where my experiment was broken.

Ties plus a positional slice

The first thing to mention is that the value column is full of ties. Out of 6,000 rows, about 5,400 shared their value with at least one other row. The measurement only has so much precision, so the same number shows up again and again.

Sorting by a column with ties gives you a partial order, not a total one. The sort will correctly put all rows with 7.2 value in between rows with 7.1 and 7.3. But internal arrangements of rows with 7.2 is never guaranteed. And this is harmless if you don't care about the order inside a tie group. Now, if you pair sorting with iloc[1500:4500] you might start caring - I know I should have. If the positions 1500 and 4500 land in the middle of tie groups, then sorting and slicing might deliver different samples each time you perform them.

The part that made it hardware-dependent

Yet it didn't. The sample was reproducible on a single machine, no matter how many time you run it. It's only when I changed the host I'd get variation and potential complete inversion of experiment results. My code didn't use accelerators or CUDA libraries, so why different hosts would have different outcomes?

The answer is it's because the thing breaking the ties is NumPy's sort implementation, and NumPy selects its sort routine based on the CPU it is running on. Some processors
support a vectorised instruction set called AVX-512, which lets NumPy sort through a code path that processes several values at once. Processors without it take another path. Both are correct - each produces a properly sorted array -but the value ties are resolved differently. No arithmetic changed. Not one number was computed differently. The ordering of equal values changed, and a positional slice turned that invisible difference into a different sample - which moved the apparent success rate of the whole experiment from 8/15 to 14/15 on identical input data.

One possible way to confirm it is by running the identical pipeline across different cloud runtimes - I executed them on 6 separate ones. They split into exactly two groups with two different results, and the split followed one thing: whether the CPU reported AVX-512 support. Furthermore, it's impossible to know beforehand which hardware you land on. Neither runtime type, nor vendor would tell you in advance if AVX-512 will be present or not. I had to keep the diagnostic fingerprint in my experiment script always on to know if the sample will be reproduced or not.

What the defect did, and what it didn't

Let me be precise - the sort defect did not cause my result to crumble. Had the method genuinely worked, my imperfect sort would not have made it fail - it would have produced real but variable outcomes, differing by host machine. At worst, I'd have had a numerical reproducibility problem and nothing more.

This variability muddied the water considerably and obscured the real problem. Every architecture gave a plausible answer. Different runs gave different plausible answers. Variability in success and hits seemed to be intrinsic to the experiment and consistent with its principal claim.

That's why the chain more data → collapse → traced to the sort bug was my first instinct and was completely wrong. Instead, two separate facts about the same project need to be stated clearly:

The method didn't work. That was always true; a larger sample made it visible.
The pipeline was non-deterministic across architectures. That was also always true, and it is what stopped me from establishing (1) months earlier.

They share connection even though they don't share the cause. The collapse is what finally made me take the anomaly seriously. The anomaly is why the collapse took so long
to be admitted and addressed.

The fix, and the thing the fix is an instance of

There are several ways to fix this. I chose to make the sort key total, by adding a tiebreaker that is unique per row:

df_sorted = df.sort_values(['value', 'id'])   # 'id' breaks every tie
middle = df_sorted.iloc[1500:4500]

Now every row has a definite position, no ties remain for any implementation to resolve differently, and every machine produces a byte-identical ordering. After the change, all six runtimes produced identical output down to a hash of the selected sample. Sample reproducibility has since held across a CPU vendor change, an AVX-512 boundary, and both NumPy and Python version bumps.

Again, fixing sample reproducibility didn't save the experiment from being refuted. It just meant that from then on I could distinguish between different sources of variability. Which is a must have for learning anything from an experiment at all.

There is a second fix worth mentioning, because it also works. Many sort functions accept a "stable" option (kind='stable' in NumPy), which preserves the input order of tied elements. Stability is a property of the algorithm, not of the hardware, so a stable sort would have given me the same pool on every machine. Had I reached for it, this post would have a shorter middle section.

I prefer the total key for reasons that are real but not dramatic. Stability makes the output a function of the input order - a property nothing in my code sets, records, or checks. A total key makes it a function of the data: shuffle the dataframe first and you get an identical pool. And the rule is visible. sort_values(['value', 'id']) states in one line how ties are broken; kind='stable' defers the question to however the rows happened to arrive.

Neither protects you from the larger thing. If the underlying dataset changes - a new release, a different query, more rows - the pool changes too, and no sort key saves you. That's a separate problem with a separate answer, which in my case was to commit the exact snapshot the results came from.

But using a total sort key was just a recipe for this particular experiment. There's a wider lesson to be learnt from it that could be applied to future experiments as well, even if recipes will look different.

The sort key was a proxy. The rows were the payload.

If my dataframe had held nothing but value, ties would have been genuinely harmless. Any tie ordering produces the same multiset of numbers, and nothing downstream could tell the difference. In that world the slice is safe and this post doesn't exist.

But the rows carried something else - in my case molecular structures, and the downstream code depended heavily on which structures it received. value was
only a convenient handle for choosing among them.

So when I sorted on value alone, it was as if I made a claim in my python code: rows with equal value are interchangeable for my purposes. That claim is true of the numbers. It is false of the molecules. And nothing in the code marked the difference, because the proxy and the payload travel in the same object and get selected by the same
operation.

That's the general shape, and it isn't specific to sorting:

Know which column you actually care about, and don't let a different column
stand in for it silently. If a proxy has to do the selecting, be explicit
about what the proxy's equivalences mean for the payload.

The total-key fix works precisely because it retires the claim. With a unique tiebreaker there are no equivalences left to be wrong about.

You have probably seen this bug before

If it feels familiar, it's the classic SQL pagination bug wearing different
clothes:

SELECT * FROM items ORDER BY created_at LIMIT 20 OFFSET 40;

If created_at isn't unique - and timestamps collide more often than people expect - then "page 3" isn't well defined. The same row can appear on two pages
or none.

And it fails for exactly the reason above. Nobody paginates in order to collect timestamps. You want the rows; created_at is the proxy you're ordering them by. Equal timestamps do not mean interchangeable rows, but ORDER BY created_at asserts that they do. The fix is the same: add a unique tiebreaker, and the false claim disappears. SQL doesn't offer the stable-sort shortcut - tables are unordered by definition, so there's no input order to preserve. Either you supply a total key, or which rows land on which page is up to the planner.

So, the practical rule and the one underneath it:

Any time you select data by position - a slice, a limit, an offset, "the top N" - the sort that defines those positions must be total.

And more generally:

When you order by one thing to select another, you are asserting that the first thing's ties are irrelevant to the second.

And one about the debugging, which cost me the most:

When two things are wrong at once, the noisy one hides the quiet one. My sort defect produced results that varied architecture to architecture, and that variation was exactly the shape of the evidence I needed to see that the method itself wasn't working. I spent months reading one problem's symptoms as the other's noise.

This came out of a reinforcement-learning experiment on molecular lead optimization; the code, data and full write-up are at github.com/farabhi/rl-lead-optimization. Fixing this made the dataset reproducible across machines. It did not make the experiment reproducible - that's a separate problem, and a stranger one, which I'll write about next.