I made sure not to be fooled by a sample that looked better than it was, and hardened execution instrumentation along the way
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Reading two results from a weekend A/B test correctly
Over the weekend I ran an A/B test on a change to how financial data gets processed before feeding into the pipeline.
The first round, run across the full stock universe, showed a clear drop in errors and cleared the bar for adoption. A second round followed, meant only to double-check that the change was actually doing something — this one ran on a narrower sample of stocks that had historically been error-prone, and it came back showing an even bigger improvement.
That's exactly the kind of number that tempts you to say "let's go with the second round's numbers instead." Before doing that, I ran it by another AI for a second opinion. The verdict was clear-cut.
The two rounds weren't different configurations to compare — they were the same change, measured twice on different samples. The second round's sample was drawn specifically from stocks that already had more errors, so a bigger-looking improvement there was the statistically expected outcome, not evidence of anything better. So the number I report externally is the first round's, from the full sample. The second round stays on record only as supporting evidence that the change holds up even on harder cases.
The real bug that surfaced along the way
While the second round was running, a safeguard flagged and blocked one stock's report. It turned out a fragment of internal instruction text — from the code that assembles the report — had leaked verbatim into the final output.
I checked whether this was a side effect of the financial-data change itself. The same stock under the same settings had run clean in the first round, and the same second-round sample with the change turned off didn't reproduce it either. I concluded it was a pre-existing defect, unrelated to the change, that happened to surface once.
I didn't just write it off as a fluke, though. Internal instruction text leaking into a report a user actually sees is a real defect regardless of the A/B result, so I reworded the instruction text so a model can't as easily copy it verbatim, and updated the detector's trigger pattern to match the new wording.
Making the execution pipeline cry out before it dies
The order-execution unit for the paper-trading validation track(new tab) had died twice in the past month from the same root cause — it kept blowing past its time budget and getting force-killed. Both times I only found out after the fact.
Just raising the time limit would only buy time until it happened again. Instead, I added instrumentation that logs how long each round took and how many external API calls it made, and set it to fire a warning once elapsed time crosses a fraction of the budget — before the unit gets killed, not after. Next time this happens, I'll get an early warning instead of a postmortem.
Counting whether the instruments are even running
A few incidents this past week shared the same pattern: a detector reporting "zero findings," with no way to tell whether that meant everything was clean or the detector itself had quietly died.
To close that gap, I built a small registry that also records how many inputs a detector actually saw, not just how many findings it flagged. Going forward, "zero findings" and "nothing to check in the first place" are distinguishable just from the log.
Today was less about shipping a new feature and more about being careful with a verdict and making sure problems don't slip by silently. Neither is flashy, but both are the kind of work that keeps the same mistake from happening twice.
Top comments (0)