<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ultimate Trading System</title>
    <description>The latest articles on DEV Community by Ultimate Trading System (@ultimatetradingsystem).</description>
    <link>https://dev.to/ultimatetradingsystem</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254898%2Fb97ed31c-c0c6-4175-89a8-155cc84e08ef.png</url>
      <title>DEV Community: Ultimate Trading System</title>
      <link>https://dev.to/ultimatetradingsystem</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ultimatetradingsystem"/>
    <language>en</language>
    <item>
      <title>Train, test, held, reserve: why one hold-out is not enough once you have looked at it</title>
      <dc:creator>Ultimate Trading System</dc:creator>
      <pubDate>Sat, 21 Nov 2026 02:48:46 +0000</pubDate>
      <link>https://dev.to/ultimatetradingsystem/train-test-held-reserve-why-one-hold-out-is-not-enough-once-you-have-looked-at-it-4af2</link>
      <guid>https://dev.to/ultimatetradingsystem/train-test-held-reserve-why-one-hold-out-is-not-enough-once-you-have-looked-at-it-4af2</guid>
      <description>&lt;p&gt;Anyone who tests a trading rule on past prices learns the first lesson quickly. A rule tuned on a stretch of history looks good on that stretch, because it was shaped to fit it. So you keep some history aside, tune on the rest, and check the rule on the part it never saw. That part is usually called a hold-out, or out-of-sample data.&lt;/p&gt;

&lt;p&gt;It is a sound idea. It also wears out faster than most people expect, and it wears out through a perfectly reasonable habit: looking at the result and then doing something about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  How a hold-out gets used up
&lt;/h2&gt;

&lt;p&gt;Take a made-up example. You search one stretch of history and come away with twenty candidate rules. You run all twenty on the hold-out. Most do poorly. Three do well. You keep the three and write down their hold-out scores as your evidence.&lt;/p&gt;

&lt;p&gt;Those scores are no longer clean. The hold-out was supposed to judge the rules. Instead, it chose them. You kept those three because they scored well there, so of course they score well there.&lt;/p&gt;

&lt;p&gt;Coins make it plain. Give twenty people a coin each and have them flip ten times. Keep the three with the most heads. Their sheets look impressive, and they would look just as impressive if every coin were fair, because you selected on the very thing you are now admiring. Ask the same three to flip again and they go back to being ordinary.&lt;/p&gt;

&lt;p&gt;The same wearing-out also happens more quietly. The hold-out result disappoints, so you go back, change a setting and check again. A little better. You change another. Each round trip is a small choice made with the hold-out's help. After enough of them, the rule has been fitted to the hold-out as surely as it was fitted to the first stretch. It only took longer, and nothing dishonest happened at any step.&lt;/p&gt;

&lt;p&gt;What comes out of this is a principle that is simple to say and hard to keep: the history that judges a rule must never be the history that chose it. The moment a stretch of history influences a choice, it has changed sides. It is now part of the choosing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four parts instead of two
&lt;/h2&gt;

&lt;p&gt;If looking and choosing use history up, the answer is to decide in advance which history may be used up, and to keep some back that is not used at all. One arrangement splits history four ways: train, test, held and reserve.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Train&lt;/strong&gt; and &lt;strong&gt;test&lt;/strong&gt; are the working parts. This is where rules are found, compared, adjusted and thrown out. You may look at these as often as you like. They are expected to wear out, and a good score here should impress nobody.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Held&lt;/strong&gt; is history no choice was ever made on. The survivors of the first two parts are judged here. Because nothing was picked or tuned on it, a score on held means something that a score on the working parts cannot.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reserve&lt;/strong&gt; comes last of all. It exists because of the problem above. Once you have seen the held result, you are a person who has looked. You may be tempted to adjust something, or just to carry forward the survivors that did best on held, which is the twenty-coins mistake again. Reserve is the reading taken after all of that, on history still untouched.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping the last part honest
&lt;/h2&gt;

&lt;p&gt;A four-way split only helps if it is kept. Three habits do most of the work.&lt;/p&gt;

&lt;p&gt;Decide the split before you start, and write it down. Boundaries moved after seeing results are choices too.&lt;/p&gt;

&lt;p&gt;Decide beforehand what a pass looks like. If a rule falls short on reserve, the honest result is that it fell short. The temptation will be to explain it away.&lt;/p&gt;

&lt;p&gt;Look at reserve once. Look, adjust and look again, and you have turned it into one more working part. Then there is nothing left to judge with.&lt;/p&gt;

&lt;p&gt;Even kept perfectly, the split has a limit. A rule that passes on every part has passed on the past. No arrangement of old prices can stand in for prices that have not happened yet. What the split does is narrower and still valuable: it stops a rule from being marked by the same history that picked it.&lt;/p&gt;

&lt;p&gt;Ultimate Trading System, the software we build, splits history these four ways, train, test, held and reserve, so the history that judges a rule is never the history that chose it. Its Held and Reserve steps judge the survivors on history no choice was ever made on, held first and reserve last of all. The free trial needs no exchange account, because Construct and Paper Books work on public data. The work runs on a machine of your own; nothing is hosted: &lt;a href="https://ultimate-trading-system.com" rel="noopener noreferrer"&gt;https://ultimate-trading-system.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is software, not financial advice. Trading carries a substantial risk of loss, and passing these tests does not guarantee future results.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written by the AI assistant of the maker of Ultimate Trading System.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>backtesting</category>
      <category>trading</category>
      <category>datascience</category>
      <category>statistics</category>
    </item>
    <item>
      <title>How good does "nothing" look? Why a backtest needs a no-information baseline</title>
      <dc:creator>Ultimate Trading System</dc:creator>
      <pubDate>Fri, 20 Nov 2026 06:15:25 +0000</pubDate>
      <link>https://dev.to/ultimatetradingsystem/how-good-does-nothing-look-why-a-backtest-needs-a-no-information-baseline-2kme</link>
      <guid>https://dev.to/ultimatetradingsystem/how-good-does-nothing-look-why-a-backtest-needs-a-no-information-baseline-2kme</guid>
      <description>&lt;p&gt;A backtest ends with a number. A rule was run over some years of past prices, and it came out ahead. The natural reaction is to be pleased. The more useful reaction is to ask a dull question: ahead of what?&lt;/p&gt;

&lt;p&gt;Most people compare against zero. If the rule finished above where it started, it is called good. But zero is the wrong mark, and a few coins show why.&lt;/p&gt;

&lt;h2&gt;
  
  
  A room full of coin flippers
&lt;/h2&gt;

&lt;p&gt;Picture a hundred people in a room. Each flips a coin ten times and writes down the result. Nobody in the room has any skill at flipping coins. Even so, when the sheets are collected, a handful will almost certainly show eight heads or more. Sort the sheets from best to worst and the one on top looks like talent.&lt;/p&gt;

&lt;p&gt;It is not talent. It is what the best of a hundred looks like when nothing is going on.&lt;/p&gt;

&lt;p&gt;A search for a trading rule works the same way. You never test just one rule. You try a longer look-back and a shorter one, a tighter threshold and a looser one, this exit and that exit. Every combination is one more person in the room. Try a few thousand, and the best of them will show a handsome score on the history you searched, whether or not any of them knows a thing about prices.&lt;/p&gt;

&lt;p&gt;So the number at the top of the list answers the wrong question. It says how the best of many tries scored. It does not say how much of that score the search would have produced anyway, with nothing behind it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building "nothing" on purpose
&lt;/h2&gt;

&lt;p&gt;The fix is to measure that directly. Before judging a rule, build a version of it that cannot possibly know anything, and put it through exactly the same process.&lt;/p&gt;

&lt;p&gt;There are a few ordinary ways to do this. One is to replace the rule's signals with coin flips: the same number of entries, placed at random. Another is to keep the rule's own signals but shuffle them in time, so each one still exists but no longer sits beside the prices it was reacting to. A third is to shuffle the labels the rule was fitted to, so that whatever it learns, it learns from noise. The general idea has an ordinary name, a permutation test.&lt;/p&gt;

&lt;p&gt;What these share is that the link between signal and price is cut on purpose, and everything else is left alone. Same history. Same costs. Same scoring. And, most important, the same search: if the real run tried a few thousand settings and kept the best, the no-information run tries a few thousand settings and keeps the best too.&lt;/p&gt;

&lt;p&gt;Then you look at what "nothing" scored.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the baseline tells you, and what it does not
&lt;/h2&gt;

&lt;p&gt;First, "nothing" is rarely zero. Imagine a made-up market that drifted upward for the whole stretch. Random buying ends ahead there. Imagine another that went sideways, with a cost on every trade. Almost everything ends behind. The baseline moves with the history, which is exactly why a fixed mark like zero misleads.&lt;/p&gt;

&lt;p&gt;Second, one shuffle is not enough. A single no-information run can land high or low by luck, like a single flipper. Repeat it many times and you get a spread: where "nothing" usually lands, and how far it sometimes reaches. That spread is the real mark.&lt;/p&gt;

&lt;p&gt;Third, it changes what counts as a result. A rule that sits inside the spread has shown nothing, however good its number looks alone. A rule that stands clear of the spread has earned a second look, and only that. It has passed one test on one stretch of history. The next step is history it has never seen.&lt;/p&gt;

&lt;p&gt;A baseline does not tell you what a rule will do next. It removes one way of fooling yourself, which is mistaking the best of many tries for a discovery. That one is worth removing first, because it is easy to fall into and cheap to check. It also keeps effort honest: search harder and the real score climbs, but so does the score of "nothing". Watching the two rise together is a humbling and useful thing to see.&lt;/p&gt;

&lt;p&gt;This is one of the things we built Ultimate Trading System to do without being asked. Each run is repeated with votes that carry no information, so the board shows how good "nothing" looks before anything is judged against it. A survivor then faces history it never saw, measured against four simpler ways of trading the same stretch. The free trial needs no exchange account, because Construct and Paper Books work on public data. The work runs on a machine of your own; nothing is hosted: &lt;a href="https://ultimate-trading-system.com" rel="noopener noreferrer"&gt;https://ultimate-trading-system.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is software, not financial advice. Trading carries a substantial risk of loss, and passing these tests does not guarantee future results.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written by the AI assistant of the maker of Ultimate Trading System.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>backtesting</category>
      <category>trading</category>
      <category>datascience</category>
      <category>statistics</category>
    </item>
  </channel>
</rss>
