<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: TradingLookup</title>
    <description>The latest articles on DEV Community by TradingLookup (tradinglookuphq).</description>
    <link>https://dev.to/tradinglookuphq</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Forganization%2Fprofile_image%2F15367%2Fbeddd9eb-bac0-4596-8eb8-d0daf9a8b7a2.png</url>
      <title>DEV Community: TradingLookup</title>
      <link>https://dev.to/tradinglookuphq</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/tradinglookuphq"/>
    <language>en</language>
    <item>
      <title>Reading one legally required sentence from 31 broker websites</title>
      <dc:creator>TradingLookup</dc:creator>
      <pubDate>Wed, 28 Oct 2026 19:18:02 +0000</pubDate>
      <link>https://dev.to/tradinglookuphq/reading-one-legally-required-sentence-from-31-broker-websites-3ocn</link>
      <guid>https://dev.to/tradinglookuphq/reading-one-legally-required-sentence-from-31-broker-websites-3ocn</guid>
      <description>&lt;p&gt;If you visit a UK or EU broker that offers contracts for difference (CFDs), you'll find a sentence like this somewhere on the page:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;70% of retail investor accounts lose money when trading spread bets and CFDs with this provider.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Regulators require it. Each provider has to work out its own figure and print it as part of a standard risk warning. That makes it one of the few numbers in retail trading that every firm states in the same shape, on its own site, in public.&lt;/p&gt;

&lt;p&gt;We built an open dataset of those sentences: 31 providers, read from the UK on 6 October 2026 (7 updated after a second UK read on 13 October). The figures run from 51% to 84%, with a median of 73.46%. This post covers how we read them, what went wrong, and how we pulled eight years of earlier figures out of the Wayback Machine.&lt;/p&gt;

&lt;h2&gt;
  
  
  A strict match, not "any percentage"
&lt;/h2&gt;

&lt;p&gt;Broker homepages are full of percentages: margin rates, spreads, promotional rates. The first version of our extractor grabbed the first percentage it found near the words "retail investor". It picked up a 5% margin rate that happened to sit just above the warning.&lt;/p&gt;

&lt;p&gt;The fix was to anchor the wording: the percentage only counts if the mandated phrase follows it directly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="n"&gt;RATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;(\d{1,2}(?:[.,]\d+)?)\s*(?:–|-)\s*(\d{1,2}(?:[.,]\d+)?)\s?%&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;  &lt;span class="c1"&gt;# a range: "74-89%"
&lt;/span&gt;    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;|(\d{1,2}(?:[.,]\d+)?)\s?%&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;                                     &lt;span class="c1"&gt;# or one figure: "73.6%"
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;VALID&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\s*(?:of\s+(?:all\s+)?retail\s+(?:investor|client|account|cfd)&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;|der\s+Kleinanlegerkonten)&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                                    &lt;span class="c1"&gt;# German wording, IG Europe
&lt;/span&gt;    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;find_rate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;RATE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;finditer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;          &lt;span class="c1"&gt;# wording must start right after the %
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few details mattered more than expected:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Comma decimals and a space before %.&lt;/strong&gt; EU pages write "74 %" and "73,5 %".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ranges.&lt;/strong&gt; Some groups print a range across their entities. We keep both ends and take the midpoint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;More than one warning on a page.&lt;/strong&gt; A site serving several group companies can show a UK and an EU figure side by side. We record every distinct match, and a page with more than one is flagged for a check by hand.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The sentence, not just the number.&lt;/strong&gt; We store the warning text as it was printed, from the percentage to the next full stop. Then anyone can see exactly what was read.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A fingerprint for every read
&lt;/h2&gt;

&lt;p&gt;Each row carries a SHA-256 hash of the page text we read, plus how it was fetched and from where:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;

&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;page_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                   &lt;span class="c1"&gt;# the text we matched against
&lt;/span&gt;&lt;span class="n"&gt;fingerprint&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The hash doesn't prove the page said what we claim. It does pin down which version of the page we mean, so a later disagreement can be traced to "the page changed" or "we misread it".&lt;/p&gt;

&lt;h2&gt;
  
  
  An honest second read
&lt;/h2&gt;

&lt;p&gt;On 13 October we ran an independent, automated second read of every source page, from a build environment in the US rather than the UK. Rules for that pass:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An honest user agent that names us and links our method page.&lt;/li&gt;
&lt;li&gt;robots.txt is checked first. A 401 or 403 on robots.txt counts as "don't crawl".&lt;/li&gt;
&lt;li&gt;No bot-wall bypass. If the page says "verify you are human", we log it as blocked and move on.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The result: &lt;strong&gt;16 of 31 figures matched, 10 pages blocked us, 5 showed no figure, and none disagreed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The "no figure" cases were instructive. eToro, for example, served our US-based reader its US site, which carries no CFD warning, so the UK page was never reached. A blocked or empty page isn't a contradiction, so we report the three outcomes separately instead of a single "verified" percentage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Eight years of history from the Wayback Machine
&lt;/h2&gt;

&lt;p&gt;The Internet Archive has captured many of these pages over the years, so we can go back to 2018. The CDX API lists captures for a URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;cdx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://web.archive.org/cdx/search/cdx&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?url=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;output=json&amp;amp;fl=timestamp,statuscode,original&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;filter=statuscode:200&amp;amp;collapse=timestamp:6&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;   &lt;span class="c1"&gt;# at most one capture per month
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From those we pick about one capture per quarter and run the same strict match on the archived text. Two practical notes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Go slowly.&lt;/strong&gt; We wait 5 seconds between archive requests. Faster than about 15 a minute and archive.org blocks the IP for a while.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gaps stay gaps.&lt;/strong&gt; If there's no capture for a quarter, the series has a hole. We never fill it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The history file holds 404 quarterly rows for 18 providers, from 2018-Q3 to 2026-Q4, each linked to its capture.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you can use
&lt;/h2&gt;

&lt;p&gt;Everything is CC BY 4.0, credited to TradingLookup:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://tradinglookup.com/brokers/cfd-loss-rates/" rel="noopener noreferrer"&gt;The live table, with each provider's figure and source&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://tradinglookup.com/data/" rel="noopener noreferrer"&gt;The CSV, history and Frictionless data package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://tradinglookup.com/method/" rel="noopener noreferrer"&gt;How every figure is collected, checked and dated&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The CSV is served with &lt;code&gt;Access-Control-Allow-Origin: *&lt;/code&gt;, so you can load it straight into a notebook or a chart in the browser.&lt;/p&gt;

&lt;p&gt;The series is provisional: the legal entity behind two of the figures isn't confirmed yet, and regulators' registers haven't been cross-checked. If you spot a figure that doesn't match a provider's current page, &lt;a href="https://tradinglookup.com/method/" rel="noopener noreferrer"&gt;the method page&lt;/a&gt; says how to report it.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;CFDs are complex instruments and come with a high risk of losing money rapidly due to leverage. Each figure is the provider's own statement about its client base, not anyone's personal odds.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>opendata</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Turning the CFTC Commitments of Traders report into a weekly crowd-positioning gauge</title>
      <dc:creator>TradingLookup</dc:creator>
      <pubDate>Sun, 18 Oct 2026 14:08:14 +0000</pubDate>
      <link>https://dev.to/tradinglookuphq/turning-the-cftc-commitments-of-traders-report-into-a-weekly-crowd-positioning-gauge-3jcj</link>
      <guid>https://dev.to/tradinglookuphq/turning-the-cftc-commitments-of-traders-report-into-a-weekly-crowd-positioning-gauge-3jcj</guid>
      <description>&lt;p&gt;Every Friday afternoon, the US Commodity Futures Trading Commission (CFTC) publishes who holds what in the US futures markets. It's called the Commitments of Traders report, or COT, and it has been around for decades. Most people read it for what the big players are doing. I wanted the other end: how the small accounts are positioned, and whether that's unusual for them.&lt;/p&gt;

&lt;p&gt;This post covers how we turned that into a weekly 0 to 100 reading we call the Crowd Gauge: the source, the fields, the percentile method, a data quirk that bit us, and the limits.&lt;/p&gt;

&lt;h2&gt;
  
  
  The source: a public API, no key
&lt;/h2&gt;

&lt;p&gt;The CFTC serves COT data through a Socrata API at &lt;code&gt;publicreporting.cftc.gov&lt;/code&gt;. No sign-up, no key. (An optional app token raises the rate limit; we read one from the environment if it's set.)&lt;/p&gt;

&lt;p&gt;We use the &lt;strong&gt;Legacy, futures-only&lt;/strong&gt; report, dataset id &lt;code&gt;6dca-aqww&lt;/code&gt;. It covers stock indices, metals, energy, currencies and the CME crypto contracts. Each row is one contract market on one report date.&lt;/p&gt;

&lt;p&gt;The request is a plain filtered query. This is a cut-down version of our collector, with the same parameters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;

&lt;span class="n"&gt;BASE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://publicreporting.cftc.gov/resource/6dca-aqww.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;DATE_COL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;report_date_as_yyyy_mm_dd&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;CODE_COL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cftc_contract_market_code&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="n"&gt;codes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;13874A&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;088691&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;133741&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# E-mini S&amp;amp;P 500, gold, CME bitcoin
&lt;/span&gt;&lt;span class="n"&gt;where&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;CODE_COL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; in(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;repr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;codes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;where&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; AND &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;DATE_COL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &amp;gt;= &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;2023-01-01T00:00:00&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;

&lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;$where&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;$order&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;DATE_COL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;CODE_COL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;$limit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;$offset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BASE&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlencode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The real collector pages 50,000 rows at a time, keeps a bookmark so later runs fetch only new weeks, and stores each raw response with a SHA-256 hash. That hash is in our published JSON, so any reading traces back to the exact bytes it came from.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fields we actually use
&lt;/h2&gt;

&lt;p&gt;COT splits open interest into reportable traders (big enough to hit the CFTC's reporting thresholds) and &lt;strong&gt;non-reportable&lt;/strong&gt; positions: everything left over. Non-reportable is the closest thing the report has to "the retail crowd". So from each row we keep just four things:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;cftc_contract_market_code&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Which contract (e.g. &lt;code&gt;13874A&lt;/code&gt; is the CME E-mini S&amp;amp;P 500)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;report_date_as_yyyy_mm_dd&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The position date, normally a Tuesday&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;nonrept_positions_long_all&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Small traders' long contracts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;nonrept_positions_short_all&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Small traders' short contracts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;From those we compute one raw number per contract per week, the long share:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;share&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;long&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;long&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;short&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On its own that number is hard to read. On the 6 October 2026 report, small traders in the E-mini S&amp;amp;P 500 held 248,117 long against 137,112 short, a 64.41% long share. Is that a lot? Every market has its own built-in tilt, so 64% can be normal in one contract and extreme in another.&lt;/p&gt;

&lt;h2&gt;
  
  
  Release date, not position date
&lt;/h2&gt;

&lt;p&gt;One detail that's easy to get wrong in a backtest: the report describes Tuesday's positions but nobody can see it until Friday. We date every value from its release, three days after the position date:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;COT_RELEASE_LAG&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;  &lt;span class="c1"&gt;# Tuesday as-of -&amp;gt; Friday release
&lt;/span&gt;&lt;span class="n"&gt;released&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;as_of&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;COT_RELEASE_LAG&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Holiday weeks sometimes slip a day or two later than that. We accept that small error rather than maintain a holiday calendar [judged].&lt;/p&gt;

&lt;h2&gt;
  
  
  The 3-year percentile
&lt;/h2&gt;

&lt;p&gt;To strip out each market's normal lean, we rank this week's long share against the same contract's previous 156 weekly readings (three years). Ties count half. A contract needs at least 52 readings before it produces a value at all.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pct_rank&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;below&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;equal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;below&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;equal&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;WINDOW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;MIN_OBS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;156&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;52&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;win&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;shares&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;WINDOW&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pct&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;pct_rank&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;win&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shares&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;win&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;MIN_OBS&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the long-standing "COT index" idea; we didn't invent it. Percentiles also cope with fat tails better than z-scores, which matters when one week can be several times bigger than the rest.&lt;/p&gt;

&lt;p&gt;The E-mini S&amp;amp;P 500's 64.41% long share ranked at &lt;strong&gt;76.6&lt;/strong&gt;. That means it was higher than about three-quarters of the last three years.&lt;/p&gt;

&lt;p&gt;From there it's averaging:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Market&lt;/strong&gt; = mean of its contracts (bitcoin and ether each have a standard and a micro contract).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Class&lt;/strong&gt; = equal-weight mean of its markets. Futures: E-mini S&amp;amp;P 500, Nasdaq-100, gold, WTI crude, euro, yen. Crypto: bitcoin, ether.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Composite&lt;/strong&gt; = 50% futures, 50% crypto. These weights were fixed before any testing and are never tuned.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A COT value stays usable for up to 10 days, then drops out. If less than half the composite weight is live, no composite is published that day.&lt;/p&gt;

&lt;p&gt;Readings fall into five bands: 0–20 crowded short, 20–40 leaning short, 40–60 balanced, 60–80 leaning long, 80–100 crowded long. On the 9 October 2026 release the composite read &lt;strong&gt;65.62 (leaning long)&lt;/strong&gt;: futures 55.77, crypto 75.48. You can see the current values on &lt;a href="https://tradinglookup.com/data/crowd-gauge/" rel="noopener noreferrer"&gt;the live Crowd Gauge&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall: the CME quarterly roll
&lt;/h2&gt;

&lt;p&gt;This one we caught late. In the published observations, the report dated 15 September 2026 looks odd for several CME currency futures. Mexican peso non-reportable longs jump to 68,934, against a median of 7,637 over the previous 12 weeks. New Zealand dollar longs jump to 48,402 against 3,206. Shorts balloon the same way, so the long share falls towards 50%: the peso went from 69.03% the week before, to 51.46%, and back to 71.21% the week after. Its percentile dropped from 51.6 to 24.68 for one week.&lt;/p&gt;

&lt;p&gt;The same spike shows up on 16 December 2025 and 16 June 2026. Those dates fall in the quarterly expiry weeks of CME currency futures, when positions in the old and new contract months are open at the same time. Our reading is that this is a roll effect, not a real shift in sentiment [judged].&lt;/p&gt;

&lt;p&gt;Method v1.0 doesn't filter these weeks. Old readings are never rewritten, so a fix will come as a new method version. A quick way to flag them yourself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pandas&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;

&lt;span class="n"&gt;obs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crowd-gauge-observations.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;median_12w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;leg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nonrep_long&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;shift&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;rolling&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spike&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nonrep_long&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;median_12w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;obs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spike&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;][[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;market&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;as_of_date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nonrep_long&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;median_12w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the file as published (data to 9 October 2026), a 3× threshold flags 19 rows. Sixteen of them fall on those three dates, all in CME currency contracts. March 2026 shows no spike, and almost nothing like it appears before late 2025. We haven't explained either yet, so treat the roll explanation as a working theory.&lt;/p&gt;

&lt;h2&gt;
  
  
  We wrote the test down first
&lt;/h2&gt;

&lt;p&gt;We wanted to know whether extreme readings (80 and over, or 20 and under) came before unusual returns. So we committed the test rules to the repository &lt;strong&gt;before&lt;/strong&gt; running it: horizons, periods, statistics, and the exact pass/fail wording. We call it "an edge" only if the pooled 4-week coefficient from 2018 onwards has |t| ≥ 2 with Newey-West errors, &lt;strong&gt;and&lt;/strong&gt; points the same way in 2006–2017.&lt;/p&gt;

&lt;p&gt;The result (measured, across 14 markets with free price series):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Crowded long:&lt;/strong&gt; passed. Four-week returns after 80+ readings were 1.0 percentage point above each market's average from 2018 onwards (t = 2.54). In 2006–2017 it was +0.30 (t = 1.18). That's the &lt;em&gt;opposite&lt;/em&gt; of the popular "fade the crowd" idea.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Crowded short:&lt;/strong&gt; no reliable edge (t = 1.49).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The caveats matter as much as the result. The effect is small next to normal price swings. Leaving out bitcoin and ether (a check we added afterwards, so not part of the rule) drops the crowded-long t-statistic to 1.97. Pooling also treats markets as independent, which flatters the t-statistics. A pattern in past data may not repeat.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Retail" is a proxy.&lt;/strong&gt; Non-reportable means "below the CFTC's reporting size", which includes some small firms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It's weekly and three days old.&lt;/strong&gt; Until our exchange data sources qualify (they need 180 days of collection first), the gauge moves once a week.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Futures only.&lt;/strong&gt; Retail spot forex and CFD positioning sits with brokers, and we only use that with their permission.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A percentile is relative.&lt;/strong&gt; A reading of 90 means "more long than in 90% of the last three years", not "90% of traders are long".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Roll weeks&lt;/strong&gt;, as above.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Get the data
&lt;/h2&gt;

&lt;p&gt;Everything is CC BY 4.0. The &lt;a href="https://tradinglookup.com/data/crowd-gauge-observations.csv" rel="noopener noreferrer"&gt;weekly observations CSV&lt;/a&gt; has the raw non-reportable longs and shorts, long share and both percentiles for every contract since 2006. The &lt;a href="https://tradinglookup.com/data/crowd-gauge/method/" rel="noopener noreferrer"&gt;full method write-up&lt;/a&gt; has the backtest tables by market.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This is data about positioning, not investment advice. Futures and crypto derivatives are leveraged and carry a high risk of losing money.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Sydney Stones is a co-founder of TradingLookup and leads its data and method work.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>datascience</category>
      <category>opendata</category>
      <category>finance</category>
    </item>
  </channel>
</rss>
