<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Slipio</title>
    <description>The latest articles on DEV Community by Slipio (@slipio).</description>
    <link>https://dev.to/slipio</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248912%2F6d7f036d-cb79-4a04-8667-41de93033988.png</url>
      <title>DEV Community: Slipio</title>
      <link>https://dev.to/slipio</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/slipio"/>
    <language>en</language>
    <item>
      <title>Engineering a Football Intelligence Platform: Simulations, Statistical Models, Scanners and Automated Workflows</title>
      <dc:creator>Slipio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 05:23:12 +0000</pubDate>
      <link>https://dev.to/slipio/engineering-a-football-intelligence-platform-simulations-statistical-models-scanners-and-3fe0</link>
      <guid>https://dev.to/slipio/engineering-a-football-intelligence-platform-simulations-statistical-models-scanners-and-3fe0</guid>
      <description>&lt;p&gt;&lt;em&gt;Written by the Slipio team. Slipio is our product.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Football looks like a perfect machine-learning problem: decades of results, thousands of matches a week, and a clean target. In practice the hard parts are rarely the model. They are time, truth and honesty: knowing exactly what was knowable when, knowing when a number in your database is final, and knowing whether your model is actually any good.&lt;/p&gt;

&lt;p&gt;This is a tour of how we built &lt;a href="https://slipio.com/" rel="noopener noreferrer"&gt;Slipio&lt;/a&gt;, a football intelligence web app that turns data, simulation and calibrated models into match probabilities with plain-English reasons. The stack is a Java 21 / Spring Boot backend on PostgreSQL, a Qdrant vector store, and React front ends. We will go layer by layer and say where things went wrong, because most of what we know came from that.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Ingestion: a feed is not the truth
&lt;/h2&gt;

&lt;p&gt;Fixtures, odds, statistics, match events and line-ups come from a commercial data provider. The ingestion layer is a set of resource-specific workers behind a dispatcher, and every request passes through a rate budget. Each priority level has its own ceiling, so a backfill cannot starve the jobs that collect pre-match data.&lt;/p&gt;

&lt;p&gt;Three rules came out of real incidents:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An empty response is not "not found".&lt;/strong&gt; Throttling also returns an empty body. Treat empty as "unknown, retry later", never as proof that something does not exist.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A present row is not a final row.&lt;/strong&gt; The provider writes match statistics &lt;em&gt;while the match is being played&lt;/em&gt;. We once settled a corners selection against statistics captured 33 minutes into the match. Settlement now waits until the statistics were last updated at least three hours after kick-off. This costs nothing in practice, because the median statistics row lands about two days after kick-off anyway.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never derive a season from a calendar year.&lt;/strong&gt; A European 2024 season runs from August 2024 to May 2025. One historical odds import resolved every January–May fixture against the wrong season. Every miss came back as "skipped", which looks exactly like a competition we do not track. A third of the corpus went unpriced, and the job still reported COMPLETED. Resolve a fixture on its provider id alone, and take the season from the match you find.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2. Historical data and the leakage problem
&lt;/h2&gt;

&lt;p&gt;A feature leaks when it carries information that did not exist, or that we had not recorded, at the moment the prediction is made. A leaking model looks excellent offline and cannot reproduce that in production.&lt;/p&gt;

&lt;p&gt;We think about three instants for every read:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Instant&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Decision&lt;/td&gt;
&lt;td&gt;When the prediction is made (a horizon from 72 hours to 5 minutes before kick-off)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Event time&lt;/td&gt;
&lt;td&gt;When the thing the row describes happened&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Record time&lt;/td&gt;
&lt;td&gt;When &lt;em&gt;we&lt;/em&gt; stored it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These give two clocks. A &lt;em&gt;publication&lt;/em&gt; clock admits a row whose event time comes before the decision. A &lt;em&gt;knowledge&lt;/em&gt; clock also requires that we had recorded it by then. Some of the rules are obvious: the result of the match being predicted is never admissible. Others are not. A weather feature may only be the &lt;strong&gt;forecast&lt;/strong&gt; that existed at the decision. The measured weather the match was played in is knowable six hours after kick-off, and training on it while serving a forecast is a train/serve skew. Injury rows have no return date, so "currently injured" has to be bounded by recency. A player's "current club" pointer is overwritten on every sync, so it must never stand in for a past squad.&lt;/p&gt;

&lt;p&gt;A time-bounded read looks like this (simplified):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;team_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;corners&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yellow_cards&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;match_statistics&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;fixtures&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;match_id&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'FINISHED'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kickoff_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;updated_at&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kickoff_at&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="kt"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'3 hours'&lt;/span&gt;  &lt;span class="c1"&gt;-- final, not in-play&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nb"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;     &lt;span class="c1"&gt;-- knowledge clock&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;CAST&lt;/code&gt; is there for a reason. PostgreSQL infers an untyped bind parameter from its neighbour, so &lt;code&gt;:decision + interval '21 days'&lt;/code&gt; comes back as an interval and fails at runtime. So does &lt;code&gt;(:x IS NULL OR col = :x)&lt;/code&gt; once a caller passes null.&lt;/p&gt;

&lt;p&gt;The rules are not just a document. A catalogue in code lists every leakage rule, and an architecture test fails the build if a new loader query loses its time bound or if a feature source is not classified. Every training run writes a leakage audit first, and the gate refuses to train on a hash-based or out-of-order split. Stages that leak knowingly are quarantined with a reason and a probe, not hidden.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Feature engineering
&lt;/h2&gt;

&lt;p&gt;The V1 feature vector has around 580 features: rolling form windows (season, last 10, last 5, venue splits), opponent-adjusted and time-decayed rates, expected goals, Elo ratings, rest and congestion, availability, and market features built from de-vigged prices.&lt;/p&gt;

&lt;p&gt;One engineering habit matters more than any single feature.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Version the calculation separately from the schema.&lt;/strong&gt; Adding features bumps a calculation version, and the regeneration pass rewrites vectors in place. Re-keying the stored vectors instead would have stranded every live prediction until regeneration caught up.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prices need care too. A bookmaker's "Home / Draw / Away" labels are also used for Asian handicaps, half-time markets and corners 1X2, and over/under is quoted across many lines. Any "latest price per bookmaker" key must include both market and line. Without them, the query usually picks the &lt;em&gt;wrong&lt;/em&gt; price instead of blending it.&lt;/p&gt;

&lt;p&gt;To remove the bookmaker margin from three-way markets we use the power method rather than proportional normalisation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;implied_i = 1 / odds_i              # sums to more than 1: that excess is the margin
find k by bisection such that  Σ implied_i^k = 1
fair_i   = implied_i^k
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Proportional de-vigging leaves the book's longshot loading in place and creates a false favourite–longshot edge. Two-way markets stay proportional, because there the power method measured neutral or worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Simulation: Poisson, over-dispersion and shared samples
&lt;/h2&gt;

&lt;p&gt;Each match is played out many times in a Monte Carlo engine. Each side's goal rate λ combines attack and defence strengths, a league base rate, and a home-advantage ratio. It is then anchored to the de-vigged market total and to an Elo-based balance between the sides. The anchors do not commute: market-then-Elo and Elo-then-market give the same total but a different split. A test that asserted otherwise failed straight away.&lt;/p&gt;

&lt;p&gt;Real goal counts are more variable than a pure Poisson allows, so the main engine adds a Gamma mixture, which gives a negative binomial:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rate  ~ Gamma(shape = k, scale = λ / k)   # mean λ
goals ~ Poisson(rate)
# marginal: negative binomial, mean λ, variance λ(1 + λ/k)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Things we learned the hard way:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Size truncation for the distribution you actually use.&lt;/strong&gt; Twenty-five goals a side is generous for a Poisson but not for a negative binomial at high λ. The grid now grows adaptively to a tolerance and reports the probability mass it discarded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shuffle samples allocated from a grid.&lt;/strong&gt; Allocating an exact score grid into samples emits them in probability order. If anything later pairs index &lt;em&gt;i&lt;/em&gt; across arrays, sorted order quietly creates a false correlation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monte Carlo noise is real.&lt;/strong&gt; An internal audit found that at 10,000 iterations, the "most likely score" for one fixture came out as 0–1 on 86 seeds out of 100 and 0–2 on the other 14, and the stored run held the 0–2. The fix is to compute the score grid analytically where you can, or to sample far more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One world per fixture.&lt;/strong&gt; The same audit found that reading different markets from separate runs gave inconsistent answers. The Prediction Set engine therefore draws one shared sample set per match and settles every selection against it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Dixon–Coles, the low-score correction, exists in the codebase but is &lt;strong&gt;off&lt;/strong&gt;. On our corpus it made log-loss worse, probably because it fights the over-dispersion over the 1–1 cell. We do not use a bivariate Poisson.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Machine-learning models and calibration
&lt;/h2&gt;

&lt;p&gt;Five prediction types (match result, over/under 2.5 goals, both teams to score, corners and cards) each have a model. Where a market price exists, the served probability blends the learner's output with the de-vigged price (the blend weight is fitted on validation log-loss) and then calibrates it with vector scaling or Platt scaling.&lt;/p&gt;

&lt;p&gt;The order matters: &lt;strong&gt;blend, then calibrate&lt;/strong&gt;. A calibrator only works on the distribution it was fitted to, so it has to be fitted on the &lt;em&gt;blended&lt;/em&gt; validation probabilities.&lt;/p&gt;

&lt;p&gt;We publish the result. Every pre-kick-off prediction is checked against the final score on a public record. As of 16 November 2026 the record had checked 26,725 predictions since 5 September. For match result, the average stated chance was 52.7% and the pick happened 53.3% of the time. Calibration metrics have their own trap: one of ours called "ECE" was really the mean |confidence − correct|, on which a perfectly calibrated 1X2 model scores about 0.5. The corrected metric got a new name, not a new meaning under the old one.&lt;/p&gt;

&lt;p&gt;The honest headline: measured against the closing price, our models reached &lt;strong&gt;parity&lt;/strong&gt; and none beats it. We say so because it is the result, and because it shapes the product. Slipio explains probabilities. It does not promise an edge.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. The Scanner: evidence with a baseline
&lt;/h2&gt;

&lt;p&gt;The Scanner checks every upcoming fixture for patterns: goals and both-teams-to-score rates, form streaks, corners, cards and referee tendencies, player shots and cards, line-up changes and availability. Its central idea is that &lt;strong&gt;a frequency means nothing without a baseline&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A goals pattern qualifies only if the combined rate is at least 20 points above the same pattern's rate in that competition over its last 150 finished matches with the statistic known, each team is at least 10 points above it, and the combined rate is at least 50%. Every insight is then ranked by one formula:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rank     = signal × sampleReliability × freshness × (0.7 + 0.3 × dataConfidence) × certainty
strength = round(100 × rank)    # "insight strength", never a probability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Small-data traps are handled explicitly. A 0–0 match with zero cards is treated as &lt;em&gt;unknown&lt;/em&gt; rather than "no bookings", because some feeds report goals but not bookings. Only provider-confirmed line-ups with eleven starters count as "official". A whole scan is about fifteen bulk SQL statements (&lt;code&gt;LATERAL&lt;/code&gt; joins over &lt;code&gt;unnest(uuid[])&lt;/code&gt;), cached per window with stale-while-revalidate and single flight. The UI calls these patterns history, not predictions, and shows the sample behind each one.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Building a Prediction Set
&lt;/h2&gt;

&lt;p&gt;A Prediction Set combines several selections. Two pieces of code make it honest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Settlement-aware maths.&lt;/strong&gt; Lines settle the Asian way: half lines win or lose, whole lines can push (stake returned), and quarter lines split into two half-stakes. Expected values go through the same settlement code the result checker uses, never through win-or-bust arithmetic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correlation from shared samples.&lt;/strong&gt; Same-match selections are evaluated together on each simulated match, so "over 2.5 goals" plus "both teams to score" correctly comes out more likely than the product of the two. Contradictory selections come out at zero.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A rules layer decides what may share a set at all: no match result together with double chance, one outcome per market, and no correct score alongside a market the score already decides. Quoted prices go through a plausibility filter first, because the odds feed contains stale outlier quotes.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Retrieval and context, honestly scoped
&lt;/h2&gt;

&lt;p&gt;Slipio stores news and match reports, embeds them (a 1024-dimension multilingual model served locally) and keeps the vectors in Qdrant. The rule is simple: &lt;strong&gt;structured platform data is authoritative, and retrieved documents are context&lt;/strong&gt;. They can be quoted and dated. They never change a number. A stricter knowledge index adds as-of bounds and version checks. That index is still an internal preview, so we do not present it as a product feature.&lt;/p&gt;

&lt;p&gt;Where Slippy, the in-app assistant, uses a language model to write prose, a guard drops any number that is not in the facts it was given. The language model never computes a probability.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Agent pipeline orchestration
&lt;/h2&gt;

&lt;p&gt;Slipio Agents (on eligible plans) run a fixed pipeline on a schedule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Fixture Scout → Scanner Analyst → Price Validator → Simulation Analyst
             → Rules Guardian → Set Architect → Prediction Set
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;No step is a language model.&lt;/strong&gt; Each one is a deterministic service with bounded settings, and the stages run in sequence even where the canvas draws a fork. Every run executes an immutable, saved version of the agent's settings, so a later edit never reaches a run in progress. The runner reports append-only stage events with sequence numbers, and clients poll them. The node-state rule is strict: a node's state comes only from reported events, never from elapsed time. The simulation step reads the normal pipeline's numbers, but it runs its baseline inside a rollback-only transaction, so automated runs never add rows to the public prediction record.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Freshness and performance: the partition lesson
&lt;/h2&gt;

&lt;p&gt;Odds history is our largest table: tens of millions of rows, range-partitioned by season. A query that filters by match id alone gives the planner no way to prune partitions, so it scans all of them in parallel. One fixtures-with-odds endpoint took more than five minutes. Adding &lt;code&gt;season = m.season&lt;/code&gt; to a correlated subquery does &lt;strong&gt;not&lt;/strong&gt; fix it, because the value is not a constant at plan time. What works is a literal season plus a bounded list of ids, sent in chunks.&lt;/p&gt;

&lt;p&gt;Then the data grew again. Recent fixtures carry tens of thousands of quote rows each, so a 100-id chunk became millions of rows, and the planner switched to a sequential scan. A read that took 62 seconds dropped to 143 ms with 10-id chunks. &lt;strong&gt;Size chunks by rows, not ids&lt;/strong&gt;, and re-measure with &lt;code&gt;EXPLAIN (ANALYZE, BUFFERS)&lt;/code&gt; whenever capture rates change.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Our models are at parity with the closing price, not ahead of it.&lt;/li&gt;
&lt;li&gt;Some context, such as team news and line-ups, is shown beside the probabilities. Showing it does not mean the probability was adjusted for it. Our rule is that a signal the models train on is a feature and must not also be applied as a simulation modifier, which would count it twice.&lt;/li&gt;
&lt;li&gt;Corners and cards are noisier than goals. On corners, a log-loss difference under about 0.02 is inside the noise a deterministic learner shows from an ordinary data refresh. Find a market's noise band before calling a difference a regression.&lt;/li&gt;
&lt;li&gt;Probabilities are estimates: a quarter of the outcomes rated 75% should not happen.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The most useful engineering decision we made was to measure ourselves in public and write down every time we were wrong.&lt;/p&gt;




&lt;blockquote&gt;
&lt;p&gt;Slipio is for over-18s. Predictions are estimates, not certainties, and never betting advice. Slipio takes&lt;br&gt;
no bets and never takes money on what happens in a match.&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>java</category>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
