<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: David Mashiah</title>
    <description>The latest articles on DEV Community by David Mashiah (@mashiah).</description>
    <link>https://dev.to/mashiah</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4219114%2Fe189016f-f25e-496b-a31a-25ddbba52439.jpeg</url>
      <title>DEV Community: David Mashiah</title>
      <link>https://dev.to/mashiah</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mashiah"/>
    <language>en</language>
    <item>
      <title>Censored demand, worked through ten days of numbers</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Sat, 21 Nov 2026 15:42:47 +0000</pubDate>
      <link>https://dev.to/mashiah/censored-demand-worked-through-ten-days-of-numbers-2n3n</link>
      <guid>https://dev.to/mashiah/censored-demand-worked-through-ten-days-of-numbers-2n3n</guid>
      <description>&lt;p&gt;&lt;em&gt;When you run out of stock, your sales record stops measuring demand. Ten days of numbers show by how much, which part of demand you can still measure exactly, and which part your data can never show you.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Sales and demand are joined by one line: &lt;code&gt;sales = min(demand, stock)&lt;/code&gt;. On a day you run out, the sales figure records your stock, not the market. Statisticians call that a right-censored observation: the true value is at least what you recorded, and you do not know how much more.&lt;/p&gt;

&lt;p&gt;I wrote about why this happens and how it feeds itself in &lt;a href="https://davidmashiah.com/notes/the-sale-you-did-not-make" rel="noopener noreferrer"&gt;a note about the sale you did not make&lt;/a&gt;. This one works it through with numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ten days, ten units on the shelf
&lt;/h2&gt;

&lt;p&gt;Say true demand over ten days was &lt;code&gt;6, 9, 12, 8, 15, 7, 11, 10, 13, 9&lt;/code&gt;. Its average is exactly 10. You stock 10 units every morning.&lt;/p&gt;

&lt;p&gt;Recorded sales are &lt;code&gt;6, 9, 10, 8, 10, 7, 10, 10, 10, 9&lt;/code&gt;. Their average is 8.9. The record understates demand by 11 percent, and 11 units of demand went unserved without leaving a trace.&lt;/p&gt;

&lt;p&gt;Five days show sales of exactly 10. On four of them demand was higher. On one, demand was exactly 10. In the sales record those five days are identical. Nothing tells you which ones hid more demand.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you can still measure exactly
&lt;/h2&gt;

&lt;p&gt;The bottom of the distribution survives. Censoring only cuts values off at the stock level, so every question about demand below 10 has the same answer in the sales record as in the true demand.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;P(demand ≤ 7)&lt;/th&gt;
&lt;th&gt;0.2 true, 0.2 from sales&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;P(demand ≤ 8)&lt;/td&gt;
&lt;td&gt;0.3 true, 0.3 from sales&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;P(demand ≤ 9)&lt;/td&gt;
&lt;td&gt;0.5 true, 0.5 from sales&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;P(demand ≤ 10)&lt;/td&gt;
&lt;td&gt;0.6 true; the sales record cannot tell&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The average is wrong, but the shape of demand below the stock level is exactly right. That turns out to matter more than the average.&lt;/p&gt;

&lt;h2&gt;
  
  
  The stock level you actually need
&lt;/h2&gt;

&lt;p&gt;For a product that spoils, the classical answer is the newsvendor rule. Write &lt;em&gt;Cu&lt;/em&gt; for the cost of being one unit short, the margin you did not earn, and &lt;em&gt;Co&lt;/em&gt; for the cost of one unit left over. Stock the amount where the probability of covering demand equals &lt;code&gt;Cu / (Cu + Co)&lt;/code&gt;. Where that rule comes from is worked through in &lt;a href="https://davidmashiah.com/notes/newsvendor-model" rel="noopener noreferrer"&gt;a note on the newsvendor model&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Low margin first. Say each unit costs 6 and sells for 10: &lt;em&gt;Cu&lt;/em&gt; = 4, &lt;em&gt;Co&lt;/em&gt; = 6, ratio 0.4. You need the smallest stock that covers demand at least 40 percent of the time. &lt;code&gt;P(demand ≤ 8) = 0.3&lt;/code&gt; is not enough and &lt;code&gt;P(demand ≤ 9) = 0.5&lt;/code&gt; is, so the answer is 9. The censored record gives 9 as well. For this product, the biased data still leads to the right decision.&lt;/p&gt;

&lt;p&gt;Now high margin. Say each unit costs 3 and sells for 10: &lt;em&gt;Cu&lt;/em&gt; = 7, &lt;em&gt;Co&lt;/em&gt; = 3, ratio 0.7. From the true demand, &lt;code&gt;P(demand ≤ 10) = 0.6&lt;/code&gt; is not enough and &lt;code&gt;P(demand ≤ 11) = 0.7&lt;/code&gt; is, so you should stock 11. The censored record cannot show that. It has never seen demand above 10, because you never had more than 10. You stay at 10, keep selling out, and the data keeps agreeing with you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why that is the trap
&lt;/h2&gt;

&lt;p&gt;Censoring does not hurt every product equally. It hurts the ones where the right stock level sits above what you have been stocking, which are the products where running out is most expensive. Those are exactly the ones a sales-based forecast keeps under-ordering.&lt;/p&gt;

&lt;p&gt;There are two ways out. The first is to record the requests you could not fill: date, item, quantity. Then the sell-out days stop being censored. The second is to stock above your usual level on purpose now and then, to see the demand you have been cutting off. The research on learning from censored demand does essentially that, adjusting stock as it learns rather than trusting the sales record alone.&lt;/p&gt;

&lt;p&gt;The same bias reaches the score of any forecast tested on that history. A stockout day records a low sale as low demand, so a forecast that copies the stockouts looks accurate. &lt;a href="https://github.com/davidmashiah/forelint" rel="noopener noreferrer"&gt;forelint&lt;/a&gt;, the library I wrote for auditing forecasts, can leave stockout days out of scoring with &lt;code&gt;exclude="stockout"&lt;/code&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;Huh, W. T. and Rusmevichientong, P. A Nonparametric Asymptotic Analysis of Inventory Planning with Censored Demand. Mathematics of Operations Research 34(1), 2009.&lt;/li&gt;
&lt;li&gt;Huh, W. T., Levi, R., Rusmevichientong, P. and Orlin, J. B. Adaptive Data-Driven Inventory Control with Censored Demand Based on Kaplan-Meier Estimator. Operations Research 59(4), 2011.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/censored-demand-by-hand" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>math</category>
      <category>analytics</category>
      <category>statistics</category>
    </item>
    <item>
      <title>Averaging threw the answer away before the model ever saw it</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Fri, 20 Nov 2026 15:43:25 +0000</pubDate>
      <link>https://dev.to/mashiah/averaging-threw-the-answer-away-before-the-model-ever-saw-it-42b9</link>
      <guid>https://dev.to/mashiah/averaging-threw-the-answer-away-before-the-model-ever-saw-it-42b9</guid>
      <description>&lt;p&gt;&lt;em&gt;Twice now I have written about a model that predicts how stiff a perforated sheet is and falls apart on shapes it was not trained on, and twice I treated that as a fact about the model. This time I looked at the quantity it is trained to predict. That quantity is an average, and the average deletes the thing that tells two shapes apart.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Correction, 31 August 2026.&lt;/strong&gt; Version 2 of the deposit below recomputes this study on duplicate-free data and, in its own words, removes the central claim of version 1 — the one this note is built on. The three numbers' advantage over the 2,304-pixel baseline falls from +0.364 at a 100 per cent win rate to +0.045 under a random forest, and is absent on C11. The pixel baseline, which this note reports as predicting nothing, recovers from -0.008 to +0.37 on C22 once the duplicates are gone. What survives is narrower and, I think, more useful: the descriptor's advantage is stability, not magnitude. Held-out R² spans +0.02 to +0.41 for the three numbers against -42.99 to +0.34 for raw pixels — its worst case across the twelve component-and-learner combinations tested is +0.02, where the pixels reach -42.99. Every figure below is from version 1 and has not been re-run here.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Take a metal sheet, punch a repeating pattern of holes in it, and ask how stiff the result is. Answering properly means solving a small physics problem on one repeating tile, which is slow enough that nobody wants to do it inside a design loop. So you train a model to guess the answer from a picture of the tile. It works beautifully on the hole shapes you trained it on and collapses on the ones you did not.&lt;/p&gt;

&lt;p&gt;I have written about that collapse twice. Both times I was, without noticing, assuming the same thing everyone else in this literature assumes: that the failure lives in the model, or in how few shapes it has been shown. Fix the architecture, or buy more families of shape.&lt;/p&gt;

&lt;p&gt;This note is about the other possibility. The number the model is trained to reproduce is computed by taking an average. Averages lose things. If what they lose is the very thing that distinguishes one family of shapes from another, then no model and no quantity of data can recover it, because it was gone before any learning started.&lt;/p&gt;

&lt;p&gt;That turns out to be measurably true, and once you measure the missing part you can hand it to the model as three numbers, which then beat all 2,304 pixels of the picture by a distance.&lt;/p&gt;

&lt;p&gt;The field names, for anyone searching onward: mechanical metamaterials for the sheets, numerical homogenization for the slow calculation, surrogate for the model that replaces it, out-of-distribution generalization for the way these models fail. I will not lean on those words again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What an average is allowed to keep
&lt;/h2&gt;

&lt;p&gt;The stiffness of the patterned sheet comes out of the tile calculation by averaging the internal stresses over the whole tile. That average &lt;em&gt;is&lt;/em&gt; the answer. It is not a summary of the answer computed afterwards; it is the definition.&lt;/p&gt;

&lt;p&gt;Averaging is a projection, which is a formal way of saying it is a doorway with a size limit. Some of what the tile can do passes through and the rest does not, and what does not pass is not arbitrary. The average keeps the two ways the tile can slide bodily as a rigid object. Every other way the material inside the tile can move against itself is projected out.&lt;/p&gt;

&lt;p&gt;Now hold that next to the learning problem. The model is fitted to reproduce the average. If two tiles differ only in a respect the average does not record, the model is never once shown that they differ. More data does not help, because the extra data carries the same blind spot. A better architecture does not help, because it is the target that is blind.&lt;/p&gt;

&lt;p&gt;This is a claim about identifiability rather than about difficulty. It does not say the model is badly built. It says its answer key is missing a column.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does the discarded part exist?
&lt;/h2&gt;

&lt;p&gt;An argument like that is only worth anything if the discarded content is real, and if it separates tiles that the stiffness cannot separate at all. So I went and measured it.&lt;/p&gt;

&lt;p&gt;Same tile, same equations, same periodic boundaries, but with the mass put back in. Instead of asking how the tile responds to being squeezed, ask what it does when it rings: which frequencies, and what shape the motion takes at each one. Two of those shapes are the rigid slides the average keeps. Everything above them is precisely what the average throws out.&lt;/p&gt;

&lt;p&gt;For any one of those shapes the motion splits, exactly and point by point, into three parts: how much of it is stretching, how much is shearing, how much is rotating. Weight each part by where the metal actually is, normalise the three to sum to one, and you have three numbers describing the character of that mode.&lt;/p&gt;

&lt;p&gt;The extractor is checked against the one case with an exact answer. On a uniform sheet the modes are plane waves whose characters are known in closed form, and it returns them to six decimal places. The rigid modes come back at 1e-13 instead of zero. Refining the mesh drops the error by a factor of 16.0, where exact second-order convergence predicts 16. Rotating the solution arbitrarily inside a degenerate pair moves the answer in the fourteenth decimal place. Ten tests, all passing, and the homogenization solver underneath has its own six.&lt;/p&gt;

&lt;p&gt;Then the evidence. Four of the twenty-two families in the benchmark are exactly isotropic in plane: circular holes, square holes, cross-shaped holes, six-pointed star holes. Their symmetry group forces the two stiffnesses to be equal, and they come out equal to machine precision. A model trained on those four has never observed directional stiffness in its target, because there is none to observe.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;family&lt;/th&gt;
&lt;th&gt;stiffness ratio&lt;/th&gt;
&lt;th&gt;rotational fraction of the softest discarded mode&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;square&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.69&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;circle&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.70&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;star&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.76&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;cross&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.89&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Identical in the target, to machine precision, and spanning 0.19 in the thing the target discards, with per-tile distributions that barely overlap. Across the whole pool the rotational fraction runs from about 0.64 to 0.95 and is essentially uncorrelated with the stiffness ratio, which is exactly what you would expect of a quantity the stiffness tensor does not encode.&lt;/p&gt;

&lt;p&gt;So the discarded information exists, and it is directional. That is the same kind of information whose absence broke the model in the first study.&lt;/p&gt;

&lt;h2&gt;
  
  
  three numbers against 2,304
&lt;/h2&gt;

&lt;p&gt;A nice argument. The test of it is whether the discarded content is usable.&lt;/p&gt;

&lt;p&gt;So: take the softest discarded mode, take its three character numbers, add the tile's solid fraction, which is free and says how much metal is left, and give those four numbers to the same model in place of the picture. Train on four families, score on the eighteen it has never seen, and repeat over 120 paired runs with independently drawn family subsets and model seeds, so every arm sees identical subsets and only the features differ.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;features&lt;/th&gt;
&lt;th&gt;numbers in&lt;/th&gt;
&lt;th&gt;cost&lt;/th&gt;
&lt;th&gt;held-out score&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;solid fraction alone&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;free&lt;/td&gt;
&lt;td&gt;+0.082&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;the raw picture&lt;/td&gt;
&lt;td&gt;2,304&lt;/td&gt;
&lt;td&gt;free&lt;/td&gt;
&lt;td&gt;-0.008&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;picture and fraction&lt;/td&gt;
&lt;td&gt;2,305&lt;/td&gt;
&lt;td&gt;free&lt;/td&gt;
&lt;td&gt;+0.019&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fraction and mode character, coarsened&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;0.25&lt;/td&gt;
&lt;td&gt;+0.356&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fraction and mode character, full&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;1.16&lt;/td&gt;
&lt;td&gt;+0.448&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Cost is in units of one physics solve, the thing being predicted. Score is on the scale where zero is what you get by ignoring the input and always answering with the average, so the third row is a model that has been handed a complete description of the tile and is doing worse than a constant.&lt;/p&gt;

&lt;p&gt;Four numbers, three of which come from a calculation costing a quarter of the answer it replaces, beat the free control on 97 of every 100 paired runs.&lt;/p&gt;

&lt;p&gt;The same contrast in one pair of numbers: on tiles drawn from its own training families the picture model scores 0.994, which is the accuracy this literature reports and the reason anyone uses these models at all. On the eighteen unseen families that same model reaches -0.306, and the four-number model reaches +0.508.&lt;/p&gt;

&lt;p&gt;It is not the learner. Both of those are the same learner.&lt;/p&gt;

&lt;h2&gt;
  
  
  not any spectrum, and not symmetry
&lt;/h2&gt;

&lt;p&gt;Two objections had to be closed before I would believe my own result.&lt;/p&gt;

&lt;p&gt;First: perhaps any spectral description of a shape transfers, and elasticity has nothing to do with it. There is published prior art to test that against. The Laplace-Beltrami spectrum, used as shape-DNA, is a rotation-invariant spectral descriptor for exactly this kind of design work, and it is purely geometric. Computed on the same tiles in the same harness it reaches +0.106, and its scale-free ratios reach +0.038. The standard hand-crafted geometric bundle, seven Hu moment invariants plus the radially averaged two-point correlation function, reaches +0.173. The elastic spectrum reaches +0.356. It is not that any spectrum transfers. It is that this one does.&lt;/p&gt;

&lt;p&gt;Second: perhaps this is a symmetry problem, and the answer is an architecture that knows about symmetry. I cannot reimplement one of those, but I can impose the tile's exact symmetry group by brute force: train on all eight mirrored and rotated copies of every tile, average each prediction over the eight copies of the test tile, and transform the target along with it. At eight times the training and inference cost, the median moves by -0.010. The spread narrows and the middle does not move.&lt;/p&gt;

&lt;p&gt;That is a lower bound on what a properly equivariant model would do and not a test of one, and I claim to have outperformed nothing. What it supports is the weaker statement I need: imposing the symmetry does not recover the information, which is what you expect if the information is missing from the target rather than mishandled by the model.&lt;/p&gt;

&lt;p&gt;The sharpest version is against a convolutional network, which is the fair geometric baseline. The network on the image reaches +0.098. Concatenating the four numbers into its head lifts it to +0.179. The four numbers alone, on a plain multilayer perceptron with no image at all, reach +0.362, beating the network on 83 percent of subsets.&lt;/p&gt;

&lt;p&gt;Throwing the image away and keeping the physics is worth more than adding the physics to the image.&lt;/p&gt;

&lt;h2&gt;
  
  
  which of the four numbers does the work
&lt;/h2&gt;

&lt;p&gt;Worth knowing, because this is the part that says what the finding means rather than that it happened.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;features&lt;/th&gt;
&lt;th&gt;count&lt;/th&gt;
&lt;th&gt;held-out score&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;the softest mode's character&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;+0.409&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;all six modes' characters&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;+0.385&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;every descriptor computed&lt;/td&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;td&gt;+0.332&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;the frequency ratios&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;+0.299&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;the softest frequency alone&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;+0.202&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;the six raw frequencies&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;+0.038&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three numbers beat twenty-four. Adding the other twenty-one makes transfer worse, which is the signature of a model absorbing information specific to the families it was trained on.&lt;/p&gt;

&lt;p&gt;The six raw frequencies are nearly worthless and their five ratios are strong. So the transferable quantity is the shape of the spectrum and not its scale: how stiff a tile is in absolute terms is a fact about its family and does not survive leaving it.&lt;/p&gt;

&lt;p&gt;Inside the three, the content is rotational. Rotation alone is worth +0.146 and stretching alone +0.057; rotation and shear together +0.349; all three +0.388. The thing that travels is a ratio, how the tile's softest internal motion divides between rotating and shearing.&lt;/p&gt;

&lt;p&gt;That has a name in mechanics. It is the micropolar, or Cosserat, degree of freedom: the ability of the material inside a cell to spin relative to the cell. The ordinary first-order theory of a continuous material has no slot for it, which is precisely why the averaged stiffness cannot record it.&lt;/p&gt;

&lt;p&gt;And it helps the component it ought to help. On the stiffness component whose transfer collapsed in the first study the three numbers are worth +0.409. On the component that did not collapse they are worth -0.088.&lt;/p&gt;

&lt;h2&gt;
  
  
  what this doesn't fix
&lt;/h2&gt;

&lt;p&gt;Absolute accuracy on unseen shapes is still poor. A score of +0.36 is a large relative improvement on a very low base, not a replacement for the physics solve. The honest description of the four numbers is a cheap diagnostic and a prior.&lt;/p&gt;

&lt;p&gt;Two families are consistently made worse: chevrons and a two-hole cell. The honeycomb that was mispredicted about sixfold in the first study improves from -66 to -16, a fourfold reduction in squared error and still wrong. Sixteen of twenty-two families improve under a forest and fourteen of twenty-two under a neural network, and the two most extreme families are improved without being repaired.&lt;/p&gt;

&lt;p&gt;The clean split between the two stiffness components only holds at full resolution. Everything here is two-dimensional, linear-elastic, one void phase, one stiffness contrast. Nothing shows this is the right descriptor in three dimensions or at large strain.&lt;/p&gt;

&lt;p&gt;And the obvious follow-up failed. If these numbers carry the transferable content, they ought to tell you which families to buy, which is the question left open by the previous study. They do not. The range of rotational fraction across a candidate training set correlates with the outcome at about +0.25 to +0.32, which is weak, and the free control, the range of solid fraction, beats it at +0.40. Knowing which families hold the content is not the same as the model being able to use it.&lt;/p&gt;

&lt;p&gt;The comparison this work does not contain is against a trained equivariant graph network. That is the missing piece, and I would rather say so than let the orbit-averaging experiment stand in for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  the part that isn't about metal
&lt;/h2&gt;

&lt;p&gt;Strip the engineering out and the claim is about labels.&lt;/p&gt;

&lt;p&gt;If your label is a summary statistic, everything the summary discarded is invisible to anything trained on it. Written down, that is obvious. In practice it is not, because the label is the part of the pipeline nobody treats as a variable. It arrives, it is correct, and all the attention goes to the model and the data.&lt;/p&gt;

&lt;p&gt;Three consequences I would actually act on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A benchmark for generalization should state which degrees of freedom its training families hold fixed. A training set that is uniformly symmetric cannot teach a model what asymmetry looks like, however large it is.&lt;/li&gt;
&lt;li&gt;More data inside a design space cannot repair a blindness in the target. In the first study more data and a finer mesh made cross-family transfer &lt;em&gt;worse&lt;/em&gt;, which is the opposite of what a coverage explanation predicts.&lt;/li&gt;
&lt;li&gt;A richer input is not automatically a safer one. The full picture let the model fit its training families almost exactly, and it was that fit which failed to travel. Four physically chosen numbers could not fit as well, and went further.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Everything is open: the extractor and its validation suite, the descriptors at every resolution and coarsening setting, every result file behind the numbers above, and the preprint, at &lt;a href="https://doi.org/10.5281/zenodo.22137052" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.22137052&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;Bastek, J.-H., Kumar, S., Telgen, B., Glaesener, R. N. and Kochmann, D. M. &lt;a href="https://doi.org/10.1073/pnas.2111505119" rel="noopener noreferrer"&gt;Inverting the structure-property map of truss metamaterials by deep learning&lt;/a&gt;. &lt;em&gt;PNAS&lt;/em&gt; 119(1), 2022. Reports scores above 0.98 across all elastic constants, measured inside the design space it was trained on. The accuracy is real; the scope is the point.&lt;/li&gt;
&lt;li&gt;Roberts, A. J. &lt;a href="https://doi.org/10.1093/imatrm/tnaf001" rel="noopener noreferrer"&gt;Accurate families of multi-continuum micromorphic homogenisations&lt;/a&gt;. &lt;em&gt;Transactions of Mathematics and its Applications&lt;/em&gt; 9(1), 2025. Builds richer continuum theories by keeping the sub-cell modes that averaging removes. This note borrows the same modes as an instrument rather than as a model.&lt;/li&gt;
&lt;li&gt;Wang, L., Chan, Y.-C., Liu, Z., Zhu, P. and Chen, W. &lt;a href="https://doi.org/10.1007/s00158-020-02523-5" rel="noopener noreferrer"&gt;Data-driven metamaterial design with Laplace-Beltrami spectrum as shape-DNA&lt;/a&gt;. &lt;em&gt;Structural and Multidisciplinary Optimization&lt;/em&gt; 61, 2020. The geometric spectral descriptor tested here as the decisive control.&lt;/li&gt;
&lt;li&gt;Hendriks, R., Menkovski, V., Doškář, M., Geers, M. G. D. and Rokoš, O. &lt;a href="https://doi.org/10.1016/j.cma.2025.117867" rel="noopener noreferrer"&gt;Similarity-equivariant graph neural networks for homogenization of metamaterials&lt;/a&gt;. &lt;em&gt;CMAME&lt;/em&gt; 439, 2025. The architecture-side answer to the same failure, and the comparison this work does not contain.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/averaging-threw-the-answer-away" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>python</category>
      <category>statistics</category>
    </item>
    <item>
      <title>There was a random number hiding inside my measurement</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Thu, 19 Nov 2026 15:43:36 +0000</pubDate>
      <link>https://dev.to/mashiah/there-was-a-random-number-hiding-inside-my-measurement-2ada</link>
      <guid>https://dev.to/mashiah/there-was-a-random-number-hiding-inside-my-measurement-2ada</guid>
      <description>&lt;p&gt;&lt;em&gt;I had a clean negative result: a physical quantity that works well when computed carefully and stops working when computed cheaply. Useful but unaffordable, and worth reporting. It was wrong. The collapse was a random number my own solver had put there, and the check I ran to catch exactly that class of problem came back clean.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Correction, 31 August 2026.&lt;/strong&gt; Every figure in this note was computed on a dataset later found to be 61.6 per cent duplicated, and 92.3 per cent duplicated at the 24x24 resolution where these descriptors were computed. Version 2 of the deposit recomputes the study on duplicate-free data and removes the central claim of the work this note reports on. The mechanism described here — a degenerate eigenpair making the descriptor part physics and part random seed, and a fidelity check that compared rankings while discrimination was being destroyed — is unaffected, because it is a fact about the solver rather than about the data it ran on. The numbers attached to it are not, and have not been re-run.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a short note about a result I reported, believed, and had to withdraw, and about why the safeguard I had in place did not fire. The engineering is a detail. The failure mode is not.&lt;/p&gt;

&lt;p&gt;The quantity in question describes how a small patterned tile of metal moves when it vibrates: what proportion of the motion is rotating, what proportion is shearing, what proportion is stretching. Three numbers. They are useful because a machine-learning model given them transfers to shapes it has never seen, where the same model given a picture of the tile does not.&lt;/p&gt;

&lt;p&gt;Computing them needs an eigenvalue solve, which costs about as much as the thing it is supposed to be a cheap substitute for. The obvious move is to shrink the tile first, solve on a coarser grid, and see how much of the signal survives.&lt;/p&gt;

&lt;h2&gt;
  
  
  The negative result
&lt;/h2&gt;

&lt;p&gt;I ran it at four resolutions and watched the value fall away.&lt;/p&gt;

&lt;p&gt;At full resolution the three numbers were worth +0.388 on the transfer task. Coarsened to a third of that resolution, which is a tenth of the work, they were worth +0.051. That is below the free control of simply telling the model how much metal is in the tile. Useful at full price, worthless at a discount.&lt;/p&gt;

&lt;p&gt;That is a perfectly publishable shape of finding. It is monotone, it has a plausible mechanism, and it is the kind of limitation you are supposed to report rather than bury. I wrote it up as one.&lt;/p&gt;

&lt;h2&gt;
  
  
  where the random number came from
&lt;/h2&gt;

&lt;p&gt;Most of the tile families in this benchmark have four-fold symmetry. A square has it, a circular hole in a square frame has it, a cross has it. When a shape has that symmetry, its second and third vibration modes come out at the same frequency.&lt;/p&gt;

&lt;p&gt;Two modes at one frequency is not a coincidence to be cleaned up. It means the solver has been asked a question with no single answer. Any mixture of the two is equally a valid mode, and which particular mixture comes back depends on where the iterative solver started, which was a random vector.&lt;/p&gt;

&lt;p&gt;My three numbers were computed from one of those modes, which is to say from a single eigenvector. They are quadratic in it, so they are not invariant to the mixture: rotate the pair against each other and the numbers change. I had built a descriptor whose value was part physics and part the seed of a random number generator, and I had not noticed, because at full resolution the physics part is much the larger of the two.&lt;/p&gt;

&lt;p&gt;Coarsening is what exposed it. A coarser grid pushes the two frequencies closer together, the mixture becomes less determined, and the random component grows. So the descriptor degraded smoothly as I made it cheaper.&lt;/p&gt;

&lt;p&gt;That is the part worth sitting with. The artefact did not look like an artefact. It looked like physics. A quantity that decays as you spend less on it is the most ordinary result in numerical work, and it is the one shape of curve that would never have made me suspicious.&lt;/p&gt;

&lt;h2&gt;
  
  
  the check that passed
&lt;/h2&gt;

&lt;p&gt;I was not careless about the coarsening. I had a fidelity check for exactly this: compute the frequencies on the coarse grid and on the fine one, and correlate them across the whole pool of tiles. If the coarse grid still ranks tiles the way the fine grid does, the coarsening is faithful.&lt;/p&gt;

&lt;p&gt;It scored 0.87 to 0.93. It gave no warning at any resolution.&lt;/p&gt;

&lt;p&gt;And it was telling the truth. The coarsening &lt;em&gt;was&lt;/em&gt; faithful in the sense the check measured. The ranking was preserved. What had been destroyed was the ability to tell apart tiles that sit close together in that ranking, and a correlation cannot see that, because a correlation is a statement about order and the loss was inside the order.&lt;/p&gt;

&lt;p&gt;My check verified a property. It was not the property I depended on.&lt;/p&gt;

&lt;h2&gt;
  
  
  what it looks like fixed
&lt;/h2&gt;

&lt;p&gt;The repair is not subtle once the diagnosis is right. Do not compute the three numbers from one mode. Compute them from every mode at that frequency and average, which is an operation that does not care how the solver split the pair, then start the solver from a fixed vector so the run is reproducible. Separately, coarsen by averaging the metal in each block rather than by thresholding it, because thresholding erases thin struts and thin struts are most of what these tiles are.&lt;/p&gt;

&lt;p&gt;The invariance is now checked directly: rotate the mixture at random inside the degenerate pair and the answer moves in the fourteenth decimal place.&lt;/p&gt;

&lt;p&gt;Under the old definition the three numbers fell from +0.388 at full resolution to +0.051 at a third of it, below the free control. Under the corrected definition, taken with that free control alongside them, the curve does not fall off at all:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;eigensolve resolution&lt;/th&gt;
&lt;th&gt;cost, in physics solves&lt;/th&gt;
&lt;th&gt;held-out score&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;full&lt;/td&gt;
&lt;td&gt;1.16&lt;/td&gt;
&lt;td&gt;+0.448&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;half&lt;/td&gt;
&lt;td&gt;0.25&lt;/td&gt;
&lt;td&gt;+0.356&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a third&lt;/td&gt;
&lt;td&gt;0.11&lt;/td&gt;
&lt;td&gt;+0.246&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The conclusion inverts. The finding is not that the physics is unaffordable. It is that the physics is cheap, and I had been measuring my own solver's starting vector.&lt;/p&gt;

&lt;p&gt;The superseded version of the extractor ships in the deposit alongside the corrected one, so anyone can rerun the comparison and watch the bad curve appear.&lt;/p&gt;

&lt;h2&gt;
  
  
  the general version
&lt;/h2&gt;

&lt;p&gt;Two things I would take out of this into work that has nothing to do with metal.&lt;/p&gt;

&lt;p&gt;The first is about ties. If a quantity you compute is defined only up to a choice, and some library makes that choice arbitrarily on your behalf, then part of what you have measured is the library. Degenerate eigenvalues are one instance. So is an argmax over equal scores, a sort that is not stable, a cluster label, an unseeded initialisation. The tell is that the ambiguity is usually invisible while the signal is strong, and only surfaces as noise when you make the problem harder, which is the same moment you are most willing to believe you have found a real limit.&lt;/p&gt;

&lt;p&gt;The second is about checks. A validity check is a claim about one property, and it is easy to pick one that is cheap to compute rather than the one your conclusion rests on. Mine compared rankings when what I needed was discrimination. It passed honestly and told me nothing, which is worse than having no check at all, because a check that passes buys you the confidence to stop looking.&lt;/p&gt;

&lt;p&gt;The three numbers, the corrected extractor, the superseded one, and every result file are here — now at version 2, recomputed on duplicate-free data: &lt;a href="https://doi.org/10.5281/zenodo.22137052" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.22137052&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;Roberts, A. J. &lt;a href="https://doi.org/10.1093/imatrm/tnaf001" rel="noopener noreferrer"&gt;Accurate families of multi-continuum micromorphic homogenisations in multi-D space-time via dynamical systems theory&lt;/a&gt;. &lt;em&gt;Transactions of Mathematics and its Applications&lt;/em&gt; 9(1), 2025. The sub-cell modes this descriptor is built on. Anyone building features from them on a symmetric cell will meet the same degeneracy.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/a-random-number-in-my-measurement" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>python</category>
      <category>statistics</category>
    </item>
    <item>
      <title>A scratched label still scans, and that is not luck</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Tue, 17 Nov 2026 15:43:13 +0000</pubDate>
      <link>https://dev.to/mashiah/a-scratched-label-still-scans-and-that-is-not-luck-2k7f</link>
      <guid>https://dev.to/mashiah/a-scratched-label-still-scans-and-that-is-not-luck-2k7f</guid>
      <description>&lt;p&gt;&lt;em&gt;Seven months moving flowers to shops taught me that a label is a physical object before it is data. It gets wet, creased, dragged across a crate edge. Most of them still scanned, and I assumed the scanner was being clever. It is not. Somebody decided in advance how much of that label they were willing to lose.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Seven months moving flowers to shops taught me that a label is a physical object before it is data.&lt;/p&gt;

&lt;p&gt;It gets wet, because the product it is stuck to has to stay wet. It gets creased when the case is stacked under something heavier. It gets dragged across a crate edge on the way off the truck. By the time a case reached a shop it had been handled six or seven times and the label showed it.&lt;/p&gt;

&lt;p&gt;Most of them still scanned. I assumed the scanner was doing something clever with a blurry picture.&lt;/p&gt;

&lt;p&gt;It is not. The clever part happened before the label was printed. The symbol is carrying spare parity, deliberately, and how much it carries is a decision somebody made when they chose which kind of symbol to print. Choose one way and a damaged label tells you it is damaged. Choose the other way and it tells you what it was supposed to say.&lt;/p&gt;

&lt;p&gt;Those are different products, and most of a supply chain buys the first one without noticing there was a choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  A check digit only ever says no
&lt;/h2&gt;

&lt;p&gt;Walk the carriers on a single pallet and count what each one can actually do.&lt;/p&gt;

&lt;p&gt;The retail unit carries an EAN-13, which ends in one modulo-10 check digit. The shipping case carries an ITF-14, also one modulo-10 check digit. The logistics label on the pallet carries a GS1-128, which is Code 128 underneath, and Code 128 protects itself with a single modulo-103 check character.&lt;/p&gt;

&lt;p&gt;Every one of those detects. None of them corrects. A failed check produces the beep that is not the good beep, and the recovery mechanism is a person turning the case around and trying again.&lt;/p&gt;

&lt;p&gt;That is a perfectly good design while a person is standing there. It stops being a good design the moment the read is supposed to happen without one, which is the direction every warehouse is moving.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Reed–Solomon buys instead
&lt;/h2&gt;

&lt;p&gt;Irving Reed and Gustave Solomon published the construction in 1960, in a paper called &lt;em&gt;Polynomial Codes over Certain Finite Fields&lt;/em&gt;. The idea is small enough to state in a sentence.&lt;/p&gt;

&lt;p&gt;Take your &lt;strong&gt;k&lt;/strong&gt; data symbols and treat them as the coefficients of a polynomial over a finite field. Evaluate that polynomial at &lt;strong&gt;n&lt;/strong&gt; different points and ship all &lt;strong&gt;n&lt;/strong&gt; results. A polynomial of degree less than &lt;strong&gt;k&lt;/strong&gt; is pinned down by any &lt;strong&gt;k&lt;/strong&gt; of its values, so any &lt;strong&gt;k&lt;/strong&gt; symbols that survive are enough to rebuild the original.&lt;/p&gt;

&lt;p&gt;You spent &lt;strong&gt;n − k&lt;/strong&gt; symbols on parity. In return you can repair up to &lt;strong&gt;(n − k) / 2&lt;/strong&gt; damaged symbols when you do not know which ones are damaged, and up to &lt;strong&gt;n − k&lt;/strong&gt; when you do. Reed–Solomon codes are maximum distance separable, meaning their minimum distance is exactly &lt;strong&gt;n − k + 1&lt;/strong&gt;. That is the best any code can do for that overhead, and it is a proved bound rather than a property of somebody's implementation.&lt;/p&gt;

&lt;p&gt;The load-bearing word is &lt;strong&gt;symbol&lt;/strong&gt;. The code counts damaged symbols, not damaged bits. Eight bad bits sitting inside one byte cost one symbol. The same eight bad bits spread across eight bytes cost eight.&lt;/p&gt;

&lt;p&gt;Physical damage on a label is contiguous. A water drop, a crease, a forklift scrape: each takes out a patch, and a patch maps to a short run of adjacent symbols. Symbol-oriented correction is matched to the way labels actually die. That is not a coincidence, and it is why the 2D symbols use this family rather than bolting a second check digit onto a linear one.&lt;/p&gt;

&lt;p&gt;The whole mechanism, worked by hand with four numbers mod 7, has &lt;a href="https://davidmashiah.com/notes/reed-solomon-by-hand" rel="noopener noreferrer"&gt;its own note&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four levels on a QR code are a commercial decision
&lt;/h2&gt;

&lt;p&gt;QR codes, specified in ISO/IEC 18004, carry Reed–Solomon parity over GF(256): 256 symbols, one byte each. The spec exposes the overhead as a dial with four settings.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;L&lt;/th&gt;
&lt;th&gt;About 7 percent of codewords recoverable. The default in most generators, and the wrong default for anything that will be handled.&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;M&lt;/td&gt;
&lt;td&gt;About 15 percent. The usual compromise for print that stays indoors and flat.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q&lt;/td&gt;
&lt;td&gt;About 25 percent.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H&lt;/td&gt;
&lt;td&gt;About 30 percent. The setting for a label that will get wet, abraded, or partly covered by another label.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every step up that list buys resilience with data capacity. More parity in a symbol of fixed size means fewer bytes of payload; the same payload at a higher level means a physically larger symbol. On a sticker that has to fit on the side of a wet bucket, that is a real trade with a real cost, and it is usually made by whoever configured the label printer rather than by anyone who has watched the cases come back.&lt;/p&gt;

&lt;p&gt;There is a detail in the QR spec worth borrowing. The field that records which correction level was used is itself protected, by a separate (15, 5) BCH code. The metadata has to survive before the payload can be decoded at all, so it gets its own code sized for its own job. The layering is deliberate.&lt;/p&gt;

&lt;p&gt;Data Matrix takes the other approach. Its current version, ECC 200 in ISO/IEC 16022, does not offer a dial: parity is fixed per symbol size. GS1 DataMatrix, which is GS1's own profile of that symbology and is spelled as one word to distinguish it, is what pharmaceutical serialisation runs on under the European Falsified Medicines Directive. That is a good sanity check on all of the above. A pack marking that has to stay readable through distribution, storage and handling all the way to a dispensing counter is a 2D symbol with real correction, not a linear one with a check digit.&lt;/p&gt;

&lt;h2&gt;
  
  
  RFID is a detection story, and I have been sloppy about this
&lt;/h2&gt;

&lt;p&gt;The shorthand you hear, and that I have used loosely myself, is that error-correcting codes are what make barcodes and RFID reliable. The first half is right. The second half is not, and the difference matters operationally.&lt;/p&gt;

&lt;p&gt;EPC UHF Gen2, standardised as ISO/IEC 18000-63, protects its air-interface messages with cyclic redundancy checks: a CRC-5 on the query, a CRC-16 on the packet. A CRC detects. It does not repair.&lt;/p&gt;

&lt;p&gt;The recovery mechanism there is retransmission. A read that fails its check is thrown away and the interrogator asks again, and because it is asking hundreds of times a second, throwing one away is nearly free.&lt;/p&gt;

&lt;p&gt;So the honest version is that the 2D symbols on your cases &lt;strong&gt;repair&lt;/strong&gt; damage, and the RFID layer &lt;strong&gt;detects&lt;/strong&gt; damage and gets another go. Both hand you an identifier you can act on. They fail differently, though, and the shorthand hides exactly the way the second one fails: it depends on there being another chance. A tag that passes through the field once, at speed, at a bad angle, behind a pallet of something wet, does not get another chance. There is no parity waiting to reconstruct what that read should have been.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I built one over 32 symbols instead of 256
&lt;/h2&gt;

&lt;p&gt;Barcodes work over GF(256) because their symbol is a byte, and a byte is what the payload already is. That is inheritance, not optimisation.&lt;/p&gt;

&lt;p&gt;The toolkit I published works over GF(25), which is 32 symbols, because the data was not bytes. It was sequence data over a small alphabet: four nucleotides, or twenty amino acids. Forcing a twenty-letter alphabet through byte-sized arithmetic leaves most of the field unused and makes every parity symbol cost far more than the information it is protecting.&lt;/p&gt;

&lt;p&gt;GF(32) holds a twenty-letter alphabet in one symbol with room to spare. Same construction, same distance bound, field size chosen by the alphabet rather than inherited from a machine word.&lt;/p&gt;

&lt;p&gt;That is the part that transfers, and it is the reason I think this belongs on a site about distribution rather than only in a repository. Reed–Solomon is not a barcode feature. It is a parameterised construction with two questions attached: what is my alphabet, and what does my damage look like? Answer those honestly and the parameters follow. Answer them by accepting a library default and you have made the decision anyway, just without knowing what you chose.&lt;/p&gt;

&lt;p&gt;Both outputs are open: the encoding work at &lt;a href="https://doi.org/10.6084/m9.figshare.32885588" rel="noopener noreferrer"&gt;https://doi.org/10.6084/m9.figshare.32885588&lt;/a&gt; and the toolkit with the error correction in it at &lt;a href="https://doi.org/10.6084/m9.figshare.32963456" rel="noopener noreferrer"&gt;https://doi.org/10.6084/m9.figshare.32963456&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What none of it fixes
&lt;/h2&gt;

&lt;p&gt;Correction protects a symbol between the printer and the scanner. It has no opinion whatsoever about whether the number was right when it was printed.&lt;/p&gt;

&lt;p&gt;The failure I actually saw on a loading bay was almost never a corrupted barcode. It was a case wearing the correct, intact, perfectly readable label for the wrong contents. Or a pallet built from a pick list that had been amended out loud and never written down. Or two orders for the same shop consolidated by someone who was right about it, and one order later that was moved by someone who was not.&lt;/p&gt;

&lt;p&gt;Parity will reconstruct a damaged wrong identifier into a pristine wrong identifier and hand it over with no flag on it at all. The confidence is the dangerous part. A clean scan reads as a settled question, and downstream nothing ever asks it again.&lt;/p&gt;

&lt;p&gt;Data integrity and data correctness are separate properties. A chain can have solved the first one completely, all the way down to proved distance bounds on the symbols, and still be wrong about what is on the truck. The mathematics is silent on that, and it is the more expensive of the two problems.&lt;/p&gt;

&lt;p&gt;The scratched label scanning is not luck. It is somebody's decision about how much of the symbol they were prepared to lose, made once, upstream, probably by default. Whether the number on it was ever true is a different question, and nothing in the code answers it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The GF(25) encoding and Reed–Solomon toolkit this note draws on is open: &lt;a href="https://doi.org/10.6084/m9.figshare.32963456" rel="noopener noreferrer"&gt;https://doi.org/10.6084/m9.figshare.32963456&lt;/a&gt;. This page at &lt;a href="https://davidmashiah.com/notes/a-scratched-label-still-scans" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt; is the canonical version.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reed, I. S. and Solomon, G. &lt;a href="https://doi.org/10.1137/0108018" rel="noopener noreferrer"&gt;Polynomial Codes Over Certain Finite Fields&lt;/a&gt;. &lt;em&gt;Journal of the Society for Industrial and Applied Mathematics&lt;/em&gt; 8(2), 1960. The original construction, and the source of the two-parity-symbols-per-corrected-error result this note leans on.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.iso.org/standard/83389.html" rel="noopener noreferrer"&gt;ISO/IEC 18004:2024&lt;/a&gt;, &lt;em&gt;QR code bar code symbology specification&lt;/em&gt;, 4th edition. Defines the four error-correction levels and their recovery capacities.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gs1.org/standards/barcodes-epcrfid-id-keys/gs1-general-specifications" rel="noopener noreferrer"&gt;GS1 General Specifications&lt;/a&gt;. The symbology and check-digit rules for the linear barcodes discussed here, including the single-digit check that can only ever reject.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/a-scratched-label-still-scans" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>algorithms</category>
      <category>math</category>
      <category>logistics</category>
      <category>barcodes</category>
    </item>
    <item>
      <title>Cross-validation, and the split that hides a model's failure</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Sun, 15 Nov 2026 16:03:47 +0000</pubDate>
      <link>https://dev.to/mashiah/cross-validation-and-the-split-that-hides-a-models-failure-1hbc</link>
      <guid>https://dev.to/mashiah/cross-validation-and-the-split-that-hides-a-models-failure-1hbc</guid>
      <description>&lt;p&gt;&lt;em&gt;Cross-validation is how most models get their score. Six points worked through by hand show how k-fold works, and how the same model scores 0.80 or −0.67 depending only on how the data was split.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A model has to be scored on data it was not trained on, or the score only measures memory. Holding back one test set works, but with little data the score then &lt;a href="https://davidmashiah.com/notes/train-test-split" rel="noopener noreferrer"&gt;depends heavily&lt;/a&gt; on which rows landed in it. Cross-validation fixes that by testing on every row exactly once.&lt;/p&gt;

&lt;p&gt;It also has a failure that does not show in the number. I ran into it in my research, which I wrote about in &lt;a href="https://davidmashiah.com/notes/my-model-scored-perfect" rel="noopener noreferrer"&gt;a note on a model that scored perfectly&lt;/a&gt;. This note works both parts through by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  How k-fold cross-validation works
&lt;/h2&gt;

&lt;p&gt;Split the data into &lt;code&gt;k&lt;/code&gt; equal parts, called folds. Train on &lt;code&gt;k − 1&lt;/code&gt; of them and test on the one left out. Repeat &lt;code&gt;k&lt;/code&gt; times, so each fold is the test set once, and combine the &lt;code&gt;k&lt;/code&gt; test results into one score.&lt;/p&gt;

&lt;p&gt;Every row is used for testing once and for training &lt;code&gt;k − 1&lt;/code&gt; times. Five and ten are the usual choices for &lt;code&gt;k&lt;/code&gt;. With &lt;code&gt;k&lt;/code&gt; equal to the number of rows it is called leave-one-out.&lt;/p&gt;

&lt;h2&gt;
  
  
  Six points, three folds
&lt;/h2&gt;

&lt;p&gt;The true relationship is &lt;code&gt;y = x²&lt;/code&gt;, and there are six measurements in two groups. Group A: &lt;code&gt;x = 0, 0.5, 1&lt;/code&gt;. Group B: &lt;code&gt;x = 2, 2.5, 3&lt;/code&gt;. The model is a straight line fitted by least squares, and the score is R², where 1 is perfect and 0 means no better than always answering with the average.&lt;/p&gt;

&lt;p&gt;Ordinary 3-fold cross-validation, with the rows dealt into folds without regard to group: fold 1 holds &lt;code&gt;x = 0&lt;/code&gt; and &lt;code&gt;2.5&lt;/code&gt;, fold 2 holds &lt;code&gt;0.5&lt;/code&gt; and &lt;code&gt;3&lt;/code&gt;, fold 3 holds &lt;code&gt;1&lt;/code&gt; and &lt;code&gt;2&lt;/code&gt;. Every fold mixes one point from each group.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test fold&lt;/th&gt;
&lt;th&gt;Line fitted on the other four&lt;/th&gt;
&lt;th&gt;Predictions (true)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;x = 0, 2.5&lt;/td&gt;
&lt;td&gt;&lt;code&gt;y = 3.517x − 2.153&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;−2.15 (0), 6.64 (6.25)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;x = 0.5, 3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;y = 2.483x − 0.602&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0.64 (0.25), 6.85 (9)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;x = 1, 2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;y = 3.000x − 0.625&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2.38 (1), 5.38 (4)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Pooling all six test predictions, the mean squared error is 2.23 and R² is 0.80. (How those two scores relate, and when to use RMSE or MAE instead, is in &lt;a href="https://davidmashiah.com/notes/rmse-vs-mae" rel="noopener noreferrer"&gt;a note on error metrics&lt;/a&gt;.) A straight line through a curve, and cross-validation calls it a decent model.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same model, split by group
&lt;/h2&gt;

&lt;p&gt;Now split by group instead: train on group A and test on group B, then the reverse. This is group k-fold, and with one group per fold it is called leave-one-group-out.&lt;/p&gt;

&lt;p&gt;Trained on group B, the line is &lt;code&gt;y = 5x − 6.083&lt;/code&gt;. On group A it predicts −6.08, −3.58 and −1.08, against true values of 0, 0.25 and 1. Trained on group A, the line is &lt;code&gt;y = x − 1/12&lt;/code&gt;. On group B it predicts 1.92, 2.42 and 2.92, against 4, 6.25 and 9.&lt;/p&gt;

&lt;p&gt;The mean squared error is now 18.68 and R² is &lt;code&gt;−0.67&lt;/code&gt;: worse than ignoring &lt;code&gt;x&lt;/code&gt; altogether. Same six points, same model. The only change is which points were allowed to sit together in a fold.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tzkn20qwr8krh9pmog5.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tzkn20qwr8krh9pmog5.jpeg" alt="Chart of the six points on the curve y = x², group A at x = 0, 0.5, 1 and group B at x = 2, 2.5, 3. A line trained on group A only stays near zero and misses group B; a line trained on group B only drops to about −6 at x = 0 and misses group A." width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Each group-trained line fits its own points and misses the other group. Random folds never ask the model to do this.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the random folds flattered it
&lt;/h2&gt;

&lt;p&gt;In the random folds, every test point had neighbours from its own group in the training set. The model never had to reach anything new. It only had to fill gaps between points it had already seen, and a straight line can fill small gaps in a curve quite well.&lt;/p&gt;

&lt;p&gt;That is the general rule. Cross-validation measures performance on data like the test folds. If the folds mix groups that will never be mixed in real use, the score answers a question nobody is asking.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this looked like in my research
&lt;/h2&gt;

&lt;p&gt;I trained models to predict the stiffness of metal plates with patterns of holes. A network trained on round holes scored essentially perfectly on held-out round-hole cells. Trained on one family and tested on another, 20 of the 30 combinations scored below zero.&lt;/p&gt;

&lt;p&gt;Splitting by row would have kept the near-perfect score. Splitting by family, the equivalent of the group split above, showed the model had learned the families it saw rather than how stiffness works. The worked version of that collapse is in &lt;a href="https://davidmashiah.com/notes/out-of-distribution-by-hand" rel="noopener noreferrer"&gt;a note on out-of-distribution failure&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choosing the split
&lt;/h2&gt;

&lt;p&gt;Ask what a new case will look like in use, and make the folds match. Several measurements from the same patient, machine, customer or design family belong in the same fold, which is what &lt;code&gt;GroupKFold&lt;/code&gt; and &lt;code&gt;LeaveOneGroupOut&lt;/code&gt; do in scikit-learn. Data in time order needs folds that train on the past and test on the future, as &lt;code&gt;TimeSeriesSplit&lt;/code&gt; does, because a random split lets the model see tomorrow.&lt;/p&gt;

&lt;p&gt;When the grouping is unclear, run both splits. A large gap between them is not noise. It is the size of what the model has memorised.&lt;/p&gt;

&lt;p&gt;How much a split flatters a model can be measured instead of guessed. On 823 retail items from the public M5 data, plain K-fold scored a gradient-boosting forecast 3.3 percent better than holding out whole items, with a 95 percent interval of 0.8 to 5.7 percent. That leak is small. In the metamaterial research above the same kind of gap was far larger, which is why it has to be measured for each dataset. &lt;a href="https://github.com/davidmashiah/forelint" rel="noopener noreferrer"&gt;forelint&lt;/a&gt;, the library I wrote for this, runs the comparison with &lt;code&gt;split_optimism&lt;/code&gt;, pairing each leaky split with an honest one on the same test rows.&lt;/p&gt;

&lt;p&gt;The same failure at full scale, in a competition I took part in, is in &lt;a href="https://davidmashiah.com/notes/validation-ranked-my-models-backwards" rel="noopener noreferrer"&gt;the note on how my validation ranked five models backwards&lt;/a&gt;: neighbouring wells kept in training made leave-one-out validation flatter the models that borrowed from them.&lt;/p&gt;

&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;Roberts, D. R. et al. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography 40(8), 2017.&lt;/li&gt;
&lt;li&gt;Bergmeir, C. and Benítez, J. M. On the use of cross-validation for time series predictor evaluation. Information Sciences 191, 2012.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/cross-validation" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>python</category>
      <category>statistics</category>
    </item>
    <item>
      <title>It won on the steady sellers and lost on the rest</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Sun, 08 Nov 2026 06:33:02 +0000</pubDate>
      <link>https://dev.to/mashiah/it-won-on-the-steady-sellers-and-lost-on-the-rest-32ij</link>
      <guid>https://dev.to/mashiah/it-won-on-the-steady-sellers-and-lost-on-the-rest-32ij</guid>
      <description>&lt;p&gt;&lt;em&gt;On one Walmart store, a machine-learning forecast was 3.9 percent worse than a 28-day moving average on absolute error, and the worse forecast for 58 percent of the 823 items. Split by demand class, the result is two opposite ones: it wins on the steady sellers and loses by about 10 percent on the intermittent and lumpy items, which are most of the store.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The usual way to choose a forecast is to compare one error number per model and pick the smallest. This note shows, on real retail data, why that number alone can point the wrong way for most of the catalogue. Everything below was measured with &lt;a href="https://github.com/davidmashiah/forelint" rel="noopener noreferrer"&gt;forelint&lt;/a&gt;, the open-source library I wrote for auditing forecast backtests, and the script that produces every figure is in its repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The data is the public M5 competition set: daily unit sales from Walmart. I took one store, CA_1, and one department, FOODS_3: 823 items. Each forecast covers the next 28 days, and the backtest repeats it at ten forecast origins 28 days apart, so the ten forecasts do not overlap and every comparison rests on ten separate results.&lt;/p&gt;

&lt;p&gt;Two forecasts compete. One is a gradient-boosting model from scikit-learn, trained on all items together, with six features: the item, the day of the week, sales 28 and 35 days earlier, and 7- and 28-day averages of those. It is deliberately plain. The other is a 28-day moving average: each item's mean daily sales over its last four weeks, repeated forward.&lt;/p&gt;

&lt;h2&gt;
  
  
  Overall
&lt;/h2&gt;

&lt;p&gt;On absolute error the model is worse. Its MAE was 1.742 against 1.676 for the moving average, a forecast value added of −3.9 percent, and it was the worse of the two on 58 percent of items.&lt;/p&gt;

&lt;p&gt;On squared error it looks slightly better: RMSE 3.44 against 3.51. But a Diebold-Mariano test across the ten origins gives p = 0.20, so that difference is not established. Squared error weights big misses heavily, and big misses happen on the items that sell a lot; &lt;a href="https://davidmashiah.com/notes/rmse-vs-mae" rel="noopener noreferrer"&gt;the RMSE versus MAE note&lt;/a&gt; works through why the two can disagree.&lt;/p&gt;

&lt;h2&gt;
  
  
  By demand class
&lt;/h2&gt;

&lt;p&gt;The Syntetos-Boylan classification sorts each item by how often it sells and how much the sale sizes vary, using its history before the first forecast: smooth, erratic, intermittent or lumpy. Breaking the same backtest down by class, on absolute error:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Demand class&lt;/th&gt;
&lt;th&gt;Items&lt;/th&gt;
&lt;th&gt;Value added vs moving average&lt;/th&gt;
&lt;th&gt;Items where the model is worse&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Smooth&lt;/td&gt;
&lt;td&gt;152&lt;/td&gt;
&lt;td&gt;+5.0%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Erratic&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;+1.4%&lt;/td&gt;
&lt;td&gt;33%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Intermittent&lt;/td&gt;
&lt;td&gt;463&lt;/td&gt;
&lt;td&gt;−10.5%&lt;/td&gt;
&lt;td&gt;68%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lumpy&lt;/td&gt;
&lt;td&gt;171&lt;/td&gt;
&lt;td&gt;−9.2%&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ford020vqi5ctmgnkkf43.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ford020vqi5ctmgnkkf43.jpeg" alt="Engineering drawing. Bars of forecast value added on mean absolute error against a 28-day moving average, by demand class, for 823 items in Walmart store CA_1, department FOODS_3, from the M5 data, over 10 forecast origins 28 days apart. Smooth, 152 items: +5.0 percent. Erratic, 33 items: +1.4 percent. Intermittent, 463 items: −10.5 percent. Lumpy, 171 items: −9.2 percent; the negative bars are red. Right: on absolute error the model is worse, −3.9 percent on MAE, worse on 58 percent of items. On squared error, RMSE 3.44 against 3.51, p = 0.20. It is worse than the average on 23 percent of smooth items, 68 percent of intermittent items and 69 percent of lumpy items. It is biased on 24 percent of items, and 10 percent of items hold 74 percent of the squared error. Measured with forelint 0.2.0." width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The model earns its keep on the steady sellers and loses about a tenth of its accuracy on everything that sells in bursts.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The model wins on the 152 smooth items and loses on the 463 intermittent and 171 lumpy ones, which together are 77 percent of the store. On about two thirds of the intermittent and lumpy items, the moving average was the better forecast.&lt;/p&gt;

&lt;p&gt;That changes the decision. The overall number says the moving average is a little better. The breakdown says use the model for the steady sellers, where it is better, and a simpler forecast or a method built for intermittent demand for the rest. One model for every item is the choice the overall number hides.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two more things worth knowing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Bias, item by item.&lt;/strong&gt; Across the store the model's bias was small, −0.6 percent of sales. But its mean error was significantly different from zero on 24 percent of items, against under 1 percent for the moving average: too high on some items and too low on others, cancelling out in the total. Item-level bias is what turns into empty shelves and overstock, as &lt;a href="https://davidmashiah.com/notes/forecast-bias" rel="noopener noreferrer"&gt;the forecast bias note&lt;/a&gt; works through.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Concentration.&lt;/strong&gt; The 82 items with the largest error, a tenth of the store, held 74 percent of the model's squared error. Improving the squared-error score mostly means fixing that list, and the audit names the items.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would check on any forecast
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Value added against a simple baseline, on the error measure that matches the cost of a miss.&lt;/li&gt;
&lt;li&gt;The same comparison per demand class. Steady items and intermittent items reward different methods, and one average mixes them.&lt;/li&gt;
&lt;li&gt;Whether a difference is real: a Diebold-Mariano test or a model confidence set across several forecast origins whose windows do not overlap. The days of one forecast are not independent, and overlapping windows share target days.&lt;/li&gt;
&lt;li&gt;Bias per item, not only in total, and which items carry the error.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;forelint does all four in one call, on the cross-validation table most forecasting code already produces. It installs with &lt;code&gt;pip install forelint&lt;/code&gt;. The full M5 script is in &lt;a href="https://github.com/davidmashiah/forelint" rel="noopener noreferrer"&gt;its repository&lt;/a&gt;, and the story of how the library started is in &lt;a href="https://davidmashiah.com/notes/validation-ranked-my-models-backwards" rel="noopener noreferrer"&gt;the note on how my validation ranked five models backwards&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Updated 8 November 2026: re-measured with forelint 0.2.0 at ten forecast origins whose windows do not overlap. Earlier versions of this note used a single origin, then overlapping origins, neither of which supports a valid significance test.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/forecast-value-added-by-demand-class" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>forecasting</category>
    </item>
    <item>
      <title>RMSE vs MAE: two forecasts with the same average error</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Fri, 06 Nov 2026 15:02:01 +0000</pubDate>
      <link>https://dev.to/mashiah/rmse-vs-mae-two-forecasts-with-the-same-average-error-1be4</link>
      <guid>https://dev.to/mashiah/rmse-vs-mae-two-forecasts-with-the-same-average-error-1be4</guid>
      <description>&lt;p&gt;&lt;em&gt;Two demand forecasts miss by exactly the same amount on average. One is steady, one has a single bad day. MAE cannot tell them apart. RMSE can, and five days of numbers show why, and when that is the difference you care about.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every error metric turns a list of misses into one number, and each one does it by deciding how much a big miss counts compared with a small one. MAE counts every unit of error the same. MSE and RMSE count big misses more. That single choice is most of what separates them.&lt;/p&gt;

&lt;p&gt;The example below is a demand forecast, because the difference is easy to feel there. When you &lt;a href="https://davidmashiah.com/notes/the-sale-you-did-not-make" rel="noopener noreferrer"&gt;sell perishables&lt;/a&gt;, being off by one unit every day and being off by five units once are very different problems.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three formulas
&lt;/h2&gt;

&lt;p&gt;For each case, the error is the prediction minus the true value, &lt;code&gt;e = ŷ − y&lt;/code&gt;. Over &lt;code&gt;n&lt;/code&gt; cases:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MAE&lt;/strong&gt;, mean absolute error: the average size of the miss, &lt;code&gt;MAE = Σ|e| / n&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MSE&lt;/strong&gt;, mean squared error: the average of the squared misses, &lt;code&gt;MSE = Σe² / n&lt;/code&gt;. Squaring makes every error positive and makes large ones count far more. Its units are squared, units of demand squared here, which is why it is hard to read.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RMSE&lt;/strong&gt;, root mean squared error: the square root of MSE, &lt;code&gt;RMSE = √MSE&lt;/code&gt;. That brings it back to the units of the data while keeping the extra weight on large misses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five days, two forecasts
&lt;/h2&gt;

&lt;p&gt;True demand over five days was &lt;code&gt;12, 9, 15, 10, 14&lt;/code&gt;. Forecast A predicted &lt;code&gt;13, 8, 16, 9, 15&lt;/code&gt;. Forecast B predicted &lt;code&gt;12, 9, 15, 10, 9&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Forecast A&lt;/th&gt;
&lt;th&gt;Forecast B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Errors&lt;/td&gt;
&lt;td&gt;+1, −1, +1, −1, +1&lt;/td&gt;
&lt;td&gt;0, 0, 0, 0, −5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MAE&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MSE&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;5.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RMSE&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;2.24&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;R²&lt;/td&gt;
&lt;td&gt;0.81&lt;/td&gt;
&lt;td&gt;0.04&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;By MAE the two forecasts are identical: both miss by one unit per day on average. A misses by one every day. B is perfect for four days and then misses by five.&lt;/p&gt;

&lt;p&gt;RMSE separates them. A's squared errors are all 1, so its MSE is 1 and its RMSE is 1. B's squared errors are four zeros and a 25, so its MSE is &lt;code&gt;25 / 5 = 5&lt;/code&gt; and its RMSE is &lt;code&gt;√5 = 2.24&lt;/code&gt;. One bad day more than doubled it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What R² adds
&lt;/h2&gt;

&lt;p&gt;R² compares the model's MSE with the variance of the true values, which is the MSE of a forecast that always says the average. Here the average demand is 12 and the variance is 5.2, so &lt;code&gt;R² = 1 − MSE / 5.2&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That gives 0.81 for A and 0.04 for B. B is barely better than predicting 12 every day, because its one miss of 5 costs almost as much squared error as the spread of the demand itself. R² is built on squared error, so it inherits RMSE's view of big misses. I used it throughout &lt;a href="https://davidmashiah.com/notes/out-of-distribution-by-hand" rel="noopener noreferrer"&gt;a note on out-of-distribution failure&lt;/a&gt;, where it goes below zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which one to use
&lt;/h2&gt;

&lt;p&gt;Use MAE when every unit of error costs about the same, and when you want a number people can read directly: off by one unit a day. It is also less swayed by a few odd cases, which is good when they are bad data and bad when they are the cases that matter.&lt;/p&gt;

&lt;p&gt;Use RMSE when a large miss is disproportionately expensive. Running out of stock on your busiest day, or a stiffness prediction that is wildly wrong for one design, is that kind of error. RMSE is also what least-squares fitting minimises, so it is the natural score for a model trained that way.&lt;/p&gt;

&lt;p&gt;Report both when you can. RMSE is always at least as large as MAE, and they are equal only when every error has the same size, as in forecast A. A large gap between them says the error is concentrated in a few cases, and those are the cases to look at.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one MAE and RMSE both miss
&lt;/h2&gt;

&lt;p&gt;Neither metric shows direction. Forecast A's errors average +0.2, close to balanced. Forecast B's average −1: every miss it makes is an under-forecast. For perishables, a forecast that always runs short means empty shelves on the days demand is highest, and its sales history then hides that demand, as worked through in &lt;a href="https://davidmashiah.com/notes/censored-demand-by-hand" rel="noopener noreferrer"&gt;a note on censored demand&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;So add the mean error, the bias, next to MAE or RMSE. Size tells you how wrong a forecast is. Bias tells you which way, and that is often the part that decides what to do.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/rmse-vs-mae" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>python</category>
      <category>statistics</category>
    </item>
    <item>
      <title>My load test had 1,000 players. None of them were late.</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Mon, 02 Nov 2026 06:42:58 +0000</pubDate>
      <link>https://dev.to/mashiah/my-load-test-had-1000-players-none-of-them-were-late-59cp</link>
      <guid>https://dev.to/mashiah/my-load-test-had-1000-players-none-of-them-were-late-59cp</guid>
      <description>&lt;p&gt;&lt;em&gt;Before the first event I ran 1 vs 100 against a thousand simulated players, and nothing broke. Then a lecture hall of real people broke it with a fraction of that. The test measured load correctly. It just never contained the one thing real people always do: arrive late.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I tested 1 vs 100 properly before anyone played it. A script opened a thousand simulated players over real WebSockets, started a question, and had all of them answer in the same instant: the thundering herd, the worst moment a live quiz has.&lt;/p&gt;

&lt;p&gt;It held. No dropped connections, the question reached every player in tens of milliseconds, and the reveal came back in under half a second.&lt;/p&gt;

&lt;p&gt;At the first real event, it broke anyway. Not under load. Under people.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the test could not contain
&lt;/h2&gt;

&lt;p&gt;Here is what the script did, in order: connect every player, wait until every one of them has joined, and only then press Start.&lt;/p&gt;

&lt;p&gt;That is the one order a real room never follows. In a lecture hall, people are still scanning the QR code when the host decides the room looks full. The game had a rule for anyone who joined after Start: watch, do not play. My thousand bots never triggered it, because not one of them was late. Everyone in the real room after the first 45 or so was, and their phones showed every question and never let them answer. &lt;a href="https://davidmashiah.com/notes/the-room-filled-after-i-pressed-start" rel="noopener noreferrer"&gt;The rule and the fix are their own note.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The test was faithful about load and wrong about time. It measured how many people the server could hold, and assumed they would all arrive at once, on cue, and stay on the page. Real people arrive late, switch tabs and reload. None of that is load, and all of it is what broke.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first ten minutes
&lt;/h2&gt;

&lt;p&gt;The first people with the problem came to me, and I did not know what it was. I went around to them, reassured them, and opened my laptop next to them to look. Technically, nothing was wrong anywhere I could see. The server was healthy and the game was running.&lt;/p&gt;

&lt;p&gt;I told them their answers were being counted. I believed it. They were not, and I did not know that yet.&lt;/p&gt;

&lt;p&gt;When more and more people had the same problem, I went outside to deal with it properly, making phone call after phone call to understand what was happening.&lt;/p&gt;

&lt;h2&gt;
  
  
  Not starting again
&lt;/h2&gt;

&lt;p&gt;After about ten minutes the event coordinator came out to find me and asked what was going on. Her suggestion was to start the entire event again.&lt;/p&gt;

&lt;p&gt;I asked her not to. Restarting would have thrown away the game for everyone who was playing fine, to fix it for the people who were not. Instead I asked her to tell the crowd that anyone hit by the bug would go through to the next stage rather than be eliminated.&lt;/p&gt;

&lt;p&gt;That kept the event moving and kept the people who had been wronged in the game. Meanwhile I changed the rule and deployed the fix while the event was still running, and the stranded players came back in at the next question.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a load test is for
&lt;/h2&gt;

&lt;p&gt;A load test answers one question well: can the server take this many people at once. It is worth running, and mine was right.&lt;/p&gt;

&lt;p&gt;It cannot answer the question that decided the night: what happens when people do not behave like a script. That needs a different test, one that is written from watching a real room: people joining after the start, leaving the page, coming back. Measuring the wrong thing precisely felt like safety. It was only confidence.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/none-of-them-were-late" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;, with the rest of my notes on real-time systems, machine learning reliability and supply chains.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>performance</category>
      <category>scalability</category>
      <category>testing</category>
    </item>
    <item>
      <title>One Durable Object per room, and a rule that stranded a hall</title>
      <dc:creator>David Mashiah</dc:creator>
      <pubDate>Sat, 31 Oct 2026 21:09:37 +0000</pubDate>
      <link>https://dev.to/mashiah/one-durable-object-per-room-and-a-rule-that-stranded-a-hall-59dc</link>
      <guid>https://dev.to/mashiah/one-durable-object-per-room-and-a-rule-that-stranded-a-hall-59dc</guid>
      <description>&lt;p&gt;&lt;em&gt;At a faculty event the vice dean played 1 vs 100 against a room of students. About 45 of them got into the game. Everyone after that got a phone that showed the questions and never let them answer. Nothing had crashed. A rule that was correct on paper had met a real room.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;1 vs 100 is a live multiplayer quiz for a lecture hall: one person on stage against the crowd on their phones, everyone answering the same question against the same clock.&lt;/p&gt;

&lt;p&gt;The first time it ran at a real faculty event, the vice dean was The One. About 45 students got into the game. The rest of the room was holding phones that showed every question and never offered a single button to press.&lt;/p&gt;

&lt;p&gt;The server was fine. Nothing ran out of memory, nothing timed out. The game was doing exactly what I had told it to do.&lt;/p&gt;

&lt;h2&gt;
  
  
  How a room works
&lt;/h2&gt;

&lt;p&gt;Each game is one of Cloudflare's Durable Objects, addressed by the six-digit PIN on the projector. Every phone in the room opens a WebSocket to that one object, and that object is the only place the game exists.&lt;/p&gt;

&lt;p&gt;A Durable Object handles one event at a time. A hall full of people answering in the same second cannot corrupt the scores, because there is never a second thread to race against. There are no locks in the code because there is nothing for them to protect.&lt;/p&gt;

&lt;p&gt;There is also no shared database. The room keeps its own state in the object's built-in storage: the players, their scores, the question in flight. That is what lets a room survive being evicted from memory and pick up where it was. When the game ends, an alarm deletes the room's data. Each game is its own small database, and it lives exactly as long as the game.&lt;/p&gt;

&lt;p&gt;Answers arrive in bursts, so broadcasts are batched, at most four a second, instead of one per answer. Idle sockets use WebSocket Hibernation, so a quiet room costs nothing. It runs on Cloudflare's free plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule that was right on paper
&lt;/h2&gt;

&lt;p&gt;The game has knockout rounds. Get a question wrong and you are out. So I wrote a rule for late joiners that seemed obvious: if you join after the game has started, you watch. You cannot walk into a knockout halfway through and claim you survived questions you never saw.&lt;/p&gt;

&lt;p&gt;In a test, that rule never fires. Everyone is already in the lobby when you press Start, because you are the one deciding when to press it.&lt;/p&gt;

&lt;p&gt;A real lecture hall does not work like that. People are still walking in, still finding the QR code, still typing a name, when the host decides the room looks full enough. The host pressed Start with about 45 people in. Every phone that finished joining a second later was a spectator, for the whole game.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fixing it while it was running
&lt;/h2&gt;

&lt;p&gt;We kept the event going. Whoever had been stranded was carried into the next round by hand, so nobody lost their place for something that was not their fault.&lt;/p&gt;

&lt;p&gt;Meanwhile I changed the rule, and deployed it during the event. In a round where nobody can be eliminated, a latecomer now joins as a full player immediately. In a knockout round they watch the current question and come in alive at the next one, so nobody is knocked out by a question they never saw. The only thing that keeps someone out for good is an actual elimination.&lt;/p&gt;

&lt;p&gt;The part I am most glad of: anyone already stuck as a spectator did not have to do anything. When the next question started, the server brought them back in. By the following round, most of the room was playing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two players who were knocked out
&lt;/h2&gt;

&lt;p&gt;Two players were knocked out who should not have been, and by the end of the event I did not know why.&lt;/p&gt;

&lt;p&gt;It turned out not to be the server at all. Both had switched away from the tab in the middle of the game. When they came back, the phone reloaded the page, and in a knockout round that cost them their place.&lt;/p&gt;

&lt;p&gt;That is the other half of the same lesson. A phone is not a test browser. It suspends tabs it is not showing, it reloads them when it wants the memory back, and nobody in a lecture hall keeps one tab open for an hour.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;p&gt;The architecture held. One object per room, no shared state, a single thread: none of that was the problem, and none of it needed to change.&lt;/p&gt;

&lt;p&gt;What failed was an assumption about people. Practice is not the same as the real thing, and the difference is never in the parts you tested. It is the queue at the door, and the tab someone switched away from.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;First published on &lt;a href="https://davidmashiah.com/notes/the-room-filled-after-i-pressed-start" rel="noopener noreferrer"&gt;davidmashiah.com&lt;/a&gt;, with the rest of my notes on real-time systems, machine learning reliability and supply chains.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>scalability</category>
      <category>serverless</category>
    </item>
  </channel>
</rss>
