The latest articles on DEV Community by Muhammad Waqas (@muhammadwaqasmbd). https://dev.to/muhammadwaqasmbd https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4101193%2F0b90a076-e158-4c87-959d-8af6f158b2b5.jpg https://dev.to/muhammadwaqasmbd en Muhammad Waqas Sun, 30 Aug 2026 09:22:43 +0000 https://dev.to/muhammadwaqasmbd/we-spent-two-days-bisecting-a-prompt-change-the-regression-was-noise-4m4e https://dev.to/muhammadwaqasmbd/we-spent-two-days-bisecting-a-prompt-change-the-regression-was-noise-4m4e <p>Quality went from 0.81 to 0.78. Someone had edited a prompt that week. Obvious culprit, obvious investigation.</p> <p>Nobody had measured that re-running the same prompt scores 0.77-0.84 across seeds.</p> <p>0.78 was never a regression. It was Tuesday.</p> <p>The number was real. The comparison was not, because nobody measured the instrument before trusting it.</p> <p>So now I do this in order, and the order is the whole point:</p> <ol> <li><p>Calibrate the judge. Can it separate a known-good answer from a known-bad one? A judge returning 3/4 for everything gives you a rock-steady dashboard that would stay green if the agent returned Lorem Ipsum.</p></li> <li><p>Measure the noise floor. Run each case across several seeds. That spread is the resolution of your instrument.</p></li> <li><p>Then gate. A delta smaller than the noise floor is not a small regression. It is no information at all.</p></li> </ol> <p>A gate that fires on noise gets marked flaky and gets continue-on-error added within a month. Then you have no gate.</p> <p>How many of your eval numbers have a measured error bar?</p> <p>Calibrate the judge, measure the noise floor, then gate in that order.</p> <p>Github Repo: <a href="https://lnkd.in/dbfwtsM6" rel="noopener noreferrer">https://lnkd.in/dbfwtsM6</a></p> mlops regression evaluation agents