Originally published on AI Tech Connect.
Three things to get right before you size a test Your variance is bigger than you think. LLM outputs vary session to session even with temperature at zero, because the input distribution — real user phrasing — is far noisier than any offline eval set. Underestimate variance and your sample-size calculation will be wrong before you collect a single row. A p-value from a dashboard you checked five times this week is not the p-value you think it is. Fixed-horizon significance testing assumes you look once, at the end. Peeking and stopping early inflates your false-positive rate, sometimes dramatically. "Better on the primary metric" is not the same as "safe to ship." Latency, cost per query and refusal rate can all move in the wrong direction while your headline metric goes up. You need…
Top comments (0)