A language model samples. Ask it the same question twice and you get two answers. Most monitoring tools ask once, record the draw, plot it on a dashboard, and call tomorrow's second draw a trend.
We tested how bad this gets. Fifteen identical runs of one prompt against one company. Four models stayed inside ranges of 44 to 79 points. Perplexity's sonar moved 232 points across the same fifteen runs and contradicted itself twice in a single afternoon. Nothing changed between runs except the sampling.
If your metric is one draw, a trend line built from it cannot separate a real movement from the model's own variance. We report the distribution instead of the point. Methodology and the reliability study are written up here https://veritaslinks.com/compare/profound-vs-veritaslinks
Top comments (0)