At Landvex we're about to scale collection and annotation across several document families, mixed image and file inputs. The obvious failure mode is multiplying training material first, then discovering the test set was contaminated — or that "accuracy" was measured against a binary genuine/fake label we no longer believe in.
I'm leaning toward: define the ground truth per family, lock a held-out test set before we scale annotation and GPU training, and only then measure whether more data actually moves the dimensions we care about.
Does that match how you've seen high-stakes document or KYC systems do evaluation? What did you lock besides the files themselves — the label schema? Annotator instructions? The exact questions the model is scored on?
Also slightly paranoid about test items leaking back into reference examples that reviewers can see. How do you keep those walls up in practice?
Top comments (0)