Before scoping any AI feature, spend one afternoon answering six questions about the data it will run on: where it lives, who owns it, how fresh it is, whether ground truth exists, what's missing, and who is allowed to see it. Most of the "the model isn't good enough" failures I've seen were data-readiness failures discovered in month three instead of hour three. The audit is cheap. Skipping it is how projects die.
Why do AI builds stall on data?
AI builds stall on data because the pilot ran on a clean export and production runs on the real system. Real systems carry conflicting definitions, status columns that changed meaning, scanned PDFs, and access rules nobody mapped. None of that shows up in a demo, and all of it shows up around week five.
The real thing is fourteen tables with three different definitions of "customer," a status column that has meant four things over the years, PDFs that are scans, and a set of fields nobody outside finance is cleared to read.
Regulated industries feel this hardest, which is a large part of why AI projects die faster in fintech and healthtech: the model was fine, and the data-access question was never asked out loud.
The six questions, and what a bad answer sounds like
The six questions are location and access path, ownership, freshness needed versus actual, ground truth, gaps and dirt, and permissions. Each one has a green answer and a red one, and the red answers are specific enough that you can hear them coming in a scoping call.
- Location and access path. Good: a read replica or an API with a named owner. Bad: "someone runs an export monthly and emails it."
- Ownership. A person, with a name, who can answer questions about the schema. "The vendor" or "the old team" means nobody.
- Freshness, needed versus actual. A daily batch is fine for a monthly report and useless for a support agent answering about an order placed an hour ago.
- Ground truth. You should be able to show me two hundred examples of the right answer, produced by someone whose judgment you trust. If the answer is "we'll know it when we see it," you can't build an eval set, and you can't build the feature.
-
Gaps and dirt. Null rates, free-text fields that should be enums, IDs that don't join, dates stored as text. Ten minutes of
GROUP BYandCOUNT(*)reveals most of it. - Permissions. PII, contract terms, per-tenant isolation, retention rules. If the AI feature will read something a junior employee couldn't, that's a design constraint from day one.
A concrete example
One project wanted a support-ticket classifier trained and evaluated on the existing "category" field. A two-hour audit found that 40 percent of tickets were tagged "Other," that twelve categories had been renamed twice, and that busy agents had been picking the first item in the dropdown. The ground truth was fiction.
That afternoon saved us from three sprints of tuning a model against a label nobody believed. The first sprint became relabelling five hundred tickets with two reviewers each, and the classifier that followed had something honest to be measured against. The same trade shows up on every finding:
| What the audit turns up | Cost if you find it in hour three | Cost if you find it in month three |
|---|---|---|
| No usable ground truth | One sprint of relabelling | Three sprints tuning against a label nobody believes |
| Monthly emailed export, no API | A scoping decision | A rebuild of the ingest path |
| PII the feature can't legally read | A design constraint | A launch held by legal review |
| Schema owner left the company | An hour finding a replacement | Weeks reverse-engineering columns |
What to do with the answers
Score each of the six questions green or red, amber only if you must. Every red item becomes sprint zero, before a model is touched, and the answers go into the scope document so the client and the team hold the same picture. Re-run all six at every phase.
Data readiness decays. An owner leaves, a schema changes, a new source appears, a retention rule tightens.
At Shanti Infosoft this audit is part of how we scope every AI engagement, and it's the single most common reason a quoted timeline changes before a contract is signed. It's also the first thing we run when a team brings us in for AI consulting on a stalled build, because the fix is usually upstream of the model. I'd rather have that conversation in the first week than the fifteenth. If you want a second pair of eyes on your six answers, grab a slot and bring your schema.
If I asked you tomorrow morning for two hundred examples of the right answer, how long would it take you to find them?
Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 firm whose 80+ engineers ship AI systems into production.
Top comments (0)