AI automation and RAG projects usually do not fail because the demo is impossible.
They fail because the scope is quoted before the uncomfortable parts are clear:
- what outcome the workflow is supposed to improve;
- which data is allowed;
- what must stay human-approved;
- how evidence quality will be checked;
- what a correct output looks like;
- what happens when the system fails.
Here is the six-point triage pass I use before treating an AI automation, RAG, or agent scope as implementation-ready.
1. Outcome clarity
Green: one measurable workflow outcome is named.
Amber: the client lists tools but not the decision or handoff to improve.
Red: the request is only "add AI" or "build an agent".
If this is red, do not quote a full build. Quote a scope review or discovery slice first.
2. Data boundary
Green: allowed sources and excluded data are explicit.
Amber: sources exist, but ownership or access is unclear.
Red: production secrets or regulated data are required just to scope the project.
The first review should work from redacted data. Credentials should not be needed for basic scope triage.
3. Human approval
Green: the first milestone has a clear staff approval point.
Amber: staff can review outputs, but only after user-visible action.
Red: the first milestone changes accounts, refunds money, sends advice, or takes customer-facing action automatically.
For most early scopes, "draft and route" is a better first milestone than "decide and send".
4. Evidence quality
Green: there are recent examples, edge cases, and source docs.
Amber: docs exist, but duplicates or stale procedures are visible.
Red: the agent must infer policy from memory, Slack history, or scattered chat logs.
RAG is not a rescue mechanism for messy source ownership. It only makes the mess easier to retrieve.
5. Evaluation path
Green: success can be checked with a small labeled set.
Amber: success is subjective, but reviewers are available.
Red: no one can say what a correct output looks like.
If the team cannot review 30 to 50 examples, they are probably not ready to automate the decision.
6. Rollback plan
Green: the workflow can fail closed and route to a human.
Amber: manual fallback exists but is not documented.
Red: a bad output can create irreversible customer, financial, or compliance impact.
The first implementation slice should be reversible.
How to quote the next step
Mostly green: quote an implementation slice with acceptance tests.
Mixed amber: quote a written review, evidence map, and first milestone plan.
Any red: quote risk triage and data-boundary clarification before touching implementation.
I published the checklist as a public page here:
https://mindtrovertlabs-sketch.github.io/scopegrade-storefront/agency-scope-triage-checklist.html
There is also a fictional sample of the async review output:
https://mindtrovertlabs-sketch.github.io/scopegrade-storefront/sample-deliverable.html
No credentials, production access, client data, or call should be required for this kind of first-pass review. The goal is simply to decide whether to quote, narrow, or defer.
Top comments (0)