This case is invented. An office of a dozen people uses a help-desk tool to answer staff questions by email. Someone suggests adding a generative AI helper. The helper would read each ticket, draft a reply and, for simple cases, send the reply and close the ticket on its own. They want to know which reviews to finish before the helper touches live tickets.
We read two public NIST guides: the AI Risk Management Framework (AI RMF 1.0) and its Generative AI Profile (NIST AI 600-1). NIST wrote both for voluntary use. Here are six checks, each tied to those pages.
Word key
- Generative AI (GAI): a model that creates new text, images or similar output from a prompt. It is often called a foundation model when it is trained on a wide range of data.
- Confabulation: NIST's word for output that sounds sure of itself but is wrong or made up. People also call this a "hallucination."
- Automation bias: the tendency to trust a machine's answer too quickly, even when a person should look again.
- Prompt injection: a trick where hidden instructions are placed in data the model later reads, so it does something the team did not intend.
- Third-party tool: a model, app or data source that another company provides, not one you built in-house.
Check 1. What exact job will the helper do, and what must it never do?
The AI RMF asks teams to map the context of use before they measure or manage risk. NIST's Generative AI Profile is a companion to that framework. It is meant to help an organization apply the same four functions — GOVERN, MAP, MEASURE and MANAGE — to generative AI.
Our advice: write the job in plain words, on one page. In our example, "draft a reply for a person to review" is a different job from "send the reply and close the ticket." Name the actions that stay with a person, such as refunds, account changes or anything that cannot be undone easily.
Check 2. What data can it see, and where do its answers go?
NIST lists data privacy as a risk that generative AI can make worse. The harm it describes is leakage or unauthorized use of personal or other sensitive data, such as health, location or biometric details. The AI RMF also says a closer first look may be needed when a system is trained on large collections of sensitive or protected data, such as personal details. The same is true when its output can affect people, directly or indirectly.
Our tip: list every data source the helper can read, and every place a draft can be stored or sent. If a ticket holds a name, payroll figure or login detail, decide whether that field is stripped, masked or blocked before the model sees it. Do not paste real personal data into a public chat box "just to try it."
Check 3. Will a person review the output before it acts?
NIST defines confabulation as confidently stated but false content, which can mislead a user. It also describes a "human-AI configuration" risk. People may lean on the system too much (automation bias or over-reliance). They may also treat it as more human than it is.
The AI RMF's safety section points to practical controls. Test in a realistic setting and watch the system while it runs. Keep a way to shut it down, change it, or let a person step in when it drifts from the planned job.
Our advice: a person should approve the output first for any action that leaves the team. That includes an email to a staff member or a change in a record. "Draft only" is the safer starting point in our example. Let the helper send mail on its own only after you have evidence it handles that job well.
Check 4. Have you tested it on realistic examples, including ones it should refuse?
The AI RMF says accuracy checks should use clearly defined, realistic test sets that match real use. The test method should be written down. For systems already in use, it says validity and reliability are often checked by ongoing testing or monitoring, not only a one-time trial.
Our advice: build a small set of past tickets (with personal details removed). It should cover easy cases, odd cases and cases the helper should not answer. Score drafts against what a skilled person would have sent. Run the same set again after any change to the model, the prompt or the connected tools.
Check 5. If a vendor is in the mix, what did you inventory?
The AI RMF notes that third-party data, software and hardware can speed up a project, and can also make risk harder to measure. The Generative AI Profile suggests an inventory of every third-party entity that can reach your content. It also suggests an approved list of generative AI tools and services. It also flags information-security risks such as indirect prompt injection, where harmful instructions sit in data the model later retrieves.
Our tip: write down the vendor, the model name and version, and where data is processed. Also record how long it is kept, and whether your prompts may be used to train a shared model. If the helper can read mail or tickets, ask how the vendor limits prompt injection from that content.
Check 6. Who is on the hook if it goes wrong, and how do you switch it off?
The Generative AI Profile ties governance to incident response. Name human oversight roles, plan how you will detect and report a problem, and practice the response. The AI RMF, as noted above, includes the ability to shut a system down or put a person back in control.
Our advice: agree, in writing, who can turn the helper off and how fast that happens. Also agree how you will tell people who already received a bad reply. In our example, that includes a switch that stops automatic sends without taking down the help-desk itself.
Tick these off before the helper acts on real work
- [ ] The job and the "never do" list are written in plain words.
- [ ] A data list shows what the helper can read and where answers go.
- [ ] A person approves any action that is sent out or that changes a record.
- [ ] A realistic test set, including "should refuse" cases, has been scored and saved.
- [ ] Each vendor, model and data flow is on an inventory, with a retention and training-use answer.
- [ ] A named person can switch the helper off, and the team has practiced that step.
Bring these questions to your engineering or automation team
- Which actions will a person still approve, and which ones (if any) will the helper take alone?
- What personal or sensitive fields are blocked before a ticket reaches the model?
- How will we notice a confident but wrong draft, and what is the fallback?
- When we change the model or the prompt, which tests do we run again?
- If a vendor's service is down or misbehaves, how do we keep answering tickets by hand?
Read the docs yourself
- NIST, AI Risk Management Framework (AI RMF 1.0), NIST AI 100-1: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
- NIST, AI RMF overview: https://www.nist.gov/itl/ai-risk-management-framework
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- NIST publication page for NIST AI 600-1: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
This checklist was prepared by Codewave Consulting LLC using public NIST publications. Treat it as background reading, not as professional advice on your systems. A qualified engineer should look at your data, tools and risks before an AI helper is connected to real work. Our company is an E-Verified employer. We proudly support H-1B visa sponsorship for qualified resources. Eligibility is reviewed individually; sponsorship does not guarantee a role, selection or government approval.
Our firm: Codewave Consulting LLC serves as a Direct Vendor for Whirlpool and works from Michigan. Next to our staffing practice, we build custom software, design AI-assisted workflows, and engineer digital products. More at codewaveconsulting.com.
Top comments (0)