In an episode published November 6, 2024, May Habib, CEO and cofounder of Writer, spelled out what she requires before an enterprise trusts generative AI with real work. I would use that bar on an internal tool. Agree what good looks like, name an owner, control the data, and keep a person able to check the result.
What has to be true before the business will trust the tool?
Habib said Writer asks for two things before the build starts.
Number one, a recognition that AI is a team sport, right? If we are brought in by IT versus line of business, right, we are really insistent on being close to the the business and not building science projects that, you know, aren't necessarily connected to to reality.
May Habib, CEO and cofounder of Writer, on Chain of Thought episode 1, November 6, 2024
The second demand is dedicated people who own the program inside the company. She said Writer will train them. They do not all have to be engineers, but they do have to understand how the engineers get the outcome. Then she set the standard I would copy:
We don't want AI to be magic inside of a company. We need it to be just software that people really understand.
May Habib, CEO and cofounder of Writer, on Chain of Thought episode 1, November 6, 2024
Trust also fails when nobody has agreed the business logic. Habib tied that directly to whether a team can hand work to a model:
When we say we align to business logic, folks need to agree on what that logic is, and especially for the data that's in people's minds, right, and the subject matter kind of experts' brains, you know, there's quite a bit of work to align on how you're going to do something if you're going to now trust AI to do that thing.
May Habib, CEO and cofounder of Writer, on Chain of Thought episode 1, November 6, 2024
She put a lot of marketing work in that category. The open question is brand voice and a shared picture of good, once you already believe a model can draft the text.
She was also done, by that November, with showing a thin prototype to the business. She said her team builds the evals even when the customer has none, because the first version a business user sees has to be strong. She thinks Writer is probably the only generative AI company that employs QA people with PhDs for that job. You may not hire that bench. You can still refuse the demo until an owner has signed a small eval set.
I'd start with three names on a page: the business owner, the technical owner, and the subject matter expert whose judgment defines a good answer.
Which data controls should you put in before the first user?
Habib described mission critical apps as a compound system: models, retrieval over customer data, and guardrails. On retrieval, the customer connects data repositories and Writer processes that data. She said the word RAG does not appear in the product. On accuracy without a large fine-tune, she said domain models for financial services, customer support, creative work, and medical work ship preconfigured, and that capturing business logic with examples takes far less data than fine-tuning properly.
For a tool you build yourself, write the controls in the repo:
- List every source the tool can read, and who approved that access.
- Keep the example set that defines a good answer separate from the corpus the tool may quote. Version the examples.
- If the tool calls another system, check the payload first. Habib said tool use was still a production gap in that interview: you have to be sure you are not moving low quality output from one system to the next. She said a lot of work remained at the model layer before parallel tool use was something she would trust.
- Write guardrail rules where the owner can read them. Habib said Writer builds guardrails with language models, in a high trust way, and goes beyond naive rejects. Hiding that logic in a prompt the owner never sees fails her "just software" test.
This sketch is a gate I would run before a demo. It is not a system Writer runs.
# Sketch only. A pre-ship gate for an internal tool.
def trust_gaps(tool: dict) -> list[str]:
gaps = []
owners = tool.get("owners", {})
if not owners.get("business") or not owners.get("engineering"):
gaps.append("name a business owner and an engineering owner")
if not tool.get("agreed_good"):
gaps.append("record the business logic the owner signed")
if not tool.get("data_sources"):
gaps.append("list sources the tool can read, and who approved them")
if not tool.get("examples_version"):
gaps.append("version the examples that define a good answer")
if tool.get("calls_other_systems") and not tool.get("payload_checks"):
gaps.append("check data before it moves to the next system")
if not tool.get("eval_cases"):
gaps.append("add eval cases before the first business demo")
return gaps
How do you explain an answer the model cannot explain?
Explainability means you can answer why the model did that in terms a person can check. Simple models often trace a decision to features. For a large language model that is much harder. If you ask it why it answered, it will produce a fluent rationale after the fact. That narration may not match the computation. Treat it as a story, not a cause.
I would log what a reviewer can actually open: retrieved sources, the example included in the prompt, the tool that ran, and the payload that came back. On a high stakes call, keep a person who can override. Habib described insurance claims of $100,000 and up, where automated passes miss detail. Writer supports human decision makers there with quick answers and an initial readout of evidence, including from electronic health records. The person still decides.
Inputs you can show, plus an override, are the visibility you can build this week.
What governance is enough for an internal tool?
AI governance is how "we built it" becomes "we can run it": an inventory of models and agents in production, limits on what each one can access and do, an audit trail of decisions and actions, and a named owner. NIST's AI Risk Management Framework organizes the work as Govern, Map, Measure, and Manage. You can copy the habit without seeking a certificate. Write the system down, bound what it can touch, measure it, and name who answers for it.
If the tool affects people in the EU, the EU AI Act can apply wherever your team sits. It tiers risk: a small set of uses is prohibited, and high-risk uses such as hiring, credit, and critical infrastructure carry duties on data quality, documentation, human oversight, and transparency. Name your tier on purpose, not in a sprint review. If the tool only drafts internal notes, say that in the inventory. If it touches credit, hiring, or infrastructure decisions, treat the high-risk duties as in scope until counsel says otherwise.
Habib's November 2024 examples: bankers drafting S-1s, analyzing earnings, and building pitch books; capital markets and hedge fund work such as industry questions, company writeups, and investment committee memos; claim adjudication; healthcare. She said those settings need high accuracy, which is why she wants a compound system around the model and a high bar before anything reaches the business.
Checklist
- Name a business owner and a technical owner, and include the people who do the work.
- Write the agreed definition of a good result, including logic that still sits with subject matter experts.
- List data sources, who approved them, and the checks on anything passed to another system.
- Version a small example set, and keep an eval set the owner has signed before the first business demo.
- Log retrieved inputs and tool results, keep a human override on high stakes decisions, and record the system in an inventory with an audit trail.
The full conversation, with the transcript, is on Chain of Thought.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)