Seeing a lot of teams lately rush an LLM into the first version of every product. Model costs keep dropping, so it feels cheap to add. But cheap to call is not the same as cheap to get right.
The rough rule I keep coming back to:
- If the product is basically useless without the model (doc parsing, matching, summarising messy input), it belongs in v1. Then the real work is evals and failure handling, not the prompt.
- If the core loop works with a person doing the step by hand for now, ship that first and add the model once you know what "correct" looks like.
The tradeoff cuts both ways. AI in v1 can make early feedback noisy, because you can't tell if users dislike the product or the model output. Adding it later can mean reworking the UX around it.
For regulated stuff (finance, health, compliance docs) I'd lean toward human in the loop first almost every time, just so there is a clean record of what the right answer was.
For whatever you're building right now, is the AI the actual hypothesis or more of a nice extra?
Top comments (0)