The checklist we go through before shipping AI features to real users

Reliqio — Thu, 12 Nov 2026 17:01:31 +0000

Every AI feature looks good in a demo. You show it to the team on Friday, everyone's impressed, and then on Monday real users start typing things into it that nobody on the team would ever type.

We build AI-powered B2C products together with founders at Reliqio, mostly in healthtech, fintech and e-commerce. Somewhere along the way we started keeping a list of things to check before an AI feature goes live. It started as a few notes in a doc and kept growing every time something surprised us. This is the current version.

Do we even need a model here?

This one gets skipped a lot, including by us. If a filter, a sensible default or a couple of if-statements solve the problem, just ship that. It's cheaper and it won't make things up.

Our rough rule: AI makes sense when the input is messy. Free text, photos, voice notes, long PDFs. If the whole input fits into three dropdowns, you probably don't need an LLM.

What does the user see when it's wrong?

Not "if". It will be wrong sometimes.

So for each feature we ask: what's the worst realistic wrong answer, and what happens next? Depending on the product that might mean showing where the answer came from, or letting the AI only suggest while the user makes the final call. In health and money apps some questions just shouldn't be answered by a model at all. We send those to fixed, reviewed content or to a real person.

Test with real messages, not ours

Prompts written by the team are way too clean. Real people write half a sentence, switch languages midway, paste a screenshot of text, or ask three unrelated things in one message.

We collect 30-50 real examples before launch (beta users, support chats, user interviews) and keep them in the repo. Every time someone touches the prompt or switches the model, we run them again and compare. It's not a perfect eval, but it catches the obvious regressions before users do.

Know the cost per user

A few cents per request sounds like nothing until you multiply it by daily active users and the number of times each of them taps the button. We log tokens for every call and work out cost per active user before launch. Finding out from the invoice is no fun.

What usually helps: shorter prompts, a smaller model for the simple stuff, caching answers to questions that come up over and over, and a per-user limit.

Try it on a slow phone

Six seconds is fine on office Wi-Fi. On a phone in the subway it feels like the app froze.

Stream the response so the first words show up fast. Show some progress for longer tasks. And test on a throttled connection at least once. If it takes longer than a few seconds anyway, it's probably better as a background job with a notification when it's ready.

Look at what goes into the prompt

People tell consumer apps a lot. Symptoms, debts, home addresses. Before launch we literally print out what goes into each prompt, remove what the model doesn't need, and check that the provider's data retention settings match what the privacy policy says.

If you can't explain in one sentence what happens to the user's data, it's not ready yet.

Have an off switch

Providers go down. Model updates sometimes change behavior with no warning. Every AI feature we ship is behind a feature flag and has a fallback that works without AI. If it breaks at 11pm on a Saturday, someone can turn it off from a dashboard instead of pushing a hotfix through app review.

Honestly, the model is usually the easy part. Most of the work is everything around it.

If your team checks something we missed, I'd like to hear about it in the comments.

DEV Community: Reliqio