AI tools can take a product from an empty repository to a convincing demo quickly. That is useful, but a smooth happy path does not answer a harder question:
What happens when users, permissions, retries, and partial failures do not follow the demo script?
This does not mean AI-generated code is automatically bad. Human-written code can fail in the same ways. The practical problem is that generation often moves faster than the team’s ability to define boundaries, write negative tests, and record why a launch-critical behavior should be trusted.
You do not have to review the whole universe to make progress. Pick one 3–6-screen journey—signup, onboarding, checkout, or create / save / export—and run these nine checks.
1. Name the exact user and outcome
Do not start with “review the backend” or “check our Supabase app.” Write one sentence that identifies:
- the user or role;
- the first action;
- the expected final state; and
- the data, money, or permission that changes.
Example:
A standard member completes checkout once and receives exactly one active entitlement tied to the correct account.
Specific wording gives you observable pass and fail conditions.
2. Record the version and environment
A test result is difficult to trust if the target keeps changing. Record:
- the commit or deployed version;
- local, preview, or staging environment;
- test accounts and roles;
- sanitized data used; and
- anything that cannot be tested safely.
The honest conclusion is then “this behavior was checked under these conditions,” not “the app is safe.”
3. Map each trust decision
Write the flow in plain language:
- The user takes an action.
- The client sends a request.
- The server identifies the user and role.
- A rule decides what is allowed.
- Data or an entitlement changes.
- The user sees a result.
At every step, ask what identity is trusted, what values the caller can choose, and where the rule is actually enforced.
4. Try the wrong user or tenant
For an authorized test account, safely ask whether changing an object or tenant identifier can expose or modify another user’s data.
The UI hiding a button is not enough. The relevant server or database boundary should reject an unauthorized action.
Use sanitized test records only. Do not probe systems you do not own or have permission to test.
5. Repeat the same action
Retries happen because browsers refresh, networks time out, users double-click, and providers redeliver events.
Repeat the same safe request or test event and check whether it creates:
- duplicate records;
- duplicate entitlements;
- contradictory status;
- repeated notifications; or
- an incorrect charge in a test-mode system.
Record the expected idempotent behavior before running the check.
6. Change the order
External events may arrive late or out of order. In a controlled test environment, check whether a delayed cancellation, completion, or retry can overwrite a newer valid state.
If order matters, the system should use durable state rules rather than assuming events always arrive in the demo sequence.
7. Stop halfway
Ask what remains if the flow fails after one write but before the next.
For example:
- a payment record exists but no entitlement was granted;
- a file was stored but its database record failed;
- a user was invited but the authorization relationship was not created.
The desired outcome may be a transaction, a retryable state, cleanup, or a clearly visible recovery path. What matters is that the half-finished state is intentional and observable.
8. Test both denial and legitimate use
Every fix or rule needs two checks:
- a negative test showing the unwanted action is rejected; and
- a normal-path test showing the correct user can still finish the flow.
Without both, it is easy to “fix” unauthorized access by breaking access for everyone or to stop duplicates by blocking valid retries.
9. Write down evidence and limits
For each check, record:
- starting role and state;
- exact safe action;
- expected result;
- observed result;
- relevant screenshot, log, or test output with secrets removed; and
- anything blocked or still outside scope.
If the behavior cannot be reproduced, say so. If access or setup prevents the check, record the blocker. Uncertainty is a result; it should not be turned into a dramatic finding.
A small next step
If you want a second set of eyes without opening your codebase, the AI App One-Flow Preflight reviews one submitted 3–6-screen journey from a sanitized recording of up to 10 minutes or up to eight screenshots.
For USD 129, the fixed deliverable is a flow map, up to three evidence-linked hypotheses, ten manual tests with expected results, a Now / Next / Later action plan, and one asynchronous follow-up. Turnaround is within two business days after complete accepted materials.
Applying is free, and a misfit application stops before checkout:
This is an evidence-limited product review, not source-code review, production testing, a security audit, penetration test, certification, defect guarantee, or approval to launch.
Disclosure: This article was prepared with AI assistance and manually reviewed for scope, factual claims, and safety boundaries.
Top comments (0)