DEV Community: AppDeploy Team

Vibe Code with Confidence: Testing AI-Built Apps with QA Agents

AppDeploy Team — Sun, 03 May 2026 10:01:44 +0000

TL;DR:

When AI writes the code, you can't trust the same AI to prove it works.
We built an independent, black-box QA agent that treats every deployed app like a real user would: clicking, navigating, and verifying outcomes.
Fast enough and cheap enough to run on every deploy.
We call this autonomous end-to-end QA: black-box testing that runs after deployment and returns QA snapshots, visual bug reports, and logs back into the chat.

We used to trust code because a human owned it. Someone wrote it, reviewed it, lost sleep over it, and could explain why it works. AI-built apps (often called vibe coding) break that assumption. Features can ship after a handful of prompts and automated diffs, and no one can honestly claim they understand every line that just changed. QA becomes the source of truth: automated, continuous proof that the system still behaves correctly.

In a chat-native deployment platform such as AppDeploy, this is doubly true. Here is what happens every time you deploy an app with AppDeploy: once the build completes, a QA agent automatically runs a test suite to verify the quality of the app.

With AppDeploy, this starts as a test-driven development (TDD) approach where the tests are defined by what the app should do before implementation. The coding agent writes the tests first, then implements the app until the suite passes. If the agent finds bugs, AppDeploy returns structured feedback to the coding agent, including a description of each failure, the relevant screenshot, and browser console errors. The coding agent uses that feedback to fix the issues and calls AppDeploy to redeploy. QA reruns automatically, and the cycle repeats until the tests pass.

A simple way to think about the loop:

You prompt
A coding agent builds
AppDeploy deploys
A QA agent runs end-to-end tests on the deployed app
Results go back to chat
The agent fixes and redeploys

What a QA agent does for AI-built apps

So how do you QA your AI-built apps?

You cannot vibe your way into confidence by asking the same agent to run a Playwright script once and call it done. When the builder and the checker share the same assumptions, you get agreement, not assurance. Real QA means independent evidence that the system works in the environment it will actually run in, across the flows that matter, with the dependencies it really uses, and under the failure modes it will eventually hit. The whole point is to replace "it looks right" with "it is proven."

Why the coding agent can't fully test itself (unit and end-to-end testing)

Writing code and proving code are different jobs

A coding agent is great at producing plausible implementations quickly. Verification is different: it needs an independent notion of truth and a willingness to fail the change. When the same agent is both builder and judge, the incentives and the failure modes line up in the wrong direction.

The same brain problem

If the agent writes the implementation and the tests, both tend to reflect the same mental model. That creates correlated failure: the tests pass because they validate what the agent thought the system should do, not necessarily what the system must do. You get internal consistency, not external correctness.

"Green" is easy to optimize for

Even without malicious intent, an agent optimizing for completion will drift toward low-resistance tests: happy paths, shallow assertions like "returns 200," snapshots of whatever output it produced, or mocks that bypass the risky parts. What does a passing suite even mean, if the suite wasn't designed to fail?

What this means for AI-built app platforms

Let the agent help generate tests and suggest scenarios, but don't let it be the final authority. The platform needs independent, repeatable verification in realistic environments, with contracts and gates that can't be waved away by a single "looks good" run.

Why real QA is difficult, especially with a backend

Given the limits from the last section, you end up doing something close to black-box QA. You treat the system like a user would. You click, you call APIs, you run flows, you verify outcomes. That gets you away from "same brain" testing and away from tests that are optimized to go green. It is also where things get hard.

State is a problem

Tests are only reliable if you can set up, control, and reset state, and that's hard once real data and history enter the picture.

Backend state is harder than frontend state

Frontend state usually lives in one session; backend state lives across databases, caches, queues, and services, and it persists between runs.

Background jobs make tests flaky

If the system has async elements (queues, jobs, webhooks), the test has to guess when to check, so it sometimes checks too early and fails.

Failures, retries, and timeouts are a big part of QA

A lot of backend bugs only show up when something is slow or fails, then you get retries, partial success, or the same event processed twice.

How we built our QA agent

The goal is simple: treat the app like a user would. Click, type, navigate, and verify outcomes. Do it fast. Do it cheap. Do it reliably. Do it without "cheating" in ways a real user cannot.

Test plan: where it comes from

A coding agent is quite good at writing a thorough test plan. It can read the spec, routes, and UI structure and produce coverage that a human QA would recognize.

In AppDeploy, this is a test-driven development (TDD) approach where the tests are defined by what the app should do before implementation. The coding agent writes the tests first, then implements the feature. The catch is execution: the same agent that builds the feature will cut corners when it runs the plan. It will prefer the shortest path to "green." So you split responsibilities: generation can be shared, verification must be independent.

Sanity vs full tests

We want two modes.

Sanity is what runs every time. It is short, high signal, and focuses on "is the app basically alive." Login, one core flow, one write, one read, one permission check. The most well-trodden happy path.

Full is what runs when risk is higher. First deploy or big diffs. Auth changes. Payment flows. Migrations. New dependencies. Full is slower, but it is targeted. It should not be "run everything always."

Speed vs accuracy

We need QA under a minute to keep the iteration loop tight. The hard part is that accuracy is expensive.

If we use a top-tier model for every tiny decision (every hover, every scroll, every "what button is this"), we might get slightly better decisions, but we blow the budget and the latency. If we send the model the entire DOM, a full screenshot, and the full interaction history on every step, we might get more context, but we slow everything down and drown the model in noise.

Building a QA agent is balancing these tradeoffs: when do we spend intelligence, and when do we run on rails. What context is essential, and what is just bulk.

Handling backend state

Backend state is the main thing that breaks black-box QA. We solve it by isolating runs so they can't contaminate each other. That means a dedicated backend per run, or hard isolation at the database and queue level.

We then make state management explicit. Either we reset to a known baseline between tests, or we keep a shared baseline and only run tests that are designed to be state-safe. Once isolation and resets are real, we can safely run tests in parallel without one flow poisoning another.

The QA agent loop

At this point, the hard part isn't the model. It's the system around it: how we split work, what we send to the model, and how we keep runs fast, cheap, and reliable.

We build it to be cheap and fast, without letting it cheat. We use a multi-agent setup: a strong manager model plans, routes, and decides when to escalate, while smaller worker models execute steps quickly and in parallel. We parallelize whenever flows don't share state, and we keep deterministic intervention points: hard timeouts, bounded retries, and clear escalation rules.

For perception, we use both VDOM and pixels. The VDOM is great when it's clean and accessible. Pixels are how we survive overlays, canvases, and cases where the DOM lies. DOM compression into some VDOM is critical, otherwise we drown in noise and latency.

We keep actions as close as possible to real user actions. Click, type, select, scroll. We avoid injecting big scripts that do things a user can't do, because that produces fake greens. And we handle the boring realities that decide whether black-box QA works: scroll and below-the-fold elements, modals and toasts, focus traps, mobile vs web viewports, and interaction differences across devices.

The feedback loop back into the coding agent

The point of black-box QA in an AI-built app platform is not a report. It's a loop. QA produces a hard signal, and the coding agent uses that signal to make a focused fix, then QA reruns to confirm the fix actually worked.

We feed the coding agent more than just "pass or fail." We expose what the app actually did during the run: runtime errors, browser console logs, and the error logs produced by the QA cycles themselves. That extra telemetry is what turns a vague failure into something the agent can debug. The loop is simple but essential: run black-box QA, capture the smallest set of high-signal errors, patch, rerun the same flow, repeat until it's clean.

What this unlocks

AI-built apps make building cheap. QA agents make shipping cheap.

Once black-box QA is automated and wired into the loop, a few things become possible at the product level. We can let users iterate aggressively without turning every prompt into a production gamble. We can catch breakage the moment it's introduced, not when a user stumbles on it. And we can keep raising quality over time, because every failure becomes a new guardrail for the next run.

Key takeaway

If code can appear without a human author, confidence has to come from somewhere else. A QA agent is that somewhere else: continuous, external verification that turns AI-built apps from demos into software you can actually run. On AppDeploy, this runs automatically on every deploy, closing the loop between the coding agent and the live app.

FAQ

Can an AI agent test its own code?

It can help generate test plans and suggest scenarios, but it should not be the final authority. When the same agent writes the code and the tests, both reflect the same assumptions, which creates correlated failures. Independent verification in the real runtime environment is what produces trustworthy results.

What is black-box QA for AI-built apps?

Black-box QA means testing the deployed app the way a real user would: clicking, navigating, calling APIs, and verifying outcomes, without access to the source code or internal state. This avoids the same brain problem where the builder's assumptions leak into the tests.

How fast does QA need to be for AI-built apps?

Fast enough to run on every deploy without breaking the iteration loop. On AppDeploy, the QA agent completes a sanity pass in under a minute. A full test suite runs when risk is higher (first deploy, auth changes, new dependencies) and takes longer, but is targeted rather than exhaustive.

How does AppDeploy handle QA for apps with a backend?

Each QA run gets an isolated backend environment so tests cannot contaminate each other. State is either reset to a known baseline between tests or tests are designed to be state-safe. This isolation makes it possible to run flows in parallel without one poisoning another.

Build & Deploy an AI Nutrition Tracker from ChatGPT or Claude

AppDeploy Team — Sat, 02 May 2026 19:17:12 +0000

Most nutrition apps fail in the same place: they ask you to adapt to the product.

You search through someone else's food database or barcode workflow. You accept someone else's dashboard. You pay for the features that should have been basic. And when your goals change, the app usually does not change with you.

So we stopped looking for the right tracker and built a personal AI nutrition tracker in Claude.

Within a minute, AppDeploy turned the conversation into a live URL. A few iterations later, we were logging meals by text and photo, saving recurring foods to a pantry, tracking protein and calories, and getting reminders on our own schedule. We called it Calorit.

The point was not to make another tracker. It was to make a personal tracker that adapts to your habits instead of forcing you into a generic subscription product.

That became the frame:

Memory: it learns your recurring foods and preferences.
Personalization: calories, protein, macros, reminders, and tone are all editable through chat.
Ownership: no tracker subscription, no vendor-defined feature roadmap, no waiting for a product team.

The same prompt-deploy-iterate loop works in ChatGPT with AppDeploy too. This run happened in Claude.

This post is about how that conversation went, what got built, and why your version of Calorit can look nothing like ours.

Who this is for

This tutorial is for people who want a personal AI calorie or macro tracker without building backend infrastructure, setting up a database, wiring auth, thinking about storage, or handling AI integrations. You describe the app in Claude or ChatGPT, and AppDeploy turns it into a live webapp.

What we wanted from an AI nutrition tracker

Before typing anything, the requirements were short:

Log meals by typing or by photo - no searching through a food database
One screen that shows calories and protein remaining for today
A weight trend chart that doesn't require a separate app
An AI that learns the foods we actually eat and stops asking the same clarifying questions
Scheduled routines - reminders, daily resets, morning coaching - all tuned to our local clock, not the server's

Just as important were the things we didn't want: no streaks, no badges, no shame messages, no premium tier locked behind a paywall.

Step 1: Prompt Claude or ChatGPT

The first message to Claude was rough:

Build a calorie and macro tracker. I want to type or photograph what I ate
and have the app estimate calories, protein, carbs, and fat against a daily
target I set during onboarding.

Claude generated the app. AppDeploy deployed it. Within a minute we had a live URL. We opened it on a phone, signed in with Google, ran through a four-step onboarding, and logged a real lunch. It was missing things. That was fine - we stayed in the same chat and kept going.

This is chat-native deployment in practice: turning ideas described in an AI chat into a live app, without leaving the chat or needing to understand infrastructure. For the full platform workflow, see how AppDeploy works.

Step 2: Iterate on the app in chat

The first version was usable but bare. We didn't switch to an issue tracker to plan what to add next. We just kept typing into the same chat.

"I keep eating the same Yogurt Pro - stop guessing the macros, let me save it." The next deploy added a pantry. We snapped the nutrition label and the AI extracted the macros directly from the photo. From then on, "had Yogurt Pro after training" used the exact label numbers, not an estimate.

"Remember that I avoid whey." The app grew a long-term memory of food preferences that the model reads back into the system prompt on every chat call. Suggestions stopped recommending the thing we won't eat.

"Ping me at 21:00 if I haven't logged anything, and again in the morning if yesterday ran heavy." Two scheduled jobs got added, smart enough to fire at 21:00 and 08:00 in our local timezone. The cron runs every fifteen minutes server-side and only triggers per user when the local hour matches the target hour for that user's profile timezone.

"The dashboard is too noisy. Strip it down to calories remaining and protein remaining." Done in the next deploy.

Each of those was one chat turn. Each one redeployed automatically. The URL never changed.

After each deploy, AppDeploy ran QA against the live app, checking sign-in, onboarding, meal logging from text and photo, pantry product extraction from a label, and the scheduled coach messages.

The full AI nutrition tracker prompt

By the end of the session the app had everything we wanted. Here is the prompt that got us to our starting point - the app has kept evolving since, but this is a solid foundation to build from:

Build a personal AI calorie and macro tracking app called Calorit.
It should work like this:

- User signs in with Google. First time, run a four-step onboarding:
  current weight (kg), height (cm), birthday, sex, activity level,
  timezone, target weight, daily calorie deficit, protein target in
  g/kg, and optional food preferences in plain text.
  Calculate daily calorie target using Mifflin-St Jeor BMR x activity
  multiplier minus the deficit. Protein target = bodyweight x g/kg.

- Dashboard shows calories remaining, protein remaining, recent meal
  entries, and a weight trend chart. Let the user log weight directly
  from the chart.

- Chat is the main way to log food. The user types what they ate
  ("Yogurt Pro and an apple") or attaches a photo of the food
  or a nutrition label, or both. Extract the items, estimate calories,
  protein, carbs, fat, and write a short friendly analysis. If a saved
  pantry product matches by name, use its saved macros. The user should
  also be able to say things like "actually that was 300g not 200g" to
  update an entry, or "delete lunch" to remove one - ask for
  confirmation before deleting. If the message is unclear, ask a
  follow-up question instead of guessing. Each entry gets a meal type
  (breakfast, lunch, dinner, snack). Add quick-action buttons for
  "daily summary", "macro review", and "suggest a meal".

- History page with a calendar. Each day shows total calories, protein,
  carbs, and number of meals. Tapping a day opens the entries for that
  day with per-item macros.

- Settings page to edit the profile, manage a pantry of saved products
  (with macros - let the user photograph a nutrition label to fill them
  in), and toggle push notifications.

- Save food preferences and patterns the AI notices over time (e.g.
  "avoids whey", "prefers high-protein dinners") and read them back
  on every chat call so suggestions get better without the user
  repeating themselves.

- Evening notification at 21:00 in the user's local timezone: if
  nothing logged, nudge them; if logged, send a quick snapshot of
  the day. Morning notification at 08:00 only if yesterday went
  25%+ over target - one short supportive message, no shaming.

Light UI, blue accents, no streaks, no badges, no gamification.

You don't have to start with this. Start simpler and iterate - that is how we did it.

What you need

A ChatGPT or Claude subscription
The AppDeploy app for ChatGPT or the AppDeploy connector for Claude

That's it. No server, no hosting account, no Git, no third-party tracker subscription.

What the AI nutrition tracker includes

After deploying, sign in and run a short four-step onboarding: body composition, daily context, target calibration, and food preferences. After that, the app has four main views:

Dashboard - calories and protein remaining at a glance, today's recent entries, and a weight trend chart with a built-in widget for logging today's weight directly from the graph.
Chat - the main logging interface. Log meals by text, photo, or both. The AI extracts macros and writes a short analysis. Entries can be added, updated, or deleted through natural conversation - the AI confirms before removing anything. Quick-action buttons for daily summary, macro review, and meal suggestions.
History - a calendar view of past entries. Each day shows total calories, protein, carbs, and meal count. Tap a day to browse individual entries with their full macro breakdowns.
Settings & Pantry - edit the profile, save common products with their macros by photographing the nutrition label, and toggle push notifications.

The evening check-in fires at 21:00 in your local timezone - today's snapshot if you logged, a nudge if you didn't. The morning coach only fires after a day that ran 25%+ over target - one short, supportive reset message, no streak to break.

Try the Calorit demo

You can explore a working version of the app here:

Open the Calorit demo

What AppDeploy handled

Calorit is not just a static page. It needs sign-in, user profiles, meal history, saved pantry products, image uploads, scheduled notifications, and AI calls.

AppDeploy handled the deployment layer: hosting, database, storage, auth, scheduled jobs, notifications, and live QA after each deploy. We stayed in the chat and kept changing the app.

Customize the tracker for your own goals

Because Calorit came out of a conversation, every part of it is up for grabs. There is no feature request to file and no maintainer to convince. You just say what you want next.

A few directions the same starter app could fork into:

A vegetarian who tracks iron and B12. Add the micronutrients to the daily target, surface them on every meal extraction, flag low days.
An endurance athlete on training blocks. On days tagged as a long run, raise the calorie target by 600 and the carb target by 80g.
A parent cooking for the family. Add a "family meal" mode that takes one logged recipe and divides the macros across each person by bodyweight.
Someone tracking carbs with a clinician's guidance. Lead with carbs instead of calories, add optional glycemic-load estimates, and only share summaries with explicit consent.
Someone who wants a calmer experience. Drop the AI analysis text, keep the numbers, shorten the morning message to one sentence.

Each of those is one prompt. None require touching code, opening a CLI, or convincing a vendor to add a feature. The customization is the point.

Frequently asked questions

Can I build an AI nutrition tracker without coding?

Yes. We typed in plain English the whole time. The AI wrote the code; AppDeploy deployed it. If you can describe what you want, you can build and keep changing an app like this.

What does AppDeploy do, exactly?

AppDeploy is the deployment layer that connects AI chat to a live, running application. You describe what you want in ChatGPT or Claude, the AI generates the code, and AppDeploy handles everything after that: bundling, hosting, database, auth, scheduled jobs, push notifications, and built-in AI. The result is a real deployed app at a live URL - not a preview, not a sandbox. You never leave the chat.

Is there a Calorit subscription or an app to install?

Neither. There is no Calorit company. It is an app you generate in your own chat, on your own AppDeploy account. You do not need a third-party tracker subscription. AppDeploy itself is free to connect.

Can I really keep changing it after the first deploy?

Yes. Every message in the same chat is another iteration - new field, removed screen, different coach tone, different unit system, an extra metric. AppDeploy redeploys automatically and the URL stays the same.

Can the tracker log meals from photos?

Yes. Calorit can log meals from text, food photos, nutrition-label photos, or a mix of them. For saved pantry products, it can reuse the nutrition-label values you reviewed and saved instead of estimating from scratch.

Can I use this as an AI calorie tracker?

Yes. Calorit can track calories, protein, macros, weight trends, and reminders. The same app can also be reshaped into a macro tracker, carb tracker, or meal-logging assistant.

How accurate are the calorie estimates?

For packaged foods saved to your pantry, Calorit uses the nutrition-label values you reviewed and saved instead of estimating from scratch. For freeform meals and photos, treat the numbers as useful estimates, not lab-grade measurements. The point is consistency over precision: you log every day instead of nothing, and the trend reveals more than perfect arithmetic ever would.

Is this medical advice?

No. Calorit is a personal tracking app, not medical advice, diagnosis, or treatment. Use it for general self-tracking. If you need clinical guidance, work with a qualified professional.

What data should I avoid putting into the app?

Avoid regulated health data, medical records, government IDs, payment card data, secrets, or anything you would not want stored by a personal web app. If your use case involves compliance obligations, design the app accordingly before collecting real user data.

What if I don't want a nutrition tracker at all?

Then describe a different one. The same loop - chat, generate, deploy, iterate - works for whatever you want to build. See the gallery for other examples.

Build your own AI nutrition tracker

Connect AppDeploy to ChatGPT or Claude
Paste the prompt above into a new chat - or start simpler and iterate
Sign in, run the four-step onboarding, log your first meal in the chat tab

The same pattern works for sleep tracking, habit logs, training journals, symptom diaries, and plenty of other personal data apps that today live in spreadsheets or half-used subscriptions.

The first evening check-in fires at 21:00 your time. You won't have to remember.

DEV Community: AppDeploy Team

Vibe Code with Confidence: Testing AI-Built Apps with QA Agents

What a QA agent does for AI-built apps

Why the coding agent can't fully test itself (unit and end-to-end testing)

Writing code and proving code are different jobs

The same brain problem

"Green" is easy to optimize for

What this means for AI-built app platforms

Why real QA is difficult, especially with a backend

State is a problem

Backend state is harder than frontend state

Background jobs make tests flaky

Failures, retries, and timeouts are a big part of QA

How we built our QA agent

Test plan: where it comes from

Sanity vs full tests

Speed vs accuracy

Handling backend state

The QA agent loop

The feedback loop back into the coding agent

What this unlocks

Key takeaway

FAQ

Can an AI agent test its own code?

What is black-box QA for AI-built apps?

How fast does QA need to be for AI-built apps?

How does AppDeploy handle QA for apps with a backend?

Build & Deploy an AI Nutrition Tracker from ChatGPT or Claude

Who this is for

What we wanted from an AI nutrition tracker

Step 1: Prompt Claude or ChatGPT

Step 2: Iterate on the app in chat

The full AI nutrition tracker prompt

What you need

What the AI nutrition tracker includes

Try the Calorit demo

What AppDeploy handled

Customize the tracker for your own goals

Frequently asked questions

Can I build an AI nutrition tracker without coding?

What does AppDeploy do, exactly?

Is there a Calorit subscription or an app to install?

Can I really keep changing it after the first deploy?

Can the tracker log meals from photos?

Can I use this as an AI calorie tracker?

How accurate are the calorie estimates?

Is this medical advice?

What data should I avoid putting into the app?

What if I don't want a nutrition tracker at all?

Build your own AI nutrition tracker