DEV Community: Ayush Niroula

Asking ChatGPT 'would people buy this?' is the worst market research you can do

Ayush Niroula — Sun, 25 Oct 2026 08:45:03 +0000

Everyone does it. You paste your landing page into a chatbot and ask: "Would people buy this?"

It says something encouraging, suggests a few tweaks, and you feel validated. That's the problem.

Three reasons one chatbot answer misleads you

1. It's one voice. Your market is students on prepaid data, parents who ask their partner first, and someone who lost money to an online course last year. One assistant answering "as a typical customer" averages all of them away.

2. It tends to agree with you. Chat assistants are tuned to be helpful and agreeable. "Would people buy this?" is a question that invites a yes.

3. Its numbers are too neat. Ask a model to rate purchase intent from 1 to 5 and you get narrow, unrealistic distributions. A 2025 paper on semantic similarity rating (arXiv 2510.08338) found that letting the model answer in its own words, then scoring those words against anchor statements, comes much closer to real survey panels.

What to do instead

Build a population, not a persona, and let more than one model speak.

That's what MarketMirror does. In one Fitloop run (an invented fitness app), 2,999 simulated people were written and interviewed: a third by Claude, a third by Codex, a third by Gemini. Each was asked about money this month, their phone and data plan, who they ask before buying and a time they got scammed. Then they saw the offer.

Here's what happened when the same offer was answered by different models (all simulated):

People written by	Answered by	Yes / maybe / no
Claude (1,000)	Claude	1.3% / 75.6% / 23.1%
Gemini (1,000)	Claude (stand-in)	0.9% / 69.7% / 29.4%
Codex (999)	Codex	0.0% / 5.7% / 94.3%

One model said "maybe". The other said "no" almost every time. If you'd asked just one of them, you'd have walked away with very different confidence about the same launch.

(The Gemini row is Gemini-written people answered by Claude: Gemini's quota ran out mid-run. Those people answered like Claude's, which says the model doing the answering shaped the result far more than the one that wrote the person.)

What this buys you

A range, not a verdict. When models disagree, the honest answer is "between these two", and you see it.
The objection that survived every model. All three groups named the same top objection: trust (59.6% overall). That's the one to fix first.
Free-text reasons per person, grouped by audience, so you know who says no and why.

The limits (read these)

The people are synthetic. Agreement between models is a robustness check on the simulation, not evidence about real buyers. Use it to choose what to test for real, then run that real test small.

Try it

Open source (AGPL-3.0). Works from Claude Code, Codex, Cursor, Gemini or any MCP agent. Ask it in plain words.

👉 https://github.com/DarkWorldCoder/market-mirror (a ⭐ helps other people find it)

Written with help from an AI assistant; every number was checked against the saved run results.

We showed 8 real Facebook ads to 120 simulated buyers. It ranked them backwards, and still found the leak.

Ayush Niroula — Sun, 25 Oct 2026 04:34:50 +0000

Simulation tools love to show the run where they were right. This is a run where one was wrong, and why it was still worth doing.

The setup

csca.ai sells prep for the CSCA exam (the test for studying in China) to students and parents in South and Southeast Asia. They ran 11 Facebook ads for a free mock test across Nepal, Indonesia, Bangladesh, Vietnam and Sri Lanka, and let us use the real results.

Real results (Meta Ads Manager): $56.05 spent, 274,557 impressions, 4,466 clicks, 1,500 landing page views, 16 sign-ups.

Then we gave MarketMirror the 8 that got most of the budget, plus a brief, without the results. It built 120 simulated students and parents (8 audiences × 15 people, each with a short life interview), put the ads in realistic Facebook feeds three times over with different random seeds, and ran a survey on the sign-up offer.

Where it was wrong: clicks

We compared the simulated click ranking with the real one. Rank correlation: -0.84. That's not "a bit off", that's almost backwards. The simulation's favourite ad was the Nepal broad audience; the real winner was the Indonesia parents ad at 2.32% CTR.

If you'd used it to pick which ad gets the budget, it would have pointed you the wrong way. We'd rather say that out loud than hide the run.

Where it was right: the leak

Look at the real funnel again: 1,500 people reached the page and 16 signed up. About 1 in 100.

The simulated buyers had already said why, in their own words and their own languages:

"Gratis tapi minta password lagi? Jangan." (Free, but asking for a password again? No way.) (Indonesia)

"Phone number dhele spam asbhe." (Give them my number and the spam starts.) (Bangladesh)

"Free sign-up? They want my phone and email. Harvesting data for sure." (Nepal)

In the survey, trust ("looks like a scam") was the top blocker for 56% of the simulated people, and privacy for 28%. The real sign-up form asks for full name, email, phone number, country and a password, for a free test.

The ad wasn't the main problem. The page after it was.

What it suggested next

From the real ad data plus the run, MarketMirror's ideas step wrote five next tests. Two of them:

Zero-friction entry: a 3-question quiz first, no email, no personal data.
Privacy-first page: say plainly how the data is used, cut the form to 3 fields.

These are proposals, not results. The point is the order: fix the page before buying more clicks.

What we take from it

Don't use a simulation to rank ads by CTR. At least not until a backtest on your own past ads says it can. (MarketMirror has backtest and calibrate for exactly this.)
Do use it to find the "why". Objections, in buyers' own words, by audience, before you spend.
Real data in, next test out. The useful loop is: export real results → compare → test the fix on simulated buyers → run the small real test.

Try it

MarketMirror is open source (AGPL-3.0) and runs from Claude Code, Codex, Cursor, Gemini or any MCP agent.

👉 https://github.com/DarkWorldCoder/market-mirror: if it saves you one bad launch, a ⭐ helps other people find it.

Real ad results shared with csca.ai's permission. Simulated buyers are synthetic; their quotes are labelled. Written with help from an AI assistant; every number was checked against the saved data.

I simulated 2,999 buyers with three AI models. The one answering mattered more than the one that wrote them.

Ayush Niroula — Sat, 24 Oct 2026 03:20:50 +0000

I'm building MarketMirror, an open-source tool that tests ads, prices, messages and sales pitches on simulated buyers before any money goes into a real campaign. It ends with a plain-language report that says go, iterate or no-go.

The obvious objection to any "synthetic customers" tool is: aren't you just measuring the language model's mood? I wanted to see that effect directly, so I ran one market with three model providers at once. Every number below is simulated. Fitloop, the product, is an invented fitness app.

The setup

The advertised offer was a 7-day free trial with no card, then USD 4.99 a month. The brief and the landing page still asked for a card at sign-up, on purpose: the test is meant to catch that.

I requested 3,000 people in six audiences across India, the Philippines and Kenya. They belonged to five communities: two WhatsApp groups, two Facebook groups and r/budgetfitness. A third each were written and interviewed by Claude Sonnet, Codex (gpt-6-luna) and Gemini 3.8 Flash. In the backend's settings:

LLM_POOL=claude_code:sonnet,codex:gpt-6-luna,antigravity:gemini-3.8-flash-low
LLM_POOL_STRICT=true          # keeps rounds and tests on each person's own provider (until a provider runs out; see below)

Each provider writes an equal share of every audience and interviews its own people. It then answers for them in the feeds, social rounds, survey and pitch, so one model speaks for a person from start to finish, unless a provider runs out mid-run (which happened here; see below). In total, 2,999 people were written and 2,987 were interviewed. The one person whose calls failed is counted, not filled in.

How one simulated person is built

Each audience is spread over slots (age, gender, city, income, life stage), so you don't get 500 copies of the same 24-year-old. Then, for each person:

a 150-250 word persona;
a stereotype guard, in code with no model call, so that no two people in an audience share both a job and a city (in this run it changed 20 people);
a 12-question life interview: how money is this month, what phone and data plan they have, a time they got scammed, who they ask before spending, what annoys them in ads;
follow-ups and a decision card: triggers, stoppers, dealbreakers, trust cues, and where what they say differs from what they do.

Here's one person as the model wrote him. Yash is 42 and lives in Pune. He drives an app cab and runs a mobile-accessories stall at weekends. He once lost ₹6,000 to a crypto course, so now he reads reviews and asks his wife and mother before buying. His says-vs-does line: he says he'd do 15 minutes of exercise while waiting for rides, but his shifts run long, so he probably won't.

What they went through

./mm cli feedtest --simulation $SIM --creatives @creatives.json   # add --seeds 3 for a range; this run used one seed
./mm cli survey --simulation $SIM --offer "7-day trial, no card" \
  --messages @messages.txt --split --pricing USD
./mm cli pitch --simulation $SIM --script @pitch.md --rebuttals @rebuttals.md

In practice, a single market-test config runs all of this.

Feed test: five ads placed among organic posts in Facebook, Instagram, TikTok and Reddit feeds. Each person sees one ad, assigned at random. In total there were 5,140 person-platform sessions.
Social rounds: four Reddit-like rounds on OASIS produced 3,565 actions, and 1,259 of those were "do nothing".
Survey: answers come back as free text. Semantic similarity rating (SSR, arXiv 2510.08338) then compares each answer against five anchor statements using local embeddings, which gives a 1-5 distribution without another model call. When you ask a model for a number directly, you get narrow, unrealistic rating distributions. The SSR paper reports that rating the model's own words comes much closer to real survey panels.
Pitch: every person hears the pitch, raises an objection, gets my rebuttal, and decides again.
Analyst panel: five simulated analysts read the run independently.

The headline (simulated)


Decision	iterate (medium confidence)
Would start the trial	0.7% yes, 50.4% maybe, 48.9% no
Purchase intent (1-5)	2.88 ± 1.14
Top objection	trust / "looks like a scam", 59.6%
Best ad	`fb_parents`, expected CTR 1.02%
Analysts	5 of 5: iterate

The signals disagree, and the report says so. The survey alone points to no-go, while the feed test and the analysts point to iterate.

The models disagreed

People written by	Played by	Yes / maybe / no (%)
Claude (1,000)	Claude Sonnet 5.5	1.3 / 75.6 / 23.1
Gemini (1,000)	Claude Sonnet 5.5 (stand-in)	0.9 / 69.7 / 29.4
Codex (999)	Codex	0.0 / 5.7 / 94.3

The stand-in row wasn't planned. Gemini's login ran out of quota for about four days mid-run, so Claude played the Gemini-written people (MM_PROVIDER_STANDIN=antigravity=claude). The results label those people "antigravity (played by claude)". That accident became the most useful comparison in the run: people written by Gemini but played by Claude answered like Claude's people, not like Codex's.

So in this run, the model doing the answering shaped the survey far more than the model that wrote the person. All three groups still named trust as the top objection. I read every rate above as a range between two temperaments, not as one market's answer. The dashboard's "Do the models agree?" section is built for this. It splits results by provider with 95% ranges and checks whether the would-convert ranges overlap and whether the top objection and best message match. Agreement is a robustness check on the simulation, not evidence about real buyers. This run can't tell me whether Codex's people are more realistic or just more negative. Only real data can settle that.

The pitch made it worse

Before the rebuttals: 1.1% yes, 64.0% maybe, 34.8% no. After them: 1.1% yes, 44.6% maybe, 54.2% no. The pitch scored 23 out of 100. The trust rebuttal moved 44.9% of the people who raised trust, and the price rebuttal moved 8.9%. The report's fix was to stop arguing and instead show the exact charge after day 7, the cancel steps and the local price. That matched the analysts' themes: no local payment option (UPI), a USD price competing with free YouTube, and a card wall that kills the trial.

What it took

Writing 2,999 full-depth people took 2 h 40 min through three login gateways. That includes a restart midway to apply a fix; a cache meant finished people weren't rewritten. The social rounds then kept hitting usage limits, so I moved the rest of the run to an Anthropic API key. From there it took 40 minutes for four rounds, 23 for the survey of 2,999 plus its retest, and 37 for the pitch. MarketMirror didn't meter the tokens used inside the rounds, survey and pitch, so I can't give you a cost for this run.

Limits you should know

The people are synthetic. Detailed biographies don't make them representative.
Attention, fatigue, social proof and landing friction include hand-set priors.
The spread across seeds measures simulation variation, not predictive confidence. Test-retest (76.7% gave the same verdict here) measures consistency, not agreement with humans.
Fitloop is invented, so none of this has been checked against real outcomes. backtest and calibrate exist for that, but they need your paired real data.

Try it

git clone https://github.com/DarkWorldCoder/market-mirror.git
cd market-mirror
bash install.sh
./mm configure
./mm provider check
./mm start
./mm cli quickstart --out scenarios/acme

Quickstart asks about your product, buyers and channels. It prepares the people, then stops before the remaining tests so you can review them. Preparation itself uses model calls, so start with a small cohort. From Claude Code, Codex, Cursor or Gemini, you can install the plugin or MCP server and ask in plain words.

The repo is at https://github.com/DarkWorldCoder/market-mirror (AGPL-3.0). If you've run real surveys or ad tests and have opinions on how to validate something like this, I'd like to hear them.

I wrote this with help from an AI assistant and checked every number against the run's saved results.

10,000 Times Cleaner Code Just By Using This Tool.

Ayush Niroula — Wed, 25 Sep 2024 03:45:58 +0000

Ever stared at your codebase thinking, “Wow, that’s a lot of console.log clutter!"? Or maybe you’ve spent way too much time hunting down those rogue print statements you used while debugging. Well, good news—Eradicate (era) is here to rescue you from that mess! 🦸‍♀️

Era is a slick open-source tool that automatically cleans your codebase, wiping out unwanted logs, prints, and debugging statements across multiple languages. No more manual code cleanup or dealing with bloated files filled with leftover logs. It’s like Marie Kondo for your code! ✨

In this story, I’m going to show you how era works, why it’s a game-changer, and how you can get started ASAP.

🎯 Why Era? Because Nobody Has Time to Manually Clean Code! ⏳
Picture this: You’re deep in the middle of a project, dropping console.log and print() statements left and right to debug your code. By the time you finish, you’ve got a sea of logs floating around your files, making it hard to see the real logic.

Enter Eradicate (era). It’s like having a digital cleaning assistant that sweeps through your codebase, finds those pesky outputs, and wipes them out faster than you can say “refactor.” Whether you’re working on a personal project or contributing to a large team, era automates the boring cleanup part, so you can focus on the fun stuff — like, you know, actually coding. 😎

⚡️ Key Features You’ll Love
Multi-language support: Clean up logs in JavaScript, Python, PHP, Go, Ruby, and more. No matter the language, era has your back. 💻
Customizable Config: You decide what gets cleaned and what stays — because not all logs deserve the boot.
Target Specific Files/Folders: Just finished working on a feature? Run era on a single file or folder. Easy peasy.
Ignore What Matters: Don’t want to clean certain files or directories (like node_modules)? Era makes it simple to exclude what you don’t want to touch.
🚀 How to Get Started with Era
Ready to clean up your code like a boss? Here’s your step-by-step guide.

Initialize Era in Your Project First things first — you need to let era know where to get started. Run the following command to initialize era in your project:

era init

This creates a shiny new erd.yaml config file in your root directory. It’s your personalized cleanup map where you define exactly what needs scrubbing.

Clean Up Logs and Prints Like a Pro Time to make your code spotless! Run the clean command, and era will scan your project and clean out all those pesky logs:

era clean

By default, era will check your erd.yaml file for cleaning rules. If it doesn’t find one, no worries—it’s got default settings ready to go. 🧼✨

Target a Specific File or Folder Don’t need to clean the whole project? No problem. You can target individual files or folders:

To clean a specific file:

era remove -f path/to/yourfile.js

Or to clean a whole folder:

era remove -d path/to/yourfolderEra will carefully check the file or folder, hunt down the unwanted logs, and leave the rest untouched. 🕵️‍♀️

🔧 The Magic Behind Era’s Config File
Let’s talk about the secret sauce: the erd.yaml configuration file. Here’s a breakdown of what it does and how you can tweak it to fit your project like a glove:yml

Root: "."
ReportPath: "era-reports"
IgnoreKeyword:
  - "erd:ignore"
  - "erd:ignoreAll"
IgnoreFileExtensions:
  - ".exe"
  - ".dll"
  - ".zip"
  - ".tar"
IgnoreDirs:
  - "node_modules"
  - ".git"
  - "vendor"
IgnoreFiles: []
ListenType: "command"

Root: This is where era starts its search — usually your project’s root directory.
ReportPath: After each run, era will generate a report in this directory, so you can see exactly what was cleaned. 📊
IgnoreKeyword: Add custom tags to your code like erd:ignore to stop era from cleaning certain sections.
IgnoreFileExtensions: These are the file types era won’t touch — think binaries or compressed files.
IgnoreDirs: Directories like node_modules or .git are automatically excluded to save you time. No one needs to clean those, right? 😅
ListenType: Currently, this is set to “command,” meaning era listens for cleanup commands from you. More listen types may come in the future — stay tuned!
🎉 The Best Part: It’s Free and Open-Source!
Yup, you heard that right. Eradicate (era) is open-source, meaning you can use it, modify it, and even contribute back to it! It’s all about helping fellow developers keep their codebases clean without the hassle.

💡 Final Thoughts
Eradicate isn’t just a tool — it’s a timesaver, a productivity booster, and a clutter-buster. Whether you’re working solo or part of a larger team, using era helps you maintain a tidy codebase with minimal effort.

If you’re tired of manually cleaning up your files, give era a spin. It’s free, easy to set up, and — trust me — it’ll save you hours of time.

Join the revolution! 🚀 You can find the code on Github and contribute to making this tool even better.

Happy Coding! 💻✨

Eradicate (era): A Powerful Open-Source Tool for Code Cleanup

Ayush Niroula — Tue, 24 Sep 2024 01:12:56 +0000

Thrilled to announce my open-source project Eradicate (era) 🎉 era is a versatile tool that simplifies cleaning and transforming codebases across multiple languages. From removing unwanted logs to customizing file cleanup, era helps developers maintain cleaner and more efficient projects.

Check it out, give it a try, and contribute to making development a little bit easier for everyone! 🌟

https://github.com/wadmit/era