DEV Community: Sarah Pan

I Asked Three AI Models the Same Question. Now I Had Three Problems.

Sarah Pan — Thu, 23 Jul 2026 06:29:05 +0000

I used to think that asking multiple AI models was a good way to verify an answer.

If GPT gave me an answer I wasn’t sure about, I would ask Claude. If Claude disagreed, I would open Gemini and try again.

In theory, the third model should settle it.

In practice, I often ended up with three different answers, three convincing explanations, and no idea which one to trust.

More answers didn’t mean more certainty

This happens to me most often when I’m working on questions with one correct answer.

One model chooses A and explains why B is wrong. Another chooses B and gives an explanation that sounds just as reasonable. The third sometimes introduces an entirely different interpretation of the question.

The strange part isn’t that they disagree. Different models are trained and tuned differently, so disagreement should be expected.

The strange part is how confident every answer sounds.

None of them says, “I might be misunderstanding this.” Each one presents its reasoning as if the problem has already been settled.

At that point, comparing answers becomes less like fact-checking and more like listening to three persuasive people argue about something I don’t understand well enough to judge.

Majority vote isn’t always the answer

A common solution is to ask several models and choose the answer that appears most often.

That can help, but only if their errors are independent.

If two models rely on the same common misconception, the majority can still be wrong. Models may also interpret the wording in similar ways or reproduce the same widely repeated but incorrect information.

Two matching answers are not automatically two separate pieces of evidence.

There’s another problem: if I ask the models one after another and include previous responses, they may start agreeing simply because they’ve seen the same reasoning. That creates consensus, but not necessarily accuracy.

Agreement can be useful. It just isn’t proof.

The disagreement is actually the useful part

I’ve started treating model disagreement as a signal to slow down.

Instead of asking, “Which model should I trust?” I try to find the exact point where their reasoning separates.

Did they interpret a word differently? Did one of them use an incorrect formula? Did they make different assumptions? Is one answer based on information that may be outdated?

Once the conflict becomes specific, it is much easier to verify.

For example, instead of giving one model the other model’s full response and asking which is correct, I can ask:

Model A chose A, while Model B chose B. Identify the exact assumption or reasoning step that causes the disagreement. Do not choose a winner yet.
Then I can ask each model to challenge its own answer:

What is the strongest reason your answer could be wrong? What evidence or test would distinguish between the competing answers?

This doesn’t guarantee a correct result, but it changes the models’ job. They are no longer competing to sound convincing. They are helping define what needs to be checked.

My current workflow

When an answer matters, I now use a simple process:

Ask each model independently, without showing it the other answers.
Compare the conclusions and list the exact points of disagreement.
Ask each model to critique its own reasoning.
Verify the disputed claim using a reliable source, calculation, or test.
Choose the answer supported by evidence—not the one written with the most confidence.

For coding questions, the final judge might be a test. For math, it might be recalculating the result step by step. For current facts, it should be a reliable and recent source.

The important part is that another AI model is not automatically the final authority.

More models can reveal uncertainty, not remove it

I still think using multiple models is valuable. A second model can catch assumptions the first one missed, and disagreement can expose weak points that would otherwise remain hidden.

But multiple models don’t automatically create a verification system.

Sometimes they produce consensus. Sometimes they produce a better answer. And sometimes they simply give you several polished versions of uncertainty.

The real advantage is not that one of them must be right. It’s that their disagreement can show you where verification is actually needed.

So now, when GPT, Claude, and Gemini give me three different answers, I don’t immediately ask for a fourth.

First, I ask them why they disagree.

Adding One Limitation Made the AI Recommendation More Confident

Sarah Pan — Wed, 22 Jul 2026 07:09:46 +0000

I recently tested two versions of the same product description to see whether a small wording change would affect how an AI evaluated it.

The first version was simple:

Gives developers access to multiple AI models through one API.

The second version added one sentence:

TokenBay gives developers access to multiple AI models through one API. It is most useful for teams that regularly compare or switch models, but may be unnecessary for projects that only use one provider.
I expected the second version to make the product sound less appealing. After all, product copy usually tries to remove doubt, not introduce it. Saying that some people may not need the product felt a little like putting a warning label on the homepage.

The model reacted in the opposite way.

Its recommendation became more specific and, strangely, more confident. Instead of describing the product as a generally useful multi-model platform, it explained that TokenBay made sense for teams testing several providers, comparing outputs, or trying to avoid maintaining separate accounts and integrations. It also noted that a small project committed to one model would probably not benefit as much.

Nothing about the product had changed. The model simply had a clearer boundary around when the product was useful.

That made me realize that limitations can give an AI something important: a reason to rule a product out.

Most product descriptions only explain why someone should use the product. They list features, benefits, and broad claims about who it is for. The result often sounds positive but vague. A platform is “flexible,” “powerful,” and “built for modern teams,” which could describe half the software products currently asking for my email address.

A limitation narrows the picture. Once the description says that the product may be unnecessary for single-provider projects, the intended user becomes easier to identify. The model can compare the product against a real situation instead of trying to interpret a list of features.

I do not think the lesson is that every product page should suddenly lead with everything the product cannot do. That would be honest, but perhaps not excellent marketing. The useful part is giving enough context for the product to be evaluated properly.

For example, these two statements communicate very different levels of information:

Supports multiple AI models.
and:

Designed for teams that regularly test or switch models and do not want to maintain separate provider integrations.

The first tells you what exists. The second tells you when it matters.

Adding a limitation makes that distinction even clearer:

Probably unnecessary if your application only relies on one provider.

Now the product has a recognizable shape. It is no longer trying to be useful to everyone, which makes the recommendation feel less like generic promotion and more like an actual judgment.

This may matter more as people increasingly ask AI systems to compare tools for them. A model needs enough information to decide not only why a product fits, but also when another option would make more sense. Without that boundary, the safest answer is often vague: the product “could be useful depending on your needs.”

With a clear limitation, the model can say something more helpful.

The part I found most interesting was that the limitation did not weaken the recommendation. It made the recommendation easier to justify.

Maybe good product descriptions should not only help a model rule the product in. They should also help it rule the product out.

That sounds slightly uncomfortable from a marketing perspective, but it may be what makes the recommendation believable.

Do Students Still Need to Learn Coding the Old Way?

Sarah Pan — Tue, 21 Jul 2026 07:24:43 +0000

A friend of mine teaches at a university, and she has been noticing that more students are using AI heavily for coding assignments and projects. Some of them cannot explain most parts of the code they submit, and they struggle to build the same thing from scratch, but their projects can still work surprisingly well. They are able to turn an idea into something people can actually use from making small apps, 2D & 3D games, connect APIs. When they meet some troubles in their code, they first asked their AI to debug or not go to the Office Hour or discuss with friends. A few years ago, students would usually need much more traditional programming knowledge before they could even get to that stage, so this shift makes me wonder what learning to code is supposed to look like now.

I do not think this means traditional programming skills have suddenly become useless. Students still need enough technical understanding to notice when the AI is producing something unreliable and to understand why a project becomes difficult to maintain when they finding a job. AI can generate a lot of code very quickly, but it can also generate a very large mess with impressive confidence. Without some knowledge of data structures, students may be able to make a project look finished without really knowing whether it is stable.

At the same time, it may no longer make sense to require students to spend months proving they can build everything without assistance before they are allowed to create something useful. In most real jobs, people already use frameworks, libraries, documentation, search engines, code examples, and now AI assistants. The ability to work with tools has always been part of programming. What has changed is how much those tools can now do, and how quickly someone with limited technical experience can reach a working first version.

The biggest change may be who gets to build. A design student with a strong visual idea can create an interactive prototype. A film student can make a tool for organizing footage or planning a production. A humanities student who understands a niche community can build something for users that a traditional engineering team might never have considered. In this environment, creativity, taste, subject knowledge, and the ability to describe a problem clearly become much more valuable, because technical execution is no longer the only barrier between an idea and a working product.

This does not necessarily mean that art or humanities students will become more competitive than engineering students. It may mean that the advantage is shifting toward people who can combine several kinds of knowledge. An engineering student who understands users, design, and business may become far more effective with AI. A creative student who learns enough technical basics to evaluate and improve AI-generated code may also become capable of building things that were previously out of reach.

Maybe the real question is no longer whether students should learn programming in the traditional way, but which parts of traditional programming still matter most. Should they spend as much time memorizing syntax, or more time understanding systems, testing, debugging, and making good technical decisions? Should assignments measure whether students can write every line alone, or whether they can explain, verify, and improve what AI helped them produce? These are also questions that universities and instructors need to take seriously.

From what I have seen, many computer science and engineering programs still discourage or completely prohibit the use of AI and vibe coding in coursework. That concern is understandable, especially when teachers need to know whether students actually understand the material, but the curriculum also has to keep pace with the tools students will encounter in the workplace. If schools continue teaching as if these tools do not exist, they may leave students at a disadvantage rather than protect their learning.

I am curious how other developers and teachers see this. If you were designing a programming curriculum for students who will grow up with AI coding tools, what would you keep, and what would you change?

Building a Model-Agnostic Agent Harness: What Breaks When You Switch LLMs

Sarah Pan — Mon, 20 Jul 2026 08:01:00 +0000

I’ve seen quite a few “multi-model agents” lately.

Usually there’s a dropdown in the corner of the screen that allows switching from one model to another. Which is useful. I also enjoy using dropdowns. It makes everything feel very configurable.

But then, when you click on that dropdown…

The model may return an answer that is in a different format than you asked for, or which calls into tools that either omit or include one that was previously not called at all.

The dropdown worked perfectly.

The agent, unfortunately, has developed a new personality.

The model changed. Everything around it noticed.

Imagine a small agent that reads a support ticket, looks up the customer’s account, and returns a suggested reply.

One model might return exactly what you asked for:
{ "priority": "high", "action": "refund", "reply": "I’ve issued a refund..." }

Another may return the same information with a short introduction:

`Here is the recommended response:

{
"priority": "high",
"action": "refund",
"reply": "I’ve issued a refund..."
}`

A third may decide recommended_action is a much nicer field name than action.

To a person, these answers are basically the same. To the next function in the pipeline, they may be three completely different situations.

This is where “supports multiple models” starts becoming less about access and more about everything wrapped around the model.

Tool use is not identical either

The differences become more obvious when the agent has tools.

One model may look up the customer account immediately. Another may ask a follow-up question first. A third may write a confident response without checking the account at all, which is efficient in the same way skipping all your tests is efficient.

Even when models have access to the same tools, they do not always make the same decisions about when to use them.

That means the harness has to do more than pass messages back and forth. It needs to check whether the required steps actually happened.

Did the model call the account tool before recommending a refund? Did it receive a valid result? Did it finish the task, or did it simply announce that it was finished?

Apparently, “done” is also a matter of opinion.

A failed agent does not always return an error

The annoying failures are rarely the dramatic ones.

A timeout is easy to notice. A rate-limit error is clear. You can retry or switch providers.

The stranger case is when the API returns 200 OK, the model gives you a polished answer, and the workflow has absolutely no idea what to do next.

The request succeeded. The task did not.

So a useful multi-model agent needs its own definition of success. The output may need to pass a schema check. Certain tools may need to appear in the trace. A value may need to fall within an expected range. If those checks fail, the harness can retry, repair the response, or move the task to another model.

Without that layer, model switching is mostly optimistic copy-pasting.

The less photogenic part of multi-model systems

We’ve been running into this while working on TokenBay.

Connecting several models through one API is the neat part you can show in a demo. The less photogenic part is making sure the rest of the workflow does not panic every time the model changes.

In practice, that means normalizing outputs, checking tool calls, handling different failure patterns, and keeping the application’s internal contract stable even when the model is not.

The models do not need to behave identically. They probably never will.

The workflow just needs to know what to do when they don’t.

A model dropdown can give users a choice. A real multi-model agent needs enough plumbing behind that choice to make switching boring.

And boring, in production, is usually a compliment.

50,000 AI Dramas a Month. So Why Have You Never Seen One?

Sarah Pan — Fri, 17 Jul 2026 05:48:37 +0000

On Douyin (China's TikTok), the total number of AI-generated comic/anime-style dramas that had ever aired before 2026 was around 58,000 titles.

In Q1 2026 alone that number jumped to roughly 180,000. About 50,000 of those were added in March alone. So, in one quarter, China produced more than double the entire back catalog that took years to build.

The reason is the cost curve, and it's not subtle:

Traditional live-action drama: ~$1,400 per minute of footage
AI-generated comic drama: as low as $14 per minute

That's a 100x difference. A full traditional series costs $40K–170K. A full AI comic-drama series costs $7K–21K.

Meanwhile, of the ~61,000 comic dramas released in 2025, only 96 crossed 100 million views. A hit rate of 0.16%.
And this isn't staying in China: Southeast Asia (Thailand, Malaysia, Philippines) is picking it up fast, with around a third of internet users there already watching this kind of content monthly. However, the mainstream Western audiences largely haven't joined in. In Spain, the highest-engagement country in Europe, only about 7% of internet users have watched one in the past month.

So the question I keep coming back to: is this what an industry actually maturing looks like, or is it just what it looks like right before the correction hits?

Curious if anyone's tracking similar cost-collapse → content-flood patterns in other media categories (stock photography, game asset generation, etc), feeling like the same shape.

The API Call Succeeded. The Workflow Still Broke.

Sarah Pan — Thu, 16 Jul 2026 06:24:26 +0000

We're building TokenBay, a unified API that routes requests across GPT, Claude, Gemini, DeepSeek, and a few dozen other models. A few weeks ago we ran a fairly boring internal test: send the same structured-output prompt to five different models and compare what came back. It turned into one of the more useful debugging sessions we've had this quarter, mostly because nothing about it looked broken at first.

The task was simple. Extract product, budget, and location from a user request, return JSON. Every request succeeded. Every model returned something a human would call correct. And our downstream workflow — the thing that actually parses the response and does something with it — broke anyway.

The responses all looked fine

Here's the input we sent:

I need a laptop under $800. I'm in California.

GPT-5.4 came back with exactly what we expected:

{ "product": "laptop", "budget": 800, "location": "California" }

Claude Opus 4.8 gave us the right values, but wrapped in a sentence first:

`Here is the extracted information:

{
"product": "laptop",
"budget": "$800",
"location": "California"
}`

Still correct, if you're a human reading it. Gemini 3.1 changed the field names entirely:

{ "item": "laptop", "max_price": 800, "region": "California" }

Also correct. DeepSeek V4 followed the schema but represented the missing currency differently again. Four models, four slightly different shapes, and our parser — which was expecting exactly product, budget, and location — choked on three out of four.

A unified endpoint doesn't mean unified behavior

This is the part that's easy to underestimate when you're evaluating a multi-model API. The pitch is usually about the request layer: one key, one endpoint, swap the model name and you're done. That part is true, and it's genuinely useful. But it quietly implies the models are interchangeable once you can call them the same way, and they aren't.

They interpret the same instruction differently. They make different assumptions about missing values. They disagree on how strict "return JSON" actually means. Two models can agree completely on the answer and disagree completely on how that answer should be formatted — and that difference is invisible in a demo, because a person is reading the output. In production, the next thing reading it is usually another piece of software, and it's a lot less forgiving than a person skimming a chat window.

Tightening the prompt helped, but not enough

Our first move was the obvious one — make the instruction stricter. We went from:

Return the result as JSON.
to
Return only valid JSON. Do not include any explanation or Markdown.

That fixed the Claude wrapper-text problem immediately. It didn't fix everything. Some models represented a missing field as null, some as an empty string, some just omitted the key entirely:

{ "product": "laptop", "budget": 800, "location": null }
{ "product": "laptop", "budget": 800, "location": "" }
{ "product": "laptop", "budget": 800 }

All three are valid JSON. None of them are the same contract. At that point it was clear we weren't asking the models for a format — we needed to ask our own workflow to enforce a schema, because the models were never going to agree on one for us.

The actual fix was a normalization layer, not a smarter prompt

Prompt tuning bought us a few percentage points, but it's fragile by nature — a model can follow an instruction 99 times out of 100 and still surprise you on the hundredth call. If that hundredth response ends up in a billing flow or an automated action, "usually correct" isn't a bar we were willing to accept.

So we stopped passing raw model output directly into the app. Everything now goes through a small adapter first: pull JSON out of any surrounding text, map alternate field names onto our internal schema, coerce currency strings into numbers, collapse empty strings to null, and reject anything incomplete outright. A simplified version of what that looks like:

type RawResult = Record<string, unknown>;

type ProductRequest = {
  product: string;
  budget: number | null;
  location: string | null;
};

function normalizeResult(raw: RawResult): ProductRequest {
  const product = raw.product ?? raw.item;
  const rawBudget = raw.budget ?? raw.max_price;
  const location = raw.location ?? raw.region ?? null;

  if (typeof product !== "string" || product.trim() === "") {
    throw new Error("Missing product");
  }

  let budget: number | null = null;

  if (typeof rawBudget === "number") {
    budget = rawBudget;
  } else if (typeof rawBudget === "string") {
    const parsed = Number(rawBudget.replace(/[^0-9.]/g, ""));
    if (!Number.isNaN(parsed)) {
      budget = parsed;
    }
  }

  return {
    product: product.trim(),
    budget,
    location:
      typeof location === "string" && location.trim() !== ""
        ? location.trim()
        : null,
  };
}

Nothing here is clever, and that's kind of the point — the real production version handles a longer list of edge cases (arrays instead of strings, nested objects, models that helpfully "correct" your currency to EUR), but the core idea doesn't change: stop trusting the raw shape and normalize before anything downstream touches it.

This also changed what we count as a failure

Before this, our fallback logic only triggered on request failure — timeout, rate limit, a 4xx or 5xx from the provider. If a request came back as 200 OK, we treated it as done.

But a 200 with an unusable payload isn't a success, it's a failure wearing a success status code. So we moved the fallback trigger downstream:

Send request → Receive response → Parse output → Validate schema → Normalize → Continue, or retry with another model

The question changed from "did the API respond?" to "did the API produce something the application can actually use?" Those aren't the same question, and building fallback logic around the first one instead of the second one will pass every test you write until it doesn't.

What we test now

We used to eyeball a handful of outputs and call the prompt "good enough." Now the test suite specifically checks for the boring stuff: extra text wrapped around the JSON, renamed fields, numbers returned as strings, omitted optional fields, inconsistent handling of missing values, and whether the normalized result from every model passes the exact same downstream checks. A response can be fluent, accurate, and still completely unusable to a parser — that's its own failure mode, and it needs its own test.

The takeaway

Being able to call five models through one API makes them easier to reach. It doesn't make them interchangeable. The call can succeed, the model can understand the task perfectly, the answer can even be right — and the workflow can still break quietly downstream. Routing the request to a different model is a config change. Deciding what the rest of your application is allowed to assume about that model's output is the actual engineering work, and it's the part that doesn't show up in a benchmark chart.

We ran into this while building TokenBay, a unified API for routing across GPT, Claude, Gemini, DeepSeek and other models. If you've hit similar cross-model quirks, I'd be curious to hear how you handled them.

The Best AI Model Is Usually the Wrong Question

Sarah Pan — Wed, 15 Jul 2026 06:25:50 +0000

The more AI becomes part of ordinary product infrastructure, the less sensible it feels to treat a single model as a permanent foundation.

Models will keep improving. Prices will keep moving. New providers will keep appearing.

A good workflow should expect that.

Why we started building TokenBay

This is also one of the reasons we started building TokenBay.

Not because developers need another long list of AI models, but because testing and changing models should not require rebuilding everything around them.

The idea is simple: one account, one API, and access to different models without having to manage a separate setup for every provider.

We are still learning where this is most useful, but the problem has become clearer to me:

The difficult part is not finding a powerful model.

It is building a workflow that does not become outdated every time a better one arrives.

Three AI APIs Now Look Almost Compatible. They Still Aren’t Interchangeable.

Sarah Pan — Tue, 14 Jul 2026 06:31:55 +0000

OpenAI, Claude, and Gemini now look surprisingly similar at the API level. In some cases, switching providers requires little more than changing an API key, a base URL, and a model name.

But this compatibility disappears almost as soon as the first successful response comes back.

Tool calls behave differently. Reasoning controls use different parameters. Streaming events arrive in different shapes. Rate limits, retries, usage data, and error handling each follow their own rules.

AI APIs are becoming syntactically compatible. They are still far from operationally interchangeable.

Google and Anthropic both provide compatibility with the OpenAI SDK. That means the same basic client can now be pointed at three different providers:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.API_KEY,
  baseURL: process.env.BASE_URL
});

const response = await client.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    {
      role: "user",
      content: "Explain vector databases in one sentence."
    }
  ]
});

console.log(response.choices[0].message.content);

The configuration changes, but most of the request does not

// OpenAI
baseURL: "https://api.openai.com/v1"

// Claude
baseURL: "https://api.anthropic.com/v1/"

// Gemini
baseURL: "https://generativelanguage.googleapis.com/v1beta/openai/"

For a prototype, this is genuinely useful. A developer can test several models without rewriting the entire application. Anthropic even describes its compatibility layer as a way to evaluate Claude using an existing OpenAI-based setup, while Google provides similar support for Gemini.

The problem is that a real AI product needs much more than a text response.

A production system needs to know when to retry, how long to wait, how to record token usage, what to do when a tool call fails, and whether a model stopped because it finished or because it reached a limit.

That is where the APIs begin to separate again.

Area	What changes between providers
Reasoning	Different controls, supported values and response formats
Tool calling	Different schemas, validation behaviour and edge cases
Streaming	Different event types and chunk structures
Errors	Different status details and retry guidance
Rate limits	Different limits for requests, input tokens and output tokens
Usage	Different token categories and accounting rules
New features	Native APIs usually receive capabilities before compatibility layers

Gemini, for example, maps OpenAI-style reasoning settings to its own thinking configuration. Its newer native Interactions API can also return structured steps such as thoughts, tool activity and model output—not simply one message string.

Claude’s native Messages API has its own request model and headers. Its rate limits can separately track requests per minute, input tokens and output tokens. An application that treats every 429 response identically may retry too aggressively, wait unnecessarily, or send traffic to a fallback model that has the same limitation.

Even successful responses can hide differences. Two providers might both return a tool call, but disagree on argument validation or how parallel calls are represented. The code passes the first demo, then breaks when a model returns something slightly unexpected.

This suggests that the useful abstraction is not simply:

generateText(prompt, model)

const result = await generate({
  provider: "anthropic",
  model: "selected-model",
  messages,
  tools,
  timeout: 30_000
});

console.log({
  text: result.text,
  toolCalls: result.toolCalls,
  usage: result.usage,
  finishReason: result.finishReason
});

The wrapper should not pretend every model behaves identically. It should provide a stable application-facing format while still exposing provider-specific capabilities when they matter.

This is also one of the problems we are thinking about at TokenBay: not simply placing several model names behind one endpoint, but reducing the operational friction that appears after the first successful request.

The APIs are converging enough to make experimentation easier. That is real progress. But changing three configuration values is not the same thing as changing providers safely in production.

The first request is becoming standardized. Everything after it is where the integration work still lives.

People Blamed AI for These UI Bugs. I Started Wondering Why.

Sarah Pan — Mon, 13 Jul 2026 06:14:57 +0000

A few days ago, I came across a post showing a strangely broken interface in Meituan, one of China’s most widely used food delivery and local services apps.

In one screenshot, a delete button had expanded into a huge red block, occupying almost half the screen. Other users joined the discussion with their own examples: text cut off by containers, overlapping components, oddly placed buttons, and stores more than ten kilometers away appearing before nearby options.

The screenshots were funny, but the comments were more interesting.

“Please bring the frontend developers back,” one person joked. Another claimed that product managers were now using AI to write frontend code. Someone else wondered whether design and testing had been removed from the process too.

No one in the discussion knew how these interfaces were actually built. Still, many people reached for the same explanation: AI.

That reaction stayed with me. A few years ago, users seeing a broken interface would probably blame a rushed developer or insufficient testing. Now, they imagine a product manager prompting an AI model and shipping whatever comes back.

We do not know whether that happened here. But the fact that it already sounds believable tells us something about how software work is changing.

The Bug Doesn’t Prove Anything, but the Reaction Does

A broken interface is not evidence of AI-generated code. The giant delete button could have come from a layout error. Cropped text might be caused by an unexpected content length or poor adaptation to a particular screen size. Incorrectly ranked stores may be a location or recommendation problem rather than a frontend issue.

Human developers produced all of these bugs long before generative AI arrived.

At the same time, the comments are not based on pure imagination. Meituan has released NoCode, an AI coding tool that allows users to create and deploy websites and software tools through natural-language conversations. According to the company, it can be used for product prototypes, internal tools, data analysis, and web portals.

That does not connect NoCode to these screenshots. It does, however, show why users can easily imagine software being built differently now.

A product manager can describe an interface and receive working code. A designer can turn a visual concept into a prototype without waiting for an engineer. A developer can generate components, tests, documentation, and product copy in the same workflow. Work that once moved between several specialists can increasingly happen in one conversation.

But this also changes where mistakes come from. When producing a page becomes easier, teams can move from idea to release much faster. Review, testing, and judgment do not automatically speed up with it.

An AI model can generate a button in seconds. It does not necessarily ask whether that button still works when the product name takes up three lines, the user has enabled larger fonts, the network response is delayed, or a promotional label is inserted above it at the last minute.

Real products rarely look like the clean examples used in a prompt. They contain missing images, unusually long names, changing prices, conflicting campaigns, old components, multiple screen sizes, and experiments running for different groups of users. The interface often fails not in the state someone designed, but in the state nobody remembered to describe.

This is why “the AI generated working code” and “the product is ready” are two very different claims.

When Execution Gets Cheaper, Judgment Becomes the Bottleneck

The screenshots made me think that AI may be changing the bottleneck of software development.

In the past, a team could understand the problem but still struggle to build the solution. Turning an idea into a functioning product required time, technical knowledge, and coordination between product managers, designers, developers, and testers.

AI reduces some of that friction. It makes technical execution available to more people and allows each person to cover a larger part of the product process.

But easier execution does not guarantee a better product. It may simply allow a team to produce more decisions—and more mistakes—in less time.

Once almost anyone can generate an interface, the difficult part becomes knowing what the interface should do. Someone still needs to understand why a nearby restaurant should appear before one fifteen kilometers away, why a delete action needs a clear warning, and why a button that looks acceptable on one screen may become unusable on another.

These are not only design or engineering questions. They are product questions because they require someone to connect technical behavior with the user’s actual situation.

This is where I think the value of “understanding the product” is growing. Not because product managers will automatically replace developers, but because AI rewards people who can move between an idea and its implementation without losing sight of the user.

A product manager who can generate code but cannot evaluate it may only create bad software faster. A developer who accepts every AI suggestion without understanding the product faces the same problem. The vulnerable skill is not programming or design itself. It is execution without judgment.

The Future Developer May Own More of the Product

This does not convince me that developers are disappearing. It does make me think their work will become broader.

Writing a familiar component from scratch may become less valuable. Understanding how that component fits into a larger system, what can break, and whether the generated solution will remain maintainable is harder to automate.

The same expansion is happening in other roles. Designers can test interactions directly. Product managers can build early prototypes. Developers can contribute to product decisions instead of receiving a finished requirement and translating it into code.

The boundaries between these jobs may become less rigid, creating a more general kind of product builder: someone who can understand a user problem, use AI to create a solution, and recognize when the result is misleading, fragile, or simply wrong.

Deep expertise will still matter. Large platforms cannot be maintained by prompts alone. But smaller teams may be able to build more, and companies may expect each person to take responsibility for a larger part of the final experience.

We may never know whether AI had anything to do with the strange Meituan interface. It could have been an ordinary layout bug, a failed experiment, or a release that moved too quickly.

But the comments revealed a shift that feels more important than the bug itself. Users already believe that software may be generated by AI and lightly reviewed by humans. The question is no longer whether AI can produce an interface.

It is whether someone still knows the product well enough to notice when the interface makes no sense.

The screenshots and discussion came from a public Xiaohongshu post. The comments are users’ speculation and do not confirm Meituan’s internal development process. Meituan describes its AI coding product in the company’s official NoCode announcement.

I Changed Only the Product Description. AI Changed Its Recommendation.

Sarah Pan — Fri, 10 Jul 2026 04:17:59 +0000

As doing the product marketing these days, I’ve been thinking about a slightly uncomfortable question: when an AI recommends a product, how much of that decision is actually about the product, and how much is about the way the product is described?

So, I compared product recommendations across different AI models.One pattern is that products with clearer information seemed easier for the models to recommend. But that experiment involved real brands, so there were too many possible explanations. The models might have seen those brands before. Some brands might have appeared more frequently in their training data. Their websites, reviews, and documentation were also completely different.

So I also tried a smaller and more controlled experiment.

I created a fictional product called ModelDock and wrote three descriptions of it. The underlying product was supposed to be the same every time. The only thing I changed was how much information the AI received—and how that information was organized.

The setup

I gave the AI this scenario:

"A startup needs reliable access to multiple AI models but wants to avoid managing separate provider accounts and APIs."

I then asked whether ModelDock would be suitable for that startup. The model had to choose Yes, No, or Insufficient information, give a confidence score, explain its evidence, list its assumptions, and identify missing information.

I also told it to treat ModelDock as fictional and rely only on the provided description. I wanted to reduce the chance that it would invent outside knowledge about the product.

(This was a small test with one response per version, so I wouldn’t treat it as a scientific result. )

Version A: The vague description

The first description was intentionally generic:

"ModelDock is an AI platform for developers and teams."

The response was:

"Recommendation: Insufficient information
Confidence: 10/10"

At first, seeing a confidence score of 10 surprised me. But the model wasn’t highly confident in the product. It was highly confident that there wasn’t enough information to evaluate it.

It pointed out that “an AI platform for developers and teams” could mean almost anything: a model aggregator, a development environment, a fine-tuning platform, or another kind of AI tool. The description never mentioned multiple models, a unified API, account management, or reliability.

Honestly, that seems fair. If a human showed me that sentence, I wouldn’t know what the product actually did either.

Version B: The specific description

For the second version, I kept the information in normal prose but explained the product more clearly:

ModelDock is a unified AI API platform that allows developers to access
multiple AI models through one account and one API. It is designed for
teams that want to compare models or avoid managing multiple providers.
This time, the response changed:

Recommendation: Yes
Confidence: 10/10

The product itself had not changed. I had only added a clearer explanation of what it did, who it was for, and what problem it solved.

The model could now connect the startup’s needs directly to the description. The startup wanted multiple models without multiple accounts and APIs; ModelDock claimed to provide multiple models through one account and one API.

It still identified missing information, including supported models, pricing, uptime, latency, and SLAs. But those gaps no longer prevented a recommendation because the core product–problem fit was clear.

This was the biggest change in the experiment. Moving from a vague description to a specific one changed the answer from “I can’t evaluate this” to a confident “Yes.”

Version C: The structured description

For the third version, I organized the information as JSON:

{
  "product": "ModelDock",
  "category": "Unified AI API platform",
  "target_users": [
    "AI developers",
    "startups",
    "multi-model application teams"
  ],
  "core_features": [
    "Access to multiple AI models through one API",
    "One account for multiple model providers",
    "Centralized token and usage management",
    "Model comparison without switching platforms"
  ],
  "best_for": [
    "Building multi-model AI applications",
    "Testing and comparing different models",
    "Reducing API and account management overhead"
  ],
  "not_for": [
    "Users looking for a no-code chatbot",
    "Teams that only need one model provider"
  ]
}

The result was still positive:

Recommendation: Yes
Confidence: 9/10

I originally expected the structured version to receive the highest confidence score. Instead, it scored slightly lower than the natural-language version.

What changed was not the direction of the recommendation, but the quality of the reasoning.

The model matched the startup to the listed target users, connected individual requirements to specific features, and clearly separated evidence from assumptions. It also raised more detailed concerns: reliability, model coverage, single-point-of-failure risk, vendor lock-in, pricing, latency, and data privacy.

The JSON did not make the AI automatically “like” the product more. It gave the AI more material to inspect.

What I took away from this

Here is the full comparison:

Description	Recommendation	Confidence	What changed
Vague	Insufficient information	10/10	The model could not verify product–problem fit
Specific prose	Yes	10/10	The core use case became clear
Structured JSON	Yes	9/10	The reasoning became more traceable and critical

My first instinct was to summarize the result as “structured information gets better recommendations.” But the actual result was more nuanced.

Clarity changed the decision. Structure changed the reasoning.

The specific prose version already contained enough information for a recommendation. The structured version did not increase the confidence score, but it helped the model explain exactly why the product fit—and where the remaining risks were.

That distinction matters. A website can be beautifully written and still leave both humans and AI unsure about what the product actually does. At the same time, turning everything into JSON is not a magic trick that guarantees a recommendation.

The most useful information was surprisingly ordinary: what the product is, who it is for, what problem it solves, what it does well, and what it is not designed for.

I’m planning to repeat this experiment with more runs and different models, because three responses are obviously not enough to make a broad claim. But even this small test changed how I think about product documentation. Clearer information does not make a weak product better. It simply makes the product easier to evaluate.

And in an environment where AI systems increasingly compare tools and generate recommendations, being evaluable may become almost as important as being discoverable.

Have you ever tested how an AI describes or recommends your own product? I’d be curious to know whether it understood the product—or confidently filled in the blanks.

Your website was built for humans. But can AI understand it?

Sarah Pan — Thu, 09 Jul 2026 03:34:11 +0000

Most product websites are built for human visitors, and that makes sense. Short headlines, clean sections, feature cards, testimonials, a call-to-action button. A good landing page helps a person quickly figure out what a product does and whether it's worth trying.

While working on TokenBay, though, I kept running into a different problem. People aren't only finding products through Google or Product Hunt anymore. They're asking AI tools things like "what should I use to improve my brand visibility" or "compare these two products for my use case." And I started wondering: if a website reads clearly to a human, does that mean it reads clearly to an AI? My guess is no.

Half the SaaS homepages out there lean on lines like "build smarter with AI," "unlock growth," "supercharge your workflow." They read fine to a human skimming a page. But hand that same page to a system trying to summarize or compare products, and those phrases don't give it much to work with. They don't say what category the product is in, who it's for, what it actually solves, or when it's the wrong choice.

This isn't a brand-new idea, and I don't want to pretend it is. Jeremy Howard's llms.txt proposal already does something adjacent: a markdown file at the root of a site that tells an LLM what content is worth reading, similar in spirit to robots.txt or sitemap.xml. But llms.txt is mostly about which pages matter — it's a reading list. What I kept wanting was something one level more specific: not "here's our content," but "here's who this product is for, and who it's explicitly not for." Closer to what schema.org markup does for search engines, just reframed for how an LLM decides whether to recommend something.

So I started experimenting with a small structured profile alongside TokenBay's site. Here's roughly what that looks like:

{
  "product_name": "TokenBay",
  "one_sentence_pitch": "TokenBay helps brands organize their public information so AI tools can understand and recommend them more accurately.",
  "category": "AI visibility platform",
  "target_users": [
    "startup founders",
    "small businesses",
    "marketing teams",
    "consumer brands"
  ],
  "core_features": [
    "AI-readable brand profile",
    "structured product information",
    "competitor and keyword positioning",
    "recommendation-readiness audit"
  ],
  "best_for": [
    "brands that want to appear in AI-generated recommendations",
    "teams with scattered or unclear public information",
    "products that are difficult to explain in one sentence"
  ],
  "not_best_for": [
    "teams expecting instant guaranteed ranking",
    "companies with no public-facing product",
    "brands that have not defined their positioning yet"
  ]
}

The JSON itself is almost boring, and that's kind of the point. It forces you to actually answer basic questions instead of hiding behind marketing language: what is this, who's it for, what does it help with, when is it useful, and when is it not. Strip the values out and you get a general-purpose template: product_name, one_sentence_pitch, category, target_users, core_features, main_use_cases, best_for, not_best_for, competitors_or_alternatives, proof_points.

The fields that matter most probably aren't the feature list — they're category, target_users, best_for, and not_best_for. AI recommendations are mostly about fit. If someone asks for a tool for a specific situation, the system needs to know not just what a product does, but when it should and shouldn't be suggested.

not_best_for is the one I keep coming back to. Most product pages avoid saying who they're not for — everyone wants to sound like a fit for everyone. But "we're not for teams expecting instant guaranteed ranking" tells you more about a product in one line than most homepages do in five paragraphs.

Where would a file like this even live? I haven't settled on this, but the obvious candidate is something like /.well-known/ai-profile.json, following the same pattern robots.txt and llms.txt already use — a fixed, guessable location so a crawler doesn't have to hunt for it.

I also put together a small checker to catch when a profile is missing the basics:

required_fields = [
    "product_name",
    "one_sentence_pitch",
    "category",
    "target_users",
    "core_features",
    "best_for",
    "not_best_for"
]

def check_profile(profile):
    missing = [f for f in required_fields if not profile.get(f)]
    if missing:
        return {"status": "incomplete", "missing_fields": missing}
    return {
        "status": "ready",
        "message": "This profile has enough structure for an AI system to understand the basics."
    }

Running it against the TokenBay profile above returns:

{
  "status": "ready",
  "message": "This profile has enough structure for an AI system to understand the basics."
}s

I want to be upfront about two problems with this, because I think glossing over them would make the idea sound more solved than it is.

First: no major AI provider has confirmed that it actually reads structured files like this at inference time, and llms.txt — the closer-to-established version of this idea — has so far shown little measurable effect on how AI search crawlers behave in independent testing. So treat everything here as a hypothesis I'm testing, not a proven technique. I'd rather say that plainly than dress it up as a growth hack.

Second: a self-reported JSON file has the same trust problem as meta keywords did twenty years ago. Nothing stops a product from writing a best_for list that's really just marketing copy in a different shape, or showing crawlers a rosier profile than what a human visitor actually sees. Any version of this that matters long-term probably needs some way to check the profile against the real product — reviews, usage data, something outside the vendor's own claims. I don't have a clean answer for that part yet.

With those caveats out of the way, here's why I still think the underlying instinct is right: it's not really about gaming AI answers. It's about reducing ambiguity. A product with scattered, vague, or inconsistent public information is going to get summarized inconsistently by any system trying to parse it — human or AI. Structure doesn't guarantee a recommendation, but it does remove one obvious way to get miscategorized or skipped.

That's the thing I keep coming back to while building TokenBay: AI visibility might not be about producing more content. It might just start with organizing the information you already have so it's easier to classify, compare, and trust.

A good-looking website still matters. A JSON profile won't fix everything on its own, and right now nobody can promise you it moves the needle. But it's a decent starting point, and a good gut-check for whether your product is actually easy to describe in the first place. I'm running a small side-by-side test soon — same set of products, with and without a structured profile, checked against how a couple of different AI tools describe and recommend them — and I'll write up whatever I find, good or bad.

Comparing How ChatGPT and Gemini Recommend Developer Tools

Sarah Pan — Wed, 08 Jul 2026 06:21:52 +0000

Most of us assume that different LLMs should recommend roughly the same tools because they're answering the same question.

After running the same prompts through ChatGPT and Gemini, I found something much more interesting that the models often recommended different products—not because one was "right" and the other was "wrong", but because they evaluated software using different priorities.

To understand these differences, I conducted a small experiment.
Experiment Design

I selected five common categories frequently searched by developers:

AI Workflow
AI Coding
Vector Databases
API Platforms
AI Ecosystem

Each category contained four recommendation prompts (Best, Startup, Open Source, Underrated), resulting in 20 identical prompts being sent to both ChatGPT and Gemini.

Observation 1
One of the first things I noticed was that ChatGPT and Gemini were not as different as I expected. Across 20 identical prompts, several recommendations were exactly the same.
Both models identified n8n as the best AI workflow platform, Cursor as the best AI coding assistant, Pinecone as the best vector database, Qdrant as the preferred choice for RAG, Postman as the leading API testing tool, and Bruno as an underrated API client.

These agreements appeared mostly in categories with relatively mature ecosystems, where a few products have become widely recognized by developers. In these cases, both models seemed to converge on the same industry leaders despite being trained independently.

However, this consensus quickly disappeared in newer areas such as LLM platforms and AI agent frameworks. ChatGPT consistently favored products like OpenAI and LangChain, while Gemini leaned toward Anthropic and CrewAI. Rather than indicating that one model was more accurate than the other, these differences suggested that emerging categories still lack a universally accepted "default choice."
As a result, the models relied more heavily on their own evaluation strategies.This was my first takeaway: LLMs appear to reach consensus more easily in mature markets than in rapidly evolving ones.

Observation 2
After comparing the recommendation results, I shifted my attention to the products themselves. Instead of asking why a model recommended a particular tool, I examined the official websites of the most frequently recommended products.

A clear pattern quickly emerged. Almost every product maintained comprehensive technical documentation, an active GitHub repository (when open source), well-structured API references, practical examples or tutorials, and an active developer community.

Importantly, this observation does not prove that ChatGPT or Gemini directly retrieve information from these sources. LLMs generate responses differently, and their recommendations are not simple website lookups. Nevertheless, these characteristics consistently appeared among the products that both models considered worth recommending.

One possible explanation is that products with rich, structured, and consistently maintained technical content are easier for AI systems to understand and describe accurately. Clear documentation reduces ambiguity. Well-organized examples provide concrete usage scenarios. Active communities generate stable technical discussions that reinforce a product's identity over time.

Rather than revealing where LLMs obtain information, this experiment suggests something different: the products most frequently recommended by AI tend to make themselves easier to understand.

Observation 3
Same Prompt. Different Priorities. The most interesting discovery came from the follow-up comparisons.

Whenever ChatGPT and Gemini recommended different products, I asked each model to explain its reasoning and compare its own choice with the alternative. Surprisingly, the differences rarely came down to factual disagreements. Instead, the models seemed to optimize for different priorities.

Across multiple categories, ChatGPT consistently emphasized broad applicability. Its explanations frequently highlighted developer experience, integration ecosystems, documentation quality, production readiness, and flexibility. For example, when recommending n8n over Langflow, ChatGPT argued that AI is often only one component of a larger business workflow, making general-purpose automation more valuable for most developers.

Gemini, by contrast, placed much greater emphasis on technical specialization. Its recommendations focused on AI-native architectures, distributed systems, scalability, reasoning capabilities, and purpose-built engineering design. When recommending Langflow, Gemini described it as a platform designed specifically for LLM orchestration, RAG pipelines, and agent-based development rather than general workflow automation.

The same pattern appeared repeatedly. ChatGPT preferred Weaviate because of its developer-friendly experience and integrated AI features, whereas Gemini recommended Milvus for its distributed architecture and large-scale performance. ChatGPT favored OpenAI because of its mature ecosystem, documentation, and multimodal tooling, while Gemini highlighted Anthropic's reasoning ability, instruction following, and long-context performance.

In other words, the models were not simply choosing different products. They were using different evaluation frameworks. ChatGPT generally optimized for versatility and practical adoption, while Gemini more often optimized for technical specialization and architectural strength.

Discussion：
Different models may be answering slightly different questions
Another observation became apparent while reading the explanations themselves.

ChatGPT frequently reinterpreted ambiguous prompts before answering. In several comparisons, it explicitly explained how it had interpreted phrases such as "workflow platform" or "vector database," sometimes broadening the definition before making a recommendation. Gemini, on the other hand, tended to answer the prompt more literally and focused on the most technically specialized interpretation.

This difference may explain why two models can recommend different products even when given exactly the same prompt. The disagreement is not necessarily caused by different knowledge or different training data. Instead, it often begins with a different understanding of what the user is actually asking.

For developers, this is an important reminder. AI recommendations are not objective rankings. They are the result of interpretation, reasoning, and prioritization. Asking a slightly different question, or asking a different model, may produce a completely different recommendation.

Conclusion

This experiment started with a simple question: Would ChatGPT and Gemini recommend the same developer tools?

Both models showed strong agreement in mature technology categories, suggesting that some products have achieved broad recognition across the AI ecosystem. However, when recommendations diverged, the differences rarely reflected factual disagreement. Instead, they revealed different ways of evaluating software.

ChatGPT generally prioritized versatility, ecosystem maturity, and practical developer workflows. Gemini more often emphasized architectural design, technical specialization, and engineering performance. Neither approach is inherently better—they simply optimize for different definitions of what makes a product "best."

Perhaps the most useful lesson is not about which model is more accurate. It is that AI recommendations should be understood as informed perspectives rather than definitive rankings. For product teams, this also highlights an emerging challenge. As more developers rely on LLMs to discover software, being easy for AI to understand may become almost as important as being easy for humans to use.