I Changed Only the Product Description. AI Changed Its Recommendation.

As doing the product marketing these days, I’ve been thinking about a slightly uncomfortable question: when an AI recommends a product, how much of that decision is actually about the product, and how much is about the way the product is described?

So, I compared product recommendations across different AI models.One pattern is that products with clearer information seemed easier for the models to recommend. But that experiment involved real brands, so there were too many possible explanations. The models might have seen those brands before. Some brands might have appeared more frequently in their training data. Their websites, reviews, and documentation were also completely different.

So I also tried a smaller and more controlled experiment.

I created a fictional product called ModelDock and wrote three descriptions of it. The underlying product was supposed to be the same every time. The only thing I changed was how much information the AI received—and how that information was organized.

The setup

I gave the AI this scenario:

"A startup needs reliable access to multiple AI models but wants to avoid managing separate provider accounts and APIs."

I then asked whether ModelDock would be suitable for that startup. The model had to choose Yes, No, or Insufficient information, give a confidence score, explain its evidence, list its assumptions, and identify missing information.

I also told it to treat ModelDock as fictional and rely only on the provided description. I wanted to reduce the chance that it would invent outside knowledge about the product.

(This was a small test with one response per version, so I wouldn’t treat it as a scientific result. )

Version A: The vague description

The first description was intentionally generic:

"ModelDock is an AI platform for developers and teams."

The response was:

"Recommendation: Insufficient information
Confidence: 10/10"

At first, seeing a confidence score of 10 surprised me. But the model wasn’t highly confident in the product. It was highly confident that there wasn’t enough information to evaluate it.

It pointed out that “an AI platform for developers and teams” could mean almost anything: a model aggregator, a development environment, a fine-tuning platform, or another kind of AI tool. The description never mentioned multiple models, a unified API, account management, or reliability.

Honestly, that seems fair. If a human showed me that sentence, I wouldn’t know what the product actually did either.

Version B: The specific description

For the second version, I kept the information in normal prose but explained the product more clearly:

ModelDock is a unified AI API platform that allows developers to access
multiple AI models through one account and one API. It is designed for
teams that want to compare models or avoid managing multiple providers.
This time, the response changed:

Recommendation: Yes
Confidence: 10/10

The product itself had not changed. I had only added a clearer explanation of what it did, who it was for, and what problem it solved.

The model could now connect the startup’s needs directly to the description. The startup wanted multiple models without multiple accounts and APIs; ModelDock claimed to provide multiple models through one account and one API.

It still identified missing information, including supported models, pricing, uptime, latency, and SLAs. But those gaps no longer prevented a recommendation because the core product–problem fit was clear.

This was the biggest change in the experiment. Moving from a vague description to a specific one changed the answer from “I can’t evaluate this” to a confident “Yes.”

Version C: The structured description

For the third version, I organized the information as JSON:

{
  "product": "ModelDock",
  "category": "Unified AI API platform",
  "target_users": [
    "AI developers",
    "startups",
    "multi-model application teams"
  ],
  "core_features": [
    "Access to multiple AI models through one API",
    "One account for multiple model providers",
    "Centralized token and usage management",
    "Model comparison without switching platforms"
  ],
  "best_for": [
    "Building multi-model AI applications",
    "Testing and comparing different models",
    "Reducing API and account management overhead"
  ],
  "not_for": [
    "Users looking for a no-code chatbot",
    "Teams that only need one model provider"
  ]
}

The result was still positive:

Recommendation: Yes
Confidence: 9/10

I originally expected the structured version to receive the highest confidence score. Instead, it scored slightly lower than the natural-language version.

What changed was not the direction of the recommendation, but the quality of the reasoning.

The model matched the startup to the listed target users, connected individual requirements to specific features, and clearly separated evidence from assumptions. It also raised more detailed concerns: reliability, model coverage, single-point-of-failure risk, vendor lock-in, pricing, latency, and data privacy.

The JSON did not make the AI automatically “like” the product more. It gave the AI more material to inspect.

What I took away from this

Here is the full comparison:

Description	Recommendation	Confidence	What changed
Vague	Insufficient information	10/10	The model could not verify product–problem fit
Specific prose	Yes	10/10	The core use case became clear
Structured JSON	Yes	9/10	The reasoning became more traceable and critical

My first instinct was to summarize the result as “structured information gets better recommendations.” But the actual result was more nuanced.

Clarity changed the decision. Structure changed the reasoning.

The specific prose version already contained enough information for a recommendation. The structured version did not increase the confidence score, but it helped the model explain exactly why the product fit—and where the remaining risks were.

That distinction matters. A website can be beautifully written and still leave both humans and AI unsure about what the product actually does. At the same time, turning everything into JSON is not a magic trick that guarantees a recommendation.

The most useful information was surprisingly ordinary: what the product is, who it is for, what problem it solves, what it does well, and what it is not designed for.

I’m planning to repeat this experiment with more runs and different models, because three responses are obviously not enough to make a broad claim. But even this small test changed how I think about product documentation. Clearer information does not make a weak product better. It simply makes the product easier to evaluate.

And in an environment where AI systems increasingly compare tools and generate recommendations, being evaluable may become almost as important as being discoverable.

Have you ever tested how an AI describes or recommends your own product? I’d be curious to know whether it understood the product—or confidently filled in the blanks.