<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Aryaman Sinha</title>
    <description>The latest articles on DEV Community by Aryaman Sinha (@aryamansi17).</description>
    <link>https://dev.to/aryamansi17</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1191273%2F2d0132bb-8f8c-4b6b-9a7f-627a9d8abd56.jpeg</url>
      <title>DEV Community: Aryaman Sinha</title>
      <link>https://dev.to/aryamansi17</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aryamansi17"/>
    <language>en</language>
    <item>
      <title>Why asking AI to rate your product doesn't work (and what does)</title>
      <dc:creator>Aryaman Sinha</dc:creator>
      <pubDate>Sat, 21 Nov 2026 18:59:46 +0000</pubDate>
      <link>https://dev.to/aryamansi17/why-asking-ai-to-rate-your-product-doesnt-work-and-what-does-2n8n</link>
      <guid>https://dev.to/aryamansi17/why-asking-ai-to-rate-your-product-doesnt-work-and-what-does-2n8n</guid>
      <description>&lt;p&gt;If you have ever pasted a product idea into ChatGPT or Claude and asked "would people buy this? rate it 1 to 5", you probably got a 4. Try a different idea and you will probably get a 4 again, or a 3.&lt;/p&gt;

&lt;p&gt;That is not because every idea is decent. It is because language models play it safe when you ask them for a number. A 2025 paper measured this properly, and found a fix that is surprisingly simple.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the paper tested
&lt;/h2&gt;

&lt;p&gt;The paper is &lt;em&gt;LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings&lt;/em&gt; (arXiv 2510.08338), by researchers at PyMC Labs and Colgate-Palmolive.&lt;/p&gt;

&lt;p&gt;They had real data from 57 product concept surveys, with 9,300 people. Each person saw a personal care product concept and answered "How likely are you to purchase the product?" on a 1 to 5 scale.&lt;/p&gt;

&lt;p&gt;Then they asked LLMs the same question, with each LLM playing one of the real respondents (same age, income, location and so on), and compared the results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Asking for a number fails
&lt;/h2&gt;

&lt;p&gt;When the model was asked to reply with a number from 1 to 5, the average looked reasonable, but the spread was wrong. Almost every answer was a 3 or a 4. Real people use the whole scale.&lt;/p&gt;

&lt;p&gt;The paper measures this with KS similarity, where 1.0 means the spread of ratings is identical to the real one. Asking for a number directly scored &lt;strong&gt;0.26&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: let it answer in words
&lt;/h2&gt;

&lt;p&gt;The method that worked is called &lt;strong&gt;Semantic Similarity Rating (SSR)&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Ask the AI customer the question, but let it answer in its own words. For example: "I like the idea, but $29 is a lot for me right now."&lt;/li&gt;
&lt;li&gt;Write a few reference sentences for each rating, from "I would definitely not buy this" (1) to "I would definitely buy this" (5).&lt;/li&gt;
&lt;li&gt;Turn the answer and the reference sentences into embeddings, which are lists of numbers that capture meaning.&lt;/li&gt;
&lt;li&gt;Measure how close the answer is to each reference sentence. Subtract the smallest closeness, and normalise so the five numbers add up to 1.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The result is not one guessed number but a probability for every rating. The answer above might come out as 8% for 1, 34% for 2, 30% for 3, 20% for 4 and 8% for 5: it leans towards "no", with some uncertainty, which is how a real person feels about it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fupa59y6oxgfxf8aqdutt.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fupa59y6oxgfxf8aqdutt.png" alt="Synthetic Consumer Research: survey AI customers before you build" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With this method the KS similarity went from 0.26 to &lt;strong&gt;0.88&lt;/strong&gt;, and the ranking of products reached about &lt;strong&gt;90% of human test-retest reliability&lt;/strong&gt;. That last number means it agreed with real results nearly as well as real people agree with themselves when they take the same survey twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two findings that matter in practice
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Personas matter.&lt;/strong&gt; Without demographics, the overall average still looked right, but the model could not tell which products people preferred. Correlation dropped from 92% to 50%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not every split is reliable.&lt;/strong&gt; Age and income effects were reproduced well. Gender and region effects were not.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;I turned the method into an open-source agent skill that works in Claude Code, Codex, Cursor, Google Antigravity and Gemini CLI:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/AryamanSi17/synthetic-consumer-research" rel="noopener noreferrer"&gt;https://github.com/AryamanSi17/synthetic-consumer-research&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You describe a product, and the agent:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;writes personas for your target market,&lt;/li&gt;
&lt;li&gt;asks each one separately (one fresh LLM call per persona, so the answers don't copy each other),&lt;/li&gt;
&lt;li&gt;scores the answers with SSR,&lt;/li&gt;
&lt;li&gt;reports the mean rating, the share who would rate it 4 or 5, results by age and income, and the main objections in the personas' own words.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Install it in any agent with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx skills add AryamanSi17/synthetic-consumer-research
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is also a free demo where you can type a customer answer and see the rating probabilities:&lt;br&gt;
&lt;strong&gt;&lt;a href="https://huggingface.co/spaces/AryamanSi/semantic-similarity-rating-demo" rel="noopener noreferrer"&gt;https://huggingface.co/spaces/AryamanSi/semantic-similarity-rating-demo&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What it is good for, and what it is not
&lt;/h2&gt;

&lt;p&gt;Use it as a fast first filter: ranking two or three concepts, testing a price, or finding the objections you had not thought of. Before you spend money on a real panel, it costs a few cents and a few minutes.&lt;/p&gt;

&lt;p&gt;Do not use it as a replacement for real customers on big decisions. The method was validated on US personal care products. Other categories, countries and languages are less tested, and the model only knows what was in its training data.&lt;/p&gt;

&lt;p&gt;If you have real survey data, you can check how close the synthetic survey gets with the &lt;code&gt;--human&lt;/code&gt; option, which is the best way to calibrate it for your market.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Paper: Maier, Aslak, Fiaschi et al. (2025), arXiv:2510.08338. Official reference implementation: &lt;a href="https://github.com/pymc-labs/semantic-similarity-rating" rel="noopener noreferrer"&gt;pymc-labs/semantic-similarity-rating&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>startup</category>
      <category>opensource</category>
      <category>productivity</category>
    </item>
  </channel>
</rss>
