<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Fathy Shalaby</title>
    <description>The latest articles on DEV Community by Fathy Shalaby (@fathy_shalaby).</description>
    <link>https://dev.to/fathy_shalaby</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247945%2Fc7963a62-67f1-403c-88b7-386c8ef85083.png</url>
      <title>DEV Community: Fathy Shalaby</title>
      <link>https://dev.to/fathy_shalaby</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fathy_shalaby"/>
    <language>en</language>
    <item>
      <title>How to Choose a Product Experiment: 55 Methods by Risk and Evidence</title>
      <dc:creator>Fathy Shalaby</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:49:39 +0000</pubDate>
      <link>https://dev.to/fathy_shalaby/how-to-choose-a-product-experiment-55-methods-by-risk-and-evidence-4n76</link>
      <guid>https://dev.to/fathy_shalaby/how-to-choose-a-product-experiment-55-methods-by-risk-and-evidence-4n76</guid>
      <description>&lt;p&gt;Most ideas lose when tested. Name the riskiest belief, decide how strong the evidence must be, then pick the cheapest of 55 product experiments that gets you there.&lt;/p&gt;

&lt;p&gt;Product experiments are small, planned tests of one belief your plan depends on, with a pass signal written before you start. To choose one, name the belief, name its risk (desirability, usability, feasibility or viability), decide how strong the evidence must be, then pick the cheapest method that reaches that strength. This guide sorts 55 methods that way, with time and effort for each.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Most ideas lose when tested. Only one third of the ideas tested at Microsoft improved the metric they were designed to improve, and about 10% of Google's controlled experiments led to business changes (&lt;a href="https://exp-platform.com/Documents/2023-03-11EncyclopeiaMLDSABTestingFinal.pdf" rel="noopener noreferrer"&gt;Kohavi and Longbotham, 2023&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Test the riskiest belief, not the whole idea. Marty Cagan's four risks are value, usability, feasibility and business viability (&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;SVPG, 2017&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Evidence comes in strengths. What people say is weakest; money or a signature is strongest. Strategyzer's library for &lt;em&gt;Testing Business Ideas&lt;/em&gt; rates 44 experiments from 1 to 5 on cost, setup time, run time and evidence strength.&lt;/li&gt;
&lt;li&gt;Cheap does not mean weak. In that library, a concierge test scores 1 of 5 for cost and 5 of 5 for evidence.&lt;/li&gt;
&lt;li&gt;Write the pass signal before you start. Moving it afterwards is how most teams fool themselves.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why do product ideas need experiments at all?
&lt;/h2&gt;

&lt;p&gt;Most product ideas do not fail because the team built badly. They fail on a guess nobody checked.&lt;/p&gt;

&lt;p&gt;The record from teams that measure is humbling. Ron Kohavi and Roger Longbotham collect it in one place: a third of ideas tested at Microsoft moved their target metric, Jim Manzi wrote that about 10% of Google's experiments led to business changes, and Netflix is reported to consider 90% of what it tries to be wrong (&lt;a href="https://exp-platform.com/Documents/2023-03-11EncyclopeiaMLDSABTestingFinal.pdf" rel="noopener noreferrer"&gt;Kohavi and Longbotham, 2023&lt;/a&gt;). These companies employ excellent product people. Their first guesses still lose most of the time.&lt;/p&gt;

&lt;p&gt;Skipping the check costs more later. CB Insights studied 431 venture-backed companies that shut down since 2023 and found reasons for 385 of them: 70% ran out of capital, 43% had poor product-market fit, 29% cited bad timing and 19% had unsustainable unit economics (&lt;a href="https://www.cbinsights.com/research/report/startup-failure-reasons-top/" rel="noopener noreferrer"&gt;CB Insights, 2026&lt;/a&gt;). Companies could give more than one reason. Poor fit and broken unit economics are exactly the doubts a cheap experiment can probe months before the money runs out.&lt;/p&gt;

&lt;p&gt;A product experiment is how you check a guess before you spend real money on it. You do not need a research team, and you do not need all 55 methods in this guide. You need the one test that could change your mind about the belief your plan depends on most.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a product experiment?
&lt;/h2&gt;

&lt;p&gt;A product experiment is a small, planned test of one belief, with a result you define before you start. Good product experiments have four parts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The belief&lt;/strong&gt;, written as a sentence that could be false. "Freelancers lose work because follow-ups slip," not "freelancers need this."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The method&lt;/strong&gt;: the smallest action that produces evidence about that belief.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The measure&lt;/strong&gt;: one number, plus one thing that must not get worse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The pass signal&lt;/strong&gt;: the result that would make you say yes, set in advance.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Teresa Torres puts the core move plainly: stop testing whole ideas and test the assumptions they depend on (&lt;a href="https://www.producttalk.org/2023/10/assumption-testing/" rel="noopener noreferrer"&gt;Product Talk, 2023&lt;/a&gt;). An idea like "a reminder app for vets" holds a dozen beliefs. Most are safe. One or two would kill it, and those are the ones worth an experiment. If you are starting from a feature request, the &lt;a href="https://www.specky.space/blog/product-problem-statement-template" rel="noopener noreferrer"&gt;product problem statement template&lt;/a&gt; turns it into a belief you can test.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are the four product risks an experiment can test?
&lt;/h2&gt;

&lt;p&gt;Every belief carries a kind of risk, and the kind decides which methods fit. Marty Cagan of Silicon Valley Product Group names four (&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;SVPG, 2017&lt;/a&gt;):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;th&gt;Cagan's question&lt;/th&gt;
&lt;th&gt;Strategyzer's word&lt;/th&gt;
&lt;th&gt;Usual owner&lt;/th&gt;
&lt;th&gt;Cheap first test&lt;/th&gt;
&lt;th&gt;Strong test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Value&lt;/td&gt;
&lt;td&gt;Will customers buy it, or choose to use it?&lt;/td&gt;
&lt;td&gt;Desirability&lt;/td&gt;
&lt;td&gt;Product manager&lt;/td&gt;
&lt;td&gt;Customer interview, fake door&lt;/td&gt;
&lt;td&gt;Concierge, letter of intent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usability&lt;/td&gt;
&lt;td&gt;Can users figure out how to use it?&lt;/td&gt;
&lt;td&gt;Tagged as desirability&lt;/td&gt;
&lt;td&gt;Product designer&lt;/td&gt;
&lt;td&gt;Paper prototype&lt;/td&gt;
&lt;td&gt;Usability test on a real build&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Feasibility&lt;/td&gt;
&lt;td&gt;Can engineers build it with the time, skills and technology we have?&lt;/td&gt;
&lt;td&gt;Feasibility&lt;/td&gt;
&lt;td&gt;Lead engineer&lt;/td&gt;
&lt;td&gt;Expert interview, technical spike&lt;/td&gt;
&lt;td&gt;Wizard of Oz, mash-up&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Business viability&lt;/td&gt;
&lt;td&gt;Does it work for the rest of the business?&lt;/td&gt;
&lt;td&gt;Viability&lt;/td&gt;
&lt;td&gt;Product manager&lt;/td&gt;
&lt;td&gt;Pricing survey, market research&lt;/td&gt;
&lt;td&gt;Funnel test, letter of intent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Bland and Osterwalder's &lt;em&gt;Testing Business Ideas&lt;/em&gt; (Wiley, November 2019, 368 pages) groups risk into three questions: do customers want it, can we build it, and will it earn sustainable revenue (&lt;a href="https://www.wiley.com/en-us/Testing+Business+Ideas%3A+A+Field+Guide+for+Rapid+Experimentation-p-9781119551447" rel="noopener noreferrer"&gt;Wiley&lt;/a&gt;; &lt;a href="https://www.strategyzer.com/library/testing-business-ideas-book-summary" rel="noopener noreferrer"&gt;Strategyzer, 2025&lt;/a&gt;). Its experiment library tags usability-style tests, such as card sorting and clickable prototypes, as desirability. Cagan keeps usability separate, and so do we, because "they want it" and "they can use it" fail in different ways and need different tests.&lt;/p&gt;

&lt;p&gt;Torres adds a fifth category, ethical: should we build this at all? Cagan's advice on order is blunt: tackle the big risks early, especially value and business viability. For a deeper look at what counts as proof for each risk, read &lt;a href="https://www.specky.space/blog/what-counts-as-evidence-four-product-risks" rel="noopener noreferrer"&gt;what counts as evidence for the four product risks&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How strong is the evidence from each kind of experiment?
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;The higher the step, the more you can trust the result.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Not all evidence is equal. A kind word and a signed letter are both "positive feedback," but only one should move your roadmap. Alex Osterwalder makes the same point: what customers say and what they do are two different things, so interviews and surveys give valuable but weak evidence, while a simulated sale gives strong evidence (&lt;a href="https://www.strategyzer.com/library/how-strong-is-your-innovation-evidence" rel="noopener noreferrer"&gt;Strategyzer, 2017&lt;/a&gt;).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strength&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;th&gt;Treat it as&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;Someone told you something&lt;/td&gt;
&lt;td&gt;An interview, a survey&lt;/td&gt;
&lt;td&gt;A reason to run a stronger test&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;Real behaviour, but cheap or indirect&lt;/td&gt;
&lt;td&gt;An ad click, a search trend&lt;/td&gt;
&lt;td&gt;A clue about where to look&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;Someone took a real action&lt;/td&gt;
&lt;td&gt;Finishing a task, leaving an email&lt;/td&gt;
&lt;td&gt;Good evidence for that specific action&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;Someone gave up something of value&lt;/td&gt;
&lt;td&gt;A payment, a signed letter&lt;/td&gt;
&lt;td&gt;The strongest evidence you can get before launch&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Specky uses the same ladder. A belief can only be marked Supported by what people do, or by money or a signature. Anything weaker stays an Early signal, however good it sounded.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does a stronger test always cost more?
&lt;/h3&gt;

&lt;p&gt;No. Strategyzer's experiment library poster rates every method from 1 to 5, and cost and evidence strength move independently (&lt;a href="https://f.hubspotusercontent40.net/hubfs/4952096/Testing%20Business%20Ideas_ExperimentLibrary-Poster.pdf" rel="noopener noreferrer"&gt;Strategyzer experiment library&lt;/a&gt;). A sample:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Cost (1-5)&lt;/th&gt;
&lt;th&gt;Evidence strength (1-5)&lt;/th&gt;
&lt;th&gt;What the pairing tells you&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/customer-interview-template" rel="noopener noreferrer"&gt;Customer interview&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Cheap and fast, but only words&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/discovery-survey-template" rel="noopener noreferrer"&gt;Discovery survey&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Counts a problem; proves no demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/paper-prototype-test-template" rel="noopener noreferrer"&gt;Paper prototype&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;The cheapest rung for a flow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/search-trend-analysis-template" rel="noopener noreferrer"&gt;Search trend analysis&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Strangers' behaviour for almost nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/404-test-template" rel="noopener noreferrer"&gt;404 test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Also rated 1 for setup and run time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/concierge-mvp-template" rel="noopener noreferrer"&gt;Concierge&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Low cost, top evidence, but slow to run&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/wizard-of-oz-test-template" rel="noopener noreferrer"&gt;Wizard of Oz&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Real service, faked automation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/technical-spike-template" rel="noopener noreferrer"&gt;Extreme programming spike&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Strong evidence on feasibility only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/presale-interest-page-template" rel="noopener noreferrer"&gt;Presale&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Money on the table&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/single-feature-mvp-template" rel="noopener noreferrer"&gt;Single-feature MVP&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Strong, but you are now building&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The lesson: the expensive part of a strong test is usually your time, not your budget. A concierge test costs little money but weeks of manual work.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you choose the right product experiment?
&lt;/h2&gt;

&lt;p&gt;Answer five questions, in order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;What is the belief?&lt;/strong&gt; Write it as one sentence that could turn out to be false.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What kind of risk is it?&lt;/strong&gt; Do they want it (desirability)? Can they use it (usability)? Can we do it (feasibility)? Will it make money (viability)?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What would it cost to be wrong?&lt;/strong&gt; A guess that decides the next six months deserves stronger evidence than one that decides a button colour. Barry O'Reilly makes the same point about evidence thresholds: a rocket launch needs a higher bar than a sign-up flow (&lt;a href="https://www.thoughtworks.com/insights/articles/how-implement-hypothesis-driven-development" rel="noopener noreferrer"&gt;Thoughtworks, 2014&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What is the cheapest test that gives enough evidence?&lt;/strong&gt; Start low on the ladder when you know little. Climb as the stakes rise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What would count as a pass?&lt;/strong&gt; Write it down before you begin. O'Reilly's reason is the right one: signals stated in advance reduce biased readings of the result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A rule of thumb we use: if being wrong costs a week, what people say is enough. If it costs a quarter, get what people do. If it costs a hire, a year or your runway, get money or a signature.&lt;/p&gt;

&lt;p&gt;When you have several beliefs and cannot tell which is riskiest, map them first. Our guide to &lt;a href="https://www.specky.space/blog/de-risking-bets-assumption-mapping" rel="noopener noreferrer"&gt;assumption mapping&lt;/a&gt; and the free &lt;a href="https://www.specky.space/tools/assumption-map" rel="noopener noreferrer"&gt;assumption map tool&lt;/a&gt; sort beliefs by impact and evidence, so the test goes where it matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  Product experiment decision table: which test for which risk and evidence level?
&lt;/h2&gt;

&lt;p&gt;Find the row for your risk. Move right until you reach the evidence strength that being wrong deserves. Pick the cheapest method in that cell.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;th&gt;What people say&lt;/th&gt;
&lt;th&gt;Early signal&lt;/th&gt;
&lt;th&gt;What people do&lt;/th&gt;
&lt;th&gt;Money or a signature&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Desirability: is the problem real?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/customer-interview-template" rel="noopener noreferrer"&gt;Customer interview&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/discovery-survey-template" rel="noopener noreferrer"&gt;discovery survey&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/forum-research-template" rel="noopener noreferrer"&gt;forum research&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/search-trend-analysis-template" rel="noopener noreferrer"&gt;Search trends&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/support-ticket-analysis-template" rel="noopener noreferrer"&gt;support tickets&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/product-analytics-check-template" rel="noopener noreferrer"&gt;analytics check&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/diary-study-template" rel="noopener noreferrer"&gt;Diary study&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Move on to the offer row&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Desirability: do they want this offer?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/product-box-exercise-template" rel="noopener noreferrer"&gt;Product box&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/brochure-test-template" rel="noopener noreferrer"&gt;brochure&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/validation-survey-template" rel="noopener noreferrer"&gt;validation survey&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/online-ad-test-template" rel="noopener noreferrer"&gt;Online ad&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/404-test-template" rel="noopener noreferrer"&gt;404 test&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/email-campaign-test-template" rel="noopener noreferrer"&gt;email campaign&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/fake-door-test-template" rel="noopener noreferrer"&gt;Fake door&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/landing-page-test-template" rel="noopener noreferrer"&gt;landing page&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/feature-stub-test-template" rel="noopener noreferrer"&gt;feature stub&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/concierge-mvp-template" rel="noopener noreferrer"&gt;Concierge&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/crowdfunding-test-template" rel="noopener noreferrer"&gt;crowdfunding&lt;/a&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Usability: can they use it?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Rarely useful: people misjudge their own ease of use&lt;/td&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/web-traffic-analysis-template" rel="noopener noreferrer"&gt;Web traffic analysis&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/paper-prototype-test-template" rel="noopener noreferrer"&gt;Paper prototype&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/clickable-prototype-test-template" rel="noopener noreferrer"&gt;clickable prototype&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/usability-test-template" rel="noopener noreferrer"&gt;usability test&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/card-sorting-template" rel="noopener noreferrer"&gt;card sorting&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;Not the right question; usability is shown by doing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Feasibility: can we deliver it?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/partner-supplier-interview-template" rel="noopener noreferrer"&gt;Partner and supplier interviews&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/expert-stakeholder-interview-template" rel="noopener noreferrer"&gt;expert interviews&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/technical-spike-template" rel="noopener noreferrer"&gt;Technical spike&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/pretend-to-own-test-template" rel="noopener noreferrer"&gt;Pretend to own&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/mash-up-mvp-template" rel="noopener noreferrer"&gt;mash-up&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/wizard-of-oz-test-template" rel="noopener noreferrer"&gt;Wizard of Oz&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/concierge-mvp-template" rel="noopener noreferrer"&gt;Concierge&lt;/a&gt; (paid delivery by hand)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Viability: will it make money?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/pricing-survey-template" rel="noopener noreferrer"&gt;Pricing survey&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/sales-team-feedback-template" rel="noopener noreferrer"&gt;sales team feedback&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/investor-feedback-test-template" rel="noopener noreferrer"&gt;investor feedback&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/online-market-research-template" rel="noopener noreferrer"&gt;Online market research&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/presale-interest-page-template" rel="noopener noreferrer"&gt;presale interest&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/funnel-test-template" rel="noopener noreferrer"&gt;Funnel test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://www.specky.space/blog/letter-of-intent-template" rel="noopener noreferrer"&gt;Letter of intent&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/referral-program-test-template" rel="noopener noreferrer"&gt;referral program&lt;/a&gt;, &lt;a href="https://www.specky.space/blog/pop-up-store-test-template" rel="noopener noreferrer"&gt;pop-up store&lt;/a&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Typical time&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1 day to 2 weeks&lt;/td&gt;
&lt;td&gt;1 day to 1 week&lt;/td&gt;
&lt;td&gt;2 days to 3 weeks&lt;/td&gt;
&lt;td&gt;2 to 4 weeks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Typical effort&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Low to high&lt;/td&gt;
&lt;td&gt;Medium to high&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Times and effort come from the method tables below and assume a small team. Read across a row to see how to climb. Read down a column to compare risks at the same strength.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which product experiments answer which question? The full library of 55
&lt;/h2&gt;

&lt;p&gt;Every method below links to its own guide, with a copyable template, a pass signal set in advance and a worked example.&lt;/p&gt;

&lt;h3&gt;
  
  
  Desirability: is the problem real, and who has it?
&lt;/h3&gt;

&lt;p&gt;Start here when you are still unsure the pain exists. Most of these cost a few days and no code.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/customer-interview-template" rel="noopener noreferrer"&gt;Customer interview&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have a hunch and need the real story&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/day-in-the-life-study-template" rel="noopener noreferrer"&gt;A day in the life&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;People cannot describe work they do on autopilot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/discovery-survey-template" rel="noopener noreferrer"&gt;Discovery survey&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want to know how common a problem is in one group&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/short-customer-survey-template" rel="noopener noreferrer"&gt;Survey&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You know the group and just need to settle one choice&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/forum-research-template" rel="noopener noreferrer"&gt;Discussion forums&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;2 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want people's exact words without taking their time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/storyboard-test-template" rel="noopener noreferrer"&gt;Storyboard&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;2 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;The situation is hard to explain in words&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/speed-boat-exercise-template" rel="noopener noreferrer"&gt;Speed boat&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have a product and want polite people to criticise it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/search-trend-analysis-template" rel="noopener noreferrer"&gt;Search trend analysis&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want a one-day read on whether people look for this&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/web-traffic-analysis-template" rel="noopener noreferrer"&gt;Web traffic analysis&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have steady visits and need to pick the step to test&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/support-ticket-analysis-template" rel="noopener noreferrer"&gt;Customer support analysis&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;2 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have a live product and a full inbox&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/product-analytics-check-template" rel="noopener noreferrer"&gt;Analytics&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;The behaviour is already logged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/diary-study-template" rel="noopener noreferrer"&gt;Diary study&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;The problem comes and goes, and people misremember it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Once you have run a handful of interviews, &lt;a href="https://www.specky.space/blog/analyze-customer-interviews-synthesis-template" rel="noopener noreferrer"&gt;analyse them with a synthesis template&lt;/a&gt; before you count anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  Desirability: will people want this offer?
&lt;/h3&gt;

&lt;p&gt;Use these once the problem looks real. Now you are finding out if your particular answer gets a reaction.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/product-box-exercise-template" rel="noopener noreferrer"&gt;Product box&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Your team cannot agree on the main benefit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/brochure-test-template" rel="noopener noreferrer"&gt;Brochure&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You need to check your offer is clear&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/validation-survey-template" rel="noopener noreferrer"&gt;Validation survey&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;People have seen the offer and you want to hear what would stop them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/online-ad-test-template" rel="noopener noreferrer"&gt;Online ad&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want a reaction from strangers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/link-tracking-test-template" rel="noopener noreferrer"&gt;Link tracking&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You are comparing messages or channels&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/404-test-template" rel="noopener noreferrer"&gt;404 test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want the cheapest possible check on curiosity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/email-campaign-test-template" rel="noopener noreferrer"&gt;Email campaign&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have a list that chose to hear from you&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/social-media-demand-test-template" rel="noopener noreferrer"&gt;Social media campaign&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Your customers gather somewhere you are welcome&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/data-sheet-test-template" rel="noopener noreferrer"&gt;Data sheet&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Your buyers evaluate on numbers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/explainer-video-test-template" rel="noopener noreferrer"&gt;Explainer video&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;It is easier to show than to describe&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/fake-door-test-template" rel="noopener noreferrer"&gt;Fake door&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You can state the offer in one sentence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/feature-stub-test-template" rel="noopener noreferrer"&gt;Feature stub&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You want to catch current users in the moment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/landing-page-test-template" rel="noopener noreferrer"&gt;Simple landing page&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You can send strangers to one clear page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/boomerang-competitor-test-template" rel="noopener noreferrer"&gt;Boomerang (competitor testing)&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Good alternatives already exist&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/buy-a-feature-template" rel="noopener noreferrer"&gt;Buy a feature&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Customers say everything matters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/split-test-plan-template" rel="noopener noreferrer"&gt;Split test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You have the traffic to prove a change caused a result&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Usability: can people use it, and will they come back?
&lt;/h3&gt;

&lt;p&gt;These put something close to the real thing in front of people. Ease of use and repeat use are separate questions, so read each row carefully.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/paper-prototype-test-template" rel="noopener noreferrer"&gt;Paper prototype&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You are still working out the steps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/clickable-prototype-test-template" rel="noopener noreferrer"&gt;Clickable prototype&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;The flow makes sense and you want to test it for real&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/usability-test-template" rel="noopener noreferrer"&gt;Usability test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Something is built and people are getting stuck&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/card-sorting-template" rel="noopener noreferrer"&gt;Card sorting&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You are deciding how to name or organise things&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/3d-printed-prototype-test-template" rel="noopener noreferrer"&gt;3D print&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;td&gt;It is a thing people hold, wear, or mount&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/life-sized-prototype-template" rel="noopener noreferrer"&gt;Life-sized prototype&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;3 weeks&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;td&gt;Size and space change the experience&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/single-feature-mvp-template" rel="noopener noreferrer"&gt;Single-feature MVP&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;3 weeks&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;td&gt;Lighter tests passed and you need to see repeat use&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/concierge-mvp-template" rel="noopener noreferrer"&gt;Concierge&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;3 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You can deliver the outcome by hand and ask for money&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Feasibility: can we build or deliver it?
&lt;/h3&gt;

&lt;p&gt;Some ideas are loved and still impossible. Use these when the doubt is on your side of the table.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/partner-supplier-interview-template" rel="noopener noreferrer"&gt;Partner and supplier interviews&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Your plan depends on someone outside your company&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/expert-stakeholder-interview-template" rel="noopener noreferrer"&gt;Expert stakeholder interviews&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Someone could block or fund this&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/technical-spike-template" rel="noopener noreferrer"&gt;Extreme programming spike&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;One technical unknown could sink the plan&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/pretend-to-own-test-template" rel="noopener noreferrer"&gt;Pretend to own&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You want to learn what one real delivery takes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/mash-up-mvp-template" rel="noopener noreferrer"&gt;Mash-up&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;Existing tools could do the job if connected&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/wizard-of-oz-test-template" rel="noopener noreferrer"&gt;Wizard of Oz&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;The hard part is automating it, and the experience needs testing first&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Viability: will it make money?
&lt;/h3&gt;

&lt;p&gt;Interest is cheap. These tests get closer to the moment someone gives up something of value.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/sales-team-feedback-template" rel="noopener noreferrer"&gt;Sales force feedback&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;Deals stall and nobody agrees why&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/investor-feedback-test-template" rel="noopener noreferrer"&gt;Investor interviews and pitches&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You want your advantage stress-tested&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/pricing-survey-template" rel="noopener noreferrer"&gt;Pricing survey&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You have no idea what to charge&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/mock-purchase-intent-test-template" rel="noopener noreferrer"&gt;Mock purchase intent&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want to compare two prices&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/online-market-research-template" rel="noopener noreferrer"&gt;Online market research&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;3 days&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You need to know if the sums could ever work&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/presale-interest-page-template" rel="noopener noreferrer"&gt;Presale interest&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;You want to see if interest survives a price&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/funnel-test-template" rel="noopener noreferrer"&gt;Funnel test&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people do&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You need to know what a customer costs to reach&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/referral-program-test-template" rel="noopener noreferrer"&gt;Referral program&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You want to know if users will vouch for you&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/letter-of-intent-template" rel="noopener noreferrer"&gt;Letter of intent&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;4 weeks&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;td&gt;You sell to businesses and can reach who signs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/crowdfunding-test-template" rel="noopener noreferrer"&gt;Crowdfunding&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;4 weeks&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;td&gt;It is a consumer product and you can deliver&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/pop-up-store-test-template" rel="noopener noreferrer"&gt;Pop-up store&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Money or a signature&lt;/td&gt;
&lt;td&gt;2 weeks&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;td&gt;People need to see or touch it before buying&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Not sure where to start?
&lt;/h3&gt;

&lt;p&gt;If the idea is still a blur, or nothing here fits, begin with one of these.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Evidence strength&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Reach for it when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/lean-canvas-template" rel="noopener noreferrer"&gt;Lean Canvas&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;What people say&lt;/td&gt;
&lt;td&gt;1 day&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;The idea is unstructured, or your team disagrees&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.specky.space/blog/custom-product-experiment-template" rel="noopener noreferrer"&gt;Other&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Early signal&lt;/td&gt;
&lt;td&gt;1 week&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;td&gt;None of the 54 methods above fits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What does climbing the evidence ladder look like in practice?
&lt;/h2&gt;

&lt;p&gt;This sequence is made up to show the idea. Imagine you think independent gym owners lose members because they do not notice who has stopped coming.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Five &lt;a href="https://www.specky.space/blog/customer-interview-template" rel="noopener noreferrer"&gt;customer interviews&lt;/a&gt; confirm owners have tried to fix this. That is what people say, and it costs a week.&lt;/li&gt;
&lt;li&gt;A &lt;a href="https://www.specky.space/blog/fake-door-test-template" rel="noopener noreferrer"&gt;fake door&lt;/a&gt; shows owners will leave an email for an offer. Say 40 of 400 visitors from a gym-owners' newsletter do, which is 10%. Now you have behaviour.&lt;/li&gt;
&lt;li&gt;A &lt;a href="https://www.specky.space/blog/concierge-mvp-template" rel="noopener noreferrer"&gt;concierge test&lt;/a&gt; sends weekly lists by hand to five gyms, and three pay for a second month. Now you have commitment.&lt;/li&gt;
&lt;li&gt;Only now do you build, starting with a &lt;a href="https://www.specky.space/blog/single-feature-mvp-template" rel="noopener noreferrer"&gt;single-feature MVP&lt;/a&gt;. Brief your coding agent with the &lt;a href="https://www.specky.space/blog/ai-coding-agent-prd-template" rel="noopener noreferrer"&gt;AI coding agent PRD template&lt;/a&gt;, and record why you went ahead in a &lt;a href="https://www.specky.space/blog/product-decision-log-template-small-teams" rel="noopener noreferrer"&gt;product decision log&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Using the times in the tables, steps 1 to 3 take about five weeks: one for interviews, one for the fake door, three for the concierge. Each step cost more than the last, and each one was earned by the one before it. If any had failed, you would have stopped early and saved everything after it.&lt;/p&gt;

&lt;p&gt;Notice what the sequence skipped. Nobody ran a survey to ask whether gym owners "would use" a churn tool, and nobody built a dashboard to see if it got traffic. Each test answered the next riskiest question and nothing more. For where this sits in a full discovery cycle, see the &lt;a href="https://www.specky.space/guides/product-discovery-process" rel="noopener noreferrer"&gt;product discovery process guide&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many product experiments and participants do you need?
&lt;/h2&gt;

&lt;p&gt;There is no fixed number of experiments. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. Strategyzer's summary of the book suggests a practical cadence: run three to five small experiments on your highest-priority assumptions in the first month, and start with the smallest test that could confirm or kill each one (&lt;a href="https://www.strategyzer.com/library/testing-business-ideas-book-summary" rel="noopener noreferrer"&gt;Strategyzer, 2025&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Pair weak and strong. One "say" test tells you what to look for; one "do" test tells you whether it holds. Two "say" tests that agree are still two opinions.&lt;/p&gt;

&lt;p&gt;Participant numbers depend on the method:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Usability tests:&lt;/strong&gt; five people per round. Jakob Nielsen's model found five users uncover about 85% of usability problems and 15 find nearly all, so he advises spending the budget on several small rounds (&lt;a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" rel="noopener noreferrer"&gt;Nielsen Norman Group, 2000&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Surveys:&lt;/strong&gt; aim for about 50 valid answers from one named segment before you trust a percentage. Our &lt;a href="https://www.specky.space/blog/discovery-survey-template" rel="noopener noreferrer"&gt;discovery survey guide&lt;/a&gt; explains why.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split tests:&lt;/strong&gt; these need real traffic. Kohavi and Longbotham note that a 50/50 split gives the most statistical power once a short ramp-up has ruled out obvious errors. If you lack the visitors, use a &lt;a href="https://www.specky.space/blog/fake-door-test-template" rel="noopener noreferrer"&gt;fake door&lt;/a&gt; or &lt;a href="https://www.specky.space/blog/landing-page-test-template" rel="noopener noreferrer"&gt;landing page&lt;/a&gt; instead. Our &lt;a href="https://www.specky.space/blog/pm-guide-ab-testing-hypothesis-to-decision" rel="noopener noreferrer"&gt;guide to A/B testing&lt;/a&gt; covers the maths.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Letters of intent and presales:&lt;/strong&gt; a handful can be enough when each one is a real buyer with a budget.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What are the most common mistakes with product experiments?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Testing the idea instead of the riskiest belief.&lt;/strong&gt; "Will this work?" cannot fail cleanly. "Will owners pay €50 a month for a weekly list?" can.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Picking the method you already know.&lt;/strong&gt; Teams that like surveys survey everything. Let the risk choose the method.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating what people say as what they do.&lt;/strong&gt; Twenty enthusiastic interviews are still the bottom rung.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writing the pass signal afterwards.&lt;/strong&gt; A result you judge after seeing it is a story, not a test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split-testing without traffic.&lt;/strong&gt; A few hundred visits rarely prove that a change caused a result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Skipping the unglamorous risks.&lt;/strong&gt; Founders love desirability tests. Cagan's advice is to tackle value and business viability early, and feasibility can sink a loved idea just as fast.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Climbing too fast.&lt;/strong&gt; A crowdfunding campaign before a single interview spends weeks to learn what a conversation could have told you.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  When should you not run a product experiment?
&lt;/h3&gt;

&lt;p&gt;Skip the experiment when the decision is cheap and easy to reverse; shipping and watching is the test. Our piece on &lt;a href="https://www.specky.space/blog/reversible-irreversible-product-decisions-framework" rel="noopener noreferrer"&gt;reversible and irreversible product decisions&lt;/a&gt; helps you tell the two apart. Skip it, too, when you will build regardless of the result. A test you would ignore is theatre. And when none of the 55 methods fits your setting, such as a year-long public-sector sale, &lt;a href="https://www.specky.space/blog/custom-product-experiment-template" rel="noopener noreferrer"&gt;design a custom experiment&lt;/a&gt; rather than forcing one.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is it ethical to test something that does not exist yet?
&lt;/h3&gt;

&lt;p&gt;Several methods, including the fake door, the 404 test, presales and the Wizard of Oz, show people something that is not fully real. Keep them honest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tell people on the next screen that the feature is not ready, and thank them.&lt;/li&gt;
&lt;li&gt;Never take money you cannot refund or deliver on. Say when delivery would happen.&lt;/li&gt;
&lt;li&gt;If a person secretly does the work in a Wizard of Oz test, tell participants afterwards when it would matter to them.&lt;/li&gt;
&lt;li&gt;Ask before recording anyone, and only keep the email addresses people chose to give you, for the purpose you stated.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How to run this in Specky
&lt;/h2&gt;

&lt;p&gt;Fill in a &lt;a href="https://www.specky.space/blog/lean-canvas-template" rel="noopener noreferrer"&gt;Lean Canvas&lt;/a&gt; on the Assumptions page, and Specky drafts the assumptions behind it. Each one appears on a board by impact against evidence, with the riskiest numbered. Click a note, pick a test, and the plan fills in with steps, a pass signal, and a time to run.&lt;/p&gt;

&lt;p&gt;Who does the work depends on the method. Some tests are AI interviews in Vouch, at &lt;a href="https://vouch.specky.space" rel="noopener noreferrer"&gt;vouch.specky.space&lt;/a&gt;. For some, Specky builds a page and collects responses. For some, it reads your connected analytics. For the rest, you run the test and Specky keeps the plan and the result. Each guide above says which.&lt;/p&gt;

&lt;p&gt;Recording a result moves the note and updates any bet resting on it. One contradicted assumption marks the whole bet as contradicted. A bet only counts as supported when every assumption under it is.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is a product experiment?
&lt;/h3&gt;

&lt;p&gt;A product experiment is a small, planned test of one belief about a product, with a pass signal defined before you start. It lets a team learn whether something is true before spending real money on it. A good one names the belief, the method, the number it will measure and the result that would count as a yes.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I choose the right product experiment?
&lt;/h3&gt;

&lt;p&gt;Write the belief you are least sure about, name its risk (desirability, usability, feasibility or viability), and decide how strong the evidence must be given what being wrong would cost. Then pick the cheapest method that reaches that strength. Early on, conversations are usually enough. As stakes rise, ask people to act or pay.&lt;/p&gt;

&lt;h3&gt;
  
  
  What are the four types of product risk?
&lt;/h3&gt;

&lt;p&gt;Marty Cagan's four risks are value (will they buy or use it?), usability (can they figure it out?), feasibility (can we build it with our time, skills and technology?) and business viability (does it work for the rest of the business?). Strategyzer's &lt;em&gt;Testing Business Ideas&lt;/em&gt; uses three, desirability, feasibility and viability, and its library tags usability tests as desirability.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the strongest evidence in product validation?
&lt;/h3&gt;

&lt;p&gt;Money or a signature is the strongest evidence, such as a payment, a pledge or a signed letter of intent. It beats what people do, which beats what people say. Osterwalder's rule of thumb is that evidence gets stronger the closer it comes to a real purchase, which is why a simulated sale outranks any interview or survey.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I start with customer interviews or a landing page?
&lt;/h3&gt;

&lt;p&gt;Start with interviews when you are still unsure the problem is real, because you need people's own words before you can write an offer. Start with a landing page when the problem is clear and you can state your offer in one sentence. Many teams run both inside two weeks: interviews first, then a page built from what they heard.&lt;/p&gt;

&lt;h3&gt;
  
  
  How many experiments should I run before building?
&lt;/h3&gt;

&lt;p&gt;There is no fixed number. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. As a cadence, Strategyzer's book summary suggests three to five small experiments on your highest-priority assumptions in the first month. Stop early when one fails clearly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;Silicon Valley Product Group — The Four Big Risks (Marty Cagan, 2017)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://f.hubspotusercontent40.net/hubfs/4952096/Testing%20Business%20Ideas_ExperimentLibrary-Poster.pdf" rel="noopener noreferrer"&gt;Strategyzer — Testing Business Ideas Experiment Library (poster)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.wiley.com/en-us/Testing+Business+Ideas%3A+A+Field+Guide+for+Rapid+Experimentation-p-9781119551447" rel="noopener noreferrer"&gt;Wiley — Testing Business Ideas: A Field Guide for Rapid Experimentation (Bland and Osterwalder, 2019)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.strategyzer.com/library/testing-business-ideas-book-summary" rel="noopener noreferrer"&gt;Strategyzer — Testing Business Ideas: Book Summary and Key Takeaways (2025)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.strategyzer.com/library/how-strong-is-your-innovation-evidence" rel="noopener noreferrer"&gt;Strategyzer — How Strong Is Your Innovation Evidence? (Alex Osterwalder, 2017)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://exp-platform.com/Documents/2023-03-11EncyclopeiaMLDSABTestingFinal.pdf" rel="noopener noreferrer"&gt;Kohavi and Longbotham — Online Controlled Experiments and A/B Tests, Encyclopedia of Machine Learning and Data Science (2023)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.cbinsights.com/research/report/startup-failure-reasons-top/" rel="noopener noreferrer"&gt;CB Insights — Top reasons startups fail (report, March 2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.producttalk.org/2023/10/assumption-testing/" rel="noopener noreferrer"&gt;Product Talk — Assumption Testing: Everything You Need to Know to Get Started (Teresa Torres, 2023)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.thoughtworks.com/insights/articles/how-implement-hypothesis-driven-development" rel="noopener noreferrer"&gt;Thoughtworks — How to Implement Hypothesis-Driven Development (Barry O'Reilly, 2014)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" rel="noopener noreferrer"&gt;Nielsen Norman Group — Why You Only Need to Test with 5 Users (2000)&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;This story was originally published on 9 November 2026 at &lt;a href="https://www.specky.space/blog/how-to-choose-a-product-experiment" rel="noopener noreferrer"&gt;https://www.specky.space/blog/how-to-choose-a-product-experiment&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Fathy Shalaby is a scientist by training (master's). He worked at LexisNexis and Canva, and writes How to Be a Product Engineer.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>startup</category>
      <category>productivity</category>
      <category>ux</category>
    </item>
    <item>
      <title>The Four Product Risks: What Counts as Evidence for Each</title>
      <dc:creator>Fathy Shalaby</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:48:48 +0000</pubDate>
      <link>https://dev.to/fathy_shalaby/the-four-product-risks-what-counts-as-evidence-for-each-39j2</link>
      <guid>https://dev.to/fathy_shalaby/the-four-product-risks-what-counts-as-evidence-for-each-39j2</guid>
      <description>&lt;p&gt;AI made feasibility cheap to prove. Value, usability and viability still need their own evidence, and a great prototype demo counts for less than it feels. The Risk Receipt shows what you've actually retired.&lt;/p&gt;

&lt;p&gt;The four product risks are value (will customers buy or use it?), usability (can they figure it out?), feasibility (can we build it?) and viability (does it work for the business?). Marty Cagan's framing still holds, but AI changed the economics: feasibility is now cheap to test, while the other three still need their own evidence. A prototype demo says far less about value than it feels.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cagan's four big risks are independent: evidence for one tells you little about another (&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;SVPG, 2017&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;In CB Insights' 2026 analysis of 385 failed VC-backed startups with known causes, 43% cited poor product-market fit and 19% unsustainable unit economics. Those are value and viability problems, not build problems (&lt;a href="https://www.cbinsights.com/research/report/startup-failure-reasons-top/" rel="noopener noreferrer"&gt;CB Insights, 2026&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Five test users find about 85% of usability problems, so usability evidence is cheap to buy (&lt;a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" rel="noopener noreferrer"&gt;Nielsen Norman Group, 2000&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Value evidence is behaviour with a cost attached, usually gathered before people see your solution.&lt;/li&gt;
&lt;li&gt;A Risk Receipt keeps each claim, its evidence and an honest "counts / weak / unverified / unexamined" verdict together, so nobody mistakes four nice demo calls for validation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What are the four product risks?
&lt;/h2&gt;

&lt;p&gt;Cagan set out the four big risks in a 2017 SVPG article, and they have become the standard checklist for product discovery (&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;SVPG, 2017&lt;/a&gt;):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;th&gt;The question&lt;/th&gt;
&lt;th&gt;Owner in Cagan's model&lt;/th&gt;
&lt;th&gt;Strongest evidence&lt;/th&gt;
&lt;th&gt;Weakest evidence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Value&lt;/td&gt;
&lt;td&gt;Will customers buy it, or users choose to use it?&lt;/td&gt;
&lt;td&gt;Product manager&lt;/td&gt;
&lt;td&gt;Behaviour with a cost: pilots, pre-orders, workarounds they already pay for&lt;/td&gt;
&lt;td&gt;"I'd use this", demo enthusiasm&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usability&lt;/td&gt;
&lt;td&gt;Can users figure out how to use it?&lt;/td&gt;
&lt;td&gt;Product designer&lt;/td&gt;
&lt;td&gt;Unassisted task attempts by target users&lt;/td&gt;
&lt;td&gt;Stakeholder walkthroughs, "looks clean"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Feasibility&lt;/td&gt;
&lt;td&gt;Can engineers build it with the time, skills and technology we have?&lt;/td&gt;
&lt;td&gt;Lead engineer&lt;/td&gt;
&lt;td&gt;A working path on realistic data, with a list of what was skipped&lt;/td&gt;
&lt;td&gt;A happy-path demo on seeded data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Viability&lt;/td&gt;
&lt;td&gt;Does it work for the rest of the business?&lt;/td&gt;
&lt;td&gt;Product manager&lt;/td&gt;
&lt;td&gt;Cost-to-serve, pricing tested on a buyer, sign-off from sales, legal, support&lt;/td&gt;
&lt;td&gt;"We'll figure out pricing later"&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In Cagan's words, viability is "whether this solution also works for the various aspects of our business". That covers pricing, cost to serve, sales, legal, compliance and support.&lt;/p&gt;

&lt;p&gt;The four risks are independent. A feature can be trivially feasible, perfectly usable and still have no value. A feature can have obvious value and be unviable, for example when each use costs more in inference than the customer pays.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why did AI change which product risks matter most?
&lt;/h2&gt;

&lt;p&gt;When building was expensive, feasibility was the gate everything queued behind. Teams spent discovery time on the other three risks while engineering estimated. Now that a working prototype takes an afternoon, the queue is gone, and so is the forced pause where the other risks used to get examined.&lt;/p&gt;

&lt;p&gt;Benedikt Kantus put it bluntly in an October 2026 essay: "AI has made feasibility almost trivial. It has not touched the other three." (&lt;a href="https://www.leadinginproduct.com/p/vibe-coding-product-management" rel="noopener noreferrer"&gt;Leading in Product, 2026&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Failure data agrees about where products die. CB Insights studied 431 VC-backed companies that shut down since 2023 and could identify causes for 385. "Ran out of capital" topped the list at 70%, but the report calls that the final cause rather than the root one. Underneath it: poor product-market fit at 43%, bad timing at 29% and unsustainable unit economics at 19% (&lt;a href="https://www.cbinsights.com/research/report/startup-failure-reasons-top/" rel="noopener noreferrer"&gt;CB Insights, 2026&lt;/a&gt;). And for products that do survive, Pendo's 2019 analysis found 80% of features in the average product are rarely or never used (&lt;a href="https://www.pendo.io/resources/the-2019-feature-adoption-report/" rel="noopener noreferrer"&gt;Pendo, 2019&lt;/a&gt;). Value risk, not build risk, is the expensive one.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is feasibility really solved?
&lt;/h3&gt;

&lt;p&gt;For prototypes, mostly. For production, less clearly. In METR's randomised trial, 16 experienced open-source developers working on 246 real issues in their own repositories took 19% longer with early-2025 AI tools, even though they had forecast a 24% speedup and believed afterwards they had been 20% faster (&lt;a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/" rel="noopener noreferrer"&gt;METR, 2025&lt;/a&gt;). A February 2026 follow-up with 57 developers found only weak evidence of a speedup, with confidence intervals that include zero, and METR itself warns the data is an unreliable measure (&lt;a href="https://metr.org/blog/2026-02-24-uplift-update/" rel="noopener noreferrer"&gt;METR, 2026&lt;/a&gt;). The honest reading: the first version is cheap; the edge cases, reliability and maintenance are not free.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a Risk Receipt?
&lt;/h2&gt;

&lt;p&gt;The problem isn't that PMs stopped caring about the other risks. It's that a polished prototype feels like evidence for all four. Demo it to five people, hear "this is great", and the team walks away believing value, usability and viability are covered. Each risk has its own currency of evidence, and most teams pay in the wrong one.&lt;/p&gt;

&lt;p&gt;A Risk Receipt is a short record you attach to any meaningful product bet. For each of the four product risks it answers three questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;What's the claim?&lt;/strong&gt; One sentence, specific enough to be wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What evidence do we have, and what kind is it?&lt;/strong&gt; Linked to the source, not paraphrased.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does that evidence actually count for this risk?&lt;/strong&gt; This is the part teams skip.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The third question is the point. Evidence has an exchange rate. Some evidence is strong currency for one risk and close to worthless for another.&lt;/p&gt;

&lt;h2&gt;
  
  
  What counts as evidence for each risk?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Value: pay in behaviour, not compliments
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Counts:&lt;/strong&gt; customers describing the problem unprompted, with a recent, concrete example. Existing workarounds (spreadsheets, scripts, manual processes) that cost them real time. Commitments with a cost attached: a calendar slot, a pilot with their data, a pre-order, an introduction to their boss.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't count:&lt;/strong&gt; "I'd use this." Enthusiasm during a demo. A prototype's click-through rate in a moderated session. Upvotes on an idea board from people who will never pay.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The tell:&lt;/strong&gt; if the strongest evidence for value is someone reacting to your prototype, you've collected usability evidence and labelled it value. People react politely to anything competent-looking you show them. Value evidence usually comes from before they saw your solution, in how they talk about the problem. Cheap ways to buy it include a &lt;a href="https://www.specky.space/blog/fake-door-test-template" rel="noopener noreferrer"&gt;fake door test&lt;/a&gt; or a &lt;a href="https://www.specky.space/blog/presale-interest-page-template" rel="noopener noreferrer"&gt;presale interest page&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Usability: pay in observed tasks, not opinions
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Counts:&lt;/strong&gt; watching a real target user attempt a real task without coaching, and recording where they hesitate or fail. Task completion on the first attempt. The questions they ask out loud.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't count:&lt;/strong&gt; "It looks clean." A stakeholder walkthrough. Your own team using it every day (they already know where everything is).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The tell:&lt;/strong&gt; if nobody outside the team has attempted a task alone, usability is unverified, no matter how good the prototype looks.&lt;/p&gt;

&lt;p&gt;This is the one risk AI prototyping genuinely helps with. A clickable, data-backed prototype gets you real task observation weeks earlier than a static mockup. It's a real gain, just a gain for one risk. Our &lt;a href="https://www.specky.space/blog/usability-test-template" rel="noopener noreferrer"&gt;usability test template&lt;/a&gt; covers how to run the sessions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Feasibility: pay in a working path, including the ugly parts
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Counts:&lt;/strong&gt; a prototype that runs against realistic data volumes, real permissions and real latency. An engineer's honest list of what the prototype skipped: auth, migrations, error states, the integration that's rate-limited.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't count:&lt;/strong&gt; a demo on the happy path with seeded data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The tell:&lt;/strong&gt; ask "what did this prototype not have to handle?" If the answer is long, feasibility is partly retired, not fully.&lt;/p&gt;

&lt;h3&gt;
  
  
  Viability: pay in numbers someone outside product would sign
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Counts:&lt;/strong&gt; a cost-to-serve estimate (for AI features, cost per use at expected volume). A pricing hypothesis you've said out loud to a buyer. A check with whoever owns support, sales, security or legal. For B2B, whether a buyer's procurement process allows this.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't count:&lt;/strong&gt; "We'll figure out pricing later." Assuming usage costs are negligible because the prototype was cheap to run for one person.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The tell:&lt;/strong&gt; if no one outside the product team has seen the bet, viability is unexamined.&lt;/p&gt;

&lt;h2&gt;
  
  
  A worked example: four happy customers, one empty receipt
&lt;/h2&gt;

&lt;p&gt;Say you're a two-person founding team building a B2B tool, and a customer call sparks an idea: an AI assistant that drafts weekly status updates from project activity. You prototype it over a weekend. It works. You show it to four customers on calls. All four say they love it.&lt;/p&gt;

&lt;p&gt;Here's the honest Risk Receipt:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;th&gt;Claim&lt;/th&gt;
&lt;th&gt;Evidence&lt;/th&gt;
&lt;th&gt;Counts?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Value&lt;/td&gt;
&lt;td&gt;Team leads will use weekly drafts instead of writing updates themselves&lt;/td&gt;
&lt;td&gt;4 positive demo reactions&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Weak&lt;/strong&gt;: reactions to a solution, not behaviour. No one described writing updates as a pain before you showed them.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usability&lt;/td&gt;
&lt;td&gt;Leads can review and send a draft in under 5 minutes&lt;/td&gt;
&lt;td&gt;None observed&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Unverified&lt;/strong&gt;: nobody tried it alone.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Feasibility&lt;/td&gt;
&lt;td&gt;Drafts stay accurate on real project data&lt;/td&gt;
&lt;td&gt;Weekend prototype on your own workspace&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Partial&lt;/strong&gt;: no test on a messy customer workspace.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Viability&lt;/td&gt;
&lt;td&gt;Cost per draft stays well under what the plan charges&lt;/td&gt;
&lt;td&gt;Not estimated&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Unexamined&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Four enthusiastic calls, and the receipt says you've retired almost nothing. That's not a reason to kill the idea. It's a reason to run three cheap checks before building the production version:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Value:&lt;/strong&gt; go back to two of those customers and ask how they write updates today, how long it takes, and what happened the last time one was late. Listen for a workaround.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Usability:&lt;/strong&gt; give one customer the prototype on their own data and watch them use it, unguided, for one cycle.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Viability:&lt;/strong&gt; multiply your per-draft model cost by realistic weekly volume across your customer base, and compare it with plan price.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;The viability maths (illustrative numbers).&lt;/strong&gt; Suppose a draft costs €0.06 in model calls on your own workspace, but a real customer workspace has 5 times more activity, so €0.30 per draft. A customer with 40 team leads generates 160 drafts a month, or €48 in model cost, against a plan priced at €60 a month. That is 80% of revenue gone on one feature. Better to know on day three than in month three.&lt;/p&gt;

&lt;p&gt;Each check takes days, not weeks. That's the real dividend from cheap feasibility: you can afford to buy real evidence for the other three risks before you commit.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many customers or users do you need for each risk?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Usability:&lt;/strong&gt; small numbers work. Nielsen's analysis found 5 users uncover about 85% of usability problems, and recommended three rounds of 5 with fixes in between over one round of 15 (&lt;a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" rel="noopener noreferrer"&gt;Nielsen Norman Group, 2000&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Value:&lt;/strong&gt; the count matters less than the cost of the signal. Two customers who each give you a pilot with their own data outweigh twenty who say "sounds great". Look for at least 3 independent commitments before treating value as retired for a segment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feasibility:&lt;/strong&gt; one realistic run, reviewed by the engineer who will own it, plus a written list of what was skipped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Viability:&lt;/strong&gt; one cost model with stated assumptions and one conversation with each function that has to say yes (sales, support, legal or security).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are starting points for a small team, not benchmarks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do receipts beat a discovery checklist?
&lt;/h2&gt;

&lt;p&gt;You could turn the four risks into a checklist. Teams do. Checklists fail predictably: someone ticks "value validated" because a demo went well, and the tick hides what the evidence was.&lt;/p&gt;

&lt;p&gt;A receipt keeps the evidence attached to the claim. Three months later, when the feature is underperforming and someone asks "why did we build this?", the answer isn't a memory or a slide. It's a record: here's what we believed, here's what we saw, here's which risks we knowingly left open. For the structure that keeps those records linked over time, see our &lt;a href="https://www.specky.space/blog/product-memory-decision-graph-small-ai-teams" rel="noopener noreferrer"&gt;decision graph for small AI teams&lt;/a&gt;. If your team ships without receipts at all, check it against the &lt;a href="https://www.specky.space/blog/stop-the-feature-factory-10-warning-signs-your-team-is-shipping-without-evidence" rel="noopener noreferrer"&gt;10 warning signs of a feature factory&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;That last part matters. Shipping with an open risk is often the right call, because small teams can't afford to retire everything. The failure is shipping with an open risk you thought was closed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Counting demo reactions as value.&lt;/strong&gt; The single most common error. Reactions are usability evidence at best.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Testing usability with insiders.&lt;/strong&gt; Colleagues and friendly customers already know the product's logic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating a prototype as proof of production feasibility.&lt;/strong&gt; Ask for the skipped list every time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Leaving viability to "later".&lt;/strong&gt; For AI features especially, cost per use can turn a loved feature into a loss-maker.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trying to retire every risk.&lt;/strong&gt; Pick the one or two that would kill the bet and buy evidence for those first. Our &lt;a href="https://www.specky.space/blog/de-risking-bets-assumption-mapping" rel="noopener noreferrer"&gt;assumption mapping guide&lt;/a&gt; helps rank them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we run Risk Receipts in Specky
&lt;/h2&gt;

&lt;p&gt;Customer signals (interview notes, support threads, call transcripts, analytics events) land in the Product Graph as typed nodes, each keeping its source. A drafted insight or decision cites the specific nodes it rests on, so "four customers loved it" can't quietly become "customers need this". For the value check, Alex, Specky's AI research agent, can run short interviews about customers' current process before they see your solution, and the transcripts come back as evidence. For early demand, a validation page measures whether people take an action with a cost attached. Judgement about which risks to leave open stays with you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with one receipt this week
&lt;/h2&gt;

&lt;p&gt;You don't need a tool to start. Pick the bet your team is most excited about right now, the one with the best prototype. Write four lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Value:       [claim] | [evidence + link] | counts / weak / unverified / unexamined
Usability:   [claim] | [evidence + link] | counts / weak / unverified / unexamined
Feasibility: [claim] | [evidence + link] | counts / weak / unverified / unexamined
Viability:   [claim] | [evidence + link] | counts / weak / unverified / unexamined
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the most exciting bet has the emptiest receipt, you've found where your next week of discovery should go. You can also sketch the assumptions visually with the free &lt;a href="https://www.specky.space/tools/assumption-map" rel="noopener noreferrer"&gt;assumption map tool&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Feasibility used to be the expensive question. Now it's the cheap one. The teams that win this cycle won't be the ones who build fastest. They'll be the ones who can show what they knew when they decided.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What are Marty Cagan's four big risks?
&lt;/h3&gt;

&lt;p&gt;They are value risk (will customers buy or choose to use it), usability risk (can users figure out how to use it), feasibility risk (can engineers build it with the time, skills and technology available) and business viability risk (does it work for the rest of the business). Cagan published them on SVPG in 2017 and assigns value and viability to the product manager.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which of the four product risks is most often underestimated?
&lt;/h3&gt;

&lt;p&gt;Value risk. A working prototype and polite reactions create a strong feeling of validation that isn't backed by behaviour. CB Insights' 2026 study found 43% of failed startups with known causes cited poor product-market fit. Viability is a close second for AI features, where cost per use can outgrow the price.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does a prototype test value risk?
&lt;/h3&gt;

&lt;p&gt;Rarely on its own. Showing a prototype mainly tests usability and some feasibility. Value evidence comes from behaviour with a cost: people describing the problem unprompted, existing workarounds, pilots, pre-orders or introductions. You can use a prototype in a value test, for example by asking for a paid pilot, but the reaction alone doesn't count.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you test business viability risk?
&lt;/h3&gt;

&lt;p&gt;Write down the numbers someone outside product would need to approve: cost to serve per customer, expected price, support load, and any legal, security or procurement constraints. Then show them to those people. For AI features, multiply cost per use by realistic volume and compare it with plan price before committing to build.&lt;/p&gt;

&lt;h3&gt;
  
  
  Did AI make feasibility risk disappear?
&lt;/h3&gt;

&lt;p&gt;No, it made the first version cheap. Prototypes now take hours, but production work on mature codebases is a different matter: METR's 2025 trial found experienced developers took 19% longer with AI tools on their own repositories, and its 2026 follow-up found only weak evidence of a speedup. Keep asking what a prototype didn't have to handle.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href="https://www.svpg.com/four-big-risks/" rel="noopener noreferrer"&gt;SVPG, Marty Cagan — The Four Big Risks (2017)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.leadinginproduct.com/p/vibe-coding-product-management" rel="noopener noreferrer"&gt;Leading in Product, Benedikt Kantus — vibe coding and product management (2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.cbinsights.com/research/report/startup-failure-reasons-top/" rel="noopener noreferrer"&gt;CB Insights — The top reasons startups fail (2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" rel="noopener noreferrer"&gt;Nielsen Norman Group — Why You Only Need to Test with 5 Users (2000)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/" rel="noopener noreferrer"&gt;METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://metr.org/blog/2026-02-24-uplift-update/" rel="noopener noreferrer"&gt;METR — Uplift update on late-2025 AI tools (2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.pendo.io/resources/the-2019-feature-adoption-report/" rel="noopener noreferrer"&gt;Pendo — The 2019 Feature Adoption Report&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;This story was originally published on 16 October 2026 at &lt;a href="https://www.specky.space/blog/what-counts-as-evidence-four-product-risks" rel="noopener noreferrer"&gt;https://www.specky.space/blog/what-counts-as-evidence-four-product-risks&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Fathy Shalaby is a scientist by training (master's). He worked at LexisNexis and Canva, and writes How to Be a Product Engineer.&lt;/p&gt;

</description>
      <category>startup</category>
      <category>productivity</category>
      <category>ai</category>
      <category>ux</category>
    </item>
    <item>
      <title>What Is a Product Engineer? Role, Skills, Pay and Career Path</title>
      <dc:creator>Fathy Shalaby</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:45:43 +0000</pubDate>
      <link>https://dev.to/fathy_shalaby/what-is-a-product-engineer-role-skills-pay-and-career-path-2eml</link>
      <guid>https://dev.to/fathy_shalaby/what-is-a-product-engineer-role-skills-pay-and-career-path-2eml</guid>
      <description>&lt;p&gt;A product engineer talks to users, decides what to build, ships it and checks whether it worked. Here is the role, how it differs from software engineering and product management, what it pays and how to grow into it.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;product engineer&lt;/strong&gt; is a software engineer who owns what gets built as well as how it gets built. They talk to users, choose the smallest useful change, ship it across the stack and check whether it actually helped. The title is common at product-led software companies such as PostHog, Intercom and incident.io. In hardware and manufacturing, the same title means a different job.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The idea predates the hype: Gergely Orosz described nine traits of "product-minded" engineers in 2019 (&lt;a href="https://blog.pragmaticengineer.com/the-product-minded-engineer/" rel="noopener noreferrer"&gt;The Pragmatic Engineer, 2019&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;PostHog's handbook sets a high bar: its product engineers talk to users, decide what to build and also do customer support (&lt;a href="https://posthog.com/blog/what-is-a-product-engineer" rel="noopener noreferrer"&gt;PostHog&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Pay data is thin but real: at Intercom, Levels.fyi reports a median of €85.3K total compensation for product engineers in Ireland, from 21 submissions (&lt;a href="https://www.levels.fyi/companies/intercom/salaries/software-engineer/title/product-engineer" rel="noopener noreferrer"&gt;Levels.fyi, Aug 2026&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Demand is moving towards broad ownership: openings for mobile and front-end specialists nearly halved in the year to November 2026 (&lt;a href="https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late" rel="noopener noreferrer"&gt;Lenny's Newsletter, Nov 2026&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The skill that separates the role from fast coding is measurement: checking user outcomes, not merges.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What does a product engineer do?
&lt;/h2&gt;

&lt;p&gt;A product engineer connects a customer problem to working software and stays with it after release. A typical scope includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Talking to users&lt;/strong&gt; in calls, support tickets and usage data, to find the repeated obstacle behind a request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shaping scope&lt;/strong&gt;: turning a problem into the smallest change that can be tested.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Making product and technical trade-offs&lt;/strong&gt; together, such as cutting an edge case that costs a week but affects 2% of users.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shipping full-stack changes&lt;/strong&gt;, with instrumentation so the team can see what happened.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Following up&lt;/strong&gt;: reviewing the data, talking to the users who tried it, then improving, expanding or removing the change.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;PostHog's Product Engineer Handbook pushes this furthest. Its product engineers own pricing, revenue and user experience, and every one of them also does customer support. The handbook's summary is blunt: "Product engineers talk to users. They decide what to build." (&lt;a href="https://posthog.com/blog/what-is-a-product-engineer" rel="noopener noreferrer"&gt;PostHog&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Not every company goes that far. Some use the title for an engineer who joins discovery but works with a product manager who owns strategy and priorities. Agree the boundaries before work starts.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does a product engineer's week look like?
&lt;/h3&gt;

&lt;p&gt;An illustrative week on a 5-person B2B team:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Day&lt;/th&gt;
&lt;th&gt;Work&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Monday&lt;/td&gt;
&lt;td&gt;Reads 30 support tickets tagged "export"; joins one customer call&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tuesday&lt;/td&gt;
&lt;td&gt;Proposes two options: a CSV fix (1 day) and a scheduled report (5 days); picks the CSV fix as the first slice&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wednesday&lt;/td&gt;
&lt;td&gt;Ships the fix behind a feature flag with an event on successful export&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Thursday&lt;/td&gt;
&lt;td&gt;Reviews a teammate's change; checks error logs for the new code path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Friday&lt;/td&gt;
&lt;td&gt;Compares export success before and after; emails the 4 customers who raised it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The code took one day. The other four went on understanding, verifying and closing the loop. That ratio is the job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where did the term "product engineer" come from?
&lt;/h2&gt;

&lt;p&gt;The role grew out of a mindset that engineers and engineering leaders wrote about long before AI coding tools.&lt;/p&gt;

&lt;p&gt;In 2019, Gergely Orosz described the product-minded software engineer: someone who wants to know why a feature exists, how people use it and what the data says. He observed that such engineers often become the go-to people for product managers and frequently move into team lead roles (&lt;a href="https://blog.pragmaticengineer.com/the-product-minded-engineer/" rel="noopener noreferrer"&gt;The Pragmatic Engineer, 2019&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Companies then turned the mindset into a title. Intercom and incident.io advertise their engineering roles under the Product Engineer title, and PostHog published a full handbook on the role. PostHog's handbook also collects definitions from incident.io, Cursor's Lee Robinson and Orosz, which share one idea: outcomes for users matter more than the specific implementation (&lt;a href="https://posthog.com/blog/what-is-a-product-engineer" rel="noopener noreferrer"&gt;PostHog&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;AI coding tools sped the trend up. In Stack Overflow's 2025 survey, 84% of developers used or planned to use AI tools and 51% of professional developers used them daily (&lt;a href="https://survey.stackoverflow.co/2025/ai" rel="noopener noreferrer"&gt;Stack Overflow, 2025&lt;/a&gt;). When typing code costs less, knowing what code is worth typing is the scarcer skill.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does "product engineering" mean?
&lt;/h2&gt;

&lt;p&gt;"Product engineering" has three common meanings. Search results mix them, so check which one a job ad or article means.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;th&gt;Where you see it&lt;/th&gt;
&lt;th&gt;What the work is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Product engineering in software&lt;/td&gt;
&lt;td&gt;Product-led software companies, startups&lt;/td&gt;
&lt;td&gt;Engineers who own user outcomes, not only tickets (this article)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Product engineering in hardware and manufacturing&lt;/td&gt;
&lt;td&gt;Electronics, semiconductors, automotive, consumer goods&lt;/td&gt;
&lt;td&gt;Taking a physical product from design into reliable volume production: tolerances, yield, testing, cost&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Product engineering services"&lt;/td&gt;
&lt;td&gt;IT outsourcing and consultancies&lt;/td&gt;
&lt;td&gt;An agency designing and building a software product for a client&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The salary aggregators blend the first two, which matters when you read pay data below.&lt;/p&gt;

&lt;h2&gt;
  
  
  Product engineer vs software engineer vs product manager: what is the difference?
&lt;/h2&gt;

&lt;p&gt;The three roles overlap. The difference is where accountability sits.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Product engineer&lt;/th&gt;
&lt;th&gt;Software engineer&lt;/th&gt;
&lt;th&gt;Product manager&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Core question&lt;/td&gt;
&lt;td&gt;"Did this change solve the user's problem?"&lt;/td&gt;
&lt;td&gt;"Is this correct, secure and maintainable?"&lt;/td&gt;
&lt;td&gt;"Is this the right problem and bet for the business?"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Owns&lt;/td&gt;
&lt;td&gt;Problem-to-outcome for a scoped area&lt;/td&gt;
&lt;td&gt;Implementation quality within agreed scope&lt;/td&gt;
&lt;td&gt;Priorities, strategy, the case for the bet&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Talks to users&lt;/td&gt;
&lt;td&gt;Regularly&lt;/td&gt;
&lt;td&gt;Sometimes&lt;/td&gt;
&lt;td&gt;Constantly&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Writes production code&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Rarely&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Measured by&lt;/td&gt;
&lt;td&gt;User and product metrics after release&lt;/td&gt;
&lt;td&gt;Delivery quality, reliability, review&lt;/td&gt;
&lt;td&gt;Business and customer outcomes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Typical failure&lt;/td&gt;
&lt;td&gt;Ships fast, skips the follow-up&lt;/td&gt;
&lt;td&gt;Builds the spec perfectly; the spec was wrong&lt;/td&gt;
&lt;td&gt;Owns documents, not outcomes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A product engineer is not a product manager who can code, and not a software engineer who attends more meetings. They hold an engineering quality bar and a product outcome at the same time. On teams without a PM, they also pick up prioritisation for their area.&lt;/p&gt;

&lt;p&gt;The product builder is a related but different role: a product person who prototypes, usually without owning production code. See &lt;a href="https://www.specky.space/blog/product-builder-vs-product-engineer" rel="noopener noreferrer"&gt;product builder vs product engineer&lt;/a&gt; for that comparison.&lt;/p&gt;

&lt;h2&gt;
  
  
  What skills does a product engineer need?
&lt;/h2&gt;

&lt;p&gt;Orosz's nine traits are still the best checklist for product engineer skills. Here they are, with what each looks like in practice:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Trait (Orosz, 2019)&lt;/th&gt;
&lt;th&gt;In practice&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1. Proactive with product ideas and opinions&lt;/td&gt;
&lt;td&gt;Proposes an alternative when a spec looks wrong&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2. Interest in the business, user behaviour and data&lt;/td&gt;
&lt;td&gt;Reads the dashboard before the ticket&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3. Curiosity about "why?"&lt;/td&gt;
&lt;td&gt;Asks what problem a feature solves before estimating&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4. Strong communication with non-engineers&lt;/td&gt;
&lt;td&gt;Explains a trade-off to sales or support without jargon&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5. Offering product/engineering trade-offs upfront&lt;/td&gt;
&lt;td&gt;"We can ship 80% of this in a third of the time if we drop X"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6. Pragmatic handling of edge cases&lt;/td&gt;
&lt;td&gt;Solves rare cases with copy or support, not code, when that is enough&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7. Quick product validation cycles&lt;/td&gt;
&lt;td&gt;Gets feedback before the feature is production-ready&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8. End-to-end ownership&lt;/td&gt;
&lt;td&gt;Owns rollout and verification, not just the merge&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9. Product instinct from repeated learning cycles&lt;/td&gt;
&lt;td&gt;Predicts which ideas will work and checks that prediction&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On top of those, the role in 2026 needs three technical skills:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Full-stack range&lt;/strong&gt;: enough front end, back end and data to ship a change without three handoffs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Instrumentation&lt;/strong&gt;: adding the events and checks that show whether users reached the outcome.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reviewing AI-generated code&lt;/strong&gt;: in the same Stack Overflow survey, 46% of developers distrusted the accuracy of AI tools and only 33% trusted it. Treat generated code as a draft you are accountable for.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How much does a product engineer earn?
&lt;/h2&gt;

&lt;p&gt;Product engineer salary data is limited because many companies file the role under "software engineer". Two named sources, with their limits:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Scope&lt;/th&gt;
&lt;th&gt;Figure&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Levels.fyi, Intercom&lt;/td&gt;
&lt;td&gt;Product Engineers in Ireland, 21 submissions, updated August 2026&lt;/td&gt;
&lt;td&gt;Median total compensation €85.3K; average €71K at level 1, €81.3K at level 2, €112K at level 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Indeed&lt;/td&gt;
&lt;td&gt;All "product engineer" postings in the US, about 2,500 salaries over 36 months, updated November 2026&lt;/td&gt;
&lt;td&gt;Average base $123,785 plus about $3,500 cash bonus&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: &lt;a href="https://www.levels.fyi/companies/intercom/salaries/software-engineer/title/product-engineer" rel="noopener noreferrer"&gt;Levels.fyi&lt;/a&gt;, &lt;a href="https://www.indeed.com/salaries/product-engineer-Salaries" rel="noopener noreferrer"&gt;Indeed&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Read them carefully. The Levels.fyi figure is one company in one country. The Indeed figure mixes software product engineers with hardware and manufacturing product engineers, which are different jobs. For a software role, compare offers against software engineer bands at the same level and location; the title alone rarely carries a premium.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a product engineer job description look like?
&lt;/h2&gt;

&lt;p&gt;Copy and adapt this template. The decision rights section is the part most job ads leave out.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Product Engineer, [area]&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Outcome you own:&lt;/strong&gt; [user outcome, e.g. "customers can export the data they need without contacting support"].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You will:&lt;/strong&gt; talk to at least 2 customers a month; propose and scope changes; ship full-stack code with tests and instrumentation; review results 2 and 6 weeks after release; write up what you learned.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You decide:&lt;/strong&gt; scope and implementation for your area, and whether to iterate, expand or remove a shipped change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You agree with product/leadership:&lt;/strong&gt; which problems your area takes on this quarter.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You have:&lt;/strong&gt; shipped production software end to end; examples of changing a plan because of user evidence; clear written communication.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;We evaluate:&lt;/strong&gt; a walkthrough of a feature you shipped, the evidence behind it and what changed afterwards.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  How does product engineering work, step by step?
&lt;/h2&gt;

&lt;p&gt;Say a 6-person team hears that preparing a weekly report "takes forever" (illustrative). "Add AI summaries" is a proposed solution, not a confirmed problem.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Find the real job.&lt;/strong&gt; Ask 5 users to walk through the last time they prepared the report. Note the repeated steps and workarounds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate evidence from guesses.&lt;/strong&gt; "3 of 5 users copied figures into a spreadsheet" is an observation. "Everyone wants an AI report" is an assumption.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose the smallest useful change.&lt;/strong&gt; Compare a saved report (2 days), a data export (1 day) and a generated summary (2 weeks) against the actual bottleneck.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the behaviour and boundaries.&lt;/strong&gt; Who can use it, what data it reads, what it must not do, how a user corrects it. A copyable structure is in the &lt;a href="https://www.specky.space/blog/ai-coding-agent-prd-template" rel="noopener noreferrer"&gt;AI coding-agent PRD template&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ship with a check.&lt;/strong&gt; Pick one measure tied to the job, such as report completion on the next weekly cycle, plus an error or quality check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Review and decide.&lt;/strong&gt; Combine the numbers with 2 or 3 follow-up conversations. Improve, expand or remove.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the discovery half of this loop, see &lt;a href="https://www.specky.space/blog/customer-discovery-for-product-engineers" rel="noopener noreferrer"&gt;customer discovery for product engineers&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you know whether product engineering is working?
&lt;/h2&gt;

&lt;p&gt;Measure outcomes, not output. Three findings show why self-reported speed is a weak signal:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Perceived speed can mislead.&lt;/strong&gt; In METR's July 2025 randomised trial, 16 experienced open-source developers took 19% longer on 246 real issues when allowed AI tools, yet believed afterwards that AI had made them 20% faster (&lt;a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/" rel="noopener noreferrer"&gt;METR, 2025&lt;/a&gt;). METR's February 2026 follow-up says its newer data is too unreliable to give a current estimate (&lt;a href="https://metr.org/blog/2026-02-24-uplift-update/" rel="noopener noreferrer"&gt;METR, 2026&lt;/a&gt;). The lesson is not "AI is slow"; it is "measure, don't assume".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speed can cost stability.&lt;/strong&gt; The 2025 DORA report, based on nearly 5,000 technology professionals, found 90% use AI at work and that AI adoption still has a negative relationship with software delivery stability (&lt;a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report" rel="noopener noreferrer"&gt;Google Cloud, 2025&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Engineers join too late.&lt;/strong&gt; Atlassian's State of Product 2026 survey found 80% of teams do not involve engineers in ideation, problem definition or roadmap creation (&lt;a href="https://www.mindtheproduct.com/inside-atlassians-state-of-product-2026-key-takeaways/" rel="noopener noreferrer"&gt;Mind the Product, 2025&lt;/a&gt;). A product engineering team should be the exception.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Useful measures: time from customer signal to shipped change, share of changes with a stated outcome, share reviewed after release, and how many changes were removed because they did not help.&lt;/p&gt;

&lt;h2&gt;
  
  
  When should a startup hire a product engineer?
&lt;/h2&gt;

&lt;p&gt;Hire a product engineer when the founders are close to customers but cannot carry every product decision and every implementation. The role fits best when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the product is past the first prototype and needs reliable releases;&lt;/li&gt;
&lt;li&gt;users are reachable, so engineers can talk to them directly;&lt;/li&gt;
&lt;li&gt;the team wants fewer handoffs, not another layer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The trend supports this shape. In November 2026, Lenny Rachitsky reported that demand for mobile and front-end specialists had nearly halved in a year, and read it as smaller engineering teams taking broader ownership (&lt;a href="https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late" rel="noopener noreferrer"&gt;Lenny's Newsletter, Nov 2026&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Do not hire one to settle a strategy question. If nobody can name the customer, the problem and the outcome, a broad product engineer role will absorb the confusion rather than resolve it.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you become a product engineer?
&lt;/h2&gt;

&lt;p&gt;From a software engineering role, take responsibility for one user problem rather than adding meetings.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Join 2 customer conversations a month&lt;/strong&gt; and write down what you observed, separately from what you concluded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Propose two solutions for your next ticket&lt;/strong&gt;, including a smaller, non-AI option.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;State the expected outcome before coding&lt;/strong&gt;, with the metric you will check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ship one narrow change with instrumentation&lt;/strong&gt; and review it after 2 weeks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Share what you learned&lt;/strong&gt;, including when the result missed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build a decision portfolio&lt;/strong&gt;: problem, evidence, trade-offs, what shipped, what changed. A polished demo shows implementation skill; this shows product judgment.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;PMs moving the other way, towards hands-on building, face a different path; see &lt;a href="https://www.specky.space/blog/should-product-managers-become-builders-pm-builder-role" rel="noopener noreferrer"&gt;should product managers become builders?&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common mistakes with the product engineer role
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Title without ownership.&lt;/strong&gt; Calling engineers "product engineers" while a PM still hands them finished specs changes nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unlimited scope.&lt;/strong&gt; One person cannot own strategy, research, design, code, QA, analytics and support for a whole product. Name what they do not own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merges as the success metric.&lt;/strong&gt; Merged and deployed are delivery states, not proof of user value.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Skipping design and research.&lt;/strong&gt; Product engineering complements specialists; it does not remove them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating AI output as finished.&lt;/strong&gt; Generated code still needs review, tests and the context of why the work exists.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where Specky fits
&lt;/h2&gt;

&lt;p&gt;Product engineers need the "why" behind a change as much as the ticket. Specky reads customer calls, feedback, Slack, Jira and analytics into a Product Graph, then drafts specs and tickets with the customer evidence cited inline, so the engineer can see which users asked and what they said. After launch it checks whether the bet worked. Through its MCP server, coding agents in Cursor or Claude Code get the same context. See how it works for &lt;a href="https://www.specky.space/for/technical-pms" rel="noopener noreferrer"&gt;technical PMs and product engineers&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is a product engineer a product manager who can code?
&lt;/h3&gt;

&lt;p&gt;No. A product engineer is an engineer first, accountable for production-quality software, who also owns user outcomes for their area. A product manager owns priorities and the business case across a wider scope. Some teams without PMs give product engineers prioritisation too, but that is a team design choice, not part of the definition.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between a product engineer and a full-stack engineer?
&lt;/h3&gt;

&lt;p&gt;Full-stack describes technical range: front end, back end and data. Product engineer describes accountability: the user outcome, not just the implementation. Most product engineers work full-stack because it lets them ship a change without handoffs, but a full-stack engineer who only builds specs as written is not doing product engineering.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do product engineers replace product managers?
&lt;/h3&gt;

&lt;p&gt;Not on current evidence. Open PM roles in November 2026 were at their highest in nearly four years (&lt;a href="https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late" rel="noopener noreferrer"&gt;Lenny's Newsletter&lt;/a&gt;). The roles divide the work differently: product engineers own outcomes within a scoped area, while PMs choose which problems the company bets on. For the bigger question, read &lt;a href="https://www.specky.space/blog/will-ai-replace-product-managers" rel="noopener noreferrer"&gt;will AI replace product managers?&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should product engineers use AI coding agents?
&lt;/h3&gt;

&lt;p&gt;Yes, for bounded work with clear acceptance criteria, but measure the result rather than trusting the feeling of speed. METR's 2025 trial found experienced developers 19% slower with AI tools while believing they were faster. Give agents the product context, review the output like a colleague's code, and check user outcomes after release.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is product engineering a good career path?
&lt;/h3&gt;

&lt;p&gt;For engineers who enjoy users and outcomes as much as code, yes. Orosz observed in 2019 that product-minded engineers often become the go-to people for product managers and frequently move into team lead roles. The role also builds the evidence-based judgment that &lt;a href="https://www.specky.space/blog/pm-role-is-splitting-which-pm-are-you-becoming" rel="noopener noreferrer"&gt;newer PM specialisations&lt;/a&gt; increasingly ask for.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is a product engineer in manufacturing?
&lt;/h3&gt;

&lt;p&gt;In manufacturing and hardware, a product engineer takes a physical product from design to reliable volume production, working on tolerances, testing, yield and cost. It shares the title, not the job, with the software role described here. Salary aggregators often mix the two, so filter pay data by industry before comparing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href="https://posthog.com/blog/what-is-a-product-engineer" rel="noopener noreferrer"&gt;PostHog — What is a product engineer? (Product Engineer Handbook)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://blog.pragmaticengineer.com/the-product-minded-engineer/" rel="noopener noreferrer"&gt;The Pragmatic Engineer — The Product-Minded Software Engineer (Gergely Orosz), 2019&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.levels.fyi/companies/intercom/salaries/software-engineer/title/product-engineer" rel="noopener noreferrer"&gt;Levels.fyi — Intercom Product Engineer Salaries, 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.indeed.com/salaries/product-engineer-Salaries" rel="noopener noreferrer"&gt;Indeed — Product Engineer salary in United States, 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late" rel="noopener noreferrer"&gt;Lenny's Newsletter — State of the tech job market: late 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.mindtheproduct.com/inside-atlassians-state-of-product-2026-key-takeaways/" rel="noopener noreferrer"&gt;Mind the Product — Four things we learned from Atlassian's State of Product 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://survey.stackoverflow.co/2025/ai" rel="noopener noreferrer"&gt;Stack Overflow — 2025 Developer Survey: AI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/" rel="noopener noreferrer"&gt;METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 2025&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://metr.org/blog/2026-02-24-uplift-update/" rel="noopener noreferrer"&gt;METR — Uplift update, February 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report" rel="noopener noreferrer"&gt;Google Cloud — Announcing the 2025 DORA report&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;This story was originally published on 4 November 2026 at &lt;a href="https://www.specky.space/blog/what-is-a-product-engineer" rel="noopener noreferrer"&gt;https://www.specky.space/blog/what-is-a-product-engineer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Fathy Shalaby is a scientist by training (master's). He worked at LexisNexis and Canva, and writes How to Be a Product Engineer.&lt;/p&gt;

</description>
      <category>career</category>
      <category>productivity</category>
      <category>startup</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
