<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Slava Kotenev</title>
    <description>The latest articles on DEV Community by Slava Kotenev (@slava_kotenev_159).</description>
    <link>https://dev.to/slava_kotenev_159</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4206374%2F5168e910-353a-4aa0-9849-60079e568b32.jpg</url>
      <title>DEV Community: Slava Kotenev</title>
      <link>https://dev.to/slava_kotenev_159</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/slava_kotenev_159"/>
    <language>en</language>
    <item>
      <title>Two weeks in one page: an AI adoption checklist for a small business</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Sat, 07 Nov 2026 05:19:01 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/two-weeks-in-one-page-an-ai-adoption-checklist-for-a-small-business-5d8m</link>
      <guid>https://dev.to/slava_kotenev_159/two-weeks-in-one-page-an-ai-adoption-checklist-for-a-small-business-5d8m</guid>
      <description>&lt;p&gt;Over the last two weeks I published a series on practical AI adoption: pilots, thresholds, documents, review queues, acceptance sets, data, costs. This is the whole thing on one page, in the order you would use it. It is written for a small business or a small team with no data science department.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you build anything
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Pick one process&lt;/strong&gt;, named in one sentence, with one owner who can say "accepted". Not "customer experience", but "incoming support emails" or "supplier invoices".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check whether you need AI at all.&lt;/strong&gt; If the rules are stable, the volume is small, or one error is very expensive, a script, a template or a lookup table is the better answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure the baseline.&lt;/strong&gt; Time a real sample of at least 100 items: minutes per item, error rate, monthly volume.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the target as a number&lt;/strong&gt; before any result exists: share handled without a person, accuracy on those, cost per item.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do the payback arithmetic on one page&lt;/strong&gt;: saving minus review, running and error cost, divided into the build cost. Run it again with pessimistic inputs.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Design the pilot
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Choose the cheapest rung that could hit the target&lt;/strong&gt;: rules, then classic ML, then a constrained LLM, and only then agents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide what goes to a person&lt;/strong&gt;: a confidence threshold plus hard rules, a named reviewer, a time limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size the review queue&lt;/strong&gt;: items per day times share to review times minutes per review. If it exceeds the people you have, change the scope.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Settle the data question&lt;/strong&gt; with whoever owns it: what is in the input, whether it may leave your systems, what the provider keeps, what lands in logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build a golden set&lt;/strong&gt; of about 100 real, labelled items covering typical, hard and must-refuse cases. Freeze it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the one-page brief&lt;/strong&gt;: scope, baseline, target, data, golden set, human path, timeline, cost, risks, and the decision rule.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xosg003g88piv4docoo.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xosg003g88piv4docoo.jpg" alt="Stacked checklist layers, top one highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Run and judge it
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Run for 2 to 6 weeks&lt;/strong&gt;, not months, with a checkpoint in the middle.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge on the golden set and on live traffic&lt;/strong&gt;, against the number you wrote down. Read every failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide by the rule you fixed in advance&lt;/strong&gt;: scale, change, or stop. Stopping is a valid, cheap outcome.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  After go-live
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Monitor&lt;/strong&gt; input shape, confidence distribution, share going to humans, and the override rate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit a random sample&lt;/strong&gt; of the automated path weekly, because nobody reviews it otherwise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rerun the golden set&lt;/strong&gt; on every change of model, prompt, threshold or data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch cost and latency&lt;/strong&gt; as quality signals, and give every alert an owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Review monthly&lt;/strong&gt; for half an hour: trends, the ten worst items, fresh examples for the golden set, the threshold.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  If you hire someone
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Ask how you will know it works, what happens when it is wrong, where your data goes, what it costs to run after launch, what is out of scope, and what happens if you stop after the pilot.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The shortest version
&lt;/h2&gt;

&lt;p&gt;Pick one process. Measure it. Write the target as a number. Use the simplest tool that can reach it. Keep a human behind the doubtful cases. Test on frozen examples. Watch it after launch. Decide by a rule you wrote before you started.&lt;/p&gt;

&lt;p&gt;All demos behind this series use public data and are open: &lt;a href="https://github.com/TerrKot/ai-ticket-triage-demo" rel="noopener noreferrer"&gt;ticket triage&lt;/a&gt;, &lt;a href="https://github.com/TerrKot/ai-invoice-extraction-demo" rel="noopener noreferrer"&gt;invoice extraction&lt;/a&gt;, &lt;a href="https://github.com/TerrKot/ai-document-qa-demo" rel="noopener noreferrer"&gt;document Q&amp;amp;A&lt;/a&gt;. Their numbers are test numbers, not client results. Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>smallbusiness</category>
      <category>automation</category>
      <category>checklist</category>
    </item>
    <item>
      <title>Eight questions to ask an AI contractor before you sign</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Fri, 06 Nov 2026 05:18:17 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/eight-questions-to-ask-an-ai-contractor-before-you-sign-2bdl</link>
      <guid>https://dev.to/slava_kotenev_159/eight-questions-to-ask-an-ai-contractor-before-you-sign-2bdl</guid>
      <description>&lt;p&gt;Hiring someone to build an AI feature is hard to judge from a portfolio, because demos look good by construction. The questions below separate people who have shipped measurable work from people who sell the word "AI". For each one I list what a good answer sounds like and what should worry you. I write this as someone who sells this kind of work, so treat it as a checklist you can apply to me as well.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. "How will we know it works?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a number, a test set and a method, agreed before building: "X% handled automatically at Y% accuracy on a labelled sample from your data".&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; "it will be very accurate", or accuracy quoted from a generic benchmark with no link to your data.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. "What happens when it is wrong or unsure?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a described human path: confidence threshold, review queue, fallback, who handles what and how fast.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; "the model rarely makes mistakes", or no plan for exceptions.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. "What do you need from us, and when?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a concrete list: sample data, a process owner, someone to label examples, access. With dates.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; nothing needed, because "AI will figure it out".&lt;/p&gt;

&lt;h2&gt;
  
  
  4. "Where will our data go?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a clear answer on which services see it, retention terms, whether personal data is redacted, and what is stored in logs. Willingness to do this in writing.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; vague reassurance, or no idea which provider terms apply.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. "What exactly will you deliver, and what is out of scope?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a short, written scope with explicit exclusions and acceptance criteria tied to question 1.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; an open-ended "AI solution", or a scope that grows without a change process.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flcixbkg9qqh4syita09a.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flcixbkg9qqh4syita09a.jpg" alt="Documents with one item highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  6. "What will it cost to run after launch?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; an estimate of per-item or monthly cost (API, hosting, monitoring) and who pays for tuning as data changes.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; only the build cost is discussed, or running cost is "negligible" with no numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. "Can I see something you built, and what did you measure?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a real example with the method and numbers, including the limits, or an open demo with its caveats stated. An honest "this was a public benchmark, not a client result" is a good sign.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; unnamed "Fortune 500 clients", screenshots without metrics, results that are suspiciously perfect.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. "What if we stop after the pilot?"
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Good:&lt;/strong&gt; a pilot with a decision rule agreed upfront (scale, change, stop), and you keep the code, the data and the documentation.&lt;br&gt;
&lt;strong&gt;Worry:&lt;/strong&gt; lock-in, an unclear ownership of the result, or pressure to commit to a long contract before any measured pilot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Small signals that count
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;They ask about your process before talking about models.&lt;/li&gt;
&lt;li&gt;They say what AI is &lt;strong&gt;not&lt;/strong&gt; good for in your case.&lt;/li&gt;
&lt;li&gt;They propose the simplest thing that could work.&lt;/li&gt;
&lt;li&gt;They are comfortable writing the acceptance number down.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you get a few of these right, you already know more than most buyers do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Insist on a measurable target and a method before work starts.&lt;/li&gt;
&lt;li&gt;Ask about errors, data, running cost and exit, not only features.&lt;/li&gt;
&lt;li&gt;Prefer evidence with limits over perfect-looking claims.&lt;/li&gt;
&lt;li&gt;Start with a short, measured pilot.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>business</category>
      <category>freelance</category>
      <category>management</category>
    </item>
    <item>
      <title>After go-live: what to monitor so quality doesn't silently decay</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Thu, 05 Nov 2026 05:18:38 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/after-go-live-what-to-monitor-so-quality-doesnt-silently-decay-2kc4</link>
      <guid>https://dev.to/slava_kotenev_159/after-go-live-what-to-monitor-so-quality-doesnt-silently-decay-2kc4</guid>
      <description>&lt;p&gt;Passing acceptance on launch day says the feature worked on the data you tested. It says nothing about next quarter. Inputs change, products change, a provider updates a model, a form gets a new field. Without monitoring, quality decays quietly and you hear about it from an angry customer. Here is the small set of signals I would put in place before go-live.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Volume and shape of inputs
&lt;/h2&gt;

&lt;p&gt;The cheapest early warning. Track items per day, average length, language mix, share of empty or malformed inputs, and the distribution of categories. A sudden shift (a new product launch, a changed template, a broken upstream feed) often precedes a quality drop.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;drift_alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;today&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.25&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;today&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mf"&gt;1e-9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  2. The confidence distribution
&lt;/h2&gt;

&lt;p&gt;If you route by a confidence threshold, watch where the scores sit. A shrinking share above the threshold means the model is less sure than at launch: more items go to people, and the cost model starts to slip. A growing share above it with no improvement elsewhere is also a flag, because over-confidence is how silent errors get through.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr9n5tsuoy76or4l1l58d.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr9n5tsuoy76or4l1l58d.jpg" alt="Stacked cubes, a few highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. The share going to humans
&lt;/h2&gt;

&lt;p&gt;The simplest business metric of the system. Track it daily with a weekly trend. It should be stable around the number you measured in the pilot. Alert on a move of more than a few points in either direction.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The override rate
&lt;/h2&gt;

&lt;p&gt;How often reviewers change the proposed answer. This is the most direct quality signal you get for free. Break it down by category: one weak category hides in an average.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Sampled audits
&lt;/h2&gt;

&lt;p&gt;Automatic metrics miss what nobody reviews. Each week, pull a small random sample (20 to 50 items) from the &lt;strong&gt;automated&lt;/strong&gt; path, the ones no person saw, and have someone check them against the correct answer. This is the only way to estimate the real error rate of what the system sends without supervision.&lt;/p&gt;

&lt;p&gt;Keep the sample random, record the result, and plot the trend.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. The golden set, rerun
&lt;/h2&gt;

&lt;p&gt;Rerun the frozen golden set on a schedule and on every change (model version, prompt, threshold, upstream data). A drop on the golden set with unchanged inputs points at the model or the prompt, not the data. Keep a version log of scores.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Operational health
&lt;/h2&gt;

&lt;p&gt;The boring part that causes most outages: latency, error rates, timeouts, queue age, cost per item, and API spend against a limit. Cost creep is quality decay's twin: a prompt change that doubles tokens is a regression too.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. The feedback channel
&lt;/h2&gt;

&lt;p&gt;Give users and reviewers an obvious way to flag a bad result, and read what comes in. It is rarely statistically useful, but it finds the failure types you did not think to measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  A monthly review
&lt;/h2&gt;

&lt;p&gt;Once a month, 30 minutes, same agenda:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Trend of the key metrics above, against the launch baseline.&lt;/li&gt;
&lt;li&gt;The ten worst items from the sample and the review queue.&lt;/li&gt;
&lt;li&gt;Add 10 to 20 fresh items to the golden set.&lt;/li&gt;
&lt;li&gt;Decide on the threshold: tune it on fresh labels, or leave it.&lt;/li&gt;
&lt;li&gt;Note provider or data changes since last time.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Write down who owns each alert and what they do when it fires. An alert with no owner is a log line.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Watch inputs, confidence, share to humans and override rate.&lt;/li&gt;
&lt;li&gt;Audit a random sample of the automated path every week.&lt;/li&gt;
&lt;li&gt;Rerun the golden set on every change.&lt;/li&gt;
&lt;li&gt;Track cost and latency as quality signals too.&lt;/li&gt;
&lt;li&gt;Give every alert an owner and a monthly review.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>monitoring</category>
      <category>mlops</category>
      <category>automation</category>
    </item>
    <item>
      <title>A one-page brief for an AI pilot (template)</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Wed, 04 Nov 2026 05:18:05 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/a-one-page-brief-for-an-ai-pilot-template-5ak1</link>
      <guid>https://dev.to/slava_kotenev_159/a-one-page-brief-for-an-ai-pilot-template-5ak1</guid>
      <description>&lt;p&gt;A pilot brief should fit on one page. If it needs ten, the scope is not yet small enough to pilot. Below is the template I would use, with a note on why each field is there. Copy it, fill it in a meeting, and refuse to start until every line has an answer, even if the answer is "unknown, to be measured in week 1".&lt;/p&gt;

&lt;h2&gt;
  
  
  The template
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI PILOT BRIEF

1. Process        One process, named in a sentence.
2. Owner          One person who can say "accepted".
3. Users          Who uses the output, how many.
4. Baseline       Volume per month; minutes per item; error rate today
                  (measured on a sample of N = ___ items, date ___).
5. Target         Number(s) that mean success, per metric, with the
                  measurement method.
6. Data           Source, format, who owns it, personal data? (Y/N),
                  can it go to an external service? (Y/N, who decided).
7. Golden set     Size, who labels it, where it lives, frozen on ___.
8. Human path     What goes to a person, who handles it, time limit.
9. Out of scope   What we are explicitly not doing.
10. Timeline      Start, checkpoint, decision date. Length: 2 to 6 weeks.
11. Cost          Pilot budget, expected running cost per month.
12. Risks         Top three and what we do about each.
13. Decision      What happens at the end: scale, change, or stop,
                  and the rule that picks between them.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Notes on the fields that get skipped
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1 and 9, scope.&lt;/strong&gt; The most common failure is a vague process ("customer experience"). Write one sentence naming the input and the output. Then write what you are &lt;em&gt;not&lt;/em&gt; doing; it prevents scope creep in week 3.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4, baseline.&lt;/strong&gt; If nobody has timed the manual work, put the measurement into week 1. Without it you will end the pilot unable to say whether anything improved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5, target.&lt;/strong&gt; Write the number before any result exists, and say how it is measured. Example: "at least 70% of items handled without a person, at least 97% accuracy on those, measured on the golden set and on a week of live traffic."&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0l3t4qezmifcc0ctryn5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0l3t4qezmifcc0ctryn5.jpg" alt="Stack of pages with one page highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6, data.&lt;/strong&gt; Two questions people avoid: is there personal data in it, and has someone with authority decided whether it may leave your systems? A name next to the decision is better than a hope.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8, human path.&lt;/strong&gt; The pilot is not "AI replaces the person". It is "AI handles what it is sure about, a person handles the rest". Say who and how fast, or the doubtful items will sit in a pile.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12, risks.&lt;/strong&gt; Typical top three: data quality worse than expected, reviewers overloaded, and the target set too high or too low. Each needs a response, such as "re-measure after week 1" or "lower the share automated".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;13, decision.&lt;/strong&gt; Fix the rule in advance: for example, "if the target is met on both the golden set and live traffic, scale to the next process; if met only on the golden set, extend two weeks; otherwise stop". It protects the team from arguing about a result after the fact.&lt;/p&gt;

&lt;h2&gt;
  
  
  A word on length and tone
&lt;/h2&gt;

&lt;p&gt;One page is a discipline, not a formatting preference. Short sentences, numbers where possible, plain words. The audience includes the owner, the person who pays and the person who will review the output every day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Use one page and refuse to start with blanks.&lt;/li&gt;
&lt;li&gt;Measure the baseline and write the target as numbers first.&lt;/li&gt;
&lt;li&gt;Name the owner, the human path and the stop rule.&lt;/li&gt;
&lt;li&gt;Fix the decision rule before you see results.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>management</category>
      <category>template</category>
    </item>
    <item>
      <title>Rules, classic ML or an LLM: a decision ladder</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Tue, 03 Nov 2026 05:17:50 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/rules-classic-ml-or-an-llm-a-decision-ladder-34mp</link>
      <guid>https://dev.to/slava_kotenev_159/rules-classic-ml-or-an-llm-a-decision-ladder-34mp</guid>
      <description>&lt;p&gt;"Should we use an LLM for this?" is usually asked as if there were two options: do it by hand, or call a large model. There are at least four, and they differ in cost, speed, and how well you can test them. The simplest way to choose is a ladder: start at the bottom, and climb only when the rung below fails a target you wrote down in advance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rung 0: do not automate yet
&lt;/h2&gt;

&lt;p&gt;If the process is rare, unstable or poorly understood, the right move is to document it and measure the manual baseline. Automation of a process nobody can describe produces a fast version of confusion.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rung 1: rules and lookups
&lt;/h2&gt;

&lt;p&gt;Regexes, validation, tables, thresholds, if-statements.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pick it when:&lt;/strong&gt; the decision can be explained in a few lines and the inputs have a stable shape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You get:&lt;/strong&gt; exact repeatability, zero per-call cost, a one-line answer to "why did it do that?".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It fails when:&lt;/strong&gt; the input is free text with many phrasings, or the rules multiply faster than anyone can maintain them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Rung 2: classic machine learning
&lt;/h2&gt;

&lt;p&gt;TF-IDF plus logistic regression, gradient boosting, small classifiers trained on your labelled examples.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pick it when:&lt;/strong&gt; you have (or can get) a few hundred to thousands of labelled examples and the output is a category, a number or a ranking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You get:&lt;/strong&gt; fast, cheap, runs on a CPU, calibrated probabilities you can threshold, easy to evaluate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It fails when:&lt;/strong&gt; you have no labels, the answer is a free-form sentence, or the task needs broad world knowledge.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In my public demos, a plain TF-IDF model with a calibrated threshold handled about 76% of support tickets automatically at 97.7% accuracy on a held-out set (public Banking77 data, test numbers). It was enough because the task is classification into 77 known intents.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft36cuqoz5n0aojlrx9ax.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft36cuqoz5n0aojlrx9ax.jpg" alt="Rungs of a stack, one lit" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Rung 3: an LLM, constrained
&lt;/h2&gt;

&lt;p&gt;A language model with a tight prompt, a fixed output schema, validation of the result, and a fallback to a human.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pick it when:&lt;/strong&gt; the input is varied text, labels are scarce, or the output must be generated or extracted flexibly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You get:&lt;/strong&gt; wide coverage with little training data, flexible handling of new phrasings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You pay with:&lt;/strong&gt; per-call cost, latency, non-determinism, provider terms and data handling, and a harder testing job.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make it testable:&lt;/strong&gt; structured output, validators (checksums, ranges, allowed values), a golden set, and a confidence or consistency signal that routes doubtful items to a person.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Rung 4: an LLM with tools or an agent
&lt;/h2&gt;

&lt;p&gt;Multi-step behaviour, calling APIs, taking actions. Climb here last, and only with a narrow scope, permissions you can audit, and a human approving anything irreversible.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question at every rung
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;What is the target, as a number?&lt;/li&gt;
&lt;li&gt;What does the cheapest rung that could reach it score on a golden set?&lt;/li&gt;
&lt;li&gt;If it falls short, &lt;strong&gt;why&lt;/strong&gt;? Add the missing capability, not the biggest model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A useful habit is to run the lower rung as a baseline even when you plan an LLM. If a regex plus a classifier scores within a point or two, you have saved a lot of money and risk. If it scores far below, you now have a number that justifies the step up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Combine them
&lt;/h2&gt;

&lt;p&gt;The best systems are often layered: rules for the easy and the forbidden cases, a classifier for the bulk, an LLM for the long tail, and a human queue behind all three.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Treat the choice as a ladder, not a binary.&lt;/li&gt;
&lt;li&gt;Write the target first, then climb only when a rung misses it.&lt;/li&gt;
&lt;li&gt;Always keep the lower rung as a baseline.&lt;/li&gt;
&lt;li&gt;Layer the rungs rather than betting everything on one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Personal data and third-party LLM APIs: a checklist before the first request</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Mon, 02 Nov 2026 05:18:28 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/personal-data-and-third-party-llm-apis-a-checklist-before-the-first-request-gpf</link>
      <guid>https://dev.to/slava_kotenev_159/personal-data-and-third-party-llm-apis-a-checklist-before-the-first-request-gpf</guid>
      <description>&lt;p&gt;The first request to an external language-model API is the moment data leaves your control. Most teams make that decision implicitly, in a prototype, and discover later that the prototype was sending names, emails and account numbers to a third party. Here is the checklist I would walk through before the first request. It is an engineering checklist, &lt;strong&gt;not legal advice&lt;/strong&gt;: the final answers depend on your jurisdiction, your contracts and your lawyer.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What data is actually in the input?
&lt;/h2&gt;

&lt;p&gt;Take 50 real inputs and look. Typical surprises: names in signatures, phone numbers in free text, order numbers that identify a person, attachments with scanned IDs. If you cannot say what is in the input, you cannot say what you are sending.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Do you need it to do the job?
&lt;/h2&gt;

&lt;p&gt;Often the model needs the &lt;em&gt;question&lt;/em&gt;, not the identity. Strip or replace what the task does not use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="n"&gt;EMAIL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[\w.+-]+@[\w-]+\.[\w.-]+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;PHONE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\+?\d[\d\s().-]{7,}\d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;PHONE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[PHONE]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;EMAIL&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[EMAIL]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Regexes catch the easy cases only. Names and addresses need a proper detector, and you still have to test what slips through on your own data.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2w3l5uj99ozzzh6u0jcq.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2w3l5uj99ozzzh6u0jcq.jpg" alt="Layers of a stack with a control point highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. What do the provider's terms say?
&lt;/h2&gt;

&lt;p&gt;Read, with the person who owns compliance:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Retention:&lt;/strong&gt; is the input stored, for how long, can it be switched off?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Training:&lt;/strong&gt; is your data used to train models by default? Business and API tiers often differ from consumer products.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sub-processors and location:&lt;/strong&gt; where is it processed?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deletion and audit:&lt;/strong&gt; can you request deletion, and is there a data processing agreement?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Write the answers down with the date. Terms change.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Is there a lawful basis and a notice?
&lt;/h2&gt;

&lt;p&gt;If the data is about people, someone has to decide on the legal basis and whether your privacy notice already covers this use. That is a decision for the data owner, not for the developer, but the developer should ask for it before the launch, not after.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Do you need the data to leave at all?
&lt;/h2&gt;

&lt;p&gt;Compare honestly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A smaller model or classic ML on your own server (no external call).&lt;/li&gt;
&lt;li&gt;A provider with a region and retention setup that fits.&lt;/li&gt;
&lt;li&gt;Redaction plus an external model.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cheapest compliant option is the one where the sensitive part never goes out.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Logs, caches and prompts are data too
&lt;/h2&gt;

&lt;p&gt;Check where else the text lands: application logs, error trackers, analytics, prompt caches, evaluation sets shared in chat. A perfect API policy does not help if the raw text is in your logging stack forever.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Secrets and access
&lt;/h2&gt;

&lt;p&gt;API keys in environment variables, not in the repo or chat. Separate keys per environment, a spend limit, and a way to rotate them. Limit who can read the raw logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Plan for the incident
&lt;/h2&gt;

&lt;p&gt;Decide in advance: who is told, how fast, and how you would find out which requests contained what. That is only possible if you log request metadata (not the raw sensitive content).&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Look at real inputs before deciding anything.&lt;/li&gt;
&lt;li&gt;Send the minimum, scrub the rest, and test the scrubber.&lt;/li&gt;
&lt;li&gt;Read retention, training and location terms, and date your notes.&lt;/li&gt;
&lt;li&gt;Get the legal basis decided by the data owner.&lt;/li&gt;
&lt;li&gt;Treat logs and caches as part of the same data flow.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>privacy</category>
      <category>security</category>
      <category>llm</category>
    </item>
    <item>
      <title>The golden set: 100 labelled examples as the acceptance test for an AI feature</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Sun, 01 Nov 2026 05:17:44 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/the-golden-set-100-labelled-examples-as-the-acceptance-test-for-an-ai-feature-k5m</link>
      <guid>https://dev.to/slava_kotenev_159/the-golden-set-100-labelled-examples-as-the-acceptance-test-for-an-ai-feature-k5m</guid>
      <description>&lt;p&gt;If you ask three people whether an AI feature "works", you will get three answers, each based on the last few examples they happened to see. A golden set replaces opinions with a number: a fixed collection of real inputs with agreed correct answers, run every time something changes. About a hundred items is enough to start.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a hundred
&lt;/h2&gt;

&lt;p&gt;With 100 items, one mistake moves the score by one point, and you can still read every failure by hand. That is the point of the first set: not statistical perfection, but a measurement the whole team can inspect. As the feature matures you grow it; you do not start with thousands.&lt;/p&gt;

&lt;p&gt;A caution on precision: with 100 items the uncertainty is wide. A 95% score can plausibly hide a true rate anywhere from roughly 89% to 98%. Treat it as a tripwire, not a guarantee, and say so in the acceptance document.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes into it
&lt;/h2&gt;

&lt;p&gt;Sample from &lt;strong&gt;real&lt;/strong&gt; data from the process, not from examples someone invented. Then deliberately cover:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The typical case&lt;/strong&gt;, in proportion to how common it is (most of the set).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The boring variations&lt;/strong&gt;: different phrasings, formats, languages, typos.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Known hard cases&lt;/strong&gt;: the pairs people confuse, the odd layouts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Things that must be refused or escalated&lt;/strong&gt;: out-of-scope questions, missing fields, items the system should hand to a human.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A few adversarial ones&lt;/strong&gt;: instructions hidden in the input, empty input, absurdly long input.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tag each item with its category so you can report per-category results, not just one average that hides a weak spot.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzzkbqgdlubstzisofw1p.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzzkbqgdlubstzisofw1p.jpg" alt="Stacked sheets with one highlighted row" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Who labels it
&lt;/h2&gt;

&lt;p&gt;The process owner, or someone they trust, decides the correct answer. Two people labelling independently, then discussing disagreements, finds the ambiguous items early. Ambiguity found at this stage is a requirements problem, and it is cheaper to fix now than after launch.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to use it
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Freeze it.&lt;/strong&gt; The set is not training data. Never tune on it after you start reporting from it, or the score only reflects what you memorised.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run it on every change&lt;/strong&gt;: a new prompt, a new model version, a new threshold.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the pass bar before the first run&lt;/strong&gt;, as a number per category. Example: "at least 95% correct on automated items, and every must-refuse item refused".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read the failures&lt;/strong&gt;, all of them. Each one is either a bug, a labelling error, or a missing rule.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep a changelog&lt;/strong&gt; of scores per version so regressions are visible.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A minimal runner is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;system&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Refresh it slowly
&lt;/h2&gt;

&lt;p&gt;Every month or so, add 10 to 20 fresh real items, especially ones that came out of the review queue as corrections. Retire items that no longer reflect the process. Keep a separate untouched holdout if you tune a lot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Start with about 100 real, labelled items.&lt;/li&gt;
&lt;li&gt;Cover typical, hard, and must-refuse cases and tag them.&lt;/li&gt;
&lt;li&gt;Freeze it, write the pass bar first, run it on every change.&lt;/li&gt;
&lt;li&gt;Be honest about the uncertainty that comes with a small set.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>machinelearning</category>
      <category>productivity</category>
    </item>
    <item>
      <title>My demo refused 96% of off-topic questions but only 39% of tricky ones. I know because of a test set</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Sun, 01 Nov 2026 04:43:56 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/my-demo-refused-96-of-off-topic-questions-but-only-39-of-tricky-ones-i-know-because-of-a-test-set-lk2</link>
      <guid>https://dev.to/slava_kotenev_159/my-demo-refused-96-of-off-topic-questions-but-only-39-of-tricky-ones-i-know-because-of-a-test-set-lk2</guid>
      <description>&lt;p&gt;Here are two numbers from my document Q&amp;amp;A demo. Ask it about something that is not in the knowledge base at all, and it refuses correctly 96.1% of the time. Ask it a question that sits close to a document but is not answered by it, and it refuses only 39% of the time.&lt;/p&gt;

&lt;p&gt;If I had tried the demo by hand, I would have asked a few questions, seen it say "not in the docs" a couple of times, and called it good. The 39% only showed up because there was a labelled test set. That set is the subject of this post: a &lt;strong&gt;golden set&lt;/strong&gt;, the cheapest acceptance test an AI feature can have.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it is
&lt;/h2&gt;

&lt;p&gt;A golden set is a list of real inputs with the answer a human accepts for each one. Nothing more. It is not training data and it is not a benchmark you publish. It is the thing you run after every change to ask one question: &lt;strong&gt;did it get better or worse?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Without it, every change is judged by whichever three examples someone happens to try. With it, "works pretty well" becomes a number you can argue about.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvrz853pcajjo4y2risob.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvrz853pcajjo4y2risob.jpg" alt="Labelled examples checked against the system" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How I would build one in an afternoon
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Take real inputs.&lt;/strong&gt; 100 is enough to start. Pull them from the actual process: tickets, invoices, questions people really ask. Not examples you invented to look nice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cover the ugly cases on purpose.&lt;/strong&gt; Roughly: most of it typical, a quarter hard or ambiguous, and a handful that must be refused or sent to a human. The ugly part is where the demo above found its 39%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Label it with the person who owns the process.&lt;/strong&gt; The owner says what "correct" means. If two people disagree on a label, that disagreement is a finding, write it down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Freeze it.&lt;/strong&gt; Do not tune the system on these examples. Tune on a separate split and touch the golden set only to report. In my ticket-triage demo the threshold is chosen on a validation split and the test set is used once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the acceptance rule before you run it.&lt;/strong&gt; For example: "at least 97% correct on what is handled automatically, everything else goes to a person." Then the run is a pass or a fail, not an opinion.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What it showed me in my own demos
&lt;/h2&gt;

&lt;p&gt;These are public demos on public datasets (test data, not client results):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ticket triage.&lt;/strong&gt; If everything were automated, accuracy would be 90.2%. Handle only the confident 75.9% and accuracy on those is 97.7%. The golden set is what made the trade-off visible.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Document Q&amp;amp;A.&lt;/strong&gt; Out-of-base questions are refused 96.1% of the time, but near-miss questions only 39.4%. The cited answers hit 93% precision against a 95% target. I reported the gap instead of hiding it, and it points to the next fix: a reranker or an LLM for exactly those near-miss cases.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In both cases the number told me where to work next. A hand-tried demo would have told me nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Leakage.&lt;/strong&gt; If the system has seen the golden examples while being tuned, the score is fiction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Too clean.&lt;/strong&gt; A set made only of easy cases gives 99% and a false sense of safety.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never refreshed.&lt;/strong&gt; Real inputs drift. Add new failures to the set whenever production surfaces one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One number for everything.&lt;/strong&gt; Report per category. A fine average can hide a category that fails every time.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;A golden set is real inputs plus answers a human accepts.&lt;/li&gt;
&lt;li&gt;Build it before the pilot, with the process owner.&lt;/li&gt;
&lt;li&gt;Include the ugly cases deliberately.&lt;/li&gt;
&lt;li&gt;Freeze it, and write the pass rule first.&lt;/li&gt;
&lt;li&gt;Report per category, and add every new production failure to it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you can only do one thing before your next AI pilot, label 100 examples. It is the cheapest hour of the project.&lt;/p&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>automation</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Human-in-the-loop: designing the review queue</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Sat, 31 Oct 2026 05:18:35 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/human-in-the-loop-designing-the-review-queue-43m4</link>
      <guid>https://dev.to/slava_kotenev_159/human-in-the-loop-designing-the-review-queue-43m4</guid>
      <description>&lt;p&gt;"Doubtful cases go to a human" is easy to say and easy to get wrong. If the review queue is slow, confusing or endless, people stop trusting it, start rubber-stamping, and the accuracy you measured in the pilot quietly disappears. Here is how I would design the queue so it holds up.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes into the queue
&lt;/h2&gt;

&lt;p&gt;Two kinds of items belong there:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Low confidence.&lt;/strong&gt; Anything below the threshold you agreed on. In my &lt;a href="https://github.com/TerrKot/ai-ticket-triage-demo" rel="noopener noreferrer"&gt;ticket-triage demo&lt;/a&gt; (public data, test numbers) about 24% of tickets fall below the line, and that share is a design input, not a surprise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hard rules.&lt;/strong&gt; Items that must always be checked regardless of confidence: amounts above a limit, a failed checksum, a customer flagged as sensitive.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Keep the two reasons separate in the record. They have different fixes later.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnz8n5ypfud7oygv4be0x.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnz8n5ypfud7oygv4be0x.jpg" alt="Gate splitting items into passed and held" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Size the queue before you build it
&lt;/h2&gt;

&lt;p&gt;A queue is a workload, so do the arithmetic early:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;items_per_day × share_to_review × minutes_per_review / 60 = reviewer_hours_per_day
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that number is bigger than the people available, either the threshold or the scope is wrong. Better to find out on paper.&lt;/p&gt;

&lt;h2&gt;
  
  
  Show the reviewer what they need in one screen
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The item itself, and the system's proposed answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why&lt;/strong&gt; it is in the queue: low confidence, which rule fired.&lt;/li&gt;
&lt;li&gt;The evidence: the cited passage, the extracted field highlighted in the source, the runner-up answer.&lt;/li&gt;
&lt;li&gt;One-click actions: accept, correct, reject. Correcting should take fewer clicks than rewriting.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the reviewer has to open three other systems to decide, the average review time doubles and your cost model breaks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Order and age
&lt;/h2&gt;

&lt;p&gt;Don't sort by arrival only. Sort by deadline and by risk. Show the age of each item and alert when anything sits longer than your service level. An unattended queue is an outage in slow motion.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the queue teach the system
&lt;/h2&gt;

&lt;p&gt;Every correction is a labelled example. Store the original proposal, the final answer, who changed it and when. Then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Re-tune the threshold monthly on the fresh labels.&lt;/li&gt;
&lt;li&gt;Look at the top confusions (the pairs the model mixes up) and fix those first.&lt;/li&gt;
&lt;li&gt;Track the &lt;strong&gt;override rate&lt;/strong&gt;: how often reviewers change the proposal. A rising rate is an early warning that quality is drifting.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Guard against rubber-stamping
&lt;/h2&gt;

&lt;p&gt;If reviewers accept 99.5% of items in two seconds each, they are not reviewing. Seed a small share of known-answer items into the queue and measure whether they are caught. It is an honest check on the human half of the system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Define why an item enters the queue, and record the reason.&lt;/li&gt;
&lt;li&gt;Compute reviewer hours before building.&lt;/li&gt;
&lt;li&gt;Give the reviewer evidence and one-click actions on one screen.&lt;/li&gt;
&lt;li&gt;Treat every correction as training data and track the override rate.&lt;/li&gt;
&lt;li&gt;Test the humans too, with seeded items.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>ux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Back-of-envelope ROI for automation: a formula and a worked example</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Fri, 30 Oct 2026 05:18:21 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/back-of-envelope-roi-for-automation-a-formula-and-a-worked-example-25b</link>
      <guid>https://dev.to/slava_kotenev_159/back-of-envelope-roi-for-automation-a-formula-and-a-worked-example-25b</guid>
      <description>&lt;p&gt;Most automation projects get approved on a feeling: "it will save a lot of time". A one-page calculation does better, and it takes ten minutes. Below is the formula I would use, a worked example, and the inputs that most often make it wrong.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;All numbers in the example are illustrative, invented for the arithmetic. They are not a client result.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The formula
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;monthly_saving = volume × share_automated × minutes_saved_per_item / 60 × hourly_cost
monthly_cost   = running_cost + review_cost + error_cost
net_monthly    = monthly_saving − monthly_cost
payback_months = build_cost / net_monthly
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four of these terms hide the real work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;share_automated&lt;/strong&gt; is not a wish, it is a measured number from a pilot (for example, the share of items that clear a confidence threshold at an agreed accuracy).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;review_cost&lt;/strong&gt; is the time people still spend checking what the system hands back. Forget it and the saving is overstated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;error_cost&lt;/strong&gt; is (wrong automated items per month) × (cost of fixing one). It is the term that punishes a threshold set too low.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;minutes_saved_per_item&lt;/strong&gt; comes from timing the manual process on a real sample, not from a guess.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A worked example (illustrative numbers)
&lt;/h2&gt;

&lt;p&gt;Take a process with 2,000 items a month, 6 minutes each by hand, at a loaded cost of $25 per hour.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Manual cost today: 2,000 × 6 / 60 × 25 = &lt;strong&gt;$5,000 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Suppose a pilot shows 75% of items can be automated at the agreed accuracy, and each automated item still costs 0.5 minutes of spot-check.&lt;/li&gt;
&lt;li&gt;Saving: 2,000 × 0.75 × (6 − 0.5) / 60 × 25 = &lt;strong&gt;$3,437 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Running cost (API, hosting, monitoring time): &lt;strong&gt;$400 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Errors: 1.5% of 1,500 automated items = 22 wrong, at $15 each to fix = &lt;strong&gt;$338 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Net: 3,437 − 400 − 338 = &lt;strong&gt;about $2,700 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;If the build costs $8,000, payback is &lt;strong&gt;about 3 months&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fng3sgy5xp719eil9fij3.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fng3sgy5xp719eil9fij3.jpg" alt="Stacked layers: saving, running cost, errors" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What breaks the calculation
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Volume that is smaller than it feels.&lt;/strong&gt; Count last quarter's real items. A process that "never stops" is often 200 items a month.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A baseline nobody measured.&lt;/strong&gt; If you don't know the 6 minutes, the whole saving is a story.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ignoring the exception path.&lt;/strong&gt; The 25% that goes to humans is usually the hard 25%, and it may take longer than the average item did before.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating the build as the only cost.&lt;/strong&gt; Monitoring and re-tuning continue after go-live.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Counting saved hours as saved money.&lt;/strong&gt; If nobody's payroll changes, the real benefit is capacity, speed or fewer errors. Say which one you mean.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Sensitivity beats a single answer
&lt;/h2&gt;

&lt;p&gt;Re-run the sum with automated share at 55% and error cost doubled. If the project still pays back within your tolerance, it is robust. If it only works at your best case, you have learned that for the price of a spreadsheet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;Put volume, baseline, measured automation share, review time and error cost on one page before any build starts. If the numbers don't clear your payback threshold, the right answer is "not this process", and that is a good outcome for ten minutes of arithmetic.&lt;/p&gt;

&lt;p&gt;Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>productivity</category>
      <category>business</category>
    </item>
    <item>
      <title>Q&amp;A over internal documents: make the system say "not in the docs"</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Thu, 29 Oct 2026 05:17:57 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/qa-over-internal-documents-make-the-system-say-not-in-the-docs-22oh</link>
      <guid>https://dev.to/slava_kotenev_159/qa-over-internal-documents-make-the-system-say-not-in-the-docs-22oh</guid>
      <description>&lt;p&gt;An assistant that answers questions about your documents will be wrong sometimes. What destroys trust is not the wrong answer, it is the &lt;em&gt;confident&lt;/em&gt; wrong answer to a question the documents never covered. So the first design question is not "how good is retrieval?" but "when does the system refuse?"&lt;/p&gt;

&lt;p&gt;Here is a small demo that treats refusal as a first-class result. The data is public and the numbers are test numbers, not a client result.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the demo does
&lt;/h2&gt;

&lt;p&gt;One function, &lt;code&gt;KB.answer(question)&lt;/code&gt;, returns three things: the best passage as the citation, a confidence that it really holds the answer, and a decision: cite it, or refuse.&lt;/p&gt;

&lt;p&gt;It deliberately returns the passage, not a generated sentence. Generating prose on top of a cited passage is the step that needs a language model; the part that decides whether to answer at all does not.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data:&lt;/strong&gt; SQuAD 2.0 dev (Wikipedia, CC BY-SA 4.0), 35 articles. Wikipedia stands in for a company's documentation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrieval:&lt;/strong&gt; BM25.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decision:&lt;/strong&gt; a small model over retrieval features: the top score, its margin over the runner-up, IDF-weighted coverage of the question's terms, and the share of terms the passage doesn't know.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Threshold:&lt;/strong&gt; tuned on separate articles for a 95% precision target.&lt;/li&gt;
&lt;li&gt;No LLM, no GPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Evaluating without leaking
&lt;/h2&gt;

&lt;p&gt;Articles are split into a tuning set and a test set. A third of the test articles are deliberately left out of the knowledge base, and their questions must be refused. Without that group you only measure how well the system answers questions it was built for, which is the easy case.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0bf0ovte1husywpcsgu8.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0bf0ovte1husywpcsgu8.jpg" alt="Layers of documents with one passage highlighted" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Results (test KB: 406 passages, 4,904 questions)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Question about a topic NOT in the base&lt;/td&gt;
&lt;td&gt;refused correctly in 96.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Answerable question: right passage in top-3&lt;/td&gt;
&lt;td&gt;87.8% (top-1: 75.3%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Answerable question: cited with confidence&lt;/td&gt;
&lt;td&gt;61.2% of questions, 93.2% of those cited point to the right passage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Answerable plus out-of-base, overall&lt;/td&gt;
&lt;td&gt;42.5% cited, 90.4% of cited are correct&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The refusal rate on out-of-base questions is the number I would put in an acceptance document. It says the system stays quiet when it has nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The target was 95% precision. On the test articles the cited answers reached 93% (answerable only) and 90% (with out-of-base questions mixed in). The threshold is a sampled estimate, and a pilot re-tunes it on the client's own documents and questions.&lt;/li&gt;
&lt;li&gt;Questions that are &lt;em&gt;close&lt;/em&gt; to a document but not answered by it (SQuAD's "unanswerable" ones) are refused only 39% of the time. Lexical signals cannot see that difference. This is where a reranker or an LLM belongs in a real build, and I report the number instead of hiding it.&lt;/li&gt;
&lt;li&gt;Wikipedia is not your wiki. Expect to measure again on real material.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Give refusal its own metric and its own test set.&lt;/li&gt;
&lt;li&gt;Always include questions about things that are not in the base.&lt;/li&gt;
&lt;li&gt;Return the citation, so a person can verify in one click.&lt;/li&gt;
&lt;li&gt;Report the failure mode you have not solved.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Code and numbers: &lt;a href="https://github.com/TerrKot/ai-document-qa-demo" rel="noopener noreferrer"&gt;ai-document-qa-demo&lt;/a&gt;. Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>rag</category>
      <category>search</category>
      <category>python</category>
    </item>
    <item>
      <title>Extracting invoice fields with validation: what auto-posts and what goes to a human</title>
      <dc:creator>Slava Kotenev</dc:creator>
      <pubDate>Wed, 28 Oct 2026 05:17:58 +0000</pubDate>
      <link>https://dev.to/slava_kotenev_159/extracting-invoice-fields-with-validation-what-auto-posts-and-what-goes-to-a-human-512b</link>
      <guid>https://dev.to/slava_kotenev_159/extracting-invoice-fields-with-validation-what-auto-posts-and-what-goes-to-a-human-512b</guid>
      <description>&lt;p&gt;Pulling an invoice number or a total out of text is the easy half of the job. The useful half is deciding, for every field of every document, whether it goes straight into the accounting system or lands on a person's desk. Here is how a small demo handles it. Everything below is test data from a public synthetic dataset, not a client result.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the demo does
&lt;/h2&gt;

&lt;p&gt;Input: the OCR text of an invoice. Output: invoice number, date, IBAN and gross total, each with a confidence. Fields with low confidence are routed to a human instead of being posted.&lt;/p&gt;

&lt;p&gt;The data is &lt;code&gt;mychen76/invoices-and-receipts_ocr_v1&lt;/code&gt;: OCR text plus ground-truth fields. The invoices are synthetic with a fixed layout, and only the invoice-type documents are used (484 of about 2,200; the rest are receipts with a different schema).&lt;/p&gt;

&lt;h2&gt;
  
  
  Candidates first, then a ranker
&lt;/h2&gt;

&lt;p&gt;For each field the code generates candidates with patterns, then a small logistic regression picks the right one using context: the words around the candidate, its position, its format. This matters most for the total, because an invoice contains many amounts (net, tax, line items, gross) and "the biggest number" is not the answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Validation beats confidence for IBANs
&lt;/h2&gt;

&lt;p&gt;For IBANs the demo does not trust a score at all. It validates the ISO 13616 checksum. If OCR confused one character (S for 5, O for 0, a dropped or doubled character) and exactly one repair makes the checksum pass, it applies the repair. If no unique valid value exists, the field goes to a human.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;iban_ok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;upper&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;rearranged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;digits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;36&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rearranged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;digits&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;97&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A checksum is a hard gate a model cannot talk its way past.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fexc952c07z7e3rbr4bd7.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fexc952c07z7e3rbr4bd7.jpg" alt="Stack of documents with one field highlighted for review" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Results on 74 held-out documents (test data)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Docs scored&lt;/th&gt;
&lt;th&gt;Accuracy when automated&lt;/th&gt;
&lt;th&gt;Automated share&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Invoice number&lt;/td&gt;
&lt;td&gt;74&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;97.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invoice date&lt;/td&gt;
&lt;td&gt;74&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IBAN&lt;/td&gt;
&lt;td&gt;56&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gross total&lt;/td&gt;
&lt;td&gt;74&lt;/td&gt;
&lt;td&gt;98.6%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;96.4% of clean documents were fully automated (all four fields), and 98.2% of those had every field correct. Thresholds were tuned on a validation split for a 98% target; the held-out set was used for reporting only.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;74 documents with a fixed synthetic layout is a small, friendly test. Real, varied invoices will score lower, and a pilot starts by measuring on the client's own sample.&lt;/li&gt;
&lt;li&gt;In 18 of the 74 documents the dataset's IBAN label itself fails the checksum while the OCR text passes it, so the label is corrupted. Those are excluded from the IBAN score and counted separately in &lt;code&gt;results.json&lt;/code&gt;. Check your ground truth before blaming the model.&lt;/li&gt;
&lt;li&gt;The input is OCR text. Turning scans and PDFs into that text is a separate, replaceable step.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Decide per field what is allowed to auto-post, with a number as the bar.&lt;/li&gt;
&lt;li&gt;Use deterministic validation (checksums, date ranges, sum checks) wherever it exists.&lt;/li&gt;
&lt;li&gt;Route what you cannot validate to a human, and make that route cheap.&lt;/li&gt;
&lt;li&gt;Audit your labels before you measure.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Code and numbers: &lt;a href="https://github.com/TerrKot/ai-invoice-extraction-demo" rel="noopener noreferrer"&gt;ai-invoice-extraction-demo&lt;/a&gt;. Portfolio and contacts: &lt;a href="https://t.me/Kotenev01_bot" rel="noopener noreferrer"&gt;@Kotenev01_bot&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: &lt;a href="https://t.me/kotenev01_en" rel="noopener noreferrer"&gt;@kotenev01_en&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>python</category>
      <category>finance</category>
    </item>
  </channel>
</rss>
