<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Harpreet Sahota</title>
    <description>The latest articles on DEV Community by Harpreet Sahota (@harpreetsahota).</description>
    <link>https://dev.to/harpreetsahota</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4213631%2Fae3b78ab-aed7-477c-bb84-5c293c70b9e9.png</url>
      <title>DEV Community: Harpreet Sahota</title>
      <link>https://dev.to/harpreetsahota</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/harpreetsahota"/>
    <language>en</language>
    <item>
      <title>Developer Content That Doesn't Feel Like Marketing</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Thu, 19 Nov 2026 14:00:33 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/developer-content-that-doesnt-feel-like-marketing-4b2e</link>
      <guid>https://dev.to/harpreetsahota/developer-content-that-doesnt-feel-like-marketing-4b2e</guid>
      <description>&lt;h1&gt;
  
  
  Developer Content That Doesn't Feel Like Marketing
&lt;/h1&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Developers do not reject marketing because they hate products. They reject it because it hides the work.&lt;/li&gt;
&lt;li&gt;Good developer content helps someone build, evaluate, debug, compare, or decide.&lt;/li&gt;
&lt;li&gt;The best technical content proves credibility by being useful before it asks for anything.&lt;/li&gt;
&lt;li&gt;If the post cannot survive without adjectives, social proof, or a vague promise, it is probably not ready.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;Developer content fails when it borrows the surface of technical writing but keeps the incentives of a campaign. It says "seamless," "powerful," and "production-ready," then gives the reader no working path, no tradeoffs, and no way to decide whether the product is worth their time.&lt;/p&gt;

&lt;p&gt;That is not a writing problem first. It is a trust problem.&lt;/p&gt;

&lt;p&gt;This post gives a practical way to plan developer content that does not feel like marketing. We will define what the reader is actually trying to do, where product claims turn into noise, which content formats earn attention, and how to build a lightweight content system that helps DevRel, product, engineering, and sales without sounding like any of them wrote it by committee.&lt;/p&gt;

&lt;h2&gt;
  
  
  Table of Contents
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Problem&lt;/li&gt;
&lt;li&gt;What Developer Content Is For&lt;/li&gt;
&lt;li&gt;Why Marketing Language Breaks Trust&lt;/li&gt;
&lt;li&gt;The Six Tests&lt;/li&gt;
&lt;li&gt;Formats That Earn Attention&lt;/li&gt;
&lt;li&gt;A Small Content System&lt;/li&gt;
&lt;li&gt;Measurement&lt;/li&gt;
&lt;li&gt;Summary&lt;/li&gt;
&lt;li&gt;Next Steps&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Problem
&lt;/h2&gt;

&lt;p&gt;Most developer content starts with the wrong question:&lt;/p&gt;

&lt;p&gt;"What do we want to say about the product?"&lt;/p&gt;

&lt;p&gt;That question almost always produces weak work. It points the writer toward features, launch messages, positioning phrases, and internal excitement. The result may be technically adjacent, but it is still a pitch. Developers can usually feel that within the first few paragraphs.&lt;/p&gt;

&lt;p&gt;The better question is:&lt;/p&gt;

&lt;p&gt;"What is the developer trying to get done?"&lt;/p&gt;

&lt;p&gt;That question changes the shape of the piece. A developer might be trying to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;decide whether a tool belongs in their stack&lt;/li&gt;
&lt;li&gt;understand a new model, API, SDK, or framework&lt;/li&gt;
&lt;li&gt;compare one approach against another&lt;/li&gt;
&lt;li&gt;get from a blank project to a working example&lt;/li&gt;
&lt;li&gt;debug an error that blocks adoption&lt;/li&gt;
&lt;li&gt;estimate cost, latency, risk, or maintenance burden&lt;/li&gt;
&lt;li&gt;explain a technical decision to the rest of their team&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those are real jobs. Content that helps with them does not need to pretend to be neutral, but it does need to be useful before it is persuasive.&lt;/p&gt;

&lt;p&gt;That distinction matters for AI and devtool companies because the product is often not self-explanatory. The buyer may be technical. The user may be more technical. The evaluator may be skeptical. A polished launch post is not enough. They need evidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Developer Content Is For
&lt;/h2&gt;

&lt;p&gt;Developer content has one job: reduce the distance between curiosity and informed action.&lt;/p&gt;

&lt;p&gt;That action can be small. It might be cloning a repo, running a notebook, joining a waitlist, opening the docs, comparing an architecture, or sending the link to a teammate. But the post should move the reader from a fuzzy state to a sharper one.&lt;/p&gt;

&lt;p&gt;Useful developer content usually does one of five things.&lt;/p&gt;

&lt;h3&gt;
  
  
  It Teaches
&lt;/h3&gt;

&lt;p&gt;Teaching content gives the reader a concept they can reuse. It does not only say "our tool supports structured outputs." It explains why structured outputs matter, where they fail, how schema validation changes the workflow, and what to check before relying on them.&lt;/p&gt;

&lt;h3&gt;
  
  
  It Demonstrates
&lt;/h3&gt;

&lt;p&gt;Demonstration content shows the thing working. This can be a tutorial, notebook, example app, CLI walkthrough, benchmark, or integration. The important part is that the reader can see the mechanism.&lt;/p&gt;

&lt;h3&gt;
  
  
  It Compares
&lt;/h3&gt;

&lt;p&gt;Comparison content helps the reader make a decision. It names tradeoffs plainly. It does not pretend every product is the best product for every case.&lt;/p&gt;

&lt;h3&gt;
  
  
  It Diagnoses
&lt;/h3&gt;

&lt;p&gt;Diagnostic content helps the reader recognize a problem. It might explain why a retrieval system is producing weak answers, why a vision model fails on a certain class of images, or why a community program is active but not useful.&lt;/p&gt;

&lt;h3&gt;
  
  
  It Translates
&lt;/h3&gt;

&lt;p&gt;Translation content turns product, engineering, or community signal into language a specific developer audience can use. Good DevRel often lives here. The work is not just broadcasting outward; it is turning what developers say back into product and content decisions.&lt;/p&gt;

&lt;p&gt;If a post does none of these, it is probably decoration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Marketing Language Breaks Trust
&lt;/h2&gt;

&lt;p&gt;Developers are not allergic to persuasion. They are allergic to missing substance.&lt;/p&gt;

&lt;p&gt;Words like "simple," "powerful," "fast," and "production-ready" are not automatically bad. They become bad when the post asks the reader to accept them without proof.&lt;/p&gt;

&lt;p&gt;For developer audiences, proof tends to look like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a runnable example&lt;/li&gt;
&lt;li&gt;a concrete workflow&lt;/li&gt;
&lt;li&gt;a before-and-after comparison&lt;/li&gt;
&lt;li&gt;a clear constraint&lt;/li&gt;
&lt;li&gt;a failure mode&lt;/li&gt;
&lt;li&gt;a benchmark with enough context to interpret it&lt;/li&gt;
&lt;li&gt;a decision rule&lt;/li&gt;
&lt;li&gt;an honest explanation of who the tool is not for&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The weakest content replaces those things with mood. It talks about innovation, speed, delight, scale, and transformation. The reader still does not know what to do next.&lt;/p&gt;

&lt;p&gt;This is especially dangerous in AI content. AI products already sit in a noisy market. Everyone claims better quality, lower cost, faster workflows, and safer deployment. If your post sounds like everyone else's post, the reader has no reason to trust your implementation.&lt;/p&gt;

&lt;p&gt;The fix is not to remove all product language. The fix is to make the product claim carry weight.&lt;/p&gt;

&lt;p&gt;Weak claim:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Our platform makes evaluation easy.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Stronger claim:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This workflow gives you a repeatable eval set, stores failed examples, and lets the team review regressions before shipping a prompt change.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The second version is still a product story. It just has a mechanism.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Six Tests
&lt;/h2&gt;

&lt;p&gt;Before publishing developer content, run it through these six tests.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Does It Help a Developer Do Something?
&lt;/h3&gt;

&lt;p&gt;The post should make the reader more capable. That does not always mean code. A good strategy post can help a founder decide what to hire for. A good architecture post can help an engineering lead avoid a bad migration. A good tutorial can help someone run the first working example.&lt;/p&gt;

&lt;p&gt;If the reader only learns that your company is excited, rewrite it.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Does It Name the Real Object?
&lt;/h3&gt;

&lt;p&gt;Generic content hides the object. Strong content names it.&lt;/p&gt;

&lt;p&gt;Instead of "modern AI workflows," say "a support triage workflow that classifies tickets, retrieves policy context, drafts a response, and logs low-confidence cases for review."&lt;/p&gt;

&lt;p&gt;Instead of "developer engagement," say "community-answered forum questions, working example submissions, package downloads from new projects, or office-hours questions from active users."&lt;/p&gt;

&lt;p&gt;Specific nouns make the piece harder to fake and easier to trust.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Can the Reader Inspect the Mechanism?
&lt;/h3&gt;

&lt;p&gt;The reader should be able to see how the result happens.&lt;/p&gt;

&lt;p&gt;For code content, that may mean commands, scripts, diagrams, repo links, model IDs, prompts, config, or expected outputs. For strategic content, it may mean a checklist, scorecard, operating cadence, or decision tree.&lt;/p&gt;

&lt;p&gt;The point is not to expose every internal detail. The point is to give the reader enough structure to evaluate the claim.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Does It Admit Constraints?
&lt;/h3&gt;

&lt;p&gt;Trust rises when the content says where the approach breaks.&lt;/p&gt;

&lt;p&gt;This is not weakness. It is technical taste.&lt;/p&gt;

&lt;p&gt;If a model needs a GPU, say so. If the community program will not work before there is a clear developer audience, say so. If the benchmark is narrow, say so. If an integration is useful but not the right default for small teams, say so.&lt;/p&gt;

&lt;p&gt;Developers do not expect perfection. They expect the author to know the edges.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Does It Match the Reader's Stage?
&lt;/h3&gt;

&lt;p&gt;Awareness content and activation content are different jobs.&lt;/p&gt;

&lt;p&gt;If the reader has never heard of the problem, start with context and examples. If the reader is already evaluating tools, give them comparison, architecture, pricing shape, migration risk, and proof. If the reader is trying to adopt, give them a working path and a way to recover when it fails.&lt;/p&gt;

&lt;p&gt;A common mistake is trying to make one post serve every stage. The result is usually too broad to be useful.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Does It Create a Feedback Path?
&lt;/h3&gt;

&lt;p&gt;Developer content should not be a one-way broadcast.&lt;/p&gt;

&lt;p&gt;The post should give readers a way to respond, build, question, or take the next step. That might be a GitHub issue, a community thread, a docs feedback link, an office-hours invitation, or a plain contact form.&lt;/p&gt;

&lt;p&gt;For a consulting site like this one, the natural path is simple: if the piece describes a problem your team is facing, use the &lt;a href="https://employabledatascientist.com/contact/" rel="noopener noreferrer"&gt;contact form&lt;/a&gt; and tell us where the work is getting stuck.&lt;/p&gt;

&lt;h2&gt;
  
  
  Formats That Earn Attention
&lt;/h2&gt;

&lt;p&gt;The format should follow the reader's job. Do not start by deciding "we need a blog post." Start with the decision or action the reader needs to make.&lt;/p&gt;

&lt;h3&gt;
  
  
  Runnable Tutorial
&lt;/h3&gt;

&lt;p&gt;Best for activation. The reader already has some intent and wants a working path.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the product has a developer-facing API, SDK, model, CLI, or integration&lt;/li&gt;
&lt;li&gt;setup friction is a known adoption blocker&lt;/li&gt;
&lt;li&gt;developers need to see the exact workflow before trusting the claim&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Good tutorial content includes prerequisites, commands, expected results, common failures, and next steps. It does not need a long brand intro.&lt;/p&gt;

&lt;h3&gt;
  
  
  Technical Explainer
&lt;/h3&gt;

&lt;p&gt;Best for education. The reader needs a mental model before they can act.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the product category is confusing&lt;/li&gt;
&lt;li&gt;there is a new architecture, model type, or workflow to explain&lt;/li&gt;
&lt;li&gt;sales calls keep producing the same conceptual questions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Good explainers make the reader sharper even if they do not buy.&lt;/p&gt;

&lt;h3&gt;
  
  
  Comparison Guide
&lt;/h3&gt;

&lt;p&gt;Best for evaluation. The reader is choosing between approaches.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;developers compare you with a known default&lt;/li&gt;
&lt;li&gt;the market uses vague category language&lt;/li&gt;
&lt;li&gt;your advantage depends on a specific workload, constraint, or team shape&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Good comparison content is not a disguised attack page. It should help the right reader choose well, even when that means choosing something else.&lt;/p&gt;

&lt;h3&gt;
  
  
  Failure-Mode Post
&lt;/h3&gt;

&lt;p&gt;Best for trust. The reader wants to know whether you understand reality.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;your audience has been burned by shallow demos&lt;/li&gt;
&lt;li&gt;the topic involves reliability, safety, cost, latency, or data quality&lt;/li&gt;
&lt;li&gt;adoption depends on understanding edge cases&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Failure-mode content works because it signals competence. It says, "We have seen where this breaks."&lt;/p&gt;

&lt;h3&gt;
  
  
  Teardown
&lt;/h3&gt;

&lt;p&gt;Best for advanced audiences. The reader wants judgment, not a beginner path.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;you can inspect a public system, demo, model, architecture, or workflow&lt;/li&gt;
&lt;li&gt;the analysis teaches transferable lessons&lt;/li&gt;
&lt;li&gt;your audience values taste and technical decision-making&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A good teardown is specific without being petty. The goal is to learn.&lt;/p&gt;

&lt;h3&gt;
  
  
  Operating Playbook
&lt;/h3&gt;

&lt;p&gt;Best for founders and team leads. The reader needs a system, not a tutorial.&lt;/p&gt;

&lt;p&gt;Use when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the problem crosses content, DevRel, product, and engineering&lt;/li&gt;
&lt;li&gt;the team needs a cadence or scorecard&lt;/li&gt;
&lt;li&gt;the decision is organizational&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is where DevRel strategy content can be especially useful. A post about a 30/60/90 day DevRel plan, a developer journey audit, or a content operating system gives the reader something they can try immediately.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Small Content System
&lt;/h2&gt;

&lt;p&gt;One good post is useful. A small content system is better.&lt;/p&gt;

&lt;p&gt;For a technical team, the system can be simple:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect the questions developers already ask.&lt;/li&gt;
&lt;li&gt;Sort them by stage: awareness, evaluation, activation, retention.&lt;/li&gt;
&lt;li&gt;Pick one content job per piece.&lt;/li&gt;
&lt;li&gt;Choose the format that matches the job.&lt;/li&gt;
&lt;li&gt;Publish with a clear next step.&lt;/li&gt;
&lt;li&gt;Review what the piece taught the team.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first step is the most important. Your best content ideas are usually not hidden in a brainstorming session. They are already sitting in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;sales call notes&lt;/li&gt;
&lt;li&gt;support tickets&lt;/li&gt;
&lt;li&gt;GitHub issues&lt;/li&gt;
&lt;li&gt;Discord or Slack threads&lt;/li&gt;
&lt;li&gt;docs search logs&lt;/li&gt;
&lt;li&gt;failed onboarding calls&lt;/li&gt;
&lt;li&gt;conference questions&lt;/li&gt;
&lt;li&gt;internal product debates&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The work is to turn those signals into public artifacts that reduce future friction.&lt;/p&gt;

&lt;p&gt;For example, if five prospects ask whether your AI feature works with private data, do not publish "How We Think About Enterprise AI." Write the concrete piece:&lt;/p&gt;

&lt;p&gt;"How to Evaluate Private Data Workflows for LLM Features"&lt;/p&gt;

&lt;p&gt;Then show the threat model, architecture options, failure modes, and questions the buyer should ask. That post will still help your product, but it will do it by helping the reader think.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measurement
&lt;/h2&gt;

&lt;p&gt;Developer content should be measured, but not every piece should be judged by the same metric.&lt;/p&gt;

&lt;p&gt;A tutorial might be measured by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;completed runs&lt;/li&gt;
&lt;li&gt;repo clones&lt;/li&gt;
&lt;li&gt;package installs&lt;/li&gt;
&lt;li&gt;docs clicks&lt;/li&gt;
&lt;li&gt;support tickets avoided&lt;/li&gt;
&lt;li&gt;activation from the example path&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An explainer might be measured by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;qualified traffic&lt;/li&gt;
&lt;li&gt;time on page&lt;/li&gt;
&lt;li&gt;newsletter signups&lt;/li&gt;
&lt;li&gt;sales-call mentions&lt;/li&gt;
&lt;li&gt;internal reuse by sales or solutions teams&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A comparison guide might be measured by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;assisted pipeline&lt;/li&gt;
&lt;li&gt;demo requests&lt;/li&gt;
&lt;li&gt;product-qualified signups&lt;/li&gt;
&lt;li&gt;follow-up questions from high-intent readers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A community or DevRel strategy post might be measured by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;founder inquiries&lt;/li&gt;
&lt;li&gt;recruiting conversations&lt;/li&gt;
&lt;li&gt;saved/shared references&lt;/li&gt;
&lt;li&gt;invitations to advise on program design&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The point is not to force every post into a revenue spreadsheet. The point is to decide what kind of signal the post is supposed to create before you publish it.&lt;/p&gt;

&lt;p&gt;If you cannot name the signal, you probably have not named the reader's job clearly enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Developer content does not feel like marketing when it respects the reader's time.&lt;/p&gt;

&lt;p&gt;That means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;lead with the problem, not the product&lt;/li&gt;
&lt;li&gt;use specific nouns&lt;/li&gt;
&lt;li&gt;show the mechanism&lt;/li&gt;
&lt;li&gt;admit constraints&lt;/li&gt;
&lt;li&gt;choose the format that matches the reader's stage&lt;/li&gt;
&lt;li&gt;create a useful next step&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The content can still support sales. It can still explain a product. It can still drive inquiries. But it earns that right by making the reader more capable first.&lt;/p&gt;

&lt;p&gt;That is the standard technical teams should hold themselves to: useful enough to stand alone, specific enough to be trusted, and direct enough that the next step is obvious.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;If your team needs a sharper developer content system, start by auditing the last ten questions developers asked you. Turn the most repeated one into a post that helps them decide, build, or debug.&lt;/p&gt;

&lt;p&gt;If the problem is bigger than one post, we provide &lt;a href="https://employabledatascientist.com/services/devrel-strategy/" rel="noopener noreferrer"&gt;DevRel consulting for developer tools&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/services/brand-content/" rel="noopener noreferrer"&gt;technical content&lt;/a&gt;. &lt;a href="https://employabledatascientist.com/contact/?service=content" rel="noopener noreferrer"&gt;Talk to us&lt;/a&gt; about what your developers are asking, what your content needs to prove, and where the current system is getting stuck.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why do developers reject marketing-style developer content, and what counts as proof instead?
&lt;/h3&gt;

&lt;p&gt;Developers reject marketing because it hides the work, not because they hate products. Proof looks like a runnable example, a concrete workflow, a before-and-after comparison, a clear constraint, a failure mode, a benchmark with context, or an honest note on who the tool is not for.&lt;/p&gt;

&lt;h3&gt;
  
  
  What six tests should developer content pass before it is published?
&lt;/h3&gt;

&lt;p&gt;The six tests ask whether the content helps a developer do something, names the real object, lets the reader inspect the mechanism, admits constraints, matches the reader's stage, and creates a feedback path. If the reader only learns that the company is excited, it should be rewritten.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which developer content format fits which reader need?
&lt;/h3&gt;

&lt;p&gt;The format should follow the reader's job. A runnable tutorial is best for activation, a technical explainer for education, a comparison guide for evaluation, and a failure-mode post for trust. Teardowns suit advanced audiences, and operating playbooks suit founders and team leads.&lt;/p&gt;

&lt;h3&gt;
  
  
  How should a team measure tutorials differently from explainers and comparison guides?
&lt;/h3&gt;

&lt;p&gt;Not every piece should be judged by the same metric. Tutorials can be measured by completed runs, repo clones, package installs, and docs clicks, explainers by qualified traffic and newsletter signups, and comparison guides by assisted pipeline and demo requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where can a team find ideas for developer content that helps instead of pitches?
&lt;/h3&gt;

&lt;p&gt;Collect the questions developers already ask in sales call notes, support tickets, GitHub issues, Discord or Slack threads, and docs search logs. If five prospects ask about private data, write a concrete evaluation guide rather than a generic enterprise AI post.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/developer-content-that-doesnt-feel-like-marketing/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devrel</category>
      <category>writing</category>
      <category>testing</category>
    </item>
    <item>
      <title>The Accidental AI Power User</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Wed, 18 Nov 2026 14:00:34 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/the-accidental-ai-power-user-188m</link>
      <guid>https://dev.to/harpreetsahota/the-accidental-ai-power-user-188m</guid>
      <description>&lt;h1&gt;
  
  
  The Accidental AI Power User
&lt;/h1&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Every company has someone quietly using AI to move faster with spreadsheets, reports, macros, docs, and internal workflows.&lt;/li&gt;
&lt;li&gt;They may not call themselves a developer, but they are already doing lightweight software work.&lt;/li&gt;
&lt;li&gt;These people are exactly who companies should enable, not ignore.&lt;/li&gt;
&lt;li&gt;With the right AI training, they can become a serious productivity multiplier without needing to become full-time engineers.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;You have probably seen this person already.&lt;/p&gt;

&lt;p&gt;They are not on the engineering team. They may work in operations, finance, sales, HR, customer support, analytics, logistics, or compliance. They are the person everyone goes to when the spreadsheet is broken, the report needs cleanup, the process needs a shortcut, or the team needs a weird internal workaround by Friday.&lt;/p&gt;

&lt;p&gt;Then they start using AI through a chat box.&lt;/p&gt;

&lt;p&gt;Suddenly they are writing better Excel formulas. They are generating macros. They are cleaning data faster. They are building little automations. They are asking the AI to explain scripts, rewrite reports, debug errors, and turn messy instructions into working steps.&lt;/p&gt;

&lt;p&gt;They may not think of themselves as technical.&lt;/p&gt;

&lt;p&gt;But they are already becoming something important: an accidental AI power user.&lt;/p&gt;

&lt;h2&gt;
  
  
  Table of Contents
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The Person Already Doing The Work&lt;/li&gt;
&lt;li&gt;Why This Is More Than A Productivity Hack&lt;/li&gt;
&lt;li&gt;They May Not Be Developers, But They Are Building&lt;/li&gt;
&lt;li&gt;Why They Eventually Hit A Ceiling&lt;/li&gt;
&lt;li&gt;What The Right Training Unlocks&lt;/li&gt;
&lt;li&gt;Why Companies Should Pay Attention&lt;/li&gt;
&lt;li&gt;Summary&lt;/li&gt;
&lt;li&gt;Next Steps&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Person Already Doing The Work
&lt;/h2&gt;

&lt;p&gt;Inside many companies, there is someone who quietly keeps work moving.&lt;/p&gt;

&lt;p&gt;They know the spreadsheet with twelve tabs and three hidden dependencies. They know which export from the enterprise system has to be cleaned before finance can use it. They know which report always breaks because one department names the same field three different ways.&lt;/p&gt;

&lt;p&gt;Before AI, they were already improvising:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;nested Excel formulas&lt;/li&gt;
&lt;li&gt;pivot tables&lt;/li&gt;
&lt;li&gt;macros&lt;/li&gt;
&lt;li&gt;CSV cleanup&lt;/li&gt;
&lt;li&gt;copy-paste workflows&lt;/li&gt;
&lt;li&gt;reporting templates&lt;/li&gt;
&lt;li&gt;lightweight scripts&lt;/li&gt;
&lt;li&gt;internal docs&lt;/li&gt;
&lt;li&gt;process checklists&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now they have a chat box that can help them think through the work.&lt;/p&gt;

&lt;p&gt;They ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;"Write an Excel formula that does this."&lt;/li&gt;
&lt;li&gt;"Explain why this macro is failing."&lt;/li&gt;
&lt;li&gt;"Turn these steps into a script."&lt;/li&gt;
&lt;li&gt;"Clean this messy list."&lt;/li&gt;
&lt;li&gt;"Help me compare these two exports."&lt;/li&gt;
&lt;li&gt;"Write a formula that checks these conditions."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And it works often enough that their output changes.&lt;/p&gt;

&lt;p&gt;They become faster. They become more ambitious. They start solving problems they used to route to someone else.&lt;/p&gt;

&lt;p&gt;That is not a small thing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Is More Than A Productivity Hack
&lt;/h2&gt;

&lt;p&gt;It is easy to dismiss this as someone getting better at Excel.&lt;/p&gt;

&lt;p&gt;That misses the larger shift.&lt;/p&gt;

&lt;p&gt;This person is learning to translate business intent into executable logic.&lt;/p&gt;

&lt;p&gt;They are not just asking for a formula. They are describing a rule.&lt;/p&gt;

&lt;p&gt;They are not just asking for a macro. They are describing a workflow.&lt;/p&gt;

&lt;p&gt;They are not just cleaning a report. They are formalizing a process that used to live in someone's head.&lt;/p&gt;

&lt;p&gt;That is software thinking, even if it does not look like software engineering.&lt;/p&gt;

&lt;p&gt;AI gives non-developers a new interface into that kind of work. It lets them express intent in natural language and get back a formula, script, macro, checklist, or structured plan.&lt;/p&gt;

&lt;p&gt;This is why &lt;a href="https://employabledatascientist.com/blog/english-is-the-new-interface/" rel="noopener noreferrer"&gt;English is becoming a new interface&lt;/a&gt;. The person does not need to know every syntax rule before they can start. They can describe what they want, inspect the result, and iterate.&lt;/p&gt;

&lt;h2&gt;
  
  
  They May Not Be Developers, But They Are Building
&lt;/h2&gt;

&lt;p&gt;There is a category of work between "business user" and "software engineer."&lt;/p&gt;

&lt;p&gt;It includes people who:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;build spreadsheet models&lt;/li&gt;
&lt;li&gt;automate repetitive reporting&lt;/li&gt;
&lt;li&gt;connect tools with exports and imports&lt;/li&gt;
&lt;li&gt;write formulas&lt;/li&gt;
&lt;li&gt;adapt templates&lt;/li&gt;
&lt;li&gt;create internal processes&lt;/li&gt;
&lt;li&gt;document workflows&lt;/li&gt;
&lt;li&gt;build no-code or low-code tools&lt;/li&gt;
&lt;li&gt;ask technical questions better than their job title suggests&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;AI makes this group more powerful.&lt;/p&gt;

&lt;p&gt;They can now ask for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;better formulas&lt;/li&gt;
&lt;li&gt;explanations of unfamiliar code&lt;/li&gt;
&lt;li&gt;macro improvements&lt;/li&gt;
&lt;li&gt;simple scripts&lt;/li&gt;
&lt;li&gt;data validation checks&lt;/li&gt;
&lt;li&gt;report templates&lt;/li&gt;
&lt;li&gt;workflow diagrams&lt;/li&gt;
&lt;li&gt;automation ideas&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This does not make them senior engineers. It makes them better operators.&lt;/p&gt;

&lt;p&gt;And in many companies, better operators create enormous leverage.&lt;/p&gt;

&lt;p&gt;They remove bottlenecks. They reduce busywork. They make reports cleaner. They help teams make fewer manual mistakes. They create small internal tools that save hours every week.&lt;/p&gt;

&lt;p&gt;That is exactly the kind of person companies should enable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why They Eventually Hit A Ceiling
&lt;/h2&gt;

&lt;p&gt;The chat box gets them started.&lt;/p&gt;

&lt;p&gt;But without training, they will hit a ceiling.&lt;/p&gt;

&lt;p&gt;They may not know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;how to break a problem into smaller steps&lt;/li&gt;
&lt;li&gt;how to test whether a formula is correct&lt;/li&gt;
&lt;li&gt;how to spot hallucinated code&lt;/li&gt;
&lt;li&gt;how to protect sensitive company data&lt;/li&gt;
&lt;li&gt;when a macro is risky&lt;/li&gt;
&lt;li&gt;when a spreadsheet should become a proper internal tool&lt;/li&gt;
&lt;li&gt;how to document what they built&lt;/li&gt;
&lt;li&gt;how to ask AI for alternatives&lt;/li&gt;
&lt;li&gt;how to review generated scripts&lt;/li&gt;
&lt;li&gt;how to avoid turning one-off fixes into fragile infrastructure&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the danger.&lt;/p&gt;

&lt;p&gt;The same person who can save the company hours can also accidentally create a hidden dependency nobody understands.&lt;/p&gt;

&lt;p&gt;The answer is not to shut them down.&lt;/p&gt;

&lt;p&gt;The answer is to train them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What The Right Training Unlocks
&lt;/h2&gt;

&lt;p&gt;The right AI training does not try to turn every operations person into a software engineer.&lt;/p&gt;

&lt;p&gt;It teaches them how to use AI tools with better judgment.&lt;/p&gt;

&lt;p&gt;Good training helps them learn:&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Describe The Problem Clearly
&lt;/h3&gt;

&lt;p&gt;Better prompts start with better problem framing. What is the input? What is the output? What rule should be applied? What edge cases matter?&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Ask For Stepwise Help
&lt;/h3&gt;

&lt;p&gt;Instead of asking AI to do everything at once, they learn to ask for a plan, inspect it, then generate one piece at a time.&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Test The Output
&lt;/h3&gt;

&lt;p&gt;They learn to use sample rows, expected results, edge cases, and sanity checks.&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Debug Safely
&lt;/h3&gt;

&lt;p&gt;They learn to paste errors, ask for explanations, compare fixes, and avoid blindly accepting the first answer.&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Protect Company Data
&lt;/h3&gt;

&lt;p&gt;They learn what not to paste into a tool, how to anonymize examples, and when to use approved systems.&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Know When To Escalate
&lt;/h3&gt;

&lt;p&gt;They learn when a formula is fine, when a macro needs review, and when the work should go to engineering.&lt;/p&gt;

&lt;h3&gt;
  
  
  How To Document Their Work
&lt;/h3&gt;

&lt;p&gt;They learn to ask AI to explain the formula, comment the macro, write a handoff note, or create a small internal guide.&lt;/p&gt;

&lt;p&gt;This kind of training turns raw enthusiasm into useful leverage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Companies Should Pay Attention
&lt;/h2&gt;

&lt;p&gt;Companies spend a lot of time thinking about AI strategy at the executive level.&lt;/p&gt;

&lt;p&gt;That matters.&lt;/p&gt;

&lt;p&gt;But a huge amount of AI adoption will happen in the middle of the organization, where people are doing messy work with spreadsheets, reports, internal systems, and recurring processes.&lt;/p&gt;

&lt;p&gt;That is where the accidental AI power user lives.&lt;/p&gt;

&lt;p&gt;If you enable them, they can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;reduce manual reporting work&lt;/li&gt;
&lt;li&gt;clean up data workflows&lt;/li&gt;
&lt;li&gt;build better internal templates&lt;/li&gt;
&lt;li&gt;automate repetitive tasks&lt;/li&gt;
&lt;li&gt;improve team documentation&lt;/li&gt;
&lt;li&gt;catch errors earlier&lt;/li&gt;
&lt;li&gt;prototype internal process improvements&lt;/li&gt;
&lt;li&gt;communicate better with engineering&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you ignore them, they will still use AI. They will just do it without shared standards, training, or review.&lt;/p&gt;

&lt;p&gt;This is the same point I made in &lt;a href="https://employabledatascientist.com/blog/ai-coding-for-internal-software-teams/" rel="noopener noreferrer"&gt;AI coding for internal software teams&lt;/a&gt;: the choice is not AI or no AI. The choice is trained adoption or unmanaged adoption.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The person using AI to write Excel formulas, debug macros, clean reports, and automate small workflows may not call themselves a developer.&lt;/p&gt;

&lt;p&gt;But they are already doing the kind of work that makes a company faster.&lt;/p&gt;

&lt;p&gt;With the right training, they can become much more than "the spreadsheet person." They can become a productivity multiplier across the team.&lt;/p&gt;

&lt;p&gt;You do not need to turn them into engineers. You need to teach them how to use AI with better judgment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;Look around your company for the person who already does this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;fixes the spreadsheet&lt;/li&gt;
&lt;li&gt;cleans the report&lt;/li&gt;
&lt;li&gt;writes the macro&lt;/li&gt;
&lt;li&gt;figures out the workaround&lt;/li&gt;
&lt;li&gt;documents the messy process&lt;/li&gt;
&lt;li&gt;quietly uses AI to move faster&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That person is not a risk to suppress. They are potential to train.&lt;/p&gt;

&lt;p&gt;If you want to unlock that person, or a whole team of them, &lt;a href="https://employabledatascientist.com/contact/" rel="noopener noreferrer"&gt;let's chat&lt;/a&gt;. I run practical AI training sessions that help business teams use AI tools for formulas, macros, internal workflows, and lightweight automation without creating a mess your engineers have to clean up later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is an accidental AI power user in a company?
&lt;/h3&gt;

&lt;p&gt;An accidental AI power user is someone outside engineering, in areas like operations, finance, sales, HR, or compliance, who quietly uses AI chat tools to write Excel formulas, generate macros, clean data, and build small automations. They may not call themselves a developer, but they are already doing lightweight software work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why is using AI for Excel formulas and macros more than a productivity hack?
&lt;/h3&gt;

&lt;p&gt;The person is learning to translate business intent into executable logic. Asking for a formula describes a rule, asking for a macro describes a workflow, and cleaning a report formalizes a process that lived in someone's head. That is software thinking, even if it does not look like software engineering.&lt;/p&gt;

&lt;h3&gt;
  
  
  What ceiling do AI power users hit without training?
&lt;/h3&gt;

&lt;p&gt;Without training they may not know how to break a problem into steps, test a formula, spot hallucinated code, protect sensitive company data, or document their work. They can also turn one-off fixes into fragile infrastructure, creating a hidden dependency nobody understands.&lt;/p&gt;

&lt;h3&gt;
  
  
  What should AI training for business teams cover?
&lt;/h3&gt;

&lt;p&gt;Good training teaches describing the problem clearly, asking for stepwise help, testing output with sample rows and edge cases, debugging safely, protecting company data, knowing when to escalate to engineering, and documenting the work. It does not try to turn operations staff into software engineers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should companies restrict employees who use AI chat tools for spreadsheets and macros?
&lt;/h3&gt;

&lt;p&gt;No. The answer is not to shut them down but to train them. If ignored, they will still use AI, only without shared standards, training, or review. The choice is trained adoption or unmanaged adoption.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/accidental-ai-power-user/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>writing</category>
    </item>
    <item>
      <title>Hands-on Point Tracking in Video with Meta AI CoTracker3 and FiftyOne</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Tue, 17 Nov 2026 14:00:33 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/hands-on-point-tracking-in-video-with-meta-ai-cotracker3-and-fiftyone-jd</link>
      <guid>https://dev.to/harpreetsahota/hands-on-point-tracking-in-video-with-meta-ai-cotracker3-and-fiftyone-jd</guid>
      <description>&lt;p&gt;Meta AI released &lt;strong&gt;CoTracker3&lt;/strong&gt;, a state-of-the-art model for tracking points in video. Point tracking represents one of the most critical foundational tasks in computer vision, yet it is often misunderstood or conflated with traditional object tracking.&lt;/p&gt;

&lt;p&gt;While object tracking follows bounding boxes or semantic masks around entire items, point tracking follows individual pixels, physical surface features, and keypoint trajectories across consecutive video frames.&lt;/p&gt;

&lt;p&gt;Whether you are building physical AI systems, evaluating robotic manipulation policies, compensating for camera motion, or testing physical consistency in generative video world models, point tracking provides the sub-pixel telemetry required to understand motion in three-dimensional space.&lt;/p&gt;

&lt;p&gt;Here is a hands-on technical breakdown of how CoTracker3 works and how to convert point tracking inference into an interactive FiftyOne dataset.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       COTRACKER3 POINT TRACKING ARCHITECTURE
+-------------------------------------------------------------+
| Input Video Sequence (Frames 0 to T)                        |
|       |                                                     |
|       v                                                     |
| Dense Feature Pyramid Extraction (CNN / Transformer)        |
|       |                                                     |
|       v                                                     |
| Query Points Selection (Grid or Custom Keypoints)           |
|       |                                                     |
|       v                                                     |
| Sliding-Window Transformer Trajectory Estimation            |
|   - Multi-frame correlation matching                        |
|   - Occlusion and visibility prediction                     |
|       |                                                     |
|       v                                                     |
| Output Trajectory Tensor (T, N, 2) + Visibility (T, N)      |
|       |                                                     |
|       v                                                     |
| FiftyOne Dataset Visualization (Interactive Inspection)     |
+-------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Point Tracking vs. Object Tracking: The Structural Difference
&lt;/h2&gt;

&lt;p&gt;In standard object detection and multi-object tracking (MOT):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A model detects a person, car, or tool using a bounding box.&lt;/li&gt;
&lt;li&gt;A tracking algorithm (e.g., ByteTrack, DeepSORT) associates boxes across frames using Kalman filters and appearance embeddings.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This approach fails when you need to know how an object is physically moving or deforming:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A robotic gripper grasping an elastic object changes shape continuously.&lt;/li&gt;
&lt;li&gt;A dancing human's limbs articulate rapidly, causing bounding boxes to fluctuate wildly without capturing joint mechanics.&lt;/li&gt;
&lt;li&gt;A vehicle turns, rotating its surface features into and out of line-of-sight.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Point tracking&lt;/strong&gt; operates at the physical coordinate level. Given an initial set of 2D coordinates $(x_0, y_0)$ at frame $t_0$, the model tracks the exact trajectory $(x_t, y_t)$ across every subsequent frame $t$, even when the physical point is temporarily occluded behind another object.&lt;/p&gt;

&lt;h2&gt;
  
  
  How CoTracker3 Operates
&lt;/h2&gt;

&lt;p&gt;CoTracker3 formulates point tracking as an iterative trajectory refinement task using an efficient transformer architecture:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Multi-Scale Feature Extraction:&lt;/strong&gt; The video sequence is processed through a convolutional backbone to generate multi-scale visual feature maps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dense Point Attention:&lt;/strong&gt; Rather than tracking each point independently (as in classical Lucas-Kanade optical flow), CoTracker3 tracks groups of points jointly. By attending to relationships between neighboring points, the model leverages physical rigidity and spatial context to resolve ambiguous motion.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sliding-Window Temporal Processing:&lt;/strong&gt; CoTracker3 processes video in overlapping temporal windows, enabling quasi-online inference for streaming video while retaining long-range temporal context to recover points after lengthy occlusions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Visibility and Confidence Heads:&lt;/strong&gt; For every tracked point at every frame, the network outputs both the 2D coordinate $(x, y)$ and a visibility flag $v \in [0, 1]$, indicating whether the point is directly visible or occluded.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Real-World Applications in Physical AI and Robotics
&lt;/h2&gt;

&lt;p&gt;Point tracking is rapidly becoming a cornerstone of production AI workflows:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Robotic Teleoperation and Imitation Learning:&lt;/strong&gt; Tracking key contact points on human demonstrations to map fingertip trajectories directly to robotic manipulators.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Physical Invariant Validation in World Models:&lt;/strong&gt; Generative video models frequently produce physically impossible trajectories (e.g. objects warping or drifting unnaturally). Running CoTracker3 across generated video allows automated pipelines to score physical trajectory consistency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dynamic Camera Ego-Motion Estimation:&lt;/strong&gt; Tracking stationary background points to separate camera pan and tilt from independent foreground object motion.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Hands-On: Running CoTracker3 and Exporting to FiftyOne
&lt;/h2&gt;

&lt;p&gt;To inspect point tracks visually, convert the raw tensor outputs from CoTracker3 into FiftyOne keypoint or polyline annotations.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;

&lt;span class="c1"&gt;# Load video sample in FiftyOne&lt;/span&gt;
&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cotracker3_demo"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sample&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Sample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filepath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/home/harpreet/workspace/yt_shorts/CoTracker3.mp4"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_sample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Load Meta AI CoTracker3 model from PyTorch Hub&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"facebookresearch/co-tracker"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cotracker3_offline"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cuda&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Load video frames as tensor (B, T, C, H, W)&lt;/span&gt;
&lt;span class="c1"&gt;# For demo purposes, assume video_tensor is loaded and normalized&lt;/span&gt;
&lt;span class="c1"&gt;# video_tensor = load_video_tensor(sample.filepath)&lt;/span&gt;

&lt;span class="c1"&gt;# Define grid of query points or sample salient keypoints&lt;/span&gt;
&lt;span class="c1"&gt;# grid_size = 20 (tracking 400 points concurrently)&lt;/span&gt;
&lt;span class="c1"&gt;# pred_tracks, pred_visibility = model(video_tensor, grid_size=20)&lt;/span&gt;

&lt;span class="c1"&gt;# Format trajectories as FiftyOne Keypoint annotations per frame&lt;/span&gt;
&lt;span class="c1"&gt;# FiftyOne displays keypoint trajectories interactively across video playback&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Converted CoTracker3 trajectories into FiftyOne video annotations."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once loaded into FiftyOne, you can scrub through the video timeline, filter tracks by visibility confidence, and inspect exactly where trajectories lose tracking or recover from occlusions.&lt;/p&gt;

&lt;p&gt;For related reading on video evaluation and robotics data pipelines, review our field notes on &lt;a href="https://employabledatascientist.com/blog/boilingbench-multimodal-fiftyone-analysis/" rel="noopener noreferrer"&gt;multimodal sensor curation&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/10-robotics-github-repos-3d-perception-vla-calibration/" rel="noopener noreferrer"&gt;open-source robotics repositories&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To deepen your understanding of dataset curation and visual AI pipelines, explore Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is the difference between point tracking and object tracking?
&lt;/h3&gt;

&lt;p&gt;Object tracking follows bounding boxes or instance masks around entire semantic objects. Point tracking tracks specific sub-pixel 2D coordinate points across frames, maintaining trajectory correspondence even as objects deform, articulate, or undergo severe occlusion.&lt;/p&gt;

&lt;h3&gt;
  
  
  What technical improvements does CoTracker3 introduce over CoTracker v1 and v2?
&lt;/h3&gt;

&lt;p&gt;CoTracker3 introduces an efficient transformer architecture that tracks points both quasi-online and offline, achieves superior temporal persistence through long occlusions, and dramatically reduces GPU memory overhead while maintaining high trajectory accuracy.&lt;/p&gt;

&lt;h3&gt;
  
  
  How is point tracking used in physical AI and robotics?
&lt;/h3&gt;

&lt;p&gt;Point tracking provides foundational motion telemetry for robot visual servoing, dynamic deformation tracking, policy verification, camera motion compensation, and physical invariant testing in world models.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where can developers inspect point trajectories interactively?
&lt;/h3&gt;

&lt;p&gt;Point trajectories can be formatted as keypoints or polylines and visualized frame-by-frame inside the open-source FiftyOne App.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/hands-on-point-tracking-cotracker3-fiftyone/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>visionmultimodal</category>
      <category>ai</category>
      <category>robotics</category>
      <category>computervision</category>
    </item>
    <item>
      <title>DreamTac: 703 Franka Trajectories With Gel Fingertip Video for Robot Learning</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Thu, 12 Nov 2026 14:00:27 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/dreamtac-703-franka-trajectories-with-gel-fingertip-video-for-robot-learning-kj5</link>
      <guid>https://dev.to/harpreetsahota/dreamtac-703-franka-trajectories-with-gel-fingertip-video-for-robot-learning-kj5</guid>
      <description>&lt;p&gt;You can see the exact moment a robot touches something when its fingertip is a gel pad printed with a marker grid. The grid deforms where the object presses, so contact shows up as a change in the image. DreamTac puts two of these fingertips on a Franka arm, alongside a wrist camera and a third-person camera.&lt;/p&gt;





&lt;h2&gt;
  
  
  What DreamTac records
&lt;/h2&gt;

&lt;p&gt;DreamTac is a &lt;a href="https://huggingface.co/datasets/cloudfan/DreamTac" rel="noopener noreferrer"&gt;Peking University release&lt;/a&gt; of a Franka Emika Panda working through contact-rich tabletop tasks. The &lt;a href="https://huggingface.co/datasets/Voxel51/DreamTac" rel="noopener noreferrer"&gt;Voxel51 conversion&lt;/a&gt; turns it into native multimodal MCAP episodes. It holds 703 trajectories across 12 tasks, 506,078 frames and 7.03 hours. Each frame carries all four views, so the set contains 2,024,312 camera frames.&lt;/p&gt;

&lt;p&gt;The two fingertips are Xense Photon vision-based tactile sensors. All four cameras record on one 20 fps clock, which is what lets you line up a touch in the fingertip with what the arm was doing.&lt;/p&gt;

&lt;p&gt;Each episode has these streams and fields:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stream&lt;/th&gt;
&lt;th&gt;What it holds&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/camera&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Third-person view at 224x224, as &lt;code&gt;foxglove.CompressedVideo&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/wrist-camera&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The same, from the wrist&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/tactile-left&lt;/code&gt;, &lt;code&gt;/tactile-right&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;The two Xense Photon fingertips&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/end-effector-pose&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Position and orientation in the Franka base frame&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/end-effector.plot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The same pose as &lt;code&gt;x&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;z&lt;/code&gt; and a rotation vector in radians&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/gripper.plot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gripper opening&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/instruction&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Task text, written at each change, so sub-steps appear on the timeline&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Episode-level fields are &lt;code&gt;task&lt;/code&gt;, &lt;code&gt;episode_index&lt;/code&gt;, &lt;code&gt;instruction&lt;/code&gt;, &lt;code&gt;num_frames&lt;/code&gt;, &lt;code&gt;duration&lt;/code&gt;, &lt;code&gt;fps&lt;/code&gt;, &lt;code&gt;end_effector_path_m&lt;/code&gt;, &lt;code&gt;end_effector_z_min_m&lt;/code&gt;, &lt;code&gt;end_effector_z_max_m&lt;/code&gt;, &lt;code&gt;gripper_min&lt;/code&gt;, &lt;code&gt;gripper_max&lt;/code&gt; and &lt;code&gt;tactile_sensor&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 12 tasks
&lt;/h2&gt;

&lt;p&gt;The tasks range from 2 episodes to 104. &lt;code&gt;clean_whiteboard&lt;/code&gt; has the most episodes at 104 (72.4 minutes, mean path 1.11 m). &lt;code&gt;pick_bread&lt;/code&gt; has the most footage: 99 episodes, 105,603 frames and 88.0 minutes, with the longest mean path at 1.83 m. At the other end, &lt;code&gt;cut_banana&lt;/code&gt; is 38 episodes and 8.8 minutes, with a mean path of 0.20 m, and &lt;code&gt;insert_usb&lt;/code&gt; is 2 episodes and 0.5 minutes.&lt;/p&gt;

&lt;p&gt;The instructions are plain sentences, such as "Clean the whiteboard with an eraser." and "Cut the banana with a knife." The Mahjong tasks all use the red dragon tile: &lt;code&gt;pick_hongzhong&lt;/code&gt; picks it up, and the three &lt;code&gt;push_hongzhong&lt;/code&gt; variants push it to a target.&lt;/p&gt;

&lt;h2&gt;
  
  
  Load it and query it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utils&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;huggingface&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fouh&lt;/span&gt;

&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fouh&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_from_hub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"Voxel51/DreamTac"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"DreamTac"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;persistent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Open an episode and play it. The wrist camera, the third-person camera and both fingertips sit on one timeline, so you can scrub to the frame where the marker grid first distorts and read off what the scene looked like at that instant.&lt;/p&gt;

&lt;p&gt;To start with the trajectories that moved the furthest, sort on the path length of the end effector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;view&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"end_effector_path_m"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To isolate one task, match on the &lt;code&gt;task&lt;/code&gt; field. This pulls up the knife episodes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ViewField&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;

&lt;span class="n"&gt;view&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"task"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"cut_banana"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Known limits
&lt;/h2&gt;

&lt;p&gt;The task mix is uneven. The three &lt;code&gt;push_hongzhong&lt;/code&gt; tasks are the same push from three different start positions, and &lt;code&gt;insert_usb&lt;/code&gt; holds two trajectories against &lt;code&gt;pick_usb&lt;/code&gt;'s 102. A model trained on everything will see the large tasks far more often than the small ones. Stratify by task if you want a fair read on &lt;code&gt;insert_usb&lt;/code&gt;, &lt;code&gt;pick_hongzhong&lt;/code&gt; (18 episodes) or &lt;code&gt;cut_banana&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Arm joint angles are not in the source and are not carried over. If your policy needs joint state, this dataset cannot supply it. The gripper is a single opening scalar.&lt;/p&gt;

&lt;p&gt;The conversion drops some trajectories. The release excludes 21 trajectories from a &lt;code&gt;no_tactile&lt;/code&gt; batch and 4 that failed source validation, which leaves the 703 you load. Episode and frame counts match the release's own &lt;code&gt;source_qc_summary.json&lt;/code&gt; on all 12 tasks.&lt;/p&gt;

&lt;p&gt;The source ships one Zarr v2 group per task in the Open-X-Tactile layout, with every trajectory laid end to end. The per-frame clock restarts at each boundary, so the conversion cuts episodes there and each one starts at zero. The four image streams arrive as raw &lt;code&gt;uint8&lt;/code&gt; arrays and are re-encoded to H.264.&lt;/p&gt;

&lt;p&gt;A tactile image shows deformation, and deformation is not force. For a comparison, &lt;a href="https://employabledatascientist.com/blog/taf-tactile-force-fiftyone-analysis/" rel="noopener noreferrer"&gt;TaF-Tactile-Force&lt;/a&gt; pairs a tactile camera with a six-axis force sensor on one clock. For how gel sensors, force vectors and taxel grids differ as data, see &lt;a href="https://employabledatascientist.com/blog/tactile-datasets-robot-learning-lerobot/" rel="noopener noreferrer"&gt;Robots Can See. They Still Can't Feel&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  License and source
&lt;/h2&gt;

&lt;p&gt;The source release is distributed under CC BY 4.0, and the conversion uses the same license. The paper is Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation (Lou, Ye, Fu, Cen, Chi and Lyu, &lt;a href="https://arxiv.org/abs/2606.08737" rel="noopener noreferrer"&gt;arXiv:2606.08737&lt;/a&gt;). The Voxel51 changes are the MCAP conversion, H.264 encoding of the image streams, splitting the concatenated task groups at their episode boundaries, and encoding pose, gripper and instruction as message streams.&lt;/p&gt;

&lt;p&gt;For more of my FiftyOne dataset notes, see my &lt;a href="https://voxel51.com/author/harpreet" rel="noopener noreferrer"&gt;Voxel51 author page&lt;/a&gt;, and to curate your own robot data in the same tool, &lt;a href="https://link.voxel51.com/harpreet" rel="noopener noreferrer"&gt;try Voxel51&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is the DreamTac dataset?
&lt;/h3&gt;

&lt;p&gt;DreamTac is a Peking University release of a Franka Emika Panda doing contact-rich tabletop tasks. It has 703 trajectories across 12 tasks, 506,078 frames and 7.03 hours, with four cameras on one 20 fps clock.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does DreamTac show contact without a force sensor?
&lt;/h3&gt;

&lt;p&gt;Each fingertip is a gel pad printed with a marker grid, and the grid deforms where an object presses. The moment of contact is visible in the tactile video rather than inferred from a force reading.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I load DreamTac in FiftyOne?
&lt;/h3&gt;

&lt;p&gt;Call load_from_hub from fiftyone.utils.huggingface with Voxel51/DreamTac and persistent=True, then launch the App. Sorting by end_effector_path_m shows the trajectories that moved the furthest.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does DreamTac include robot arm joint angles?
&lt;/h3&gt;

&lt;p&gt;No. Arm joint angles are not in the source release and the conversion does not carry them. The gripper is a single opening scalar, and the end effector pose is in the Franka base frame.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is the DreamTac task mix balanced?
&lt;/h3&gt;

&lt;p&gt;No. The three push_hongzhong tasks are the same push from three start positions, and insert_usb holds 2 trajectories against 102 for pick_usb. Stratify by task before you compare results across tasks.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/dreamtac-fiftyone-analysis/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>robotics</category>
      <category>visionmultimodal</category>
      <category>computervision</category>
      <category>python</category>
    </item>
    <item>
      <title>Scene Complexity Metrics: Measuring Clutter and Density Beyond mAP</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Wed, 11 Nov 2026 14:30:46 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/scene-complexity-metrics-measuring-clutter-and-density-beyond-map-3acp</link>
      <guid>https://dev.to/harpreetsahota/scene-complexity-metrics-measuring-clutter-and-density-beyond-map-3acp</guid>
      <description>&lt;p&gt;A model reports 0.84 mean average precision (mAP) on your benchmark test set. Management approves deployment, and marketing drafts the launch announcement.&lt;/p&gt;

&lt;p&gt;Three days later, field reports arrive: the model is failing systematically. In crowded retail aisles, busy warehouse loading docks, or messy construction sites, false detections proliferate and target objects are repeatedly missed.&lt;/p&gt;

&lt;p&gt;The failure is not that the model suddenly degraded. The failure is that aggregate mAP told only half the story.&lt;/p&gt;

&lt;p&gt;In real-world computer vision, aggregate metrics mask critical operational failure modes. To understand where and why a model will break, you must measure performance against explicit scene complexity metrics: visual clutter, instance crowding, and background texture entropy.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       SCENE COMPLEXITY EVALUATION ARCHITECTURE
+-------------------------------------------------------------+
| Input Test Suite                                            |
|       |                                                     |
|       v                                                     |
| Compute Complexity Scores per Sample:                       |
|   1. Visual Clutter Score (Edge Density &amp;amp; Entropy)          |
|   2. Object Density &amp;amp; Spatial Crowding (IoU Overlap)        |
|   3. Background High-Frequency Distractor Power             |
|       |                                                     |
|       v                                                     |
| Stratify Validation Suite into Complexity Buckets:          |
|   [Simple / Clean] --&amp;gt; [Moderate] --&amp;gt; [Extreme Clutter]     |
|       |                                                     |
|       v                                                     |
| Plot Performance vs. Complexity Curve (Find Breaking Point) |
+-------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Blind Spot of Aggregate mAP
&lt;/h2&gt;

&lt;p&gt;Mean average precision calculates the area under the precision-recall curve, averaged across classes and IoU thresholds. Because the majority of images in standard datasets represent moderate or clean viewing conditions, aggregate mAP is dominated by easy samples.&lt;/p&gt;

&lt;p&gt;A model can maintain an overall mAP of 0.80 by scoring 0.95 on simple scenes while scoring 0.35 on complex, cluttered environments. When that model is deployed into a complex operational setting, performance collapses immediately.&lt;/p&gt;

&lt;p&gt;To build reliable vision systems, you must measure performance as a function of scene complexity.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four Key Dimensions of Scene Complexity
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Visual Clutter Score
&lt;/h3&gt;

&lt;p&gt;Visual clutter reflects the density of competing visual features in an image. Cluttered scenes force feature backbones to process high-entropy noise alongside target objects.&lt;/p&gt;

&lt;p&gt;A reliable operational proxy for visual clutter is &lt;strong&gt;Edge Density&lt;/strong&gt;: the ratio of edge pixels to total image pixels:&lt;/p&gt;

&lt;p&gt;$$D_{edge} = \frac{1}{W \cdot H} \sum_{x, y} E(x, y)$$&lt;/p&gt;

&lt;p&gt;Where $E(x, y)$ is the binary edge map generated by a Canny edge detector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;compute_clutter_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Compute visual clutter via Canny edge density."""&lt;/span&gt;
    &lt;span class="n"&gt;img&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;imread&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;IMREAD_GRAYSCALE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;img&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"Could not load image at {image_path}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;edges&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Canny&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;img&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threshold1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threshold2&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;edge_density&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count_nonzero&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;edges&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;edges&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;edge_density&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2. Dense Object Distributions and Crowding
&lt;/h3&gt;

&lt;p&gt;When objects appear in dense clusters—such as pallets in a logistics depot or pedestrians on a crosswalk—bounding box overlap increases significantly.&lt;/p&gt;

&lt;p&gt;Measure crowding using the &lt;strong&gt;Maximum Pairwise IoU&lt;/strong&gt; and &lt;strong&gt;Instance Count per Frame&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Calculate pairwise intersection-over-union between all ground truth annotations in the frame.&lt;/li&gt;
&lt;li&gt;High pairwise IoU ($&amp;gt; 0.4$) directly correlates with Non-Maximum Suppression (NMS) failure, where adjacent true objects suppress each other.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Instance Scale and Aspect Diversity
&lt;/h3&gt;

&lt;p&gt;A scene containing objects of vastly disparate scales (e.g., a foreground worker next to a distant forklift) stresses multi-scale feature pyramids. Measure scale entropy to isolate whether failures stem from extreme scale differentials.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Background Complexity and Distractor Textures
&lt;/h3&gt;

&lt;p&gt;High-frequency background textures—such as chain-link fences, dappled foliage sunlight, or textured concrete—frequently trigger false positive detections in modern backbones. Measuring subband spatial frequency entropy reveals background distractor strength.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Complexity Dimension&lt;/th&gt;
&lt;th&gt;Measurement Algorithm&lt;/th&gt;
&lt;th&gt;Vulnerable Model Component&lt;/th&gt;
&lt;th&gt;Remediation Strategy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Visual Clutter&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Edge Density &amp;amp; Subband Entropy&lt;/td&gt;
&lt;td&gt;Feature Backbone Attention Layers&lt;/td&gt;
&lt;td&gt;Multi-scale feature aggregation, context gating&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Object Crowding&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Pairwise IoU Distribution&lt;/td&gt;
&lt;td&gt;Non-Maximum Suppression (NMS)&lt;/td&gt;
&lt;td&gt;Soft-NMS, transformer-based direct set prediction (DETR)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale Disparity&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Bounding Box Area Entropy&lt;/td&gt;
&lt;td&gt;Feature Pyramid Networks (FPN)&lt;/td&gt;
&lt;td&gt;High-resolution sliding window inference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Background Noise&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High-Frequency Gradient Energy&lt;/td&gt;
&lt;td&gt;Classifier Head (False Positives)&lt;/td&gt;
&lt;td&gt;Hard negative background mining&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Operationalizing Complexity Curves
&lt;/h2&gt;

&lt;p&gt;Stop reporting single-number mAP scores to executive stakeholders. Instead, generate performance-vs-complexity curves:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Compute clutter and density scores across your entire validation suite.&lt;/li&gt;
&lt;li&gt;Partition the evaluation dataset into quartile buckets: Low, Moderate, High, and Extreme Complexity.&lt;/li&gt;
&lt;li&gt;Compute precision, recall, and localization error within each quartile bucket.&lt;/li&gt;
&lt;li&gt;Pinpoint the exact complexity inflection point where model accuracy drops below your operational threshold.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This stratification tells your engineering team exactly what type of data collection or architectural adjustment is required to bridge the gap.&lt;/p&gt;

&lt;p&gt;For deeper insights into systematic evaluation frameworks, explore our guides on &lt;a href="https://employabledatascientist.com/blog/evaluating-ai-agents-for-production-beyond-llm-as-a-judge/" rel="noopener noreferrer"&gt;evaluating production AI systems&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/10-robotics-github-repos-3d-perception-vla-calibration/" rel="noopener noreferrer"&gt;robotics dataset evaluation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To learn how to implement complexity metrics and automated evaluation curves in Python, audit Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why is mean average precision (mAP) insufficient for evaluating production vision models?
&lt;/h3&gt;

&lt;p&gt;mAP averages performance across all validation samples, allowing high accuracy on clean, simple images to mathematically mask catastrophic failure on dense, cluttered scenes.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is visual clutter and how is it quantified?
&lt;/h3&gt;

&lt;p&gt;Visual clutter measures the density of task-irrelevant visual information, high-frequency texture transitions, and competing edge contours across a scene, typically quantified using subband entropy or edge density algorithms.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does instance crowding degrade object detection performance?
&lt;/h3&gt;

&lt;p&gt;When multiple objects overlap, bounding box proposals collide. Non-maximum suppression (NMS) algorithms struggle to differentiate between genuine adjacent instances and duplicate proposals of the same object, leading to false negatives.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where can engineers learn how to benchmark vision models against scene complexity?
&lt;/h3&gt;

&lt;p&gt;Comprehensive frameworks for complexity stratification and evaluation benchmarking are covered in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/scene-complexity-metrics-beyond-map/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>computervision</category>
      <category>ai</category>
      <category>datascience</category>
    </item>
    <item>
      <title>2026-Humanoid-IKEA-Assembly-Challenge: 35 Motors, Seven Cameras, in FiftyOne</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Wed, 11 Nov 2026 14:00:35 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/2026-humanoid-ikea-assembly-challenge-35-motors-seven-cameras-in-fiftyone-39i4</link>
      <guid>https://dev.to/harpreetsahota/2026-humanoid-ikea-assembly-challenge-35-motors-seven-cameras-in-fiftyone-39i4</guid>
      <description>&lt;p&gt;A humanoid assembling IKEA furniture logs 35 motors at once, and for each motor it records position, velocity, acceleration and torque. The 2026 Humanoid IKEA Assembly Challenge adds seven camera streams per episode, including infrared pairs from both wrists, and a human-annotated list of the subtasks the robot is working through. This FiftyOne version is a 6-episode, 30.7-minute subset, parsed so the torque plots sit on the same timeline as the wrist cameras.&lt;/p&gt;





&lt;h2&gt;
  
  
  What is in it
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://huggingface.co/datasets/Voxel51/2026-Humanoid-IKEA-Assembly-Challenge" rel="noopener noreferrer"&gt;2026 Humanoid IKEA Assembly Challenge subset&lt;/a&gt; comes from the &lt;a href="https://huggingface.co/datasets/BitRobot/2026-humanoid-ikea-assembly-challenge" rel="noopener noreferrer"&gt;BitRobot release&lt;/a&gt;. It keeps the shortest episode from each of six recording days spanning the collection window, and converts them to native multimodal MCAP episodes. Each of the 6 episodes runs 7,368 to 11,099 frames.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stream&lt;/th&gt;
&lt;th&gt;What it holds&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/head-camera&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Stereo head camera, 1280x480 side by side&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/left-wrist-camera&lt;/code&gt;, &lt;code&gt;/right-wrist-camera&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;RGB wrist views&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/left-wrist-ir1&lt;/code&gt;, &lt;code&gt;/left-wrist-ir2&lt;/code&gt;, &lt;code&gt;/right-wrist-ir1&lt;/code&gt;, &lt;code&gt;/right-wrist-ir2&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Infrared pairs from the wrist depth cameras&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/body-joint-state&lt;/code&gt;, &lt;code&gt;/body-joint-command&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;All 35 motors: position, velocity, acceleration, torque, each with a timeline plot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/left-gripper-state&lt;/code&gt;, &lt;code&gt;/right-gripper-state&lt;/code&gt; and command counterparts&lt;/td&gt;
&lt;td&gt;State and commands for the Dex1-1 hands&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/end-effector-state&lt;/code&gt;, &lt;code&gt;/end-effector-action&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;End-effector state and action, plus &lt;code&gt;/pivot.plot&lt;/code&gt; and &lt;code&gt;/gripper-controls.plot&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/ego-pose&lt;/code&gt;, &lt;code&gt;/odometry.plot&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Pose as &lt;code&gt;foxglove.PoseInFrame&lt;/code&gt;; body height, velocity and yaw rate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;/body-imu.plot&lt;/code&gt;, &lt;code&gt;/secondary-imu.plot&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;The two IMU plots&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/instruction&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The annotated subtask sequence&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each episode also has the fields &lt;code&gt;episode_name&lt;/code&gt;, &lt;code&gt;task&lt;/code&gt;, &lt;code&gt;recording_day&lt;/code&gt;, &lt;code&gt;subtasks&lt;/code&gt;, &lt;code&gt;num_subtasks&lt;/code&gt;, &lt;code&gt;num_frames&lt;/code&gt; and &lt;code&gt;duration&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The subtask sequence is the part that makes the telemetry readable. The labels run in this order: move to table, move table base, pick table leg, insert table leg to table base, rotate leg to tighten, rotate table base, flip table.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read torque against what the hands see
&lt;/h2&gt;

&lt;p&gt;Load the dataset and launch the App:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utils&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;huggingface&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fouh&lt;/span&gt;

&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fouh&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_from_hub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"Voxel51/2026-Humanoid-IKEA-Assembly-Challenge"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"2026-Humanoid-IKEA-Assembly-Challenge"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;persistent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Open an episode and play it. The wrist RGB views, the infrared pairs and the motor plots share one timeline, so you can scrub to the instruction change for "rotate leg to tighten" and check what the wrist camera shows while the arm torques move.&lt;/p&gt;

&lt;p&gt;To start with the longest recording, sort on a per-episode field:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;view&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"num_frames"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;view&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same pattern works on &lt;code&gt;duration&lt;/code&gt;, &lt;code&gt;num_subtasks&lt;/code&gt; or &lt;code&gt;recording_day&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Know the limits before you build on it
&lt;/h2&gt;

&lt;p&gt;Six episodes is a small sample. Each one is the shortest episode from its recording day, so the subset leans toward short runs and says little about how long assemblies behave. Use it to check the stream layout, build a viewer or test a parser. For a policy-training claim you need the full BitRobot release.&lt;/p&gt;

&lt;p&gt;The camera payloads are the source JPEGs passed through unchanged. The conversion from ROS 2 CDR to the FiftyOne MCAP flavor made one repair: the source recordings declare a &lt;code&gt;SportModeState&lt;/code&gt; schema that omits its trailing &lt;code&gt;path_point&lt;/code&gt; array, and the field is restored at decode so the odometry stream reads. If you decode the original files with a strict schema, expect that mismatch.&lt;/p&gt;

&lt;p&gt;The source dataset is released under &lt;a href="https://creativecommons.org/licenses/by/4.0/" rel="noopener noreferrer"&gt;CC-BY-4.0&lt;/a&gt;, and this subset uses the same license. For more robot-learning datasets, from arms to humanoids, see &lt;a href="https://employabledatascientist.com/blog/lerobot-v3-datasets-vla-training-fiftyone/" rel="noopener noreferrer"&gt;LeRobot v3.0 robot-learning datasets for VLA training, from Franka arms to humanoids&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For more of my FiftyOne dataset notes, see my &lt;a href="https://voxel51.com/author/harpreet" rel="noopener noreferrer"&gt;Voxel51 author page&lt;/a&gt;, and to curate your own robot data in the same tool, &lt;a href="https://link.voxel51.com/harpreet" rel="noopener noreferrer"&gt;try Voxel51&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is in the 2026 Humanoid IKEA Assembly Challenge subset?
&lt;/h3&gt;

&lt;p&gt;Six MCAP episodes of 7,368 to 11,099 frames each, 30.7 minutes in total. Each episode has seven camera streams, telemetry for 35 motors, gripper and end-effector streams, base odometry, two IMUs and the annotated subtask sequence.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which sensors does each episode record?
&lt;/h3&gt;

&lt;p&gt;The cameras are a stereo head camera, left and right wrist RGB cameras, and the infrared pairs from both wrist depth cameras. The body streams carry all 35 motors with position, velocity, acceleration and torque, and each has a timeline plot.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I load the dataset in FiftyOne?
&lt;/h3&gt;

&lt;p&gt;Call load_from_hub with Voxel51/2026-Humanoid-IKEA-Assembly-Challenge and persistent=True, then launch the App. Sort by num_frames to start with the longest episode.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which subtasks are annotated?
&lt;/h3&gt;

&lt;p&gt;The instruction stream holds a human-annotated sequence: move to table, move table base, pick table leg, insert table leg to table base, rotate leg to tighten, rotate table base, and flip table. Each episode also stores the sequence in a subtasks field and a num_subtasks count.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is this the full Humanoid IKEA Assembly Challenge?
&lt;/h3&gt;

&lt;p&gt;No. It is a six-episode subset of the BitRobot release, made of the shortest episode from each of six recording days. It is a sample for inspecting the format, not a training set.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/2026-humanoid-ikea-assembly-challenge-fiftyone-analysis/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>robotics</category>
      <category>visionmultimodal</category>
      <category>computervision</category>
      <category>python</category>
    </item>
    <item>
      <title>Why Duplicate Training Data Silently Corrupts Vision Models</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:30:42 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/why-duplicate-training-data-silently-corrupts-vision-models-3dbi</link>
      <guid>https://dev.to/harpreetsahota/why-duplicate-training-data-silently-corrupts-vision-models-3dbi</guid>
      <description>&lt;p&gt;What is worse than bad training data? Duplicate training data that silently corrupts your model while convincing you that performance is flawless.&lt;/p&gt;

&lt;p&gt;Every duplicate and near-duplicate image in your dataset amplifies sampling bias, wastes compute, and forces your model to overfit. Most dangerously, traditional validation metrics will not warn you. When near-identical frames exist across both your training and validation splits, evaluation metrics measure memorization rather than generalization.&lt;/p&gt;

&lt;p&gt;The result is an artificial spike in precision-recall curves, inflated mean average precision (mAP), and high confidence scores that unravel the moment the model encounters the real world.&lt;/p&gt;

&lt;p&gt;To prevent this silent failure, engineering teams must implement strategic, cluster-aware dataset pruning.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-----------------------------------------------------------------+
|          THE IMPACT OF DUPLICATE DATA ON VISION MODELS          |
+-----------------------------------------------------------------+
  [Train-Val Leakage]  --&amp;gt; Inflated validation metrics (false mAP)
  [Loss Imbalance]     --&amp;gt; Gradient updates over-index on duplicates
  [Diversity Loss]     --&amp;gt; Representation collapses to oversampled modes
+-----------------------------------------------------------------+
                                |
                                v
+-----------------------------------------------------------------+
|                    STRATEGIC PRUNING STEPS                      |
+-----------------------------------------------------------------+
  1. Cryptographic check for bitwise exact duplicates (MD5/SHA)
  2. Embedding similarity clustering for near-duplicate sets
  3. Selective pruning: preserve rare angles, remove redundancy
+-----------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Anatomy of Silent Corruption
&lt;/h2&gt;

&lt;p&gt;When video streams or burst captures are sampled without adequate temporal spacing, consecutive frames capture essentially identical geometry, lighting, and object postures.&lt;/p&gt;

&lt;p&gt;This causes three distinct structural failures:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Precision-Recall Curve Distortion:&lt;/strong&gt; When evaluating object detectors, predictions on memorized duplicate frames yield high-confidence true positives with zero localization error. These easy hits artificially inflate the area under the PR curve, masking severe failure modes on novel, diverse test samples.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feature Representation Collapse:&lt;/strong&gt; In self-supervised and supervised representation learning, gradient updates are dominated by the most frequent visual patterns. If 20% of your dataset consists of near-identical frames of an empty warehouse aisle, the model allocates disproportionate representational capacity to that specific background.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overconfidence on Minor Variations:&lt;/strong&gt; Models trained on redundant data fail to learn invariance to physical camera motion. Instead of learning that a forklift remains a forklift across continuous viewing angles, the model memorizes specific pixel configurations.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Exact Duplicates vs. Semantic Near-Duplicates
&lt;/h2&gt;

&lt;p&gt;Deduplication requires two complementary detection strategies:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Exact Bitwise Duplicates
&lt;/h3&gt;

&lt;p&gt;These are identical files or duplicate frame exports. They can be detected in linear time using cryptographic hashing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;find_exact_duplicates&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_paths&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;hashes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;duplicates&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;image_paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;hasher&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;md5&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rb"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;hasher&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;digest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hasher&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;digest&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;duplicates&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;duplicates&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exact bitwise duplicates should be pruned immediately without hesitation.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Semantic Near-Duplicates
&lt;/h3&gt;

&lt;p&gt;Near-duplicates share the same physical scene but vary slightly in camera position, minor illumination changes, compression artifacts, or subtle foreground movement. Cryptographic hashes fail completely on these samples.&lt;/p&gt;

&lt;p&gt;Detecting near-duplicates requires computing pairwise cosine similarity across dense feature embeddings:&lt;/p&gt;

&lt;p&gt;$$\text{Sim}(u, v) = \frac{u \cdot v}{|u| |v|}$$&lt;/p&gt;

&lt;p&gt;Using FiftyOne, you can identify and group near-duplicate clusters efficiently:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;brain&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fob&lt;/span&gt;

&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"surveillance_feed_raw"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Index embeddings for similarity search&lt;/span&gt;
&lt;span class="n"&gt;fob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compute_similarity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"clip-vit-base32-torch"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;brain_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"img_sim"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Find near-duplicate pairs with cosine similarity &amp;gt; 0.98&lt;/span&gt;
&lt;span class="n"&gt;duplicate_map&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_duplicates&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;similarity_threshold&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.98&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;brain_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"img_sim"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"Identified {len(duplicate_map)} near-duplicate groups."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Trap of Blind Deduplication: Why Selective Pruning Matters
&lt;/h2&gt;

&lt;p&gt;A common operational error is running an automated script that deletes all but one frame from every near-duplicate cluster.&lt;/p&gt;

&lt;p&gt;Blind pruning frequently destroys valuable edge cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In continuous video, an object transition (e.g. an assembly line component shifting alignment or a pedestrian starting to cross) may occur within a near-duplicate sequence.&lt;/li&gt;
&lt;li&gt;Near-duplicate sequences often contain subtle lighting variations or extreme camera angles that provide valuable regularization.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Strategic Pruning Rules
&lt;/h3&gt;

&lt;p&gt;Follow a disciplined three-phase pruning protocol:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Prune exact duplicates first:&lt;/strong&gt; Remove identical files and verify that no sample ID exists in both training and test splits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cluster near-duplicates:&lt;/strong&gt; Group samples with similarity $&amp;gt; 0.96$ into explicit equivalence classes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Apply diversity-preserving selection:&lt;/strong&gt; Within each near-duplicate cluster, select representatives that maximize visual diversity (e.g., the frame with the highest contrast, the sharpest focus, or the most extreme viewing angle) and prune the redundant interior frames.&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Deduplication Stage&lt;/th&gt;
&lt;th&gt;Target Similarity&lt;/th&gt;
&lt;th&gt;Detection Method&lt;/th&gt;
&lt;th&gt;Preservation Rule&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Exact Clones&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$1.00$ bitwise&lt;/td&gt;
&lt;td&gt;MD5 / SHA-256 Hashing&lt;/td&gt;
&lt;td&gt;Purge completely&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Near Clones&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$&amp;gt; 0.985$ cosine&lt;/td&gt;
&lt;td&gt;Embedding similarity&lt;/td&gt;
&lt;td&gt;Keep 1 representative per burst&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Semantic Siblings&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.94 - 0.98$ cosine&lt;/td&gt;
&lt;td&gt;Clustering (DBSCAN/k-NN)&lt;/td&gt;
&lt;td&gt;Select frames with diverse angles/lighting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Distinct Scenes&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$&amp;lt; 0.90$ cosine&lt;/td&gt;
&lt;td&gt;Baseline distribution&lt;/td&gt;
&lt;td&gt;Preserve all valid instances&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Building Reliable Production Datasets
&lt;/h2&gt;

&lt;p&gt;Eliminating redundant duplicates restores integrity to your evaluation splits, ensures that gradient updates reflect genuine visual variance, and cuts training time without sacrificing performance.&lt;/p&gt;

&lt;p&gt;For companion reading on dataset curation and evaluation, see our articles on &lt;a href="https://employabledatascientist.com/blog/robot-data-quality-scoring/" rel="noopener noreferrer"&gt;finding edited copies with pHash and FiftyOne&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/evaluating-ai-agents-for-production-beyond-llm-as-a-judge/" rel="noopener noreferrer"&gt;evaluating AI agent pipelines&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To master hands-on dataset curation and pruning workflows, audit Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How do duplicate images cause train-test data leakage?
&lt;/h3&gt;

&lt;p&gt;When identical or near-duplicate frames appear in both training and test splits, the model is evaluated on samples it has already memorized. This creates artificially inflated validation metrics that collapse upon live deployment.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does duplicate data inflate mean average precision (mAP)?
&lt;/h3&gt;

&lt;p&gt;Duplicates create artificial clusters in high-confidence regions of precision-recall curves. The model repeatedly scores easy True Positives on memorized duplicate samples, lifting aggregate area under the curve despite poor generalization on novel images.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between exact duplicates and near-duplicates?
&lt;/h3&gt;

&lt;p&gt;Exact duplicates have identical pixel values and can be detected using MD5 or SHA-256 hashes. Near-duplicates share identical scenes with minor variations in crop, lighting, compression, or sensor angle, requiring perceptual hashing or embedding cosine distance to detect.&lt;/p&gt;

&lt;h3&gt;
  
  
  How can teams implement selective dataset pruning?
&lt;/h3&gt;

&lt;p&gt;Techniques for cluster-based deduplication and strategic pruning are demonstrated step-by-step in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/duplicate-training-data-silent-model-corruption/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>computervision</category>
      <category>datascience</category>
      <category>ai</category>
    </item>
    <item>
      <title>Why Open Weight Models Matter for Small Businesses</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:00:36 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/why-open-weight-models-matter-for-small-businesses-32ni</link>
      <guid>https://dev.to/harpreetsahota/why-open-weight-models-matter-for-small-businesses-32ni</guid>
      <description>&lt;h1&gt;
  
  
  Why Open Weight Models Matter for Small Businesses
&lt;/h1&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Open weight models are AI models you can download, inspect, modify, and run on hardware you control.&lt;/li&gt;
&lt;li&gt;They matter for small businesses because they remove per-token pricing, vendor lock-in, and the requirement to send sensitive data to a third party.&lt;/li&gt;
&lt;li&gt;The real barrier is the system around the model: hardware, evaluation, integration, and maintenance.&lt;/li&gt;
&lt;li&gt;The distance between downloading a model and having a working system is where most small businesses stall.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;In July 2026, 26 organizations including Microsoft, NVIDIA, Meta, and Hugging Face signed an open letter framing open weight models as a foundation for competitive AI ecosystems. The letter drew a parallel to open-source software in the 1980s.&lt;/p&gt;

&lt;p&gt;The macro argument is about national competitiveness. The practical argument is about what happens when a small business tries to use one of these models.&lt;/p&gt;

&lt;p&gt;Open weight models are accessible. Anyone can download a capable model from Hugging Face for free. Almost no small business can turn that download into a reliable, evaluated system integrated with their actual workflows.&lt;/p&gt;

&lt;h2&gt;
  
  
  Table of Contents
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;What Open Weight Models Are&lt;/li&gt;
&lt;li&gt;The Open-Source Parallel&lt;/li&gt;
&lt;li&gt;Why Small Businesses Should Care&lt;/li&gt;
&lt;li&gt;The Real Challenges&lt;/li&gt;
&lt;li&gt;From Download to Working System&lt;/li&gt;
&lt;li&gt;Summary&lt;/li&gt;
&lt;li&gt;Next Steps&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What Open Weight Models Are
&lt;/h2&gt;

&lt;p&gt;An open weight model is a trained AI model whose parameters (the weights) are published so anyone can download, inspect, modify, and run them.&lt;/p&gt;

&lt;p&gt;In practical terms, this means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You download the model files. You own the copy.&lt;/li&gt;
&lt;li&gt;You run it on hardware you control. No API call to a third party.&lt;/li&gt;
&lt;li&gt;You inspect the model's behavior, test it on your data, and adapt it.&lt;/li&gt;
&lt;li&gt;You pay for compute, not for tokens.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is distinct from a hosted or closed model, where you send requests to a provider's API and pay per use. With a closed model, the provider controls the model, the pricing, the data path, and the update schedule.&lt;/p&gt;

&lt;p&gt;Open weight does not always mean fully open. Some models publish weights but not training data or code. Licensing varies. The core property that matters here is that you can run the model yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Open-Source Parallel
&lt;/h2&gt;

&lt;p&gt;The open letter signed by Microsoft, NVIDIA, and others compared open weight models to the open-source software movement of the 1980s.&lt;/p&gt;

&lt;p&gt;The parallel is worth taking seriously.&lt;/p&gt;

&lt;p&gt;Open-source software did not win by being better than every commercial alternative. It won by being diffusable and adaptable. Linux is not the best operating system for every workload. It is the operating system you can inspect, modify, and deploy without asking permission. That property let it spread into servers, embedded systems, phones, and infrastructure worldwide.&lt;/p&gt;

&lt;p&gt;Open weight models have a similar property. A 7-billion-parameter model is not a frontier model. But it is a model you can run on a single GPU, fine-tune on your data, and embed in a product without a vendor relationship. For many business tasks (classification, summarization, extraction, search, drafting), that is enough capability and the right kind of control.&lt;/p&gt;

&lt;p&gt;The argument from the letter is that this diffusion into every sector is what builds a competitive ecosystem. The argument for a small business is simpler: it gives you options you did not have before.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Small Businesses Should Care
&lt;/h2&gt;

&lt;p&gt;Four reasons matter for small and local businesses.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cost Predictability
&lt;/h3&gt;

&lt;p&gt;Hosted model pricing scales with usage. For a business with variable or growing workloads, that means unpredictable costs. A per-token API bill that doubles when usage doubles is a budget problem disguised as a usage metric.&lt;/p&gt;

&lt;p&gt;Open weight models flip this. You pay for the hardware to run the model. The model itself is free or low-cost. Your cost is fixed and predictable. If you run 10,000 inferences or 10 million, the model cost is the same.&lt;/p&gt;

&lt;p&gt;For a business doing repetitive AI work (document processing, support triage, data extraction), this is the difference between a line item and a liability.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Control
&lt;/h3&gt;

&lt;p&gt;When you call a hosted model API, your input data travels to the provider. For many businesses, that is acceptable. For businesses handling sensitive client data in legal, healthcare, finance, or insurance, it may not be.&lt;/p&gt;

&lt;p&gt;Open weight models run on your infrastructure. The data never leaves your environment. You control what is logged, what is retained, and who can access it. For regulated industries or businesses with confidentiality obligations, this is a requirement, not a preference.&lt;/p&gt;

&lt;h3&gt;
  
  
  No Vendor Lock-In
&lt;/h3&gt;

&lt;p&gt;A hosted model provider can change pricing, deprecate a model, change terms of service, or shut down. When your product depends on that provider, you absorb the risk.&lt;/p&gt;

&lt;p&gt;With an open weight model, you control the model. If you need to switch to a different model, you can. If you need to keep running the same model for years, you can. Nobody deprecates a model you have already downloaded.&lt;/p&gt;

&lt;h3&gt;
  
  
  Right-Sizing
&lt;/h3&gt;

&lt;p&gt;Most business tasks do not need a frontier model.&lt;/p&gt;

&lt;p&gt;Document classification, named entity extraction, summarization, semantic search, draft generation, form pre-filling. These are well-served by models in the 3B to 13B parameter range. These models run on a single consumer or mid-range GPU. They are fast, cheap to operate, and good enough for production if you evaluate them properly.&lt;/p&gt;

&lt;p&gt;The frontier models get the press. The workhorse models do the work.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Real Challenges
&lt;/h2&gt;

&lt;p&gt;Open weight models are accessible. Deploying them well takes work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Hardware
&lt;/h3&gt;

&lt;p&gt;You need hardware to run the model. The requirements depend on model size.&lt;/p&gt;

&lt;p&gt;A 3B parameter model in 4-bit quantization can run on a modern laptop with 8GB of VRAM or even CPU-only with acceptable latency for low-volume use. A 7B model wants 8 to 12GB of VRAM. A 13B model wants 16 to 24GB. A 70B model wants multiple high-end GPUs.&lt;/p&gt;

&lt;p&gt;For a small business, the realistic starting point is a single workstation GPU or a cloud GPU instance. The cost is manageable. The decision about which hardware to buy or rent requires understanding the model size, quantization options, and expected load.&lt;/p&gt;

&lt;p&gt;Learning this is not hard. Learning it by accident is. Most businesses do not have someone who knows the difference between FP16, INT8, and 4-bit quantization, or why it matters for their use case.&lt;/p&gt;

&lt;h3&gt;
  
  
  Expertise
&lt;/h3&gt;

&lt;p&gt;Running an open weight model in production means making several decisions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which model to use (size, license, modality, context length)&lt;/li&gt;
&lt;li&gt;Which quantization format (GGUF, AWQ, GPTQ) and what it costs in quality&lt;/li&gt;
&lt;li&gt;Which inference server to run (vLLM, llama.cpp, TGI, Ollama)&lt;/li&gt;
&lt;li&gt;How to serve it (API endpoint, batch processing, embedded library)&lt;/li&gt;
&lt;li&gt;How to handle concurrent requests and throughput&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each of these is a solvable problem. Together, they form a knowledge barrier. A business that has never deployed a model before will hit this wall quickly. The information is available, but assembling it into a working deployment requires either time or help.&lt;/p&gt;

&lt;h3&gt;
  
  
  Evaluation
&lt;/h3&gt;

&lt;p&gt;A hosted model provider handles evaluation internally. You trust their benchmarks and their model improvements. When you run your own model, you are responsible for knowing whether it works.&lt;/p&gt;

&lt;p&gt;Evaluation means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Building a test set of representative inputs and expected outputs&lt;/li&gt;
&lt;li&gt;Running the model against that set&lt;/li&gt;
&lt;li&gt;Measuring quality against a defined standard&lt;/li&gt;
&lt;li&gt;Re-running evaluation when you change models, prompts, or parameters&lt;/li&gt;
&lt;li&gt;Monitoring production outputs for drift or degradation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most businesses do not have this capability. It is a measurement design problem, not a tooling problem. You need to define what "good" means for your specific task, collect examples, and measure against that standard repeatedly.&lt;/p&gt;

&lt;p&gt;Evaluation is the part of AI that teams underbuild first and regret first.&lt;/p&gt;

&lt;h3&gt;
  
  
  Maintenance
&lt;/h3&gt;

&lt;p&gt;Models do not maintain themselves.&lt;/p&gt;

&lt;p&gt;Open weight models are static artifacts. They do not update when the world changes. If your business data, language, or requirements shift, the model's performance may degrade without warning.&lt;/p&gt;

&lt;p&gt;Maintenance means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Monitoring output quality over time&lt;/li&gt;
&lt;li&gt;Re-evaluating after business changes&lt;/li&gt;
&lt;li&gt;Swapping or fine-tuning models when needed&lt;/li&gt;
&lt;li&gt;Keeping the inference stack updated and secure&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A hosted provider handles this invisibly. When you run your own model, this becomes your responsibility.&lt;/p&gt;

&lt;h2&gt;
  
  
  From Download to Working System
&lt;/h2&gt;

&lt;p&gt;The distance between downloading a model and having a working system is the entire distance between a tool and a product.&lt;/p&gt;

&lt;p&gt;Any business can go to Hugging Face, search for a model, and download it. The model is free. The download is easy.&lt;/p&gt;

&lt;p&gt;Almost no small business can do the next steps alone:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Assess the task and pick the right model.&lt;/li&gt;
&lt;li&gt;Provision appropriate hardware.&lt;/li&gt;
&lt;li&gt;Configure quantization and inference serving.&lt;/li&gt;
&lt;li&gt;Build an evaluation suite and define quality bars.&lt;/li&gt;
&lt;li&gt;Integrate the model into existing workflows and systems.&lt;/li&gt;
&lt;li&gt;Monitor performance, maintain the deployment, and adapt when things change.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is the work. The model is a component. The system around the model is the product.&lt;/p&gt;

&lt;p&gt;The open letter from Microsoft and NVIDIA argues that open weights will diffuse AI into "factories, hospitals, farms, classrooms, and main street businesses." That diffusion will not happen because the models are free. It will happen because someone builds the systems that make the models useful in those settings.&lt;/p&gt;

&lt;p&gt;For a small business, that someone is either an internal hire who learns the stack or a consultant who already knows it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Open weight models give small businesses access to capable AI without per-token pricing, vendor lock-in, or data exposure.&lt;/p&gt;

&lt;p&gt;The barrier is the system around the model: hardware, evaluation, integration, and maintenance. These are solvable problems, but they require expertise most small businesses do not have and should not need to build from scratch.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;If you are a small business exploring AI, start with the task, not the model. Define what you want the system to do, what data it will touch, and what quality bar it needs to meet. That definition tells you whether an open weight model fits and which one to start with.&lt;/p&gt;

&lt;p&gt;If you need help with that assessment, whether model selection, deployment, evaluation, or integration, we do this work. &lt;a href="https://employabledatascientist.com/contact/" rel="noopener noreferrer"&gt;Get in touch&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is an open weight model and how does it differ from a hosted model?
&lt;/h3&gt;

&lt;p&gt;An open weight model is a trained AI model whose weights are published so anyone can download, inspect, modify, and run it. You pay for compute, not tokens, while a hosted provider controls the model, pricing, data path, and update schedule. Open weight does not always mean fully open.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why do open weight models matter for small businesses?
&lt;/h3&gt;

&lt;p&gt;Four reasons matter: cost predictability, data control, no vendor lock-in, and right-sizing. Costs become fixed hardware costs instead of per-token bills, data stays on your infrastructure, nobody can deprecate a model you downloaded, and most business tasks do not need a frontier model.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much hardware does running an open weight model need at different sizes?
&lt;/h3&gt;

&lt;p&gt;A 3B parameter model in 4-bit quantization can run on a modern laptop with 8GB of VRAM or even CPU-only for low-volume use. A 7B model wants 8 to 12GB of VRAM, a 13B model wants 16 to 24GB, and a 70B model wants multiple high-end GPUs.&lt;/p&gt;

&lt;h3&gt;
  
  
  What makes deploying open weight models harder than downloading them?
&lt;/h3&gt;

&lt;p&gt;The barrier is the system around the model: hardware, expertise, evaluation, and maintenance. Open weight models are static artifacts that can degrade without warning, and running your own model makes you responsible for knowing whether it works, which hosted providers handle internally.&lt;/p&gt;

&lt;h3&gt;
  
  
  What steps turn a downloaded open weight model into a working system?
&lt;/h3&gt;

&lt;p&gt;Six steps: assess the task and pick the model, provision hardware, configure quantization and inference serving, build an evaluation suite with quality bars, integrate into existing workflows, and monitor and maintain the deployment. The model is a component, and the system around it is the product.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/why-open-weight-models-matter-for-small-businesses/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>openweightmodels</category>
      <category>machinelearning</category>
      <category>testing</category>
    </item>
    <item>
      <title>Detecting Hidden Outliers in Visual AI Datasets with Embeddings and UMAP</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Mon, 09 Nov 2026 14:30:30 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/detecting-hidden-outliers-in-visual-ai-datasets-with-embeddings-and-umap-59f6</link>
      <guid>https://dev.to/harpreetsahota/detecting-hidden-outliers-in-visual-ai-datasets-with-embeddings-and-umap-59f6</guid>
      <description>&lt;p&gt;Your model's most dangerous vulnerability is rarely a sub-optimal hyperparameter configuration. It is hidden outliers lurking undetected inside your training and evaluation data.&lt;/p&gt;

&lt;p&gt;In production, vision models frequently fail on rare scenarios that were never surfaced during development: unusual sensor artifacts, unrepresented object variants, corrupted metadata, or bizarre edge-case interactions. Traditional statistical checks—like tracking image resolution, file size, or mean pixel values—fail to detect semantic and compositional anomalies.&lt;/p&gt;

&lt;p&gt;By leveraging foundation model embeddings, density-based anomaly algorithms like Local Outlier Factor (LOF), and dimensionality reduction with UMAP, you can systematically reveal hidden dataset structure and isolate rogue outliers before training begins.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       EMBEDDING-BASED OUTLIER DETECTION PIPELINE
+-------------------------------------------------------------+
| Raw Image Dataset                                           |
|       |                                                     |
|       v                                                     |
| Foundation Model Embedding (CLIP / DINOv2 / SigLIP)          |
|       |                                                     |
|       +-----------------------------+                       |
|       |                             |                       |
|       v                             v                       |
| Local Outlier Factor (LOF)    UMAP 2D Projection            |
| [Quantitative Anomaly Score]  [Visual Cluster Exploration]  |
|       |                             |                       |
|       +-----------------------------+                       |
|       v                                                     |
| Visual Inspection &amp;amp; Triage in FiftyOne                      |
+-------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Three Classes of Visual Outliers
&lt;/h2&gt;

&lt;p&gt;When inspecting anomalies in visual datasets, outliers typically fall into three distinct technical categories:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Quality and Sensor Anomalies:&lt;/strong&gt; Corrupted frames, extreme lens flare, severe compression blockiness, dropped video packets, or infrared sensor bleed. These represent data pipeline defects that should almost always be scrubbed or quarantined.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Semantic and Content Anomalies:&lt;/strong&gt; Rare, valid physical scenarios that occur with low frequency: an overturned truck on a highway, an animal running across an industrial floor, or a worker wearing non-standard personal protective equipment. These are valuable operational edge cases that must be preserved and reinforced.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Annotation and Formatting Bugs:&lt;/strong&gt; An image tagged with a label from an entirely different ontology, bounding boxes scaled incorrectly, or labels assigned to inverted coordinate axes.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Identifying which category an outlier belongs to determines whether you prune the sample, fix its metadata, or curate more examples to support it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Extracting Dense Semantic Embeddings
&lt;/h2&gt;

&lt;p&gt;Pixel-level distances cannot distinguish between a dark cat and a black leather jacket. Foundation vision models trained via self-supervised learning (DINOv2) or contrastive language-image pretraining (CLIP, SigLIP) solve this by encoding high-level semantic, structural, and stylistic properties into dense feature vectors.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fiftyone.brain&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fob&lt;/span&gt;

&lt;span class="c1"&gt;# Load dataset
&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;industrial_inspection_v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Compute embeddings using a foundation vision backbone
&lt;/span&gt;&lt;span class="n"&gt;fob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compute_similarity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clip-vit-base32-torch&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;embeddings_field&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clip_embeddings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;brain_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clip_sim&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Computed dense embeddings across all dataset samples.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Density-Based Anomaly Scoring with Local Outlier Factor
&lt;/h2&gt;

&lt;p&gt;A common mistake in outlier detection is calculating global Euclidean or Mahalanobis distance from the dataset centroid. Visual datasets are inherently multimodal: images of outdoor scenes form one dense cluster, while indoor scenes form another with completely different local density.&lt;/p&gt;

&lt;p&gt;Local Outlier Factor (LOF) solves this by comparing the local density of an image's embedding to the density of its $k$-nearest neighbors:&lt;/p&gt;

&lt;p&gt;$$\text{LOF}&lt;em&gt;k(p) = \frac{\sum&lt;/em&gt;{o \in N_k(p)} \frac{\text{lrd}(o)}{\text{lrd}(p)}}{|N_k(p)|}$$&lt;/p&gt;

&lt;p&gt;Where $\text{lrd}(p)$ is the local reachability density of point $p$. An LOF score substantially greater than $1.0$ indicates that the sample resides in a sparse region relative to its neighbors, flagging it as an anomaly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn.neighbors&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LocalOutlierFactor&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="c1"&gt;# Extract embeddings array from FiftyOne dataset
&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clip_embeddings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="c1"&gt;# Fit Local Outlier Factor
&lt;/span&gt;&lt;span class="n"&gt;lof&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LocalOutlierFactor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_neighbors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;contamination&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.02&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;outlier_labels&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit_predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;anomaly_scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;negative_outlier_factor_&lt;/span&gt;

&lt;span class="c1"&gt;# Store anomaly scores back onto the dataset
&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;outlier_score&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;anomaly_scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tolist&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="c1"&gt;# Query the top 1% most severe anomalies
&lt;/span&gt;&lt;span class="n"&gt;outliers_view&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;outlier_score&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Isolated top &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;outliers_view&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; outlier candidates for inspection.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Visualizing Dataset Manifolds with UMAP
&lt;/h2&gt;

&lt;p&gt;Quantitative anomaly scores tell you &lt;em&gt;which&lt;/em&gt; images are unusual, but dimensionality reduction tells you &lt;em&gt;why&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Uniform Manifold Approximation and Projection (UMAP) preserves both local and global neighborhood structures when projecting high-dimensional embeddings down to a 2D plane:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Compute 2D UMAP visualization coordinates
&lt;/span&gt;&lt;span class="n"&gt;fob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compute_visualization&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clip_embeddings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;umap&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;brain_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;umap_2d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Launch FiftyOne interactive app to inspect clusters
&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the interactive 2D scatter plot:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tight, dense clusters&lt;/strong&gt; represent standard, well-represented operational scenarios.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Isolated satellites and rogue points&lt;/strong&gt; floating outside major clusters pinpoint rare outliers, mislabeled samples, and corrupted images.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Outlier Type&lt;/th&gt;
&lt;th&gt;UMAP Geometry&lt;/th&gt;
&lt;th&gt;Typical Root Cause&lt;/th&gt;
&lt;th&gt;Action Plan&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Pipeline Defect&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Far off the manifold, isolated single point&lt;/td&gt;
&lt;td&gt;Corrupted sensor frame, black frame&lt;/td&gt;
&lt;td&gt;Quarantine and remove from training set&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Annotation Error&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Sits inside Cluster A, but labeled as Class B&lt;/td&gt;
&lt;td&gt;Human labeler typo or ambiguous definition&lt;/td&gt;
&lt;td&gt;Correct ground truth annotation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Operational Edge Case&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Small cluster of 3–5 points detached from main body&lt;/td&gt;
&lt;td&gt;Rare physical scenario in the real world&lt;/td&gt;
&lt;td&gt;Preserve, balance, and collect companion cases&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Turning Anomaly Detection into Practice
&lt;/h2&gt;

&lt;p&gt;Running outlier detection during data ingestion prevents corrupted frames from skewing model weights and ensures that critical real-world edge cases receive explicit attention.&lt;/p&gt;

&lt;p&gt;To explore related evaluation workflows, review our guide on &lt;a href="https://employabledatascientist.com/blog/curating-golden-trajectories-adversarial-agent-red-teaming/" rel="noopener noreferrer"&gt;curating golden trajectories for red-teaming&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/boilingbench-multimodal-fiftyone-analysis/" rel="noopener noreferrer"&gt;analyzing vision datasets&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To get hands-on experience running embedding searches, LOF audits, and UMAP clustering on real datasets, audit Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What makes visual outliers dangerous in machine learning datasets?
&lt;/h3&gt;

&lt;p&gt;Visual outliers can corrupt model decision boundaries, cause catastrophic gradients during training, or represent critical real-world edge cases that models fail to recognize in production. Detecting them early prevents both training instability and operational blind spots.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why are foundation model embeddings better than pixel metrics for outlier detection?
&lt;/h3&gt;

&lt;p&gt;Raw pixel comparisons only capture low-level color and brightness shifts. Foundation model embeddings (such as CLIP, DINOv2, or SigLIP) map images into a structured semantic space where distance reflects conceptual, compositional, and contextual similarity.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does Local Outlier Factor (LOF) differ from simple Euclidean distance?
&lt;/h3&gt;

&lt;p&gt;Global Euclidean distance assumes uniform density across the dataset. Local Outlier Factor compares the local density of an object to the local densities of its k-nearest neighbors, correctly identifying anomalies even in regions with variable cluster densities.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where can developers practice embedding-based curation workflows?
&lt;/h3&gt;

&lt;p&gt;Complete hands-on code examples and guided dataset curation labs are available in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/detecting-hidden-outliers-visual-ai-embeddings/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>computervision</category>
      <category>python</category>
      <category>datascience</category>
    </item>
    <item>
      <title>EAGLE-3 Speculative Decoding in vLLM for Qwen3-30B-A3B-Instruct-2507</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Mon, 09 Nov 2026 14:00:23 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/eagle-3-speculative-decoding-in-vllm-for-qwen3-30b-a3b-instruct-2507-5h1d</link>
      <guid>https://dev.to/harpreetsahota/eagle-3-speculative-decoding-in-vllm-for-qwen3-30b-a3b-instruct-2507-5h1d</guid>
      <description>&lt;h1&gt;
  
  
  EAGLE-3 Speculative Decoding in vLLM for Qwen3-30B-A3B-Instruct-2507
&lt;/h1&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The old notebook version of this workflow is no longer the best way to explain EAGLE-3: it is tied to Colab, an older Qwen3-32B pairing, and a one-off benchmark run.&lt;/li&gt;
&lt;li&gt;A cleaner current setup is &lt;code&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/code&gt; plus &lt;code&gt;RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3&lt;/code&gt;, served through vLLM with the model-specific &lt;code&gt;eagle3&lt;/code&gt; draft configuration.&lt;/li&gt;
&lt;li&gt;The tokenizer and chat-template side still works cleanly in &lt;code&gt;transformers==5.12.1&lt;/code&gt;, which makes it a good companion stack for client-side prompt rendering and sanity checks.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;Speculative decoding is one of the few inference optimizations that can make a large model feel meaningfully faster without changing its output distribution. The original draft behind this post had the right subject, but it had the wrong shape for 2026: a Colab export, a stale verifier-speculator pairing, and a benchmark section that mattered less than the serving path itself. This rewrite keeps the important idea and updates everything else. You will see what EAGLE-3 is doing, why &lt;code&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/code&gt; is a better current example than the older Qwen3-32B setup, how to launch the matching Red Hat AI speculator in vLLM, and how to sanity-check the tokenizer flow with &lt;code&gt;transformers==5.12.1&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Requirements
&lt;/h2&gt;

&lt;p&gt;Use this setup if you want to follow the post as written:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.10+&lt;/li&gt;
&lt;li&gt;&lt;code&gt;transformers==5.12.1&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;requests&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;a current &lt;code&gt;vllm&lt;/code&gt; build with EAGLE-3 speculative decoding support&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You also need enough GPU memory for the verifier model you choose. &lt;code&gt;Qwen3-30B-A3B-Instruct-2507&lt;/code&gt; is still a serious model even though the speculator itself is small.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;You are comfortable running shell commands and short Python scripts.&lt;/li&gt;
&lt;li&gt;You understand the difference between a verifier model and a draft or speculator model.&lt;/li&gt;
&lt;li&gt;You have used an OpenAI-compatible endpoint before, or can follow a &lt;code&gt;curl&lt;/code&gt; example.&lt;/li&gt;
&lt;li&gt;You are testing an interactive or low-concurrency workload where speculative decoding can actually help.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Table of Contents
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Problem&lt;/li&gt;
&lt;li&gt;Background&lt;/li&gt;
&lt;li&gt;Approach&lt;/li&gt;
&lt;li&gt;Example&lt;/li&gt;
&lt;li&gt;Failure Modes &amp;amp; Caveats&lt;/li&gt;
&lt;li&gt;Practical Guidance&lt;/li&gt;
&lt;li&gt;Summary&lt;/li&gt;
&lt;li&gt;Next Steps&lt;/li&gt;
&lt;li&gt;Resources&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Problem
&lt;/h2&gt;

&lt;p&gt;Notebook exports age badly when the ecosystem around them moves fast.&lt;/p&gt;

&lt;p&gt;That is exactly what happened to the original file. It still had the bones of a useful post, but it mixed blog prose with Colab boilerplate, installed everything in one cell, and centered the story on &lt;code&gt;Qwen/Qwen3-32B&lt;/code&gt; plus a benchmark snippet instead of the real question a reader cares about:&lt;/p&gt;

&lt;p&gt;How do you run a current EAGLE-3 speculator with vLLM in a way that is easy to understand, easy to verify, and still compatible with the modern &lt;code&gt;transformers&lt;/code&gt; stack?&lt;/p&gt;

&lt;p&gt;That is the problem this rewrite solves.&lt;/p&gt;

&lt;h2&gt;
  
  
  Background
&lt;/h2&gt;

&lt;p&gt;Speculative decoding uses two models together:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A small draft model proposes several next tokens.&lt;/li&gt;
&lt;li&gt;The large verifier model checks those tokens in parallel.&lt;/li&gt;
&lt;li&gt;Correct draft tokens are accepted, and the final output stays consistent with normal verifier-only decoding.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The reason people care is simple: large models are usually slow because generation is sequential. If the draft model guesses well, the verifier can accept multiple tokens per pass instead of only one.&lt;/p&gt;

&lt;p&gt;EAGLE-3 is the current high-water mark in the EAGLE family because it moves beyond earlier feature-prediction approaches and uses multi-layer fusion with direct token prediction. In practice, the important part is not the paper vocabulary. It is that EAGLE-3 has matured into a deployable path in vLLM, with published draft checkpoints that are specific to particular verifier models.&lt;/p&gt;

&lt;p&gt;That model-specific coupling matters. An EAGLE-3 draft model is not a generic accelerator. It is trained to match one verifier family and often one exact verifier checkpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  Approach
&lt;/h2&gt;

&lt;p&gt;For this rewrite, the best current compromise between freshness and practical clarity is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Verifier: &lt;code&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Speculator: &lt;code&gt;RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This pairing is better than the older notebook example for three reasons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;It reflects a more current Qwen release than the original draft.&lt;/li&gt;
&lt;li&gt;It has an official EAGLE-3 speculator model card with a direct vLLM serve recipe.&lt;/li&gt;
&lt;li&gt;Its tokenizer and chat-template flow still behave cleanly in &lt;code&gt;transformers==5.12.1&lt;/code&gt;, which makes the surrounding Python examples easier to trust.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The post keeps the serving path deliberately boring:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Verify the tokenizer and chat template in &lt;code&gt;transformers&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Launch vLLM with the matching &lt;code&gt;eagle3&lt;/code&gt; speculator.&lt;/li&gt;
&lt;li&gt;Send one OpenAI-compatible chat request.&lt;/li&gt;
&lt;li&gt;Measure whether the workload is actually the kind that benefits from speculative decoding.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Example
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Install a current stack
&lt;/h3&gt;

&lt;p&gt;Keep the client-side dependencies lean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="s2"&gt;"transformers==5.12.1"&lt;/span&gt; requests vllm
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you already manage &lt;code&gt;vllm&lt;/code&gt; separately, the main thing to preserve from this post is the &lt;code&gt;transformers==5.12.1&lt;/code&gt; tokenizer path and the exact verifier-speculator pairing.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Sanity-check the tokenizer path in Transformers
&lt;/h3&gt;

&lt;p&gt;This is a small check, but it is worth doing because it proves the tutorial is aligned with the current &lt;code&gt;transformers&lt;/code&gt; API instead of only pasting model-card commands.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;__future__&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;annotations&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;


&lt;span class="n"&gt;MODEL_ID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;PROMPT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Explain speculative decoding in one sentence.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Render a Qwen chat prompt with the current Transformers stack.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL_ID&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PROMPT&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;
    &lt;span class="n"&gt;rendered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply_chat_template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;tokenize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;add_generation_prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rendered&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a healthy setup, the rendered prompt should use the Qwen chat markers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;&amp;lt;|im_start|&amp;gt;user&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;your prompt text&lt;/li&gt;
&lt;li&gt;&lt;code&gt;&amp;lt;|im_start|&amp;gt;assistant&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the exact behavior I wanted to preserve from the old Qwen-based post while moving the model choice forward.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Launch vLLM with the EAGLE-3 speculator
&lt;/h3&gt;

&lt;p&gt;The current long-form command is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-tp&lt;/span&gt; 1 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--speculative-config&lt;/span&gt; &lt;span class="s1"&gt;'{
    "model": "RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3",
    "num_speculative_tokens": 3,
    "method": "eagle3"
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example sticks to the published default of &lt;code&gt;3&lt;/code&gt; speculative tokens because that is the safest place to start. The speculator is small, but the verifier is not, so adjust tensor parallelism to match your hardware rather than copying &lt;code&gt;-tp 1&lt;/code&gt; blindly.&lt;/p&gt;

&lt;p&gt;If your vLLM build supports model-embedded &lt;code&gt;speculators_config&lt;/code&gt;, there is also a shorter path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That shorthand is convenient, but the long-form command is better for learning because it makes the verifier-speculator relationship explicit.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Send a quick chat request
&lt;/h3&gt;

&lt;p&gt;Once the server is up, test it with a predictable prompt rather than a creative one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl http://localhost:8000/v1/chat/completions &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "model": "Qwen/Qwen3-30B-A3B-Instruct-2507",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain why speculative decoding helps code generation."}
    ],
    "max_tokens": 192,
    "temperature": 0.2,
    "stream": false
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can also call the same endpoint from Python:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;__future__&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;annotations&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;


&lt;span class="n"&gt;API_URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://localhost:8000/v1/chat/completions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;MODEL_NAME&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Send one chat request to a local vLLM server.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;MODEL_NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;messages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;system&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;You are a concise assistant.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Explain why speculative decoding helps code generation.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;temperature&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stream&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;API_URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;choices&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;message&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This client code is intentionally plain. The interesting part of the tutorial is the server-side acceleration, not the SDK wrapper.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Caveats
&lt;/h2&gt;

&lt;p&gt;Speculative decoding is useful, but it is not universal.&lt;/p&gt;

&lt;h3&gt;
  
  
  It works best when the workload is predictable
&lt;/h3&gt;

&lt;p&gt;Code generation, structured outputs, templated responses, and low-temperature Q&amp;amp;A are usually good fits. Free-form creative writing is often a worse fit because the draft model has a harder time guessing the next token sequence well.&lt;/p&gt;

&lt;h3&gt;
  
  
  High concurrency can erase the benefit
&lt;/h3&gt;

&lt;p&gt;Speculative decoding often shines in interactive or low-concurrency serving. If the GPU is already saturated with many concurrent requests, the extra draft-model work can stop helping.&lt;/p&gt;

&lt;h3&gt;
  
  
  The draft model must match the verifier
&lt;/h3&gt;

&lt;p&gt;This is the mistake most new readers make. You cannot casually mix an EAGLE-3 speculator trained for one model with a different verifier. The pairing in this post is specific:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Faster does not mean cheaper if you measure the wrong thing
&lt;/h3&gt;

&lt;p&gt;Time to first token can move differently from total throughput. Measure both. A setup that feels better for a long response may still have slightly higher startup overhead.&lt;/p&gt;

&lt;h3&gt;
  
  
  The verifier is still the expensive model
&lt;/h3&gt;

&lt;p&gt;The speculator is only the helper. You do not get to ignore memory planning just because the draft checkpoint is small.&lt;/p&gt;

&lt;h2&gt;
  
  
  Practical Guidance
&lt;/h2&gt;

&lt;p&gt;If you want this setup to be useful outside a toy demo, start here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keep &lt;code&gt;num_speculative_tokens=3&lt;/code&gt; for your first run, because that is the published default for this speculator.&lt;/li&gt;
&lt;li&gt;Use a predictable prompt family first, such as code, extraction, JSON-like formatting, or constrained Q&amp;amp;A.&lt;/li&gt;
&lt;li&gt;Compare speculative and non-speculative runs on the same prompt set before drawing conclusions.&lt;/li&gt;
&lt;li&gt;Watch tokens per second, time to first token, and time per output token together.&lt;/li&gt;
&lt;li&gt;If acceptance is consistently high, experiment with a slightly larger speculative token count.&lt;/li&gt;
&lt;li&gt;If your real workload is mostly creative writing or large offline batches, do not assume speculative decoding will help.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also a broader 2026 lesson here. The newest published verifier-speculator pairing overall is not always the best tutorial choice. For example, Gemma 4 speculators are newer, but the updated Qwen path above is easier to explain cleanly and stays closer to the original Qwen-based draft while still being current.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sanity check
&lt;/h3&gt;

&lt;p&gt;Before you benchmark anything, verify that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;transformers.__version__&lt;/code&gt; is &lt;code&gt;5.12.1&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;AutoTokenizer.from_pretrained("Qwen/Qwen3-30B-A3B-Instruct-2507")&lt;/code&gt; loads successfully&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;apply_chat_template(...)&lt;/code&gt; renders the Qwen chat markers&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;vllm serve ... --speculative-config ...&lt;/code&gt; starts without model-pairing errors&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/v1/chat/completions&lt;/code&gt; returns a normal assistant message&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The original notebook had a useful topic but an outdated delivery mechanism. A better 2026 version is a real post built around a current verifier-speculator pair, a current tokenizer check, and a serving example that makes the moving parts obvious.&lt;/p&gt;

&lt;p&gt;If you want a practical EAGLE-3 starting point today, &lt;code&gt;Qwen/Qwen3-30B-A3B-Instruct-2507&lt;/code&gt; plus &lt;code&gt;RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3&lt;/code&gt; is a cleaner choice than the old Qwen3-32B example. It keeps the Qwen workflow readable, works cleanly with &lt;code&gt;transformers==5.12.1&lt;/code&gt; on the client side, and maps directly onto the current vLLM speculative decoding story.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Run the same prompt set once with standard vLLM and once with the EAGLE-3 speculator enabled.&lt;/li&gt;
&lt;li&gt;Measure whether your actual workload looks more like code or structured output than creative generation.&lt;/li&gt;
&lt;li&gt;Try a larger tensor-parallel setting if your hardware requires it for the verifier.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Related reading:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;For reducing model size before serving, read &lt;a href="https://employabledatascientist.com/blog/compress-qwen3-5-with-autoround-transformers-5-12-1/" rel="noopener noreferrer"&gt;Compress Qwen3.5 with AutoRound&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;For evaluation risk before deployment, read &lt;a href="https://employabledatascientist.com/blog/codec-contamination-detection-transformers-qwen3-qwen2-5-gemma3/" rel="noopener noreferrer"&gt;CoDeC Contamination Detection&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If serving performance is now a product constraint, &lt;a href="https://employabledatascientist.com/contact/?service=ai-consulting" rel="noopener noreferrer"&gt;talk to us&lt;/a&gt;. We can help compare latency, quality, cost, and reliability instead of optimizing one benchmark in isolation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Resources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507" rel="noopener noreferrer"&gt;Qwen3-30B-A3B-Instruct-2507 model card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3" rel="noopener noreferrer"&gt;Red Hat AI EAGLE-3 speculator for Qwen3-30B-A3B-Instruct-2507&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://vllm.ai/blog/2025-12-13-speculators-v030" rel="noopener noreferrer"&gt;vLLM Speculators v0.3.0 blog post&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.redhat.com/articles/2026/06/12/how-speculative-decoding-delivers-faster-llm-inference" rel="noopener noreferrer"&gt;Red Hat article on speculative decoding in production&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://huggingface.co/papers/2503.01840" rel="noopener noreferrer"&gt;EAGLE-3 paper page on Hugging Face&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/huggingface/transformers/releases" rel="noopener noreferrer"&gt;Transformers release history&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How does EAGLE-3 speculative decoding speed up large language model serving?
&lt;/h3&gt;

&lt;p&gt;A small draft model proposes several next tokens and the large verifier model checks them in parallel. Correct draft tokens are accepted, the final output stays consistent with normal verifier-only decoding, and the verifier can accept multiple tokens per pass instead of one.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you launch vLLM with an EAGLE-3 speculator for Qwen3-30B-A3B-Instruct-2507?
&lt;/h3&gt;

&lt;p&gt;Run vllm serve for Qwen/Qwen3-30B-A3B-Instruct-2507 with a speculative-config that names RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3, num_speculative_tokens set to 3, and method eagle3. Adjust tensor parallelism to match your hardware instead of copying -tp 1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can an EAGLE-3 draft model be used with any verifier model?
&lt;/h3&gt;

&lt;p&gt;No. An EAGLE-3 draft model is not a generic accelerator. It is trained to match one verifier family and often one exact verifier checkpoint, so mixing a speculator trained for one model with a different verifier is the mistake most new readers make.&lt;/p&gt;

&lt;h3&gt;
  
  
  When does speculative decoding not help with vLLM serving?
&lt;/h3&gt;

&lt;p&gt;High concurrency can erase the benefit, because if the GPU is already saturated the extra draft-model work can stop helping. Free-form creative writing and large offline batches are also poor fits, while code, structured outputs, templated responses, and low-temperature Q&amp;amp;A work better.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which metrics should you measure when benchmarking EAGLE-3 speculative decoding?
&lt;/h3&gt;

&lt;p&gt;Watch tokens per second, time to first token, and time per output token together, and compare speculative and non-speculative runs on the same prompt set. Time to first token can move differently from total throughput, so faster does not always mean cheaper.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/serve-qwen3-30b-a3b-instruct-2507-faster-with-eagle-3-and-vllm/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llmsystems</category>
      <category>machinelearning</category>
      <category>python</category>
    </item>
    <item>
      <title>Critical Image Quality Metrics That Break Computer Vision in Production</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Sun, 08 Nov 2026 14:31:07 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/critical-image-quality-metrics-that-break-computer-vision-in-production-3982</link>
      <guid>https://dev.to/harpreetsahota/critical-image-quality-metrics-that-break-computer-vision-in-production-3982</guid>
      <description>&lt;p&gt;Deploying computer vision models into production exposes them to an unforgiving physical reality. In training benchmarks like COCO or ImageNet, images are curated, reasonably illuminated, and centered. In the physical world, your model receives inputs from dirty lenses, vibrating robot mounts, changing daylight, and aggressive video compression codecs.&lt;/p&gt;

&lt;p&gt;Even experienced human annotators struggle to identify objects in dark, blurry, or low-contrast frames. Yet engineering teams routinely deploy models trained on clean data and expect them to perform reliably across all environmental conditions.&lt;/p&gt;

&lt;p&gt;When vision models fail in production, the failure is rarely a flaw in model architecture. It is almost always a failure to understand, measure, and control critical image quality metrics.&lt;/p&gt;

&lt;p&gt;Here is how four fundamental image quality factors degrade model performance, and how to measure them systematically.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       FOUR CRITICAL IMAGE QUALITY DIMENSIONS
+-------------------------------------------------------+
|  1. BLUR &amp;amp; DEFOCUS        |  2. ILLUMINATION &amp;amp; RANGE  |
|  - Motion blur            |  - Underexposure (shadow) |
|  - Camera shake           |  - Overexposure (clipping)|
|  - Defocus aberration     |  - Dynamic range loss     |
+---------------------------+---------------------------+
|  3. CONTRAST DYNAMICS     |  4. CODEC ARTIFACTS       |
|  - Weber/Michelson score  |  - Block compression      |
|  - Shadow boundary fusion |  - Chroma subsampling     |
|  - Internal false edges   |  - Quantization noise     |
+-------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  1. Blur and Defocus: Attenuating High-Frequency Gradients
&lt;/h2&gt;

&lt;p&gt;Blur occurs across three distinct physical mechanisms:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Motion blur:&lt;/strong&gt; Rapid object movement relative to exposure time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Camera shake:&lt;/strong&gt; High-frequency vibration of the sensor mount (common in robotics and automotive payloads).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Defocus:&lt;/strong&gt; Objects sitting outside the camera lens's depth of field.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All three share a common mathematical effect: they act as a spatial low-pass filter. Convolutional kernels and attention patches rely on high-frequency edge transitions to locate object boundaries. When edges blur, localization regressors lose precision, leading to low intersection-over-union (IoU) scores and missed detections.&lt;/p&gt;

&lt;h3&gt;
  
  
  Measuring Blur with Laplacian Variance
&lt;/h3&gt;

&lt;p&gt;The standard quantitative metric for blur is the variance of the Laplacian:&lt;/p&gt;

&lt;p&gt;$$\sigma^2_{Lap} = \text{Var}\left( \nabla^2 I \right)$$&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;compute_blur_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Compute blur metric using the variance of the Laplacian.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;img&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;imread&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IMREAD_GRAYSCALE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;img&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Could not load image at &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;laplacian&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Laplacian&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;img&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cv2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CV_64F&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;laplacian&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;var&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Images scoring below a calibrated threshold (typically $\sigma^2 &amp;lt; 100$, depending on sensor resolution) represent severe blur candidates that should either be filtered, re-captured, or explicitly used in dedicated robustness validation splits.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Illumination and Dynamic Range Clipping
&lt;/h2&gt;

&lt;p&gt;Both underexposure and overexposure devastate feature extraction by destroying dynamic range:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Underexposure:&lt;/strong&gt; Compresses signal into the camera's sensor noise floor. Feature extraction layers amplify Poisson shot noise rather than visual semantic structure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overexposure:&lt;/strong&gt; Saturates pixels to maximum intensity ($255$), clipping specular highlights and erasing texture gradients entirely.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To detect illumination failures, compute histogram clipping ratios:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Under-saturation ratio: Fraction of pixels with grayscale intensity $Y &amp;lt; 15$.&lt;/li&gt;
&lt;li&gt;Over-saturation ratio: Fraction of pixels with intensity $Y &amp;gt; 240$.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If either ratio exceeds 15% of the total bounding box area, feature representations degrade significantly.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Contrast Dynamics: The False Proposal Trap
&lt;/h2&gt;

&lt;p&gt;Contrast is often treated as a simple visibility check, but extreme contrast presents subtle failure modes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Low contrast:&lt;/strong&gt; Fuses object boundaries into adjacent backgrounds, causing false negatives in segmentation masks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;High contrast:&lt;/strong&gt; Harsh direct sunlight creates sharp intra-object shadows. Region proposal networks mistake these internal shadow edges for separate objects, generating multiple fragmented proposals that bypass non-maximum suppression (NMS).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Measure Michelson contrast across the bounding box region:&lt;/p&gt;

&lt;p&gt;$$C = \frac{L_{max} - L_{min}}{L_{max} + L_{min}}$$&lt;/p&gt;

&lt;p&gt;Calibrate your training distribution to ensure balanced representation across both high-contrast direct sunlight and low-contrast diffuse lighting conditions.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Codec Compression and Compounding Penalties
&lt;/h2&gt;

&lt;p&gt;Video pipelines rarely feed uncompressed frames to inference servers. Video streams undergo H.264/H.265 compression, chroma subsampling ($4:2:0$), and bitrate throttling.&lt;/p&gt;

&lt;p&gt;Modern architectures handle moderate, uniform compression well. However, when compression compounds with low lighting or motion blur, high-frequency DCT coefficients are aggressively quantized. The resulting blocking artifacts introduce grid-aligned edges that confound convolutional backbones.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Quality Factor&lt;/th&gt;
&lt;th&gt;Diagnostic Metric&lt;/th&gt;
&lt;th&gt;Failure Mode&lt;/th&gt;
&lt;th&gt;Production Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Blur&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Laplacian Variance ($\text{Var}(\nabla^2 I)$)&lt;/td&gt;
&lt;td&gt;Localization collapse, boundary jitter&lt;/td&gt;
&lt;td&gt;Fast shutter speeds, motion deblur preprocessing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Underexposure&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Low-intensity histogram skew ($Y &amp;lt; 15$)&lt;/td&gt;
&lt;td&gt;Sensor noise amplification, missed detections&lt;/td&gt;
&lt;td&gt;Auto-exposure bracketing, gamma correction&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Overexposure&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Pixel saturation clipping ($Y &amp;gt; 240$)&lt;/td&gt;
&lt;td&gt;Blown highlights, texture erasure&lt;/td&gt;
&lt;td&gt;Neutral density filters, HDR sensors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Compression&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Blockiness metric / Bitrate per pixel&lt;/td&gt;
&lt;td&gt;False grid edges, chroma blurring&lt;/td&gt;
&lt;td&gt;Increase bitrate allocation on motion frames&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Auditing Quality Across the Life Cycle
&lt;/h2&gt;

&lt;p&gt;Do not wait for production downtime to discover how poor image quality impacts your vision models. Measure and monitor these four dimensions during dataset ingestion, training set curation, and live inference logging.&lt;/p&gt;

&lt;p&gt;For deeper reading on robust dataset curation, consult our analysis on &lt;a href="https://employabledatascientist.com/blog/boilingbench-multimodal-fiftyone-analysis/" rel="noopener noreferrer"&gt;multimodal sensor curation&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/dexterous-teleoperation-tactile-policies/" rel="noopener noreferrer"&gt;tactile and visual policy evaluation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To learn how to implement quantitative quality gates in Python, explore Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How does motion blur specifically degrade object detection models?
&lt;/h3&gt;

&lt;p&gt;Motion blur acts as a spatial low-pass filter, attenuating high-frequency gradients that convolutional filters and vision transformers depend on for boundary detection and feature localization. This causes boundary regression loss to spike.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the Laplacian variance metric and how is it used?
&lt;/h3&gt;

&lt;p&gt;Laplacian variance convolves an image with the Laplacian kernel to measure the variance of second-order spatial derivatives. Sharp images with clear edges yield high variance, while blurred images yield low variance scores.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does excessive contrast trigger false positive duplicate detections?
&lt;/h3&gt;

&lt;p&gt;Extreme contrast exaggerates internal object shadows and surface textures, causing region proposal networks to split single objects into multiple fragmented candidate boxes that survive non-maximum suppression.&lt;/p&gt;

&lt;h3&gt;
  
  
  How can teams systematically audit image quality across production pipelines?
&lt;/h3&gt;

&lt;p&gt;Step-by-step implementations and quality auditing workflows are detailed in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/critical-image-quality-metrics-computer-vision/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>computervision</category>
      <category>machinelearning</category>
      <category>datascience</category>
      <category>python</category>
    </item>
    <item>
      <title>Why Squeaky-Clean Datasets Break Production Vision Models</title>
      <dc:creator>Harpreet Sahota</dc:creator>
      <pubDate>Sun, 08 Nov 2026 14:01:08 +0000</pubDate>
      <link>https://dev.to/harpreetsahota/why-squeaky-clean-datasets-break-production-vision-models-1mb7</link>
      <guid>https://dev.to/harpreetsahota/why-squeaky-clean-datasets-break-production-vision-models-1mb7</guid>
      <description>&lt;p&gt;In machine learning engineering, data cleaning is treated as an unquestioned virtue. Teams spend weeks scrubbing datasets: removing blurry frames, discarding underexposed captures, filtering unusual camera angles, and eliminating partial occlusions.&lt;/p&gt;

&lt;p&gt;The resulting dataset is pristine. Training loss drops smoothly, validation accuracy hits record highs, and the team celebrates a successful data cleaning sprint.&lt;/p&gt;

&lt;p&gt;Then the model goes live.&lt;/p&gt;

&lt;p&gt;Within hours, performance degrades. The model misidentifies objects under normal outdoor glare, misses vehicles during light drizzle, and loses tracking whenever a robot camera vibrates.&lt;/p&gt;

&lt;p&gt;The pursuit of pristine data backfired. By over-sanitizing the training distribution, the team inadvertently stripped the structured real-world noise the model needed to learn robust physical representations.&lt;/p&gt;

&lt;p&gt;Controlled variety beats pristine uniformity every time. Here is how to clean datasets without destroying real-world robustness.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       THE PRISTINE DATA PARADOX VS. ROBUST PRESERVATION
+-----------------------------------+-----------------------------------+
|     OVER-CLEANED DATASET          |     CONTROLLED ROBUST DATASET     |
+-----------------------------------+-----------------------------------+
| - Only perfect 12:00 PM sunlight  | - Calibrated lighting spectrum    |
| - Zero motion blur or vibration   | - Sensor mount vibration modeled  |
| - No lens flares or rain streaks  | - Realistic environmental noise   |
| - Uncluttered, centered objects   | - Natural occlusions preserved    |
|                                   |                                   |
| OUTCOME:                          | OUTCOME:                          |
| Brittle in production; collapses   | Generalizes across field conditions|
| under physical camera noise.      | without localized blind spots.    |
+-----------------------------------+-----------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Clean Data Paradox
&lt;/h2&gt;

&lt;p&gt;Deep learning models do not learn abstract physical principles; they learn statistical correlations across training features.&lt;/p&gt;

&lt;p&gt;When you remove all imperfect frames from your training pool, you make two fatal assumptions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;You assume production inputs will be clean:&lt;/strong&gt; Field cameras are exposed to temperature swings, condensation, dirt, motion, and varying optical degradation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You remove regularizing gradients:&lt;/strong&gt; Realistic physical variations force neural networks to ignore superficial pixel artifacts and focus on durable geometric and structural invariants.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;When an over-cleaned model encounters real-world sensor noise, it treats that noise as a meaningful semantic feature, leading to erratic false positives and missed detections.&lt;/p&gt;

&lt;h2&gt;
  
  
  Structured Real-World Noise vs. True Pipeline Bugs
&lt;/h2&gt;

&lt;p&gt;Not all noise is created equal. Successful data-centric engineering requires drawing a strict distinction between &lt;strong&gt;destructive bugs&lt;/strong&gt; and &lt;strong&gt;structured physical noise&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Noise Type&lt;/th&gt;
&lt;th&gt;Physical Cause&lt;/th&gt;
&lt;th&gt;Should You Clean It?&lt;/th&gt;
&lt;th&gt;Engineering Action&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Corrupted Frame&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Dropped UDP video packet, zero-byte read&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;YES&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Purge immediately; corrupts backprop gradients&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Label Misalignment&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Human annotator misread guideline&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;YES&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Correct ground truth annotation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Motion Blur&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Physical camera movement during exposure&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;NO&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Preserve; calibrate exposure distributions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Sensor Glare&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Low-angle sun hitting camera optics&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;NO&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Preserve; essential for outdoor operational invariance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Partial Occlusion&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Object behind a pillar or barrier&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;NO&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Preserve; teaches contextual inference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Low Light / Grain&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High ISO gain in dusk or dawn settings&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;NO&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Preserve; prevents darkness blindness&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The Solution: Create a Data Preservation List
&lt;/h2&gt;

&lt;p&gt;Before launching any data cleaning sprint, write an explicit &lt;strong&gt;Data Preservation List&lt;/strong&gt;. This document functions as an operational contract defining what variations must survive the cleaning process.&lt;/p&gt;

&lt;h3&gt;
  
  
  Components of a Data Preservation List
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Physical Environmental Conditions:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Illumination: Require at least 20% of samples to feature low-light, dawn/dusk, or harsh specular glare conditions.&lt;/li&gt;
&lt;li&gt;Weather and Atmosphere: Explicitly retain fog, rain streaks, and dust scatter.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sensor and Optical Imperfections:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Retain a calibrated percentage of frames exhibiting motion blur, defocus, and camera shake matching your deployment hardware.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Occlusion and Truncation Thresholds:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Mandate that objects occluded up to 60% remain in the training set with explicit occlusion flags rather than being discarded.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Boundary and Edge Case Documentation:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Log all preserved non-standard samples in an auditable metadata manifest. Documenting why a difficult sample was kept is far more valuable than maintaining an artificially perfect quality score.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conducting Empirical Noise Correlation Studies
&lt;/h2&gt;

&lt;p&gt;Rather than guessing which noise levels are acceptable, run empirical correlation studies:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Take a clean baseline validation set.&lt;/li&gt;
&lt;li&gt;Incrementally inject calibrated physical noise (Gaussian blur, luminance shifts, salt-and-pepper noise, cutout occlusions).&lt;/li&gt;
&lt;li&gt;Measure the model's accuracy decay curve against each noise dimension.&lt;/li&gt;
&lt;li&gt;If performance drops by more than 15% under modest, realistic noise levels, your training distribution is over-sanitized and requires re-injection of structured noise.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For deeper architectural perspectives on evaluation and physical AI data, review our field notes on &lt;a href="https://employabledatascientist.com/blog/robot-data-quality-scoring/" rel="noopener noreferrer"&gt;mapping the robot data quality stack&lt;/a&gt; and &lt;a href="https://employabledatascientist.com/blog/building-agent-evaluation-cicd-pipeline-regression-gates/" rel="noopener noreferrer"&gt;building agent evaluation CI/CD pipelines&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;To practice balancing data quality and environmental robustness with real code, audit Harpreet Sahota's Coursera course: &lt;a href="https://www.coursera.org/learn/hands-on-data-centric-visual-ai" rel="noopener noreferrer"&gt;Hands-on Data Centric Visual AI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is the clean data paradox in computer vision?
&lt;/h3&gt;

&lt;p&gt;The clean data paradox occurs when teams sanitize every minor flaw, shadow, blur, and lighting variance from a training dataset, producing models that achieve perfect benchmark scores but fail under real-world operational noise.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is structured real-world noise?
&lt;/h3&gt;

&lt;p&gt;Structured noise refers to non-random, physically grounded visual variations—such as rain streaks, camera vibration, lens glare, seasonal foliage changes, or sensor dust—that models must encounter in training to build invariant representations.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is a Data Preservation List?
&lt;/h3&gt;

&lt;p&gt;A Data Preservation List is an engineering specification that explicitly defines which physical imperfections, edge cases, and noise patterns must be preserved during data cleaning sprints to avoid creating blind spots.&lt;/p&gt;

&lt;h3&gt;
  
  
  How can teams balance data quality and real-world robustness?
&lt;/h3&gt;

&lt;p&gt;Practical frameworks for balancing noise preservation and dataset quality are taught in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This post first appeared as &lt;a href="https://employabledatascientist.com/blog/why-squeaky-clean-datasets-fail-in-production/" rel="noopener noreferrer"&gt;the original article&lt;/a&gt;. I'm Harpreet Sahota, Hacker-in-Residence at &lt;a href="https://employabledatascientist.com" rel="noopener noreferrer"&gt;The Employable Data Scientist&lt;/a&gt;, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the &lt;a href="https://employabledatascientist.com/podcast/" rel="noopener noreferrer"&gt;Artists of Intelligence&lt;/a&gt; podcast. For new field notes by email, &lt;a href="https://harpreetsahota.substack.com" rel="noopener noreferrer"&gt;subscribe on Substack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>visionmultimodal</category>
      <category>ai</category>
      <category>testing</category>
      <category>robotics</category>
    </item>
  </channel>
</rss>
