<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vyankatesh</title>
    <description>The latest articles on DEV Community by Vyankatesh (@vyankatesh).</description>
    <link>https://dev.to/vyankatesh</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4255659%2F64ee3b7a-2377-45de-9a68-d2d72bdebd55.png</url>
      <title>DEV Community: Vyankatesh</title>
      <link>https://dev.to/vyankatesh</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vyankatesh"/>
    <language>en</language>
    <item>
      <title>Are AI-assisted commits reverted more often? I checked 663 matched pairs. The honest answer is 'can't tell'</title>
      <dc:creator>Vyankatesh</dc:creator>
      <pubDate>Fri, 20 Nov 2026 12:10:48 +0000</pubDate>
      <link>https://dev.to/vyankatesh/are-ai-assisted-commits-reverted-more-often-i-checked-663-matched-pairs-the-honest-answer-is-33j1</link>
      <guid>https://dev.to/vyankatesh/are-ai-assisted-commits-reverted-more-often-i-checked-663-matched-pairs-the-honest-answer-is-33j1</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Results are aggregate; no developer, repo or commit is named. This is observational: "associated with", never "causes".&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; are commits that &lt;em&gt;disclose&lt;/em&gt; AI help (e.g. a &lt;code&gt;Co-Authored-By: Claude&lt;/code&gt; trailer) reverted or followed by a bug-fix more often than comparable human-only commits?&lt;/p&gt;

&lt;p&gt;I wrote down the hypotheses and definitions &lt;em&gt;before&lt;/em&gt; pulling any data. The short answer: &lt;strong&gt;no clear difference, but the study is too small to say much.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Method in brief
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Public GitHub APIs only, through my own authenticated &lt;code&gt;gh&lt;/code&gt;. No repo was cloned and no code was run.&lt;/li&gt;
&lt;li&gt;Repos with 50+ stars, not forks or archived, active in the last year.&lt;/li&gt;
&lt;li&gt;Each AI-disclosed commit was matched 1:1 to a human-only commit &lt;strong&gt;from the same repo&lt;/strong&gt;, within 30 days, with similar diff size, and from the &lt;strong&gt;same author&lt;/strong&gt; when possible (82.5% of pairs).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;663 matched pairs, 378 repos&lt;/strong&gt;, commits from 2026-03-12 to 2026-05-04.&lt;/li&gt;
&lt;li&gt;Primary model: conditional logistic regression on the matched pairs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Matching on size worked: median diff was 102 changed lines for AI-disclosed commits and 103 for controls.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqkzrkhv4vdu2tvmghm82.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqkzrkhv4vdu2tvmghm82.png" alt="Diff size distribution" width="800" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 1: reverts are too rare to learn anything
&lt;/h2&gt;

&lt;p&gt;Within 30 days, &lt;strong&gt;1 of 663&lt;/strong&gt; AI-disclosed commits was reverted, versus &lt;strong&gt;2 of 663&lt;/strong&gt; controls. Odds ratio 0.51, 95% CI &lt;strong&gt;0.05 to 5.57&lt;/strong&gt;. That interval spans "much safer" to "5x worse". I checked my revert detector against the 264k commits in the downloaded histories: about 0.2% of commits get reverted, so reverts are just rare. Detecting a 2x effect at that rate would take on the order of 10,000 pairs (rough calculation).&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 2: the fix heuristic is mostly noise
&lt;/h2&gt;

&lt;p&gt;My pre-registered "fix follow-up" rule: a later commit within 14 days, touching a same file, with &lt;code&gt;fix|bug|hotfix|regression&lt;/code&gt; in the message. On a feasible 220-pair subset: OR &lt;strong&gt;1.15 (0.75-1.77)&lt;/strong&gt;, consistent with no difference.&lt;/p&gt;

&lt;p&gt;But I then read 100 flagged follow-ups and judged only &lt;strong&gt;about 30%&lt;/strong&gt; to be plausible fixes (95% CI 22-40%). Common errors: shared &lt;code&gt;package.json&lt;/code&gt;/README/lockfiles, a keyword in the body of a non-fix commit, merges of the same fix. So that "null" mostly reflects a noisy measurement. A stricter, &lt;em&gt;exploratory&lt;/em&gt; variant gave the same picture (OR 1.16, 0.72-1.86).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvif6ls0ejux9tw907zes.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvif6ls0ejux9tw907zes.png" alt="Odds ratios" width="800" height="375"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What went wrong, and what I changed
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sample shortfall:&lt;/strong&gt; target was 2,000 treatment commits; I got 663 pairs. The 50-star filter removed about 93% of repos, and I stopped collection early for time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bug I caught:&lt;/strong&gt; my first fix analysis silently dropped 39 pairs where a commit had no follow-up candidates. Fixed; the paper uses the corrected numbers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feasibility cut:&lt;/strong&gt; the fix outcome only covers pairs with at most 20 candidate follow-ups, which favours quieter periods.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Threats to validity (the important ones)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Only &lt;em&gt;disclosed&lt;/em&gt; AI use is visible, so this studies developers who disclose.&lt;/li&gt;
&lt;li&gt;Controls may contain undisclosed AI code, which biases toward "no difference".&lt;/li&gt;
&lt;li&gt;A revert is a weak quality signal.&lt;/li&gt;
&lt;li&gt;The search sample is capped at 100 results per query-day and favours the latest commits of each day; it covers only 39 days.&lt;/li&gt;
&lt;li&gt;One unblinded reviewer judged the heuristic; recall is unknown.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;For this sample: &lt;strong&gt;no evidence of a difference, and little ability to detect one.&lt;/strong&gt; A real answer needs about 10x more pairs and a validated way to link fixes to the commits they fix (issue links or SZZ-style tracing), plus some idea of how reliably people disclose.&lt;/p&gt;

&lt;p&gt;Related reading (arXiv): &lt;a href="https://arxiv.org/abs/2509.14745" rel="noopener noreferrer"&gt;On the Use of Agentic Coding&lt;/a&gt;, &lt;a href="https://arxiv.org/abs/2601.18341" rel="noopener noreferrer"&gt;Agentic Much? Adoption of Coding Agents on GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Preprint-style write-up, not peer reviewed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;By Vyankatesh Rohokale&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>github</category>
      <category>research</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I statically scanned 618 public agent skills. Here's what the scanner actually found</title>
      <dc:creator>Vyankatesh</dc:creator>
      <pubDate>Fri, 20 Nov 2026 12:09:47 +0000</pubDate>
      <link>https://dev.to/vyankatesh/i-statically-scanned-618-public-agent-skills-heres-what-the-scanner-actually-found-2ppc</link>
      <guid>https://dev.to/vyankatesh/i-statically-scanned-618-public-agent-skills-heres-what-the-scanner-actually-found-2ppc</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Everything here is aggregate: no skill, author or repo is named.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Agent skills are folders of instructions (a &lt;code&gt;SKILL.md&lt;/code&gt;) plus optional scripts that coding agents load on demand. The text is read by the model as trusted guidance, and the scripts can run with your permissions. That makes public skill collections a supply-chain surface. So I asked a simple question: &lt;strong&gt;how risky do public skills look to a static scanner?&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I did
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Found public GitHub repos by topic (&lt;code&gt;agent-skills&lt;/code&gt;, &lt;code&gt;claude-skills&lt;/code&gt;, &lt;code&gt;claude-code-skills&lt;/code&gt;), then drew a seeded, stratified sample across four star ranges (0-9, 10-99, 100-999, 1000+).&lt;/li&gt;
&lt;li&gt;Collected 622 skills from 387 repos. &lt;strong&gt;Nothing was installed, imported or run.&lt;/strong&gt; Each skill went into its own scratch directory and was only read by the scanner.&lt;/li&gt;
&lt;li&gt;Scanned each with an open-source static scanner (SkillSpector v2.12.0, &lt;code&gt;--no-llm&lt;/code&gt;). 618 scanned fine; 4 timed out and were excluded.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Headline numbers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;63.6%&lt;/strong&gt; of skills had at least one finding (95% CI 59.7-67.3).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;40.1%&lt;/strong&gt; had at least one HIGH or CRITICAL finding (95% CI 36.3-44.0).&lt;/li&gt;
&lt;li&gt;Median risk score was &lt;strong&gt;11&lt;/strong&gt;; 36% scored 0; the 90th percentile was 100.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyivampeloqtrbds8vef3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyivampeloqtrbds8vef3.png" alt="Distribution of risk scores" width="800" height="492"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The most common categories were "MCP least privilege" (27% of skills), "excessive agency" (25%), privilege escalation (18%) and data exfiltration (16%). Prompt-injection findings appeared in 6.5% of skills.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8e412hdxt8we08ztkuh9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8e412hdxt8we08ztkuh9.png" alt="Finding categories" width="800" height="486"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The interesting part: score tracks code, not popularity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Stars and risk were unrelated (Spearman rho = 0.01, 95% CI -0.07 to 0.09).&lt;/li&gt;
&lt;li&gt;Age had a weak negative link (rho = -0.10), which I'd treat as exploratory.&lt;/li&gt;
&lt;li&gt;What mattered was &lt;strong&gt;executable surface&lt;/strong&gt;: skills that ship scripts had a median score of 52; markdown-only skills had a median of &lt;strong&gt;0&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqfssvq39uajbj52oqy3d.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqfssvq39uajbj52oqy3d.png" alt="Stars vs risk" width="800" height="523"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  A high score is not a verdict
&lt;/h2&gt;

&lt;p&gt;Of the 25 skills with a CRITICAL finding, 23 were CRITICAL only because of "data flow" findings, usually an environment variable (like an API key) flowing into an HTTP request. In the ones I read, the destination was a fixed first-party API endpoint. The other two were outdated dependencies pinned in lockfiles.&lt;/p&gt;

&lt;p&gt;I also read 10 randomly chosen skills. Of the 3 flagged HIGH or CRITICAL, &lt;strong&gt;none&lt;/strong&gt; warranted that severity in my judgment. One "memory poisoning" hit was a mocking library's docs saying "clear call history". That's 3 skills judged by a single reviewer, so treat it as a hint that severity labels over-state risk, not as an error rate.&lt;/p&gt;

&lt;p&gt;I did not find a skill I'd call intentionally malicious. That is &lt;strong&gt;not&lt;/strong&gt; a claim that none exist: static scanning can't see runtime behaviour, and I didn't estimate how many real problems it misses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Static only; no false-negative estimate.&lt;/li&gt;
&lt;li&gt;Sampled via GitHub topics, so it's not representative of all skills; strata are equal-weighted, not population-weighted.&lt;/li&gt;
&lt;li&gt;One scanner, one version, LLM component off.&lt;/li&gt;
&lt;li&gt;Skills within a repo are correlated, so intervals are probably a bit too narrow.&lt;/li&gt;
&lt;li&gt;The spot-check is n=10, single reviewer, unblinded.&lt;/li&gt;
&lt;li&gt;I didn't analyse topical categories (e.g. "data science" vs "devops").&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;SkillSpector is a static scanner: it lowers risk but doesn't remove it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;Use static scores to &lt;strong&gt;triage capabilities&lt;/strong&gt; (network, credentials, subprocess, unpinned dependencies), not to decide who's malicious. Be most careful with skills that ship scripts, and read what a skill does with your credentials before you enable it. Larger, behaviourally validated studies are the right way to find out how common genuinely malicious skills are.&lt;/p&gt;

&lt;p&gt;Related work I found useful (all on arXiv): &lt;a href="https://arxiv.org/abs/2601.10338" rel="noopener noreferrer"&gt;Agent Skills in the Wild&lt;/a&gt;, &lt;a href="https://arxiv.org/abs/2602.06547" rel="noopener noreferrer"&gt;"Do Not Mention This to the User"&lt;/a&gt;, &lt;a href="https://arxiv.org/abs/2605.28588" rel="noopener noreferrer"&gt;Exploring the Emerging Threats of the Agent Skill Ecosystem&lt;/a&gt;, &lt;a href="https://arxiv.org/abs/2602.14211" rel="noopener noreferrer"&gt;SkillJect&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Preprint-style write-up, not peer reviewed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;By Vyankatesh Rohokale&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>opensource</category>
      <category>research</category>
    </item>
  </channel>
</rss>
