<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Tariq Baloch</title>
    <description>The latest articles on DEV Community by Tariq Baloch (@tariqbaloch786).</description>
    <link>https://dev.to/tariqbaloch786</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4251632%2F23bde18c-ecf1-4f94-9cd4-50d38c17fdd9.jpg</url>
      <title>DEV Community: Tariq Baloch</title>
      <link>https://dev.to/tariqbaloch786</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/tariqbaloch786"/>
    <language>en</language>
    <item>
      <title>I tested an AI agent on real API migrations. Green CI was not proof.</title>
      <dc:creator>Tariq Baloch</dc:creator>
      <pubDate>Wed, 18 Nov 2026 10:27:19 +0000</pubDate>
      <link>https://dev.to/tariqbaloch786/i-tested-an-ai-agent-on-real-api-migrations-green-ci-was-not-proof-2iph</link>
      <guid>https://dev.to/tariqbaloch786/i-tested-an-ai-agent-on-real-api-migrations-green-ci-was-not-proof-2iph</guid>
      <description>&lt;p&gt;For five weeks I have been testing an agent that drafts API-upgrade pull requests. It watches a provider's releases (Stripe, the OpenAI SDK, Twilio), finds the lines in a repository that a breaking change affects, and opens one draft PR with one commit per change. It never merges.&lt;/p&gt;

&lt;p&gt;I tested it on public repositories whose maintainers had already done the same migration by hand, so the agent's fix could be compared with theirs. Here is what came out, including a result I had to withdraw.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;First Stripe run, 12 repositories:&lt;/strong&gt; 1 correct fix. The model kept deleting or nulling fields, because nothing told it where a removed field had moved.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Second Stripe run, 10 new repositories:&lt;/strong&gt; reported as "8 of 10". Withdrawn (below).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The 6 valid Stripe repositories:&lt;/strong&gt; 3 fixed and correct, 3 held back because they are pinned to an older API version where the fix would break at runtime.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OpenAI Node SDK v3 to v4, 10 repositories that reached the model:&lt;/strong&gt; 5 pass, 2 partial, 3 fail. A rerun gave 4 / 2 / 4 on first attempts. The answers vary from run to run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same 10 on a stronger paid model:&lt;/strong&gt; 6 / 1 / 3. One more correct fix, and the same three hard cases failed. The model is not the main limit.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The number I withdrew
&lt;/h2&gt;

&lt;p&gt;The agent's own report said 8 of 10 fixes matched the maintainers. Before publishing, four of the diffs were compared line by line with the maintainers' real commits.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;4 of the 10 comparisons were invalid. The "migration commit" only re-formatted code, or the code had already migrated.&lt;/li&gt;
&lt;li&gt;2 fixes read the right field but would have broken the code. The repositories were pinned to an older Stripe API version in which the new field does not exist. One would have marked every paying user as expired.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The automatic check had marked all of these correct, because it looked for the new field name and nothing else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Green is not proof
&lt;/h2&gt;

&lt;p&gt;The most useful thing I have is a list of cases where every check passed and the code was still wrong.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A cast that hid the error.&lt;/strong&gt; A fix read a field that does not exist, through an &lt;code&gt;as any&lt;/code&gt; cast. The typecheck passed because of the cast.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tests that did not cover the line.&lt;/strong&gt; Two fixes dropped a value instead of reading it from its new place. The test suites passed, because no test looked at that value.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A string nobody read.&lt;/strong&gt; In a PHP demo the code was fixed, but the request still said:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;'expand' =&amp;gt; ['invoice.payment_intent']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;That field was removed. The usage search did not look inside strings, the agent's self-review passed it, and the tests built their objects by hand. CI was green and the PR was marked ready. I found it by reading the branch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Webhook payloads.&lt;/strong&gt; A maintainer told me this one: the SDK types follow the SDK version, but the payload a webhook delivers follows the version set on the provider's endpoint. When they drift apart, the compiler is happy and the handler is wrong. In their words: "the slow part isn't the fix, it's noticing."&lt;/p&gt;

&lt;p&gt;The agent's own review step is the weakest link. In eight PHP runs it passed four things that were wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I think now
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Finding the affected lines is the reliable part.&lt;/li&gt;
&lt;li&gt;Writing the fix is workable for moved and renamed fields. It is weak on wrappers and on changes that span many files.&lt;/li&gt;
&lt;li&gt;Verifying the fix is the hard part. Most small repositories have no test that exercises the changed line.&lt;/li&gt;
&lt;li&gt;The honest product is an impact report with a draft fix for a person to review, not an automatic upgrade.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is not proven
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;No outside developer has installed it on their own repository yet.&lt;/li&gt;
&lt;li&gt;Nothing was sent to a real provider API. That the fixed requests are accepted rests on published specs.&lt;/li&gt;
&lt;li&gt;The sets are small. These are early results, not an accuracy rate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Full write-up with the table and every caveat: &lt;a href="https://agent.tariqbaloch.com/writeup.html" rel="noopener noreferrer"&gt;https://agent.tariqbaloch.com/writeup.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One question, if you maintain an API integration: when an API you depend on changed, how did you find out, and what took the time?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>api</category>
      <category>testing</category>
    </item>
  </channel>
</rss>
