<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vitaliy Dmitriev</title>
    <description>The latest articles on DEV Community by Vitaliy Dmitriev (@vitaliy_dmitriev).</description>
    <link>https://dev.to/vitaliy_dmitriev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4163505%2Fcf22ba5c-74df-4ba1-951a-f0f3ded6e32e.jpg</url>
      <title>DEV Community: Vitaliy Dmitriev</title>
      <link>https://dev.to/vitaliy_dmitriev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vitaliy_dmitriev"/>
    <language>en</language>
    <item>
      <title>How I approve AI agent commands without staying at the terminal</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Tue, 17 Nov 2026 14:54:45 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/how-i-approve-ai-agent-commands-without-staying-at-the-terminal-3pa0</link>
      <guid>https://dev.to/vitaliy_dmitriev/how-i-approve-ai-agent-commands-without-staying-at-the-terminal-3pa0</guid>
      <description>&lt;h1&gt;
  
  
  How I approve AI agent commands without staying at the terminal
&lt;/h1&gt;

&lt;p&gt;I let AI agents do a growing share of my infrastructure work: investigate alerts, prepare changes and propose fixes. Eventually, a fix involves running a command that changes something. I want to review that command, but I don't want to sit at a terminal waiting for the permission prompt.&lt;/p&gt;

&lt;p&gt;So I moved the approval conversation to a Slack DM. The agent prepares a command, a small Bash script posts it, and I reply &lt;code&gt;A42 ok&lt;/code&gt; in the thread. The script checks the answer and hands the command back to the agent.&lt;/p&gt;

&lt;p&gt;The gate is about 120 lines of Bash. I currently use it only for non-production changes in sessions I started myself. Production changes still need me in the session, and scheduled agents only write proposals and exit.&lt;/p&gt;

&lt;p&gt;That scope matters. This is a convenient way to collect approval from a human. Its security still depends on the permissions around the agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why move the prompt?
&lt;/h2&gt;

&lt;p&gt;A terminal permission prompt works while I'm watching the session. If I walk away during a long task, the agent stops at the first command that needs an answer. The investigation may be finished, the change prepared, and everything sits there waiting for me to return to the right terminal.&lt;/p&gt;

&lt;p&gt;A DM lets me review the proposal from my phone. The agent can keep working on unrelated steps while the blocked action waits. The same approach could support scheduled sessions later, but I haven't enabled that yet.&lt;/p&gt;

&lt;p&gt;I also want the approval request to show the actual command. "Restart the service" leaves too much interpretation between what I approve and what the agent eventually runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Freeze, post, reply, release
&lt;/h2&gt;

&lt;p&gt;The flow is deliberately small:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The agent writes the proposed command to a file. The approval script copies it into its state directory, makes that copy read-only, and records its SHA-256 hash and expiry.&lt;/li&gt;
&lt;li&gt;The script sends a DM containing a summary, the full command, a hash prefix, a proposal ID and the expiry time.&lt;/li&gt;
&lt;li&gt;I reply in that message's thread. A correction needs a new proposal.&lt;/li&gt;
&lt;li&gt;The script polls the thread. When it accepts an approval, it marks the proposal as used under a file lock and returns the frozen command. The agent is instructed to execute that command unchanged.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A synthetic example looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[approval A42] staging: scale demo-api to 3 replicas to absorb the queue backlog
    kubectl scale deployment/demo-api --replicas=3 -n demo --kubeconfig ~/.kube/staging
sha256 &amp;lt;hash-prefix&amp;gt;. Reply in this thread `A42 ok` or `A42 no`.
Expires 2026-11-13 14:30 UTC.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One proposal holds one operational action. If the agent needs three unrelated changes, it sends three proposals. I don't want to approve the first line and accidentally agree to everything below it.&lt;/p&gt;

&lt;p&gt;The hash binds the proposal to the saved command text. It doesn't freeze the environment in which that command runs. In this example, the kubeconfig could change while I'm deciding. A command that invokes a script also depends on whatever that script contains when it executes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What counts as approval
&lt;/h2&gt;

&lt;p&gt;The script checks the configured approver's Slack user ID and reads replies only from the proposal's thread. The approver's identity comes from configuration, not from something the agent writes in its request.&lt;/p&gt;

&lt;p&gt;It accepts a small vocabulary: &lt;code&gt;ok&lt;/code&gt;, &lt;code&gt;yes&lt;/code&gt;, &lt;code&gt;approve&lt;/code&gt;, &lt;code&gt;no&lt;/code&gt; or &lt;code&gt;reject&lt;/code&gt;, optionally preceded by the proposal ID. "ok, but use 2 replicas" is not approval. The agent needs to prepare a new command and ask again.&lt;/p&gt;

&lt;p&gt;The saved command is hashed again before approval is released. A mismatch produces &lt;code&gt;CHANGED&lt;/code&gt;. Making the copy read-only helps prevent accidental edits; it doesn't make the file tamper-proof against a process that controls the same account or can replace the saved metadata.&lt;/p&gt;

&lt;p&gt;A proposal expires after four hours. An approval posted before the deadline can be collected during a ten-minute grace period afterwards, to allow for polling delay. A reply posted after the deadline doesn't revive it. That grace period also means execution can happen after the original expiry, so the expiry isn't a guarantee that the command still makes sense against current infrastructure.&lt;/p&gt;

&lt;p&gt;The last valid reply observed before the proposal is consumed wins. I can send &lt;code&gt;no&lt;/code&gt; after &lt;code&gt;ok&lt;/code&gt; while it is still pending. Once the script has released the command, a later message isn't a cancellation mechanism.&lt;/p&gt;

&lt;h3&gt;
  
  
  A detail worth getting right: edited replies
&lt;/h3&gt;

&lt;p&gt;Slack distinguishes a message's original timestamp from its edit timestamp. Filtering out message subtypes alone doesn't reject edited replies. An old message changed to &lt;code&gt;ok&lt;/code&gt; after the deadline could otherwise look like an approval posted in time. &lt;a href="https://docs.slack.dev/reference/events/message/" rel="noopener noreferrer"&gt;Slack documents the separate &lt;code&gt;edited&lt;/code&gt; field here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For this workflow, I would reject edited replies and require a fresh message. This is a revised example of the reply filter, not the complete approval script:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;jq -r --arg u &lt;span class="s2"&gt;"$APPROVER_ID"&lt;/span&gt; --arg p &lt;span class="s2"&gt;"$PROPOSAL_TS"&lt;/span&gt; \
      --arg id &lt;span class="s2"&gt;"$ID"&lt;/span&gt; --argjson exp &lt;span class="s2"&gt;"$EXPIRES"&lt;/span&gt; &lt;span class="s1"&gt;'
  if .ok != true then error("Slack API request failed") else . end
  | [ .messages[]
      | select(.user == $u
               and (.bot_id | not)
               and (.subtype | not)
               and (.edited | not)
               and ((.ts | tonumber) &amp;gt; ($p | tonumber))
               and ((.ts | tonumber) &amp;lt;= $exp))
      | (.text | ascii_downcase
               | gsub("^\\s+|[\\s.!]+$"; "")) as $text
      | (($id | ascii_downcase) + " ") as $prefix
      | (if ($text | startswith($prefix))
         then ($text | ltrimstr($prefix)) else $text end) as $vote
      | select(["ok", "yes", "approve", "no", "reject"]
               | index($vote))
      | {ts: (.ts | tonumber), vote: $vote}
    ]
  | sort_by(.ts) | last | .vote // ""'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The surrounding code still needs to fetch the correct DM and thread, handle pagination, validate the local state and expiry, and consume the approval under a lock. A failed API call or malformed response must stop approval processing. It cannot become permission to run.&lt;/p&gt;

&lt;h2&gt;
  
  
  One approval can be consumed once. Execution is a separate question.
&lt;/h2&gt;

&lt;p&gt;The script reports its decision through exit codes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Exit code&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;th&gt;Meaning for the agent&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;&lt;code&gt;APPROVED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Receives the frozen command for execution&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;REJECTED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Drops the proposed action&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;code&gt;EXPIRED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Drops it; a new proposal needs fresh evidence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;&lt;code&gt;CHANGED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Stops and reports that the saved command changed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;&lt;code&gt;USED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Does not reuse this approval&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;&lt;code&gt;PENDING&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Waits or works on an independent step&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;USED&lt;/code&gt; means the approval was consumed. It doesn't mean the command ran successfully, or even started.&lt;/p&gt;

&lt;p&gt;If the script marks a proposal used and the session crashes before execution, the action may never happen. If a command changes the system but its result gets lost, retrying it could repeat the change. The lock prevents two checks from consuming the same proposal; it doesn't control what the agent does with the returned command.&lt;/p&gt;

&lt;p&gt;That's why I wouldn't describe this as exactly-once execution. After an ambiguous result, inspect the target before retrying. Prefer idempotent operations where possible, and get a new approval if another attempt is needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why polling instead of buttons?
&lt;/h2&gt;

&lt;p&gt;Buttons would be nicer. They also need a running handler, using either an HTTP endpoint or a Socket Mode connection. &lt;a href="https://docs.slack.dev/tools/bolt-js/concepts/socket-mode/" rel="noopener noreferrer"&gt;Socket Mode avoids exposing a public HTTP endpoint&lt;/a&gt;, but it still needs a process to receive interactions.&lt;/p&gt;

&lt;p&gt;The polling version uses a Slack bot token, &lt;code&gt;curl&lt;/code&gt;, &lt;code&gt;jq&lt;/code&gt;, hashing tools and &lt;code&gt;flock&lt;/code&gt; on the Linux host. It runs only while a proposal is open. That was enough for my current volume without adding another service to operate.&lt;/p&gt;

&lt;p&gt;I poll every 30 seconds. That interval needs to fit the app's API limits, especially if several proposals are waiting. Slack's &lt;a href="https://docs.slack.dev/reference/methods/conversations.replies/" rel="noopener noreferrer"&gt;&lt;code&gt;conversations.replies&lt;/code&gt; documentation&lt;/a&gt; gives internal customer-built apps Tier 3 limits, while commercially distributed apps outside the Marketplace can face a one-request-per-minute limit. Handle rate limits and &lt;code&gt;Retry-After&lt;/code&gt;; don't assume this interval works for every installation.&lt;/p&gt;

&lt;p&gt;A crash while waiting should leave the action blocked. A crash around approval consumption needs the more careful recovery described above.&lt;/p&gt;

&lt;h2&gt;
  
  
  The boundaries I still rely on
&lt;/h2&gt;

&lt;p&gt;The script checks an environment label against a non-production allowlist. It doesn't parse the command and prove what infrastructure it targets. A proposal labelled "staging" could still contain a command pointing at production. Reading the command helps, but the enforceable boundary is a credential that cannot write to production.&lt;/p&gt;

&lt;p&gt;The agent can also bypass this workflow if its tools already let it execute arbitrary commands with write access. Instructions to ask first are useful, but they don't remove that capability. A stronger design would put write credentials behind a separate executor that validates approval and runs the approved artifact itself.&lt;/p&gt;

&lt;p&gt;Commands go to Slack as written, so they must not contain inline secrets. References to protected files or environment variables can keep secret values out of the message, but those values remain part of the execution context. They aren't covered by the command hash.&lt;/p&gt;

&lt;p&gt;Finally, the Slack account is part of the approval chain. Someone controlling that session can answer as me. I'm willing to use that arrangement for the limited non-production workflow described here. I haven't extended it to production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this changes for me
&lt;/h2&gt;

&lt;p&gt;The useful part is being able to leave a session without turning a routine approval into a reason to stay at my desk. The agent prepares the action, I review it where I can respond, and the blocked step can continue.&lt;/p&gt;

&lt;p&gt;I can see this fitting between two workflows I already understand: agents that only investigate and prepare changes, and narrowly scoped automation that can run known operations without asking each time. Moving between them depends on the action, credentials and recovery path, not on whether the approval arrives in a terminal or a chat window.&lt;/p&gt;

&lt;p&gt;For now, the small script solves the interruption problem. The permissions around it still decide what the agent can actually change.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>automation</category>
    </item>
    <item>
      <title>AI Can Prepare a DevOps Change. What Makes It Ready to Approve?</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Fri, 06 Nov 2026 10:11:11 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/ai-can-prepare-a-devops-change-what-makes-it-ready-to-approve-59ep</link>
      <guid>https://dev.to/vitaliy_dmitriev/ai-can-prepare-a-devops-change-what-makes-it-ready-to-approve-59ep</guid>
      <description>&lt;h1&gt;
  
  
  AI Can Prepare a DevOps Change. What Makes It Ready to Approve?
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Part 3 of 3 in "Terraform with an AI co-worker: access, testing, review".&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;An AI agent had prepared a Terraform change: a storage bucket, message queues and an application identity across several environments. The tests showed that the application could use them. Before approving each next step, I asked a separate agent to review the work.&lt;/p&gt;

&lt;p&gt;The reviewers found a rollback that would have tried to delete a bucket we hadn't created, a script that could save credentials under the wrong environment's name, and a handover document that promised access which wasn't ready.&lt;/p&gt;

&lt;p&gt;These are the details that turn a small infrastructure ticket into extra work for several teams. They also explain why getting an agent to write the code is only part of automating DevOps routines.&lt;/p&gt;

&lt;p&gt;Nine reviews ran over one week. Each took between half a minute and four minutes. That was the review runtime, not a measurement of the total effort or time saved. I still had to assess findings, resolve questions and approve the next step.&lt;/p&gt;

&lt;p&gt;The reviews also missed a misleading claim in a draft of this series. That failure changed how I brief reviewers.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://dev.to/vitaliy_dmitriev/i-gave-an-ai-agent-a-terraform-task-it-found-a-credentials-problem-4n5o"&gt;Part 1&lt;/a&gt; covers access, &lt;a href="https://dev.to/vitaliy_dmitriev/terraform-applied-successfully-did-the-ai-agents-change-actually-work-57bj"&gt;Part 2&lt;/a&gt; covers testing. This part is about the evidence I needed before accepting the agent's work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Give the reviewer something it can check
&lt;/h2&gt;

&lt;p&gt;The reviewer was a separate agent with a fresh context. It didn't see the conversation in which the work was done. That gave it some distance from the author's assumptions, but it could still make similar mistakes or inherit those assumptions through its briefing.&lt;/p&gt;

&lt;p&gt;I gave it the ticket requirements, the material to review, and read-only access to the real environments where available. The material could be a plan, a code change, a script, a rollout procedure or a handover document. The author's conclusions were claims to verify.&lt;/p&gt;

&lt;p&gt;The review instructions were straightforward:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Read-only review: no deployments or changes, and never print secret values.&lt;/li&gt;
&lt;li&gt;Check relevant vendor documentation rather than relying on memory.&lt;/li&gt;
&lt;li&gt;Rank findings by severity and give a clear verdict, with any conditions or verification gaps.&lt;/li&gt;
&lt;li&gt;Keep the report short enough to use for a decision.&lt;/li&gt;
&lt;li&gt;Account for the actual setting: one operator, a private network and realistic operational risks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last point helped keep the reviews useful. A small change doesn't need every control a large organization might operate. It does need someone to notice that its rollback could delete somebody else's bucket.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rollback would have undone too much
&lt;/h2&gt;

&lt;p&gt;In one test environment, someone had created the bucket by hand before the ticket existed. The agent brought it under Terraform's management through an &lt;em&gt;import&lt;/em&gt;, rather than creating a new bucket.&lt;/p&gt;

&lt;p&gt;The merge request reviewer spotted the consequence. Terraform tracks the imported bucket just like a resource it created. Removing its resource definition and applying that configuration would normally propose deleting the bucket. Reverting our code wasn't enough to restore the situation before our work.&lt;/p&gt;

&lt;p&gt;The bucket still contained an empty folder marker. With &lt;code&gt;force_destroy&lt;/code&gt; disabled, that object would normally prevent Terraform from deleting it. &lt;em&gt;But a rollback procedure shouldn't depend on a leftover object blocking a destructive operation.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The procedure gained an explicit step to remove the imported bucket from Terraform's state before applying the reverted configuration. That leaves the bucket itself in place. The resource definition must also be absent from the rollback configuration, or a later plan could try to create it again.&lt;/p&gt;

&lt;p&gt;For the person approving the work, the useful question was: does rollback undo our change without undoing somebody else's?&lt;/p&gt;

&lt;h2&gt;
  
  
  The script accepted an environment name. It didn't use it to select the source.
&lt;/h2&gt;

&lt;p&gt;After deployment, an existing script (that is not created as a part of subj effort) copied Terraform outputs into the secrets store used by build jobs.&lt;/p&gt;

&lt;p&gt;The script accepted an environment name as an argument. But it read outputs from whichever environment Terraform currently had selected. If those differed, the script could save one environment's credentials under another environment's name.&lt;/p&gt;

&lt;p&gt;The second credentials-and-procedure review found that mismatch by reading the script. A later review found another problem: it reported "saved successfully" even when the secrets store refused the write.&lt;/p&gt;

&lt;p&gt;The rollout procedure gained an explicit check that Terraform's selected environment matched the intended deployment. That addressed the environment mismatch at the procedure level. Finding a bug in a helper script and permanently fixing that script are separate pieces of work; a procedure check still depends on being followed.&lt;/p&gt;

&lt;p&gt;This is one reason I want automation reviewed beyond the new code. &lt;em&gt;Existing helper scripts are part of the deployment too&lt;/em&gt;, including their assumptions and misleading success messages.&lt;/p&gt;

&lt;h2&gt;
  
  
  A successful plan hadn't proved deployment access
&lt;/h2&gt;

&lt;p&gt;Part 1 described the move to temporary personal logins. In the first credentials review, the reviewer noticed that all the plans had run with state locking disabled.&lt;/p&gt;

&lt;p&gt;Terraform's state records which real resources it manages. A deployment needs to update that state, and locking protects it against concurrent changes. The successful plans hadn't demonstrated either the required state writes or the disabled locking operations. Those permissions needed separate examination. Successfully acquiring a lock would not, by itself, prove permission to write the state either.&lt;/p&gt;

&lt;p&gt;The read-only access checks described in Part 1 followed from that finding. They provided evidence about permissions without making a test write. They weren't an end-to-end deployment test.&lt;/p&gt;

&lt;p&gt;For an approval decision, "the plan ran" and "the deployment has the access it needs" are different claims.&lt;/p&gt;

&lt;h2&gt;
  
  
  The handover document was ahead of the deployment
&lt;/h2&gt;

&lt;p&gt;The agent also drafted a page for the developers who would upload and consume the files. A reviewer compared it with the live environments and found two must-fix problems.&lt;/p&gt;

&lt;p&gt;First, the page said credentials were ready in all four test environments. Only one had them. Following that document would have sent developers looking for access that wasn't available yet.&lt;/p&gt;

&lt;p&gt;Second, the page told the consuming application to skip files it had already processed. Elsewhere, it said that re-running an hour meant uploading the same marker file again. An application following both instructions could ignore a deliberate re-run.&lt;/p&gt;

&lt;p&gt;Neither problem would have appeared in a test of the infrastructure alone. The handover needed to match both the deployed system and the application's intended behaviour.&lt;/p&gt;

&lt;h2&gt;
  
  
  A plausible resource count wasn't a verified plan
&lt;/h2&gt;

&lt;p&gt;The merge request description said the production plan would create 13 resources. The final reviewer noted that nobody had run a production plan since the last resources were added. The number was arithmetic, not evidence.&lt;/p&gt;

&lt;p&gt;The production procedure then required the saved plan to show exactly 13 creates, 0 changes and 0 deletes, or stop. A final pre-mortem review, prompted with "assume this deployment went wrong; how did it happen?", inspected that saved plan and confirmed the count.&lt;/p&gt;

&lt;p&gt;The count was one check, not the whole approval. Thirteen creates can still be the wrong thirteen resources. The review also examined whether the production steps could change or delete existing resources.&lt;/p&gt;

&lt;p&gt;Its verdict was "go", with four conditions. One was to record the names of keys already present in the production secret before adding new ones, and confirm that the previous version remained readable for rollback.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the nine reviews fitted
&lt;/h2&gt;

&lt;p&gt;The reviews covered different parts of the work. They weren't nine passes over the same code.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Checkpoint&lt;/th&gt;
&lt;th&gt;What was reviewed&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Before writing code&lt;/td&gt;
&lt;td&gt;Plan and supporting facts&lt;/td&gt;
&lt;td&gt;No critical or major issues; five minor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Merge request&lt;/td&gt;
&lt;td&gt;Code, scope and rollback&lt;/td&gt;
&lt;td&gt;Ready, with caveats&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Credentials and deployment procedure&lt;/td&gt;
&lt;td&gt;Three review rounds&lt;/td&gt;
&lt;td&gt;One major issue in each of the first two rounds; both addressed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before test rollout&lt;/td&gt;
&lt;td&gt;Rollout steps and helper script&lt;/td&gt;
&lt;td&gt;Three major, six minor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Handover&lt;/td&gt;
&lt;td&gt;Instructions for application teams&lt;/td&gt;
&lt;td&gt;Two must-fix items&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before production&lt;/td&gt;
&lt;td&gt;Final merge request check&lt;/td&gt;
&lt;td&gt;Ready, with caveats&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before production&lt;/td&gt;
&lt;td&gt;Pre-mortem and saved plan&lt;/td&gt;
&lt;td&gt;Go, with four conditions&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The credentials and deployment procedure needed three rounds. The first found the unverified state-write access. The second confirmed the response to that finding and caught the environment mismatch in the script. The third found no critical or major issues, only six minor ones.&lt;/p&gt;

&lt;p&gt;My default cap is two rounds, with a follow-up justified by a critical or major finding. Each follow-up checks the response to earlier findings before looking for new problems. I allow a third round to verify the response to a major issue from round two when the change involves production, an irreversible operation or a security control. This work qualified because it created production credentials.&lt;/p&gt;

&lt;p&gt;The cap limits review effort - &lt;em&gt;it doesn't make an unresolved major issue acceptable&lt;/em&gt;. If one remains at the cap, I have to stop the affected work or decide how to resolve it. Minor findings go onto a short list for human judgement rather than triggering another automatic round.&lt;/p&gt;

&lt;p&gt;Reviewers can almost always find something else to improve ;) I need a process that ends in an informed decision.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the reviewers still got wrong
&lt;/h2&gt;

&lt;p&gt;The most revealing miss was in an early draft of Part 2. It said that a full deployment "would have deleted existing stored files". Two rounds of AI review passed that sentence.&lt;/p&gt;

&lt;p&gt;The problem was in the briefing. The authoring agent had supplied a summary for reviewers to treat as ground truth. It said the deployment would have "deleted existing storage objects", without identifying the objects. The reviewers checked the article against that summary instead of the actual Terraform plan.&lt;/p&gt;

&lt;p&gt;Then I asked: was data loss confirmed, or inferred?&lt;/p&gt;

&lt;p&gt;The agent went back to the plan. The deletions were two empty folder markers that a colleague had deliberately removed from the code. Part 2 was corrected to say that.&lt;/p&gt;

&lt;p&gt;The reviewers had checked consistency with the author's explanation. They hadn't verified the explanation itself. A fresh context doesn't fix that.&lt;/p&gt;

&lt;p&gt;There were other limits too. The production pre-mortem claimed that the year-2000 date used in test filenames would stop a future consumer from treating them as real data. It wouldn't. The date was a naming convention. The relevant safety condition was that no application was reading those queues yet.&lt;/p&gt;

&lt;p&gt;One merge request reviewer couldn't reach the real Terraform state and marked the plan numbers as unverifiable. That was a useful result. It told me exactly which part of the evidence was missing.&lt;/p&gt;

&lt;p&gt;Some decisions also &lt;em&gt;belonged elsewhere&lt;/em&gt;. The reviewers asked for confirmation of the queue visibility timeout, which determines how long a message stays hidden during processing, and whether repeatedly failing messages should go to a separate queue. The application developers needed to settle those questions before production.&lt;/p&gt;

&lt;p&gt;I also retained responsibility for exceptions to my own rules. One reviewer flagged a step that required a secrets-store token in the shell. We wrote a small helper to issue a short-lived login for that step. The reviewer identified the problem; it didn't decide which access arrangement I should accept.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would automate for a team
&lt;/h2&gt;

&lt;p&gt;This case gives me a practical starting point for helping a company automate DevOps routines: choose one recurring change and work through the whole path from ticket to handover.&lt;/p&gt;

&lt;p&gt;I would first identify what the agent can investigate, which evidence it must return, where an independent check adds value, and who approves the next action. The same exercise exposes old scripts, unclear ownership and documentation gaps that can slow down human engineers too.&lt;/p&gt;

&lt;p&gt;For this task, the agent investigated access, wrote code, built tests, and prepared and checked the rollout. My work included setting the review conditions, challenging findings, resolving exceptions and deciding whether the evidence supported proceeding. That division of work is the part I would want to make repeatable.&lt;/p&gt;

&lt;p&gt;I would measure the total effort before extending it: preparation, reviewing findings, rework and interruptions, as well as agent runtime. Nine quick reviews demonstrate that this extra checking was feasible for this change. They don't establish a time-saving percentage or prove that every ticket needs the same process.&lt;/p&gt;

&lt;p&gt;A team trying this should finish with a concrete approval package: the proposed change, the evidence behind it, unresolved questions and a rollback procedure. The reviewer needs access to the underlying plan, diff or live facts, with the author's summary clearly separated from them.&lt;/p&gt;

&lt;p&gt;The misleading sentence in Part 2 is why I care about that separation. Adding another agent was easy. Making it check the right evidence took more thought.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This is Part 3 of "Terraform with an AI co-worker: access, testing, review". &lt;a href="https://dev.to/vitaliy_dmitriev/i-gave-an-ai-agent-a-terraform-task-it-found-a-credentials-problem-4n5o"&gt;Part 1&lt;/a&gt; covered access; &lt;a href="https://dev.to/vitaliy_dmitriev/terraform-applied-successfully-did-the-ai-agents-change-actually-work-57bj"&gt;Part 2&lt;/a&gt; covered testing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Technical references: &lt;a href="https://developer.hashicorp.com/terraform/cli/commands/state/rm" rel="noopener noreferrer"&gt;removing a resource from Terraform state&lt;/a&gt;, &lt;a href="https://developer.hashicorp.com/terraform/language/backend/s3" rel="noopener noreferrer"&gt;S3 backend permissions and locking&lt;/a&gt;, &lt;a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#force_destroy" rel="noopener noreferrer"&gt;S3 bucket force_destroy&lt;/a&gt;, and &lt;a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html" rel="noopener noreferrer"&gt;SQS dead-letter queues&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>terraform</category>
      <category>codereview</category>
    </item>
    <item>
      <title>Replacing long-lived AWS keys: testing Vault's options with an AI co-worker</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Fri, 30 Oct 2026 10:08:24 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/replacing-long-lived-aws-keys-testing-vaults-options-with-an-ai-co-worker-3en9</link>
      <guid>https://dev.to/vitaliy_dmitriev/replacing-long-lived-aws-keys-testing-vaults-options-with-an-ai-co-worker-3en9</guid>
      <description>&lt;h1&gt;
  
  
  What we are testing before choosing how Vault issues AWS credentials, what &lt;code&gt;assumed_role&lt;/code&gt; revealed, and where AI-generated tests still needed an engineer.
&lt;/h1&gt;




&lt;p&gt;Replacing long-lived AWS keys sounds straightforward: issue credentials that expire, and stop leaving permanent access in build systems, laptops and old scripts.&lt;/p&gt;

&lt;p&gt;We already use HashiCorp Vault to manage secrets. It can issue AWS credentials in four different ways. Before choosing one for wider use, we wanted to understand how each behaves when access is granted, restricted or revoked.&lt;/p&gt;

&lt;p&gt;I used an AI-assisted workflow to help with the evaluation. Agents prepared test plans, reviewed one another's work, wrote scripts and turned execution logs into reports. I reviewed the scripts, ran them in a sandbox and checked the conclusions.&lt;/p&gt;

&lt;p&gt;All four plans are ready. Two options have been tested. This article covers the comparison we are working through and what the first option, &lt;code&gt;assumed_role&lt;/code&gt;, taught us. The AI loop implementation will get its own article.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four ways to issue access
&lt;/h2&gt;

&lt;p&gt;The goal is to replace keys distributed to people and automated jobs with access that lasts only as long as needed. Shorter lifetimes reduce the window in which a stolen credential can be used. Permissions still matter: a short-lived administrator credential can do plenty of damage before it expires.&lt;/p&gt;

&lt;p&gt;Vault's four credential types take different approaches:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;How it works&lt;/th&gt;
&lt;th&gt;Main question for our evaluation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;assumed_role&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gets temporary credentials for an AWS role, which defines the available permissions.&lt;/td&gt;
&lt;td&gt;How do duration limits, audit trails and early revocation behave?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;iam_user&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Creates an IAM user and key, then deletes the user when its Vault lease expires.&lt;/td&gt;
&lt;td&gt;How reliable are creation and cleanup, and how much authority must Vault hold?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;federation_token&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gets temporary credentials limited by both the issuing user's permissions and additional session policies.&lt;/td&gt;
&lt;td&gt;How do those policies combine to limit access?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;session_token&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gets temporary credentials based on the issuing IAM user's permissions, without a separate Vault role policy to narrow them.&lt;/td&gt;
&lt;td&gt;Does this give callers only the permissions they need?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are documented differences, not four completed test results. In particular, the &lt;code&gt;iam_user&lt;/code&gt; key does not expire by itself: Vault has to remove it as part of deleting the user. With the STS options, AWS enforces expiry. With &lt;code&gt;iam_user&lt;/code&gt;, we also depend on Vault successfully cleaning up.&lt;/p&gt;

&lt;p&gt;We gave each option a ticket and a sandbox test. The practical questions were how quickly access becomes usable, what callers can do, whether we can trace their actions, and how access ends. We also wanted to understand the consequences if Vault's issuing credentials leaked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the AI loops helped
&lt;/h2&gt;

&lt;p&gt;The workflow produced a plan for each ticket, including possible failures, then had separate AI reviewers check it. They wrote scripts for setup, testing and cleanup, and produced reports from the logs after I ran them.&lt;/p&gt;

&lt;p&gt;The agent did not receive the AWS keys or Vault token. I supplied credentials to the scripts when running them, and every live action in this evaluation came from a script I ran. That &lt;em&gt;left me responsible&lt;/em&gt; for reviewing what those scripts would do.&lt;/p&gt;

&lt;p&gt;This gave us a repeatable way to evaluate the options. It also meant fixes from the first test could be carried into later scripts. The separate reviewers were useful, although one invalid test still made it through.&lt;/p&gt;

&lt;h2&gt;
  
  
  First result: &lt;code&gt;assumed_role&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;For this test, Vault used an IAM user allowed to assume roles matching a particular name prefix. Each role defined what the issued credentials could do.&lt;/p&gt;

&lt;p&gt;That &lt;em&gt;still leaves an underlying long-lived key in Vault&lt;/em&gt; in our test setup. We are evaluating how to replace keys handed out to people and jobs, rather than claiming to eliminate every static key at once.&lt;/p&gt;

&lt;h3&gt;
  
  
  Access worked on the first attempt
&lt;/h3&gt;

&lt;p&gt;In our test, the first request succeeded one second after Vault issued the credentials, with no retries. That matters for automated jobs that request access and immediately use it.&lt;/p&gt;

&lt;p&gt;Our session name included the Vault login and role. We saw it in CloudTrail, AWS's audit log, so we could connect the activity to the requester. That is more useful than a shared static key which identifies an IAM user but often says little about which person or job used it. The naming template needs to fit within the 64-character session-name limit.&lt;/p&gt;

&lt;p&gt;The permission checks also behaved as intended. The test role could list S3 buckets; listing EC2 regions was denied. The issuing user could not assume a role outside its allowed prefix and could not list IAM users.&lt;/p&gt;

&lt;p&gt;Those checks supported the intended restrictions. A leaked issuing key could still request fresh sessions for the roles it was allowed to assume, so protecting it remains important.&lt;/p&gt;

&lt;h3&gt;
  
  
  Two duration limits need to agree
&lt;/h3&gt;

&lt;p&gt;AWS rejected an &lt;code&gt;AssumeRole&lt;/code&gt; session shorter than 15 minutes. At the upper end, Vault and AWS handled excessive requests differently:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Vault capped a request at its configured &lt;code&gt;max_sts_ttl&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;AWS rejected a request above the role's maximum session duration.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If Vault allows two hours but the AWS role allows only one, a two-hour request fails. These settings need to be configured together so callers get predictable behaviour.&lt;/p&gt;

&lt;h3&gt;
  
  
  Revoking in Vault did not stop access in AWS
&lt;/h3&gt;

&lt;p&gt;This was the most important operational finding. Revoking the Vault lease did not invalidate the issued role credentials. Without an AWS-side permission change, they remained usable until their AWS expiry.&lt;/p&gt;

&lt;p&gt;AWS provides a way to block older role sessions using a deny policy with a cutoff time. Its standard “revoke active sessions” action uses that approach. It also affects legitimate users of the same role whose sessions started before the cutoff, so they need fresh credentials.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;That is something to establish before an incident.&lt;/em&gt; “Revoke access” can mean different things in Vault and AWS, and the emergency procedure can interrupt other work.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test that answered the wrong question
&lt;/h2&gt;

&lt;p&gt;We also wanted to measure how quickly the AWS-side deny policy took effect. The generated report said it had “no effect within two minutes”.&lt;/p&gt;

&lt;p&gt;Reading the log, I found that the script was checking access with &lt;code&gt;sts:GetCallerIdentity&lt;/code&gt;, essentially asking AWS “who am I?”. AWS allows this request even when an identity policy explicitly denies it. It could not tell us whether the session had lost permission to do useful work.&lt;/p&gt;

&lt;p&gt;The conclusion became “not measured”. I asked the agent to replace the check with an action that could actually be denied. The script has been corrected, but we still need a valid timing result.&lt;/p&gt;

&lt;p&gt;This showed a weakness in the review process. The plan, script and report all relied on the same wrong assumption. I only caught it when I checked what the API call actually proved.&lt;/p&gt;

&lt;h2&gt;
  
  
  What still needed engineering work
&lt;/h2&gt;

&lt;p&gt;The first run stopped because my Vault admin policy lacked access to the new test paths. I cleaned up using the rollback steps, then made about eight fixes to the script: safer cleanup, stricter checks and a permissions check before creating resources. I updated the Vault policy through our normal code review process.&lt;/p&gt;

&lt;p&gt;The second run failed at the revoke step because of another missing permission. This time cleanup ran automatically.&lt;/p&gt;

&lt;p&gt;The workflow carried these fixes into the later scripts. That was useful, but the work still needed someone who understood the permissions, checked cleanup and could challenge a report's conclusion.&lt;/p&gt;

&lt;p&gt;For context, the workflow's own estimate of its first week's model usage was about $14, excluding my interactive AI sessions and my time. My involvement was very roughly one to two hours per ticket. These are estimates, not a measured comparison with doing the work manually.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we still need before choosing
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;iam_user&lt;/code&gt; test has also run; its results will be covered in the follow-up. It requires Vault to create and manage IAM users and keys, which makes the issuing identity's permissions and reliable cleanup especially important.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;federation_token&lt;/code&gt; and &lt;code&gt;session_token&lt;/code&gt; plans are reviewed but have not run yet. We still need those results before choosing an approach for wider use.&lt;/p&gt;

&lt;p&gt;We have useful results for &lt;code&gt;assumed_role&lt;/code&gt;, but not enough evidence to choose among all four options yet. The AI workflow helped prepare and document the tests. My job was still to check that those tests answered the questions we needed answered.&lt;/p&gt;

&lt;p&gt;References: &lt;a href="https://developer.hashicorp.com/vault/docs/secrets/aws" rel="noopener noreferrer"&gt;Vault AWS secrets engine&lt;/a&gt;, &lt;a href="https://developer.hashicorp.com/vault/api-docs/secret/aws" rel="noopener noreferrer"&gt;Vault AWS API&lt;/a&gt;, &lt;a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" rel="noopener noreferrer"&gt;AWS AssumeRole&lt;/a&gt;, &lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html" rel="noopener noreferrer"&gt;revoking role session permissions&lt;/a&gt;, and &lt;a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html" rel="noopener noreferrer"&gt;GetCallerIdentity&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>vault</category>
      <category>devops</category>
      <category>ai</category>
    </item>
    <item>
      <title>Terraform Applied Successfully. Did the AI Agent's Change Actually Work?</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Tue, 20 Oct 2026 08:31:01 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/terraform-applied-successfully-did-the-ai-agents-change-actually-work-57bj</link>
      <guid>https://dev.to/vitaliy_dmitriev/terraform-applied-successfully-did-the-ai-agents-change-actually-work-57bj</guid>
      <description>&lt;h1&gt;
  
  
  Terraform Applied Successfully. Did the AI Agent's Change Actually Work?
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Part 2 of 3 in "Terraform with an AI co-worker: access, testing, review".&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Part 1 is &lt;a href="https://dev.to/vitaliy_dmitriev/i-gave-an-ai-agent-a-terraform-task-it-found-a-credentials-problem-4n5o"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The task looked small: create an S3 bucket, two SQS queues and a service user across four test environments and production. An AI agent was doing the work.&lt;/p&gt;

&lt;p&gt;Before deploying, it ran Terraform plans. In two test environments, those plans included other teams' access keys, permission changes and the deletion of two empty S3 folder markers. None of that belonged to our task.&lt;/p&gt;

&lt;p&gt;That was the first useful result of testing, before we had deployed anything. Later, the agent would upload real objects, deliberately trigger access denials, and discover that some of its own checks were wrong.&lt;/p&gt;

&lt;p&gt;In Part 1, I covered how we restricted the agent's AWS access. This part is about the evidence I wanted before accepting its work as done.&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan included someone else's work
&lt;/h2&gt;

&lt;p&gt;A Terraform plan previews proposed changes. It doesn't apply them, which makes it a useful first check. It also only covers what the configuration and state let Terraform see, so it isn't a complete inventory of an AWS account.&lt;/p&gt;

&lt;p&gt;In our case, a colleague's cleanup had been merged into the repository but hadn't been deployed. A full apply would have removed two empty folder markers, created access keys and permission policies for other teams, and changed existing permission policies.&lt;/p&gt;

&lt;p&gt;The folder-marker deletions were intended by their owner. This wasn't a confirmed data-loss scenario. The problem was that nobody deploying our change had reviewed or approved those unrelated changes for this rollout.&lt;/p&gt;

&lt;p&gt;They would have gone live as a side effect of our task, at a time their owners hadn't chosen.&lt;/p&gt;

&lt;p&gt;We used &lt;code&gt;-target=module.&amp;lt;name&amp;gt;&lt;/code&gt; to limit this deployment, then checked the resulting plan against the expected resource addresses and actions. Anything outside that scope, including any deletion, stopped the deployment before approval. The unrelated changes were reported separately.&lt;/p&gt;

&lt;p&gt;Targeting needs a caveat: it includes dependencies, and HashiCorp recommends it only for exceptional situations. It doesn't mean "touch exactly this module and nothing else." The resulting plan still needs inspection, and a subsequent full plan is needed to see what remains pending.&lt;/p&gt;

&lt;p&gt;There is another distinction worth making. We used &lt;code&gt;terraform plan -lock=false&lt;/code&gt; for the initial inspection with read-only credentials. &lt;strong&gt;The credentials restricted writes; &lt;code&gt;-lock=false&lt;/code&gt; only disabled state locking.&lt;/strong&gt; It isn't a general safety option. Keep locking enabled for normal deployment work, and don't use an unlocked inspection while another operation may be changing the same state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Could we prove the bucket rule worked?
&lt;/h2&gt;

&lt;p&gt;The bucket had a rule that mattered to the application.&lt;/p&gt;

&lt;p&gt;Each hour, an upstream system uploads data files, followed by a small marker file saying the upload is finished. The marker triggers an S3 notification to an SQS queue. A retry must not overwrite that marker and trigger another object-created event for the same hour.&lt;/p&gt;

&lt;p&gt;We enforced conditional writes for markers through a bucket policy. The uploader had to send &lt;code&gt;If-None-Match: *&lt;/code&gt;, meaning "create this object only if the key doesn't already exist." Ordinary data files didn't need that header.&lt;/p&gt;

&lt;p&gt;Reading the policy back would show that it had been saved. I wanted to know whether it actually enforced those rules.&lt;/p&gt;

&lt;p&gt;Before deploying to the real environments, the agent created a throwaway bucket and queue in a test account and exercised the same policy:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Attempt&lt;/th&gt;
&lt;th&gt;Expected&lt;/th&gt;
&lt;th&gt;Observed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Upload a marker without the required header&lt;/td&gt;
&lt;td&gt;&lt;code&gt;403 AccessDenied&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Refused&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upload a marker with &lt;code&gt;If-None-Match: *&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Success&lt;/td&gt;
&lt;td&gt;Accepted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upload the same marker again with the header&lt;/td&gt;
&lt;td&gt;&lt;code&gt;412 PreconditionFailed&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Refused&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upload an ordinary data file without the header&lt;/td&gt;
&lt;td&gt;Success, no notification&lt;/td&gt;
&lt;td&gt;Accepted, no notification observed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inspect queue messages&lt;/td&gt;
&lt;td&gt;Marker notification, separate from AWS's setup test event&lt;/td&gt;
&lt;td&gt;One marker notification observed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The agent then removed the resources it had created for the test.&lt;/p&gt;

&lt;p&gt;There is an important limit here: preventing marker overwrites doesn't guarantee exactly-once processing. S3 notifications and standard SQS queues can deliver duplicates. Seeing one message in a test doesn't change that contract. The downstream consumer &lt;em&gt;still needs to be idempotent&lt;/em&gt;, and we passed that requirement to its developers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test with the application's permissions
&lt;/h2&gt;

&lt;p&gt;After each deployment, the agent ran a smoke test using the application's service-user credentials. A successful upload with a privileged deployment identity wouldn't tell us whether the application could do the same thing.&lt;/p&gt;

&lt;p&gt;Here is the production output, with names shortened:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;identity: user/&amp;lt;app-user&amp;gt;-prod
content folders: input/content/  output/content/
1 put without header (expect AccessDenied): AccessDenied
2 put with If-None-Match (expect OK):        OK
3 duplicate put (expect PreconditionFailed): PreconditionFailed
4 parquet without header (expect OK):        OK
5 other bucket (expect AccessDenied):        AccessDenied
6 input queue: smoke-test events=1 (expect 1)
7 output queue: non-test events=0 (expect 0)
cleanup: OK
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Line 5 mattered as much as the successful upload: the identity couldn't access the other bucket we tested. Line 7 checked that the test hadn't produced an event in the wrong queue during the observation period.&lt;/p&gt;

&lt;p&gt;The production queues were dedicated to this application, brand new and not yet in use. That made it possible to inspect the test messages without competing with a live consumer. &lt;em&gt;This procedure would need a different design for an active queue.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The test objects used a year-2000 date to make them recognizable, and cleanup removed only the objects created by the test. The fake date was a naming convention, not protection against real processing. The fact that the queues weren't in use &lt;em&gt;was the relevant condition here&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Some of the tests needed debugging too
&lt;/h2&gt;

&lt;p&gt;In the first test environment, uploads initially produced no visible messages. The configuration looked correct, but it took roughly five to eight minutes before we observed notifications flowing.&lt;/p&gt;

&lt;p&gt;That was our observation, not proof of a particular AWS root cause or a delay to expect every time. S3 notifications can be delayed, and SQS message counts are approximate. The useful response was bounded waiting and checking the actual messages, rather than immediately changing a configuration that might already be correct.&lt;/p&gt;

&lt;p&gt;The agent also produced two false alarms in its safety checks. One treated an empty response as a failure; another expected a field that its query hadn't returned. There were smaller script bugs too: a check inspected the wrong part of the data, and extra command output broke a later step.&lt;/p&gt;

&lt;p&gt;A guard that stops on uncertainty can be useful. But if the guard itself is broken, "the deployment was blocked" doesn't tell you much.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;We needed to verify that it recognized valid results as well as unexpected ones.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Those script and checking bugs were fixed before the production rollout. AWS timing behaviour remained something the tests had to accommodate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lessons learned: what I now ask for before accepting "done"
&lt;/h2&gt;

&lt;p&gt;For this change, the evidence was fairly concrete:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A reviewed plan whose proposed actions matched the task.&lt;/li&gt;
&lt;li&gt;Behavioural tests on throwaway resources before deployment.&lt;/li&gt;
&lt;li&gt;Smoke tests in every target environment using the application identity.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Negative tests&lt;/em&gt; for operations that should be refused.&lt;/li&gt;
&lt;li&gt;An explicit account of test limitations, delays and cleanup.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These checks can live in scripts or a testing framework. They don't need an LLM to decide whether &lt;code&gt;AccessDenied&lt;/code&gt; matches an expected denial. The agent was useful for investigating the environment, preparing the tests and working through unexpected results. The repeatable comparisons should remain explicit.&lt;/p&gt;

&lt;p&gt;The most useful part wasn't the final successful apply. It was discovering what that apply would have included, and then checking that the deployed resources behaved as the application required.&lt;/p&gt;

&lt;p&gt;That's the evidence I want alongside an agent's summary.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Next, in Part 3: reviewing the agent's work with independent agents, what those reviewers caught, and why we stop after three rounds.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Technical references: &lt;a href="https://developer.hashicorp.com/terraform/cli/commands/plan" rel="noopener noreferrer"&gt;Terraform plan and targeting&lt;/a&gt;, &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html" rel="noopener noreferrer"&gt;S3 conditional-write enforcement&lt;/a&gt;, &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html" rel="noopener noreferrer"&gt;S3 notification delivery&lt;/a&gt;, and &lt;a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html" rel="noopener noreferrer"&gt;SQS approximate message counts&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>terraform</category>
      <category>aws</category>
    </item>
    <item>
      <title>One missing name, a Kubernetes namespace full of restarts</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Fri, 16 Oct 2026 07:40:39 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/one-missing-name-a-kubernetes-namespace-full-of-restarts-3njn</link>
      <guid>https://dev.to/vitaliy_dmitriev/one-missing-name-a-kubernetes-namespace-full-of-restarts-3njn</guid>
      <description>&lt;h1&gt;
  
  
  One missing name, a Kubernetes namespace full of restarts
&lt;/h1&gt;

&lt;p&gt;Compare these two commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl delete deployment &lt;span class="nt"&gt;-n&lt;/span&gt; prod
&lt;span class="c"&gt;# error: resource(s) were provided, but no name was specified&lt;/span&gt;

kubectl rollout restart deployment &lt;span class="nt"&gt;-n&lt;/span&gt; prod
&lt;span class="c"&gt;# deployment.apps/api restarted&lt;/span&gt;
&lt;span class="c"&gt;# deployment.apps/billing restarted&lt;/span&gt;
&lt;span class="c"&gt;# deployment.apps/checkout restarted&lt;/span&gt;
&lt;span class="c"&gt;# ...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first refuses to run. The second attempts to restart every Deployment in the namespace, without confirmation or an explicit &lt;code&gt;--all&lt;/code&gt;. It's &lt;a href="https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/" rel="noopener noreferrer"&gt;documented behavior&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;That makes this a surprisingly dangerous bit of shell:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl rollout restart deployment &lt;span class="nv"&gt;$SERVICE&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; prod
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;SERVICE&lt;/code&gt; is empty, the command gets a much bigger audience than intended.&lt;/p&gt;

&lt;p&gt;Each Deployment still follows its own update strategy. But with rolling updates, those rollouts can overlap across the namespace. Caches warm up, connections reconnect, and replacement pods compete for capacity together. Per-Deployment rollout settings don't coordinate that load across services.&lt;/p&gt;

&lt;p&gt;We wanted to remove this particular footgun without taking away routine troubleshooting access. That led to a Jenkins job, and then to a more interesting question: what should its credential be allowed to do?&lt;/p&gt;

&lt;h2&gt;
  
  
  RBAC doesn't see your intention
&lt;/h2&gt;

&lt;p&gt;There is no dedicated Kubernetes “restart” API. For a Deployment, kubectl patches the pod template with a timestamp:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;kubectl.kubernetes.io/restartedAt&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-01-01T12:00:00Z"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The changed template triggers a rollout. Without a name, kubectl lists the Deployments and sends a separate patch for each one.&lt;/p&gt;

&lt;p&gt;RBAC authorizes those requests individually. It cannot express “allow any one Deployment, but reject a command that targets several.” A &lt;code&gt;resourceNames&lt;/code&gt; allowlist can restrict which Deployments are patchable, but it still permits patching every allowed name.&lt;/p&gt;

&lt;p&gt;We removed &lt;code&gt;patch&lt;/code&gt; on Deployments from users and provided the restart through Jenkins instead. Users kept their existing scaling and pod-deletion access.&lt;/p&gt;

&lt;p&gt;There is an important limit here: our users also retained &lt;code&gt;update&lt;/code&gt; on Deployments. They could still change the template with a PUT request. We were preventing an easy accident, not preventing users from modifying workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the target explicit
&lt;/h2&gt;

&lt;p&gt;The job requires a resource name, validates its format, checks that the object exists, and takes the namespace from a fixed choice list. The restart path always uses &lt;code&gt;kind/name&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight groovy"&gt;&lt;code&gt;&lt;span class="c1"&gt;// After validating name and checking ns against the allowed namespaces:&lt;/span&gt;
&lt;span class="n"&gt;withEnv&lt;/span&gt;&lt;span class="o"&gt;([&lt;/span&gt;&lt;span class="s2"&gt;"NS=${ns}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"NAME=${name}"&lt;/span&gt;&lt;span class="o"&gt;])&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;sh&lt;/span&gt; &lt;span class="s1"&gt;'''
        set -eu
        : "${NS:?Namespace is required}"
        : "${NAME:?Deployment name is required}"
        kubectl get "deployment/$NAME" -n "$NS" -o name
        kubectl rollout restart "deployment/$NAME" -n "$NS"
        kubectl rollout status "deployment/$NAME" -n "$NS" --timeout=180s
    '''&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The shell rejects an empty name before kubectl runs. Even without that check, &lt;code&gt;deployment/&lt;/code&gt; fails instead of expanding the operation to the namespace.&lt;/p&gt;

&lt;p&gt;The single-quoted Groovy script matters too: validated input reaches the shell through environment variables and stays quoted, rather than becoming part of the script text.&lt;/p&gt;

&lt;p&gt;One easy permission to miss is &lt;code&gt;watch&lt;/code&gt;, which &lt;code&gt;rollout status&lt;/code&gt; needs. Otherwise the restart can succeed while the build fails to observe it. A status timeout also doesn't undo the restart; the rollout can continue after the build fails.&lt;/p&gt;

&lt;p&gt;The job records who requested the operation, the namespace, and the target in its build history. It also supports scaling to zero and deleting a pod, but those have different failure modes. Neither a rolling update nor a controller-owned pod guarantees zero downtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  Don't give the restart button cluster-admin
&lt;/h2&gt;

&lt;p&gt;The existing Jenkins deployer credential in our setup had cluster-admin access. Reusing it would have made this a very expensive restart button.&lt;/p&gt;

&lt;p&gt;Pipeline validation cannot compensate for that. Anyone who can modify or replay the Pipeline may be able to run different commands with credentials available to it. Jenkins permissions and credential scope matter as much as the form fields.&lt;/p&gt;

&lt;p&gt;We used a dedicated service account, bound only in the intended namespaces through RoleBindings. The restart path needs this Deployment rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deployments"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The other methods need their own permissions. None needs cluster-admin, Secret reads, or &lt;code&gt;pods/exec&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This credential still permits a namespace-wide Deployment restart. The job limits the normal workflow to one target; Kubernetes permissions limit where a modified workflow could act. With unrestricted Deployment &lt;code&gt;patch&lt;/code&gt;, it can also change more than the restart annotation.&lt;/p&gt;

&lt;p&gt;That distinction is the useful part of the exercise. We made the everyday operation harder to mistype, while keeping the automation's access narrow enough to inspect and test.&lt;/p&gt;

&lt;p&gt;And for a runbook or local script, the smallest useful improvement is just:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl rollout restart &lt;span class="s2"&gt;"deployment/&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SERVICE&lt;/span&gt;:?Set&lt;span class="p"&gt; SERVICE first&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; prod
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An empty variable should produce an error, not a namespace full of restarts ;)&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>jenkins</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>Resetting an Early 2011 MacBook Pro firmware password with a CH341B</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Wed, 14 Oct 2026 08:43:41 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/resetting-an-early-2011-macbook-pro-firmware-password-with-a-ch341b-47b5</link>
      <guid>https://dev.to/vitaliy_dmitriev/resetting-an-early-2011-macbook-pro-firmware-password-with-a-ch341b-47b5</guid>
      <description>&lt;h1&gt;
  
  
  Resetting an Early 2011 MacBook Pro firmware password with a CH341B
&lt;/h1&gt;

&lt;p&gt;I bought an Early 2011 MacBook Pro on OLX, listed as broken. It turned out to power on, but there was another problem: it had an EFI firmware password.&lt;/p&gt;

&lt;p&gt;The laptop came without an SSD or HDD, so there was no previous owner's disk or data to recover. I wanted to get the machine usable again.&lt;/p&gt;

&lt;p&gt;That turned into a small hardware project involving a CH341B programmer, an in-circuit clip, flashrom, and a Python script. After reflashing the firmware, the password was gone and I was able to boot macOS.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hardware and the starting point
&lt;/h2&gt;

&lt;p&gt;My machine was a 15-inch Early 2011 MacBook Pro with a Macronix MX25L6406E flash chip. Its capacity is 8 MiB, or 8,388,608 bytes.&lt;/p&gt;

&lt;p&gt;I used a CH341B programmer and an in-circuit clip, so I didn't desolder the chip. For chip access and connection details, I followed this &lt;a href="https://gist.github.com/willzhang05/e5b5563cdc65514dfb7ca131e03ca4b2" rel="noopener noreferrer"&gt;existing EFI firmware-password removal guide&lt;/a&gt;. The underlying technique isn't my discovery; my contribution was documenting my setup and adding a small helper for modifying the dump.&lt;/p&gt;

&lt;p&gt;I didn't photograph the process before putting the laptop back together, so this is a write-up of my repair, not a complete board-level connection guide. Before connecting a programmer, check the chip's voltage requirements, the programmer's actual supply and signal levels, clip orientation, and the board's power-isolation requirements. A similar-looking programmer or logic board isn't enough to establish compatibility.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read first, keep the original
&lt;/h2&gt;

&lt;p&gt;Using flashrom on macOS, I read the chip twice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;flashrom &lt;span class="nt"&gt;-p&lt;/span&gt; ch341a_spi &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"MX25L6406E/MX25L6408E"&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; dump1.bin

&lt;span class="nb"&gt;sudo &lt;/span&gt;flashrom &lt;span class="nt"&gt;-p&lt;/span&gt; ch341a_spi &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"MX25L6406E/MX25L6408E"&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; dump2.bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The programmer was a CH341B, although flashrom's programmer option is named &lt;code&gt;ch341a_spi&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Then I compared the dumps and kept an original copy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cmp dump1.bin dump2.bin
shasum &lt;span class="nt"&gt;-a&lt;/span&gt; 256 dump1.bin dump2.bin
&lt;span class="nb"&gt;cp &lt;/span&gt;dump1.bin original.bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;cmp&lt;/code&gt; produces no output when the files match. The hashes provide another easy comparison. If the reads differ, stop and investigate before modifying anything.&lt;/p&gt;

&lt;p&gt;Matching reads are a useful check, not proof that everything is correct. A consistently bad connection can still produce repeatable output. The expected file size and plausible contents matter too. Keep a separate private backup of the original, beyond the working directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Replace bytes, don't delete them
&lt;/h2&gt;

&lt;p&gt;The modification concerns regions identified by the &lt;code&gt;$SVS&lt;/code&gt; signature. The important practical detail is that the firmware image must keep its layout: removing bytes shifts everything that follows.&lt;/p&gt;

&lt;p&gt;The helper in my gist replaces the selected bytes with &lt;code&gt;0xFF&lt;/code&gt; instead. The output stays the same size as the input.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 patch_svs.py dump1.bin modified.bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script checks for an 8 MiB input, locates the signatures, prints the proposed ranges and surrounding bytes, and asks for confirmation before creating the output.&lt;/p&gt;

&lt;p&gt;There is a limitation worth making explicit: it finds the end of each region by looking for the next run of eight &lt;code&gt;0xFF&lt;/code&gt; bytes. That's a heuristic, not a full parser for the firmware format. The script also processes every matching signature it finds.&lt;/p&gt;

&lt;p&gt;So don't treat it as a universal MacBook password-reset tool. Review the reported ranges for the specific image, check that the output size is unchanged, and inspect the differences before writing anything back. An unchanged size alone doesn't prove the modification is correct.&lt;/p&gt;

&lt;p&gt;Also, use distinct input and output files. The current helper doesn't prevent you from choosing the original as the output, which would overwrite it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Flashing and the result
&lt;/h2&gt;

&lt;p&gt;Once the modified image was ready, I reflashed the chip. The write command in my notes is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;flashrom &lt;span class="nt"&gt;-p&lt;/span&gt; ch341a_spi &lt;span class="nt"&gt;-V&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"MX25L6406E/MX25L6408E"&lt;/span&gt; &lt;span class="nt"&gt;-w&lt;/span&gt; modified.bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Flashrom normally verifies the contents after writing unless verification is disabled. The uppercase &lt;code&gt;-V&lt;/code&gt; enables verbose output; it isn't the verification switch.&lt;/p&gt;

&lt;p&gt;Writing firmware is the destructive part of this process. Keep the original available and remember that restoring it still depends on being able to communicate reliably with the chip.&lt;/p&gt;

&lt;p&gt;In my case, reflashing removed the firmware password and I could boot macOS. That's the result I verified, rather than a claim that every function of the laptop was tested afterward.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this applies to
&lt;/h2&gt;

&lt;p&gt;This worked on my Early 2011 machine. I haven't established compatibility with other models or firmware revisions. It isn't an Activation Lock bypass or a way to decrypt FileVault-protected data.&lt;/p&gt;

&lt;p&gt;Apple's &lt;a href="https://support.apple.com/en-gb/102384" rel="noopener noreferrer"&gt;documented procedure for a forgotten Intel Mac firmware password&lt;/a&gt; is an in-person service appointment with proof of purchase. The procedure above is an independent hardware-repair approach for a machine I purchased, not Apple's supported recovery workflow.&lt;/p&gt;

&lt;p&gt;I've kept the &lt;a href="https://gist.github.com/vi7/5f4224c8d9cfb11b0900222b2d98d137" rel="noopener noreferrer"&gt;commands and Python helper in this gist&lt;/a&gt;. If you're working on a similar repair, the useful starting point is an original dump you can trust, a backup you haven't overwritten, and a clear understanding of what the script proposes to change.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Cover photo: © Raimond Spekking / CC BY-SA 4.0 (via Wikimedia Commons). Reference photograph; not my repair.&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>hardware</category>
      <category>python</category>
      <category>laptop</category>
      <category>repair</category>
    </item>
    <item>
      <title>I gave an AI agent a Terraform task. It found a credentials problem.</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Wed, 14 Oct 2026 08:31:37 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/i-gave-an-ai-agent-a-terraform-task-it-found-a-credentials-problem-4n5o</link>
      <guid>https://dev.to/vitaliy_dmitriev/i-gave-an-ai-agent-a-terraform-task-it-found-a-credentials-problem-4n5o</guid>
      <description>&lt;h1&gt;
  
  
  I gave an AI agent a Terraform task. It found a credentials problem.
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Part 1 of 3 in "Terraform with an AI co-worker: access, testing, review".&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The ticket was ordinary: create an S3 bucket, two SQS queues and an IAM user across five environments, using the existing Terraform setup.&lt;/p&gt;

&lt;p&gt;I gave it to an AI agent. It prepared the code and ran its checks, but deployment hit a problem. The AWS profiles Terraform expected weren't available on the machine. Investigating those profiles led to something more interesting than the ticket itself: permanent administrator keys shared with automated jobs.&lt;/p&gt;

&lt;p&gt;The agent stopped and reported the problem. It didn't ask me to paste the keys into the chat or start replacing credentials that other workloads depended on.&lt;/p&gt;

&lt;p&gt;That pause was probably the most useful part of this stage of the work.&lt;/p&gt;

&lt;p&gt;This series follows the same Terraform task through three stages: sorting out access, testing the change, and reviewing the work. This first part is about the credentials the agent could use and what we could verify before allowing a deployment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The profiles were familiar. The credentials behind them were less reassuring.
&lt;/h2&gt;

&lt;p&gt;Our Terraform configuration expected one AWS profile per environment, with names like &lt;code&gt;stage-user-terraform&lt;/code&gt; and &lt;code&gt;prd-user-terraform&lt;/code&gt;. Each pointed to an IAM user with long-lived access keys and administrator permissions. The Terraform state was stored in S3 in the AWS organization's management account.&lt;/p&gt;

&lt;p&gt;The keys weren't on the agent's machine. Using read-only queries, it found that the existing keys had been used that day by automated data jobs. In two environments, the users already had two keys each, which is the AWS limit per IAM user.&lt;/p&gt;

&lt;p&gt;Creating another key for those users would first require deleting one. That would turn a small infrastructure ticket into a credential rotation for workloads outside its scope.&lt;/p&gt;

&lt;p&gt;There were several tempting shortcuts: copy an existing secret from wherever it was stored, replace a key to make room, or use a more privileged identity because it happened to work. None belonged in this task.&lt;/p&gt;

&lt;p&gt;Also, AWS doesn't let you retrieve an existing secret access key after creation. Inspecting key metadata and usage is different from having the credential itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rules helped. They didn't remove the permissions.
&lt;/h2&gt;

&lt;p&gt;The agent had standing instructions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Read-only investigation was pre-approved; important changes needed human approval.&lt;/li&gt;
&lt;li&gt;It was not to delete or modify resources it hadn't created without explicit authorization.&lt;/li&gt;
&lt;li&gt;Missing credentials and ambiguous ownership were reasons to stop and ask.&lt;/li&gt;
&lt;li&gt;A refused operation was not permission to find another way around the refusal.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These rules explain the behavior I wanted. They don't make an administrator role read-only.&lt;/p&gt;

&lt;p&gt;That distinction matters here because the SSO role we eventually used still had administrator permissions. We improved credential handling, but the agent's restraint and the approval workflow remained part of the boundary. A dedicated role with narrower permissions is the next step.&lt;/p&gt;

&lt;p&gt;There was also a small local example. A Terraform run left a generated file in the repository, and the safety layer blocked the agent's attempt to delete it. The agent moved it into a holding directory and reported the location, leaving final deletion to me.&lt;/p&gt;

&lt;p&gt;That preserved the file, but I wouldn't present it as proof that the guardrail worked perfectly. Moving a file also changes the working tree. Whether that is an acceptable alternative depends on what the restriction is meant to protect. The useful detail was that the agent reported the alternative instead of silently removing the file another way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the profile names, change how we authenticate
&lt;/h2&gt;

&lt;p&gt;The agent proposed four options. We chose one that left the Terraform code and the shared keys alone: keep the expected profile names, but resolve them through personal AWS IAM Identity Center logins, commonly called AWS SSO.&lt;/p&gt;

&lt;p&gt;Those logins provide temporary role credentials instead of another permanent key. We put the profiles in a separate AWS config file, selected explicitly when running Terraform.&lt;/p&gt;

&lt;p&gt;Here is an illustrative version, with placeholders:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# ~/.aws/config-terraform
# Profile configuration only, no secret access keys.
&lt;/span&gt;&lt;span class="nn"&gt;[sso-session company]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;sso_start_url&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;https://&amp;lt;your-sso-portal&amp;gt;/start&lt;/span&gt;
&lt;span class="py"&gt;sso_region&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;us-east-1&lt;/span&gt;
&lt;span class="py"&gt;sso_registration_scopes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;sso:account:access&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[profile stage-user-terraform]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;sso_session&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;company&lt;/span&gt;
&lt;span class="py"&gt;sso_account_id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;stage-account-id&amp;gt;&lt;/span&gt;
&lt;span class="py"&gt;sso_role_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;your-permission-set&amp;gt;&lt;/span&gt;
&lt;span class="py"&gt;region&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;us-east-1&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c"&gt;# Existing backend profile name; this is NOT the AWS root user.
&lt;/span&gt;&lt;span class="nn"&gt;[profile root-user-terraform]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;sso_session&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;company&lt;/span&gt;
&lt;span class="py"&gt;sso_account_id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;management-account-id&amp;gt;&lt;/span&gt;
&lt;span class="py"&gt;sso_role_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;your-permission-set&amp;gt;&lt;/span&gt;
&lt;span class="py"&gt;region&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;us-east-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;root-user-terraform&lt;/code&gt; name is unfortunate, but it was already what the backend expected. Here it refers to an SSO role in the management account, not root credentials.&lt;/p&gt;

&lt;p&gt;We logged in with the separate configuration and selected it inside a subshell for the Terraform commands. An investigation command looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;AWS_CONFIG_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.aws/config-terraform"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  aws sso login &lt;span class="nt"&gt;--sso-session&lt;/span&gt; company

&lt;span class="o"&gt;(&lt;/span&gt;
  &lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;AWS_CONFIG_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.aws/config-terraform"&lt;/span&gt;
  &lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;AWS_PROFILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;root-user-terraform
  &lt;span class="nb"&gt;cd &lt;/span&gt;infra &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1
  terraform workspace &lt;span class="k"&gt;select &lt;/span&gt;stage &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1
  terraform plan &lt;span class="nt"&gt;-lock&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt; &lt;span class="nt"&gt;-target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;module.my-change
&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The paths, workspace and module address are examples. This assumes an initialized backend and an AWS provider configuration that explicitly selects the environment profile. The backend and provider authenticate separately; setting &lt;code&gt;AWS_PROFILE&lt;/code&gt; alone doesn't separate their access.&lt;/p&gt;

&lt;p&gt;The two plan flags also need context. We used &lt;code&gt;-lock=false&lt;/code&gt; during the investigation to avoid a remote lock write, and targeting to inspect the ticket's scope. These are not defaults I'd use for deployment. An unlocked plan can become stale if another operation changes the state, and targeting can omit changes outside the selected dependency graph.&lt;/p&gt;

&lt;p&gt;The separate config file reduces accidental profile use, and the subshell keeps these environment settings from remaining in my interactive shell. Neither prevents a process with access to the file and cached SSO session from selecting the profile again. Exiting the subshell doesn't revoke the session.&lt;/p&gt;

&lt;p&gt;Before using this pattern, check credential sources and precedence too. Existing environment credentials or static credentials under the same profile names can interfere with the intended login. The Terraform S3 backend and the AWS provider also need versions that support your SSO configuration.&lt;/p&gt;

&lt;p&gt;So what did we actually improve? We avoided distributing another long-lived secret, left the pipeline credentials untouched, and used personal role sessions with better attribution than a shared IAM user. &lt;em&gt;The remaining admin permissions still deserved attention.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Could it write the state? We gathered evidence without trying a write.
&lt;/h2&gt;

&lt;p&gt;Reading infrastructure was only part of the access question. An eventual apply also needed permission to update the state in the management account and use the configured locking mechanism.&lt;/p&gt;

&lt;p&gt;We kept this investigation read-only:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Run a Terraform plan with locking disabled for each environment we could authenticate to. This checked state reads and the AWS reads needed by that plan.&lt;/li&gt;
&lt;li&gt;Inspect the state bucket's policy. There was no bucket policy.&lt;/li&gt;
&lt;li&gt;Use the IAM policy simulator to evaluate the relevant S3 actions against the bucket and object resources. The simulated result was &lt;code&gt;allowed&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The plan demonstrated that the reads it actually performed worked. The simulation supplied evidence for write permissions, but it wasn't an end-to-end write test. No bucket policy also doesn't, by itself, mean writes are allowed.&lt;/p&gt;

&lt;p&gt;AWS documents limitations in the simulator, including resource-policy simulation for IAM roles and differences from live request evaluation. A result depends on the policies and context being evaluated. I wouldn't turn &lt;code&gt;allowed&lt;/code&gt; into a promise that the eventual apply will succeed.&lt;/p&gt;

&lt;p&gt;An independent reviewer agent rechecked the evidence and inspected encryption and organization controls. The reported bucket encryption didn't require a customer-managed KMS key. It also confirmed that SCPs don't restrict identities in the management account. That last point makes the account more sensitive, not less.&lt;/p&gt;

&lt;p&gt;Locking permissions need their own check. S3 lock files require object permissions on the lock path; a DynamoDB-backed lock needs the relevant table permissions instead. An unlocked plan doesn't exercise either.&lt;/p&gt;

&lt;p&gt;Two environments had expired SSO logins and weren't checked. We reported them as unverified rather than extending the result from the other environments.&lt;/p&gt;

&lt;h2&gt;
  
  
  Access wasn't the only thing blocking an apply
&lt;/h2&gt;

&lt;p&gt;The real plan also exposed a mismatch between the shared Terraform configuration and the deployed environments. A full apply in two test environments would have included other teams' changes and deleted existing storage objects, well outside this ticket.&lt;/p&gt;

&lt;p&gt;We chose a targeted deployment approach for this change and left the broader reconciliation as separate work. That didn't make the plan automatically safe: &lt;code&gt;-target&lt;/code&gt; can include dependencies and leave other changes unapplied. I'll return to the plan's scope and the testing in Part 2.&lt;/p&gt;

&lt;p&gt;Getting the right credentials was necessary. It wasn't permission to apply everything those credentials could reach.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this changed for me
&lt;/h2&gt;

&lt;p&gt;The agent prepared the change and investigated the access problem. A person still runs the apply.&lt;/p&gt;

&lt;p&gt;We didn't rotate shared keys, create another administrator key or change the Terraform code just to get past the missing profiles. We found a way to use temporary personal credentials, while being explicit that those credentials still carried too much power for a narrowly scoped task.&lt;/p&gt;

&lt;p&gt;For a team starting with AI-assisted infrastructure work, that is a useful first question: what identity will the agent use, and what can that identity actually do?&lt;/p&gt;

&lt;p&gt;In this case, the missing login exposed an access arrangement worth revisiting. I would rather get that finding before deployment than discover the dependency on shared admin keys halfway through replacing one.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This is Part 1 of "Terraform with an AI co-worker: access, testing, review".&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;em&gt;Part 2 will cover testing: using a temporary bucket and a real upload to check the change, cleaning up the test resources, and handling the unrelated changes in the plan.&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Part 3 will cover review: what independent AI reviewers checked, what they caught, and which conclusions still needed human judgement.&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Technical references
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html" rel="noopener noreferrer"&gt;AWS: managing IAM access keys&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/cli/latest/topic/config-vars.html" rel="noopener noreferrer"&gt;AWS: credential precedence&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html" rel="noopener noreferrer"&gt;AWS: IAM policy simulator&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html" rel="noopener noreferrer"&gt;AWS: SCP effects on permissions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developer.hashicorp.com/terraform/language/backend/s3" rel="noopener noreferrer"&gt;HashiCorp: S3 backend permissions and locking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developer.hashicorp.com/terraform/tutorials/state/resource-targeting" rel="noopener noreferrer"&gt;HashiCorp: resource targeting&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>terraform</category>
      <category>aws</category>
      <category>devops</category>
      <category>ai</category>
    </item>
    <item>
      <title>I let an AI agent make an infrastructure change</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Wed, 07 Oct 2026 07:59:37 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/i-let-an-ai-agent-make-an-infrastructure-change-lbd</link>
      <guid>https://dev.to/vitaliy_dmitriev/i-let-an-ai-agent-make-an-infrastructure-change-lbd</guid>
      <description>&lt;h1&gt;
  
  
  I let an AI agent make an infrastructure change
&lt;/h1&gt;

&lt;p&gt;An AI agent recently took an infrastructure ticket from "please create these queues" to deployed and tested.&lt;/p&gt;

&lt;p&gt;It also:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;found a bug I would probably have shipped;&lt;/li&gt;
&lt;li&gt;briefly broke a consumer;&lt;/li&gt;
&lt;li&gt;confidently gave me the wrong explanation for some infrastructure versus code drift.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All three were useful.&lt;/p&gt;

&lt;p&gt;The first one showed me what agents can do.&lt;/p&gt;

&lt;p&gt;The other two showed me what they need before you let them touch infrastructure.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ticket
&lt;/h2&gt;

&lt;p&gt;Typical BAU ticket:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Create a few message queues, configure dead-lettering and permissions, and make them available to the application.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Normally this is a small DevOps task.&lt;/p&gt;

&lt;p&gt;But infrastructure changes are rarely isolated. The broker was shared, there were several environments, existing applications depended on it, and some configuration was generated from code.&lt;/p&gt;

&lt;p&gt;So I decided not to do the change myself.&lt;/p&gt;

&lt;p&gt;I gave it to an AI coding agent running in a terminal.&lt;/p&gt;

&lt;p&gt;Its permissions were deliberately asymmetric:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It could read almost everything. It could change almost nothing without my approval.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;My rules were simple:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;production is read-only;&lt;/li&gt;
&lt;li&gt;non-production changes require approval;&lt;/li&gt;
&lt;li&gt;changes must be reversible;&lt;/li&gt;
&lt;li&gt;don't guess when something is ambiguous.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent followed a fixed workflow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;explore
  ↓
plan + pre-mortem
  ↓
independent review
  ↓
questions to human
  ↓
implement
  ↓
non-production rollout
  ↓
test
  ↓
review
  ↓
handoff
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This turned out to matter more than the model itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first useful thing: it didn't start coding
&lt;/h2&gt;

&lt;p&gt;The agent first explored the environment.&lt;/p&gt;

&lt;p&gt;That immediately found several things I would have been tempted to skip.&lt;/p&gt;

&lt;h3&gt;
  
  
  The ticket was ambiguous
&lt;/h3&gt;

&lt;p&gt;The non-production broker was shared by several environments.&lt;/p&gt;

&lt;p&gt;The queue names in the ticket didn't contain an environment identifier, so blindly creating them would have made different test environments share the same queues.&lt;/p&gt;

&lt;p&gt;The agent stopped and asked which environments actually needed the change.&lt;/p&gt;

&lt;p&gt;Good catch.&lt;/p&gt;

&lt;h3&gt;
  
  
  There was already a pattern
&lt;/h3&gt;

&lt;p&gt;Other team members and I had implemented something very similar before.&lt;/p&gt;

&lt;p&gt;Instead of inventing a new configuration, the agent found the existing implementation and used it as a reference.&lt;/p&gt;

&lt;p&gt;This is one of the least glamorous but most valuable things an agent can do:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;look around before inventing things.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  It found a potential time bomb
&lt;/h3&gt;

&lt;p&gt;The planned queue configuration had a dangerous default. Under the expected failure scenario, messages could accumulate in memory while the consumer was unavailable. Enough messages could eventually trigger the broker's memory protection and affect unrelated applications. The agent suggested adding a size limit and changing the storage policy.&lt;/p&gt;

&lt;p&gt;That was useful.&lt;/p&gt;

&lt;p&gt;But &lt;em&gt;I didn't trust the suggestion&lt;/em&gt; just because it sounded reasonable.&lt;/p&gt;

&lt;p&gt;I asked for an independent review.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second agent found the real problem
&lt;/h2&gt;

&lt;p&gt;A separate agent reviewed the plan. Its first finding was marked &lt;strong&gt;CRITICAL&lt;/strong&gt;. There was a regular expression containing an escaped dot.&lt;/p&gt;

&lt;p&gt;Something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;amq\.default
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Perfectly valid regex. Except this value eventually ended up inside JSON. And &lt;code&gt;\.&lt;/code&gt; isn't a valid JSON escape.&lt;/p&gt;

&lt;p&gt;The interesting part was when this would fail. Not during deployment. Not during the usual validation.&lt;/p&gt;

&lt;p&gt;The broker only read that particular configuration during a restart.&lt;/p&gt;

&lt;p&gt;So the change could have been deployed successfully, tested successfully, and then caused a failure weeks later when somebody restarted a node.&lt;/p&gt;

&lt;p&gt;That's exactly the kind of infrastructure bug I want automation to catch.&lt;/p&gt;

&lt;p&gt;The fix was tiny.&lt;/p&gt;

&lt;p&gt;The important change was in the process:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;don't validate the template. Validate the actual artefact the system will consume.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Render it. Parse it. Do that for every environment.&lt;/p&gt;

&lt;p&gt;The reviewer also found a few other problems in the planned tests and configuration.&lt;/p&gt;

&lt;p&gt;This was a good reminder that an agent doesn't have to produce the final answer to be useful.&lt;/p&gt;

&lt;p&gt;Sometimes &lt;em&gt;its best job&lt;/em&gt; is simply:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Here are five things you haven't thought about."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Then we changed the infrastructure
&lt;/h2&gt;

&lt;p&gt;After I answered its questions, the agent implemented the change. It didn't import an entire configuration file.&lt;/p&gt;

&lt;p&gt;It made targeted changes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;create these objects
change these permissions
leave everything else alone
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That sounds obvious, but it's an important property for infrastructure automation.&lt;/p&gt;

&lt;p&gt;A command that says "make the whole configuration look like this" can have a very different blast radius from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;create X
create Y
modify Z
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then it tested the change using the real application identity.&lt;/p&gt;

&lt;p&gt;It tested the happy path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;publish
→ consume
→ fail
→ retry
→ retry
→ retry
→ dead-letter
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It also tested the negative cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Could the application access things it shouldn't?&lt;/li&gt;
&lt;li&gt;Could it create things it shouldn't?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The answer needed to be no.&lt;/p&gt;

&lt;p&gt;And then came one of my favourite tests. The agent deliberately filled the queue until the configured limit was reached.&lt;/p&gt;

&lt;p&gt;If you configure a limit but never actually hit it, you don't really know whether the limit works. A limit you haven't triggered is still partly a theory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then I let it restart the nodes
&lt;/h2&gt;

&lt;p&gt;The infrastructure change also needed a rolling restart.&lt;/p&gt;

&lt;p&gt;Before doing that, the agent checked for configuration differences between the repository and the live system. It also checked whether there were messages that could be affected by the restart. There was one queue containing messages that would not survive the restart.&lt;/p&gt;

&lt;p&gt;It asked, and I approved it.&lt;/p&gt;

&lt;p&gt;The restart began, and everything looked great: all nodes were healthy, replicas were synchronized, no broker alarms, the dashboard was green.&lt;/p&gt;

&lt;p&gt;Then the agent found a problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The dashboard was green. The application wasn't.
&lt;/h2&gt;

&lt;p&gt;One test consumer had stopped consuming, while its Pod was healthy.&lt;/p&gt;

&lt;p&gt;The cluster was healthy, the monitoring dashboard was mostly green, BUT the application was still broken.&lt;/p&gt;

&lt;p&gt;The restart had caused the application to reconnect and redeclare its queue. The queue already existed with slightly different settings from what the application expected.&lt;/p&gt;

&lt;p&gt;The broker rejected the declaration and the application stopped consuming, while its health check didn't notice.&lt;/p&gt;

&lt;p&gt;This lasted for about ten minutes. No customer impact, because it was a test environment.&lt;/p&gt;

&lt;p&gt;But it exposed a very important problem with the original verification:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I was checking server health, not client health.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The agent had done what I asked, the cluster was healthy, but that wasn't enough. :(&lt;/p&gt;

&lt;p&gt;So we changed the procedure - before every disruptive operation, we now capture the important client-side state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;consumers
connections
restart counts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the operation, we compare it. Cluster health tells you that the cluster is healthy, but it doesn't tell you that your applications are happy.&lt;/p&gt;

&lt;p&gt;Those are two different questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  One more miss: the agent got drift wrong
&lt;/h2&gt;

&lt;p&gt;Earlier, the agent had found three broker policies present in the live system but missing from Git. It confidently called them manual drift. I believed it and even warned the team.&lt;/p&gt;

&lt;p&gt;Later, I asked it to check Git history. Turns out two policies &lt;em&gt;had&lt;/em&gt; been in the repo, but were removed months earlier. The broker's config import added settings without deleting old ones, so the policies stayed live. Worse, those two policies were still protecting dead-letter queues. Removing them as “drift” would have made things less reliable.&lt;/p&gt;

&lt;p&gt;Only one policy was actually created manually. The agent had checked the current repo, skipped its history, and filled the gap with a plausible explanation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Plausible isn't verified.&lt;/strong&gt; Now any drift claim needs a history check and evidence before anyone acts on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  So we changed the rules
&lt;/h2&gt;

&lt;p&gt;Each failure became a new rule.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;New rule&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A restart broke a consumer&lt;/td&gt;
&lt;td&gt;Notify the human immediately when user-facing behaviour changes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cluster was healthy but client was broken&lt;/td&gt;
&lt;td&gt;Compare client-side state before and after disruptive operations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Drift" was based on current Git state only&lt;/td&gt;
&lt;td&gt;Check Git history before calling something drift&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Review repeated facts from the plan&lt;/td&gt;
&lt;td&gt;Reviewers must independently verify important facts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A plausible statement had no evidence&lt;/td&gt;
&lt;td&gt;Every important fact must have a command, query or source behind it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A configuration file could fail only after restart&lt;/td&gt;
&lt;td&gt;Validate the exact artefact consumed by the system&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is probably the most important part of the whole experiment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The agent is allowed to be wrong.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The system around the agent shouldn't make the same mistake twice.&lt;/p&gt;

&lt;p&gt;The workflow itself now contains these checks, so I don't have to remember them next time.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means for a business
&lt;/h2&gt;

&lt;p&gt;If you're thinking about using AI to automate DevOps or infrastructure work, I wouldn't start by giving an agent production access. I'd start much more boringly.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Give it read access first
&lt;/h3&gt;

&lt;p&gt;Let it investigate.&lt;/p&gt;

&lt;p&gt;Let it correlate logs and metrics.&lt;/p&gt;

&lt;p&gt;Let it find configuration.&lt;/p&gt;

&lt;p&gt;Let it prepare a change.&lt;/p&gt;

&lt;p&gt;You can learn a lot about the agent without giving it the ability to break anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Gate writes with credentials
&lt;/h3&gt;

&lt;p&gt;This is important.&lt;/p&gt;

&lt;p&gt;Don't rely on:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Don't touch production."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a prompt.&lt;/p&gt;

&lt;p&gt;Instead:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The production credential literally cannot modify production.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a control.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Keep approval where the blast radius begins
&lt;/h3&gt;

&lt;p&gt;The agent can prepare the change.&lt;/p&gt;

&lt;p&gt;The human approves it.&lt;/p&gt;

&lt;p&gt;You can gradually automate more once you understand the failure modes.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Make the agent ask questions efficiently
&lt;/h3&gt;

&lt;p&gt;One message with five numbered questions is useful.&lt;/p&gt;

&lt;p&gt;Five separate interruptions are annoying.&lt;/p&gt;

&lt;p&gt;A good agent should come back with something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. These four environments need the queues. Agree?
2. Existing pattern A looks closest. Use it?
3. This policy may affect memory usage. Apply it?
4. One queue contains messages that won't survive restart. Proceed?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That turns the human into &lt;em&gt;a decision maker&lt;/em&gt; instead of a human API.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Turn mistakes into guardrails
&lt;/h3&gt;

&lt;p&gt;This is where I think agentic automation becomes really interesting.&lt;/p&gt;

&lt;p&gt;A mistake shouldn't just produce:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The AI made a mistake."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It should produce:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"What rule can we add so neither the AI nor the next engineer makes this mistake again?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's how the system gets better.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually changed for me
&lt;/h2&gt;

&lt;p&gt;The agent didn't replace the DevOps engineer.&lt;/p&gt;

&lt;p&gt;It changed what the DevOps engineer was doing.&lt;/p&gt;

&lt;p&gt;Instead of spending the morning:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;read ticket
find config
search Git
check cluster
write YAML
run commands
test
write documentation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I spent much more of the time asking:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What do we actually need to change?

What could go wrong?

How will we know it worked?

What must never be allowed?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are much more valuable questions to automate around.&lt;/p&gt;

&lt;p&gt;And there is an interesting business implication here.&lt;/p&gt;

&lt;p&gt;You don't necessarily need an AI that can autonomously run your entire infrastructure. You need an AI that can take a well-defined piece of infrastructure work and move it through a &lt;strong&gt;safe, repeatable process&lt;/strong&gt;, while escalating the decisions that actually require human judgement.&lt;/p&gt;

&lt;p&gt;That's a much smaller problem.&lt;/p&gt;

&lt;p&gt;And, in my experience, a much more useful one.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you're starting with AI in DevOps
&lt;/h2&gt;

&lt;p&gt;Three habits are worth stealing even if you never use an AI agent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prove your limits.&lt;/strong&gt; If you configure a timeout, retry count or size limit, trigger it deliberately and see what happens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check clients, not only servers.&lt;/strong&gt; "The cluster is healthy" does not mean "the application works."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent didn't replace judgement. It made the places where judgement was missing much easier to see.&lt;/p&gt;

&lt;p&gt;And that's probably the part of this experiment I found most valuable.&lt;/p&gt;




&lt;p&gt;I'm experimenting with AI-assisted infrastructure automation, particularly around DevOps and on-prem environments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Would you trust an AI agent to make a production infrastructure change if the permissions, approval gates and rollback path were designed correctly?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I'm curious where other teams draw that line.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>automation</category>
      <category>sre</category>
    </item>
    <item>
      <title>Don't put an LLM where a bash script will do</title>
      <dc:creator>Vitaliy Dmitriev</dc:creator>
      <pubDate>Tue, 06 Oct 2026 08:58:11 +0000</pubDate>
      <link>https://dev.to/vitaliy_dmitriev/dont-put-an-llm-where-a-bash-script-will-do-728</link>
      <guid>https://dev.to/vitaliy_dmitriev/dont-put-an-llm-where-a-bash-script-will-do-728</guid>
      <description>&lt;h1&gt;
  
  
  Don't put an LLM where a bash script will do
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Or: how I almost gave a language model a login, a service account and a cron job to check four numbers.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Recently I was debugging a production Prometheus exporter that was restarting during the night. It wasn't crashing in the usual sense. It was getting killed by Kubernetes because its liveness probe timed out.&lt;/p&gt;

&lt;p&gt;The reason was fairly simple: the exporter depended on another service that became slow when the nightly jobs started. The probe had a short timeout, so Kubernetes eventually decided the exporter was dead and restarted it.&lt;/p&gt;

&lt;p&gt;The fix was boring: increase the probe timeout, adjust the monitoring timeout, deploy the change.&lt;/p&gt;

&lt;p&gt;The interesting question was what happens after the deployment.&lt;/p&gt;

&lt;p&gt;The fix only really matters at 3 a.m. That's when the nightly load happens. I didn't want to wake up at 3 a.m. just to check whether the exporter survived. So I needed some kind of automated verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious AI solution
&lt;/h2&gt;

&lt;p&gt;I had been using an AI agent during the investigation and it was actually quite useful. It helped correlate the restarts with the nightly load, look through metrics and logs, find a problem with one of my assumptions, and prepare the ticket and merge requests.&lt;/p&gt;

&lt;p&gt;So the next idea seemed reasonable:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Run the agent every morning for a few days and ask it whether the fix worked.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The agent could query Prometheus, look at the deployment and send me a message. Then I started thinking about what would actually be required to run an agent unattended.&lt;/p&gt;

&lt;p&gt;It would need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a dedicated non-root user;&lt;/li&gt;
&lt;li&gt;its own authentication to the model provider;&lt;/li&gt;
&lt;li&gt;read-only access to the monitoring system;&lt;/li&gt;
&lt;li&gt;some kind of K8S credentials if it needed its data;&lt;/li&gt;
&lt;li&gt;a restricted tool set;&lt;/li&gt;
&lt;li&gt;a timeout and a way to stop it;&lt;/li&gt;
&lt;li&gt;logs;&lt;/li&gt;
&lt;li&gt;and probably some tests around the whole thing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's a lot of infrastructure for a question that turned out to be:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Did the exporter restart?&lt;/li&gt;
&lt;li&gt;Was it up during the whole observation period?&lt;/li&gt;
&lt;li&gt;What was the maximum successful scrape duration?&lt;/li&gt;
&lt;li&gt;Did the alert fire?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's four PromQL queries and four comparisons.&lt;/p&gt;

&lt;p&gt;At that point I stopped.&lt;/p&gt;

&lt;h2&gt;
  
  
  The boring solution
&lt;/h2&gt;

&lt;p&gt;I wrote a shell script instead. It was around 80 lines, most of which were notification and error handling. The actual check was tiny.&lt;/p&gt;

&lt;p&gt;Something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;query&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    curl &lt;span class="nt"&gt;-sk&lt;/span&gt; &lt;span class="nt"&gt;--max-time&lt;/span&gt; 25 &lt;span class="nt"&gt;-G&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PROM&lt;/span&gt;&lt;span class="s2"&gt;/api/v1/query"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
        &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"query=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; |
        jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'
            select(.status == "success") |
            .data.result[0].value[1] // "none"
        '&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

check&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;local &lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;local &lt;/span&gt;&lt;span class="nv"&gt;query&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;local &lt;/span&gt;&lt;span class="nv"&gt;condition&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$3&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

    &lt;span class="nb"&gt;local &lt;/span&gt;value
    &lt;span class="nv"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;query &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$query&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        errors+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;: query failed"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"none"&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        errors+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;: no data"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    jq &lt;span class="nt"&gt;-en&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; v &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
        &lt;span class="s1"&gt;'($v | tonumber) as $v | '&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$condition&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;||&lt;/span&gt;
        failures+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt; = &lt;/span&gt;&lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

check restarts &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'...'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'$v == 0'&lt;/span&gt;

check &lt;span class="nb"&gt;uptime&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'...'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'$v == 1'&lt;/span&gt;

check scrape_time &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'...'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'$v &amp;lt; 25'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The real script had a few more checks and notification handling, but the principle was the same.&lt;/p&gt;

&lt;p&gt;There were three possible results:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CONFIRMED&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Everything was within the expected limits.&lt;/p&gt;

&lt;p&gt;No notification during the observation period. On the last day, send one short confirmation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;REGRESSED&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Something went wrong.&lt;/p&gt;

&lt;p&gt;Send me a message with the relevant result and the rollback procedure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;INCONCLUSIVE&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The monitoring query failed, there was no data, or the thing being checked no longer existed.&lt;/p&gt;

&lt;p&gt;Send a message too.&lt;/p&gt;

&lt;p&gt;This last case is important:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Silence should mean "checked and everything is fine", not "the checker failed".&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So I also wrote one JSONL record for every run. Nothing fancy, just enough to know that the check actually happened.&lt;/p&gt;

&lt;h3&gt;
  
  
  One useful detail
&lt;/h3&gt;

&lt;p&gt;The script also checked that the deployment being verified was still the one I intended to verify.&lt;/p&gt;

&lt;p&gt;I used an identifier associated with the newly deployed workload rather than simply asking Prometheus about "the exporter".&lt;/p&gt;

&lt;p&gt;That matters because deployments can change while your verification job is waiting.&lt;/p&gt;

&lt;p&gt;If somebody rolls back or deploys another version, the original workload disappears. In that case the script reports &lt;code&gt;INCONCLUSIVE&lt;/code&gt; instead of accidentally verifying the new deployment.&lt;/p&gt;

&lt;p&gt;It's a small detail, but it prevents a particularly annoying kind of false positive.&lt;/p&gt;

&lt;h2&gt;
  
  
  systemd instead of cron
&lt;/h2&gt;

&lt;p&gt;I could have used cron, but systemd made the rest of this surprisingly simple.&lt;/p&gt;

&lt;p&gt;The service looked roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;
&lt;span class="py"&gt;ConditionPathExists&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;!/etc/example/PAUSE&lt;/span&gt;

&lt;span class="nn"&gt;[Service]&lt;/span&gt;
&lt;span class="py"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;oneshot&lt;/span&gt;
&lt;span class="py"&gt;User&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;example-verify&lt;/span&gt;
&lt;span class="py"&gt;ExecStart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/libexec/example-verify/verify.sh&lt;/span&gt;
&lt;span class="py"&gt;TimeoutStartSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;300&lt;/span&gt;

&lt;span class="py"&gt;StateDirectory&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;example-verify&lt;/span&gt;
&lt;span class="py"&gt;LoadCredential&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;notifier.env:/path/to/notifier.env&lt;/span&gt;

&lt;span class="py"&gt;ProtectSystem&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;strict&lt;/span&gt;
&lt;span class="py"&gt;ProtectHome&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;PrivateTmp&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the timer contained the few dates on which I wanted the check to run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Timer]&lt;/span&gt;
&lt;span class="py"&gt;OnCalendar&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;...&lt;/span&gt;
&lt;span class="py"&gt;OnCalendar&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;...&lt;/span&gt;
&lt;span class="py"&gt;OnCalendar&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;...&lt;/span&gt;

&lt;span class="py"&gt;Persistent&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There were a couple of nice things here.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;LoadCredential=&lt;/code&gt; lets systemd provide the notification credential to the service without copying it into the script's environment or configuration.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;ConditionPathExists=!PAUSE&lt;/code&gt; gives me a simple kill switch. Create the file and systemd doesn't start the service.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And because the timer had a finite number of scheduled runs, it didn't become another forgotten cron job that would still be running six months later ;)&lt;/p&gt;

&lt;p&gt;There was also a small systemd gotcha. I initially used:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="py"&gt;RuntimeMaxSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;300&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;because that's what I wanted: kill the process after five minutes.&lt;/p&gt;

&lt;p&gt;Turns out &lt;code&gt;RuntimeMaxSec&lt;/code&gt; doesn't apply to &lt;code&gt;Type=oneshot&lt;/code&gt; services.&lt;/p&gt;

&lt;p&gt;The correct option here was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="py"&gt;TimeoutStartSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;300&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;systemd-analyze verify&lt;/code&gt; happily caught it. It's worth running, costs nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the LLM actually helped
&lt;/h2&gt;

&lt;p&gt;This isn't an anti-LLM story, on the contrary the LLM was useful during the investigation.&lt;/p&gt;

&lt;p&gt;The interesting part was figuring out &lt;strong&gt;what should be checked&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;For example, one of the initial metrics I was looking at seemed to show that the exporter had enough headroom, while second review showed that this conclusion was wrong.&lt;/p&gt;

&lt;p&gt;The apparent maximum duration was based on successful scrapes. When a scrape exceeds the timeout, it can fail instead of giving you a nice metric saying "this scrape took 27 seconds". So a graph showing a maximum of 9 seconds doesn't necessarily mean the exporter never took longer than 9 seconds. It can mean that 9 seconds was the longest successful scrape.&lt;/p&gt;

&lt;p&gt;The failed scrapes were the important part. That kind of investigation is where an LLM is useful:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;correlate different signals;&lt;/li&gt;
&lt;li&gt;suggest what to look at;&lt;/li&gt;
&lt;li&gt;challenge assumptions;&lt;/li&gt;
&lt;li&gt;generate queries;&lt;/li&gt;
&lt;li&gt;review a proposed solution;&lt;/li&gt;
&lt;li&gt;write the boring documentation and ticket text.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  My current rule
&lt;/h2&gt;

&lt;p&gt;I ended up with a fairly simple rule:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If the pass/fail condition can be written as a comparison, use a script. / If you still need to figure out what should be compared, an LLM can be useful.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The important part is not "bash good, AI bad". It's rather about using the right amount of machinery.&lt;/p&gt;

&lt;p&gt;An LLM is useful when there is ambiguity and you need judgement. A script is useful when the decision has already been made and needs to be repeated reliably (not "re-decided each and every time").&lt;/p&gt;

&lt;p&gt;In this case, using an agent would have meant building a small security boundary around a system whose job was basically:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;query
query
query
query
compare
notify
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The shell script was easier to understand, easier to secure and easier to debug.&lt;/p&gt;

&lt;p&gt;And it was considerably cheaper ;)&lt;/p&gt;

&lt;h2&gt;
  
  
  One last surprise
&lt;/h2&gt;

&lt;p&gt;After three nights the check reported exactly what I wanted: no restarts and scrape times comfortably below the new timeout. So I deleted the timer, the service, the script and the temporary user.&lt;/p&gt;

&lt;p&gt;Except I didn't.&lt;/p&gt;

&lt;p&gt;The cleanup command appeared to run successfully but didn't remove the files. The reason was my shell configuration.&lt;/p&gt;

&lt;p&gt;My interactive shell had an &lt;code&gt;rm -i&lt;/code&gt; alias. The non-interactive environment inherited it. Without a terminal to answer the confirmation prompt, &lt;code&gt;rm -i&lt;/code&gt; effectively did nothing.&lt;/p&gt;

&lt;p&gt;That was a good reminder that automation doesn't necessarily run in a clean environment. Your shell configuration is part of the environment too.&lt;/p&gt;

&lt;p&gt;If you use interactive-only aliases, make sure they are actually limited to interactive shells:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$-&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt;i&lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;AGENT_SHELL&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;alias rm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'rm -i'&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use whatever environment variable or detection mechanism makes sense for your agent.&lt;/p&gt;

&lt;p&gt;The broader lesson was probably the same as the one from the beginning:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use the clever tool to figure out what needs to be checked. Use the boring tool to check it.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>ai</category>
      <category>kubernetes</category>
      <category>monitoring</category>
    </item>
  </channel>
</rss>
