An AI Agent That Keeps Your Infrastructure Code Honest

1. The 3 AM Problem

Lets picture a night that most of us cloud engineers know. An alarm goes off at 3 AM indicating that a service cannot connect to its database. The fix ? just one missing network rule. The correct approach here is a simple code change, a review, and a pipeline run, which takes time that nobody has during an outage. As a result, the cloud engineer opens the AWS console, manually adds the rule, and the errors cease.
The incident is over but here comes a new problem, the repository now says something different from production.The next deployment could discreetly reverse the fix and cause the outage to occur again, or the discrepancy might remain unnoticed, leading to a loss of trust in the code

Many tools can indicate that an event has occurred. Terraform can show a discrepancy, and AWS Config can identify a resource that violates a rule, but both only provide a notification, both stop at the alert. Still, one has to determine who made the change, find the appropriate file, craft the corresponding code, and submit a pull request. This task can sometimes get delayed, causing the gap to widen.
Here we proposes an AI agent that takes care of this task. It notices the change, gathers the evidence, and writes a pull request for a person to review. The objective is to maintain a dynamic architecture, a repository that evolves alongside production rather than diverging from it.

 

2. The Idea in One Picture

The agent turns a late-night fix into a reviewed pull request

 
Figure 1. The agent turns a late night fix into a pull request, and a human makes the final call.
The agent works in three steps:

• Observe : Detect changes made directly in AWS
• Understand : Compare the changes with the Terraform code and gather the relevant details
• Ask : Open a pull request for a person to review and approve.  

It follows three simple rules:

• It proposes, and a human decides. It never changes production and never pushes straight to the main branch.
• It shows its evidence like who changed what, and when.
• When it is unsure whether a change should be kept, it asks instead of assuming.   ##3. How System Works The AI agent observes and understands, but the human always makes the final decision.

 
Observe: AWS Config keep track of changes to the resource , and EventBridge passes the notable ones to a function. The function learns who made the change with the help of CloudTrail event, and uses the CloudTrail lookup API only as a fallback. We only set aside changes when they match a known deployment and pass our expected outcome checks. Other changes become candidates for further investigation, they are not confirmed problems, since a person may have made a good change, nor are they ignored, as a failed deployment can appear similar. Because AWS events can arrive late or be missed, Additionally, a scheduled check for resources changed outside of Terraform serves as a backup measure.

Understand: Amazon Bedrock agent jumps in with three helpful tools! One tracks who made changes and when, another reads the relevant Terraform file from GitHub alongside the current AWS rule, and the last one prepares the pull request. It checks the code for consistency with the live setup and focuses on simple edits like inline or standalone rules. For more complex elements such as variables or dynamic blocks, it opens an issue with evidence instead. Before submitting the pull request, it validates the changes, ensures proper formatting, and checks the Terraform plan to confirm that only one rule is added. If anything seems off, it pauses to ask for clarification.

Ask!: The agent makes the edits on its own branch and kicks off a pull request to share what it found out. It's got a bunch of extra safeguards hanging around to keep things safe!

• The agents has read only acesses , so it cannot change or delete anything.
• Its GitHub access is also limited to its own branches, and the main branch is protected.
• Automatic checks run on its own branch: formatting, validation, a Terraform plan whose output must show only the intended change, and security rules.
• A person reviews and merges every change.

An incorrect guess results in a review comment, rather than an outage. If a pipeline run overrides a manual change before anyone has a chance to review it, the pull request is marked as stale and closed.
 

Under the Hood: The AWS Pieces

The design has three main parts: detecting a change, investigating it, and preparing a pull request. Human review remains the final step.

The full architecture: three layers, four decisions, and a Humane at the end.
 

 

The event that wakes the agent

The workflow uses two Amazon EventBridge rules to detect relevant changes to security groups.

1. AWS Config events

AWS Config records configuration changes to supported resources. An EventBridge rule listens for configuration-item changes involving EC2 security groups:

{
  "source": ["aws.config"],
  "detail-type": ["Config Configuration Item Change"],
  "detail": {
    "configurationItem": { "resourceType": ["AWS::EC2::SecurityGroup"] }
  }
}

When an event arrives, the workflow retrieves the latest configuration and compares it with the previous version, when available. It then checks the result against the Terraform configuration.

This matters because multiple changes can happen close together. The event payload may not represent the latest state, so the workflow should verify the current configuration before preparing a proposal.

2. CloudTrail events

A second EventBridge rule listens for relevant EC2 API calls recorded by CloudTrail:

{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["ec2.amazonaws.com"],
    "eventName": ["AuthorizeSecurityGroupIngress", "RevokeSecurityGroupIngress", "ModifySecurityGroupRules"]
  }
}

Every CloudTrail record has a userIdentity section that shows who made the chnages, IAM user or a role that was taken on during a session. The orchestrator compares it with the known deployment role, but identity alone proves nothing, so a change is set aside only when it matches a known deployment and the result is verified. Changes made by AWS services acting for you, for example through service linked roles, go into a quiet batched report instead of individual pull requests.

Two Terraform details are worth knowing:

• Rules can be inline or separate resources like aws_vpc_security_group_ingress_rule, and mixing them causes conflicts, so the agent matches the existing style or escalates.
• The backup check runs terraform plan -refresh-only -detailed-exitcode, and exit code 2 means the state differs from real infrastructure, so it is investigated as a possible out-of-band change.

A Walkthrough

During an incident, an engineer opens port 8080 in the console. The agent confirms who made the change, sees that the code has no matching rule, and proposes this addition for a person to accept or reject.

+  ingress {
+    from_port   = 8080
+    to_port     = 8080
+    protocol    = "tcp"
+    cidr_blocks = ["203.0.113.0/24"]
+  }

The range is only an illustration (203.0.113.0/24). A real run copies it from the verified live rule and writes no code if it is missing. The agent's pull request reads like this:

Candidate drift on sg-12345: TCP 8080 added by [identity] at [time]. This needs a decision: if it was a temporary fix, close this pull request and restore the intended configuration.

After the merge, a new Terraform plan should report no changes. That shows code and production agree, not that the change was safe or intended.
Timing and cost have not been measured.
 

When the Agent Should Hold Back ?

Not every change needs to be coded, and the agent doesn’t just assume that people will easily adopt it. Just because a rule is currently in place doesn’t mean it should stick around.

• Mistake or unclear. The agent opens an issue or proposes restoring the original setup, and writes no code.
• A temporary fix.The agent inquires whether the change should be permanent. If it is not, a follow-up issue is created with an owner and an expiry date.
• A deployment. If a change matches a confirmed deployment, it is set aside. If a deployment fails or is unexpected, it is marked as a problem.
• An AWS service. When AWS services make changes on your behalf, they compile all those actions into a streamlined, easyto read batched report. This way, you can stay informed without all the noise!
• Not in the repository, or an unsupported pattern. The agent opens an issue with the evidence and a suggested snippet.
• Many changes at once. The agent waits briefly and opens one pull request that covers them all.

In summary, the agent initiates a pull request only when the code pattern is viable and adoption seems likely. These behaviors have been designed but have not yet been tested.

Honest Limits

• Code needs human review. The generated code may be wrong. Automated checks help, but a human still needs to review every pull request.
• Cloud resource details can be changed. Someone could change tags or descriptions to mislead the agent. The agent treats them as data, not instructions, but this protection still needs testing.
• Too many pull requests can be tiring. Reviewers might start approving changes without checking them properly. The agent tries to avoid unnecessary pull requests and group related changes together.
• AWS events can be missed. Some events may arrive late or not arrive at all. Scheduled checks help find changes that were missed.
• Some repositories may not work well. Repositories that use complex modules or dynamic blocks may need manual review instead of automatic pull requests.
• The design has a limited scope. It currently covers only security groups in one AWS account and Region. Supporting other resources and multiple accounts would need more work.
• Success needs to be measured. We need to check how often pull requests are accepted, how many alerts are incorrect, and how quickly changes are detected. Cost and speed also need testing.
• The system hasn't been fully tested. This is a proposed design, not a finished product. It needs to be built and tested before it can be used in production.  

Why It Matters

Cloud teams need to fix problems quickly while keeping their code up to date. During an incident, fixing the problem comes first, and updating the code is often left for later. This agent aims to handle that extra work automatically while leaving the final decision to a person.

In the future, the same idea could be used for databases, permissions, and other tools. But each new step should be tested carefully to make sure it works as expected.

When an engineer fixes a problem at 3 AM, they can focus on restoring the service. Later, they find a pull request waiting for review, making it easier to bring the code back in sync with the actual infrastructure.

DISCLAIMER: As English is not my first language and I am not a professional technical writer, Ai assited tool Grammarly was employed during the writing process to assist with grammar, phrasing, formatting, and overall readability.