DEV Community: Lalit Bagga

My AI Agent Remembered Staging. It Still Didn’t Know If It Was Healthy.

Lalit Bagga — Fri, 02 Oct 2026 00:55:38 +0000

My CloudOps agent remembered that I preferred the production environment.

Then I opened a new conversation and asked about deployment health. It recalled production, checked the deployment status tool, and returned a fresh result.

That was useful. But it raised a more important question:

What should the agent remember, and what should it always check again?

This was a small experiment in my personal AWS lab using Amazon Bedrock AgentCore Memory. The deployment data was synthetic, and the agent did not access a real production system.

The Agent Already Had One Tool

In the previous stage, I gave the agent a read only deployment status tool:

Question -> AgentCore Harness -> AgentCore Gateway -> Deployment_status Lambda

The Lambda supported one environment: production. It returned a sample service version, instance count, alarms, health status, and a fresh observation time.

The agent could check a current result, but it couldn't remember which environment I preferred when I started a different conversation.

So I added managed AgentCore Memory with the USER_PREFERENCE strategy.

The design now had two sources:

Memory -> which environment I prefer
Tool   -> what that environment reports now

I wanted to make sure the agent didn't confuse them.

Remembering Across Sessions Took Time

I first told the agent:

My preferred environment is production.

When I asked again in the same session, it remembered immediately.

Then I started a new session for the same actor. The first attempt did not recall the preference.

At first, that looked like i have done something wrong. But i wasn't

Upon checking documentation i found that AgentCore creates long-term memories asynchronously. The conversation must be stored, processed, and turned into a preference that can be retrieved later. After I allowed time for that processing and opened another session, the agent remembered production.

It then called the deployment-status tool and returned a new health result.

That gave me the first lesson:

Long-term memory is not an immediate key-value write.

If your application needs a preference to be available instantly in a new session, you should account for that processing delay.

A Different Actor Did Not Get My Preference

Next, I used a different actor and asked for the preferred environment.

The agent did not return the original actor's production preference.

This was the result I wanted. The preference belonged to the actor, not globally to the agent.

It also clarified the difference between an actor and a session:

A session represents one conversation.
An actor represents the person or system whose memory can continue across conversations.

A session ID alone is not a durable user identity. The application still has to assign the correct actor identity.

Staging Exposed the Real Boundary

I then changed the original actor's preference from production to staging.

After the new preference was processed, another session remembered staging.

But the deployment-status tool only supported production.

This was the real test.

The agent remembered that I preferred staging, but it could not retrieve staging health. Instead of guessing, it said that the current health was unknown.

Memory: "This actor prefers staging."

Tool: "Staging is not supported."

Answer: "I cannot determine staging health."

That is exactly how I want this boundary to work.

Memory answered:

Which environment does this person care about?

The tool answered:

What does the deployment source report now?

Remembering an environment name did not make that environment healthy, supported, or current.

The Rule I Would Keep

The main lesson from this experiment is simple:

Use memory to decide what to look up. Use a current source to decide what is true.

Memory removes repetition. It should not create confidence without evidence.

What Comes Next

This stage gave the agent continuity without letting remembered context replace current evidence.

In the next stage, I use AgentCore Code Interpreter to analyze the given deployment history. That creates another boundary to test a correct looking answer is not enough unless I can inspect the generated code and the actual execution result.

References

I Gave My Agent a Deployment Tool. It Could Finally Answer.

Lalit Bagga — Fri, 25 Sep 2026 00:52:40 +0000

My first CloudOps agent could not tell me whether production was healthy.

That was the correct answer. It had a model and a system prompt, but it had no connection to deployment data.

For the next experiment, I asked the same question again:

Is production healthy?

This time, the agent returned the service name, deployed version, instance count, active alarms, and observation time.

The model had not suddenly learned how to monitor an environment. I had given it a controlled way to ask another system for evidence.

The Missing Piece Was a Tool

I kept the architecture deliberately small:

User
  |
  | "Is production healthy?"
  v
AgentCore Harness
  |
  | selects get_deployment_status
  v
AgentCore Gateway
  |
  | invokes a registered target which is AWS Lambda
  v
AWS Lambda
  |
  | returns a deterministic deployment snapshot
  v
Tool grounded answer

The Harness still used Amazon Nova Micro. Memory remained disabled for this stage.

I added one read only Lambda named AgentCoreCloudOps-DeploymentStatus. It did not connect to my real production systems or to the existing three-tier application. It returned a deterministic snapshot for a sample service named blog-api.

That limitation was intentional. Before connecting an agent to real operational systems, I wanted to understand the path from a question to a tool call and back.

I Defined Healthy in Code

I did not ask the model to decide what healthy meant.

The Lambda compared three conditions:

def determine_status(snapshot):
    versions_match = (
        snapshot["desired_version"] == snapshot["running_version"]
    )
    instances_healthy = (
        snapshot["healthy_instances"] == snapshot["desired_instances"]
    )
    alarms_clear = len(snapshot["active_alarms"]) == 0

    return (
        "healthy"
        if versions_match and instances_healthy and alarms_clear
        else "unhealthy"
    )

The handler also rejected missing or unsupported environments. Only production was accepted:

environment = event.get("environment")

if not isinstance(environment, str) or not environment:
    return {
        "ok": False,
        "error": "environment is required and must be a string.",
    }

if environment != "production":
    return {
        "ok": False,
        "error": f"Unsupported environment: {environment}",
    }

SAM and Docker Gave Me a Local Lambda Test

I defined the function with AWS Serverless Application Model

Resources:
  DeploymentStatusFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: AgentCoreCloudOps-DeploymentStatus
      Runtime: python3.12
      Architectures:
        - arm64
      Handler: app.lambda_handler
      CodeUri: .
      MemorySize: 128
      Timeout: 5

SAM supplied the Lambda project and invocation workflow. Docker supplied a local Linux container that matched the Lambda runtime closely enough to test the handler before deploying it.

The first local attempt reported:

Running AWS SAM projects locally requires Docker.
Have you got it installed and running?

Docker Desktop was already running. The message described the symptom, not the cause.

Debug output showed that the installed SAM CLI was trying to use Docker API version v1.35, while the Docker engine required at least v1.40. Updating SAM fixed the compatibility problem.

After updating SAM, I moved into the Lambda project and invoked the function with the production event:

cd lambda/deployment_status

sam local invoke DeploymentStatusFunction \
  --event events/production.json

SAM mounted the function under /var/task and returned:

{
  "environment": "production",
  "service": "blog-api",
  "deployment_id": "deploy-1042",
  "desired_version": "2026.09.20.1",
  "running_version": "2026.09.20.1",
  "healthy_instances": 3,
  "desired_instances": 3,
  "active_alarms": [],
  "status": "healthy",
  "observed_at": "2026-09-22T01:26:17Z",
  "ok": true
}

This invocation ran inside Docker, so it did not need an active AWS login. The next test moved the same function into AWS.

I built and deployed the SAM application:

sam build
sam deploy --guided --region us-east-2

I then invoked the deployed Lambda directly through AWS:

aws lambda invoke \
  --function-name AgentCoreCloudOps-DeploymentStatus \
  --region us-east-2 \
  --cli-binary-format raw-in-base64-out \
  --payload '{"environment":"production"}' \
  /tmp/deployment-status-response.json

The AWS CLI saved the function response to a file, so I inspected it separately:

cat /tmp/deployment-status-response.json

The deployed function returned:

{
  "environment": "production",
  "service": "blog-api",
  "deployment_id": "deploy-1042",
  "desired_version": "2026.09.20.1",
  "running_version": "2026.09.20.1",
  "healthy_instances": 3,
  "desired_instances": 3,
  "active_alarms": [],
  "status": "healthy",
  "observed_at": "2026-09-22T01:47:28Z",
  "ok": true
}

This verified the deployed Lambda independently. AgentCore Gateway and the Harness were not involved yet.

The Tool Schema Told the Agent What It Could Ask

Deploying the Lambda did not automatically make it an agent tool.

I created a JSON schema that described one operation:

{
  "name": "get_deployment_status",
  "description": "Get the deployment health of the blog-api service in a supported environment. Use this tool when asked whether production is healthy, which version is running, whether instances are healthy, or whether deployment alarms are active.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "environment": {
        "type": "string",
        "description": "The environment to inspect. Currently only production is supported."
      }
    },
    "required": ["environment"]
  }
}

The schema was more than documentation. It gave the model a name, purpose, and input contract for the operation.

AgentCore Gateway registered the Lambda as DeploymentStatusTarget and exposed the operation through a managed tool interface. The Gateway used AWS_IAM for inbound authorization, and the Harness referenced the Gateway as an agentcore_gateway tool.

The responsibilities were now separate:

Harness  -> decides whether to call the tool
Gateway  -> exposes and routes the tool call
Lambda   -> validates the input and calculates health
Model    -> explains the structured result

That separation was the main reason for the experiment. The model could select the deployment status operation, but it did not define the health rule. This Gateway exposed no deployment or rollback operation, and the Lambda itself was read-only.

The Gateway Existed, but the Protocol Was Wrong

My first Gateway deployment appeared successful. The Lambda target also existed.

I first asked the Gateway to list the tools it exposed:

agentcore invoke \
  --gateway CloudOpsGateway \
  "list-tools"

That command failed with:

--target-name is required for HTTP gateways.
Available targets:

I had created the Gateway with no MCP protocol configured. The client therefore treated it as an HTTP Gateway and expected a target name.

That was not the interface I wanted. The Harness needed to discover the Lambda operation as a tool through the Model Context Protocol.

I changed the Gateway configuration to:

{
  "name": "CloudOpsGateway",
  "protocolType": "MCP",
  "authorizerType": "AWS_IAM"
}

I validated and redeployed the corrected Gateway configuration:

agentcore validate
agentcore deploy --dry-run
agentcore deploy

I then invoked the Lambda tool directly through the Gateway:

agentcore invoke \
  --gateway CloudOpsGateway \
  "call-tool" \
  --tool "DeploymentStatusTarget___get_deployment_status" \
  --input '{"environment":"production"}'

{
  "environment": "production",
  "service": "blog-api",
  "deployment_id": "deploy-1042",
  "desired_version": "2026.09.20.1",
  "running_version": "2026.09.20.1",
  "healthy_instances": 3,
  "desired_instances": 3,
  "active_alarms": [],
  "status": "healthy",
  "ok": true
}

This verified the Gateway to Lambda path without involving the Harness. After that, I attached the Gateway to the Harness and redeployed the project.

This was a useful failure. A resource can exist, its target can exist, and the connection can still be wrong for the client using it.

The Same Question Finally Produced Evidence

Finally, I invoked the Harness and explicitly asked it to use the deployment status tool:

agentcore invoke \
  --harness CloudOpsHarness \
  --verbose \
  "Use the deployment status tool to check production. Is production healthy? Include the versions, instance count, active alarms, and observation time."

This time it selected DeploymentStatusTarget, invoked the Lambda, and summarized the returned fields:

Environment: production
Service: blog-api
Deployment ID: deploy-1042
Desired version: 2026.09.20.1
Running version: 2026.09.20.1
Healthy instances: 3
Desired instances: 3
Active alarms: 0
Status: healthy

The Harness concluded that production was healthy because all three instances were running the desired version and the tool reported no active alarms.

What Comes Next

The tool solved one problem: the agent could retrieve a current result instead of guessing.

It did not solve continuity between conversations. In the next stage, I added AgentCore Memory so the agent could remember a user's preferred environment across sessions.

References

I Deployed My First AgentCore Harness and Asked It About Production

Lalit Bagga — Thu, 17 Sep 2026 01:49:20 +0000

My first CloudOps agent gave me the right answer by refusing to answer.

I asked:

Is production healthy?

The agent said it could not verify the current state because it had no access to live infrastructure, deployment data, monitoring systems, or production environments.

That was not a disappointing first result. It was the boundary I wanted to prove.

This was Stage 1 of a larger project. The end goal is a small CloudOps agent that can answer the same question by calling a Lambda that only reads deployment status. Before adding that tool, I wanted to understand what Amazon Bedrock AgentCore gives me when the agent has only a model, a prompt, and a managed agent loop.

Why I Started with Harness

The AgentCore CLI currently offers two paths. I can bring an agent loop written with a framework such as Strands, or I can use a managed Harness. With the Harness, I declare the model, prompt, tools, and memory in configuration while AgentCore runs the loop.

I chose Harness because I wanted the first experiment to isolate the managed platform. No custom orchestration code. No Lambda. No Gateway. No durable memory.

The architecture was deliberately small:

User
  |
  | prompt
  v
AgentCore managed Harness
  |  CloudOps system prompt
  |  memory disabled
  |  no infrastructure tools configured by me
  v
Amazon Nova Micro
  |
  v
CloudWatch logs, metrics, and traces

AWS describes the managed Harness as the path where AgentCore runs the agent loop from configuration. The alternative gives the developer control of the loop in code. The distinction sounds simple in documentation. Deploying both the configuration and its infrastructure made it concrete.

The Boundary Was the Product

I named the Harness CloudOpsHarness and gave it a narrow system prompt:

You are a CloudOps assistant in an early learning stage.

You currently have no access to live infrastructure, deployment data,
monitoring systems, or production environments.

Never claim that an environment is healthy or unhealthy without current
tool evidence.

Do not imply that you can deploy, roll back, restart, or modify resources.
Those capabilities have not been provided.

Memory was disabled, and I configured no external tools.

The First Deployment Failed Before It Reached AWS

My first Harness name was descriptive enough to become a problem.

The physical name generated by the CLI combined the project and Harness names. That composed value exceeded the Harness name limit, so CDK synthesis failed. agentcore validate still said the project was valid because the local schema was valid, the failure appeared when the infrastructure was synthesized.

Shortening the resource name to CloudOpsHarness fixed it.

That gave me the first useful lesson: a friendly logical name can become a longer physical name after a deployment system adds context. Local validation and infrastructure synthesis are different checks.

The next dry run reached another boundary:

✓ Synthesize CloudFormation
✗ Check bootstrap status
AWS environment needs bootstrapping.

I reran the dry run with permission to bootstrap the environment:

agentcore validate
agentcore deploy --dry-run --yes

This time the dry run completed and the stack was ready to deploy.

Model Access Was a Separate Failure Domain

The initial configuration used an Anthropic model. The Harness deployed, began producing the expected refusal, and then the Bedrock streaming call failed because the account had not submitted the details required for Anthropic model access.

That was not an AgentCore hosting failure. It was a prerequisite for model access.

For this learning stage, I changed the model to Amazon Nova Micro:

{
  "modelProvider": "bedrock",
  "modelId": "us.amazon.nova-micro-v1:0",
  "tools": [],
  "skills": [],
  "memory": {
    "enabled": false
  }
}

The next deployment completed successfully.

What `agentcore dev` Taught Me

I expected a command named dev to mean “run only on my machine.” The managed Harness flow challenged that assumption.

The command validated the project, synchronized CDK dependencies, built and synthesized the CDK project, checked AWS bootstrap and stack status, and persisted deployment state.

The lesson was not that the command was wrong. The lesson was that names such as dev are not a security or cost boundary. I need to read the proposed actions and inspect status before assuming where something will run.

The Tests I Ran

Once the Harness was running with Nova, I tested the questions that mattered for this stage.

1. Can it admit that it does not know?

When I asked whether production was healthy, it explained that it could not verify the environment without current operational evidence. It suggested signals such as application metrics, logs, health checks, database performance, and user feedback.

That response passed the test. It did not turn general CloudOps knowledge into a fabricated observation.

2. Can a user talk it into a capability it does not have?

I tried to override the prompt and also requested a production rollback. Neither test gave the assistant new infrastructure access.

3. Does a session behave like memory?

A follow up in the same session recognized that I had asked about production health before. A new session provided the comparison.

That did not mean I had added AgentCore Memory. AgentCore Runtime sessions can preserve ephemeral context across invocations in one session. AgentCore Memory is the separate capability intended for structured information that must persist beyond that lifecycle.

That distinction is going to matter later. A remembered user preference may be useful. A remembered statement that production was healthy can become dangerously stale.

Finding the Trace Was Its Own Experiment

The Harness worked, but these commands did not:

agentcore traces list
agentcore logs --since 30m

Both returned:

No runtimes defined in agentcore.json

The reason was visible in the project structure. The managed resource was declared under harnesses, while those CLI commands looked for configured runtimes. The Harness still had a backing Runtime log group in CloudWatch. it simply was not discovered through that configuration path.

I went to CloudWatch Logs directly and found standard Runtime logs plus OpenTelemetry output. At first, the trace exporter reported HTTP 400. CloudWatch Transaction Search had not finished enabling span ingestion.

AWS documents Transaction Search as a setup that each account needs once to view AgentCore spans and traces. After its status changed to enabled, I invoked the Harness again and opened the trace in CloudWatch.

The complete path was visible:

The model span reported:

CloudWatch displayed the cost as $0.000. I treat that as a rounded console value for one small invocation, not proof that the call was free.

The same token attributes appeared on several parent spans. Those spans describe one nested request path, not several model calls, so I did not add the repeated numbers together.

AWS's AgentCore Observability guide says agents hosted on Runtime receive automatic OpenTelemetry instrumentation through the CLI deployment path. Seeing the root invocation, agent loop, event loop cycle, and model call in one timeline made that statement much more useful than reading it alone.

Cleaning Up Without Deleting the Learning

After the tests, I removed the AWS Harness and deployed that deletion. I then restored the local Harness files so the experiment remained reproducible.

The final check was:

AgentCore Status (target: default, us-east-2)

Harnesses
  CloudOpsHarness: Local only

That status is stronger evidence than a successful deletion command by itself. It says the CLI can still see the local definition but no deployed Harness for the selected target.

The CDK bootstrap stack and CloudWatch evidence are separate resources with separate lifecycles. Removing the Harness does not mean the AWS account has returned to a completely untouched state.

Conclusion

The biggest lesson is that an agent can be correctly deployed and still be unable to answer the business question.

That is not a prompt problem. It is an architecture problem.

References

My Kubernetes App Moved to EKS Unchanged. Everything Around It Didn't.

Lalit Bagga — Fri, 11 Sep 2026 02:25:02 +0000

My Kubernetes application moved from my home server to Amazon EKS without an application code change.

The platform around it did not.

The EKS control plane took 5 minutes 51 seconds to create. Once the delivery path was working, Argo CD deployed two healthy application replicas and corrected a manual scaling change in 5 seconds.

But reaching that result exposed three assumptions about networking, worker size and AWS access that did not exist in my home lab.

That was the test I wanted.

Why I Returned to EKS

I started this series because I wanted an always on Kubernetes environment without an always on EKS bill.

My Debian server and k3s gave me an inexpensive place to learn every day. I then built a pull based delivery path: CI tested and published an immutable image, Git recorded the approved image digest, and Argo CD made the cluster follow Git.

The home lab proved that the design worked on k3s. It did not prove that I could move the same workload to a managed cloud cluster.

So I created a temporary EKS environment for one question:

Could I keep the application and delivery contract while changing the Kubernetes platform underneath them?

This was a portability test, not a production EKS design.

What I Kept the Same

I reused the important parts of the existing project:

the same Node.js application
the same container image, selected by an immutable digest
the same Helm chart
the same Git repository as the deployment record
the same pull-based Argo CD model
the same health, readiness and version endpoints.

The application CI still had no Kubernetes credentials. It could test code, build an image and propose a digest change in Git, but it could not deploy directly to either cluster.

The delivery path remained:

CI tests and publishes the image
  -> Git records the approved digest
  -> Argo CD reads Git from inside the cluster
  -> Kubernetes runs the declared release

I made one portable extension to the Helm chart. The home environment could keep a private ClusterIP Service, while the EKS values could request an AWS load balancer. The chart stayed shared, the environment-specific choice stayed in a small values file.

The Smallest Useful AWS Test

I used Terraform to create only what the experiment needed:

one EKS 1.36 cluster
one managed AMD64 worker node
two public subnets in separate Availability Zones
private control plane access for the worker
a public control plane endpoint restricted to my current IP
the standard EKS networking add-ons
AWS Load Balancer Controller
Argo CD Core with no public user interface; and
one temporary Network Load Balancer for the application.

One worker reduced the cost, but it did not provide high availability. If that node failed, the workload had nowhere else to run. That limitation was acceptable for a short portability experiment.

The Worker Could Not Join

The EKS control plane became active, but the worker did not join the cluster.

I had restricted the public API to my computer and disabled private access. That protected the public endpoint, but it also left the worker without a path to the control plane.

I enabled the private endpoint for communication inside AWS while keeping public administrative access restricted.

The application had moved unchanged. Its network paths still required an AWS specific decision.

AWS Rejected My Planned Worker

I planned to use a t3.medium worker. AWS rejected that instance type for the account's eligible configuration.

Instead of choosing a larger machine at random, I queried the eligible AMD64 choices and selected one c7i-flex.large worker with 2 vCPUs and 4 GiB of memory.

This changed the infrastructure, not the application. The same container and Helm release still ran after the worker choice changed.

That is a useful portability boundary: Kubernetes can preserve the workload contract, but the cloud provider still controls which underlying compute choices are available.

The Access Problem Was Operational, Not Kubernetes

The long-running AWS operation also exposed an authentication problem. My interactive login stopped refreshing reliably while I was creating and checking the cluster.

For this temporary lab, I created a temporary role for the remaining setup and teardown work, then removed that access during cleanup.

The important lesson was not to weaken authentication. It was to treat operator access as part of the platform design. A repeatable EKS workflow needs a stable, non-personal AWS role with only the permissions the workflow requires.

Git Deployed the Workload

After the cluster components were ready, I merged the EKS configuration through a pull request. The CI checks completed successfully in 13 seconds.

Argo CD, running privately inside EKS, then read the merged Git state and applied the Helm release. CI never received a kubeconfig and never called the cluster.

The final checks showed:

Argo CD was Synced and Healthy
both application replicas were ready
the running image matched the immutable digest in Git
/health returned a healthy response
/version returned the expected application version and
Argo CD had no public Service or user interface.

The same application and delivery decision had crossed from k3s to EKS. Git remained the durable record on both platforms.

I Changed the Cluster Behind Git's Back

I then repeated the simplest self-healing test from my home lab.

Git declared two application replicas. I manually scaled the live EKS Deployment down to one.

Argo CD detected that the cluster no longer matched Git and restored the second replica in 5 seconds.

On k3s, the same type of test had taken 42 seconds. This single observation does not prove that EKS is generally faster. Reconciliation timing depends on polling, refresh timing and cluster conditions. It proves only that this EKS run corrected this specific drift in 5 seconds.

The Cost Test

I also treated cost as an engineering result rather than an afterthought.

EKS control-plane creation: 5 minutes 51 seconds
Estimated environment cost: approximately US$0.24-US$0.27 per hour
Final charge: to be replaced with the AWS billing result when it becomes available

The estimate covered the temporary control plane, worker, storage, public IPv4 use and load balancer. I kept it as an estimate because billing data had not appeared when I prepared the first draft.

Teardown Was Part of the Test

A temporary cloud environment is not finished when the application works. It is finished when its chargeable resources are gone.

I removed the application load balancer first, destroyed the Terraform-managed infrastructure and removed the temporary access used for the experiment. The teardown script completed successfully.

I deliberately kept identifiers, cluster endpoints, credentials, IP addresses and kubeconfig data out of the repository and article evidence.

What Was Actually Portable

The test gave me a more precise answer than “Kubernetes is portable.”

The application code, container contract, immutable image, Helm release structure and GitOps delivery model moved cleanly.

The cloud platform still required AWS-specific engineering:

worker-to-control-plane networking
available instance selection
operator and controller identity
load balancer integration and
cost-aware teardown.

Kubernetes made the workload portable. It did not make every platform decision portable.

That is the boundary I wanted to find. My home k3s server remains the practical always on learning environment. EKS becomes a short-lived validation target when I need to prove that the platform contract still works in AWS.

Argo CD Fixed My Drift, Then Deployed My Bad Release

Lalit Bagga — Thu, 03 Sep 2026 21:38:24 +0000

This project started with a simple goal: run Kubernetes without keeping an EKS
cluster online every day.

In I Wanted Kubernetes Without an Always-On EKS Bill,
I built an always-on k3s lab on my home server and proved that I could deploy,
update, and roll back an application.

The rollback worked, but it exposed the next problem. Kubernetes restored
Version 2 while the saved YAML still declared Version 3. I corrected the file
manually, but the recovery depended on repairing the running cluster and its
saved instructions separately.

In The Rollback Worked. My Next Deploy Could Break It Again,
I designed a safer path. The automated build process would test and publish an
exact image, then stop at a Git pull request. Git would record the reviewed
version. Argo CD, running inside Kubernetes, would make the cluster follow that
record.

Now I needed to prove that the design worked outside a diagram.

I followed one release from source code to running Pods. Then I tested two
opposite failures:

The cluster was wrong while Git was correct.
Git contained a bad setting while the cluster followed it correctly.

Those experiments showed both the value and the limit of GitOps. Automation can
make the cluster match Git, but it cannot decide whether the human-approved
version in Git is a good one.

CI Built the Release but Did Not Deploy It

The GitHub Actions workflow—my continuous integration, or CI, worker—ran the
application tests and checked the Kubernetes package before building anything.
Its job was to prove and publish a release, not to change the cluster.

After validation, Buildx created a Linux AMD64 image with the full source commit
baked into /version:

docker buildx build \
  --platform linux/amd64 \
  --build-arg "APP_VERSION=$GITHUB_SHA" \
  --tag "$image_name:$GITHUB_SHA" \
  --provenance=mode=max \
  --sbom=true \
  --push \
  application

After publishing the image, CI read its registry digest. A digest is the image's
content fingerprint: if the image changes, the digest changes. CI updated one
field in gitops/environments/homelab/values.yaml. For the final measured
release, the automated pull request contained this one-line change:

-  digest: sha256:170bcc91b9c4bfcbdc194c004cb38d6c89094e9b49a1a128ed3a0507976fef97
+  digest: sha256:f8303e07e737f40fa5479b73a148261cf4221f4d4b52ebf095b74d2cb4ef5be5

The workflow opened the pull request and stopped. A human reviewed and merged
the digest. That merge—not a deployment command in CI—was the release
instruction.

Argo CD Made the Cluster Follow Git

Argo CD was the worker inside Kubernetes. It repeatedly compared the version
recorded in Git with the version running in the cluster and worked to make them
agree. This comparison-and-correction loop is called reconciliation.

I installed a pinned, resource-limited Argo CD Core deployment with no permanent
API server or public UI. Its Services remained internal ClusterIP Services.

The Application reads both the Helm chart and homelab values from the public
repository. Automated sync, pruning, and self-healing are enabled:

syncPolicy:
  automated:
    enabled: true
    prune: true
    selfHeal: true

At the healthy baseline, Argo CD reported Synced, Healthy, and operation
Succeeded. Two application Pods were ready with zero restarts.

I retained the complete artifact trace:

application commit
    -> registry digest
    -> Git desired state
    -> Deployment image
    -> both Pod runtime image IDs
    -> /version response

The Git declaration, Deployment, and both Pod image IDs agreed on the same
registry digest. /version returned the source commit baked into that image.

The four Argo CD components used a combined 19 millicores of CPU and 85 MiB of
memory in the retained steady-state sample. That measurement describes this
home lab; it is not production sizing guidance.

First, I Changed the Cluster Behind Git’s Back

Git declared two replicas. I deliberately changed the live Deployment to one:

kubectl -n delivery-api scale deployment/delivery-api --replicas=1

The first poll already showed one replica and Argo CD OutOfSync. Git remained
unchanged with replicaCount: 2.

Forty-two seconds after the mutation, the Deployment was back to two replicas
and the Application was Synced again.

This proved self-healing for that specific field. Git remained correct, so Argo
CD could restore the live cluster from the durable declaration.

The next failure reversed that relationship.

Then I Put a Bad Setting in Git

I merged a reviewed configuration change from:

readinessMode: ready

to:

readinessMode: fail

The change was valid YAML. It passed the Helm schema and lint checks. At runtime,
the new Pod stayed alive but /ready returned HTTP 503.

Forty-nine seconds after the merge, the evidence showed:

Argo CD sync:    Synced
Argo CD health:  Progressing
New Pod:         Running, not Ready
Old Pods:        Running, Ready

Synced meant the cluster matched Git. Progressing meant Kubernetes could not
complete the rollout.

GitOps had not malfunctioned. It had correctly delivered a harmful declaration.

The Deployment used maxUnavailable: 0, so the two old ready replicas remained
while the replacement failed readiness. One sampled request through the private
Service returned {"status":"ready"}. That proves availability at that moment,
not zero downtime across the experiment.

Recovery Had to Correct Git

Patching only the live Deployment would leave Git declaring the bad readiness
mode. Argo CD could then reapply it.

I reverted the bad merge in Git:

git revert --mainline 1 69ac4fe6de392aed4affc573ea20a2bb3f4ee59f
git push

Argo CD pulled the revert and restored readinessMode: ready. Forty-two seconds
after the recovery commit, the Application was Synced and Healthy with two
updated, ready replicas.

Argo CD did not choose the known-good version. I chose it by reverting Git; Argo
CD reconciled that decision.

I Timed a Fresh Release End to End

The first image promotion happened before Argo CD was installed, so I did not
use it for a continuous promotion-to-healthy claim. I ran another application
release with Argo CD already active and started the monitor before merging the
digest pull request.

Event	Observed result
CI workflow window	74 seconds
Validation job	14 seconds
Image publication and promotion job	50 seconds
Promotion merge to Argo CD operation start	32 seconds
Argo CD operation start to healthy	11 seconds
Promotion merge to healthy release	43 seconds

The monitor sampled every two seconds, so the healthy timestamp has up to two
seconds of sampling uncertainty.

During the rolling update, the Deployment already declared the new digest while
a request still reached an old ready Pod and returned the previous commit. A few
seconds later, both updated replicas were ready and /version returned:

{"service":"delivery-api","version":"84db89f7f968c5860b07a7efd6a42e21895d9e5c"}

That was expected. A rolling update changes replicas gradually; the Deployment
declaration and every serving Pod do not switch at the same instant.

What the Implementation Proved

Application CI published the artifact and proposed the environment change
without receiving cluster authority. Git retained the reviewed digest. Argo CD
pulled and reconciled it. Kubernetes reported rollout health.

The failure tests also exposed the limits:

Self-healing corrected live drift only because Git remained correct.
Argo CD faithfully applied the bad readiness declaration from Git.
Recovery required a human decision recorded as a Git revert.
One successful request did not prove zero request loss.
A single-node k3s cluster could not demonstrate high availability.

GitOps did not remove operational judgment. It moved release and recovery
decisions into a place where they were reviewable, durable, and reconcilable.

Conclusion

The first article created an affordable, always-on Kubernetes lab. The second
used its rollback problem to design one reviewable release path. This
implementation proved how that path behaved when either the cluster or Git was
wrong.

The common lesson across the three articles is that a working cluster is only
one part of a delivery platform. The approved release, permission to deploy,
health signals, and recovery decision also need clear owners.

The code, chart, GitOps configuration, runbooks, and sanitized evidence are in
the public
pull-based-kubernetes-delivery repository.

The Rollback Worked. My Next Deploy Could Break It Again.

Lalit Bagga — Thu, 27 Aug 2026 19:06:29 +0000

My application was working again. That should have been the end of the problem.

It was not.

In [the first article in this series(https://blog.lalitbagga.com/posts/why-i-chose-k3s-debian-homelab), I built an always on Kubernetes lab on my home server. I deployed a small application, changed it from Version 2 to Version 3, and then practised a rollback.

Kubernetes restored Version 2 successfully. But the YAML file on my computer the file containing the instructions for the application still said Version 3.

I had repaired what was running, but I had not repaired the instructions that I would use next time. If I applied that file again, I could accidentally bring Version 3 back.

I corrected the YAML manually and confirmed that the file and cluster agreed. That solved the immediate problem, but it also exposed a weakness in my process: recovery depended on me remembering to repair two places.

That became the starting point for this project.

The Real Problem Was Two Different Answers

Kubernetes always has a live answer to the question, “What should run now?” My
YAML file had a saved answer to the same question.

After the rollback, those answers were different:

Running in Kubernetes: Version 2
Saved in YAML:          Version 3

Neither answer was hidden. The weakness was that nothing continuously checked
whether they agreed.

I wanted one durable place to record the version I had chosen. If somebody changed the cluster directly, the system should notice the difference and restore the recorded choice. If the recorded choice itself was bad, recovery should mean changing that record not making another temporary cluster edit.

That is the problem GitOps is meant to address.

I Considered Who Should Be Allowed to Change the Cluster

My first idea was the most direct one: let the application pipeline deploy the new version.

A pipeline, usually called continuous integration or CI, is an automated worker. It runs tests and builds the application whenever code changes. I could also give it the password like Kubernetes credentials needed to change the cluster.

That would be convenient, but it would give the application build process a second, much more powerful job. A mistake or compromised workflow could move directly from building code to changing the cluster.

I wanted a smaller boundary:

CI may build and publish a release.
CI may propose which release should run.
CI may not change Kubernetes directly.

The proposal would become a pull request. I would review it and merge it into Git, the same version history system used for the application code. Git would therefore hold the durable answer to “Which version should this environment run?”

This was not about removing every credential from CI. It still needed limited GitHub permission to publish an image and open a pull request. The important restriction was precise: application CI would not receive a Kubernetes credential or permission to deploy to the cluster.

I Designed the Process Before Choosing the Tools

At this point, I could describe the process without naming a Kubernetes tool:

Developer changes the application
        ↓
CI tests it and builds a release
        ↓
CI opens a pull request with the exact release
        ↓
A human reviews and merges the change
        ↓
A worker inside Kubernetes reads Git
        ↓
That worker makes the cluster match Git

This operating model is called GitOps. Git stores the desired state the version we intend to run. Software inside the cluster repeatedly compares that record with reality and corrects differences.

I chose Argo CD for that inside-the-cluster worker. Argo CD watches the Git repository and reconciles the cluster. “Reconcile” simply means compare what is running with what should be running, then work to make them agree.

This separation gave each part one main responsibility:

CI proves that the application can be built.
Git records the reviewed version choice.
Argo CD makes Kubernetes follow that choice.
Kubernetes reports whether the application becomes healthy.

I Needed an Exact Version, Not a Moving Name

A release also needed an identity that could not silently change.

Container images often use names such as latest or version-3. These are tags: convenient labels that can be moved to different image contents later. That would weaken Git as the record of exactly what I approved.

I therefore chose an image digest. A digest is a fingerprint calculated from the image contents. If the contents change, the fingerprint changes. Git would store that exact fingerprint, so the selected release could not change while keeping the same name.

CI would publish the image, learn its digest, and open a pull request changing only that value. It would then stop and wait for review.

I Used Helm to Keep the Kubernetes Instructions Together

The application needed several Kubernetes instructions: how to run it, how to reach it, how to check its health, and which image digest to use.

I packaged those instructions with Helm. Helm is a way to create a reusable Kubernetes template. The structure stays the same while selected values such as the image digest or number of copies can change.

Before allowing Argo CD to use the Helm package, I planned to test it on its own: install it, verify the application, upgrade it, roll it back, remove it, and install it again in a clean location.

This separated two questions:

Does the Kubernetes package work correctly?
Does the automated delivery process use it correctly?

If both failed at the same time, troubleshooting would be much harder.

I Chose a Small Application That Could Fail Safely

A successful first deployment would not prove much. I needed an application that could help test both healthy and unhealthy releases without risking real data.

I built a small Node.js service that could answer four simple questions:

/health   Is the program alive?
/ready    Should Kubernetes send it user traffic?
/version  Which source version is running?
/metrics  How many requests has it handled?

The health and readiness checks were deliberately separate. This allowed me to create a safe bad release: the program would keep running, but its readiness check would fail, so Kubernetes would refuse to send new traffic to it.

The version response would help trace one change from source code, through the built image and Git, to the running application.

I Kept Argo CD Private

My home server already ran other services and had limited free memory. I did not need another public management screen.

I chose the smaller Argo CD Core installation. Its management components would remain reachable only inside the cluster, and I gave them explicit CPU and memory limits.

Argo CD still needed meaningful power inside Kubernetes to do its job. Keeping it private did not make it harmless. I therefore limited this application to its repository, its Kubernetes namespace, and the types of resources its Helm package created.

I Defined the Failure Tests Before Claiming Success

A green installation screen would only prove that the tools started. I wrote the tests before implementing the system:

Test	Question it had to answer
Follow one release	Can I connect the source change to the built image, the version recorded in Git, and the running application?
Change the cluster by hand	Does Argo CD notice that the cluster no longer matches Git and correct it?
Approve a bad readiness setting	What happens when Git is followed correctly but the chosen configuration is harmful?
Revert the Git change	Can I recover by correcting the durable record instead of editing only the cluster?
Time a normal release	How long does a reviewed version take to become healthy?
Measure resources	Does Argo CD fit on the shared home server without creating pressure?

The two failure cases were intentionally different.

Changing the cluster by hand would make the cluster wrong while Git remained correct. Argo CD should restore the version recorded in Git. This disagreement is called drift.

The readiness experiment would test the opposite problem: Git itself would contain a harmful setting. Argo CD should apply it faithfully, while Kubernetes should report that the new copy was not ready. Recovery would require reverting the Git change.

That distinction became the most important result of the project.

What the Design Did Not Promise

This design did not make Git automatically correct. A reviewer could still approve a bad value, and Argo CD could apply it perfectly. Argo CD also still needed permission to change the cluster.

The single server lab could not prove high availability. One successful request also could not prove that no user request was ever lost.

The goal was smaller and testable: keep direct cluster access out of application CI, record the exact selected release in Git, and observe how the system behaved when either the cluster or Git was wrong.

What Came Next

The rollback in my first article taught me that recovering the running application and correcting its saved instructions were separate jobs. This design joined them through one reviewable path.

Part 2
implements that path. It follows one release from CI to Git and into Kubernetes, then records what happened when I changed the cluster by hand, approved a bad readiness setting, and recovered by reverting Git.

The complete implementation and sanitized evidence are in the public
pull-based-kubernetes-delivery repository.

I Turned a $7 Lightsail VM Into a Staging Server

Lalit Bagga — Thu, 20 Aug 2026 19:37:38 +0000

My side projects had a staging gap.

Local Docker was useful for checking an application before a release, but it still ran on my laptop. A larger AWS setup with a VPC, load balancer, container platform, and deployment pipeline would answer a different question and create much more infrastructure than I needed.

I wanted something in the middle: one server with a stable address, controlled access, repeatable deployment, a health check, and enough recovery behavior to be useful during development.

So I tested Amazon Lightsail as a small staging environment instead of treating it as a simplified hosting tutorial.

The Smallest Useful Staging Stack

I deliberately kept the architecture narrow:

Developer laptop
  ├── Terraform → Lightsail instance, SSH key, static IP, firewall
  └── deploy.sh → SSH/SCP → Docker Compose
                              └── Node.js app
                                   ├── /health
                                   └── /version

The instance ran Ubuntu 24.04 in ca-central-1 on the Micro 1 GB Linux plan with public IPv4. AWS currently lists that bundle at $7 per month, billed hourly up to the monthly maximum.

There was no database, load balancer, Kubernetes cluster, or CI/CD system. The point was to learn whether one developer could operate a useful staging VM without quietly building a small platform team around it.

Step 1: Define More Than an Instance

The Terraform resource for the VM was only one part of the environment:

resource "aws_lightsail_instance" "lightsail_instance" {
  name              = "${var.project_name}-lightsail-instance"
  availability_zone = var.availability_zone
  blueprint_id      = var.blueprint_id
  bundle_id         = var.bundle_id
  key_pair_name     = aws_lightsail_key_pair.lightsail_instance_key.name
  user_data         = file("${path.module}/../scripts/bootstrap.sh")
}

I also needed a custom SSH public key, a static IP, the attachment connecting that address to the instance, and explicit firewall rules.

The static IP was not just extra Terraform. Lightsail's default public IPv4 address can change when an instance is stopped and started. A staging endpoint is much easier to use when its address stays stable, and Lightsail lets the same static address move to a replacement instance later.

The SSH key had a similarly clear boundary. Terraform uploaded only the public key:

resource "aws_lightsail_key_pair" "lightsail_instance_key" {
  name       = "${var.project_name}-key"
  public_key = file(pathexpand(var.ssh_public_key_path))
}

The private key never entered Terraform, the repository, or the article.

Step 2: Replace the Default Firewall Rules

A base Lightsail Linux instance can start with SSH on port 22 and HTTP on port 80 open to every address. That is convenient for a first connection, but it was wider than this experiment required.

I replaced those defaults with two rules restricted to my temporary public /32 address:

resource "aws_lightsail_instance_public_ports" "lightsail_instance_firewall" {
  instance_name = aws_lightsail_instance.lightsail_instance.name

  port_info {
    protocol  = "tcp"
    from_port = 22
    to_port   = 22
    cidrs     = [var.developer_cidr]
  }

  port_info {
    protocol  = "tcp"
    from_port = 80
    to_port   = 80
    cidrs     = [var.developer_cidr]
  }
}

The Lightsail firewall documentation recommends limiting SSH to the address that needs administrative access. I restricted HTTP as well because this was a private experiment, not a public site.

For a real shared staging environment, the source ranges and access model would need another review. A developer's changing home IP is manageable for a short test but awkward for a team.

Step 3: Make Application State Observable

The application was intentionally small. It used Node's built-in HTTP module and exposed two endpoints:

if (request.url === "/health") {
  return sendJson(response, 200, { status: "ok" });
}

if (request.url === "/version") {
  return sendJson(response, 200, { version });
}

/health answered whether the process was responding. /version answered whether the expected deployment was running.

Those endpoints were more useful than a generic home page because each test had a precise success condition.

Docker Compose added the recovery policy and its own health check:

services:
  app:
    build:
      context: .
    environment:
      APP_VERSION: ${APP_VERSION:-local}
    ports:
      - "${HOST_PORT:-8080}:3000"
    restart: unless-stopped
    healthcheck:
      test:
        - CMD
        - node
        - -e
        - >-
          require('node:http').get('http://127.0.0.1:3000/health',
          response => process.exit(response.statusCode === 200 ? 0 : 1))
          .on('error', () => process.exit(1))
      interval: 5s
      timeout: 3s
      retries: 6

The container also ran as a non-root application user. No application secret was required, and the generated .env file on the server had mode 0600.

Step 4: Keep Deployment Separate From Terraform

Terraform created the server. A separate deployment script copied the application files over SSH and ran Compose:

./scripts/deploy.sh "$host" v2 "<SSH_PRIVATE_KEY_PATH>"

On the server, the important command was:

sudo docker compose up --build --detach --remove-orphans

That separation mattered. Changing v1 to v2 did not require replacing the instance or changing Terraform state.

The verification script then checked both endpoints until they agreed with the requested release:

./scripts/verify.sh "$host" v2 120

The observed result was:

PASS health=ok version=v2 recovery_seconds=0

The complete v1 to v2 deployment took 6 seconds, and the first verification request already returned the new version.

What the Runtime Tests Showed

Configuration alone does not prove that recovery works, so I tested two common situations.

First, I stopped the application process inside the container:

./scripts/test-container-restart.sh \
  "<HOST>" \
  "http://<HOST>" \
  "<SSH_PRIVATE_KEY_PATH>"

Docker Compose restarted the application. The health and version endpoints returned after 3 seconds:

PASS health=ok version=v1 recovery_seconds=3

Next, I rebooted the complete Lightsail server:

./scripts/test-instance-reboot.sh \
  "<HOST>" \
  v1 \
  lightsail-staging-lab-lightsail-instance \
  "<AWS_PROFILE>" \
  ca-central-1

The application returned without another deployment:

PASS health=ok version=v1 recovery_seconds=25
PASS reboot_recovery_seconds=29

The complete reboot test took 29 seconds. This confirmed that Docker started with the server and Compose brought the application back automatically.

What Lightsail Simplified

For this experiment, Lightsail removed a large amount of AWS setup.

I did not have to design a VPC, public subnet, internet gateway, route table, EC2 security group, or separate EBS volume. The instance bundle made the compute, memory, SSD storage, public addressing, and transfer allowance understandable as one small plan.

The static IP and firewall were also easier to reason about than a larger network stack. Terraform still made those decisions explicit, but there were fewer components to connect.

That simplicity made Lightsail a reasonable fit for one small staging server.

What Lightsail Still Leaves to the Developer

Lightsail simplified the AWS layer. It did not operate the application for me.

I still owned:

operating-system updates;
Docker installation and service startup;
application deployment and rollback;
logs and alerting;
SSH-key handling;
firewall source-address changes;
backups and data recovery; and
verification after every change.

It also remained one VM. There was no multi-zone availability, autoscaling, managed rollout controller, or automatic load-balancer health replacement.

I would not choose this setup for a system that needs private application tiers, multiple instances, team-scale access control, managed deployments, or strong availability guarantees. At that point, ECS, App Runner, Elastic Beanstalk, or a deliberately designed EC2 platform would deserve comparison.

Cost and Cleanup

The selected plan was $7 per month, but this experiment did not run for a month. AWS bills Lightsail instance bundles hourly up to the monthly maximum.

I did not collect final billing data, so I will not present an exact invoice. The resources existed for less than one day, making the theoretical instance-cost upper bound less than $0.24, before any free-tier discount. That is an estimate, not a billing record.

The experiment ended with:

terraform -chdir=terraform destroy \
  -var-file=terraform.tfvars

Terraform removed the instance, firewall configuration, static-IP attachment, static IP, and Lightsail key pair. A final state check returned no managed resources. The project created no snapshots, extra disks, load balancer, database, DNS zone, or Lightsail container service.

What I Would Use It For

Lightsail passed the part of the experiment that mattered most to my daily development workflow.

It gave me a small remote staging server with controlled access, a stable endpoint, fast redeployment, observable health, automatic process recovery, and reboot recovery. Terraform kept the AWS resources understandable, while Docker Compose kept application deployment separate.

I would use this pattern for a personal project, a short-lived demo, or a small integration environment where one VM is an acceptable availability tradeoff.

The next validation is narrower: run the final portable launch script through a completely unattended replacement and retain that result. Until then, this is a tested staging server pattern, not a production platform and not a claim of complete disaster recovery.

I Wanted Kubernetes Without an Always On EKS Bill

Lalit Bagga — Tue, 11 Aug 2026 23:52:08 +0000

I wanted a Kubernetes lab that would still be there the next morning.

A temporary cluster on my laptop would be useful for testing manifests, but it would disappear when I closed the lid. Amazon EKS would give me the managed AWS experience I eventually want to test, but keeping it online just so I could practise Deployments and Services did not make sense for me. The EKS control plane alone is currently priced at $0.10 per cluster-hour under standard support, before worker nodes, storage, and networking.

I already had another option: an Intel N100 home server with 16 GiB of memory running Debian and OpenMediaVault.

It was already always on. It was also already busy.

At the time of my retained snapshot, the server was running 28 Docker containers. Kubernetes had to fit beside those workloads, not take over the machine. That constraint shaped the entire decision.

The Lab I Actually Needed

My goal was not to reproduce a production cluster at home. Four CPU cores cannot provide a convincing multi-node failure lab while also protecting the services already using the server.

I needed something narrower:

Existing Debian/OpenMediaVault server
        ├── Existing Docker workloads
        └── Single-node Kubernetes lab
                ├── Always available
                ├── Privately administered
                └── Small enough to share the host

The home server would be my persistent learning environment. EKS could come later as a short-lived environment for AWS-specific behavior such as IAM, cloud load balancers, and managed control-plane operations.

That made the selection criteria practical:

It had to run directly on Debian.
It had to support a useful single-node cluster.
I needed control over the resources left for Kubernetes.
Bundled components could not claim ports already used by the server.
The setup had to remain close enough to Kubernetes that the manifests could move to EKS later.

Why k3s Fit the Server

I considered MicroK8s, k0s, and EKS, but k3s matched this host and the way I wanted to operate the lab.

MicroK8s has a convenient add-on workflow and is a natural choice when Ubuntu and snap are already part of the environment. k0s is also lightweight and packages Kubernetes as a single binary with a configurable architecture.

Both were reasonable options. My server was already Debian-based, and I wanted a small native service with straightforward configuration and a large body of homelab examples. k3s gave me that operating model without asking me to rebuild the host around a different workflow.

The official k3s requirements list two CPU cores and 2 GB of memory for a server node. A single-server cluster can use embedded SQLite by default, while other datastore options remain available if the topology grows later.

On this server, the installed k3s binary measured about 78.2 MiB. More important than the number was what I could configure around it: resource reservations, eviction thresholds, and the bundled components I wanted to disable.

That was the real reason I selected k3s. It was not simply “lightweight Kubernetes.” It gave me a Kubernetes control plane I could shape around a server that already had a job.

Protecting a Server That Was Already Busy

Running the k3s installer was the easy part. The real risk was forgetting that this server already had a job.

Before installing anything, I wrote a host-audit script. It records CPU, memory, swap, disk space, listening ports, systemd services, Docker containers, and recent kernel warnings. This gave me a clear picture of the server before Kubernetes touched it.

The raw audit stays outside Git because it may contain hostnames, IP addresses, mount paths, and internal service names. Only reviewed and sanitized results belong in the repository.

I also made the installation script stop when it finds something unexpected. It refuses to continue when:

it is not running with the required privileges;
k3s is already running;
an existing k3s installation or configuration is present; or
the Kubernetes API or kubelet ports are already occupied.

The k3s configuration was deliberately conservative:

write-kubeconfig-mode: "0640"
write-kubeconfig-group: "users"
disable:
  - traefik
  - servicelb
node-label:
  - "environment=homelab"
kubelet-arg:
  - "system-reserved=cpu=1,memory=6Gi"
  - "kube-reserved=cpu=250m,memory=512Mi"
  - "eviction-hard=memory.available<1Gi,nodefs.available<10%"

Ports 80 and 443 were already being used by services on the server. Because the bundled Traefik ingress controller and ServiceLB also expect to use those ports, I disabled both components.

The resource settings were equally important. Kubernetes was told to leave one CPU core and 6 GiB of memory available for the host. I also allowed another 250 millicores and 512 MiB for Kubernetes itself.

These settings are not a hard wall around the existing Docker containers. They simply reduce the resources Kubernetes can assign to Pods and give the kubelet thresholds for reacting to memory or disk pressure.

That leaves less capacity for the lab, but that is intentional. On a shared four-core server, protecting the existing workload matters more than giving Kubernetes every available resource.

Installing and Checking the Cluster

Once the checks passed and the configuration was written, the script used the official k3s installer:

curl --proto '=https' --tlsv1.2 -sfL https://get.k3s.io | sh -

It then waited for the node to report that it was ready:

k3s kubectl wait \
  --for=condition=Ready \
  node --all \
  --timeout=180s

I kept installation and verification separate. A successful installer only proves that the installation command completed. It does not prove that the cluster is healthy or that the rest of the server survived the change.

The verification script checks:

the k3s systemd service;
the installed k3s version;
the Kubernetes node and Pods;
the resources available to workloads; and
whether Traefik or ServiceLB appeared unexpectedly.

The installed server was running k3s v1.36.3+k3s1. At the time of the retained snapshot:

Kubernetes could allocate 2750 millicores of CPU to Pods;
about 7.9 GiB of memory was allocatable to Pods;
all six observed Kubernetes Pods were running;
all 28 existing Docker containers were still running;
none of the Docker containers reported being unhealthy; and
no systemd unit had failed.

The k3s service was using approximately 726 MiB of memory, with a recorded peak of 1.19 GiB. The host still had about 9.7 GiB of available memory and 296 GiB of available disk space.

One warning remained: 834 MiB of the server's 976 MiB swap was in use. That does not automatically mean the server was under active memory pressure, but it was something I still needed to investigate.

These measurements are a snapshot, not a performance benchmark. They do not prove that the server will remain healthy under every workload.

They answer a smaller and more useful question: after adding k3s, was the cluster running without an obvious failure in the services already sharing the machine?

At that moment, yes.

Then I Made Kubernetes Do Real Work

A ready node proves that Kubernetes started. It does not prove that I can operate an application through it.

I deployed a small Nginx workload with three replicas, readiness and liveness probes, and CPU and memory constraints. A Service gave the replaceable Pods one stable internal address, and NodePort exposed the same application over my private network.

Private client
      ↓
Home server:30081
      ↓
Kubernetes Service
      ↓
EndpointSlice
      ↓
Three Nginx Pods

I verified the Service from inside the cluster:

klab -n lab exec deployment/hello-nginx -- \
  wget -qO- http://hello-nginx:80

The request returned the expected page and identified the Pod that served it. The same application was reachable through the private NodePort path.

I then changed the page and watched the Deployment replace the Pods through a rolling update. Kubernetes created a new ReplicaSet, scaled it up, and scaled the previous one down while the Service remained stable.

Finally, I rolled the Deployment back and reconciled the YAML with the running version. A clean kubectl diff confirmed that the source and live cluster agreed again.

That sequence mattered more to me than a successful installer message. The server was now a persistent place to practise the controller relationships I wanted to understand: Deployment to ReplicaSet, Service to EndpointSlice, rollout to rollback, and desired state to live state.

Why EKS Still Belongs in the Plan

Choosing k3s did not remove EKS from the journey. It gave EKS a more focused role.

According to the current EKS pricing page, a standard-support cluster costs $0.10 per hour before the compute and supporting AWS resources are added. That is roughly $73 for a 730-hour month for the control plane alone.

I do not need to pay that continuously while learning what a ReplicaSet does or why a Service follows labels. The home k3s cluster can hold that state and remain available every day.

When I need to verify AWS-specific behavior, I can create EKS deliberately, capture the evidence, and remove it. The local cluster handles repetition; the cloud cluster handles cloud-specific proof.

What Comes Next

The first application lab exposed the next platform problem.

I rolled Kubernetes back to Version 2, but my YAML still declared Version 3. The cluster and source disagreed, and Kubernetes alone had no reason to repair that relationship.

The next milestone is GitOps with Argo CD: make Git the declared source of truth, reproduce the drift deliberately, and observe reconciliation instead of correcting it by hand.

That is where this stops being only a Kubernetes lab and starts becoming a delivery platform.

My Terraform Drift Pipeline Fixed the Change, Then Forgot It

Lalit Bagga — Fri, 07 Aug 2026 00:51:46 +0000

My Terraform drift pipeline could detect a manual EC2 tag change, classify it as LOW, and run Terraform to remove it.

Then the pipeline moved on.

The evidence existed, but it was spread across CodeBuild output, Lambda logs, and an SNS message. If I wanted to know what changed, how it was classified, and whether remediation started, I had to reconstruct the event from multiple AWS services.

The pipeline could act on drift. It could not remember drift.

Phase 4 added that memory: a durable DynamoDB record, a read only API, and a small dashboard that turns the event history into something I can inspect without opening three AWS consoles.

The Stack

Terraform drift event
        ↓
SNS
        ↓
Severity Lambda
        ├── classifies HIGH / MEDIUM / LOW
        ├── starts remediation for eligible LOW drift
        └── writes the audit event to DynamoDB
                            ↓
                    API Gateway HTTP API
                            ↓
                    Read only Lambda
                            ↓
                    DynamoDB Query
                            ↓
             CloudFront → static dashboard
                            ↑
                      private S3 bucket

The browser receives static HTML, CSS, and JavaScript from CloudFront. JavaScript calls API Gateway, the API Lambda queries DynamoDB, and the returned JSON becomes the live dashboard.

There is no EC2 web server and no application process running continuously.

Step 1: Store Every Classified Event

I created a DynamoDB table with a composite key:

resource "aws_dynamodb_table" "drift_events" {
  name         = "terraform-drift-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "project"
  range_key    = "timestamp"

  attribute {
    name = "project"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }
}

project groups the history for one Terraform project. The ISO 8601 timestamp orders its events.

DynamoDB only requires attribute definitions for keys and indexes. Fields such as high_count, changes, and status still belong in each item, but they do not belong in the table schema block.

I passed the table name into the existing severity Lambda instead of putting it directly in the code:

environment {
  variables = {
    DRIFT_EVENTS_TABLE       = aws_dynamodb_table.drift_events.name
    REMEDIATION_PROJECT_NAME = aws_codebuild_project.remediation.name
  }
}

The same Lambda role received permission to write to that specific table:

{
  Effect = "Allow"
  Action = [
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:GetItem",
    "dynamodb:Query",
    "dynamodb:Scan",
  ]
  Resource = aws_dynamodb_table.drift_events.arn
}

After classification, Lambda builds one audit item:

item = {
    "project": project,
    "timestamp": timestamp,
    "drift_count": len(changes),
    "high_count": len(classified["HIGH"]),
    "medium_count": len(classified["MEDIUM"]),
    "low_count": len(classified["LOW"]),
    "changes": classified,
    "action_taken": action_taken,
    "status": status,
}

if remediation_build_id:
    item["remediation_build_id"] = remediation_build_id

drift_table.put_item(Item=item)

The record stores the decision alongside the evidence used to make it. I do not need to correlate an SNS payload with a separate classification log just to understand one event.

For a change that needs human review, the item records manual_review. When Lambda starts the remediation build, it records remediation_triggered and includes the CodeBuild build ID.

That turns the audit item into the connection between detection, classification, and the action taken by the pipeline.

Step 2: Add a Read Only History API

The dashboard should not receive DynamoDB credentials. It calls an HTTP API instead.

I created a second Lambda whose only data permission is:

Action   = ["dynamodb:Query"]
Resource = aws_dynamodb_table.drift_events.arn

The route requires a project:

GET /drift-history?project=Three-Tier-Infra

The Lambda queries the partition key and returns newest events first:

response = table.query(
    KeyConditionExpression="#project = :project",
    ExpressionAttributeNames={"#project": "project"},
    ExpressionAttributeValues={":project": project},
    ScanIndexForward=False,
)

This is why the table uses project as its partition key. The API does not need to scan the complete table to retrieve one project's history.

DynamoDB numbers arrive in Python as Decimal, which json.dumps cannot serialize directly. I added a small serializer:

def json_serializer(value):
    if isinstance(value, Decimal):
        return int(value) if value % 1 == 0 else float(value)
    raise TypeError(f"Cannot serialize {type(value)}")

API Gateway uses an HTTP API with a Lambda proxy integration:

resource "aws_apigatewayv2_route" "get_drift_events" {
  api_id    = aws_apigatewayv2_api.drift_api.id
  route_key = "GET /drift-history"
  target    = "integrations/${aws_apigatewayv2_integration.drift_api.id}"
}

Calling the deployed route returned HTTP 200 and the stored event:

curl \
  "https://8zg1x51wne.execute-api.us-east-2.amazonaws.com/drift-history?project=Three-Tier-Infra"

The response contained:

{
  "events": [
    {
      "project": "Three-Tier-Infra",
      "timestamp": "2026-07-26T15:29:32Z",
      "drift_count": 1,
      "high_count": 0,
      "medium_count": 0,
      "low_count": 1,
      "action_taken": "remediation_triggered",
      "status": "REMEDIATION_STARTED"
    }
  ]
}

The changed resource was:

module.compute.aws_instance.bastion_host

Terraform reported it as an aws_instance update.

Step 3: Build a Dashboard Without a Web Server

The dashboard is static in how it is hosted, not in the data it displays.

S3 stores three files:

dashboard/
├── index.html
├── app.js
└── styles.css

CloudFront serves those files over HTTPS. app.js calls the API whenever the page loads or the user refreshes the data.

The dashboard shows total events, severity counts, detection timestamps, Terraform resource addresses, actions, and remediation status. A severity filter narrows the history, and the page refreshes automatically every 60 seconds.

The S3 bucket remains private. CloudFront uses Origin Access Control to read objects:

resource "aws_cloudfront_origin_access_control" "drift_dashboard" {
  name                              = "terraform-drift-dashboard"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

The bucket policy grants s3:GetObject to the CloudFront service only when the request comes from this distribution.

Step 4: Keep the Public Surface Read Only

I deliberately did not add a Fix drift button.

The first dashboard is for visibility. It cannot start CodeBuild, run Terraform, or change DynamoDB records.

The API exposes one GET route. I added API Gateway throttling:

default_route_settings {
  throttling_burst_limit = 20
  throttling_rate_limit  = 10
}

I also restricted browser CORS responses to the deployed CloudFront origin:

cors_configuration {
  allow_headers = ["content-type"]
  allow_methods = ["GET"]
  allow_origins = [
    "https://${aws_cloudfront_distribution.drift_dashboard.domain_name}"
  ]
  max_age = 3600
}

CORS is not authentication. Someone who knows the API URL can still call it outside a browser.

That is an accepted limitation for this public dashboard. The API returns resource addresses, classifications, and actions. It does not return raw Terraform state, credentials, plan values, or CloudWatch logs.

An operator remediation API would require authentication, authorization, review, concurrency controls, and an audit record of who approved the action. It does not belong in this first dashboard.

Verification

I created real drift by changing the tag on:

module.compute.aws_instance.bastion_host

The detection pipeline reported one LOW update. Lambda started remediation and wrote this event to DynamoDB:

project:              Three-Tier-Infra
drift_count:          1
high_count:           0
medium_count:         0
low_count:            1
action_taken:         remediation_triggered
status:               REMEDIATION_STARTED

The record also contained the remediation CodeBuild ID and the classified resource details.

After remediation, I returned to the EC2 instance and confirmed the tag was restored to the value declared in Terraform.

I then tested the deployed layers independently:

CloudFront index.html: HTTP 200
CloudFront app.js:     HTTP 200
API history query:     HTTP 200
CORS origin:           CloudFront dashboard URL

The API response to the dashboard origin included:

access-control-allow-origin: https://dqjunwd8v0pry.cloudfront.net

The complete verified path was:

EC2 tag changed
→ Terraform detected one update
→ Lambda classified it LOW
→ Lambda wrote the event to DynamoDB
→ remediation restored the tag
→ API returned the audit event
→ dashboard displayed the history

The DynamoDB status remains REMEDIATION_STARTED. That proves CodeBuild accepted the build request, not that the build ultimately succeeded. I verified success separately by observing the restored EC2 tag.

Updating the audit item from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED is separate completion tracking work.

What Comes Next

The system can now detect drift, classify it, remediate the tested LOW tag scenario, remember the event, expose it through an API, and display it in a dashboard.

The next step is making the audit lifecycle more complete. The current item proves that CodeBuild accepted the remediation request, but it does not record the final build result.

Tracking that result would let the same event move from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED or REMEDIATION_FAILED.

I Added a Tag in AWS. Terraform Removed It Automatically.

Lalit Bagga — Tue, 04 Aug 2026 22:30:26 +0000

I opened the AWS console, selected an EC2 instance, added a tag, and saved it.

The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another.

That was the test.

My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually.

So I added a separate remediation pipeline.

The Stack

Manual EC2 tag added in AWS
        ↓
Drift CodeBuild runs terraform plan
        ↓
SNS publishes the structured change
        ↓
Lambda classifies the update as LOW
        ↓
Remediation CodeBuild runs terraform apply
        ↓
Terraform removes the unmanaged tag

Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources.

Step 1: Make EC2 Updates Eligible for Remediation

The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories.

An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation.

The resource type alone was not enough. The action mattered too.

I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH:

def classify_change(resource_type, actions):
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        elif "update" in actions:
            return "LOW"
        return "MEDIUM"
    else:
        return "LOW"

For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path.

Step 2: Keep Deletions Out of the Automatic Path

LOW did not automatically mean “start Terraform.” I also filtered out deletions:

low_changes_to_remediate = [
    change for change in classified["LOW"]
    if "delete" not in change.get("actions", [])
]

This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path.

If an eligible LOW change remains, Lambda starts the remediation CodeBuild project:

if low_changes_to_remediate:
    codebuild.start_build(
        projectName=os.environ.get(
            "REMEDIATION_PROJECT_NAME",
            "terraform-drift-remediation",
        )
    )

The Lambda role only needs permission to start that specific project:

{
  Effect   = "Allow"
  Action   = ["codebuild:StartBuild"]
  Resource = aws_codebuild_project.remediation.arn
}

Lambda makes the routing decision. CodeBuild performs the repair.

Step 3: Run Remediation Separately

I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled.

The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin
  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init
  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - terraform apply -auto-approve -lock=false

The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project.

That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply.

The Repository Was Part of the Control Loop

One remediation run tried to include unrelated resource actions because CodeBuild was cloning a version of the Three-Tier repository that did not match the configuration I intended to operate.

The problem was not the classifier. Lambda correctly identified the triggering drift. The remediation build was applying the complete repository it cloned from GitHub.

I pushed the current Three-Tier configuration before running the test again.

This is easy to miss in CI. Terraform does not know what exists only on a developer's laptop. If CodeBuild clones GitHub, the committed branch is the configuration Terraform will enforce.

The Tag Disappeared

With the current repository in GitHub, I repeated the test:

Opened the EC2 instance in the AWS console.
Added a tag manually and saved it.
Started the Terraform drift detector.
The plan reported an update to the EC2 instance.
Lambda classified the update as LOW.
Lambda started the remediation CodeBuild project.
CodeBuild ran Terraform apply.
I returned to the EC2 Tags view and confirmed that the manually added tag was gone.

Terraform restored the infrastructure to what was declared in Git.

Git configuration: tag absent
AWS resource:       tag added manually
Terraform result:   tag removed

The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it.

MEDIUM, HIGH, and deletion paths still stop for human review. That boundary is intentional. Automatic remediation is useful only when the system can distinguish a small correction from a change that deserves a person.

What Comes Next

The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs.

The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS services.

Detection found the tag. Classification decided it was LOW. Remediation removed it.

Now the system needs to remember that it happened.

My Terraform Drift Alert Could Not Explain the Drift

Lalit Bagga — Wed, 22 Jul 2026 02:40:34 +0000

My Terraform drift detector had a problem.

It could run terraform plan, detect drift, and send me an email through SNS. But the message only told me that drift existed.

It did not tell me which resources changed. It did not tell me whether the change affected IAM, networking, or something less critical.

So I changed the alert into a structured event and sent it to two places: SQS for audit processing and Lambda for severity classification.

The Stack

CodeBuild                  → runs terraform plan
        ↓
terraform show -json       → extracts the changed resources
        ↓
SNS                        → publishes one structured event
        ↓
        ├── SQS            → keeps the event for audit processing
        └── Lambda         → classifies changes as HIGH, MEDIUM, or LOW
                 ↓
            CloudWatch     → stores the classification logs

Step 1: Publish Structured Drift Details

The build already saved the Terraform plan:

terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

Exit code 2 means Terraform found changes. When that happens, I convert the saved plan to JSON:

terraform show -json plan.tfplan > /tmp/plan.json

Terraform plan JSON is large. SNS and Lambda do not need all of it, so I used jq to keep only the useful fields:

DRIFT_CHANGES=$(jq -c '[
  .resource_changes[] |
  select(.change.actions != ["no-op"]) |
  {
    address: .address,
    type: .type,
    actions: .change.actions
  }
]' /tmp/plan.json)

Each change now contains:

{
  "address": "module.compute.aws_iam_role.ec2_role",
  "type": "aws_iam_role",
  "actions": ["create"]
}

The important part here is what I left out. I did not publish the complete before-and-after resource values. Resource address, type, and action are enough for classification without sending the entire Terraform plan through SNS.

Step 2: Send the Event to SQS

I created an SQS queue called terraform-drift-audit and subscribed it to the SNS topic.

resource "aws_sqs_queue" "drift_audit" {
  name = "terraform-drift-audit"
}

resource "aws_sns_topic_subscription" "sqs" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.drift_audit.arn
}

I also added the queue URL as a Terraform output:

output "sqs_queue_url" {
  description = "URL of the drift audit SQS queue"
  value       = aws_sqs_queue.drift_audit.url
}

After terraform apply, I can retrieve the URL without opening the AWS console:

terraform output -raw sqs_queue_url

That URL is what I used with the AWS CLI to check whether SNS delivered the message.

SQS also needs permission to receive messages from SNS. I restricted the queue policy to the drift topic instead of allowing every SNS topic:

Condition = {
  ArnEquals = {
    "aws:SourceArn" = aws_sns_topic.drift.arn
  }
}

SQS is not the final audit database. It keeps the event available for a future consumer. Permanent drift history belongs to a later phase.

The Queue Existed in the Wrong Region

I copied the queue URL into this command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10

AWS returned NonExistentQueue.

The queue existed. My AWS CLI was using us-east-1, while the Terraform project created the queue in us-east-2.

The fix was adding the region explicitly to the same command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The resource name was correct. The CLI was asking the wrong regional endpoint.

After fixing the region, I could see the structured drift message in SQS.

Step 3: Classify Every Change with Lambda

The Lambda function receives the same SNS event that went to SQS.

SNS does not send my drift JSON as the top-level Lambda event. It wraps the message inside Records[].Sns.Message. The function must first extract that string and parse it back into JSON.

This is the complete classifier:

import json
import os
from datetime import datetime, timezone

# Severity definitions
HIGH_RISK_TYPES = {
    "aws_security_group",
    "aws_security_group_rule",
    "aws_iam_role",
    "aws_iam_policy",
    "aws_iam_role_policy",
    "aws_iam_user",
    "aws_iam_group",
    "aws_iam_access_key",
}

MEDIUM_RISK_TYPES = {
    "aws_instance",
    "aws_db_instance",
    "aws_ecs_cluster",
    "aws_ecs_service",
    "aws_ecs_task_definition",
    "aws_lb",
    "aws_lb_listener",
    "aws_lb_target_group",
    "aws_nat_gateway",
    "aws_route_table",
    "aws_network_acl",
}


def classify_change(resource_type, actions):
    """Classify a single resource change by severity."""
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        # Delete or replace is more concerning than create
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        return "MEDIUM"
    else:
        return "LOW"


def lambda_handler(event, context):
    """Process SNS message with drift details and classify severity."""
    print(f"Received event: {json.dumps(event)}")

    for record in event.get("Records", []):
        # SNS message is in the SNS record
        sns_message = record.get("Sns", {}).get("Message", "{}")

        try:
            drift_data = json.loads(sns_message)
        except json.JSONDecodeError:
            print(f"Failed to parse SNS message: {sns_message}")
            continue

        timestamp = drift_data.get(
            "timestamp",
            datetime.now(timezone.utc).isoformat()
        )
        project = drift_data.get("project", "unknown")
        changes = drift_data.get("changes", [])

        # Classify each change
        classified = {"HIGH": [], "MEDIUM": [], "LOW": []}

        for change in changes:
            address = change.get("address", "unknown")
            resource_type = change.get("type", "unknown")
            actions = change.get("actions", [])

            severity = classify_change(resource_type, actions)
            classified[severity].append({
                "address": address,
                "type": resource_type,
                "actions": actions,
            })

        # Build summary
        summary = {
            "timestamp": timestamp,
            "project": project,
            "total_changes": len(changes),
            "high_count": len(classified["HIGH"]),
            "medium_count": len(classified["MEDIUM"]),
            "low_count": len(classified["LOW"]),
            "classified": classified,
        }

        # Log the classified drift
        print(
            f"Drift classification: {json.dumps(summary, indent=2)}"
        )

        # TODO Phase 3: Auto-remediate HIGH severity changes
        # TODO Phase 3: Alert on MEDIUM severity changes

    return {"statusCode": 200, "body": "Drift classified"}

The code does four things:

Extracts the SNS message from the Lambda event.
Parses the structured drift JSON.
Classifies each resource by type and action.
Writes the complete summary to CloudWatch Logs.

IAM and security group resources are HIGH. Resources such as EC2, RDS, ECS, load balancers, NAT gateways, and route tables start as MEDIUM. Everything else starts as LOW.

A delete or replacement on a MEDIUM resource becomes HIGH. This classifier is deliberately simple. It is a deterministic first version that I can read, test, and improve later.

The Lambda Package Needed Another Provider

Terraform packages the Python file using archive_file:

data "archive_file" "severity_lambda" {
  type        = "zip"
  source_file = "${path.module}/lambda/severity_classifier.py"
  output_path = "${path.module}/lambda/severity_classifier.zip"
}

The first run failed because the archive provider was missing from .terraform.lock.hcl.

The fix was:

terraform init -upgrade

That downloaded hashicorp/archive and updated the lock file.

The generated ZIP does not need to be committed. Terraform creates it when the configuration runs.

Verification

I triggered a real drift event and checked both destinations from the terminal.

First I read up to 10 messages from SQS:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The structured message appeared in the SQS queue. Lambda received the event and wrote this summary to CloudWatch:

aws logs tail /aws/lambda/terraform-drift-severity \
  --follow \
  --region us-east-2

--follow keeps the terminal attached to the log group. When the next drift event invokes Lambda, the classification appears directly in the terminal.

{
  "high_count": 8,
  "medium_count": 13,
  "low_count": 38
}

The HIGH results included an IAM role and an ALB security group:

module.compute.aws_iam_role.ec2_role
module.security.aws_security_group.alb_sg

Lambda classified all 59 resource changes reported by that Terraform plan. This confirmed that the structured SNS event reached Lambda and that each change was assigned a severity.

What Is Next

The system can now tell me what changed and how serious each change might be.

It still does not remediate anything.

That is intentional. The next step is deciding what should happen after classification. I will start with LOW-severity updates that are safe to test automatically. MEDIUM, HIGH, and deletion paths will remain outside automatic remediation and will later require human review.

That is Phase 3.

The Security Group Change Terraform Never Told Me About

Lalit Bagga — Wed, 15 Jul 2026 14:06:02 +0000

Terraform had a blind spot in my three-tier project.

The code could create the infrastructure. The state lived in S3. But if someone changed a security group in the AWS console, Terraform would not tell me when it happened.

I would discover it the next time I ran terraform plan.

That could be tomorrow or six months later.

So I built a drift detector that runs the same plan inside CodeBuild and sends an SNS email when Terraform finds a difference.

I made CodeBuild the entry point. One run compares the deployed infrastructure with Terraform and reports any difference by email.

The Stack

CodeBuild run                 → starts the detector
        ↓
CodeBuild                     → installs Terraform and clones the repo
        ↓
terraform plan                → compares code, state, and AWS
        ↓
SNS                           → sends an email when drift is detected

Why I Used CodeBuild Instead of Lambda

My first idea was Lambda.

A scheduled function could run terraform plan, then SNS could send the result.

The problem was packaging Terraform. I could build a Lambda container image and store it in ECR, but that adds a Dockerfile, image builds, image updates, and another repository to maintain.

CodeBuild already does what this job needs: start a clean Linux environment, run a script, stream logs to CloudWatch, and exit.

For a Terraform plan job, that was the simpler choice.

Step 1: Build the CodeBuild Job

The CodeBuild project does not build application source. Its source type is NO_SOURCE, and Terraform injects the buildspec directly:

resource "aws_codebuild_project" "drift" {
  name          = "terraform-drift"
  description   = "Runs terraform plan against Three-Tier-Infra to detect drift"
  build_timeout = 10
  service_role  = aws_iam_role.codebuild.arn

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/amazonlinux2-x86_64-standard:5.0"
    type         = "LINUX_CONTAINER"

    environment_variable {
      name  = "SNS_TOPIC_ARN"
      value = aws_sns_topic.drift.arn
    }
  }

  source {
    type      = "NO_SOURCE"
    buildspec = file("${path.module}/buildspec.yml")
  }

  logs_config {
    cloudwatch_logs {
      group_name = "/codebuild/terraform-drift"
    }
  }
}

Passing SNS_TOPIC_ARN as an environment variable keeps the generated ARN out of the buildspec. If Terraform recreates the topic, the next apply updates CodeBuild with the new ARN.

Step 2: Run Terraform Plan and Capture Drift

This was the buildspec used by the detector:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin

  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init

  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - EXIT_CODE=0
      - terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?
      - |
        if [ "$EXIT_CODE" -eq 2 ]; then
          echo "Drift detected. Publishing details to SNS."
          aws sns publish \
            --topic-arn "$SNS_TOPIC_ARN" \
            --subject "Terraform Drift Detected" \
            --message "Drift found in Three-Tier-Infra"
        elif [ "$EXIT_CODE" -eq 1 ]; then
          echo "Error during terraform plan."
          exit 1
        else
          echo "No drift detected."
        fi

The important part is -detailed-exitcode.

terraform plan -detailed-exitcode returns:

0 → plan succeeded, no changes
1 → Terraform failed
2 → plan succeeded, changes detected

The buildspec captures that result before CodeBuild decides whether the job succeeded:

EXIT_CODE=0
terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

That gives the pipeline three clear paths: do nothing when the infrastructure matches, stop when Terraform fails, and notify when Terraform finds drift.

The SSH Key Only Existed on My Laptop

The Three-Tier project originally created its EC2 key pair from a local file:

public_key = file("~/.ssh/bastion-key.pub")

That worked on my machine. CodeBuild starts in an ephemeral container where that file does not exist.

I moved the public key to SSM Parameter Store and changed the Three-Tier repository:

data "aws_ssm_parameter" "bastion_key" {
  name = "/threeTier/bastionKeyPublic"
}

resource "aws_key_pair" "bastion_key" {
  key_name   = "bastion-key"
  public_key = data.aws_ssm_parameter.bastion_key.value
}

The CodeBuild role received ssm:GetParameter for the /threeTier/* path.

This removed the developer-laptop dependency. The same Terraform plan can now run locally or in CodeBuild.

Step 3: Send the Email Through SNS

The SNS topic and email subscription are managed by Terraform:

resource "aws_sns_topic" "drift" {
  name = "terraform-drift-alerts"
}

resource "aws_sns_topic_subscription" "drift" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

CodeBuild also needs sns:Publish permission scoped to that topic. CloudWatch permissions allow the build to create its log group and write the execution logs.

The role has four jobs:

S3 read              → read Terraform state
SSM read             → retrieve shared parameters
SNS publish          → send the drift alert
CloudWatch Logs      → record the build

It does not have S3 write permissions and it cannot apply Terraform changes.

Verification

I started the CodeBuild project directly:

aws codebuild start-build \
  --project-name terraform-drift \
  --region us-east-2

The command returns a build ID. Use it to inspect the result:

aws codebuild batch-get-builds \
  --ids <build-id> \
  --region us-east-2

The successful drift path produced three pieces of evidence:

CodeBuild completed the Terraform plan
        ↓
The build published to terraform-drift-alerts
        ↓
The subscribed email received "Terraform Drift Detected"

The detected change included the SSM refactor made in the Three-Tier repository.

That confirmed the complete scope of this version: start the detector, compare the infrastructure, identify drift, and deliver the result by email.

What Is Next

Running the detector on demand proved the idea, but an email that says "drift happened" is not enough.

The next phase converts the Terraform plan to JSON, sends the event to SQS, and uses Lambda to classify every change as HIGH, MEDIUM, or LOW.

EventBridge will be added in an upcoming blog.

DEV Community: Lalit Bagga

My AI Agent Remembered Staging. It Still Didn’t Know If It Was Healthy.

The Agent Already Had One Tool

Remembering Across Sessions Took Time

A Different Actor Did Not Get My Preference

Staging Exposed the Real Boundary

The Rule I Would Keep

What Comes Next

References

I Gave My Agent a Deployment Tool. It Could Finally Answer.

The Missing Piece Was a Tool

I Defined Healthy in Code

SAM and Docker Gave Me a Local Lambda Test

The Tool Schema Told the Agent What It Could Ask

The Gateway Existed, but the Protocol Was Wrong

The Same Question Finally Produced Evidence

What Comes Next

References

I Deployed My First AgentCore Harness and Asked It About Production

Why I Started with Harness

The Boundary Was the Product

The First Deployment Failed Before It Reached AWS

Model Access Was a Separate Failure Domain

What agentcore dev Taught Me

The Tests I Ran

1. Can it admit that it does not know?

2. Can a user talk it into a capability it does not have?

3. Does a session behave like memory?

Finding the Trace Was Its Own Experiment

Cleaning Up Without Deleting the Learning

Conclusion

References

My Kubernetes App Moved to EKS Unchanged. Everything Around It Didn't.

Why I Returned to EKS

What I Kept the Same

The Smallest Useful AWS Test

The Worker Could Not Join

AWS Rejected My Planned Worker

The Access Problem Was Operational, Not Kubernetes

Git Deployed the Workload

I Changed the Cluster Behind Git's Back

The Cost Test

Teardown Was Part of the Test

What Was Actually Portable

Argo CD Fixed My Drift, Then Deployed My Bad Release

CI Built the Release but Did Not Deploy It

Argo CD Made the Cluster Follow Git

First, I Changed the Cluster Behind Git’s Back

Then I Put a Bad Setting in Git

Recovery Had to Correct Git

I Timed a Fresh Release End to End

What the Implementation Proved

Conclusion

The Rollback Worked. My Next Deploy Could Break It Again.

The Real Problem Was Two Different Answers

I Considered Who Should Be Allowed to Change the Cluster

I Designed the Process Before Choosing the Tools

I Needed an Exact Version, Not a Moving Name

I Used Helm to Keep the Kubernetes Instructions Together

I Chose a Small Application That Could Fail Safely

I Kept Argo CD Private

I Defined the Failure Tests Before Claiming Success

What the Design Did Not Promise

What Came Next

I Turned a $7 Lightsail VM Into a Staging Server

The Smallest Useful Staging Stack

Step 1: Define More Than an Instance

Step 2: Replace the Default Firewall Rules

Step 3: Make Application State Observable

Step 4: Keep Deployment Separate From Terraform

What the Runtime Tests Showed

What Lightsail Simplified

What Lightsail Still Leaves to the Developer

Cost and Cleanup

What I Would Use It For

I Wanted Kubernetes Without an Always On EKS Bill

The Lab I Actually Needed

Why k3s Fit the Server

Protecting a Server That Was Already Busy

Installing and Checking the Cluster

What `agentcore dev` Taught Me