DEV Community: Atul Vishwakarma

The Google File System Explained: How Google Built Storage That Expects to Fail

Atul Vishwakarma — Fri, 21 Aug 2026 02:30:00 +0000

In 2003, three Google engineers — Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung — published a paper describing the storage system they'd built to keep up with Google's data. It's called The Google File System (GFS), and it went on to influence nearly every large-scale storage system that came after it, most directly the Hadoop Distributed File System (HDFS).

This is the first post in a series where I break down classic systems papers that quietly shaped how we build infrastructure today. We'll follow the same structure every time, so you always know what to expect. Today: GFS.

You can read the original paper here: The Google File System (PDF). All credit for the design and ideas below belongs to the original authors and Google.

Problem Statement

By the early 2000s, Google was crawling, indexing, and processing the web — which meant generating and chewing through enormous volumes of data across a lot of cheap machines. They needed a file system that could:

Store many terabytes (eventually petabytes) of data reliably.
Run on inexpensive, commodity hardware rather than expensive specialized storage arrays.
Stay available and correct even though individual machines would constantly fail.
Support workloads that mostly appended data rather than randomly overwriting it — think log files, crawl results, and intermediate computation output.
Serve hundreds of clients concurrently with high throughput, without needing every read or write to be blazingly low-latency.

No off-the-shelf file system at the time was designed around these specific realities. So Google built its own.

Why the Older Approach Struggled

Traditional file systems — including distributed ones like AFS at the time — were built around a different set of assumptions:

Failure was treated as an exception, not a routine event. Systems assumed disks and machines mostly worked, and failure handling was bolted on rather than core to the design.
Files were assumed to be relatively small, and systems optimized for fine-grained I/O rather than huge, multi-gigabyte files.
Random writes were a first-class use case. Filesystems were built to support arbitrary overwrites anywhere in a file, which added complexity that Google's workloads didn't actually need.
Client-side caching of file data was a default assumption, which brings cache-coherence complexity — but Google's workloads mostly streamed through huge files once, so caching data blocks bought little.
POSIX compliance was treated as a requirement, which constrains the API and adds overhead that isn't always earning its keep for a specialized internal system.

None of this made older systems "wrong" — they were solving a different problem. But applied directly to Google's scale and access patterns, these assumptions became a poor fit. So instead of retrofitting an existing file system, Google's engineers designed one from scratch around their actual workload.

Core Architecture

At a high level, a GFS cluster has three kinds of participants: a single master, many chunkservers, and the clients that read and write data.

The Master

One master process holds all the metadata in memory: the file namespace, which files map to which chunks, and where those chunks currently live. Crucially, the master is only involved in control decisions — it tells clients where to find data, but it never touches the actual bytes being read or written. That separation is what keeps it from becoming a bottleneck even as the cluster scales.

Chunkservers

Files are split into fixed-size 64 MB chunks, and each chunk is replicated (three copies, by default) across different chunkservers and different racks. Chunkservers just store chunks as regular files on Linux and serve reads/writes for the byte ranges clients ask for.

Clients

A client asks the master "which chunkserver holds this chunk?", caches that answer briefly, and then talks directly to the relevant chunkserver for the actual data transfer. Metadata traffic and data traffic are cleanly split.

Why 64 MB chunks?

This is one of the more counterintuitive design choices, and it's worth calling out on its own. A large chunk size means:

Clients need to talk to the master far less often, since one chunk covers a lot of data.
The total number of chunks — and therefore the size of the master's in-memory metadata — stays manageable even at huge scale.
Sequential reads and writes, which dominate Google's workload, get to be highly efficient.

Writes and leases

For mutations, the master grants a temporary "lease" to one replica, making it the primary for that chunk. The primary decides the order in which concurrent writes are applied, and the other replicas follow that same order. Data itself is pushed along a pipelined chain between chunkservers — not broadcast from the client to every replica at once — which makes better use of available network bandwidth.

Record append

GFS added an operation not found in traditional file systems: atomic record append. Multiple clients can append to the same file concurrently, and GFS guarantees each individual append lands atomically somewhere in the file — without the clients needing any external locking. This turned out to be a great fit for producer-consumer style workloads, like many worker processes writing results into one shared output file.

Key Trade-offs

GFS makes almost none of its decisions for free — every design win is paired with something given up:

Choice	What you gain	What you give up
Single master	Simple design, easy global coordination, easy to reason about	A single point that must be carefully protected (via replication and fast recovery) from becoming a bottleneck or outage source
Large (64 MB) chunks	Less master traffic, smaller metadata footprint, efficient sequential I/O	Small files or hot files can create load imbalance ("hot spots") on the few chunkservers that hold them
Relaxed consistency model	Simpler, faster implementation; no expensive coordination on every write	Applications must be written defensively — using checksums, unique record IDs, and append-friendly patterns — to tolerate duplicate or padded data
No client-side data caching	No cache-coherence complexity	Every read of "hot" data goes back to a chunkserver rather than a local cache
Optimized for append-heavy, sequential workloads	Excellent throughput for its target use case	Not a good general-purpose file system — small random writes are supported but never optimized

The overarching theme: GFS repeatedly chose simplicity and throughput for a specific known workload over generality. That's a defensible trade only because Google controlled both the file system and the applications running on it.

Failure Cases or Limitations

The paper is refreshingly honest about where the system falls short:

Single-client write throughput was disappointing. The paper reports it was roughly half of the theoretical network limit, largely because of how their networking stack interacted with the pipelined replication scheme.
Small files can create hot spots. If a single-chunk file (like an executable pushed to hundreds of machines at once) suddenly gets many concurrent readers, the few chunkservers holding it can get overloaded. Google's real-world fix was operational — higher replication for such files and staggered rollout — rather than architectural.
The relaxed consistency model isn't free for application developers. Because concurrent writes can leave a region "consistent but undefined" (all replicas agree, but the content may be a merge of multiple writers), applications must be written defensively — self-validating records, checksums, and idempotency-friendly identifiers become mandatory, not optional.
The master is a structural bottleneck risk. Even with fast restart and log replication, one process holding all metadata in memory sets a ceiling on file count and requires real engineering (checkpointing, prefix-compressed namespace, binary search over metadata) to keep operations fast at scale.
It is explicitly not POSIX-compliant and not general-purpose. GFS was tuned tightly to Google's own workloads. Bolting it onto a very different access pattern (lots of small random writes, for instance) would likely perform poorly.

What Modern Engineers Can Learn

Even if you'll never build a file system, several ideas from this paper show up constantly in modern distributed systems design:

Design for the workload you actually have, not a hypothetical general one. GFS's biggest wins came from refusing to support use cases (random writes, POSIX semantics) that weren't part of the real workload.
Separate control plane from data plane. The master handles metadata and coordination; chunkservers handle the actual bytes. This pattern — keep the "brain" out of the high-volume data path — recurs everywhere from Kubernetes to modern object stores.
Treat failure as the default state, not an edge case. Fast recovery, replication, and constant background repair aren't bolted on after the fact; they're central to the design from day one.
Relaxing a guarantee can be a legitimate design choice — but only if you push the resulting complexity to a place where it can be handled well (in this case, onto application-level conventions like checksums and idempotent writers).
A single coordinator isn't automatically a scalability bug. If you keep it out of the hot path and make it fast to recover, a centralized component can radically simplify a system without becoming its bottleneck.

How This Maps to AWS / Kubernetes / DevOps

If you work with modern cloud infrastructure, you've already used GFS's descendants, even if the vocabulary is different:

Amazon S3 mirrors the core GFS idea: a durable, replicated, distributed object store optimized for large sequential objects rather than small random-access files, with a metadata layer that's kept separate from the actual data path.
HDFS, the storage layer behind much of the original Hadoop ecosystem, is a near-direct open-source implementation of GFS's ideas: a NameNode (the master) and DataNodes (the chunkservers), large fixed-size blocks, and replication for fault tolerance.
Kubernetes' control plane vs. data plane split echoes the same architectural instinct: the API server and etcd (control plane) coordinate and store metadata/desired state, while the actual application traffic flows directly between pods, never through the control plane.
Kafka's partition replication and leader election resemble GFS's primary/lease mechanism — one replica is elected to order writes for a period, and others follow that order, with automatic re-election on failure.
DevOps takeaway: when you design internal tooling or services, ask the same question GFS's authors asked: what does our actual workload look like, and which "standard" guarantees can we safely relax to buy simplicity and throughput? That question — more than any specific technology — is the real lesson of this paper.

Next in this series: MapReduce (Dean & Ghemawat, 2004).

Source: Ghemawat, S., Gobioff, H., & Leung, S-T. "The Google File System." SOSP '03. Original PDF

What I’ve Learned Building a CI/CD Cloud Pipeline from Scratch

Atul Vishwakarma — Fri, 14 Aug 2026 02:30:00 +0000

When I first began my transition from a Full Stack Developer to focusing heavily on Cloud & DevOps, the learning curve felt like an absolute mountain.

Between managing the infrastructure for thousands of daily orders and squeezing in certification study time during my 3.5-hour daily commute on the Mumbai local trains, my overarching goal was simple: stop doing things manually.

Over the last few months, I have documented every step of this journey right here on Atul Codes. From writing my first Dockerfile to provisioning cloud infrastructure with code, it has been an incredible phase of growth.

Here is a look back at the biggest lessons I’ve learned while building a modern CI/CD cloud pipeline from scratch.

Phase 1: Escaping "It Works on My Machine"

The first major bottleneck I had to solve was environment inconsistency. Realizing that manual server configurations were slowing down our development cycles pushed me straight into containerization.

Learning Docker: I started by figuring out How to Dockerize a Node.js App. Packaging the runtime and the code together changed how I viewed deployments entirely.
The Power of Automation: But building containers manually wasn't enough. The real "aha!" moment came when I learned to Automate Docker Builds with GitHub Actions, turning code commits into instant, deployable images.
Embracing the Terminal: Of course, containerization comes with its own headaches. I spent hours debugging until I learned How to Fix the "Port Already in Use" Error, and I quickly realized that mastering the 5 Linux Bash Commands Every DevOps Engineer Uses Daily was non-negotiable for troubleshooting headless servers.

Phase 2: Killing "ClickOps" with Infrastructure as Code

Once the applications were containerized, I needed a place to run them. The AWS Management Console is great for learning, but clicking through UI menus to spin up servers is dangerous for production.

Entering the Terraform Ecosystem: I forced myself to learn HashiCorp Configuration Language (HCL). Writing the code to Provision an AWS EC2 Instance with Terraform was a massive milestone. It took away the fear of breaking the cloud because everything was version-controlled.
Building a Second Brain: As I studied deeper for my Terraform Associate certification, the syntax became overwhelming. To save time, I compiled my Ultimate Terraform Cheatsheet for DevOps Engineers—which has become my most used daily reference.
Optimizing the Workflow: To write all this HCL effectively, I had to upgrade my environment. Finding the Top 5 VS Code Extensions for DevOps and Cloud Engineering completely streamlined my development process.

Phase 3: The Reality of Production & Security

Building infrastructure is fun, but securing it is what actually keeps you employed.

Securing the Pipeline: Early on, I realized the immense danger of leaked keys. I immediately audited my workflows and documented How to Stop Hardcoding AWS Credentials using environment variables and AWS CLI profiles.
Protecting the Wallet: The cloud is cheap until you make a typo. Setting up AWS Billing Alerts gave me the peace of mind to experiment with new architectures without the fear of a $1,000 surprise bill.
Exploring the Future: With security locked down, I started exploring beyond traditional servers, successfully Deploying My First Serverless Function on AWS Lambda.

What’s Next for Atul Codes?

Writing these articles over the last quarter hasn't just been about sharing knowledge—it has been the ultimate forcing function for my own upskilling. When you have to explain a complex topic clearly to someone else, you uncover the gaps in your own understanding.

So, what is the next big goal?

I am currently diving deep into AWS ECS Fargate for scalable container orchestration, and preparing to knock out my HashiCorp Terraform Associate certification. You can expect a deep-dive series on both of those topics very soon.

Thank you to everyone who has read, commented, and shared my posts so far. Let’s keep building!

☕ If my tutorials have helped you on your own cloud journey, consider buying me a coffee to support the blog!

My Top 5 VS Code Extensions for DevOps and Cloud Engineering

Atul Vishwakarma — Fri, 07 Aug 2026 02:30:00 +0000

As a Cloud & DevOps Engineer, Visual Studio Code isn't just a text editor—it is my absolute command center.

Whether I am writing Infrastructure as Code (IaC), debugging a broken CI/CD pipeline, or managing containerized microservices, switching back and forth between a browser, a terminal, and an editor is a massive productivity killer. The right extensions bring all of that context directly into your workspace.

If you are setting up your development environment on a fresh Linux install or just looking to optimize your daily workflow, here are the top 5 VS Code extensions I use every single day at Atul Codes.

1. HashiCorp Terraform

If you are writing Terraform modules to provision cloud infrastructure, this extension is non-negotiable.

Writing HCL (HashiCorp Configuration Language) without syntax highlighting is painful. The official HashiCorp extension provides auto-completion for resource blocks, real-time syntax validation, and instantly formats your code when you save.

Why I love it: It highlights missing required variables before I even run terraform validate or terraform plan, saving me countless failed deployment attempts.

2. Docker

When you are containerizing applications, writing Dockerfiles and managing docker-compose.yaml files can get messy.

The official Docker extension by Microsoft makes it incredibly easy to build, manage, and deploy containerized applications right from VS Code. It provides syntax highlighting for your Dockerfiles and gives you a visual UI in the sidebar to see exactly which containers are running and which images are taking up space on your hard drive.

Why I love it: I can right-click any running container in the sidebar and select "View Logs" to instantly troubleshoot without typing out long docker logs <container-id> commands in the terminal.

3. YAML (by Red Hat)

In the DevOps world, you are going to write a lot of YAML. From GitHub Actions workflows to Kubernetes manifests, YAML is everywhere.

The YAML extension by Red Hat is a lifesaver. It provides comprehensive auto-completion, hover support, and strict validation.

Why I love it: YAML relies entirely on exact indentation. One misplaced space will break an entire GitHub Actions pipeline. This extension catches those formatting errors instantly, so you don't have to wait for your CI/CD pipeline to fail to realize you made a typo.

4. AWS Toolkit

Since I spend the vast majority of my time architecting on AWS, having the AWS Toolkit installed is a massive time-saver.

This extension lets you interact with AWS services like S3, Lambda, and CloudWatch directly from your editor. You can view your S3 buckets, invoke serverless functions locally, and even read CloudWatch logs without ever logging into the AWS Management Console.

Why I love it: It securely integrates with my local AWS CLI profiles, meaning I can safely switch between environments without risking deploying to the wrong account.

5. GitLens — Git supercharged

Git is the backbone of everything we do, but tracking down who changed a specific line of infrastructure code (and why) can take some digging.

GitLens supercharges the built-in Git capabilities of VS Code. It adds inline blame annotations—meaning you can click on any line of code and see exactly who wrote it, when they wrote it, and the commit message associated with it.

Why I love it: It makes team collaboration seamless. If a deployment script breaks, I instantly know which commit introduced the bug. (Note: GitLens has a fantastic free version, but if you manage massive repositories and need advanced visual commit graphs).

The Takeaway

Your code editor should work for you, not against you. By loading up VS Code with the right tools, you can catch errors earlier, type less boilerplate code, and stay focused on building resilient cloud infrastructure.

Did I miss your favorite DevOps extension? Let me know in the comments below!

☕ Did you discover a new favorite extension today? Buy me a coffee to help keep Atul Codes running!

The 5 Linux Bash Commands Every DevOps Engineer Uses Daily

Atul Vishwakarma — Fri, 31 Jul 2026 02:30:00 +0000

When you are first learning cloud engineering, you spend a lot of time clicking through beautiful web dashboards.

But what happens when your application crashes at 2:00 AM, the web server goes offline, and the only access you have is an SSH connection to a headless Linux box? You can't rely on a graphical user interface anymore. You need the command line.

Whether you are deploying microservices or debugging a massive production outage, the Linux terminal is where the real DevOps work happens. Here are the 5 Bash commands I use daily to troubleshoot broken servers.

1. `tail` (Watching Live Logs)

When an application fails, your first instinct should be to check the logs. But log files can be gigabytes in size. Opening a 2GB file in a text editor will crash your server entirely.

Instead, we use tail to look at just the very end of the file. By adding the -f (follow) flag, you can watch the logs stream in real-time.

# Watch the Nginx error log live
tail -f /var/log/nginx/error.log

Why it saves you: You can run tail -f, trigger your broken API endpoint in another window, and watch the exact error message pop up instantly on your screen.

2. `grep` (Finding the Needle in the Haystack)

If tail is for watching logs live, grep is for searching through them after the fact. grep allows you to search for specific text patterns inside massive files.

If a user complains that their payment failed, you don't want to read a million lines of logs. You just want to see the lines containing the word "ERROR" or "500".

# Find every line containing "500 Internal Server Error"
grep "500 Internal Server Error" /var/log/app/production.log

Pro Tip: Add the -i flag to make your search case-insensitive!

3. `awk` (Parsing Structured Data)

Sometimes grep gives you too much information. If you have a log file and you only want to extract a specific piece of data—like a list of IP addresses that are spamming your server—you need awk.

awk reads text line by line and splits it into columns (which it calls fields).

# Print ONLY the first column (e.g., the IP address) from an access log
awk '{print $1}' /var/log/nginx/access.log

Why it saves you: When paired with other commands, awk allows you to quickly isolate data so you can block malicious IPs or count exactly how many times a specific user ID failed to log in.

4. `chmod` (Fixing "Permission Denied")

The most common error you will face as a junior DevOps engineer is Permission denied.

This usually happens because the user running your application (like www-data or ubuntu) does not have the right to read a configuration file or execute a deployment script. chmod (change mode) fixes this by modifying the file permissions.

# Make a deployment script executable
chmod +x deploy.sh

Why it saves you: When your CI/CD pipeline suddenly fails because a script won't run, a quick chmod +x is usually the magic bullet.

5. `htop` (Monitoring System Resources)

Is your server crashing because of a memory leak in your code, or because the CPU is maxed out at 100%? You need to know immediately.

While top comes pre-installed on most systems, htop is the vastly superior, colorful, and interactive version. It provides a visual breakdown of your CPU cores, RAM usage, and exactly which processes are eating up your resources.

# Launch the interactive resource monitor
htop

Why it saves you: One glance at htop tells you if you need to quickly upgrade your server size (scale up) or if a rogue process needs to be killed immediately.

The Takeaway

You don't need to be a Linux kernel expert to be a great DevOps engineer, but you do need to be comfortable in the terminal. The next time you build a project, skip the UI. Force yourself to navigate the filesystem, check the logs, and manage your scripts entirely through Bash.

☕ Did this reference guide help you out? Buy me a coffee to help keep Atul Codes running!

Deploying Your First Serverless Function on AWS Lambda

Atul Vishwakarma — Fri, 24 Jul 2026 02:30:00 +0000

For a long time, deploying an application meant provisioning a server, installing an operating system, configuring a runtime environment, and paying for that server to run 24/7—even if nobody was visiting your site.

Then came Serverless.

Serverless doesn't mean there are no servers. It simply means you don't have to manage them. With AWS Lambda, you just write your code, upload it, and AWS handles the rest. It automatically scales based on traffic, and the best part? You only pay for the exact milliseconds your code is running.

(If you followed my previous tutorial on Setting Up AWS Billing Alerts, you already know how to keep your costs under control!)

Today, we are going to build your first serverless API endpoint using AWS Lambda and API Gateway. Let's dive in.

Step 1: Create the Lambda Function

Log into your AWS Management Console and search for Lambda in the top search bar.

Once you are on the Lambda dashboard, click the bright orange Create function button.

Configure your function with the following settings:

Author from scratch: Leave this selected.
Function name: MyFirstServerlessAPI
Runtime: Select Node.js 24.x (or the latest available Node.js version).
Architecture: x86_64

Scroll down to the bottom and click Create function.

Step 2: Write the Serverless Code

You will be redirected to your new function's dashboard. Scroll down to the Code source section. You will see an integrated code editor with a file named index.mjs.

AWS provides a basic "Hello from Lambda" template, but let's update it so it behaves like a real API endpoint that returns JSON data.

Replace the default code with this:

export const handler = async (event) => {
  // We can log the incoming request for debugging
  console.log("Incoming event: ", JSON.stringify(event, null, 2));

  // Create our JSON payload
  const responseBody = {
    message: "Success! You just built your first Serverless API.",
    timestamp: new Date().toISOString(),
    yourRole: "Cloud Engineer in the making"
  };

  // Return the HTTP response
  const response = {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify(responseBody),
  };

  return response;
};

After pasting the code, click the Deploy button just above the code editor. If you don't click deploy, AWS won't save your changes!

Step 3: Add API Gateway

Right now, your code exists in the cloud, but there is no way for the outside world to communicate with it. We need to attach a "trigger." In this case, we want an HTTP URL that triggers our code whenever someone visits it.

To do this, we use Amazon API Gateway.

Scroll up to the top of your function dashboard and look at the Function overview diagram. Click on the + Add trigger button.

Configure the trigger with these settings:

Select a source: Search for and select API Gateway.
Intent: Select Create a new API.
API type: Select HTTP API (this is cheaper and simpler than REST API).
Security: Select Open (this makes the endpoint public so we can test it in our browser).

Click Add.

Step 4: Test Your Live API Endpoint

AWS will take a few seconds to configure the API Gateway and attach it to your Lambda function.

Once it is done, you will see API Gateway listed in your Function overview. Scroll down to the Configuration tab, and click on Triggers.

You will see an API endpoint URL that looks something like this: https://2eb2yh0j28.execute-api.us-east-1.amazonaws.com/default/MyFirstServerlessAPI

Click that link!

It will open a new tab in your browser, and you should immediately see your formatted JSON response:

{
  "message": "Success! You just built your first Serverless API.",
  "timestamp": "2026-06-09T14:18:40.430Z",
  "yourRole": "Cloud Engineer in the making"
}

The Takeaway

Congratulations! You just deployed a scalable, highly available microservice.

Think about what you didn't have to do today: You didn't configure a VPC, you didn't open ports in a security group, and you didn't install Node.js on a Linux server. AWS handled the entire underlying infrastructure, allowing you to focus entirely on writing business logic.

That is the true power of Serverless.

If this tutorial helped you deploy your first Lambda function, consider ☕ buying me a coffee to support my work on Atul Codes!

How to Set Up AWS Billing Alerts (Before You Get a $1,000 Bill)

Atul Vishwakarma — Fri, 17 Jul 2026 02:30:00 +0000

If you talk to anyone who is just starting to learn Cloud Engineering, you will quickly find out that their biggest fear isn't failing to understand VPCs or IAM roles.

Their biggest fear is waking up to a $1,000 surprise bill because they left an EC2 instance running over the weekend.

Last week, we talked about how to prevent hackers from running up your bill by securing your credentials. Today, we are going to protect your wallet from your own mistakes.

I am going to show you exactly how to set up an AWS Budget that automatically emails you the second your monthly AWS spend crosses $5.00. Do this before you spin up another server.

Step 1: Open the AWS Billing Dashboard

Log in to your AWS Management Console. In the top right corner, click on your account name and select Billing and Cost Management from the dropdown menu.

Step 2: Create a New Budget

On the left-hand navigation pane, look under the "Budgets and Planning" section and click on Budgets. Then, click the bright orange Create budget button.

AWS gives you two options here: a simplified template or an advanced setup. We are going to use the custom route to ensure we get exactly what we want.

Select Customize (advanced).
Choose Cost budget as your budget type.
Click Next.

Step 3: Set Your $5 Threshold

Now we need to tell AWS how much we expect to spend.

Budget name: Give it a clear name, like My-5-Dollar-Limit.
Period: Monthly.
Budget effective dates: Recurring budget.
Budgeting method: Fixed.
Budget amount: Enter 5.00.

This tells AWS that your absolute ceiling for the month is 5 bucks.

Step 4: Configure the Alert (The Most Important Step)

Creating a budget is useless if AWS doesn't actually notify you when you cross it. We need to set an alert.

Scroll down and click Add an alert threshold.

Threshold: Enter 100% of actual amount. (This means AWS will trigger the alert the moment your bill hits $5.00).
Notification preferences: Enter your primary email address in the "Email recipients" box.

Pro Tip: You can actually set multiple alerts! I like to set one at 50% ($2.50) and one at 100% ($5.00) just to have an early warning.

Step 5: Review and Create

Click Next through the remaining optional screens (you do not need to attach any specific Actions for this basic alert).

Review your settings on the final page, and hit Create budget.

The Takeaway

That's it! You now have a financial safety net.

If you ever accidentally leave a database running, or if you provision a resource that falls outside of the AWS Free Tier, you will get an email alert before the costs spiral out of control.

Building hands-on projects is the only way to master AWS, and you should not let the fear of a surprise bill hold you back. Now that your account is secure, get back to building!

If this tutorial gave you some peace of mind today, you can ☕ buy me a coffee here to support my work!

Stop Hardcoding AWS Credentials: How to Manage Secrets Securely

Atul Vishwakarma — Fri, 10 Jul 2026 02:30:00 +0000

It is a cloud engineer's worst nightmare.

You finish a long coding session, type git push origin main, and go to sleep. You wake up the next morning to hundreds of automated emails from AWS and a billing dashboard showing a $5,000 charge.

What happened? You accidentally pushed your AWS Access Key and Secret Key to a public GitHub repository. Within seconds, automated bots scraped your keys and spun up dozens of expensive EC2 instances across the globe to mine cryptocurrency.

Today, we are going to make sure this never happens to you. Here is how to stop hardcoding your AWS credentials and manage your secrets securely.

The Bad Way: Hardcoding in Your App

When you are just learning how to use the AWS SDK (like boto3 for Python or the AWS SDK for Node.js), it is incredibly tempting to just paste your keys directly into your code so it works:

// ❌ NEVER DO THIS!
const AWS = require('aws-sdk');

const s3 = new AWS.S3({
    accessKeyId: "AKIAIOSFODNN7EXAMPLE",
    secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
});

If this code gets pushed to GitHub, your AWS account is completely compromised. We need to separate our code from our configuration.

The Good Way (Local Development): AWS CLI Profiles

Instead of putting credentials in your code, you should store them securely on your local machine using the AWS Command Line Interface (CLI).

If you have the AWS CLI installed, open your terminal and run:

aws configure

It will prompt you to enter your Access Key, Secret Key, default region (e.g., us-east-1), and output format.

Once you do this, AWS creates a hidden file on your machine. If you are developing on a Linux system (like Fedora) or macOS, this file lives at ~/.aws/credentials. On Windows, it is at C:\Users\YOUR_USERNAME\.aws\credentials.

Now, when you run your code, the AWS SDK will automatically look for this hidden file and use the credentials inside it. Your code becomes completely clean:

// ✅ THE SAFE WAY
const AWS = require('aws-sdk');

// The SDK automatically pulls from your ~/.aws/credentials file!
const s3 = new AWS.S3();

The Good Way (Pipelines & CI/CD): Environment Variables

If you are running your code inside a Docker container or a GitHub Actions pipeline, you can't manually run aws configure. Instead, you use Environment Variables.

AWS SDKs are programmed to automatically look for these specific variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

If you are working locally with a .env file, it should look like this:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_DEFAULT_REGION=us-east-1

Crucial Step: You must add .env to your .gitignore file. If you track your .env file in Git, you are right back to the nightmare scenario we talked about at the beginning!

The Ultimate Fix (Production): AWS Secrets Manager

The methods above are perfect for personal projects, local development, and simple CI/CD pipelines.

However, when you are dealing with production microservices at scale, managing dozens of .env files becomes a massive security liability. In a true production environment, the best practice is to migrate credential management to AWS Secrets Manager.

By using Secrets Manager, you eliminate hardcoded secrets entirely, automate key rotation, and establish SOC2-aligned auditability so you know exactly which microservice accessed which secret and when.

The Takeaway

Security in the cloud isn't just an afterthought; it is step zero. By utilizing aws configure locally and environment variables in your pipelines, you can push your code to GitHub with complete peace of mind.

(If you are using these secured credentials to write infrastructure code, be sure to bookmark my Ultimate Terraform Cheatsheet to speed up your workflow!)

If this security tip helped you out today, you can ☕ buy me a coffee here to support my work!

Infrastructure as Code 101: Provisioning an AWS EC2 Instance with Terraform

Atul Vishwakarma — Thu, 02 Jul 2026 12:30:00 +0000

If you are currently managing your AWS infrastructure by logging into the console, clicking through menus, checking boxes, and hitting "Create," you are practicing what DevOps engineers jokingly call "ClickOps."

While clicking around is fine for learning, it is a nightmare for production. Manual clicks are impossible to track, prone to human error, and incredibly slow to replicate. If your server goes down, you have to remember exactly which boxes you checked six months ago to rebuild it.

The solution is Infrastructure as Code (IaC).

Today, we are diving into Terraform, the industry standard for IaC. I am going to show you how to stop clicking and start coding by provisioning a basic Linux AWS EC2 instance using just a few lines of code.

What is Terraform?

Created by HashiCorp, Terraform is a tool that allows you to define both cloud and on-premise resources in human-readable configuration files. It uses a declarative language called HCL (HashiCorp Configuration Language).

"Declarative" means you just tell Terraform what you want (e.g., "I want one t2.micro EC2 instance"), and Terraform figures out how to communicate with AWS APIs to make it happen.

Prerequisites

Before we write our code, you need three things:

An AWS Account (the Free Tier is perfect).
The AWS CLI installed and configured on your machine with your IAM credentials (so Terraform has permission to talk to your account).
Terraform installed on your local machine.

Step 1: Define the Provider

Terraform can manage infrastructure on AWS, Azure, Google Cloud, and even GitHub! To tell Terraform we want to talk specifically to AWS, we need to configure a "Provider."

Create a new folder on your computer, open it in your terminal, and create a file called main.tf.

Add this code:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1" # Change this to your preferred region
}

Step 2: Write the EC2 Resource Block

Now, we tell Terraform exactly what resource we want to build. Below your provider block in main.tf, add the following code:

resource "aws_instance" "my_first_server" {
  ami           = "ami-0c7217cdde317cfec" # Ubuntu 22.04 LTS in us-east-1
  instance_type = "t2.micro"

  tags = {
    Name = "AtulCodes-Terraform-Server"
  }
}

Breaking down the code:

resource: Tells Terraform we are creating a new piece of infrastructure.
"aws_instance": The specific type of resource (an EC2 instance).
"my_first_server": This is just a local name we use to reference this server within our Terraform code. AWS never sees this name.
ami: The Amazon Machine Image ID. Note: AMIs are region-specific! The one above is for Ubuntu in us-east-1. If you are in ap-south-1 (Mumbai), you will need to grab a different AMI ID from the AWS console.
instance_type: We are using t2.micro because it qualifies for the AWS Free Tier.
tags: This is how we actually name the server inside the AWS Console.

Step 3: The Core Terraform Workflow

Terraform relies on three magical commands to get your infrastructure into the cloud: init, plan, and apply.

1. Initialize the Directory

Open your terminal in your project folder and run:

terraform init

This command looks at your main.tf file, sees you want to use AWS, and downloads the necessary AWS plugins in the background.

2. Plan the Deployment

Next, we want to see what Terraform is going to do before it actually does it.

terraform plan

You will get a large output showing exactly what resources will be created. Look for the green + signs. At the bottom, it should say: Plan: 1 to add, 0 to change, 0 to destroy.

3. Apply the Code

If the plan looks good, it is time to deploy!

terraform apply

Terraform will show you the plan one more time and ask you to type yes to confirm. Type yes and hit enter. Within a few seconds, Terraform will communicate with AWS and spin up your server!

Step 4: Verify in the AWS Console

Don't just take my word for it. Log into your AWS Management Console, navigate to the EC2 Dashboard, and click on Instances.

You will see a brand-new server named AtulCodes-Terraform-Server sitting right there, in a "Running" state. You provisioned that without clicking a single button in the UI!

Step 5: Destroy the Evidence (Crucial!)

One of the best features of IaC is how easy it is to clean up. If you leave this server running, AWS might eventually charge you for it.

To tear down everything we just built, run:

terraform destroy

Type yes when prompted. Terraform will gracefully delete the EC2 instance it created.

The Takeaway

You have officially taken your first step into Infrastructure as Code. By writing your infrastructure as code, you can version control it in Git, review it in Pull Requests, and ensure your staging environments are exact replicas of your production environments.

Next week, I am dropping my Ultimate Terraform Cheatsheet! It will cover all the essential commands, state management, and variable syntax you need for your daily DevOps workflows. Make sure you hit the subscribe button so you don't miss it.

If this tutorial helped you deploy your first Terraform resource, you can ☕ buy me a coffee here to support my work!

TIL: How to Fix the "Port Already in Use" Error in Docker

Atul Vishwakarma — Tue, 30 Jun 2026 03:30:00 +0000

We have all been there. You just built a fresh Docker image, you confidently type your docker run command into the terminal, and BAM. You are hit with a massive wall of red text that looks something like this:

Error starting userland proxy: listen tcp4 0.0.0.0:3000: bind: address already in use.

This error (often referred to as EADDRINUSE) simply means that something else on your computer is already using the port you are trying to assign to your Docker container.

Today I Learned (TIL) the fastest way to find that rogue process and terminate it so you can get back to coding. Here is the two-step fix for Mac and Linux users.

Step 1: Find the Rogue Process

To find out exactly which application is hoarding your port, we will use the lsof (List Open Files) command.

Open your terminal and type this command, replacing 3000 with whatever port is giving you trouble:

lsof -i :3000

This will return a list of processes using that port. Look under the PID (Process ID) column. You need that number.

Step 2: Kill the Process

Now that we have the Process ID, we can force it to shut down using the kill command. Pass the -9 flag to ensure it terminates immediately, followed by the PID you found in Step 1:

kill -9 14235

That’s it! Your port is now free. You can run your docker run -p 3000:3000 my-app command again, and it will spin up perfectly.

Bonus: What if it's a "Ghost" Docker Container?

Sometimes, the port is being blocked because you started a Docker container earlier, forgot about it, and left it running in the background.

If lsof doesn't solve your problem, quickly check your active Docker containers:

docker ps

Find the container using your port, grab its Container ID, and stop it gracefully:

docker stop <CONTAINER_ID>

📚 Catch up on this Docker Series:

Part 1: How to Dockerize a Node.js App in 5 Easy Steps
Part 2: Automate Your Docker Builds with GitHub Actions

If this quick tip saved you some debugging time today, you can ☕ buy me a coffee here to support my work!

TerraGuard AI: Adding an AI Brain to Terraform Drift Detection — Design Decisions, Trade-offs, and Lessons Learned

Atul Vishwakarma — Mon, 29 Jun 2026 05:30:00 +0000

Repo: https://github.com/vatul16/TerraGuard-AI — full source, Terraform infra, drift detection engine, and CI/CD workflows.

When I finished TerraTier, the 3-tier AWS architecture project I wrote about last time, I wanted to build something that pushed in a different direction. TerraTier was about the infrastructure layer — networking, security boundaries, secrets management. This project is about the operational layer: what happens after the infrastructure is running, when someone inevitably makes a change outside of Terraform.

The result is TerraGuard AI — an event-driven drift detection system that catches manual AWS console changes, classifies their risk using an LLM, and routes context-rich alerts intelligently instead of just firing a generic notification every time a description field changes. A Node.js API on ECS Fargate, a PostgreSQL database on RDS, and an ALB are the infrastructure being monitored. GitHub Actions, Python, and Groq's API for Llama 3.3 are how the detection and analysis work.

This article is about the decisions behind why it works the way it does, and what I'd change.

The problem with standard drift detection

The textbook implementation of Terraform drift detection is: run terraform plan -detailed-exitcode on a cron job, check the exit code, send an alert if it's non-zero. This works, and it takes maybe 30 lines of shell script to implement. The problem isn't the detection — it's what happens to the humans on the other end of the alert.

In any environment where more than one person touches the AWS console, drift is nearly constant. Someone tags a security group directly in the console, or updates a description, or changes a retention period on a log group that wasn't in Terraform yet. Each of these fires the exact same alert as "port 22 opened to 0.0.0.0/0 on a production security group." After the third or fourth time the on-call engineer wakes up to an alert and finds a tag change, they start doing what humans do with noisy signals: they ignore them.

Alert fatigue is a well-documented problem in SRE literature, but most drift detection tooling solves it by giving you more configuration options — exception lists, ignore patterns, threshold tuning. That's fighting symptoms. The actual cause is that the alerting system has no concept of severity. It knows something changed; it doesn't know what that means.

The insight that shaped this project is that detection and classification are two separate problems, and they need different tools. Terraform is genuinely excellent at detection — it has the full resource schema, it talks directly to the AWS APIs, and its plan output is deterministic and reliable. It has no idea what a change means from a security or business perspective. That's a reasoning problem, and it's exactly what language models are good at.

So TerraGuard AI uses Terraform to detect drift, and an LLM to classify it. The two tools do the parts they're each actually suited for.

Decision 1: AI for classification, not detection

This distinction is worth dwelling on, because it's easy to imagine AI being used to find drift — some vector-similarity approach comparing current state to desired state, or something. That would be using AI to badly replicate something Terraform already does perfectly.

The LLM layer in this project only ever sees the parsed output of terraform plan. Drift is already confirmed by the time the AI gets involved. What the AI does is answer a different set of questions: Is this change a security risk or a configuration nuance? Should it be reverted or adopted into the Terraform codebase? What is the exact command someone should run to remediate it, right now, without having to look anything up?

The prompt asks Groq's Llama 3.3 to act as a Senior Cloud Security Engineer and return a structured JSON response with a risk level (CRITICAL / HIGH / MEDIUM / LOW / INFO), a score out of 10, an impact summary, a recommended action (REVERT / ADOPT / INVESTIGATE / MONITOR), and a concrete remediation command. The temperature is set to 0.1 — not 0, because zero-temperature responses can be oddly mechanical, but close enough to zero that the analysis is consistent and factual rather than creative.

{
  "risk_level": "CRITICAL",
  "risk_score": 9,
  "category": "Security",
  "summary": "Security group ingress rule added exposing SSH port 22 to all internet traffic.",
  "impact": "Any host on the internet can attempt SSH brute-force or exploitation against ECS tasks.",
  "action": "REVERT",
  "remediation": "terraform apply -target=aws_security_group.ecs_tasks",
  "reasoning": "Opening port 22 to 0.0.0.0/0 is a critical security misconfiguration..."
}

Based on that response, the routing is simple: CRITICAL or HIGH goes to Slack immediately, plus a GitHub Issue for the audit trail. MEDIUM, LOW, or INFO creates a GitHub Issue that goes into the backlog — no Slack notification, no on-call page.

The practical effect is that the Slack channel only gets messages that actually need a human to look at something now. The rest creates a self-organizing backlog of minor drift that can be handled during normal working hours. That's not a feature — it's the entire point.

Decision 2: Groq over AWS Bedrock

The obvious choice for an AWS-native project would be Amazon Bedrock. The LLM call would live inside the VPC, authentication would use the same IAM role the rest of the infrastructure uses, and there would be no external API dependency.

I chose Groq instead, for two reasons that turned out to matter more than I expected.

The first is speed. Groq's custom LPU hardware runs Llama 3.3 70B at a genuinely unusual pace — we're talking single-digit seconds for a complete risk analysis. Bedrock inference at a comparable model size is slower, and when drift detection runs in a GitHub Actions workflow with a 6-hour cron cadence, latency matters less than it would in a real-time system. But for local development and testing, where you're running the detector manually and waiting for output, the speed difference makes iteration significantly less frustrating.

The second reason is cost. Groq's free tier is 14,400 requests per day, which is more than enough for a drift detection workflow that runs every 6 hours and only calls the API when drift is actually found. Bedrock charges per-token with no free tier. For a personal portfolio project that also needs to pay for ECS, RDS, NAT Gateway, and an ALB, not paying for inference is a meaningful constraint.

The trade-off is the external dependency. The drift detector now requires an outbound HTTPS connection from the GitHub Actions runner to api.groq.com, and if Groq is down or rate-limiting, the analysis step fails. I handled this with a fallback in the Python script: if the Groq call fails or returns unparseable JSON, the detector falls back to a MEDIUM risk assessment with a manual review recommendation rather than crashing entirely. The drift is still logged and acted on; it just lacks the AI analysis.

Decision 3: GitHub Actions over Lambda

The architecturally "purer" approach to scheduled drift detection is a Lambda function triggered by EventBridge on a cron schedule, with an additional EventBridge rule listening to CloudTrail events for real-time detection when a console change happens. That's closer to the enterprise architecture in the project's original design.

I built it on GitHub Actions instead, and I think that was the right call for this stage.

The practical argument: GitHub Actions is free, already configured for the project, and doesn't require provisioning, IAM-scoping, or maintaining a separate compute resource. A Lambda function adds at least three more Terraform resources to manage, an ECR image or deployment package to maintain, and CloudWatch Logs to check when something goes wrong — all of which is real work that doesn't directly demonstrate anything new. The drift detection workflow in GitHub Actions does the same job with less infrastructure surface area and a simpler debugging story (Actions tab in GitHub, logs right there).

The conceptual argument: the drift detector is not a latency-sensitive workload. A 6-hour detection window is fine for a portfolio project, and even for many real production environments, catching a security group change within 6 hours is a meaningful improvement over catching it when someone manually reviews the AWS console. Real-time detection via CloudTrail + EventBridge is in the project roadmap, but it's a layer added on top of a working baseline, not a prerequisite.

Where this decision has a genuine cost: the GitHub Actions runner needs AWS credentials. That means AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as repository secrets — long-lived credentials stored in GitHub, which is a less secure pattern than OIDC role assumption. The Terraform IAM code in the project already scaffolds the OIDC setup (aws_iam_openid_connect_provider, aws_iam_role with a GitHub OIDC trust policy), and migrating to keyless authentication is the highest-priority item on the roadmap. I built with access keys first to get the workflow working without debugging OIDC trust policies at the same time, which is a trade-off I'd make again in the same situation.

Decision 4: Minimal infrastructure being monitored

A reasonable design question for a drift detector is: what infrastructure should it monitor? You could make the case that the more complex the infrastructure, the more impressive the demo — VPCs with 20 resources, multiple environments, Auto Scaling Groups, the works.

I went the opposite direction. The monitored infrastructure is as simple as it can be while still being realistic: one ECS Fargate service, one RDS PostgreSQL database, one ALB. That's a pattern that appears in real production environments constantly — it's the architecture for a backend API behind a load balancer with persistent storage. Nothing exotic.

The reason for keeping it simple is that the interesting thing in this project is the drift detection and classification layer, not the infrastructure. A 40-resource VPC with multiple tiers and Auto Scaling Groups would take most of the explanation budget to describe, and would make the CI/CD and drift detection workflows harder to follow for someone reading the code. By keeping the infrastructure simple and well-understood, someone reading the repository can quickly get past "what does this infrastructure do" and get to "how does the drift detection actually work" — which is the part worth explaining.

Decision 5: Separating infrastructure and application deployment

The most practically important single line in the entire ecs.tf file is this:

lifecycle {
  ignore_changes = [task_definition]
}

Without this, terraform apply and the deploy-app.yml GitHub Actions workflow would fight each other. The deploy workflow updates the ECS task definition to use a new ECR image URI (tagged with the git SHA) every time new code is pushed to app/**. If Terraform's ECS service resource didn't ignore task definition changes, the next terraform apply — triggered by the drift detector or by a manual infra change — would overwrite the CI/CD-deployed image with whatever image URI is in the Terraform variables. Deploying a new version of the app would immediately cause "drift" against the Terraform-managed state.

The ignore_changes lifecycle setting tells Terraform: "I know this attribute might differ from what's in the config, and that's intentional — CI/CD owns it, not you." This is the standard pattern for separating infrastructure management (Terraform's job) from application deployment (CI/CD's job) on ECS, and it's one of those things that sounds obvious in retrospect but takes a debugging session to discover if you haven't seen it before.

The deploy workflow itself follows a pattern worth understanding: it doesn't just push an image and call aws ecs update-service --force-new-deployment. It downloads the current live task definition from AWS, replaces only the container image URI, registers the modified definition as a new revision, and then updates the service to use that revision. The reason for this sequence is that the task definition contains Secrets Manager ARNs, IAM role ARNs, environment variable sets, and health check configuration that are all managed by Terraform. Overwriting the entire task definition from a CI/CD workflow would blow away those fields. Downloading the live definition and patching only the image preserves everything else.

Decision 6: Secrets Manager with ECS secrets references, not environment variables

The Node.js application gets its database connection details through a combination of plain environment variables and a secrets block in the ECS task definition.

Non-sensitive values — DB_HOST, DB_PORT, DB_NAME, DB_USER, NODE_ENV, PORT — are passed as plain environment variables. They're not sensitive: knowing the RDS hostname doesn't help an attacker get into the database.

The password is different. The ECS secrets block references the Secrets Manager secret by ARN, and at container startup, the ECS agent fetches the value, injects it as DB_PASSWORD in the container's environment, and never writes it anywhere that shows up in the task definition JSON, CloudWatch logs, or the AWS console. The only evidence it existed is the ARN reference.

secrets = [
  {
    name      = "DB_PASSWORD"
    valueFrom = aws_secretsmanager_secret.db_password.arn
  }
]

This also required a specific IAM policy attached to the ECS execution role (not the task role — the agent that starts containers, not the container itself):

Action   = ["secretsmanager:GetSecretValue"]
Resource = aws_secretsmanager_secret.db_password.arn

The scoping matters. The policy grants access to exactly one secret ARN, not "Resource": "*". If an attacker found a way to escalate privileges from inside the container to the execution role, they could read one specific secret with a known ARN, not any secret in the account. That's a meaningful reduction in blast radius for a one-line change.

A race condition I hit and what it taught me

When I added the Node.js app port (3000) to the Terraform configuration — replacing the placeholder 80 that was there from the initial setup — running terraform apply produced an error I hadn't seen before:

Error: deleting ELBv2 Target Group: ResourceInUse: Target group is currently in use by a listener or a rule

What happened: changing the port on an ALB target group forces replacement (Terraform creates a new target group, then tries to delete the old one). But the ALB listener still referenced the old target group when Terraform tried to delete it. The listener update and the target group deletion were running in the wrong order — or more precisely, in parallel, and the deletion raced against the listener update and lost.

The fix was simply running terraform apply a second time. The first apply created the new target group and partially updated the listener, but the listener change hadn't completed by the time the old target group deletion ran. The second apply found the listener already pointing to the new target group and successfully deleted the old one.

I mention this because it's the kind of error that's alarming-looking and turns out to be completely benign — and knowing that pattern ("Terraform apply failed with ResourceInUse, run it again") is one of those things that takes a long time to learn from documentation and about five seconds to learn from experience. If you hit it: apply again.

The deeper lesson is about Terraform's dependency graph. Terraform builds an explicit dependency graph from the depends_on and implicit references in your configuration, and parallelizes everything it can within that graph. When you're modifying a resource that has something depending on it and you're replacing the resource rather than updating it in-place, the order of operations for the replacement can be non-obvious. Looking at the terraform plan output more carefully — specifically the -/+ indicators for "destroy and create replacement" — before applying would have flagged this in advance.

What I'd build next

The most significant gap in the current design is real-time detection. Running terraform plan every 6 hours means there's a window of up to 6 hours between a manual console change and the drift detection alert. For most configuration changes that's probably fine. For a security group rule opening port 22 to the internet, it's not.

The architecture for closing that gap exists: an EventBridge rule that listens to CloudTrail events for specific API calls (AuthorizeSecurityGroupIngress, ModifyDBInstance, AttachRolePolicy, and similar high-risk mutations), triggering a Lambda function that runs the plan-and-classify pipeline on demand, within seconds of the change. That's the enterprise-grade pattern that the project's original design was based on, and it's what I'd add in v2.

OIDC authentication for GitHub Actions is the other immediate priority. Storing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as repository secrets is functional but carries unnecessary risk — those credentials persist until they're rotated, and rotating them requires remembering they exist. The OIDC setup in iam.tf is already scaffolded; it's a workflow change and a secret deletion away from being the default.

A drift history store is something I've sketched but not built. Right now, every drift detection run is stateless — the result goes to Slack or GitHub Issues, and that's the end of it. If you wanted to answer "how often does our infrastructure drift, in which direction, and is it getting better or worse over time?", there's nowhere to look. A DynamoDB table storing detection results per run, with a simple dashboard, would turn TerraGuard AI from a reactive alerting tool into something with genuine observability over infrastructure compliance trends.

Closing thoughts

The most useful framing I found for this project is that it's not an AI project with some infrastructure bolted on, and it's not an infrastructure project with some Python bolted on. It's an operations automation project that happens to use AI for the part where you'd otherwise need a human to make a judgment call.

That framing matters because it determines what you actually have to understand to build it well. The Groq integration is a dozen lines of Python — the interesting engineering is in understanding when to call it, with what inputs, and what to do with the output, which requires thinking about the infrastructure, the CI/CD pipeline, the IAM model, and the alerting routing as a system. None of those are complicated individually. The interesting work is making them compose.

The full source is at https://github.com/vatul16/terraguard-ai — Terraform, Python detector, GitHub Actions workflows, and the Node.js app. I'm actively looking for Cloud/DevOps Engineer roles as I transition from full-stack development, and I'd genuinely enjoy talking through any part of this with someone who has questions about a specific decision. You can find me on LinkedIn.

Previous project: TerraTier — Production-grade 3-tier AWS architecture with Terraform, Auto Scaling Groups, two ALBs, four subnet tiers, and SSM Session Manager.

Automate Your Docker Builds with GitHub Actions

Atul Vishwakarma — Thu, 25 Jun 2026 12:30:00 +0000

Over the last two weeks, we solved two massive DevOps problems. First, we stopped deploying manually by building a CI/CD pipeline with GitHub Actions. Then, we solved the "it works on my machine" problem by containerizing a Node.js app with Docker.

Today, we are going to combine these two superpowers.

Running docker build manually on your laptop every time you change a line of code is tedious. Instead, we are going to configure GitHub Actions to automatically build your Docker image and push it to Docker Hub the second you push your code to the main branch.

Let's automate it.

Prerequisites

A free Docker Hub account.
A basic Node.js application with a Dockerfile (If you don't have one, follow my 5-step Docker tutorial here).
A GitHub repository with an active Actions workflow (Learn how to set one up in 10 minutes here).

Step 1: Create a Docker Hub Access Token

To let GitHub securely push images to your Docker Hub account, we need to give it a "password" in the form of an Access Token.

Log in to Docker Hub.
Click your profile picture in the top right and select Account Settings.
Click on Personal access tokens then Generate new token.
Name it github-actions-token, give it "Read & Write" permissions, and click Generate.
Copy this token immediately (you won’t be able to see it again).

Step 2: Add Your Secrets to GitHub

Now, we need to hand that token over to GitHub safely. Never hardcode passwords in your code!

Go to your repository on GitHub.
Click the Settings tab at the top.
On the left sidebar, scroll down to Secrets and variables and click Actions.
Click the green New repository secret button.

Create two distinct secrets:

Name: DOCKER_USERNAME | Secret: Your actual Docker Hub username.
Name: DOCKER_PASSWORD | Secret: The Access Token you copied in Step 1.

Step 3: Update Your GitHub Actions Workflow

Open up the .github/workflows/ci.yml file we created a couple of weeks ago. We are going to add a new job to our pipeline.

Replace your existing file contents with this updated YAML code:

name: Node.js CI/CD and Docker Build

on:
  push:
    branches: [ "main" ]

jobs:
  build-and-push-docker-image:
    runs-on: ubuntu-latest

    steps:
    # Step A: Check out the repository code
    - name: Checkout code
      uses: actions/checkout@v3

    # Step B: Log in to Docker Hub using our secure secrets
    - name: Log in to Docker Hub
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKER_PASSWORD }}

    # Step C: Build the Docker image and push it to Docker Hub
    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        # Replace 'yourusername' with your actual Docker Hub username
        tags: yourusername/my-node-app:latest

Make sure to change yourusername on the very last line to your actual Docker Hub username!

Step 4: Push and Watch the Automation

Save your file, commit the changes, and push it up to GitHub.

git add .github/workflows/ci.yml
git commit -m "feat: add docker build and push automation"
git push origin main

Head over to the Actions tab in your GitHub repository. You will see your new workflow spinning up. GitHub is now renting a tiny Ubuntu server, checking out your code, logging into Docker Hub, building your image based on your Dockerfile, and pushing the final product to the cloud.

Once you get the green checkmark, go check your Docker Hub profile. Your brand-new image will be sitting right there, ready to be pulled by any server in the world.

The Takeaway

You have just built a professional-grade continuous delivery pipeline.

From now on, all you have to do is write your code and type git push. The robots handle the rest. This is the exact workflow used by top tech companies, and you just implemented it for free.

Next week, we are going to look at a quick debugging trick: how to fix the dreaded "Port Already in Use" error when working with Docker locally.

If you are enjoying this DevOps series, you can ☕ buy me a coffee here to support my work.

How to Dockerize a Node.js App in 5 Easy Steps

Atul Vishwakarma — Tue, 23 Jun 2026 12:30:00 +0000

"But it works on my machine!" If you are a developer, you have likely said this exact phrase. Your code runs perfectly on your laptop, but the moment you hand it to a coworker or try to deploy it to a production server, it crashes. The Node version is wrong, a dependency is missing, or the environment variables are mismatched.

This is exactly the problem Docker solves.

Docker allows you to package your application and all of its dependencies into a single, standardized unit. Today, I am going to show you how to containerize a simple Node.js application in 5 easy steps so it runs perfectly anywhere, every single time.

Before we write the code, let's clear up the biggest point of confusion for beginners.

The Core Concept: Image vs. Container

People often use these terms interchangeably, but they are completely different things:

The Docker Image: This is the recipe. It is a static, read-only file that contains your source code, libraries, dependencies, and tools. An image does not "run."
The Docker Container: This is the cake. It is the running, active instance of your Docker Image. You can start, stop, and delete a container.

You build an Image once, and you can spin up a hundred Containers from it.

Prerequisites

Docker Desktop installed on your machine.
A basic understanding of the terminal.
Node.js installed locally (just for our initial setup).

Step 1: Create a Simple Node.js App

We need something to containerize. Let's build a dead-simple Express server. Open your terminal, create a new folder, and initialize a project:

mkdir my-docker-app
cd my-docker-app
npm init -y
npm install express

Next, create a file named server.js and paste in this code:

const express = require('express');
const app = express();
const PORT = 3000;

app.get('/', (req, res) => {
  res.send('Hello from inside a Docker Container! 🐳');
});

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Finally, open your package.json file and add a start script:

"scripts": {
  "start": "node server.js"
}

Step 2: Write the Dockerfile

The Dockerfile is a text document that contains all the commands needed to assemble your Image. In the root of your project folder, create a new file named exactly Dockerfile (no file extension).

Add the following instructions:

# 1. Specify the base image
FROM node:18-alpine

# 2. Set the working directory inside the container
WORKDIR /usr/src/app

# 3. Copy package.json and package-lock.json first
COPY package*.json ./

# 4. Install dependencies
RUN npm install

# 5. Copy the rest of your application code
COPY . .

# 6. Expose the port your app runs on
EXPOSE 3000

# 7. Define the command to run your app
CMD ["npm", "start"]

Why do we copy the package.json file before the rest of the code? Docker builds images in layers and caches them. If you change a single line in server.js, Docker doesn't need to re-install all your npm packages. It uses the cached layer for the dependencies, making your builds incredibly fast.

Step 3: Add a .dockerignore File

You wouldn't push your node_modules folder to GitHub, and you shouldn't copy it into your Docker image either. Your container should install its own clean dependencies.

Create a file named .dockerignore in your root directory:

node_modules
npm-debug.log
.git

Step 4: Build the Docker Image

Now it is time to bake the recipe. Open your terminal in the root of your project and run this command:

docker build -t my-node-app .

The -t flag "tags" your image with a readable name (my-node-app).
The . at the very end tells Docker to look for the Dockerfile in the current directory. Do not forget the period!

Step 5: Run the Docker Container

Your image is built. Now, let's bring it to life by spinning up a container.

docker run -p 3000:3000 -d my-node-app

The -p 3000:3000 flag maps port 3000 on your physical laptop to port 3000 inside the isolated container.
The -d flag runs the container in "detached" mode so it runs in the background, keeping your terminal usable.

Open your web browser and navigate to http://localhost:3000.

The Takeaway

Congratulations! You just built and ran your first Dockerized application.

Because the environment is bundled directly into the image, you could send this exact my-node-app image to a colleague running Windows, a friend on a Mac, or a Linux server on AWS, and it will run flawlessly. No configuration required.

Next week, we are going to take this a step further and combine it with the CI/CD pipeline we built last time. I will show you how to automatically build this Docker image using GitHub Actions every time you push code.

Make sure to subscribe to the blog so you don't miss it!

If this tutorial helped you finally understand Docker, you can ☕ buy me a coffee here to support my work!

DEV Community: Atul Vishwakarma

The Google File System Explained: How Google Built Storage That Expects to Fail

Problem Statement

Why the Older Approach Struggled

Core Architecture

Key Trade-offs

Failure Cases or Limitations

What Modern Engineers Can Learn

How This Maps to AWS / Kubernetes / DevOps

What I’ve Learned Building a CI/CD Cloud Pipeline from Scratch

Phase 1: Escaping "It Works on My Machine"

Phase 2: Killing "ClickOps" with Infrastructure as Code

Phase 3: The Reality of Production & Security

What’s Next for Atul Codes?

My Top 5 VS Code Extensions for DevOps and Cloud Engineering

1. HashiCorp Terraform

2. Docker

3. YAML (by Red Hat)

4. AWS Toolkit

5. GitLens — Git supercharged

The Takeaway

The 5 Linux Bash Commands Every DevOps Engineer Uses Daily

1. tail (Watching Live Logs)

2. grep (Finding the Needle in the Haystack)

3. awk (Parsing Structured Data)

4. chmod (Fixing "Permission Denied")

5. htop (Monitoring System Resources)

The Takeaway

Deploying Your First Serverless Function on AWS Lambda

Step 1: Create the Lambda Function

Step 2: Write the Serverless Code

Step 3: Add API Gateway

Step 4: Test Your Live API Endpoint

The Takeaway

How to Set Up AWS Billing Alerts (Before You Get a $1,000 Bill)

Step 1: Open the AWS Billing Dashboard

Step 2: Create a New Budget

Step 3: Set Your $5 Threshold

Step 4: Configure the Alert (The Most Important Step)

Step 5: Review and Create

The Takeaway

Stop Hardcoding AWS Credentials: How to Manage Secrets Securely

The Bad Way: Hardcoding in Your App

The Good Way (Local Development): AWS CLI Profiles

The Good Way (Pipelines & CI/CD): Environment Variables

The Ultimate Fix (Production): AWS Secrets Manager

The Takeaway

Infrastructure as Code 101: Provisioning an AWS EC2 Instance with Terraform

What is Terraform?

Prerequisites

Step 1: Define the Provider

Step 2: Write the EC2 Resource Block

Breaking down the code:

Step 3: The Core Terraform Workflow

1. Initialize the Directory

2. Plan the Deployment

3. Apply the Code

Step 4: Verify in the AWS Console

Step 5: Destroy the Evidence (Crucial!)

The Takeaway

TIL: How to Fix the "Port Already in Use" Error in Docker

Step 1: Find the Rogue Process

Step 2: Kill the Process

Bonus: What if it's a "Ghost" Docker Container?

TerraGuard AI: Adding an AI Brain to Terraform Drift Detection — Design Decisions, Trade-offs, and Lessons Learned

The problem with standard drift detection

Decision 1: AI for classification, not detection

Decision 2: Groq over AWS Bedrock

Decision 3: GitHub Actions over Lambda

Decision 4: Minimal infrastructure being monitored

Decision 5: Separating infrastructure and application deployment

Decision 6: Secrets Manager with ECS secrets references, not environment variables

A race condition I hit and what it taught me

What I'd build next

Closing thoughts

Automate Your Docker Builds with GitHub Actions

Prerequisites

Step 1: Create a Docker Hub Access Token

Step 2: Add Your Secrets to GitHub

Step 3: Update Your GitHub Actions Workflow

1. `tail` (Watching Live Logs)

2. `grep` (Finding the Needle in the Haystack)

3. `awk` (Parsing Structured Data)

4. `chmod` (Fixing "Permission Denied")

5. `htop` (Monitoring System Resources)