Monthly Golden Image Build Process using Packer & Ansible

Introduction:

In IT operations,Imagine We're working in a cloud organization that deploys hundreds of EC2 instances every month. Each instance needs to be secure, compliant.Manually configuring each instance is a nightmare. Instead, you want a golden image — a reusable AMI that’s pre-hardened and provisioned with all necessary tools.

This is the point at which Packer becomes relevant.

Problem Statement:

In most organizations, EC2 instances are launched frequently to support various workloads. But here's the catch — each instance needs to be:

Secure and compliant

Equipped with monitoring and security agents

Consistently configured

Real Time Scenario:

Suppose you’re part of a security-conscious enterprise. Every EC2 instance must:

1. Follow CIS benchmarks

2. Have CrowdStrike and Qualys agents installed

Instead of configuring each instance post-launch, you want to create a golden AMI that’s already hardened and provisioned. This image will serve as the base for all future deployments — saving time and ensuring consistency.

Tools Involved:

• Packer
• AWS EC2
• Ansible
• Gitlab CI/CD
• Amazon SSM

Architecture Diagram:

This workflow automates the creation of a secure AMI by:

Launching a temporary EC2 instance from a base image.

Running provisioning scripts to:

Apply OS hardening (CIS benchmarks, firewall rules, SSH configs).

Creating a new AMI from the configured instance.

Terminating the temporary instance.

Implementation Steps :

Step1: Install Packer

Step2: Create Packer Template with Ansible Provisioning

This Packer template automates the creation of a custom Amazon Machine Image (AMI) by launching a temporary EC2 instance in a specific AWS VPC and subnet, using a designated SSH key pair for secure access.

Code block

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.8"
      source  = "github.com/hashicorp/amazon"
    }
    ansible = {
      version = "~> 1"
      source = "github.com/hashicorp/ansible"
    }  
  }
}
variable "ami_prefix" {
  type = string
  default = ""
}

variable "reference_image" {
  type = string
  default = ""
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

variable "privatekey"{
  type = string
  default = ""
}

source "amazon-ebs" "amazon_linux" {
  ami_name      = "${var.ami_prefix}-${local.timestamp}"
  instance_type = "t2.micro"
  region        = "ap-south-1"
  vpc_id = "vpc-07b2ce11f9b189f3b"
  subnet_id = "subnet-063ebc661edd9fb37"
  security_group_id = "sg-04e9ae673095b02e9"
  ssh_interface = "private_ip"
  associate_public_ip_address = true
  ssh_keypair_name = "runner_key"
  ssh_private_key_file = var.privatekey

  source_ami_filter {
    filters = {
      name                = "${var.reference_image}"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = [""]
  }
  ssh_username = "ec2-user"
}

build {
  name = "learn-packer"
  sources = [
    "source.amazon-ebs.amazon_linux"
  ]

  provisioner "shell" {
  inline = [
    "sleep 20",
    "echo '--- Running AMI pre-check ---'",
    "set -e",
    "sudo mkdir -p /usr/lib",
    "# Ensure SFTP subsystem path exists",
    "if [ ! -f /usr/lib/sftp-server ]; then",
    "  if [ -f /usr/libexec/openssh/sftp-server ]; then",
    "    sudo ln -s /usr/libexec/openssh/sftp-server /usr/lib/sftp-server",
    "    echo 'Linked /usr/libexec/openssh/sftp-server -> /usr/lib/sftp-server';",
    "  else",
    "    echo 'Warning: sftp-server not found, installing openssh-server...';",
    "    sudo yum install -y openssh-server || sudo apt-get install -y openssh-server;",
    "  fi;",
    "fi",

    "# Basic network sanity check",
    "sudo yum clean all || true",
    "sudo yum update -y || true",
    "echo '--- Pre-check complete ---'"
  ]
  }

  provisioner "shell" {
  inline = [
    "sudo mkdir -p /tmp/.ansible",
    "sudo chmod 777 /tmp/.ansible"
  ]
  }

  provisioner "ansible" {
    playbook_file = "./playbook/main.yml"
    use_proxy = false
    extra_arguments = ["--vault-password-file=/home/gitlab-runner/.vault_pass",
    "-e", "ansible_remote_tmp=/tmp/.ansible",
    "-e", "ansible_local_tmp=/tmp/.ansible",
    "-e", "ansible_scp_if_ssh=True",
    "-e", "ansible_python_interpreter=/usr/bin/python3",
    "-e", "ansible_ssh_transfer_method=scp"]
  }

}

Ansible Main Block:

---
- name: Create users and providing sudo access
  hosts: all
  become: true
  gather_facts: true
  vars_files:
    - ../vars/useradd.yml
    - ../vars/vault.yml
  roles:
    - ../roles/useradd
    - ../roles/sudo

- name: Set hostnames
  hosts: all
  become: true
  gather_facts: false
  vars_files:
    - ../vars/var.yml
  roles:
    - ../roles/hostnamectl

- name: Enable or Set miscellaneous services
  hosts: all
  gather_facts: false
  become: true
  roles:
     - ../roles/ssh
     - ../roles/login_banner
     - ../roles/services
     - ../roles/timezone
    # - ../roles/fs_integrity
    #  - ../roles/selinux
    #  - ../roles/firewalld
    #  - ../roles/log_management
     - ../roles/rsyslog
    #  - ../roles/cron
    #  - ../roles/journald

Ansible User Creation block:

---
- hosts: all
  become: true
  gather_facts: true
  vars_files:
    - ../vars/useradd.yml
    - ../vars/vault.yml
  roles:
    - ../roles/useradd

Outcome:

This template builds a custom AMI by:

Launching a VM in a specific VPC and subnet.

Using a defined SSH key pair for access.

Running shell scripts and Ansible to configure the instance.

Saving the final image with a unique name for future use.

Note: Now, I've also configured a CI/CD variable in GitLab to securely store the private key content used for SSH access in the Packer build.

GitLab CI/CD Variable for Private Key
In GitLab, CI/CD variables allow us to store sensitive data like passwords, tokens, or SSH keys securely.

I've created a variable (e.g., PRIVATE_KEY) that contains the entire private key content (not just the path).

This variable is injected into the pipeline at runtime, allowing tools like Packer to use it without hardcoding the key or exposing it in our repository.

packer validate -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl

Step3: Gitlab Pipeline Stages

In this, GitLab CI/CD job automates the AMI creation process by:

Securely injecting an SSH private key from a CI/CD variable.

Validating and building a Packer template.

Provisioning an EC2 instance in a specific VPC/subnet.

Saving the final image for future use.

default: 
    tags:
      - gitlab_runner

stages:
  - image_build

Image Build: 
  stage: image_build
  script: 
    - echo "$SSH_PRIVATE_KEY" > runner_key.pem
    - chmod 400 runner_key.pem
    - packer init .
    - echo "Validating packer template..."
    - packer validate -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl
    - echo "Building AMI..."
    - packer build -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl

Conclusion: In this article, we tackled a common challenge in cloud operations i.e., ensuring every EC2 instance is secure, compliant, and consistently configured — without manual intervention.

By combining Packer, Ansible, and GitLab CI/CD, we built a fully automated pipeline that:

• Launches a temporary EC2 instance

• Applies CIS hardening and installs security agents

• Saves a golden AMI for future use

• Secures credentials using GitLab CI/CD variables

This approach not only boosts security and compliance but also saves hours of manual effort, reduces human error, and ensures every deployment starts from a trusted baseline.

Thanks,
Susseta Bose