DEV Community

DarkEdges
DarkEdges

Posted on

Building CIS-Hardened, SBOM-Attested CentOS 9 Golden Images with Packer, QEMU and PingAccess - entirely on WSL2

#devsecops #packer #supplychainsecurity #linux

Most "golden image" pipelines assume a cloud builder or a beefy Linux box. This one runs end-to-end on a Windows laptop, inside WSL2 with nested KVM, and produces a CentOS 9 Stream QCOW2 image that is:

  • CIS Level 1 hardened (via the official ansible-lockdown role)
  • Pre-loaded with PingAccess 8.3.5 on a JRE 17 runtime
  • Shipped with a signed SBOM and VEX attestations so a Trivy scan tells you what's actually exploitable - not just what's theoretically vulnerable

Everything is driven by a single ./build.sh. Here's how it fits together, and the WSL2-specific gotchas I had to solve along the way.

The shape of the pipeline

The build is split into two Packer sources that run sequentially, because the second one boots the artifact the first one produced.

CentOS 9 Stream ISO
        │  Build 1 - "harden"
        ▼
  kickstart install ──► CIS Level 1 hardening (Ansible) ──► base SBOM
        │
        ▼
  output/base/centos9-base.qcow2
        │  Build 2 - "pingaccess"
        ▼
  JRE 17 + PingAccess ──► VEX/SBOM artifacts ──► app SBOM
        │
        ▼
  output/app/centos9-app.qcow2
        │  "sbom" target
        ▼
  Trivy scan (VEX-suppressed) + cosign-signed VEX + CVE report
Enter fullscreen mode Exit fullscreen mode

build.sh exposes each stage as a target so you can iterate on just the part you're changing:

./build.sh harden        # ISO → CIS-hardened base image
./build.sh pingaccess    # base image → Java + PingAccess + security artifacts
./build.sh sbom          # scan + sign an existing app image
./build.sh               # all of the above, end to end
Enter fullscreen mode Exit fullscreen mode

Build 1: a CIS-compliant base image from a kickstart

The foundation is an automated CentOS 9 Stream install. The interesting part is the partition layout - CIS expects several mount points to be isolated filesystems with restrictive flags, so the kickstart lays them out as separate LVM logical volumes:

part /boot            xfs   512 MiB
volgroup centos_vg
  lv_root      /                6 GiB   xfs
  lv_tmp       /tmp            512 MiB  xfs   nodev,nosuid,noexec
  lv_var       /var              2 GiB  xfs
  lv_varlog    /var/log        512 MiB  xfs
  lv_audit     /var/log/audit  512 MiB  xfs
  lv_home      /home           512 MiB  xfs   nodev,nosuid
  lv_swap      swap              1 GiB
Enter fullscreen mode Exit fullscreen mode

SELinux is left enforcing, the firewall is enabled with only SSH open, and unused network services are stripped out.

Hardening itself is delegated to the maintained ansible-lockdown.rhel9_cis role rather than a hand-rolled script. A few overrides keep it Packer-friendly:

rhel9cis_level_1: true
rhel9cis_level_2: false
os_check: false                 # role asserts != CentOS; Stream == RHEL 9 controls
skip_reboot: true               # Packer powers off after all provisioners
rhel9cis_rule_5_2_2: false      # "Defaults use_pty" hangs sudo under pipelining
rhel9cis_sudoers_exclude_nopasswd_list: [packer]   # keep build sudo mid-play
Enter fullscreen mode Exit fullscreen mode

That last group is the kind of thing you only learn by watching a build hang at 80%: CIS rule 5.2.2 adds Defaults use_pty to sudoers, which deadlocks sudo when Ansible pipelining is on and no PTY is allocated.

Build 2: PingAccess, the noexec-/tmp problem, and a lean image

Build 2 boots the hardened base over SSH (key-based now - password auth is gone) and layers on Java 17 and PingAccess.

The first surprise: /tmp is 512 MiB and mounted noexec thanks to the CIS layout. That breaks two assumptions at once - the PingAccess archive doesn't fit, and you can't execute staged binaries from there. The fix is to stage and extract under /opt (which lives on the 6 GiB root LV):

- name: "PA | Extract PingAccess archive"
  ansible.builtin.unarchive:
    src: /opt/pa_staging.zip
    dest: /opt/pa_extract/
    remote_src: true
Enter fullscreen mode Exit fullscreen mode

To keep the image lean, the role deletes the PingAccess SDK samples after install (nobody runs the Maven sample projects in production), and the final Packer step does dnf clean all + fstrim -av. Combined with disk_discard = "unmap" and disk_compression = true, the TRIM is the single biggest lever on final image size.

The interesting part: shipping security context inside the image

A plain vulnerability scan of any real product image is mostly noise - hundreds of CVEs in transitive dependencies that aren't reachable, aren't loaded, or are already mitigated. The goal here was an image that carries its own answer to "is this actually exploitable?"

SBOMs generated inside the VM

Rather than scanning the QCOW2 from the host (which gives you the host's view, and made my base and app SBOMs come out identical at one point), SBOM generation runs inside each VM during its build via a small generate_sbom Ansible role. It downloads syft, scans /, writes the SBOM to /opt/security/sbom/, then fetches a copy back to the host and removes the binary:

- name: "SBOM | Generate SBOM"
  ansible.builtin.shell:
    cmd: >
      syft / --output spdx-json
      --file {{ sbom_output_dir }}/{{ sbom_filename }}
      {% for e in sbom_excludes %}--exclude {{ e }} {% endfor %}

- name: "SBOM | Fetch SBOM to build host"
  ansible.builtin.fetch:
    src: "{{ sbom_output_dir }}/{{ sbom_filename }}"
    dest: "{{ sbom_host_dest }}"
    flat: true
Enter fullscreen mode Exit fullscreen mode

Now you get a true base SBOM (OS only) and a true app SBOM (OS + JRE + PingAccess), and both are embedded in the image for later audit.

VEX: turning "vulnerable" into "not affected"

OpenVEX statements describe whether a product is actually affected by a given CVE. The pipeline collects *.openvex.json statements, merges them with vexctl, signs the consolidated document with cosign, and embeds it at /opt/security/vex/:

cosign generate-key-pair --output-key-prefix cosign   # COSIGN_PASSWORD="" for unattended
cosign sign-blob --key cosign.key --yes consolidated.openvex.json
Enter fullscreen mode Exit fullscreen mode

(Keyless OIDC signing is tempting but hangs in an unattended build - a local key with an empty password is the right call for CI.)

Trivy + VEX = signal, not noise

Finally, Trivy scans the mounted image filesystem with the VEX document applied, so suppressed findings are shown but explained:

trivy fs --vex consolidated.openvex.json \
  --severity CRITICAL,HIGH,MEDIUM,LOW \
  --show-suppressed --format json image-root/
Enter fullscreen mode Exit fullscreen mode

A small Python summariser (no jq dependency) turns the JSON into a per-severity CVE count, including how many were suppressed by VEX. There's a NO_VEX=1 switch to see the raw, unsuppressed picture when you want it.

Mounting a QCOW2 offline without a running VM

To scan the image's filesystem (not just packages), build.sh mounts the QCOW2 read-only via qemu-nbd. Two tricks make this reliable:

qemu-nbd --connect=/dev/nbd0 --snapshot app.qcow2   # CoW overlay → no exclusive lock
kpartx -av /dev/nbd0
pvscan --cache
vgchange -ay centos_vg
udevadm settle
mount -t xfs -o ro,norecovery /dev/centos_vg/lv_root /mnt/packer-scan
Enter fullscreen mode Exit fullscreen mode

--snapshot writes changes to a throwaway overlay, so the original file is never locked exclusively - you can even scan while being careful about a running VM. And because the root filesystem lives on LVM, you can't just mount the partition; you have to pvscan/vgchange to activate the volume group first. The norecovery mount option avoids XFS log replay on a read-only device.

Running it

A companion run.sh boots either image under QEMU/KVM with port forwarding wired up:

./run.sh           # boot the app image (PingAccess)
./run.sh ssh       # boot + drop into an SSH session
./run.sh stop      # graceful shutdown over SSH, SIGTERM fallback
Enter fullscreen mode Exit fullscreen mode 
2222  → 22     SSH
19000 → 9000   PingAccess Admin UI
13000 → 3000   PingAccess Engine
Enter fullscreen mode Exit fullscreen mode

WSL2 lessons worth stealing

  • Use -cpu host. The default qemu64 vCPU lacks SSE4.2/XSAVE and the CentOS 9 kernel panics on boot.
  • Build to an ext4 path, not /mnt/c or /mnt/e. QCOW2 I/O over the 9P/NTFS bridge is unreliable; build on the Linux filesystem and copy out afterward.
  • q35 machine type is required - CentOS 9's initramfs uses AHCI for CD-ROM access.
  • scp_extra_args = -O forces legacy SCP mode, which Ansible's file transfers need against the hardened sshd.
  • Trivy reads embedded SBOM/SPDX files and will report CVEs for components listed there even after you've deleted them from disk. Skip those directories in the scan, or it'll look like your cleanup didn't work.

Takeaway

The result is a reproducible, single-command build that turns an ISO into a hardened, application-ready image whose security posture travels with it: an embedded SBOM that says what's inside, signed VEX attestations that say what's actually exploitable, and a Trivy report that separates signal from noise - all on a laptop.

If you're building golden images, consider making the SBOM and VEX first-class build artifacts rather than an afterthought bolted on at scan time. The difference between "412 CVEs" and "412 CVEs, 6 actually relevant, here's the signed proof" is the difference between a report nobody reads and one a security team can act on.

GitHub

The repository is at https://github.com/darkedges/centos9-qemu

Top comments (0)