DEV Community

Sireesharaju Kamparaju
Sireesharaju Kamparaju

Posted on AI-assisted

Patching Ubuntu Linux Servers with Ansible & AWX — A Production Approach — My DevOps Journey

#ansible #devops #automation #linux

After writing about Windows Server patching, I wanted to do the same for our Ubuntu Linux servers. The approach is the same in terms of thinking — check first, patch safely, validate after — but Linux has its own things to handle, especially around stopping your application cleanly before touching the system.

Before I run anything on our live UAT or production servers, I always test the playbook in my homelab first. That's just how I work — if it doesn't run cleanly in the lab, it doesn't go near a live server. This blog is based on what I actually do at work, so hopefully it saves you some time.

One thing I want to flag early — in our environment we use InterSystems IRIS to manage our client servers. Before patching I stop IRIS using either iris stop iris or /usr/utils/ishut — both do the same thing. If you're following this for your own environment, just swap this out for whatever application you need to stop before patching. The principle is the same — never patch a server with your application still running.

In this blog I'll cover:

  • Pre-check — kernel version, disk space, pending reboots, update scan
  • Patch — stop application, run updates, reboot, start application
  • Post-check — new kernel, uptime, installed packages
  • Validate — kernel comparison before vs after, final connectivity check

Each phase runs as a separate AWX job template using Ansible tags — same as my Windows patching blog. One server at a time (serial: 1) so if something goes wrong on one server it doesn't touch the rest.

The Architecture — How It Runs in AWX 

AWX Workflow: Ubuntu Patching
  │
  ├── Job Template 1: Pre-Check        (tags: precheck)
  │         └── [On Success]
  ├── Approval Node: Review Pre-Check Logs
  │         └── [On Approval]
  ├── Job Template 2: Patch            (tags: patch)
  │         └── [On Success]
  ├── Job Template 3: Post-Check       (tags: postcheck)
  │         └── [On Success]
  └── Job Template 4: Validate         (tags: validate)
Enter fullscreen mode Exit fullscreen mode

I always add a manual Approval node between Pre-Check and Patch. After the pre-check runs and saves the log, the workflow pauses — I review the output, check the kernel versions, disk space and application status, then approve before patching starts. That human checkpoint before touching a live server is something I'd never skip.

The Playbook Variables 

vars:
  log_dir: "/etc/ansible/patchlogs"
  app_stop_cmd: "/usr/utils/ishut"
  app_start_cmd: "/usr/utils/ikick.myapp"
  reboot_timeout: 300
Enter fullscreen mode Exit fullscreen mode

log_dir — where patch logs get saved on the Ansible controller. Every server gets its own timestamped log file so you always have something to refer back to after a patching window.

app_stop_cmd — which is the same as iris stop iris since our application is realted to intersystems we use iris stop iris. Change this to whatever stops your application cleanly.

app_start_cmd — brings everything back up after the reboot. Swap this out for your own application start command.

Important: Before you run this in your environment, check what application you need to stop before patching. The playbook will handle the stop and start — you just need to know the right command for your setup.

Pre-Tasks — Setting Up Log Files

Before anything runs I set up the log directory and file paths on the controller. I use tags: [always] so these run regardless of which phase I'm running:

pre_tasks:
  - name: Build run timestamp on controller
    delegate_to: localhost
    run_once: true
    set_fact:
      date_str: "{{ lookup('ansible.builtin.pipe', 'date +%Y%m%d_%H%M%S') }}"
    tags: [always]

  - name: Ensure log directory exists on controller
    delegate_to: localhost
    run_once: true
    ansible.builtin.file:
      path: "{{ log_dir }}"
      state: directory
      mode: "0755"
    tags: [always]

  - name: Build per-host log file paths
    set_fact:
      precheck_file: "{{ log_dir }}/{{ inventory_hostname }}_precheck_{{ date_str }}.log"
      postcheck_file: "{{ log_dir }}/{{ inventory_hostname }}_postcheck_{{ date_str }}.log"
    tags: [always]
Enter fullscreen mode Exit fullscreen mode

Phase 1 — Pre-Check

This is the phase I care about the most. I never approve the patch without going through what this produces.

1. Get Current and Available Kernel Version 

- name: Get current kernel version before patching
  ansible.builtin.command: uname -r
  register: kernel_version_before
  changed_when: false
  tags: [precheck]

- name: Get latest kernel available in apt
  ansible.builtin.shell: >
    apt-cache madison linux-image-generic |
    awk '{print $3}' | head -1
  register: latest_kernel_version
  changed_when: false
  tags: [precheck]

- name: Show kernel comparison
  ansible.builtin.debug:
    msg: >
      {{ inventory_hostname }} |
      Running: {{ kernel_version_before.stdout }} |
      Available: {{ latest_kernel_version.stdout }} |
      Update needed: {{ kernel_version_before.stdout != latest_kernel_version.stdout }}
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

This tells me straight away if a kernel update is actually needed. If the server is already on the latest kernel — nothing happens. No unnecessary reboots, no unnecessary application downtime.

2. Check Disk Space 

- name: Check disk space on root partition
  ansible.builtin.shell: df -h / | awk 'NR==2 {print $4}'
  register: disk_space_free
  changed_when: false
  tags: [precheck]

- name: Check disk usage percentage
  ansible.builtin.shell: >
    df / | awk 'NR==2 {print $5}' | sed 's/%//'
  register: disk_usage_pct
  changed_when: false
  tags: [precheck]

- name: Fail if disk space is critically low
  ansible.builtin.fail:
    msg: >
      Disk usage on {{ inventory_hostname }} is at
      {{ disk_usage_pct.stdout }}%. Not safe to patch —
      clean up disk space first.
  when: disk_usage_pct.stdout | int > 85
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

If disk usage is over 85% the playbook stops for that server. I learned early on that kernel updates need room and a nearly-full disk mid-patch is not something you want to deal with.

3. Check for Pending Reboots 

- name: Check if a reboot is already pending
  ansible.builtin.stat:
    path: /var/run/reboot-required
  register: reboot_pending
  tags: [precheck]

- name: Fail if reboot already pending
  ansible.builtin.fail:
    msg: >
      {{ inventory_hostname }} already has a pending reboot.
      Reboot the server first before patching.
  when: reboot_pending.stat.exists
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

Same logic as Windows — if there's already a pending reboot, we stop. Patching a server that's already waiting for a reboot causes more problems than it solves.

4. Check Uptime and System Load 

- name: Get system uptime
  ansible.builtin.command: uptime
  register: pre_uptime
  changed_when: false
  tags: [precheck]

- name: Get system load average
  ansible.builtin.shell: >
    cat /proc/loadavg | awk '{print $1, $2, $3}'
  register: load_avg
  changed_when: false
  tags: [precheck]

- name: Warn if load average is high
  ansible.builtin.debug:
    msg: >
      WARNING: {{ inventory_hostname }} has high load:
      {{ load_avg.stdout }}. Consider patching at a quieter time.
  when: load_avg.stdout.split()[0] | float > 2.0
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

5. Scan Available Updates 

- name: Update apt cache
  ansible.builtin.apt:
    update_cache: yes
    cache_valid_time: 3600
  tags: [precheck]

- name: List available security updates
  ansible.builtin.shell: >
    apt list --upgradable 2>/dev/null |
    grep -i security || echo "No security updates available"
  register: security_updates
  changed_when: false
  tags: [precheck]

- name: Count available updates
  ansible.builtin.shell: >
    apt list --upgradable 2>/dev/null |
    grep -c upgradable || echo 0
  register: update_count
  changed_when: false
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

6. Check Application Status Before Patch 

- name: Check IRIS application status before patch
  ansible.builtin.shell: >
    ps aux | grep -v grep | grep -i iris ||
    echo "IRIS not running"
  register: app_status_pre
  changed_when: false
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

7. Save Pre-Check Log 

- name: Save pre-check summary to controller
  delegate_to: localhost
  ansible.builtin.copy:
    dest: "{{ precheck_file }}"
    mode: "0644"
    content: |
      ============================================
      PRE-CHECK REPORT
      ============================================
      Host              : {{ inventory_hostname }}
      Date              : {{ date_str }}
      Kernel (running)  : {{ kernel_version_before.stdout }}
      Kernel (available): {{ latest_kernel_version.stdout }}
      Update needed     : {{ kernel_version_before.stdout != latest_kernel_version.stdout }}
      Disk free         : {{ disk_space_free.stdout }}
      Disk usage %      : {{ disk_usage_pct.stdout }}%
      Pending reboot    : {{ reboot_pending.stat.exists }}
      Uptime            : {{ pre_uptime.stdout | trim }}
      Load average      : {{ load_avg.stdout }}
      Updates available : {{ update_count.stdout }}
      IRIS status       : {{ app_status_pre.stdout | trim }}
      --------------------------------------------
      Available security updates:
      {{ security_updates.stdout }}
  tags: [precheck]
Enter fullscreen mode Exit fullscreen mode

This is what I review in AWX before clicking Approve. If anything looks off — high disk usage, unexpected load, IRIS already down — I investigate before touching anything.

Phase 2 — Patch

Once I've reviewed the pre-check log and approved the workflow in AWX, the patch phase kicks in.

1. Stop IRIS Before Patching 

- name: Stop IRIS before patching
  ansible.builtin.shell: "{{ app_stop_cmd }}"
  register: app_stop_result
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Wait 30 seconds after stopping IRIS
  ansible.builtin.wait_for:
    timeout: 30
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Verify IRIS has stopped
  ansible.builtin.shell: >
    ps aux | grep -v grep | grep -i iris ||
    echo "IRIS not running"
  register: app_verify_stopped
  changed_when: false
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Fail if IRIS is still running
  ansible.builtin.fail:
    msg: >
      IRIS is still running on {{ inventory_hostname }}.
      Cannot patch safely — investigate and stop manually.
  when:
    - kernel_version_before.stdout != latest_kernel_version.stdout
    - "'IRIS not running' not in app_verify_stopped.stdout"
  tags: [never, patch]
Enter fullscreen mode Exit fullscreen mode

I don't just run the stop command and move on — I always verify IRIS has actually stopped before proceeding. If it's still running, the playbook fails and patching doesn't happen. This step has saved me more than once.

Note for your environment: Replace {{ app_stop_cmd }} with your own application stop command — whether that's stopping Tomcat, nginx, a custom service or anything else your server runs.

2. Install Updates 

- name: Update apt cache
  ansible.builtin.apt:
    update_cache: yes
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Install security and critical updates
  ansible.builtin.apt:
    upgrade: dist
  register: patch_result
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Remove unused packages after upgrade
  ansible.builtin.apt:
    autoremove: yes
    autoclean: yes
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]
Enter fullscreen mode Exit fullscreen mode

3. Reboot If Required 

- name: Check if reboot is required after patching
  ansible.builtin.stat:
    path: /var/run/reboot-required
  register: reboot_required_file
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Reboot the server if required
  ansible.builtin.reboot:
    reboot_timeout: "{{ reboot_timeout }}"
    msg: "Rebooting after kernel update — Ansible AWX patching"
    pre_reboot_delay: 10
  when:
    - kernel_version_before.stdout != latest_kernel_version.stdout
    - reboot_required_file.stat.exists
  tags: [never, patch]

- name: Wait for server to come back online
  ansible.builtin.wait_for_connection:
    delay: 20
    timeout: "{{ reboot_timeout }}"
  when:
    - kernel_version_before.stdout != latest_kernel_version.stdout
    - reboot_required_file.stat.exists
  tags: [never, patch]
Enter fullscreen mode Exit fullscreen mode

4. Start IRIS After Reboot 

- name: Start IRIS after patching
  ansible.builtin.shell: "{{ app_start_cmd }}"
  register: app_start_result
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Wait for IRIS to fully start
  ansible.builtin.wait_for:
    timeout: 60
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Verify IRIS started successfully
  ansible.builtin.shell: >
    ps aux | grep -v grep | grep -i iris ||
    echo "IRIS not running"
  register: app_verify_started
  changed_when: false
  when: kernel_version_before.stdout != latest_kernel_version.stdout
  tags: [never, patch]

- name: Fail if IRIS did not start
  ansible.builtin.fail:
    msg: >
      IRIS failed to start on {{ inventory_hostname }} after patching.
      Manual intervention required immediately.
  when:
    - kernel_version_before.stdout != latest_kernel_version.stdout
    - "'IRIS not running' in app_verify_started.stdout"
  tags: [never, patch]
Enter fullscreen mode Exit fullscreen mode

If IRIS doesn't come back up, the playbook fails with a clear message and the next server doesn't get touched. That's the right behaviour — you want to know immediately if something went wrong before continuing down the fleet.

Phase 3 — Post-Check 

- name: Get kernel version after patching
  ansible.builtin.command: uname -r
  register: kernel_version_after
  changed_when: false
  tags: [never, postcheck]

- name: Get uptime after reboot
  ansible.builtin.command: uptime
  register: post_uptime
  changed_when: false
  tags: [never, postcheck]

- name: Get list of recently installed packages
  ansible.builtin.shell: >
    grep "$(date +%Y-%m-%d)" /var/log/dpkg.log |
    grep " install " | awk '{print $4}' ||
    echo "No packages installed today"
  register: installed_packages
  changed_when: false
  tags: [never, postcheck]

- name: Check disk space after patching
  ansible.builtin.shell: df -h / | awk 'NR==2 {print $4}'
  register: disk_space_after
  changed_when: false
  tags: [never, postcheck]

- name: Save post-check summary to controller
  delegate_to: localhost
  ansible.builtin.copy:
    dest: "{{ postcheck_file }}"
    mode: "0644"
    content: |
      ============================================
      POST-CHECK REPORT
      ============================================
      Host              : {{ inventory_hostname }}
      Date              : {{ date_str }}
      Kernel (before)   : {{ kernel_version_before.stdout }}
      Kernel (after)    : {{ kernel_version_after.stdout }}
      Kernel changed    : {{ kernel_version_before.stdout != kernel_version_after.stdout }}
      Disk free (after) : {{ disk_space_after.stdout }}
      Uptime (after)    : {{ post_uptime.stdout | trim }}
      --------------------------------------------
      Packages installed today:
      {{ installed_packages.stdout }}
  tags: [never, postcheck]
Enter fullscreen mode Exit fullscreen mode

Phase 4 — Validate 

- name: Summarize patch result
  ansible.builtin.debug:
    msg: |
      ============================================
      PATCH VALIDATION SUMMARY
      ============================================
      Host          : {{ inventory_hostname }}
      Kernel Before : {{ kernel_version_before.stdout }}
      Kernel After  : {{ kernel_version_after.stdout | default('N/A') }}
      Kernel Changed: {{ kernel_version_after is defined and kernel_version_before.stdout != kernel_version_after.stdout }}
      Status        : {{
        'PATCH SUCCESSFUL - KERNEL UPDATED'
        if (kernel_version_after is defined and kernel_version_before.stdout != kernel_version_after.stdout)
        else 'NO KERNEL CHANGE - verify packages or already up to date'
      }}
  tags: [never, validate]

- name: Final ping test
  ansible.builtin.ping:
  tags: [never, validate]

- name: Final IRIS status check
  ansible.builtin.shell: >
    ps aux | grep -v grep | grep -i iris ||
    echo "IRIS not running"
  register: final_app_status
  changed_when: false
  tags: [never, validate]

- name: Confirm IRIS is running at end of validation
  ansible.builtin.debug:
    msg: >
      {{ inventory_hostname }} — IRIS status after patching:
      {{ final_app_status.stdout | trim }}
  tags: [never, validate]
Enter fullscreen mode Exit fullscreen mode

Setting This Up in AWX

Step 1 — Create Four Job Templates

Job Template Playbook Job Tags
Ubuntu Pre-Check UAT-UBUNTU-PATCHING-WITH-Logs.yaml precheck
Ubuntu Patch UAT-UBUNTU-PATCHING-WITH-Logs.yaml patch
Ubuntu Post-Check UAT-UBUNTU-PATCHING-WITH-Logs.yaml postcheck
Ubuntu Validate UAT-UBUNTU-PATCHING-WITH-Logs.yaml validate

Step 2 — Create the Workflow

  • AWX UI → Templates → Add → Add Workflow Template
  • Name: Ubuntu Patching Workflow
  • Click Save → Workflow Visualiser

Chain using On Success with an Approval node between Pre-Check and Patch:

Pre-Check → [On Success] → Approval → [Approved] → Patch → [On Success] → Post-Check → [On Success] → Validate

Step 3 — What I Check Before Approving

When the pre-check completes and the workflow pauses for approval, here's what I actually look at in the log before clicking Approve:

  • Is the kernel update actually needed or is the server already up to date?
  • Is disk space healthy — anything over 85% I investigate first
  • Is there already a pending reboot — if yes I sort that first
  • Is IRIS running as expected before we stop it
  • Does the load average look normal for this time of day

If everything looks fine — approve and let it run. If anything looks off — I reject and investigate before rescheduling.

Errors I Hit Along the Way

Error 1 — IRIS Not Stopping Cleanly

The stop command returned success but IRIS was still running when the verify task checked.

Fix — added the verify step and fail condition after stopping. Never trust the stop command's exit code alone — always check the process is actually gone.

Error 2 — Server Taking Too Long to Come Back

On some of our heavier servers, the default 300 second reboot timeout wasn't enough after a big kernel update.

Fix — increased reboot_timeout to 600 for those servers:

reboot_timeout: 600
Enter fullscreen mode Exit fullscreen mode

Error 3 — Kernel Version Not Changing After Patch

The patch ran and showed updates installed but uname -r showed the same kernel before and after.

This usually means the new kernel was installed but the reboot didn't happen or /var/run/reboot-required wasn't created. Check manually:

ls /boot/vmlinuz-* | sort -V | tail -1
Enter fullscreen mode Exit fullscreen mode

This shows the newest kernel installed on disk regardless of what's currently running.

Error 4 — IRIS Starting Too Quickly After Reboot

The IRIS start command ran before some system services it depends on were fully up after reboot.

Fix — added a 60 second wait after the server comes back before starting IRIS:

- name: Wait for system services to settle after reboot
  ansible.builtin.wait_for:
    timeout: 60
Enter fullscreen mode Exit fullscreen mode

Error 5 — Disk Check Failing on LVM

The standard df -h / check was giving strange results on servers using LVM thin provisioning.

Fix — switched to:

df --output=pcent / | tail -1 | tr -d ' %'
Enter fullscreen mode Exit fullscreen mode

What This Gives You

Like my Windows patching blog, this setup gives you proper production confidence:

  • Never patch blindly — pre-check tells you exactly what's going to happen before it happens
  • Application safety — IRIS stops before patching, gets verified as stopped, comes back up and gets verified as running — at both ends
  • Full audit trail — pre and post logs per server per patching run saved on the controller
  • Smart skipping — if the kernel is already up to date nothing happens, no unnecessary downtime
  • Visibility in AWX — four separate phases each visible in the workflow, easy to see exactly where something failed
  • Human checkpoint — manual approval before patching means someone always reviews before touching production

A reminder for your own environment: The application stop and start commands in this playbook are specific to our setup using InterSystems IRIS. Before running this against your own servers, update app_stop_cmd and app_start_cmd to match whatever application you need to manage during patching.

Thanks for reading.

Drop your questions in the comments — happy to help!

— Sireesha

Top comments (0)