DEV Community

Sireesharaju Kamparaju
Sireesharaju Kamparaju

Posted on

Ansible Vault — Managing Secrets Securely in Ansible and AWX — My DevOps Journey By Sireesha

#awx #ansible #security #devops

One thing I kept putting off when I started with Ansible was properly securing my secrets. Passwords, API keys, tokens — they were just sitting in plain text inside my vars files. I knew it was wrong but it worked and I kept moving forward.
Then I started thinking about what happens when this goes into GitLab. Anyone with access to the repo can see every password. That's when I properly sat down and learned Ansible Vault — and honestly I wish I'd done it from day one.
In this blog I'll walk through how I use Ansible Vault from the command line and then how I handle it properly inside AWX so scheduled jobs and workflow templates can still run without someone manually typing a password every time.

What is Ansible Vault?
Ansible Vault is a built-in feature that lets you encrypt sensitive data — passwords, keys, tokens — so they can be safely stored in your playbooks and pushed to GitLab without exposing anything.
The beauty of it is that it works right inside your existing YAML files. Your playbook structure doesn't change — Vault just encrypts the values that need protecting.

What I Was Doing Before — The Wrong Way
This is what my vars file looked like before Vault:

# vars/main.yml
ubuntu_sudo_password: MyPassword123
db_password: SuperSecret456
api_token: abcd1234efgh5678
Enter fullscreen mode Exit fullscreen mode

Pushed straight to GitLab. Anyone with repo access could read every single one of those. Not good.

Encrypting a Single Value with Ansible Vault
The first thing I learned was how to encrypt just a single variable value — not the entire file. This is cleaner because the rest of the vars file stays readable.

ansible-vault encrypt_string 'MyPassword123' --name 'ubuntu_sudo_password'
Enter fullscreen mode Exit fullscreen mode

It asks you to create a vault password — this is the master password you'll use to decrypt later. Enter it twice:

New Vault password:
Confirm New Vault password:
Enter fullscreen mode Exit fullscreen mode

The output looks like this:

ubuntu_sudo_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66386439653236336462626566653337386235396138623934363161623364663834623437333132
          3865663966343437326235373961316435623365663866610a663834623437333132386566396634
          34373236353337386235396138623934363161623364663834623437333132386566396634343732
          3635333738623539613862393436316162336466383462343733313238656639363434373236353337
Enter fullscreen mode Exit fullscreen mode

Copy this entire block and paste it into your vars file:

# vars/main.yml
ubuntu_sudo_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66386439653236336462626566653337386235396138623934363161623364663834623437333132
          ...
db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          ...
Enter fullscreen mode Exit fullscreen mode

Now this file is safe to push to GitLab. Anyone who opens it sees encrypted gibberish — not the actual passwords.

Encrypting an Entire Vars File
If you have a lot of secrets, encrypting the whole file is quicker:

ansible-vault encrypt vars/secrets.yml
Enter fullscreen mode Exit fullscreen mode

To edit it:

ansible-vault edit vars/secrets.yml
Enter fullscreen mode Exit fullscreen mode

To decrypt it permanently (be careful with this one): (I didn't try this)

ansible-vault decrypt vars/secrets.yml
Enter fullscreen mode Exit fullscreen mode

Running Playbooks with Vault from Command Line
This is how I was running playbooks before I set up AWX properly. Every time I ran a playbook that used vaulted variables, I added --ask-vault-pass:

ansible-playbook patch.yml -i inventory/hosts.yml --ask-vault-pass
Enter fullscreen mode Exit fullscreen mode

It prompts to enter vault password:

Vault password:
Enter fullscreen mode Exit fullscreen mode

Enter vault password and the playbook runs, decrypting the secrets on the fly. Nothing is ever stored in plain text on the server.
This works perfectly from the command line. But the moment you move to AWX — scheduled jobs, workflow templates, CI/CD — you can't have AWX stopping and waiting for someone to type a password. That's where the AWX Vault credential comes in.

The Problem with --ask-vault-pass in AWX
Imagine you've set up your patching workflow to run every Sunday night at 2AM. AWX kicks off the job automatically — but your playbook uses vaulted passwords. AWX has no way to ask you for the vault password at 2AM.
The job just fails.
That's exactly the problem I hit. I had my refresh.yml scheduled for every Monday at 5AM and it kept failing because it couldn't decrypt the vault variables. I was confused at first — it ran fine from the command line. Then I realised AWX needs the vault password stored as a credential so it can decrypt automatically at runtime.

Solving It in AWX — Adding a Vault Credential
AWX has a built-in credential type specifically for Ansible Vault. Here's how I set it up:

AWX UI → Credentials → Add ->Click Save

The vault password is stored encrypted inside AWX — no one can read it back out. It's just referenced by name.

Attaching the Vault Credential to a Job Template
Now I attach this credential to any job template that uses vaulted variables:

  • AWX UI → Templates → open your job template → Edit
  • In the Credentials field — click the search icon
  • You'll see your existing machine credential already there
  • Search for ansible-vault-cred and select it as well
  • A job template can have multiple credentials — one for machine access, one for vault
  • Click Save

Now when AWX runs this job template — whether manually, on a schedule, or as part of a workflow — it automatically decrypts the vault variables using the stored credential. No one needs to type anything.

Example — Patching Workflow with Vault
Let me show you exactly how this works end to end with my patching workflow.
My pre_patch_check.yml playbook connects to servers using credentials stored in a vaulted vars file:

# vars/secrets.yml (encrypted with ansible-vault)
ubuntu_sudo_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          ...
db_connection_string: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          ...
Enter fullscreen mode Exit fullscreen mode

Myplaybook

---
- name: Pre-patch checks
  hosts: all
  vars_files:
    - vars/secrets.yml
  tasks:
    - name: Check application status
      ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}/health"
        headers:
          Authorization: "Bearer {{ api_token }}"
      register: health_check
Enter fullscreen mode Exit fullscreen mode

From command line I run it with:

ansible-playbook pre_patch_check.yml -i inventory/hosts.yml --ask-vault-pass
Enter fullscreen mode Exit fullscreen mode

In AWX — the job template has both machine-cred and ansible-vault-cred attached. When the patching workflow kicks off at 2AM on Sunday, AWX decrypts the secrets automatically and the whole workflow runs without anyone being awake to babysit it.
That's proper production automation.

Rotating Vault Password
Every now and then you should change your vault master password — especially if someone who knew it leaves the team. Here's how to rekey all your encrypted files at once:

ansible-vault rekey vars/secrets.yml
Enter fullscreen mode Exit fullscreen mode

It asks for the old password, then the new one. After rekeying, update the vault credential in AWX with the new password too — otherwise your scheduled jobs will start failing.

Errors I Hit Along the Way
Error 1 — Decryption Failed

ERROR! Decryption failed (no vault secrets would unlock any vault file/string)
Enter fullscreen mode Exit fullscreen mode

Fix — I had forgotten to attach the ansible-vault-cred to the job template. Once I added it, the scheduled job ran perfectly.
Error 3 — Committed Vault Password File to GitLab
Early on I accidentally committed my .vault_pass file to GitLab. As soon as I realised:

# Remove from tracking
git rm --cached .vault_pass

# Add to gitignore
echo ".vault_pass" >> .gitignore

git commit -m "removed vault password file from tracking"
git push origin dev
Enter fullscreen mode Exit fullscreen mode

And immediately changed my vault password using ansible-vault rekey. Lesson learned — always add .vault_pass to .gitignore before creating it.

My Golden Rules for Ansible Vault

After going through all of this, here's what I now follow without exception:

  • Never store plain text passwords in vars files — encrypt everything sensitive from day one
  • Always add .vault_pass to .gitignore before creating the file
  • Use AWX Vault credentials for any job that runs on a schedule or in a workflow
  • Rekey when team members leave — treat it like changing a shared password
  • Test vault decryption after any rekey before your next scheduled job runs

What's Next
Now that secrets are properly managed, the next blog I'm planning is AWX Notifications — setting up Slack and email alerts so when a patching workflow fails at 2AM, I know about it first thing in the morning without having to log into AWX to check.

Drop your questions in the comments — happy to help!
— Sireesha

Top comments (0)