Cloud automation with Ansible series: Inventory caching

WORDS BY Sašo Stanovnik

We’ve previously talked about using a dynamic inventory for cloud automation with Ansible and promised to also explore inventory caching. Here we are! We’re looking at what inventory caching is, why we need it, and how it can speed up our day-to-day operations with Ansible.

Inventory caching basics

As Ansible uses a dynamic inventory, this means that it must read remote state before executing any operations. This is frequently done in batches of operations, called Ansible playbooks, where creating an inventory only occurs once at the beginning. Ignoring manually requesting refreshes, of course. Those are useful when your playbook creates a new host on a cloud provider, and you want the same playbook to be aware of that new host. It is also quite useful for updating the inventory cache for later executions, effectively simulating a static inventory.

For numerous ad-hoc commands however, connecting to an infrastructure provider’s API and establishing an inventory each and every time can get time-consuming. Think generating many different reports, executing quick fixups or queries against nodes, testing connectivity and so on. Let’s look at how inventory caching works in practice!

Creating some machines

But first, we need some machines to use our inventory to keep this example self-contained. Let’s just create one to keep things simple. Here’s our very simple playbook. Don’t worry about the loop, we’ll get to using that later.

- hosts: localhost
  vars:
    your_pubkey: YOUR_PUBKEY
  tasks:
    - community.digitalocean.digital_ocean_sshkey:
        name: steampunk-pubkey
        state: present
        ssh_pub_key: "{{ your_pubkey }}"
      register: pubkey

    - community.digitalocean.digital_ocean_droplet:
        name: "{{ item }}"
        unique_name: true
        ssh_keys:
          - "{{ pubkey.data.ssh_key.fingerprint }}"
        size: s-1vcpu-1gb
        region: fra1
        image: centos-8-x64
        state: present
      loop:
        - steampunk-cachelet-1

We’ll need to install the DigitalOcean community Ansible collection to do this. Since inventory caching isn’t available as a release for this collection (we’ll get to this later!), we install it directly from the upstream repository. Then, all that’s left is to run the playbook!

# this makes things easy to clean up
mkdir steampunk-trials && cd steampunk-trials/
python3 -m venv .venv && source .venv/bin/activate

pip install -U pip wheel
pip install ansible-core==2.11.1
ansible-galaxy collection install \
    git+https://github.com/ansible-collections/community.digitalocean.git

export DO_API_TOKEN=<YOUR API TOKEN>

ansible-playbook playbook.yml

Enabling and using the inventory cache

To use the inventory plugin, we need to create an appropriate file. We went through all the details in our previous blog post. Let’s call it digitalocean.yml with the following contents.

plugin: community.digitalocean.digitalocean

compose:
  ansible_host: do_networks.v4 | selectattr('type','eq','public') | map(attribute='ip_address') | first
  ansible_user: "'centos'"
  ansible_ssh_common_args: "'-o StrictHostKeyChecking=no'"

# define caching variables
cache: true
cache_plugin: ansible.builtin.jsonfile
cache_connection: digitalocean-cache

Now, every time the inventory plugin is run, the cache is consulted instead of the API. We do need to explain what each of the three new (bottommost) settings do.

cache: true just enables caching with any caching plugin. The second, cache_plugin, selects a caching plugin, which is a fancy way of saying how the cache is stored. jsonfile is a persistent cache, but there are also non-persistent, in-memory cache plugins, such as memory. Use ansible-doc -t cache -l to get a list of cache plugins installed on your system.

Lastly, cache_connection is a slightly complicated name in our case, as it doesn’t represent a connection, but instead a path to the directory that will store the cache files. In our case, this will be the digitalocean-cache/ directory in our current working directory.

These settings are also available as environment variables. You can read more about them in the plugin’s documentation page, where these are documented alongside other plugin-specific configuration settings.

Try caching out for yourself multiple times, and you’ll notice that the first invocation is quite a bit slower than later ones.

time ansible-inventory -i digitalocean.yml --graph
time ansible-inventory -i digitalocean.yml --graph

We get the same output each time, with different execution times:

all:
  |--@ungrouped:
  |  |--steampunk-cachelet
________________________________________________________
Executed in    1.04 secs   fish           external 
   usr time  308.45 millis    0.00 micros  308.45 millis 
   sys time   48.87 millis  1176.00 micros   47.69 millis 

@all:
  |--@ungrouped:
  |  |--steampunk-cachelet
________________________________________________________
Executed in  360.57 millis    fish           external 
   usr time  313.74 millis  879.00 micros  312.86 millis 
   sys time   47.91 millis   92.00 micros   47.81 millis 

Subsequent executions are three times faster! This results in a very noticeable speedup when running ad-hoc commands, as the command line is much more responsive. The test was, of course, repeated several times, and this is a representative example of the timings. We’re on quite a good and low-latency connection, with 5 milliseconds RTT to the DigitalOcean API, so any additional delays caused by a slower or unstable connection would only enlarge the difference.

Let’s now look at the contents of the cache file at digitalocean-cache/ansible_inventory_community.digitalocean.digitalocean*:

[
    {
        "backup_ids": [],
        "created_at": "2021-05-31T15:24:42Z",
        "disk": 25,
        "features": [
            "private_networking"
        ],
        "id": 8916516,
        "image": {...},
        "kernel": null,
        "locked": false,
        "memory": 1024,
        "name": "steampunk-cachelet",
        "networks": {...},
        "next_backup_window": null,
        "region": {...},
        "size": {...},
        "size_slug": "s-1vcpu-1gb",
        "snapshot_ids": [],
        "status": "active",
        "tags": [],
        "vcpus": 1,
        "volume_ids": [],
        "vpc_uuid": "a8f18t2-78ab-8168-0252-123456789123"
    }
]

The above output is a bit abbreviated for clarity, but what we see is a saved state of the inventory. Well, not quite. This is an implementation detail of the configuration plugin, which in this case, is a direct output from the API. To get the actual inventory, you need to run ansible-inventory, which will consult this file instead of querying a remote API.

How and why to invalidate the inventory cache

There are, of course, times when you need invalidate, or update, the inventory cache. A common occurrence is you adding a host in a playbook, where you need to access that additional host in subsequent plays. Let’s try to do an oopsie by modifying the playbook.

  [...]
  loop:
    - steampunk-cachelet-1
    - steampunk-cachelet-2

We’ve added another VM, leaving the previous one up. If you’ve been following the commands in this post, you’ll now get something strange when you run the inventory command again:

$ ansible-inventory -i digitalocean.yml --graph
@all:
  |--@ungrouped:
  |  |--steampunk-cachelet-1

There’s only one host! Indeed there is, and caching is to blame. To solve this quickly and with some brute force, we can just delete the caching directory, which in our case, is digitalocean-cache/.

$ rm -rf digitalocean-cache/
$ ansible-inventory -i digitalocean.yml --graph
@all:
  |--@ungrouped:
  |  |--steampunk-cachelet-1
  |  |--steampunk-cachelet-2

It works! This goes for removals as well as additions, so caching is a bit tricky to manage, since you need to be very sure that your backing state hasn’t changed.

To solve this a bit more cleanly, we can update the cache in the playbook directly with the inclusion of an Ansible meta task, which forces an inventory refresh. If you are using any new hosts in subsequent plays, you need to do this even if you are not using caching, since the inventory is only constructed at the beginning of the playbook.

- name: Refresh the inventory, whether cached or not
  meta: refresh_inventory

If you just want to have a TTL for the inventory cache, you can add the cache_timeout key to your inventory plugin configuration file. This variable contains, in seconds, how long the cache should be valid for. After this time, the cache is not valid and Ansible will automatically update your inventory from the source of truth.

[...]
# invalidate the cache after 60 minutes
cache_timeout: 3600
[...]

Of course, you can disable the cache in the configuration file. This will not invalidate it, but it will prevent Ansible from using it until you enable it again.

Caching does not come free

Inventory plugin maintainers must implement caching themselves, as this is not automatically done in the Ansible framework. If a user has caching turned on and the plugin doesn’t support it, Ansible will not show a warning but instead just disregard any caching directives that may be present.

As mentioned before, the latest stable DigitalOcean Ansible collection release has some issues with caching. Since we care about the community (and this blog post!), the Steampunk team prepared a fix for the DigitalOcean inventory plugin.

If you’d like to get assistance in developing high-quality Ansible content, you can contact XLAB Steampunk! Keep an eye out for our blog posts where we explore the deep trenches of Ansible, how to navigate them, and how to maintain best practices when writing your own collections.

If you’re into it, also sign up for our monthly newsletter to stay up-to-date with best practices and more!