Become-ing a user securely in Ansible

Using become_user in Ansible requires the acl package to be installed on the remote host. This is a note to my future self, reminding me of the error, the root cause, and the solution. Maybe it helps someone else as well. 🙂

Running into a familiar problem

Once upon a time, one could use become_user in an Ansible task, and things would “just work”™. It turns out that Ansible wasn’t doing this as securely as it could have. These days, Ansible sets (or tries to set) the appropriate access controls on uploaded files and directories. Legacy Ansible tasks will therefore run into errors if the acl package isn’t installed on the remote host.

So why is this interesting, and how did I get here? Well, while updating some old Ansible playbooks for a client recently,¹ I ran across a problem that I’d seen before, but had never written up properly. This time, I decided to make notes for my future self and anyone else who might need to know about this problem and its solution.

Ansible is a great tool for provisioning and system configuration. The fact that I can define a system’s configuration state with something akin to code, which I can then keep in Git, is brilliant. It dovetails well with my needs as a programmer and sysadmin.

I’ve been using Ansible for a few years now, and sometimes it shows. Some of my playbooks don’t use current best practices, which is odd for me, because I’m usually really picky about adhering to best practices. Such things help teams work together more easily because there are fewer “surprises” in code.

But if you don’t touch things very often, stuff gets outdated. Thus, playbooks and tasks that once ran flawlessly no longer work. Here is the instance that struck me most recently:

- name: install base requirements
  ansible.builtin.pip:
      requirements: "{{ project_dir }}/base.txt"
      virtualenv: "{{ venv_path }}"
      virtualenv_command: "virtualenv --python=/usr/bin/python3"
  become_user: <functional_user>

This task uses pip to install the base requirements for a Python application and ensures that a virtual environment has been set up in advance.² The task also does this using a functional user account. A functional user is a user account that you create for the sole purpose of running the application. We’re not in the 90s anymore, and hence we don’t want to run our services as root or some other privileged user. Having a functional user account is nice because it encapsulates an application’s code and configuration into one location. Of course, this all assumes that the application isn’t containerised, something that one runs across often when dealing with legacy software.

Running the playbook containing this task, I got the following error message:

fatal: [old-fashioned-test-system]: FAILED! => {"msg": "Failed to set
permissions on the temporary files Ansible needs to create when becoming an
unprivileged user (rc: 1, err: chown: changing ownership of
'/var/tmp/ansible-tmp-1763739956.7741241-2042672-91787264687983/': Operation
not permitted\nchown: changing ownership of
'/var/tmp/ansible-tmp-1763739956.7741241-2042672-91787264687983/AnsiballZ_pip.py':
Operation not permitted\n}). For information on working around this, see
https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}

I’d seen similar messages a couple of times in the past and had re-solved the underlying problem each time. So, it was high time I documented everything more thoroughly.

Understanding the root cause

So what does this error message mean, and what should we do to fix the situation? An obvious thing to try would be to look up the documentation link mentioned in the error message. Unfortunately, that link location no longer exists. Fortunately, the Internet Archive has our back and has kept a copy for us. The bit that we’re interested in is the Becoming an Unprivileged User section.

The gist of the problem is that, for modules where pipelining isn’t possible, Ansible needs to set the access control flags on the temporary file that it sends to the remote host. After all, we don’t want every Tom, Dick and Harry being able to see what files Ansible is shuffling around. By installing the acl package, we ensure that the setfacl command exists on the remote host. Ansible then uses setfacl to set file-level access controls. This is more secure than the previous behaviour.

For me, a big signal here is that I need to upgrade my Ansible installation. This will, in turn, require me to upgrade my Debian installation, which requires planning and a lot of time. Hence, upgrading Ansible isn’t going to happen in a hurry. At least, not right now. :-/ But, as fellow Kiwi Rachel Hunter once said, “It won’t happen overnight, but it will happen”. 🙂

So yeah, I have to confess, I’m currently using an ancient Ansible version:

$ ansible --version
ansible 2.10.17
  config file = /<project_path>/playbooks/ansible.cfg
  configured module search path = ['/<home_path>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]

And yes, I’m still on Debian bullseye. sigh. It can be hard to upgrade an operating system while also getting things done, all right?

A solution and its nuances

Ok, back to the story at hand. As mentioned a couple of times, the solution to the problem above is to install the acl package. That turns out to be really easy. Install the package with the appropriate OS package manager module.³ You’ll need a task that looks something like this:

- name: install general packages
  ansible.builtin.apt:
    name: [
      'acl',
      .... other OS packages
    ]

Simply installing acl isn’t necessarily sufficient, however. After having installed the setfacl command, you might find that you run into the next security-related problem: a missing temporary directory on the remote host that has the correct permissions for the functional user. In other words, you might get a warning like this one:

[WARNING]: Module remote_tmp /home/<functional_user>/.ansible/tmp did not exist and was created with a mode of
0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with
the correct permissions manually

The solution to this part of the problem is to create the appropriate remote temporary directory for the functional user account. The best place to put this task is straight after that which created the functional user, e.g.:

# avoid warnings tmp dir being created with incorrect permissions
- name: create the Ansible temporary dir for the <functional> user
  ansible.builtin.file:
    path: /home/<functional>/.ansible/tmp
    owner: <functional>
    group: <functional>
    mode: '0700'
    state: directory

where I’ve left the value <functional> as a placeholder for the actual functional user account name.

Note that the task requiring become_user will also need become: True so that the become process works as expected.⁴ Thus, you might need to update the task to include this parameter as well as the other changes mentioned here.

So, the original task to install the app’s base requirements with pip, erm, becomes:⁵

- name: install base requirements
  ansible.builtin.pip:
      requirements: "{{ project_dir }}/base.txt"
      virtualenv: "{{ venv_path }}"
      virtualenv_command: "{{ ansible_python.executable }} -m venv"
  become: True
  become_user: <functional_user>

where I’ve fixed the virtualenv_command to replace virtualenv with (effectively) python3 -m venv to create the virtual environment.

Important: On Debian/Ubuntu, it’s also necessary to install the python3-venv package for the venv module to be available. Thus, you’ll need a task which runs beforehand that looks like this one:

- name: install dependencies for <app>
  ansible.builtin.apt:
      name: [
          'python3-pip',
          'python3-venv',
          ... other packages ...
      ]
      update_cache: yes
      cache_valid_time: 3600

I think that’s got most of the t’s dotted and i’s crossed!

Time-travelling love letters

This blog post is a love letter to my future self so that I find the solution more easily should I stumble across this issue again. Hullo, future me! 👋

As a side note, it was pretty cool to find that I’d put lots of detail into commit messages in the repo containing my Ansible configuration. For example:

    Create an Ansible tmp dir for <functional> user

    Some Ansible processes copy files to the remote system and if they use
    `become_user` then a new temporary directory is created to handle the
    files in that case. When this happens, the `remote_tmp` module gives a
    warning that a `tmp` dir is created with particular permissions and this
    might be a Bad Thing (TM) in certain conditions and recommends creating
    the temporary directory explicitly.

This (and other related commit messages) made piecing together the ideas outlined in this article much less onerous. Having described things now all in one place should make fixing everything in the future much easier.

Thank you, past me, for taking the time to do that!

1. If you need a persistent, thorough, and experienced software developer with Linux and DevOps experience, give me a yell! I’m available for freelance Python/Perl backend development and maintenance work. Contact me at [e-mail] and let’s discuss how I can help solve your business’ hairiest problems. ↩

2. Come to think of it, the fact that I use virtualenv here rather than the venv module from Python’s standard library tells me that there’s more here that I need to modernise. ↩

3. Since I’m on Debian, I’m using apt. If you’re on another Linux distribution, you’ll have to use an appropriate other package manager. ↩

4. ansible-lint, for example, will tell you this. ↩

5. Future me might be wondering why I used ansible_python.executable when the pip module documentation clearly states that one should use {{ ansible_python_interpreter }}. It turns out that only the {{ ansible_python }} dict was available in my configuration (i.e. in the Ansible facts), hence the need to use {{ ansible_python.executable }}. This variable pointed to /usr/bin/python3, which I didn’t want to hard-code in the task. Hopefully, future me will understand! ↩