Cover image for Speed up Ansible with Mitogen!

Speed up Ansible with Mitogen!

sshnaidm profile image Sergey ・4 min read

Speed up Ansible with Mitogen!

Ansible is one of most popular Configuration Management Systems nowadays, after it was acquired by Red Hat in 2015 Ansible has reached numbers of thousands of contributors and became maybe one of most used deployment and orchestration tools. Its use-cases are quite impressive.
Ansible works by SSH connections to remote hosts. It opens SSH session, logs in to the shell, copy python code via network and create a temporary file on remote hosts with this code. In the next step, it executes the current file with python interpreter. All this workflow is pretty heavy and there are multiple ways to make it faster and lighter.

One of these ways is using SSH pipelines which reuses one SSH session for copying python code of multiple tasks and prevent opening multiple sessions, which saves a lot of time. (Just don’t forget to disable requiretty settings for sudo on the remote side in /etc/sudoers)

The new way to speed up Ansible is a great python library called Mitogen. If somebody like me was not familiar with it — this library allows fast execution of python code on a remote host and Ansible is only one of its cases. Mitogen uses UNIX pipes on remote machines while passing "pickled" python code compressed with zlib. This allows to run it fast and without much traffic. If you're interested you can read details about how it works in its "How it works" page. But we'll focus today on Ansible related part of it.
Mitogen in specific circumstances can speed up your Ansible in a few times and significantly lower your bandwidth. Let's check the most popular use cases and figure out if it's helpful for us.

The most popular use cases for me to run Ansible are: creating configuration files on a remote host, packages installation, downloading and uploading files from and to a remote host. Maybe you want to check other use cases, please leave comments to this article.

Let's start rolling!
Configuring Mitogen for Ansible is pretty simple:
Install the Mitogen module:

pip install mitogen

Then either configure environment variables or set configuration options in ansible.cfg file, both options are fine:
Let's assume /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy is your path to installed Mitogen library.

export ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy
export ANSIBLE_STRATEGY=mitogen_linear


strategy = mitogen_linear
strategy_plugins = /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy

Prepare Ansible in virtualenv, with and without Mitogen enabled:

virtualenv mitogen_ansible
./mitogen_ansible/bin/pip install ansible==2.7.10 mitogen
virtualenv pure_ansible
./pure_ansible/bin/pip install ansible==2.7.10

Please pay attention that Mitogen 0.2.7 doesn't work with Ansible 2.8 (for May 2019)

Create aliases:

alias pure-ansible-playbook='$(pwd)/pure_ansible/bin/ansible-playbook'
alias mitogen-ansible-playbook='ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy:$(pwd)/mitogen_ansible/lib/python3.7/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear $(pwd)/mitogen_ansible/bin/ansible-playbook'

Now let's try the playbook that creates file on remote:

- hosts: all
  gather_facts: false
    - name: Create files with copy content module
        content: |
          test file {{ item }}
        dest: ~/file_{{item}}
      with_sequence: start=1 end={{ n }}

And run it with Mitogen and without while creating 10 files:

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m2.603s
user    0m1.152s
sys     0m0.096s

time pure-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m5.908s
user    0m1.745s
sys     0m0.643s

Right now we see improvement in x2 times. Let's check it for 20, 30, ..., 100 files:

time pure-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m51.775s
user    0m8.039s
sys     0m6.305s

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m4.331s
user    0m1.903s
sys     0m0.197s

Eventually, we improved execution time in more than 10 times!

Now let's try different scenarios and see how it improves:

  • Scenario of uploading files from the local host to remote (with copy module):
    Uploading files

  • Scenario of creating files on the remote host with copy module:
    Creating files

  • Scenario with fetching files from the remote host to local:
    Fetching files

Let's try the last scenario on a few (3) remote hosts, for example uploading files scenario:
Uploading files to multiple hosts

As we can see the Mitogen saves us both time and bandwidth in these scenarios. But if the bottleneck is not Ansible,
but for example I/O of disk or network, or somewhere else, then it's hard to expect from Mitogen to help of course.
Let's run for example packages installation with yum/dnf and python modules installation with pip.
Packages were pre-cached to avoid dependencies on network glitches:

- hosts: all
  gather_facts: false
    - name: Install packages
      become: true
          - samba
          - httpd
          - nano
          - ruby
        state: present

    - name: Install pip modules
      become: true
          - pytest-split-tests
          - bottle
          - pep8
          - flask
        state: present

With Mitogen it takes 12 seconds, as well as with pure Ansible.
In Mitogen for Ansible page you can see additional benchmarks
and measurements. As the page declares:

Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible

That's why it's important to find where your bottlenecks are and if they are related to Ansible operations, Mitogen will
help you to solve it and speed your playbooks up significantly.


Editor guide
eduardosufan profile image
Eduardo Zaqui Sufán

Hi Sergey, thanks for the tutorial. I'm trying to make it work but is not taken my ANSIBLE_STRATEGY_PLUGINS variable.

I have configure it in my playbooy as:

  • hosts: local .... strategy: mitogen_linear strategy_plugins: /usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy Here execution says "strategy_plugins" is not command valid...

And as a environment execution in my apy:

command = ['ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy', 'ANSIBLE_STRATEGY=mitogen_linear', 'ansible-playbook', '-ihosts', 'ansible_scripts/inventory.yml']
process = subprocess.Popen(command, stdout=subprocess.PIPE)

Here is not finding directory
FileNotFoundError: [Errno 2] No such file or directory: 'ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy': 'ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy'

I'm using ansible 2.7.10
And python 3.6
Do you know how I can make it work?

sshnaidm profile image
Sergey Author

Hi, Eduardo
firstly worth to check if path exists:

ls /usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy

you don't need specify 'strategy_plugins' in playbook, it's only for configuration file ansible.cfg, see my example in the article
is the path exists, just export it:

export ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.6/site-packages/ansible_mitogen/plugins/strategy

and then run your playbook

eduardosufan profile image
Eduardo Zaqui Sufán

Ok, now I see mitogen strategy working. But execution measurement time is the same with and without mitogen in my case (about 30min execution).
Do you know in which cases time execution will be very similar between strategies?

PD: right now I'm deploying just 1 host, a college says to me maybe I can find improvements with 2 or more hosts.

Thread Thread
sshnaidm profile image
Sergey Author

Yes, if ansible is not your bottleneck and you have much more long tasks, you can't see the improvement really, that's what I'm writing in the article in last example.