Upgrading Salt to Python 3

As of January 1st, 2020, Python 2 has officially reached the end of life and will no longer receive any updates (except the last one in April). Various Linux distributions will support it for a while; however, many organizations already switched to Python 3 or plan to do so in the near future.

It is time to switch your Salt install to Python 3:

• Salt Nitrogen (2017.7.0) is the first release that supported Py3
• Salt Oxygen (2018.3.0) gained significant Py3 improvements
• Salt Fluorine (2019.2.0) can be considered more-or-less mature
• Salt 3001 Sodium (the one after Salt 3000 Neon) is going to deprecate Python 2.7

This guide describes how to switch the repo from Py2 to Py3, identify and uninstall the old packages, then install the Py3 version. It contains detailed checklists that will help you to plan and perform the migration. It can be also applied to regular upgrades. However, it is not a 100% automatic foolproof recipe you can apply blindly.

• 10 key principles
• Plan the upgrade process
• Survey your master and minion fleet
• Upgrade the Salt Master
• Upgrade Salt on Windows
• Upgrade Salt on Ubuntu and Debian
• Upgrade Salt on CentOS and Amazon Linux
• Upgrade Salt on other operating systems
• Monitor the upgrade process
• Upgrade automation
• Future developments

10 key principles

Over the years of using Salt, I developed a set of ten opinionated guiding principles that will help you maintain and upgrade it. You do not have to follow these guidelines, and I often break them too. However, they are implied throughout the article.

1. Have a QA environment

QA environment can be manual or automated. The goal is to be able to test the required combinations of Operating Systems, Salt versions (master & minion), and possible highstates. Always do staged/gradual rollouts. You’ll save a lot of time if your QA environment supports snapshots.

2. Run the same Salt version everywhere

Master and minion versions should be the same (the only exceptions are [1] and [5]). This is the most traveled path. You won’t need to adapt your states to different feature sets. Also, the upgrade process is much easier.

3. Avoid rare operating modes

Less traveled paths and settings that significantly change how Salt operates (transport, multiprocessing, zmq filtering, caching, undocumented things), tend to result in nasty surprises. If you want to change a setting, give it a try in your QA environment for a while.

4. Control the bootstrap/upgrade process

• Do not use the “latest” repo prefix (except in a QA environment), and do not rush to upgrade to a latest major version
• Use the “archive” repo prefix, or pin Salt packages down to a minor version (you never know when another 2019.2.1 will happen)
• Always install specific version when you bootstrap new minions (e.g., through the script_args Salt Cloud option)

Yes, a minor upgrade will require more work. But it is work you can control and decide when to do. You can’t control work that appears out of the blue when minions automatically upgrade and something breaks.

5. Upgrade the master first

When you upgrade Salt, upgrade the master first. Older master can (and eventually will) break things.

6. Back up the keys

Set up periodic backups of the /etc/salt/pki master directory. Other than that, your Salt master should be disposable, and states/pillars should be kept in a version control system (or pulled from external data sources).

7. Have a backup access path

Have a plan B to access your servers when Salt goes down. Use salt-ssh, fabric, ansible, ssh, winrm, or any other tool. It is also helpful to keep a list of minion IPs (you can periodically extract them from the master grain cache).

8. Use Salt as an O&M control plane

Avoid making your production workload depend on Salt. If you uninstall Salt, your production environment should remain operational (at least for a while).

If you have the resources, you may also want to consider the last two principles:

9. Mirror the Salt repo

I never had to do this, but I’ve seen rare connectivity issues with the official repo. Also, for some folks, the minion bootstrap process was broken when SaltStack removed (and then quickly returned back) the buggy 2019.2.1 release. This is less of an issue if you follow the principle [4].

10. Build your own packages

Bugs will happen, and you need to fix them quickly without waiting for the next release. Also, you may want to integrate new features before they are shipped. Custom packages are a way to achieve that. They give you full control over the Salt code you run.

I ran a patched salt-master from a virtualenv for some time in 2012 or 2013, but I no longer do that. Instead, I prefer the more lightweight ways to patch Salt.

Plan the upgrade process

1. Survey your master and minion fleet

A list of useful commands is provided below. There are many dimensions you should inspect:

• Operating systems, their architectures, and versions
• Available repositories
• Python versions
• Master and minion versions
• Salt dependencies

The main goal is to find all the existing combinations, so you can exhaustively test the upgrade process. Also check out the Py3 support list.

2. Prepare the QA environment

Ideally, you want to be able:

1. Test the same OS/Salt versions
2. Test the existing highstates
3. Test the actual workload that runs on top of your infra

You’ll need to test the upgrade process multiple times, so a fast feedback cycle is necessary (e.g., it would be nice to quickly snapshot/restore your QA environment).

3. Update the states

Make sure your states are compatible with the target Salt version. Read the official release notes and search for any open issues.

4. Decide on the upgrade process

For example:

1. Perform the backups (master/minion keys, alternate connection paths, etc.)
2. Upgrade OS packages, run autoclean/autoremove. Optionally reboot if you have new kernels.
3. Disable any periodic Salt jobs and other stuff that can interfere with the upgrade process. Try minion blackouts
4. Upgrade master to the target version, while staying on Py2
5. Ensure that the existing process to bootstrap new minions uses the target Salt version (Py3)
6. Upgrade minions to the target version, while staying on Py2. If that is not possible, upgrade to the latest available for each distro.
7. Upgrade Salt master to Py3
8. Upgrade Salt minions to Py3 for each OS separately

Consider the minion upgrade order. Your environment may require some specific upgrade sequence.

The goal is to remove the old Salt version and its dependencies (to avoid having packages from different repos), then bootstrap the Py3 one from scratch.

If your minions are disposable, you can avoid the in-place upgrade process and instead rebuild them from scratch. Another option is blue/green deployment.

5. Test the upgrade process multiple times

1. Manually upgrade your masters
2. Make sure their minions are reachable after the upgrade
3. Manually upgrade a single minion for each OS
4. Make sure the upgraded minions are reachable, the highstates can be applied, and the business apps are functional
5. Automate the above steps
6. Test the automation multiple times
7. Test the backup options (master/minion keys, alternate connection paths, etc.)
8. Test the bootstrap process for new minions

6. Do the upgrade

1. Decide how many upgrade stages you want
2. Decide on the appropriate schedule
3. Inform the team(s)
4. Perform and verify the backups (master/minion keys, alternate connection paths, etc.)
5. Switch the bootstrap for new minions to the target version
6. Upgrade the master
7. Upgrade the minions
8. Monitor the result

Survey your master and minion fleet

Listing versions

% sudo salt '*' test.version

% sudo salt '*' pkg.version salt-minion

% sudo salt '*' test.versions_report

(you may find a lot of variation in the installed dependencies!)

% sudo salt '*' grains.item pythonversion

% sudo salt-run manage.versions

Master:
    2019.2.3
Minion requires update:
    ----------
    minion1:
        2018.3.5
    minion2:
        2018.3.5
    minion3:
        2018.3.5
    win10:
        2018.3.5

sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion1
      - minion2
      - minion3
      - win10
  result:
      2018.3.5

Listing repos and keys

% sudo salt '*' pkg.list_repos
% sudo salt '*' pkg.get_repo_keys

Combine with jq to get more compact output:

% sudo salt -G "os:Ubuntu" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main"
"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2019.2.3 bionic main"

% sudo salt -G "os:Debian" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main"
"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.3 stretch main"

% sudo salt -G "os:CentOS" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[].baseurl'

"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2018.3.5/"
"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2019.2.3/"

Targeting py2 minions

% sudo salt -C "G@os:Ubuntu and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Debian and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:CentOS and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Windows and G@pythonversion:0:2" grains.item pythonversion saltversion

Other useful grains for targeting

• cpuarch
• lsb_distrib_codename
• lsb_distrib_id
• lsb_distrib_release
• os
• osarch
• oscodename
• osfinger
• osmajorrelease
• osrelease
• osversion
• pythonexecutable
• pythonversion
• saltpath
• saltversion

Surveys

Any module calls (test.version, cmd.run, grains.item, etc.) can be combined with the survey runner:

sudo salt-run survey.hash '*' grains.item os osfinger pythonversion saltversion
|_
  ----------
  pool:
      - minion3k
  result:
      {u'osfinger': u'Ubuntu-18.04', u'saltversion': u'2018.3.5', u'os': u'Ubuntu', u'pythonversion': [2, 7, 17, u'final', 0]}
|_
  ----------
  pool:
      - minion2
  result:
      {u'osfinger': u'CentOS Linux-7', u'saltversion': u'2018.3.5', u'os': u'CentOS', u'pythonversion': [2, 7, 5, u'final', 0]}
|_
  ----------
  pool:
      - minion1
  result:
      {u'osfinger': u'Debian-9', u'saltversion': u'2018.3.5', u'os': u'Debian', u'pythonversion': [2, 7, 13, u'final', 0]}
|_
  ----------
  pool:
      - win10
  result:
      {u'osfinger': u'Windows-10', u'saltversion': u'2018.3.5', u'os': u'Windows', u'pythonversion': [2, 7, 14, u'final', 0]}

Listing package origins

The goal is to find packages that were installed from a specific origin, so that you can remove them during the upgrade process.

Ubuntu/Debian

I found the following snippet on Ask Ubuntu. If you know a more straightforward way to do the same without installing aptitude or other packages, I’m all ears.

% apt-cache policy $(dpkg -l | awk 'NR >= 6 { print $2 }') |
  awk '/^[^]/ { split($1, a, ":"); pkg = a[1] }
    nextline == 1 { nextline = 0; printf("%-40s %-50s %s\n", pkg, $2, $3) }
    /\*\*\*/ { nextline = 1 }' | grep saltstack

The output can vary depending on the OS:

salt-common https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
salt-minion https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main

python-jinja2 https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-common https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-minion https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main

Additionally, you can inspect how a Salt minion was installed in the first place. You need to figure out which packages are installed explicitly (the rest of the dependencies can be auto removed). Check out the /var/log/apt/history.log:

Debian:

Start-Date: 2020-02-18 05:15:02
Commandline: apt-get install -y -o DPkg::Options::=--force-confold procps pciutils python-yaml
Requested-By: vagrant (1000)
Install: python-yaml:amd64 (3.12-1), libyaml-0-2:amd64 (0.1.7-2, automatic)
End-Date: 2020-02-18 05:15:02

Start-Date: 2020-02-18 05:15:05
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg2 apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: gnupg2:amd64 (2.1.18-8~deb9u4), apt-transport-https:amd64 (1.4.9)
End-Date: 2020-02-18 05:15:06

Start-Date: 2020-02-18 05:15:35
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion
Requested-By: vagrant (1000)

Ubuntu:

Start-Date: 2020-02-18 05:15:08
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: apt-transport-https:amd64 (1.6.12)
End-Date: 2020-02-18 05:15:08

Start-Date: 2020-02-18 05:15:49
Commandline: apt-get install -y -o DPkg::Options::=--force-confold python2.7 python-apt python-requests python-yaml procps pciutils
Requested-By: vagrant (1000)
Install: python-six:amd64 (1.11.0-2, automatic), python-openssl:amd64 (17.5.0-1ubuntu1, automatic), python-yaml:amd64 (3.12-1build2), python-requests:amd64 (2.18.4-2ubuntu0.1), python-certifi:amd64 (2018.1.18-2, automatic), python-chardet:amd64 (3.0.4-1, automatic), python-enum34:amd64 (1.1.6-2, automatic), python-cryptography:amd64 (2.1.4-1ubuntu1.3, automatic), python-cffi-backend:amd64 (1.11.5-1, automatic), python-ipaddress:amd64 (1.0.17-1, automatic), python-pkg-resources:amd64 (39.0.1-2, automatic), python-apt:amd64 (1.6.5ubuntu0.2), python-urllib3:amd64 (1.22-1ubuntu0.18.04.1, automatic), python-idna:amd64 (2.6-1, automatic), python-asn1crypto:amd64 (0.24.0-1, automatic)
Upgrade: python2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), python2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-stdlib:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04)
End-Date: 2020-02-18 05:15:56

Start-Date: 2020-02-18 05:16:11
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion

The bootstrap-salt.sh script also logs everything to /tmp/bootstrap-salt.log:

pciutils is already the newest version (1:3.5.2-1).
procps is already the newest version (2:3.3.12-3+deb9u1).
...
Selecting previously unselected package python-yaml.
Preparing to unpack .../python-yaml_3.12-1_amd64.deb ...
Unpacking python-yaml (3.12-1) ...
Setting up libyaml-0-2:amd64 (0.1.7-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python-yaml (3.12-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20161130+nmu1+deb9u1).
ca-certificates set to manually installed.
wget is already the newest version (1.18-5+deb9u3).
The following NEW packages will be installed:
  apt-transport-https gnupg2
...
The following additional packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common
Suggested packages:
  debtags dbus-user-session pinentry-gnome3 tor apache2 | lighttpd | httpd
  python-apt-dbg python-apt-doc python-cryptography-doc
  python-cryptography-vectors python-enum34-doc python-jinja2-doc
  python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
  python-socks python-setuptools-doc python-mysqldb python-pycurl
  python-tornado-doc python-twisted python-ntlm python-augeas
Recommended packages:
  sfdisk e2fprogs
The following NEW packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common salt-minion

So, for Debian/Ubuntu you may want to remove the salt-minion, python-requests, python-yaml, and python-apt explicitly, then auto remove the dependencies.

CentOS/Amazon

% yum list installed | grep saltstack

PyYAML.x86_64 3.11-1.el7 @saltstack
python-zmq.x86_64 15.3.0-3.el7 @saltstack
salt.noarch 2018.3.5-1.el7 @saltstack
salt-minion.noarch 2018.3.5-1.el7 @saltstack
zeromq.x86_64 4.1.4-7.el7 @saltstack

The /var/log/yum.log is not very informative:

Feb 18 08:15:37 Updated: rpm-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:37 Updated: rpm-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-build-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-python-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Installed: python-chardet-2.2.1-3.el7.noarch
Feb 18 08:15:38 Installed: python-kitchen-1.1.1-5.el7.noarch
Feb 18 08:15:38 Installed: libyaml-0.1.4-11.el7_0.x86_64
Feb 18 08:15:38 Updated: libxml2-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Installed: libxml2-python-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Updated: python-urlgrabber-3.10-9.el7.noarch
Feb 18 08:15:39 Updated: yum-3.4.3-163.el7.centos.noarch
Feb 18 08:15:39 Installed: yum-utils-1.1.31-52.el7.noarch
Feb 18 08:15:39 Installed: PyYAML-3.11-1.el7.x86_64
Feb 18 08:15:39 Updated: chkconfig-1.7.4-1.el7.x86_64
Feb 18 08:16:15 Installed: lz4-1.7.5-3.el7.x86_64
Feb 18 08:16:15 Updated: systemd-libs-219-67.el7_7.3.x86_64
Feb 18 08:16:15 Installed: python-ipaddress-1.0.16-2.el7.noarch
Feb 18 08:16:15 Installed: python-markupsafe-0.11-10.el7.x86_64
Feb 18 08:16:15 Installed: python-six-1.9.0-2.el7.noarch
Feb 18 08:16:15 Installed: libtommath-0.42.0-6.el7.x86_64
Feb 18 08:16:15 Installed: libtomcrypt-1.17-26.el7.x86_64
Feb 18 08:16:16 Installed: python2-crypto-2.6.1-16.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-1.0-8.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
Feb 18 08:16:16 Installed: python-tornado-4.2.1-5.el7.x86_64
Feb 18 08:16:16 Installed: python-urllib3-1.10.2-7.el7.noarch
Feb 18 08:16:16 Installed: python-requests-2.6.0-8.el7_7.noarch
Feb 18 08:16:17 Installed: python-babel-0.9.6-8.el7.noarch
Feb 18 08:16:18 Installed: python-jinja2-2.7.2-4.el7.noarch
Feb 18 08:16:18 Installed: libsodium-1.0.18-1.el7.x86_64
Feb 18 08:16:18 Installed: python2-msgpack-0.5.6-5.el7.x86_64
Feb 18 08:16:18 Updated: cryptsetup-libs-2.0.3-5.el7.x86_64
Feb 18 08:16:20 Updated: systemd-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: systemd-python-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: openpgm-5.2.122-2.el7.x86_64
Feb 18 08:16:20 Installed: zeromq-4.1.4-7.el7.x86_64
Feb 18 08:16:21 Installed: python-zmq-15.3.0-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-futures-3.1.1-5.el7.noarch
Feb 18 08:16:21 Updated: pciutils-libs-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: pciutils-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-psutil-2.2.1-5.el7.x86_64
Feb 18 08:16:26 Installed: salt-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Installed: salt-minion-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Updated: systemd-sysv-219-67.el7_7.3.x86_64
Feb 18 08:16:26 Updated: libgudev1-219-67.el7_7.3.x86_64

However, the /tmp/bootstrap-salt.log shows precisely which packages were requested and which are merely dependencies:

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 PyYAML x86_64 3.11-1.el7 saltstack 160 k
 yum-utils noarch 1.1.31-52.el7 base 121 k
Updating:
 chkconfig x86_64 1.7.4-1.el7 base 181 k
Installing for dependencies:
 libxml2-python x86_64 2.9.1-6.el7_2.3 base 247 k
 libyaml x86_64 0.1.4-11.el7_0 base 55 k
 python-chardet noarch 2.2.1-3.el7 base 227 k
 python-kitchen noarch 1.1.1-5.el7 base 267 k
Updating for dependencies:
 libxml2 x86_64 2.9.1-6.el7_2.3 base 668 k
 python-urlgrabber noarch 3.10-9.el7 base 108 k
 rpm x86_64 4.11.3-40.el7 base 1.2 M
 rpm-build-libs x86_64 4.11.3-40.el7 base 107 k
 rpm-libs x86_64 4.11.3-40.el7 base 278 k
 rpm-python x86_64 4.11.3-40.el7 base 83 k
 yum noarch 3.4.3-163.el7.centos base 1.2 M

Transaction Summary
================================================================================
Install 2 Packages (+4 Dependent packages)
Upgrade 1 Package (+7 Dependent packages)

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 salt-minion noarch 2018.3.5-1.el7 saltstack 37 k

Below is a similar log for Amazon Linux:

Installing:
 m2crypto x86_64 0.31.0-4.el7 saltstack-repo 287 k
 python-crypto x86_64 2.6.1-2.el7 saltstack-repo 470 k
 python-msgpack x86_64 0.4.6-1.el7 saltstack-repo 73 k
 python-zmq x86_64 15.3.0-3.el7 saltstack-repo 520 k
Updating for dependencies:
 PyYAML x86_64 3.11-1.el7 saltstack-repo 160 k
 procps-ng x86_64 3.3.10-26.amzn2 amzn2-core 292 k
 python-requests noarch 2.6.0-7.amzn2 amzn2-core 95 k
Installing for dependencies:
 libsodium x86_64 1.0.16-1.el7 saltstack-repo 140 k
 openpgm x86_64 5.2.122-2.el7 saltstack-repo 172 k
 python2-typing noarch 3.5.2.2-4.el7 saltstack-repo 39 k
 zeromq x86_64 4.1.4-7.el7 saltstack-repo 555 k

So, for CentOS/Amazon you may want to remove the salt-minion, PyYAML, m2crypto, python-zmq, python-crypto, and python-msgpack explicitly, then auto remove the dependencies.

Listing dependencies using pip

If you installed some Salt dependencies via pip, it is worth inspecting them too:

% sudo salt -C "G@kernel:Linux" cmd.run 'pip freeze --local'

% sudo salt -t 30 -C "G@os:Windows" cmd.run 'pip.exe freeze --local' cwd='C:\salt\bin\scripts\'

On CentOS, Amazon, and Windows, the command shows a full list of python packages, so you’ll need to compare it with a default one to find what is different.

Upgrade the Salt Master

From the FAQ:

Q: Can I run different versions of Salt on my Master and Minion?

A: This depends on the versions. In general, it is recommended that Master and Minion versions match.When upgrading Salt, the master(s) should always be upgraded first. Backwards compatibility for minions running newer versions of Salt than their masters is not guaranteed.

There are two common ways to upgrade a Salt master:

Option 1

The in-place upgrade process:

1. Backup the /etc/salt/pki directory (just in case)
2. Remove the old Salt repo
3. Uninstall the old salt-master package and all of its dependencies
4. Add the new repo (py3)
5. Install the new salt-master

Option 2

If you can’t disconnect all minions at once during the upgrade process, it is possible to bring up an additional master server and migrate minions one-by-one:

1. Set up new salt-master (py3)
2. Copy the /etc/salt/pki directory from old master to the new one, to ensure that master and minion keys are the same
3. Migrate the minions:

# migrate_minion.sls
migrate_to_new_master:
  file.replace:
    - name: "{{ salt['config.get']('conf_file') }}"
    # Alternative variant if you keep master address in a separate config file
    # - name: "{{ salt['file.join'](salt['config.get']('config_dir'), 'minion.d', 'master.conf') }}"
    - pattern: '^master:\s+{{ salt["config.get"]("master") | regex_escape }}$'
    - repl: 'master: new-master.example.com'

# Restart salt minion after the the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltstack.com/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: migrate_to_new_master

To be safe, you can migrate minions in batches:

% sudo salt -G 'master:old-master.example.com' --batch-size 10% state.apply migrate_minion

Upgrade Salt on Windows

With winrepo

Things are very easy in Windows land if you use winrepo-ng:

1. Update your winrepo definitions

If you do not want to bring the whole winrepo-ng, just put the following two files into salt://win/repo-ng:

• salt-minion-py2.sls
• salt-minion-py3.sls

2. Rebuild the package database

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.refresh_db

3. List the currently installed version

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.version salt-minion

3. Make sure the target version is available

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.list_available salt-minion-py3

4. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.install salt-minion-py3 version=2019.2.3

You might want to add the --batch-size X argument for a staged rollout.

Without winrepo

Do not want to use winrepo at all? Below is an alternative way to upgrade Windows minions.

1. Fetch the installer

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

You can also include a checksum to prevent redownloads (works in Salt 2018.3.0 or greater):

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe \
  source_hash=$(curl -s http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe.sha256 |\
  iconv -f utf-16 -t utf-8 | awk '{print $1;}' | tr A-Z a-z)

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

The default download location is fine, because the Salt installer doesn’t remove the cache dir since 2017.7.2.

2. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" cmd.run \
  'START /B c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe /S'

Upgrade Salt on Ubuntu and Debian

There are a few key pieces:

1. Remove the old repo and key

remove_salt_repo:
  pkgrepo.absent:
    - name: deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main
# - name: deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main
    - keyid: 0E08A149DE57BFBE

It is also possible to delete all existing Salt repos and keys on both Ubuntu and Debian:

{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

apt-get -y remove --auto-remove salt-minion
apt-get -y remove --auto-remove python-requests python-yaml python-apt

4. Install the minion

While it is possible to just apt-get install the necessary packages:

apt-get -y -o DPkg::Options::=--force-confold install python3-requests python3-yaml python3-apt
apt-get -y -o DPkg::Options::=--force-confold install salt-minion

I feel that using salt-bootstrap.sh is simpler and supports more platforms:

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3

Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.

5. The resulting state

# upgrade/ubuntu_debian.sls
{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
    # - keyid: 0E08A149DE57BFBE
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

install_at:
  pkg.installed:
    - name: at

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        apt-get -y remove --auto-remove salt-minion
        apt-get -y remove --auto-remove python-requests python-yaml python-apt
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

6. Run the upgrade command

% sudo salt -C "( G@os:Ubuntu or G@os:Debian ) and G@pythonversion:0:2" state.apply upgrade.ubuntu_debian

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on CentOS and Amazon Linux

There are a few key pieces:

1. Remove the old repo

remove_salt_repo:
  pkgrepo.absent:
    - name: saltstack

It is also possible to delete all existing Salt repos on any rpm-based distro:

{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

yum -y remove salt-minion
yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
yum -y autoremove

4. Install the minion without starting it

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3

Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.

5. Reuse the old config

The snippet below should be adapted to your specific environment:

[-f /etc/salt/minion.rpmsave] && \
( cp -f /etc/salt/minion /etc/salt/minion.bak; \
mv -f /etc/salt/minion.rpmsave /etc/salt/minion )

6. Start the salt-minion service

systemctl is-enabled salt-minion.service || \
(systemctl preset salt-minion.service && \
systemctl enable salt-minion.service)
systemctl start salt-minion.service

7. The resulting state

# upgrade/centos_amazon.sls
{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

install_at:
  pkg.installed:
    - name: at

atd_service:
  service.running:
    - name: atd
    # - enable: True

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        yum -y remove salt-minion
        yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
        yum -y autoremove
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3
        [-f /etc/salt/minion.rpmsave] && \
        ( cp -f /etc/salt/minion /etc/salt/minion.bak; \
        mv -f /etc/salt/minion.rpmsave /etc/salt/minion )
        systemctl is-enabled salt-minion.service || \
        (systemctl preset salt-minion.service && \
        systemctl enable salt-minion.service)
        systemctl start salt-minion.service" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

8. Run the upgrade command

% sudo salt -C "( G@os:CentOS or G@os:Amazon ) and G@pythonversion:0:2" state.apply upgrade.centos_amazon

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on other operating systems

If you have done this on any other operating system (Red Hat, Fedora, SUSE, Gentoo, Arch, Raspbian, macOS, Solaris, FreeBSD, OpenBSD, etc.), feel free to share your experience and possible gotchas. I’ll be happy to update the article with any specific recipes!

Monitor the upgrade process

The upgrade process can take some time. There are three ways to monitor minions as they reconnect:

Watch the master log

% sudo tail -f /var/log/salt/master

Watch the master event bus

% sudo salt-run state.event tagmatch='salt/minion/*/start' pretty=true

List the Py2/Py3 minion count

% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0]).\(.[1]).\(.[2])"' | sort | uniq -c

      1 "2.7.13"
      1 "2.7.17"
      1 "2.7.5"
      1 "3.5.3"
      1 "3.6.8"
      1 "3.6.9"

% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0])"' | sort | uniq -c

      3 "2"
      3 "3"

Upgrade automation

If you need more control over the upgrade process (handle retries and upgrade failures, define specific orchestration sequence, etc.) you can use Salt orchestration or preferably write a custom runner. Another variant is to use salt-api and pepper.

An out-of-band upgrade using fabric or salt-ssh is also an option.

Future developments

In 2019 SaltStack introduced salt-bin and pop-build projects to make self-contained Salt binaries. Hopefully, they will simplify the dependency handling and make the upgrade process easier.

Need to run different Salt versions simultaneously?

The next post will explain how to write backward-compatible Salt states (e.g., to deal with module.run syntax differences, or for public formulas). Subscribe to read it as soon as the draft is done.

Comments

[mrx]

this seems like a crazy complicated upgrade path

I install salt master and agents using pip,

heres my bootstrap script,

#!/bin/bash
# salt minion installer
export PYTHONIOENCODING=utf8
VENVPATH="/opt/salt"

# get latest py3 version
[ -f /bin/python3 ] && PYPATH=/bin/python3
[ -f /bin/python3.6 ] && PYPATH=/bin/python3.6
[ -f /bin/python3.7 ] && PYPATH=/bin/python3.7
[ -f /bin/python3.8 ] && PYPATH=/bin/python3.8
[ -f /bin/python3.9 ] && PYPATH=/bin/python3.9
[ -f /bin/python3.10 ] && PYPATH=/bin/python3.10

[ -z "${PYPATH}" ] && { echo "No python3 detected, exiting"; exit 1; }

# upgrade pip
$PYPATH -m pip install --upgrade pip

# create venv
[ -d "${VENVPATH}/bin" ] || { cd "/opt"; $PYPATH -m venv salt; }

# install pkgs
[ -f "${VENVPATH}/bin/salt" ] || /opt/salt/bin/pip3 install salt pyinotify dictor

ln -sf $VENVPATH/bin/salt-minion /usr/bin/salt-minion
ln -sf $VENVPATH/bin/salt-call /usr/bin/salt-call

then if i need to upgrade, just do pip install -U salt-minion on each host

the entire thing is in a virtualenv which can be wiped out and recreated easily

[Yoshy]

Un grand merci pour cette ressource fort utile, du moins pour moi et mon projet actuel :)