DEV Community: Max Arnold

Upgrading Salt to Python 3

Max Arnold — Wed, 19 Feb 2020 18:36:22 +0000

As of January 1st, 2020, Python 2 has officially reached the end of life and will no longer receive any updates (except the last one in April). Various Linux distributions will support it for a while; however, many organizations already switched to Python 3 or plan to do so in the near future.

It is time to switch your Salt install to Python 3:

Salt Nitrogen (2017.7.0) is the first release that supported Py3
Salt Oxygen (2018.3.0) gained significant Py3 improvements
Salt Fluorine (2019.2.0) can be considered more-or-less mature
Salt 3001 Sodium (the one after Salt 3000 Neon) is going to deprecate Python 2.7

This guide describes how to switch the repo from Py2 to Py3, identify and uninstall the old packages, then install the Py3 version. It contains detailed checklists that will help you to plan and perform the migration. It can be also applied to regular upgrades. However, it is not a 100% automatic foolproof recipe you can apply blindly.

10 key principles
Plan the upgrade process
Survey your master and minion fleet
Upgrade the Salt Master
Upgrade Salt on Windows
Upgrade Salt on Ubuntu and Debian
Upgrade Salt on CentOS and Amazon Linux
Upgrade Salt on other operating systems
Monitor the upgrade process
Upgrade automation
Future developments

10 key principles

Over the years of using Salt, I developed a set of ten opinionated guiding principles that will help you maintain and upgrade it. You do not have to follow these guidelines, and I often break them too. However, they are implied throughout the article.

1. Have a QA environment

QA environment can be manual or automated. The goal is to be able to test the required combinations of Operating Systems, Salt versions (master & minion), and possible highstates. Always do staged/gradual rollouts. You’ll save a lot of time if your QA environment supports snapshots.

2. Run the same Salt version everywhere

Master and minion versions should be the same (the only exceptions are [1] and [5]). This is the most traveled path. You won’t need to adapt your states to different feature sets. Also, the upgrade process is much easier.

3. Avoid rare operating modes

Less traveled paths and settings that significantly change how Salt operates (transport, multiprocessing, zmq filtering, caching, undocumented things), tend to result in nasty surprises. If you want to change a setting, give it a try in your QA environment for a while.

4. Control the bootstrap/upgrade process

Do not use the “latest” repo prefix (except in a QA environment), and do not rush to upgrade to a latest major version
Use the “archive” repo prefix, or pin Salt packages down to a minor version (you never know when another 2019.2.1 will happen)
Always install specific version when you bootstrap new minions (e.g., through the script_args Salt Cloud option)

Yes, a minor upgrade will require more work. But it is work you can control and decide when to do. You can’t control work that appears out of the blue when minions automatically upgrade and something breaks.

5. Upgrade the master first

When you upgrade Salt, upgrade the master first. Older master can (and eventually will) break things.

6. Back up the keys

Set up periodic backups of the /etc/salt/pki master directory. Other than that, your Salt master should be disposable, and states/pillars should be kept in a version control system (or pulled from external data sources).

7. Have a backup access path

Have a plan B to access your servers when Salt goes down. Use salt-ssh, fabric, ansible, ssh, winrm, or any other tool. It is also helpful to keep a list of minion IPs (you can periodically extract them from the master grain cache).

8. Use Salt as an O&M control plane

Avoid making your production workload depend on Salt. If you uninstall Salt, your production environment should remain operational (at least for a while).

If you have the resources, you may also want to consider the last two principles:

9. Mirror the Salt repo

I never had to do this, but I’ve seen rare connectivity issues with the official repo. Also, for some folks, the minion bootstrap process was broken when SaltStack removed (and then quickly returned back) the buggy 2019.2.1 release. This is less of an issue if you follow the principle [4].

10. Build your own packages

Bugs will happen, and you need to fix them quickly without waiting for the next release. Also, you may want to integrate new features before they are shipped. Custom packages are a way to achieve that. They give you full control over the Salt code you run.

I ran a patched salt-master from a virtualenv for some time in 2012 or 2013, but I no longer do that. Instead, I prefer the more lightweight ways to patch Salt.

Plan the upgrade process

1. Survey your master and minion fleet

A list of useful commands is provided below. There are many dimensions you should inspect:

Operating systems, their architectures, and versions
Available repositories
Python versions
Master and minion versions
Salt dependencies

The main goal is to find all the existing combinations, so you can exhaustively test the upgrade process. Also check out the Py3 support list.

2. Prepare the QA environment

Ideally, you want to be able:

Test the same OS/Salt versions
Test the existing highstates
Test the actual workload that runs on top of your infra

You’ll need to test the upgrade process multiple times, so a fast feedback cycle is necessary (e.g., it would be nice to quickly snapshot/restore your QA environment).

3. Update the states

Make sure your states are compatible with the target Salt version. Read the official release notes and search for any open issues.

4. Decide on the upgrade process

For example:

Perform the backups (master/minion keys, alternate connection paths, etc.)
Upgrade OS packages, run autoclean/autoremove. Optionally reboot if you have new kernels.
Disable any periodic Salt jobs and other stuff that can interfere with the upgrade process. Try minion blackouts
Upgrade master to the target version, while staying on Py2
Ensure that the existing process to bootstrap new minions uses the target Salt version (Py3)
Upgrade minions to the target version, while staying on Py2. If that is not possible, upgrade to the latest available for each distro.
Upgrade Salt master to Py3
Upgrade Salt minions to Py3 for each OS separately

Consider the minion upgrade order. Your environment may require some specific upgrade sequence.

The goal is to remove the old Salt version and its dependencies (to avoid having packages from different repos), then bootstrap the Py3 one from scratch.

If your minions are disposable, you can avoid the in-place upgrade process and instead rebuild them from scratch. Another option is blue/green deployment.

5. Test the upgrade process multiple times

Manually upgrade your masters
Make sure their minions are reachable after the upgrade
Manually upgrade a single minion for each OS
Make sure the upgraded minions are reachable, the highstates can be applied, and the business apps are functional
Automate the above steps
Test the automation multiple times
Test the backup options (master/minion keys, alternate connection paths, etc.)
Test the bootstrap process for new minions

6. Do the upgrade

Decide how many upgrade stages you want
Decide on the appropriate schedule
Inform the team(s)
Perform and verify the backups (master/minion keys, alternate connection paths, etc.)
Switch the bootstrap for new minions to the target version
Upgrade the master
Upgrade the minions
Monitor the result

Survey your master and minion fleet

Listing versions

% sudo salt '*' test.version

% sudo salt '*' pkg.version salt-minion

% sudo salt '*' test.versions_report

(you may find a lot of variation in the installed dependencies!)

% sudo salt '*' grains.item pythonversion

% sudo salt-run manage.versions

Master:
    2019.2.3
Minion requires update:
    ----------
    minion1:
        2018.3.5
    minion2:
        2018.3.5
    minion3:
        2018.3.5
    win10:
        2018.3.5

sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion1
      - minion2
      - minion3
      - win10
  result:
      2018.3.5

Listing repos and keys

% sudo salt '*' pkg.list_repos
% sudo salt '*' pkg.get_repo_keys

Combine with jq to get more compact output:

% sudo salt -G "os:Ubuntu" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main"
"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2019.2.3 bionic main"

% sudo salt -G "os:Debian" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main"
"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.3 stretch main"

% sudo salt -G "os:CentOS" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[].baseurl'

"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2018.3.5/"
"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2019.2.3/"

Targeting py2 minions

% sudo salt -C "G@os:Ubuntu and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Debian and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:CentOS and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Windows and G@pythonversion:0:2" grains.item pythonversion saltversion

Other useful grains for targeting

cpuarch
lsb_distrib_codename
lsb_distrib_id
lsb_distrib_release
os
osarch
oscodename
osfinger
osmajorrelease
osrelease
osversion
pythonexecutable
pythonversion
saltpath
saltversion

Surveys

Any module calls (test.version, cmd.run, grains.item, etc.) can be combined with the survey runner:

sudo salt-run survey.hash '*' grains.item os osfinger pythonversion saltversion
|_
  ----------
  pool:
      - minion3k
  result:
      {u'osfinger': u'Ubuntu-18.04', u'saltversion': u'2018.3.5', u'os': u'Ubuntu', u'pythonversion': [2, 7, 17, u'final', 0]}
|_
  ----------
  pool:
      - minion2
  result:
      {u'osfinger': u'CentOS Linux-7', u'saltversion': u'2018.3.5', u'os': u'CentOS', u'pythonversion': [2, 7, 5, u'final', 0]}
|_
  ----------
  pool:
      - minion1
  result:
      {u'osfinger': u'Debian-9', u'saltversion': u'2018.3.5', u'os': u'Debian', u'pythonversion': [2, 7, 13, u'final', 0]}
|_
  ----------
  pool:
      - win10
  result:
      {u'osfinger': u'Windows-10', u'saltversion': u'2018.3.5', u'os': u'Windows', u'pythonversion': [2, 7, 14, u'final', 0]}

Listing package origins

The goal is to find packages that were installed from a specific origin, so that you can remove them during the upgrade process.

Ubuntu/Debian

I found the following snippet on Ask Ubuntu. If you know a more straightforward way to do the same without installing aptitude or other packages, I’m all ears.

% apt-cache policy $(dpkg -l | awk 'NR >= 6 { print $2 }') |
  awk '/^[^]/ { split($1, a, ":"); pkg = a[1] }
    nextline == 1 { nextline = 0; printf("%-40s %-50s %s\n", pkg, $2, $3) }
    /\*\*\*/ { nextline = 1 }' | grep saltstack

The output can vary depending on the OS:

salt-common https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
salt-minion https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main

python-jinja2 https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-common https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-minion https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main

Additionally, you can inspect how a Salt minion was installed in the first place. You need to figure out which packages are installed explicitly (the rest of the dependencies can be auto removed). Check out the /var/log/apt/history.log:

Debian:

Start-Date: 2020-02-18 05:15:02
Commandline: apt-get install -y -o DPkg::Options::=--force-confold procps pciutils python-yaml
Requested-By: vagrant (1000)
Install: python-yaml:amd64 (3.12-1), libyaml-0-2:amd64 (0.1.7-2, automatic)
End-Date: 2020-02-18 05:15:02

Start-Date: 2020-02-18 05:15:05
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg2 apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: gnupg2:amd64 (2.1.18-8~deb9u4), apt-transport-https:amd64 (1.4.9)
End-Date: 2020-02-18 05:15:06

Start-Date: 2020-02-18 05:15:35
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion
Requested-By: vagrant (1000)

Ubuntu:

Start-Date: 2020-02-18 05:15:08
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: apt-transport-https:amd64 (1.6.12)
End-Date: 2020-02-18 05:15:08

Start-Date: 2020-02-18 05:15:49
Commandline: apt-get install -y -o DPkg::Options::=--force-confold python2.7 python-apt python-requests python-yaml procps pciutils
Requested-By: vagrant (1000)
Install: python-six:amd64 (1.11.0-2, automatic), python-openssl:amd64 (17.5.0-1ubuntu1, automatic), python-yaml:amd64 (3.12-1build2), python-requests:amd64 (2.18.4-2ubuntu0.1), python-certifi:amd64 (2018.1.18-2, automatic), python-chardet:amd64 (3.0.4-1, automatic), python-enum34:amd64 (1.1.6-2, automatic), python-cryptography:amd64 (2.1.4-1ubuntu1.3, automatic), python-cffi-backend:amd64 (1.11.5-1, automatic), python-ipaddress:amd64 (1.0.17-1, automatic), python-pkg-resources:amd64 (39.0.1-2, automatic), python-apt:amd64 (1.6.5ubuntu0.2), python-urllib3:amd64 (1.22-1ubuntu0.18.04.1, automatic), python-idna:amd64 (2.6-1, automatic), python-asn1crypto:amd64 (0.24.0-1, automatic)
Upgrade: python2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), python2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-stdlib:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04)
End-Date: 2020-02-18 05:15:56

Start-Date: 2020-02-18 05:16:11
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion

The bootstrap-salt.sh script also logs everything to /tmp/bootstrap-salt.log:

pciutils is already the newest version (1:3.5.2-1).
procps is already the newest version (2:3.3.12-3+deb9u1).
...
Selecting previously unselected package python-yaml.
Preparing to unpack .../python-yaml_3.12-1_amd64.deb ...
Unpacking python-yaml (3.12-1) ...
Setting up libyaml-0-2:amd64 (0.1.7-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python-yaml (3.12-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20161130+nmu1+deb9u1).
ca-certificates set to manually installed.
wget is already the newest version (1.18-5+deb9u3).
The following NEW packages will be installed:
  apt-transport-https gnupg2
...
The following additional packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common
Suggested packages:
  debtags dbus-user-session pinentry-gnome3 tor apache2 | lighttpd | httpd
  python-apt-dbg python-apt-doc python-cryptography-doc
  python-cryptography-vectors python-enum34-doc python-jinja2-doc
  python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
  python-socks python-setuptools-doc python-mysqldb python-pycurl
  python-tornado-doc python-twisted python-ntlm python-augeas
Recommended packages:
  sfdisk e2fprogs
The following NEW packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common salt-minion

So, for Debian/Ubuntu you may want to remove the salt-minion, python-requests, python-yaml, and python-apt explicitly, then auto remove the dependencies.

CentOS/Amazon

% yum list installed | grep saltstack

PyYAML.x86_64 3.11-1.el7 @saltstack
python-zmq.x86_64 15.3.0-3.el7 @saltstack
salt.noarch 2018.3.5-1.el7 @saltstack
salt-minion.noarch 2018.3.5-1.el7 @saltstack
zeromq.x86_64 4.1.4-7.el7 @saltstack

The /var/log/yum.log is not very informative:

Feb 18 08:15:37 Updated: rpm-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:37 Updated: rpm-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-build-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-python-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Installed: python-chardet-2.2.1-3.el7.noarch
Feb 18 08:15:38 Installed: python-kitchen-1.1.1-5.el7.noarch
Feb 18 08:15:38 Installed: libyaml-0.1.4-11.el7_0.x86_64
Feb 18 08:15:38 Updated: libxml2-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Installed: libxml2-python-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Updated: python-urlgrabber-3.10-9.el7.noarch
Feb 18 08:15:39 Updated: yum-3.4.3-163.el7.centos.noarch
Feb 18 08:15:39 Installed: yum-utils-1.1.31-52.el7.noarch
Feb 18 08:15:39 Installed: PyYAML-3.11-1.el7.x86_64
Feb 18 08:15:39 Updated: chkconfig-1.7.4-1.el7.x86_64
Feb 18 08:16:15 Installed: lz4-1.7.5-3.el7.x86_64
Feb 18 08:16:15 Updated: systemd-libs-219-67.el7_7.3.x86_64
Feb 18 08:16:15 Installed: python-ipaddress-1.0.16-2.el7.noarch
Feb 18 08:16:15 Installed: python-markupsafe-0.11-10.el7.x86_64
Feb 18 08:16:15 Installed: python-six-1.9.0-2.el7.noarch
Feb 18 08:16:15 Installed: libtommath-0.42.0-6.el7.x86_64
Feb 18 08:16:15 Installed: libtomcrypt-1.17-26.el7.x86_64
Feb 18 08:16:16 Installed: python2-crypto-2.6.1-16.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-1.0-8.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
Feb 18 08:16:16 Installed: python-tornado-4.2.1-5.el7.x86_64
Feb 18 08:16:16 Installed: python-urllib3-1.10.2-7.el7.noarch
Feb 18 08:16:16 Installed: python-requests-2.6.0-8.el7_7.noarch
Feb 18 08:16:17 Installed: python-babel-0.9.6-8.el7.noarch
Feb 18 08:16:18 Installed: python-jinja2-2.7.2-4.el7.noarch
Feb 18 08:16:18 Installed: libsodium-1.0.18-1.el7.x86_64
Feb 18 08:16:18 Installed: python2-msgpack-0.5.6-5.el7.x86_64
Feb 18 08:16:18 Updated: cryptsetup-libs-2.0.3-5.el7.x86_64
Feb 18 08:16:20 Updated: systemd-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: systemd-python-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: openpgm-5.2.122-2.el7.x86_64
Feb 18 08:16:20 Installed: zeromq-4.1.4-7.el7.x86_64
Feb 18 08:16:21 Installed: python-zmq-15.3.0-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-futures-3.1.1-5.el7.noarch
Feb 18 08:16:21 Updated: pciutils-libs-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: pciutils-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-psutil-2.2.1-5.el7.x86_64
Feb 18 08:16:26 Installed: salt-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Installed: salt-minion-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Updated: systemd-sysv-219-67.el7_7.3.x86_64
Feb 18 08:16:26 Updated: libgudev1-219-67.el7_7.3.x86_64

However, the /tmp/bootstrap-salt.log shows precisely which packages were requested and which are merely dependencies:

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 PyYAML x86_64 3.11-1.el7 saltstack 160 k
 yum-utils noarch 1.1.31-52.el7 base 121 k
Updating:
 chkconfig x86_64 1.7.4-1.el7 base 181 k
Installing for dependencies:
 libxml2-python x86_64 2.9.1-6.el7_2.3 base 247 k
 libyaml x86_64 0.1.4-11.el7_0 base 55 k
 python-chardet noarch 2.2.1-3.el7 base 227 k
 python-kitchen noarch 1.1.1-5.el7 base 267 k
Updating for dependencies:
 libxml2 x86_64 2.9.1-6.el7_2.3 base 668 k
 python-urlgrabber noarch 3.10-9.el7 base 108 k
 rpm x86_64 4.11.3-40.el7 base 1.2 M
 rpm-build-libs x86_64 4.11.3-40.el7 base 107 k
 rpm-libs x86_64 4.11.3-40.el7 base 278 k
 rpm-python x86_64 4.11.3-40.el7 base 83 k
 yum noarch 3.4.3-163.el7.centos base 1.2 M

Transaction Summary
================================================================================
Install 2 Packages (+4 Dependent packages)
Upgrade 1 Package (+7 Dependent packages)

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 salt-minion noarch 2018.3.5-1.el7 saltstack 37 k

Below is a similar log for Amazon Linux:

Installing:
 m2crypto x86_64 0.31.0-4.el7 saltstack-repo 287 k
 python-crypto x86_64 2.6.1-2.el7 saltstack-repo 470 k
 python-msgpack x86_64 0.4.6-1.el7 saltstack-repo 73 k
 python-zmq x86_64 15.3.0-3.el7 saltstack-repo 520 k
Updating for dependencies:
 PyYAML x86_64 3.11-1.el7 saltstack-repo 160 k
 procps-ng x86_64 3.3.10-26.amzn2 amzn2-core 292 k
 python-requests noarch 2.6.0-7.amzn2 amzn2-core 95 k
Installing for dependencies:
 libsodium x86_64 1.0.16-1.el7 saltstack-repo 140 k
 openpgm x86_64 5.2.122-2.el7 saltstack-repo 172 k
 python2-typing noarch 3.5.2.2-4.el7 saltstack-repo 39 k
 zeromq x86_64 4.1.4-7.el7 saltstack-repo 555 k

So, for CentOS/Amazon you may want to remove the salt-minion, PyYAML, m2crypto, python-zmq, python-crypto, and python-msgpack explicitly, then auto remove the dependencies.

Listing dependencies using `pip`

If you installed some Salt dependencies via pip, it is worth inspecting them too:

% sudo salt -C "G@kernel:Linux" cmd.run 'pip freeze --local'

% sudo salt -t 30 -C "G@os:Windows" cmd.run 'pip.exe freeze --local' cwd='C:\salt\bin\scripts\'

On CentOS, Amazon, and Windows, the command shows a full list of python packages, so you’ll need to compare it with a default one to find what is different.

Upgrade the Salt Master

From the FAQ:

Q: Can I run different versions of Salt on my Master and Minion?

A: This depends on the versions. In general, it is recommended that Master and Minion versions match.When upgrading Salt, the master(s) should always be upgraded first. Backwards compatibility for minions running newer versions of Salt than their masters is not guaranteed.

There are two common ways to upgrade a Salt master:

Option 1

The in-place upgrade process:

Backup the /etc/salt/pki directory (just in case)
Remove the old Salt repo
Uninstall the old salt-master package and all of its dependencies
Add the new repo (py3)
Install the new salt-master

Option 2

If you can’t disconnect all minions at once during the upgrade process, it is possible to bring up an additional master server and migrate minions one-by-one:

Set up new salt-master (py3)
Copy the /etc/salt/pki directory from old master to the new one, to ensure that master and minion keys are the same
Migrate the minions:

# migrate_minion.sls
migrate_to_new_master:
  file.replace:
    - name: "{{ salt['config.get']('conf_file') }}"
    # Alternative variant if you keep master address in a separate config file
    # - name: "{{ salt['file.join'](salt['config.get']('config_dir'), 'minion.d', 'master.conf') }}"
    - pattern: '^master:\s+{{ salt["config.get"]("master") | regex_escape }}$'
    - repl: 'master: new-master.example.com'

# Restart salt minion after the the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltstack.com/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: migrate_to_new_master

To be safe, you can migrate minions in batches:

% sudo salt -G 'master:old-master.example.com' --batch-size 10% state.apply migrate_minion

Upgrade Salt on Windows

With winrepo

Things are very easy in Windows land if you use winrepo-ng:

1. Update your winrepo definitions

If you do not want to bring the whole winrepo-ng, just put the following two files into salt://win/repo-ng:

2. Rebuild the package database

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.refresh_db

3. List the currently installed version

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.version salt-minion

3. Make sure the target version is available

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.list_available salt-minion-py3

4. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.install salt-minion-py3 version=2019.2.3

You might want to add the --batch-size X argument for a staged rollout.

Without winrepo

Do not want to use winrepo at all? Below is an alternative way to upgrade Windows minions.

1. Fetch the installer

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

You can also include a checksum to prevent redownloads (works in Salt 2018.3.0 or greater):

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe \
  source_hash=$(curl -s http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe.sha256 |\
  iconv -f utf-16 -t utf-8 | awk '{print $1;}' | tr A-Z a-z)

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe

The default download location is fine, because the Salt installer doesn’t remove the cache dir since 2017.7.2.

2. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" cmd.run \
  'START /B c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe /S'

Upgrade Salt on Ubuntu and Debian

There are a few key pieces:

1. Remove the old repo and key

remove_salt_repo:
  pkgrepo.absent:
    - name: deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main
# - name: deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main
    - keyid: 0E08A149DE57BFBE

It is also possible to delete all existing Salt repos and keys on both Ubuntu and Debian:

{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

apt-get -y remove --auto-remove salt-minion
apt-get -y remove --auto-remove python-requests python-yaml python-apt

4. Install the minion

While it is possible to just apt-get install the necessary packages:

apt-get -y -o DPkg::Options::=--force-confold install python3-requests python3-yaml python3-apt
apt-get -y -o DPkg::Options::=--force-confold install salt-minion

I feel that using salt-bootstrap.sh is simpler and supports more platforms:

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3

Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.

5. The resulting state

# upgrade/ubuntu_debian.sls
{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
    # - keyid: 0E08A149DE57BFBE
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

install_at:
  pkg.installed:
    - name: at

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        apt-get -y remove --auto-remove salt-minion
        apt-get -y remove --auto-remove python-requests python-yaml python-apt
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

6. Run the upgrade command

% sudo salt -C "( G@os:Ubuntu or G@os:Debian ) and G@pythonversion:0:2" state.apply upgrade.ubuntu_debian

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on CentOS and Amazon Linux

There are a few key pieces:

1. Remove the old repo

remove_salt_repo:
  pkgrepo.absent:
    - name: saltstack

It is also possible to delete all existing Salt repos on any rpm-based distro:

{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

yum -y remove salt-minion
yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
yum -y autoremove

4. Install the minion without starting it

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3

5. Reuse the old config

The snippet below should be adapted to your specific environment:

[-f /etc/salt/minion.rpmsave] && \
( cp -f /etc/salt/minion /etc/salt/minion.bak; \
mv -f /etc/salt/minion.rpmsave /etc/salt/minion )

6. Start the salt-minion service

systemctl is-enabled salt-minion.service || \
(systemctl preset salt-minion.service && \
systemctl enable salt-minion.service)
systemctl start salt-minion.service

7. The resulting state

# upgrade/centos_amazon.sls
{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

install_at:
  pkg.installed:
    - name: at

atd_service:
  service.running:
    - name: atd
    # - enable: True

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        yum -y remove salt-minion
        yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
        yum -y autoremove
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3
        [-f /etc/salt/minion.rpmsave] && \
        ( cp -f /etc/salt/minion /etc/salt/minion.bak; \
        mv -f /etc/salt/minion.rpmsave /etc/salt/minion )
        systemctl is-enabled salt-minion.service || \
        (systemctl preset salt-minion.service && \
        systemctl enable salt-minion.service)
        systemctl start salt-minion.service" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

8. Run the upgrade command

% sudo salt -C "( G@os:CentOS or G@os:Amazon ) and G@pythonversion:0:2" state.apply upgrade.centos_amazon

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on other operating systems

If you have done this on any other operating system (Red Hat, Fedora, SUSE, Gentoo, Arch, Raspbian, macOS, Solaris, FreeBSD, OpenBSD, etc.), feel free to share your experience and possible gotchas. I’ll be happy to update the article with any specific recipes!

Monitor the upgrade process

The upgrade process can take some time. There are three ways to monitor minions as they reconnect:

Watch the master log

% sudo tail -f /var/log/salt/master

Watch the master event bus

% sudo salt-run state.event tagmatch='salt/minion/*/start' pretty=true

List the Py2/Py3 minion count

% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0]).\(.[1]).\(.[2])"' | sort | uniq -c

      1 "2.7.13"
      1 "2.7.17"
      1 "2.7.5"
      1 "3.5.3"
      1 "3.6.8"
      1 "3.6.9"

% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0])"' | sort | uniq -c

      3 "2"
      3 "3"

Upgrade automation

If you need more control over the upgrade process (handle retries and upgrade failures, define specific orchestration sequence, etc.) you can use Salt orchestration or preferably write a custom runner. Another variant is to use salt-api and pepper.

An out-of-band upgrade using fabric or salt-ssh is also an option.

Future developments

In 2019 SaltStack introduced salt-bin and pop-build projects to make self-contained Salt binaries. Hopefully, they will simplify the dependency handling and make the upgrade process easier.

Need to run different Salt versions simultaneously?

The next post will explain how to write backward-compatible Salt states (e.g., to deal with module.run syntax differences, or for public formulas). Subscribe to read it as soon as the draft is done.

Patching Salt Modules

Max Arnold — Mon, 27 Jan 2020 13:26:36 +0000

Let’s say you were bitten by a critical bug in Salt (2019.2.0 & 2019.2.1 were painful for many of us) that had an existing but unreleased fix. Or a cool feature that you badly need has landed in the old develop branch and is waiting forever to be migrated to the new master. Even if it was released, maybe you can’t yet upgrade Salt across the whole minion fleet. Or maybe you were wrongly expecting that your PR against the master branch would be included in the nearest release but were told that the upcoming release is feature frozen, and you have to wait another 4 months for the next one.

As a DevOps engineer, you need to get things done quickly. Fortunately, SaltStack has a built-in way to patch itself! After reading this guide, you’ll be able to patch almost any part of Salt code, even if you run multiple versions simultaneously. Some of the ways are hacky, but bugs are bugs, and you need to fix them in the most efficient way.

The methods described are mostly suitable for applying a couple of simple fixes. If you have a large or constantly evolving patchset, you’ll keep yourself sane by making your own fork and installing it via pip or custom packages.

Determining affected minions
How to sync Salt modules
Extracting upstream patches
Overriding a whole module
Adding a new function
Version-aware guard
Version-aware override
Non-syncable modules
The dreaded utils namespace
Self-patching
Bootstrapping patched minions
Useful commands
Future developments
If you need more help
Unexplored areas

Determining affected minions

The first thing to do is understand what minion versions you have (you can discover that there are older minions you forgot to upgrade):

% sudo salt '*' test.version

minion3:
    2019.2.1
minion2:
    2019.2.1
minion1:
    2019.2.3

Another way to do it is the manage.versions runner (it includes the master version too):

% sudo salt-run manage.versions

Master:
    2019.2.3
Minion requires update:
    ----------
    minion2:
        2019.2.1
    minion3:
        2019.2.1
Up to date:
    ----------
    minion1:
        2019.2.3

If that is too verbose, use the survey runner:

% sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion2
      - minion3
  result:
      2019.2.1
|_
  ----------
  pool:
      - minion1
  result:
      2019.2.3

And if all you want is just version counters, use the following command:

% sudo salt '*' test.version --out json --static | jq '.[]' | sort | uniq -c
      2 "2019.2.1"
      1 "2019.2.3"

Now you need to understand which of these versions you need to target:

If you found a bug, chances are you already know that
Otherwise, search the list of issues and the list of pull requests (including closed ones)

The general advice is to look at the PR contents and see if the changes are applicable to your installed Salt versions.

How to sync Salt modules

In the official docs, you can find the up-to-date list of syncable modules (contributed by Jamie Bliss in #50633). The majority of module types could be loaded (synced) from a fileserver, except the ones with a footnote.

The simplest way to sync them all across a minion fleet is to use the following command:

% sudo salt '*' saltutil.sync_all

Alternatively, you can use the individual saltutil.sync_* commands to sync different kinds of modules.

In Salt Neon, you can also use the new saltutil.sync_* states to do the same.

Some module types could be synced to a master (unfortunately, excluding rosters):

% sudo salt-run saltutil.sync_all

Or individually via the saltutil.sync_* runner commands.

Also, custom modules are automatically synced whenever a highstate is performed.

Extracting upstream patches

The general advice is to avoid grabbing the latest modules from the upstream Git repo and putting them into your salt:// file tree. The only scenario when it’ll likely end up well is when a module doesn’t exist yet in your Salt version and doesn’t depend on any other modules. In all other cases, it is much safer to extract a specific patch and adapt it to apply cleanly to your specific Salt version.

Let’s say you want to backport the ability to pass arbitrary pip arguments through pip.installed from Neon to 2019.2.3 (PR #52327).

A simple way to extract the feature is to open the https://github.com/saltstack/salt/pull/52327.diff link, download the patch and remove unnecessary hunks/files that are related to tests and docs. You can see the resulting patch here: pip-extra-args-orig.diff

The patch won’t apply cleanly; however, if you inspect the salt/modules/pip.py.rej and salt/states/pip_state.py.rej, you’ll see that they contain non-functional changes and could be ignored. For more complex conflicts, you’ll need to spend some time to port the patch.

By comparing the original salt-2019.2.3 tree with the patched one (using the diff -Naur -x '*.orig' -x '*.rej' salt-2019.2.3{.orig,} command), you can get the patch that applies cleanly to 2019.2.3: pip-extra-args-2019-2-3.diff

Overriding a whole module

Grab the patched modules. For the example above, that would be salt/modules/pip.py and salt/states/pip_state.py.
Put them into the salt://_modules/ and salt://_states/ folders respectively.
Sync the modules:

% sudo salt minion1 saltutil.sync_modules,saltutil.sync_states ,

minion1:
    ----------
    saltutil.sync_modules:
        - modules.pip
    saltutil.sync_states:
        - states.pip_state

Viola! Now you can use the extra_args option:

$ sudo salt minion1 sys.state_doc pip.installed | grep -A 8 extra_args

            extra_args
                pip keyword and positional arguments not yet implemented in salt

                    pandas:
                      pip.installed:
                        - name: pandas
                        - extra_args:
                          - --latest-pip-kwarg: param
                          - --latest-pip-arg

                Warning:

                    If unsupported options are passed here that are not supported in a
                    minion's version of pip, a `No such option error` will be thrown.

Below is an example of how to use this new option to install a package into the /root/.local directory (instead of /usr/local):

install_poetry:
  pip.installed:
    - name: poetry
    - extra_args:
      - --user

Adding a new function

Sometimes you want to extend a module with just one new function, but do not want to override the whole module because you are afraid of possible future version conflicts. Let’s say you want to add the delete function to sdb.sqlite3 module (PR #51479).

The original module is named sqlite3.py. Custom one should have a unique file name that contains the original one (see #50812 and #52521 to understand why). Let’s name it as sqlite3_custom.py:

# The imports below are copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__ () return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__ ():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount

Put the following configuration snippet to /etc/salt/minion.d/mysqlite.conf on a minion and restart it:

mysqlite:
  driver: sqlite3
  database: /tmp/sdb.sqlite
  table: sdb
  create_table: True

Then put the above sqlite3_custom.py module into the salt://_sdb/ directory and sync it:

% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom

Let’s try it:

% sudo salt minion1 sdb.set sdb://mysqlite/password 12345678

minion1:
    True

% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    12345678

% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 58, in delete
        return salt.utils.sdb.sdb_delete(uri, __opts__ , __utils__ )
      File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 108, in sdb_delete
        return loaded_db[fun](query, profile=profile)
      File "/var/cache/salt/minion/extmods/sdb/sqlite3_custom.py", line 28, in delete
        conn, cur, table = _connect(profile)
    NameError: name '_connect' is not defined
ERROR: Minions returned with non-zero exit code

Oops! Our custom function needs to have access to _connect, any related imports, private functions, and module-level variables. Salt loader only exposes public functions (without leading underscores), so any public function calls from the same module should be changed to __salt__ ['sqlite3.function']() syntax.

Let’s fix this specific example to make it work:

# The import below is copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

# This private function is imported from the original module
from salt.sdb.sqlite3 import _connect

# The import below is copied from the original module
try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__ () return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__ ():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount

Let’s try it again:

% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom

% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    1

% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    None

It works! Please note, that because the overlay module is named differently than the original one, all that initialization code in the upstream module is still triggered. Also, direct imports could potentially have unintended side effects.

CAVEAT: your custom delete function will be silently ignored (even with a version-aware guard) if the upstream sqlite3.py module gets its own delete function (i.e., after a Salt upgrade).

Version-aware guard

It would be nice if a patched module just stopped working with any Salt version that doesn’t match the target one. Then any upgrades would lead to a loud failure, and you won’t forget to remove or update the patched module. Let’s implement exactly that!

For example, there is an issue #54755 (pip failures even when not using pip) that badly affected Salt 2019.2.1. The fix is located in #54826; however, it can’t be applied cleanly on top of 2019.2.1 because the module was changed multiple times between the releases. Fortunately, all these changes are relevant, so we can just grab the state/pip_state.py module from 2019.2.2 or 2019.2.3. You can inspect the changes using the following command on two extracted release tarballs:

% diff -u salt-2019.2.{1,3}/salt/states/pip_state.py

...
DIFF HERE
...

Put the pip_state.py module from 2019.2.3 into the salt://_states/ directory and sync it:

% sudo salt minion1 saltutil.sync_states

minion1:
    - states.pip_state

After that, the state that triggered the error will run just fine:

# pip_test_state.sls
/tmp/file.txt:
  file.managed:
    - contents: |
        Content
    - unless:
      - 'grep "Content" /tmp/file.txt'

% sudo salt minion2 state.apply pip_test_state

minion1:
----------
          ID: /tmp/file.txt
    Function: file.managed
      Result: True
     Comment: unless condition is true
     Started: 06:47:19.338294
    Duration: 4058.666 ms
     Changes:

Summary for minion1
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 4.059 s

Now comes the most important part. We need to add a version-aware guard to the module, so it will only work on 2019.2.1. There are multiple ways to check Salt version:

Compare __grains__ ['saltversion'] with a string like '2019.2.1'
Compare __grains__ ['saltversioninfo'] with a list like [2019, 2, 1, 0]
Compare the salt.utils.versions.LooseVersion instances
Use the salt.utils.versions.warn_until helper
Compare salt.version. __saltstack_version__ with an instance of salt.version.SaltStackVersion
Compare release codenames using the new Compare release codenames using the new salt_version execution module in Salt Neon
Use the new salt.utils.versions.warn_until_date helper in Salt Neon

Option (5) seems to be the most robust (it should work with release candidates and git snapshots too, and also support the new versioning scheme).

We need to place the version comparison in a module function, which is triggered as early as possible. For the majority of modules, that would be __virtual__ () or __init__ (opts). The added lines are marked with --- VERSION CHECK ---:

def __virtual__ ():
    '''
    Only load if the pip module is available in __salt__
    '''
    # --- VERSION CHECK ---
    from salt.version import SaltStackVersion, __saltstack_version__ as sv
    ver = SaltStackVersion(2019, 2, 1, 0, u'', 0, 0, None)
    if sv != ver:
        msg = 'The %s module is not compatible with %s' % ( __name__ , ver)
        log.critical(msg)
        raise Exception(msg)
    # --- VERSION CHECK END ---

    if HAS_PKG_RESOURCES is False:
        return False, 'The pkg_resources python library is not installed'
    if 'pip.list' in __salt__ :
        return __virtualname__
    return False

Now, if you apply the pip.installed state to 2019.2.3 minion, it will fail because the module is not loaded:

% sudo salt minion1 state.single pip.installed ipython

minion1:
    Data failed to compile:
----------
    Specified state 'pip.installed' was not found

Another option is to place this check into the installed(...) function (right after the docstring and before the function body). This is more risky, because the module will be loaded and some parts could be executed before the check. On the positive side, the failure is more visible:

% sudo salt minion1 state.single pip.installed ipython

minion1:
----------
          ID: ipython
    Function: pip.installed
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "/var/cache/salt/minion/extmods/states/pip_state.py", line 675, in installed
                  raise Exception(msg)
              Exception: The salt.loaded.ext.states.pip_state module is not compatible with 2019.2.1
     Started: 07:49:24.007851
    Duration: 4.435 ms
     Changes:

Summary for minion1
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 4.435 ms

A couple of essential details to summarize:

An overlay module needs to be named exactly as the original one (see the Overriding a whole module section)
If an overlay module fails with an exception, the original one won’t be loaded
A module that extends the original one needs to contain its name in the filename (see the Adding a new function section)
If an extension module fails with an exception, the original one will be still loaded
Because the loader is lazy, the warning would be only triggered when the function is used
If a module is used implicitly (i.e., not by your states), avoid raising exceptions and instead find other ways to inject the warning

Version-aware override

So, we got a little safety guard that helps us to detect when our custom modules are outdated. But the rabbit hole is much deeper because we can run multiple Salt versions at the same time and may need to have different versions of custom modules.

The solution is to use Salt environments. First, set up the file_roots master setting:

file_roots:
  base:
    - /srv/salt
  v20190201:
    - /srv/salt_20190201

Second, make sure that any custom modules for 2019.2.1 are placed into the corresponding /srv/salt_20190201/ subdirs (_modules, _states, _grains, etc.) and not into the base environment /srv/salt dir:

% tree /srv
├── pillar
│   └── top.sls
├── salt
│   ├── top.sls
├── salt_20190201
│   └── _modules
│   └── test.py

There are two ways to pin minions to specific saltenvs:

Using the top file
Using the saltenv minion setting

Unfortunately, the second approach doesn’t affect how the saltutil.sync_* functions work (no modules are synced). And the explicit saltutil.sync_all saltenv=v20190201 will sync the modules to all minions, ignoring the saltenv setting. So, the only option left is to specify saltenvs in the top file:

base:
  '*': []
v20190201:
  'saltversion:2019.2.1':
    - match: grain

Now the saltutil.sync_modules function will work as intended and sync our custom test.py only to minion2:

% salt '*' test.version
minion1:
    2019.2.3
minion2:
    2019.2.1

% sudo salt '*' saltutil.sync_modules
minion1:
minion2:
    - modules.test

By adding more environments, you can distribute different custom modules to different minions using any targeting criteria (including the compound matchers).

CAVEAT 1: custom modules are cached and won’t be automatically deleted when you update Salt (at least for Ubuntu packages; other distros may contain uninstaller scripts that clear these caches). To mitigate that, run saltutil.clear_cache before the update (and maybe saltutil.refresh_* if necessary).

CAVEAT 2: Fileserver environments won’t be usable for proxymodules until the #56222 is merged. However, you can fix this problem by using the self-patching method and then proceed as usual.

If you are paranoid, you can use environments in combination with version-aware guards to prevent modules from being run on updated minions or if you accidentally mess up Salt environments.

Non-syncable modules

Some module types do not have a corresponding saltutil.sync_* command. For example, before Salt Neon, it was impossible to sync custom executors.

Below is a customized direct_call.py executor that logs the function name:

# -*- coding: utf-8 -*-
# direct_call.py
from __future__ import absolute_import, print_function, unicode_literals
import logging

log = logging.getLogger( __name__ )

def execute(opts, data, func, args, kwargs):
    log.info("Executing the %s function", func)
    return func(*args, **kwargs)

There are three ways to distribute this custom executor module.

1. Place the module into a cache dir

The cache dir is defined by the extension_modules config option (/var/cache/salt/minion/extmods, there is no need to change it).

# override_executor.sls
/var/cache/salt/minion/extmods/executors:
  file.directory:
    - makedirs: True

/var/cache/salt/minion/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /var/cache/salt/minion/extmods/executors

Run sudo salt minion1 state.apply override_executor to apply the state.

The biggest downside is that the cache directory is volatile and may be cleaned (e.g., if you run saltutil.clear_cache, saltutil.sync_all, state.highstate, or state.apply commands).

2. Module search path

Salt has multiple settings to control the search path for each module type (extension_modules and various *_dirs options). First, you need to configure the minion. Place the following snippet into /etc/salt/minion.d/extmods.conf and restart the minion (unfortunately, pillar-based minion configuration doesn’t work in this case):

# /etc/salt/minion.d/extmods.conf
executor_dirs:
  - /srv/extmods/executors

Then run sudo salt minion1 state.apply override_executor to apply the following state:

# override_executor.sls
/srv/extmods/executors:
  file.directory:
    - makedirs: True

/srv/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /srv/extmods/executors

3. Setuptools entry points

This is the most complex method; however, it allows installing arbitrary sets of custom modules along with their dependencies. I started writing some examples for this blog post but then abandoned the idea for the sake of brevity. If you are interested in how to do that, visit the Packaging Modules documentation page.

Do not forget to add a version-aware guard whenever you override a built-in module.

The dreaded utils namespace

Utils are a special breed of modules (despite being syncable through saltutil.sync_utils). Sometimes they are used through the loader (the __utils__ dunder), but most of the time they are imported directly:

% find salt/ -name '*.py' -exec grep -l ' __utils__ \[' \{\} \; | sort -u | wc -l
     159

% find salt/ -name '*.py' -exec grep -l ' salt\.utils' \{\} \; | sort -u | wc -l
    1101

There are a couple of problems with utils:

They are nested, and Salt loader can’t override nested modules (e.g., salt.utils.docker.translate) because it only supports the module.function notation
Although custom utils are added to sys.path if the utils cache directory is present, they aren’t importable as salt.utils.*
Even if they were importable, the minion process wouldn’t reload already imported modules unless restarted
When utils aren’t called via the __utils__ dunder, __opts__ and other dunders won’t be injected by the loader
Custom Jinja filters do not work because they are added at import time via the @jinja_filter decorator, but the loader is lazy and won’t import anything unless it is used somewhere

This explains the long list of issues that people have when they try to override utils: #32500, #46841, #46911, #51719, #51958, #52136, #52222, #53666, #55232…

There are two attempts to partially solve this use-case (#52001 and #53167 by Alberto Planas), but so far none of them has been released.

I was able to find two ways to deal with this mess when a patch touches both regular modules and utils:

Copy all the changed functions (and functions that use these functions!) from the utils patch into the modules that use it, and change all the calls accordingly
Alternatively, put the patched utils module into the _utils dir, then change all the calls to use the __utils__ notation

With both methods, you’ll end up running two versions of functions from the utils module (the old one is imported as usual, and the newer one is inlined or called from the overlay module).

Self-patching

The trick is described by DatuX in #51932. For example, let’s fix the horrible typo that prevented Salt 2019.2.1 from starting when the ping_interval option was set (see #54777). The patch is simple:

diff --git a/salt/minion.py b/salt/minion.py
index d1c0c00751..28b4956a49 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -2730,7 +2730,7 @@ class Minion(MinionBase):
                     self._fire_master('ping', 'minion_ping', sync=False, timeout_handler=ping_timeout_handler)
                 except Exception:
                     log.warning('Attempt to ping master failed.', exc_on_loglevel=logging.DEBUG)
- self.remove_periodic_callbback('ping', ping_master)
+ self.remove_periodic_callback('ping')
             self.add_periodic_callback('ping', ping_master, ping_interval)

         # add handler to subscriber

Put the patch into the salt://patches/files/2019.2.1-periodic-callback.diff file, then create the following state and put it into salt://patches/periodic_callback.sls:

{% if grains.saltversion == "2019.2.1" %}

# Just in case there is no patch command installed
patch:
  pkg.installed

# We use directory name here (saltpath) to apply patches that touch multiple files
periodic_callback_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/2019.2.1-periodic-callback.diff
    - strip: 2
    - require:
        - pkg: patch

# Set the ping_interval option after the patch is applied
set_ping_interval:
  file.managed:
    - name: '{{ salt["file.join"](salt["config.get"]("config_dir"), "minion.d", "ping_interval.conf") }}'
    - contents: |
        ping_interval: 60
    - onchanges:
        - periodic_callback_patch

# Restart salt minion after the patch is applied and the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltstack.com/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: set_ping_interval

{%- endif %}

The state will apply the patch, set the required config option, and restart the minion. However, there are a couple of things that could be improved:

The state is idempotent, but to improve the overall performance, we can prevent it from running the second time. You can use grain to mark the patch as applied.
The file.patch state is not implemented for Windows (see #44783)
There are other Windows-specific tweaks that are missing

Bootstrapping patched minions

All these patching methods are super useful, but it is also essential to know how to apply them to new minions. Earlier I described a few commands that you can use to sync the modules manually. Now it is time to automate that.

There are many ways to bootstrap Salt minions: salt-bootstrap, salt-ssh, different cloud drivers (including the saltify one), salt-run manage.bootstrap, and countless custom scripts. The most important thing is to apply any overrides/patches to minions as early as possible (before any other states are run). There are multiple ways to do so:

You do not need to do anything when you apply a highstate. With the autoload_dynamic_modules minion config option (True by default), Salt runs saltutil.sync_all automatically before applying the highstate.
If for some reason, you disabled the above option, you can place a state that syncs modules at the beginning of the top.sls file, so it will be applied first
Configure the startup_states minion setting
Add a master reactor config for the salt/minion/*/start event
Hook into the salt-cloud bootstrap process
Write a custom bootstrap script

If you need to restart a minion once for the patch to take effect, you can combine methods (3) or (4) with the restart technique. Methods (5) and (6) allow you to patch minions before they are started.

Useful commands

A couple of commands to troubleshoot custom modules:

salt-call together with print or logging statements (or with any interactive Python debugger like ipdb)
saltutil.clear_cache to clear the extmods directory
cp.list_master prefix=_modules/ (or any other directory) to see which custom modules are available on the fileserver
saltutil.list_extmods to see the list of synced modules
sys.list_* to list different kinds of modules

Future developments

We can expect Salt loader refactoring to happen in 2020
In 2019 SaltStack introduced POP (warn: gated content) as a standalone plugin system that is better than Salt loader. So far, no integration plans have been announced.

If you need more help

Hopefully, this guide has enough details to help you apply any patches you want. However, if you need more help to do that, I’m available for part-time consulting:

I can port any existing upstream patch (that is not overly invasive) to an older Salt version. Examples: 1, 2, 3, 4
I can help integrate the patch into your Salt bootstrapping process
My availability is about ~8-12 hours per week

See the home page footer for my contact email.

Unexplored areas

There are a couple of rabbit holes I wanted to explore but had no time to do so:

How to override just one function without copying a whole module?
Overriding modules using providers
Why raising exceptions from patched core grains is bad (and what to do instead)?
Different salt-cloud hooks that could be used to bootstrap patched minions
How to prevent salt-minion from autostarting (even on Debian/Ubuntu systems) so you can patch it?
Writing custom bootstrap scripts
Distributing modules via setuptools entry points (example)
Patching Salt minions on Windows
Patching Salt master
Loader internals (module search path, file loading algorithm)

This is not a commitment, but I hope to cover these topics in the future (if this post turns out to be useful).

Want to learn more about SaltStack patching?

There are a couple of missing aspects I plan to cover in the future (and maybe release an expanded guide in the form of a small & focused ebook). Subscribe to signal your interest in this topic.

Text editor plugins for Salt states and YAML/Jinja

Max Arnold — Sat, 10 Nov 2018 15:19:10 +0000

Image by Justin Palmer

If you have ever written a couple of Salt states, these problems may sound familiar:

A missing colon after the YAML key or a missing space after the colon
Data failed to compile: State ‘foo’ in SLS ‘bar’ is not formed as a list
A ParserError or ScannerError while running the state.highstate
A nested dictionary, indented with two spaces instead of four
State requirements at the wrong indentation level
Indentation with mixed tabs and spaces, which took a surprisingly long time to debug

And even if your states are valid YAML, they still can have no sense for Salt because your wathc_in requisite has a typo. Sprinkle a few Jinja tags here and there, and the errors can get even fancier…

Is there anything that could be done?

These issues definitely have come up before (#802). Smart developers probably have this figured out, right?

Indeed, it is possible to validate your states, in a semi- or completely automated way, even before they touch your infrastructure. For Salt, you have the following options:

Use the salt-call --local --retcode-passthrough state.show_sls YOUR_STATE either manually or using a commit hook. This triggers the state rendering process and can catch tons of possible errors.
Test your states using Test Kitchen together with kitchen-salt and some verifier. A helpful guide on this topic is available here

However, all of this feels a bit heavy. You need to commit and push to run your tests, spin up VMs or containers, dive into Ruby code, and write tests for all possible pillar/grain permutations. Such a long feedback cycle could easily lead to git commit -m "Fuck YAML" in your commit history and a subsequent amend/rebase/squash dance (and you can’t squash the obscene commit notifications sent to the rest of your team!).

Is there something more lightweight?

Well, your development environment can help you. The first layer of defense can work right in your text editor as you type, providing immediate feedback. It will catch your typos before they get committed and deployed (and let the CI system deal with more complicated stuff). An ideal editor plugin can:

Help you deal with YAML indentation levels and data structures
Know and highlight the most common Salt state keywords
Highlight Jinja brackets and tags
Autocomplete the keywords

Below I’ll cover plugins for 9 different editors, so you do not have to switch away from your favorite one. Also, I will give a few installation/configuration tips to save you some googling. These plugins aren’t ideal, but they are open source, and you can contribute to any of them if you want (I did).

Atom
Eclipse
Emacs
Kate
Midnight Commander
Sublime Text
PyCharm
Vim
Visual Studio Code
Final tips

If your favorite editor is not covered here, or you want to share a nifty hack that helps you reduce the number of YAML mistakes, ping me on Twitter.

Atom

Out of the box, Atom understands *.sls files as YAML. Plus, you could install atom-jinja2 and manually switch a file grammar to YAML (Jinja Templates) by pressing Ctrl+Shift+L.

For better results, you can install both atom-salt and atom-jinja2. Just click Edit -> Preferences -> Install, search for atom-salt or atom-jinja and install them.

Features

The first atom-salt language scope is source.yaml.salt, which is automatically enabled for *.sls files and provides syntax highlighting (YAML+Jinja combo)
The second scope source.python.salt is not associated with any file extension and instead is enabled for any files which start with #!py. This means you get syntax highlighting for py, pydsl, and pyobjects Salt renderers.
Also, it should open Salt docs via the salt-doc command, but it didn’t work for me for some reason
Atom-jinja2 provides the source.yaml.jinja scope, which is enabled for *.jinja, *.j2 and a bunch of other extensions
Supports syntax highlighting and folding
Has a couple of Jinja-related snippets

You can assign specific language scope to any extension with something like this in your ~/.atom/config.cson:

"*":
  core:
    customFileTypes:
      "source.python.salt": [
        "py.sls"
        "pysls"
      ]

Other things to try

Language-salt - an alternative Atom plugin for Salt (enabled for *.sls files, provides the source.salt scope and a bunch of snippets)
Aligner-salt (GitHub) to visually align YAML attributes
The indentation guides should be enabled by default by Salt plugin, but you can enforce it by clicking Edit -> Preferences -> Editor -> Show Indent Guide

Eclipse

I was unable to find a good enough solution for Eclipse, but below are two directions you can try to explore:

Go to Help -> Eclipse Marketplace, search for “yedit” and install it. Then go to Window -> Preferences -> General -> Editors -> File Associations and associate the *.sls extension with YEdit. This enables syntax highlighting for plain YAML files (no luck with Jinja though).
Another plugin is called LiClipseText. It has support for both YAML and Jinja syntaxes, but can’t enable them simultaneously for a single file. It should be possible though, and there are some hints if you want to contribute.

Indentation guides are available via a separate indent-guide plugin.

Emacs

Salt-mode is your best bet. It is based on yaml-mode and mmm-mode and is automatically enabled for *.sls files. The simplest way to install is through MELPA.

Features

Syntax highlighting for *.sls files (also possible to enable *.jinja syntax support)
Indentation and alignment of expressions and statements
Spell checking of comments with flyspell
Use salt-mode-browse-doc to browse the documentation of the state module at point
Quickly move between Salt state functions

Things to be aware of

No auto-completion yet
It uses yaml-mode under the hood, which needs a new maintainer. If you are willing to step in, there are a number of issues for you to fix.

Other things to try

Indent-tools together with hydra looks like a good combo for whitespace-sensitive languages like YAML and Python.
Highlight-indent-guides is worth looking at (although it adds more visual clutter). Also, there is a crosshair highlighting mode.
Install dash-at-point if you use Dash API Documentation Browser
To highlight other Jinja-templated configuration files, establish the convention of naming them using the *.jinja extension and add the following lines to your init.el (see the README for more details):

(mmm-add-mode-ext-class nil "\\.jinja\\'" 'jinja2)

Kate

Kate-jinja2-highlighting has a bunch of syntax highlighters for different file types with Jinja on top.

Features

Highlights *.jinja, *.jinja2 and *.j2 using Jinja2 syntax
Highlights *.yaml, *.yml and *.sls files using YAML+Jinja2 syntax (if you apply the patch below)
If you want to enable indentation guide, toggle the Settings -> Configure Kate -> Editor Component -> Appearance -> Show indentation lines

To install it, copy the *.xml files into the ~/.local/share/org.kde.syntax-highlighting/syntax/ folder. Then go to Settings -> Configure Kate -> Editor Component -> Open/Save -> Modes & Filetypes, select Markup/Jinja2/YAML, open the Variables drop-down, and then set both indent-width tab-width to 2 and Indent using to Spaces.

You also need to associate *.sls files with the syntax highlighter:

sed -i'' -e 's/\(\*\.yml\)"/\1;*.sls"/' yaml-jinja2.xml

The sed command above is equivalent to the patch below:

--- yaml-jinja2.xml.orig    2018-10-25 02:39:31.000000000 +0700
+++ yaml-jinja2.xml 2018-10-25 02:39:40.000000000 +0700
@@ -3,7 +3,7 @@
 <!-- Author: Dr Orlovsky MA <maxim@orlovsky.info> //-->
 <!-- Autogenerated from yaml.xml -->
 <language name="Jinja2/YAML" version="1.2" kateversion="2.3" section="Markup"
- extensions="*.yaml;*.yml" mimetype="text/yaml"
+ extensions="*.yaml;*.yml;*.sls" mimetype="text/yaml"
           author="Dr Orlovsky MA (dr.orlovsky@gmail.com)" license="LGPL">
   <highlighting>
     <contexts>

Midnight Commander

Midnight Commander is an oldie but goodie for those of us who used orthodox file managers back in the 90’s and can’t live without one. Personally, I use it to quickly navigate the filesystem tree and view the files without opening them in my primary editor. According to this ticket, out of the box, mcedit highlights *.yml and *.yaml files and understands simple Jinja variables (no support for tags though). Also, it does not understand *.jinja files.

To enable *.sls highlighting you need to make a copy of the default Syntax file and tweak it a bit:

mkdir -p ~/.config/mc/mcedit
sed -e 's/\(|YML)\)/|sls\1/' /usr/share/mc/syntax/Syntax > ~/.config/mc/mcedit/Syntax

The sed command above is equivalent to the patch below:

--- /home/user/.config/mc/mcedit/Syntax.orig 2017-08-05 04:03:29.000000000 +0700
+++ /home/user/.config/mc/mcedit/Syntax 2018-10-10 19:50:11.390542735 +0700
@@ -271,7 +271,7 @@
 file ..\*\\.cl$ OpenCL\sProgram
 include opencl.syntax

-file ..\*\\.(ya?ml|YML)$ YAML\sFile
+file ..\*\\.(ya?ml|sls|YML)$ YAML\sFile
 include yaml.syntax

 file .\*\\.osl$ OSL\sProgram

PyCharm

Well, it looks there are no Salt-specific plugins for PyCharm. Ansible users are luckier here - PyCharm has support for JSON Schemas, and someone from RedHat has created one. This means code auto-completion for Ansible-specific YAML data structures right out of the box! Plus, there is a separate YAML/Ansible plugin (GitHub).

However (aside from using Ansible plugins), you can tweak PyCharm to make it work with Salt states. Click File -> Settings -> Languages & Frameworks -> Python Template Languages, select Jinja2 in the Template language drop-down, click the “+” button and select YAML. This should enable syntax highlighting for YAML+Jinja combo. To associate it with the *.sls files, go to File -> Settings -> Editor -> File Types, find YAML and add the *.sls extension.

Other features

You can reformat YAML code with the Reformat Code action or via a shortcut: Cmd + Alt + L on MacOS / Ctrl + Alt + L on Windows and Linux
To customize YAML code style settings, visit File -> Settings -> Editor -> Code Style -> YAML
Indentation guides are enabled by default, but you can disable them via File -> Settings -> Editor -> General -> Appearance -> Show indent guides

Sublime Text

The extension is available on GitHub. To install it click Preferences -> Package Control -> Install Package, search for saltstack and click to install. It is also recommended to install the separate extension for Jinja (go to Package Control and search for jinja2).

Features

Syntax highlighting for *.sls files
Syntax highlighting for *.j2 files if you install the Jinja extension
Several snippets for Jinja (forked from sublime-jinja2)
Several snippets for often used Salt state modules like file and pkg

Indentation guides are enabled by default, but you may also want to set "indent_guide_options": ["draw_normal", "draw_active"] in the settings to highlight currently active guide.

Vim

You need both salt-vim and vim-jinja2-syntax.

Features

Syntax highlighting for *.sls files and Saltfile
Syntax highlighting for *.jinja2, *.j2 and *.jinja files
Proper indentation settings
Space key will try to fold/unfold an area
A visually selected block might be indented and unindented with < and >

If you are using Vim 8 or higher, you may skip the Pathogen/Vundle route and install the plugins directly:

mkdir -p ~/.vim/pack/plugins/start
cd !$
git clone https://github.com/saltstack/salt-vim.git salt.vim
git clone https://github.com/Glench/Vim-Jinja2-Syntax.git jinja2.vim

Then add the following lines to your .vimrc:

syntax on
set nocompatible
set modeline
filetype plugin indent on
let g:sls_use_jinja_syntax = 1

You might also consider vim-indentguides, vim-indent-guides or indentLine. Some folks use crosshair highlighting.

Visual Studio Code

The VSCode extension for SaltStack is available on the Marketplace. To install it click File -> Preferences -> Extensions, then search for saltstack and click install. The source code is available on GitHub

Features

Has syntax highlighting for *.sls files (understands YAML+Jinja combo)
Autocompletion of Jinja brackets
Folding of indented lines in YAML
Also supports *.j2 and *.jinja extensions

If you get annoyed by extra closing bracket added by autocompletion {%%}}, add the following snippet to your settings.json file (File -> Preferences -> Settings, then press the {} icon to open User Settings):

"[sls]": { "editor.autoClosingBrackets": "never" }

Final tips

For teams

If you have a team of engineers who regularly contribute to a common Salt state tree, it is essential to establish some YAML/Jinja style conventions.

You could add the following boilerplate to each state file, but it is easy to forget and requires some discipline:

# -*- coding: utf-8; mode: salt; -*-
# vim: ft=sls

Alternatively, you could ask engineers to set up their editors properly. Or you can adopt http://editorconfig.org/ - just create a .editorconfig in your project root, and any compatible editor will automatically pick up your coding convention.

Also, it makes sense to add .jinja or .j2 suffix to all config templates managed by Salt, to help most editors recognize the syntax.

YAML docs

YAML Cheatsheet (also could be used with Dash)
The YAML Format (from the Symfony docs)
YAML Multiline (a nice cheatsheet for YAML multiline strings)
Understanding YAML (from the Salt docs)
YAML Idiosyncrasies (from the Salt docs)
YAML Specification (authoritative and comprehensive, but not so easy to read)

Jinja docs

Jinja Template Designer Documentation (please choose the appropriate Jinja version to match the one installed in your Salt environment)
Understanding Jinja (from the Salt docs)

Online YAML parsers/linters

These could be helpful if you don’t want to or can’t use any editor plugins:

https://github.com/Inveracity/jinjabread (made specifically for Salt)
http://yaml-online-parser.appspot.com/
https://www.json2yaml.com/
http://www.yamllint.com/

Also, YAML is a superset of JSON, so you could just use square brackets for lists and curly ones for dictionaries:

/etc/http/conf/http.conf:
  file.managed:
    - source: salt://apache/http.conf
    - mode: 644
    - template: jinja
    - context: { custom_var: "override" }
    - defaults: { custom_var: "default value", other_var: 123 }

If you want to be notified about tips like this one, you can subscribe to the mailing list.

Also, you can follow me on Twitter where I periodically post things like this:

DEV Community: Max Arnold

Upgrading Salt to Python 3

10 key principles

1. Have a QA environment

2. Run the same Salt version everywhere

3. Avoid rare operating modes

4. Control the bootstrap/upgrade process

5. Upgrade the master first

6. Back up the keys

7. Have a backup access path

8. Use Salt as an O&M control plane

9. Mirror the Salt repo

10. Build your own packages

Plan the upgrade process

1. Survey your master and minion fleet

2. Prepare the QA environment

3. Update the states

4. Decide on the upgrade process

5. Test the upgrade process multiple times

6. Do the upgrade

Survey your master and minion fleet

Listing versions

Listing repos and keys

Targeting py2 minions

Other useful grains for targeting

Surveys

Listing package origins

Ubuntu/Debian

CentOS/Amazon

Listing dependencies using pip

Upgrade the Salt Master

Option 1

Option 2

Upgrade Salt on Windows

With winrepo

1. Update your winrepo definitions

2. Rebuild the package database

3. List the currently installed version

3. Make sure the target version is available

4. Run the upgrade command

Without winrepo

1. Fetch the installer

2. Run the upgrade command

Upgrade Salt on Ubuntu and Debian

1. Remove the old repo and key

2. Stop the salt-minion service

3. Remove the packages

4. Install the minion

5. The resulting state

6. Run the upgrade command

Upgrade Salt on CentOS and Amazon Linux

1. Remove the old repo

2. Stop the salt-minion service

3. Remove the packages

4. Install the minion without starting it

5. Reuse the old config

6. Start the salt-minion service

7. The resulting state

8. Run the upgrade command

Upgrade Salt on other operating systems

Monitor the upgrade process

Watch the master log

Watch the master event bus

List the Py2/Py3 minion count

Upgrade automation

Future developments

Need to run different Salt versions simultaneously?

Patching Salt Modules

Contents

Determining affected minions

How to sync Salt modules

Extracting upstream patches

Overriding a whole module

Adding a new function

Version-aware guard

Version-aware override

Non-syncable modules

1. Place the module into a cache dir

2. Module search path

3. Setuptools entry points

Listing dependencies using `pip`