Using Spack to build the VFX Refence Platform

Exploring Spack as an alternative to Rez for VFX and Animation

If you are a software developer in the VFX/Animation industry then chances are you've heard of Rez. Rez is widely used to manage complex software builds and environments, and it was recently added to the Academy Software Foundation.

I first discovered Rez in 2013 while looking for a replacement for Luma's in-house environment management tool. Over the following two years I worked with the author and other contributors on the Rez 2.0 rewrite. When colleagues asked me why we couldn't use tool X instead of Rez, my answer was that our industry is unique. Only we are required to build, deploy, and simultaneously utilize such a multitude of libraries and versions. The ultimate badge of our special status is that this "dependency hell" became so intractable that software vendors and studio leaders united to establish the VFX Reference Platform in order to reduce the build surface area.

In my experience, a software development team in our industry needs a tool that meets at least these requirements:

• manage builds of numerous software packages, each using its own build tools, such as cmake, autoconf, scons, etc
• specify variations for package builds, allowing definition of build matrices
• resolve transitive dependencies from a loose set of version range requirements
• load environment variables associated with released packages into a shell environment
• ability to build behind a firewall, without direct internet access

Unbeknownst to me, while we were making Rez, a different community made a tool that addresses these core requirements and eclipses Rez in many ways.

That tool is Spack, built by the supercomputing and scientific computing community. This is a serious tool, backed by (as you might expect) some sophisticated engineering, with a robust layered configuration suitable for small and large organizations.

In this article I track what I've learned while building the VFX Reference Platform using Spack.

Comparison to Rez

Before we dive in, here's a mile-high overview of the two projects.

Popularity

The first thing that stands out about Spack is that its user base is many times larger than Rez, if we use Github stars and forks as a metric.

Project health

Next up, according to www.repocheck.com, Spack is a healthier project, with far more closed PRs and issues, and a healthier ratio of opened to closed.

In the last month Rez saw 2 completed PRs while Spack had a whopping 472.

Features

After reading through the docs and experimenting a bit, I found that Spack has solutions to many of my complaints with Rez.

With Spack you can...

1. Build a package and all of its dependencies with a single command.
2. Use package repos to share build recipes, akin to Homebrew on Mac. Imagine a repo that's curated and tested by the ASWF that can be used by the entire industry.
3. Simplify working in air-gapped environments by creating local mirrors of source code tarballs
4. Automatically generate CI pipelines that will create a dependency graph of CI jobs to build each package in parallel.
5. Host caches of binary builds for easy distribution without recompilation.
6. Generate an optimized docker container with your compiled binaries.

Rez to Spack cheat sheet

Rez	Spack	Notes
package	package	package.py in Spack is responsible for build recipe and environment
request	spec	see below for a comparison of version specifiers
variant	variant	variants in Spack are explicitly named and can encompass multiple dependencies
resolve a request	concretize a spec	concretization can run in multiple processes
rez env	spack load	update a shell env with specified packages

Diving in

Ok, time to get our hands dirty.

The Goal: Build as much of the VFX Reference Platform as possible using Spack.

📘 Note

If you're following along at home, I'm using a custom fork of Spack that fixes a few package recipes and adds some new capabilities (see "Managing environment variables", below). You can see all of the changes here.

Also, the steps in this article are compiled into some reproducible shell scripts in this repo:

chadrik / spack-vfx-demo

Companion scripts for the article "Using Spack to build the VFX Reference Platform"

Build the VFX Reference Platform using Spack

git clone https://github.com/chadrik/spack-vfx-demo
cd spack-vfx-demo
./demo.sh

View on GitHub

I chose to use docker so that my tests would be completely reproducible. I love that I can use docker, and what's more there's a spack/centos7 image. This is already feeling more modern than Rez.

I'm mounting a local directory as a volume in order to reuse builds across docker invocations, and to avoid maxing out the docker disk limit.

mkdir ./spack-builds
git clone https://github.com/chadrik/spack --branch vfx-demo
docker run -it --rm \
  -v $(pwd)/spack-builds:/opt/spack/opt/spack/ \
  -v $(pwd)/spack/lib:/opt/spack/lib \
  -v $(pwd)/spack/var/spack:/opt/spack/var/spack \
spack/centos7:v0.19.0 

Now that we're in the container, let's see if we can locate the OpenEXR package:

$ spack list exr
fakexrandr  openexr  py-annexremote
==> 3 packages

More info:

$ spack info openexr
CMakePackage:   openexr

Description:
    OpenEXR Graphics Tools (high dynamic-range image file format)

Homepage: https://www.openexr.com/

Preferred version:  
    3.1.5     https://github.com/AcademySoftwareFoundation/openexr/archive/refs/tags/v3.1.5.tar.gz

Safe versions:  
    3.1.5     https://github.com/AcademySoftwareFoundation/openexr/archive/refs/tags/v3.1.5.tar.gz
    2.3.0     https://github.com/AcademySoftwareFoundation/openexr/releases/download/v2.3.0/openexr-2.3.0.tar.gz
    2.2.0     http://download.savannah.nongnu.org/releases/openexr/openexr-2.2.0.tar.gz
    2.1.0     http://download.savannah.nongnu.org/releases/openexr/openexr-2.1.0.tar.gz
    2.0.1     http://download.savannah.nongnu.org/releases/openexr/openexr-2.0.1.tar.gz
    1.7.0     http://download.savannah.nongnu.org/releases/openexr/openexr-1.7.0.tar.gz
    1.6.1     http://download.savannah.nongnu.org/releases/openexr/openexr-1.6.1.tar.gz
    1.5.0     http://download.savannah.nongnu.org/releases/openexr/openexr-1.5.0.tar.gz
    1.4.0a    http://download.savannah.nongnu.org/releases/openexr/openexr-1.4.0a.tar.gz
    1.3.2     http://download.savannah.nongnu.org/releases/openexr/openexr-1.3.2.tar.gz

With this configuration, the latest version of openexr available in Spack is 3.1.5, but there are a number of earlier versions available as well.

A quick test of version specs

As with Rez, Spack can resolve loose version requirements into specific set of package versions and their dependencies.

In Spack terms, this is called "concretizing a spec".

Below is a quick test of a few version specifiers. All of these found openexr 3.1.5:

spack spec openexr@3.1      # >= 3.1.0
spack spec openexr@3:4      # >= 3.0.0, < 5
spack spec openexr@3:3.1    # >= 3.0.0, < 3.2

And as expected, the command below did not find openexr 3.1.5:

spack spec openexr@3:3.1.0  # >= 3.0.0, < 3.1.0

This demonstrates two things:

• upper bounds are inclusive. i.e. <=
• omitted sub-versions float to the highest value when used as an upper bound, and lowest versions when used as a lower bound

If you need a quick refresher while you're at the console, you can run spack help --spec.

Setting up the compiler

Before we do a build we need to install gcc 9.3.1, since this is the version required by VFX Reference Platform CY2021 and CY2022 and it's not in the container.

yum install -y centos-release-scl && yum install -y devtoolset-9
spack compiler find /opt/rh/devtoolset-9/root/usr/bin/

Alternatively, you can build your own gcc:

spack install gcc@9.3.1

📘 Note

The above gcc installation instructions need to be repeated each time the container is started, unless you make a new container with that step baked in. This is because anything installed into the docker file system is ephemeral, and will not be persisted between invocations.

Building OpenImageIO

As a way to test a complex build with numerous dependencies, I chose to tackle OpenImageIO.

First, we'll inspect the full list of packages required to build OpenImageIO, with dependencies locked down versions that are compatible with the VFX Reference Platform:

spack spec openimageio +python ^openexr@3.1 ^python@3.9 ^py-numpy@1.20 ^boost@1.76

Some details:

• +python: enables the variant of openimageio which will build the python bindings
• ^: requested dependency versions. I want to lock down the dependencies to match those in CY2022.

For reference, the package.py file for openimageio looks like this:

from spack.package import *
from spack.pkg.builtin.boost import Boost


class Openimageio(CMakePackage):
    """OpenImageIO is a library for reading and writing images, and a bunch of
    related classes, utilities, and applications."""

    homepage = "https://www.openimageio.org"
    url = "https://github.com/OpenImageIO/oiio/archive/refs/tags/v1.8.15.tar.gz"

    version("2.4.8.0", sha256="e8402851970d48c718917060fa79bb7d75a050316556337cd0d4f0d0f971c904")
    version("2.2.7.0", sha256="857ac83798d6d2bda5d4d11a90618ff19486da2e5a4c4ff022c5976b5746fe8c")
    version("1.8.15", sha256="4d5b4ed3f2daaed69989f53c0f9364dd87c82dc0a09807b5b6e9008e2426e86f")

    # Core dependencies
    depends_on("cmake@3.2.2:", type="build")
    depends_on("boost@1.53:", type=("build", "link"))

    # TODO: replace this with an explicit list of components of Boost,
    # for instance depends_on('boost +filesystem')
    # See https://github.com/spack/spack/pull/22303 for reference
    depends_on(Boost.with_default_variants, type=("build", "link"))
    depends_on("libtiff@4.0:", type=("build", "link"))
    depends_on("openexr@2.3:", type=("build", "link"))
    depends_on("libpng@1.6:", type=("build", "link"))
    depends_on("libjpeg", type=("build", "link"))

    # Optional dependencies
    variant("ffmpeg", default=False, description="Support video frames")
    depends_on("ffmpeg@3.0:", when="+ffmpeg")

    variant("jpeg2k", default=False, description="Support for JPEG2000 format")
    depends_on("openjpeg@2.0:", when="+jpeg2k")

    variant("python", default=False, description="Build python bindings")
    extends("python", when="+python")
    depends_on("py-numpy", when="+python", type=("build", "run"))
    depends_on("py-pybind11", when="+python", type=("build", "run"))

    variant("qt", default=False, description="Build qt viewer")
    depends_on("qt@5.6.0:+opengl", when="+qt")

    conflicts("target=aarch64:", when="@:1.8.15")

    def cmake_args(self):
        args = ["-DUSE_FFMPEG={0}".format("ON" if "+ffmpeg" in self.spec else "OFF")]
        args += ["-DUSE_OPENJPEG={0}".format("ON" if "+jpeg2k" in self.spec else "OFF")]
        args += ["-DUSE_PYTHON={0}".format("ON" if "+python" in self.spec else "OFF")]
        args += ["-DUSE_QT={0}".format("ON" if "+qt" in self.spec else "OFF")]
        if "+python" in self.spec:
            args += ["-DPYTHON_VERSION={0}".format(self.spec["python"].version)]
        args += ["-Wno-deprecated-declarations"]
        return args

To build, just replace spack spec with spack install:

spack install openimageio +python ^openexr@3.1 ^python@3.9 ^py-numpy@1.20 ^boost@1.76

Drumroll.......

and.... it failed installing ncurses.

External packages

After some experimentation, it appears this error occurs with all versions of ncurses in Spack when using gcc 9.3.1, but not gcc 4.8.5.

As a quick way around this problem, I decided to install ncurses via yum.

Spack is designed to build just about everything from scratch by default -- including many libraries that would typically be installed by a package manager like yum or apt (this is because Spack is itself a package manager). However, there is a way to register "external" packages that are not managed by Spack's repository of package build recipes.

First, we start by installing ncurses:

yum install -y ncurses-devel

Next, run the following:

$ spack external find
==> The following specs have been detected on this system and added to /root/.spack/packages.yaml
autoconf@2.69    binutils@2.27.44  coreutils@8.22  diffutils@3.3     flex@2.5.37  git@1.8.3.1  groff@1.22.2   m4@1.4.16      pkg-config@0.27.1  swig@2.0.10  texinfo@5.1
automake@1.13.4  bison@3.0.4       curl@7.29.0     findutils@4.5.11  gawk@4.0.2   gmake@3.82   libtool@2.4.2  openssh@7.4p1  subversion@1.7.14  tar@1.26

This finds a bunch of libraries, but the output shows that ncurses is not among them. The docs say that special handling must be added for external packages to be found, so for now we can add it manually:

vi ~/.spack/packages.yaml

Then we add the following to the bottom of the file:

  ncurses:
    externals:
    - spec: ncurses@5.9.14+termlib
      prefix: /usr

📘 Note

When you register an external package, you are telling Spack that it is a drop-in replacement for the same package built by Spack. This means if it's a library the external package should include headers, which often means you need to install a "devel" package.

Of the external packages that were found, there is one devel package we need to install:

yum install -y binutils-devel

Ok, let's try installing again:

spack install openimageio +python ^openexr@3.1 ^python@3.9 ^py-numpy@1.20 ^boost@1.76

Success!

After a little setup and trial and error, we just built a quarter of the VFX Reference Platform with one command.

Building the VFX Reference Platform

Now let's see if we can do even more. Rather than create an ad hoc build as before, we will create a Spack "environment", which is like a python virtualenv.

First, activate shell support:

. /opt/spack/share/spack/setup-env.sh

Then create and activate the environment:

spack env create vfx2022
spack env activate -p vfx2022

Next we use spack add to add each of the packages to build.

Below are the VFX Reference Platform 2022 libraries that I could find that have matching versions in Spack (though openvdb is out of date). Incredibly, there are only 4 libraries missing from the builtin Spack repo: Aces, Ptex, FBX, and OpenColorIO.

spack add openimageio +python
spack add openexr@3.1
spack add python@3.9
spack add py-numpy@1.20
spack add boost@1.76
spack add qt@5.15 +webkit
spack add intel-tbb@2020.3
spack add intel-mkl@2020
spack add opensubdiv@3.4
spack add alembic@1.8
spack add openvdb@8

As before, we can use abstract specs when defining our environment, which will be "concretized" into specific versions and variants.

Let's take a quick pitstop to see how the environment is managed:

cat /opt/spack/var/spack/environments/vfx2022/spack.yaml 

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [openimageio+python, openexr@3.1, python@3.9, py-numpy@1.20, boost@1.76,
    qt@5.15+webkit, intel-tbb@2020.3, intel-mkl@2020, opensubdiv@3.4, alembic@1.7,
    openvdb@8]
  view: true
  concretizer:
    unify: true

To concretize our specs within a Spack environment we use spack concretize, and the result will be cached as long as the inputs in spack.yaml don't change.

Here's the result of concretization:

[vfx2022] [root@96edee2040aa ~]# spack concretize
==> Error: No version for 'python' satisfies '@2.7.18' and '@3.9.8'. Consider setting `concretizer:unify` to `when_possible` or `false` to relax the concretizer strictness.

The error indicates one of the new packages added since building openimagio is hardwired to use python@2.7.18. There's no obvious workflow for conflict resolution. Score one point for Rez, with its handy (but often overwhelming!) conflict resolutions graphs.

After some trial and error I discovered that the problem is qt. Looking at the qt package.py there's a note that webkit requires python2 for building, so I'll remove that variant for now.

spack rm qt
spack add qt@5.15  # add without +webkit

Before we concretize there's one more external dependency to setup to avoid a build problem triggered by qt's deps:

yum install -y harfbuzz-devel
echo -e "  harfbuzz:\n    externals:\n    - spec: harfbuzz@1.7.5\n      prefix: /usr" >> ~/.spack/packages.yaml
spack add harfbuzz@1.7.5

Then we re-concretize:

spack concretize

Here's the output from just the openimageio portion:

Spack installs the built packages into a centralized location, so all of the packages that were already installed during the build of openimageio are skipped this time around. Previously installed packages are prefixed with [+] while those which still need to be built have [-].

📘 Note

Spack generates a hash for each spec. This hash is a function of the full provenance of the package, so any change to the spec affects the hash. Spack uses this value to compare specs and to generate unique installation directories for every combinatorial version.

In the output above we can see that ^py-numpy@1.20.3 needs to be rebuilt. This is because in the vfx2022 environment I added a new requirement that was omitted when I built openimagio the first time: intel-mkl@2020. Since py-numpy depends on this package, a new variant of py-numpy with this altered dependency needs to be built.

Time to build all the packages that have been added to the environment:

spack install

We just built most of the VFX Reference Platform!

Managing environment variables

One of the most useful features of Rez is its ability to take a set of loose package versions and create a new shell with environment variables that are composed from the resolved packages. For example, rez env maya-2020 bifrost would set MAYA_LOCATION and PATH to load this verson of Maya and ensure that the Bifrost plugin is on the MAYA_MODULE_PATH.

Spack does not have this exact feature, but it has the building blocks to implement it. The spack load command is able to load the environment variables of explicitly provided packages, but unlike Rez or spack spec it does not accept abstract specs -- you must know exactly what you want loaded.

I forked Spack to see how hard it would be to make this work more like rez env, and it was pretty approachable. I needed to add a few simple features to existing commands:

1. Add spack load --concretize to treat the list of input specs as abstract specs to concretize.
2. Add a concretization mode that only considers installed (i.e. built) packages, and errors if it cannot resolve successfully.
3. Add spack load --deptype to control which dependency types are loaded: we only want "run" deps, so that we skip build-only dependencies.

You can see all of these changes in this PR.

With those fixes in place, here's the workflow:

Deactivate the vfx2022 environment and make sure there are no packages currently loaded:

[vfx2022] [root@661cfada1db4 ~]# spack env deactivate
[root@661cfada1db4 ~]# spack find --loaded
==> 0 loaded packages

Now load the runtime dependencies of openimagio into the current environment:

[root@661cfada1db4 ~]# spack load --deptype run --concretize openimageio

Let's confirm that we've loaded openimageio and its runtime dependencies:

[root@661cfada1db4 ~]# yum install -y which
[root@661cfada1db4 ~]# which oiiotool
/opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.3.1/openimageio-2.4.8.0-rdvmyrbos6q6om6apzqvunbnx5h6egv7/bin/oiiotool
[root@661cfada1db4 ~]# spack find --loaded
-- linux-centos7-x86_64 / gcc@9.3.1 -----------------------------
openimageio@2.4.8.0  py-numpy@1.20.3  py-pybind11@2.10.1  py-setuptools@59.4.0  python@3.9.15
==> 5 loaded packages

I still have more testing to do, but it works in my basic scenarios. And obviously we'd want to make a command that's a shortcut for spack load --deptype run --concretize since that's a mouthful.

I jumped on the Spack Slack workspace and the author and contributors were open to the idea of adding these features. But life intervened and I haven't followed through with a PR yet.

Installed package layout

The install directory seems compatible with placing on network storage.

The installed package structure is something like this:

/opt/spack/opt/spack/linux-centos7-haswell/gcc-9.3.1/gdbm-1.19-mqmfeifz6htulhb44thxp6mpw7z7x7sg

Breaking this apart:

<install_dir>/<os>/<compiler>/<package>-<version>-<variant_spec_hash>

At the root of the install directory there is a .spack-db folder that appears to hold a file-based database of the contained packages. This is a nice touch for speeding up access since there can be a lot of folders here, especially because package variants are stored flat. It does mean that you can't simply delete packages from this directory: you have to use the proper command to keep the database in sync.

A few sharp edges

Most of the problems that I encountered were with the recipes rather than the tool itself. This makes sense since it's obviously not plausible to test every combination of OS, compiler, and package. Luckily, solving my various recipe issues was pretty straightforward.

Here are some other issues that I observed:

• When using spack install or spack spec, concretizing an abstract spec that does not resolve to an existing spec will not error: it will complain later when it can't find the source.
• spack install failed building boost with -j20 but worked when omitting the flag.
• spack spec can be very slow. This may be improved by registering more external packages, which would reduce the number of dependencies to solve for.
• support for Windows is still in progress. The docs state "while this work is still in an early stage, it is currently possible to set up Spack and perform a few operations on Windows."
• The compiler must be configured before doing any spec concretization, as it will only find installed packages built with the active compiler

Closing thoughts

Rez is an environment management tool that grew build capabilities, whereas Spack seems to be the opposite: a build manager with some basic environment management. Making a robust build tool is many times harder than environment management. With a few days of work I was able to add functionality to Spack along the lines of rez-env. Making Rez's build functionality as capable as Spack would be... a lot harder. Multiple orders of magnitude more difficult.

I think the payoff of Spack would be high, especially in terms of enabling our industry to cooperate on build problems that we're all currently solving on our own. Imagine if the ASWF hosted a repo of battled-tested package recipes to build the VFX Reference Platform and ASWF libraries. With Spack that would be achievable with relatively little effort.

Spack has the potential to be a compelling package manager for the VFX/Animation industry, with a bit of work. It's certainly interesting how much overlap there is between our industry and supercomputing community in terms of build and deployment concerns. Who knows, maybe it could grow into a fruitful partnership.

Let me know what you think in the comments!

Comments

[infintropy]

@chadrik very cool insights. Wondering if awsf/rez could leverage this with its own interoperability layer as they received downstream of pip. Have you used Singularity/Apptainer to leverage more lift/shift to the concept of existing packages? Its another offering from hpc that is very nice and I know animalogic to be using it in prod

[Chad Dombrova]

Treating apps as containers like Singularity/Apptainer aims to do is my dream solution, so I’m paying attention to that space, but I’m now at a Windows-based facility so it may be a dream deferred, for now.

[Donal McMullan]

Hey Chad - thanks for sharing this. It does look really interesting - is it something you're continuing to work on?

A couple of other projects from the HPC community that may have potential in vfx pipelines are Globus XIO (wrt 'OpenFileIO') and GridFTP (for transcontinental/intercontinental data transfer with UDT).

[Chad Dombrova]

Hi Donal, unfortunately I won’t have much time to continue this experiment myself, but I’m happy to help others or get involved with working groups that may want to push it forward.

Thanks for the software recs, will check them out.

[Alex Fuller]

Nice write-up @chadrik I didn't know of Spack until now, thanks for sharing.

I've been playing around with another approach via Buildstream: buildstream.build/index.html not for VFX platform but by making a fork of carbonOS to streamline a custom immutable Linux distro that can use something like distrobox+rocky on top, but maybe looking at it later to make VFX ref platforms as a Flatpak runtime.

Its design was I think more inspired for catering Flatpak deployment and immutable Linux distros with a declarative way of specifying a huge permutation of build systems, compilers, architectures, etc. and making it work in a CI way with Linux containers to sandbox it up. I'm not familiar with Spack but I do know Rez, and I guess Buildstream on the surface looks more like Spack than Rez.

The environment variable or package deployment to a centralised network share probably is lacking in Buildstream or out of scope, its intended use is to deploy something like Flatpak runtimes and apps and deploying multiple versions via OSTree and could de-duplicate the final result and not need environment variables to repath stuff in $PATH - immutable OCI overlays can also output too, then you cache it to the machine itself over some self-hosted protocol.

I think Rez itself or something similar could be grafted on the end of it as a custom output plugin of Buildstream though...