Pip broken under Mac OS X Catalina? TLDR;
pip install --global-option=build_ext --global-option="-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/" --global-option="-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/" --prefer-binary -r requirements.txt
As Apple has announced, Mac OS Catalina now runs under its own read only file system:
macOS Catalina runs on a dedicated, read-only system volume called Macintosh HD. This volume is completely separate from all other data to help prevent the accidental overwriting of critical operating system files. [1]
Sounds great. But if you need to change that volume, or if you use software that expects to be able to find files on that read only volume that aren't there and can't be added, you're kind of stuck.
One example is Python's pip which, for some packages, will expect to find header files located in /usr/include. Apple gave developers a get-out-of-jail-free card in Mojave:
As a workaround, an extra package is provided which will install the headers to the base system. In a future release, this package will no longer be provided. [2]
and in true Apple fashion, the package was taken away. This creates problems if you want to install commonly used, but maybe not so commonly maintained, libraries. One example is Pillow, an image processing library. This will well and truly fail to compile if you just try pip install Pillow. It looks for the headers and libraries, and doesn't find them.
What can we do? Well, for pip there's a few options. The first is simply not to compile at all, but to prefer a binary. This is as simple in some cases as the --prefer-binary option. But what if, like Pillow, there is no precompiled binary for your platform? Or for reasons of information security, you have to compile it yourself?
Fortunately, pip can be directed to look in different places using the "global-options" flag. To get Pillow to build, use the command at the top of this article, reproduced below:
pip install --global-option=build_ext --global-option="-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/" --global-option="-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/" --prefer-binary -r requirements.txt
To understand this more fully, take a look at https://stackoverflow.com/questions/18783390/python-pip-specify-a-library-directory-and-an-include-directory.
If you need to do a similar function as part of other install tools, common environment variables are LD_LIBRARY_PATH (deprecated[3], but still commonly used), LIBRARY_PATH and INCLUDE_PATH. Setting these to the library paths (/Library ... /usr/lib) and include paths (/Library ... /usr/include) respectively may help.
[1]https://support.apple.com/en-gb/HT210650
[2]https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035624
[3]https://stackoverflow.com/questions/18241517/c-include-path-vs-ld-library-path
Top comments (0)