A few months back, Google shared about Scalable Nearest Neighbors, ScaNN (Paper, Code) for efficient vector similarity search. It seemed to beat the SOTA benchmarks on angular distance (i.e., >2x throughput for a given recall level).
ANN Benchmarks on the GloVe embeddings (dim=100) (source)
Recently, I found some time to try it out but was frustrated by how tricky it was to install on a Mac. Here are the steps I took to install it successfully.
First, we install the necessary compilers.
brew install bazel brew install llvm brew install gcc
Then, we set up our Python version via
brew update && brew upgrade pyenv pyenv --version > pyenv 1.2.21 pyenv install 3.8.6. # Doesn't work with 3.9 yet pyenv local 3.8.6 python --version > Python 3.8.6
Now, we create our virtual environment.
python -m venv .venv source .venv/bin/activate pip install --upgrade pip
ScaNN is part of the google-research repo which is huge. There are more than 200 directories in there and we don’t need all of them. Thus, we’ll do the following to only checkout the ScaNN directory.
git clone --depth 1 --filter=blob:none --no-checkout https://github.com/google-research/google-research.git git checkout master -- scann cd scann
Next, we’ll need to install the Python dependencies.
pip install wheel python configure.py # There might be complaints about "tensorflow 2.3.1 requires numpy<1.19.0,>=1.16.0, but you'll have numpy 1.19.2 which is incompatible." but it's fine
Several issues prevent a direct installation and we’ll be manually fixing them here.
First, we’ll update
.bazel-query.sh. (It’s not absolutely necessary to update
.bazel-query.sh but I thought we do it anyway for completeness). We should replace:
Then, we’ll need to update the C++ imports by replacing (there are four of these):
Now, we can build it via
bazel. Instead of using
clang-8 as specified, I just used the latest version of
clang and it worked fine.
CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/gcc/bin/gcc bazel build -c opt --copt=-mavx2 --copt=-mfma --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --cxxopt="-std=c++17" --copt=-fsized-deallocation --copt=-w :build_pip_pkg
If it builds successfully, we should see output similar to this.
INFO: Elapsed time: 316.366s, Critical Path: 206.32s INFO: 1066 processes: 319 internal, 747 local. INFO: Build completed successfully, 1066 total actions
Then, we build the Python wheel:
And now we can install it:
pip install scann-1.1.1-<replace with your package suffix>
You can test if the installation was successful in Python:
import scann scann.scann_ops_pybind.builder() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: builder() missing 3 required positional arguments: 'db', 'num_neighbors', and 'distance_measure'
You should get the error if installation was successful. Here’s a sample demo on using it.
Top comments (3)
Thanks, I had some issues installing on the 1.0.0 version from source, it became easier once they released the pip package.
I wrote up a post about using it on docker if you are interested.
whenever I do git checkout master -- scann I get this error after it processes some already: error: cannot create standard output pipe for fetch-pack: Too many open files. I then encounter some issues when trying to change my .bazelrc file because I dont see where to make the '#include change'. I suspect these issues are not allowing my bazel build correctly cause I then get this error when I run 'CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/gcc/bin/gcc bazel build -c opt --copt=-mavx2 --copt=-mfma --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --cxxopt="-std=c++17" --copt=-fsized-deallocation --copt=-w :build_pip_pkg':
ERROR: /Users/akoke/google-research/scann/BUILD.bazel:6:10: no such package 'scann/scann_ops/py': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
had to do
git checkout master -- scann