Biohub - Cell Tracking During Development
Abstruct
- A guide on how to download python wheel files for custom packages(like Zarr)and import them in internet-disabled Kaggle notebooks.
- Resolves dependency version mismatch issues between local and Linux-based Kaggle execution environments.
Introduction and Background
Kaggle code competitions require you to submit your final predictions by running a notebook inside a container with the internet connection disabled.
This is a security precaution to prevent cheating, but it introduces a major headache: if you need to use an external library(such as zarr)that is not pre-installed in the default Kaggle notebook environment, you cannot simply run !pip install zarr. Doing so will fail because the notebook has no internet access to fetch packages from PyPI.
In this article, I will explain a reliable method to download, upload, and install any custom packages completely offline inside Kaggle.
Content
Overall Flow
The general workflow for offline package installation is shown below. In short, we download the necessary wheel(.whl)files on our local PC, upload them to Kaggle as a private Dataset, attach it to our notebook, and install from the local mount path.
Step 1: Download Wheel Files Locally
We must download the packages targeting the specific OS and Python version of the Kaggle environment.
Kaggle notebooks run on Linux, using Python 3.12. If you download wheel files using default Windows/macOS settings, they will not be compatible.
Open your command prompt or terminal, and run the following command. The key is to force the Python version to 3.12 and the platform to manylinux:
pip download zarr -d ./zarr_wheels_fixed --only-binary=:all: --platform manylinux2014_x86_64 --python-version 3.12 --implementation cp
This will download zarr and all of its dependencies(including libraries like numcodecs)as .whl files into the zarr_wheels_fixed directory.
Step 2: Upload Wheels to Kaggle as a Dataset
(1)Navigate to the Kaggle Datasets page and click "New Dataset".
(2)Drag and drop all the .whl files downloaded inside your zarr_wheels_fixed folder.
(3)Name your dataset(e.g., zarr-offline-whl)and choose your visibility preference(Private is fine).
(4)Click "Create" to compile the dataset.
Step 3: Add the Dataset and Install Offline
(1)In your Kaggle notebook editor, toggle the settings panel on the right side and ensure "Internet" is turned off.
(2)Click "Add Input" at the top right of the notebook. Search for your uploaded dataset under your profile and click add.
(3)In the very first cell of your notebook, run the following pip install command, directing pip to look into the local mounted path of your dataset:
!pip install --no-index --find-links=/kaggle/input/datasets/aaaa1597/zarr-offline-installation-wheels/zarr_wheels_fixed zarr
This tells pip not to query index servers(--no-index)and instead look inside the wheels directory. The library will install immediately without any internet connection.
Summary
Even in competitions with internet disabled, you can use any library freely using this method.
Note
For the zarr library, you can easily install it by simply running the following code block at the beginning of your notebook. Please feel free to use it:
!pip install --no-index --find-links=/kaggle/input/datasets/aaaa1597/zarr-offline-installation-wheels/zarr_wheels_fixed zarr
Japanese Version of This Series
You can read the original Japanese version of this article on Zenn:
Conclusion
In the next part, we will dive deep into the source code of the baseline pipeline, explaining OME-Zarr indexing, Min-Max normalization, and spatiotemporal cell tracking.
I hope this helps!
Top comments (0)