DEV Community: SHIGRAF SALIK

How to Install `pairtools` Properly in Python 3.12

SHIGRAF SALIK — Thu, 10 Apr 2025 11:46:44 +0000

If you're trying to install pairtools—a toolkit for parsing, sorting, and analyzing Hi-C paired-end sequencing data—but run into linking issues with pysam, you're not alone.

A common and frustrating issue is this:

ImportError: libchtslib.cpython-312-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

Even after using --no-build-isolation, you might still see errors like:

/tmp/pip-build-env-*/overlay/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so => not found

This happens because the compiled shared object is wrongly linked to temporary build paths. In this guide, I'll show you how to fix this for good.

✅ Goal

To install pairtools in an isolated environment (like GitHub Codespaces), ensuring that:

pysam is installed from source.
Shared libraries are not linked to /tmp or other temporary build paths.
All 17 tests pass using pytest.

📦 Step-by-Step Installation

🔄 1. Start Fresh (Use a Clean Codespace or Environment)

Delete and recreate the Codespace or virtual environment to prevent cached artifacts from interfering.

🔧 2. Install System Dependencies

sudo apt update && sudo apt install -y build-essential libhts-dev python3-dev

This ensures your system has the compilers and headers required for building C extensions like those in pysam.

❌ 3. Uninstall Existing Builds

Remove any previously installed versions of pysam or pairtools:

pip uninstall pysam pairtools -y

🧪 4. Build `pysam` from Source

This is crucial. We want to avoid using binary wheels:

pip install "pysam==0.23.0" --no-binary pysam --no-cache-dir --verbose

✅ Check for .so files:

ls /home/codespace/.python/current/lib/python3.12/site-packages/pysam/*.so

You should see something like:

libchtslib.cpython-312-x86_64-linux-gnu.so
csamtools.cpython-312-x86_64-linux-gnu.so

🔍 5. Verify Proper Linking of `pysam`

ldd /home/codespace/.python/current/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so

Ensure the output does not reference any /tmp paths.

Expected:

linux-vdso.so.1 => ...
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f...)

📂 6. Clone and Install `pairtools`

Navigate to your cloned pairtools directory:

cd /workspaces/pairtools

Install in editable mode with --no-build-isolation:

pip install -e .[all] --no-build-isolation --no-cache-dir --verbose

The [all] includes all optional dependencies, and --no-cache-dir avoids reuse of broken builds.

🧬 7. Confirm `pairtools` Linking

ldd /workspaces/pairtools/pairtools/lib/parse_pysam.cpython-312-x86_64-linux-gnu.so

You should see it link to:

/home/codespace/.python/current/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so

There should be no reference to /tmp/pip-build-env-*.

✅ 8. Run Tests

Now validate the setup:

/home/codespace/.python/current/bin/python -m pytest -v

You should see:

collected 17 items
...
17 passed in ...

Note: At the time of writing this article, the Pairtools had 17 tests in total. Kindly recheck it to see if there has been any addition to the tests using:

pytest --collect-only

🧹 If Issues Persist: Manual Build

If you still see broken paths in the ldd output, do this:

cd /workspaces/pairtools
rm -rf build dist *.egg-info pairtools/lib/*.so
python setup.py build_ext --inplace --verbose
pip install -e .[all] --no-build-isolation --no-cache-dir --verbose

Also check and clear any conflicting environment variables:

unset LD_LIBRARY_PATH
unset PYTHONPATH

🧠 Why This Works

--no-build-isolation forces pip to use your installed pysam, not a temp one.
Building from source ensures .so files are in the correct, persistent location.
Cleaning build artifacts ensures you're not pulling in a mislinked .so.

💡 Pro Tip: Verify Shared Object Path

Always check your .so dependencies using:

ldd <your-module>.so

If it points to /tmp, rebuild from scratch.

📌 Final Words

Installing pairtools correctly in Python 3.12, especially in ephemeral environments like GitHub Codespaces, requires care. By following this guide, you can ensure proper linking and a smooth installation with all tests passing.

A Comprehensive Guide to Installing Pairtools

SHIGRAF SALIK — Fri, 28 Mar 2025 18:07:13 +0000

Pairtools is a powerful Python package for processing paired-end sequencing data, widely used in bioinformatics for tasks like Hi-C analysis. However, installing it can be tricky due to dependency linking issues, particularly with pysam. In this article, I'll walk you through a step-by-step process (that I used to install pairtools) to install pairtools successfully, avoiding common pitfalls like the dreaded ImportError caused by broken library links. By the end, you'll have a working setup, verified by running pairtools' test suite.
This guide is based on real-world troubleshooting in a GitHub Codespace environment as of March 28, 2025, using pairtools 1.1.3 and pysam 0.23.0.

Setup

Pairtools works best in Unix based systems due to it's dependence on pysam which in turn utilizes htslib.
Therefore this installation is primarily meant for Linux platform, although the results could be replicated for MacOS too.
For Windows machine, there is a workaround method to install it by using tools like gcc.

Why Installation Can Be Challenging

Pairtools relies on pysam, a Python wrapper for the htslib C library, which involves compiling Cython extensions (e.g., parse_pysam.so). During installation, these extensions must link to pysam's shared libraries (e.g., libchtslib.so). A common issue arises when pip's build isolation creates temporary environments, causing pairtools to link against paths like /tmp/pip-build-env-…/pysam/libchtslib.so. These paths vanish post-installation, leading to runtime errors like:

ImportError: /tmp/pip-build-env-k1zoq23s/overlay/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

Our goal is to ensure pairtools links to a persistent pysam installation, avoiding temporary paths. GitHub Codespaces, with its Ubuntu-based environment, provides a clean slate to achieve this - if set up correctly.

Prerequisites

GitHub Codespace: Access to a Codespace for your pairtools repository (e.g., forked from open2c/pairtools).
Basic Terminal Skills: Familiarity with running commands in a Linux terminal.
Repository Cloned: Your Codespace should have the pairtools repo checked out (e.g., at /workspaces/pairtools). ## Step-by-Step Installation Guide ### Step 1: Start with a Fresh Codespace A clean environment is critical to avoid cached build artifacts that can cause linking errors.
Action: Delete your existing Codespace and create a new one. - Go to the GitHub Codespaces dashboard (https://github.com/codespaces). - Find your current Codespace for the pairtools repo, delete it, and click "New Codespace" to start fresh.
Why: This ensures no stale dependencies or mislinked libraries interfere. ### Step 2: Install System Dependencies Pysam and pairtools require system libraries and development tools for compilation.
Command:

bash sudo apt update && sudo apt install -y build-essential libhts-dev python3-dev

Details: - build-essential: Provides gcc, make, and other compilation tools. - libhts-dev: Headers for htslib, used by pysam. - python3-dev: Python development headers for building extensions.
Why: These ensure successful compilation of pysam and pairtools from source. ### Step 3: Uninstall Existing Packages Remove any prior installations to prevent conflicts.
Command:

bash pip uninstall pysam pairtools -y

Why: A clean slate avoids version mismatches or leftover files. ### Step 4: Install pysam from Source Build pysam explicitly to place its shared libraries in a persistent location.
Command:

bash pip install "pysam==0.23.0" - no-binary pysam - no-cache-dir - verbose

Details: - - no-binary pysam: Forces a source build, avoiding precompiled wheels that might mislink. - - no-cache-dir: Prevents reusing cached files that could cause issues. - - verbose: Shows build output for debugging if needed.
Verification: - Check installed .so files:

bash ls /usr/local/python/3.12.1/lib/python3.12/site-packages/pysam/*.so

Expect files like libchtslib.cpython-312-x86_64-linux-gnu.so.
- Verify linking:

bash ldd /usr/local/python/3.12.1/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so

Ensure no /tmp/ paths appear; expect system libraries like /lib/x86_64-linux-gnu/libc.so.6.

Why: A source-built pysam ensures its libraries are correctly placed and linked. Note: The exact path (/usr/local/python/3.12.1/) may vary slightly depending on your Codespace's Python setup. Replace it with /home/codespace/.python/current/ if that's your environment. ### Step 5: Install pairtools Without Build Isolation Install pairtools in editable mode, linking it to the installed pysam.
Commands:

bash cd /workspaces/pairtools rm -rf build dist *.egg-info pairtools/lib/*.so # Clean old build artifacts pip install -e .[all] - no-build-isolation - no-cache-dir - verbose

Details: - -e: Editable mode, useful for development. - [all]: Installs optional dependencies (e.g., bioframe). - - no-build-isolation: Uses the current environment's pysam, avoiding temporary build paths.
Verification: - Check linking:

bash ldd /workspaces/pairtools/pairtools/lib/parse_pysam.cpython-312-x86_64-linux-gnu.so

Look for:

/usr/local/python/3.12.1/lib/python3.12/site-packages/pysam/libchtslib.cpython-312-x86_64-linux-gnu.so (0x…)

No /tmp/ paths should appear.

Why: - no-build-isolation ensures pairtools' extensions link to the installed pysam. ### Step 6: Run Tests to Confirm Test the installation to ensure everything works.
Command:

bash /usr/local/python/3.12.1/bin/python -m pytest -v

Expected Output: All 17 tests should collect and run, showing pass/fail results. No ImportError for libchtslib.so.
Why: This verifies that pairtools and its dependencies are fully functional. ### Step 7: Use pairtools Try it out!
Command:

bash /usr/local/python/3.12.1/bin/pairtools - help

Expected Output: Displays the command-line help, confirming the CLI is operational. ## Troubleshooting Common Issues ### Linking Still Points to /tmp/
Symptom: ldd shows /tmp/pip-build-env-…/pysam/libchtslib.so => not found.
Fix: - Clean and rebuild manually:

bash cd /workspaces/pairtools rm -rf build dist *.egg-info pairtools/lib/*.so python setup.py build_ext - inplace - verbose pip install -e .[all] - no-build-isolation - no-cache-dir - verbose

- Recheck with ldd.

NumPy Version Warnings

Symptom: Warnings like numpy 1.26.4 is incompatible with bioframe 0.7.2.
Fix: - Downgrade NumPy:

bash pip install "numpy<2"

- Rerun tests.

Missing System Dependencies

Symptom: Build fails with errors about missing htslib or gcc.
Fix: Reinstall system dependencies (Step 2). ## Why This Works
Fresh Codespace: Eliminates cached artifacts that cause linking errors.
Source-Built pysam: Ensures shared libraries are in a stable location.
No Build Isolation: Forces pairtools to use the installed pysam, avoiding temporary paths.
Ubuntu Environment: Native Linux in Codespaces provides reliable library linking compared to WSL quirks. ## Conclusion Installing pairtools in GitHub Codespaces doesn't have to be a headache. By starting fresh, building pysam from source, and installing pairtools without build isolation, you can sidestep linking issues and get to work on your paired-end sequencing projects. This method has been battle-tested to resolve the ImportError for libchtslib.so, ensuring a robust setup as of March 2025. Now that your pairtools is installed, explore its capabilities - parse Hi-C data, run stats, or dive into its test suite for inspiration. Happy coding!