I just joined a new project — one that runs on a big HPC cluster. I opened the project README.md and saw something like this:
singularity exec docker://python:3.11 python train_model.py --data /shared/datasets/train
I had no idea what singularity was. 😅 So I did what felt natural — typed the Docker command instead:
docker run python:3.11 python train_model.py --data /shared/datasets/train
And the terminal replied with:
bash: docker: command not found
I messaged my project manager. His reply was short:
"We don't use Docker here. We use Singularity."
I stared at the message, thinking: "I have been using Docker for years. I know docker build, docker run, docker push like the back of my hand. And now none of that works here?"
That's how it all started.
🤔 "Why Not Docker?"
I loved Docker. Years of packaging apps in containers, deploying to production, running ML training pipelines. It was part of how I worked every day.
So I asked my project manager to install Docker on the cluster. His reply came quickly:
"We can't. There are legal reasons we can't use Docker here. Company policy. But also — Singularity is a better fit for what we do."
He didn't go into all the legal details, but explained the technical side in a Google Meet session. Docker needs a background daemon running all the time — a service that sits there waiting for your commands. On a shared HPC cluster where hundreds of researchers submit jobs, that adds complexity and overhead. Singularity doesn't need any daemon — you just run it directly. No background process. And you are the same user inside the container as you are outside. No switching to root, no permission confusion.
That last part sounded too good to be true. But it was true.
| What? | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Main idea | Isolation (microservices) | Integration (HPC) |
| Background daemon? | Yes (always running) | No (daemonless) |
| Who are you inside? | root by default | Same user as host |
| Image format | Multi-layer, managed by daemon | Single .sif file |
| File access | Must mount volumes | Auto-mounts $HOME
|
| Network | Isolated by default | Host network |
💡 Docker wants to isolate your app from the system. Singularity wants to integrate your app with the system.
That made sense. I started learning Singularity and writing down every Docker command I knew next to its Singularity equivalent. This article is that cheat sheet.
📦 Image Management
First thing I needed was a Python image. In Docker, I would type docker pull python:3.11. In Singularity, the command is almost the same — with one small twist:
singularity pull docker://python:3.11
See that docker:// prefix? All my Docker Hub images still work. Every image I had ever used — GPU-enabled, data science, notebook servers — still available.
But instead of layers hidden in Docker's storage, I got a single file:
ls -lh python_3.11.sif
# -rwxr-xr-x 1 dalirnet dalirnet 385M Feb 25 09:15 python_3.11.sif
That .sif file is the image. A real file in my directory. I can cp it, scp it to another node, or rsync it anywhere. Try that with Docker — you need a registry, accounts, push, pull... With Singularity, you just copy a file.
Want to delete it? rm python_3.11.sif. No docker system prune, no dangling images.
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Pull image | docker pull ubuntu:22.04 |
singularity pull docker://ubuntu:22.04 |
| Build image | docker build -t myimage . |
singularity build myimage.sif myimage.def |
| List images | docker images |
ls *.sif (they're just files!) |
| Inspect image | docker inspect myimage |
singularity inspect myimage.sif |
▶️ Running Containers
In Docker, I would do docker run python:3.11 python script.py. In Singularity, the keyword is exec instead of run (run exists too — more on that in a second):
singularity exec python_3.11.sif python --version
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Run a command | docker run myimage cmd |
singularity exec myimage.sif cmd |
| Interactive shell | docker run -it myimage bash |
singularity shell myimage.sif |
| Default command | docker run myimage |
singularity run myimage.sif |
| Background | docker run -d myimage |
singularity instance start myimage.sif name |
📂 File System & Volumes
This is where Singularity first surprised me. I typed singularity shell python_3.11.sif, then ls — and saw all my files. My notebooks. My training scripts. My config files. Everything from my home folder, right there.
In Docker, you see an empty filesystem unless you mount your folder with -v. In Singularity, your home directory, current working directory, /tmp, and system paths like /proc and /sys are automatically available ✅.
So my old Docker habit:
docker run -v $(pwd):/workspace -w /workspace python:3.11 python analysis.py
Becomes just:
singularity exec docker://python:3.11 python analysis.py
No -v. No -w. Your current directory is already there.
When you need folders outside your home directory, use --bind:
singularity exec \
--bind /shared/datasets:/data \
--bind /scratch/$USER:/scratch \
myimage.sif python train.py --data /data/train --output /scratch/checkpoints
You can also make binds read-only: --bind /shared/datasets:/data:ro.
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Bind mount | -v /host:/container |
--bind /host:/container |
| Current dir | -v $(pwd):/app |
Already there |
| Read-only | -v /host:/container:ro |
--bind /host:/container:ro |
| Multiple | -v /a:/a -v /b:/b |
--bind /a:/a,/b:/b |
| Working dir | -w /app |
--pwd /app |
⚙️ Environment Variables
This one caught me off guard. In Docker, the container starts with a clean environment. If you need an API key inside, you pass it explicitly:
docker run -e API_KEY=abc123 myimage python train.py
Singularity does the opposite — it inherits your entire host environment by default ✅. Your $PATH, your custom variables — all available inside the container.
At first I thought this was great. Then I hit a bug where my container's Python was fighting with my host's Python paths because environment variables were leaking in. That's when I learned about --cleanenv:
# Clean environment — recommended for reproducible experiments
singularity exec --cleanenv myimage.sif python train.py
# Clean env + only the variables you need
singularity exec --cleanenv --env API_KEY=abc123 myimage.sif python train.py
# Or use an env file
singularity exec --cleanenv --env-file .env myimage.sif python train.py
My advice: always use --cleanenv for training runs. The inherited environment is handy for quick interactive work, but for anything reproducible, you want a clean slate.
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Set variable | -e VAR=value |
--env VAR=value |
| Inherit host env | No | Default |
| Env file | --env-file .env |
--env-file .env |
| Clean env | Default | --cleanenv |
🌐 Networking
In Docker, every container gets its own isolated network. Want to run a notebook server? You need port mapping:
docker run -p 8888:8888 mynotebook-image
In Singularity, there is no network isolation. The container uses the host network directly. Start a service on port 8888 inside the container, and it is on port 8888 on your machine. No mapping needed ✅:
singularity exec myimage.sif python -m notebook --port 8888
Running notebook servers, dashboards, monitoring tools — no more figuring out port mappings. Just start the service and go to localhost:port.
The downside? Two users on the same node starting something on port 8888 will conflict. But our cluster setup handles that by assigning different ports.
| Docker 🐳 | Singularity 🔬 | |
|---|---|---|
| Default | Isolated bridge | Host network |
| Port mapping | -p 8888:8888 |
Not needed |
| Custom network | docker network create |
Not available |
🔄 Running Services (Instances)
Sometimes you need something running in the background — a notebook server, a database, a monitoring dashboard. In Docker, you use -d to detach. Singularity has instances:
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Start | docker run -d --name nb myimage |
singularity instance start myimage.sif nb |
| List | docker ps |
singularity instance list |
| Stop | docker stop nb |
singularity instance stop nb |
| Exec | docker exec nb ls |
singularity exec instance://nb ls |
| Logs | docker logs nb |
Check instance-specific logs |
One important difference: in Docker, the daemon keeps your containers alive even if you log out. In Singularity, instances are tied to your session. If you disconnect from the cluster, they stop. For long-running services, you need a job scheduler — which is usually how HPC clusters work anyway.
📄 Definition Files
Every Docker user knows the Dockerfile. Singularity has its own version called a definition file (.def). Different structure, same idea — a recipe for building your image.
A Dockerfile:
FROM python:3.11-slim
RUN pip install numpy pandas scikit-learn matplotlib
COPY analysis.py /app/
WORKDIR /app
CMD ["python", "analysis.py"]
The same thing as a Singularity definition file:
Bootstrap: docker
From: python:3.11-slim
%post
pip install numpy pandas scikit-learn matplotlib
%files
analysis.py /app/
%environment
export LC_ALL=C
%runscript
cd /app
exec python analysis.py
| Dockerfile | Singularity | Purpose |
|---|---|---|
FROM |
Bootstrap: docker + From:
|
Base image |
RUN |
%post |
Build commands |
COPY |
%files |
Copy files in |
ENV |
%environment |
Environment vars |
CMD |
%runscript |
Default command |
ENTRYPOINT |
%startscript |
Instance command |
LABEL |
%labels |
Metadata |
WORKDIR |
Set in %runscript
|
Working dir |
Most of the time, you don't need a .def file at all. If you already have a Docker image, convert it directly:
singularity build myenv.sif docker://myregistry/gpu-image:latest
I only started writing .def files when I needed a custom environment that didn't exist on Docker Hub. For everything else, docker:// was enough.
🧪 Sandbox Mode
When I need to experiment with new packages before committing to a build, I create a writable sandbox — a draft environment I can mess around in, then freeze into a clean image:
singularity build --sandbox myenv/ myenv.def # create writable folder
singularity shell --writable myenv/ # shell in and install stuff
# pip install some-new-package
# python -c "import some_new_package" # test it
sudo singularity build myenv.sif myenv/ # freeze when happy
In Docker, you would do docker run -it myimage bash then docker commit, but the sandbox approach feels more intentional.
🎮 GPU Access
This is where Singularity really shines.
In Docker, you need --gpus all:
docker run --gpus all gpu-image:latest python train.py
In Singularity, you add --nv:
singularity exec --nv gpu-image.sif python train.py
That's it. No nvidia-docker, no container runtime config, no Docker daemon GPU passthrough setup. Just --nv and your NVIDIA drivers are available. For AMD GPUs, it's --rocm.
Combined with a job scheduler, a typical training job looks like this:
singularity exec --nv \
--bind /shared/datasets:/data \
--bind /scratch/$USER:/scratch \
gpu-image.sif \
python train.py \
--data /data/train \
--output /scratch/checkpoints \
--epochs 100 \
--batch-size 512 \
--gpus 4
Submit the job and check the results later.
| GPU | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| NVIDIA | --gpus all |
--nv |
| Specific GPU | --gpus '"device=0"' |
CUDA_VISIBLE_DEVICES=0 |
| AMD | --device /dev/kfd --device /dev/dri |
--rocm |
📤 Registry & Sharing
In Docker, sharing an image means pushing to a registry. Both sides need accounts and permissions.
In Singularity, you can push to the Singularity Library:
singularity remote login
singularity push myimage.sif library://myname/default/myimage:v1.0
But what I actually do: just copy the file.
cp myimage.sif /shared/containers/team-env.sif
Anyone on the cluster can run singularity exec /shared/containers/team-env.sif python train.py and get the exact same environment. Same packages, same versions, same GPU libraries. No registry, no login. Just a file on a shared filesystem.
| Task | Docker 🐳 | Singularity 🔬 |
|---|---|---|
| Login | docker login |
singularity remote login |
| Push | docker push user/image:tag |
singularity push image.sif library://user/image:tag |
| Share | Push to registry | Copy the .sif file |
🎼 Multi-Container Orchestration
Singularity has no built-in docker-compose alternative ⚠️. If you come from a world of docker-compose up with web servers, databases, and caches all wired together, this will feel like a step back.
But on HPC, you usually don't need it. Most workloads are single-container jobs. Your training script runs inside one environment, reads data from shared storage, and writes checkpoints to scratch space.
When you do need multiple services, you have a few options:
📜 Option 1: A simple shell script. Start each service as a Singularity instance, connect them through localhost (since they share the host network):
#!/bin/bash
singularity instance start postgres.sif db
sleep 5
singularity instance start --env DATABASE_URL=postgresql://localhost/mydb tracker.sif tracker
singularity instance list
🔧 Option 2: singularity-compose — a community tool that reads YAML files similar to docker-compose.yml. It works for simple setups, but it is not actively maintained.
| Feature | Docker Compose | Singularity Compose |
|---|---|---|
| Network isolation | ✅ Full | ❌ Host only |
| Service discovery | ✅ DNS | ⚠️ Limited |
| Health checks | ✅ Built-in | ❌ Manual |
| depends_on | ✅ Full | ⚠️ Limited |
| Secrets | ✅ Built-in | ❌ Env vars |
| Production ready | ✅ Yes | ⚠️ Not actively maintained |
🔍 Troubleshooting
Here are the problems I hit and how I solved them:
🐍 Host Python leaking in — my $PYTHONPATH and other environment variables were being inherited by the container, causing import errors. Fix: always use --cleanenv for training runs.
✏️ "Read-only file system" errors — Singularity images are read-only by default. If your script tries to write to /opt or /usr, it will fail. Fix: write to $HOME, use --bind, or use sandbox mode.
🔌 Port conflicts — two users starting a service on the same port will conflict since Singularity shares the host network. Fix: always pick a random port, or let your job scheduler assign one.
💾 Disk space — .sif files can be large (multi-GB for GPU images) and there is no layer sharing. Fix: keep shared images in /shared/containers/ instead of each person having their own copy.
| Problem | Docker fix 🐳 | Singularity fix 🔬 |
|---|---|---|
| Permission denied | --user $(id -u):$(id -g) |
Shouldn't happen (same user) |
| Can't write to a path | Mount a volume | Use $HOME, --bind, or sandbox |
| Port in use | Change port mapping | Pick a different port |
| Out of space | docker system prune |
rm *.sif or use shared images |
| Wrong Python version | Check base image |
--cleanenv to stop host env leaking in |
| Package not found | Install in Dockerfile | Install in %post or use sandbox |
💡 What I Learned
Where Singularity wins
- No daemon — just run it
- Single-file images —
cpfor instant reproducibility - Same user inside and outside — no permission headaches
- Simple GPU access — just
--nv - Works on shared HPC clusters without special privileges
Watch out for
- Building images needs root (
--fakerootor--remoteas workaround) - No network isolation
- No built-in compose/orchestration
- No layer caching — full rebuild every time
- Host environment can leak in — always use
--cleanenv
🎬 The End
That command not found on my first day scared me. I thought none of my Docker experience would transfer.
But it did 😊. Every Docker image still works — just add docker:// in front. Every concept — images, containers, mounts, environment variables — still applies. Same mental model, different tool.
Singularity taught me something unexpected: isolation is not always the goal. Docker keeps containers separate from the host. But on a shared cluster, you actually want the container to feel like part of the system. You want your files there. You want the GPU drivers to just work. You want to submit a job and not worry about daemon sockets and port mappings.
My workflow now: Docker on my MacBook for local testing. Singularity on the cluster for real training. Same images, same Dockerfiles, different last mile.
They are not competitors. They answer different questions:
- 🐳 Docker: "How do I isolate this app?"
- 🔬 Singularity: "How do I bring this app into the researcher's environment?"
Sometimes the second question is the right one.
If you found this useful, follow me here on dev.to and check out my GitHub.
Top comments (0)