Auto‑Generate Docker‑Compose for Free‑Tier ML Experiments

Free‑tier cloud credits run out faster than expected – a single Jupyter notebook on a GPU spot instance can consume a 10 GB quota in under an hour. The bottleneck is not the model, it is the time spent wiring containers, mounting data, and reproducing the same environment on a new VM. ## 1. Define the experiment once, deploy everywhere
The most reliable way to keep the environment consistent across GCP Free Tier, Azure for Students, and AWS Free Tier is to describe it in a docker-compose.yml. Write the file once, then spin up a replica with a single command.

version: "3.9"
services:
 notebook:
 image: jupyter/datascience-notebook:latest
 ports:
 - "8888:8888"
 volumes:
 - ./work:/home/jovyan/work
 environment:
 - GRANT_SUDO=yes
 command: start-notebook.sh --NotebookApp.token=''
 tensorboard:
 image: tensorflow/tensorflow:2.13.0-jupyter
 ports:
 - "6006:6006"
 volumes:
 - ./logs:/logs
 command: tensorboard --logdir=/logs --host=0.0.0.0

python

Takeaway: Store the compose file in your repo; a single docker compose up -d reproduces the exact stack on any free‑tier VM. ## 2. Automate file generation with a Python helper
Manually editing YAML for each experiment is error‑prone. The snippet below generates a compose file based on a tiny JSON spec. It works on a laptop without extra dependencies.

import json, pathlib
spec = {
 "services": {
 "notebook": {
 "image": "jupyter/datascience-notebook:latest",
 "ports": ["8888:8888"],
 "volumes": ["./work:/home/jovyan/work"],
 "environment": ["GRANT_SUDO=yes"],
 "command": "start-notebook.sh --NotebookApp.token=''"
 },
 "tensorboard": {
 "image": "tensorflow/tensorflow:2.13.0-jupyter",
 "ports": ["6006:6006"],
 "volumes": ["./logs:/logs"],
 "command": "tensorboard --logdir=/logs --host=0.0.0.0"
 }
 }
}

out = pathlib.Path('docker-compose.yml')
out.write_text('version: "3.9"\nservices:\n')
for name, cfg in spec["services"].items():
 out.write_text(out.read_text() + f' {name}:\n')
 for k, v in cfg.items():
 if isinstance(v, list):
 out.write_text(out.read_text() + f' {k}:\n')
 for item in v:
 out.write_text(out.read_text() + f' - {item}\n')
 else:
 out.write_text(out.read_text() + f' {k}: {v}\n')
print('docker-compose.yml generated')

Run it with python gen_compose.py. The script can be extended to accept command‑line arguments for GPU count, Python version, or additional services. Takeaway: A few lines of Python replace repetitive YAML edits and keep the spec source‑controlled. ## 3. And deploy on three free‑tier providers without modification

GCP Free Tier (e2‑micro + Cloud Shell)

# In Cloud Shell
git clone https://github.com/yourname/ml‑experiment.git && cd ml‑experiment
python gen_compose.py
docker compose up -d

The e2‑micro instance has only 0.6 vCPU, but the notebook runs fine for data‑preprocessing and small‑scale training. ### Azure for Students (B1S VM)

az vm create --resource-group ml-demo --name ml‑vm \
 --image UbuntuLTS --size B1s --generate-ssh-keys
az vm open-port --resource-group ml-demo --name ml‑vm --port 8888
ssh azureuser@<public-ip>
# inside VM
git clone https://github.com/yourname/ml‑experiment.git && cd ml‑experiment
python3 -m venv .venv && source .venv/bin/activate
pip install docker-compose
python gen_compose.py
docker compose up -d

The B1S VM offers 1 vCPU and 1 GiB RAM, enough for a notebook that uses CPU‑only libraries. ### AWS Free Tier (t2.micro EC2)

aws ec2 run-instances --image-id ami-0c02fb55956c7d316 \
 --instance-type t2.micro --key-name mykey --security-group-ids sg-12345 \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ml‑vm}]'
# Add inbound rules for 8888 and 6006 in the console
ssh -i mykey.pem ec2-user@<public-ip>
# inside EC2
sudo yum install -y docker
sudo systemctl start docker
sudo usermod -aG docker ec2-user
newgrp docker
git clone https://github.com/yourname/ml‑experiment.git && cd ml‑experiment
python3 -m venv .venv && source .venv/bin/activate
pip install docker-compose
python gen_compose.py
docker compose up -d

All three clouds start the same containers, mount the same host directories, and expose identical ports. No provider‑specific Dockerfiles are needed. Takeaway: A single compose file and generator script let you move between free‑tier VMs with copy‑paste commands. ## 4. Keep data cheap and persistent across runs
Free tiers recycle the root volume after a stop/start cycle. Store experiment artefacts in a mounted bucket instead of the VM filesystem. * GCP: gcsfuse my-bucket ./work

• Azure: blobfuse ./work --container-name=mycontainer --tmp-path=/mnt/blobfusetmp
• AWS: s3fs my-bucket ./work -o iam_role=auto

Mount the bucket before docker compose up. The containers see the same path (./work) whether they run locally or on a cloud VM, and the data never counts against the VM quota. Takeaway: Bind a cloud storage fuse mount to the compose volume; you stay within free compute credits while keeping unlimited data. ## 5. Validate the environment automatically
A quick health check prevents wasted credits. Add a one‑off service that runs pip list inside the notebook container and exits with status 0 only if the expected packages are present.

 validator:
 image: jupyter/datascience-notebook:latest
 command: >
 bash -c "pip list | grep -E 'numpy|pandas|scikit-learn' && exit 0 || exit 1"
 depends_on:
 - notebook

Run docker compose up validator. If the command returns non‑zero, the CI pipeline (or a simple shell script) can abort the experiment before any GPU minutes are burned. Takeaway: A minimal validator service catches missing dependencies early, protecting scarce free credits. Act today: Clone the repo, run python gen_compose.py, then launch docker compose up -d on your local machine. The same commands will work unchanged on any free‑tier VM, giving you an instant, reproducible ML sandbox without spending a single credit.

---

If this was useful: I packaged a deeper version as LinkedIn Outreach Sequence Prompt Pack.

This article was drafted with AI assistance.