In this project, I built and tore down a small, distributed S3-compatible object storage cluster on Microsoft Azure using Terraform and Silo (a MinIO fork).
Before going any further: this wasn't a massive, fault-tolerant, enterprise production cluster. It was a deliberately small, private 2-node deployment built under a strict 6-vCPU subscription cap and a tight budget. The goal was straightforward—provision the infrastructure with Infrastructure as Code (IaC), keep the storage nodes completely off the public internet, stripe data across four managed disks using erasure coding, and access the admin console securely through a Bastion host.
Getting it working taught me way more from the things that broke than the things that worked on the first try. Here is how the setup works, what went wrong along the way, and what I would do differently next time.
The Setup (Keep It Simple)
Because I only had 6 vCPUs to work with in the spaincentral region, every VM had to earn its place:
-
1 Bastion Host (
vm-acumen-bastion): The only machine with a public IP, locked down with a Network Security Group (NSG) to accept SSH only. -
2 Storage Nodes (
vm-acumen-node-1&vm-acumen-node-2): Running Ubuntu 24.04 LTS onStandard_E2ds_v4instances inside a private subnet with zero public internet exposure. -
4 Managed Disks: Two 10GB disks attached to each node (LUN 10 and 11), formatted with XFS and mounted persistently via UUID in
/etc/fstabto/data/disk1and/data/disk2. -
Silo in Docker: Running with
--network hostacross both nodes so they could talk to each other over port9000and serve the web console on port9001.
The Difficulties (What Actually Went Wrong)
1. Hitting the 6-vCPU Wall and "Ghost" Disks
Earlier iterations of this build started with more nodes, which immediately ran into Azure's 6-vCPU quota limit. Scaling back down to 2 storage nodes + 1 Bastion solved the CPU quota, but left behind a hidden FinOps trap.
When I scaled down, Azure didn't automatically delete the unattached OS disk from the old node-4. It just sat there in the resource group quietly draining credits in the background until I ran an Azure CLI check for unattached disks (managedBy==null) and wiped it manually.
- Lesson: Scaling down compute doesn't always clean up storage. Always audit for orphaned disks after changing node counts.
2. Dynamic IPs Breaking Storage Quorum (and a Terraform NicInUse Error)
Originally, I left the storage nodes' private IPs set to Dynamic in Terraform. The problem with distributed storage is that nodes rely on predictable peer addresses to maintain quorum. If Azure reboots or deallocates a VM and hands it a new internal IP, the cluster breaks.
I updated cluster.tf to enforce static private IPs (10.0.2.5 and 10.0.2.4), but accidentally changed the NIC's name attribute in the same edit (nic-cluster-node-1 → nic-acumen-node-1). Terraform tried to destroy the existing network cards to recreate them with the new name, and Azure immediately threw a 400 Bad Request: NicInUse error because those NICs were still attached to running VMs.
-
Lesson: Renaming a resource in Terraform forces a replacement. Reverting the name string let Terraform update the IP allocation in-place (
0 to add, 2 to change, 0 to destroy) in 15 seconds without touching the VMs.
3. Disk Formatting and Quorum Collisions
Terraform provisioned the blank 10GB managed disks cleanly, but formatting them with XFS, grabbing their UUIDs, writing them to /etc/fstab, and getting Silo's startup script (start-cluster.sh) aligned across both nodes took a few rough tries. Early mismatches in disk paths and metadata caused Silo to refuse startup until the drives were wiped cleanly and brought up together—finally reporting a healthy state (healed: 4, skipped: 0).
4. Reaching a Private Dashboard Without Exposing Ports
Since node-1 and node-2 had no public IPs and their NSGs blocked external access to ports 9000 and 9001, opening the Silo web dashboard in my browser required chaining an SSH tunnel through the Bastion host:
ssh -o ServerAliveInterval=60 -J acumenadmin@<BASTION_IP> -L 8081:127.0.0.1:9001 acumenadmin@10.0.2.5
Typing that out every time was so tiring, so I learnt a trick which was to move it into my local ~/.ssh/config using ProxyJump and LocalForward. After that, running ssh silo-dashboard opened the tunnel cleanly and let me manage the cluster at http://127.0.0.1:8081.
5. Real-World Downtime and Ephemeral Cloud Shell
To save credits while not actively testing, my plan was to deallocate the VMs using az vm deallocate. Then real life happened—my area got hit with a week of power outages, and I couldn't get back online to shut the VMs down. Three running VMs plus four managed disks ticking away 24/7 ate about $8 of my credits before I could get back to a terminal.
When I finally logged into Azure Cloud Shell to tear everything down, I got one final reminder: Cloud Shell sessions are ephemeral. My local ~/acumen-azure-terraform folder had reset, so running terraform destroy directly wasn't an option without re-cloning and re-initializing the backend. Since every resource lived inside dedicated resource groups (rg-acumen-infra and rg-acumen-tfstate), I used the Azure CLI to nuke the resource groups directly and stop billing immediately:
az group delete --name rg-acumen-infra --yes --no-wait
Finally, what I'd Improve Next Time
If I spin this architecture back up, there are three clear gaps I want to close:
-
Automate disk formatting: Right now, Terraform creates the disks, but XFS formatting and
/etc/fstabmounting are manual post-deploy steps. That belongs inbootstrap.sh(custom_data) or an Ansible playbook. -
Stop passing credentials in scripts: The Silo root password was passed via environment variables in
start-cluster.sh. Moving that to a locked-down.envfile (chmod 600) or Azure Key Vault is a must. -
Actually test node failure: Seeing
healed: 4, skipped: 0confirmed the drives synced properly, but I didn't get to run a destructive failover test (killingnode-2mid-upload) before tearing the lab down to save credits.
Overall, building inside a strict 6-vCPU limit forced me to be deliberate about every resource I spun up—and tearing it all down cleanly was just as important as getting the quorum healthy. If you want to inspect the Terraform files or try deploying it yourself, the full code and documentation are up on GitHub: wandile-lab/azure-silo-cluster.
Top comments (0)