DEV Community

Rafa Jimรฉnez
Rafa Jimรฉnez

Posted on

How I built a Fully Automated 2-Data Center HA Lab with PostgreSQL, HAProxy, and Bind9 (using Vagrant)

#vagrant #postgres #bind9 #dns

๐Ÿ“– Introduction

When I started my journey into Site Reliability Engineering (SRE), I quickly realized that understanding how systems behave in distributed environments is one of the most valuable skills you can have. But there's a catch: setting up a realistic multi-server environment to practice high availability (HA) and disaster recovery (DR) is expensive, complex, and time-consuming.

So I decided to build my own.

This article is about how I created a fully automated 2-data center lab that runs entirely on my laptop using Vagrant and VirtualBox. It simulates a production-like architecture with PostgreSQL replication, a load balancer, and dynamic DNSโ€”all with a single command: vagrant up.

๐ŸŽฏ What Problem Does This Solve?

In the real world, infrastructure engineers, SREs, and DevOps practitioners need to understand how systems behave when things go wrong. But:

  • Cloud environments cost money when you leave them running.
  • Physical hardware is expensive and not portable.
  • Manual setups are error-prone and hard to reproduce.

This lab solves all of those problems. It's:

  • Free (open source).
  • Portable (runs on your laptop).
  • Reproducible (everything is code).
  • Safe (you can break it and rebuild it in seconds).

๐Ÿ—๏ธ The Architecture

The lab consists of 6 virtual machines distributed across two simulated data centers:

  • DC1: db1 (PostgreSQL Primary) + app1 (Flask App)
  • DC2: db2 (PostgreSQL Replica) + app2 (Flask App)
  • Common Infrastructure: lb (HAProxy Load Balancer) + dns (Bind9 DNS Server)

Network:

  • DC1: 10.10.1.0/24
  • DC2: 10.10.2.0/24

This separation simulates two physically distinct locations, allowing me to test real-world scenarios like network partitions and data center failures.

๐Ÿ› ๏ธ Technologies Used

  • Vagrant โ€“ Infrastructure orchestration.
  • VirtualBox โ€“ Hypervisor.
  • PostgreSQL 16 โ€“ Streaming replication (Active/Passive).
  • Flask โ€“ Python API application.
  • HAProxy โ€“ Load balancing (round-robin).
  • Bind9 โ€“ Dynamic DNS resolution.

๐Ÿงช What Can You Test?

This lab is designed to be a safe playground for real-world scenarios:

1. Controlled Switchover

Promote the replica to primary with zero downtime.

vagrant ssh db1
sudo -u postgres psql -c "SELECT pg_switch_wal();"
sudo systemctl stop postgresql
exit

vagrant ssh db2
sudo pg_ctl promote -D /var/lib/postgresql/16/main
Enter fullscreen mode Exit fullscreen mode

2. Disaster Recovery (DR)

Simulate the complete loss of DC1 and verify that the system continues to work using DC2 alone.

vagrant halt db1 app1
vagrant ssh db2
sudo pg_ctl promote -D /var/lib/postgresql/16/main
curl http://10.10.2.10:5000/items
Enter fullscreen mode Exit fullscreen mode

3. Dynamic DNS Failover

Update Bind9 records to point to the new primary database:

nsupdate -k /etc/bind/keys.conf << EOF
update delete db.lab.local A
update add db.lab.local 5 A 10.10.2.20
send
EOF
Enter fullscreen mode Exit fullscreen mode

4. Node Recovery

Rejoin a failed primary node back into the cluster as a replica after recovery.

๐Ÿ“‚ The Code

Everything is open source and available on GitHub:

๐Ÿ”— https://github.com/rafajimenezdev/2dc-ha-lab

Repository structure:

2dc-ha-lab/
โ”œโ”€โ”€ README.md                 # Full documentation
โ”œโ”€โ”€ LICENSE                   # MIT License
โ”œโ”€โ”€ Vagrantfile               # VM definitions
โ”œโ”€โ”€ images/
โ”‚   โ””โ”€โ”€ arch_whitebg.png      # Architecture diagram
โ”œโ”€โ”€ provision/
โ”‚   โ”œโ”€โ”€ db1.sh                # Primary DB setup
โ”‚   โ”œโ”€โ”€ db2.sh                # Replica DB setup
โ”‚   โ”œโ”€โ”€ app.sh                # Flask app setup
โ”‚   โ”œโ”€โ”€ lb.sh                 # HAProxy setup
โ”‚   โ””โ”€โ”€ dns.sh                # Bind9 DNS setup
โ””โ”€โ”€ app/
    โ”œโ”€โ”€ app.py                # Flask application
    โ””โ”€โ”€ requirements.txt      # Python dependencies
Enter fullscreen mode Exit fullscreen mode

๐Ÿš€ How to Get Started

1. Clone the repository 

git clone https://github.com/rafajimenezdev/2dc-ha-lab.git
cd 2dc-ha-lab
Enter fullscreen mode Exit fullscreen mode

2. Start the environment 

vagrant up
Enter fullscreen mode Exit fullscreen mode

3. Test the application 

curl http://10.10.1.100/items
Enter fullscreen mode Exit fullscreen mode

4. Insert data 

curl -X POST http://10.10.1.100/items -H "Content-Type: application/json" -d '{"id": 42}'
Enter fullscreen mode Exit fullscreen mode

๐Ÿ“Š Validation Tests

After vagrant up, run these tests to ensure everything is working:

# Check VM status
vagrant status

# Test connectivity
ping 10.10.1.10
ping 10.10.2.10

# Test load balancer
for i in {1..5}; do curl http://10.10.1.100/; done

# Test API
curl http://10.10.1.100/items
curl -X POST http://10.10.1.100/items -H "Content-Type: application/json" -d '{"id": 42}'
curl http://10.10.1.100/items

# Verify replication
curl http://10.10.2.10:5000/items
Enter fullscreen mode Exit fullscreen mode

๐Ÿ› Common Issues & Fixes

SSH Timeouts During Boot

Problem: Vagrant gets stuck at SSH auth method: private key.

Solution: Add this to your Vagrantfile:

vb.customize ["modifyvm", :id, "--cableconnected1", "on"]
Enter fullscreen mode Exit fullscreen mode

Workaround: If the VM gets stuck, open the VirtualBox GUI and bring the VM window to the foreground.

VirtualBox Host-Only Adapter Conflicts

Problem: Error VERR_INTNET_FLT_IF_NOT_FOUND.

Solution:

  1. Open VirtualBox โ†’ File โ†’ Tools โ†’ Network Manager.
  2. Go to Host-only Networks.
  3. Ensure you have two adapters:
    • 10.10.1.1/24 (DC1)
    • 10.10.2.1/24 (DC2)
  4. Disable DHCP on both.

Hyper-V Conflict

Problem: VirtualBox fails to start VMs.

Solution: Disable Hyper-V:

bcdedit /set hypervisorlaunchtype off
Enter fullscreen mode Exit fullscreen mode

๐Ÿ’ก Lessons Learned

  1. Infrastructure as Code is powerful. Being able to destroy and recreate the entire environment with a single command (vagrant destroy -f && vagrant up) changes the way you think about testing and experimentation.

  2. DNS is critical in failover scenarios. Hardcoding IPs is a trap. Using dynamic DNS (nsupdate) makes the system much more resilient and realistic.

  3. Simplicity wins. While it would be easy to add more complexity (e.g., Kubernetes, service meshes), starting with a simple, solid foundation helps you understand the fundamentals.

  4. The community is the best resource. Sharing this project on Reddit and other platforms has given me invaluable feedback and new ideas.

๐Ÿค What's Next?

I'm planning to:

  • Add tc (traffic control) to simulate network latency and packet loss between DCs.
  • Write a follow-up article on "Chaos Engineering in a Local Lab."
  • Create a video demo showing the failover and DR scenarios in action.

๐Ÿ“ข Let's Connect!

If you enjoyed this article or have questions about the lab, feel free to reach out:

Built with โค๏ธ for the infrastructure community.

Top comments (0)