DEV Community: Sunbird Analytics

How to Securely Migrate PostgreSQL to AWS RDS with Zero Downtime (Docker & Terraform)

Sunbird Analytics — Sun, 12 Apr 2026 14:30:16 +0000

How to Migrate PostgreSQL to AWS RDS with Zero Downtime (Docker & Terraform)

1. Introduction & Source Code

This article demonstrates a professional, zero-downtime homogeneous database migration from a local "on-premise" PostgreSQL database to AWS RDS.

It demonstrates the process that AWS DMS runs "under the hood" when migrating a homogeneous database. Instead of relying on a managed service, this architecture manually implements PostgreSQL Native Logical Replication (Publication/Subscription) operating securely over a Site-to-Site IPsec VPN tunnel. It includes a Python script simulating live order traffic to prove that data continuously syncs during the migration process.

You can download the full source code for this project on GitHub:
Download the Source Code

2. Architecture Overview

1. The On-Premise Environment

Simulated using Docker, this environment contains the source PostgreSQL database, a pgAdmin dashboard, and a live Python application generating mock e-commerce transactions.

Fig 1: Simulated on-premise environment using interconnected Docker containers.

2. The Cloud Environment

Provisioned via Terraform, this environment features a secure VPC, private subnets, an RDS PostgreSQL instance, and a Virtual Private Gateway (VGW) ready to receive the VPN connection.

Fig 2: Cloud infrastructure layout provisioned dynamically via Terraform.

3. The Combined Migration Architecture

The strongSwan container establishes an IPsec tunnel to the AWS VGW, allowing the AWS RDS instance to securely reach back into the local Docker network and subscribe to the live transaction feed.

Fig 3: End-to-end logical replication architecture connected over a secure Site-to-Site VPN tunnel.

3. Prerequisites

Docker & Docker Compose: To run the local environment.
Terraform: To provision the AWS infrastructure.
AWS CLI: Configured with credentials that have permission to create VPCs, RDS instances, and VPNs.
psql: Installed locally (or accessible via your pgAdmin container) to execute the schema dump.

4. Step-by-Step Execution Guide

Phase 1: Start the On-Premise Infrastructure

Navigate to the on-prem directory: cd on-prem
Start the Docker containers: docker-compose up -d --build
Verify the live traffic generator is working by checking the logs: docker logs -f fake_live_traffic
Open your browser and navigate to http://localhost:5050 to access pgAdmin.
Log in using admin@sunbirdanalytics.com and password sunbird_analytics.
Add the local server using host source_db, username sunbird_analytics, and password sunbird_analytics. You will see the live_orders table actively populating.

Phase 2: Gather Required IP Addresses

Before provisioning the cloud, you need two critical IP addresses.

Find your Public IP: Run curl ifconfig.me in your terminal. Save this for Terraform.
Find your Source DB Docker IP: Run docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' source_db.
Open scripts/main.md and replace the IP address in the CREATE SUBSCRIPTION connection string with this Docker IP.

Phase 3: Provision the AWS Infrastructure

Navigate to the cloud setup directory: cd ../cloud-setup
Initialize Terraform: terraform init
Deploy the infrastructure (replace the placeholder with your actual public IP from Phase 2): terraform apply -var="my_ip=YOUR_PUBLIC_IP_HERE"
Type yes to confirm. This will take roughly 5-10 minutes to provision the RDS instance and VPN.
Once complete, retrieve your new RDS endpoint via the AWS Console or by running:

aws rds describe-db-instances \
 --region us-west-1 \
--db-instance-identifier target-cloud-db \
--query "DBInstances[0].Endpoint.Address" \
--output text

Phase 4: Map Your Infrastructure with Sunbird Insyte

Once terraform apply finishes and everything is successfully provisioned on AWS, you can visualize your new cloud environment.

Use Sunbird Insyte to automatically map all of your newly provisioned infrastructure. It will generate a clear overview of all your provisioned resources, giving you total visibility into your active cloud setup.

Fig 4: Verifying the deployed AWS VPC and resources automatically with Sunbird Insyte.

Phase 5: Configure the Site-to-Site VPN Tunnel

AWS has generated the keys for your VPN tunnel, but your local Docker container needs them to establish the connection.

Log into the AWS Management Console and navigate to the VPC Dashboard -> Site-to-Site VPN Connections.
Select your newly created VPN and click Download configuration (Vendor: Generic).
Open the downloaded file and find the IPSec Tunnel #1 section to locate your Pre-Shared Key (PSK) and the Virtual Private Gateway IP (<AWS_TUNNEL_IP>).
Update vpn_config/ipsec.secrets in your project folder:

   <YOUR_PUBLIC_IP> <AWS_TUNNEL_IP> : PSK "YOUR_PRE_SHARED_KEY_HERE"

5.Update vpn_config/ipsec.conf in your project folder:

   leftid=<YOUR_PUBLIC_IP>
   ...
   right=<AWS_TUNNEL_IP>

6.Start Tunnel: docker restart aws_ipsec_vpn then docker exec aws_ipsec_vpn ipsec up aws-tunnel-1.

Phase 6: Inject Local Network Routes

Even with the tunnel "Up," the pgAdmin and Postgres containers need a map to find AWS.

Find VPN Container IP: docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' aws_ipsec_vpn
Add Route to pgadmin and local db containers so they know to send traffic destined for AWS through the VPN tunnel:

   docker exec -u root pgadmin_dashboard ip route add 10.0.0.0/16 via <VPN_IP>

   docker exec -u root source_db apt-get update && docker exec -u root source_db apt-get install -y iproute2
   docker exec -u root source_db ip route add 10.0.0.0/16 via <VPN_IP>

3.Test Connectivity: From the pgAdmin container, ping the AWS RDS endpoint to confirm connectivity:

   docker exec -it pgadmin_dashboard ping <YOUR_RDS_ENDPOINT>

4.If the ping is successful, you have established a secure connection from your local Docker environment to AWS! You can now also connect to the AWS RDS instance from pgAdmin using the <YOUR_RDS_ENDPOINT>, username SunbirdAdmin, and the password sunbird_analytics you set in Terraform.

Fig 5: Connecting directly to the AWS cloud server via the local pgAdmin dashboard.

Phase 7: Execute the Migration Using PostgreSQL Publication and Subscription

1. Schema Migration

Export the schema from your Docker Postgres:

docker exec source_db pg_dump -s -U sunbird_analytics -d sunbird_analytics > schema.sql

Import the schema into AWS RDS. Open the schema.sql script and Copy all the text inside that file. Then, in pgAdmin open a query window connected from the cloud server to the postgres database on AWS RDS. Paste the entire schema.sql content and execute it. This will create all the tables and structures needed on AWS.

2. On the Source DB (Local Docker)

Create the "Publisher." This tells your local DB to start broadcasting every change to the transaction logs.

-- Run this in the 'sunbird_analytics' database on Localhost
CREATE PUBLICATION sunbird_migration_pub FOR ALL TABLES;

3. On the Target DB (AWS RDS)

Get the IP address of your local Docker Postgres container. This is the "host" that AWS will connect to.

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' source_db

Create the "Subscriber." This tells AWS to reach through the VPN and start sucking in data.

-- Run this in the 'sunbird_analytics' database on AWS RDS
CREATE SUBSCRIPTION sunbird_migration_pub
CONNECTION 'host=172.20.0.2 port=5432 user=sunbird_analytics password=sunbird_analytics dbname=sunbird_analytics'
PUBLICATION sunbird_migration_pub;

4. Reset Sequences (CRITICAL)

Postgres sequences (for ID columns) don't stay in sync during replication. You must manually "bump" them on AWS so the next ID used is correct.

Run this on AWS RDS:

-- This script resets all sequences to the current max ID + 1
SELECT setval(pg_get_serial_sequence('live_orders', 'id'), coalesce(max(id), 1)) FROM live_orders;

Need Help With Your Infrastructure?

Navigating complex database migrations and modernizing your cloud infrastructure doesn't have to be a solo mission.

👉 Explore Our Cloud Solutions at Sunbird Analytics

The Free AWS Risk, Compliance, FinOpS & Auditing Platform

Sunbird Analytics — Fri, 10 Apr 2026 23:38:54 +0000

1. The Need for Automated Cloud Governance
Maintaining a secure and cost-effective AWS environment manually is virtually impossible as your infrastructure scales. By leveraging Sunbird Insyte, teams can instantly identify security vulnerabilities and FinOps optimisation targets through a unified dashboard. Let's break down how simple it is to run a comprehensive scan of your cloud infrastructure.

2. Running Your First Infrastructure Audit
To begin, navigate to the Infrastructure Audits section in the Sunbird Insyte console. From there, click the Run Audit Scan button. A configuration modal will prompt you to select your target AWS region. In our example, we select us-east-1 and click Confirm & Run.

The system will queue your scan for processing. After a few moments, clicking Refresh Scan Status will update the status indicator to a green "Succeeded", confirming that your environment data has been successfully ingested and analysed.

3. Reviewing Security Vulnerabilities
With the scan complete, head over to the Security tab located under the Governance menu. The Security Posture dashboard immediately highlights the risk overview.

For example, a typical audit might reveal a total of 135 findings, with a specific focus on 5 Critical Risks. The dashboard clearly flags severe issues, such as:

AWS LAMBDA: Potential secrets found in Lambda function source code (e.g., hardcoded Secret Keywords).
ECS: Potential secrets detected within ECS task definition environment variables.
IAM: Highly permissive policies attached directly to users, or overly broad administrative access.
S3: Buckets with public-read access policies exposing data to the public internet.

4. Uncovering FinOps Opportunities
Cloud optimisation isn't just about security; it's also about managing costs. Switching to the FinOps tab provides deep visibility into your cloud spend. The dashboard displays a straightforward Cost Insights panel showcasing the current 30-day cost—such as $29.00 in our demo environment—alongside the forecasted spend.

Below the overview, Sunbird Insyte breaks down your cost distribution across different AWS services and maps them directly to actionable Strategy Overviews. Some highly effective recommendations generated by the platform include:

Right-sizing task allocations and utilizing Fargate Spot for ECS workloads.
Monitoring utilisation and right-sizing based on precise metrics for EC2.
Implementing S3 Lifecycle policies and storage tiering for long-term storage.

5. Exporting Actionable PDFs
The real value of an audit is in how easily the data can be shared with stakeholders and engineers. On both the Security and FinOps pages, Sunbird Insyte provides an Export PDF Report button. Clicking this generates a beautifully formatted, comprehensive PDF document outlining every finding, risk analysis, and FinOps recommendation ready to be saved and distributed.

Stop guessing about your cloud security posture and monthly bill. Try running your first automated audit today with Sunbird Insyte.

Are You Wasting 70% of Your AWS Budget on Non-Prod Instances?

Sunbird Analytics — Fri, 10 Apr 2026 13:08:51 +0000

1. The Voluntary Cloud Tax
Watching an AWS bill speedrun your IT budget is pure panic. Stop paying the voluntary cloud tax for dev servers that nobody turned off on Friday.

2. The Idle Math
Let's look at the math: there are 168 hours in a week, and your developers are allegedly working 40 of them. That means for almost 130 hours a week, your non-production servers are just sitting there. They are active for maybe a quarter of the time.

The other 76%, they're completely idle, doing absolutely nothing, just quietly burning company cash in a data center somewhere.

3. The Baseline Architecture
Here is a baseline setup: we've got a standard Application Load Balancer sitting in front of an Auto Scaling Group with three T3 mediums. If we run a cost breakdown right now, it wants to charge us $108 a month. For a development server, that's a bit too much.

Here is the complete Terraform configuration used to provision the baseline architecture, including the Launch Template, Auto Scaling Group, and networking components.

provider "aws" {
  region = "us-east-1"
}

# 1. Fetch the latest Amazon Linux OS
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }
}

# 2. Minimal Networking for the Load Balancer
resource "aws_vpc" "dev_vpc" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "dev_subnet_a" {
  vpc_id                  = aws_vpc.dev_vpc.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
}

resource "aws_subnet" "dev_subnet_b" {
  vpc_id                  = aws_vpc.dev_vpc.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = "us-east-1b"
  map_public_ip_on_launch = true
}

resource "aws_security_group" "dev_sg" {
  name   = "dev-sg"
  vpc_id = aws_vpc.dev_vpc.id

  # Allow inbound HTTP traffic
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Allow all outbound traffic (so the servers can download updates)
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_internet_gateway" "dev_igw" {
  vpc_id = aws_vpc.dev_vpc.id
}

resource "aws_route_table" "dev_public_rt" {
  vpc_id = aws_vpc.dev_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.dev_igw.id
  }
}

resource "aws_route_table_association" "a" {
  subnet_id      = aws_subnet.dev_subnet_a.id
  route_table_id = aws_route_table.dev_public_rt.id
}

resource "aws_route_table_association" "b" {
  subnet_id      = aws_subnet.dev_subnet_b.id
  route_table_id = aws_route_table.dev_public_rt.id
}

# 3. The Launch Template
resource "aws_launch_template" "dev_web" {
  name_prefix            = "dev-web-template"
  image_id               = data.aws_ami.amazon_linux.id
  instance_type          = "t3.medium"
  vpc_security_group_ids = [aws_security_group.dev_sg.id]

  # This script automatically installs a web server when the instance boots!
  user_data = base64encode(<<-EOF
              #!/bin/bash
              yum update -y
              yum install -y httpd
              systemctl start httpd
              systemctl enable httpd

              # Grab the private IP of the specific EC2 instance
              INSTANCE_IP=$(hostname -I | awk '{print $1}')

              # Build the webpage
              echo "<html><body>" > /var/www/html/index.html
              echo "<h1>Hello from the Dev Environment!</h1>" >> /var/www/html/index.html
              echo "<p>Traffic is currently being handled by Server IP: $INSTANCE_IP</p>" >> /var/www/html/index.html
              echo "</body></html>" >> /var/www/html/index.html
              EOF
  )
}

# 4. The Auto Scaling Group (Deploys 3 Instances)
resource "aws_autoscaling_group" "dev_web_asg" {
  name                = "dev-web-asg"
  vpc_zone_identifier = [aws_subnet.dev_subnet_a.id, aws_subnet.dev_subnet_b.id]
  desired_capacity    = 3
  min_size            = 3
  max_size            = 3

  launch_template {
    id      = aws_launch_template.dev_web.id
    version = "$Latest"
  }
  target_group_arns = [aws_lb_target_group.dev_web_tg.arn]
}

# 5. The Load Balancer
resource "aws_lb" "dev_alb" {
  name               = "dev-web-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.dev_sg.id]
  subnets            = [aws_subnet.dev_subnet_a.id, aws_subnet.dev_subnet_b.id]
}

resource "aws_lb_target_group" "dev_web_tg" {
  name     = "dev-web-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.dev_vpc.id
}

# The Listener (Tells the ALB where to send traffic)
resource "aws_lb_listener" "dev_listener" {
  load_balancer_arn = aws_lb.dev_alb.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.dev_web_tg.arn
  }
}

Let's run terraform apply and deploy our setup. Terraform does its thing, we check the AWS console, and the instances are running. If we hit the load balancer URL, we get our test page. Refresh it a couple of times, the IPs change, so traffic is routing perfectly. The architecture is solid, but the billing department is still crying.

4. Implementing Scheduled Scaling

Now, let's cut the AWS bill by treating the servers like office lights, automatically turning them off when people go home using a couple of lines of Terraform.

Here's the magic trick: scheduled scaling. We drop in two aws_autoscaling_schedule resources into our configuration.

# --- SCHEDULED SCALING (The Cost Saver) ---
resource "aws_autoscaling_schedule" "scale_down" {
  scheduled_action_name  = "scale-down-evening"
  min_size               = 0
  max_size               = 0
  desired_capacity       = 0
  recurrence             = "0 18 * * 1-5" # 6 PM Mon-Fri
  autoscaling_group_name = aws_autoscaling_group.dev_web_asg.name
}

resource "aws_autoscaling_schedule" "scale_up" {
  scheduled_action_name  = "scale-up-morning"
  min_size               = 3
  max_size               = 3
  desired_capacity       = 3
  recurrence             = "0 8 * * 1-5" # 8 AM Mon-Fri
  autoscaling_group_name = aws_autoscaling_group.dev_web_asg.name
}

The first block tells the group to scale down to zero instances at 6 PM every weekday. The second block tells it to wake back up and spin up three instances at 8 AM every weekday.

We run a quick terraform apply to push the update. Check the AWS console again, and look under the autoscaling group, and our instances officially have a bedtime.

5. The Cost Savings

Let's actually look at the math to see if it was worth it. To calculate our baseline cost without the schedule, we run a standard Infracost breakdown:

infracost breakdown --path .

This command returns our $108/month estimate based on instances running 24/7. But what happens when we apply our bedtime? We can feed Infracost a custom usage file (infracost-usage.yml) that simulates our 40-hour work week instead of the default 730-hour month.

# infracost-usage.yml
version: 0.1
resource_usage:
  aws_autoscaling_group.dev_web_asg:
    instances: 1 # This simulates 40 hours/week out of the standard 730-hour month

We run the breakdown again, this time passing in the custom usage file:

infracost breakdown --path . --usage-file infracost-usage.yml

Our estimated bill drops from $108 to $47. It took 5 minutes of code to cut the bill in half.

Amazon S3 Files: The End of Data Silos (And the Security Risks to Watch)

Sunbird Analytics — Thu, 09 Apr 2026 22:19:30 +0000

1. What is Amazon S3 Files?
For years, a fundamental division has existed in cloud storage: you either used object storage (like Amazon S3) for massive scale and low costs, or block/file storage (like EFS or EBS) for applications that require a standard file system. Syncing data between the two was a tedious process requiring custom pipelines.

AWS has erased that boundary with the introduction of Amazon S3 Files. S3 Files is a shared file system feature that connects any AWS compute resource directly with your data in Amazon S3. It provides fast, direct access to your S3 buckets as files with complete NFS file system semantics, bringing the simplicity of a file system to the limitless scale of S3.

2. Game-Changing Benefits for Cloud Workloads
By effectively turning your S3 bucket into a traditional file system, you instantly eliminate duplicate storage. Data engineers, ML models, and containerized applications can read and write to the same central S3 bucket in real time.

No Code Changes Required: Standard Python libraries, shell scripts, and native ML frameworks can interact with S3 directly, oblivious to the fact that it's object storage on the backend.

Massive Scalability: S3 Files supports up to 25,000 compute resources (EC2, EKS, ECS, Lambda, Fargate) accessing the same dataset simultaneously.

Cost Optimization: By avoiding data replication between object stores and file systems, AWS claims S3 Files can deliver up to 90% lower costs. Intelligent caching ensures only active working sets are loaded onto high-performance layers.

Uncovering the New Security Risks While the operational advantages are massive, S3 Files introduces complex new attack vectors. Exposing object storage through a network file system interface means your network security boundary is now inextricably tied to your data security boundary.

Over-Permissive Network Access: S3 Files requires mount targets inside your VPC. If your security groups are misconfigured, unauthorized resources—or bad actors who have breached an EC2 instance—could silently mount your S3 bucket and access or exfiltrate your entire dataset using standard OS commands.

Identity and Access Management (IAM) Gaps: Organizations typically lock down S3 using bucket policies. However, S3 Files uses a combination of IAM policies, file system policies, and Access Points. Misaligning these layers can result in "shadow access" where users bypass intended bucket restrictions via the file system mount.

Ransomware Threats: Because S3 Files allows standard file writes and overwrites, a compromised container with write access to an S3 file system could potentially encrypt or delete files at a massive scale, behaving exactly like a traditional on-premise ransomware attack.

4. Securing S3 Files with Sunbird Insyte
Adopting Amazon S3 Files means your compliance and security audits must evolve immediately. The days of simply checking S3 Bucket Policies are over; you now have to audit VPC endpoints, NFS security groups, and S3 File system access points in tandem.

This is where Sunbird Insyte steps in. Insyte continuously monitors your AWS infrastructure and automatically flags risky configurations related to S3 Files. Our platform checks for:

Exposed mount targets with overly broad security group rules.
Misconfigurations between IAM policies and file system access points.
Missing encryption settings for data in transit and at rest.
Instead of manually querying your environment to see who has mounted what, Insyte gives you a single pane of glass to verify that your new S3 file systems align with strict compliance frameworks like ISO 27001 and SOC 2.