DEV Community: Dimeji Ojewunmi

Migration & Modernization From Azure to AWS

Dimeji Ojewunmi — Wed, 21 Oct 2026 23:05:49 +0000

My Hands-On Migration of a Node.js App and PostgreSQL Database from Azure to AWS

My migration lab, from a blocked local network to PostgreSQL on RDS and a private application on EC2.

Full load: 100%. Three tables validated. Zero mismatched records.

Those were the results on my AWS DMS dashboard. The data had made it from Azure to Amazon RDS, and the row counts agreed.

There was still work hiding behind that success: sequence-backed IDs, defaults, constraints, and permissions needed attention before the target database was ready for the application.

I built this lab to practice moving a Node.js order-management app and its PostgreSQL database to AWS. I used an Azure VM as the simulated on-premises source. There were only three tables, which made it easy to compare the data while I worked through the networking, database cutover, and application move.

The database copy was one step. Getting the application to use it—and then making the application reachable through the load balancer—took more work.

I originally wanted to run this from home

My original plan was to run the source locally and connect it to AWS. I ran into connectivity blockers with my MTN and Airtel 5G router setups before I could get to the migration itself.

I still wanted to carry out the project, so I moved the source environment to an Azure VM. There, I could configure the private network and VPN connection myself.

I kept the application and PostgreSQL on a private machine, with AWS on a separate network and a VPN between them. That let me practice the migration steps I had set out to learn.

The MTN and Airtel 5G router limitations remained unresolved. Using Azure let me continue, and I ended up working through the network configuration in both clouds.

My lab setup

Source (Azure, simulating on-prem): an Ubuntu VM with a private address, a Node.js/Express order-management app on port 3000, and PostgreSQL 16 containing customers, products, and orders.

Target (AWS): a VPC (10.16.0.0/16), private Amazon RDS for PostgreSQL, AWS DMS, and a private EC2 instance running the application. An internet-facing Application Load Balancer (ALB) terminates HTTPS and forwards traffic to the private app on port 3000.

Migration connectivity: a route-based IPsec Site-to-Site VPN between Azure (172.16.0.0/16) and AWS (10.16.0.0/16). The Azure VPN connection showed Connected; on AWS, one VPN tunnel was UP and the other DOWN. That was sufficient for the lab, but not a highly available production VPN design.

                 Migration path (private traffic)
  Azure simulated on-prem                    AWS
  172.16.0.0/16                              10.16.0.0/16
  +----------------------+   IPsec VPN   +----------------------+
  | Ubuntu VM 172.16.0.4|===============>| DMS replication      |
  | Node.js :3000       |                | RDS PostgreSQL       |
  | PostgreSQL 16       |                | Private subnets      |
  +----------------------+                +----------------------+

                 Final application request path
  Browser --HTTPS :443--> Public ALB --HTTP :3000--> Private EC2
                                                   |
                                                   +--TCP :5432--> Private RDS

Security note: This diagram shows the logical data flow, not every route table, gateway, or security-group rule. The public ALB is the entry point; the EC2 application and RDS database do not need public IPs.

Phase 1 — Setting up the source in Azure

The source began with an Azure virtual network, a private workload subnet, a VPN gateway subnet, and an Ubuntu VM hosting both the application and PostgreSQL. I gave the VM no public NIC address. That decision made connectivity part of the exercise from the beginning: every connection to the source needed a deliberate path.

Figure 1: The Azure Ubuntu VM used as the simulated on-premises source. Its private IP is 172.16.0.4; the public IP shown lower on the page belongs to the NAT gateway.

Figure 2: The Azure VNet contains separate workload, VPN gateway, and Bastion subnets. The workload subnet is associated with private-SG.

The database contained customers, products, and orders. The application could read and write data and exposed a health endpoint. That gave the migration a practical acceptance test: after the move, could the application still use the database the way it did before?

Copying three tables would be easy to observe. Preserving their behavior would take more care.

Phase 2 — “Connected” was the first checkpoint

Before DMS could move a row, the two private networks needed a working conversation.

I configured compatible IPsec tunnel parameters on the Azure VPN gateway and AWS virtual private gateway. In Azure, a Local Network Gateway represented the AWS VPN endpoint and remote address space. In AWS, the private route tables needed a route back to 172.16.0.0/16 through the virtual private gateway.

Azure showed Connected. The next step was to check what the AWS side could actually tell me.

Figure 3: The Azure VPN gateway uses the VpnGw1AZ SKU and a route-based VPN. Its public IP is the Azure-side VPN endpoint.

Figure 4: The Azure Local Network Gateway represents the AWS tunnel endpoint and the remote AWS address space, 10.16.0.0/16.

Figure 5: The Azure gateway connection list shows azure-to-aws-tunnel1 in the Connected state.

Follow the connection into AWS

On AWS, I created azure-onprem-cgw to represent the Azure VPN gateway's public IP, 4.255.22.3. I attached azure-migration-vgw to Demo-VPC, then linked the gateways through the azure-to-aws-vpn Site-to-Site VPN connection.

Figure 6: The AWS customer gateway is Available and points to the Azure VPN gateway public IP. Although this page lists a BGP ASN, the VPN connection in this lab uses static routing.

Figure 7: The AWS virtual private gateway is Available and its VPC attachment state is Attached.

The AWS tunnel view gave me the other half of the connectivity check: Tunnel 1 was Up, with outside IP 3.227.45.240, matching the endpoint configured in Azure's Local Network Gateway. Tunnel 2 was Down, so this lab had only one operational tunnel.

Figure 8: AWS reports one operational tunnel and explicitly warns that both tunnels are not in use.

The tunnel status was only one piece of the path. I checked the remote network in two places: the VPN connection's static routes and the VPC subnet's route table. The VPN static route listed the Azure address space, 172.16.0.0/16. The priv-RT route table, associated with priv-subnet, directed that same address space to the virtual private gateway.

Figure 9: The Site-to-Site VPN uses static routing and lists 172.16.0.0/16 as an Available remote prefix.

Figure 10: The Azure route, 172.16.0.0/16, is Active and targets the virtual private gateway. A separate route to 10.200.0.0/24 is Blackhole; it does not establish connectivity for that prefix and is distinct from the Azure route.

Give PostgreSQL a path through the firewall

The Azure network security group allowed TCP 5432 from the AWS VPC range, 10.16.0.0/16, to the source VM, 172.16.0.4. This captured the source-side database rule used for replication.

Figure 11: The Azure inbound rule allows PostgreSQL traffic from 10.16.0.0/16 to 172.16.0.4 on TCP port 5432.

At this point, Connected had become a useful starting point for investigation. I still needed routes in both directions and firewall rules that permitted the actual traffic. Replication needed PostgreSQL on TCP 5432. The application would later need its own path on 3000—a distinction that would matter again when I reached the load balancer.

I also treated the single operational tunnel as a lab limitation. A production design should validate redundant tunnels, failover behavior, monitoring, and appropriate routing resilience.

Phase 3 — Connecting DMS to the source and target

With the private network path established, I provisioned private Amazon RDS for PostgreSQL and set up AWS Database Migration Service (DMS) endpoints for the Azure source and AWS target. The source was PostgreSQL 16 and the RDS target reported PostgreSQL 18.3, so this exercise also required attention to engine-version differences and schema compatibility.

Figure 12: The DMS source endpoint connects to migrationdb on the Azure VM using dms_user and port 5432. The screenshot shows SSL mode None.

Figure 13: The DMS target endpoint connects to migrationdb on Amazon RDS using port 5432, with SSL mode Require.

Verify connectivity from the DMS replication instance

The DMS replication instance, azure-postgres-migration, had private IP 10.16.2.71 in the AWS VPC and no public IP. I tested each database endpoint from that instance to verify the connection used by the migration service.

Figure 14: The replication instance is Available, uses private IP 10.16.2.71, and lists its VPC, subnet group, and security group.

Figure 15: The source endpoint connection test from azure-postgres-migration is Successful, providing a database connectivity check across the private migration path.

Figure 16: The target endpoint connection test from the same replication instance is Successful. These tests establish endpoint access at the time of testing; migration and data validation are checked separately.

Both endpoint tests returned Successful. That gave me evidence that the replication instance could connect to each database.

I chose full load plus change data capture (CDC): copy the existing rows, then apply changes made while the source remained in use. The initial copy would get the target populated; CDC would keep changes moving until the cutover boundary.

I organized the migration around these checkpoints:

Establish and test private connectivity and database access.
Create and test DMS source and target endpoints.
Run the pre-migration assessment and review warnings.
Run full load and CDC, monitoring task and table status.
Compare data and validate application read/write behavior.
Freeze source writes, confirm target consistency, and stop replication.
Reconfigure and move the application to AWS, then verify end-to-end access.

Phase 4 — The dashboard reached 100%

Before relying on the task results, I checked the pre-migration assessment. Its summary had a little more nuance than a simple pass or fail.

Figure 17: The assessment summary reports 26 passed checks, 2 warnings, 15 skipped checks, and 0 failures.

The assessment reported 43 checks: 26 passed, 2 warnings, 15 skipped, and 0 failed. The warnings and skipped checks still called for manual review.

Then came the encouraging result: full load reached 100%, with replication ongoing. The table statistics and direct SQL comparisons gave me two more ways to inspect what had arrived.

Figure 18: The DMS task shows “Load completed, replication ongoing”: full load has reached 100%, and CDC is still running.

Figure 19: DMS reports three completed tables and three validated tables, with zero mismatched records, validation errors, or pending records.

Figure 20: The source and RDS target each return 4 customers, 3 products, and 3 orders. The terminal also records an initial failed login followed by a successful TLS connection.

At one comparison point, both databases returned 4 customers, 3 products, and 3 orders. DMS validation subsequently reported 3 validated tables, 0 mismatched rows, and 0 validation errors.

Those checks established something valuable about the copied data. They still left an application question open: what would happen when the target needed to create the next record?

The work the row counts could not reveal

After replication, I had to repair sequence-backed IDs, defaults, and missing unique and foreign-key constraints on the target. The application user also needed the required sequence permissions.

These details shape everyday application behavior: how a new row gets its ID, which values appear automatically, and which relationships the database enforces. Existing rows can look correct while that behavior still needs work.

One of the permission changes in this lab was:

GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO appuser;

The precise schema changes depend on the source DDL and migration configuration; the command above is not a universal production permission recipe. In a real migration, I would migrate schema objects explicitly, compare source and target DDL, validate constraints and sequences, and test inserts and updates before approving cutover.

That changed how I read the green dashboard. It was evidence that the data checks had passed. I still needed to exercise the operations the application would depend on after cutover.

Phase 5 — The cutover happened in `.env`

Cutover introduced a boundary the earlier screenshots could not capture by themselves: the moment new writes belonged on AWS.

DMS had copied and validated the data, but the application still had its original database configuration. Its .env pointed to the local PostgreSQL host. Until that changed, the application would keep connecting to the source database, regardless of what the DMS dashboard showed.

After validating the data, I stopped source writes, completed the migration checks, and stopped the DMS replication task. Then came the configuration change that made the migration real for the application: I changed the PostgreSQL host in .env from the local host to the Amazon RDS endpoint.

The essential change looked like this. DB_HOST is an illustrative variable name; use the name your application's database configuration actually reads.

# Before: PostgreSQL on the application VM
DB_HOST=localhost

# After: Amazon RDS PostgreSQL
DB_HOST=database-migration.cojc20qugnja.us-east-1.rds.amazonaws.com

These are before-and-after alternatives, not two entries to keep in the same file. The rest of the connection configuration also needed to match the target: database name, port, application credentials, and SSL settings. A running Node.js process must reload its configuration—typically through a service restart—to use the changed .env value.

At this stage, the application was still on the Azure VM. Its database connection now crossed the private VPN to RDS:

Before: Azure app -> local PostgreSQL
After:  Azure app -> Site-to-Site VPN -> Amazon RDS PostgreSQL

That separated the database cutover from the later application move to EC2. I could test the target database with the application before changing the application's hosting environment as well.

I then tested target-side writes. Additional customer records were inserted on AWS after cutover, so the later target count intentionally differed from the earlier source comparison.

Figure 21: The DMS task is stopped, with full load at 100%. This records the replication state; the write freeze and application cutover checks are separate steps.

The earlier count of four customers remained a useful migration checkpoint. It was no longer the expected count forever. Once AWS accepted new writes, comparing its current data with an old source snapshot required that context.

The application now pointed to RDS. Its compute still lived in Azure; moving that to EC2 was the next step.

Phase 6 — Moving the app to private EC2

I copied the application to a private AWS EC2 instance over the established private network path, installed its Node.js dependencies, and recreated its connection configuration with the RDS endpoint selected during database cutover. The transfer excluded .env and node_modules, so dependencies and environment settings were set up on EC2 separately. The application ran as a systemd service, and its /health endpoint exercised a database connectivity query.

Figure 22: The AWS private-server instance uses private IP 10.16.2.132 in priv-subnet, within Demo-VPC, and has no public IPv4 address. This screenshot establishes its network placement, not application readiness.

The target app's connection path was:

Private EC2 app (:3000) -> RDS security group (:5432) -> PostgreSQL

I used SSL for the database connection, but the lab configuration disabled server-certificate verification (rejectUnauthorized: false). That is not suitable for production. A production migration should use RDS CA certificates and verify the server identity, with credentials in a managed secret store rather than hard-coded or committed .env files.

Phase 7 — Serving the private application over HTTPS

The final request path had several distinct hops: a browser reaching the public ALB, the ALB reaching private EC2, and the application reaching RDS.

I configured an HTTP listener on 80 to redirect to 443, an HTTPS listener on 443 using an ACM certificate, and a target group forwarding to the app on HTTP 3000. The health check used /health. The ALB was internet-facing; the application instance remained private.

User enters https://app.example.com
              |
              v
ALB listener :443 (TLS termination)
              |
              v
Target group :3000 (health check /health)
              |
              v
Private EC2 security group allows :3000 FROM ALB security group
              |
              v
Node.js service :3000 -> RDS :5432

The EC2 security group allowed TCP 3000 from the ALB security group, so the load balancer could forward requests to the application while the instance remained private.

One more detail worth getting right: a wildcard ACM certificate only matches appropriate domain names. Testing an ALB-generated hostname does not prove a custom DNS name or public DNS cutover was configured. I verified the application path through the ALB, but I am not claiming a completed custom-domain DNS cutover here.

Figure 23: The order-management application displays six customers, three products, and three orders. Its original environment label was hard-coded; this screenshot alone does not establish the serving host, database endpoint, or HTTPS/ALB path.

The shortcuts I would revisit

I got the lab working, but I would not carry every choice into a production setup. The single VPN tunnel and disabled database certificate verification are two clear examples. Before using this approach for a live workload, I would go back through these checks:

Network: VPN tunnel redundancy, both-direction routes, SG/NSG policy, DNS resolution, and private-only database access.
Database: DDL parity, indexes, constraints, sequences, roles, permissions, row counts, validation errors, application transactions, and a tested recovery path.
Cutover: approved freeze window, final CDC lag checks, explicit source-of-truth decision, rollback criteria, and stakeholder sign-off.
Application: process supervision, health checks, logs, connection pool settings, TLS validation, secrets management, and ALB target health.
Operations: monitoring, backup/restore exercises, runbooks, cost controls, and post-cutover support.

There is still more I can do with this app

For now, the application runs on EC2. I would like to build on that by containerizing it, storing the image in ECR, and trying ECS/Fargate or EKS. Terraform would also give me a way to recreate the infrastructure without repeating the console setup.

Those would be separate exercises. This lab stops at the application and database migration; secrets management, CI/CD, and better monitoring are still on my list, along with RDS high availability and the second VPN tunnel.

Looking back

I started this project wanting to connect my local environment to AWS. The MTN and Airtel 5G router blockers changed the setup, but I was able to continue with Azure as the source.

The part that stays with me is the database cutover. I could see the rows in RDS, but my app was still configured to use local PostgreSQL. Changing the host in .env was the step that made the application use the migrated database. Moving the application itself to EC2 came afterward.

These are the details I wanted to get hands-on experience with when I started. Having worked through them, I have more to check than a completed DMS task: the app's connection settings, its ability to read and write, and the path a request takes to reach it.

Thanks for reading and following along with my lab. Cheers!

Migration & Modernization

Dimeji Ojewunmi — Thu, 08 Oct 2026 01:07:28 +0000

Migrated an App and PostgreSQL Database from Azure to AWS—And the “Successful” Database Migration Still Needed Fixing

An end-to-end migration lab: private networking, Site-to-Site VPN, AWS DMS full load + CDC, database cutover, and a private EC2 application behind an HTTPS load balancer.

Migration is easy in architecture diagrams. Draw an arrow from Azure to AWS, add a cloud icon, and everyone goes home happy. Real migrations involve routes, ports, replication state, application secrets, database defaults, and at least one moment when an ALB says 504 Gateway Timeout instead of saying hello.

This was a hands-on lab, not a claim about a production enterprise cutover. I built an Azure environment to simulate an on-premises source and migrated a Node.js application and PostgreSQL database to AWS. The goal was to practice the decisions, checks, and failure modes that matter in a real migration.

What I built

Source (Azure, simulating on-prem): an Ubuntu VM with a private address, a Node.js/Express order-management app on port 3000, and PostgreSQL 16 containing customers, products, and orders.

                 Migration path (private traffic)
  Azure simulated on-prem                    AWS
  172.16.0.0/16                              10.16.0.0/16
  +----------------------+   IPsec VPN   +----------------------+
  | Ubuntu VM 172.16.0.4|===============>| DMS replication      |
  | Node.js :3000       |                | RDS PostgreSQL       |
  | PostgreSQL 16       |                | Private subnets      |
  +----------------------+                +----------------------+

                 Final application request path
  Browser --HTTPS :443--> Public ALB --HTTP :3000--> Private EC2
                                                   |
                                                   +--TCP :5432--> Private RDS

Phase 1 — Establish the source environment

I created the Azure virtual network, private workload subnet, VPN gateway subnet, and an Ubuntu VM without a public NIC address. The source VM hosted the application and PostgreSQL database. A private VM is a good simulation of an on-prem server: reaching it requires an intentional management or network path, rather than a casual SSH connection from the internet.

Figure 1: The Azure VM used as the simulated on-premises source.

Figure 2: Azure network segmentation for the workload and VPN gateway.

The source database contained three tables: customers, products, and orders. The application exposed a health endpoint and supported reading and writing data, giving me something meaningful to validate after migration.

Phase 2 — Connect Azure to AWS over IPsec

The Azure VPN gateway and AWS virtual private gateway were configured with compatible tunnel parameters. I created an Azure Local Network Gateway representing the AWS VPN endpoint and remote address space. On the AWS side, the private route tables needed a route back to 172.16.0.0/16 through the virtual private gateway.

Figure 3: Azure VPN gateway used for the lab connection.

Figure 4: The Azure-to-AWS VPN connection in the Connected state.

The important lesson: a VPN connection showing Connected is not proof that your application traffic can flow. Both directions need working routes; firewalls, network security groups, and AWS security groups must permit the actual source/destination traffic. For database replication, that meant PostgreSQL TCP port 5432, not the application port 3000.

I also treated the single operational tunnel as a lab limitation. A production design should validate redundant tunnels, failover behavior, monitoring, and appropriate routing resilience.

Phase 3 — Prepare AWS RDS and DMS

I provisioned private Amazon RDS for PostgreSQL and set up AWS Database Migration Service (DMS) endpoints for the Azure source database and the AWS target. The source was PostgreSQL 16 and the RDS target reported PostgreSQL 18.3, so this exercise also required attention to engine-version differences and schema compatibility.

Figure 5: AWS DMS source endpoint for the Azure-hosted PostgreSQL database.

Figure 6: AWS DMS target endpoint for Amazon RDS PostgreSQL.

I used full load plus change data capture (CDC). Full load copies the existing rows; CDC then applies ongoing changes while the source is still in use. This is valuable because a migration is rarely just “copy a database and hope nobody edits anything.”

The migration plan was:

Establish and test private connectivity and database access.
Create and test DMS source and target endpoints.
Run the pre-migration assessment and review warnings.
Run full load and CDC, monitoring task and table status.
Compare data and validate application read/write behavior.
Freeze source writes, confirm target consistency, and stop replication.
Reconfigure and move the application to AWS, then verify end-to-end access.

Phase 4 — Assess, replicate, validate

Figure 7: The DMS task configured for the migration.

Figure 8: Pre-migration assessment results.

The pre-migration assessment reported 43 checks: 26 passed, 2 warnings, 15 skipped, and 0 failed. “Skipped” and “warning” do not mean “ignore”; they tell you where manual review is still necessary.

Figure 9: DMS table validation showed three validated tables, with no reported mismatches or validation errors.

Figure 10: Comparing source and target row counts during migration.

At one comparison point, the source and target both showed 4 customers, 3 products, and 3 orders. These were point-in-time counts—not a substitute for comprehensive data validation. DMS validation subsequently reported 3 validated tables, 0 mismatched rows, and 0 validation errors.

The gotcha: rows migrated, but the schema was not application-ready

The most useful lesson in this lab was that a healthy-looking DMS task is not the same as a fully functional database cutover.

After replication, I had to repair PostgreSQL schema behavior on the target: sequence-backed IDs, defaults, and missing unique/foreign-key constraints. I also had to grant the application user the required sequence permissions. For example, in a controlled lab where that level of access is appropriate:

GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO appuser;

Why it matters: a row-count dashboard can be green while the first real customer insert fails. Databases do not give bonus points for beautiful dashboards.

Phase 5 — Cut over the database

Once I had validated the data, I stopped writes to the source, completed the migration checks, and stopped the DMS replication task. I then tested target-side writes. Additional customer records were inserted on the AWS side after cutover, so later target counts were intentionally different from the earlier source comparison.

Figure 11: Replication stopped after the database cutover.

The distinction is important: a post-cutover target row count should not be compared blindly against an earlier source snapshot. You need a defined validation window, a write-freeze boundary, and an understanding of which system became the source of truth.

Phase 6 — Move the Node.js app to private EC2

I copied the application to a private AWS EC2 instance over the established private network path, installed its Node.js dependencies, and configured it to connect to the RDS endpoint. The transfer excluded .env and node_modules, allowing dependencies and configuration to be created appropriately on the target. The application ran as a systemd service, and its /health endpoint exercised a database connectivity query.

The target app's connection path was:

Private EC2 app (:3000) -> RDS security group (:5432) -> PostgreSQL

Phase 7 — Put the private application behind HTTPS

The ALB was internet-facing; the application instance remained private. I configured an HTTP listener on 80 to redirect to 443, an HTTPS listener on 443 using an ACM certificate, and a target group forwarding to the app on HTTP 3000. The health check used /health.

User enters https://app.example.com
              |
              v
ALB listener :443 (TLS termination)
              |
              v
Target group :3000 (health check /health)
              |
              v
Private EC2 security group allows :3000 FROM ALB security group
              |
              v
Node.js service :3000 -> RDS :5432

The 504 Gateway Timeout I hit was a practical reminder that the ALB listener port and application port are separate. The listener accepted HTTPS on 443, but the target instance's security group initially did not allow the ALB to reach the application on port 3000. Once I allowed TCP 3000 from the ALB security group, the target could respond.

Figure 12: The order-management application used for end-to-end testing. The page's original environment label was hard-coded and is not, by itself, proof of which server handled the request.

What I would check before calling a real production migration complete

Network: VPN tunnel redundancy, both-direction routes, SG/NSG policy, DNS resolution, and private-only database access.
Database: DDL parity, indexes, constraints, sequences, roles, permissions, row counts, validation errors, application transactions, and a tested recovery path.
Cutover: approved freeze window, final CDC lag checks, explicit source-of-truth decision, rollback criteria, and stakeholder sign-off.
Application: process supervision, health checks, logs, connection pool settings, TLS validation, secrets management, and ALB target health.
Operations: monitoring, backup/restore exercises, runbooks, cost controls, and post-cutover support.

Future modernization (not implemented in this lab)

After a successful rehost, the next stage could be containerizing the app, pushing images to Amazon ECR, deploying to ECS/Fargate or EKS, managing infrastructure with Terraform, using AWS Secrets Manager, adding CI/CD, and strengthening observability. I'd also revisit RDS high availability and VPN redundancy.

Those are next steps, not things I am claiming to have delivered in this migration.

Final thoughts

The biggest takeaway wasn't a service name. It was learning that connectivity, replication, schema fidelity, application behavior, and cutover are five different things—and every one must be validated independently.

A green DMS status is good news. A working application that can read and write safely after cutover is better news.

It's been a long read, thank you for staying through to the end... Cheers!

DORA METRICS EXPORTER

Dimeji Ojewunmi — Thu, 08 Aug 2024 20:42:15 +0000

Introduction to Dora Metrics

DORA Metrics are key performance indicators used to measure the effectiveness of a software delivery process. They help teams understand the efficiency of their DevOps practices by tracking key aspects of the CI/CD pipeline.

Below are the outlined reason stipulate why we need to leverage the need of Dora Metrics in an organization

Deployment Frequency: Measures how often deployments occur. High frequency indicates a fast, iterative development process.
Lead Time for Changes: The time it takes from committing code to deploying it in production. Short lead times suggest an efficient pipeline.
Change Failure Rate: The percentage of deployments that result in a failure in production. Lower rates indicate more stable and reliable releases.
Time to Restore Service: The time it takes to recover from a failure in production. Faster recovery times imply a more resilient system.

Installation Setup

The following git command below are directories and files to be opened in the cause of our Dora metric exporter installation

mkdir dora_metrics_exporter
cd dora_metrics_exporter
python3 -m venv venv
touch requirements.txt .env exporter.py
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
touch requirements.txt .env exporter.py

`requirements.txt`context below

flask
prometheus_client
requests
python-dotenv

Run the script below right after the opening of 'requirements.txt'

pip install -r requirements.txt

`.env` context below

GITHUB_TOKEN=your_github_token
REPO_OWNER=holadmex (GitHub Username)
REPO_NAME=hng_boilerplate_expressjs (Project Repository Name)

`exporter.py` context below

# exporter.py
from flask import Flask
from prometheus_client import start_http_server, Gauge
import requests
import time
import os
import logging
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Configure logging
logging.basicConfig(level=logging.INFO)

app = Flask(__name__)

# Metrics
DEPLOYMENT_FREQUENCY = Gauge('deployment_frequency', 'Frequency of deployments')
LEAD_TIME_FOR_CHANGES = Gauge('lead_time_for_changes', 'Lead time for changes')
CHANGE_FAILURE_RATE = Gauge('change_failure_rate', 'Rate of change failures')
TIME_TO_RESTORE_SERVICE = Gauge('time_to_restore_service', 'Time to restore service')

# GitHub Configuration
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
REPO_OWNER = os.getenv('REPO_OWNER')
REPO_NAME = os.getenv('REPO_NAME')

HEADERS = {
    'Authorization': f'token {GITHUB_TOKEN}'
}

# Function to fetch data from GitHub API
def fetch_github_data():
    logging.info("Fetching GitHub data")
    url = f'https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/stats/code_frequency'
    response = requests.get(url, headers=HEADERS)
    response.raise_for_status()
    logging.info("Data fetched successfully")
    return response.json()

def calculate_metrics():
    logging.info("Calculating metrics")
    try:
        data = fetch_github_data()
        # Perform calculations for DORA metrics here

        # Dummy values for demonstration
        DEPLOYMENT_FREQUENCY.set(10)
        LEAD_TIME_FOR_CHANGES.set(5)
        CHANGE_FAILURE_RATE.set(0.1)
        TIME_TO_RESTORE_SERVICE.set(2)
        logging.info("Metrics calculated and set")
    except Exception as e:
        logging.error(f"Error calculating metrics: {e}")

@app.route('/metrics')
def metrics():
    return ""

if __name__ == '__main__':
    # Start Prometheus metrics server
    logging.info("Starting Prometheus metrics server")
    start_http_server(8001)
    logging.info("Prometheus metrics server started on port 8001")
    # Schedule metric calculations
    while True:
        calculate_metrics()
        time.sleep(300)  # Fetch data every 5 minutes

The code below execute the Dora Metric Exporter installation

python exporter.py
OR
nohup python exporter.py & (This cmd runs your application in the background without your terminal being held hostage)

NB:

On completion of the installation, Dora metrics job needs to be configured or added to Prometheus config file in order to export the generated metrics from Dora exporter to Prometheus which will be visualized on Grafana dashboard through Grafana data source.

scrape_configs:
  - job_name: 'dora_metrics'
    static_configs:
      - targets: ['localhost:8001']

DORA EXPORTER METRICS QUERY TO BE VISUALISED ON GRAFANA DASHBOARD

deployment_frequency
lead_time_for_changes
change_failure_rate
time_to_restore_service

Thank you for reading to the end,
Happy Hands On Practice!.

Python Messaging Queue App Development

Dimeji Ojewunmi — Tue, 16 Jul 2024 09:46:01 +0000

Hello guys,

Today we'll be developing a Python Messaging Queue Test, which consist of the following 'Tools' as our requirement.

RabbitMQ
Celery
Ngrok
Nginx

Lets talk about the theoretical aspect of the core tools we'll be using in the course of the task, courtesy of (HNG Internship)

What is RabbitMQ?

RabbitMQ is an open-source messaging software that implements the Advanced Message Queuing Protocol (AMQP) and is widely used for managing messaging queues. RabbitMQ here also serves as Amazon Simple Queue Service(SQ) distributed messaging system which is fully managed by AWS, makes it easy to decouple and scale microservices, distributed systems, and serverless applications.

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. its main purpose is to handle "asynchronous" task queue/job queue that handles the execution of background tasks.

Asynchronous programming is a form of programming that allows a unit of work (like a function or a task) to start processing before the previous unit of work finishes which also means its serves task to function independently.

What is Ngrok?

Ngrok is a tool that creates secure tunnels from a public endpoint (like the internet) to a locally running web service or application. It allows developers to expose their local development stage environment to the internet securely, making it easier to test and share work-in-progress applications with team members without deploying them to a live environment.

What is Nginx?

Nginx is a web server and reverse proxy server that is widely used for serving web content, handling load balancing, and acting as a proxy for email services. It's known for its speed, reliability, and low resource usage mainly in development to production stage.

Lets hop into the business of the day, getting hands on the project task as we progress in the course of the blog post.

Installation Prerequisite

sudo apt-get update

sudo apt-get install rabbitmq-server

sudo apt install nginx -y

Linux | ngrok documentation

Learn how to install ngrok on a remote Linux device to provide secure access and management.

ngrok.com

(Redirection link to installl Ngrok)

On successful completion of installing Ngrok, log on to

ngrok | API Gateway, IoT Device Gateway, Secure Tunnels for Containers, Apps & APIs

ngrok is a secure ingress platform that enables developers to add global server load balancing, reverse proxy, firewall, API gateway and Kubernetes Ingress to applications and APIs.

ngrok.com

and sign up, to have a generate token which will be integrated with your project directory on your Linux terminal environment. Run the command below

ngrok authtoken <your generate token>

Hold still, we'll be writing few couple blocks of code using Python and installing the required Python Framework/Library to execute our Python blocks of code functions.

First we'll open a directory on our Linux Vm (e.g Ubuntu)

mkdir messaging-queue-app

cd messaging-queue-app

touch app.py .env ngnix.conf

Created app.py File context

import os
from flask import Flask, request, Response
from celery import Celery
from dotenv import load_dotenv
from datetime import datetime
import logging
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

# Load environment variables
load_dotenv()

# Initialize Flask application
app = Flask(__name__)

# Configure Celery
app.config['CELERY_BROKER_URL'] = 'pyamqp://guest@localhost//'
app.config['CELERY_RESULT_BACKEND'] = 'rpc://'
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)

# Configure logging
log_path = os.path.join(os.path.expanduser('~'), 'messaging_system.log')
logging.basicConfig(level=logging.INFO, filename=log_path, filemode='a', format='%(asctime)s - %(message)s')

# Celery task to send email
@celery.task
def send_email(to_email):
    from_email = os.getenv('EMAIL_USER')
    from_password = os.getenv('EMAIL_PASSWORD')
    smtp_server = 'smtp.mail.yahoo.com'
    smtp_port = 465
    subject = 'Official Email from Flask Application'
    body = 'This is a test email sent from the Flask application.'
    msg = MIMEMultipart()
    msg['From'] = from_email
    msg['To'] = to_email
    msg['Subject'] = subject
    msg.attach(MIMEText(body, 'plain'))
    try:
        logging.info(f'Connecting to SMTP server {smtp_server}:{smtp_port}')
        with smtplib.SMTP_SSL(smtp_server, smtp_port) as server:
            logging.info('Logging in to SMTP server')
            server.login(from_email, from_password)
            logging.info('Sending email')
            text = msg.as_string()
            server.sendmail(from_email, to_email, text)
            logging.info(f'{datetime.now()} - Email sent to {to_email}')
    except Exception as e:
        logging.error(f'Failed to send email: {e}')

# Flask routes
@app.route('/', methods=['GET'])
def index():
    if 'sendmail' in request.args:
        send_email.apply_async(args=[request.args.get('sendmail')])
        logging.info(f'{datetime.now()} - Email queued for: {request.args.get("sendmail")}')
        return 'Email queued.'
    elif 'talktome' in request.args:
        logging.info(f'{datetime.now()} - Logged the current time.')
        return 'Logged the current time.'
    else:
        return 'Welcome to the messaging system. Use /?sendmail=email or /?talktome.'

@app.route('/logs', methods=['GET'])
def logs():
    if os.path.exists(log_path):
        with open(log_path, 'r') as f:
            log_content = f.readlines()
        filtered_logs = [line for line in log_content if 'Email sent to' in line or 'Email queued for' in line]
        return Response(''.join(filtered_logs), mimetype='text/plain')
    else:
        return "Log file not found.", 404

# Main entry point
if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

NB:

The Python Messaging queue application development here was setup to make use of (Yahoomail SMTP Send). You you're to put in your email address, and also generate an app token in your Yahoomail account security path, and make the configuration changes in the (.env)

Created .env File context

EMAIL_USER="ojewumi_dimeji@ymail.com"
EMAIL_PASSWORD="clndcnbjfpaelrj1"
SMTP_SERVER="smtp.mail.yahoo.com"
SMTP_PORT=465

Created nginx.conf File context

server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

On filling each files with their respective blocks of codes, there are few more Python requirement Frameworks/libraries to be installed right in our project directory using the (virtual Linux environment cmd)

Installation

sudo apt-get install python3-venv

Right after the python3-venv installation, execute the following command below on your Linux terminal. This command create a separate virtual environment in your project directory for your block of code function to carry out its functions.

python3 -m venv messaging-queue
source messaging-queue/bin/activate

On carry out the above, we'll commence installation of our Python Frameworks/Libraries right inside our created (venv messaging-queue) which you will be redirected to on executing the (source messaging-queue/bin/activate) cmd

pip install celery
pip install flask
pip install python-dotenv
pip freeze > requirements.txt (This CMD output all the installed Python pip installation for the application to run successfully)

Freight not, lol, we're 95% done to the completion of our project on carrying out the following command to execute our Python Messaging Queue Test.

First step:

sudo rabbitmq-server (This CMD launch our RabbitMQ sever to run for the course of our python messaging queue app test)

sudo rabbitmq-server -detached (This helps to run your RabbitMQ sever in the background without the terminal being held hostage)

Open another sperate terminal on executing each rabbitmq-server and celery cmd

"Optional, but a must on first attempt with the cmd above and below"

Second step:

celery -A app.celery worker --loglevel=info (This opens your Celery monitoring environment on checking for errors and successful ?sendmail execution of your python messaging queue app)

celery -A app.celery worker --loglevel=info --detach (This helps to run your Celery in the background without the terminal being held hostage.)

Third step:

python3 app.py (This CMD lunch your web app development URL to be accessed on your localhost web browser.)

nohup python3 app.py > app.log 2>&1 & (This helps to lunch the app in the background without the terminal being held hostage, and also output the return function to app.log.)

Here are the following path to the application web redirection on sending your queued email locally without Ngrok

localhost:5000
localhost:5000/?sendmail=happy-learning@gmail.com (Email you which to send message to)
localhost:+5000/?talktome
localhost:5000/logs

Finally, on exposing our `Local Development Test` to other team members, carry out the installation below;

sudo apt-get install screen

On completion of the installation, run;

screen -S ngrok

ngrok http 5000 (You should see an outputted endpoint "https://e7c9-44-202-0-101.ngrok-free.app/" which serves as your URL to be publicly accessed.)

Here are the following path to the application web redirection on sending your queued email with Ngrok

https://e7c9-44-202-0-101.ngrok-free.app/
https://e7c9-44-202-0-101.ngrok-free.app:5000/?sendmail=happy-learning@gmail.com
https://e7c9-44-202-0-101.ngrok-free.app:5000/?talktome
https://e7c9-44-202-0-101.ngrok-free.app:5000/logs

We've come to the End of the Task,

Celebrate yourself for not holding back on the success of completing the Python Messaging Queue App Development,

Happy reading,
Thank you.
Dimeji...

Linux Bash Script User Creation

Dimeji Ojewunmi — Wed, 03 Jul 2024 19:06:07 +0000

Hi Enthusiast DevOps Engineer,

Today we'll be having a deep insight on the need to write an automated users bash script in an organization, and also having all the metrics and password auto generated stored in a LOG_FILE.

The main reason for the creation of the users bash scrip is to mitigate repetitive user/group creation task manually, which might be daunting in most cases, and also to have an autonomous control of employees who have assess to the organization Linux workflow environment.

Before i proceed, the user bash script creation is in curtesy of (HNG Internship) program, which has just kicked off for the year 2024 "11th cohort", helping various beginners and intermediate tech enthusiast in having a real world project experience. To learn more about HNG internship program, kindly visit

hng.tech

Now lets delve to the business of the day, seat back, make use of your seat belt, while we journey through this together.

Firstly, we'll create a text file called users filled with the content below, which is dependent on our bash script file to execute its functions.

light; sudo,dev,www-data
 idimma; sudo
 mayowa; dev,www-data
 dmex; devops,sysops
 iosegbo; sysops

Secondly we'll be creating a file called create_users.sh which will run as our bash script file with the code below

#!/bin/bash

# Autogenerate Password and Metrics will be sent here
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"


# This will create a /var/secure file, and also assign a <write, read & execution> permission to the created file.
mkdir -p /var/secure
chmod 700 /var/secure

# Create or clear the log and password files
> $LOG_FILE
> $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

# Function to generate random password
generate_password() {
    echo $(openssl rand -base64 12)
}


# Read the input file, which is the <users> file we created to be dependent on our bash script code function execution
INPUT_FILE=$1
# Process each line in the file
while IFS= read -r line; do
  # Ignore characters before the semicolon
  after_semicolon="${line#*;}"

  # Split the line into items separated by commas
  IFS=',' read -ra items <<< "$after_semicolon"

  # Ensure a group exists for each item
  for item in "${items[@]}"; do
    item=$(echo "$item" | xargs)  # Trim whitespace
    if [ ! -z "$item" ]; then
      if ! getent group "$item" > /dev/null; then
        echo "Creating group: $item"
        sudo groupadd "$item"
      else
        echo "Group already exists: $item"
      fi
    fi
  done
done < "$INPUT_FILE"


# Process each line in the input file
while IFS=';' read -r user groups; do
    # Trim whitespace
    user=$(echo "$user" | xargs)
    groups=$(echo "$groups" | xargs)


    # Create user with home directory and primary group
    if ! id "$user" &>/dev/null; then
        useradd -m "$user"
        echo "User $user was created successfully." >> $LOG_FILE
    else
        echo "User $user already exists." >> $LOG_FILE
    fi

    # Set user's groups
    if [ -n "$groups" ]; then
        usermod -aG $groups "$user"
        echo "User $user added to group: $groups." >> $LOG_FILE
    fi

    # Auto Generate and set password
    password=$(generate_password)
    echo "$user:$password" | chpasswd
    echo "$user,$password" >> $PASSWORD_FILE
    echo "Password for user $user set." >> $LOG_FILE


done < "$INPUT_FILE"

Thirdly, we'll have to assign an execution permission to the created bash script file chmod 700 create_users.sh

Lastly, the command to execute our bash script file is

sudo ./create_users.sh

The command above will execute the above bash script function which is written as a code, enabling all the users to be assigned to the specified group and also enabling each users to have an assigned encrypted password which will be redirected or logged to a LOG_FILE encoded in the script.

Here are the list of commands to execute on our Linux VM to check the following output of our bash script block of code

sudo cat /var/log/user_management.log (This cmd checks for the activities which has taken place in the course of the bash script execution i.e "MERTICS").

sudo cat /var/secure/user_passwords.txt (This cmd checks for the auto generated password assigned to each users).

To confirm validity of your user creation, simply execute the following command below

su dmex

Then copy the encrypted password generated for dmex under sudo cat /var/secure/user_passwords.txt

Thank you for taking your time in reading through to the completion of the blog post, do have a wonderful time.