DEV Community: Okesanya Odunayo

Secure and Fast Static Website Deployment on AWS using Pulumi

Okesanya Odunayo — Mon, 07 Apr 2025 03:30:49 +0000

This is a submission for the Pulumi Deploy and Document Challenge: Fast Static Website Deployment

What I Built

In this project, I used Pulumi as an Infrastructure as Code (IaC) tool to create a secure and scalable static website infrastructure on AWS. The solution sets up an Amazon S3 bucket to host static site files and integrates it with Amazon CloudFront for fast, global content delivery. To ensure secure HTTPS access, I restricted permissions to only CloudFront, obtained an SSL certificate using AWS Certificate Manager (ACM), and set up a custom domain with Route 53.

Everything was written in Python using pulumi's Python SDK, which makes the codebase easy to read and maintain. The result is a modern react static website setup that is ready for production, cost-effective, and secure, all fully automated with pulumi.

Live Demo Link

CloudFront Links
https://d8fuic0sibn8l.cloudfront.net/
https://d8fuic0sibn8l.cloudfront.net/nonexistent-page
Custom Domain Links
https://challenge.drintech.online
https://challenge.drintech.online/nonexistent-page

Project Repo

DrInTech22 / Pulumi_Static_Website

Secure and Fast Static Website Deployment on AWS using Pulumi

This is a repository to set up a secure and fast static website deployment using pulumi.

Table of content

Overview

This project uses Pulumi as an IAC tool to set up a secure and fast static website deployment. Pulumi is a unique IAC tool that lets you define your infrastructure using your favorite programming language. With Pulumi, we'll set up the infrastructure (S3 bucket, CloudFront, and Route 53) on AWS and securely deploy the static website files to this infrastructure using Python.

Technologies

Pulumi: An Infrastructure as Code (IaC) tool that provisions and manages cloud resources using Python.
Python: A programming language that provides modules to interact with Pulumi and AWS…

View on GitHub

My Journey

I started this project by exploring the Pulumi documentation since it was my first time using Pulumi. It turned out to be a great experience because the Pulumi documentation is developer-friendly and easy to follow. Being a fan of Python and quite familiar with it, I chose to use the Pulumi Python SDK to set up the static website infrastructure and deploy a static React website to it.

My development journey was rewarding because I gained a better understanding of the technologies, but it was also filled with unexpected challenges. At one point, I even thought I had downloaded malware on my machine, lol. I began the journey using the helpful static website template provided by the Pulumi documentation. Let's go over the steps and the challenges I faced during development.

Summary of Technologies Used

Pulumi: An Infrastructure as Code (IaC) tool that provisions and manages cloud resources using Python.
Python: A programming language that provides modules to interact with Pulumi and AWS resources.
S3 Bucket: An object storage service that stores static website files and assets, serving as the origin for CloudFront.
CloudFront: A Content Delivery Network (CDN) that caches and securely delivers website content globally with low latency. These contents are cached at edge locations closer to your users.
Route 53: A service that manages DNS records, enabling the use of a custom domain for the website.

Architecture

Prerequisites

To get started, the following are required:

Python3, pip, and python3-venv installed
A basic understanding of Python
AWS CLI installed and configured

Step 1: Pulumi Installation

The command below installs pulumi on linux machine, refreshes the shell so the changes can reflect, and confirms the installation.

curl -fsSL https://get.pulumi.com | sh
source ~/.bashrc
pulumi version

Step 2: Setup Project Template

Next, I created a new folder website/ and setup the static website template. The static website template bundles sample files and configurations to deploy static website with S3 and CloudFront.

mkdir website && cd website
pulumi new static-website-aws-python

Since I'm using the pulumi new command for the first time, I needed to sign in to Pulumi Cloud to authenticate my CLI. I found this interesting because, after authentication, I could immediately start building the infrastructure. Pulumi automatically manages the state file remotely for me, unlike other IaC tools where I have to manually set up the backend file to manage the state file remotely before focusing on automation.

After authentication, I configured the project and stack for the website deployment. This include:
- setting the project name, description and stack name.
- selecting pip as the toolchain for installing dependencies.
- Setting us-east-1 as the region to deploy our aws resources such as S3 buckets.
- confirming the index, error document and website path.

After completing the configuration, pulumi proceeds to install the dependencies defined in the requirements.txt but fails due to python env error. I did not have python3-venv installed. Luckily for me, python provided me with the right command to install the correct version in the error message.

Pulumi manages the infrastructure setup as project and stack. Projects represent the entire infrastructure definition. They include metadata, runtime, and some configuration to manage and execute your infrastructure code. A project is defined in the pulumi.yaml file.

A stack is an instance of the project with its own configuration and state. Stacks include environment-specific configurations that allow us to deploy the infrastructure to different environments such as dev, staging, and prod. A stack is defined in the pulumi.dev.yaml.

After setting up the project and stack, pulumi generated the following files for me.

pulumi.yaml: This represents the project. It includes metadata, runtime and configs to manage the infrastructure code.

name: website
description: A Python program to deploy a static website on AWS
runtime:
  name: python
  options:
    toolchain: pip
    virtualenv: venv
config:
  pulumi:tags:
    value:
      pulumi:template: static-website-aws-python

pulumi.dev.yaml: This file represents the stack for the development environment. It includes environment-specific configurations such as the AWS region, the path to static files, and HTML documents.

config:
  aws:region: us-east-1
  website:indexDocument: index.html
  website:errorDocument: error.html
  website:path: ./www

__main__.py: This is the main file that contains the code defining the infrastructure. It specifies the resources to be deployed and their configurations.
requirements.txt: The Python dependencies needed to deploy the infrastructure using Pulumi are listed here.

pulumi>=3.0.0,<4.0.0
pulumi-aws>=6.0.2,<7.0.0
pulumi-synced-folder>=0.0.0,<1.0.0

venv/: This folder manages the Python dependencies for this project in an isolated virtual environment.
www/: This folder contains sample static web files, including index.html and error.html.
.gitignore: Files listed here are not tracked by Git. This prevents us from pushing dependencies to our repository.

A breakdown of the __main__.py

This file is where all the main operations happen. The breakdown highlights different parts of the code that work together to set up the static website infrastructure and how I address deprecated aspects of the code. After the detailed breakdown, I will adjust these components to deploy a more secure, personalized, and fast static website infrastructure.

Import the required modules

import pulumi
import pulumi_aws as aws
import pulumi_synced_folder as synced_folder

This imports the necessary modules to create AWS resources and sync files to an S3 bucket.

Read Configuration Settings

config = pulumi.Config()
path = config.get("path") or "./www"
index_document = config.get("indexDocument") or "index.html"
error_document = config.get("errorDocument") or "error.html"

This reads environment-specific settings from the stack using pulumi.Config and passes them as variables.

Create an S3 Bucket & Enable Static Website Hosting

bucket = aws.s3.BucketV2(
    "bucket",
    # website={
    #     "index_document": index_document,
    #     "error_document": error_document,
    # },
)
bucket_website = aws.s3.BucketWebsiteConfigurationV2(
    "bucket",
    bucket=bucket.bucket,
    index_document={"suffix": index_document},
    error_document={"key": error_document},
)

This creates an S3 bucket and sets it up for static website hosting. It also specifies the index and error documents. The bucketV2 class no longer accepts the website argument for configuring the bucket as a website, as it would cause errors. Instead, the BucketWebsiteConfigurationV2 class is used to configure the bucket for website hosting.

Set Bucket Ownership Controls

ownership_controls = aws.s3.BucketOwnershipControls(
    "ownership-controls",
    bucket=bucket.bucket,
    rule={
        "object_ownership": "ObjectWriter",
    },
)

This sets permissions that define who owns newly uploaded objects. "ObjectWriter" means the uploader retains ownership. I will modify this later on for enhanced security.

Allow Public Access to Objects

public_access_block = aws.s3.BucketPublicAccessBlock(
    "public-access-block",
    bucket=bucket.bucket,
    block_public_acls=False,
)

This allows public read access to objects by turning off the bucket public ACL blocking. In other words, it makes the bucket public so anyone can access it. However, this is not recommended for security best practices and it will be adjusted later because it lets anyone access our static files insecurely over HTTP.

Upload static files to S3 bucket

bucket_folder = synced_folder.S3BucketFolder(
    "bucket-folder",
    acl="public-read",
    bucket_name=bucket.bucket,
    path=path,
    opts=pulumi.ResourceOptions(depends_on=[ownership_controls, public_access_block]),
)

This uses pulumi_synced_folder to upload the static files located at path to the S3 bucket. It applies the public-read ACL settings to the uploaded objects, allowing public users to access the website files. It ensures that ownership_controls and public_access_block are set first. This order is important to avoid permission conflicts and ensure that the object ACL aligns with the bucket ownership settings.

Create a CloudFront CDN for Caching & Distribution

cdn = aws.cloudfront.Distribution(
    "cdn",
    enabled=True,
    origins=[
        {
            "origin_id": bucket.arn,
            "domain_name": bucket_website.website_endpoint,
            "custom_origin_config": {
                "origin_protocol_policy": "http-only",
                "http_port": 80,
                "https_port": 443,
                "origin_ssl_protocols": ["TLSv1.2"],
            },
        }
    ],

This creates a CloudFront distribution to serve the website using a CDN. The origin (source) is set to the S3 bucket website endpoint and uses "http-only" because S3 website hosting does not support HTTPS natively. This means CloudFront will internally fetch the static files from S3 over HTTP.

Configure CloudFront Cache Behavior

default_cache_behavior={
    "target_origin_id": bucket.arn,
    "viewer_protocol_policy": "redirect-to-https",
    "allowed_methods": [
        "GET",
        "HEAD",
        "OPTIONS",
    ],
    "cached_methods": [
        "GET",
        "HEAD",
        "OPTIONS",
    ],
    "default_ttl": 600,
    "max_ttl": 600,
    "min_ttl": 600,
    "forwarded_values": {
        "query_string": True,
        "cookies": {
            "forward": "all",
        },
    },
},

This setup forces users to always access the website through CloudFront using HTTPS (redirect-to-https). It also includes request methods, cookies, caching rules, and TTL (cache time) settings. The default TTL caches your static files for 600 seconds (10 minutes), which is suitable for content that changes often.

For example, if your static website is still in the early stages of development and you release several builds a day, this setting works well. Use 3600 seconds (1 hour) or 86400 seconds (24 hours) if your static website is more stable and rarely updated.

If you're using a longer TTL and you update any or all objects in the bucket, CloudFront offers an option to invalidate the cache for specific or all objects. This ensures that your latest changes are visible to users.

Configure CloudFront Error Handling & Security

price_class="PriceClass_100",
custom_error_responses=[
    {
        "error_code": 404,
        "response_code": 404,
        "response_page_path": f"/{error_document}",
    }
],
restrictions={
    "geo_restriction": {
        "restriction_type": "none",
    },
},
viewer_certificate={
    "cloudfront_default_certificate": True,
},

PriceClass_100 limits the CloudFront distribution to use edge locations in the U.S., Canada, and Europe to save on costs. If you need wider coverage for your users, you can look into other CloudFront price classes.

404 errors from the S3 bucket are managed by serving the errorDocument (error.html). You can use geo_restriction to control regional access or keep your content available worldwide. CloudFront uses its default SSL certificate, enabling public users to securely access your content over HTTPS.

Export Public URLs & Endpoint

pulumi.export("originURL", pulumi.Output.concat("http://", bucket_website.website_endpoint))
pulumi.export("originHostname", bucket.website_endpoint)
pulumi.export("cdnURL", pulumi.Output.concat("https://", cdn.domain_name))
pulumi.export("cdnHostname", cdn.domain_name)

This exports the necessary URL and hostname for S3 and CloudFront to access the newly deployed static website.

Step 3: Deploy the static website

Now that we understand how the infrastructure code works, let's deploy the infrastructure and access the sample static website.

Run pulumi up. This command validates and sets up the infrastructure. After a successful validation, it confirms if I would like to deploy the infrastructure.
After I select 'yes', pulumi deploys the infrastructure. I can track the deployed stack on the Pulumi console. The console offers an easy-to-use dashboard where I can monitor deployments, view logs (updates), and see the resources deployed to a stack.
The CloudFront distribution took the longest time to create. Once the deployment is complete, I will be able to access the sample static website over HTTP using the S3 originURL and over HTTPS using the CloudFront cdnURL.
The the error page is viewed by adding a nonexistent path to the URL, for example, https://cloudfronturl.net/nonexistent-page.

During deployment, Pulumi displayed a warning: "website_endpoint deprecated." The website_endpoint is an output property that is no longer supported. Pulumi Co-Pilot, available on the console, was helpful in fixing the warning.

To resolve it, I replaced the website_endpoint property with the bucket_regional_domain_name property, as suggested by Pulumi Co-Pilot, in the CDN configuration and export section at the bottom of the code. Then I used the pulumi preview command to confirm that the warning was resolved.

To deploy any custom website to S3, you need to update the path in the stack file (pulumi.dev.yaml) to point to the location of the static website files.

I wanted to make things more interesting by deploying a React application I built to the S3 bucket. I moved the react_website/ folder to the root of the repository. The production build, which contains the static files, is located in the react_website/build/ directory. The configuration in the dev stack is updated to deploy the React application.

config:
  aws:region: us-east-1
  website:indexDocument: index.html
  website:errorDocument: error.html
  website:path: ./react_website/build

pulumi up command deploys the React application. The newly deployed static website can be accessed using any of the provided URL links. I tested the error page by add a non-existent path to the URL and it works fine. The next phase is were things gets tricky and almost messy.

Step 4: Secure the website deployment

This step was the most challenging, requiring the most troubleshooting. I frequently switched between Pulumi documentation and AWS console to resolve unexpected issues.

Currently, the website can be accessed via HTTP using the bucket endpoint and via HTTPS using the cdnURL. Disabling BucketPublicAccessBlock makes the bucket and its objects publicly accessible, which poses a security risk. I set up a secure deployment to ensure users can only access the static website securely through the CloudFront URL.

To enhance security, I restricted the bucket permissions so that only CloudFront can fetch objects from the bucket. I modified the public_access_block settings to block all public access to the S3 bucket. This ensures that no one can access the S3 bucket or read the objects unless explicitly permitted. In other words, the bucket is now private.

public_access_block = aws.s3.BucketPublicAccessBlock(
    "public-access-block",
    bucket=bucket.bucket,
    block_public_acls=True,
    block_public_policy=True,
    ignore_public_acls=True,
    restrict_public_buckets=True,
)

Next, I set object_ownership to BucketOwnerEnforced. This setting disables ACLs and ensures the bucket owner is the owner of all objects. With this configuration, access to the objects can only be explicitly granted using bucket policies.

ownership_controls = aws.s3.BucketOwnershipControls(
    "ownership-controls",
    bucket=bucket.bucket,
    rule={
        "object_ownership": "BucketOwnerEnforced",
    },
)

I created CloudFront Origin Access Identity (OAI). An OAI is a virtual identity that can be associated with a CloudFront distribution to secure access to private S3 bucket.

oai = aws.cloudfront.OriginAccessIdentity("oai")

Then I attached the OAI to the CloudFront distribution by switching from custom_origin_config to s3_origin_config. The s3_origin_config sets up CloudFront to access the S3 bucket through the OAI.

cdn = aws.cloudfront.Distribution(
    ...
    origins=[
        {
            "origin_id": bucket.arn,
            "domain_name": bucket.bucket_regional_domain_name,
            "s3_origin_config": {
                "origin_access_identity": oai.cloudfront_access_identity_path,
            },
        }
    ],
    ...
)

I Created an S3 bucket policy that allows only the CloudFront OAI to access the S3 bucket. The policy grants limited permission, allowing CloudFront to only fetch objects from the bucket. The code below creates the policy and attaches it to the bucket.

bucket_policy = aws.s3.BucketPolicy(
    "bucketPolicy",
    bucket=bucket.id,
    policy=pulumi.Output.all(bucket.arn, oai.iam_arn).apply(lambda args: f"""{{
        "Version": "2012-10-17",
        "Statement": [
            {{
                "Effect": "Allow",
                "Principal": {{"AWS": "{args[1]}"}},
                "Action": "s3:GetObject",
                "Resource": "{args[0]}/*"
            }}
        ]
    }}""")
)

When I ran pulumi up to test the new security changes, I encountered an 'AccessControlListNotSupported: The bucket does not allow ACLs' error for all the objects. This error occurred because the synced_folder module attaches ACLs to objects and uploads them to the S3 bucket. However, the BucketOwnerEnforced setting disables ACLs, which conflicts with synced_folder.

Although the Pulumi documentation specified that the synced_folder ACL property can be set to None, I still encountered errors asking for the value when I executed the code. I manually wrote the code to upload objects to the S3 bucket using the bucketObject class from the S3 submodule and a nested for loop to iterate through all the files. Pulumi Co-Pilot was helpful for this.

I then replaced the synced_folder code with the new one generated.

for root, _, files in os.walk(path):
    for file in files:
        file_path = os.path.join(root, file)
        key = os.path.relpath(file_path, path)

        aws.s3.BucketObject(
            key,
            bucket=bucket.bucket,
            source=pulumi.FileAsset(file_path),
            opts=pulumi.ResourceOptions(depends_on=[ownership_controls, public_access_block]),
        )

In the code above, I used the os module to get the path of the static files on my machine. I imported the module at the beginning of the code.

import os

Now, when I ran pulumi up and tried to access the website, I got an access denied error. Checking my S3 bucket, it looked like a disaster had struck—nothing was there. So, I ran pulumi destroy && pulumi up to rebuild the infrastructure from scratch. While this brought the objects back, the access denied error was still there.

I reviewed all the configurations in the code and double-checked the settings for CloudFront and the S3 bucket in the console, but everything seemed correct. While going through the CloudFront distribution dashboard for the third time, I noticed that the default root object option was not set. It didn't seem like a problem because it was optional, and the setup worked fine without it until I set the OAI. When you set OAI, some optional settings become mandatory.

The default root object tells CloudFront which object to return when users access the CloudFront URL, and in this case, it's index.html. I asked Pulumi Copilot for the right argument to set it up in the CDN configuration.

cdn = aws.cloudfront.Distribution(
    "cdn",
    enabled=True,
    default_root_object="index.html",
    ...
)

I deployed the new changes, expecting everything to work smoothly. But when I tried accessing the website using the CloudFront URL, a weird file named "download" with no extension got downloaded to my computer. I deleted it right away. For a moment, I worried that I had downloaded malware and that my secure website was under attack, after just setting up enhanced security for it.

I realized I needed to see what was happening in my infrastructure to answer questions like: Did my request reach the CloudFront distribution? Was it successful? Did it return an error, and if so, what was the error?

I switched to the CloudFront dashboard, navigated to the logging tab, and set up a logging destination for CloudFront to send logs to a CloudWatch log group I had created for a Lambda function in my previous project.

Then I tried accessing the link again so the request and response would be logged, allowing me to gather information to troubleshoot this unexpected behavior. I accessed the log from CloudWatch, but I couldn't understand it right away. I pasted it into Pulumi Copilot for analysis.

With the response from Pulumi Copilot, I realized that the code I used to upload objects didn't set the correct file type for them. Instead, it set the file type for all objects to application/octet-stream instead of text/html, which is needed to display the index.html.

This explains why a strange file was downloaded, which I suspected was my index.html, and I could confidently open it to confirm.

I asked Pulumi Copilot again for a solution to set the file type in the nested for loop code that uploads the objects.

import mimetypes
...

for root, _, files in os.walk(path):
    for file in files:
        file_path = os.path.join(root, file)
        key = os.path.relpath(file_path, path)
        content_type, _ = mimetypes.guess_type(file_path)
        print(f"Uploading: {file_path} as {key}")

        aws.s3.BucketObject(
            key,
            bucket=bucket.bucket,
            source=pulumi.FileAsset(file_path),
            content_type=content_type or "application/octet-stream",
            opts=pulumi.ResourceOptions(depends_on=[ownership_controls, public_access_block]),
        )

The nested loop now uses the mimetypes module to set the correct file type for each uploaded object. I ran pulumi up, and it finally succeeded, or so I thought. My React static website is now securely accessible through the CloudFront URL link. When I tried the origin URL, it returned an access denied error, which was expected. However, when I tried to access the error page using the CloudFront URL, I also received an access denied error, which was unexpected.

So, I turned to the trusty old troubleshooting library (Stack Overflow). I discovered that because CloudFront and the S3 bucket are secured with OAI, the S3 bucket sends a 403 error code to CloudFront for pages that don't exist, instead of a 404. I tweaked the CloudFront settings to show the error page and send a 404 error response to users when it receives a 403 error code from the S3 bucket.

...
custom_error_responses=[
    {
        "error_code": 403,
        "response_code": 404,
        "response_page_path": f"/{error_document}",
    }
]
...

I deployed the latest changes, and now I can access both the main page and the error page of the static website. The next step is to personalize it with a custom domain.

Setup a Custom Domain

Sharing the CloudFront URL with my friends to access my secure, fast, and beautiful static website felt boring. I wanted a custom domain that reflects a bit about me and what the website is about. I followed the steps in the Pulumi documentation to set up a custom domain for the static website.

The steps include:

Adding the domain and subdomain configuration to the stack.
Importing the new configurations into my code.
Creating and validating a new SSL/TLS certificate with ACM. I used a DNS validation method, which involves creating a DNS record in the hosted zone to confirm domain ownership.
Adjusting the CloudFront code to handle requests for the custom domain by using the aliases argument.
Creating a Route 53 alias A record that maps our subdomain to the CloudFront domain.

You might be wondering why we use an alias A record instead of a CNAME record, which is commonly used for mapping one domain to another. The Route 53 alias A record is recommended for mapping domains to AWS services like CloudFront. With an alias A record, DNS queries are resolved internally, making them fast and more efficient.

After setting up the custom domain, I was able to access my static website using the personalized domain. When I create a new production build for the React application, all I need to do is run pulumi up. Pulumi handles the rest by automating the deployment to the secure S3 bucket.

Using Pulumi

This is my first time using Pulumi as an IAC tool, and I find it amazing. Pulumi lets me use Python, which is my favorite programming language. This made Pulumi feel familiar to me, like it was a part of my workflow. Like other IAC tools, Pulumi saves me the hassle of clicking through the AWS console to set up infrastructure and allows me to develop repeatable infrastructure. But unlike other IAC tools, Pulumi lets me automate right away without first manually configuring the remote state file.

The Pulumi console and Pulumi Copilot (my personal assistant for this project) were very helpful in troubleshooting, referencing, and successfully completing this project. The Pulumi console provided updates (logs) on the changes I made, making it easy to track changes and reference past execution outputs. Pulumi Copilot, my intelligent personal assistant, helped me fix deprecated arguments, generate bucket policies, analyze logs, and provided possible solutions.

In fact, when I wanted to secure my static website infrastructure, I copied the entire code provided by the template and gave prompts to Pulumi Copilot to set up a secure website infrastructure. This simplified the process, allowing me to focus on adjusting and reviewing the code using the pulumi documentation to make a few tweaks. It's incredible to have an AI assistant that understands the infrastructure tool I'm using. It makes infrastructure provisioning easy.

Below are some of my key prompts:

Prompt 1

can you adjust this code to set the content type for each file uploaded? 

for root, _, files in os.walk(path):
    for file in files:
        file_path = os.path.join(root, file)
        key = os.path.relpath(file_path, path)

        aws.s3.BucketObject(
            key,
            bucket=bucket.bucket,
            source=pulumi.FileAsset(file_path),
        )

Prompt 2

The option acl=None is not working. It still ask for acl value on runtime. Is there any good alternative to S3BucketFolder? 

bucket_folder = synced_folder.S3BucketFolder(
    "bucket-folder",
    acl=None,
    bucket_name=bucket.bucket,
    path=path,
    opts=pulumi.ResourceOptions(depends_on=[ownership_controls, public_access_block]),
)

Prompt 3

this is the cloudfront log I got from cloudwatch. Any reason why I can't access my static files via cloudfront URL after using OAI?

{
    "date": "2025-04-02",
    "time": "18:01:10",
    "x-edge-location": "LOS50-P2",
    "sc-bytes": "594",
    "c-ip": "102.89.44.119",
    "cs-method": "GET",
    "cs(Host)": "d39ymmysanotpr.cloudfront.net",
    "cs-uri-stem": "/",
    "sc-status": "200",
    "cs(Referer)": "-",
    "cs(User-Agent)": "Mozilla/5.0%20(Windows%20NT%2010.0;%20Win64;%20x64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/135.0.0.0%20Safari/537.36",
    "cs-uri-query": "-",
    "cs(Cookie)": "-",
    "x-edge-result-type": "Miss",
    "x-edge-request-id": "-lB3Ff7SJ8HEdpSZeS8d8AwV8kjflOVlQdgaI_-yqDYQ-TenMjY2nA==",
    "x-host-header": "d39ymmysanotpr.cloudfront.net",
    "cs-protocol": "https",
    "cs-bytes": "486",
    "time-taken": "0.480",
    "x-forwarded-for": "-",
    "ssl-protocol": "TLSv1.3",
    "ssl-cipher": "TLS_AES_128_GCM_SHA256",
    "x-edge-response-result-type": "Miss",
    "cs-protocol-version": "HTTP/2.0",
    "fle-status": "-",
    "fle-encrypted-fields": "-",
    "c-port": "17055",
    "time-to-first-byte": "0.480",
    "x-edge-detailed-result-type": "Miss",
    "sc-content-type": "application/octet-stream",
    "sc-content-len": "238",
    "sc-range-start": "-",
    "sc-range-end": "-"
}

Kindly note this submission was published April 06, 8:42PM PDT.

Reference Links

Thanks for reading. Your suggestions and feedbacks are highly welcome. Kindly drop them in the comment section.

Perfect Development Workflow with Mirrord: Every Developer’s Hero

Okesanya Odunayo — Tue, 20 Aug 2024 20:20:08 +0000

Introducing mirrord, a tool designed to streamline developers' workflows. With mirrord, you can test your code in the cloud in seconds!

Introduction

Remember the long, tedious process of building an application, probably dockerizing it, then deploying to staging for testing, only to encounter bugs that are only realized in the cloud environment? These are the incidents that inevitably slow down the deployment process.

Well, what if you could catch those bugs before your code reaches the final testing stage and avoid occasionally breaking the staging server with failed tests?

The hero, mirrord, is here. Less troubles! mirrord enhances your development workflow by bringing the cloud environment closer to your local setup. This lets you test your code in the context of a cloud environment throughout the development process.

Yes, you heard it right. Mirrord is bringing the cloud environment to your local setup!

The Pain Points of Development Environments

When developing a Kubernetes application, the primary challenges usually stem from setting up a local Kubernetes environment and passing your code through a lengthy CI/CD pipeline, followed by waiting for the code to be deployed to the cloud before testing can commence.

To circumvent this frustrating delay and lengthy process, developers are often tempted to save cloud testing for the final stages of development. However, this approach piles up bugs and issues that need to be addressed later in the cloud. This scenario can be best avoided by testing in the cloud from the very beginning of development.

There are still more challenges developers face in their development workflow. Let's discuss these issues in more detail.

Hurdles of dockerization and deployment

Developers often need to dockerize their applications and go through a lengthy CI process before they can start testing in the cloud. These steps can be time-consuming and sometimes complex, diverting focus from actual development tasks. This shift in focus consumes valuable time that could be better spent on developing the application.

Discrepancy between the local and cloud environment

When you deploy your app to the cloud, it faces a different environment. It must handle real-time traffic, interact with external services, and work with new configurations. This can lead to unexpected bugs that may break the application and make the server unusable for a period of time.

Challenges with shared staging servers

In many development teams, a shared staging server is used for testing code before it goes live. When one developer's code causes the staging server to crash, it creates a bottleneck.

Other developers are forced to wait until the server is back up and running before they can continue their work. This not only slows down individual progress but also disrupts the entire team's workflow, leading to frustration and decreased productivity.

Debugging

Debugging in a cloud environment can be challenging. When you encounter a bug on the shared server, you have two options.

Bring a copy of the cloud environment down to your local environment to replicate the bug.
Start debugging on the remote server.

Let's consider how these options will serve us well in an environment of about 300 microservices.

The first option requires you to replicate over 300 services and manage their individual dependencies and configurations. Beside the technical difficulty, such a massive workload can fry your laptop. I don't think you would love to tick this option.

The second option lets you start debugging right away, but there's a downside. Your work can be disrupted when another developer's code breaks the server. Additionally, spending too much time on the shared server can slow down other developers who also need to test their code, causing delays for the entire team.

How Mirrord Saves the Day!

Mirrord provides solutions to the seemingly endless troubles developers face in their workflow. You can get your code running in your cloud environment with just a single button click.

It provides your code with everything it needs to operate like it's on the cloud—access to other applications, databases, configurations, and most importantly, real-time traffic.

Mirrord operates in the memory of your local process in your friendly environment and as an agent pod in the cloud. The agent pod duplicates incoming requests to the target pod (the cloud version of your code) and sends a copy to your local process. Your code interacts with this traffic, and you can take actions against the results —tweak, modify, and test your code again.

From your local IDE, your code operates like it's living in the staging server. However, the server still handles requests and sends out responses seamlessly. It doesn't even know you were there!

Core advantages of mirrord

Mirrord makes developers' lives easier. Let’s take a look at what you will enjoy from using mirrord.

Simplicity

mirrord makes things easy for developers by offering a straightforward setup process and user-friendly commands. With just a few steps, you can configure mirrord to integrate seamlessly into your development environment. Get started quickly with minimal effort.

Improved Productivity

mirrord abstracts away the need for frequent git pushes and the delays associated with CI/CD pipelines, reducing both the time and stress involved in the development process. Instead of waiting for code to be pushed, built, and deployed, you can focus directly on writing and testing your code.

Faster Deployment

With the cloud environment integrated into the developer’s workflow, and the complete elimination of containerization and zero stress of CI process, you now experience faster development workflow which equals faster time to market of applications.

Efficient debugging

No more reading logs to tackle remote bugs. With mirrord, you can replicate remote bugs directly in your local environment, allowing you to debug them more effectively. Debugging now becomes a seamless part of your development process.

Shift left on cloud testing

You no longer have to save cloud testing for the final step. Test your code in the cloud continuously as you develop for faster iterations and improvements. This allows you to catch and fix issues earlier in the development cycle, leading to higher quality code.

Saves the production-like environment

mirrord helps save the staging server. By allowing you to test your code on the staging server without actually deploying it, mirrord removes the experience of frequent disruptions and downtimes in the shared staging server.

Collaboration

mirrord makes team collaboration much easier. With the new feature, mirrord for teams, you can now mirror traffic from the same target pod. This means multiple team members can test their code against similar traffic at the same time, producing more consistent and reliable results

Unique features of mirrord

Easy to use plugins

Mirrord offers extra flexibility with its powerful plugins, which can be easily integrated into your local Integrated Development Environment (IDE). These plugins are designed to enhance your development experience by offering a range of functionalities that simplify your workflow. Begin testing with just a single click.

Regulate traffic flow

Mirrord lets you control how traffic is directed to your local service. You can choose to mirror traffic to the target pod and send a copy to your local process, or you can configure your local process to handle requests exclusively by stealing the traffic if you are confident in your code.

You can also streamline your testing by exposing your code to specific traffic using filters. These flexible options enable you to test your code under different configurations, ensuring that it performs well in the context of the cloud.

Granular configuration

mirrord offers flexible configurations for how your local process responds to traffic. For example, if both the target pod and your local process write to the database, it can result in duplicate data. You can configure your local process to write to the remote database or let the target pod handle it while your local process writes to a local instance of your database.

This level of control helps you tailor the development environment to your specific needs and manage how your local process accesses certain resources in your cluster.

Resource cleanup

Mirrord is quite efficient and reliable. It does a few things in the cluster to bring the remote functionalities to your cluster environment but unlike other development tools, it doesn't leave your cluster messy with stale pods or resources.

When you terminate a mirrord session, it properly cleans up its resources in your cluster, ensuring that no resources are left behind that could disrupt your environment or accidentally incur costs.

Customer centric

Facing a problem connecting your code to the cloud? No worries. Mirrord team is available to support you and assist in resolving any problems you may face. They provide assistance through various channels, including email, public Discord, support tickets, private Discord, Slack, or Teams.

You can also schedule a live demo. Mirrord team will guide you in fully utilizing the functionalities of mirrord within your organization and assist in resolving any issues you may encounter.

Additionally, Mirrord values user suggestions. If there is a feature you would like to see implemented, just let them know. They are absolutely committed to enhancing your development experience.

Conclusion

In summary, mirrord saves developers troubles in their development workflow by making it easier to test their code in a cloud environment from the comfort of their local IDE. This significantly saves time, improves developer productivity, and reduces the chances of bugs sneaking into the production-like environment.

On the business side, this tremendously improves the time-to-market of products. But it doesn't end there!

Don't forget, mirrord is every developer's hero, and collaboration is key to every business's success. You wouldn't want to limit this tremendous capability of mirrord to a single person on your team. Empower your team with a better and faster workflow and promote collaboration with the new feature - mirrord for teams, which recently launched on the 17th, June. Wondering what is so amazing about this new feature?

Mirrord for teams extends the capabilities of mirrord by providing additional functionalities such as concurrent use, enhanced security and deployment-level mirroring. Register here to get started.

You want to hear more hero stories of mirrord in action? Read the user stories to see how Mirrord helped them.

The Problem Docker Solves: Beginner's Introduction to Docker

Okesanya Odunayo — Thu, 16 May 2024 11:54:03 +0000

Introduction

The recent talk of the town - a revolution to how applications are deployed. When developers build and successfully run an application, most times the application fails on deployment to different environments.

Docker takes an application and everything it needs (dependencies and runtime), then puts them in an isolated environment. This makes the application independent of other processes and enables it to run smoothly across different environments (servers). Interesting, isn't it? Let’s dig deeper into that!

This article explains docker, the problems it solves, basic concepts and benefits of using docker.

The PROBLEM

Redundant configurations
Conflicting dependencies (Dependency hell issue)
Slow deployment
Scaling

After an application is built, the next step is to deploy to production environments. Each production server is manually configured - installing application dependencies, libraries, which makes the deployment process slow and cumbersome.

To make matters worse, some applications usually require similar dependencies but of different versions leaving us with the hunt of compatible versions of dependencies for these applications. In several scenarios, we’re left with no other option but to use a deprecated version that can support both applications. This issue is popularly known as the Dependency Matrix Hell.

Also, applications that perfectly work on a developer’s machine might fail to run on a production environment. Some underlying processes and applications running on the production server makes the production environment incompatible and different, causing the application to fail. This makes the developer yell “It works on my machine!!”.

The SOLUTION - DOCKER

Docker is an open-source containerization platform. The free availability of docker to a wider audience contributed to the vast adoption of this containerization technology which positively transformed the deployment game in the industry.

Docker allows developers to package an application together with its dependencies, libraries and runtime into lightweight, isolated, self-contained units (containers).

This isolated and self-dependent functionality prevents conflicts with underlying processes (other applications and system processes), and ensures the application runs the same way in any environment. Thereby eliminating the popular “it works on my machine” problem - healing to every developer's headache.

Since the application is packaged with its configuration requirements and runtime, this further eliminates the need to manually configure each production server. All you need on the production server is docker installed!

Basic Docker Terminologies

Image

A docker image is a lightweight, independent and standalone package. It contains the application code, dependencies, libraries and runtime. With a docker image, you can easily deploy several containers.

Most images are built upon existing images or parent images. For example, a front-end application image can be built on a httpd-server image which provides server functionality and configurations to the front-end application.

Images are immutable and cannot be modified after creation. Images are built using a dockerfile. Dockerfile is a file with defined steps (instructions) to create a docker image.

Container

A container is a lightweight, self-sustained unit that runs the application code and the dependencies. It is the running instance of the application created from the docker image.

Containers pick up only the necessary components needed for execution of the application from the docker image. This makes them lightweight, portable, and easy to run across different environments (servers).

Docker registry

A docker registry is a remote storage that enables distribution of docker images. It serves as a central storage for uploading and downloading images. Docker Hub is the official public registry for Docker images.

Docker Hub provides two types of repositories (storage types):

Public repositories: Images in this repo are generally available to the public. They are free and open-source images.
Private repositories: Private repositories are usually created by an organization. Images found here are only accessible by members of the organization.

Containers Vs Virtual Machines

Both containers and virtual machines are capable of running applications in an isolated environment, but it is important to understand their key differences.

Virtual machines are virtual computers within your physical computer that run a full guest operating system. Each virtual machine shares the underlying hardware resources with the host operating system (your personal OS) and other virtual machines. This makes them heavy and compute-resource demanding.

Containers are lightweight, isolated environments that include only the application and required dependencies. Each container shares the operating system kernel with the host and other containers. This makes them light-weight and easy to deploy.

Benefits of Docker

Consistency: With docker, applications run successfully and behave the same way across different environments from development to production.
Portability: Containers are lightweight, this makes it easier to deploy applications to several environments and even store the image in docker registries.
Isolation: Isolation of each container prevents conflicts between containers and host machine, eliminating the occurrence of dependency matrix hell - where different applications require similar dependencies but of different versions.
Collaboration: since the application runs successfully across all platforms, this will foster collaboration between teams, such as the developers and operations team.
Scalability: Docker empowers developers to create or destroy several instances of the application in response to user demands.

Conclusion

With the elimination of ‘it works on my machine problem’ and ‘dependency hell issue’, organizations now experience consistent deployment and efficient scalability, thanks to docker. As Docker clears obstacles in your deployment journey, your applications are only a few commands away from a successful deployment.

I have also written a short story to further help visualize the problem-solving feature of Docker on X (formerly twitter). You can check it out here.

Evolution of DevOps! How it started

Okesanya Odunayo — Sun, 05 May 2024 11:20:19 +0000

Introduction

To develop an application, there are set of steps that mints the project requirements into a live application. The need for an organized approach towards software development led to the adoption of two major software development practices:

Waterfall model (traditional)
Agile methodology (classical)

These software development approaches are also known as SDLC (Software Development Life Cycle).

In this article, we'll look at traditional software development approaches, the issues they imposed, and how we transitioned from these traditional practices to DevOps practices(modern).

Waterfall Model

Waterfall Model is also referred to as linear-sequential life cycle. It is a step-by-step approach towards software development where each phase must be completed before moving on to the next one.

It is the earliest SDLC practice. Each current phase is independent of the next phase leading to minimal collaboration between the separate teams and any required changes to final product will require restarting the SDLC.

A case study to visualize this will be an e-commerce web app built for a client.

Following the waterfall model, the steps to follow would be:

Requirements - Design - Development - Testing - Deployment - Maintenance

Requirements are gathered from the client, the web app is built, then taken to the testing phase to be tested, and finally deployed. After that, it is reviewed by the client. What if our client wants some adjustments like more functionality, animations or a different layout to attract customers to products on the e-commerce web app?

That would be a lot of mess because we just followed a linear approach, where going back to the previous phase/step is not allowed. The only feasible solution is to restart the web app development right from step one, which is the requirement then integrating client required changes in the requirement.

This shows how the waterfall model is an inefficient, time-consuming and rigid process that is not adaptable to changes into products/software.

Waterfall Model Design

The waterfall approach divides software development into pre-defined phases. The outcome of one phase acts as a sort of input for the next phase.

Advantages of the Waterfall Model

The waterfall model gives a clear, defined set of steps to follow.
It is good for smaller projects where requirements are well-defined with no need for future adjustment.

Disadvantages of the Waterfall Model

Inflexibility: The linear structure of the waterfall model makes it difficult to incorporate a change once a phase is completed as it’s not easy to make changes along the way. Changes to the software would necessitate restarting the software development process.
Delay: Since each phase must be completed before going to the next one, there’s a period of idleness for team members in other phases.

We became very uncomfortable with this time-consuming linear approach, we wanted a system that is adaptable to any change our client desires, where we wouldn't have to start over to implement changes and that led us to the Agile Methodology.

Agile Methodology

Agile methodology is an iterative approach toward the software development life cycle making it more flexible and adaptable to changes.

Unlike the sequential process of waterfall, Agile breaks the entire SDLC into a loop of small series of steps (plan- build - test - deploy) known as sprint. The product is developed bit by bit, feature by feature. At the end of each sprint, review is gotten from client and required changes are integrated at the beginning of the next sprint.

This way, Agile breaches the gap between clients and developers. Now let’s visualize Agile in action with our client's e-commerce app.

Following the Agile method, the e-commerce app is built into small chunks, which go through the development cycle of packaging, testing and deployment known as sprint. At the end of each sprint, feedback is gotten from users which is leveraged to integrate changes into the app (chunks of code). Then the app goes through the sprint again.

This loop process allows us to effectively integrate changes requested by our clients from every feedback into the development lifecycle while also shortening the time it takes for benefits to be delivered to users.

Some benefits of Agile Methodology

Increased flexibility and adaptability.
Faster delivery of software or products.
Improved quality of software.
Customer satisfaction.

Birth of DevOps

Now we’re able to deliver quality software faster but there’s still one issue left unaddressed.

Each team focuses solely on their tasks and sequentially moves the software (web app) to the next phase without any form of collaboration to ensure the software's success in the next phase. The next phase has little visibility of how the software was built in the previous phase. Most of the time, we say that there are silos (barriers) between these teams.

If a bug occurs in production mode, the operations team in charge of deployment and monitoring will blame the testing team for not thoroughly testing the app, while the testing team will point fingers at the developer's team for producing poor-quality code.

We can almost hear the dev teams saying, "We spent a sleepless night reviewing and testing our codes, the operations team is simply not doing their job." From this point, we can see that no team wants to take the blame; this is commonly known as the BLAME GAME.

This blame causes delays in products and the teams forget they’re all responsible for ensuring the product is a success. So, we decided to bring Development and the Operations team (including testing team) together, where these teams can together oversee the process in each phase and collaborate with enhanced communication at every phase in the development cycle. And that was how we birth DevOps.

DevOps bridged the gap between developers and operators. It is an improvement of Agile. It employs continuous integration, continuous delivery, continuous deployment and automated testing to release software more quickly with greater efficiency.

Lifecycle of DevOps and Tools.

The DevOps lifecycle is a set of phases involved in the software development process that incorporates DevOps practices to ensure delivery of high-quality software applications.

Plan: This stage involves planning the software development process. This includes identifying the objectives, requirements and resources needed for the project. Project management tools like Jira, Asana are used in this phase.
Code: In this stage, the development team writes code for the application and makes use of version control tools like git,svn or bitbucket to keep track of changes and manage the codebase.
Build: This stage involves compiling the code from codebases and building the application with the help of build automation tools such as Jenkins, Travis CI, or CircleCI to automate the build process.
Test: In this stage, the application is tested to ensure it meets the requirements, functions correctly and is free of bugs. Automated testing tools such as Selenium, JMeter, and Appium are used to automate the testing process.
Release: In this stage, the application is prepared for release to the production environment. Release automation tools such as Ansible, Puppet, or Chef to automate the deployment process.
Deploy: In this stage, the application is deployed to the production environment. This stage employs containerization and orchestration tools such as Docker, Kubernetes, or Mesos to manage and deploy the application in a scalable and efficient manner.
Operate: This stage focuses on application monitoring and maintenance. Monitoring and logging tools such as Nagios, ELK Stack, or Splunk are used to monitor the application's performance.
Monitor: In this stage, the team analyzes application performance and user feedback to identify areas for improvement. Tools used in this stage include monitoring tools like Prometheus, Grafana, and others.

Benefits of DevOps

Enhanced collaboration between developers and operations team.
Increased efficiency by automating processes and reducing manual tasks.
Greater scalability and availability.
Increased productivity of business.
Lowers the failure rate of new releases.

Conclusion

In this article, you’ve learned about traditional software development practices, their benefits, how we conceived DevOps from traditional practices, lifecycle and benefits of DevOps.

If you enjoyed reading, you could give a heart (up to 10). You can also connect with me on X (formerly twitter). Adios - DrIntech✍️