DEV Community: J. Alexander

Setting Up an S3 Backend for Terraform State Management

J. Alexander — Mon, 30 Dec 2024 03:41:23 +0000

Setting Up an S3 Backend for Terraform State Management

Introduction

Managing Terraform state files in a team environment requires a robust and secure backend solution. AWS S3 provides an excellent option for storing Terraform state files remotely. In this tutorial, we'll create a production-ready S3 backend with versioning and encryption enabled.

Prerequisites

AWS Account
Terraform installed locally
AWS CLI configured with appropriate credentials

Project Structure

Let's start by creating our project directory:

mkdir tf-remote-backend
cd tf-remote-backend

tf-remote-backend/
├── backend.tf
├── main.tf
├── variables.tf
├── outputs.tf
└── README.md

Step 1: Create the Main Configuration

Create

main.tf:

provider "aws" {
  region = "us-east-1"
}

resource "random_string" "bucket_suffix" {
  length  = 8
  special = false
  upper   = false
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-state-backend-${random_string.bucket_suffix.result}"

  lifecycle {
    prevent_destroy = false
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

Step 2: Set Up Variables and Outputs

Create variables.tf:

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

Create outputs.tf:

output "s3_bucket_name" {
  value = aws_s3_bucket.terraform_state.id
}

Step 3: Initialize and Create the S3 Bucket

Run these commands:

terraform init
terraform apply

Step 4: Configure the Backend

After the S3 bucket is created, update backend.tf:

terraform {
  backend "s3" {
    bucket = "your-bucket-name-from-output"
    key    = "terraform.tfstate"
    region = "us-east-1"
    encrypt = true
  }
}

Step 5: Initialize the Backend

terraform init -migrate-state

Verification

Check your S3 bucket:

aws s3 ls s3://your-bucket-name

You should see:

2024-12-12 19:05:09       5474 terraform.tfstate

Key Features Implemented

Unique Bucket Names: Using random suffix
Versioning: Track state file changes
Encryption: Secure state file storage
Standard Storage: Optimal for frequent access

Best Practices

Always enable versioning for state files
Use encryption for sensitive data
Create unique bucket names
Implement proper IAM policies
Regular backup verification

Conclusion
You now have a production-ready Terraform backend using AWS S3. This setup provides:

Secure state file storage
Team collaboration capability
Version history
Encrypted data storage

Resources

Terraform S3 Backend Documentation
AWS S3 Documentation

Automating Python Flask Deployment with Google Cloud Run: A Step-by-Step Guide

J. Alexander — Mon, 30 Dec 2024 03:24:00 +0000

Automating Python Flask Deployment with Google Cloud Run: A Step-by-Step Guide

Introduction

In today’s digital landscape, automating the deployment of web applications is crucial for businesses and developers. Flask, a popular Python web framework, offers a lightweight and flexible solution for creating web applications. When combined with Google Cloud Run and Google Cloud Build, developers can harness the full potential of these technologies to create robust and scalable applications.

In this step-by-step guide, we will walk you through the process of creating a Flask app, containerizing it with Docker, and deploying it on Google Cloud Run using Google Cloud Build and GitHub triggers. You will learn how to set up a CI/CD pipeline, deploy the Flask app, and handle common issues.

Prerequisites

The prerequisites for this project include:

Flask and Python: Familiarity with Flask, a Python web framework, and Python programming language is essential.
Google Cloud Platform (GCP) Account: You will need a GCP account to create a project, access the necessary services, and deploy your Flask app on GCP.
Git and GitHub: Familiarity with the Git version control system and GitHub platform is necessary to clone the project, manage code changes, and set up the CI/CD pipeline using GitHub Actions.
Docker: Knowledge of Docker and containerization concepts is required to containerize your Flask app using Docker and create Docker images for deployment.
Google Cloud Build: To configure the CI/CD pipeline for the deployment of the Flask app, one must be familiar with Google Cloud Build, a workflow automation tool.

It is recommended to have a good understanding of these prerequisites before starting the project to ensure a smooth development and deployment process. If you are not familiar with any of these technologies, it is advisable to refer to the respective documentation and tutorials to gain the necessary knowledge.

Creating the Flask App

First, let's create a simple Flask application. Here's the code for
main.py:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Welcome to Python Flask World v3.0 from cicd'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

This basic Flask app will serve a welcome message at the root URL.

Creating the Dockerfile

Next, we'll create a Dockerfile to containerize our Flask application. Here’s the content of the Dockerfile:

FROM python:3.7-slim
RUN pip install flask
WORKDIR /myapp
COPY main.py /myapp/main.py
CMD ["python", "/myapp/main.py"]

This Dockerfile uses a slim Python image, installs Flask, sets the working directory, copies the main.py file, and specifies the command to run the Flask app.

Building and Pushing the Docker Image

We will use Google Cloud Build to build and push our Docker image. Here’s the cloudbuild.yaml configuration:

steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/YOUR-PROJECT-NAME/runimage', '.']
# Push the container image to Container Registry
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/YOUR-PROJECT-NAME/runimage']
# Deploy container image to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: gcloud
  args: ['run', 'deploy', 'runwithcicd', '--image', 'gcr.io/devops-442409/runimage', '--region', 'us-central1', '--allow-unauthenticated']
images:
- gcr.io/YOUR-PROJECT-NAME/runimage
options:
  logging: CLOUD_LOGGING_ONLY

This configuration file defines the steps to build and push the Docker image to Google Container Registry.

Deploying the App on Google Cloud Run

Finally, we will deploy the Docker image to Google Cloud Run using the following command in our cloudbuild.yaml:

- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: gcloud
  args: ['run', 'deploy', 'runwithcicd', '--image', 'gcr.io/YOUR-PROJECT-NAME/runimage', '--region', 'us-central1', '--allow-unauthenticated']

This command deploys the Docker image to Cloud Run, making it accessible via a public URL.

Setting Up from a GitHub Repo with Cloud Build Triggers

To automate the deployment process, we will set up a CI/CD pipeline using a GitHub repository and Cloud Build triggers.

GitHub Repository Setup:

Step 1: Create a new repository on GitHub or use an existing one.

Example:

Repository name: flask-cloud-run
Description: A simple Flask app deployed using Google Cloud Run

Step 2: Add your Flask application code (main.py) and Dockerfile to the repository.

Example:
flask-cloud-run/
├── main.py
└── Dockerfile

Step 3: Add the cloudbuild.yaml file to the root of the repository.

Example:

flask-cloud-run/
├── main.py
├── Dockerfile
└── cloudbuild.yaml

Cloud Build Trigger Setup:

Step 1: Open the Google Cloud Console and navigate to Cloud Build.
Step 2: Click on "Triggers" in the left-hand menu and then click "Create Trigger."

Step 3: Configure the trigger:

Name: Give your trigger a meaningful name, such as flask-cloud-run-trigger.

Event: Choose the event that will trigger the build. For example, select "Push to a branch."

Source: Select "GitHub" as the source and connect your GitHub repository.

Branch: Specify the branch to monitor, such as main or master.

Step 4: Set up the build configuration:
Build Configuration: Choose "cloudbuild.yaml" as the build configuration file.

Step 5: Save and create the trigger.
Now, whenever you push changes to the specified branch in your GitHub repository, Cloud Build will automatically start the build process.

For a successful Cloud Run deployment via Cloud Build, I granted these key permissions:

Cloud Build service account needed:
roles/run.admin (Cloud Run Admin)
roles/iam.serviceAccountUser (Service Account User)

Critical Permission Added:
Granted iam.serviceAccounts.actAs permission on the Compute Engine default service account

Conclusion

By following these steps, you can automate the deployment of your Python Flask application using Google Cloud Run. This CI/CD pipeline ensures that your application is always up-to-date and easily scalable. Try it out and let me know how it works for you!

Cloud Resume Challenge: Hosting a React CV using S3, CloudFront, Route 53, Lambda, DynamoDB and GitHub actions for CI/CD

J. Alexander — Mon, 09 Sep 2024 12:14:48 +0000

Introduction

I will be attempting to complete the Cloud Resume Challenge. By doing the Cloud Resume challenge, I was inspired to take and pass the AWS Cloud Practitioner certification. Shout out to Forrest Brazeal for coming up with such a fantastic project and book for those looking to enter the cloud IT field.
Through this, I'll walk you through the process I used to deploy my React CV resume website using AWS services and GitHub Actions for continuous integration and deployment (CI/CD). I will cover the setup of AWS Organizations, AWS S3, CloudFront, Route 53,Lambda, DynamoDB, GitHub Actions and Terraform.
Since completing the Cloud Resume challenge, I have passed the AWS Cloud Practitioner and AWS Certified Solutions Architect exams.

Step 1: Setting Up Your AWS Account, Organization, IAM

Overview of AWS Accounts and IAM

AWS accounts are typically managed through AWS Organizations. AWS (Amazon Web Services) provides a secure and scalable infrastructure for deploying web applications. An AWS account is necessary to access and manage AWS services. AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely.

First, sign up for an AWS account if you don't have one. After logging into the root user account, navigate to the IAM console. Enable IAM from the console and create a new IAM user by setting up AWS Organizations in the root account. Create 2 Organizational Units (OUs) to hold the production and test account. ![Screenshot of AWS IAM console showing the creation of a new IAM user]Next, choose the Identity Source and create a new user in the Identity Source, entering your username and credentials to receive a one-time password for signing into the AWS access portal. Copy your sign-in credentials into notepad. Then, create an Admin Permission Set using the predefined AdministratorAccess. Set up account access for the Identity Center Admin User by going to AWS accounts, giving administrative access to your Organizational Units, and adding the user created in step 3 with Administrator Access to the user in the Test account of the Organization. Finally, sign into the AWS Portal using your username and one-time password, then set the new password. You can also perform command line or programmatic access using short-term access keys.

Step 2: Create the React CV website and deploy it to GitHub

I purchased a custom domain name from Namecheap and transferred the name server to Route 53. First, I created an S3 bucket named to match my custom domain, using the subdomain WWW. To host my website in the S3 bucket, I disabled Block all public access. After building the React project with npm run build, I uploaded the static files to the bucket via the AWS Console. I enabled static website hosting and set index.html as the index document, then updated the bucket policy to allow public read access by adding a policy for PublicReadGetObject.

For setting up a custom domain, I created a CNAME record in my domain hosting provider, pointing to the S3 bucket endpoint. Next, I deployed the React app to CloudFront by creating a new distribution, using the S3 bucket endpoint as the origin, enabling automatic object compression, and setting up custom domain names with HTTPS.

To use the root domain, I transferred the domain to Route 53, created a Route 53 public-hosted zone, updated the DNS name servers, and created DNS records for both the www and root domains using Aliases.

Finally, I set up a CI/CD pipeline using GitHub Actions. This automated the deployment process by installing dependencies, running unit tests, uploading files to S3, and invalidating the CloudFront cache. I created an IAM user with a policy granting the necessary permissions, added AWS credentials to GitHub secrets, and configured a GitHub Actions workflow to deploy the app whenever changes are pushed to the master branch.

name: Build and Deploy React App to CloudFront
on:
  push:
    branches: [ master ]
jobs:
  build-and-deploy:
    name: Build and Deploy
    runs-on: ubuntu-latest
    env:
      BUCKET: <bucket-name>
      DIST: build
      REGION: us-east-1
      DIST_ID: <ID>

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.REGION }}

    # - uses: actions/setup-node@v2
    #   with:
    #     node-version: '14'

    - name: Install Dependencies
      run: |
        node --version
        npm ci --production

    - name: Build Static Website
      run: npm run build

    - name: Copy files to the production website with the AWS CLI
      run: |
        aws s3 sync --delete ${{ env.DIST }} s3://${{ env.BUCKET }}

    - name: Copy files to the production website with the AWS CLI
      run: |
        aws cloudfront create-invalidation \
          --distribution-id ${{ env.DIST_ID }} \
          --paths "/*"

Connecting RetroHitCounter to DynamoDB Using AWS Lambda and API Gateway

To connect the RetroHitCounter to a DynamoDB database using an AWS Lambda function and API Gateway, start by setting up a DynamoDB table named 'HitCounter' with a primary key of type String. Then, create a Lambda function in Python to interact with the DynamoDB table, incrementing the hit count each time it is invoked. Ensure the Lambda function has the necessary IAM role permissions to read and write to the DynamoDB table. Next, create an API Gateway, setting up a new REST API with a resource (e.g., /hits) and a GET method that triggers the Lambda function. Enable CORS for the GET method to allow cross-origin requests. Finally, update your React application to fetch the hit counter value from the API Gateway endpoint and display it using RetroHitCounter. This involves fetching the hit count in a React component's useEffect hook and integrating the RetroHitCounter component to show the hit count.

Code Snippets

Lambda Function in Python:

import json
import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('HitCounter')

def lambda_handler(event, context):
    response = table.get_item(Key={'id': 'counterId'})

    if 'Item' in response:
        hits = response['Item']['hits']
    else:
        hits = 0

    hits += 1

    table.put_item(
        Item={
            'id': 'counterId',
            'hits': hits
        }
    )

    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*' 
        },
        'body': json.dumps({'hits': hits})
    }

IAM Role Policy:

{
    "Effect": "Allow",
    "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
    ],
    "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/HitCounter"
}

React Component:

import React, { useState, useEffect } from 'react';
import RetroHitCounter from 'react-retro-hit-counter';
import jelani from '../assets/img/jelaniprofilepic.jpeg';
import { Link, animateScroll as scroll } from 'react-scroll';
import { Navbar, Nav } from 'react-bootstrap';
import { Container } from 'react-bootstrap';

const Navigate = (props) => {
    const [hits, setHits] = useState(0);

    useEffect(() => {
        fetch('https://YOUR_API_GATEWAY_URL/hits')
            .then(response => response.json())
            .then(data => setHits(data.hits))
            .catch(error => console.error('Error fetching hits:', error));
    }, []);

    const titles = props.titles.map((title, index) => (
        <Container style={{ paddingRight: 0, marginRight: 0 }} className="nav-item justify-content-end" key={index}>
            <Nav.Link
                as={Link}
                eventKey={index}
                activeClass="active"
                style={{ cursor: "pointer" }}
                className="nav-link"
                spy={true}
                to={props.hrefs[index]}
                smooth={"easeInOutQuart"}
                duration={1500}
            >
                {title}
            </Nav.Link>
        </Container>
    ));

    return (
        <Navbar className="navbar navbar-expand-lg navbar-dark bg-primary fixed-top" id="sideNav" collapseOnSelect expand="lg" bg="dark" variant="dark">
            <button className="navbar-brand" onClick={() => scroll.scrollToTop()} style={{ background: 'none', border: 'none', padding: 0 }}>
                <span className="d-none d-lg-block">
                    <img className="img-fluid img-profile rounded-circle mx-auto mb-2" style={{ cursor: "pointer" }} src={jelani} alt="" />
                </span>
            </button>
            <Navbar.Toggle aria-controls="responsive-navbar-nav" />
            <Navbar.Collapse id="responsive-navbar-nav">
                <Nav className="mr-auto">
                    {titles}
                    <RetroHitCounter
                        hits={hits}
                        withBorder={true}
                        withGlow={false}
                        minLength={4}
                        size={40}
                        padding={4}
                        digitSpacing={3}
                        segmentThickness={4}
                        segmentSpacing={0.5}
                        segmentActiveColor="#76FF03"
                        segmentInactiveColor="#315324"
                        backgroundColor="#222222"
                        borderThickness={7}
                        glowStrength={0.5}
                    />
                </Nav>
            </Navbar.Collapse>
        </Navbar>
    );
}

export default Navigate;

Make sure to replace https://YOUR_API_GATEWAY_URL/hits with your actual API Gateway endpoint URL. This setup will fetch the hit counter value from DynamoDB through the Lambda function and display it using RetroHitCounter.

Step 3: Setting Up AWS Lambda Function with Terraform

Now, let's dive into deploying your backend Lambda function using Terraform. This approach allows you to define your infrastructure as code, ensuring consistency and reproducibility across environments.

Terraform Code for AWS Lambda

Create a new Terraform file, let's call it lambda.tf, and include the following code to define your Lambda function:

# lambda.tf

# Define the AWS Lambda function

resource "aws_lambda_function" "hitcounter" {
  filename         = "${path.module}/lambda/func.zip"
  function_name    = "myfunction"
  role             = aws_iam_role.lambda_exec_role.arn
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.12"
  memory_size      = 128
  timeout          = 3

  environment {
    variables = {
      ENV_VAR_KEY = "value"
    }
  }

  tracing_config {
    mode = "Active"
  }

  lifecycle {
    create_before_destroy = true
  }
}

# Define IAM role for Lambda execution
resource "aws_iam_role" "lambda_exec_role" {
  name               = "lambda_exec_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Action    = "sts:AssumeRole"
      }
    ]
  })
}

# Define IAM policy for Lambda function permissions
resource "aws_iam_policy" "lambda_policy" {
  name        = "lambda_policy"
  description = "Policy for Lambda function"

  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = "logs:*"
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = [
          "dynamodb:GetItem",
          "dynamodb:PutItem"
        ]
        Resource = "arn:aws:dynamodb:us-west-2:123456789012:table/HitCounter"
      },
      {
        Effect   = "Allow"
        Action   = "lambda:*"
        Resource = "*"
      }
    ]
  })
}

# Attach IAM policy to Lambda execution role
resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role       = aws_iam_role.lambda_exec_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn
}

# Archive Lambda function code
data "archive_file" "func" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/"  # Replace with the directory containing your Lambda function code
  output_path = "${path.module}/lambda/func.zip"
}

Deploying with Terraform

*Make sure Terraform is installed and configured with AWS credentials.
*Place your Lambda function code in the lambda/ directory relative to your Terraform configuration.
*Initialize Terraform (terraform init) and apply the configuration (terraform apply) to deploy your Lambda function.

Step 4: Setting Up CI/CD with GitHub Actions
To automate deployment and ensure continuous integration with your backend Lambda function, integrate GitHub Actions with your repository.

GitHub Actions Workflow
Update your existing GitHub Actions workflow (from Step 2) to include deployment steps for your Lambda function. Here's an example snippet to add to your workflow file:

# .github/workflows/main.yml

name: Build and Deploy Backend Lambda

on:
  push:
    branches:
      - master

jobs:
  build-and-deploy:
    name: Build and Deploy
    runs-on: ubuntu-latest
    env:
      BUCKET: <bucket-name>
      DIST: build
      REGION: us-east-1
      DIST_ID: <CloudFront Distribution ID>
      LAMBDA_FUNCTION_NAME: "myfunction"

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.REGION }}

      - name: Install Dependencies
        run: |
          npm ci --production

      - name: Build Lambda Function ZIP
        run: |
          cd lambda/
          zip -r func.zip .

      - name: Deploy Lambda Function with Terraform
        run: |
          terraform init
          terraform apply -auto-approve

      - name: Invalidate CloudFront Cache
        run: |
          aws cloudfront create-invalidation \
            --distribution-id ${{ env.DIST_ID }} \
            --paths "/*"

Conclusion
By using Terraform for infrastructure as code and GitHub Actions for CI/CD, you've automated the deployment of your backend Lambda function, ensuring efficient development and deployment workflows for your Cloud Resume Challenge project.

DEV Community: J. Alexander

Setting Up an S3 Backend for Terraform State Management

Setting Up an S3 Backend for Terraform State Management

Introduction

Prerequisites

Project Structure

Step 1: Create the Main Configuration

Step 2: Set Up Variables and Outputs

Step 3: Initialize and Create the S3 Bucket

Step 4: Configure the Backend

Step 5: Initialize the Backend

Verification

Automating Python Flask Deployment with Google Cloud Run: A Step-by-Step Guide

Automating Python Flask Deployment with Google Cloud Run: A Step-by-Step Guide

Introduction

Table of Contents:

Prerequisites

Creating the Flask App

Creating the Dockerfile

Building and Pushing the Docker Image

Deploying the App on Google Cloud Run

Setting Up from a GitHub Repo with Cloud Build Triggers

GitHub Repository Setup:

Example:

Cloud Build Trigger Setup:

For a successful Cloud Run deployment via Cloud Build, I granted these key permissions:

Conclusion

Cloud Resume Challenge: Hosting a React CV using S3, CloudFront, Route 53, Lambda, DynamoDB and GitHub actions for CI/CD

Introduction

Step 1: Setting Up Your AWS Account, Organization, IAM

Overview of AWS Accounts and IAM

Step 2: Create the React CV website and deploy it to GitHub

Connecting RetroHitCounter to DynamoDB Using AWS Lambda and API Gateway

Code Snippets

Step 3: Setting Up AWS Lambda Function with Terraform

Terraform Code for AWS Lambda

Deploying with Terraform