DEV Community: Ekemini Udongwo

Implementing an On-Demand Video Streaming Platform on AWS Using Terraform

Ekemini Udongwo — Tue, 29 Oct 2024 08:06:25 +0000

In this project, we will deploy an on-demand video streaming platform utilizing AWS services and infrastructure automation with Terraform. The solution will incorporate AWS Route 53 for DNS management, CloudFront as a Content Delivery Network (CDN) to optimize content delivery, AWS Certificate Manager for SSL certificates, and Amazon S3 for video file storage. The following diagram outlines the infrastructure architecture that will be implemented.

Prerequisites:

Domain name
Knowledge of Terraform

Note: Before we move forward, make sure you have Terraform set up on your local machine, including the necessary provider, and that you’ve initialized your project. If you’re all set, let’s get started!

1. Register a domain on route53

First, log into the AWS Management Console and head over to the Route 53 service. From there, find the “Registered Domains” section and click on “Register Domain.”
Type in the domain name you want and check its availability. If it’s available, just fill in your details, and AWS will email you a verification link. Once verified, your new domain should be ready within a few minutes to an hour.

2. Create some variables

Let's create a vars.tf file and add some variables to it which will be used by other resources.

vars.tf

variable "domain" {
  type    = string
  default = "your-domain.com"
}

variable "cloudfront_priceClass" {
  type = string
  default = "PriceClass_100"
}

variable "environment" {
  type = string
  default = "dev"
}

3. Create the S3 bucket

Next, we need to create an S3 bucket to store our video or videos we wanna serve to our users. In your text editor, create a S3.tf file where our S3 configuration is going to be written.

# Create an S3 bucket
resource "aws_s3_bucket" "video_bucket" {
  bucket = "kemzzy-new-video-bucket"

  tags = {
    Name = "Video bucket"
  }
}

# Enable versioning
resource "aws_s3_bucket_versioning" "video_bucket_versioning" {
  bucket = aws_s3_bucket.video_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

This creates our bucket and also enables versioning on it.

4. Upload a video to the S3 bucket

Next we upload our video to the created S3 bucket

# replace with the path to the video you want to upload
resource "aws_s3_object" "new_video" {
  bucket       = aws_s3_bucket.video_bucket.id
  key          = "awesome-video"
  source       = "path-to-your-video-file"
  etag         = filemd5("path-to-your-video-file")
  content_type = "video/mp4"

  depends_on = [aws_s3_bucket.video_bucket]
}

make sure to upload a mp4 video format otherwise you might have an issue (Speaking from personal experience )

5. Create and verify a SSL certificate with ACM

In other to use our domain with cloudfront we need a domain with SSL enabled. You can use the following terraform config to create and validate your SSL certificate for your domain.

 Create an ACM certificate
resource "aws_acm_certificate" "video_cert" {
  domain_name               = var.domain
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

# Cert validation
resource "aws_acm_certificate_validation" "validate_cert" {
  certificate_arn         = aws_acm_certificate.video_cert.arn
  validation_record_fqdns = [for record in aws_route53_record.certificate_validation : record.fqdn]

  depends_on = [ aws_route53_record.certificate_validation ]
}

# create a record to use for validation
resource "aws_route53_record" "certificate_validation" {
  for_each = {
    for dvo in aws_acm_certificate.video_cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.primary.zone_id

  depends_on = [ aws_acm_certificate.video_cert ]
}

6. Create a CloudFront origin access identity

Now we can proceed to create our OAI for our cloudfront distribution. Create another file called main.tf

resource "aws_cloudfront_origin_access_identity" "cloudfront_OAI" {
  comment = "OAI for video streaming S3 bucket"
}

Great, now that we’ve tackled that part, let’s move on to creating our CloudFront distribution!

resource "aws_cloudfront_distribution" "s3_distribution" {
  origin {
    domain_name = aws_s3_bucket.video_bucket.bucket_regional_domain_name
    origin_id   = aws_s3_bucket.video_bucket.id

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.cloudfront_OAI.cloudfront_access_identity_path
    }
  }

  enabled         = true
  is_ipv6_enabled = true
  comment         = "play video"

  aliases = [var.subdomain]


  default_cache_behavior {
    allowed_methods  = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = aws_s3_bucket.video_bucket.id

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "allow-all"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  }

  price_class =var.cloudfront_priceClass

  restrictions {
    geo_restriction {
      restriction_type = "none"
      locations        = []
    }
  }

  tags = {
    Environment = var.environment
  }

  viewer_certificate {
    acm_certificate_arn = data.aws_acm_certificate.cert.arn
    ssl_support_method  = "sni-only"

  }
}

6. Access the video through the CloudFront distribution

To set this up, we’ll create a bucket policy that restricts read access exclusively to our CloudFront service. This setup adds a layer of security by making sure users can only access content through CloudFront, not directly from the bucket. Simply add this policy to your *s3.tf * file.

# Bucket policy to allow read access from CloudFront and prevent direct access to our bucket
resource "aws_s3_bucket_policy" "video_stream" {
  bucket = aws_s3_bucket.video_bucket.id
  policy = <<POLICY
  {
    "Version": "2012-10-17",
    "Statement": [
         {
            "Sid": "AllowLegacyOAIReadOnly",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${aws_cloudfront_origin_access_identity.cloudfront_OAI.id}"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::${aws_s3_bucket.video_bucket.bucket}/*"
        }
      ]
  }
  POLICY
}

7. Create an outputs file to get your outputs (optional)

Create an output.tf file and add these outputs

output "cloudfront_url" {
  value = aws_cloudfront_distribution.s3_distribution.domain_name
}

output "domain_url" {
  value = aws_route53_record.video_subdomain.name
}

8. Deploy your infrastructure

Phew! That was a journey, but you’ve made it to the finish line. Now, it’s time to bring your project to life! Run terraform validate to check for any issues, then use terraform plan -out="tfplan" to generate a plan, and finally, terraform apply "tfplan" to deploy your infrastructure to AWS.

Now, to watch the video, just use your domain name, like your-domain.com/awesome-video, or the CloudFront URL provided in the output. Enjoy!

Conclusion

Congratulations on completing this tutorial. Make sure your you run terraform destroy when you are done to avoid accumulating any bills.

Automating User and Group Creation with a Bash Script

Ekemini Udongwo — Mon, 01 Jul 2024 14:42:38 +0000

As a DevOps engineer, it is quite a common task to create, update and delete users and groups. Automating this process can save time and reduce errors, especially when onboarding multiple new developers. In this article, we'll walk through a Bash script designed to automate the creation of users and their respective groups, set up home directories, generate random passwords, and log all actions.

Objectives

Create users and their personal groups.
Add users to specified groups.
Set up home directories with appropriate permissions.
Generate and securely store random passwords.
Log all actions for auditing purposes.

Requirements

Input File: A text file containing usernames and groups, formatted as username;group1,group2.
Log File: A log file to record all actions.
Password File: A file to securely store generated passwords.

Let's Begin

First and foremost, at the beginning of any shell script we're writing, we start the first line with a line called a "shebang", sounds catchy right?


#!/bin/bash

The presence of a shebang indicates that a file is executable.

Now that that is out of the way we can move on to the juicy details.

Script Initialization

The script starts by defining the locations of the log file and the password file(you can name these file whatever you want, ):

LOGFILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.csv"

Check File Input

The script checks if an input file is provided as an argument:

if [ -z "$1" ]; then
  echo "Usage: $0 <name-of-text-file>"
  exit 1
fi

If no file is provided, the script exits with a usage message, telling our users how to use our script.

Create Log and Password Files

This code snippet creates the necessary directories and files, setting appropriate permissions:

mkdir -p /var/secure
touch $LOGFILE $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

We make a directory called "/var/secure/" which will keep our passwords. After that we create the two files that were defined above for logging and saving passwords. Modifying our $PASSWORD_FILE with chmod 600 will ensure that only the user with appropriate permissions can view it(which happens to be the current user we are logged in as).

Generate Random Password Function

We're gonna write a function to generate some random and secure passwords for our different users.

generate_random_password() {
    local length=${1:-10} # Default length is 10 if no argument is provided
    tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c $length
}

The function takes an argument which is how long you want the passwords to be, if none is passed, it defaults to 10.

tr -dc 'A-Za-z0-9!?%+=': This command is used to translate and delete characters that are not in the regex A-Za-z0-9!?%+=
< /dev/urandom: uses the Linux kernel's random number generator and passes the result to the command above.
| head -c $length: outputs the length of the random string specified.

Logging Function

log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> $LOGFILE
}

Takes an argument and logs the date and time of the message in the $LOGFILE

Create User Function

The create_user() function handles the creation of users and their groups. It takes two arguments: username and groups.

create_user() {
  local username=$1
  local groups=$2

  if getent passwd "$username" > /dev/null; then
    log_message "User $username already exists"
  else
    useradd -m $username
    log_message "Created user $username"
  fi

  # Add user to specified groupsgroup
  groups_array=($(echo $groups | tr "," "\n"))

  for group in "${groups_array[@]}"; do
    if ! getent group "$group" >/dev/null; then
      groupadd "$group"
      log_message "Created group $group"     
    fi
    usermod -aG "$group" "$username"
    log_message "Added user $username to group $group"
  done

  # Set up home directory permissions
  chmod 700 /home/$username
  chown $username:$username /home/$username
  log_message "Set up home directory for user $username" 

  # Generate a random password
  password=$(generate_random_password 12) 
  echo "$username:$password" | chpasswd
  echo "$username,$password" >> $PASSWORD_FILE
  log_message "Set password for user $username"
}

Let's break it down.....

Checking if User Already Exists

if id "$username" > /dev/null; then
  log_message "User $username already exists"
else
  useradd -m $username
  log_message "Created user $username"
fi

If a user exists our script logs it and moves on else the user is created and then logged.

NOTE: > /dev/null is passing the response to null so that it doesn't interfere with our logs.

NOTE: We are not going to create a personal group for each user because unix does this for us automatically when we create a new user

Adding User to Specified Groups

We then proceed to add the users to their different groups

groups_array=($(echo $groups | tr "," "\n"))
for group in "${groups_array[@]}"; do
  if ! getent group "$group" > /dev/null; then
    groupadd "$group"
    log_message "Created group $group"   
  fi
  usermod -aG "$group" "$username"
  log_message "Added user $username to group $group"
done

The code snippet splits the groups string by commas and saves them in a groups_array variable. It then loops through each group and adds the user to it, also making sure to create the group if it doesn't exist:

Setting Up Home Directory Permissions

chmod 700 /home/$username
chown $username:$username /home/$username
log_message "Set up home directory for user $username"

chmod 700 /home/$username:: Sets the home directory permissions so only the user has full access (read, write, execute).

chown $username:$username /home/$username: Changes the ownership of the home directory to the specified user and their group.

echo "Set up home directory for user $username" | tee -a $LOGFILE: Logs a message indicating the home directory setup to a specified log file.

Assigning A Random Password To Each User

password=$(generate_random_password 12) 
echo "$username:$password" | chpasswd
echo "$username,$password" >> $PASSWORD_FILE
log_message "Set password for user $username"

This snippet uses the function we wrote earlier to create a password, update the user's password to the generated password, saves the username and the corresponding password to the $PASSWORD_FILE and finally logs a message indicating it has been successfully changed.

Reading the Input File

The script reads the input file line by line, calling the create_user() function for each line and passing the $username and $groups as arguments:

while IFS=';' read -r username groups; do
  create_user "$username" "$groups"
done < "$1"

Now Put It All Together...

#!/bin/bash

# Log file location
LOGFILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.csv"

# Check if the input file is provided
if [ -z "$1" ]; then
  echo "Error: No file was provided"
  echo "Usage: $0 <name-of-text-file>"
  exit 1
fi

# Create log and password files
mkdir -p /var/secure
touch $LOGFILE $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

generate_random_password() {
    local length=${1:-10} # Default length is 10 if no argument is provided
    LC_ALL=C tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c $length
}

# Function to create a user
create_user() {
  local username=$1
  local groups=$2

  if getent passwd "$username" > /dev/null; then
    echo "User $username already exists" | tee -a $LOGFILE
  else
    useradd -m $username
    echo "Created user $username" | tee -a $LOGFILE
  fi

  # Add user to specified groupsgroup
  groups_array=($(echo $groups | tr "," "\n"))

  for group in "${groups_array[@]}"; do
    if ! getent group "$group" >/dev/null; then
      groupadd "$group"
      echo "Created group $group" | tee -a $LOGFILE      
    fi
    usermod -aG "$group" "$username"
    echo "Added user $username to group $group" | tee -a $LOGFILE
  done

  # Set up home directory permissions
  chmod 700 /home/$username
  chown $username:$username /home/$username
  echo "Set up home directory for user $username" | tee -a $LOGFILE

  # Generate a random password
  password=$(generate_random_password 12) 
  echo "$username:$password" | chpasswd
  echo "$username,$password" >> $PASSWORD_FILE
  echo "Set password for user $username" | tee -a $LOGFILE
}

# Read the input file and create users
while IFS=';' read -r username groups; do
  create_user "$username" "$groups"
done < "$1"

echo "User creation process completed." | tee -a $LOGFILE

Conclusion

This script provides an efficient way to manage user and group creation, ensuring that all necessary steps are handled securely and logged for auditing. By automating these tasks, SysOps engineers can save time and reduce the risk of errors during user onboarding.

To learn more and kickstart your programming journey you can visit:
https://hng.tech/internship or https://hng.tech/premium

Feel free to reach out with questions or suggestions for improvements. Happy automating!