DEV Community: Irlan Dos Santos

Visualize Data with AWS QuickSight

Irlan Dos Santos — Thu, 19 Dec 2024 17:28:30 +0000

Overview

Amazon QuickSight empowers users to analyze datasets and create insightful visualizations with ease. This guide walks you through the process of analyzing Netflix's extensive dataset of shows and movies to create a dynamic dashboard brimming with actionable insights.

Project Objectives

Here’s the roadmap to transform raw data into a polished dashboard:

Upload the Dataset: Store data in an Amazon S3 bucket.
Create a QuickSight Account: Set up and access Amazon QuickSight.
Connect Data: Link the S3-hosted dataset with QuickSight.
Visualize Data: Build graphs, charts, and perform analyses.
Publish Dashboard: Share your findings via a professionally styled dashboard.

Instructions

Dataset Preparation

Download Required Files:
- netflix_titles.csv: Contains information about Netflix's titles.
- manifest.json: A configuration file to guide QuickSight in reading the dataset.

Store Dataset in Amazon S3

Log in to your AWS Management Console.
Navigate to the S3 service and create a new bucket:
- Name the bucket (e.g., quicksight-project-dataset).
- Choose a region near you for optimal performance.
Upload netflix_titles.csv.
Copy the S3 URL of the uploaded dataset for later use.

Open the manifest.json file and replace the placeholder URL with the Amazon S3 URL of your dataset.

Set Up Amazon QuickSight

Search for Amazon QuickSight in your AWS console.

Sign up for the Enterprise Free Trial (ensure optional features like paginated reports are unchecked to avoid charges).

Enter your details for your QuickSight account. Make sure the email you use matches your AWS account email to avoid errors.

Select Select S3 buckets and look for the bucket you created earlier.

Make sure you UNCHECK ANY OPTIONAL ADD-ON options.

Stop and double-check:
Did you uncheck "ANY OPTIONAL ADD-ON options" checkbox? Make sure to do this so your AWS account doesn't get charged!

Click Finish.

Connect S3 Bucket to QuickSight

Navigate to the Datasets section in QuickSight and select New Dataset.

Choose S3 as the data source and enter a name (e.g., Netflix_Data).
Copy the S3 URL of manifest.json, paste it into the required field, and connect.

You will see this screen; click Visualize.

On this screen, click X.

Any Errors?

No stress! Errors can happen, especially when you're connecting your dataset with QuickSight. Here are the top errors and solutions:

Common Errors and Fixes

QuickSight doesn't have permissions to access the database
- Cause: QuickSight might not be connected to your S3 bucket.
- Solution:
  1. Select the profile icon on the top-right of the QuickSight dashboard.
  2. Go to Manage QuickSight and then Security & permissions.
  3. Under QuickSight access to AWS services, select Manage and add your S3 bucket.
Manifest file cannot be parsed as valid JSON
- Cause: The manifest.json file is incorrectly formatted or doesn't match the S3 region.
- Solution:
  1. Ensure your QuickSight account is in the same region as your S3 bucket.
  2. Check your S3 bucket's region in the AWS Management Console.
  3. Compare it with the region displayed in your QuickSight dashboard (profile icon > region details).
  4. Verify that manifest.json is formatted correctly:
  5. Ensure no brackets are missing.
  6. Confirm the S3 bucket URI in the file is accurate.
Unable to connect to the S3 bucket
- Cause: The S3 bucket may not be associated with your AWS account, or your QuickSight email may not match.
- Solution:
  - Ensure your QuickSight email matches your AWS account email.
  - If they differ, delete your QuickSight account and recreate it using the same email as your AWS account.
Not enough SPICE capacity
- Cause: SPICE (QuickSight's in-memory engine) capacity is region-specific.
- Solution:
  - Use the same region where you created your QuickSight account.
  - Free SPICE capacity is only available in the region where QuickSight was initially set up.

Tips

Double-check JSON formatting before uploading.
Always ensure QuickSight and S3 regions match.
Regularly verify permissions for AWS services in Manage QuickSight.

By addressing these common issues, you'll be well-prepared to connect your dataset successfully!

Create Your Visualizations

Some Explanation about the Fields:

Data: The head of a column, where information is cataloged.
Visuals: The type of graphs you can choose.
Add Data, Y-Axis Value, Group/Color: Attributes you can edit to adjust how data appears in graphs.

Explore the dataset and view available fields. -Just play combining the DATA groups, and change the type of graphs:

Click on release_year to the Y-axis to visualize title releases over time.
Experiment with chart types like bar charts, pie charts, and donut charts.
Group data by categories like type (TV shows vs. movies) for deeper insights.
Resize and rearrange charts for clarity and aesthetics.

Advanced Analysis

Create a horizontal 100% stacked bar chart to show proportions of TV shows vs. movies by release year.
Present the number of titles per year in a detailed table.
Identify the date with the highest number of new titles (e.g., January 1, 2020).
Filter for categories like Action & Adventure, TV Comedies, and Thrillers.
Filter to show titles released in or after 2015 for selected genres.

Polishing Your Dashboard

Add clear and descriptive titles to all charts:
- "Titles by Release Year"
- "Proportion of Movies vs. TV Shows"
- "Genre Distribution Over Time"
Publish the dashboard and export it as a PDF for sharing.

Wrap-Up

Congratulations! You’ve created a comprehensive QuickSight dashboard. Here’s a summary of what you achieved:

Uploaded data to Amazon S3.
Connected S3 to QuickSight.
Built visualizations that uncover meaningful trends.
Published and exported a polished dashboard.

Don’t forget to clean up your AWS resources to avoid unnecessary charges:

Delete your QuickSight account. -On the dashboard, go to top right corner, click on the user icon, and then Manage QuickSight.

Select Account Settings from the left hand navigation panel, and then Manage at the bottom of the page.

Empty and delete your S3 bucket. -If you prefer, you can use the cloudshell and run the command:


bash
aws s3 rb s3://YourBucketName --force

Good Job!!!

Host a Website on Amazon S3

Irlan Dos Santos — Thu, 19 Dec 2024 15:28:57 +0000

Host a Website on Amazon S3

Irlan Dos Santos

Introducing Today's Project!

What is Amazon S3?

Amazon S3 (Simple Storage Service) is a highly scalable and durable cloud storage solution offered by AWS. While its primary purpose is to store objects (files) and their metadata, it also includes features like static website hosting, making it a versatile tool for developers.

How does Amazon S3 work for hosting?

Static website hosting with Amazon S3 is perfect for websites that don’t require server-side scripting (e.g., PHP). S3 buckets serve as containers for your website’s files, delivering content over the internet with low latency and high availability.

One thing I didn't expect in this project was:

Since I was already familiar with the steps, there were no surprises during the process. This project helped solidify my knowledge.

This project took me:

Approximately 20 minutes from start to finish, thanks to the user-friendly AWS interface.

How I Set Up an S3 Bucket

Steps to Create the S3 Bucket:

Duration: The bucket creation process was quick and straightforward, taking only 1 minute.
Region Selection: I selected us-east-1 (Northern Virginia) for its reliability, proximity, and lower cost.
Naming: S3 bucket names must be globally unique. For this project, I used a descriptive name: nextwork-website-project-irlan.

Why is the S3 bucket important?

The bucket serves as the root directory for your website files. It ensures proper organization, access control, and efficient content delivery.

Upload Website Files to S3

What Files Were Uploaded?

index.html: The main HTML file that structures and displays the website’s content.
Image assets: A .zip file containing all visual elements, such as logos, banners, and images, to enhance the website’s look and feel.

Why are these files critical?

The index.html file acts as the website's entry point, ensuring users land on a functioning homepage. Meanwhile, images improve user experience and engagement by adding visual appeal.

How were the files uploaded?

Navigated to the Objects tab in the S3 bucket dashboard.
Used the Upload button to add the required files.
Ensured files were set to the Standard Storage Class, which balances cost and performance for static websites.

Static Website Hosting on S3

Definition:

Website hosting refers to the process of making your website accessible to users on the internet. Amazon S3 simplifies this by enabling static website hosting directly from an S3 bucket.

Steps to Enable Static Website Hosting:

Go to the bucket’s Properties tab.
Scroll to the Static Website Hosting section and click "Edit."
Select "Enable" to activate hosting.
Specify index.html as the Index Document (the homepage users see).
Optionally, add error.html for handling 404 errors.

Why are permissions (ACL) important?

Access Control Lists (ACLs) dictate who can access the files in the bucket. Public access must be explicitly enabled for website hosting to function properly.

Bucket Endpoints

What is a Bucket Endpoint?

Once hosting is enabled, Amazon S3 generates a unique URL for your website. This endpoint allows users to access your website via their browser.

Example Endpoint URL:

http://nextwork-website-project-irlan.s3-website-us-east-1.amazonaws.com/

Initial Error Encountered:

When I first visited the bucket endpoint, I encountered a 403 Forbidden - Access Denied error. This occurred because the files were not publicly accessible, which is a common oversight during initial setup.

Success!

How I Resolved the Access Error:

To make the files accessible:

Navigated to the Objects tab in the bucket.
Selected all files and clicked Actions > Make Public (using ACL).
Confirmed the changes.
Refreshed the browser, and the website was successfully displayed.

Result:

The website is now live and accessible at the endpoint URL, showcasing a functional static website hosted on Amazon S3.

Using Terraformer to Import and Manage VMware Infrastructure with Terraform and AWS Backup Strategy

Irlan Dos Santos — Thu, 19 Dec 2024 13:46:52 +0000

Organizations with hybrid cloud infrastructures often face challenges in managing on-premises resources and integrating them with cloud solutions for backup and recovery. This article outlines a strategy to use Terraformer for importing VMware infrastructure, manage it with the Terraform vSphere provider, and integrate AWS as a reliable backup and restore solution. This approach ensures efficient infrastructure management and disaster recovery capabilities.

Hybrid Infrastructure Overview

On-Premises Infrastructure

The local network infrastructure described consists of a dedicated VMware ESXi host with six virtual machines (VMs) configured to provide essential network and IT services for the company. Key components include:

Active Directory (AD) Server: Domain controller managing user authentication, DHCP, and DNS services.
Backup Server: Utilizing Veeam Backup & Replication for local and cloud-based backups.
File Server: Centralized storage for shared files and documents.
Terminal Server (TS): Hosts the ERP system for remote access by users.
Firebird Database Server: Supports the ERP system and accounting applications.
PFsense Firewall: Provides network security and VPN capabilities for remote access.

Cloud Integration via AWS

AWS serves as the cloud-based backup solution for redundancy and disaster recovery. The strategy involves storing Veeam backups in Amazon S3 and utilizing AWS Glacier for long-term archival.

Step 1: Use Terraformer to Import VMware Resources

1.1 Install and Configure Terraformer

Download Terraformer: Install Terraformer from its GitHub repository.
Set Environment Variables:

   export VSPHERE_USER="administrator@vsphere.local"
   export VSPHERE_PASSWORD="your_password"
   export VSPHERE_SERVER="vcenter.example.com"

1.2 Run Terraformer Import

Run the following command to import existing VMware resources:

terraformer import vsphere \
  --resources=vm,datastore,network \
  --connect=vcenter.example.com \
  --user=administrator@vsphere.local \
  --password=your_password \
  --path-output="./output"

--resources: Specifies the resources to import (e.g., VMs, datastores, networks).
--path-output: Directory to store generated Terraform configuration files.

1.3 Review Generated Files

Terraformer generates:

.tf Files: Represent the imported VMware infrastructure.
Terraform State File: Tracks the current state of imported resources.

Organize and modularize the configuration files as needed.

Step 2: Manage VMware Resources with Terraform vSphere Provider

2.1 Configure the vSphere Provider

Set up the Terraform vSphere provider in a provider.tf file:

provider "vsphere" {
  user           = "administrator@vsphere.local"
  password       = "your_password"
  server         = "vcenter.example.com"
  allow_unverified_ssl = true
}

2.2 Refine Imported Resources

Move the imported .tf files to your Terraform project directory and refine the configuration. For example, to manage a virtual machine:

resource "vsphere_virtual_machine" "ad_server" {
  name             = "AD-Server"
  resource_pool_id = "resgroup-123"
  datastore_id     = "datastore-456"

  num_cpus   = 2
  memory     = 4096
  guest_id   = "windows9_64Guest"

  network_interface {
    network_id   = "network-789"
    adapter_type = "vmxnet3"
  }

  disk {
    label            = "disk0"
    size             = 50
    eagerly_scrub    = false
    thin_provisioned = true
  }
}

2.3 Apply Changes

Run Terraform commands to apply the refined configuration:

terraform init
terraform plan
terraform apply

Step 3: Backup and Restore Strategy with AWS

3.1 Configure Veeam for AWS S3

Create an S3 Bucket: Use Terraform to create a bucket for storing backups:

   resource "aws_s3_bucket" "veeam_backup" {
  bucket = "veeam-backup-bucket"
  acl    = "private"

  # Define lifecycle rules for Intelligent-Tiering
  lifecycle_rule {
    id      = "intelligent-tiering"
    enabled = true

    # Transition to Intelligent-Tiering frequent access tier after creation (automatic)
    transition {
      days          = 0
      storage_class = "INTELLIGENT_TIERING"
    }

    # Transition to Intelligent-Tiering infrequent access tier after 30 days
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    # Transition to Intelligent-Tiering archive access after 90 days
    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    # Transition to deep archive access tier after 120 days
    transition {
      days          = 120
      storage_class = "DEEP_ARCHIVE"
    }   
   }
  }

  # Enable server-side encryption
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

Integrate with Veeam: Configure Veeam Backup & Replication to use the S3 bucket for offsite backups.

3.2 Regular Backup and Testing

Schedule daily backups to S3 with periodic testing of restore capabilities.
Implement incremental backups to optimize storage and bandwidth.

3.3 Monitor and Optimize

Use Amazon CloudWatch to monitor S3 bucket activity.
Optimize costs by reviewing storage lifecycle policies and access patterns.

Conclusion

By combining Terraformer, the Terraform vSphere provider, and AWS backup strategies, you can:

Simplify Management: Import and manage VMware resources declaratively.
Enhance Resilience: Leverage AWS for reliable offsite backups.
Streamline Operations: Automate infrastructure tasks for consistent and scalable management.

This hybrid approach ensures that your on-premises infrastructure is both robust and prepared for disaster recovery scenarios, delivering long-term operational efficiency.

How to Deploy Resources on AWS Using Terraform

Irlan Dos Santos — Tue, 17 Dec 2024 20:32:39 +0000

How to Deploy AWS Resources Using Terraform

1. Prerequisites

Before starting, ensure the following tools are installed and configured:

Terraform - Install Terraform
AWS CLI with configured Access KeyID and secret AccessKey.

Create a folder, exemple:my_test_deploy, open VS Code and open the folder you just created.
Create a new file and name it as main.tf, and paste the terraform code bellow.

2. Backend State Storage (Optional)

In a production environment, save your state file remotely. To enable S3 remote state storage, update the following block:

terraform {
  backend "s3" {
    bucket  = "your-s3-bucket-name"
    key     = "terraform/state.tfstate"
    region  = "us-east-1"
    encrypt = true
  }
}

3. Terraform Configuration

Required Providers and Versions

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.16"
    }
  }
  required_version = ">= 1.2.0"
}

provider "aws" {
  region = "us-east-1"
}

Create VPC

resource "aws_vpc" "my_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "my_vpc"
  }
}

Create Public Subnet

resource "aws_subnet" "public_subnet" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"

  tags = {
    Name = "public-subnet"
  }
}

Create Private Subnet

resource "aws_subnet" "private_subnet" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = "us-east-1b"

  tags = {
    Name = "private-subnet"
  }
}

Create Internet Gateway

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.my_vpc.id

  tags = {
    Name = "my-igw"
  }
}

Create Route Table for Public Subnet

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.my_vpc.id

  tags = {
    Name = "public-route-table"
  }
}

Add Route to Internet Gateway

resource "aws_route" "public_internet_route" {
  route_table_id         = aws_route_table.public_rt.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.igw.id
}

Associate Public Subnet with Route Table

resource "aws_route_table_association" "public_assoc" {
  subnet_id      = aws_subnet.public_subnet.id
  route_table_id = aws_route_table.public_rt.id
}

Launch an EC2 Instance

resource "aws_instance" "app_server_irlan" {
  ami           = "ami-01816d07b1128cd2d"   # Replace with the desired AMI ID
  instance_type = "t2.micro"

  tags = {
    Name = "app_server_irlan"
  }
}

4. Outputs

Output VPC ID

output "vpc_id" {
  value = aws_vpc.my_vpc.id
}

Output Public Subnet ID

output "public_subnet_id" {
  value = aws_subnet.public_subnet.id
}

Output EC2 Instance Public IP

output "instance_public_ip" {
  value = aws_instance.app_server_irlan.public_ip
}

Terraform Workflow

Open a terminal, and go to the folder you created earlier.

Step 1: Initialize Terraform

Run the following command to initialize the working directory and download the AWS provider plugin:

terraform init

Step 2: Review the Terraform Plan

Verify the resources that will be created:

terraform plan

Step 3: Deploy the Resources

Apply the Terraform script to provision resources on AWS:

terraform apply

Type yes when prompted to confirm the deployment.

5. Verify the Deployment

After applying the script:

Check the AWS Management Console for the created VPC, subnets, and EC2 instance.
Use the outputs printed to access the public subnet ID, EC2 instance public IP, and VPC ID.
To get outputs again at any time, run:

terraform output

6. Clean Up

To destroy all resources created by Terraform:

terraform destroy

Confirm by typing yes when prompted.

Conclusion

Using Terraform to deploy infrastructure on AWS provides a scalable and consistent approach. This script automates the creation of a VPC, public and private subnets, an Internet Gateway, and an EC2 instance. You can modify or extend it further to include security groups, load balancers, or RDS instances as needed.