DEV Community: Waleed Elginady

AWS Re-invent 24..What's new??

Waleed Elginady — Wed, 08 Jan 2025 20:04:42 +0000

[(https://www.aboutamazon.com/news/aws/what-is-aws-reinvent)]

Key New Services and Capabilities Announced at AWS re:Invent 2024

Amazon EC2 Trn2 Instances

Designed for training and deploying foundation models.
Up to 20.8 petaflops of FP8 compute and 1.5 TB high-bandwidth memory.

Amazon EC2 Trn2 UltraServers

Built for high-performance AI workloads with 64 Trainium2 chips.
Offers 83.2 petaflops of FP8 compute.

AWS Trainium3

First chip using a three-nanometer semiconductor process.
Up to twice as fast as Trainium2, 40% more energy-efficient.

Amazon S3 Tables

Fully managed Apache Iceberg tables for analytics workloads.
Achieves up to 3x faster query throughput.

Amazon Aurora DSQL

Serverless distributed SQL database with 4x faster reads and writes.
99.999% multi-region availability.

Amazon DynamoDB Global Tables

Enhanced multi-region capabilities with strong consistency.

Amazon Bedrock Model Distillation

Smaller, faster models that maintain use-case-specific accuracy.

Amazon Q Developer Enhancements

Automatic unit test generation and code reviews.
Integration with GitLab for a unified DevSecOps experience.

Amazon SageMaker Enhancements

HyperPod flexible training plans and task governance for generative AI models.

Amazon Kendra Generative AI Index

Improved search accuracy for generative AI tasks.

Amazon Bedrock Marketplace

Access to over 100 foundation models for generative AI development.

Amazon Bedrock Guardrails

Configurable safeguards for building generative AI applications.
These announcements reflect AWS's commitment to pushing the boundaries of cloud computing and AI capabilities, providing customers with innovative tools to enhance their operations and applications.

Navigating AWS: Crafting a Secure Architecture with Documentation and Well-Architected Framework

Waleed Elginady — Sun, 31 Dec 2023 23:47:28 +0000

Cover Photo Credit[EDURECA.com]
Introduction
When I first started using AWS, I came to understand how crucial it was to follow the documentation and architectural guidelines of AWS. They turned to become my compass points for navigating this web of interconnectivity.

Step 1: Leveraging AWS Documentation
The AWS documentation turned as a reliable friend. I delved deeply into the extensive library, learning tips, tricks, and best practices. Every page contained keys to releasing AWS services' full potential.

AWS Documentation

Step 2: Embracing Well-Designed Architectural Principles
The principles of architecture served as my compass. I welcomed ideas like security, dependability, and scalability and let them shape my design into a robust and flexible framework.

AWS Well-Architected and the Pillars

Step 3: Getting Acquainted with AWS Services
Equipped with an understanding of architectural principles and documentation, I went back and reviewed AWS services. This time, I tackled them with a deeper comprehension and matched my architectural objectives with their functionalities.

AWS Free Tier

Step 4: Sketching Out My Blueprint
Every decision I made in my architectural design was informed by documentation and design principles. I used security layers, redundancy plans, and scalability measures that were all inspired by the guiding principles.

Draw AWS architecture diagrams

Step 5: Setting Up IAM Roles and Policies
I was guided towards least privilege access and strong policy frameworks by the architectural principles, while the documentation provided comprehensive instructions on setting up IAM.

AWS Identity and Access Management

Step 6: Launching My EC2 Instances
I carefully chose instances, set up security groups, and mapped out network settings to correspond with the intended architecture, guided by best practices in documentation and architectural plans.

Amazon EC2

Step 7: Implementing S3 for Storage
The documentation gave me detailed instructions on how to set up S3, and the principles of architecture inspired me to design for data durability and encryption, resulting in the creation of a secure storage ecosystem.

Amazon S3

Step 8: Deploying RDS for Databases
My database selections and security configurations were informed by architectural principles and documentation insights, which made sure my data fortress was built to last.

Amazon Relational Database Service

Step 9: Configuring Networking with VPC
My VPC configuration was supported by architectural principles and AWS documentation, which matched subnets, route tables, and gateways for secure communication in accordance with best practices.

Amazon Virtual Private Cloud

Step 10: Monitoring and Security Measures
While architectural principles stressed a proactive security approach, documentation described monitoring instruments. I implemented CloudWatch and configured alerts using the defense-in-depth approach as a guide.

Security, Identity, and Compliance on AWS

Step 11: Testing and Optimization
My optimization tactics were driven by the architectural concepts of ongoing improvement and the testing requirements provided in the documentation, which ensured that my architecture was not only functional but also optimally tuned for efficiency.

AWS Optimization and Licensing Assessment

Conclusion
My experience with AWS was entwined with the knowledge gained from documentation and design principles. They shaped every aspect of my architecture; they were more than just advisors. They continue to be my compass as I travel this voyage, showing me the way to a reliable and secure Amazon infrastructure.

Terraform and AWS Overview

Waleed Elginady — Sat, 18 Mar 2023 09:50:12 +0000

What is terraform?

Terraform is an open-source infrastructure-as-code (IaC) tool developed by HashiCorp. It allows you to define, manage, and automate your infrastructure as code, making it easier to provision and manage your resources on AWS and other cloud platforms.

Capabilities...

HashiCorp Configuration Language (HCL), a high-level language that offers a straightforward and declarative syntax to define your resources, may be used with Terraform to define your infrastructure. Creating and managing complex infrastructures across various environments and geographies is made simple by this.

Terraform for AWS....

Infrastructure-as-Code: Terraform allows you to define your infrastructure as code, which means that you can store, version, and manage your infrastructure the same way you manage your application code.
Multi-Cloud Support: Terraform supports multiple cloud providers, including AWS, Azure, and Google Cloud Platform, allowing you to manage your infrastructure across multiple clouds with a single tool.
Modularity and Reusability: Terraform allows you to modularize your infrastructure code, making it easy to reuse code across different environments and projects.
State Management: Your infrastructure's present status is monitored by Terraform, allowing it to precisely decide what needs to be added, changed, or removed.
Plan and Apply: Terraform includes a plan command that shows you what changes will be made to your infrastructure before applying them, allowing you to review and approve changes before they are implemented.

Where to start?

To use Terraform with AWS, you must first create a Terraform configuration file (typically named "main.tf") in which you define your infrastructure resources using the HCL syntax. You can then use the Terraform CLI to initialize your project, plan and apply your changes, and manage your infrastructure.

Here are the high-level steps to use Terraform with AWS:

Install Terraform: From the official website, you may download Terraform and set it up locally.
Create a Terraform Configuration File: Create a main.tf file and define your AWS resources using the HCL syntax.
Configure Your AWS Credentials: Create environment variables for your AWS access and secret key or use a machine-configured AWS profile.
Initialize Your Project: Run the terraform init. command to initialize your Terraform project and download the necessary providers.
Plan Your Changes: To examine what modifications will be made to your infrastructure, use the terraform plan command.
Apply Your Changes: Run the terraform apply command to apply your changes and create your AWS resources.

Some of the key use cases for using Terraform with AWS include:

Infrastructure automation: Terraform can be used to automate the provisioning of AWS resources, such as setting up VPCs, subnets, security groups, and other networking components.

Application deployment: Terraform can be used to deploy and manage applications on AWS, such as deploying a web application on an EC2 instance and setting up a load balancer.

Resource scaling: Terraform can be used to scale AWS resources, such as increasing the size of an EC2 instance, adding more instances to an Auto Scaling Group, or increasing the capacity of an RDS database.

Compliance and governance: Terraform can be used to enforce compliance and governance policies on AWS resources, such as ensuring that security groups are configured correctly or that all resources are tagged with the appropriate metadata.

Multi-account management: Terraform can be used to manage multiple AWS accounts, enabling teams to maintain consistent infrastructure across multiple environments and accounts.

Overall, using Terraform with AWS can help organizations manage infrastructure more effectively, reduce errors and inconsistencies, and improve collaboration and agility in managing resources on the cloud.

File Streaming with AWS Media Services Workshop

Waleed Elginady — Sun, 17 Jul 2022 17:22:41 +0000

AWS #biginner #S3

In this workshop, you will learn how to process videos for web streaming using AWS services for storage and media transcoding.
This workshop is for developers, operators, or video engineers who want to learn the basics of processing file-based media with AWS Media Services. No prior experience is necessary, but familiarity with the AWS Console, Amazon S3, and AWS Lambda can help.

Steps

We will create an S3(Simple Storage Service) bucket for storing media
We will create an AWS Elemental MediaConvert Job to transcode videos into Apple HTTP Live Streaming (HLS) adaptive bitrate format (ABR) for delivery over the web
We will configure the (Identity and Access Management)IAM policies required to grant service access to your content.

Hint

Costs for running this workshop in your own account, using our provided content (2m long), will be less than 1$.

Storage

In this section, you’ll use the console to create an S3 Bucket to store outputs from MediaConvert jobs. To enable access from the test player to your content, you must allow public read access and apply a Cross Origin Resource Sharing (CORS) policy. Typically, an origin server for content would be fronted by a Content Delivery Network like Amazon Cloudfront before going into production, but we will omit this step to keep the workshop shorter in length. A CDN improves performance to end users and reduces load on backend origin infrastructure.

Create Bucket

1- In the AWS Management Console choose Services then select S3 under Storage.

2- Choose +Create Bucket.

3- Provide a globally unique name for your bucket such as vod-YOURLASTNAME.

4- Select the Region you've chosen to use for this workshop from the dropdown.

5- Choose Create bucket in the lower right of the page.

6- From the S3 console select the bucket you just created.

7- Select the Properties tab and click edit on the Static website hosting tile.

8- Select Enable to use this bucket to host a static site.

9- Enter index.html in the Index document box. Note: S3 requires an index document configuration even though we won’t be using it.

10- Select Save changes.

11- Under the Permissions tab click on CORS configuration and enter the following XML into the CORS configuration editor and click Save. This will allow a browser to request S3 content from any origin.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

Create an IAM Role

For MediaConvert to read and write from an S3 Bucket and emit status events to CloudWatch, we need to create an Identity and Access Management (IAM) role for MediaConvert to assume. An IAM Role defines a set of permissions to be assumed by a trusted entity, such as a user, service, or application.

1-From the AWS Management Console, click on Services and then 2-select IAM in the Security, Identity & Compliance section.

2-Select Roles in the left navigation bar and then choose Create role.

3-Select AWS Service and MediaConvert for the role type, then click on the Next:Permissions button. Note: Selecting a role type automatically creates a trust policy for your role that allows AWS services to assume this role on your behalf. If you were creating this role using the CLI, AWS CloudFormation or another mechanism, you would specify a trust policy directly.

4-Choose Next:Tags

5-Choose Next:Review.

6-Enter vod-MediaConvertRole for the Role name.

7-Choose Create role.

8-Type vod-MediaConvertRole into the filter box on the Roles page and choose the role you just created.

9-Scroll to the top of the page for your new Role and note down the ARN. You will use this in the following modules.

Transcoding

A MediaConvert job reads an input video from S3 and converts the video to output groups that in turn can have multiple outputs. A single media convert job can create multiple versions of the input video in different codecs and packages. For simplicity, in this module, you will create a MediaConvert job that takes a single input video and converts the video into an Apple HLS output group. This is an example of an adaptive bitrate stack that encodes the source into multiple files at different bit rates and divides them into short chunks, usually between 5-10 seconds long. Along with text-based manifest files, the media chunks for each of the encoded streams are hosted via an HTTP server (S3).

1-Open the MediaConvert console for the region you are completing the lab within

2-Select Jobs from the side bar menu.

3-Select Create job to open the Create job page.

4-Select Input 1 on the Job panel to open the form for Input 1.
5-Enter the file name s3://mediaimmersion/mediaconvert/VANLIFE.m2ts in the box under Input 1.

Create Job

1-Under Output Groups on the Job panel, select Add.

2-Check the Apple HLS box then click on Select.

3-Make sure Apple HLS is selected in the Job panel in the Output groups section so that Apple HLS form is loaded.

4-Fill in HLS in the Custom group name box with any name, this helps you identify groups easily if you have many in a job or template.
5-In the Destination Browse for bucket name you created from the previous step to and/or construct the output S3 path as follows: s3://vod-YOURLASTNAME/assets/VANLIFE/HLS/

6-Leave the remaining settings as the default.
7-Scroll down to the Outputs panel and click on Add output two times so that there are a total of three outputs.

8-Select Output 1 from the Output Groups section Jobs panel to go to the Output settings form for Output 1.
9-Select System-Avc_16x9_360p_29_97fps_1200kbps in the preset dropdown.

10-Enter _360 in the Name modifier box to append the resolution to the stream name.

11-Enter $dt$ in the Segment modifier box to append a datetime to each media segment to uniquely identify the media and prevent accidental overrwrites.

12-Select Output 2 from the Output Groups section Jobs panel to go to the Output settings form for Output 2.
13-Select System-Avc_16x9_540p_29_97fps_3500kbps in the preset dropdown.

14-Enter _540 in the Name modifier box

15-Enter $dt$ in the Segment modifier box.

16-Select Output 3 from the Output Groups section Jobs panel to go to the Output settings form for Output 3.

17-Select System-Avc_16x9_720p_29_97fps_5000kbps in the preset dropdown.

18-Enter _720 in the Name modifier box
19-Enter $dt$ in the Segment modifier box.

20-Select AWS Integration from the Job settings section of the Job panel to open the Job settings form.

21-Select the role you created in the previous section of this lab from the Service role dropdown.

22-Scroll to the bottom of the page and select Create

23-Wait for the job to complete. Monitor the status of the job by refreshing the Job details page using the refresh button.

Playback

To play the videos, you must make the manifest and segment files public. In order to do so, you need to remove the restrictions imposed by S3 that prevent accidental public objects, then make the MediaConvert output objects public.

1-Navigate to your Media Bucket and select Permissions, then Public access settings and edit.

2-Uncheck the “Block new public ACLs…” and “Remove public access…” options as shown below

3-Save and type ‘confirm’
4-Now that you have permissions to make our object public, navigate to the root of your Media Bucket by selecting Overview.
5-Check the box next to the ‘assets’ folder, select Actions, and finally Make public
6-Get the Object URL for the manifest file located in /assets/VANLIFE/HLS/ it should look like:
https://s3.amazonaws.com//assets/VANLIFE/HLS/VANLIFE.m3u8

7-Play the stream using the Safari browser by clicking on the Link for the object or use the JW Player Stream Tester .

Congratulations! You have successfully created an adaptive bitrate video-on-demand stream using AWS Elemental MediaConvert and used S3 as an origin server.

Cloud Computing & AWS Graph talks

Waleed Elginady — Wed, 13 Jul 2022 08:27:21 +0000

AWS #cloudcomputing

Cloud Computing

Cloud computing is the on-demand availability of computer system resources,(including servers, storage, databases, networking, software, analytics, and intelligence), without direct active management by the user. Large clouds often have functions distributed over multiple locations, each location being a data center.

Cloud Computing History timeline

-In the 1950s John McCarthy created a time-sharing concept, enabling a group of users to use an expensive mainframe simultaneously.
-In the 1960s, the American computer scientist Joseph Carl Robnett Licklider described a theory of interconnected computing systems.
-In 1972, IBM released VM (Virtual Machine) operating system, exhibiting the behavior of dedicated hardware.
-In the 1990s, telecommunications companies started offering “Virtualized” Private Networks (VPNs) as rentable services.
-In 1996 The term “cloud computing” was coined within a Compaq internal document.
-In 1997 Professor Ramnath Chellapa of Emory University defined cloud computing as “A computing paradigm, Where the boundaries of computing will be determined by economic rationale, rather than technical limits alone.”

-In 1999 the arrival of Salesforce.com pioneered the concept of delivering enterprise applications via a simple website.
-In 2002 launching of Amazon Web Service (AWS).

Cloud Computing Benefits

Whether you experience a natural disaster, power failure, or other crisis, having your data stored in the cloud ensures it is backed up and protected in a secure and safe location. Being able to access your data again quickly allows you to conduct business as usual, minimizing any downtime and loss of productivity.
Besides factors behind this change include things like:
-Reduced costs: Moving to cloud computing may reduce the cost of managing and maintaining your IT systems.
-Scalability: Your business can scale up or scale down your operation and storage needs quickly to suit your situation,
-Improved time to market: Cloud applications improve collaboration by allowing dispersed groups of people to meet virtually and easily share information in real time and via shared storage.
-Better collaboration: Collaboration in a cloud environment gives your business the ability to communicate and share more easily outside of the traditional methods.
-Increased flexibility. The cloud is a great way to run a business since it offers many advantages and only a few disadvantages.

Risks of cloud computing

-Privacy agreement and service level agreement: You will need to have suitable agreements in place with your service providers before services commence.
-Security and data protection: consider how your data will be stored and secured when outsourcing to a third party.
-Location of data: investigate where your data is being stored and which privacy and security laws will apply to the data. you will also need to be aware of the legislation and regulation requirements in that geographic location.

Cloud Computing Market Size & Shares

What is AWS

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.
Amazon Web Services (AWS) was officially released in 2006. However, its development started earlier in 2003. The first available service was SQS; it was developed in November 2004, followed by EC2. AWS was mainly focused on website and application hosting in those days. Nowadays you can expect to run any type of workload in the AWS cloud.
AWS Services

Services

Amazon Web Services offers a wide variety of services, including compute power, database engines, resources for authentication, services for migration of data and storage solutions.

AWS customers

Resources:

1- cloud computing image
https://forum.huawei.com/enterprise/en/everything-you-need-to-know-about-cloud-computing-simply-explained/thread/682381-893?page=2
2-cloud computing history
https://www.netsolutions.com/insights/what-is-cloud-computing/
3- cloud computing benefits and risks
https://www.business.qld.gov.au/running-business/digital-business/online-risk-security/cloud-computing/benefits#:~:text=Whether%20you%20experience%20a%20natural,downtime%20and%20loss%20of%20productivity.
https://www.poketors.com/2013/09/benefits-of-cloud-computing-infographic.html
https://www.sdxcentral.com/cloud/definitions/what-is-cloud/top-risks-of-cloud-computing/
4-Cloud Computing Market Size & shares
https://www.statista.com/statistics/500541/worldwide-hosting-and-cloud-computing-market/
https://www.theregister.com/2022/05/02/cloud_market_share_q1_2022/
5-AWS
https://aws.amazon.com/what-is-aws/?nc1=f_cc
https://www.clickittech.com/aws/why-aws-cloud/
https://trailhead.salesforce.com/en/content/learn/modules/aws-cloud/discover-the-aws-service-categories