DEV Community: KrushiVasani

Unlocking the Power of Big Data with Amazon EMR

KrushiVasani — Sun, 09 Jul 2023 04:23:15 +0000

Introduction:
In today's data-driven world, businesses are recognizing the potential of leveraging big data processing and analytics frameworks like Apache Hadoop and Apache Spark. However, operating these technologies in on-premises data lake environments can present several challenges, including lack of agility, high costs, and administrative headaches. To overcome these hurdles, many organizations are turning to Elastic MapReduce (EMR), a managed service offered by Amazon Web Services (AWS). EMR allows businesses to harness the power of scalable EC2 instances and run distributed frameworks like Hadoop, Spark, HBase, Presto, and Flink.

In this article, we will explore what EMR is and how it solves common problems associated with on-premises big data environments. We will delve into the concept of EMR clusters, discuss different storage options available with EMR, highlight supported tools, and explore the benefits of using EMR, including cost savings, ease of deployment, enhanced security, and seamless integration with other AWS services.

EMR Empowering Big Data Processing:
Elastic MapReduce (EMR) is a managed Hadoop framework provided by Amazon Web Services (AWS) that enables businesses to process massive volumes of data using scalable EC2 instances. With EMR, organizations can efficiently analyze and derive insights from their data, thanks to the flexibility and power of distributed computing.

EMR offers the ability to run various distributed frameworks, including Apache Spark, HBase, Presto, and Flink, alongside the core Hadoop ecosystem. This versatility allows businesses to choose the right tools for their specific big data processing and analytics needs, without the burden of managing the underlying infrastructure.

Understanding EMR Clusters:
EMR clusters are collections of Amazon EC2 instances that work together to process data. Each instance within a cluster plays a specific role, determined by its node type. The three primary node types in an EMR cluster are:

Leader Node (Master Node):
Manages the cluster by coordinating job and task distribution
Tracks the status and health of the cluster

Worker Node (Core Node):
Runs tasks and stores data in the Hadoop Distributed File System (HDFS)

Task Node (Slave Node):
Runs tasks but does not store data

Storing Data in EMR:
EMR provides three storage options for managing data:

Hadoop Distributed File System (HDFS):
HDFS is a distributed and scalable file system for Hadoop.
It stores data across multiple instances in the cluster and creates replicas for fault tolerance.
Primarily used for intermediate results, as data is lost once the cluster is terminated.

EMR File System (EMRFS):
EMRFS allows direct access to data stored in Amazon S3.
Input and output data can be stored in S3, enabling easy data reuse and accessibility.

Local File System:
In this storage option, data is stored on the local disks of the cluster's instances.
Typically used for temporary or non-persistent data.

Supported Tools and Flexibility

EMR supports a wide range of tools and frameworks that can be installed on the cluster to meet specific data processing requirements. Some of the supported tools include:

Apache Zeppelin:

An interactive notebook for data exploration, visualization, and collaboration.

Apache Hadoop:
The core framework for distributed processing and storage of big data.

HBase:
A scalable, distributed database that provides random access to large amounts of structured data.

Hive:
A data warehousing and SQL-like query language for querying and analyzing data stored in Hadoop.

ZooKeeper:
A coordination service used to manage distributed systems.
EMR Benefits: Unlocking Potential (250 words)
By adopting Amazon EMR, businesses can realize several benefits:

Cost Savings:
EMR eliminates the need for physical hardware, enabling businesses to leverage AWS's scalable infrastructure.
Reserved instances can be used to optimize costs further.

Deployment Made Easy:
EMR simplifies the deployment of big data tools and frameworks, reducing setup and configuration time.
Organizations can customize EMR clusters to meet their specific needs, ensuring optimal performance and resource allocation.

Enhanced Security:
EMR integrates with AWS Identity and Access Management (IAM) for robust user authentication and authorization.
Data stored in EMR can be encrypted to protect sensitive information.
Secure access to the cluster can be achieved using EC2 key pairs.

Seamless AWS Integration:
EMR seamlessly integrates with other AWS services, such as Amazon S3 for data storage, IAM for security and permissions, and Virtual Private Cloud (VPC) for networking.

Conclusion:
Amazon EMR is revolutionizing big data processing by providing a managed Hadoop framework that addresses the challenges associated with on-premises data lake environments. With EMR, businesses can leverage the power of scalable EC2 instances and run distributed frameworks like Hadoop, Spark, and more. By utilizing EMR's flexible storage options, such as HDFS and EMRFS, organizations can efficiently manage their data. Additionally, EMR's support for various tools and seamless integration with other AWS services make it a compelling choice for businesses seeking to unlock the potential of big data. Embrace the power of Amazon EMR and take your data analytics to new heights.

Exploring Amazon Kinesis: Real-Time Data Streaming and Processing with Kinesis Streams and Firehose

KrushiVasani — Sat, 08 Jul 2023 14:19:24 +0000

In December 2013, Amazon Web Services (AWS) launched Kinesis, a service designed for processing real-time streaming big data. Over the years, AWS has expanded the availability of Kinesis to multiple regions, allowing integration with custom applications for real-time data processing from various sources.

Kinesis serves as a highly reliable conduit for streaming messages between data producers and data consumers. Data producers can be any source of data, such as system logs, social network data, financial information, geospatial data, mobile app data, or IoT device telemetry. Data consumers typically include applications for data processing and storage like Apache Hadoop, Apache Storm, Amazon Simple Storage Service (S3), and ElasticSearch.

Key Concepts: Kinesis vs Firehose
To work with Kinesis Streams effectively, it's important to understand some key concepts. The fundamental scaling unit in Kinesis is a shard. Each shard can handle up to 1MB or 1,000 PUTs (data writes) per second, and emit data at a rate of 2MB per second.

Shards scale linearly, meaning that adding shards to a stream increases the ingestion rate by 1MB per second and the emission rate by 2MB per second for each added shard. For example, ten shards would enable a stream to handle 10MB (10,000 PUTs) of data ingestion and 20MB of data emission per second. The number of shards is determined when creating a stream and cannot be changed through the AWS Console afterward.

Resharding, the process of dynamically adding or removing shards from a stream, is possible using the AWS Streams API. However, resharding is considered an advanced strategy and should be approached with a solid understanding of the subject.

When adding or removing shards, the cost of the stream adjusts accordingly. The default limit for shards per region is 10, but this limit can be increased by contacting Amazon Support. There is no limit to the number of shards or streams in an account.

Records are the data units stored in a stream, consisting of a sequence number, a partition key, and a data blob. Data blobs represent the payload of information within a record and have a maximum size of 1MB (before Base64-encoding). For larger data, it needs to be divided into smaller chunks before being put into a Kinesis stream.

Partition keys are used to identify different shards within a stream and enable data distribution across shards. Sequence numbers are unique identifiers for records inserted into a shard and increase monotonically. They are specific to individual shards.

Amazon Kinesis Offerings: Kinesis Streams, Firehose, and Analytics
Amazon Kinesis is divided into three service offerings:

Kinesis Streams: Captures large volumes of data from data producers and streams it into custom applications for processing and analysis. Kinesis replicates streaming data across three availability zones in AWS for reliability and availability. Scaling the ingestion and emission rates requires manually provisioning the appropriate number of shards for the expected data volume. Data can be loaded into streams using HTTPS, Kinesis Producer Library, Kinesis Client Library, or Kinesis Agent. By default, data is available in a stream for 24 hours but can be extended to 168 hours (7 days) for an additional charge. Monitoring is provided through Amazon CloudWatch.

**Kinesis Firehose: **Used for capturing and loading streaming data into other Amazon services like S3 and Redshift. Firehose can handle gigabytes of streaming data per second and supports features like data batching, encryption, and compression. Unlike Kinesis Streams, Firehose automatically scales to meet demand, eliminating the need for manual provisioning. Data can be loaded into Firehose using various methods, and it can stream data to S3 and Redshift simultaneously. Monitoring is available through Amazon CloudWatch.

Kinesis Analytics: A forthcoming product from Amazon that allows running standard SQL queries on data streams and sending the results to analytics tools for monitoring and alerting. As of now, detailed information about this service has not been released by Amazon.

Kinesis vs SQS: Key Differences
Kinesis and Amazon's Simple Queue Service (SQS) differ in their purpose and capabilities. Kinesis is designed for real-time processing of streaming big data, while SQS serves as a message queue for storing messages between distributed application components.

Benefits of Kinesis:

Routing and ordering of records based on a given key.
Multiple clients can read messages concurrently from the same stream.
Ability to replay messages up to seven days in the past.
Records can be consumed at a later time.
Provisioning enough streams ahead of time is required to meet anticipated demand.

Benefits of SQS:

Messaging semantics for tracking successful completion of work items in a queue.
Delay scheduling of messages for up to 15 minutes.
Automatic scaling to handle application demand.
Limited number of messages that can be read or written at a time compared to Kinesis, allowing for larger batches of messages in Kinesis.
By understanding the differences between Kinesis Streams, Firehose, and SQS, you can choose the most suitable service for your specific use case and requirements.

Migrating MySQL to PostgreSQL With the AWS Database Migration Service

KrushiVasani — Sat, 08 Jul 2023 13:52:01 +0000

AWS Database Migration Service (DMS) is used to transfer data and database applications between different database instances. When migrating data the source and the target databases can use the same database engine, or they can be different engines. The primary use-case for DMS is to enable and support one-time large-scale migration activities.

A secondary use-case is for frequent or long-term replication tasks. Using DMS, a migration process that would previously have involved an outage or a risky and sudden switching of instances can be avoided. Instead, setting up real-time replication across different database instances allows for migration activities to happen more slowly, in smaller steps, and with validation being performed at each stage.

In addition, DMS can also be used for indefinite backup tasks. This is usually more costly than traditional database backup strategies (periodic snapshotting for example), but when the volume of data is very big, or real-time backups are a requirement, DMS is often the most effective and efficient solution.

In this blog we will use DMS to migrate data from a database instance running the MySQL engine to an instance running the Aurora PostgreSQL engine.

Learning Objectives
This is a beginner level blog, upon completion of this blog you will be able to:

Configure source and target endpoints in DMS
Run a migration task in DMS
Connect to MySQL and PostgreSQL databases from the command-line
Generate a pre-migration task assessment in DMS.

Prerequisites
You should have a conceptual understanding of:

Amazon RDS
SQL and Databases

Steps:

In the AWS Management Console search bar, enter Database Migration Service, and click the Database Migration Service result under Services
In the left-hand menu, click Endpoints.
Click Create endpoint.

In the Endpoint type section of the Create endpoint form, ensure Source endpoint is selected:

Check the Select RDS DB instance checkbox, and in the drop-down box that appears, select mysqlsource: (have created this database in advanced)

This will populate the Endpoint configuration section of the form with values for Server name, Port, and User name.

In the Endpoint configuration section of the form, select Provide access information manually.

In the Password textbox, enter testpass.

By entering the password here you are explicitly giving the Database Migration Service access to the source database.

In the Test endpoint connection section of the form, in the VPC drop-down menu, select the VPC test. (your VPC):
In the Replication instance drop-down, ensure lab-replication-instance is selected:
To create your source endpoint, click Create endpoint:
In the Endpoints list, select the mysqlsource endpoint:
To open the Test endpoint connection form, click Actions, and Test connection:
To test the source endpoint connection, click Run test:

The test will take up to a minute to complete.

Once complete you will see Status field change to successful:

If you don't see a successful connection, it is likely that the password stored in the endpoint is incorrect. Use the management console to modify it and re-enter the password.

Connecting to the Virtual Machine using EC2 Instance Connect

In the AWS Management Console search bar, enter EC2, and click the EC2 result under Services:
To see available instances, click Instances in the left-hand menu:

(will use this instance console later in this blog)

Populating the Source Database

Steps

In the AWS Management Console search bar, enter RDS, and click the RDS result under Services:

In the left-hand side menu, click Databases:
In the list of databases, select mysqlsource:

Under the Connectivity & security heading, make a note of the Endpoint, it will be similar to the following:

You will use this endpoint in a moment to connect to the database from the Linux host.

To avoid confusion, an endpoint resource in DMS is not the same as an endpoint in RDS:

In RDS an endpoint is the hostname of the RDS instance
In DMS an endpoint is a resource that contains information about the type of database and also includes the hostname, and other connection details

In the shell browser window you accessed in the previous Step, enter ls.

You will see the following: (have created small test database files)

people.sql is a file containing sample data to be loaded into the source database. The dataset contains one table called people that has six columns.

To load the people.sql data into the source database, enter the following command, replacing source-mysql-endpoint with the endpoint you noted down earlier from RDS:

mysql -P 3306 -u admin -p -h "source-mysql-endpoint" < people.sql

You will be asked to enter a password, enter testpass, this password i have used. Please note that the password is case-sensitive.

You will see output similar to the following:

This command has two parts, the first part is from the start of the command up to the < symbol. It tells the MySQL command-line client to create a connection to the source database:

-P 3306 specifies the port to connect to
-p specifies that a password is required
-u admin specifies the username to use when connecting
-h source-mysql-endpoint specifies the hostname to connect to
The second part, < people.sql, uses a feature of the Linux Bash shell called redirection. This part of the command is feeding the contents of the people.sql file into the MySQL client's connection.

To connect to the database enter the following command, replacing source-mysql-endpoint with the endpoint you retrieved from RDS previously:

mysql -P 3306 -u admin -p -h "source-mysql-endpoint" people

You will be asked for a password, enter testpass.

This command is similar to the one you used to load the people.sql file. This time your command is not redirecting a file, instead, it is connecting to the people database you created.

A MySQL client command prompt will open.

To verify the data has been populated, enter the following SQL query in the MySQL command prompt:

SELECT * FROM people LIMIT 10;

This query selects the first 10 records from a table called people.

You will see output similar to the following:

To see how many rows there are in total in the people table, enter the following SQL query into the MySQL command prompt:

SELECT COUNT(*) AS row_count FROM people;
You will see the following output:

To exit the MySQL command prompt and return to the bash shell, enter quit.

Creating the Migration Task

Navigate to the AWS Database Migration Service.
In the left-hand menu, click Database migration tasks:
Click Create task:
In the Task configuration section of the Create database migration task form, in the Task identifier textbox, enter test-task:
Select the following values for these drop-down fields, accepting the defaults for fields not specified:

Replication instance: Instance beginning with lab-replication-instance
Source database endpoint: mysqlsource
Target database endpoint: postgrestarget-1

In the Migration type drop-down, ensure Migrate existing data is selected:

The Migration type field allows you to specify different kinds of migration:

Migrate existing data: This type is the simplest, it migrates data from the source to the target and finishes. Using this type will usually require an outage for the duration of the migration.
Migrate existing data and replicate ongoing changes: This type will capture changes to the source during the migration and apply them. With this type outages can be minimized or avoided.
Replicate data changes only: This type assumes you have already performed an initial migration of data from source to target, and want to migrate changes in the source that have occurred since. This allows for more complex migration scenarios, such as the initial migration happening outside of the AWS Database Migration Service.

Scroll down to the Table mappings section of the form, and click Add new selection rule:

Selection rules allow specifying which parts of a database to export. You can create include and exclude rules.

As an example where you would use this, imagine you have a group of one or more database tables that don't have relationships with other tables. Selection rules can be configured to migrate those tables separately from the rest of the database, enabling you to migrate your database in parts. This approach may be less risky and easier to manage than migrating the entire database in one task.

In the Schema drop-down, select Enter schema:
In the Source name field, enter people:

In the Source Table name field, enter people:

You have specified an include selection rule that will migrate the people table from the people database schema.

Scroll down to the Premigration assessment section, and check Enable premigration assessment run:
Under Assessments to run, leave the options at their defaults.
Under Assessment report storage, click Browse S3:
An S3 bucket choose dialog will open.
Select the bucket by clicking the radio button:

You have to selecte a bucket where the pre-migration task assessment data will be stored.

Under IAM role, select the role called s3-access-for-tasks:

This role should allows DMS to access S3.

To create your migration task, at the bottom of the page, click Create task:
To start your task, in the top right, click Actions and click Restart/Resume.

Understanding Basics of Authentication, Authorisation, and Identity Federation

KrushiVasani — Sat, 01 Jul 2023 11:54:38 +0000

In today's interconnected world, where online security plays a vital role, it's essential to understand the fundamental terms and protocols related to authentication, authorization, and identity federation. In this blog, we will explain these concepts in simple terms and explore some common protocols used for identity federation, such as OAuth 2.0, SAML 2.0, and OpenID Connect (OIDC). We will also delve into how these concepts apply to the AWS (Amazon Web Services) environment. Let's get started!

Authentication and Authorization:
Authentication refers to the process of verifying the identity of a user, typically done through a combination of a login and password. It answers the question, "Who are you?" Once a user's identity is established, the next step is authorization. Authorization determines what actions and resources a user is allowed to access. It answers the question, "What are you allowed to do?"

Identity Provider (IdP):
An Identity Provider, or IdP, is a system that stores and manages user data, such as email addresses and passwords. It serves as a trusted source for authentication and authorization. One commonly used IdP is Active Directory, which is widely utilized for managing user identities in many organizations.

Identity Federation:
Identity Federation allows users to use an external IdP instead of managing their own. With federation, you don't need to create your own sign-in code; instead, the IdP takes care of authentication and authorization tasks. By granting federated identities permission to use your resources, you can simplify user management and enable seamless access to various services.

OAuth 2.0, SAML 2.0, and OIDC:
These protocols are commonly used for identity federation, but each has its own characteristics and use cases. SAML 2.0 and OIDC are primarily authentication and authorization protocols, while OAuth 2.0 focuses on authorization for protected resources like APIs.

SAML 2.0, designed for enterprise usage, is more complex to implement but offers extensive functionality. It allows users to log into a corporate IdP and access other services without re-entering credentials. However, SAML 2.0 is browser-constrained and relies on browser security.

OIDC, on the other hand, is ideal for mobile and consumer-facing applications. It has a low barrier to entry and is based on OAuth 2.0. OIDC combines OAuth 2.0's authorization capabilities with an additional authentication mechanism, utilizing an ID Token in the form of a JSON Web Token (JWT).

Use Cases:
Let's explore some practical examples to better understand the use cases for these protocols:

OAuth 2.0: When you sign up for an app and agree to let it access your contacts on Facebook without sharing your login credentials. OAuth 2.0 enables API authorization without exposing sensitive information.

OIDC: When you sign in to an Identity Provider like Google and gain access to other websites, such as YouTube, without sharing your sign-in information repeatedly. OIDC simplifies the authentication process for various online services.

SAML 2.0: When you log into your corporate intranet or IdP and seamlessly access other services, like Salesforce, without the need to re-enter your credentials. SAML 2.0 streamlines the authentication and authorization process within an enterprise environment.

Identity Federation in AWS:
Amazon Web Services supports all the aforementioned protocols and provides two types of federation:

Web Identity Federation: This approach is suitable when you utilize well-known third-party IdPs like Facebook, Google, or any OIDC compatible provider. AWS enables integration with these IdPs to establish federated access.

Enterprise Identity Federation: If you use a corporate IdP compatible with SAML 2.0, you can leverage out-of-the-box integration with AWS. For example, Microsoft ADFS can be used to integrate with AWS, leveraging your existing Active Directory infrastructure. If your IdP is not compatible, you would need to develop a custom identity broker application to authenticate users, obtain temporary credentials from AWS Security Token Service (STS), and grant access to AWS resources.

AWS Services for Federated Access:
To enable federated access to your workforce, AWS provides the following services:

AWS IAM Identity Center: This service, which succeeded AWS SSO (Single Sign-On), allows you to define federated access permissions for users based on their group memberships within a centralized directory. It simplifies the management of federated access across multiple AWS accounts.

AWS Identity and Access Management (IAM): If you require the flexibility to use multiple directories or manage permissions based on user attributes, IAM provides a comprehensive solution. IAM allows you to define fine-grained access policies and manage users, groups, and roles within the AWS environment.

Conclusion:
Understanding authentication, authorization, and identity federation is crucial in today's digital landscape. By grasping the basics of these concepts and the protocols associated with them, such as OAuth 2.0, SAML 2.0, and OIDC, you can better navigate the complexities of secure user access management. In the AWS ecosystem, identity federation is supported through various protocols, enabling seamless integration with external IdPs. By leveraging AWS services like AWS IAM Identity Center and IAM, you can establish efficient and secure federated access for your workforce.

Demystifying AWS IAM: Exploring Users, Groups, Roles, and Policies

KrushiVasani — Sat, 01 Jul 2023 11:39:09 +0000

Introduction:
AWS Identity and Access Management (IAM) is a critical component of securing and managing access to AWS resources. Understanding the nuances of IAM users, groups, roles, and policies is vital for effective access control and maintaining a robust security posture. In this comprehensive blog post, we will delve into each component in detail, providing in-depth insights, best practices, and examples to help you master IAM in your AWS environment.

Section 1: IAM Users
IAM users are entities that represent people or applications requiring access to AWS resources. Creating and managing users is a fundamental step in IAM configuration. For example, let's consider a scenario where an organization needs to grant access to a group of developers who require access to an S3 bucket for code deployments. We would create individual IAM users for each developer, assign them unique credentials, and attach policies granting the necessary S3 permissions. Additionally, we would enforce strong security practices, such as enabling MFA for user authentication and setting up password policies to ensure regular password rotation.

Section 2: IAM Groups
IAM groups allow for the logical organization of users with similar access requirements. For instance, imagine a scenario where an organization has multiple departments, each with distinct access needs. Instead of individually assigning permissions to each user, we can create IAM groups for each department and assign appropriate policies to the respective groups. This simplifies access management and ensures consistent permissions across users within a department. An example would be creating an "Administrators" group with policies granting full access to AWS resources, and a "Development" group with policies providing access to specific development-related resources.

Section 3: IAM Roles
IAM roles provide a flexible mechanism for granting temporary access to users, services, or resources within or across AWS accounts. Consider an application running on an EC2 instance that requires access to other AWS services, such as accessing an S3 bucket. Instead of hardcoding access keys or credentials within the application, we can create an IAM role with the necessary permissions and associate it with the EC2 instance. The application can then assume the role and access the S3 bucket securely. Role assumption is also useful in cross-account scenarios, where one AWS account needs to access resources in another account.

Section 4: IAM Policies
IAM policies define permissions that determine what actions users, groups, or roles can perform on AWS resources. Policies are written in JSON format and consist of statements that specify the desired access rules. For example, let's say we want to grant an IAM user read-only access to specific Amazon S3 buckets. We would create an IAM policy stating that the user is allowed the "s3:GetObject" action on those specific buckets. By attaching the policy to the user, we ensure they can only perform read operations on the designated buckets. It's crucial to follow the principle of least privilege when crafting policies to prevent over-authorization.

Section 5: IAM Best Practices
Implementing IAM best practices is vital for maintaining a secure AWS environment. One key best practice is the principle of least privilege, which ensures that users, groups, and roles have only the necessary permissions to perform their tasks. Regular auditing and monitoring of IAM configurations help detect and address any potential security vulnerabilities or misconfigurations. Another best practice is utilizing IAM Access Analyzer to identify any unintended access and validate the effectiveness of IAM policies. Additionally, securing IAM users and roles against potential attacks, such as enforcing MFA, regularly rotating access keys, and implementing strong password policies, is crucial.

Section 6: Advanced IAM Concepts
In this section, we explore advanced IAM concepts that extend the capabilities of access control. Identity providers and federated access enable external identities, such as those from Active Directory or social identity providers, to access AWS resources. For example, an organization may integrate its existing Active Directory with AWS IAM, allowing users to log in using their existing AD credentials. Web identity federation enables applications to authenticate users through popular identity providers like Google or Facebook, simplifying user onboarding. AWS Organizations and consolidated billing streamline access management across multiple accounts, making it easier to manage permissions and budgets centrally. IAM also plays a vital role in securing AWS Lambda functions and API Gateway endpoints in serverless architectures.

Section 7: Conclusion
In conclusion, mastering IAM users, groups, roles, and policies is crucial for maintaining a secure and well-managed AWS environment. By following best practices, implementing strong access control measures, and regularly auditing IAM configurations, organizations can ensure that only authorized entities have the appropriate level of access to AWS resources. IAM's flexibility and scalability make it a powerful tool for managing access across diverse environments and scenarios.

By exploring the in-depth details of IAM and providing practical examples, this blog post equips readers with the knowledge and expertise to effectively leverage IAM in their AWS environments. Remember, IAM is not a one-time setup but an ongoing process that requires regular evaluation and updates to adapt to changing requirements and mitigate potential security risks. With a solid understanding of IAM, you can confidently manage access control and safeguard your AWS resources.

Understanding Role Assumption in AWS IAM for Enhanced Security and Access Management

KrushiVasani — Sat, 01 Jul 2023 11:19:26 +0000

Introduction:
AWS Identity and Access Management (IAM) is a crucial component of securing AWS resources. IAM Roles provide a powerful mechanism for granting and managing permissions within your AWS environment. In certain scenarios, it becomes necessary for one role to assume another role, allowing for controlled access delegation and establishing trust relationships. This process, known as "role assumption," enables enhanced security and simplifies access management in AWS.

In this blog post, we will delve into the concept of role assumption in AWS IAM. We will explore the steps involved in configuring role assumption, whether the roles exist within the same AWS account or across separate accounts. By understanding the intricacies of role assumption, you can effectively manage access permissions and bolster the security posture of your AWS infrastructure.

AWS IAM Roles:
IAM Roles serve as a fundamental building block of access management in AWS. They provide a way to define a set of permissions and policies that can be assumed by various entities within your AWS environment. By assigning roles to entities, you can achieve granular access control without relying on long-term access keys, improving security and simplifying access management.

Assuming Roles:
Role assumption is the process of one role taking on the permissions and policies of another role. It enables controlled access delegation and establishes trust relationships between different entities in AWS. Whether the roles are in the same account or different accounts, the steps for configuring role assumption involve modifying the trust relationship of the target role and, in some cases, attaching additional policies to the source role.

Roles in the Same Account:
When the roles exist within the same AWS account, configuring role assumption involves modifying the trust relationship of the target role. This is accomplished by specifying the source role's ARN (Amazon Resource Name) in the Principal element of the target role's trust policy. The Principal element indicates the role that is allowed to assume the target role.

Roles in Different Accounts:
For roles in different AWS accounts, enabling role assumption requires modifying the trust relationship of the target role as before. However, the source role also needs an additional policy granting sts:AssumeRole permissions for the target role's ARN. This policy establishes trust between the accounts, allowing the source role to assume the target role even across account boundaries.

Best Practices for Role Assumption:
Implementing role assumption comes with a set of best practices to ensure secure and efficient access management. The principle of least privilege should be followed, granting only the necessary permissions to roles. Regular auditing and monitoring help maintain the integrity of role-based access. Automation can simplify the configuration and management of assumed roles, reducing human error. Additionally, role chaining, web identity federation, and temporary security credentials offer advanced capabilities for specific use cases.

Troubleshooting Role Assumption:
While configuring role assumption, it's essential to be aware of potential issues that may arise. Common errors, such as incorrect JSON syntax or incorrect Principal elements, can cause role assumption to fail. Enabling logging and debugging features can aid in troubleshooting and resolving any problems that occur during the role assumption process.

Role Assumption with AWS Services:
Role assumption integrates with various AWS services, enhancing their security and access control capabilities. Amazon EC2 instance profiles leverage roles to provide secure access to AWS resources from EC2 instances. AWS Lambda functions can also assume roles, enabling fine-grained access control for serverless applications.

Integrating Role Assumption into AWS Security Solutions:
Role assumption plays a pivotal role in AWS security solutions. The AWS Security Token Service (STS) enables the issuance of temporary security credentials for role assumption. AWS Organizations and consolidated billing further enhance role assumption capabilities, simplifying access management in multi-account environments. Multi-Factor Authentication (MFA) can be enforced for role assumption, adding an extra layer of security to the process.

Conclusion:
Role assumption is a powerful feature of AWS IAM that enables granular access control and simplified access management. By configuring trust relationships and establishing role assumptions, you can ensure secure delegation of permissions within your AWS infrastructure. Implementing best practices, troubleshooting issues, and leveraging role assumption with AWS services and security solutions empowers you to maintain a robust security posture and efficiently manage access to your AWS resources.

Create VPC Peering

KrushiVasani — Sun, 19 Mar 2023 06:00:07 +0000

What :

AWS provided network connectivity between two VPC's.

When :

Multiple VPC's need to communicate or access each other's resources.

Pros :

Uses AWS backbone without traversing the Internet.

Cons :

Transitive peering is not supported. ex. if you have created peering between "VPC 1" and "VPC 2".one more peering between "VPC 1" to "VPC 3".Then "VPC 2" is not peered with "VPC 3".

How :

VPC peering request made;accepter accepts requests (either within or across accounts).

A VPC peering connection is a networking connection between two VPC's that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses.Instances in either VPC can communicate with each other as if they are within the same network.You can create a VPC peering connection between your own VPC's, or with a VPC in another AWS account. The VPC's can be in different regions (also known as an inter-region VPC peering connection). Data sent between VPC's in different regions is encrypted (traffic charges apply).

A VPC peering connection goes through various stages starting from when the request is initiated. At each stage, there may be actions that you can take, and at the end of its lifecycle, the VPC peering connection remains visible in the Amazon VPC console and API or command line output for a period of time.

Setup: VPC Peering Connection

Create the VPCs. To create a VPC peering connection, you need to have two VPCs that you want to connect. I am Creating one VPC with cidr range 10.0.0.0/16 and going to use default VPC.
Navigate to the "peering connection" in the AWS Management Console.
click on "Create Peering Connection".Select the any of the above created vpc as a requester.
Select the accepter vpc.Accepter vpc can be in a different AWS account or in an different Region.
Click on create peering connection.Now you will be able to see one peering connection in the console.but the status will be shown as "pending acceptance".
Select the peering connection and click on the actions button.select the "Accept Request" option.
Now the status of peering connection will be available.
Only thing remaining is add an entry to route table.In my case i am adding an entry to "Test-VPC" route table.created new route with the cidr range of default vpc and used the created peering connection id as a target.
Now change the route table of accepter vpc.

We have successfully created vpc peering between "Test Vpc" and "default VPC".

Pros:

Simplicity: VPC peering provides a simple way to connect VPCs, without requiring any external resources like VPNs or dedicated connections.
Cost-effectiveness: Since VPC peering relies on the cloud provider's internal network, it is usually more cost-effective than using external connections.
Security: VPC peering allows you to keep traffic between VPCs within the provider's network, which can be more secure than using external connections.
Low Latency: Since VPC peering uses the provider's internal network, it generally provides low latency connections between VPCs.

Cons:

Limited scope: VPC peering only works within a single cloud provider's infrastructure and can't connect VPCs in different cloud providers.
Bandwidth limitations: VPC peering has limits on the amount of traffic that can be transferred between VPCs, and exceeding these limits can result in degraded performance.
No transitive peering: VPC peering only supports direct connections between VPCs, so if you need to connect more than two VPCs, you'll need to set up multiple peering connections.
Potential for overlapping IP addresses: VPCs that are peered together cannot have overlapping IP address ranges, so this needs to be carefully managed to avoid conflicts.

Conclusion:

In this blog, we have discussed the steps required to create a VPC peering connection between two VPC's in the same region. VPC peering is a powerful feature that enables you to connect two or more VPC's within the same or different regions or accounts. It can be used to share resources between VPC's, facilitate communication between applications in different VPC's, and improve availability and fault tolerance of your infrastructure.

Things you must know about VPC

KrushiVasani — Mon, 23 Jan 2023 17:16:37 +0000

Amazon VPC

Amazon VPC lets you provision a logically isolated section of the Amazon Web Services (AWS) cloud where you can launch AWS resources in a virtual network that you define.
Provides complete control over the virtual networking environment including selection of IP ranges, creation of subnets, and configuration of route tables and gateways.
A VPC is logically isolated from other VPCs on AWS.
Possible to connect the corporate data center to a VPC using a hardware VPN (site-to-site).
VPCs are region wide.
A default VPC is created in each region with a subnet in each AZ.
By default, you can create up to 5 VPCs per region.
You can define dedicated tenancy for a VPC to ensure instances are launched on dedicated hardware (overrides the configuration specified at launch).
A default VPC is automatically created for each AWS account the first time Amazon EC2 resources are provisioned.
The default VPC has all-public subnets. Public subnets are subnets that have:

`1. “Auto-assign public IPv4 address” set to “Yes”.

The subnet route table has an attached Internet Gateway.`

Instances in the default VPC always have both a public and private IP address.
AZs names are mapped to different zones for different users (i.e. the AZ “ap-southeast-2a” may map to a different physical zone for a different user).

Components of a VPC:

A Virtual Private Cloud: A logically isolated virtual
network in the AWS cloud. You define a VPC’s IP address space from ranges you select.
Subnet: A segment of a VPC’s IP address range where you
can place groups of isolated resources (maps to an AZ, 1:1).Internet Gateway: The Amazon VPC side of a connection to
the public Internet.
NAT Gateway: A highly available, managed Network
Address Translation (NAT) service for your resources in a private subnet to access the Internet.
Hardware VPN Connection: A hardware-based VPN
connection between your Amazon VPC and your datacenter,
home network, or co-location facility.
Virtual Private Gateway: The Amazon VPC side of a VPN
connection.
Customer Gateway: Your side of a VPN connection.
Router: Routers interconnect subnets and direct traffic
between Internet gateways, virtual private gateways, NAT
gateways, and subnets.
Peering Connection: A peering connection enables you to
route traffic via private IP addresses between two peered VPCs.
VPC Endpoints: Enables private connectivity to services
hosted in AWS, from within your VPC without using an an
Internet Gateway, VPN, Network Address Translation (NAT)
devices, or firewall proxies.
Egress-only Internet Gateway: A stateful gateway to
provide egress only access for IPv6 traffic from the VPC to the Internet.

Options for connecting to a VPC are:

Hardware based VPN
Direct Connect
VPN CloudHub
Software VPN

Routing

The VPC router performs routing between AZs within a region.
The VPC router connects different AZs together and connects the VPC to the Internet Gateway.
Each subnet has a route table the router uses to forward traffic within the VPC.
Route tables also have entries to external destinations.
Up to 200 route tables per VPC.
Up to 50 route entries per route table.
Each subnet can only be associated with one route table.
Can assign one route table to multiple subnets.
If no route table is specified a subnet will be assigned to the main route table at creation time.
Cannot delete the main route table.
You can manually set another route table to become the main route table.
There is a default rule that allows all VPC subnets to communicate with one another – this cannot be deleted or modified.
Routing between subnets is always possible because of this rule – any problems communicating is more likely to be security groups or NACLs.

Internet Gateways

An Internet Gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet.
An Internet Gateway serves two purposes:
To provide a target in your VPC route tables for internetroutable traffic.
To perform network address translation (NAT) for instances that have been assigned public IPv4 addresses.
Internet Gateways (IGW) must be created and then attached to a VPC, be added to a route table, and then associated with the relevant subnet(s).
No availability risk or bandwidth constraints.
If your subnet is associated with a route to the Internet, then it is a public subnet.
You cannot have multiple Internet Gateways in a VPC.
IGW is horizontally scaled, redundant and HA.
IGW performs NAT between private and public IPv4 addresses.
IGW supports IPv4 and IPv6.
IGWs must be detached before they can be deleted.
Can only attach 1 IGW to a VPC at a time.
Gateway terminology:

Internet gateway (IGW) – AWS VPC side of the connection to the public Internet.
Virtual private gateway (VPG) – VPC endpoint on the AWS side.
Customer gateway (CGW) – representation of the customer end of the connection. - To enable access to or from the Internet for instances in a VPC subnet, you must do the following:
Attach an Internet Gateway to your VPC.
Ensure that your subnet’s route table points to the Internet Gateway (see below).
Ensure that instances in your subnet have a globally unique IP address (public IPv4 address, Elastic IP address, or IPv6 address).
Ensure that your network access control and security group rules allow the relevant traffic to flow to and from your instance.

Must update subnet route table to point to IGW, either:

To all destinations, e.g. 0.0.0.0/0 for IPv4 or ::/0for IPv6.
To specific public IPv4 addresses, e.g. your company’s public endpoints outside of AWS.

Egress-only Internet Gateway:

Provides outbound Internet access for IPv6 addressed instances.
Prevents inbound access to those IPv6 instances.
IPv6 addresses are globally unique and are therefore public by default.
Stateful – forwards traffic from instance to Internet and then sends back the response.
Must create a custom route for ::/0 to the Egress-Only Internet Gateway

NAT Gateway vs NAT Instance:

VPC Wizard
VPC with a Single Public Subnet:

Your instances run in a private, isolated section of the AWS cloud with direct access to the Internet.
Network access control lists and security groups can be used to provide strict control over inbound and outbound network traffic to your instances.
Creates a /16 network with a /24 subnet. Public subnet instances use Elastic IPs or Public IPs to access the Internet.

VPC with Public and Private Subnets:

In addition to containing a public subnet, this configuration adds a private subnet whose instances are not addressable from
the Internet.
Instances in the private subnet can establish outbound connections to the Internet via the public subnet using Network Address Translation (NAT).
Creates a /16 network with two /24 subnets.
Public subnet instances use Elastic IPs to access the Internet.
Private subnet instances access the Internet via Network Address Translation (NAT).

Security Groups

Security groups act like a firewall at the instance level.
Specifically, security groups operate at the network interface level.
Can only assign permit rules in a security group, cannot assign deny rules.
There is an implicit deny rule at the end of the security group.
All rules are evaluated until a permit is encountered or continues until the implicit deny.
Can control ingress and egress traffic.
Security groups are stateful.
By default, custom security groups do not have inbound allow rules (all inbound traffic is denied by default).
By default, default security groups do have inbound allow rules (allowing traffic from within the group).
All outbound traffic is allowed by default in custom and default security groups.
You cannot delete the security group that’s created by default within a VPC.
You can use security group names as the source or destination in other security groups.
You can use the security group name as a source in its own inbound rules.
Security group members can be within any AZ or subnet within the VPC.
Security group membership can be changed whilst instances are running.
Any changes made will take effect immediately.
Up to 5 security groups can be added per EC2 instance interface.
There is no limit on the number of EC2 instances within a security group.
You cannot block specific IP addresses using security groups, use NACLs instead.
You can associate a network ACL with multiple subnets; however, a subnet can only be associated with one network ACL at a time.
Network ACLs do not filter traffic between instances in the same subnet.
NACLs are the preferred option for blocking specific IPs or ranges.
Security groups cannot be used to block specific ranges of IPs.
NACL is the first line of defense, the security group is the second line.

Network ACL's

Network ACL’s function at the subnet level.
The VPC router hosts the network ACL function.
With NACLs you can have permit and deny rules.
Network ACLs contain a numbered list of rules that are evaluated in order from the lowest number until the explicit deny.
Recommended to leave spacing between network ACL numbers.
Network ACLs have separate inbound and outbound rules and each rule can allow or deny traffic.
Network ACLs are stateless, so responses are subject to the rules for the direction of traffic.
NACLs only apply to traffic that is ingress or egress to the subnet not to traffic within the subnet.
A VPC automatically comes with a default network ACL which allows all inbound/outbound traffic.
A custom NACL denies all traffic both inbound and outbound by default.
All subnets must be associated with a network ACL.
You can create custom network ACL’s. By default, each custom network ACL denies all inbound and outbound traffic until you add rules.
Each subnet in your VPC must be associated with a network ACL. If you don’t do this manually it will be associated with the default network ACL.

create static website with S3 and CloudFront

KrushiVasani — Sun, 22 Jan 2023 15:10:45 +0000

Setting up a static website using Amazon CloudFront is a great way to improve the performance and reliability of your website. Not only does CloudFront distribute your content globally via edge locations, but it also allows you to easily set up a custom domain, and use HTTPS for secure connections. In this detailed blog post, we will walk you through the process of setting up a static website using CloudFront and an S3 bucket.

Step 1: Create an Amazon S3 bucket
The first step in setting up a static website using CloudFront is to create an S3 bucket. S3 stands for Simple Storage Service, and it's a fully-managed object storage service that allows you to store and retrieve any amount of data. To create an S3 bucket, go to the S3 console and click on the "Create Bucket" button. Give your bucket a unique name, select the region where you want to create the bucket, and click "Create".

click on save changes.

Step 2: Upload your website's files to the S3 bucket
Once you have created your S3 bucket, you can upload your website's files to it. To do this, simply drag and drop your files into the bucket or use the "Upload" button to select them from your computer. Make sure that the "Index Document" and "Error Document" fields are set to the appropriate files for your website (usually "index.html" and "error.html" respectively).
index.html
<!DOCTYPE html> <html> <head> <title>Welcome</title> </head> <body> <h1>Welcome to our website!</h1> </body> </html>
error.html
<!DOCTYPE html> <html> <head> <title>Error</title> </head> <body> <h1>404 Error</h1> <p>Sorry, the page you requested could not be found.</p> <p>Please check the URL and try again.</p> </body> </html>

It's also a good practice to set the public access level to the bucket and files, this way the files can be accessed publicly and CloudFront can use them.

Step 3: Create a CloudFront distribution
Now that your website's files are stored in an S3 bucket, you can create a CloudFront distribution. CloudFront is a content delivery network (CDN) that allows you to distribute your content to users around the world. To create a CloudFront distribution, go to the CloudFront console and click on the "Create Distribution" button. Select the "Web" delivery method and select your S3 bucket as the origin.

In the create distribution page, Under “Default Cache Behavior Settings”, make sure that the “Viewer Protocol Policy” is set to “HTTP or HTTPS”.

Now you can see the website using the cloudfront's Distribution domain name.

In conclusion, by using Amazon S3 and CloudFront, you can easily set up a static website that is fast, reliable, and globally available. By following these detailed steps, you can have your static website up and running in no time and served over HTTPS, which is a bonus for security.

Serverless Video Transcoder

KrushiVasani — Wed, 17 Aug 2022 06:51:00 +0000

| AWS(Amazon Web Service)

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. In computing, a client can be a web browser or desktop application that a person interacts with to make requests to computer servers. A server can be services such as Amazon Elastic Compute Cloud (Amazon EC2), a type of virtual server.

client- server model
For example, suppose that a client makes a request for a news article, the score in an online game, or a funny video. The server evaluates the details of this request and fulfills it by returning the information to the client.

| Benefit of Amazon Web Service (AWS):

                   Benefit of AWS

| Service used in AWS

AWS Services

| Overview

Video Stream is server less video sharing web application which is being developed using node.js and some specific cloud services of amazon web services(AWS). Scenario like any youtuber upload his video and all subscribers can watch the videos in different qualities.

| Hardware Requirement:

Hardware that is essential in our system is:

A. Laptop/Computer B. Minimum 4 GB RAM

| Software Requirement:

Software that is essential in our system is:

A. Node JS B. NPM Library C. Visual Studio D. Browser E. Windows / Linux Operating system F. AWS Account G. Auth0 API

| Implementation:

Figure A:Home Page
Figure A This is our website and and here’s the first page without sign in or login .As you can see on the screen there is a four default video we have added to the firebase for the sample purpose.

                Figure B: Firebase Console

Figure B show this is firebase console. Where each video get it’s unique token id. Here the transcoding false is like boolean variable whenever the transcoding is in process it is set to the true and when it is transcoded it is set to the false and shown to the website.

Figure C: Login- Signup
Figure C show this is the login signup part for that we have used the third party Auth0 API. First of all user can login application via email and password. If user is newer then press “sign up” text and you will first create account via email id and password. Another feature like user can directly sign in using google id. So that no need to enter email and password.

Figure D: Profile Information
Figure D Shows the Raw Information of user which is fetched from the Auth0 API using the AWS Lambda Service. Which Provides the information like given_name, email ,Updated time, Nickname, Profile picture, email verified, locales etc.

Figure E: Animation while Transcoding video
In center of the page there is one plus button. Using that user can upload the video from local machine to s3 bucket. Whenever the video is transcoding in the background through the pipeline it will shown the animation on the webpage.

Figure F: Website
After completion of the Transcoding video will go to the destination bucket. From destination bucket we have fetched the video to the webpage using scripts.

Figure G: S3 bucket
Figure G shows the different quality Transcoded video which is uploaded by the user from local machine. From this user can get three different quality video like 480P,720p,720p (Web Friendly),1080p.

Figure H: Transcoded video
In S3 bucket there is a separate link of each quality video from that link we can play the video in any of the browser.

|Flow Diagram:

Flow Diagram

| Conclusion

These is amazing project because of it is server less project. In this project we have used some services of Amazon Web Services such as AWS Lambda, AWS Simple Storage Service, AWS API Gateway, AWS Multimedia Service (AWS Transcoder) and learn lots of special services provided by Amazon. From this project we could understand how server less system works and benefits of server less computing. During this project we learn lots of new things about sever less and AWS services

| Github Link:

https://github.com/KrushiVasani/Video-Transcode

Create a Load Balanced WordPress Website

KrushiVasani — Tue, 16 Aug 2022 11:44:00 +0000

| Amazon Lightsail

Lightsail is an easy-to-use virtual private server that offers everything which is needed to build a cost-effective website with a monthly plan. This approach is suitable for prototyping and test environments as well for blogs, custom sites, and e-commerce applications. In this blog, you can explore everything about Amazon Lightsail including a business scenario to spin up a WordPress site quickly with a customized look and feel, least configuration efforts, and minimal costs.

| Lightsail instance
Lightsail instance is a virtual private server (VPS) that lives in the AWS Cloud. Use your Lightsail instances to store your data, run

your code, and build web-based applications or websites. Your instances can connect to other AWS resources through both public (Internet) and private (VPC) networking. You can create, manage, and connect easily to instances right from the Lightsail console.

Prerequisites
You need to have an AWS account and some basic knowledge of working with AWS services. Following AWS services will be utilized throughout this guide.

Amazon Lightsail
Wordpress
Amazon S3
Elastic Load Balancer

*| Wordpress Website in Lightsail
*

              Figure 1: Wordpress instance

In the Above Figure, two Wordpress instances are there with the configuration like 512MB,1vCPU etc. Also created two Static IP
and attached with each instance.

                Figure 2 : Load Balancer

The load balancer is configured in the same region in which two Wordpress instances are launched. After that, both instances are attached to the load balancer under Target Instance. The load balancer will give one DNS name by which you can access the website and response will come from any of the Wordpress instances.

                    Figure 3: Website

This is the Homepage of the website where the default hello world page is already given. All the recent posts will be shown on the menu and by clicking it you can open that particular blog.

          Figure 4: Wordpress's Admin Dashboard

This is the Admin dashboard of Wordpress. From here you can configure many things like Add new pages, Menus, Add necessary plugins, change the theme of the website.

                  Figure 5: Website

Here's the blog of the Flutter Ebook app which is runnig on the Wordpress instance.

Textract using AWS

KrushiVasani — Tue, 16 Aug 2022 11:27:00 +0000

AWS Textract is a document text extraction service.

“Amazon Textract is based on the same proven, highly scalable, deep-learning technology that was developed by Amazon’s computer vision scientists to analyze billions of images and videos daily. You don’t need any machine learning expertise to use it” — AWS Docs

This post will provide a walkthrough of several use cases of AWS Textract service using AWS Lambda with Python implementations. Mainly,

Extracting Text from an S3 Bucket Image.

Prerequisites
You need to have an AWS account and some basic knowledge of working with AWS services. Following AWS services will be utilized throughout this guide.

Lamda Service
Textract Service
Simple Storage Service
Identity Access Management Service

| Extracting Text from an S3 Bucket Image

               Figure 1: Flow Diagram

When the user uploads the image to the S3 bucket and uploaded it successfully then the Lambda trigger will be activated and the Lambda function will call the Amazon Textract Service which will extract the text from 1mage. Textract will pass the extracted text to the Lamda. Now Lambda function will generate one text file of the same name as the Image name and write the extracted text into text document. This text file will be stored in the S3 bucket.

               Figure 2: Lambda Function

In the above Figure, one Lambda function is created named getTextFromS3Image.With the Lambda function, the S3 bucket trigger is attached. It will activate on all objects create events with the suffix .png.

             Figure 3:Uploading Image to S3

From the S3 bucket by clicking add files you can attach the png file and then upload it. On the completion of uploading the status will be changed to succeeded.

                 Figure 4: S3 Bucket

This is the image of the S3 bucket in which I have uploaded 2 png photos for testing the Text extraction.Within 4-5 Seconds the Text file of both the png image is generated and saved as the same name of image with .txt format.

               Figure 5: Extracted Text

Here's the final output on the left side one png file is there and on the right side is the extracted text file of the image. As you can see Amazon Textract has correctly identified all the words of the image. Now you can use this text anywhere as per your need.