DEV Community: Rohit Shrivastava

#4 Shorticles: AWS Glue’s integration with Apache Spark

Rohit Shrivastava — Tue, 16 May 2023 08:13:11 +0000

When we start writing the program for glue first time, few lines which we first encountered are:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
glueContext = GlueContext(SparkContext.getOrCreate())

The above lines of code are used to set up the necessary dependencies and initialize the GlueContext object for AWS Glue ETL (Extract, Transform, Load) jobs using Apache Spark.

Let’s break down the code line by line and understand what exactly their meaning is:

import sys: This line imports the sys module, which provides access to system-specific parameters and functions.

from awsglue.transforms import *: This line imports all the classes and functions from the awsglue.transforms module. The awsglue.transforms module provides a set of transformation functions that can be used in AWS Glue ETL jobs.

from awsglue.utils import getResolvedOptions: This line imports the getResolvedOptions function from the awsglue.utils module. The getResolvedOptions function is used to retrieve the AWS Glue job arguments and their values.

from pyspark.context import SparkContext: This line imports the SparkContext class from the pyspark.context module. SparkContext is the entry point for creating a connection to Apache Spark.

from awsglue.context import GlueContext: This line imports the GlueContext class from the awsglue.context module. GlueContext is a high-level wrapper around Apache SparkContext that provides additional AWS Glue-specific functionality.

from awsglue.job import Job: This line imports the Job class from the awsglue.job module. The Job class represents an AWS Glue job and provides methods to configure and execute the job.

glueContext = GlueContext(SparkContext.getOrCreate()): This line creates an instance of GlueContext by passing the SparkContext.getOrCreate() method as an argument.

SparkContext.getOrCreate() returns an existing SparkContext instance or creates a new one if it doesn't exist. The glueContext object is then used to interact with the AWS Glue environment and perform ETL operations.

With these lines of code, you can leverage AWS Glue’s integration with Apache Spark to write and execute ETL scripts for data processing and transformation in the AWS Glue service.

#3 Shorticle: REGEX in Python

Rohit Shrivastava — Tue, 16 May 2023 08:11:44 +0000

Regular expressions, often referred to as regex or regexp, are powerful tools for pattern matching and text manipulation. Python provides a built-in module called re that allows you to work with regular expressions.

To start using regular expressions in Python, you need to import the re module:
import re

Now let’s dive into the details of using regular expressions in Python:

Matching Patterns: The re module provides several functions for matching patterns:
re.search(pattern, string): Searches the string for a match to the pattern and returns the first occurrence.

re.match(pattern, string): Checks if the pattern matches at the beginning of the string.

re.findall(pattern, string): Returns all non-overlapping matches of the pattern in the string.

re.finditer(pattern, string): Returns an iterator yielding match objects for all matches.

Basic Patterns: Literal characters: You can search for literal characters by including them directly in the pattern. Metacharacters: These are special characters that have special meanings in regular expressions, such as ., *, +, ?, ^, $, \, [ ], |, (, and ).

Character classes: You can define custom character classes using square brackets [ ]. For example, [a-z] matches any lowercase letter.

Predefined character classes: There are several shorthand character classes like \d (digits), \w (word characters), \s (whitespace characters), and their negations \D, \W, \S, respectively.

Quantifiers:
*: Matches zero or more occurrences of the preceding element.
+: Matches one or more occurrences of the preceding element.
?: Matches zero or one occurrence of the preceding element.
{n}: Matches exactly n occurrences of the preceding element.
{n,}: Matches at least n occurrences of the preceding element.
{n,m}: Matches at least n and at most m occurrences of the preceding element.
Anchors:
^: Matches the beginning of a line.
$: Matches the end of a line.
Groups and Capturing:
( ): Creates a group. Groups can be used for capturing substrings and applying quantifiers.
\number: Backreference to a captured group by its number.
Special Sequences:
\b: Matches a word boundary.
\B: Matches a non-word boundary.
\d: Matches any decimal digit.
\s: Matches any whitespace character.
\w: Matches any alphanumeric character and underscore.
Flags: The re module supports optional flags that modify the behavior of the regular expression matching. Some common flags include:
re.IGNORECASE or re.I: Performs case-insensitive matching.
re.MULTILINE or re.M: Allows the ^ and $ anchors to match at the beginning and end of each line.
re.DOTALL or re.S: Allows the . metacharacter to match any character, including a newline.

Here’s a simple example that demonstrates some of the concepts mentioned above:
import re pattern = r"gr.y" string = "The gray cat is sitting on the green mat." match = re.search(pattern, string) if match: print("Match found:", match.group()) else: print("No match found.") matches = re.findall(pattern, string) prin
Match found: gray

Explanation:
The regular expression pattern gr.y matches any three-character sequence starting with "gr" and ending with "y", where the dot . represents any single character.
The re.search() function is used to find the first occurrence of the pattern in the string.
Since the string contains the word "gray," which matches the pattern, the re.search() function returns a match object.
The match.group() method retrieves the matched substring.
Therefore, the output is "Match found: gray".
Note: If you want to find all occurrences of the pattern in the string, you can use the re.findall() function. In this case, the output would be:
['gray']
This returns a list containing all the non-overlapping matches of the pattern in the string.

#2 Shorticle: How to read data from S3 bucket using python

Rohit Shrivastava — Sun, 14 May 2023 06:31:20 +0000

To read data from an Amazon S3 bucket using Python, you can utilize the boto3 library, which is the official AWS SDK for Python. Follow the steps below to get started:
Install the boto3 library by running the following command in your Python environment:

pip install boto3

Import the required modules in your Python script:

import boto3

Create a boto3 S3 client or resource object by providing your AWS access key ID and secret access key (or by using an IAM role with appropriate permissions). For example:

s3_client = boto3.client('s3', aws_access_key_id='YOUR_ACCESS_KEY', aws_secret_access_key='YOUR_SECRET_KEY')

Use the client or resource object to interact with your S3 bucket. To read an object (file) from the bucket, you can use the get_object() method:

response = s3_client.get_object(Bucket='your-bucket-name', Key='your-object-key')

In the above code snippet, replace 'your-bucket-name' with the name of your S3 bucket, and 'your-object-key' with the key (path) of the object you want to read.
Access the content of the object through the response object. For example, to read the object's content as text:

object_content = response['Body'].read().decode('utf-8')

You can also read the content as bytes by omitting the decode('utf-8') part. Process or use the data as needed. Here’s a complete example that demonstrates reading a text file from an S3 bucket:

import boto3
# Create an S3 client
s3_client = boto3.client(‘s3’, aws_access_key_id=’YOUR_ACCESS_KEY’, aws_secret_access_key=’YOUR_SECRET_KEY’)
# Read an object from the bucket
response = s3_client.get_object(Bucket=’your-bucket-name’, Key=’your-object-key’)
# Read the object’s content as text
object_content = response[‘Body’].read().decode(‘utf-8’)
# Process or use the content as needed
print(object_content)

Remember to replace 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', 'your-bucket-name', and 'your-object-key' with your actual credentials and bucket/object information.

Note: It’s recommended to follow AWS security best practices and avoid hardcoding your access keys in your code. Instead, consider using AWS credentials stored in a secure manner, such as environment variables, AWS profile configuration, or EC2 instance roles, depending on your execution environment.

#1 Shorticle: Why AWS Lambda Throttles?

Rohit Shrivastava — Sun, 14 May 2023 06:24:14 +0000

AWS Lambda throttling refers to the process of limiting the number of concurrent executions or the number of requests per second that a Lambda function can handle. Throttling is designed to prevent excessive resource consumption and ensure the overall stability and performance of the Lambda service.
When a Lambda function receives a high number of requests or exceeds the allocated concurrency limit, AWS may enforce throttling by limiting the rate at which new requests are accepted or the rate at which concurrent executions are allowed. Throttling can occur at different levels:

Account Level Throttling: AWS imposes account-level limits on the total number of concurrent executions across all Lambda functions within an AWS account. If the account limit is reached, any new requests or invocations may be throttled until the load reduces or the concurrency limit is increased.

Function Level Throttling: Each individual Lambda function can have its own concurrency limit, which restricts the maximum number of simultaneous executions for that function. When the function’s concurrency limit is reached, additional requests or invocations may be throttled until existing executions complete or the limit is increased.

Provisioned Concurrency Throttling: If you have provisioned concurrency enabled for your Lambda function, there is a separate limit for the number of provisioned concurrency instances. If this limit is reached, additional requests may be throttled until a provisioned concurrency instance becomes available.

When a Lambda function is throttled, AWS typically responds with a specific error code (e.g., “TooManyRequestsException” or “RequestLimitExceeded”), indicating that the function is currently unable to process the request. Throttled requests may need to be retried or handled accordingly in your application logic.

To mitigate throttling, you can take the following actions:
Monitor your function’s concurrency usage and adjust the concurrency limit based on your workload requirements.
Implement proper error handling and retry mechanisms in your application code to handle throttling errors and resubmit requests if necessary.

Consider using provisioned concurrency to pre-warm your function and ensure a specific number of concurrent instances are always available to handle requests.

Optimize your code and configuration to reduce the time and resources required for each function execution, allowing more efficient use of the available concurrency.

It’s important to note that AWS Lambda provides high scalability and performance, but it’s crucial to design your applications to handle and recover from throttling conditions to ensure smooth operation under varying workloads.
Do read about Concurrency Limit in AWS Lambda

Serverless! Is it really without servers or something else????

Rohit Shrivastava — Sun, 28 Aug 2022 11:57:00 +0000

Introduction:
In the recent years the computing technology has been evolving at a phenomenal rate. Data storage has become very affordable and computing power has vastly increased and this has allowed us to build applications that leverage these advancements in the computing space.

Cloud computing for example has made it possible for us to build such cutting-edge applications at a small amount of cost, not only for big ventures but also for small ventures or individual people. Public cloud providers like Amazon AWS, Microsoft Azure, Google Cloud Platform have removed the entry barriers and made computing very easy and affordable for common people.

We know the modern-day applications demand high level of power, high level of compute power as well as speed let's take an example if your youtube app doesn’t show you what you need to see as soon as you open the youtube app, most of you might simply close the app and move on. It means the application should be fast and responsive and at the same time, it must be scalable with almost no downtime. Serverless Computing is the new trend in cloud computing which attempts to solve many of these challenges.

Serverless in Brief:
Serverless computing will help you build the next generation of systems that can handle workloads on demand and scale indefinitely without having to provision or manage any servers. Now, if we look at the traditional architecture, there are several activities and tasks that you must perform in order to run any particular application.

First, you need to create and setup servers, install operating systems, install and setup databases, manage software patches and hardware updates, manage capacity and scaling, manage high-availability through load-balancing and so on and of course, this server infrastructure and the computing power has its own costs, which are often substantial. On the other hand, with Serverless Architecture, all these tedious tasks of managing the underlying infrastructure are abstracted away from us.

So in a way when you use serverless architecture, you won’t really feel like you are using any high-end servers, because you’re not the one who is actually taking care of it.
That’s the reason this approach is called as Serverless.

That means, Serverless Computing still uses servers, but you no longer have to worry about managing them or worry about the uptime or availability or anything that has to do with the infrastructure part. With Serverless, you kind of get to be more productive. You can completely focus on writing your code without having to worry about the underlying support system. So you simply upload your code to the cloud provider and it can be run on-demand as and when needed.

With Serverless(e.g. AWS Lambda), you write your code in the form of functions, just like you’d write a function in any programming language and these functions then run in the cloud. So, you segregate your application logic in to small independent functions or microservices and upload them to the cloud provider. These functions are stateless and can be invoked in response to different events. These events could be file uploads, database updates, in-app activity, API calls, website clicks, or sensor outputs just like those from IoT devices and so on and all this can be accomplished without having to manage any infrastructure or having to size, provision or scale a bunch of servers, and without having to worry about performance and availability. This seriously allows us to lesser focus on our application code and not get caught up in multitude of infrastructure issues.

Let’s discuss in a more technical terms, these serverless functions often run in Docker-like containers and hence several instances of these functions can be run concurrently, like in a Dockerswarm, thus making them highly scalable. If you’re new to these terms like Docker or Container, nothing to worry. You really don’t need to know about these terms to be able to build Serverless applications.

All this containerization happens behind the scenes and is completely taken care of by the cloud provider. Just for your information though, containers are a lightweight alternative to virtualization they allow you to run your code in isolated preconfigured environments.

So, in a nutshell, Serverless means “Event-driven Computing” using “Small independent stateless functions” running inside “containers in the cloud”. That’s the short definition of what Serverless is all about. So, you have your code running on the cloud platform, AWS in our case. Whenever the triggering event occurs, the cloud platform spins up a container or initializes a container, loads the function in it and executes the function. And this happens almost instantaneously, thereby allowing us to build applications that respond quickly to new information and thus enhance user experience. Once the function completes execution, it optionally returns a response back to the caller, and then finally exits or shuts down.

Pros

No necessity of server management.
Scales Quickly.
High availability and fault tolerance.
Pay as per use.
Integration of services with other AWS services to send text notifications, emails.

Cons

Level of control.
Execution time.

AWS offers Serverless Services in various categories:

Serverless Compute Services:

AWS Lambda
AWS ECS
AWS Fargate
AWS EKS

Serverless Application Integration Services:

Amazon EventBridge
Amazon SQS
AWS Step Functions
Amazon API Gateway
Amazon SNS
AWS AppSync

Serverless Database Services:

Amazon S3
Amazon RDS Proxy
Amazon DynamoDB
Amazon Aurora Serverless

Serverless Data/Analytics Integration Services:

AWS Glue
AWS Athena
AWS Redshift
Amazon Kinesis

Serverless Debugging Tools:

AWS X-Ray
AWS Cloudwatch

Serverless User Authentication & Mnagement Services:

AWS Cognito

For more information you can refer AWS Documentation

From the next articles onwards, I’ll discuss about all the above services in my SHORTICLES and detailed articles with their applications.

Thanks for reading!!!