DEV Community: Nishita Mohanty

S3 trigger AWS Lambda

Nishita Mohanty — Sun, 15 Oct 2023 05:05:47 +0000

AIM -
Here, The provided AWS Lambda function is part of a serverless project designed to automate the process of copying a specific object from one Amazon S3 bucket to another.

What is AWS Lambda function?
AWS Lambda is a serverless compute service offered by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers. With AWS Lambda, you can execute your code in response to various events, such as HTTP requests, changes to data in an S3 bucket, updates to a database, or custom events generated by other AWS services.

Brief description:-
The goal is to streamline the data transfer process and ensure that a particular object is always available in a destination bucket for various purposes such as data aggregation, sharing, backup, or further processing.

In order to achieve this via AWS Lambda function, we need to make our source S3 bucket as the trigger which triggers the Lambda function, which then acts according to the code written in it (we have written our code in Python 3.11 language). In our code we have mand performs the following tasks:

-Extracts the object key, source bucket name, and destination bucket name.
-Copies the specified object from the source S3 bucket to the destination S3 bucket.
-Records successful copy operations in the function's logs.
-Handles errors and exceptions, logging them for debugging and monitoring purposes.
In our Lambda function we have attached IAM role too in which we have given full access to S3 bucket.

Services Required:-

AWS Lambda Function: The core component of the project is an AWS Lambda function, which is written in Python 3.11 and uses the Boto3 library to interact with AWS services.
2 S3 buckets
- Source S3 Bucket: The source S3 bucket contains the data that needs to be copied.
- Destination S3 Bucket: The destination S3 bucket is where the object will be copied to.
AWS IAM: Gives full access of S3.

Diagram:-

Steps:-

- Create 2 S3 buckets and name them as source and destination buckets respectively.
(Note: Follow the standard naming convention)

Create the destination bucket following the same procedure as the source bucket.

- Create an IAM role and give full-access of S3 Bucket.

- Create a Lambda function and attach the IAM role with it.

- Write the code in the code section and then test it.

- To test the code click on "test" then click on create event.

- After successful test case attach a trigger to the lambda function i.e. the source S3 bucket.

- Then deploy the code.
(At this step the process of syncing is almost completed)

- Now, to test the working (syncing of both the S3's) upload a file in the source S3 bucket.

-At this step we can check on our destination S3 bucket that the file which we had uploaded in our source s3 bucket can be seen here.

Hence, its shows that our project is running successfully.

Note:-
Clearly mention the object name which you want to upload.
Do, mention its extension file name (example .png, .jpg etc.)

TRANSIT GATEWAY

Nishita Mohanty — Fri, 08 Jul 2022 12:57:51 +0000

How to connect multiple VPC's together??
There are 2 ways to connect multiple VPC's together:-
-> VPC PEERING
-> TRANSIT GATEWAY

What is Transit Gateway??
Transit Gateway is a service that enables us to connect their VPCs together as well as connect VPCs and their on-premises networks to a single gateway. It is used to establish communication in much simpler way.

What is VPC peering?
A VPC peering connection is a networking connection between several VPCs that enables us to route traffic between them using private IPv4 addresses or IPv6 addresses.

Why Transit Gateway is preferred over VPC Peering??
Drawback is if number of network increases then, our VPC peering also increases. Then it creates confusion.
There is a formula to establish peering connection:-
[n(n-1)]/2, where n represents number of networks/VPCs.

So, if we have 6 networks then according to the formula we are required to have 15 peering connections which is time consuming and creates confusion.

So, for this reason transit gateway is preferred over VPC peering.

DIAGRAM

How to establish connection among VPC's using transit gateway??
Step 1- Create 3 VPC's.

Step 2- Create SUBNETS for respective VPC's

Step 3- Make the subnet of VPC-1 as public subnet and make an Internet gateway and attach it to VPC-1 as we need to test the connection is communicating with other VPC's or not.

STEP 4- Create route tables for each subnet.

Now, create two more route tables for VPC-2 and VPC-3, then associate respective subnets.

STEP 5-
Now we want to test VPC-1 can communicate with VPC-2 and VPC-3 so for that we need to launch servers in respective VPC's.
**For VPC-2 and VPC-3 the servers are not required to have a public IP address.

STEP-6
We need Transit gateway to establish connections among the VPC's,
So, go to transit gateway,

STEP 7- Go to Transit Gateway Attachments,

Follow the above steps and make transit-gateway attachments for VPC-2 and VPC-3.

And wait for the attachments' state to be Available .

STEP 8- Go to Route table of VPC-1 and click on edit routes.

Now follow, the above steps and edit routes of VPC-2 and VPC-3.
->In route table of VPC-2, add CIDR range of VPC-1 and VPC-3.
->In route table of VPC-3, add CIDR range of VPC-1 and VPC-2.
Then click on save changes.

Now, we will use the server connected to VPC-1 (which is connected to internet gateway). From this server we will try to ping the other 2 servers. If pinging is successful then the established connection is working perfectly.
So, connect server of VPC-1 in Xshell.
then,
-> sudo su
-> cd
Then type ping 8.8.8.8 (to check it is connected to internet or not).

Now,
->type ping (private IP of server of VPC-2)
->type ping (private IP of server of VPC-3)

If we see pinging occurs then we are successful in building a connection between VPC-1 and other two VPCs.

We can see the pinging is happening so, the connection is successful.

SITE TO SITE CONNECTIVITY

Nishita Mohanty — Sat, 04 Jun 2022 07:53:01 +0000

How to establish site to site connectivity between your AWS account and client's on-premise network? How the setup is done?

STEP 1:-
CREATE YOUR VPC, and configure it using subnets, internet gateway, route tables etc.(use nat-gateway if you are using private subnet)

STEP 2:-
Go to Virtual Private Gateway.

Then click on Create Virtual private gateway -> Give a name -> then click on create.

Now attach it to your VPC.

STEP 3:-
Go to Customer Gateway.

Create -> Give a name tag -> Then allocate IP( This IP should be given by the on-premise client) -> Then click on create.

STEP 4:-
Go to Site to Site VPN Connection.

Then click on create -> give name -> select virtual private gateway -> select customer gateway -> change the routing options from dynamic to static -> static preferences (give the subnet details of the customer) -> click on create Vpn connection.

Then select the Vpn connectivity and wait for the status gets active and we can see that in the tunnel details the status is down.

STEP 5:-
We need to configure the route table of VPC.

Choose the route table you want to propagate.

Then edit-> route propagation-> check on enable and save.

Then again go to VPN and download configuration and set the values accordingly.

The text file which is downloaded consists of all the details of the two tunnels and it is given to the client so they will establish connectivity at their end.
When the connection would be done the tunnel's status will be up.
Which means we are successful in creating site to site connectivity.

NOTE:-
We need to ask the customers 2 things-

Public IP of gateway .
Subnet details.

APPLICATION MIGRATION

Nishita Mohanty — Tue, 10 May 2022 14:15:05 +0000

WHAT IS APPLICATION MIGRATION?
It is the process of moving data, applications, or other business components from an organization’s on-premises infrastructure to the cloud, or moving them from one cloud service to another cloud platform.

So how can we migrate our data or applications server from one AWS account of a particular region to another AWS account of same or some other region?

So this is the server (in place of on-premise I have used another AWS account)(application migration) in Mumbai region of one AWS account and we want to migrate it to the N.Virginia region of some other AWS account.

Connect the server through xshell and install apache.

Now we are required to host a website from our server which we want to migrate.

The below is the sample website (downloaded from free-css.com and sent to the server via WinScp) which we have hosted and connected it through the public IP of the server.

Now go the the AWS account where you want to migrate your data.
1st type application migration in the search bar and then select AWS application migration service.

Then click on add servers.

Then select your desired operating system. Here, we have chosen Linux.
Then select your replication preferences.
then create an IAM user and provide its access key and secret key.

Then copy the commands present in point 4 and 5 and then paste in Xshell (one by one) where your server is connected.

Then go back to the source server and the click on source server's name.

As we scroll down we can see the replication initiation steps.

Then go to launch settings and click modify (in EC2 launch template).

Give a template name.

Then give the instance type of your choice and select your security group (by doing necessary changes) .

Then configure storage to gp2.

Then add a new tag.

Then in advanced details click on the checkbox of enable IPv4.

Then go to launch templates and then select and set it as default version.

After the replication process will get over then an instance will launched in your desired AWS account where you wanted to migrate.

Then again select the source server and then click on launch test instance.

Then another instance will get launched and after it gets stopped then the real replication server will be launched which we have attach an elastic IP and then after opening its public IP in the browser we can get to see the that the website which we have deployed in the server of the AWS account from where we have migrated the data, is up and running here.

Then again select source server and then click on mark as ready for cutover.

Then launch cutover instance.

In the job log we can see the details of work done in the backend.

Then finalize cutover.

then mark as archived.

Then after that we can select the server's IP (attach an elastic IP) and can see the website is up and running.

This is the process by which we can migrate data from server of one AWS account to the server of another AWS account.

Open Vpn

Nishita Mohanty — Fri, 08 Apr 2022 15:54:57 +0000

How to connect a private server(i.e. the server without a public IP address)?
Well there is a solution for the above concern, which leads us to learn a new concept named OpenVpn.

So, what is OpenVpn?
Well OpenVpn is a type of computer network that provides users with remote access to resources on the internet.

So how can we use it to connect a server?
Well the solution to the above query is given below.

STEP 1. Launch an instance without public IP.

So, to connect the server we have to setup a VPN connectivity.
**
STEP 2. Go to **launch instances then in the search bar search for openvpn -> click on 44 results.

Then select the 1st openvpn.

STEP 3. Then launch another instance with public IP enabled -> create new key pair -> launch instances.

Now, we can see that both the instances which we have launched have same VPC ID.

STEP 3. Now we need to connect the second server i.e. with public IP to Xshell -> Then type yes and then enter multiple times.

Then openvpn will initialize

Then after some time again paste the same command (SSH client) and of root write openvpnas -> then again connect with the key pair.

Now already a openvpn (user) is created so now we have to change password.
So, type sudo passwd -> enter
Give password

Then scroll up and copy link of Admin and Client and paste it in notepad.

Now close Xshell.

Now, copy the admin's url and paste it in browser -> login to openvpn -> Agree

Now again go to instances.
Then connect the instance (without public IP) through Xshell

But we will notice that it wont get connected.

STEP 4. Now open OpenVpn -> connect -> next (multiple times) -> agree ok -> Give url i.e. copy the IP of client link and paste -> next -> accept -> username : openvpn ->Password: give the password which u have set earlier in Xshell -> Accept -> Connect -> Password -> OK

Then go to Xshell and try to connect the instance (without public key) again. Now, u will be able to connect/access the instance.

So, this is the way by which we can connect servers without public IP address through openvpn.

IAM USER

Nishita Mohanty — Thu, 10 Mar 2022 15:09:29 +0000

What is an IAM user?
An IAM User is an entity created in AWS that provides a way to interact with AWS resources. We don't need to give the root user account and password to interact with AWS services. So, an IAM user is created for the clients to use and interact with AWS resources without hampering any security.

How to create IAM user ?

(Suppose, you want to create an IAM user for admin access only)

1st - Login through root user.
2nd - Now, go to IAM user.
3rd - Click on "users" on the left column.
4th - Now press add users.
5th - Give username as "admin".
6th - Now select both "access key" and "password".
7th - Select custom password at console password and give a strong
password.
8th - Tick on the check box of "Required password reset".
9th - Then click on next: Permission
10th - Click on attach existing policy.
11th - Tick on the check box of "administration access", then
click on next: tags.
12th - Adding tag is optional, then next.
13th - Then you can see username - Admin: Then click on create
user.
14th - Then download .csv file, then click on close.

How to login into IAM user?

Go to AWS console -> Click on the IAM user -> Enter 12 digit account ID -> Give the username - Admin (whatever username you have set while creating the IAM user) -> Enter the password which you have set.

Then when you log in it will ask to change password.
After changing password, you are done and you can use the account.

So, this is how we can create IAM user for particular user groups.

How to attach a volume to an instance of different availability zone?

Nishita Mohanty — Fri, 21 Jan 2022 12:34:01 +0000

So, when we want to attach a volume of different availability zone to an instance then the concept of SNAPSHOT comes into the picture.
Suppose you have an instance, in ap-south-1a availability zone.!

Now, suppose you have a volume of 15GB in different availability zone let it be in ap-south-1b.

Then go the volume(which you want to attach) -> go to actions -> create snapshot -> add a description -> add tag (value = description name)-> create snapshot.

Now, go to Snapshots you will find that a snapshot of your volume is being created.

Now, we need to create volume from the snapshot.
So, go to actions -> click on create volume from snapshot -> give size according to your wish -> change the availability zone according to the instance with which you want to attach the volume -> Give tag -> Create volume.

Now, go to volumes you will find a new volume is created.

Now, you can add the new volume to your instance by following the below steps.
Step 1: Go to Xshell then type sudo su then type lsblk

We will fing a 8G volume is mounted into the root (/).

Step 2: Then we need to add the volume to our instance so, go to
your AWS console and then go to the volume section. Click on the volume you want to attach to your instance then go to actions -> click on attach volume, then follow the below steps.

Step 3: Then go to Xshell, type lsblk, it will show that the
extra volume is attached to the instance.

Now, type df -h, we will find that the new volume is not showing on the instance details which means the new volume is attached but not mounted into the instance.

So, to mount the volume here are the following steps:
Step1 : Type mkdir /india (making a directory)
Step2 : Type cd /india (changing directory)

Then type mount /dev/xvdf /india/. Now, our EBS volume is mounted on our AWS instance.
We can also check it by typing df -h, then we will see the 15G storage used in the our directory India.

This is the whole process by which you can connect the volume and instance of different availability zones.