Streamlining Your Data Transfer

Nathaniel Agbenyenu — Wed, 05 Apr 2023 10:30:16 +0000

Security threats have been one of the major challenges faced by many organisations and over the years, huge sums of monies have been lost as a result of one security threat or the other.

According to this report by Forbes, the top 10 security threats of 2022 include:

Credential Reuse Attack
Insider Threat
Man-in-the-Middle Attack
Phishing
Ransomware
Watering Hole Attack
Spyware
Social Engineering Attack
DDoS Attack
Cloud Crypto-mining

The report indicates that these 10 are the most malicious of **Top 50 Security Threats** that must be taken note of.

This post discusses a way of mitigating the third threat; Man-in-the-Middle Attack

A Man-in-the-Middle happens when an attacker places himself between two targets and relays messages for them. The communicating parties may think there is a secure and direct communication between them but the attacker eavesdrops and possibly changes the original data stream from the sender before delivering it to the receiver.

The workflow in many organizations involve data transfer from one system to another over a network.
Medical history, transactional information, receipts, bank statements, geographical coordinates etc. are few examples of many sensitive data shared across networks.

When files are sent over the internet, the data traverses the public internet with a high risk of being intercepted and tempered with.
A Man-in-the-Middle Attack is likely occur if the transfer is not made over a secure network. The repercussions of this can be dire as lives, money, reputations, security of nations amongst other things may be at risk if these files end up in the wrong hands.

In this session, we’ll explore how to Securely Transfer Files To And From AWS S3 Buckets By Utilizing VPC Endpoints.

VPC endpoints are virtual devices that are horizontally scaled, redundant, and highly available components of the Amazon VPC. They enable communication between Amazon VPC instances and AWS services without imposing availability risks or bandwidth constraints on network traffic.

They aid customers to connect privately to AWS by providing a secure, efficient and cost-effective way to access services from within your VPC.
They ensure security by allowing access to AWS services without going through the internet.
The network architecture is simplified since there is no need for a NAT Gateway. This makes it easy to setup and manage.

Network performance is also improved with a reduction in network cost as well. The ability to provide direct access to AWS services reduces latency and improves overall performance of applications while reducing cost.

Now let’s dive deep into the tutorial

Prerequisites: You must have an AWS account.

This tutorial requires an AWS account and a secure connection established between the account and whatever instance required to reach it.

VPC endpoints are created in specific VPCs and are accessible only to systems that have access to those VPCs. The requests may come from other AWS accounts or on-premise networks. Connectivity must be established between the networks in order to proceed.

Connectivity with other AWS accounts require VPC Peering while connectivity with on-premise networks requires establishing a Site-to-Site VPN connection between the two networks.

Ensure that a Site-to-Site VPN connection or a VPC Peering connection is established depending on your use case.

The S3 bucket to/from which the files would be accessed must also exit.

Login to your AWS account and in the search bar, type VPC, navigate to the VPC page and click on “Endpoints”.

3. Click on “Create endpoint”.

4. Fill the details below.

Enter the name of the endpoint you want to create.
Select the AWS services in the service category.
All the AWS services that are accessible via VPCEs are listed here. You can select as many services as you want but in our use case, we only need access to the S3 hence we select only that.
In the services, select the interface type of the S3 services.
There are two types of VPC endpoints; **Interface and Gateway **endpoints. We are going to work with interface endpoints because they support a host of services and are accessible via peered VPCs.

5. Specify the VPC and subnets where the endpoint would be created.

Select the VPC that as peered in Step 1.
Choose the private subnets in that VPC.
If you want to limit access to only specific IP addresses in the other account, create a security group and grant access to IPs or range of IPs allowed to reach it on port 443 as well.

6. Edit the VPCE policy to limit access to specific services.

Granting full access would give users or services in the source accounts the ability to access all AWS services in the destination account, which is not recommended. In order to implement the Principle of Least Privilege, select custom to limit access to only the services they are required to reach and in this case the designated S3 bucket.

Go ahead and create the endpoint. Next, we’ll modify the bucket’s policy to allow access from the VPCE.

7. Modify the bucket’s ACL to allow access from the VCPE.

Accessing Objects In The Bucket.

Now that we have successfully created our VPCE, we would go ahead and use it to access our S3 bucket and perform various operations on the bucket. Objects in the bucket can be accessed by the AWS CLI or CDK.

Using The AWS CLI

Below are the steps to follow when connecting with AWS CLI:
First, ensure the AWS CLI is installed on the device trying to reach the bucket.

In the examples below, replace the VPC endpoint ID

“vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com”, filename “file.txt”, the region and bucket name “my-bucket” with appropriate information.

In the details tab of the VPCE created, copy the DNS name. There are usually 4 DNS names for every VPCE. The first one indicates the endpoint at the regional level whiles the others are at the availability zone level.

Copy he first one to proceed.

Let’s say our file of interest here is file.txt

List Files In Bucket

aws s3 --region eu-west-1 \
--endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com \
ls s3://my-bucket/

Write File To Bucket

aws s3 --region eu-west-1 \ 
--endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com \ 
cp file.txt s3://my-bucket/

For instance, if there is file on the host called, file.txt. the command above is used to copy it to the bucket.

Download File From Bucket

aws s3 --region eu-west-1 \
--endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com  \
cp s3://my-bucket/file.txt file.txt

To download a file “file.txt” to the host, use the command above.

Using The AWS SDK

Various AWS clients can be employed to access S3 buckets with various programming languages. The clients have to be configured to use an endpoint URL for accessing buckets and performing operations through S3 interface endpoints. Here are some examples of such configurations:

Using The AWS Python SDK (Boto3)

s3_client = session.client(
service_name='s3',
region_name='eu-west-1',
endpoint_url='https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com'
)

Using The AWS SDK For Java 1.x

// bucket client
final AmazonS3 s3 = AmazonS3ClientBuilder.standard().withEndpointConfiguration(
        new AwsClientBuilder.EndpointConfiguration(
                "https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com",
    Regions.DEFAULT_REGION.getName()
        )
).build();
List<Bucket> buckets = s3.listBuckets();

Using The AWS SDK For Java 2.x

// bucket client
Region region = Region.EU_WEST_1;
s3Client = S3Client.builder().region(region).endpointOverride(
URI.create("https://bucket.vpce-1a2b3c4d-5e6f.s3.eu-west-1.vpce.amazonaws.com")
).build()

Refer to the official AWS documentation for further details.

References:
https://www.enisa.europa.eu/topics/incident-response/glossary/man-in-the-middle.

Photo Credit : Sean Patrick on Pexels

Get Sizes of All S3 Buckets In AWS SSO Accounts With Python

Nathaniel Agbenyenu — Fri, 10 Mar 2023 09:13:11 +0000

Let's say you've been tasked to produce an excel document containing the sizes of all the s3 buckets in multiple AWS accounts. This is a tedious task and would take a lot of time.
I was in a similar situation but then I came up with a simple python script to do that.

All the accounts were on SSO so it was simpler to achieve this having already configured SSO.

This tutorial assumes that you have set up AWS SSO and have the necessary permissions to access the S3 buckets in the specified account. You may also need to install the boto3 library if it is not already installed on your system.

Initializing the boto3 client

The boto3 client is initialized first. A session is created for the SSO profile.

Getting the list of all buckets in the account

Next, the list of all the buckets in the account are retrieved.

Calculating the size of objects in each bucket

For each of the buckets listed above, the size of the objects in the bucket(GB) are calculated. The code reads the contents of the objects and calculates the size of each. A key error is raised for buckets that are empty since there are no contents.
A dictionary of the buckets and their sizes is then created and returned.

Write to csv

A csv file named after the sso profile is then generated with the columns being the bucket name and the object size.

Bringing it all together

To run this code you need to pass the sso profile as an argument.
For example if your code file is s3_size.py and the sso profile is production, you execute it by entering the following command in the terminal

python s3_size.py production

DEV Community: Nathaniel Agbenyenu