Description:
With this implementation we are trying to automate data load and consumption process using S3 trigger and EC2.
Brief explanation:
- AWS Customer uploads invoice to S3 Bucket.
- S3 bucket has IAM Bucket policy to only allow specific customers (in this case it will be IP address) with putObject permission.
Once the object is uploaded, SNS trigger will allow EC2 Instance application to read the object from S3 bucket.
Application in EC2 will process the invoice and generates CSV from it. Which is then uploaded to target S3 bucket.
Same application writes original S3 file to document database Dynamo DB.
Below blog covers the part of configuring SNS subscription with application running in EC2.
EC2 Instance configuration
Instance configuration-
Ubuntu 18.04
T2-micro
This instance is configured as public instance, public IP – 107.22.136.3 is assigned to it. This will be needed for SNS subscription.
Security Group:
- As shown in snapshot above, 2 security groups are added.
- Port 8080 is opened for SNS end point access to EC2 instance.
- Port 22 is opened for personal IP address for developer access via SSH and SCP.
S3 Configuration
Source S3 bucket is created which is configured to trigger based on put event.
We can either keep this bucket open for public( ideally not recommended ) or block public access and interact with AWS Services or applications running on EC2. In our use case we used EC2 instance to upload object on this bucket.
To make it private, blocked public access.
Enable Events
In this case we added lifecycle policy to delete file in 24 hrs. As shown below expiration rule is set to expire/delete object after upload of 1 day.
SNS Configuration
Based on the above description, requirement is to integrate S3 trigger with SNS to automate file upload and processing. Below steps are followed to implement the same.
To add subscription to EC2, firsly we need to launch a service with-in ec2 and then configure SNS using http/https protocol.
This sample test uses python with django to enable service, but aws documentation also have other samples for reference
https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html
Below snapshot shows service in ec2 running on port 8080.
SNS Topic creation and adding subscription
SNS Access policy -
SNS Access policy shows only S3 source bucket uploads are allowed to cause the event trigger. Its also restricted by source account and ARN.
SNS Subscription -
Subscription is created using EC2 instance launched above as an endpoint. As shown in snapshot above endpoint URL is http://107.22.136.3:8080/sns.
Subscription Confirmation
In previous section where we launched application in EC2 on port 8080, now receives the request and provides subscription confirmation link below as highlighted.
On copying the link and pasting in webbrowser, receives below XML response.
S3 Upload Event notification
Once setting is done, uploaded any object on Source S3 bucket which triggers notification and json payload as shown below.
We can build rest of the process once notification is received by application. In our use case we will run data copy to another S3 bucket or DynamoDB.
Happy Learning!!!
Top comments (0)