My goal
My goal is to built image compression website by using Amazon S3 and lambda. Here is my goal architecture.
HTML and Javascript files are stored in Amazon S3 Web bucket. A client accesses this HTML page by his browser. They can upload images to the Upload bucket through the HTML page. The uploaded image is compressed by Lambda function and saved in Compressed bucket. He can view the compressed images from Compressed bucket.
What are Amazon S3 & Lambda?
I'll explain the basics of Amazon S3 and Lambda. I'll also mention what functions of S3 and Lambda were used.
What is Amazon S3?
Amazon Simple Storage Service (S3) is a managed AWS service which can be used as an object storage. It is not a block storage, not a file storage, not even a relational database; it is an object storage! ref
A bucket is a container for objects. You have to create a bucket first and then store objects in the bucket.
Features of S3 used in this article.
Features used in this article are...
Website hosting ref
You can use an S3 bucket as a website hosting site by simply uploading static page files like HTML. You need to change the S3 bucket's permissions and policies to enable public access.Amazon S3 Event Notifications ref
You can send notifications when certain events happen in your S3 bucket. For example, you can send notifications when 'New Object Created events' occur. If you set a notification destination as a Lambda function, you can activate the Lambda function.
What is Lambda? ref
Lambda is a compute service, and you don't have to prepare any machine to run code. All you have to do is coding.You create lambda functions, and the Lambda service runs these functions.
Features of Lambda used in this article.
- Lambda Layer ref A Lambda Layer can be used as a supplement data for Lambda function. It contains data like library dependencies archived in .zip file. To install packages like Pillow, I use Lambda Layer.
Creating HTML page for uploading images to S3 and viewing them from S3.
To create HTML page by S3, I referenced these documents.
I quickly created an environment for browser (JavaScript) uploads to Amazon S3 using CloudFormation.
1. Do some prerequisite tasks.
1-1 Let's make s3 buckets!
All buckets allow public access.
1-2 Let's set CORS setting on Upload and Compress bucket.
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"HEAD",
"GET",
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [
"ETag"
]
}
]
Set this CORS setting!
1-3 Let's make identity pool by Amazon Cognito.
Find the IAM role created by an Amazon Cognito for unauthenticated users, and set following role to it.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cognito-identity:GetCredentialsForIdentity",
"s3:DeleteObject",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::Upload bucket/",
"arn:aws:s3:::Upload bucket/*",
"arn:aws:s3:::Compress bucket/",
"arn:aws:s3:::Compress bucket/*"
]
}
]
}
2. Defining the Webpage
Only you have to do is putting following HTML file and javascript file in Web bucket! The name of HTML file should be 'index.html' so that this page will show up when clients access to Web bucket endpoint.
**index.html**
<!DOCTYPE html>
<html>
<head>
<!-- **DO THIS**: -->
<!-- Replace SDK_VERSION_NUMBER with the current SDK version number -->
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.1473.0.min.js"></script>
<script src="s3_photoExample.js"></script>
<script>
function getHtml(template) {
return template.join('\n');
}
listAlbums();
</script>
</head>
<body>
<h1>My Photo Albums App</h1>
<div id="app"></div>
</body>
</html>
**s3_photoExample.js**
var albumBucketName = "Upload bucket";
var compBucketName = "Compress bucket";
var bucketRegion = "reagion";
var IdentityPoolId = "identitypoolID";
AWS.config.update({
region: bucketRegion,
credentials: new AWS.CognitoIdentityCredentials({
IdentityPoolId: IdentityPoolId
})
});
var s3 = new AWS.S3({
apiVersion: "2006-03-01",
params: { Bucket: albumBucketName }
});
var s3comp = new AWS.S3({
apiVersion: "2006-03-01",
params: { Bucket: compBucketName }
});
function listAlbums() {
var albums = [];
var albumscomp = [];
// Get list of folders in Upload bucket
s3.listObjects({ Delimiter: "/" }, function (err, data) {
if (err) {
return alert("Error listing albums in 'Upload bucket' bucket: " + err.message);
} else {
albums = data.CommonPrefixes.map(function (commonPrefix) {
var prefix = commonPrefix.Prefix;
var albumName = decodeURIComponent(prefix.replace("/", ""));
return getHtml([
"<li>",
"<span onclick=\"deleteAlbum('" + albumName + "', 'Upload bucket')\">X</span>",
"<span onclick=\"viewAlbum('" + albumName + "', 'Upload bucket')\">",
albumName,
"</span>",
"</li>"
]);
});
updatePage();
}
});
// get folder lists form Compress bucket
s3comp.listObjects({ Delimiter: "/" }, function (err, data) {
if (err) {
return alert("Error listing albums in 'Compress bucket' bucket: " + err.message);
} else {
albumscomp = data.CommonPrefixes.map(function (commonPrefix) {
var prefix = commonPrefix.Prefix;
var albumName = decodeURIComponent(prefix.replace("/", ""));
return getHtml([
"<li>",
"<span onclick=\"viewAlbumcomp('" + albumName + "', 'Compress bucket')\">",
albumName,
"</span>",
"</li>"
]);
});
updatePage();
}
});
function updatePage() {
// add folder lists form Upload and Compress bucket
var allAlbums = albums.concat(albumscomp);
var message = albums.length
? getHtml([
"<p>Click on an album name to view it.</p>",
"<p>Click on the X to delete the album.</p>"
])
: "<p>You do not have any albums. Please create an album.";
var messagecomp = albumscomp.length
? getHtml([
"<p>Click on an album name to view it.</p>",
"<p>Click on the X to delete the album.</p>"
])
: "<p>You do not have any albums. Please create an album.";
var htmlTemplate = [
"<h2>Compressed Alubms</h2>",
messagecomp,
"<ul>",
getHtml(albumscomp),
"</ul>",
"<h2>Original Alubms</h2>",
message,
"<ul>",
getHtml(albums),
"</ul>",
"<button onclick=\"createAlbum(prompt('Enter Album Name:'))\">",
"Create New Album",
"</button>"
];
document.getElementById("app").innerHTML = getHtml(htmlTemplate);
}
}
function createAlbum(albumName) {
albumName = albumName.trim();
if (!albumName) {
return alert("Album names must contain at least one non-space character.");
}
if (albumName.indexOf("/") !== -1) {
return alert("Album names cannot contain slashes.");
}
var albumKey = encodeURIComponent(albumName);
s3.headObject({ Key: albumKey }, function (err, data) {
if (!err) {
return alert("Album already exists.");
}
if (err.code !== "NotFound") {
return alert("There was an error creating your album: " + err.message);
}
s3.putObject({ Key: albumKey }, function (err, data) {
if (err) {
return alert("There was an error creating your album: " + err.message);
}
alert("Successfully created album.");
viewAlbum(albumName);
});
});
}
function viewAlbumcomp(albumName) {
var albumPhotosKey = encodeURIComponent(albumName) + "/";
s3comp.listObjects({ Prefix: albumPhotosKey }, function (err, data) {
if (err) {
return alert("There was an error viewing your album: " + err.message);
}
// 'this' references the AWS.Response instance that represents the response
var href = this.request.httpRequest.endpoint.href;
var bucketUrl = href + compBucketName + "/";
var photos = data.Contents.map(function (photo) {
var photoKey = photo.Key;
var photoUrl = bucketUrl + encodeURIComponent(photoKey);
return getHtml([
"<span>",
"<div>",
'<img style="width:128px;height:128px;" src="' + photoUrl + '"/>',
"</div>",
"<div>",
"<span>",
photoKey.replace(albumPhotosKey, ""),
"</span>",
"</div>",
"</span>"
]);
});
var message = photos.length
? "<p>Click on the X to delete the photo</p>"
: "<p>You do not have any photos in this album. Please add photos.</p>";
var htmlTemplate = [
"<h2>",
"Album: " + albumName,
"</h2>",
message,
"<div>",
getHtml(photos),
"</div>",
'<button onclick="listAlbums()">',
"Back To Albums",
"</button>"
];
document.getElementById("app").innerHTML = getHtml(htmlTemplate);
});
}
function viewAlbum(albumName) {
var albumPhotosKey = encodeURIComponent(albumName) + "/";
s3.listObjects({ Prefix: albumPhotosKey }, function (err, data) {
if (err) {
return alert("There was an error viewing your album: " + err.message);
}
// 'this' references the AWS.Response instance that represents the response
var href = this.request.httpRequest.endpoint.href;
var bucketUrl = href + albumBucketName + "/";
var photos = data.Contents.map(function (photo) {
var photoKey = photo.Key;
var photoUrl = bucketUrl + encodeURIComponent(photoKey);
return getHtml([
"<span>",
"<div>",
'<img style="width:128px;height:128px;" src="' + photoUrl + '"/>',
"</div>",
"<div>",
"<span onclick=\"deletePhoto('" +
albumName +
"','" +
photoKey +
"')\">",
"X",
"</span>",
"<span>",
photoKey.replace(albumPhotosKey, ""),
"</span>",
"</div>",
"</span>"
]);
});
var message = photos.length
? "<p>Click on the X to delete the photo</p>"
: "<p>You do not have any photos in this album. Please add photos.</p>";
var htmlTemplate = [
"<h2>",
"Album: " + albumName,
"</h2>",
message,
"<div>",
getHtml(photos),
"</div>",
'<input id="photoupload" type="file" accept="image/*">',
'<button id="addphoto" onclick="addPhoto(\'' + albumName + "')\">",
"Add Photo",
"</button>",
'<button onclick="listAlbums()">',
"Back To Albums",
"</button>"
];
document.getElementById("app").innerHTML = getHtml(htmlTemplate);
});
}
function addPhoto(albumName) {
var files = document.getElementById("photoupload").files;
if (!files.length) {
return alert("Please choose a file to upload first.");
}
var file = files[0];
var fileName = file.name;
var albumPhotosKey = encodeURIComponent(albumName) + "/";
var photoKey = albumPhotosKey + fileName;
// Use S3 ManagedUpload class as it supports multipart uploads
var upload = new AWS.S3.ManagedUpload({
params: {
Bucket: albumBucketName,
Key: photoKey,
Body: file
}
});
var promise = upload.promise();
promise.then(
function (data) {
alert("Successfully uploaded photo.");
viewAlbum(albumName);
},
function (err) {
return alert("There was an error uploading your photo: ", err.message);
}
);
}
function deletePhoto(albumName, photoKey) {
s3.deleteObject({ Key: photoKey }, function (err, data) {
if (err) {
return alert("There was an error deleting your photo: ", err.message);
}
alert("Successfully deleted photo.");
viewAlbum(albumName);
});
}
function deleteAlbum(albumName) {
var albumKey = encodeURIComponent(albumName) + "/";
s3.listObjects({ Prefix: albumKey }, function (err, data) {
if (err) {
return alert("There was an error deleting your album: ", err.message);
}
var objects = data.Contents.map(function (object) {
return { Key: object.Key };
});
s3.deleteObjects(
{
Delete: { Objects: objects, Quiet: true }
},
function (err, data) {
if (err) {
return alert("There was an error deleting your album: ", err.message);
}
alert("Successfully deleted album.");
listAlbums();
}
);
});
}
3. What you will see.
When you access to this website, you will see albums in Upload and Compressed buckets. You can create new album on Original albums (Upload bucket), add photo to it, and see it.
Compress images in S3 by using S3 events and Lambda.
To make lambda function, I referenced these documents.
1. Make lambda function.
Before coding lambda function, I made lambda layer for import Pillow. I made Pillow zip file by following command and upload it to Lambda.
$pip install \
--platform manylinux2014_x86_64 \
--target=python/ \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
Pillow
$ zip -r ./my-deployment-package.zip ./python/
Then I coded Lmabda function. Values of TARGET_BUCKET_NAME and QUALITY_IMAGE are in environmental variables.
I used Chat GPT to fix errors in lambda function...
from PIL import Image
import urllib.parse
import boto3
import os
TARGET_BUCKET_PATH = os.environ['TARGET_BUCKET_PATH']
QUALITY_IMAGE = int(os.environ['QUALITY_IMAGE'])
s3 = boto3.client('s3')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
source_key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
destination_key = 'compressed' + source_key
source_file = u'/tmp/' + os.path.basename(source_key)
destination_file = source_file
file_paths = os.path.splitext(source_key)
file_path = file_paths[0]
file_extension = file_paths[1][1:].lower()
try:
s3.download_file(Bucket=bucket, Key=source_key, Filename=source_file)
img = Image.open(source_file, 'r')
img.save(destination_file, file_extension, quality = QUALITY_IMAGE)
s3.upload_file(Filename=destination_file,
Bucket=TARGET_BUCKET_PATH,
Key=destination_key)
return source_key
except Exception as e:
print(e)
raise e
2. Set IAM Role.
I set this IAM Role to Lambda function.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:reagion:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:reagion:log-group:/aws/lambda/naoCompFunction:*"
]
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::Upload bucket",
"arn:aws:s3:::Upload bucket/*",
"arn:aws:s3:::Compress bucket",
"arn:aws:s3:::Compress bucket/*"
]
}
]
}
3. Set S3 events.
Trigger event types are these. You should include Multi part upload event if you might upload large files.
Event types: s3:ObjectCreated:Put, s3:ObjectCreated:Post, s3:ObjectCreated:CompleteMultipartUpload
Finally...
When client upload image file, Lambda function automatically compress image and put it in Compressed bucket. Client can see compressed image and download it form website!!!
Summary
It was so much fun to create some systems using AWS! It was not as difficult and didn't take as much time as I had expected. In this blog, I couldn't cover any security settings like bucket policies, so I'll write about them soon.
Top comments (0)