Go straight to the Github Repo
The need to properly size and optimize images within web and app content is such a common problem that there are numerous—if not seemingly innumerable—solutions out there. Generally, you’ll find this to be just one feature of Digital Asset Managers (or DAMs), like Cloudinary, Widen, and Bynder. These services also provide many more primary features—proofing, approval workflows, asset organization, content localization, slick UIs, etc.—and you’ll pay handsomely. What if I don’t need all of that, and just want to be able to dynamically (at request time) resize images?
Isn’t there already open source for this?
There are actually quite a few public Github repos out there to address this problem. A couple of years back (in the early days of Lambda@Edge) I was looking for a solution and found this tutorial on the official AWS blog. Of course, that tutorial was just intended as a demonstration of what was capable and not instruction on building a production-ready solution per sé. In fact, both the architecture and the code were a little lacking. Still, I tried to use that blog post as a jump-off to create several subsequent solutions. The problem is that all of the blogs, tutorials, walkthroughs, and open source solutions that I’ve been able to find since then are just as incomplete, as if they’ve all learned from the same example.
TLDR;
Yeah, this is a bit long-winded, so I’ll just copy-paste the Conclusion here.
If all you need is to host, resize, optimize, and cache images, don’t break the bank on a Digital Asset Management solution. Building a robust solution is simple with a little knowledge of a handful of AWS services, and I went ahead and did it all for you with Image Flex. Check out my fully documented and production-ready solution on Github, which builds and deploys in just 2 commands!
Introducing: Image Flex
Image Flex is a robust and fully open source image resizing service built on AWS Serverless technologies and used to resize, optimize, and cache images on "the edge," on the fly. Served by CloudFront via an Origin Access Identity. Executed on Lambda@Edge. Backed by S3. Protected by AWS WAF. Provisioned via CloudFormation. Built and deployed by the Serverless Application Model (SAM) CLI.
The Approach
I started by using what I evangelize as “template-driven architecture.” That is, I don’t manually create or configure any resources. All of the infrastructure will be code, configured via a SAM/CloudFormation template, and only via that template. This is a good (if not best) practice to prevent drift between your actual resources and your template, or vice versa. Other good solutions for template-driven architecture include Terraform, Ansible, and Serverless Framework.
I systematically composed a robust serverless… er… service on AWS, organically, one resource at a time. The end product can host large, unoptimized images, resize them at request time on demand, optimize them to WebP format, store the new versions, cache the new versions on the edge, and return them to the client. To do this, the application required provisioning, security, storage, caching, and business logic (i.e., compute). AWS provided the tools and services that fulfilled all of those requirements:
- Storage: S3
- Caching: CloudFront
- Compute: Lambda@Edge
- Security: AWS WAF, Origin Access Identity
- Provisioning: CloudFormation, SAM CLI
Architecture
Provisioning
The CloudFormation template implements AWS’s Serverless Application Model (SAM) syntax. SAM provides an alternative shorthand for defining certain serverless-specific resources in a CloudFormation template. Note that SAM resource definitions can differ dramatically from the same resource definitions using straight CloudFormation syntax.
Security
The entire service is behind an AWS WAF Web access control list. I only employ the AWS Common Rule Set to defend against numerous vulnerabilities. You could further lock it down by whitelisting requester IP addresses or geo-locking requests.
I also implement an Origin Access Identity, which prevents public access to the hosting S3 bucket by only allowing the CloudFront distribution to read from it.
Storage
To host both the original and optimized images, S3 was the obvious and only choice here. A private S3 bucket resource is defined, blocking all public access except by the CloudFront distribution, which employs the Origin Access ID. The bucket definition also includes CORS settings that allow clients on any domain to GET images from the service. You might want to lock this down further by providing a list of AllowedOrigins (domains from which your images may be requested).
A second S3 bucket is defined to store logs for the CloudFront distribution (see Caching next).
Caching
Now that we have a bucket to host our images, we’ll want to put a CDN in front of it to enable caching. A CDN (content delivery network) like CloudFront will serve cached copies of the images from edge locations closest to the actual user. The definition for the CloudFront distribution specifies the hosting S3 bucket (above) as its origin, as well as the Origin Access ID to use to be allowed access to that bucket. The definition also specifies the WAF ACL to use (created earlier). Lastly and as mentioned in the previous section, the definition also enables saving logs to an S3 bucket. These logs will have an entry for every request to the CloudFront distribution.
Compute
To process our requests (and to prevent having to manually run servers) Lambdas were a perfect fit. These Lambdas scripts are actually Lambda@Edge scripts, as they will be attached to different parts of the CloudFront request/response event cycle, therefore executing on the CloudFront edge servers.
*See CloudFront Events That Can Trigger a Lambda Function
Viewer Request
The first script is triggered by CloudFront’s viewer request event, which fires before the request reaches the cache. This script will see a URL that looks like /myimage.png?w=500 and convert it to an S3 key that looks like /500/myimage.webp.
Observer Respons
The second script is triggered by CloudFront’s observer response event, which fires after CloudFront has gotten the response from the origin, but before it has returned it. This script follows this logic:
- If the response from the origin has an HTTP status of 200 (the resized image has already been previously generated), return that response.
- If the response from the origin has an HTTP status of 403 or 404, fetch the base image, resize to the requested dimensions, save the resized copy of the image to S3, and return that new image in the response.
Usage
Using the service is as simple as constructing a URL to request an image.
For example, let’s say we have a large, unoptimized photo that we want to include on a website. We don’t want to pull that image in directly, as that would negatively affect performance. We want it sized appropriately, and optimized if possible.
Our large, unoptimized base image might be 2400x1350 (a 16:9 aspect ratio):
https://my.service.url/myphoto.png
Once we upload that to the service’s S3 bucket, we can easily request different sizes using the w (width, required) and h (height, optional) query string parameters.
For example, requesting the following for the first time…
https://my.service.url/myphoto.png?w=800
… will generate, save, cache, and return the following image resized to 800wx450h (the 16:9 aspect ratio is preserved) and optimized to WebP format (if supported by the requesting browser):
https://my.service.url/800/myphoto.webp
Using the optional height parameter…
https://my.service.url/myphoto.png?w=400&h=400
… the resulting image will be 400wx400h, the aspect ratio of the original image still preserved, but the image will be clipped (as with CSS’s object-fit: cover):
https://my.service.url/400x400/myphoto.webp
Try it out
If all you need is to host, resize, optimize, and cache images, don’t break the bank on a Digital Asset Management solution. Building a robust solution is simple with a little knowledge of a handful of AWS services, and I went ahead and did it all for you with Image Flex. Check out my fully documented and production-ready solution on Github, which builds and deploys in just 2 commands!
If you like the solution, please go ahead and star that Github repo, and be sure to post any issues you come across or new features you’d like to see (you could even submit your own pull request, should you feel so motivated).
Cheers!
— Horace
Top comments (12)
Hey thank you so much the solution is flawless. one think i want to know is how does is serve the .webp, if i request my images like this my.service.url/myphoto.png?w=800, how does the .webp served.
Thanks for the praise, and you're welcome!
Two things work together to return WebP.
First, I check the "Accepts" header of the http request, which automatically gets sent by all browsers. If "webp" is in the value of that header, I know the client can accept webp format.
Second, I'm using the
sharpnpm package to manipulate the images. Sharp can create WebP images from source images.And that's it. Glad you like the solution.
Thanks for a quick reply.
I am facing the following an Error if the image is not in the bucket root,
Example: original URL: distribution/uploads/test.jpeg?w=200
Error message: "Error while getting source image object "/uploadstest.jpeg": NoSuchKey: The specified key does not exist"
If the image is in the subfolder I get the following error how can if ix it.
Ah, looks like a bug in appending the prefix ( a missing / character). I'll push a fix in about an hour.
yah it took me long to realize that a / was missing please du push the fix.
I fixed the bug. You'll just need to
git pullandnpm run update [env].now its working like a charm. im just curious if its possible to control quality wen resizing. Im losing a great deal of quality for example if i resze an image from 800x800 to 200x200 the difference in quality is highly noticable.
There should be no loss of quality when sizing down. If anything, you'd gain quality. I'm curious... can you point me to an example?
Apparently, Sharp (the image processing library used) accepts some options for quality. I added the settings and pushed the changes. It looks like that solved the issue, so let me know if that helps.
Simply did the trick thank you so much for your effort much appreciated.
Hie the solution is working great but rather i noted a huge grow on my page size after inspecting thoroughly i found out that if i upload a jpeg with 1200x800 size 88kb. If i then try to resize it to ?w=500 the size actually grows from 88kb to 265kb when it supposed to have gone down. i tried to change the sharp quality to as less a 50 but did not change a thing. thank you bro please look into it.
Hello there. I can't, I can't figure out how to install it. is there any video. Please