DEV Community

Junior Garcia
Junior Garcia

Posted on

How to create your own image CDN with Thumbor and AWS

#aws #cdn #images #thumbor

Image CDNs like Cloudinary or Imgix make it easy to dynamically optimize the aesthetics and performance of your images. Thumbor can help us to do the same but with a difference, is open-source and can be used for free to resize, apply filters, compress, and transform images. It's currently used by Wikipedia and 99Designs.

Note: This tutorial is based on
Katie Hempenius article about How to install thumbor but adding AWS EC2 and S3.

This post explains how to install Thumbor on your own server. Once installed, you can use Thumbor as an API to transform your images, we will install on AWS EC2 server and integrate it with AWS S3 to deliver the images as quickly as possible and save the transformations.

This post assumes that you have an AWS account and how to use command-line tools to set up the VM.

First of all, we are going to launch an EC2 VM with Ubuntu 18.04LTS 64-bit (x86), I used t2 micro because it's free tier but you can choose the one that best suits your needs.

Install Thumbor dependencies

Update and upgrade Ubuntu's already-installed packages:



sudo apt-get update -y && sudo apt-get upgrade -y
Enter fullscreen mode Exit fullscreen mode

Now we need to install pip, the package manager for Python. Later you'll install Thumbor with pip. 



sudo apt-get install -y python-pip
Enter fullscreen mode Exit fullscreen mode

Install Thumbor's dependencies.



#ssl packages
sudo apt-get install -y libcurl4-openssl-dev libssl-dev
Enter fullscreen mode Exit fullscreen mode 


#computer vision packages
sudo apt-get install -y python-opencv libopencv-dev
Enter fullscreen mode Exit fullscreen mode 


#image format packages
sudo apt-get install -y libjpeg-dev libpng-dev libwebp-dev webp
Enter fullscreen mode Exit fullscreen mode

Install Thumbor 



sudo pip install thumbor
Enter fullscreen mode Exit fullscreen mode

To see if Thumbor installed correctly this should work:



thumbor --help
Enter fullscreen mode Exit fullscreen mode

Hurrah! 🥳

Now we are going to start the debug server of Thumbor to make some requests and test it.



thumbor --log-level debug
Enter fullscreen mode Exit fullscreen mode

AWS EC2 Settings

Thumbor is now running.

By default, Thumbor runs on port 8888 However, this probably won't work for you (yet) because cloud providers usually require that you explicitly open firewall ports before they will accept incoming traffic, so we are going to open required ports on our Amazon EC2 instance.

To open the required ports we have to select on the left menu Network & Security -> Security Groups, we select the group assigned to our EC2 instance usually called launch-wizard-1 -> Actions -> Edit Inbound Rules,
once there we create the rules defined below and we press Save rules.

Type Protocol Port range Source Description
HTTP TCP 80 Anywhere http port
Custom TCP TCP 8888 Anywhere thumbor port
SSH TCP 22 Anywhere to accept ssh connections
HTTPS TCP 443 Anywhere https port

Once created our security rules, we should see something like this:

Screen Shot 2020-12-05 at 19.16.04.png

Try It Out

Thumbor is now accessible and ready for use 🥳👏🏻. Try it out by visiting the following URL:

http://YOUR_VIRTUAL_MACHINE:8888/unsafe/100x100/https://jrgarciadev.com/avatar.png

You should see an image that is 100 pixels wide by 100 pixels tall. Thumbor has taken the image avatar.png and size specified in the URL string and served the result. You can replace the image in the URL string (i.e., https://jrgarciadev.com/avatar.png) with any other image (e.g., https://your-site.com/kitten.jpg) and Thumbor will resize that image too.

Note: Thumbor uses HTTP by default but can be configured to use HTTPS.

I recommend you visit this article The Optimize images with Thumbor for more information on using the Thumbor API and Thumbor configuration file.

Configuring Thumbor to run in background

In a production environment, we need to configure Thumbor to start automatically
to do this we are going to make use of Linux Systemd.

Navigate to the /lib/systemd/system directory. This directory contains the service files for systemd.



cd /lib/systemd/system
Enter fullscreen mode Exit fullscreen mode

As superuser, create a thumbor. service file.



sudo touch thumbor.service
Enter fullscreen mode Exit fullscreen mode

Using vim or nano add the following configuration to Thumbor service. This configuration will run /usr/local/bin/thumbor (i.e. the Thumbor binary) once networking is available and will restart Thumbor on failure.

I prefer to use vim so the command is the bellow:



sudo vim thumbor.service
Enter fullscreen mode Exit fullscreen mode 


[Unit]

Description=Service for Thumbor image CDN

Documentation=https://thumbor.readthedocs.io/en/latest/

After=network.target

[Service]

ExecStart=/usr/local/bin/thumbor

Restart=on-failure

[Install]

WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Now we need to start Thumbor, to do this we are going to usesystemctl which is the utility used to manage systemd.



sudo systemctl start thumbor.service
Enter fullscreen mode Exit fullscreen mode

Note: If Thumbor is currently running, you should stop it before attempting to start Thumbor using systemctl.

Next, "enable" Thumbor. This means that Thumbor will automatically start on boot.



sudo systemctl enable thumbor.service
Enter fullscreen mode Exit fullscreen mode

Verify that you've successfully configured systemd by running the status command.



systemctl status thumbor.service
Enter fullscreen mode Exit fullscreen mode

The status should show that it is enabled and active.

Screen Shot 2020-12-06 at 01.11.12.png

Congratulations 🎊, now you have a Thumbor service running and ready to make transformations, apply filters, resize and convert any image.

AWS S3 Configuration

We have running our Thumbor service successfully but the aim of this post is to guide you to use Thumbor efficiently, for this reason, we are going to save the transformations in an S3 bucket so we will only do the transformation once if we do the same request again, Thumbor will send us the image that it already has saved in the bucket.

Here we go!

We have to create 2 S3 buckets with all default settings, one for the data source and the other for saving the results, I have named them thumbor-blog-source and thumbor-blog-results.

Once we have created our S3 bucket you have to upload an image on your source S3 bucket only for test, then return the server terminal and we are going to execute the following command to create a new thumbor.conf file:



thumbor-config > ./thumbor.conf
Enter fullscreen mode Exit fullscreen mode

Now we have to install a Boto3 to management the AWS authentication and AWS S3 Plugin just run next commands:



pip install boto3
Enter fullscreen mode Exit fullscreen mode

Once boto3 is installed we need to configure our AWS S3 connection to do this are going to create a file for boto3 configuration:



mkdir .aws && sudo touch .aws/config && sudo touch .aws/credentials
Enter fullscreen mode Exit fullscreen mode

Open the .aws/config file a write the next content:



[default]
region=YOUR_S3_REGION
Enter fullscreen mode Exit fullscreen mode

Save it and open the .aws/credentials file and write:



[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
Enter fullscreen mode Exit fullscreen mode

Note: YOUR_S3_REGION by default is us-east-1 or us-east-2

Then we have to install Thumbor AWS



 pip install tc_aws
Enter fullscreen mode Exit fullscreen mode

Open your thumbor.conf file created earlier, there are a lot of options but for now, we are going to focus only on the TC_AWS section (This section doesn’t exist in the default config and you’ll need to add it somewhere).



############################## TC_AWS ##########################################
TC_AWS_REGION = 'YOUR_AWS_REGION' # AWS Region

TC_AWS_ENDPOINT = None # Custom S3 endpoint URL (for GCP, Minio, etc.)

TC_AWS_STORAGE_BUCKET = 'YOUR_SOURCE_S3_BUCKET_NAME' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH = '' # S3 path prefix for Storage bucket

TC_AWS_LOADER_BUCKET = 'YOUR_SOURCE_S3_BUCKET_NAME' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH = '' # S3 path prefix for Loader bucket

TC_AWS_RESULT_STORAGE_BUCKET = 'YOUR_RESULT_S3_BUCKET_NAME' # S3 bucket for result Storage
TC_AWS_RESULT_STORAGE_ROOT_PATH = 'thumbor_results' # S3 path prefix for Result storage bucket

# put data into S3 using the Server Side Encryption functionality to
# encrypt data at rest in S3
# https://aws.amazon.com/about-aws/whats-new/2011/10/04/amazon-s3-announces-server-side-encryption-support/
TC_AWS_STORAGE_SSE = False

# put data into S3 with Reduced Redundancy
# https://aws.amazon.com/about-aws/whats-new/2010/05/19/announcing-amazon-s3-reduced-redundancy-storage/
TC_AWS_STORAGE_RRS = False


# Enable HTTP Loader as well?
# This would allow you to load watermarks in over your images dynamically through a URI
# E.g.
# http://your-thumbor.com/unsafe/filters:watermark(http://example.com/watermark.png,0,0,50)/s3_bucket/photo.jpg
TC_AWS_ENABLE_HTTP_LOADER = False

TC_AWS_ALLOWED_BUCKETS = False # List of the allowed bucket to be requested
TC_AWS_STORE_METADATA = False # Store result with metadata (for instance content-type)
################################################################################
Enter fullscreen mode Exit fullscreen mode

Now we have to uncomment and edit these lines on the thumbor.conf file:



## Defaults to: 'thumbor.loaders.http_loader'
LOADER = 'tc_aws.loaders.s3_loader'

## Defaults to: 'thumbor.storages.file_storage'
STORAGE = 'thumbor.storages.no_storage'

## Defaults to: None
RESULT_STORAGE = 'tc_aws.result_storages.s3_storage'
Enter fullscreen mode Exit fullscreen mode

And finally, we have to set as RESULT_STORAGE_STORES_UNSAFE = True option to save thumbor results in our results bucket.



################################ Result Storage ################################

## Indicates whether unsafe requests should also be stored in the Result Storage
## Defaults to: False
RESULT_STORAGE_STORES_UNSAFE = True
Enter fullscreen mode Exit fullscreen mode

Then we need to restart the Thumbor service so that it takes the new settings, write the next command on your server command line:



sudo systemctl restart thumbor.service
Enter fullscreen mode Exit fullscreen mode

Execute the status command to be sure that it's running correctly

Note: If for any reason the server is not running (sudo systemctl stop thumbor.service) you can stop it and try out with debug mode with thumbor --log-level debug

Now visit the URL of your server with the name of the image saved on the source bucket, in my case thumbor-blog-source

Here a diagram of what we have configured in thumbor :

thumbor-blog.jpg

I'm proud that you made it this far 👏🏻

If you liked it, just let me know and so I will continue with part 2 of this post, where we will delve into the Thumbor options for image optimization, security and we will also install an Nginx server and connect it with CloudFront, an authentic CDN of images! 🚀

Extra: We can see how to mount 4 Thumbor servers managed by an AWS Load Balancer.

Thanks for reading! 🙋🏻‍♂️

Top comments (5)

Collapse
 
enthusistic42 profile image
enthusistic42 • • Edited

Is thumbor installed here restricted only to files on aws s3 or will it accept any url over the internet? If yes then wont it be vulnerable to mindless or foreign requests?

Collapse
 
jrgarciadev profile image
Junior Garcia • • Edited

The final result just work with AWS s3, it won’t work with another internet url, to avoid foreign request users must disable “UNSAFE” mode in the thumbor configuration file, hope it helps you 🙋🏻‍♂️

Collapse
 
briancollins082 profile image
brian •

Thanks I was looking for the service creation part in linux

Collapse
 
ashishkpal profile image
ashishkpal •

can you please update on the part 2 which is integrating the Thumbor with the nginx and cloudfront

Collapse
 
stoneisland profile image
StoneIsland •

Here is a ready to use Thumbor S3 Docker Image which allows you to do everything what is mentioned in this article
github.com/beeyev/thumbor-s3-docker/

Read next

lhidalgo_dev profile image

Mastering DynamoDB: Batch Operations Explained

Lorenzo Hidalgo Gadea -

buzzgk profile image

The Rise of Edge SEO: Transforming Search Engine Optimization

BuzzGK -

karthiksakthiveltechie profile image

AWS CodePipeline V2 type pipelines supports to automatically retry a stage if there is a failure in the stage.

Karthik Sakthivel -

mahinur05 profile image

🚀 Deploying Node.js Application with PM2, NGINX, and SSL Configuration 🚀

Mahinur Rahman -