This is the fifth part of the Deploy Rails in Amazon ECS post. It's part of a broader series called More than 'hello world' in Docker. The series will get you from hello world in Docker to having your application deployed in AWS.
- Build Rails + Sidekiq web apps in Docker
-
Deploy Rails in Amazon ECS
- Concepts
- Push an image to ECR
- Create the RDS database, Task Definition, and Load Balancer
- Create the ECS Cluster and connect it all together
- Configure Sidekiq - we are here
- Automate Deploys with AWS CodeDeploy
In this post, we will be extending the ECS Setup we've built by adding a Sidekiq functionality. We will do this by:
- creating a Redis server that will hold tasks for Sidekiq
- creating an ECS service hosting our Sidekiq server
- connecting the web and Sidekiq service to this Redis server
13 | Creating a Task Definition for Sidekiq
For this step, follow the instructions on section 7 but changing the following configurations:
- (7.3) Put docker-rails-app-sidekiq as the name of the task definition
- (7.5) Use "sidekiq" as the container's name instead of "web".
-
(7.6) Put
sidekiq,-C,config/sidekiq.ymlfor the command. This is the command to run Sidekiq. You can specify another yml configuration file if yours is named differently.
14 | Creating a Sidekiq ECS Service
For this step, follow the instructions on section 10 but changing the ones listed below. Since this Sidekiq service will never receive traffic, we don't need to attach it to a load balancer. This service will just periodically fetch jobs from the Redis database we are about to make.
-
(10.2) Choose the task definition family of docker-rails-app-sidekiq. For the name, choose
simple-ruby-app-sidekiq. - (10.3) For the load balancer, click "None".
15 | Start an EC2 instance for Redis
Sidekiq services need a Redis database to fetch tasks from. These tasks are either scheduled to run periodically or are fed in by the web services we created in the last post.
To make things simple, we are going to create an EC2 instance to run a Redis database application server. We will then configure our Sidekiq ECS service to be able to connect to this Redis database.
(15.1) Start an EC2 instance
For our EC2 instance, choose an Ubuntu 18.04 t2.micro instance. For this example, a t2.micro is enough but do note this can differ base on how much your workload uses Redis and Sidekiq. If your application also uses Redis for caching, then it's a good idea to make this instance bigger.
Here are the networking options you should apply:
- Make sure to choose the same VPC you selected when you created the ECS Cluster in section 9.
- Select a Public Subnet.
- Select auto-assign public IP to be true.
We generally don't need the Redis database server to be accessible to the public. But to make things simple for this example, we want it to be publicly accessible so we can connect publicly to the EC2 instance and install Redis. On production environments (which generally must be more secured), you usually connect to a jumpbox instance first before being able to connect to this EC2 instance hosting Redis. The EC2 instance is then placed in a private subnet and is not assigned a public IP.
Stick to the defaults for the storage and tagging pages.
Next, we will create a security group named my-docker-redis-sg. Create a rule that allows traffic from port 22 (so you can SSH into the instance), and from port 6379 (so anyone can access our Redis instance).
Setting port 6379 accessible to the public is not best practice. We don't want anyone to be able to connect to this Redis server. Even if you have a secured password, all it might take is a few minutes for a sophisticated brute force attack to crack it.
(15.2) Configure the EC2 instance
Now that we have our EC2 instance up and running, let's connect to it. Go to the Services tab, then find EC2. On the left-side menu, click on Instances. You should be able to see the instance you just created. Once its status becomes "running", click the connect button on the upper right. You should be able to see instructions on how to connect to the instance.
If you haven't used your public key, you need to do a chmod 400 yourkey.pem to change its permissions. This is a required step for you to be able to use the key. Next, connect to the instance via the ssh command shown on the image below. If that doesn't work, it usually asks you to change root to ubuntu or ec2-user, depending on the OS that you used.
(15.3) Install Redis
After connecting to the instance, install Redis using the instructions in this Digital Ocean documentation.
(15.4) Configure Redis
There are some changes though that you have to make to your Redis configuration. These changes will allow us to connect to the Redis instance. You can do this by vi /etc/redis/redis.conf.
- Comment out
bind 127.0.0.1 ::1 - Make
protected-mode yestoprotected-mode no
(15.5) Start the Redis server
Then, type sudo systemctl restart redis.service to make your changes effective. Do a ps -ef | grep rails to ensure your process is running:
16 | Revising Task Definition
Now that we have a Redis instance and a Sidekiq ECS server, the missing piece is how we will let the Web ECS service throw jobs to the Redis database instance and how we will let the Sidekiq ECS service fetch jobs from the Redis database instance. To do this, we would have to change the task definitions of the Web and Sidekiq ECS services to include a REDIS_URL environment variable.
(16.1) On the services tab, search for ECS and click it. Click on "Task Definition" on the left-hand side menu. Then, click on the web's task definition (docker-rails-app).
(16.2) Click "Create New Revision"
(16.3) Add a REDIS_URL environment variable in the container definition for web.
You should be able to see something like this:
(16.4) Repeat steps 16.1 to 16.3 for the Sidekiq task definition (docker-rails-app-sidekiq).
17 | Updating the service
Now that we have an updated task definition for both web and Sidekiq, it's time to deploy!
(17.1) Click the simple-ruby-app service. On that page, click "Update"
(17.2) Choose the latest revision of the task definition of this service. Then tick "Force New Deployment" to make sure this new version gets deployed right away.
(17.3) Monitor your deployment here. It takes a few minutes for your changes to take effect. You should be able to see here if your new version has been deployed. Your app has been deployed if PRIMARY has a running count of 1+
(17.4) Repeat the same steps for the Sidekiq service (simple-ruby-app-sidekiq)
18 | Finished!
Using the URL from section 12, access the website once again. You should be able to click the like button. This like button will tell the web ECS service to throw a task in the Redis database server. The Sidekiq ECS service gets this task from Redis and adds 1 to the like_count, as instructed on the worker.
Deploying a dockerized Rails application in production takes a lot of heavy lifting at the start, but once the setup is done, you'd be able to enjoy the benefits of Docker and ECS: lower costs, faster deployments, faster rollbacks, easier dev onboarding and more.
Now that you've completed your own sample Rails app in Docker, you'd be able to extend this to your own production environments.
If you have any comments, suggestions or just want to let me know how this series has helped you, feel free to leave a comment below, or message me! I'd love to hear from you!
Special thanks to my editor, Allen, for making my posts more coherent.
Top comments (4)
Thanks for this great content!
It's really helpful.
I've recognized one miss instruction.
After I've deployed my simple-ruby-app-sidekiq service, it wouldn't start, and the reason was for a missing file, which is
config/sidekiq.yml.After I've created the file and pushed to the ECR, it worked fine.
Just in case if anyone else has the same problem.
Hi Raphael Jambalos, I have requirement to setup Rails, AWS, Redis and Sidekiq for Production environment. I have already setup same till staging environment and working fine where Rails server, queues and schedulers run on same machine. But this setup is not good for Production. Can it possible to run Sidekiq server on separate instance and rails server on separate one?
Great series, can't wait for the rest! Why not use Elasticache for Redis instead of an EC2 instance?
Thank you, Jason! I actually haven't tried Elasticache so I preferred to demo this via installing Redis on an EC2 instance. But I do believe using Elasticache would have been much simpler...