You may be wondering: "Why AWS Lambdas for Real World?". Well, since Amazon announced Ruby support for AWS Lambdas on November 29th, I've been reading about it because I'm a Ruby enthusiastic.
Currently, you can easily find several blog posts and tutorials explaining how to build your own Lambda functions in Ruby, most of them using the famous Hello World as example, which is good as a starting point, but, let's be honest, you won't need to build something as simple as a Hello World. You will need to face real-world issues regarding automated testing, using other services, building/deploying, handling dependencies, etc.
In this post, I'd like to share some ideas with those who, like me, started to reach a bit deeper in this matter, and discuss how to tackle these real-world issues using Ruby and Serverless Framework. Be aware that I'm not claiming that these are the best practices. My goal here is, as I mentioned, share some ideas and start a discussion about them.
The application that I'll be using as an example to illustrate these ideas is a GitHub App which will consume data from Github and calculate some metrics. The first step, which will be the focus of this post, is to receive data sent from the Github Webhook and store it in a DynamoDB table.
Application structure
The first topic we'll tackle is how to organize your application. Many guides I've found put everything in the root project folder and it's done. Particularly I don't like this approach, I tend to take care on how to organize my projects in multiple subfolders, so, in a long-term, this organization can persist and avoid some headaches. In addition, since you may have several projects, keeping a similar structure will help you to find what you're looking for. If you don't care about it, feel free to jump this section.
As far as I can see, there are three way to organize your application.
- A single project with all functions and a single serverless framework settings file
- Multiple projects, divided into application modules, each one with its own serverless framework settings file
- Multiple projects, one per function, each one with its own serverless framework settings file
Particularly I choose the option 2. I believe the first option will result in a big single project, which may be a bit confusing to newcomer developers to start to contribute, however, may be the easiest option to build integration test across different functions. The third option, in the other hand, may turn these integration tests harder to implement, however, newcomer developers would have a smaller project to understand. The option 2 is a bit of both worlds.
I decided this first module will be called "webhooks". Currently, I'm integrating my application with Github, but in the future, I may decide to integrate it with something else. That said, the Webhooks project has the following structure.
├── app
└── functions
└── github.rb
└── lib
└── log.rb
└── models
└── github_event.rb
├── spec
└── functions
└── github_spec.rb
└── support
└── fixtures
└── github
└── events
└── push.json
└── spec_helper.rb
├── serverless.yml
├── Gemfile and Gemfile.lock
├── Rakefile
├── Other files like .gitignore, .editorconfig, etc.
As you can see, I didn't lie about organizing my project in several subfolders. If you're a Rails developer, you may notice that I'm using a similar structure of Rails projects. I like the way Rails organize the projects and it's also familiar to me, which make it even easier for me to find something.
As I mentioned before, most of Hello World examples hold all files in a single root project folder, which makes very easy to the Serverless Framework settings file (serverless.yml) references the lambda function.
functions:
myFunctionName:
handler: <filename_without_extension>.<lambda_function_name>
However, in my case, the file which contains my lambda function (app/functions/github.rb) is inside multiple subfolders and, in addition, is a class method of Webhooks::Github.
To reference the handler class method inside Webhooks::Github, I needed to set it this way:
functions:
github:
handler: app/functions/github.Webhooks::Github.handler
Which corresponds to the following pattern: path/to/<filename_without_extension>.<module_name>::<class_name>.<method_name>
Automated testing
Next step: automated tests! Serverless is a new way of thinking about how to design applications, so, honestly, took me a while to really understand that a lambda function is as simple to test as a Ruby method. Once again, most of the blog post I found didn't tackle testing. In fact, I believe the only one I found discussing something about it was this one. But here we'll be working with RSpec!
As you can see in the lines 23 and 29, I'm mocking the response from Aws::DynamoDB::Client. This is needed because the save! method of the Webhooks::Github class is calling the DynamoDB client, and, because I don't have DynamoDB running locally, that's the easiest way to mock success and failure responses.
Building and deploying
Ok, let's say your application is ready to be deployed and you're excited to use your lambda functions. Once you have your Serverless framework settings file properly configured, you can deploy your app running sls deploy.
But, let's be curious and take a look at the file built by the framework: it's a zip file inside the .serverless folder with the name you set to service in serverless.yml. You will see that Serverless framework basically zip all the content of your project, including your tests, which are not needed in a production environment, and, since the dependencies of your project are not inside it, this zip file doesn't contain your project dependencies. So, basically, your lambda function will be successfully deployed but, won't work.
So, before deploying our project, we need to execute bundle install --deployment to switch Bundler to deployment mode. This way, Bundler will create a vendor folder inside your project with all dependencies. Great! Wait... All dependencies? We don't need development and test dependencies. No problem: bundle install --deployment --without test development.
Great, thanks Bundler! Wait again. As soon as you try to perform changes in the Gemfile, Bundler won't allow you to do it because you're in the deployment mode. So let's execute bundle install --no-deployment to switch back to development mode. Once you try to execute bundle install, you'll notice that Bundler is not taking care of the development and test dependencies anymore. Damn it Bundler!
Instead of executing just bundle install --no-deployment, we need to execute bundle install --no-deployment --with test development to make Bundler take care of test and development dependencies again.
Three commands to perform a single deploy. That's unacceptable! I really want to execute something similar to app deploy, and this looks very much like a rake task!
Now I can simply execute rake deploy. As you may notice, the first command of my deploy rake task is rm -Rf vendor, and I'm doing this because I don't want gems that were removed from Gemfile to be there. For a brief moment I tried to use --clean, but then Bundler started to warn me that's very dangerous, so it scared me a bit.
Please leave a comment if you know a smarter way to do it. I really appreciate that! I had hope that Serverless framework team would take care of that, but, apparently, they won't.
Last but not least, let's take care of the files that shouldn't be included in the deployment package. In this matter, Serverless framework team did a good job because, in the settings file (serverless.yml), you can set the paths that should be included and excluded from the packaging process.
package:
exclude:
- Gemfile
- Gemfile.lock
- Rakefile
- spec/**
Logging
Not a big deal here but I was getting too many logging outputs while testing my code using rspec, which was messing a bit my testing outputs: I just want to see green dots. So, I needed to build something that (a) is simple as logging should be, (b) I don't want to initialize it (Logger.new(STDOUT)) every time I need to use it, (c) I want to silence it when running tests and (d) I want to see the logs in CloudWatch. That said, I built the following solution.
In order to log something, I just need to call Log.info "Something to log".
Next step
Obviously, my next steps are to continue working in my lambda functions, however, I already know that I will need to share some code across different functions, and I want to work on that before building new functions. Fortunately, AWS also provided a solution for that: AWS Lambda Layers. But this topic will be the subject of a future post.
Looking for a job?
If you're a Ruby on Rails Developer, DevOps Engineer, Python Developer or a Fullstack Java/Python Developer (all positions available for English speakers), and you live around Frankfurt am Main or you're willing to move, we have positions available at creditshelf.
Top comments (7)
I just published a serverless plugin to handle a lot of this npmjs.com/package/serverless-ruby-...
The are a couple modifications from the approach you describe. First, the plugin excludes all files by default, so you need to whitelist (add to the package/includes section) any files or directories you want to package.
Second, I recommend
bundle install --standalone --path vendor/bundleinstead of messing withbundle install --deployment. As you discovered, the "deployment" mode is really only intended for deployment, not active development, so you always need to immediately undo it. Instead, "standalone" accomplishes the part we actually care about, specifically, it copies all of the gem files into the vendor directory in your working directory. The plugin then analyzes your Gemfile to figure out which gems should be included, or excluded if they are in the development/test groups.I've managed my deployment pipeline in a different way, using the serverless-hooks-plugin and adding the following to the serverless.yml:
Great article Jalerson ! I published an article on the same topic this morning : medium.com/lifen-engineering/circl...
This is great! Thanks so much for the overview of the process and pitfalls; can't wait to try this out.
Auto-bookmark! Can't wait to see more of what Rubyists do with Lambda.
Did you manage to install gems that have native bindings (such as mysql2) successfully?
This is super helpful! I am having issues with conflicting gems already installed on lambda and being preferred (specifically, JSON 2.1.0 is too new). Did you find anything like that?