DEV Community: Frederick Cheung

AWS Lambda Layers and Ruby

Frederick Cheung — Sun, 29 Nov 2020 14:31:41 +0000

I've been building a few things using lambda functions in ruby recently. Some of these are just standalone functions, some end up as more complicated stacks, with multiple functions working together, usually with a lot of common dependencies.

This was annoying to work with since on each sam build invocation bundler was redownloading & reinstalling these dependencies for each function,
even if you had only changed one of the functions. By the time I got to 4-5 functions in a stack it was really starting to slow down that fast iteration
that is so beneficial. This got me thinking - could I put the vendored bundle in a layer that was used by all my functions?

One Gemfile to rule them all

At the start my directory looked at little like this - the standard sam setup ¹

.
├── Gemfile
├── Gemfile.lock
├── function1
│   ├── Gemfile
│   ├── Gemfile.lock
│   └── app.rb
├── function2
│   ├── Gemfile
│   ├── Gemfile.lock
│   └── app.rb
└── template.yaml

The first thing I did was merge the Gemfiles into a single one that I could use to build the layer. In order to make sure this would never diverge from what the functions are using, I added this at the top level and symlinked it into each function ². I then added a folder called dependencies for my layer, containing just the Gemfile and Gemfile.lock (also as symlinks), and added the layer itself to the template:

Dependencies:
  Type: AWS::Serverless::LayerVersion
  Properties:
    LayerName: !Sub "${AWS::StackName}-dependencies"
    ContentUri: dependencies/
    CompatibleRuntimes:
      - ruby2.7
    LicenseInfo: 'MIT'
    RetentionPolicy: Retain
  Metadata:
    BuildMethod: ruby2.7

Each function needs to be told to use this new layer, by adding this to its properties:

Layers:
  - !Ref Dependencies

sam build / sam deploy ran ok - this all seems too easy!

Trouble in paradise

Unfortunately I've simultaneously broken things and made them slower:

sam build still wants to run bundler in each of my function directories
The layer doesn't put gems where ruby is looking for them (see this issue)
The layer installed all the gems, including test/development ones

It's tempting to fix the first issue by just removing the Gemfile from each function, but I didn't want to do that - I really value bundler
enforcing that the correct versions of gems and ruby are installed and the work it does to avoid various forms of dependency hell.

Luckily there is a way out: while sam build has a default way of building functions for a given runtime (i.e. run bundler for ruby, pip for python etc.) you can override this by setting the build method to 'makefile', in which case it will just run make instead. This is most useful for custom runtimes but pretty handy here too.

This results in functions that look like

Function1:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: function1/
    Handler: app.lambda_handler
    Runtime: ruby2.7
    Layers:
      - !Ref Dependencies
  Metadata:
    BuildMethod: 'makefile'

and a Makefile (in the directory for the function) that just copies the source into the destination directory

build-Function1:
  cp -pLR . "$(ARTIFACTS_DIR)"

You do end up with a bunch of nearly identical makefiles, which is a bit tedious but ok. Don't forget that makefiles need to use tabs, not spaces!

Finding Nemo Rubygems

While researching the second issue I came across a post
by Josh Kahn that seemed to do exactly what I wanted: it builds the layer using a custom makefile that runs bundler and then moves the output around so that the gems are in ruby's search path.

This was so close to what I wanted! Unfortunately we have some private code that we install as gems build from git, so that approach doesn't quite work - you need to preserve more of bundler's structure. I ended up letting mostly bundler just do its thing:

build-Dependencies:
  BUNDLE_WITHOUT="development:test" bundle install "--path=$(ARTIFACTS_DIR)/ruby/lib/vendor/bundle"
  rm -rf "$(ARTIFACTS_DIR)/ruby/lib/vendor/bundle/ruby/2.7.0/cache"
  rm -rf "$(ARTIFACTS_DIR)/ruby/lib/vendor/bundle/ruby/2.7.0/bin"

This uses BUNDLE_WITHOUT to avoid installing unneeded gems and --path to control the install location. The last two rm-rf steps just make the deployment package a little smaller by removing unneeded files.

On its own this isn't quite enough, because the lambda function can't see the gems. It also doesn't know that it shouldn't expect to find the development / test only gems, so it will complain about that too. The magic step is to create .bundle/config in each of the lambda functions with these contents:

---
BUNDLE_DEPLOYMENT: "true"
BUNDLE_WITHOUT: "development:test"
BUNDLE_PATH: "/opt/ruby/lib/vendor/bundle"

This does three things:

Puts bundler into deployment mode, so that it will complain if Gemfile and Gemfile.lock are out of sync
Tells bundler to ignore gems in development / test groups
Tells bundler where gems have been installed

And that's it - with this the lambda function is able to load the gems from the layer.

Summary

I'm pretty happy with this - it achieved all the goals I had when I started out:

One consistent gemfile across functions and in production vs running specs locally
Bundler runs once only per invocation of sam build
Bundler still enforcing gem versions etc.

Combined with the recent sam build --cached feature (sam 1.9.0 and higher) it gets even better: bundler is only run if Gemfile/Gemfile.lock has changed. While --cached is cool all on its own, without dependencies isolated in this fashion, it would run bundler for each function where at least one file has changed (since sam has
no knowledge of what file changes should trigger bundler).

If you want to play around with this, I've made a very simple repo that demonstrates this.

I don't love the separate gemfiles. More than once I've added something to the toplevel gemfile (used for running specs) but not the per function Gemfile, so although my specs were green the dependiences weren't correct when deployed. I've also made mistakes because the per function Gemfiles have a ruby constraint but the top level one didn't, or having 2 functions accidentally end up with different versions of the same dependency. While in theory there's nothing wrong with this, I find it a lot easier if those variations between functions don't exist. ↩
These symlinks will break if you need to run sam build -u - my workaround is to run the entire sam build process in docker (and not pass -u at all). ↩

Diagnosing a mruby crash

Frederick Cheung — Sat, 04 Jan 2020 18:23:42 +0000

I've been playing with mruby recently, after seeing several interesting talks at Rubyconf 2019. I found a few projects ( mruby-esp32, M5Stack-mruby) that were setup for building mruby on the device I had bought (an M5StickC), so after some messing around with toolchains & drivers I was up and running.

All these projects were using mruby 2.0.1 (released April 2019), but I wanted to use mruby 2.1 (Nov 2019) which has a bunch of ruby 2.6 features, some nice enhancements to the C apis (including keyword arguments) and the usual collection of bugfixes. In theory this was easy to fix: the setup for these projects was a git submodule pointing at mruby, all I needed to do was update that submodule to the tag for 2.1, recompile, and I'd be on my way.

However, when I did that, my device started crashing as soon as it loaded mruby (in the call to mrb_open) with a message like this:

Guru Meditation Error: Core  0 panic'ed (InstrFetchProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x000ef970  PS      : 0x00060730  A0      : 0x800e791d  A1      : 0x3ffb8c40  
A2      : 0x3ffb8d30  A3      : 0x3ffb8c60  A4      : 0x0000000e  A5      : 0x000ef970  
A6      : 0x00000001  A7      : 0x3ffb4a68  A8      : 0x800e7cf0  A9      : 0x00000000  
A10     : 0x3ffb4a68  A11     : 0x00000010  A12     : 0x3ffb5298  A13     : 0x3ffb4a68  
A14     : 0x00000012  A15     : 0x00000002  SAR     : 0x0000000f  EXCCAUSE: 0x00000014  
EXCVADDR: 0x000ef970  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  

ELF file SHA256: 5da51cd84380674eb796b51bd0d21ea3ee3c4f275043c281291ffd060f6d7e00

Backtrace: 0x000ef970:0x3ffb8c40 0x400e791a:0x3ffb8d70 0x400e7e33:0x3ffb8ea0 0x400d6ca1:0x3ffb8f30 0x400efb3d:0x3ffb8fb0 0x400f0ad9:0x3ffb9000 0x400da57f:0x3ffb9060 0x400da32b:0x3ffb9080 0x400da341:0x3ffb90a0 0x400da361:0x3ffb90c0 0x400d390d:0x3ffb90e0 0x40086ee9:0x3ffb9150
0x400e791a: mrb_funcall_with_block at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/vm.c:808 (discriminator 8)

0x400e7e33: mrb_funcall_argv at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/vm.c:808 (discriminator 8)

0x400d6ca1: mrb_obj_new at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/class.c:504

0x400efb3d: mrb_exc_new_str at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/error.c:563

0x400f0ad9: mrb_init_exception at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/error.c:610

0x400da57f: mrb_init_core at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/init.c:42

0x400da32b: mrb_open_core at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/state.c:43

0x400da341: mrb_open_allocf at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/state.c:71

0x400da361: mrb_open at /Users/fred/iot/M5Stick-mruby/components/mruby_component/mruby/src/state.c:63

0x400d390d: mrubyTask(void*) at /Users/fred/iot/M5Stick-mruby/main/main.cpp:38

0x40086ee9: vPortTaskWrapper at /Users/fred/iot/esp-idf/components/freertos/port.c:403

Which was more than a little intimidating, seeing as I'm totally unfamiliar with the mrb or the mrb source and my google searches turned up nothing. It's crashing inside mrb_funcall_with_block, which is the c api function for calling a ruby method, but that's about all I could glean from the stack trace.

Bisect to the rescue

That's when I remembered a great talk from Brighton Ruby 2019 about tracking down a bug in MRI. Different codebase, same idea: just git bisect on the mruby itself, with the 2.0.1 tag as the initial "good" commit and 2.1.0 as the bad one.

For those unfamiliar with git bisect, it's an awesome tool, well worth adding to your arsenal. Given an initial pair of good and bad commits it will do a binary search through the intervening history to find the commit that caused the problem. Since it's doing a binary search, instead of having to try all 900 or so commits between the 2 mruby versions, I would only need to try 10 or so (which was a good thing, since you're recompiling mruby at each step). I couldn't quite do the automated setup Alyssa did in her talk: I had to manually build my code & flash to the device for each commit git bisect asked me to check. I also got compile errors on some commits and had to include the fix from this commit.

Still it wasn't too long before it pointed the finger of blame at this commit. This was a change relating to an optimisation relating to how methods are represented.

Digging deeper

On mruby, a ruby method is represented by two things: a pointer to a function or RProc structure and some flags. Those flags tell ruby whether the pointer is a function pointer or not and whether the method takes arguments. There are only 2 flags (i.e. 2 bits are needed) and on most platforms function pointers are 4 byte aligned: the low 2 bits are never used. Therefore mruby can use those 2 bits to store the flags. As a result it doesn't need to allocate a separate data structure, reduces memory accesses required to call methods etc. (A similar trick is used to encode integers into fixnum objects).

When mruby gets a mrb_method_t (the name of this data type) it can mask off the low 2 bits to get that pointer or the high bits to get the flag information.

If however function pointers are not 4 byte aligned though things will go very wrong:

When jumping to the function, mruby ignores the last 2 bits, so will use the wrong pointer value
The low bits look like flags so mruby may use the pointer incorrectly

Apparently that's the case on my platform (you can see this on the stack trace above - the addresses are not all divisible by zero). Prior to the linked commit you had to opt into this optimisation, but afterwards it became the default.

Fixing it

Luckily the commit message had all the information I needed: you can disable the optimization by adding MRB_METHOD_T_STRUCT to the list of defines in the mruby build config file (compiling with falign-functions=4 didn't seem to do anything). With that, I was up and running on mruby 2.1.

This option is also documented in the mrbconf docs, however I doubt I would have realised its significance.

So there you have it: mruby is cool and so git bisect is a great tool even if you don't understand the codebase. (and go watch Alyssa's talk if you have a spare 25 minutes or so - I saw it in person and it is well worth the watch)

My first RubyConf

Frederick Cheung — Thu, 21 Nov 2019 19:25:31 +0000

It's been a week of firsts:

my first visit to RubyConf
my first talk at RubyConf
my first time being a guide at a conference
and now, my first post on dev.to

I've been a member of the ruby community for a long time now, back when 1.8.4 was all the rage, threads were still green and deploying rails applications involved mod_fastcgi / mod_fcgi and large amounts of hopes and prayers. Somehow I'd never made it to RubyConf, although I've had my share of RailsConfs and some of the european ruby conferences.

I don't have a reference point, but RubyConf 2019 was an all-round excellent experience.

Speaking at RubyConf

I was the usual mixture of nerves and excitement - I knew I had prepared to the best of my ability, but like many, public speaking is definitely pushing myself outside of my comfort zone. The speakers' reception was a really nice way to chat to many of my fellow speakers & pick up a few tips (@molly_struve is a font of wisdom, on top of being a great runner!). There's always some comfort too in seeing that everyone else is nervous too!

After some AV shenanigans involving USB-C to HDMI dongles that were only just fixed in time, the talk itself went pretty smoothly. At the speakers' reception one speaker mentioned they'd stopped using speaker notes ever since an unfortunate setup incident meant they had to do without. Like many others, I was horrified at the thought but I genuinely think that I didn't actually use mine - more of an emotional crutch than anything else (I probably would have freaked out if I didn't have them though!).

Everyone I spoke to afterwards was really nice about the talk and @annarankin even made some lovely sketch notes, so I'm chalking that up as success.

Guiding at RubyConf

Every year a certain number of opportunity scholars (I'm told 20 is typical) get a free ticket + travel expenses to go to RubyConf. They're all paired with a guide, who's a friendly face & direct point of contact during the conference. It's a really well run program, with about the right balance of structure vs freedom.

Although 20 scholars + 20 guides + some organisers adds up to a lot of people, we were split up into cohorts that were much more manageable. We had some great discussions on jobs, speaking and more ( I definitely picked up a few tips!).

I later realised that fellow speaker Ben Greenberg was an opportunity scholar just 2 years ago (see his writeup from back then) - so cool to see the leaps people make! I loved the way that after just a short while my scholar was striding up to speakers for questions & stickers.

Talks at RubyConf

Picking which talk to go was a struggle - most slots I was torn between at least 2-3 options! There was an amazing range, from Sandi Metz's incredible "Lucky you" on social mobility and the unfairness that is just baked into the system, to life deep in the ocean sediment or livecoding in ruby on an apple ][.

I went to quite a few mruby talks, more out of curiosity than anything else and have ended up buying a small microcontroller (the m5stickC that featured in talk on mruby's init sequence) to play with!

All in all the lineup was really strong, and I feel honoured to have been part of it.

People at RubyConf

I wasn't on top form, between jetlag, a cold/headache that I never really shook and pre talk nerves followed by post talk crash, but I still enjoyed hanging out with some fun people, walking around East Nashville \u0026 chatting about all sorts of topics.

Ruby is an amazing community to be a part of and I love seeing the positive impact it has had on so many others too. I'll be back for sure!