First, a disclaimer: the problem presented by this blog post happened when using AWS Lambda, Honeybadger, and Ruby. If you're using a different cloud provider, error reporting service and/or programming language, this might not be a problem, which doesn't mean you shouldn't be aware of it.
That said, let's start.
Honeybadger is a popular service for monitoring errors and application uptime like many others. We use it in production for a long time and it's taking care nicely, however, when we started using Honeybadger to report eventual problems in our AWS Lambda functions, a very interesting and unexpected situation happened.
We started to notice that Honeybadger eventually failed to report errors from our lambda functions, which is very tricky because we rely on it to notifies us when something goes wrong. If there are no notifications, there are no problems, right? We have more important things to do rather than watching the logs in Cloudwatch all over the day.
This week I was intentionally provoking errors in one particular lambda function, just to check if Honeybadger would report errors, but it didn't.
According to the Honeybadger documentation (Architecture Deep-Dive), the Worker, which is responsible for pulling Notice from the Queue, runs in a different thread to not block the execution of your application while reporting errors. This is great, however, it represents a problem when running in serverless functions.
The AWS Lambda Execution Context page says:
Background processes or callbacks initiated by your Lambda function that did not complete when the function ended resume if AWS Lambda chooses to reuse the execution context. You should make sure any background processes or callbacks in your code are complete before the code exits.
Which means, when your lambda function finishes the execution, the Worker, which runs in a different thread, might not have finished sending the error notification.
Fortunately, the solution is simple. However, it took me some time to find it because, until the moment I'm writing this post, it's not available in Honeybadger documentation (yes, I needed to dive into their code).
When using the Honeybadger.notify method, make sure to set sync: true:
Honeybadger.notify(error, sync: true)
When using sync: true, Honeybadger will send the notification immediately, and not pushing it into the queue to be processed by the worker: https://github.com/honeybadger-io/honeybadger-ruby/blob/master/lib/honeybadger/agent.rb#L161
Alternatively, you can also call Honeybadger.flush after calling Honeybadger.notify to force Honeybadger to send all the notifications: https://github.com/honeybadger-io/honeybadger-ruby/blob/master/lib/honeybadger/agent.rb#L314
If you want to know more about this matter, take a look at this nice article in DZone: Multi-Threaded Programming With AWS Lambda
Top comments (4)
Thanks for writing this up! We'll get our documentation updated with info about using the sync option in the Lambda environment.
Thanks Benjamin! I couldn't expect a better outcome from this blog post 👍
We have released a new version of our gem that not only auto-enables sync: true when it detects the Lamba environment, but it also adds more Lambda specific information to the payload when reporting an error. :)
That's amazing Ben! Thanks for giving me this feedback!
I will upgrade the Honeybadger gem in my lambdas asap! :D