In our last post, we introduced RubyLLM::Monitoring, a Rails engine that captures every LLM request your application makes and provides a dashboard where you can see cost, throughput, response time, and error aggregations, and lets you set up alerts so that when something interesting to you happens, you receive an email or a Slack notification.
But how did we do it? What mechanism does RubyLLM provide that we can use to capture all LLM requests? Or did we use something else?
RubyLLM event handlers
RubyLLM provides event handlers out of the box. You can use them to capture an event when a message is sent to the LLM and, for example, calculate its cost. This is how you’d use them:
Provided that you have gemini configured in config/initializers/ruby_llm.rb
chat = RubyLLM.chat provider: "gemini", model: "gemini-2.5-flash"
chat.on_end_message do |message|
Event.create(
provider: chat.model.provider,
model: chat.model.id,
input_tokens: message&.input_tokens || 0,
output_tokens: message&.output_tokens || 0
)
end
response = chat.ask("Write a short poem about Ruby")
In the code above, an event record is created when a message is completed, and the cost is calculated in an ActiveRecord callback. The solution is pretty simple and works perfectly, but it doesn’t scale very well:
- You need to add this manual tracking everywhere. Every chat instance requires this callback to be set up; otherwise you will lose that data. You can simplify it even more, but you’ll always have to set up the callback.
- Your instrumentation code and your business logic are tightly coupled, which makes both harder to maintain.
- This only works for RubyLLM::Chat instances. What about embeddings, image generation, and other operations? You’d need different mechanisms for each.
- Tracking full request metrics like latency needs more complex and intrusive code.
We needed something more comprehensive and automatic that doesn’t rely on us remembering to hook the instrumentation code everywhere. Luckily, Rails has something neat for us to use baked in.
ActiveSupport::Notifications
ActiveSupport::Notifications is Rails’ instrumentation API. It’s what Rails uses internally to track things like database queries, view rendering, controller executions, and more.
Using it is simple: you make your code emit events by calling ActiveSupport::Notifications#instrument(...), and subscribers can consume those events to do logging, monitoring, or whatever else you need. An interesting example is Prosopite, which hooks into sql.active_record events to detect N+1 queries.
This mechanism is especially important for libraries, as it decouples the business logic of the library and the business logic of the application that uses it. In the case of RubyLLM::Monitoring, the monitoring logic lives separately and subscribes to what it cares about. No coupling between RubyLLM and RubyLLM::Monitoring.
So, this is what we did in RubyLLM::Instrumentation to make RubyLLM emit events after each LLM call. RubyLLM::Monitoring, on the other hand, provides an event subscriber that captures the events and feeds them into its dashboard.
RubyLLM::Instrumentation
Instrumentation should be automatic and invisible. RubyLLM::Instrumentation achieves that: just add it to your Gemfile, run bundle install, and you’re done. RubyLLM will start emitting events for you to subscribe to.
Now, following the example above, the code becomes:
in config/initializers/ruby_llm.rb
ActiveSupport::Notifications.subscribe(/ruby_llm/) do |event|
# Do whatever you want with the event, in RubyLLM::Monitoring we store the event data in the database for later use
Event.create(
provider: event.payload[:provider],
model: event.payload[:model],
input_tokens: event.payload[:input_tokens] || 0,
output_tokens: event.payload[:output_tokens] || 0
)
end
Provided that you have gemini configured in config/initializers/ruby_llm.rb
chat = RubyLLM.chat provider: "gemini", model: "gemini-2.5-flash"
RubyLLM will emit the event, and it'll be captured by the subscriber above
response = chat.ask("Write a short poem about Ruby")
The code remains practically the same as in the original example, but the instrumentation becomes much simpler and decoupled, and there’s no need to repeat the same hook in multiple places.
In the example above, all ruby_llm events are captured, but you can subscribe to specific events. You can read more about the instrumented events and their payload in the project’s repository.
Wrapping up
RubyLLM::Instrumentation takes off the burden of manually instrumenting the code from the users’ shoulders. Originally written as part of RubyLLM::Monitoring, we extracted it into its own gem because we thought it was a fundamental tool, and as we needed it, other people might need it too to build a different monitoring tool, or an analytics tool, or set up logging differently.
Give it a try, send us feedback, and contribute if you want to!
--
If you’re building AI-powered applications with Rails and need help with architecture, optimization, or observability, get in touch.
Top comments (0)