Do you have a Ruby on Rails app that manages integrations with multiple 3rd party APIs or external data sources?
Do you struggle with keeping your application's domain logic separate from these external data sources and flows?
Do you feel like you have to customize main flows in your app for each data source or API?
Can I grow a few extra arms to raise my hands twice for all this? ✋✋✋
Here's what I'm going to cover in this post:
- How this became a problem
- The Service Object architecture we've committed to for untangling these dependencies (aka Skip to the Code)
- 2 months later - how it's been going
- The resources that were the most helpful to learn from and adapt for our use case (jump to resources)
The Problem: The Snowball Effect
I work on an eCommerce app that takes external data from our "sources" and makes that data available to our customers via a catalog. The value of our product is that it takes the data from multiple sources and combines it into one catalog.
To simplify the examples to come, let's think of the product as an online grocery store. Each data source is the maker of an item you can purchase in a grocery store. So you have farmers that supply the produce and cereal makers who supply the cereal, and the people who make Eggo waffles, etc.
So where we got into trouble with our app is that (perhaps unsurprisingly) each data source formats their data differently. They name things differently, they have different ways of matching up a SKU (an individual thing you can buy, like a snack-size pack of Oreos vs. the family-size pack of Oreos), and just think about The Thing They Are Selling differently from other suppliers.
We started out with what we call "integrations": a directory of namespaced code that houses the fetching of the external data and translating it into the common language of our Grocery Store offerings. It looks a little like this:
app/ |__ integrations/ |__ farmers/ |__ fetch_data.rb |__ format_data.rb ... |__ cereal_makers/ |__ fetch_data.rb |__ format_data.rb ... |__ catalog/ |__ update_skus.rb ... ...
At first, this was great! We took in the new data, structured it how we wanted, filled in missing data with default values, sanitized data - no problemo. But then we needed to add another integration. And - guess what?! - they didn't have an API. So now we had to make a CSV upload process that handles fetching this new integration's data. But we still wanted the same output: a catalog of consistent data across all suppliers.
And then we wanted to add yet another integration...and our minds sort of exploded, to be honest 🤯 . You had to be intensely familiar with each integration's incoming data structure and know THEIR domain-specific context like whether they use an API or CSVs to send us their data. And you had to support multiple intake processes that could all break in different ways. And we needed to funnel all this data to one, common process to populate our catalog.
The Solution: Rails Engines + Structured Service Objects
I'm not going to spend much time talking about Rails Engines (that deserves a post of its own!) but there's pretty good documentation out there about it if you want to learn more. The examples to follow don't require you to also use engines.
Isolate Code Related to External Logic
So the first change we committed to is that each integration with a different data source will have its import code live inside of a Rails Engine (we call them Components). A hand-wavey definition of an engine is ruby gem that is internal to your app. It has access to the main app's code, but has its own routing, test files, services, etc. For my team, this was an easy way to draw a line in the sand and say "if the behavior I'm working on belongs to the data source and not our core app, it goes into the engine".
Our app structure doesn't look much different now:
app/ |__ components/ # moved these integrations into a "components" directory |__ farmers/ ... |__ cereal_makers/ ... |__ catalog/ |__ update_skus.rb ... ...
but our main "Core" app Gemfile now looks like this:
# Gemfile ... gem 'farmers', path: 'components/farmers' gem 'cereal_makers', path: 'components/cereal_makers'
Now that we've pulled our integration code out of our core app, let's get down to the nitty gritty with dependency management inside our components!
Service Objects & Dependencies
To establish this pattern, we needed to commit to some non-negotiable expectations:
- Service objects will always return a "response object" with a specific shape
- Service objects maintain their own dependencies
- Service objects should handle their own failure as well as surface the failures of any dependencies
Here's what our service should look like at the end of this process:
# app/components/famers/app/services/farmers/lettuces/create_service.rb module Farmers class Lettuces class CreateService < BaseService self.build new(Farmers::Produce::InventoryManagerService.build) end def initialize(manage_inventory_service) @manage_inventory_service = manage_inventory_service end def call(lettuce_params) @lettuce_params = lettuce_params lettuce, created = create_lettuce inventory = @manage_inventory_service.call(lettuce) OpenStruct.new( created?: created, lettuce: lettuce, inventory_updated?: inventory.updated?, errors: [lettuce.errors, inventory.errors].join(', ') ) end private def create_lettuce lettuce = Lettuce.new(@lettuce_params) if lettuce.save return [lettuce, true] end [lettuce, false] end end end end
Expectation: Service objects will always return a "response object" with a specific shape
This is a personal decision for your team based on your own experiences and preferences. We have a completely separate frontend app and have struggled with error handling before. So for us, we wanted to always have our controllers return
status: :ok but have json objects that include errors if they exist.
We decided upon a structure with these guidelines:
- The mutated/created record is returned (usually with a key of the model name, like
- There is a boolean status key that indicates if the service completed its job or not (could be
- There is an
errorskey that returns the error message(s) or
We used an OpenStruct object, which is mutable (meaning a service could set
OpenStruct.new(success?: true) and then elsewhere it could be overwritten
OpenStruct.new(success?: false). This hasn't been an issue for our team, but is good to keep in mind.
Service objects maintain their own dependencies
This falls under the larger consideration of "how the heck do I manage all this state?!".
# app/components/famers/app/services/farmers/lettuces/create_service.rb module Farmers class LettucesController < ApplicationController def create results = Lettuces::CreateService .build .call(lettuce_params) ... end ... end end # app/components/famers/app/services/farmers/lettuces/create_service.rb module Farmers class Lettuces class CreateService < BaseService # bad def call(lettuce_params) @lettuce_params = lettuce_params lettuce, created = create_lettuce inventory = Farmers::Produce::InventoryManagerService .build .call(lettuce) ... end ... end end end # app/components/famers/app/services/farmers/lettuces/create_service.rb module Farmers class Lettuces class CreateService < BaseService # good self.build new(Farmers::Produce::InventoryManagerService.build) end def initialize(manage_inventory_service) @manage_inventory_service = manage_inventory_service end def call(lettuce_params) @lettuce_params = lettuce_params ... end ... end end end
Two Months Later: How It's Been Going
I can honestly say we've been loving this change! We've started converting over old service objects and creating all new ones in this style. It's made it so much easier to know:
- what is going to be changed/done when I call this service
- how will I know if it succeeded or failed?
It's made us become much more thoughtful about single-purpose services that don't need to be so generalized and vast. They can be specific and accurate to one case and then put together to handle generalized cases.
This blog post by Adam Niedzielski is the main influencer of how we handle dependency injection in Service Objects.
- It had the best examples of multiple services that rely on each other
- There is a github repo - with tests! - that give a more in-depth view on this approach
This blog post by Dave Copeland at Stitch Fix has some really great narrative about handling this same problem in a really big app with lots of developers working on the project.
- Includes some insight to how they created a gem to enforce immutable (meaning it can't be changed after the fact) service object responses
- Is opinionated (in a good way) about good vs. bad practices with service objects
This blog post on Toptal by Amin Shah Gilani is actually one of the first articles I read when searching for "a better way" of handling service objects. It ended up being a different path than we committed to, but it still a good generalized read about Rails Service Objects.
- Includes opinions on where a service object should live in your app
- Great example code
- Thoughts on what a service object should return
Top comments (5)
Hi! This looks awesome. How has it been working with Rails Engines?
I was looking at them for a project in the past, but decided not to do it at the time. It seems like there is some boilerplate needed to set them up, but they seem like a great idea.
Great question! Maybe I need to write about that next 😉.
I will be honest: engines have had their ups and downs. In my workplace, we introduced 3 engines into our app at once. One of my coworkers documented a list of all the small modifications you have to make on top of the bootstrapped engine from running that
plugin newcommand. I would recommend that if you plan on adding more than one engine - it's a great resource.
So far the pros have been:
Some of the challenges:
Overall I'm still happy with our decision to use them. But they are definitely not a tool I would recommend 100% of the time to 100% of projects. They're a very niche tool that works well in a specific context.
At current time, It's just a matter of typing
rails plugin new blorgh --mountableand you're ready to code your engine
Great post. I am still little confused(I can say exploring more about when I should opt for engines/component) about components. Will try exploring links shared, thanks for that.
I personally don't prefer returning 200 status code for all response. Its a good practice to return respective status code to FE. And in my sense, instead of constructing response with OpenStruct, constructing a PORO is great.
Thanks for commenting! Looks like there's a general curiosity about when to use Rails engines. I'm working on a post to dive deeper into the pros/cons I've personally experienced.
Totally agree about the status codes! This is unique to the way our frontend app is built and its own unique error handling. We ended up using an OpenStruct based on one of the tutorials I linked at the bottom. Looking back, we could have used many other shapes or tools, but I've been happy with the way OpenStructs enforce a specific type of notation and accessibility that we didn't have as much reliability with PORO ("plain old ruby objects"). But that is definitely a decision you can make and customize to the needs of your app!