Swagger Search: Finding an endpoint in your microservice primordial soup

IG started its service-oriented architecture journey back in 2009.

The first proof of concept consisted of three services, deployed independently from the existing monolith: a big fat J2EE Enterprise Server.

We loved the freedom from the monolith so much, that during these years the service landscape at IG has been growing organically, boasting now more than 250 services, with tens of hundreds of endpoints.

Perfect services with clear boundaries were born, but due to time pressures, laziness, or other unfortunate practicalities, these perfect babies grew into bloated teenagers, accruing functionality marginally related to them.

But then, by virtue of some kind of asexual reproduction, the fat services were split into a new set of perfect services, condemned to repeat the errors of their parents.

Do we have ...?

The evolving nature of a microservice architecture is a strength, but it comes with a lot of different challenges, one of which can be summarized in the following question:

Do we have an endpoint that provides XXX functionality?

To document what was available, we ended up using a service registry, to know which services were available, and Swagger, to know what each service provided.

So the workflow to find a candidate endpoint would be:

1. Go to the service registry.
2. Grep for a name that seemed logical. Failing that, scan the list of all service names.
3. Find the Swagger doc url for the service.
4. Scan the available endpoints for something suitable.

If no service or endpoint was found, we would fall back to the “ask around the office” methodology.

This workflow was at first mildly annoying, but as we distilled our existing services, more and more of them would have suspiciously similar names, making the process more onerous.

Swagger Search

To ease that pain, IG’s Clojure Club rolled up their sleeves and built Swagger Search: a service that collects all the Swagger documents from all the services and presents a user interface that allows you to find endpoints in your microservice architecture.

This is what it looks like:

The service allows searching through the endpoint documentation, URL, HTTP method, its parameters, their types, or any fields in the request or response bodies.

What do we mean by type and fields? If you have the following endpoint:

"post": {
 "description": "Adds a new moderator for the chat.",
 "parameters": [
   {
     "in": "body",
     "name": "body",
     "schema": {
       "$ref": "#/definitions/LiveChatModerator"
     }
   }
 ],
 }
...
"LiveChatModerator": {
 "properties": {
   "id": {
     "description": "...",
     "type": "string"
   },
   "snippet": {
     "$ref": "#/definitions/LiveChatModeratorSnippet",
     "description": "...."
   }
 },
 "type": "object"
},
"LiveChatModeratorSnippet": {
 "properties": {
   "liveChatId": {
     "description": "...",
     "type": "string"
   },
   "moderatorDetails": {
     "$ref": "#/definitions/ChannelProfileDetails",
     "description": "..."
   }
 },
 "type": "object"
}
...

The endpoint will show in queries for the types LiveChatModerator, LiveChatModeratorSnippet and ChannelProfileDetails, and in queries for id, snippet, liveChatId or moderatorDetails.

Swagger Search has built-in support for Consul and Etcd, but you can plug your own service discovery mechanism.

It comes in three different flavors: as a docker container, as an executable jar, or as a library.

IG has open sourced the project under the Apache 2 license. You can find the source and instructions at Github.

Hope you also find it useful.

Acknowledgements

Thanks a lot to all the member of IG’s Clojure Club and special thanks to all those graduates that were forced by their demonic overlords to contribute to the project.
You did a great job and you taught me that Clojure can be learned in less than a month.

Thanks also to APIs.guru for providing all those Swagger APIs that helped enormously during testing.

And of course, thanks to everybody behind Swagger. It is an awesome tool.

Kudos to everyone!