Design-first approach

I own and manage software development company Flussonic. We are a small team of 70 people that design, develop and maintain different pieces of software and services.

Knowledge sharing

Several months ago our development lead to the understanding that we need to find out some way to exchange knowledge across different teams.

As I have grown from programmers, I always prefer to add some strictness and code to our instruments, so one of the ideas was to find a way to exchange knowledge in programatical way.

For example: we have a concept of license in flussonic and a single license can have limit of streams. Different systems (about 6 of them) have different names for this limit: channel_limit, channels_count, streamsCount, stream_limit, streams

How can people from different departments communicate if they use different names for the same object?

API specification and predictability

Second problem grew from another part of our landscape. As we write in dynamic typed language erlang, it is very easy to add some ad-hoc code deeply in internals that raises some data to api as some json field. Of course after one week no developer cannot even remember about this code, but customers find this field and start relying on it. So one year ago in 2020 we couldn’t even tell full comprehensive list of fields in our api responses.

We tried to solve this issue by adding some type hinting code to erlang that added strict checking of all output data. After several months we have got full specification that reflected our code. Such approach led us to abyss of pain with our admin ui: small fix inside of code was absolutely correct but it led to new bugs in admin ui that was relying on old structure.

It became clear that generating absolutely correct spec from code is useless because it doesn’t give any promises for future and doesn’t help to reduce amount of bugs.

So we got two different issues:

• problem of structuring exchange of knowledge
• establishing clear and stable contract of api

Looks like they are two different problems, but we have found single solution for them: OpenAPI. To be more correct, we have changed our development process to API-first with support of OpenAPI.

Of course it is worthy mention that it is impossible to find out a silver bullet that solves all aspects of exchanging knowledge between people, but we could touch at least one of them.

OpenAPI

OpenAPI itself is a standard, specification on yaml/json file that describes HTTP API. HTTP is a standard de-facto nowadays and by default we use today JSON as a data format. So OpenAPI helps to describe how a system can exchange data via HTTP with JSON representation of information. JSON is defined in the way compatible with JSON-schema standard.

Sometimes people say that OpenAPI is about REST, but it is not very valuable comparison, because REST itself is not strictly defined.

OpenAPI gives so strict and language-agnostic approach, that it became default standard for HTTP API definition. All other options like RAML or blueprint are not actual today anymore. The only competitors live in other protocols, for example gRPC that uses another protocol, transport and formats.

Using OpenAPI means that you have file (with possibility to split it into different files) with strict structure and full list of your endpoints with strictly defined schema of input and output. It is up to you how to manage this file. Some people generate it from code, some people generate code from this file. We have tried both ways and consider generating spec from code a useless waste of time.

It is not a minor tech detail: what to do first and where to have source of truth, there is an abyss between these two approaches: either you try to restore specification after you have written some code, designed by your developers, either you make clear specification with a well-defined process and then code is written.

Here we come to the solution to both problems: design-first process + openapi as a technical solution for it helped us to get a clean and stable API and share API details between different teams.

Now teams can declare the API of their services in the single repository and physically share the same description of some objects. It means that same object will have the same naming across all projects. It is really wonderful!

Design-first (OpenAPI) transformation

To adopt this approach, you need to change your development process and it is never easy. If you have something working, it is not easy to change it, especially when you work in agile and have releases very often. You just cannot stop down all development to change API.

How are we passing this narrow path?

First we have selected main project that will be default for all naming styles. As we have erlang, javascript, python, rails, rust in our landscape during last 12 years, there are many things to discuss and naming may be one of the most irritating due to its irrationality. There are absolutely no rational reasons to select between snake_case and camelCase and this is why it draws so much attention and so much emotions. Just because there is nothing except emotions when you select between different naming styles.

Then we have passed first iteration and generated OpenAPI spec from existing code. This is required only once, this code will never be ever used. It is the second step. You can get badly designed, badly named, unstructured specification that is so horrible that it is hard to believe that it is yours. This is ok, at least I had this result =)

At this point we had a code generator that could generate back erlang code from openapi spec. After several iterations, we could achieve working code that is fully generated from yaml spec. This point is very important, because after it you do not change your generated code anymore. It is even a good idea to check in your CI scripts that developer hasn't edited generated code by hands.

It is important that we haven't published our API spec at this time, because it required rethinking and editing names.

Now we are in the process of redesigning API, purifying it, cleaning naming and even rethinking data model.

Other projects will have to align naming, data structure and other conventions with Flussonic to provide unified API to all our customers.

Internal changes

Main idea of this transformation is that we write specification first and then start writing code. Schema files now live in separate single repository that is used for all company. Here are main ideas that we have found during adopting this process:

• YAML described specification is much simpler than code itself, but it give so much information to manager, product owner, analytics and even sales guys, that often they do not need anything else. This is a great leverage: people with different background can discuss same ideas without digging into implementation details, but still speaking about strict and programmable code. Excellent balance between implementation full of ceremony and native human language.
• It is mandatory to use code generation from schema. You need to be sure that all pieces of code strictly implement OpenAPI constraints and it is done by robot, not by human. People should write business logic, not write dumb code and they deserve a promise that dull wrapper code is strictly controlled by logic, not written by hands.
• It is really cool to start simultaneous development by different teams right after schema is confirmed and becomes a contract.
• Linters will help you a lot. They will enforce your code style and other rules. No need to waste time in arguing with colleagues: "Please, fix your documentation, it is incorrect! Yes, of course, right after release!"
• We do not like when tech writers have access to source code. With schemas it is not an issue anymore: they can edit well defined schema and their work will not be wiped out after next "major weekly refactoring". They should even start documenting implementation before it is even planned to be done: contract embodied in schema can be defined more explicitly in description tags that tech writers can write together with product manager.

What else?

We do not have any plans to adapt gRPC at the moment, however it seems that it has some subset compatible with OpenAPI. Seems that we are ready to integrate two systems.

We have prepared our erlang implementation of openapi tooling for opening source. It has cowboy endpoint, json schema parser, etc.

Right now we are migrating Rust to OpenAPI and Rails and it seems to be a challenge. I will share details of this process if it will be interesting.

Read whole https://apisyouwonthate.com/blog, it is a wonderful deep set of information related to this post.