DEV Community: José Haro Peralta

Faster time-to-market with API-first

José Haro Peralta — Tue, 25 Oct 2022 23:10:36 +0000

A few years ago, I had the opportunity to work in a short engagement with a London startup helping them to build an MVP for one of their latest products. They wanted to trial a strategic partnership with a bigger company within the same sector. The startup would provide a solution for an untapped niche of the market, while the bigger company would act as a catalyst for customers who trust an established brand.

The idea for the MVP was very simple: we had to build a small UI using an SPA framework, and a web service exposing a REST API that would allow both integration with the SPA and with the strategic business partner.

Despite the simplicity of the idea, and although the project had started just a few weeks before I joined, some parts of the project were already a hot mess. In particular around the API.

For starters, we didn’t have actual API documentation anywhere. We had a Google Doc with some notes about the endpoints, and a few examples of the payloads we could expect. The problem was the documentation was very incomplete, half of it was wrong, and it wasn’t maintained.

Needless to say, the API implementation wasn’t tested against the contract, since there was no contract at all! Hence the backend developers were able to release API changes without notice or visibility. Unfortunately, many of these changes weren’t really intended - they were bugs. And sadly, they often broke the integration with the API client. It was frustrating both for the UI developers and for the backend developers.

The second major problem was the backend team wasn’t using a proper API framework. They used plain Flask with custom payload validation which they themselves implemented. You can picture it: hundreds of lines of code dedicated to API validation (for dates, for timestamps, for string formats, and so on), most of them untested. No wonder the project was going slow.

To get the project back on track, we had to fix the API documentation, get a proper API framework in place, and make sure no releases were allowed if they didn’t comply with the API specification. Let’s see how we tackled each of these issues.

Fixing the API design and docs

It turned out the team just wasn’t aware of REST APIs best practices and standards and didn’t know about the OpenAPI specification. So the first thing I did was to explain what OpenAPI is and how it works. Then we consolidated the API documentation in an OpenAPI specification. This allowed us to be very clear about what to expect from the API.

The process of consolidating the documentation into an OpenAPI specification also brought to light plenty of problems with the previous API design. For example, dates were being represented with a custom format that required custom validation logic both in the server and in the UI. We replaced those dates with ISO standards, which are supported by OpenAPI.

Some schemas also proved to be too flexible and therefore barely useful for validation. In fact, under the name of reusability, the same schemas were reused for different entities, resulting in very complicated schemas in which most properties were optional. To improve validation, we refactored the schemas and created a one model per entity, with different endpoints.

Another limitation of the previous design was making inappropriate use of HTTP methods and status codes. The only HTTP methods they used were GET and POST, and all the responses returned 200 status codes. This wasn’t very useful. The improper use of HTTP methods meant it wasn’t clear when we were trying to create, delete, or update a resource. And since all the responses returned 200, it appeared they were all successful and you had to inspect the actual payload in search for an “error” property to determine whether the request succeeded or not.

From a developer point of view, it may seem like a good idea to “reuse” endpoints. For example, if you capture create, update, and delete operations through a POST endpoint, you’ll have less endpoints to maintain. And if all those endpoints have similar blocks of code, you may also save a few lines. However, you can also do all these things while exposing different endpoints for each operation. This is a common mistake I often encounter among API developers: we expose our implementation details through the API interface, and that happens because we think of the code first. Instead, I encourage you to design your API first, and think about the implementation later.

Consolidating the API specification with OpenAPI was a turning point for the project. From that moment we were able to run mock servers to build and test the UI before integrating with the backend, and we were able to validate the backend implementation against the specification. We used prism to run mock servers, and Dredd to validate the server implementation (these days I’d rather use schemathesis).

Fixing the API release process

Having API documentation is very good, but without testing the implementation against the API specification before a release, it’s not much use really. Sure it helps us get a clear understanding of how the API works. But the power of API documentation is serving as a validation tool - it helps us verify that the server is correctly implemented.

To ensure the API server worked as expected, I included the Dredd test suite in the Continuous Integration server, so nobody would be able to merge and release new code unless it was validated by Dredd and therefore compliant with the API specification. Thanks to this, we stopped having silent changes to the API server. From that moment on, any changes to the server would have to be documented first, and the API server would have to comply with the changes before merging or releasing.

Improving the server implementation

The API server was implemented with Flask, a popular Python framework for building web applications. Before I joined the team, they’d been using plain Flask to build the API, and they wrote a lot of custom code to validate API payloads. This is a common mistake I see often among less experienced API developers.

Don’t get me wrong - building your own API validation layer isn’t necessarily bad, but if you go down this route, you’ll end up reinventing the wheel. APIs require complex validation logic for both payloads and URLs (path and query parameters), and this logic has to be applied throughout the API. So if you build your own API validation layer, you’ll end up creating an API framework. The thing is, there’re tons of API development frameworks out there, and very good ones, so why not use one of them?

When it comes to Flask, in particular, there’re plenty of choices. And in fairness, not all frameworks are created equal. You’ve got flasgger, restx (successor of flask-restplus), flask-RESTful, and flask-smorest, to mention a few. How do you choose among those???

When choosing a REST API development framework, you’re looking mainly for the following factors:

It supports OpenAPI out of the box: if you’re going to build a REST API, you need to use a framework that knows how OpenAPI works. Otherwise, you’ll get nasty surprises when it comes to payload validation. The easiest way to determine if a framework supports OpenAPI is by checking whether it can auto generate API documentation from your code. Flasgger and flask-smorest both do this.
Uses a robust data validation library: validating payloads is a complex business. Your data validation library must handle optional and required properties, string formats like ISO dates and UUIDs (both dates and UUIDs are string types in OpenAPI), and strict vs loose type validation (should a string pass as an integer if it can be casted?). Also, in the case of Python, you need to make sure 1 and 0 don’t pass for True and False when it comes to boolean properties. In my experience, the best data validation libraries in the Python ecosystem are pydantic and marshmallow. From the above-mentioned libraries, flasgger and flask-smorest work with marshmallow.
It validates everything: It validates request payloads, response payloads, URL path parameters, and URL query parameters. I’ve noticed some libraries only validate request payloads and provide very little support for validating URL parameters or response payloads. In those cases, make sure at least the library gives you a way to enforce your own validation rules for responses and URL parameters. If your library uses marshmallow, you can always use the marshmallow model directly to validate a response payload.
Maturity: if your business (or your job) depends on the APIs you’re building, you want to build them with robust and mature libraries. Look for libraries that have been around for a while and with lots of users and contributors. Look for libraries with an active community, in which users raise issues often, but check that the issues are quickly addressed. It also helps if the library has good documentation.

After applying this analysis, we chose to work with flask-smorest, which is a Flask plugin that allows you to easily build REST APIs using marshmallow for data validation. This not only allowed us to remove hundreds of lines of custom data validation code - it also improved data validation. Both request and response payloads were now properly validated. Also, while URL query or path parameters were not being validated before, now marshmallow was taking care of all that.

Using a proper API framework was a game changer - it gave us a properly working API. Because the framework takes care of everything in the API layer, we were able to spend less time working on the API layer, and focus our efforts on the business layer and the application’s data model. Our development speed got faster and we were able to release better software more often.

Moral of the story

APIs are deceivingly simple. At the end of the day, APIs are everywhere and we all use APIs all the time. However, in reality APIs are complex pieces of software. Anyone can build a simple API, but then anyone can also write some random code. Just like the most challenging thing in software development is writing readable and maintainable code, the most challenging thing in API development is delivering good interfaces that work as expected, are easy to consume, and easy to change.

Delivering good APIs is difficult because it requires alignment with the business: our API must meet the requirements of our organization. It requires alignment between the client- and the server-sides of the API: they must work together and use the same contract, otherwise the integration won’t work. And we must ensure that both the client and the server are implemented according to the specification.

Many skills are needed to make this happen: you need to know how to engage with the stakeholders and gather requirements, how to translate those requirements into technical details, how to design an API, how to document it, how to choose a good API framework and how to use it correctly, how to test the API, and how to ensure the implementation follows the design. And this isn’t even considering API security, deployments, and operations. Those too are extremely complex topics, but that’s for another day.

If you’re building business-critical APIs, my recommendation is don’t leave it to your most junior developers on their own. By all means engage junior developers, but make sure their work is supervised by a senior developer who knows how to build APIs. If you don’t have previous experience building APIs, stick to best practices: design the API first, document it, build according to the spec, and validate it against the spec. And don’t reinvent the wheel - use proper frameworks!

I know it sounds like a lot of work: you need to learn OpenAPI, figure out how to use API testing frameworks like Dredd and schemathesis, research API development frameworks, and so on. But it’s worth the effort. The alternative is a downward spiral of API integrations and software quality problems that will hold your progress back and will cost your business a lot of money to fix.

There’re a lot of great resources to help you get started with best API development practices. You may want to check out the works of Kin Lane, Arnaud Lauret, Mike Amundsen, Erik Wilde, Mehdi Medjaoui, Ronnie Mitra, James Higginbotham, Corey J. Ball, JJ Geewax, and yours truly, among others. Don’t say you couldn’t find any resources!

And if you need additional help, reach out to the experts directly (check previous paragraph)! Based on my own observations working with different clients and watching many API disasters, I estimate that companies without adequate talent or experience waste somewhere between $50k and $500k trying to get their APIs right. Sometimes it’s way more than that. But money isn’t the biggest problem here: it’s the legacy code that gets built along the way, the burnout among the employees (who often end up leaving the company), and the loss of business opportunities, either because the project ships late, or because it gets canceled altogether. So even if you have to pay $50k for a consulting/training session to get things right, you’ll still be much better off: you’ll have saved time and money by getting yourself on the right track from the beginning.

If you enjoyed reading this article and found it useful, you’ll like my book Microservice APIs. It teaches you everything you need to know to build APIs from the ground up, including design, documentation, implementation, testing, security, and deployments!

You can download two chapters of the book for free from this link.

You can also use the following code to obtain a 40% discount when buying the book: slperalta.

I also run a very popular series of workshops, both free and paid ones, in which I teach best practices for building microservices and APIs. I’d love to see you around in one of my workshops! An updated list of upcoming workshops is always available here: https://microapis.io/workshops.

API-first development maturity framework

José Haro Peralta — Tue, 06 Sep 2022 14:29:29 +0000

In conversations with software developers, I’ve noticed that most of them claim to be API-first in their API development strategy. In fact, Postman’s 2021 State of API Report found that 67% of respondents ranked themselves with a score of 5 or higher (on a scale of 0 to 10) in terms of embracing API-first development.

In the past years, I’ve helped many teams build and deliver APIs. Most of them claimed to be API-first. But as it turned out, being API-first took on a different meaning in every project. This isn’t surprising. Postman’s 2021 State of the API Report includes the following definitions of API-first (with the percentage of respondents that subscribed them in parentheses):

“Defining and designing APIs and schema before beginning development” (42% or respondents)
“Developing APIs before developing applications or integrations” (31% of respondents)
“Defining business requirements before defining and designing APIs” (16% of respondents)

The remaining 10% of respondents admitted they don’t know what API-first means.

Despite the diversity of perceptions around what API-first means, over the years I’ve noticed an increasing consolidation of the concept. When I’ve had the opportunity to work with the same team for a long period of time, I’ve noticed that over time the definition of API-first becomes more clear and solid. And as the concept of API-first matures, I also notice how API development methods become more sophisticated and standardized. In view of this, I’d like to propose a framework for analyzing API-first development strategies. I call it the API-first maturity framework (yes, inspired by Leonard Richardson).

I want to make it clear that this is a framework for analyzing API-first development strategies. I don’t analyze API-first strategies in general, but focus specifically on API-first development: the strategy you use to build your APIs. Since we’re talking API-first, the framework assumes that there’s always an element of API design before heading on to the implementation. However there’re no assumptions about how you design the API or how and when you document it. In fact, this framework considers different strategies for documenting APIs and evaluates them in terms of their benefits and constraints. The discussion also focuses mainly on REST since it’s the most prevalent type of API, but the same conclusions in my experience apply to GraphQL and other types of APIs.

In the early days of API development, very few people had a clear idea of how to design and document their APIs, and the API development workflow was full of hiccups. I used to say that no API server survives first contact with its client. And to an extent, this is still true even for the most advanced API development workflows. However, the benefit of a robust API development workflow is not to eliminate all integration errors, but to give you more control and observability of the errors and enable you to fix them more quickly.

The ideal is being able to deliver API integrations that work regardless of the complexity of the API and the integration. This means that both the API client and the API server development teams have a crystal clear idea of how the API works, and that both the API client and the server are implemented and validated against the API design. I assess the maturity of each API documentation methodology and each API development workflow in terms of their ability to help us reach this ideal.

I divide the analysis into two categories: 1) API modeling and documentation strategies and 2) API development workflows. For each category, I’ll discuss the most common strategies and workflows that I’ve found when working with different teams. Before heading over to the analysis, here’s the list of documentation strategies and development workflows that I’ll analyze:

API modeling and documentation strategies:

JSON Examples
Data modeling with programming languages
OpenAPI
Less standard API description languages

API development workflows:

Server first, client later
Code-first
Bi-directional testing
Developing the API client against manually crafted mocks
Developing the API client against the spec using mock servers
Validating the server against the spec using contract testing tools

I’ll begin by analyzing API modeling and documentation strategies and then I’ll analyze API development workflows. I’ll close the article with a summary and discussion.

API modeling and documentation

API modeling refers to the process of designing the API endpoints and schemas, while API documentation refers to the process of describing the API design. In this section, I analyze the most common strategies that I’ve come across for modeling and documenting APIs.

JSON Examples

With JSON Examples, we design the API endpoints and their associated payloads in the form of JSON examples. So instead of having schemas for the payloads, we have examples of what the payloads look like. This approach works fine for very simple APIs. However, APIs with complex schemas that include optional properties or properties with multiple types aren’t easy to document with JSON Examples - it means having different JSON examples for the same payload. And when the payload design changes, it means updating all the JSON examples. There are three big flaws in this method:

Incomplete documentation: JSON Examples aren’t usually exhaustive and some cases are typically missing (specially for complex payloads); it’s also very easy to produce examples that contain mistakes.
Maintenance burden: as the API design evolves, we need to update all the JSON examples. In my experience, it’s common to forget to update some examples, which ends up making the collection of examples unreliable.
High noise-to-signal ratio: With multiple examples per payload, it’s difficult to get an idea of how the API actually works. You end up having to infer the schemas by collating multiple examples.

Data modeling with programming languages

This is a surprisingly common approach to API documentation. In this approach, developers enumerate the API endpoints and model their payloads as types using programming languages (such as TypeScript). The appeal of this approach is that the defined models are reusable straight away in API client and server code.
When properly done, this approach is quite powerful, as it allows you to define reusable models, optional properties, and properties with multiple types. It also has a nice and clear syntax that everyone (who’s familiar with the chosen language) understands. The downside is that we’re mixing data types with data schemas. An API schema is not exactly a type - it’s far more precise than that. A schema allows you to do things that you can’t do with types, like defining the minimum length of a string or array, describing dependencies between properties, and so on.
The better version of this strategy is when the models can be exported to JSON Schema or at least can be used to generate JSON Examples. Any update to the models can come with an update list of JSON Examples that API client developers can use for their testing.

OpenAPI

The gold standard. The OpenAPI specification was created to satisfy the documentation needs of REST APIs, and for all of its imperfections, it really does a great job of describing how an API works.
Unfortunately, this approach isn’t quite as common among self-claimed API-first developers as one would expect. Why is that? When I ask software developers why they don’t use OpenAPI to document their API designs, the most common answer is that they find OpenAPI difficult to understand and to work with.
Behind that reasoning, there seems to be an assumption here that you must write your OpenAPI specifications by hand. However, there really are many different ways to produce OpenAPI documentation: you can use a framework to generate documentation from code, or your can use one of the many tools that make it easier to produce OpenAPI documentation, like Postman’s API Builder, Insomnia’s Designer, or Stoplight’s Studio.
Using documentation-generation tools is very effective for those less familiar with OpenAPI. What I’d say is that you must check out the output of those tools to make sure it really aligns with the API design you want to achieve, and make any necessary amendments.
OpenAPI has a large community and a vast ecosystem of tools and frameworks that make it easier to design, build, test, and work with APIs. Once you have your OpenAPI documentation at hand, you can leverage this ecosystem and turbocharge your API development process.

Less standard API description languages

A common complaint about OpenAPI is that it’s difficult to learn and to read. Consequently, over the years we’ve seen many alternatives to OpenAPI, such as RAML, WADL, API Blueprint, and others. The problem with many of these alternatives is that in most cases they aren’t really more readable or easier to learn.
Simpler description languages also tend to support less capabilities for documenting API features. Finally, being less standard, those alternatives have smaller communities, and smaller ecosystems of tools and frameworks than OpenAPI, which makes for a poor development experience.

API development workflows

API development workflow refers to the process followed to build an API. When it comes to API development workflows, the biggest differentiator is whether you have an API specification in place before you begin implementing, or whether the API specification comes after you implemented the API. As I mentioned earlier, this analysis is for API-first strategies, so in all cases I assume that there has been an element of API design, with the exception of the “Server first, client later” strategy.

Server first, client later

In this approach, you build the API server first, and then you build the API client. In the early days of API development, this was a very common approach. Surprisingly, this approach still exists. The idea is that you don’t know what the API is going to look like or how it’s going to work until it’s built, and therefore it doesn’t make sense to build the client before the server is implemented.
This approach often goes hand in hand with a lack of API design and documentation practices. I remember following this approach many years ago, and it was most frustrating. Documentation was an afterthought of the API development process, and you’d often have to hack around the endpoints or dive into the server code to figure out how the API worked.
Sadly, I’ve also encountered this approach among teams that design and document their APIs first. The reason why client developers decide to wait for the server implementation even if the documentation is available is because they don’t trust that server developers will comply with the API specification, which means that the API server isn’t validated against the specification. The obvious risk of this approach is ending up with an API server implementation that doesn’t reflect the intended API design.

Code-First

Code-first generally fits within the concept of self-documenting APIs. This approach is very popular among backend developers. The idea is to use API development frameworks that generate API documentation from our code, and to publish that documentation.
Advocates of this approach claim that this is the only way to offer reliable API documentation and to keep it up to date, since it automatically updates itself when you change the code. The problem with this approach is that it presumes that the API implementation must be the source of truth. In practice, this approach means that the server isn’t checked for compliance against the API design.
This approach makes the lives of API client developers miserable, since new server releases can easily break the existing API integration. As an API client developer, it’s very frustrating to spend hours testing your code against the server, only for a new server release to introduce changes in the API. A recent post in Reddit illustrated well the resentment this approach creates among client developers. You can argue that better team communication can prevent this kind of problem. However, if you’ve been in software long enough, you know that communication isn't enough. You need to have automatic validation in place.
Another disadvantage of this approach is that API documentation generation tools are usually limited and sometimes can be wrong. For example, most frameworks don’t allow you to create OpenAPI links. Documenting security schemes is also challenging unless you use the framework’s specific security implementation, which may not satisfy your needs.

Bi-directional testing

This approach is popular among the contract-testing community and it was popularized by Pactflow. The idea is to implement the server with a self-documenting framework and to build the API client using manually crafted mocks for testing. Then you generate the API specification for the backend and the API specification for the client. Then you compare both specifications and see whether they agree. The nice thing about this approach is that it forces conciliation between the server and the client before you release the code.
The downside of this approach is that neither the server nor the client are directly implemented against the API design. Instead of using the API specification as the single source of truth, this approach lets the server and the client generate their own specifications, and it’s up to the developers to resolve the differences between them. This can cause unnecessary friction and delays in the API development process and can make discussions more difficult, involving specific implementation details of each side of the API.

Developing the API client against manually crafted mocks

In this approach, you manually create mocks to build the API client. The most traditional version of this approach involves creating mock responses in the form of JSON examples, while recent frameworks allow you to run mock servers based on those JSON examples and some additional custom configuration.
This is the most traditional approach to building API clients, but also the most common to this date, and sadly also the most error-prone. As we discussed earlier, JSON Examples tend to be incomplete and often wrong. They’re also a maintenance burden and become deprecated as soon as the API changes.
The biggest limitation of this approach is that it doesn’t take advantage of a full API specification, and as the API design changes, there’s a potentially growing drift between JSON examples and the actual API.

Developing the API client against the spec using mock servers

In this approach, you first produce an API specification, and then you run mock servers directly against the specification. Then you build the client against those mock servers. A mock server is a fake server that replicates the behavior of the real server based on the API specification. This is the best approach for building API clients, since it gives you the closest experience to interacting with and testing against the real API server. It’s a big advantage over traditional approaches like using manually crafted mock responses, since you don't need to create and maintain JSON Examples anymore.
In my experience, running API mock servers against the specification is very effective, although not all API designs are easily mockable. For example, APIs that return time series data in which the dates must be sequential and within certain bounds are difficult to mock. APIs with badly designed data models can be very difficult to mock as well (see examples in this article). In those cases, it’s still possible to leverage mock servers as long as you can provide examples or additional configuration.
There’re many different ways to launch API mock servers and this is a list of the most commonly used tools:

Stoplight’s prism: the easiest way to launch a simple mock server locally.
Microcks: an increasingly popular framework for running mock servers locally and in the cloud, with nice visualization dashboards and many other features.
Postman: a popular choice for those who’re already familiar with Postman, allowing integration with existing collections.
Wiremock/MockLab: a classic in the API mocking space, MockLab allows you to customize your mock server with your own stubs.
(upcoming) microapis.io: in my previous work, I’ve always come across annoying limitations in the previously mentioned tools, like the inability to save results, add examples outside of the API documentation, leverage OpenAPI links, or obtain randomly successful and unsuccessful responses with varying payloads and response latencies. I'm currently building microapis.io’s mocker to address these limitations.

Validating the server against the spec using contract testing tools

In this approach, you produce an API specification first, then you build the API against the specification, and then you validate your implementation against the specification using automated API testing tools. This is the most reliable approach for building API servers, since it’s the only one that holds the server accountable and validates the implementation against the source of truth.
Unfortunately, this approach isn’t as common as it should be. One of the reasons why it isn’t so common is because it requires you to produce the API specification first, which, as we saw earlier, puts off many developers who don’t know how to work with OpenAPI. However, like I said before, generating OpenAPI specifications doesn’t need to be painful since you can use tools for that.
In this approach, you use automated API testing tools to validate your implementation. Tools like Dredd and schemathesis. These tools work by parsing your API specification and automatically generating tests that ensure your implementation complies with the specification. They look at every aspect of your API implementation, including use of headers, status codes, compliance with schemas, and so on. The most advanced of these tools at the moment is schemathesis, which I highly encourage you to check out.

Summary tables

The below tables summarize the analysis from the previous sections and scores the maturity of each strategy and workflow on a scale of 0 to 3, with 0 being the most immature and 3 being the most mature.

API documentation methods

Like I said before, OpenAPI is the gold standard for documenting REST APIs - it’s the most widely supported documentation format and it offers the highest degree of granularity for documenting API features, hence it gets the highest maturity score.
JSON Examples are the most basic form of API documentation. They tend to be incomplete and inaccurate, and they’re difficult to maintain, hence they represent the most immature type of API modeling and documentation.
Modeling with programming languages is better since it forces you to think about data models as opposed to specific examples, hence the score of 1.
Finally, using a proper API description language, even if it isn’t widely supported, is still better than using JSON Examples or programming language models, hence the score of 2.

API doc method	Benefits	Challenges	Documentation format	Maturity
OpenAPI	Highly detailed docs	Learning OpenAPI	JSON or YAML	3
Less standard description languages	Very customizable docs	Smaller ecosystem	Custom file formats	2
Programming languages	Reusable code	Inability to represent schema constraints	Language-specific files	1
JSON Examples	Real payload examples	Maintenance burden	JSON	0

API development workflows

Developing and validating against a unique source of truth is the most effective way for delivering reliable API integrations, hence why “API server tested against the API spec” and “API client tested against the API spec” get the highest rating.
On the other side of the spectrum, the “Server first, client later” strategy is the most ineffective and unreliable way to deliver API integrations since it doesn’t validate the server implementation and it rarely exposes reliable documentation, hence the score of 0.
Code-first and API client against manual mocks are sub-optimal since they don’t enforce a single source of truth, but at least they have a concept of API documentation, hence the score of 1.
Bi-directional testing forces us to conciliate the API client and server implementations, which is a neat feature to ensure reliable integrations. However the lack of a single source of truth for reference and validation causes friction and earns it the score of 2.

API Workflow	Benefits	Challenges	Maturity
API server tested against the API spec	Ensures API server accurately reflects the spec	Producing the OpenAPI documentation	3
API client tested against the API spec	Accurate reflection of how the API server works	Some APIs are not easily mocked	3
Bi-directional testing	Forces conciliation between client and server	Doesn’t rely on a single source of truth	2
API client against manual mocks	Testing against payload examples	Non-comprehensive and quickly deprecated examples	1
Code-first	Accurately documented API implementation	Unaccounted/untested API changes	1
Server first, client later	You know how the API server works	Undocumented API changes	0

Conclusions

APIs are fundamental components of modern platforms, and they give us access to many of services across the Internet. The sustainability of many businesses depends on having robust and reliable APIs. That’s why it’s important to invest in solid API modeling and documentation processes and development workflows. Hopefully, the discussion in this article has given you some insight into the different ways you can approach these tasks, and how they fit within the whole spectrum of choices.
Although I do have a preference towards the most mature API documentation and development workflows, I reckon it isn’t always easy or possible to get started straight away with them. Embracing API design, using the OpenAPI specification, and leveraging contract-testing tools require skills and knowledge that sometimes aren’t there. They also require an additional investment of time and resources that sometimes cannot be afforded. And in fairness, there are situations, when your APIs are very simple and not business-critical, in which you may be able to get away with plain JSON examples and simple development workflows.
The goal of the API-first development maturity framework is to make you aware of the benefits and constraints of each approach, so that given your specific circumstances, you can assess the risk of each approach and choose the strategy that best fits your needs.

You can download two chapters of the book for free from this link.

You can also use the following code to obtain a 40% discount when buying the book: slperalta.

I also run a very popular series of workshops, both free and paid ones, in which I teach best practices and advanced patterns for building microservices and APIs. I’d love to see you around in one of my workshops! An updated list of upcoming workshops is always available here: https://microapis.io/workshops and here https://www.eventbrite.co.uk/o/jose-haro-peralta-43771267993.

Documentation-driven development for APIs: what is it, how do you do it and why you should do it?

José Haro Peralta — Sun, 27 Feb 2022 23:18:23 +0000

The code for this post is available under: https://github.com/abunuwas/documentation-driven-development

Documentation-driven development is an approach to API development where you write the documentation first, and then implement the API as per the specification. If you have any clients of the API within your system (for example a frontend application) then you implement them against the specification as well. This approach is often also called API-first.

There’s often this idea that API changes should be driven by the backend, and that the backend can change the API at any time and then the API client (for example a frontend application) has to comply with whichever arbitrary changes were made to the backend.

Many developers have the idea that you can’t start working on an API client (for example a frontend application) until the backend API is implemented. This is simply not true: if you write the documentation first, then both the client application and the API server can be implemented at the same time.

However, not everything counts as documentation when it comes to APIs. APIs must be documented in standard formats, such as OpenAPI, so that we can leverage the benefits of documentation-driven development. I’ve seen many teams documenting their API using tools such as Confluence, Google Docs, Sharepoint, Dropbox Papers, and similar. This doesn’t work because we can’t generate standard specifications from them that we can use in combination with other tools to test our API implementation.

How does this work in practice? And is it beneficial at all? I’m going to show you how to practice documentation-driven development to develop a REST API with Flask. The approach is the same for any other kind of API and it works with any other framework.

We’ll implement a simple API to manage a list of to-do items. The schema for the to-do item will have the following attributes:

ID in UUID format
Created timestamp
Task as a string
Status as a string, which can be one of the following: pending, progress, completed.

The attributes ID created will be set by the backend and therefore will be read-only for the API client. The attribute task represents the task that has to be done, and the status attribute represents the status of the task and can only take one of the enumerated values. For best practice and reliability, we’ll invalidate any requests that include any properties not listed in the schema.

We’ll have two URL paths to manage our list of to-do items:

/todo
/todo/{item_id}

/todo represents the collection of to-do items, and we’ll use it to fetch a list of our to-do items and to create new to-do items. We’ll use /todo/{item_id} to manage specific tasks, and we’ll use it to retrieve the details of a specific task, to update it, and to delete it.

Now that we have gone through the process of designing our API, let’s document it first before we jump onto the implementation!

I mentioned at the beginning that we’d be implementing a REST API, so we’ll use OpenAPI to document it. Create a file called oas.yaml and write the following content to it:

openapi: 3.0.3
info:
  title: TODO API
  description: API that allows you to manage a to-do list
  version: 1.0.0
paths:
  /todo/:
    get:
      summary: Returns a list of to-do items
      responses:
        '200':
          description: A JSON array of tasks
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/GetTaskSchema'
    post:
      summary: Creates an task
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateTaskSchema'
      responses:
        '201':
          description: A JSON representation of the created task
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/GetTaskSchema'
 /todo/{item_id}:
    parameters:
      - in: path
        name: item_id
        required: true
        schema:
          type: string
          format: uuid
    get:
      summary: Returns the details of a task
      responses:
        '200':
          description: A JSON representation of a task
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/GetTaskSchema'
        '404':
          $ref: '#/components/responses/NotFound'
    put:
      summary: Updates an existing task
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateTaskSchema'
      responses:
        '200':
          description: A JSON representation of a task
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/GetTaskSchema'
        '404':
          $ref: '#/components/responses/NotFound'
    delete:
      summary: Deletes an existing task
      responses:
        '204':
          description: The resource was deleted successfully
        '404':
          $ref: '#/components/responses/NotFound'
components:
  responses:
    NotFound:
      description: The specified resource was not found.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  schemas:
    Error:
      type: object
      properties:
        code:
          type: number
        message:
          type: string
        status:
          type: string
    CreateTaskSchema:
      type: object
      required:
        - task
      additionalProperties: false
      properties:
        status:
          type: string
          enum:
            - pending
            - progress
            - completed
          default: pending
        task:
          type: string
    GetTaskSchema:
      type: object
      required:
        - created
        - id
        - priority
        - status
        - task
      additionalProperties: false
      properties:
        id:
          type: string
          format: uuid
        created:
          type: integer
          description: Date in the form of UNIX timestmap
        status:
          type: string
          enum:
            - pending
            - progress
            - completed
        task:
          type: string

Now that we have the API specification we are in an excellent position to start implementing the API server. If you’re working with another team that has to implement a client application for the API, such as a frontend application, make sure you make the API specification available to all teams in a central location, such as a GitHub repository or URL endpoint. I’ll write another post later on illustrating how you can do that.

To implement the API we’ll use Flask in combination with flask-smorest. Flask-smorest is a REST API framework that uses marshmallow to validate schemas. For illustration purposes we’re going to keep this super simple and the whole app will be in one file, and the list of items will be represented by an in-memory list. In a real app you want to use persistent storage and make sure different components go into different modules. The only dependency you’ll need to get the code working is flask-smorest, so make sure it’s installed. Create a file called app.py and copy the following content to it:

import time
import uuid
from flask import Flask
from flask.views import MethodView
from flask_smorest import Api, Blueprint, abort
from marshmallow import Schema, fields, EXCLUDE, validate

app = Flask(__name__)

app.config['API_TITLE'] = 'TODO API'
app.config['API_VERSION'] = '1.0.0'
app.config['OPENAPI_VERSION'] = '3.0.3'
app.config['OPENAPI_JSON_PATH'] = "api-spec.json"
app.config['OPENAPI_URL_PREFIX'] = "/"
app.config['OPENAPI_SWAGGER_UI_PATH'] = "/docs"
app.config['OPENAPI_SWAGGER_UI_URL'] = "https://cdn.jsdelivr.net/npm/swagger-ui-dist/"

API_TITLE = 'Orders API'
API_VERSION = '1.0.0'

api = Api(app)


class CreateTaskSchema(Schema):
    class Meta:
        unknown = EXCLUDE
    status = fields.String(
        default='pending',
        validate=validate.OneOf(['pending', 'progress', 'completed']),
    )
    task = fields.String()
class GetTaskSchema(CreateTaskSchema):
    created = fields.Integer(required=True)
    id = fields.UUID(required=True)
blueprint = Blueprint(
    'todo', 'todo', url_prefix='/todo',
    description='API that allows you to manage a to-do list',
)


TODO_LIST = []


@blueprint.route('/')
class TodoItems(MethodView):
@blueprint.response(GetTaskSchema(many=True))
    def get(self):
        return TODO_LIST


@blueprint.arguments(CreateTaskSchema)
@blueprint.response(GetTaskSchema, code=201)
    def post(self, item):
        item['created'] = time.time()
        item['id'] = str(uuid.uuid4())
        TODO_LIST.append(item)
        return item


@blueprint.route('/<item_id>')
class TodoItem(MethodView):
@blueprint.response(GetTaskSchema)
    def get(self, item_id):
        for item in TODO_LIST:
            if item['id'] == item_id:
                return item
        abort(404, message='Item not found.')


@blueprint.arguments(CreateTaskSchema)
@blueprint.response(GetTaskSchema)
    def put(self, update_data, item_id):
        for item in TODO_LIST:
            if item['id'] == item_id:
                item.update(update_data)
                return item
        abort(404, message='Item not found.')


@blueprint.response(code=204)
    def delete(self, item_id):
        for index, item in enumerate(TODO_LIST):
            if item['id'] == item_id:
                TODO_LIST.pop(index)
                return
        abort(404, message='Item not found.')


api.register_blueprint(blueprint)

You can run the app with the following command:

$ FLASK_APP=app:app flask run

As per the app configuration, you can visit the API documentation auto-generated from the app with a Swagger UI theme under /docs.

Great, now, how do we test this implementation to make sure it’s compliant with the specification? You can use a bunch of different tools and frameworks to accomplish this. Here I’ll show you how you can do it with Dredd. Dredd is an npm package, so to install it run the following command:

$ npm install dredd

To run dredd, execute the following command:

$ ./node_modules/.bin/dredd oas.yaml http://127.0.0.1:5000 — server “flask run”

If you run this now, you’ll get an error saying that we need to provide an example for the item_id parameter in the /todo/{item_id} URL path. You’ll see also some warnings about issues with the specification format, which you can safely ignore as the API spec is valid (you can validate that with external tools like this: https://editor.swagger.io/). Let’s go ahead and add an example fo item_id:

/todo/{item_id}:
    parameters:
      - in: path
        name: item_id
        required: true
        schema:
          type: string
          format: uuid
        example: d222e7a3-6afb-463a-9709-38eb70cc670d
    ...

If you run Dredd now again, you’ll get 5 tests passing and 3 failures. If you look closer at the failures, you’ll see that all of them are related to operations on existing resources under the /todo/{item_id} URL path. It seems Dredd is picking up the example ID we provided in the spec and expecting a resource with such ID to exist. Obviously no resource exists until we start running the API, so we want Dredd to actually create a resource first using POST /todo/, fetch the ID of the created resource, and use it to test the endpoints under the /todo/{item_id} URL path. How do we do that? Using dredd hooks!

Dredd hooks offer a simple interface that allow use to take action before and after a transaction. Every transaction is identified by a “name”, which is a combination of different parameters that uniquely identify an operation. To list all names available in your specification, run the following command:

$ ./node_modules/.bin/dredd oas.yaml http://127.0.0.1:5000 — server “flask run” -- names

The available names are each of the info blocks listed by the command:

We want the names of the successful actions under the /todo/{item_id} URL path, namely actions that return a success status code such as 200 and 204 (i.e. not those returning 404). Create a file called hooks.y and copy the following content to it:

import json
import dredd_hooks


response_stash = {}


@dredd_hooks.after('/todo/ > Creates an task > 201 > application/json')
def save_created_task(transaction):
    response_payload = transaction['results']['fields']['body']['values']['actual']
    task_id = json.loads(response_payload)['id']
    response_stash['created_task_id'] = task_id


@dredd_hooks.before('/todo/{item_id} > Returns the details of a task > 200 > application/json')
def before_get_task(transaction):
    transaction['fullPath'] = '/todo/' + response_stash['created_task_id']


@dredd_hooks.before('/todo/{item_id} > Updates an existing task > 200 > application/json')
def before_put_task(transaction):
    transaction['fullPath'] = '/todo/' + response_stash['created_task_id']


@dredd_hooks.before('/todo/{item_id} > Deletes an existing task > 204')
def before_delete_task(transaction):
    transaction['fullPath'] = '/todo/' + response_stash['created_task_id']

I’ll talk more about dredd and dredd hooks in another tutorial. You can run Dredd with these hooks with the following command:

$ ./node_modules/.bin/dredd oas.yaml http://127.0.0.1:5000 — hookfiles=./hooks.py — language=python — server “flask run”

Now everything passes:

Right, so now we are certain that our API implementation complies with the specification. If another team working on a frontend application has made similar tests, we can be quite certain that both the server and the frontend app will integrate nicely and without errors. Or at least we won’t encounter integration errors due to non-compliance with the API.

Let’s now say that we want to make a change to the API. As it turns, we’d like to be able to assign a priority to each task. This will result in a new field in task resource payload called “priority”, which may have one of the following values:

low
medium
high

The default value will be “low”.

How should we approach this change? We are practicing documentation-driven development, so first we’ll change the specification. Once we’ve updated the specification, we can change both the backend and the frontend. Any change to the backend API that doesn’t comply with the spec will result in a failed test and shouldn’t be released.

The updated schemas in oas.yaml look like this:

...
  CreateTaskSchema:
      type: object
      required:
        - task
      additionalProperties: false
      properties:
        priority:
          type: string
          enum:
            - low
            - medium
            - high
          default: low
        status:
          type: string
          enum:
            - pending
            - progress
            - completed
          default: pending
        task:
          type: string
  GetTaskSchema:
      type: object
      required:
        - created
        - id
        - priority
        - status
        - task
      additionalProperties: false
      properties:
        id:
          type: string
          format: uuid
        created:
          type: integer
          description: Date in the form of UNIX timestmap
        priority:
          type: string
          enum:
            - low
            - medium
            - high
          default: low
        status:
          type: string
          enum:
            - pending
            - progress
            - completed
        task:
          type: string

If we now run the Dredd command the test will fail:

$ ./node_modules/.bin/dredd oas.yaml http://127.0.0.1:5000 — hookfiles=./hooks.py — language=python — server “flask run”

To fix this we only need to update our marshmallow schemas:

class CreateTaskSchema(Schema):
    class Meta:
        unknown = EXCLUDE
    priority = fields.String(
        default='low',
        validate=validate.OneOf(['low', 'medium', 'high']),
    )
    status = fields.String(
        default='pending',
        validate=validate.OneOf(['pending', 'progress', 'completed']),
    )
    task = fields.String()
class GetTaskSchema(CreateTaskSchema):
    created = fields.Integer(required=True)
    id = fields.UUID(required=True)

If you run the Dredd command again, the tests now pass. In this example, updating the marshmallow schemas is sufficient to update the app. In a real application, you’ll be persisting data to a database and you’ll need to run some migrations to update your models as well.

If you want to learn more about API integrations, have a look at my new book Microservice APIs in Python”. It’s just been released through the Manning Early Access Program (MEAP), which means it’s still under development and you get a chance to give your feedback and request new content before the book is in print. Access to the book: http://mng.bz/nz48

If you’re interested in the book, you can use the code slperalta to get a 40% discount. You can also download two chapters for free from the following URL: https://www.microapis.io/resources/microservice-apis-in-python.

How employers can navigate the current software developer market

José Haro Peralta — Tue, 22 Feb 2022 10:17:34 +0000

The big skills crunch is coming and we need to rethink how we approach talent acquisition. According to a Mckinsey report from last year, most businesses expect to suffer from skills gaps soon and are looking to alleviate the problem by upskilling their workforce.

Demand for software developers is at nearly an all-time high, if not an all-time high. Developer salaries are reaching record levels. From the job boards, it appears that companies are struggling to hire senior developers more than anything else.

The ratio of senior developers to junior developers has always been small. According to Bob Martin's famous analysis, the population of software developers roughly doubles every five years. These days, however, the ratio of senior to junior developers is extraordinarily small. In the most recent StackOverflow developer survey, developers with less than 5 years of experience make up almost half of the demographics.

Traditionally, businesses have strived to keep a balance between the amount of junior and senior developers on board. The idea was to have enough senior developers to allow them to coach and guide junior developers while still being able to focus on their own work. This balance is breaking.
Amid fierce competition from the big players, many small and medium companies are giving up on their pursuit of senior developers and opting to bring in junior developers instead. What does this mean for small and medium companies?
Having teams composed mainly of junior developers is not necessarily a problem. But it's a situation you need to manage. If you have very few senior developers, you can't rely on them to be able to coach and guide your junior developers while focusing on their own work. You have to change your approach to upskilling junior developers.
Most companies don't have proper training programs for junior developers. They expect junior developers to somehow be able to grow and learn by themselves with the help and guidance of their more senior colleagues. In my experience, this works in certain situations, but it also leaves many otherwise capable developers behind. A recent thread in Hackernews captured the feelings of most developers around this problem, and one of the comments summarised how many of us see the situation:
Devs don't spend money on expensive training courses and certs to gain skills; we just get hired somewhere and then pester our co-workers.

In the current market conditions, you can't afford to leave any developer behind. You need to invest in your developers and make sure they receive the necessary support to grow into skilled developers.

How to make training effective

Now here's the important note for business managers: training your employees doesn't mean go tell them to do a course or two in Udemy or Coursera. Most online courses are good for basic training, but the real value comes from advanced training tailored to your needs and delivered in real time by subject-matter experts. You need to bring structured training as part of the job routine, and you need industry experts to deliver the training.

Let's dissect the recommendation in the previous paragraph. In particular, let's look at the following questions: why real-time training? Why training tailored to your needs? And why training delivered by subject-matter experts?

The benefits of real time training: real time training allows the instructor to interact with the audience, which has the following benefits:

The instructor can adapt the content on the fly. For example, if some of the attendants never heard of certain concepts, the instructor can explain them in advance to make sure everybody is on the same page.
The instructor can look at the audience's reactions and gauge whether they are following or not. Again, this allows the instructor to be adaptive to make sure the audience understands all the concepts.
The audience can ask questions during the lecture if they don't understand something.

Training tailored to your needs: it's great to do a course about AWS, or Kubernetes, or GraphQL, or any other topic. But if the course covers those topics exclusively from a generic point of view, there's a great chance that the content delivered doesn't address the needs of your team. If you're planning to use GraphQL, talk to the instructor about your use cases and the instructor will be able to adapt the course to your needs. That way the training will be a lot more effective for your employees, as they'll be able to relate their experience to it, and they'll be able to apply the lessons learned straight away into their job.

Training delivered by experts: you can argue that anyone can deliver training. And sure enough, everyone has something to teach. But to make the most of your corporate training, you want to engage subject matter experts. Not generalists or random professionals. You want instructors with hands-on experience in the field they're teaching, and with proven abilities to teach either through published books or courses. Think of the instructor as a consultant who'll not only teach your employees best practices around each topic, but also can advise on whether you're making the correct use of certain technologies.

But, why should I invest in training my employees?

Many companies think investing in employee training is a waste of money. After all, corporate training workshops are expensive, and your employees will be "out of work" for a few days. But the reality is far from that. From experience, I can say that the right training delivered by the right person will save you tens of thousands of dollars. Yes, there's a hidden cost to lack of training.

How's that possible? Consider the case of APIs, which is one of my subjects of expertise. If you ask around, most developers will say they have a solid understanding of how APIs and API authentication work. And sure enough, most developers have a fairly good idea of how APIs work. After all, we work with APIs all the time!

But consuming an API is a very different task from producing one. Think of it this way: we can all read books, but that doesn't mean we're all writers. It takes a special skill and knowledge to produce something. When it comes to APIs, common knowledge gaps are around available API technologies with their pros and cons, documentation standards, how to work with an OpenAPI specification, JSON Schema syntax, how to leverage API documentation for testing and validation, or to run mock servers, best practices around the design of the endpoints and resource modelling, API security and authentication, and many others.

Without guidance, each of the knowledge gaps listed in the previous paragraph typically cause development teams to lose weeks or even months doing research and learning. When deadlines are around the corner, it can be frustrating for everybody.

And it's also expensive! According to StackOverflow's 2020 developer survey, the median salary of a backend developer in the US is $120k. Assuming *roughly* 220 working days a year (11 months x 20 days), knowledge gaps cost businesses approximately $2,700 per developer per week. And assuming a developer only spends one month a year (it's usually more) filling knowledge gaps, the yearly cost sits at approximately $11k. In a team of ten developers, the cost rises to $110k per year! Bear in mind this is a conservative estimate.
These spiralling costs can be avoided by incorporating training into the daily job. Training brings expert knowledge and guidance for your team. And it also makes happy teams. Software is a knowledge-based industry, and one of the reasons developers leave their job is because they're not learning new things anymore. To build a strong, confident, happy, and productive team, you need to embrace training.

If you liked this article and would like to learn more about my work, please check out my websites:

Personal website: https://www.joseharoperalta.com
microapis.io: https://microapis.io

If you'd like to learn more about microservice APIs, feel free to check out my book Microservice APIs. You can use the following code to get a 40% discount: slperalta.

You can also download two chapters for free using the following link: https://www.microapis.io/resources/microservice-apis-in-python.

If you're currently working or looking to work with APIs and would like to arrange a training or consulting session, please feel free to reach out to me at info@algorizm.co.

How bad models ruin an API (or why design-first is the way to go)

José Haro Peralta — Sun, 09 Jan 2022 17:22:42 +0000

Documenting API schemas is important

OpenAPI allows you to define the schema of the data that’s exchanged over the API. You can define schemas for your query parameters, your request payloads, your response payloads, and more. Schemas are defined using a subset of JSON Schema syntax, and typically a schema is an object with a collection of properties. The following is an example of a simple schema with OpenAPI valid syntax:

Person:
  type: object
  properties:
    name:
      type: string 
    age: 
      type: integer

The point of having schemas in an API specification is to give clients an expectation of what the data that they’ll receive from the server will look like. In that sense, schemas are super important, since they allow client developers to build their integrations correctly.

Despite their usefulness, I often come across APIs that don’t provide schemas for their response payloads. These are typically APIs for internal consumption, usually an internal product of the company. In such cases, it seems developers think it’s an unnecessary overhead to write good documentation for the API. After all, it’s them who’re building the API server and the client at the same time. They talk to each other and they agree on what the API payloads look like. So what’s the point of documentation?

The point of documentation is that at some point the composition of the team is going to change. At some point, new team members will join, and sometime in the future the original team members probably won’t be around any longer. In that case, having documentation is extremely useful if not indispensable. Without documentation, the only way to get an idea of what the API response payloads look like is by interacting with the server and harvesting responses.

But not all schemas are the same

In some cases, we do have response schemas documented in the API, but with very poor models. I recently came across a response payload model with a schema similar to below definition:

Data:
  type: object
  properties:
    data:
      type: array
      items:
        type: array
          items:
            oneOf:
              - string
              - number
    columns:
      type: array
      items:
        type: string

When looking at the actual response payload, the data served by the API looked like this (not the actual data):

{
 "columns": [
  "date",
  "engine_type",
  "brand",
  "model",
  "value"
 ],
 "data": [
  [
   "2021–09–01",
   "electric",
   "01",
   "Tesla",
   10
  ],
  [
   "2021–09–01",
   "combustion",
   "01",
   "Ford",
   0
  ]
 ]
}

I call this a schemaless schema, or free-form schema. Schemas like this are useless. It’s not obvious how data is structured here. As it turned out, the data served by the API was meant to be represented in a table. The columns field tells us the column names of the table. The data field contains the values that must be populated for each column. In strict order.

See what the problem is here? Various problems actually. To begin with, we’re relying on positional order to determine how a value maps to a property. That’s a lot of information that the client application has to have and to manage.
It’s also problematic for the backend. The slightest change in the code could cause the values to come in the wrong order, or we might have arrays with missing elements, and everything would break or give us an erroneous representation of the data.

Schemaless schemas make testing difficult. Tools like Dredd and Schemathesis rely on your API documentation to generate tests and validate your API responses. A collection of free-form arrays like the above model will pass nearly every test, even if the length of the arrays or their contents are wrong. Schemaless schemas are also useless for API mocking, which is a fundamental part of building reliable API integrations.

The above schema can be improved with a few simple changes. Instead of relying on positional order in an array, we define an array of objects with explicit properties and values:

Data:
  type: array
  items:
    type: object
    properties:
      date: 
        type: date
      engine_type: 
        type: string
      brand:
        type: string
      model:
        type: string
      value:
        type: number

The data from this schema might look like this:

[
  {
    "date": "2021-09-01",
    "engine_type": "electric",
    "brand": "Tesla",
    "model": "01",
    "value": 10
  },
  {
    "date": "2021-09-01",
    "engine_type": "combustion",
    "brand": "Ford",
    "model": "02",
    "value": 100
  }
]

So much more understandable!! Isn’t it?? Designing good schemas is actually not that difficult. You just need to spend some time working on them. If that’s all it takes, how come we end up so often with bad schemas?

Code-first is a risky strategy

This situation is actually pretty common in APIs that have been built with a code-first approach, without much design consideration. That was indeed the case of the above-mentioned API. I could bring in a lot more examples of bad API models resulting from lack of a design stage (i.e. resulting from code-first). On one occasion, I had to work with an API in which date fields looked like this:

Date:
  type: object
  properties:
    day:
      type: integer
    month:
      type: integer
    year:
      type: integer

Apparently, the server and the client development teams couldn’t agree on what the date format should look like, so they opted for an object representation of the date. The funny thing about this model is it doesn’t even have a timezone field! In any case, date fields shouldn’t be a matter of debate. We have ISO standards. All languages and frameworks can handle ISO dates. OpenAPI accepts date and date-time as field types. If only developers had spent one minute looking at the OpenAPI documentation...

On multiple occasions, I’ve had properties which are defined as an array of just one element. Something along the lines of the below example:

Entity:
  type:
    object
  properties:
    type:
      type: array
      items:
        type: string
      minItem: 1
      maxItem: 1

I mean, why?? Why would you even do that? The property is singular (indicating it’s just one element). Arrays are for variable lengths of elements. Where in the world would you use arrays for just one element? The reason, again, was that the API had been built with a code-first approach. The part of the application that consumed this bit of data required input in the form of an array, and somewhere in the application the code would blow up if you gave it a list of more than one element.

What is missing in this case is a separation of concerns between the business layer and the API layer, and a data transfer object (DTO) that translates the input from the API into the format that is accepted by the code internals. In fact, by skipping all these things, we’ve achieved an excellent degree of tight-coupling between low-level application code and the API. The consequences of this tight-coupling are disastrous, since the slightest change in the server code can break the API and/or the client application in unexpected ways. The best part? All of this could’ve been avoided by spending five minutes designing the interface before jumping straight into the implementation.

Are we doomed?

Are we doomed? No, we’re not doomed. The moral of the story is that to build a good API, you need to spend some time designing it before jumping into the implementation. That doesn’t mean you can’t code from the beginning. If you don’t like to write raw yaml or JSON, you can use an API documentation-generation framework to drive the design of your API from your code. Just make sure you don’t commit your code and release it as the actual API implementation before the design has been agreed upon.

My recommendation is, once the design is agreed, use your documentation-generation framework to render the OpenAPI specification, make any changes to it if required, and commit the file. Your OpenAPI specification file becomes the source of truth for your API. From that moment on, drive changes to your API from that file, and test your API implementation against that file. Don’t use the dynamically generated documentation as the source of truth. Dynamically generated documentation is just a reflection of the backend code, which is not necessarily correct.

If you liked this article, you’ll also like my book Microservice APIs. The book is conceived as a one-stop guide for learning how to design and build microservices and how to drive their integration with APIs.

You can download two chapters of the book for free from this link.

And if you’re interested, you can get a 40% discount from the book’s price by using the following code on Manning’s website: slperalta.

In addition to my blog, I also upload video tutorials to YouTube.

Feel free to connect with me on LinkedIn!