This post was originally posted on Medium.
I’ve worked as a software developer for almost five years now, most of this time focused on developing backends for applications or other kinds of clients. I’ve learned a lot throughout this time and I want to share a bit of what I know in this text.
First Things First
One thing to point out before we start is the concept of an API. An API (Application Programming Interface) is, as its name says, a way to allow external parts to interact with some application, programmatically. It’s very helpful when some third-party software needs to handle data from your software, and, of course, you do not want to open your database to them. In cases like this, you simply can create a set of services, or operations, for them to use.
When someone builds an API that is accessible from some network (usually through the internet) we name it as Web API.
One of the most common standards for designing web APIs is the REST architecture. REST defines a set of properties (or rules) that constraints (for good reasons) the web API’s architecture. As long as you follow those properties, you will be fine. My tips will basically drive you through the right path for some of these properties.
The Simplistic (Yet Useful) Approach
You may think of a REST API as an answerer. Someone (a client) asks a question, the API listen to the question, thinks about the answer and then responds to it.
Thus, when designing a REST API, you must keep in mind this very simple flow:
- the request comes in;
- the API does whatever it has to do, as quickly as possible;
- sends the response to the client.
Everything that doesn’t suit this approach, shall not stay inside the API. Therefore, you shall not employ an API for operations such as timed-recurring tasks, server-sent events or caching engine.
Besides that, you should pay attention to the “as quickly as possible” statement. It’s not there by chance. Web API shall not host long-running tasks. Web APIs communication is built on top of HTTP requests and responses, and it’s very likely that all the clients will set up a timeout for those requests.
How much time defines ‘long’? Well, that’s up to you. But keep in mind that in the web world everything should be fast.
Designing Long-Running Tasks
Ok, I should not run a long-running task on my API, but I do have one to run. So, how can I design that?
In my opinion, the best design for long-running tasks is to receive the request for the long-running operation, store this intent somewhere (usually a database) and then return an answer to the requester with a URL where the client could consult the status (or the progress) of the task.
Somewhere else, there’s another software that periodically consults the storage/database, takes the tasks and run them, updating the entries while they’re being processed.
In a variation of this, the API can activate this external software to process the new request, but, of course, must not wait until it’s done.
Nonetheless, I wouldn’t recommend the API starting an external program because, well, it could fail. Misconfiguration, permission stuff, just keep away from things that may become a problem later.
Semantics Matters
HTTP is perhaps the most feared yet widely popular protocol in which the internet relies on. When it comes to REST APIs, I believe it’s heavy important to take all the advantages that this protocol allows.
HTTP is such a robust and complete protocol that it’d require an article to talk only about it. Well, this one is not about HTTP, so, I will try to simplify things here.
This protocol provides built-in features for – at least – the most frequent situations you may face while programming web applications. Do you need internationalization? HTTP has a header named “Accept-Language” to deal with this. Does the resource require authorization? HTTP has a header named “Authorization” to deal with this. Do you need to deliver compressed data to mobile clients? Wait for that, the HTTP has a header named “Accept-Encoding” to deal with this!
I barely scratched the surface so far. All the examples I mentioned is about headers, but HTTP has a lot of more fun.
With the HTTP verbs, you can accomplish different actions for the same endpoint. Let’s say you want to retrieve the user’s profile picture. You can GET it. Ok, but now the user wants to remove it. Well, you can DELETE it. Oh, what a shame, the user wants to upload a new profile picture! You can POST it. Damn it! The user wants to replace the picture. You can PUT it. That’s it. Literally it. Replace the “it” before each verb for the same URL, and you may deduce, just for looking at it, what does the method executes.
This is the “Semantics” I meant in the title of this section.
Using the built-in HTTP features, it’s simple to create requests and responses that contain meaningful data. If you keep adding proprietary headers or following the rule of “always returns success, even if it’s an error”, the processes of parsing and understanding your requests and your responses are much more expensive. By returning a “custom error object” along with a success status code, one must know its properties to grab the real error. Instead, if you just return, for instance, a 404 response, everyone will quickly notice it’s a not found resource.
Furthermore, remember, API is for programmatically using. Consequently, the much standard stuff you put on your API, easier it will be for other people to consume it.
The status code is another excellent feature of the HTTP. You can reply much information, just by sending the right status code. You could reply 404 for not found (as I mentioned), 400 for a bad request, 204 for success with no content, etc.
I will explain a little bit about my favorite status codes and when I apply each of them.
201 – Created At
In my opinion, this is the correct status code to return when you respond that a long-running task was accepted. Why? Because this response allows you to deliver a response-header named “Location”, and you should fill it with the location of the created resource. I fill this header with the absolute URL so the client can check the progress of the task.
204 – Success with No Content
Well, as the name says, everything worked just fine, but the API has no content to reply to the requester. I mention that because it’s my default response when implementing DELETEs. Well, if you requested for the API to delete something, it’s because you don’t want it anymore, right?
400 – Bad Request
What the hell is a bad request?! It’s a common question. Basically, it’s an invalid request. There are some data that is incorrect or missing, or the client requests for an action that cannot be done, etc. Really, anything that the API won’t be able to perform.
402 – Payment Required
Yes, you read it right. There’s a status code for that. I don’t really know if I use it correctly, but I use to return this whenever the authenticated user of the request lacks payment, subscription or something like this.
404 – Not Found
This one is easy, right? Well, but I usually grant another meaning for this status code: not found to you. When a user attempts to select an entry by id, I tend to respond with 404 when the entry exists but does not belong to the requesting user. So, the requester may never truly know if the entry exists or not. Of course, it’s not the right way to deal with this situation. By the book, you ought to return a 403 (forbidden), however, I particularly don’t feel comfortable using this approach.
401 – Unauthorized
You should return this status code whenever the endpoint requires authentication, but the client hasn’t sent it. Besides, you must fill a response-header named “WWW-Authenticate” with the expected authentication scheme. If the returned scheme stands for Basic Authentication, then the most popular browsers will automatically pop up a login dialog for the user inputs its username and password.
A Side Note About Basic Authentication
The Basic Authentication concatenates the username and password and submits it to the server behind a base64 string. It means that, once the request encounters the server, the server can decode the base64 string, retrieving the original username and password as plain-text. It also means that anyone between your client/requester and your server/API can decode that too. So please, please, just use Basic Authentication with SSL. Well, whenever you need to handle sensitive user data (e.g. geolocation, logins, registration forms, payment forms), you ought to communicate with the server through SSL. Seriously, it’s not that expensive.
Provide Significative URLs
Another interesting feature someone might deliver along with the API is significative URLs. Significative for those ones who don’t know your application. For example, when building the API, you could group the operations under the entity they are related to; so, everything related to customers will be under the /Customers URL. It’s helpful when tracking bugs between the client and the API.
Consistent Behavior
Some people think that Consistent Behavior means always return equal answer objects independently whether the request resulted in success or failure. No. Consistent Behavior stands for predictable response situations. Again, APIs are built to be consumed programmatically, hence when someone implements the client-side for that, it’s very important that all the possible responses are clear and well-documented. If you got a 400, read it that way; when receiving a 200, read it this way; etc.
I guess this is the substantial difference between SOAP and REST. In the SOAP architecture, you know right away the response structure, just by looking to its WSDL. In the REST architecture, you don’t have this ability, but it allows a much more flexible way to deal with the data. Although even without having a contract like a WSDL, I believe it’s important to keep some level of foreseeability – those are required information; these might be ignored; etc. It could benefit the development of both of sides – client and server.
Every Request is a Different Request
Unlike many other kinds of web applications, REST APIs are designed to be session-less. It means that each request must contain every necessary information for identification of the client, or user (or both) in addition to the action’s information itself.
Sometimes data length or data security are concerns, so one alternative is to establish a token-based identification. You log in the user using an API service that provides back a unique token to identify the user in the next requests. So, once logged in, the client sends this token in all the subsequent API calls instead of, for instance, always send the username and password.
A Side Note About Unauthorized Password Storing
When you create an API that provides user login you shall be aware of, if the client wants, it can store the password that the user has typed because the user typed it inside its software and not directly in the API.
So, in case you don’t intend to trust in the third-party app that’s consuming your API, you shall not write a login service in your API. The alternative is to bring the user to your site, so the user types his or her credentials inside your web application, which you have full control, and then calls back to the third-party informing the token for the user. This is how Facebook and Twitter accomplish their user authentication even though they have APIs for developers to use.
Another alternative consists in simply do not have your own user authentication. Once you don’t store any password for your users, you don’t need to worry about securing them.
Performance
As I mentioned before, APIs should respond to requests as fast as they can. You should understand that it’s not just the time inside your API that counts. You must to sum the network delay and consider that the client application also run some code when receiving the response from the API.
So, there are a few techniques you may employ to improve your performance, but I will focus on two of them: caching and multi-thread managing.
Caching
If you shall not keep a caching service inside your API because it doesn’t suit the API workflow, also there’s no need in going to the database every time to grab a list of available countries or currencies since they almost never change. Caching this data is important because keeping your code away from the database is a huge performance improvement you may have.
So, I will point out three ways of caching those data even when you are implementing an API. You must pick the right one for each specific situation.
Caching Service
A caching service is basically a software that only manages the cache. There are a few options available (some free, some paid) for you to take, or you can implement one yourself. I widely recommend you pick some already created alternative because they probably have thought in some situations you might forget.
When using a caching service, instead of going to the database, you will consult this service to retrieve the data you need. So, as you may observe, it’s the best approach to the situations when it’s necessary that all the requests run through your code.
Web Server Caching
Another option is to configure your web server to cache the responses for determined endpoints. This can prevent that all the requests touch your code. So, it’s important to you to notice that caching the responses through the web server might not run your code.
Client-Side Caching
The last option I present to you is the client-side cache. Utilizing the response-headers that relate to caching, you may set up the response information that your client can understand and, thus, avoid calling the API again. In this approach, the client won’t even perform a real request to the web server, so it’s definitely not running your code because it’s never even getting to the server in the first place.
Multi-thread Management
Multithread is such a complex subject it’d require an entire article just to talk only about this. So, as I did to the HTTP protocol, I will simplify things here.
REST APIs are naturally multi-thread, once they can execute multiple requests at the same time. Therefore, every time you put a thread to wait for something synchronously you are wasting CPU time because that thread could be being used to handle another request.
Many developers utilize asynchronous methods without really understanding what it does under the hood. Basically, every action that doesn’t run on the CPU could be performed asynchronously. What does run code besides CPU on a computer? Drivers: from disk read/write operations to keyboard inputs.
When you send a web request through the internet it requires CPU (because TCP utilizes CPU), but the biggest part of the process is done by your network card driver. Therefore, this could be accomplished asynchronously. When running an asynchronous code, the operating system knows that that thread is waiting for something and then it could use this thread to run some code that needs to run on the CPU, avoiding the creation of a new thread, thus avoiding wasting memory and time.
This is how utilizing asynchronous methods may help to improve scaling your API.
Fortunately, many of the most common server-side programming languages help you writing asynchronous methods by utilizing the async and await keywords. This way, it’s much easier to write some code that executes asynchronously and still looks like a synchronous one.
Conclusion
Well, those are some important stuff that I consider every web API developer should know or keep in mind when building REST APIs.
I hope this article helped you some way to become a better developer.
Thank you for reading.
Top comments (7)
It seems like in the section about Long-Running tasks, you're describing a pattern that would and could be better served by message queuing. This is great because long running tasks can happen in the background without forcing your customer to wait since this is asynchronous. My team uses the pattern:
202 - Acceptedand a JSON body that contains a URL that a client can "poll" for status if necessary.Meanwhile, the message is sent by the broker to a waiting "worker" that will process the message and perform any sort of updates needed to the application/service backing store to let anyone interested know that the work has been done.
Yes, I can recommend this practice too.
Great post, but you have missed an important http status Code. The 418 I am a teapot has to be in every API 😉
Thanks for the nice article, Mateus! I will definitely look more into headers as a way to communicate information to the consumers of my APIs.
One thing that I strongly want to point out is that there is no such thing as reusing a thread. A thread is started with a certain task and cannot be reused. Rather other threads can run and utilize the CPU cycles that would otherwise be wasted. An efficient idea of this is a ThreadPool.
Perhaps I have expressed in a wrong way, but what I did have in mind while writing about the threads was actually the thread pool system. I apologize for the misunderstanding.
Really nice article discussing the subject which is one of the keywords in market today.
I too have worked on Rest and find it really a good tool, but being pragmatic is the way to go when designing any api.
The martinfowler.com/articles/richards... provides good guidelines regarding how to be pragmatic when designing any endpoint.
Amazing article!