Designing WEB API

API Definition

An Application Programming Interface
APIs are the building blocks that allow interoperability for major business platforms on the web. APIs are how identity is created and maintained across cloud software accounts.

Use Cases

Here are some examples of developer use cases for an image upload and storage API:

• Lisa is a web developer in a startup that sells art, and she needs an easy way for the artists to upload and show photos of their work.

• Ben is a backend enterprise developer who needs to store receipts coming from the expense system into his audit and policy solution.

• Jane is a frontend developer who wants to integrate real-time customer support chat onto her company’s website.

If you do not address your developers’ needs, your API will not be successful.

API Structuring Based on Some Companies Need

• APIs for internal developers first, external developers second. e.g Slack API

• APIs for external developers first, internal developers second. e.g GitHub API

• APIs as the product e.g Stripe, Twilio

What Makes an API Great

A good API may come down to the problem you’re trying to solve and how valuable solving it is. You may be willing to use a confusing, inconsistent, poorly documented API if it means you’re getting access to a unique dataset or complex functionality. Otherwise, good APIs tend to offer clarity (of purpose, design, and context), flexibility (ability to be adapted to different use cases), power (completeness of the solution offered), hack-ability (ability to pick up quickly through iteration and experimentation), and documentation.

API Paradigm

Picking the right API paradigm is important. An API paradigm defines the interface exposing backend data of a service to other applications. When starting out with APIs, organizations do not always consider all the factors that will make an API successful.

Over the years, multiple API paradigms have emerged. REST, RPC, GraphQL, WebHooks, and WebSockets are some of the most popular standards today.

Request-Response API

Request–Response apis typically expose an interface through an http-based web server. apis define a set of endpoints. clients make http requests for data to those endpoints and the server returns responses. the response is typically sent back as json or xml. there are three common paradigms used by services to expose request–response apis: rest, rpc, and graphql.

Representational State Transfer (REST) API

REST APIs expose data as resources and use standard HTTP methods to represent Create, Read, Update, and Delete (CRUD) transactions against these resources.

HTTP methods like GET, POST, UPDATE, and DELETE inform the server about the action to be performed. Different HTTP methods invoked on the same URL provide different functionality:

Create

Use POST for creating new resources.

Read

Use GET for reading resources. GET requests never, ever change the state of the resource. They have no side effects; the GET method has a read-only semantic. GET is idempotent. Consequently, you can cache the calls perfectly.

Update

Use PUT for replacing a resource and PATCH for partial updates for existing resources.

Delete

Use DELETE for deleting existing resources.

Remote Procedure Call (RPC) API

Remote Procedure Call (RPC) is one of the simplest API paradigms, in which a client executes a block of code on another server. Whereas REST is about resources, RPC is about actions. Clients typically pass a method name and arguments to a server and receive back JSON or XML. RPC APIs generally follow two simple rules:

• The endpoints contain the name of the operation to be executed.

• API calls are made with the HTTP verb that is most appropriate: GET for read-only requests and POST for others.

GraphQL API

Unlike REST and RPC APIs, GraphQL APIs need only a single URL endpoint. Similarly, you do not need different HTTP verbs to describe the operation. Instead, you indicate in the JSON body whether you’re performing a query or a mutation.

{  
"data": {  
"user": {  
"id": "MDQ6VXNlcjY1MDI5",
"name": "AYOUB ALI", 
"company": "Slack",
"createdAt": "2009-03-19T21:00:06Z"
}
}
}

Although GraphQL has many advantages, one of its drawbacks is the complexity it adds for the API provider. The server needs to do additional processing to parse complex queries and verify parameters. Optimizing performance of GraphQL queries can be difficult, too. Internally, within a company, it’s easy to predict the use cases and debug performance bottlenecks. When working with external developers, those use cases become difficult to understand and optimize for. When opening up GraphQL to third parties, you move the expense of managing multiple incoming requests to composing complicated queries on the backend—depending on the request, the performance and impact to infrastructure can be highly variable.

Web Hooks

A WebHook is just a URL that accepts an HTTP POST (or GET, PUT, or DELETE). An API provider implementing WebHooks will simply POST a message to the configured URL when something happens.

Web Sockets

WebSocket is a protocol used to establish a two-way streaming communication channel over a single Transport Control Protocol (TCP) connection. Although the protocol is generally used between a web client (e.g., a browser) and a server, it’s sometimes used for server-to-server communication, as well.

HTTP Streaming

HTTP Streaming, the server can continue to push new data in a single long-lived connection opened by a client.
To transmit data over a persistent connection from server to client, there are two options. The first option is for the server to set the Transfer-Encoding header to chunked. This indicates to clients that data will be arriving in chunks of newline-delimited strings. For typical application developers, this is easy to parse. Another option is to stream data via server-sent events (SSE). This option is great for clients consuming these events in a browser because they can use the standardized EventSource API.

API Security

To ensure an application is secure, there are many things engineers tend to do. This includes input validation, using the Secure Sockets Layer (SSL) protocol everywhere, validating content types, maintaining audit logs, and protecting against cross-site request forgery (CSRF) and cross-site scripting (XSS).

Authentication and Authorization

Authentication

The process of verifying who you are. Web applications usually accomplish this by asking you to log in with a username and password. This combination is checked against an existing valid username/password record to ensure the request is authentic.

Authorization

The process of verifying that you are permitted to do what you are trying to do. For instance, a web application might allow you to view a page; however, it might not allow you to edit that page unless you are an administrator. That’s authorization.

OAuth

OAuth is an open standard that allows users to grant access to applications without sharing passwords with them. The latest version of the standard, OAuth 2.0, is the industry-standard protocol for authorization.

Token Generation

With OAuth, applications use an access token to call APIs on behalf of a user. The generation of this token happens in a multi-step flow. Before an application can start the OAuth flow, it needs to be registered with the API provider. During registration, developers provide a redirect URL—an application URL to which the API provider can redirect the authorizing user. The API provider issues a client ID and client secret that are unique to the application. The client ID can be public, whereas the client secret should be kept confidential. After an application is registered, the application can generate an access token by following these steps:

1. The application directs the user to the API provider for authorization. Applications typically first show authorizing users a button labeled something like “Continue with Facebook.” When users click the button, they are redirected to the API provider’s authorization URL. While redirecting, the application sends the client ID, the permissions requested (i.e., access to the user’s public profile or list of friends) in a parameter called scope, an optional unique string state parameter, and (optionally) a redirect URL.
2. The API provider seeks the user’s authorization.
3. The application exchanges an authorization code for an access token.

Scopes

OAuth scopes are used to limit an application’s access to user data. For instance, an application might only need to identify a user. Rather than requesting access to all of the user’s data, the application can request access to only the user’s profile information by means of a granular OAuth scope. During authorization, the API provider will display all the requested scopes to the user. This way, users will know what permissions they are granting to an application. Defining scopes for an API is an interesting problem. Many APIs offer simple read, write, and combo read/write scopes. For instance, the Twitter API offers three levels of permissions via scopes:

• Read only
• Read and Write
• Read, write, and access direct messages

Token Expiry and Refresh Tokens

Refresh token is a special type of token used to obtain a new access token when the current access token expires. Applications need to provide the client ID, client secret, and refresh token to generate a new access token. Refresh tokens are a standard way of renewing expired access tokens.

Design Practices

• Documentation
• Error Handling
• Extensible

Before you begin coding or writing your API specification, take a moment to ask yourself two questions: what problem are you trying to solve, and what is the impact you want to have by building this API? The answers to both of these questions must focus on the needs of the user as well as the business you have created.

• Technology Architecture
• API Specifications

Design

1. Problem
2. Solution
3. Implementation
4. Authentication
5. Error Handling
6. Validation
7. Testing

Scaling API

Making sure your API scales both in terms of use cases and load is critical for its success.

• Scaling throughput
• Evolving API design
• Paginating APIs
• Rate-limiting APIs
• Developer SDKs

To scale your API by supporting an increased number of API calls, there are many things you can do at the application level. Database query optimization, sharding databases, adding missing indexes, utilizing caching, doing expensive operations asynchronously, writing efficient code, and tuning web servers help in increasing the throughput and decreasing the latency.

Scaling Throughput

As the number of users of an API grows, the throughput—measured as the number of API calls per second—increases. In this section, we talk about various ways in which you can optimize your API to support this growth.

Finding the Bottlenecks

One of the best ways to gain insights into bottlenecks is through instrumentation. By collecting data on usage and monitoring for capacity bottlenecks, you can take advantage of data-driven insights into optimizations that will help you scale.

• Disk I/O
  Expensive database queries and local disk access often lead to disk-related bottlenecks.

• Network I/O
  Network bottlenecks in modern applications are frequently caused by dependencies on external services requiring API calls across data centers.

• CPU
  Inefficient code performing expensive computations is one of the common causes of CPU bottlenecks.

• Memory
  Memory bottlenecks typically occur when systems do not have sufficient RAM.

Solution for Bottlenecks

Adding Computing Resources

Simply adding more computing resources can help in scaling an application.
There are two ways to add more resources: -

• Vertical scaling Vertical scaling can be achieved by adding more power, like CPUs, RAM, and disk storage, to existing servers.
• Horizontal scaling Horizontal scaling is achieved by adding more server instances to your pool of resources so that the load can be distributed among them.

Database Indexes

Indexing is a way to optimize the performance of data retrieval operations on a database table. Indexes help databases to locate rows to return for a query without having to compare every row in a database table. They do this by additionally storing the index data structure.
However, adding too many indexes is not ideal, either. Each index on a table requires additional storage space, and there is a performance hit every time you add, update, or delete rows because the corresponding indexes need to be updated as well

Caching

Caching is one of the most popular and simplest techniques that web applications use to scale to very large throughput. Caching solutions like Memcached store data in memory instead of on the disk because it is much faster to read from memory. Caching is often used to store the responses to database queries. By analyzing your database logs, you can figure out the database queries that take a long time to execute and are most frequent. With caching, when you need to look up data, you first check whether it’s available in the cache. If you find the data, you return it. Otherwise, you can do a database query to find results and store them in the cache for future lookups before returning the response to the user. By caching these results in memory, you can significantly improve the scalability and performance of your application. When you implement caching, one important thing to remember is to invalidate the cache. Often you want to delete the cache whenever the corresponding data is updated. At other times, when you can tolerate data update delays, you might let the cache expire on its own. Although application-level caching for APIs is typically implemented along with your web servers, caching API results closer to end users can help to achieve even higher throughput and performance. This is referred to as edge caching.

Asynchronous Operations

If some of your API requests take a long time to execute, you might want to consider performing expensive operations outside of the request (asynchronously). This way you can serve the responses to such requests faster.

Evolving API's

Paginating API's

Paginating APIs can help with scaling. Quite often, APIs need to handle large datasets. An API call might end up returning thousands of items. Returning too many items can overload the web application backend and even slow down clients that can’t handle large datasets. For that reason, it’s important to paginate large result sets. This splits long lists of data into smaller chunks, minimizes response times for requests, and makes responses easier to handle.

Offset Based Pagination

Using limits and offsets is generally the easiest way to implement pagination. It’s also the most widely used pagination technique. To paginate this way, clients provide a page size that defines the maximum number of items to return and a page number that indicates the starting position in the list of items. Based on these values, servers storing data in a SQL database can easily construct a query to fetch results. For instance, to fetch the fifth page of items with each page’s size being 10, we should load 10 items, starting after 40 items (skipping the first 4 pages of size 10)

Cursor-Based Pagination

To address the problems of offset-based pagination, various APIs have adopted a technique called cursor-based pagination. To use this technique, clients first send a request while passing only the desired number of items. The server then responds with the requested number of items (or the maximum number of items supported and available), in addition to a next cursor. In the subsequent request, along with the number of items, clients pass this cursor indicating the starting position for the next set of items. Implementing cursor-based pagination is not very different from offset-based pagination. However, it’s much more efficient. Systems that store data in a SQL database can create queries based on the cursor values and retrieve results. Suppose that a server returns a Unix timestamp of the last record as the cursor. To fetch a page of results that are older than that given cursor, the server can construct a SQL query like the following:

SELECT * FROM items
WHERE created_at < 1507876861
ORDER BY created_at
LIMIT 10;

Having an index on the column created_at in the preceding example makes the query fast.

Rate-Limiting APIs

A rate-limiting system controls the rate of traffic sent or received on a network interface. For web APIs, rate-limiting systems are used to control how many times an application or a client is allowed to call an API during a given time interval. Traffic is allowed up to the specified rate, whereas traffic that exceeds that rate might be denied.
When an API becomes popular and suddenly sees a surge of traffic that potentially affects the availability of the application, API developers begin exploring rate-limiting options. There are two key reasons why APIs should do rate-limiting:

• To protect the infrastructure while increasing reliability and availability of the application You do not want a single misbehaving developer or user to bring your application down through a denial-of-service (DoS) attack.

• To protect your product You want to prevent abuses of your product, like mass registration of users or creation of a lot of spam content.

Implementation Strategies

There are a few common algorithms that are used to implement rate limits:

• Token bucket
• Fixed-window counter
• Sliding-window counter

SDK's

SDKs are thin abstraction layers over your API. They provide developers with the ability to work with a code library rather than making raw API calls. Developers download or refer to an SDK library and build their business logic by calling the functionality of the SDK.
Many companies wrap their APIs with SDKs, and many developers prefer to use an SDK rather than calling the API methods directly. This is due to the inherent complexity that comes with handling and making web requests. The interfaces of the SDK need to be readable and well documented, but the internals of the SDK do not need to meet these requirements. The internals of the SDK library can be optimized or even obfuscated and minified. It is important that you provide SDKs in the programming language(s) your developers use. SDKs are not like sample code; they are less portable and basically useless if they are not in the programming language of your audience. Remember that, after you launch an SDK, you need to continue to update it. The key here is to update your SDK at the same time you make your API updates. Assuming that your developers are relying heavily on your SDK, if you do not update it, they will not have access to the new platform or API features that you just launched.

Frameworks

Frameworks are additional layers of abstraction that you sometimes need to add over an API. They make it easier to use API methods by providing functionality that is closer to the use case the developer needs to implement and can further hide the complexity of the API.

API Implementation Checklist

❏ Define specific developer problem to solve
❏ Write internal API specification
❏ Get internal feedback on API specification
❏ Build API
❏ Authentication
❏ Authorization
❏ Error handling
❏ Rate-limiting
❏ Pagination
❏ Monitoring and logging
❏ Write documentation
❏ Run beta test with partners on new API
❏ Gather feedback from beta partners and make changes
❏ Create communication plan to notify developers of changes
❏ Release API changes

Credits

BOOK - Designing Web API's by Brenda Jin