In the late 1990's and the early 2000's, the predominant use of distributed APIs over the HTTP protocol involved the exchange of Extensible Markup Language (XML) formatted documents in a relatively simple remote procedure call (RPC) fashion.
One down-side of XML-based RPC APIs, if you live or work in an affected territory, is that their status with regards to applicable patents is unclear (see US7028312).
Protocols such as XML-RPC evolved into the Simple Object Access Protocol, or SOAP, giving birth to the approach widely known as Web Services. These were largely machine-to-machine interactions, and though they used the technologies of the world wide web, they were used only infrequently between web-servers and web-clients (browsers).
SOAP calls over HTTP involve either a GET or POST request to an HTTP endpoint, specifying the SOAPAction either as an HTTP header or a URL query parameter. Although SOAP can also support less common transports such as e-mail. XML namespaces are used to disambiguate the SOAP envelope elements from the message body elements defined by the web service, but this arguably does nothing to make SOAP messages more readable by humans.
Example of a SOAP request, taken from https://www.w3.org/2001/03/14-annotated-WSDL-examples:
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body>
<m:GetEndorsingBoarder xmlns:m="http://namespaces.snowboard-info.com">
<manufacturer>K2</manufacturer>
<model>Fatbob</model>
</m:GetEndorsingBoarder>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
Example of a SOAP response, from the same source:
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body>
<m:GetEndorsingBoarderResponse xmlns:m="http://namespaces.snowboard-info.com">
<endorsingBoarder>Chris Englesmann</endorsingBoarder>
</m:GetEndorsingBoarderResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
Because SOAP over HTTP is limited to GET and POST HTTP methods, it fails to get the benefit of the wider range of HTTP methods available (such as PUT, DELETE, HEAD and OPTIONS) and the full range of HTTP status codes associated with them. This can have a negative effect on cacheability and thus performance.
AJAX
In February 2005, Jesse James Garrett published an article entitled Ajax: A New Approach to Web Applications, which detailed a relatively lightweight approach developed at Google where a web client could load content from a server asynchronously (i.e. in the background). AJAX, standing for Asynchronous JavaScript And XML, soon became a popular approach for making web sites dynamic, giving birth to the "Web 2.0" concept.
Although originally concerned with loading HTML and XHTML document fragments, developers soon began to use AJAX to exchange data with web servers, and several factors (such as performance and cross-browser compatibility) led to a gradual shift from XML to JavaScript Object Notation (JSON) as the data format of choice. The acronym however, has stuck.
Once common, a web site which combined content or data from more than one "Web 2.0" source was known as a "Mash-up". Though this term has largely fallen out of use, remnants of this terminology can still be seen in organisation names such as Mashery (now part of Tibco) and Mashape (now known as Kong).
The simplicity of AJAX and the growing popularity of JavaScript libraries such as jQuery led to an explosion in the use of web APIs between the server and the browser. With the rise of mobile applications since 2007, the usage of web APIs has only grown further, and this approach, now decoupled from JavaScript, XML and the browser itself has effectively supplanted other web API technologies such as SOAP.
REST and RESTful APIs
The term REST - standing for Representational State Transfer - was coined by Roy T. Fielding, in his doctoral dissertation published in 2000; Architectural Styles and the Design of Network-based Software Architectures. It aimed to describe how networks like the World Wide Web, and software such as web applications should work. It rejects the RPC-style approach to APIs and advocates the use of features we are familiar with from the World Wide Web, such as:
- meaningful resource identifiers, such as URIs
- the passing of resource representations, as opposed to messages or function calls
- use of standard media-types and HTTP status codes
- explicit support for cacheing where appropriate
to transmit hypermedia documents (data - including text - containing links, just as hypertext refers simply to text containing links).
REST is described as an architectural style, and this is how its tenets and recommendations should be viewed. There is a lot to learn from REST, and it is very often a good touch-stone to return to when confronted with a complex design decision. REST is not "the new SOAP", they address radically different styles of APIs, and do this in a fundamentally different way. The REST style, when applied to HTTP also leverages this underlying protocol to its fullest, and thus can be very performant when well implemented, though this was not its original focus. Benefits from improvements in HTTP/2 will only further this performance capability. However, REST is not a standard with which you must comply, nor should it be seen as the one true path, to be taken as if gospel. Knowing when to break the 'rules' can be just as important as knowing when to stick to them, and it is important not to see strict REST adherence as a goal in and of itself, particularly if it conflicts with your, or your API consumers, business needs.
REST does have specific constraints which it applies to the design of the web to define its architectural style, much as architectural styles such as Palladian or neoclassical are used to divide and describe the range of building architecture. An API cannot truly be called a RESTful API if it does not exhibit all of the mandatory REST constraints. This has led to the coining of such terms as REST-like, and even RESTish, to describe APIs which display some of the aspects of the REST style, or have more in common with REST than RPC-like APIs. REST itself pre-dates the rise of web APIs, and there can be some difficulties in mapping the REST style for the web / web applications to APIs as we now know and describe them. For example, the US White House has elements of both Palladian and neoclassical architecture, but this does not make it any less of a grand building, nor any more likely to fall down any time soon.
"Like most real-world systems, not all components of the deployed Web architecture obey every constraint present in its architectural design."
Fielding's dissertation, section 6.2.5
Commonly web-APIs are referred to as RESTful APIs, meaning those that conform to most or all of the constraints imposed by the REST architectural style.
GraphQL
In January 2015 Facebook announced that its internal client APIs were being built on a new technology called GraphQL (though it had been in development in one shape or another since 2011). This is an API framework which owes a little to the RPC model, a little to REST and its use of web protocols, but also strikes out in its own direction.
GraphQL is based on a type schema, where the types of data and the relationships between them are modelled in an interface description language (IDL, also known as the Schema Description Language, or SDL) and this information is dynamically queryable by the client, using what is known as an introspection query.
Instead of having fixed functions like RPC, or fixed resource representations accessible via URLs like in REST, GraphQL is a query language similar in concept to Structured Query Language (SQL), which allows the client to request just the data it needs to perform a task, and sometimes to avoid multiple API calls to fetch all of the data required. GraphQL can also batch and combine queries. GraphQL divides queries into those that merely read data, and those that can alter it, known as 'mutations'.
Here's an example of a GraphQL query, which uses a syntax somewhat similar to JSON:
{ allPeople { people { name, homeworld { name }}}}
And an extract of the response, in JSON, from http://graphql.org/swapi-graphql
{
"data": {
"allPeople": {
"people": [
{
"name": "Luke Skywalker",
"homeworld": {
"name": "Tatooine"
}
},
{
"name": "C-3PO",
"homeworld": {
"name": "Tatooine"
}
},
{
"name": "R2-D2",
"homeworld": {
"name": "Naboo"
}
}
]
}
}
There are some potential performance downsides with this approach, because of the difficulty of cacheing GraphQL queries at the HTTP level and because clients may issue queries which have never been seen before or been adequately tested.
There is also an argument that GraphQL's type system encourages a tighter coupling between the underlying data model (such as an SQL database schema) and the data model presented to the client (though this is not always the case).
There is an interactive console provided with the Facebook GraphQL stack, called GraphiQL (pronounced 'graphical'). Other organisations provide implementations of GraphQL (such as the open source Apollo framework) and GraphQL is in use publicly by organisations such as GitHub, Shopify and Yelp.
gRPC
In recent years, the RPC style of API has made something of a resurgence in the form of Google's gRPC (the 'g' stands for gRPC, in the fashion of a recursive acronym, not Google).
gRPC is focussed on performance, as you can imagine from the amount of data Google's systems exchange, and mandates the use of HTTP/2. Messages are defined by the Protocol Buffers (protobuf) interface description language, which has both a textual / human-readable and binary serialisation. Code to generate and serialise data into protocol buffer format is usually generated from the protobuf IDL using the protoc compiler, making for a compact and performant representation, though one which cannot as easily be inspected as in the case of a textual transfer protocol such as HTTP/1.x.
A Google open-source project, gnostic aims to allow the use of the OpenAPI Specification to define APIs using the protobuf format.
Thanks to Marco Palladino for catching a thinko in this article.
Top comments (18)
Man, a topic I can talk about for hours (and spent years involved in)! Thanks Mike.
"...led to a gradual shift from XML to JavaScript Object Notation (JSON) as the data format of choice. The acronym however, has stuck."
I also wondered why the acronym never changed.
"Once common, a web site which combined content or data from more than one "Web 2.0" source was known as a "Mash-up". Though this term has largely fallen out of use, remnants of this terminology can still be seen in organisation names such as Mashable (now part of Tibco) and Mashape."
I remember Yahoo had a application you could mix and match different queries and data sources into a mashable news feed like dashboard. For the life of me I can not remember what it was called.
Side note: I'd like to point out that Roy T. Fielding doctoral dissertation was published a hole 5 years BEFORE AJAX. Conceptually making REST an older tech than AJAX. Just food for thought.
"...REST is not a standard with which you must comply, nor should it be seen as the one true path."
I am guilty of pushing to stay as close to the spec as possible. In recent years (and a number of projects) I've come to be a bit more flexible...when it makes sense.
Thank you again Mike, great article. I look forward to the next one.
Yahoo Pipes of course :)
Microsoft also had a product called Popfly that competed with Yahoo Pipes.
Thanks, will respond to your detailed points when I'm back at a computer.
Yes, Yahoo Pipes is a fond memory for many (though I never used it at the time). I think there might be open-source reimplementations of it - may be worth checking out alternativeto.net/software/yahoo-p...
I make reference to Fielding's dissertation being from 2000, but adoption of REST really picked up around 2007, some time after AJAX was coined.
When you say "spec", what do you mean? There is no REST specification, REST is just a set of constraints (one optional) by which you can recognise whether a web API leverages the HTTP protocol in the way Fielding intended.
You are confusing Mashable (the publication) with Mashery (which got acquired by Tibco).
You're right, thanks. Will correct and credit you.
Thank you! But you misspelled my last name :)
Aargh, sorry! Will fix.
Very interesting, thanks for the post!
Thanks, keep eyes peeled for the follow ups!
Nice overview Mike!
Thank you!
I'll leave you with a request for a post if you want to write it: your opinions about the future of Web APIs. Where REST future lies, how is GraphQL evolving, alternatives to protobuffers like capnproto and whatever you want to talk about :D
In this one you talked about the past, I would love to read one about present and future :)
Great, great idea. Thanks again.
Excellent overview.
Thank you.
Very very interesting. Needless to mention that GraphQL only works via HTTP POST, which is a paradox
GraphQL also supports GET over HTTP, but obviously that can run into problems with server query-string length limits.