DEV Community: Mike Ralphson

OpenAPI 3.1 - The Gnarly Bits

Mike Ralphson — Mon, 03 Apr 2023 13:08:00 +0000

Though obviously support in tooling has taken a little while to begin to appear, both for commercial and Open-Source offerings, there are already a number of resources available to help you get to grips with the latest version of the OpenAPI specification (OAS), whether you are entirely new to it, or an older hand looking to focus on the new features.

Lorna Mitchell, who championed the new webhooks feature, has a range of information available: a blog post, a video and associated slides.

Phil Sturgeon, who along with Ben Hutton and Henry Andrews from the JSON Schema community, helped drive the push to full JSON Schema Draft 2020-12 compliance, has written a blog post for the official OpenAPIs.org website on how to transition your OAS documents from v3.0.x to v3.1.0.

My fellow OpenAPI Initiative TSC members Ron Ratovsky and Darrel Miller presented a webinar on what's new in v3.1.

My fellow Postman colleague Arnaud Lauret (the API Handyman) gave a talk at the API Specifications Conference (ASC) in 2022 entitled OpenAPI 3.x Does What Swagger 2.0 Don’t.

Last but not least, the OpenAPI Initiative now has an official getting started guide.

So, with all that going on, is there anything much else to add?

Indeed there is, so let's take a top-down stroll through the OpenAPI specification, to focus on the details of what has changed.

Top-Level changes

As part of this release, we have decided to not follow SemVer anymore, and as such allow ourselves to introduce minor, but breaking changes. These changes are documented as part of the release notes.

Additions

Introduced a new top-level field - webhooks. This allows describing out-of-band registered webhooks that are available as part of the API.
The Info Object has a new summary field.
The License Object now has a new identifier field for SPDX license codes.
The Components Object now has a new entry pathItems, to allow for reusable Path Item Objects to be defined within a OpenAPI document.

Extended Functionality

Updated primitive types to be based on JSON Schema Specification Draft 2020-12. This now includes type "null".
Lifted the restriction of allowing Request Body only in HTTP methods where the HTTP 1.1 specification RFC9110 has explicitly defined semantics. While now allowed for other methods, this use is still not recommended.
Added support for the object type for spaceDelimited and pipeDelimited style values.
The Encoding Object now supports style, explode and allowReserved for the multipart/form-data media type as well.
To enable better webhooks support, expressions in the Callback Object can now also reference Path Item Objects.
When using the Reference Object, summary and description fields can now be overridden. This is the default behaviour, tooling may optionally allow merging or appending the text.
The Schema Object is now fully compliant with JSON Schema draft 2020-12 (see JSON Schema Core and JSON Schema Validation). See also, Breaking Changes.
The $ref keyword within Schema Objects can now contain relative JSON Pointers.
The Discriminator Object can now be extended with Specification Extensions, correcting an oversight in version 3.0.
Added support for mutual TLS (mutualTLS) as a security scheme type.
Security requirements (such as for API keys) can now define an array of roles that are required for execution (and not only scopes for OAuth 2.0 security schemes).
Added the jsonSchemaDialect top-level field to allow the definition of a default $schema value for Schema Objects. This allows any past or future version of JSON Schema to be used in your OAS documents, provided that tools support them.

Changes

An OpenAPI Document now requires at least one of paths, components or webhooks to exist at the top level. While previous versions required paths, now a valid OpenAPI Document can describe only webhooks, or even only reusable components. Thus, an OpenAPI Document no longer necessarily describes an API.
Anywhere in the 3.0 Specification that had a type of Schema Object | Reference Object has been replaced to be Schema Object only. With the move to full JSON Schema support, $ref is inherently part of the Schema Object and has its own defined behavior.
Extensions prefixed with x-oas- and x-oai- are now reserved for the OpenAPI Initiative.
The format keyword is now not validated by default. It is treated as an annotation. Tooling may allow opt-in validation on a best-case basis.
The allowEmpty property on parameters is now deprecated as it had confusing and less-than-useful behaviour.

Breaking changes

The specification versioning no longer follows SemVer.
The nullable keyword has been removed from the Schema Object ("null" can be used as a type value).
exclusiveMaximum and exclusiveMinimum do not accept boolean values (following JSON Schema). They are independent keywords which take a number.
Due to the compliance with JSON Schema, there is no longer interaction between required and readOnly/writeOnly in relation to requests and responses.
format (whether byte, binary, or base64) is no longer used to describe file payloads. As part of JSON Schema compliance, now contentEncoding and contentMediaType can be used to control this.
The Server Object Variable's enum array now MUST not be empty (changed from SHOULD).
The Server Object Variable's default property now MUST exist in the enum values, if such values are defined (changed from SHOULD).
responses are no longer required to be defined under the Operation Object.

Clarifications

Reworded the definition of OpenAPI Document to reflect that a document no longer must describe paths, but can describe either paths, webhooks, components or any combination of these.
Dropped the term "RESTful APIs" in favor of "HTTP APIs"
Resolution of relative references has been redefined and clarified. Note there's a difference in resolution between Schema Object References and all others.
Modification of examples to improve them and provide context for new fields and objects.
It is now clarified what happens when path template expressions do not have a corresponding path parameter.
Data types (and just primitive data types) now correspond to JSON Schema.
A new section was added to address how to handle the $schema keyword (implicitly and explicitly).
Updated some inline links to more accurate or secure locations.
Path parameter values cannot contain the unescaped characters /, ? or #.
Further explanation of where Reference Object and JSON Schema's reference should be used.
Unified wording when values are URLs/URIs.
Reworded Path Item's $ref to take into account reference and component changes.
Minor text changes to improve consistency and readability.
The description of the Reference Object has been updated to further clarify its behavior.
Further updated Schema Object's description to take into account the latest draft, and the default use of https://spec.openapis.org/oas/3.1/dialect/base as the default OAS dialect.
Reworded "Schema Vocabularies" to "Schema dialects"

Conclusion

I hope this guide proves helpful to those considering using OAS 3.1, and those migrating from earlier versions. As ever, let me know in the comments if anything is wrong, unclear or missing.

Why not get involved in the discussions around a tentative OpenAPI 4.0, codename 'Moonwalk'?

Raspberry Pi radio alarm clock - part 1

Mike Ralphson — Thu, 20 May 2021 08:19:19 +0000

Introduction

I've had this Sanyo RM5005 radio alarm clock since my 13th birthday back in 1985. It replaced a second-hand Morphy Richards model with a flip display. If you're not familiar with flip displays, they look something like this:

Anyway, it looks like the Sanyo was available in brown (or wood-effect?), black, or like mine, a white, which has now aged to a vaguely yellowish hue. A working model will set you back about £20 from the usual places, unless you happen to chance upon one in a charity shop.

I don't really want to part with this reminder of my youth (especially as it still works), but maybe doing a hack project on it would justify keeping it around?

So, the plan is to replace the innards with a Raspberry Pi. I thought the Zero WH would be a suitable board to use. They're small, cheap and more than capable of running an internet radio client and network time protocol client.

Initial teardown

Here's where it gets fun. I took many things apart as a child, as my Dad trained as an electrician and I was interested too. Unfortunately, most of things I dissassembled never worked again afterwards. Hopefully this won't be the case here.

Like I say, this still works perfectly, but isn't it just a matter of time before one of those capacitors goes bang, and the magic smoke that makes the radio work escapes? We'll be ditching all of the power supply and circuit board, except for the speaker and LED display module, if we can salvage it.

But would the Pi Zero fit?

Looks like that's a yes. The amount of space prompted me to start thinking about adding an SSD for increased storage. Why not add the backup from my old iPod and turn this into a media server too?

The tuning dial

A hi-tech solution of a plastic wheel, a small spring and a loop of cotton kept this thing in tune for over 35 years!

The 7-segment LED display

We will need to desolder the LED display or source a compatible module. It's a Sanyo SL-1042-27T. It's a very thin module, so I suspect the driver gubbins is on a separate chip. I know very little about driving LED modules, so at this stage I don't know if I can just bit-bang it from a Node.js script or what.

In fact it turns out the whole clock radio experience, including display driving, is provided by this 3M3 LM8560 chip:

It looks like these will today set you back a whopping 60p each! Here's the datasheet in case we later need pin-outs etc.

The speaker

Another outstanding question is: can we drive the speaker directly from a pair of GPIO pins, or do we need to source an audio module (or 'hat') for the Pi?

Next time

Booting and configuring the Pi.

GraphQL is King

Mike Ralphson — Thu, 04 Mar 2021 10:51:51 +0000

To the tune of "Rock 'n' Roll is King" by The Electric Light Orchestra

Listen everybody let me tell you 'bout GraphQL
It's the API architecture that's really gonna thrill your soul
She said come along with me, to a land of graph queries

She said (wam-a-lam-a bam-a-lam-a) GraphQL is king

She loves that GraphQL and she daydreams SDL
That's all she ever tells me when I ping her on the Slack channel
Says named mutations are the key, but it isn't like RPC

She says (wam-a-lam-a bam-a-lam-a) GraphQL is king

Oh let Apollo run
Run queries, run queries
Oh let's install Relay
That's how it's meant to be...

It's coupled to the DB and it gets HATEOAS flack
She threw a runtime error and she gave "200" back
She loves that paradigm, she does JSON not resource-based MIME

She said (wam-a-lam-a bam-a-lam-a) GraphQL is king

Oh let Apollo run
Run queries, run queries
Oh let's install Prisma
That's how it's meant to be

When she comes 'round and I'm reading 'bout REST / RESTful,
She says you under/over-fetch and all I resolve to do is GraphQL
I think I'm gonna choose - to use REST and HTTP2

She says (wam-a-lam-a bam-a-lam-a) GraphQL is king

(wam-a-lam-a bam-a-lam-a) GraphQL is king
(wam-a-lam-a bam-a-lam-a) GraphQL is king
(wam-a-lam-a bam-a-lam-a) GraphQL is king

She says... (wam-a-lam-a bam-a-lam-a) GraphQL is king

With apologies to Jeff Lynne and the GraphQL community

How to get your web API noticed

Mike Ralphson — Mon, 06 Jul 2020 11:19:46 +0000

Web API discovery is a somewhat sprawling and amorphous topic. There is no one right way to achieve it, and no simple silver bullet.

This post offers a number of strategies, both aimed at your human-readable documentation and your machine-readable API definitions, to make them stand out and attract potential users.

Document your API

If nothing else, provide reference documentation and examples so people know how your API works. Aim for a low "time-to-first-200-OK"; can users access the root node of your hypermedia API, or a status page, with a simple curl request?

Slate provides a quick and easy static site generator from simple markdown to a classic three-panel documentation display.

Provide an API Portal

The next stage of API publicity is the API or developer portal. This is a home base or one-stop-shop for your API users. Some of this may only be necessary if your API is a commercial offering. It should include onboarding documentation, key / authentication management, API reference, terms of service, SLAs, pricing, support / helpdesk information, a downloadable API definition, sunset / deprecation information and anything else required during the lifecycle of using your APIs. Making everything available in one place keeps your API consumers happy and keeps them coming back for updated information.

Redoc.ly has a developer portal in-a-box offering in beta which is well worth a look.

Follow industry standards

Use of industry standards and norms will speed the onboarding process for experienced API consumers. From simply using JSON requests and responses, to following IETF RFCs for problem reporting and healthcheck status, standards for the HTTP PATCH method and sunset / deprecation information, standing on the shoulders of giants saves you API design time and potential rework in the future.

standards.rest is a great starting point to ensure that you're not reinventing the wheel.

Publish an OpenAPI document

The OpenAPI Specification [OAS] is the industry-leading API description standard. By making your API documentation machine-readable, you enable a complete eco-system of tooling which supports your API through interactive documentation and testing / mocking to client and server code generation and more.

OAS version 3.0 recommends your API definition be called openapi.yaml or openapi.json though currently makes no further suggestions on discovery.

Your API definition should be prominently linked to from your API portal, and included in your repository if your project is Open-Sourced. Don't forget to validate your API definition to make sure it is OAS compliant and lint it to make sure it is complete and consistent.

Submit to APIs.guru OpenAPI Directory

If your API meets the inclusion criteria:

Public - anyone can access it as long as they follow some clearly defined steps (subscribe, pay fees, etc.).
Persistent - API is made with long-lived goal, and not for a particular event (conference, hackathon, etc.).
Useful - API should provide useful functionality not only for its owner.

then the APIs.guru OpenAPI-Directory is an excellent showcase for your API definition. It is the largest independent directory of machine-readable API definitions. Submit your API here.

The directory currently includes over 3,100 API definitions ranging from world-leading cloud vendors to self-hosted informational APIs. Inclusion in the directory automatically makes your API available to several integrated platforms and products.

Other directories / marketplaces you might consider submitting your API to include RapidAPI, the Postman Collection network and ProgrammableWeb.

`service-desc` Link relation type

One way to point to your OAS document is to use an HTTP Link header with a relation-type (rel value) of service-desc. More information can be found in RFC-8631

Schema.org WebAPI type

Another way to make your API and its definition discoverable is to use the Schema.org WebAPI type.

Using a format such as JSON Linked-Data (JSON-LD) you can embed machine-readable metadata in your web pages describing your API and linking to your documentation.

    <script type="application/ld+json">
    {
      "@context": "http://schema.org/",
      "@type": "WebAPI",
      "name": "Google Knowledge Graph Search API",
      "description": "The Knowledge Graph Search API lets you find entities in the Google Knowledge Graph. The API uses standard schema.org types and is compliant with the JSON-LD specification.",
      "documentation": "https://developers.google.com/knowledge-graph/",
      "termsOfService": "https://developers.google.com/knowledge-graph/terms",
      "provider": {
        "@type": "Organization",
        "name": "Google Inc."
      }
    }
    </script>

APIs.json

A little obscure and pending some work to update it, apis.json is "a machine readable approach that API providers can use to describe their API operations, similar to how web sites are described using sitemap.xml".

This format can come into its own when you have a number of APIs to describe, and need to link to various documentation and support pages for each.

Others?

Please let me know if you think I've missed anything important in this round-up, and I'll be sure to update and credit you.

Thanks

To Aijaz Ansari for inspiring this post with a question on Twitter.

Slate and the Future

Mike Ralphson — Tue, 23 Jun 2020 10:36:10 +0000

Two announcements today, first thanks to the hard work of my co-maintainers Matthew Peveler and Robert Lord, and many members of the Slate community, we are happy to present Slate v2.7.0.

Secondly, we have something we'd like lots of feedback on.

The Slate v3 technology preview (in the v3-tp0 branch) is a ground-up reworking of Slate using pure Node.js and the eleventy static-site-generator.

The Slate v3 technology preview uses the exact same CSS and client-side JavaScript (apart from a couple of compatibility tweaks) as Ruby Slate, so the output HTML should function exactly as in Ruby Slate.

Ports of Slate including to Node.js are nothing new and indeed my own Shins project has been tracking releases of Ruby Slate since late 2016, however, this technology preview is not based on Shins or any of the other outdated ports, and leverages eleventy to minimise the amount of custom code and dependencies required to build your documentation with Slate.

Project	Top-Level Deps	Bundled JS Deps	Docker Image Size
Slate 2.7	8	5	482MB
Slate 3-tp0	9	2	176MB
Shins	18	5	177MB

The fact that three of our core client-side JS libraries can be included in the Node.js package.json dependencies makes the process of keeping them up to date much simpler. It also paves the way to potentially move the lunr search index generation to the server side, which would allow multi-page searches.

The Future

It is important to be clear what the function of the technology preview is. It is primarily focused on getting feedback from the community.

We are not saying that Slate 2.7.0 will be the last release of Ruby Slate
We are not saying that the next major version of Slate will be Node.js-based
We are not saying that the next major version of Slate will be eleventy-based

We are saying that we recognise that the Ruby and middleman infrastructure used by Slate has historically caused the community, and maintainers, a good deal of headaches over the years.

We particularly want to hear your experiences of using the technology preview in these areas:

Compatibility with Ruby Slate
Any bugs found in building or the resultant HTML
Operating system compatibility, particularly on Windows
Should we symlink JS dependencies as now, or simply copy them?
Should we mirror the Ruby Slate source directory structure or have a clean break?
Ease of use
Speed

In addition to the concrete areas above, we would welcome feedback on the adoption of eleventy itself; we note that the project is relatively young (circa two and a half years) and has not yet reached the milestone of a stable v1.0.0 release. Also, eleventy has a fair number of open issues and this is with it using the lodash style of issue management where enhancement requests and documentation change issues are closed, but still monitored. eleventy while not being one of the most well-known static site generators, is actively maintained and it fits the requirements of supporting markdown and ejs templates while not including large unnecessary dependencies such as React or Vue.js, or being primarily focused on being a blog framework. We believe the Slate community could be beneficial to the eleventy community in terms of visibility and additional contributors. eleventy also has a well thought-out plugin system.

We know that the provided documentation is very bare-bones at the moment, but we would plan to copy over the remainder of the README and start the process of updating the Wiki if the technology preview warrants moving to the next phase.

One more thing to make clear, we are accepting Pull-Requests against the technology-preview branch, but these should be bug fixes and documentation improvements rather than new features at this stage.

Many thanks for reading, and we look forward to your feedback.

A brief history of Web APIs

Mike Ralphson — Sun, 30 Dec 2018 09:14:03 +0000

In the late 1990's and the early 2000's, the predominant use of distributed APIs over the HTTP protocol involved the exchange of Extensible Markup Language (XML) formatted documents in a relatively simple remote procedure call (RPC) fashion.

One down-side of XML-based RPC APIs, if you live or work in an affected territory, is that their status with regards to applicable patents is unclear (see US7028312).

Protocols such as XML-RPC evolved into the Simple Object Access Protocol, or SOAP, giving birth to the approach widely known as Web Services. These were largely machine-to-machine interactions, and though they used the technologies of the world wide web, they were used only infrequently between web-servers and web-clients (browsers).

SOAP calls over HTTP involve either a GET or POST request to an HTTP endpoint, specifying the SOAPAction either as an HTTP header or a URL query parameter. Although SOAP can also support less common transports such as e-mail. XML namespaces are used to disambiguate the SOAP envelope elements from the message body elements defined by the web service, but this arguably does nothing to make SOAP messages more readable by humans.

Example of a SOAP request, taken from https://www.w3.org/2001/03/14-annotated-WSDL-examples:

<SOAP-ENV:Envelope
  xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <SOAP-ENV:Body>
    <m:GetEndorsingBoarder xmlns:m="http://namespaces.snowboard-info.com">
      <manufacturer>K2</manufacturer>
      <model>Fatbob</model>
    </m:GetEndorsingBoarder>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Example of a SOAP response, from the same source:

<SOAP-ENV:Envelope
  xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <SOAP-ENV:Body>
    <m:GetEndorsingBoarderResponse xmlns:m="http://namespaces.snowboard-info.com">
      <endorsingBoarder>Chris Englesmann</endorsingBoarder>
    </m:GetEndorsingBoarderResponse>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Because SOAP over HTTP is limited to GET and POST HTTP methods, it fails to get the benefit of the wider range of HTTP methods available (such as PUT, DELETE, HEAD and OPTIONS) and the full range of HTTP status codes associated with them. This can have a negative effect on cacheability and thus performance.

AJAX

In February 2005, Jesse James Garrett published an article entitled Ajax: A New Approach to Web Applications, which detailed a relatively lightweight approach developed at Google where a web client could load content from a server asynchronously (i.e. in the background). AJAX, standing for Asynchronous JavaScript And XML, soon became a popular approach for making web sites dynamic, giving birth to the "Web 2.0" concept.

Although originally concerned with loading HTML and XHTML document fragments, developers soon began to use AJAX to exchange data with web servers, and several factors (such as performance and cross-browser compatibility) led to a gradual shift from XML to JavaScript Object Notation (JSON) as the data format of choice. The acronym however, has stuck.

Once common, a web site which combined content or data from more than one "Web 2.0" source was known as a "Mash-up". Though this term has largely fallen out of use, remnants of this terminology can still be seen in organisation names such as Mashery (now part of Tibco) and Mashape (now known as Kong).

The simplicity of AJAX and the growing popularity of JavaScript libraries such as jQuery led to an explosion in the use of web APIs between the server and the browser. With the rise of mobile applications since 2007, the usage of web APIs has only grown further, and this approach, now decoupled from JavaScript, XML and the browser itself has effectively supplanted other web API technologies such as SOAP.

REST and RESTful APIs

The term REST - standing for Representational State Transfer - was coined by Roy T. Fielding, in his doctoral dissertation published in 2000; Architectural Styles and the Design of Network-based Software Architectures. It aimed to describe how networks like the World Wide Web, and software such as web applications should work. It rejects the RPC-style approach to APIs and advocates the use of features we are familiar with from the World Wide Web, such as:

meaningful resource identifiers, such as URIs
the passing of resource representations, as opposed to messages or function calls
use of standard media-types and HTTP status codes
explicit support for cacheing where appropriate

to transmit hypermedia documents (data - including text - containing links, just as hypertext refers simply to text containing links).

REST is described as an architectural style, and this is how its tenets and recommendations should be viewed. There is a lot to learn from REST, and it is very often a good touch-stone to return to when confronted with a complex design decision. REST is not "the new SOAP", they address radically different styles of APIs, and do this in a fundamentally different way. The REST style, when applied to HTTP also leverages this underlying protocol to its fullest, and thus can be very performant when well implemented, though this was not its original focus. Benefits from improvements in HTTP/2 will only further this performance capability. However, REST is not a standard with which you must comply, nor should it be seen as the one true path, to be taken as if gospel. Knowing when to break the 'rules' can be just as important as knowing when to stick to them, and it is important not to see strict REST adherence as a goal in and of itself, particularly if it conflicts with your, or your API consumers, business needs.

REST does have specific constraints which it applies to the design of the web to define its architectural style, much as architectural styles such as Palladian or neoclassical are used to divide and describe the range of building architecture. An API cannot truly be called a RESTful API if it does not exhibit all of the mandatory REST constraints. This has led to the coining of such terms as REST-like, and even RESTish, to describe APIs which display some of the aspects of the REST style, or have more in common with REST than RPC-like APIs. REST itself pre-dates the rise of web APIs, and there can be some difficulties in mapping the REST style for the web / web applications to APIs as we now know and describe them. For example, the US White House has elements of both Palladian and neoclassical architecture, but this does not make it any less of a grand building, nor any more likely to fall down any time soon.

"Like most real-world systems, not all components of the deployed Web architecture obey every constraint present in its architectural design."

Fielding's dissertation, section 6.2.5

Commonly web-APIs are referred to as RESTful APIs, meaning those that conform to most or all of the constraints imposed by the REST architectural style.

GraphQL

In January 2015 Facebook announced that its internal client APIs were being built on a new technology called GraphQL (though it had been in development in one shape or another since 2011). This is an API framework which owes a little to the RPC model, a little to REST and its use of web protocols, but also strikes out in its own direction.

GraphQL is based on a type schema, where the types of data and the relationships between them are modelled in an interface description language (IDL, also known as the Schema Description Language, or SDL) and this information is dynamically queryable by the client, using what is known as an introspection query.

Instead of having fixed functions like RPC, or fixed resource representations accessible via URLs like in REST, GraphQL is a query language similar in concept to Structured Query Language (SQL), which allows the client to request just the data it needs to perform a task, and sometimes to avoid multiple API calls to fetch all of the data required. GraphQL can also batch and combine queries. GraphQL divides queries into those that merely read data, and those that can alter it, known as 'mutations'.

Here's an example of a GraphQL query, which uses a syntax somewhat similar to JSON:

{ allPeople { people { name, homeworld { name }}}}

And an extract of the response, in JSON, from http://graphql.org/swapi-graphql

{
  "data": {
    "allPeople": {
      "people": [
        {
          "name": "Luke Skywalker",
          "homeworld": {
            "name": "Tatooine"
          }
        },
        {
          "name": "C-3PO",
          "homeworld": {
            "name": "Tatooine"
          }
        },
        {
          "name": "R2-D2",
          "homeworld": {
            "name": "Naboo"
          }
        }
      ]
    }
}

There are some potential performance downsides with this approach, because of the difficulty of cacheing GraphQL queries at the HTTP level and because clients may issue queries which have never been seen before or been adequately tested.

There is also an argument that GraphQL's type system encourages a tighter coupling between the underlying data model (such as an SQL database schema) and the data model presented to the client (though this is not always the case).

There is an interactive console provided with the Facebook GraphQL stack, called GraphiQL (pronounced 'graphical'). Other organisations provide implementations of GraphQL (such as the open source Apollo framework) and GraphQL is in use publicly by organisations such as GitHub, Shopify and Yelp.

gRPC

In recent years, the RPC style of API has made something of a resurgence in the form of Google's gRPC (the 'g' stands for gRPC, in the fashion of a recursive acronym, not Google).

gRPC is focussed on performance, as you can imagine from the amount of data Google's systems exchange, and mandates the use of HTTP/2. Messages are defined by the Protocol Buffers (protobuf) interface description language, which has both a textual / human-readable and binary serialisation. Code to generate and serialise data into protocol buffer format is usually generated from the protobuf IDL using the protoc compiler, making for a compact and performant representation, though one which cannot as easily be inspected as in the case of a textual transfer protocol such as HTTP/1.x.

A Google open-source project, gnostic aims to allow the use of the OpenAPI Specification to define APIs using the protobuf format.

Thanks to Marco Palladino for catching a thinko in this article.

A brief history of the OpenAPI Specification

Mike Ralphson — Mon, 17 Dec 2018 09:45:28 +0000

In 2009, Tony Tam of Wordnik (an online dictionary service) had begun working on what would become the Swagger Specification. Used to describe Wordnik's JSON API, Swagger was used to drive Wordnik's developer console / documentation (which can still be seen at https://developer.wordnik.com/docs), their server integration and code-generation within the company.

Initially known as SWAGR - "why WADL, when you can SWAGR?" said Wordnik colleague Zeke Sikelianos when discussing potential names for the project with Tam (though if SWAGR ever stood for anything, it appears to be lost in the mists of time) - Tam published the first version of the Swagger Specification (1.0) in August 2011, just one month after Mashery announced I/O Docs, though there had been at least a 0.1a version used internally.

While working at Reverb Technologies, Tam published version 1.1 of the Swagger Specification in August 2012. This was a minor refinement of the specification only.

Version 1.2 of the Swagger Specification was the first version written up as a formal specification document, in March 2014, separating the specification itself from the Swagger implementation. Improvements were made to the type system, to bring it more into line with JSON Schema Draft 4. This was the first version to gain widespread adoption across the API industry, and is still in use by some API providers today.

September 2014 saw the release of the Swagger Specification 2.0. This saw a reorganisation from the two-file format of Swagger 1.2 and earlier into a single document structure, but one that could still reference sub-documents using the JSON-Reference standard. Further improvements were made to the type system / JSON Schema support, fuller API metadata support with the contact and license objects, the use of markdown in descriptions was allowed and vendor extensions (now known as specification extensions) and response headers were supported among other more minor changes. Swagger 2.0 saw more growth and adoption across the industry, and the number of supporting commercial solutions and open-source offerings increased dramatically.

In March 2015, SmartBear Software, a leading API testing and development tool company, acquired the interests in the Swagger intellectual property and open-source projects from Reverb Technologies.

In September 2015, Tam joined SmartBear Software as VP of Products, Swagger.

In December 2015, the Swagger Specification was donated by SmartBear Software to a new open-governance organisation, set up under the auspices of the Linux Foundation: the OpenAPI Initiative (https://openapis.org).

Version 2.0 of the OpenAPI Specification (as it was now known) was identical to the Swagger 2.0 specification. Although there was, and remains, some confusion between the use of the terms Swagger and OpenAPI, this was a positive first step in the transition to the new governance body and the new name for the specification.

Ten companies were founding members of the OpenAPI Initiative (3Scale, Apigee, CapitalOne, Google, IBM, Intuit, Microsoft, PayPal, Restlet and SmartBear) and the number of member organisations now stands at over 30.

The OpenAPI Specification is now managed by a Technical Steering Committee (TSC), and since the departure of Tony Tam in 2017 this has comprised long-time contributor Ron Ratovsky of SmartBear Software, Darrel Miller of Microsoft, Jeremy Whitlock and Marsh Gardiner, both now at Google, Uri Sarid of Mulesoft, and the author of this blog. The TSC draw from the wider Technical Developer Community to develop the specification in an open, transparent fashion. Membership of the OAI does not confer voting rights over content of the specification itself.

In July 2017, the OpenAPI Initiative announced the release of version 3.0.0 of the OpenAPI Specification, containing several structural improvements, closer alignment with JSON Schema draft 5, new features such as links and callbacks, standardisation on CommonMark for descriptions and many small tweaks and clarifications. Two patch releases to the 3.0 line were made in 2018. Version 3.1.0 of the specification is being actively worked on, and a release is expected in mid 2019.

The OpenAPI Initiative defines the OpenAPI Specification in these terms:

"The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for REST APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenAPI, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. Similar to what interface descriptions have done for lower-level programming, the OpenAPI Specification removes guesswork in calling a service."

Just as REST is not the new SOAP, OpenAPI is not the new WSDL. Its capacity for design-first development, wide range of tooling and cross-platform support take it to places only dreamed of by web services.

Like any API description language, the OpenAPI Specification is not capable of fully describing every conceivable API design, so retrofitting it to an existing code-base can be an uphill struggle. But as you gain familiarity with the OAS and adopt or transition to a design-first workflow, you will begin to see patterns emerge in the specification which will help shape your design decisions. Not only will you be working with the specification and not against it, you will gain commonality with other APIs described using the OAS, which will translate into familiarity and clear understanding by users of your API who also understand the OAS and perhaps have experience of many other APIs described with it.

The OpenAPI Specification has a rich ecosystem of tooling, both commercial and open-source (where more than 2,000 projects are known to exist, though not all of them yet have support for OpenAPI 3.0). This ranges from editors and low-level parsers and validators, to documentation-generators and dynamic consoles, testing tools and code-generation services.

Thanks to Tony Tam for providing a correction to this article.

Why there is no such thing as the Richardson Maturity Model

Mike Ralphson — Thu, 13 Dec 2018 08:01:14 +0000

One common yardstick used to measure the RESTishness of an API design is what has become known as the Richardson Maturity Model (or RMM). Originally known simply as the Maturity Heuristic, Leonard Richardson's scale defines the following levels of maturity of an API design:

Level Zero: One URI, one HTTP method
Level One: Many URIs, one HTTP method
Level Two: Many URIs, each supporting many HTTP methods
Level Three: Hypermedia

We can recognise Level Zero as RPC, or SOAP style APIs.

Level One APIs have the concept of resources (or representations), but all requests use one HTTP method (such as GET or POST). This fails to leverage the HTTP protocol to its fullest, and as Richardson notes, makes it very easy to destroy data by accident.

Level Two is probably the commonest level seen in production APIs today. It represents all of Fielding's mandatory REST constraints apart from HATEOAS.

Level Three is REST, including the hypermedia constraint. Though Richardson in his original discussion (https://www.crummy.com/writing/speaking/2008-QCon/act3.html) states:

"A lot of people settle for level two because hypermedia is difficult to understand and its value in the web service domain isn't as clear."

Unfortunately, the Richardson Maturity Model has been popularised by Martin Fowler in a somewhat bowdlerised form from the original. Unhelpfully, the graphic on his website, often reproduced, chooses to rename the levels above as:

Level Zero: The Swamp of POX
Level One: Resources
Level Two: HTTP verbs
Level Three: Hypermedia controls

Even if we accept the application of a maturity model to API design as being somewhat biased, implying as it does that REST APIs are more "mature" than RPC APIs (when in fact they simply serve different needs and purposes), there is little excuse for using pejorative terms like "swamp". "POX" also sounds unpleasant, but here simply means 'Plain Old XML'.

Either way, the popularised version of the RMM makes no distinction between resources and representations and only adds to the confusion between HTTP 'verbs' and HTTP methods, which the original does not.

Richardson himself at RESTFest 2015, in a talk subtitled "What Have I Done?", described the whole thing as "very embarrassing" and clarified that if you want to proceed up the steps of the Maturity Heuristic, you have to have either a technical reason or a political reason to do so. There is also no prize or "glory" at the top - sorry to disappoint. (https://vimeo.com/147237932)

Do please refer to Richardson's original work for more details on this model.

Thanks to Darrel Miller for pointing out Richardson's later talk.

The Bluffer's Guide to PC Memory (archive)

Mike Ralphson — Thu, 08 Nov 2018 10:21:39 +0000

Note

This is an example of why it's so important to put a date on your blog posts! This is one of my earliest posts (can anyone hazard a guess at the date it was written?) and in fact, pre-dates the whole notion of blogs. It was written for a FidoNet echomail area and for download via modem.

I'd love to hear your thoughts in the comments. Especially if it comes in handy for anyone using DOS emulators etc.

Starting with the Basics

Conventional memory is the PC's native type of memory, and can be accessed directly by DOS and applications programs. It can be limited physically by the amount of memory installed (usually on the motherboard) to say 256 or 512 Kb, but normally 640 Kb will be installed.

Most 8086 based PCs (usually known as XT architecture) will have 640 Kb of conventional memory and nothing else; therefore DOS, device drivers, configuration resources (FILES and BUFFERS), Terminate and Stay Resident (TSR) programs and applications must all co-exist in this 640Kb area of memory.

Why Is There a 640Kb Limit in DOS?

Simple answer: there isn't. The original Intel 8086 processor chip (as used in the first PCs) can access 1 Mb of RAM directly, and so can DOS. It was IBM's decision to reserve the area from 640 Kb to 1 Mb for the system BIOS (Basic Input / Output System), adaptor ROMs (for example for Hard Disks and Network cards), as well as areas of memory used for graphics adaptors such as MDA, Hercules, CGA, EGA and VGA.

Because there are so many adaptor cards that can be installed in a PC, it is not possible to predict which areas of this 384 Kb reserved area are being used and which are not. Some manufacturers use a BIOS of 128 Kb (IBM and Compaq notably) though most clone machines have a 64 Kb BIOS.

Thus it is not really possible to produce a machine with a full 1 Mb of conventional memory, though it is possible to use the areas of RAM installed for VGA and Mono graphics modes for additional DOS use, thus extending DOS's 'memory arena' from 640 Kb to 736 Kb. An example of a program to do this is Quarterdeck's VIDRAM which is supplied with their QEMM memory manager.

I've Got a Bloody Big Spreadsheet and I Need More Memory!

A typical cry heard in the early 1980's, which was solved by the invention of Expanded Memory (or EMS). EMS memory is located outside the conventional 1 Mb, which is accessed by a 64 Kb window. In XT architecture machines EMS is usually provided on a special memory expansion card and accessed with a device driver.

The device driver allocated 64 Kb of conventional memory, which can be used to move chunks of data between EMS and conventional memory. Most EMS managers can handle about 16 Mb of memory on 8086 and 80286 machines. It is not possible to run programs (device drivers, TSRs or applications) in EMS memory, as only small chunks are 'paged' in at any one time.

EMS on 80286, 386 and 486 Machines

With the advent of Intel's new processor chips which have inbuilt memory management functions, it became much easier to bypass the DOS 640 Kb barrier. The 80286 can have up to 16 Mb of extended memory to play with and the 80386 / 486 can access up to 32 Gigabytes (32,000 Mb).

Hang on? Extended memory, not Expanded? That's right. Intel added a new mode to their processors starting with the 286, called Protected Mode (the 8086 can only operate in what is now called Real Mode). In this mode the processor can access all of the available memory in the same manner, without having to resort to complicated paging schemes like EMS.

There were three alternatives; scrap EMS and start using Extended Memory, scrap DOS and start using OS/2 (which runs the processor in Protected Mode and can access that whole 16 Mb), or write device drivers that simulated EMS in Extended Memory.

A few people moved to OS/2 version 1, but the number of applications available was very small. OS/2 version 2 which uses the abilities of the 80386 (a third mode called Virtual Mode) to manage memory and multi-task without fear of one program corrupting another, is a much more viable operating system.

Most people didn't want to give up their favourite applications, and their favourite applications wanted EMS. The easiest solution was to write those device drivers.

Why The 80386 So Soon?

The reason the 80286 was never used as much except a faster version of the 8086 is that it has quite a major design flaw in it. You can change the processor from running Real Mode to Protected Mode with a machine code instruction, but you can't change it back!

Eventually motherboard designers came up with a hardware solution, they wired the SHUTDOWN pin of the 286 back to the RESET pin. Then a program can tell the processor to turn itself off, and this will in fact cause the 286 to reset itself into the mode it uses when it is first turned on, which happens to be Real Mode.

Unfortunately, it's not quite as easy as that, as the program has to make sure that the processor will remember what it was supposed to be doing when it switches back into Real Mode, and this involves quite a detailed knowledge of 80286 machine code and behaviour.

Enter the 80386 processor. As well as Real Mode and a Protected Mode which worked properly, it also had one important new ability. The ability to lie. With the 386 it is possible to instruct the chip to pretend that a range of memory addresses are really another range of addresses. When a program accesses a byte of memory in the 'logical' range, the processor will translate this into a different physical address and use that instead.

(Some later 286 systems with better chipsets, for example NEAT and C&T, can perform some of the memory-relocation functions of the 386, but only with the supplied driver programs.)

The 386 can even implement a virtual memory system, where it can declare areas of the 32 Gb address-space to exist not as physical memory, but as say hard disk space, so the most often used areas of memory will be held in RAM and the less often used areas will be held on disk. DOS doesn't use this technique, but OS/2 version 2 does and Windows/NT might when it is released.

As mentioned above, a new mode was also added called Virtual 8086 Mode, which allows the processor to emulate several separate Real Mode computers, each of which can be running its own programs. This is an extremely safe method of multi-tasking. Windows and OS/2 both use this method to run several programs at once, although OS/2 is more stable than Windows because Windows for some reason runs all Windows programs in the same Virtual Machine (it runs DOS programs in separate VMs though). Thus if one Windows program crashes, the whole system will go down.

OS/2 runs each DOS, Windows or OS/2 application in its own VM, offering extremely good protection against badly behaved or badly written programs.

What's Upper Memory Then?

With the advanced memory management features of the 386 (and a few 286s), it is possible to move a section of memory from one address to another (as far as programs are concerned). Using this technique, the unused areas of memory between 640 Kb and 1 Mb can be 'back-filled' with memory from above 1 Mb.

This creates Upper Memory Blocks or UMBs.

This memory looks just like normal conventional memory to DOS, and you can use it for storing programs and data just like memory below 640 Kb. Because some older programs might get upset if the 'top' of DOS memory isn't at 640 Kb, the UMBs are usually treated as a separate pool of memory, accessed with the DEVICEHIGH command in CONFIG.SYS for loading Device Drivers and the LOADHIGH (or LH) command for loading programs.

There is an undocumented feature of DOS which allows the UMBs to be joined to the rest of DOS memory, allowing COMMAND.COM to be loaded high, and this gives more memory to certain programs.

To switch DOS into this memory-allocation strategy you need a small program (such as my AUTOHIGH.EXE) which tells EMM386.EXE to enable this feature.

And the High Memory Area?

The High Memory Area (or HMA) is a 64 Kb area of Extended Memory which can be used from Real Mode because of a quirk in the way the Intel chips access memory.

With 16 bit registers the PC would only be able to access 64 Kb. Not a lot. To access the 1 Mb in the original IBM PC, the 8086 would have needed 20 bit registers. To get around this problem Intel decided to use a segmented memory scheme. The 1 Mb is divided up into 64 Kb segments, but these start every 16 bytes, thus they overlap quite a lot.

The last memory address in the first 1 Mb is FFFFFh, which is expressed in segment:offset form as FFFF:000F (to calculate the actual address, shift the segment left by one hex-digit and then add the offset).

XT architecture machines have address lines to the processor called A0 to A19. These are the 20 bits needed to access 1 Mb. Processors such as the 286 and above have further address lines from A20 upwards to allow them to access the rest of their 16 Mb.

If there was a way to allow use of the A20 line in addition to the A0 - A19 lines in Real Mode, there would be nothing stopping a program accessing the rest of the segment that starts at FFFF:0000 all the way up to FFFF:FFFF.

This would give DOS another 64 Kb of memory to use, without having to switch processor modes between Real and Protected to get at it. This is quite an advantage as switching modes is quite a slow process, and the extra 64 Kb could be used for loading DOS itself (IBMBIO.COM & IBMDOS.COM or IO.SYS & MSDOS.SYS depending upon whether your copy of DOS comes from IBM or MicroSoft).

Because not all motherboards control the address lines in the same way, a routine called an A20 handler must be installed, one that matches the motherboard or chipset in use. DOS's HIMEM.SYS contains several A20 handlers and will load the correct one when it is installed.

What About Extended Memory?

Ok, so we know about Upper Memory Blocks and the High Memory Area, can we use Extended Memory instead of Expanded Memory? I mean, you wouldn't be limited to accessing it 64 Kb at a time, and you wouldn't have to give up 64 Kb of conventional memory for a 'Page Frame' like you do with EMS.

Well, yes. You could access Extended Memory yourself, like old versions of RAMDRIVE.SYS used to do, but there isn't a standard way of checking to see which bits of Extended Memory are being used by other programs. You would have to try to communicate with each program that you suspect might be using some Extended Memory and ask it, using its API (or Application Programing Interface).

This is, to say the least, extremely tedious. To standardise use of Extended Memory, and to give it a proper abbreviation, XMS was devised. This is a management specification for Extended Memory, and allows all programs to enquire about and use Extended Memory in a standard manner.

The normal way to manage Extended Memory is with an XMS memory manager (or XMS provider) like DOS's HIMEM.SYS or Quarterdeck's QEMM.SYS

MicroSoft in particular has gone XMS mad, if you want EMS on your 386 these days, you have to load HIMEM.SYS first to convert your Extended Memory into managed XMS (HIMEM also loads an A20 handler if possible to convert 64 Kb of your Extended Memory into the High Memory Area), then load EMM386.EXE to emulate EMS in XMS. EMM386.EXE is also DOS's UMB provider, though you can have UMBs without having EMS and vice versa.

QEMM combines the features of HIMEM.SYS and EMM386.EXE and allows dynamic allocation of memory, thus your free memory is in limbo until a program asks for it using either the XMS or EMS APIs. If you have 2 Mb of managed memory, you can allocate all of it as EMS, all of it as XMS or as a bit of both.

As programs free up memory, you can then use it again for either of the memory standards.

Windows requires XMS, SmartDrive requires XMS, RamDrive requires XMS. Unfortunately, older applications and games tend to still require EMS and don't know about the newer and rather more sophisticated XMS specification.

What Version of DOS and What Sort of Memory Should I Have?

Consult the following table for some guidelines:

Processor	DOS Version	Memory Type	Where	Software Driver
8086 / 8088	3.30 or 3.31	Expanded	Add on card	H/ware specific
80286	3.30 or 3.31	Expanded	Motherboard	H/ware specific
80286	5.00 +	Extended	Motherboard	DOS / QEMM etc
80386	5.00 +	Extended	Motherboard	DOS / QEMM etc
80486	5.00 +	Extended	Motherboard	DOS / QEMM etc

The Memory Type column shows the type of Hardware memory installed, 386s can emulate Expanded memory in Extended (XMS) memory.

DOS version 3.31 is an OEM version supplied by Compaq and Tandon (and possibly a few other manufacturers) that can support hard disks above 32 Mb. If you can't get hold of it, use 3.30 with Ontrack's Disk Manager if the hard disk is bigger than 32 Mb.

If you can't get hold of DOS 3.3 for an 8086 or 80286, then use DOS 5.00 + rather than DOS 4 which is notoriously flaky.

DOS 5.00 + refers to MSDOS or PCDOS version 5.00 or higher. It does not mean DRDOS 5 (which is a DOS 3 compatible product) or DRDOS 6 which is not command-compatible with any official version of DOS.

While You're At It, How About VCPI and DPMI?

The Virtual Control Program Interface and DOS Protected Mode Interface are commonly known as DOS Extenders. DPMI effectively replaces VCPI. These allow programs to run in Protected Mode without having to change to another operating system, allowing both Code and Data to reside above 1 Mb. XMS and EMS only allow data to be stored above 1 Mb and the PC remains in Real Mode at all other times.

Programs written to use DOS extenders must load a VCPI or DPMI server program before they can run. Some memory managers such as QEMM support VCPI (but not currently DPMI I think), DOS's memory managers do not, so you may have to install a device driver, or even buy a runtime copy of the DOS extender.

Advanced DPMI servers like Borland's are royalty-free (so programmers can include them with their applications) and automatically load themselves when the application program starts up.

What Is Memory Parity?

To hold one byte of data in memory, you need 8 bits. However, as processors get faster and motherboards get more complicated it becomes necessary to be able to check that what you put into a memory location is the same as what you read from it. The PC has had this mechanism from day one, because in the early days of PCs motherboards and memory were less reliable than now, even though hardware speeds were much lower.

If you add a ninth bit to every byte of memory, you can store a 1 or 0 in that extra bit depending on whether the contents of the other 8 bits have an Even or Odd number of 1's. This provides a simple check to see that the contents of memory haven't been changed by stray electro-magnetic influences.

If you experience Memory Parity Errors, you should check that all memory (whether chips, SIMMs or SIPPs) modules are securely seated and they are all rated at the correct speed (or faster). Typical memory speeds are around 180 to 120 ns (nano-seconds) for XT machines and anywhere between 90 and 60 ns for more modern ATs.

Memory Hardware Types

Memory typically comes in three forms, chips (more properly known as DIPs - Dual Inline Packages), SIPPs (Single Inline Pin Package) and SIMMs (Single Inline Memory Module).

SIMMs are basically a replacement for SIPPs, both are small 2 x 9 cm PCBs with chips surface mounted onto them. There might be 8 or 9 chips (the 9 chip variety are those with parity and the most common), or the new breed of SIMMs with three chips, the two larger ones are 4 bits 'wide' each and the smaller one is for parity.

SIPPs have a row of pins at the base to plug into the motherboard or expansion card, SIMMs have a small edge connector, like expansion cards themselves.

SIMMs and SIPPs usually come in 256 Kb, 1Mb and now 4 Mb capacities.

DIPs can usually be identified by the last 4 digits of the part number on the chip casing. They take the form of XYYY, where X is the width in bits and YYY is the 'length' in kilo-bits. (Not all chip manufacturers use this notation.)

Thus if the code is 4256, then the chips are 4 bits wide and 256 kilo-bits (262144 bits) long, so you would need 2 to make 256 Kb because each is only half the width of one byte. You can simply say that each chip is 128 Kb in size. To make 1 Mb, you would need 8 chips, or 9 if parity is required.

Motherboards with more than 1 or 2 Mb fitted will usually have SIMM or SIPP sockets.

Static and Dynamic RAM

There are two types of RAM, that which has to be refreshed (read and rewritten) fairly often and that which does not. Dynamic RAM (the type that has to be refreshed) is cheap and therefore used for the majority of your PC's memory.

On XT machines you can actually alter the rate at which the timer circuitry refreshed the RAM, the less time spent refreshing the RAM, the more time the CPU / bus can be doing something else (an increase in PC speed of a few percent was possible). All you needed to do was slow down the rate of refresh rate until the machine stopped working, and then use a refresh rate just a tad faster than that! IBM stopped people mucking about like this with the introduction of the AT.

Static RAM does not need to be refreshed, because it holds its electrical charge better, this makes it much faster than dynamic RAM. It is also much more expensive, and is only used in critical areas of memory (see Processor Cache RAM below).

CMOS memory (Complimentary Metal-Oxide Semiconductor) can hold its charge for a very long length of time, while provided with a constant (but small) voltage. CMOS RAM (sometimes called Non-Volatile RAM or NVR) is used to hold the configuration information on AT machines and in some modems. Some CMOS chips require an external battery, while others have a small cell actually inside the chip casing. Both sorts of battery are usually trickle-charged while the machine or modem is powered from the mains.

Processor Cache RAM

Processor Cache RAM can be either internal (on the processor chip itself) or external. All 486s have an internal cache (8 Kb on Intel models, only 1 Kb on Cyrix's plug in 386sx replacement the 486SLC). Most 386s and 486s have external caches. This is to prevent the memory being a bottleneck in the system, as processor speeds start to outstrip the speed at which memory can store and retrieve information.

Processor RAM caches are always Static RAM rather than Dynamic RAM to give the speed advantage required.

Typical external cache sizes are 16, 32, 64, 128 or even 256 Kb. A cache of just a few Kb will allow the processor to fetch over 90% of instructions from the cache rather than main memory (called the hit-rate), after that the law of diminishing returns kicks in. The more cache you add over say 64 Kb the benefits you will see start to get smaller and smaller.

Hard Disk Caches

There are three types of Hard Disk Caches, internal, external and software. Most IDE, ESDI and SCSI drives will have a small read-ahead cache internally as part of the drive electronics of a few Kb. When the hard disk is asked for the data on a particular sector of the disk, the cache will buffer the next few sectors as well, as it is quite likely that they will be the next sectors asked for by the program.

External caches and software caches work in similar ways. External caches are usually situated on the drive controller card and populated with SIMMs, software caches are usually set up in Extended memory. Either kind can be a read-only or a read-write cache.

Read-only caches try to keep track of the data that is read from the disk most often as well as using read-ahead techniques similar to those described above. Areas such as the FAT (File Allocation Table) and directories of the disk are typical areas that are likely to be kept in the cache.

Hard Disk caches are useful because retrieving data from RAM is orders of magnitude faster than even the fastest hard drive.

Read-Write caches save up writes to the hard disk and tries to perform them in more efficient bursts, either when the processor doesn't have anything else to do, or to perform them in the most efficient order, meaning the drive heads don't have to travel so far.

One problem with read-write caches is that the power to the machine could be interrupted before crucial data is written to the hard disk. Most read-write caches trap the CTRL-ALT-DEL keypress, but none can prevent data loss through a power-cut or the accidental touch of the reset-button.

Replacing embedded GoogleMaps with OpenStreetMap

Mike Ralphson — Tue, 08 May 2018 09:32:19 +0000

Recently, Google announced that as of June 11th 2018 a billing account (credit/debit card) and API key would be mandatory requirements for using GoogleMaps, even if your usage currently fell below the free tier of $200 'value' per month.

If however, you feel this is an unnecessary complication, and wish to avoid nasty surprises, especially if someone else were to use your API key, then it may be time to investigate an alternative, such as OpenStreetMap.

My usage was simply to provide a dynamic location map on my website, so the steps for me were particularly simple:

Go to OpenStreetMap.org
Search for the location you want as the centre of the map, or use the arrow to Show My Location
Select the desired layers, such as Standard, Cycle Map, Transport etc
Zoom and pan as necessary, using the + and - icons and your mouse
Select the Share icon
Decide if you want to show a marker, and tick the checkbox if so
Highlight the HTML tab
Copy and paste the resultant HTML fragment into your website

For example:

<iframe width="425" height="350" frameborder="0" scrolling="no" 
marginheight="0" marginwidth="0" 
src="https://www.openstreetmap.org/export/embed.html?bbox=-2.493209838867188%2C53.50540525319918%2C-2.246360778808594%2C53.61980121473449&amp;layer=mapnik&amp;marker=53.56274386269267%2C-2.3699569702148438" 
style="border: 1px solid black"></iframe>
<br/><small>
<a href="https://www.openstreetmap.org/?mlat=53.5627&amp;mlon=-2.3700#map=12/53.5626/-2.3698">
View Larger Map</a></small>

As you can see, the key ingredients are the latitude and longitude co-ordinates of the bounding-box of the map area required. The%2Cs are URL-encoded commas. You can also tweak the iframe width and height etc to suit your needs.

Happy mapping.

Defining reusable components with the OpenAPI specification

Mike Ralphson — Mon, 29 Jan 2018 07:39:18 +0000

A pattern I'm seeing increasingly, is to compose OpenAPI specification (OAS, formerly known as Swagger) documents from multiple files. This allows re-use, easier collaboration and makes larger documents much easier to follow.

The resultant document can be parsed by tools which fully understand and implement the JSON Reference specification (i.e. they include a resolver), or it can be 'bundled' by de-referencing external references using such a JSON Reference-implementing tool.

But what should those fragments of OAS documents look like?

The OAS specification makes no rulings on this, other than the object being referenced must be of the expected type.

Thus a parameter definition in a self-contained OAS document may look like this:

...
paths:
  /:
    get:
      parameters:
      - $ref: '#/components/parameters/sort'
...
components:
  parameters:
    sort:
      name: sort
      in: query
      schema:
        type: string
        description: 'The direction of the sort'
        enum:
          - asc
          - desc

A parameter referencing an external fragment may look like this:

...
paths:
  /:
    get:
      parameters:
        - $ref: './includes/parameters.yaml#/sort'

and parameters.yaml could simply look like this, with no additional structure:

sort:
  name: sort
  in: query
  schema:
    type: string
    description: 'The direction of the sort'
    enum:
        - asc
        - desc

If we wished to compose our reusable components across subject-matter areas rather than structurally relating to the OAS, we might structure our documents like this:

paths:
  /:
    get:
      parameters:
        - $ref: './includes/parameters.yaml#/parameters/sort'

parameters:
  sort:
    name: sort
    in: query
    schema:
      $ref: '#/definitions/sortType'
definitions:
  sortType:
    type: string
    description: 'The direction of the sort'
    enum:
      - asc
      - desc

The problem with both styles of included document, is that neither is defined by any standard other than being valid JSON or YAML. Neither can be validated against a specification, even though their contents should be composed of valid OAS objects.

It raises questions such as: is it allowable to share the definition of the sortType schema across OAS v2 and OAS v3 documents, even though the definition of the Schema Object differs between the two specifications?

I would argue there are significant benefits in always structuring your included documents as fully valid OAS documents.

At the cost of the following six lines of metadata overhead (a little more in JSON), and slightly longer $ref values, our parameters.yaml document becomes:

openapi: 3.0.1
info:
  title: An include file to define sortable attributes
  version: 1.0.0
paths: {}
components:
  parameters:
    sort:
      name: sort
      in: query
      schema:
        $ref: '#/components/schemas/sortType'
  schemas:
    sortType:
      type: string
      description: 'The direction of the sort'
      enum:
        - asc
        - desc

This document now has the benefits that it can be validated, linted, converted, transformed and edited using OAS compliant tools. In the case of conversion, it means OAS v2 document fragments can be upgraded in place to OAS v3 without having to bundle the document into a monolith.

The minimal info object gives us properties to describe the expected usage of the fragment, and an ability to version it separately from the 'master' OAS documents which reference it.

Simply by the presence of the empty paths object, we can tell we are dealing with a fragment, not a fully-defined OAS document.

These reusable sub-documents can also be shared between projects, allowing for industry-specific standard components to emerge.

The Hypermedia Hitchhiker's Guide to Standardised Affordance Metadata

Mike Ralphson — Mon, 22 Jan 2018 19:00:07 +0000

REST is asking Siri "What can you do?" RPC is lookup up a list of commands from osxdaily or trying to guess.

Phil Sturgeon

Let's extrapolate from that a little.

What is Siri's response likely to be, and what would that look like from a REST API?

Siri: What can you do?

Let's bang the root of that hoopy API and see if we get back anything useful.

GET /

Holy swutting Belgium man!

{
    "_links": [
        {
            "rel": "self",
            "href": "/"
        },
        {
            "rel": "siri:whats-playing",
            "href": "/nowplaying"
        },
        {
            "rel": "siri:play-song",
            "href": "/nowplaying/{songName}"
        },
        {
            "rel": "siri:send-message",
            "href": "/messages/{recipient}/{message}"
        },
        {
            "rel": "siri:buy-book-by-title",
            "href": "/purchases/book/{title}"
        },
        {
            "rel": "siri:buy-book-by-title-and-author",
            "href": "/purchases/book/{title}/by/{author}"
        }
    ]
}

By the way, being a Linux user, I naturally had to get the information for the list above from OSXDaily, which we can think of as a static API description document (a la the OpenAPI Specification), after having to google for it, which we can think of as some kind of API discovery mechanism / registry lookup.

The perennial question now is how do we perform those link relation actions?

“Accessing any hypermedia API in the sophisticated Eastern spiral arm of the galaxy tends to pass through three distinct and recognizable phases, those of Location, Enquiry and Confusion, otherwise known as the Where, What, and How phases. For instance, the first phase is characterized by the question 'Where is this API?' the second by the question 'What can I do with this API?' and the third by the question 'Oh I give up, how am I supposed to know what to do next?”

With apologies to Douglas Adams

It may be obvious that a GET request to /nowplaying will return information on the currently-playing piece of Bach, but to rock out to some Procol Harum, do I POST to /nowplaying/whiter%20%shade%20of%20pale, or possibly PUT or even PATCH the resource representation?

If I want to buy a book by a particular author (I want the original "Last Chance to See" by Douglas Adams and Mark Cawardine, not the sequel by Mark C. and Stephen Fry), can I pass both authors? Do I concatenate them? Do I use an array?

What if I could consult the Hitchhiker's Guide to the API?

Obviously, out-of-band documentation is a REST no-no, but what about in-band documentation?

What if there was a standardised way of retrieving metadata about a resource... such as:

OPTIONS /nowplaying/:songName

Which returned a minimal, dynamically-generated OpenAPI definition for the given endpoint (including any necessary supporting definitions, such as parameters, headers, responses etc).

The OpenAPI format is used to describe the affordances of the endpoint, and the Allow response header gives us the supported HTTP methods for the endpoint to act as indices into the OpenAPI pathItem object.

The affordances metadata can be effectively cached by using the ETag and If-None-Match headers.

If the metadata hasn't changed since the last time the client checked, the API should respond to the OPTIONS request with HTTP 412 - "Precondition Failed" as per RFC7232.

Obviously, other API description formats could be returned based on content-type negotiation.

An alternative to using the OPTIONS method is to include a link with the relation type of service-desc, and link your partial OAS definition that way.

If you want to play with returning partial OpenAPI definitions, I've knocked together a proof-of-concept utility in Node.js - openapi-extract. It works with OpenAPI (fka Swagger) 2.0 and 3.0.x.