A few of us built the IG public API in around 6 months, a feat we were all tremendously proud of.
But mistakes were made, lessons were learned, but unfortunately too late.
Here is what I wish we'd done differently.
To Gateway Or Not?
We evaluated a number of API gateway products (Apigee, Layer7 and WS02), but decided to build our own. The reasoning was:
- Our trading API was latency sensitive. The fastest of the products (WSO2) added 3ms, which may not sound like much, but in trading, even online trading, it is.
- Our backend APIs were already REST, so all the fancy ESB-type integration functionality the gateways offered were of no interest to us.
- We had security solved, so didn't need OAuth (and anyway, latency went up to around 50ms with OAuth on!)
- The gateway web portals were too prescriptive for our liking - we wanted ours to be heavily IG-branded and customised, not just "white-labelled".
So we built a thin API adapter framework on top of our existing internal REST APIs, doing little more than adding basic API key security, transformation and documentation generation from source.
It turns out our security solution was inadequate, and we did need OAuth. We ended up using OpenAM (which is an excellent freemium open source security platform, by the way).
We invested too much time in our own web portal, and still, in my view have ended up with a sub-standard offering. Better to have been happy with 80% of a 3rd party offering, and ended up with a more polished overall product.
We used IG's RESTdoclet at the time to generate our API documentation, but should have used Swagger (which we subsequently moved to internally). A number of the gateways supported Swagger.
Having said that, think carefully before you generate documentation. If you put your documentation in source code, make sure that it is clearly marked as content that will be visible on the web, and subject it to rigorous review by a skilled content writer. Don't assume developers know how to write content. They usually don't.
We're now considering moving the adapter layer into a separate, how shall we say... er ... gatew. No, I can't say it.
Not so RESTful
I like REST. It offers a very simply API model which is easy to get to grips with for the average punter. We did a lot of research before making the following mistakes:
Modeling using URI
We tried to get too clever here, and model a URI hierarchy that mirrored our domain resource taxonomy. Unfortunately the model turned out to be too complicated for a simple resource tree, and we ended up with an ugly set of URIs in some instances.
Using HTTP DELETE
In finance systems, there is often very little opportunity to really delete things - at best one marks things as deleted. So that is more of an update (PUT) than DELETE. But we persisted, and found two cases where we could use DELETE:
logout: DELETE /session (the argument being that logging in is just creating (POST) a session, and logging out is deleting the session.) Intuitive as a brick.
closing a position: the problem here is that positions can also be partially closed. So we thought, no problem, just put that data in the request body. This is bad news and not widely supported, so we ended up using PUT (as we should have done).
We still have DELETE /session to logout, but nobody uses it. This hurts.
Using HTTP status codes
It seemed brilliantly simple at the time. 200 means success. 400 error range means the client needs to fix something. 500 error range means the server has to fix something.
Many objections to this will be religious, some suggesting that HTTP status codes were never intended for APIs, and only 200 should be used. OK, fine, but then you have to handle 401 and 403 security errors, so why not use 404 (not found) for lookup errors and 400 (bad request) for validation errors? And 500 range for system errors?
This all falls apart when you try to return error details in the response body. For example our CDN (Akamai) happily replaced our custom 500 error message with its own.
So in the end we stopped using 500 message response bodies, which is a shame. A good pattern here is to generate an internal GUID for a logged system error and then return that to the caller, so they have something to quote when they phone up support.
Using headers to version
We didn't want to clutter our beautiful URIs with versions, so we put the API version in the header, and applied it at the "operation" level, i.e. URI + HTTP method.
I could talk at length about versioning, but will save that for a future post. In summary I think API versioning should be as follows:
Version number at the beginning of the URI.
- non breaking change (no version number change)
- breaking change (version number change)
To B2C or B2B?
Being "agile" means we sometimes forget to think things through completely before starting to cut code. We didn't think through who would be using our API. We thought: REST is easy, so our customers will finally be able to automate their trading. They will just come, and trade... kerching!
And many did, but many struggled, and many didn't even trade: they just got our free (i.e. no kerching!) market prices. But we had a lot of support calls, ranging from bugs (which is cool) to "Can you tell me how to write a program?" (which is understandable but irritating. I was so tempted to send a link to my university degree course).
In one sense I don't regret opening up the API to the masses, since this really put our API through its paces, but the power-users of the API ended up building applications (with the help of developers they hired) - which is precisely our B2B model, and where the real money is, in my humble opinion.
But we spend so much time supporting the punters, and it's just not cost-effective.
We made the mistake of "building it and hoping they would come", without thinking who would come, and who should come.
Top comments (2)
my 2 cents:
A. APIs should definitely use HTTP status codes as part of being RESTful. For example, some times it makes sense to return just 200, but in other cases it makes more sense to return 201 (created), for a POST action for example.
B. Error messages in REST/API applications is indeed a topic that has a lot of confusion but also many do things different, but what's probably considered a standard is that you should always assign and provide an actual error code (i.e: ERR1234) for your errors so your users can easily reference it. If you're generous enough, you can also provide the textual representation of the error ("not enough credits"), or be verbose on what field validations failed for example.
Very nicely written, thank you for that!