DEV Community: Chris Williams

Ai isn't replacing your job - trust me, we've been here before.

Chris Williams — Sun, 25 Jun 2023 09:20:42 +0000

As someone who has weathered several transitions in the technology landscape, I've learned that change, though often intimidating, brings with it a wealth of opportunities.

I remember the early days of my career in web development when tools like HTML, CSS, and JavaScript frameworks were emerging. Many viewed these as threats to traditional coding roles. I recall the apprehension surrounding Dreamweaver in 1997. There was a sense that using such a tool, rather than coding HTML by hand, was somehow 'cheating.'

But in reality, these technologies didn't 'steal' jobs. They transformed them. They opened up new avenues for creativity, efficiency, and complexity in our work, enabling us to transition from hard coding to crafting more intricate, interactive sites.

Today, AI stands in a similar position. It's seen as a potential job-stealer, an automator of tasks that might leave us, developers, out of work. But if my experiences have taught me anything, it's that such innovations don't eradicate roles - they redefine them.

To the developers early in their careers, I would say this: Embrace these changes. Upskill. Adapt. AI isn’t here to replace you, but to augment your capabilities. Much like HTML, CSS, and JavaScript frameworks, AI will allow you to do more, to innovate further.

Remember, with every significant tech innovation, new roles emerge. The rise of web development tools gave birth to roles like UI/UX designers and frontend developers. Similarly, AI is creating exciting new positions like AI ethicists, data scientists, and machine learning engineers.

These technologies also democratise their respective fields. Just as HTML, CSS, and JavaScript have made the web more accessible, AI has the potential to do the same, making expertise more widely available and breaking down barriers.

Yes, fears are natural with change, but history shows that innovation typically creates as many opportunities as it disrupts. Embracing AI and adapting to its growth could lead to a richer, more diverse tech landscape.

Lastly, remember this: Technology is a tool designed to augment our capabilities, not replace them. HTML, CSS, JavaScript frameworks, and now AI, all serve to amplify our abilities to create, to solve problems, to make a difference.

As you shape your future, learn from those who have navigated past transitions. AI, like the web tools before it, is a tool in your hands. Use it wisely to create, innovate, and open up new possibilities for your career.

Moderating Actions on Stream's Activity Feeds service

Chris Williams — Sun, 11 Jul 2021 07:04:26 +0000

(headline image by Claudio Schwarz)

This is for Stream, or getstream.io's Activity Feeds service. This block is here so people who are searching for getstream.io can find it! I'm never sure how to referr to the company, as just calling it 'Stream' when discussing it in a development setting gets very confusing :(

INTRO

In this tutorial, we're going to talk about, and solve, a problem with Stream's Activity Feeds product: there's no build-in moderation tools.

Is that a problem? Yes, it really is. Skipping past the problems of the legals of who is liable for content on your platform, there's also compliance for User Generated Content on both App Store and Google Play to consider.

We're going to look at setting up your Activity Feed Types so we can create a firewall between Activities being added, and those being viewable. We're also going to talk about a way of handing image moderation: AWS's Rekognition service.

This isn't a tutorial on how to implement this, rather a guide on how to set up your Activity Feed Types so we can add moderation and other automation.

I'll cover the practical 'how-do' in another article (and I'll update this once I've written it!)

WHAT YOU'LL NEED

AWS account, and some basic understanding of what SQS is, and what Lambdas are,
Getstream account, and some knowedge of the Activity Feed service

AWS REKOGNITION

AWS Rekognition is a machine learning tool that Amazon have trained over billions of images and videos, and we can use that power just by calling a simple API. By using this, we don't have to have people looking at each image manually, and have to deal with all the problems that come with that.

You'll want to be using the 'Detecting Inappropriate Images' part.

We need to upload the image to AWS Rekognition (which is built into the AWS SDK), and it will return a series of Labels, along with a Confidence (a number on how sure AWS is that this Label is right). Then we can use those to determine if the images need to be hidden or not.

You can see in the AWS Rekognition documentation the list of Labels you can get back, and from that you can choose what kind of images you do and don't want on your platform.

STREAM

Getting the Activity Feeds Types right here is the key. We need to create a gap between the Activities created by users, and those activites being seen by other users.

Common Activity Feed Type setup

Imagine we want a platform where we have content creators and content viewers. You could set it so each Creator has a timeline feed, and each Viewer has a notification feed that is subscribed to one or many Creator's Timelines.

Something like this:

Any time creator-0001 publishes to their timeline feed, it will appear in the notification feed for viewer-9876.

You can see here, there is no way we can stop this happening. If creator-0001 posts something we don't want, we're in trouble.

Moderation Activity Feed Type setup

The solution is to create some kind of break between Creator publishing something, and the Viewer seeing it before we've had a chance to moderate it.

Stream provide a way of doing this. For each Activity Feed Type you can it up to puiblish the Activity to an SQS queue, so we can grab it from the queue, look at it, then publish it somehow.

Something like this:

This method allows us to code the link between the unpublished_timeline:creator-0001 and the published_timeline:creator-0001 feeds, and using AWS Rekognition we can inject image moderation into that step. In fact, with this step, we can do any kind of moderation or action based on the content published we want.

Your Lambda should take the code from the SQS queue (something Stream have written a tutorial on), and use the AWS SDK to send the image to Rekogntion. Once you have your Labels back, you can then choose to ''copy'' the activity from one feed to another.

There isn't a way of copying an Activity from one feed to another that I know off, so you have to re-create the Activity on the published feed, and remove it from the unpublished feed.. if someone from Stream knows a better way to do this, I would appreciate you letting me know.

CONCLUSION

With this set up, you will be able to perform any kind of moderation you wish on any Action published on your platform. Using Lambdas and SQS means your solution will scale with your demands, and you can use this model to build in any kind of events.

What about using SQS feeds to manage push notifications, or emails to users when content is published? It's also a good way of getting data out of getstream into something you can look at and use without over-consuming your API usage.

I would be super interested in knowing if this solution works for you, or how you have approached this problem! Please reach out on Twitter to @scampiuk and let me know how it goes.

A rather pointless performance test for Node

Chris Williams — Sun, 11 Oct 2020 21:07:40 +0000

So today I performance a rather pointless performance test.

I was wondering what difference the version of Node makes to simple performance tests: I would expect that the newer the version, the faster (in requests per second) it would be.

Because it should be, right?

The Test

I copied-and-pasted a simple hello-world script...

const http = require('http');

const hostname = '127.0.0.1';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

and then installed Apache Bench, or ab to hammer it:
ab -c500 -n100000 http://localhost:3000/

But how to get the different versions? Node version management is a pain on Linux, it's a nightmare on Macs, and I've not tried it on Windows but I can't imagine that it's fun.

Here's the actual useful bit of this, an app called n. Just install it via sudo npm install -g n and you've suddenly got a very powerful tool for switching node versions.

So, if you want the LTS version?

 % sudo n lts

  installing : node-v12.19.0
       mkdir : /usr/local/n/versions/node/12.19.0
       fetch : https://nodejs.org/dist/v12.19.0/node-v12.19.0-linux-x64.tar.xz
   installed : v12.19.0 (with npm 6.14.8)

Easy.

The Results

Each test was ran three times, because #science, on the latest version of each major version back to Node 5. No changes where made in the code, or the way the test was run. This was all ran from my 8th gen i5 laptop:

version     req/s  1         2          3
14.13.1      9732.52     9866.98     9894.88
13.14.0     11065.33    12051.36    12247.49
12.19.0      9261.99     9926.34     9869.66
11.15.0     12510.51    13466.22    13783.50
10.22.1     11740.82    11925.83    11727.70
 9.11.2     12702.04    13667.15    13632.01
 8.17.0     10570.54    11579.70    11914.19
 7.10.1     11062.46    11622.44    11650.05
 6.17.1      9236.92     9293.16     9593.82
 5.12.0      8991.05     9790.51     9560.93

Interesting. From version 5,6, and 7 there's an obvious increase in performance from this simple metric. After that, into version 8 it starts to flatten out, then something odd happens for 12 and 14.

What does this test show us?

It shows us, in this very limited set of hardware, for a non-real-world code example, what the requests per second would be.

That's about it.

From this, we can't tell anything, really, about the major differences between the versions, or what that version would be like to run our code in production.

So this is a pointless set of numbers?

No. Well, yes, but the reason I did this was to try to disprove my bias to the notion that 'latest is greatest'. To really test if your code is going to be benefit from a bump in version numbers, you're going too have to understand a couple of things:

What your code does, and how the new version changes that
What the performance of your code looks like currently
What, other than performance, are the reasons not to upgrade.

Most of the time, if you're moving up between minor and bug versions, security is your biggest reason to upgrade. If there is a bug that particularly affects you then that's also another great reason to move up also.

So what now?

I'm currently looking at correctly measuring the performance of our production codebases with tools such as Gatling, and tracing the code to find other bottlenecks and memory issues. Once we understand what the current numbers are, we can then use those to compare and contrast before future upgrades.

We have a policy of sticking to the LTS (Long Term Support) version for production code. This comes after a range of versions of node was being used in different projects in our stack, causing all kinds of problems testing, deploying, developing etc. The tool n has helped with quickly changing versions, but if you have a single policy for a version, it makes things a bit simpler.

Cover image was Image by jacqueline macou from Pixabay

Publishing your OpenAPI documents automatically with NodeJS and Express

Chris Williams — Mon, 13 Jul 2020 11:44:16 +0000

Writing that OpenAPI spec document is one thing, but how can you make sure it's viewable to everyone who needs it?

There are plenty of commercial tools for sharing API information around (https://www.postman.com/ is a great example), but what's a difficult thing to do if your API is for public consumption.

You could maintain a copy on your website somewhere? Make the OpenAPI spec available so people can copy-paste to https://editor.swagger.io/?

We've tried both and found that it's hard work to maintain an up-to-date version without doing some additional automation on deployment.

Training people to check the file from the repo, copy-paste it to the online editor.. yea that works as well as you would expect it too.

Why not just expose the OpenAPI document in a nice format directly on the API?

The code to do this is really, really simple.

We're going to use swagger-ui-express by Stephen Scott

    const express = require('express')
    const swaggerUi = require('swagger-ui-express')

    const router = new express.Router()
    const jsYaml = require('js-yaml')
    const fs = require('fs')

    // Our document is YAML, if yours is JSON, then you can just
    // `const openApiDocument = require('spec/openapi.json')`
    // instead.
    const openApiDocument = jsYaml.safeLoad(
        fs.readFileSync('spec/petstore.yaml', 'utf-8'),
    )

    // We can enable the explorer also!
    const options = { explorer: true }

    router.use('/api-docs', swaggerUi.serve)
    router.get('/api-docs', swaggerUi.setup(openApiDocument, 
        options))

    // If you're not using a `router`, you can use
    // `app.use('/api-docs', swaggerUi.serve,
    //    swaggerUi.setup(swaggerDocument, options));

    ....

And that's about it. Firing up our app and heading over to https://mycool.io/api-docs and you should see something like this:

Now, every time you deploy your API, the latest version of the OpenAPI document will be available for all to see.

Thanks, I hope this helps someone get their API documentation in front of the people who need it - I've also written an article on Handling API validation with OpenAPI (Swagger) documents in NodeJS and OpenAPI (Swagger) specifications that write your tests for you (sort of)

Header image by rawpixel

Using Postman's CLI tool for API testing: newman

Chris Williams — Sun, 13 Oct 2019 14:00:11 +0000

Image by Steve Buissinne from Pixabay

I need a tool to handle post-deployment testing properly

I've got a project where I'm not handing post-deployment testing with any real grace. It's been on the list of things to resolve, and I'm happy with it for the moment because of the pre-release tests, manual release testing, and monitoring post-release, but it does need solving.

I stumbled upon the newman cli tool from the good folks at getpostman.com. It's a CLI, open source tool that runs the tests you have saved in your Postman collections, and gives the typical error state output / console output you expect from any modern testing tool, which means you can integrate it into your CI/CD workflow.

For anyone who's not used Postman, it's a stunning tool for making requests to services, keeping collections of connections and such, and if you're doing almost any web-based development you need it. If you're too old-school and like using cURL for everything? Fine, it'll import and export cURL commands for you. Go check it out.

https://www.getpostman.com/

The only problem for me - I don't use Postman like that. I don't keep collections of things really, I just use it ad-hoc to test things or for a quick bit of debugging. We've got a nice collection of integration tests build around our OpenAPI specs that we rely on, so I've not had to do what others have done and create a large collection of API endpoints.

The trick here is going to be keeping the duplication down to a minimum.

Getting started: We're going to need an API to test against:

I've stored everything for this project, you can see all the files on GitHub

// src/index.js
const express = require('express')
const bodyParser = require('body-parser')
const addRequestId = require('express-request-id')

const app = express();
app.use(addRequestId())
app.use(bodyParser.json())

app.get('/', (req, res) => {
   res.json({message: 'hello world', requestId: req.id});
})

app.get('/foo', ({ id, query }, res, next) => {
    const { bar } = query
    res.json( { bar: `${bar}`, requestId: id })
})

app.post('/foo', ({ id, body }, res, next) => {
    const { bar } = body

    if (typeof bar === 'undefined' ) { 
        return res
        .status(400)
        .json({ error: 'missing `bar`', requestId: id})
    }

    res.json( { bar: `${bar}`, requestId: id } )
})

const server = app.listen(8081, function () {
   const port = server.address().port
   console.log("Example app listening to port %s", port)
})

So we've got three endpoints to use: / and /foo as GET, and /foo as POST. There's a little validation in the POST /foo endpoint. I've added express-request-id in and added it to the responses so we can make sure they're unique.

Starting the Collection

I'm learning this as I blog here, so forgive any backtracking! I've gone into postman and created a new collection called postman-newman-testing.

A nice start, an empty collection created

I went through and created and saved a request for each of the three endpoints, adding a little description for each:

Three requests added to the collection. If you have trouble with this, I recommend the very helpful documentation over at [the Postman Learning Center](https://learning.getpostman.com/)

Adding some tests:

Remembering the goal here is to create something that can help us run some post-deployment tests, so we're going to define some simple tests in the collection for each of the endpoints. I want to ensure:

I get a requestId back for each response
I get a 200 response for each
I can trigger a 500 response when I expect there to be an error
The expected values come back for the POST /foo and GET /foo endpoints

The documentation for the test scripts is all in the Postman Learning Center as you would expect, and thankfully it's going to be really familiar for anyone who's worked with tests and JS before.

So, after a little hacking around, I discovered something cool; when you make the tests, they're executed each time you execute that request, so if you're using Postman to dev with, you can't 'forget' to run the tests.

The four tests are executed each time, so we're testing the response code, shape of the response, that it's a JSON type returned, and that it's as fast as we expect it to be

Variations

I want to test two different outputs from an endpoint, success and failure, but I don't think I should have to save two different requests to do that, so how are we going to test our POST /foo endpoint? I'm going to come back to that at some point once I understand more.

Automate all the things

I've got my collection set up with all the happy-path tests, and if open the Collection Runner, and run my collection (..), then I get a nice board of green boxes telling me my API is, at a very basic level, doing what I expect it to be doing.

Everything is 5 by 5

Let's work out newman.

I've exported the collection from Postman and stored it under docs/postman-collection.json in the project root, installed newman ($ npm install --save-dev newman), and ran the command to run the tests:

There's all kinds of nice output there!

So that's amazing, I've made some simple API tests, but it's not going to do me any good for the simple reason, that in my collection all my URLs are set to http://localhost:8081, so I need to work out how to change that.

After a bit of clicking and Googling, we can do this. Postman has support for Environments - you can see them at the top-right of the main window. I created a couple ('development' and 'staging') and created a value called host in them with the http://localhost:8081 for development, and https://api.mydomain.com:3000 for the staging environment. These are a bit fiddly, the same as some of the other parts of Postman's UI, but it's possible ;)

It took too long to work this out

Next, we go into the Collection and change the host names in the saved requests to use {{host}} - the {{ }} method is how Postman handles environment variables, and could be used for things like API keys.

As you can see, I've changed the hostname to `{{host}}/` and I'm using the `development` environment, and all is well

So let's translate to the newman tool.

Ahh. Ok.

So exporting the Collection doesn't bring any of the environment variables with it. We have to export those as well.

I left the file naming scheme as it is

And now, we're going to want to use those environment configs with our newman execution:

Boom! Git controlled, command line executed testing for APIs in different environments, using a tool all devs should be using anyway for a simple post-deployment check. There's the obvious steps of adding this to your Jenkins / Gitlab / whatever pipeline which I'm not going to cover here, but I'm happy with what's been discovered over the last couple of hours.

One last thing, lets put this into the package.json file so we can re-use:

{
  "name": "postman-newman-testing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "config": {
    "environment": "development"
  },
  "scripts": {
    "debug": "nodemon src/index.js",
    "start": "node src/index.js",
    "test": "echo \"Error: no test specified\" && exit 1",
    "test-post-deploy": "newman run ./docs/postman-collection.json -e ./docs/$npm_package_config_environment.postman_environment.json"
  },
  "author": "Chris Williams <chris@imnotplayinganymore.com>",
  "license": "ISC",
  "dependencies": {
    "body-parser": "^1.19.0",
    "express": "^4.17.1",
    "express-request-id": "^1.4.1"
  },
  "devDependencies": {
    "newman": "^4.5.5",
    "nodemon": "^1.19.3"
  }
}

Then we can handle configs for environments as we like, and run

npm run test-post-deploy

to execute the test!

Conclusion

While it may be another set of tests and definitions to maintain (I would really like this to be based from our OpenAPI spec documents, but I'll figure that out later), this seems to be a great way of achieving two things:

A really simple set of tests to run post-deployment or as part of the monitoring tooling
The collection file can be handed out to devs working with the APIs: The're going to be using Postman (probably) anyway, so give them a head-start.

Postman is just one of those tools that you have to use if you're doing web-dev, or app-dev. Given that it's just 'part' of the dev toolkit, we may as well use the familiarity and use it as part of the testing tools as well.

There are some things that I would like to know a bit more about:

Be able to store the output in a file, maybe, so it's visible quickly in Jenkins
Set the severity of individual tests - so if we fail some it's an instant roll-back, if we fail some others it's a loud klaxon in the engineering office for someone to investigate, but it may be resolved by fixing forwards
Test for sad-paths , make sure that the right error response codes are coming back for things without having to create the responses for them: I think there's something you can do with the Collection Runner and a file of sample data, and then flag if tests should be red or green, but I didn't get around to that.

Thanks also to those who've replied to my on-going tweets about all things Postman over the last couple of hours, esp Danny Dainton, who also has his own Dev.to articles over at https://dev.to/dannydainton

Thanks again for the comments on previous articles, I would love to hear how you use this in your projects! Get me on https://twitter.com/Scampiuk

OpenAPI (Swagger) specifications that write your tests for you (sort of)

Chris Williams — Mon, 23 Sep 2019 22:27:59 +0000

I recently wrote the article Handling API validation with OpenAPI (Swagger) documents in NodeJS, which went into how to pass on the work of input validation to the OpenAPI spec. This follows on, showing how to lighten the testing load and ensure your API is producing exactly the output you’ve painstakingly documented.

Photo by Paul Esch-Laurent on Unsplash

Faster testing by relying on the OpenAPI spec as a single source of truth.

There is nothing, nothing, more predictable than API documentation being wrong.

It’s hard to keep that document up to date with all the other pressures of having to, you know, maintain the API. It’s simpler to push the fix or feature and then update the doc. Eventually.

I would be lying if I was looking for the solution to this exact problem, but I’ve discovered one as a by-product of two other things:

Using the API spec document for validation. We covered this in https://medium.com/@Scampiuk/handling-api-validation-with-openapi-swagger-documents-in-nodejs-1f09c133d4d2
Using the API spec document for testing. ( This guide )

This little duo means the API spec has to be up to date, else you can’t pass any of your tests. Nice, hu?

We’re going to start with as simple as a test application as possible:

npm install express

Let’s run it…

chris@chris-laptop:~/Projects/openapi-testing$ curl localhost:3000
{"version":"1.0.0"}

Ok so that’s simple and working, let’s create a spec that defines this, rather limited, API. Using OpenAPI 3 spec, we’ll be quite verbose in the way we build the objects up so we can re-use them in the future:

We can see that our GET / endpoint needs to return an object with the a property called version which has a pattern of ^\d.\d.\d$ , and it requires a header called X-Request-Id which is a UUID.

But our current endpoint doesn’t fulfill this criteria! We’ve created the thing we hate, the thing worst than no API documentation: bad API documentation. The solution? Tests.

npm install supertest chai mocha --save-dev

Once we have that installed, let’s create a nice simple test

Then in package.json , under the scripts block, add

"test": "./node\_modules/.bin/mocha --exit --timeout 10000"

This will run our test we’ve just created, exit once it’s done, have a sane time-out time.

We’ve expelled some effort to test this endpoint, but the tests are a false positive — we know the spec requires the X-Request-Id to be defined, and our test doesn’t cover that.

We’re going to look at the same tooling as we used in the previous guide, express-openapi-validate . This thing is going to ingest our spec file, and in the same way we used it previously to validate the input to an API, we’re going to use it to validate the output of the API.

npm install express-openapi-validate js-yaml app-root-path --save-dev

And now we’re going to change the index.spec.js around a bit, taking out the explicit definition of what we expect in the endpoint, and adding in the OpenApiValidator…

and run the test again…

There! It failed this time, and told us why it failed: response.headers should have required property "x-request-id"

Note we’ve not had to define that in the test: in fact we’ve taken out code for testing what shape the response is, it’s taken the spec and worked out what’s required for a GET / request. Let’s fix the endpoint.

npm install faker

( if you’ve not looked at faker before, I strongly recommend it, I’m abusing it here slightly but it’s a fantastic fake data generator))

We changed the response to set the X-Request-Id header with a UUID, and now the tests pass.

What happens if we break the format of version? We’ll change the request to send x1.0.0 instead, which doesn’t match the pattern for Version …

Tests fail, because you’re sending the wrong value.

This is crazy powerful. Now, because you’ve defined things in your spec file correctly, you can re-use patterns in your API and ensure that the spec is being fulfilled on your tests, while updating all your tests if you update the spec file. You write less lines in your tests, focus on putting the effort into the spec file (because that’s now driving your tests…) and things become simpler.

In conclusion

Using the OpenAPI spec to control how data gets into your API, and using it to build your tests around, means that it becomes the single source of truth about your API. Sure, there are ways of cheating this and not documenting all the objects, or not testing endpoints, but why do that?

By combining these two approaches, we’ve found that workflow on the API now starts with the OpenAPI specification, then building tests, then implementing the endpoints. TDD becomes almost the de facto way of approaching development. While before API development may have started by firing up Postman and thrashing through some ideas, now it’s all tested by this near-magic combination of supertest, mocha, chai, and OpenApiValidator.

There are a couple of things missing from this set-up which I’m still working on:

I would like code coverage reports via nyc to ensure that all the endpoints and response codes defined in the OpenAPI spec document are implemented
I would like the test validation to error if there are objects or properties in the API responses that are not documented — I just can’t work that one out at the moment.

I would love to hear how you use this in your projects! Get me on https://twitter.com/Scampiuk

Handling API validation with OpenAPI (Swagger) documents in NodeJS.

Chris Williams — Wed, 17 Apr 2019 20:45:07 +0000

Photo by Tim Gouw on Unsplash

This article was originally posted on my Medium blog.

I always found the hardest thing about API work was the documentation.

Sure, there are loads of nice tools out there to help you define it, provide nice front-ends and the like, but maintaining that isn’t anywhere nearly as fun as getting the actual work done. So soon enough, you’ve got stale documentation with little errors, and validation rules that don’t quite match up.

A recent NodeJS API project came my way which had out-of-date OpenAPI 3 documentation for the few endpoints it already had, but the understanding that we where going to start using it a lot more, so it needed to get up to scratch.

OpenAPI Specification (formerly Swagger Specification) is an API description format for REST APIs

I figured that if we where going to maintain this OpenAPI spec, which contained all the validation rules for the endpoints, then there must be a way we could use that to save us some time.

What if we could use that spec to enforce the validation? What if we could use it as the basis of the endpoint testing?

If we could get these two things, we have the wonderful combo of the OpenAPI spec needing to be write for the validation to work, and the validation being unable to deviate from the spec — so no more dodgy documentation where that param is documented as an int but it’s actually a float..

.. and if we can build tests based from the documentation then all our outputs have to be as defined, so the consumers of the API don’t get urked if we send an object and they’re expecting an array.

Using the OpenAPI spec to enforce the validation and be the crux of the tests enforces good definition of the API and removes all the nasty little ‘Ohh yea, that only returns X if Y’ that plagues API development IMHO.

So let’s stop waffling here and create something simple to prove how this works.

Randall Munroe — https://xkcd.com/1481/

First we’re going to spec our endpoint. To save some time, I’ve used one of the sample specifications as a base. There’s a very nice editor / visualization tool at https://editor.swagger.io/ to work with your spec files.

Here’s the sub-set of the specification we’re going to look at:

An endpoint that expects two variables in the path, {dataset} and {version} that are both strings. There are three possible variables in the post body also, one of which is required. It has two responses, a 200 that returns an array of records, and a 404. The response has a bunch of criteria also.

Let’s store this thing as /spec/api.spec.yaml

Now, a quick build of an Express app to handle responses to the path as documented:

This is as simple as it gets. So lets run it and check out if it works in Postman.

As expected, our API responds by telling us what we told it.

This all so-far-so-normal. Lets add the good stuff. Looking at the spec, we should now start adding validation into the endpoint we’ve just created — ensuring all those numbers are numbers, that the criteria is there etc.. But we don’t want to do that as we’ve already spent the time writing that all into the spec.

We’re going to install a node module called express-openapi-validate ( along with js-yaml) to handle this for us. Now we’ve got that installed, let’s change up the code a little:

Lost more going on here!

We’ve loaded the app.spec.yaml file and we’re creating an OpenApiValidator object with it, along with some interesting looking options. The options are all inherited from the parsing app ajv . Our two are :

allErrors: true, // makes it return all errors, not just the 1st
removeAdditional: "all", // Removes any undocumented params

We’re validating the spec against the request as middleware, where we’re telling it what method we’re looking for and the path, and we’ve added some error handling to give us something to display if it doesn’t all go according to plan.

Let’s mess up our request, and try again.

What, an API that returns with useful error messages?!

Okay! We’ve just added validation against the OpenAPI spec! It’s capturing the two things I broke: The removal of the required field criteria , and the incorrect type of .body.rows . It’s not a very graceful error message, but it’s telling the consumer what’s gone wrong, and you’ve not had to write any of it. It’s also returning the right status code of 400 for us. Sweet!

Let’s fix the request and try once again.

Much better, but something’s missing…

All looks as before.. but it’s stripped out the foo: "bar" from the body, because it wasn’t documented. The validator stripped it out because it was undocumented. No more sneaking in properties in post bodies and not telling anyone.

This means now, if you format your OpenAPI spec correctly, the data arrives to your code validated and correct. Now, I’m not saying it’s perfect — there is a known problem with trying to parse numbers in the path, and Express handles everything as a string, but it’s much faster than having to maintain the OpenAPI spec document -and- the validation on the endpoint.

I’m hoping that gives you enough grounding in how to approach this so you can start using your OpenAPI spec documents as the amazing resource that it is. Treat the spec well and it’ll not only provide you with documentation for the consumers of the API, but will also do a lot of the work for you.

The next thing to look at, which I’ll link to once I write it, is the other side of this, which is writing tests that ensure that the output of your API conforms to the OpenAPI spec — thus forcing you to write API responses that your consumers expect!

I would love to hear how you use this in your projects! Get me on https://twitter.com/Scampiuk

Mining for Crypto in the browser is not evil.

Chris Williams — Fri, 29 Sep 2017 21:54:12 +0000

The old argument is that the content on the internet needs paying for. Long gone are the days where everything is created by enthusiasts and people messing around in their spare time. Today’s content and platforms are high-quality and expensive to produce, and at the moment we’ve really only got three models to pay for it:

Subscription or pay-per-content. You want it? Open your wallet — subscription models are becoming more popular for the high-end content such as video and music, and that’s great. But it doesn’t really trickle down to things that you don’t really think you should be paying for, stuff that doesn’t have that real, tangible sense of value. Sure I’ll subscribe to Google Play Music, Amazon Prime, and Strava, but would I pay per month to use Twitter? No.
Ads and marketing You’re the product that’s being sold here, have no illusions. Hope you like ads. You’re targeted within 10 feet of your location and everyone knows what mood you’re currently in, and if you’re planning to buy a toaster or a coffee in the next ten minutes. Feeling a little grubby?
Donations. That thing you’ll get around to next time.

So, now there’s the idea of a fourth option. A bit of code runs in the website, extension, or app that you’re making good use of, and quietly and unobtrusively uses some of your CPU cycles to make a bit of money for it’s creator. You get the content ad-free, they get paid — this is what coin mining in browser could lead to.

Now, there’s an option to have a business model without advertising in mind. A model that isn’t trying to squeeze every little last drop of revenue out of you, because the money is being earned by you being on the site. The miner doesn’t care who you are or what shopping mood you’re in, it just get’s on and mines a tiny fraction of a given crypto-currency.

Is that so bad? Apparently people think so.

This is from two genuine reviews of an extension we tried this over the week. These got posted once it was published on Redditthat it contained the miner — something that we had put in the description of the extension from the very beginning.

No addon is worth giving the author CPU time and electricity. That’s what a coin miner takes from you.

No addon? I use a few that make my workday easier because of what their authors have created, and I try to donate or pay when I can. What if I didn’t have to, because every hour I had their tool open in the browser, it earned them money? What about websites, does the same apply here if they can offer me an advert-free, tracking free experience?

Here’s another one

he’s taking your electricity and bill so he can pay his.

Is that any difference between someone taking up your screen space to show you adverts? That just about defines all brand-awareness advert campaigns.

I’m not going to say here that this technology won’t have it’s problems. People will abuse it, and because of this others will distrust it. I personally learned about it after reading what The Pirate Bay did a weekend ago. The reaction to that where news articles that included the words ‘hidden’ and ‘secretly’, not a great first impression made to the world.

But, I’m not going to say that this is a bad technology ether. The idea of an advert free web, and a more anonymous web because of that, is something that we should be excited about.

Let’s not stamp on ways that content creators can earn enough to pay the bills.