DEV Community: You Know, For Devs

What I Learnt From Reviewing 22 CVs

You Know, For Devs — Sat, 04 Jul 2020 14:08:00 +0000

I was recently asked to look over some CVs. It’s a while since I’ve done this, so I thought that noting down my immediate reactions as I went through the CVs might help anyone having to circulate their own CV at the moment.

I’m not for a moment saying I’m right or wrong in the following views, and this is definitely not one of those posts about ‘how to write your CV to land your dream job’. I’m merely laying out the responses I had and more importantly what I felt these reponses told me about the person behind the CV. My ‘gut reaction’ may well be similar to others’ and if they are, it might be worth bearing them in mind when you share your CV.

Looks Count More Than I Thought

First, it wasn’t until I reached the ninth CV that I realised how much a CV’s appearance makes a difference. When viewing numbers one to eight I hadn’t been thinking ‘these look terrible’, but when I reached the ninth, a part of my brain said: ‘Finally! A CV that looks good.’

I think if the role was for a C++ programmer then I probably would have simply smiled to myself, ignored my response, and moved on. That’s not to say that appearance doesn’t matter–or that C++ programmers don’t worry about their appearance–but if I was trying to decide between two C++ programmers it wouldn’t be the look of their CV that was the deciding factor.

But the role for which I was reviewing this batch of CVs was a React developer and there are lots of very talented React developers out there. And when lots of people have worked with GraphQL and Docker and cloud services and D3 then it may well come down to whether or not you have made an effort to convey that you like things to ‘look nice’.

It’s worth saying too that the CVs that looked good didn’t go overboard; all they did was have a pleasing font, perhaps two columns instead of one for the layout, maybe a couple of icons and some colour, and probably didn’t span more than a couple of pages. But that was enough to convince me that these were people who would ensure that a bad-looking product didn’t get deployed.

Attention to Detail

Probably in the same vein, typos really jump out at you when you are looking at a lot of CVs. Again I’m not saying I’m right or wrong on whether this should be an important criteria, but the fact that my subconcscious reaction was to be turned off if I saw a spelling mistake–or a technology name that had been capitalised incorrectly–shows that somewhere in our psyches this stuff counts for something.

As with the previous point about looks, I’d probably see this as more relevant for front-end people.

And it might be worth saying that it’s usually not difficult to differentiate between those who have just not bothered to check their CV and those that have difficulty with spelling. I’ve worked with people in the latter camp, both senior managers and junior programmers, and it’s not a big deal–you can work with it and it would not stop me from hiring someone.

But to keep with the theme of my ‘initial reaction’, I can’t help that when I see someone write ‘Graphql’ I’m going to doubt whether the applicant really has that much experience with ‘GraphQL’; maybe it’s worth just double-checking the website of the technologies you use one last time, before sending off that CV.

Approaches v. Technologies

One other thing that I hadn’t expected was that I was drawn to mark people more highly if they gave some indication of the approach they took to their coding and development. For example, a few people mentioned having experience with TDD, which I thought was good. That doesn’t mean I’d rule out someone who didn’t have TDD experience, but it’s an approach that people need to be aware of.

However, more importantly than TDD, only a handful indicated that they liked to get involved in code reviews and saw them as a good way to share knowledge and build a consistent and high quality codebase. Again, I would not rule out someone who didn’t mention code reviews, mentoring, and so on, provided they were relatively inexperienced; but I probably would rule out someone who clearly has experience, but hasn’t thought to mention these things.

Approach From Technologies

What I was also surprised by was how the list of technologies on a person’s CV led to me forming an impression of the kind of coder that person was. If I saw GraphQL in the mix then I found myself thinking ‘modern tech stack’. It may be unfair, especially since GraphQL wasn’t mentioned in the job description that was circulated; but after reviewing all of the CVs I found that it was a pretty accurate indicator of the types of projects someone had been involved in.

There were other technologies that were similarly indicative, such as mention of ‘microservices’, or use of MongoDB, Elasticsearch, Redis, RabbitMQ, and so on. But there were very few CVs where these technologies appeared without GraphQL.

There is probably no general rule here, since the technologies that differentiate will be different in each cohort of CVs; if every CV included GraphQL then it may have been the presence of something like Docker and/or Kubernetes that told you something about the kind of people you were looking at.

Public Profile

This last point may also be controversial, but I’m going to describe it so that you can do whatever you like with the information; if someone was not on LinkedIn or didn’t have a public Git account, I found myself thinking ‘well, what exactly do they do?’

From this batch of 22 CVs, only one had answered any questions on Stack Overflow, and only one–a different person–had done any talks at Meetups. A couple had GitHub accounts, a couple were on LinkedIn and a couple had blogs.

This is no doubt unfair, so again, blame my subconscious. But I can’t help think that if I’m going to work with some people day after day, I want to be with people that are sharing links to things they’ve read, people who are trying out new technologies and languages, people who have weekend projects, or who are contributing bug fixes back to open source projects.

And although I’m not particularly a big fan of LinkedIn, I always quite like it when I see people who are at the early stages of their career making connections with everyone they met at last night’s Meetup.

None of this is to ignore the current discussion about work/life balance–to demand that people do their own education in their own time, spend all waking hours coding, attend Meetups every night of the week. But it is to recognise that we are in a fast-moving industry, and if you want your CV to be picked from amongst that batch of 22, then it’s going to be important to think about how you come across.

Qualifications

One final thing has just occurred to me…I didn’t consider the qualifications of anyone, which I imagine is down to the role that the CVs were for. You’d certainly be interested in degrees and PhDs if you were hiring a data scientist or someone to write an operating system; in these spaces theory is a major element. But for fast-moving technologies like front-end development, you can probably tell everything about a person from their experience.

Conclusion

I haven’t had to look at CVs for quite a while, so it was interesting to examine my own responses to reviewing this batch. What became clear is that I quickly ‘merged’ the CV and the person in my mind. If there were mistakes in the CV I couldn’t help but imagine someone who lacked attention to detail. If there was a little bit of colour in the CV or the person had spoken at a Meetup then I couldn’t help but think that this was a person who was prepared to go the extra mile. And if the person had a public Git account then my spontaneous reaction was that this was someone who wanted to keep learning, and really enjoyed the work they do.

I may be wrong in my judgements or harsh in my conclusions, but if you’re about to send your CV out, at least you now have a couple more insights that might help you decide how your CV is representing you.

Let TypeScript Inference Take The Strain

You Know, For Devs — Sat, 27 Jun 2020 15:13:00 +0000

I was doing a code review recently and saw something like this:

const mapEventName = (eventName: string): string => {
  switch (eventName) {
    case 'UPDATE_ABC':
      return 'ABC Updated'
    case 'UPDATE_DEF':
      return 'DEF Updated'
    default:
      return eventName
  }
}

Putting aside whether this should use a Map or some other technique instead of switch, I was interested specifically in the type information that was provided for the return value.

Something I see a lot when people adopt TypeScript is to type everything. I think it’s a good idea to add types to structures and interfaces that are passing between layers of our code. But to add extra code to indicate that a function returns a string, when it so obviously returns a string is overkill.

So what’s the alternative?

Well in this example there is no need for the return type to be specified. TypeScript has impressive powers of inference and in this function it will have no trouble deducing that the type of the return value will be a string literal. If we look at the function there are three places where a return value is specified; two of them stipulate a string constant to return, and the third says to return the parameter that was passed in–which is itself defined as a string.

Together these conditions combine such that there is no possibility that this function will return anything other than a string. TypeScript will therefore act accordingly. To verify this, take a look at the IntelliSense provided by your favourite editor. Here is the IntelliSense with the return type specified explicitly:

and here is exactly the same IntelliSense when the return type is omitted:

And if we want to reassure ourselves that TypeScript is still doing the same checking, let’s try to use the function in a position where a number is required:

const a = Math.max(mapEventName('UPDATE_ABC'), 4);

Put this in a TypeScript-enabled editor and you’ll see there is a type error, regardless of the fact that the return type is not explicitly specified:

You might ask whether missing off one declaration gains us much, but I would suggest it’s a mindset. Rather than approaching TypeScript as this rigid straightjacket that must be wrapped around all of our code, we instead see it as a flexible combination of two things:

a supercharged linter that can infer types from our code and then check that we’re not doing anything dumb, and;
a type language that we can add as we need to, to help to resolve ambiguities.

This gives us the option of a much ‘lighter touch’ approach to TypeScript.

For more on TypeScript’s type inference see Type Inference. And for a particularly clever scenario where types can be inferred from assert() statements, see ‘asserts condition’.

Why You Need a Pipeline Runner For Your Node Streams

You Know, For Devs — Sat, 20 Apr 2019 09:12:00 +0000

Basic Node Streams

A basic Node streams application would look like this:

const fs = require('fs')
const zlib = require('zlib')

fs.createReadStream(inputPath)
.pipe(zlib.createGzip())
.pipe(fs.createWriteStream(outputPath))

The key thing is that each stream provides a pipe() method, which is used to bolt on additional streams.

However, although this is correct, it doesn’t lend itself to easy manipulation; in particular we have to start with one stream and bolt other streams on to it, which throws out the symmetry and makes the whole process difficult to reason about.

A Better Model: A Pipeline of Streams

A better metaphor is of a pipeline that comprises multiple steps, each of which is a stream. In this model there is nothing ‘special’ about the first or last streams. Since version 10, Node has provided the module function stream.pipeline() that does exactly this. This function can be promisified, which makes the code even easier to read. Our previous pipeline would now look like this:

const stream = require('stream')
const util = require('util')
const pipeline = util.promisify(stream.pipeline)

const fs = require('fs')
const zlib = require('zlib')

await pipeline(
  fs.createReadStream(inputPath),
  zlib.createGzip(),
  fs.createWriteStream(outputPath)
)

Now it’s much clearer that our streams are ‘steps’ in a pipeline, and they no longer play the role of driving the pipeline.

We’ll use this pattern as our canonical pipeline because it is much easier to augment.

Separating Pipeline Creation From Running

Now that we have the idea of a pipeline as our primary concept, we can start to play with it. In particular we’ll find it extremely useful if we can separate the step of creating a pipeline from the step of running it.

We could then, for example, create a pipeline and check it for errors, even before it is run; we could check that the first step is a readable stream and the last step a writable one, for example, or ensure that all necessary resources such as databases or search engines are available before the pipeline starts.

We could also insert extra steps that wouldn’t need to be specified explicitly in the pipeline, such as pushing and popping items to and from a queue between each step, or adding handlers for step-specific errors.

Another benefit of separating pipeline creation from execution is that we could optimise the pipeline before it’s run, perhaps by merging or reordering steps. We might even create the pipeline on one machine, and send the pipeline to be run on another machine…or multiple machines if we have a way to split the work.

And not only could we do interesting things with the full pipeline, but we could even manipulate parts of the pipeline; we could run half of the pipeline, then materialise the results, and later run the rest of the pipeline using the saved results (incidentally allowing a pipeline to be upgraded, even whilst running), or we could execute individual pipeline steps as serverless functions, with all of the advantages that that would bring.

To get ourselves to a situation where these kinds of things are possible our next step is to separate the pipeline itself from its runner.

Creating a Runner

The basic form of a runner is to first create some steps:

const steps = [
  fs.createReadStream(inputPath),
  zlib.createGzip(),
  fs.createWriteStream(outputPath)
]

and then to run those steps:

await pipeline(...steps)

If we wrap this functionality in a class then we have a foundation on which to add the other features we mentioned earlier. So let’s create a class that allows us to register a set of steps with one method, and to run those steps with another method:

class Runner {
  constructor() {
    const stream = require('stream')
    const util = require('util')
    this.pipeline = util.promisify(stream.pipeline)
  }

  register(steps) {
    this.steps = steps
    return this
  }

  run() {
    return this.pipeline(...this.steps)
  }
}

Now we can use our runner like this:

const runner = new Runner()
const steps = [
  fs.createReadStream(inputPath),
  zlib.createGzip(),
  fs.createWriteStream(outputPath)
]

await runner
.register(steps)
.run()

It’s exactly the same as our previous pipeline but with this structure we’ve got lots of points at which we can insert the new kinds of functionality that we mentioned earlier, by inheriting from the base class.

For example, we could add more features by way of the register() method, such as pipeline validation, step optimisation, insertion of additional steps, and so on. (And manipulation of the pipeline is as simple as manipulating an array.)

And we could modify the run() method to send the steps on to some other server, or launch more workers, sharing the work between them.

In future posts we’ll start to flesh out how to achieve some of these more advanced features by building on this class.

Conclusion

Node streams are incredibly powerful, and are a fundamental building-block of efficient and fast data-processing and transformation applications. However, the usual model of an input stream that pipes its output to another stream is quite difficult to manipulate and reason about. Node supports a better model by way of the pipeline() function, and this post shows how it can be used to construct pipelines that comprise multiple streams, and then to separately execute those pipelines.

Getting Control Of Your .dockerignore Files

You Know, For Devs — Fri, 07 Dec 2018 09:04:01 +0000

Every now and then I’ll notice a file inside a Docker container that really shouldn’t be there. Thankfully it’s never been a .env file or an SSH key, but even so, any unused file or directory takes up space in the image and makes that image slower to build and pass around. Best practice suggests using the oft-neglected .dockerignore file to keep our secrets secret and make sure our Docker images are as lean as possible. But this file then usually becomes a maintenance nightmare as the list of exclusions grows.

This post shows how to control .dockerignore by indicating which files we want to include rather than those we want to exclude.

The Goal

For a while now I’ve been trying to come up with an easy way to exclude unecessary files from my Docker images. Obviously I want to keep secrets out, and I want to prevent any unused directories from being transferred to the Docker daemon when building–they slow down the build as well as increase the size of the resulting images.

But I also wanted to find a way to exclude files that relate to the build process itself, like Dockerfile and docker-compose.yml; it’s unnecessary clutter if they get transferred to an image and it would be better if they weren’t included at all. This is even more important should you want to share an image through public channels like Docker Hub while keeping the source that builds that image internal to your organisation; in this case there may be private information in these build files and you certainly won’t want them to be shared.

One last requirement is that when we make a mistake with our ignore rules we want to know about it. The way .dockerignore is normally used, if you mistakenly put something like ‘git’ into your file as a pattern, instead of ‘.git’, you won’t know anything about it unless you spot that a lot of data is being transferred to your build.

The rest of this post looks at how we might solve these problems.

Approach #1: Build From A Single Directory

Most of the approaches I have tried in the last year or so have revolved around different ways of organising the project directories. There are lots of ways a project could be organised, but in general, the approach is to try to keep the source, tests, documentation, generated files, and anything else that is part of the project, separate from each other.

As part of this separation, we also want to ensure that any directories that we don’t want to appear inside the Docker image get set as peers of the main directory that we want to include. This simply means that rather than having app > docs or app > test as our directories, we would instead have app, docs and test as top-level directories sitting alongside each other.

By having one directory that contains all of the files that will be needed in a build, and everything else kept safely apart in other directories, we at least start to lower the risk of mistakenly copying something important or unnecessary.

Setting the Context

Once our directories are organised then it’s a simple matter to set the Docker build context to refer to the single directory that should be included in the Docker image. By setting the context to a sub-directory we have ensured that all other directories are excluded, whether they are .git, node_modules, documentation and tests, and so on.

However, the weakness with this approach is that since the build context must contain everything that Docker will need to build the image, that, unfortunately, means we need to place Dockerfile in our source directory too.

Ignoring With `.dockerignore`

It’s easy enough to stop the Dockerfile from being copied into the image by creating a .dockerignore file and then adding ‘Dockerfile’ to it. The .dockerignore file is an ‘ignore file’ which tells the build process which files to leave out when transferring the context to the Docker daemon. For this situation it could be as simple as this:

# In .dockerignore
Dockerfile

Don’t worry that this could prevent the whole build process from working. In the case of Dockerfile, Docker will still transfer it to the daemon to guide the build process, regardless of whether we exclude it or not. But by adding it to our ignore file we stop it going any further through the process; i.e., it won’t be available to be copied into the image.

Of course, if we add a .dockerignore file to our source directory then that will be available to the Docker daemon, so we now need to ignore that as well:

# In .dockerignore
.dockerignore
Dockerfile

Problems With Approach #1

Although this is a nice simple solution, there’s still nothing to say that we might not accidentally copy something into the image that we didn’t want to. For example, when developing code on our laptop we’ll probably have a .env file full of database passwords and server names in the source directory. If we forget to exclude the .env file as well then it will most likely be copied into our image.

I also prefer not to leave the Docker-related files in the same directory as the source code since it is mixing layers–the source file of our application is now sat alongside the instructions to build our application. It’s all a bit ‘meta’.

Approach #2: A Richer `.dockerignore` File

So the next step would be to keep the build files out of the main directory, moving them to the root of the project. If we do that then we must make the root of the project the ‘build context’, and that means we’re now exposing all of our files to Docker. To exclude the peer directories we mentioned earlier we would modify the ignore file further, like this:

# In .dockerignore
.dockerignore
Dockerfile
dist
docs
test

If the code we want to put in our image is in the app directory then this .dockerignore file will allow that directory to be copied, but will exclude the dist, docs and test directories.

Problems With Approach #2

But whilst we’re excluding tests and documentation, we should also be excluding our .git directory (to keep the size down) and other build files, like docker-compose.yml and .gitlab-ci.yml (to keep things private). Now our ignore file looks like this:

# In .dockerignore
.git
.gitlab-ci.yml
.dockerignore
dist
Dockerfile
docker-compose.yml
docs
test

This approach of continually growing the ignore file works and you’ll find many blog posts that describe exactly what we’re doing here. But it soon gets very, very tedious to keep excluding all of the files and directories that are not needed in the image, and after a while, you’ll find yourself turning a blind eye to any ‘harmless’ files that are exposed or that get inadvertently copied in.

But that means we’re back to square one again because we can never really be sure that we haven’t copied in something that shouldn’t be there, or that the image we’ve created couldn’t be smaller.

Luckily it turns out that there is a much easier way to maintain these exclusions, and it’s pretty much foolproof.

Approach #3: Define Inclusions, Not Exclusions

As we all know, the .dockerignore file contains a list of file patterns to exclude. But it doesn’t just contain that. The ignore file also contains a list of file patterns to not exclude.

Say we have a blog full of Markdown files that we want to exclude from our image because they will be converted to HTML beforehand and are therefore unnecessary at runtime. But let’s say also that we still want to include the README.md file from our project because it might help anyone looking inside a running container. We could achieve this goal with the following entries in a .dockerignore file:

# In .dockerignore
# Exclude all Markdown files:
#
*.md

# Now make an exception for one file:
#
!README.md

With this file, when Docker starts the build process the only Markdown file that it will pass to the daemon is the README.md file.

Exclude Everything

So why don’t we use this feature to our advantage, and begin every .dockerignore file with a statement to exclude everything? If we start by saying that nothing will be sent to the Docker daemon when building an image, it will be almost impossible for us to accidentally include anything that’s secret or unecessary.

The pattern that we need in the .dockerignore file to ignore everything is simply an asterisk:

# In .dockerignore
# Exclude everything:
#
*

Now of course, if we exclude everything when we’re building the image then it won’t just be our secrets that don’t get copied in…nothing will! It’s like being certain of not losing a race by not entering at all.

So now we need to expand our .dockerignore file with information about the files that we do want to be included.

And those files should only be those that are mentioned in our Dockerfile.

Only Include Those Files Referenced In `Dockerfile`

Let’s assume that we’re building a Node app. And let’s also assume that we’re following best practice in our Dockerfile by installing all of the dependencies in one layer and our source code in another layer.

The ‘install dependencies’ step might look like this:

# In Dockerfile
# Copy dependency definitions and then install:
#
COPY package.json .
ENV NODE_ENV=production
RUN npm install

while the ‘copy source’ step could be as simple as this:

# In Dockerfile
# Copy our app:
#
COPY app/ .

When building the Docker image with this Dockerfile the Docker daemon must have access to both the package.json file and the app sub-directory. And since we’ve excluded everything at the top of the ignore file, every file or directory referred to in the build process must be explicitly allowed through.

We can easily add these files to our .dockerignore file with a ‘don’t exclude’ pattern:

# In .dockerignore
# Exclude everything:
#
*

# Now un-exclude package.json and the app folder:
#
!app
!package.json

That’s it…we’re done. The Docker daemon won’t receive .git or test directories, or .env, or .gitlab-ci.yml, or Dockerfile, or docker-compose.yml or anything else.

Now all we have to do is keep the files .dockerignore and Dockerfile in sync–i.e., ensuring that they both refer to the same paths and files–and nothing else will accidentally get into our Docker images.

One of the many neat features of this approach is that if we forget to add a rule we’ll hear about it. If we indicate in our Dockerfile that we want to copy in the README or the LICENSE:

# In Dockerfile
# Copy our app and the license:
#
COPY app/ .
COPY LICENSE .

we’ll get an error at build time unless we add LICENSE to the .dockerignore file:

# In .dockerignore
# Exclude everything:
#
*

# Now un-exclude package.json and the app folder:
#
!app
!LICENSE
!package.json

In fact, the patterns that we need in our .dockerignore file to specify what we want to include are simply the ADD and COPY entries in our Dockerfile, which is much simpler to manage than trying to keep track of what needs to be excluded.

Problems With Approach #3

Although this is my preferred solution and solves a lot of problems, there are still a couple of weaknesses. Although we’ll get notified by the build process if we are missing an important rule from our .dockerignore file, we won’t be notified if our rules are too liberal. So in our example above, if we later decide not to copy the LICENSE file when building, there is nothing to tell us that we can remove the pattern from .dockerignore.

And also, we still need to remember not to leave anything in the app directory that might get copied in. As it happens, the example I gave earlier of a .env file (in Approach #1) is now less of an issue since it is best kept in the root directory anyway.

Conclusion

I’ve tried a few different solutions to this problem over the years, but I think this ‘include file’ approach is about as minimal and easy to maintain as it gets. If the notion of ‘exclude everything and then un-exclude the files you want’ seems a bit quirky just flip things on their head and treat the .dockerignore file as if it was an ‘include’ file–in other words see it as ‘only the files listed will be included in the build’.

I think I’d prefer it if this was the default behaviour from Docker, because then we would get both security and minimal image size as the default behaviour, right out of the box. But unless a .dockerinclude feature is added to Docker any time soon, then the solution described here is about as good as it will get.

Using _writev() to Create a Fast, Writable Stream for Elasticsearch

You Know, For Devs — Mon, 29 Oct 2018 11:37:01 +0000

We all know how great Node streams are. But it wasn’t until I recently needed to create (yet another) writable stream wrapper for Elasticsearch that I realised just how much work the streaming APIs can do for you. And in particular how powerful the _writev() method is.

I was looking to wrap the Elasticsearch client in a writable stream so that I could use it in a streaming pipeline. I’ve done this many times before, in many different contexts–such as creating Elasticsearch modules to be used with Gulp and Vinyl–so I was all set to follow the usual pattern:

my first step would be to set up an Elasticsearch client, using the Elasticsearch API;
I’d then add a function that gets called with whatever entry should be written to the Elasticsearch server;
to speed writing up I wouldn’t write this entry straight to the server, but instead buffer each of the entries in an array (the size of which would of course be configurable). Then, once the buffer was full the entries would be written en masse to the Elasticsearch server using the bulk update API (which is much, much faster than writing records one at a time);
when the source of the data for the writable stream indicates that there is no more data to send I’d check whether there is any data still in the buffer, and if so call a ‘flush’ function;
and once all data is flushed, I’d delete the client.

None of this will probably surprise you, and you’d no doubt write an interface to Elasticsearch in much the same way yourself.

But what might surprise you–especially if you haven’t looked at Node’s Writable Streams for a while–is how many of these steps could be done for you by the Node libraries.

To kick things off, let’s create a class that extends the Node stream Writable class:

const stream = require('stream')

class ElasticsearchWritableStream extends stream.Writable {
}

module.exports = ElasticsearchWritableStream

Now we can start adding each of the features in our list.

Creating an Elasticsearch Client

The first step we described above was to create an Elasticsearch client, using the Elasticsearch API, so let’s add that to the constructor of our class:

const stream = require('stream')
const elasticsearch = require('elasticsearch')

class ElasticsearchWritableStream extends stream.Writable {
  constructor(config) {
    super()
    this.config = config

    /**
     * Create the Elasticsearch client:
     */

    this.client = new elasticsearch.Client({
      host: this.config.host
    })
  }
}

module.exports = ElasticsearchWritableStream

We can now call our class with some configuration, and we’ll have a writable stream with an Elasticsearch client:

const sink = new ElasticsearchWriteStream({ host: 'es:9200' })

Of course, this stream doesn’t do anything yet, so let’s add the method that the streaming infrastructure will call whenever some other stream wants to write a record.

Writing Records

When implementing a writable stream class, the only method we need to provide is _write() which is called whenever new data is available from the stream that is providing that data. In the case of our Elasticsearch stream, to forward the record on we only need to call index() on the client that we created in the constructor:

class ElasticsearchWritableStream extends stream.Writable {
  constructor(config) {
    ...
  }

  /**
   * When writing a single record, we use the index() method of
   * the ES API:
   */

  async _write(body, enc, next) {

    /**
     * Push the object to ES and indicate that we are ready for the next one.
     * Be sure to propagate any errors:
     */

    try {
      await this.client.index({
        index: this.config.index,
        type: this.config.type,
        body
      })
      next()
    } catch(err) {
      next(err)
    }
  }
}

Note that once we’ve successfully written our record we then call next() to indicate to the streaming infrastructure that we’re happy to receive more records, i.e., more calls to _write(). In fact, if we don’t call next() we won’t receive any more data.

Index and Type

When writing to Elasticsearch we need to provide the name of an index and a type for the document, so we’ve added those to the config that was provided to the constructor, and we can then pass these values on to the call to index(). We’ll now need to invoke our stream with something like this:

const sink = new ElasticsearchWriteStream({
  host: 'es:9200',
  index: 'my-index',
  type: 'my-type'
})

Buffering

As things stand, we already have a working writable stream for Elasticsearch. However, if we’re planning to insert hundreds of thousands of records then it will be slow, and a simple optimisation would be to buffer the records and use the bulk update API.

Bulk Update API

The bulk update API allows us to perform many operations at the same time, perhaps inserting thousands of records in one go. Rather than defining each record to be inserted as we did with the index() call, we need to create a list that contains pairs of entries; one that indicates the operation to carry out–such as an insert or update–and one that contains the data for the operation.

Using an Array

The usual ‘go to’ implementation here would be to create an array in the class constructor, and then push the rows of data into that array with each call to _write(). Then, when the array is full, construct a call to the bulk API, still within the _write() method.

The problem here though, is that in order to properly implement backpressure we need quite a sophisticated interaction with the next() function; we need to allow data to flow to our stream as long as the buffer is not full, and we need to prevent new data arriving until we’ve had a chance to write the records to Elasticsearch.

It turns out that the Node streaming API can manage the buffer and the backpressure for us.

_writev()

Although the bare minimum we need to provide in our writable stream class is a _write() method, there is another method we can create if we like, called _writev(). Where the first function is called once per record, the second is called with a list of records. In a sense, the streaming API is doing the whole create an array and store the items until the array is full and then send them on bit for us.

Here’s what our _writev() method would look like:

class ElasticsearchWritableStream extends stream.Writable {
  ...

  async _writev(chunks, next) {
    const body = chunks
    .map(chunk => chunk.chunk)
    .reduce((arr, obj) => {
      /**
       * Each entry to the bulk API comprises an instruction (like 'index'
       * or 'delete') on one line, and then some data on the next line:
       */

      arr.push({ index: { } })
      arr.push(obj)
      return arr
    }, [])

    /**
     * Push the array of actions to ES and indicate that we are ready
     * for more data. Be sure to propagate any errors:
     */

    try {
      await this.client.bulk({
        index: this.config.index,
        type: this.config.type,
        body
      })
      next()
    } catch(err) {
      next(err)
    }
  }
}

The streaming API will buffer records and then at a certain point hand them all over to our _writev() function. This gives us the main benefit of buffering data–that we can then use the bulk update API–without actually having to create and manage a buffer, or look after backpressure.

Buffer Size

If we’d created the buffer ourselves we’d have had complete control over how big the buffer is, but can we still control the buffer size if the Node streaming API is managing the buffer for us?

It turns out we can, by using the generic highWaterMark feature, which is used throughout the streams API to indicate how large buffers should be.

The best way to implement this in our writable stream is to have two parameters for our constructor:

one which will provide configuration for the Elasticsearch connection, such as server address, timeout configuration, the name of the index and type, and so on;
another which provides settings for the writable stream itself, such as highWaterMark.

This is easily added, like so:

class ElasticsearchWritableStream extends stream.Writable {
  constructor(config, options) {
    super(options)
    this.config = config

    /**
     * Create the Elasticsearch client:
     */

    this.client = new elasticsearch.Client({
      host: this.config.host
    })
  }

  ...
}

And now we can control the size of the buffer–and hence, the number of records that are being written by each call to the bulk API–by setting options in the constructor:

const esConfig = {
  host: 'es:9200',
  index: 'my-index',
  type: 'my-type'
}
const sink = new ElasticsearchWriteStream(
  esConfig,
  { highWatermark: 1000 }
)

Closing the Elasticsearch Client

All that remains from our original checklist is to close the client when there is no more data to receive. To implement this, all we need to do is to add another optional method, _destroy(). This is called by the streaming infrastructure when there is no more data, and would look something like this:

_destroy() {
  return this.client.close()
}

Conclusion

As you can see, the Node streaming API has done much of the work of buffering, for us, which means that we don’t get bogged down with trying to implement backpressure properly. By providing us with the methods _write(), _writev() and _destroy() our code ends up very clean, and focuses our attention on only the parts required to spin up and destroy a connection to Elasticsearch, and the functions required to write a single record, or a batch. The full implementation looks like this:

const stream = require('stream')
const elasticsearch = require('elasticsearch')

class ElasticsearchWritableStream extends stream.Writable {
  constructor(config, options) {
    super(options)
    this.config = config

    /**
     * Create the Elasticsearch client:
     */

    this.client = new elasticsearch.Client({
      host: this.config.host
    })
  }

  _destroy() {
    return this.client.close()
  }

  /**
   * When writing a single record, we use the index() method of
   * the ES API:
   */

  async _write(body, enc, next) {

    /**
     * Push the object to ES and indicate that we are ready for the next one.
     * Be sure to propagate any errors:
     */

    try {
      await this.client.index({
        index: this.config.index,
        type: this.config.type,
        body
      })
      next()
    } catch(err) {
      next(err)
    }
  }

  async _writev(chunks, next) {
    const body = chunks
    .map(chunk => chunk.chunk)
    .reduce((arr, obj) => {
      /**
       * Each entry to the bulk API comprises an instruction (like 'index'
       * or 'delete') and some data:
       */

      arr.push({ index: { } })
      arr.push(obj)
      return arr
    }, [])

    /**
     * Push the array of actions to ES and indicate that we are ready
     * for more data. Be sure to propagate any errors:
     */

    try {
      await this.client.bulk({
        index: this.config.index,
        type: this.config.type,
        body
      })
      next()
    } catch(err) {
      next(err)
    }
  }
}

module.exports = ElasticsearchWritableStream

Using Disqus with Netlify

You Know, For Devs — Fri, 20 Jul 2018 11:37:01 +0000

To use Disqus on Netlify you can either let Netlify do the work of embedding the required snippet, or get your static-site generator (SSG) to take the strain.

Netlify

Netlify is great. In the search for the ideal blogging platform it has the perfect architecture.

First, you write all of your posts in Markdown and keep them in a Git repo, and then use a static-site generator (SSG) to build your blog. Of course you do, you might say…where’s the news?

Well, second, whilst you might be used to running your SSG locally and then pushing the generated HTML to somewhere like S3 or a CDN, with Netlify that’s done for you, by connecting Netlify to your GitHub, GitLab or Bitbucket repo.

Yawn…you can do that with GitHub Pages, you tell me.

But here is where it gets interesting; Netlify has built a whole set of features into its build process that can help you manage the generated blog without having to modify the repo itself. You can add forms and logins to your blog, for example.

And if you want to take the site beyond a blog and into application territory you can even add AWS Lamda Functions.

Using Netlify Snippets

The easiest way to get Disqus going in Netlify is to create a snippet. Simply paste the code from Disqus’s instructions for the universal embed code.

There are two problems with this, though.

The first is that Netlify only supports placing snippets before the closing head tag or before the closing body tag. However, for Disqus to look its best in our blog it would ideally be placed before the footer. In the case of this blog (which is generated by Jekyll) that’s just before the closing article tag.

The second issue is that in order to solve the split threads and missing comments problem we’d like to be able to insert the page URL and a unique identifier into the embedded script. This is not possible in Netlify snippets though, since there is no support for variable substitution.

However, neither of these issues is a show-stopper; having the Disqus comments placed at the end of the page rather than before the footer looks a bit odd but not terrible. And I’ve seen many sites that don’t bother to put the page URL and identifier values into the Disqus script–probably because they have canonical URLs for their site pages and don’t suffer from the ambiguity that can cause the ‘split threads’ problem.

So if neither the layout or canonical URLs problems are an issue for you then using Netlify snippets is certainly the best way to go, because it is completely independent of the SSG used to generate the site.

Using the SSG

The other approach to incorporating Disqus is to get your SSG to do the work. Most SSGs will let you trigger inclusion of the snippet required with just a little bit of configuration. In the case of Jekyll it’s a simple case of adding the shortname for the particular Disqus configuration for your site, to the _config.yml file. In my case it looks like this:

.
.
.
disqus:
  shortname: you-know-for-devs
.
.
.

Jekyll won’t add the correct snippet in preview sites but it will when generating a site in production mode. This is easily done in Netlify by setting JEKYLL_ENV to the value production in the Build environment variables section of your site’s Settings:

Once the config and environment variable are added the result is that the correct snippet is embedded by Jekyll at the end of the article tag (looking much neater than at the end of body) and the page’s URL and a unique identifier are added to the configuration used to communicate with the Disqus back-end.

One thing that caught me out for a little while is that if there is no site URL set in the config then the embedded script fails to make the connection with Disqus. It’s a simple fix; just make sure to have url set in _config.yml:

.
.
.
url: "http://youknowfordevs.com" # the base hostname & protocol for your site, e.g. http://example.com
.
.
.

That’s it. Commit and push your configuration changes to your repo and let Netlify do its magic–a few seconds later you have Disqus comments on your site.

Your Development Workflow Just Got Better, With Docker Compose

You Know, For Devs — Mon, 02 Jul 2018 13:30:01 +0000

In a previous post we saw how to set up our basic Node development environment using Docker. Our next step is reduce the size of these unwieldy docker run commands. This is not just because of their unwieldiness but also because if we just type them from the command-line then we don’t have an easy way to share what we’re doing–not just with other people but with ourselves, tomorrow, when we’ve inevitably forgotten what we were doing today!

So before we forget the command we were running in the previous post, let’s lock it down in a file that we can use repeatedly.

But in what file, you ask?

Docker Compose

The tool we’re going to use to capture these kinds of commands is Docker Compose. This app will have been installed for you when you installed Docker (assuming that you took the advice of our previous post to embrace Docker). Docker Compose is an incredibly handy utility because it allows us to use a YAML file to create definitions for Docker commands, rather than having to use command-line options. This means we can easily share and version our commands.

The YAML file can also be used to manage a group of containers that we want to launch at the same time–perhaps our microservice needs a MySQL database or a RabbitMQ queue–and as if that wasn’t enough, the same file format can also be used to describe a Docker swarm stack–a collection of services that will all run together–when it comes time to deploy our application.

Just as in the previous post we suggested that applications should no longer be installed locally but instead run inside Docker containers, now we want to just as strongly argue that no activity that can be performed in the creation of your application–whether linting, testing, packaging, deploying–should be carried out without it being captured in a Docker Compose file.

But before we get too excited, let’s go back to the command we were running in the earlier post (which launches a development container in which we run Node) and let’s convert it to use Docker Compose.

A Docker Compose Configuration File

Recall that the command we were running was:

docker run -it --rm -v ${PWD}:/usr/src/app -p 127.0.0.1:3000:3000 \
  node:10.5.0-alpine /bin/sh

To turn this into a Docker Compose file, fire up your favourite editor and create a file called docker-compose.yml into which you’ve placed the following:

version: "3.2"

services:
  dev:
    image: node:10.5.0-alpine
    ports:
    - "127.0.0.1:3000:3000"
    volumes:
    - .:/usr/src/app
    command: ["/bin/sh"]

You can probably figure out which parts of the original command-line map to which entries in this Compose file, so we’ll just flag up a couple of things that might not be immediately obvious.

First, the entry dev is just the name of our service. It can be anything we like, and there can be more than one of these entries in a file. We’ll see in a moment how it’s used to indicate what we want to launch.

(A service is the term Docker Compose uses to describe running containers. The reason it doesn’t use the term container in the way that we would if we were using the docker run command is that a service has extra features such as being able to comprise more than one instance of a container.)

Next you probably noticed that the port mapping now has quotes around it; on the command line we had -p 127.0.0.1:3000:3000 whilst in the compose file we have "127.0.0.1:3000:3000". This is a best practice due to the way that YAML files are processed. If a port lower than 60 is mapped and no IP address is specified (for example, 40:40) then the parser will not treat it as 40 followed by 40, but as a base 60 number. You could just remember that you need quotes when using ports below 60, but most Docker Compose files you’ll see will have quotes placed around any port number, which is a little easier to remember.

Finally, you will also have spotted that the ${PWD} part of our docker run command has now been replaced with ., i.e., the current directory. Docker Compose doesn’t need the environment variable when mapping volumes, which makes things a bit easier. Paths in the YAML file are always relative to the file itself (and relative paths are supported).

Launching Our Development Container

Now we have our configuration set up it’s a simple matter of running the Docker Compose command with the name of our service. Run the following command and you should have launched the development environment again:

docker-compose run --rm --service-ports dev

Ok…so it’s still not the shortest command on the block–we’ll see in a future post how we can get this down further. But it’s a lot easier to remember than the long docker run command we had before. And what’s more, it will always be the same no matter what changes you make to the configuration file; any additional options we want to add to our docker run will go in our Docker Compose file, clearly documented and under source control.

Just to wrap up this section, we’ll quickly explain the parameters that we need to pass to docker-compose run. The first is --rm which is exactly the same as the option we were using with docker run–when the command has finished running our container will be deleted.

The second is --service-ports which instructs Docker Compose to make available any port mappings we define in the Compose file. It’s a little annoying to have to add this parameter, and you’ll find many discussion threads that argue that this behaviour should be the default. But the logic is fair; if we are launching a number of connected services, such as a web server and a MySQL database, we don’t necessarily want every single port to be mapped to our host machine. In the example of a web server and MySQL server for example, there is no need to expose MySQL’s port 3306 on our laptop since it is only needed by the web server connection to the backend. Docker Compose will create a network that the web server and MySQL can use to communicate with each other.

So there we have it; run that command, and we will get a shell prompt, and then we can launch our web server in exactly the same way as we did in the previous post, when using docker run:

cd /usr/src/app
node app.js

Working Directory

We said a moment ago that one of the advantages of using Docker Compose is that we can add additional options without changing the way we run the command. An example would be to get Docker to change to the working directory for us, i.e, to remove the need for the cd /usr/src/app step in our sequence, above.

To do this we only need to add the working_dir option to the YAML file:

version: "3.2"

services:
  dev:
    image: node:10.5.0-alpine
    working_dir: /usr/src/app
    ports:
    - "3000:3000"
    volumes:
    - .:/usr/src/app
    command: ["/bin/sh"]

And to stress again, we still launch our development environment in exactly the same way as we did before–the only changes are to the configuration file:

docker-compose run --rm --service-ports dev

This time our command-line prompt will have us sitting in the correct directory, and we can launch the server directly:

node app.js

Changing Launch Commands

But we can go a bit further here; we’ll rarely need to be ‘inside’ the container doing stuff, since we’ll be using our favourite editor running on our laptop (remember we’ve mapped our project directory into the container so that our laptop and the container both have access to our files). So we’ll probably find ourselves more often than not invoking our container and then running the server. So we could change the command that is run inside the container from one that launches a Bash shell to one that launches the server:

version: "3.2"

services:
  dev:
    image: node:10.5.0-alpine
    working_dir: /usr/src/app
    ports:
    - "3000:3000"
    volumes:
    - .:/usr/src/app
    command: ["/bin/sh", "-c", "node app.js"]

Making a Clean Exit

You probably spotted that the command we added was not what we might have expected:

    command: ["node", "app.js"]

but:

    command: ["/bin/sh", "-c", "node app.js"]

The background as to why is that if we use the first version of the command which simply runs node with app.js as the parameter, then when we try to exit the server with [CTRL]+C nothing will happen and we’ll have to find some other way to kill the server. This is because the Node app does not process a SIGTERM signal (a [CTRL]+C) correctly when Node is running as the primary, top-level application in a container (what you’ll often see described as running as PID 1).

However the Bash shell does handle the whole SIGTERM dance correctly, and will cleanly shut down our server when it receives [CTRL]+C. So all we need to do is run our server inside a shell.

If you need (or want) to understand this in more detail then search online for something along the lines of “pid 1 docker node” and you’ll find a number of articles. If you just want to cut to the chase then read the section Handling Kernel Signals in the best practises guidance for using Node in Docker.

Multiple Services

Of course, if we think we might need both of these commands–the one to launch a Bash shell inside the container, ready for playing around, and the one to launch the server–then instead of overwriting our first, we can just add a second entry to our Docker Compose file:

version: "3.2"

services:
  shell:
    image: node:10.5.0-alpine
    working_dir: /usr/src/app
    ports:
    - "3000:3000"
    volumes:
    - .:/usr/src/app
    command: ["/bin/sh"]

  serve:
    image: node:10.5.0-alpine
    working_dir: /usr/src/app
    ports:
    - "3000:3000"
    volumes:
    - .:/usr/src/app
    command: ["/bin/sh", "-c", "node app.js"]

We’ve changed the name of the shell version from dev to shell to indicate what it’s used for, which means we can now launch the server with:

docker-compose run --rm --service-ports serve

Don’t Repeat Yourself

One last tip involves a way to reuse the common settings we have in our file. As you can see the only difference between our two services is in the command value. Ideally we’d like to place all of the other values into some common collection and share them between both services.

This is possible in version 3.4 onwards of the Docker Compose file format by using YAML anchors:

version: "3.4"
x-default-service-settings:
  &default-service-settings
    image: node:10.5.0-alpine
    working_dir: /usr/src/app
    ports:
    - "3000:3000"
    volumes:
    - .:/usr/src/app

services:
  shell:
    << : *default-service-settings
    command: ["/bin/sh"]

  serve:
    << : *default-service-settings
    command: ["/bin/sh", "-c", "node app.js"]

So note first that the version value has been updated at the top of the document. Then, any block that we want to create for sharing goes at the top level with an x- prefix–that’s how we tell Docker Compose not to process this block as some configuration.

Within the custom block we set an anchor (the &default-service-settings part) and give it any name we want. Then finally we can refer to that block by referencing the anchor with the << syntax.

Next Steps

We’ve taken our original docker run command and converted it to use Docker Compose, making complex configurations much easier to manage. We’ve also added some additional commands to help with our development process. And we also now have a way to keep a collection of commands under source control. We can now build on this approach to:

add more directory mappings so that modules installed with npm install stay inside our container;
add entries for test containers that include runners like Mocha or TAP;
add entries for commands that help the build process, for example using Webpack or Parcel;
launch local Nginx servers that will mirror our live deployments.

We’ll drill into these techniques and more in future posts.

Good luck with your projects!

Don’t Install Node Until You’ve Read This (Or, How to Run Node the Docker Way)

You Know, For Devs — Sun, 01 Jul 2018 13:30:01 +0000

We need Node for some application or other–perhaps we’re creating a microservice or just want to follow along with a tutorial.

But most places you start with suggest that the first step is to install Node for your operating system. Perhaps you’re on a Mac so now you have to start thinking about whether you should also install Homebrew or MacPorts.

Or you’re on Ubuntu so you head in the apt-get direction…except before you know it, to get the latest version you find yourself using curl to pipe some script to your shell.

Windows? You could just use the Windows installer but as with macOS you ponder whether it’s time to embrace the Chocalatey or Scoop package managers.

In this blog post we’ll look at how skipping over all of this and heading straight to a Docker environment makes it much easier to manage your Node applications and development workflow, and whats more gets you going with best practices right from the start.

Docker First

Whichever route we go with installing Node the OS-specific way, we now have two problems; the first is that the way that we install Node is different on each platform, and damn, that’s annoying. And number two we now have Node installed globally on our laptop. Why so sad? Well now if we want to use different versions of Node for different projects we have to faff about with something like nvm. (And if you were planning on running a Python project it’s the same story, with virtualenv.)

So do yourself a favour and get Docker installed. True, how you install Docker will also be different for different platforms–Ubuntu is slightly different to Mac and Windows. But this initial effort will repay you later because now you’ll have a standard way to install Node, Ruby, Python, TensorFlow, R…whatever language you’re using for your projects–or perhaps more likely nowadays, languages–just got way easier to manage.

So assuming that you now have Docker, let’s get a development environment set up so that you can get back to that tutorial or project.

Running Node

First, create a new directory for your project:

mkdir new-project && cd new-project

and then launch the latest version of Node:

docker run -it --rm node:10.5.0-alpine

If you haven’t run this version of Node before, Docker will download it for you. After a bit of toing and froing you’ll be left with the usual Node command prompt. Type something like 5+6 and press return to check all is well, and then press [CTRL]+D to exit.

If you are reading this in the future you might want to find out what the most recent version number is; just head to the Docker Hub page for the official Node Docker image.

Interactive Containers

We executed the docker run command with a couple of options. The first–the -it part–is a combination of the two options, -i and -t. It’s these options together that mean we can interact with the running container as if it was our normal shell, accepting input from our keyboard and sending output to our display.

Disposable Containers

The --rm option causes the container to be deleted when we exit. It’s a good habit to get into to delete containers as we go along, because it gets us into the mindset that our containers are disposable. This is particularly important when it comes to deployment because we don’t want our container to hold any state internally–any updates or processing should result in writes to external services such as a connected file system, cloud storage, queues, and so on. By taking this approach it’s really easy to upgrade our images to newer versions when necessary–we just throw away the old ones and launch completely new ones.

(It will also make it easier to scale, since we can just launch a bunch more containers when we need to do more work, and provided that all state is maintained outside of the containers this becomes straightforward.)

Bonus Points: No SSH

If you really want to get into good habits with your Docker containers then also avoid the tempation to SSH into a running container to see what’s going on. There’s nothing worse than making a tweak to fix something, logging out, and then forgetting what was changed. The service may now be running again and your boss thinks your are flavour of the month, but it’s fragile. Deploy again and you overwrite those changes. Far better to fix the problem in your deployment scripts, then simply tear down the faulty service and launch another. The changes are now clear to see in source control and reproducible.

Versions

Beyond the command-line options to docker run, there are also a few things to note about the Node Docker image that we’ve used (the node:10.5.0-alpine part).

First, it’s worth being specific about the version number of Node that you are using, since it makes it easier to force updates and to know what is being deployed. If we were to only specify ‘version 10’:

docker run -it --rm node:10-alpine

or even ‘the latest version of node’:

docker run -it --rm node:alpine

then although on the first time through we’ll get 10.5.0, once the images are updated at some later point, we won’t pick up the same version on subsequent runs. At some point using node:10-alpine in the command will cause us to pick up version 10.6.0 or 10.7.0 of Node. And using node:alpine will at some point cause us to get version 11 and onwards.

However, if we choose a specific version like 10.5.0 then although we also won’t get updates automatically, it will be a simple case of updating to 10.5.1 in our build files, when we are ready to force a download of the latest changes.

This is particularly important when it comes to deploying applications later on (or sharing your code with other people), since you want to be able to control what version appears where. And perhaps more to the point, when you are troubleshooting you want to know for sure what version was used.

Controlled Updates

It’s tempting of course to want to ‘always use the latest’; after all, the latest will be faster, won’t it? And won’t it have the latest security patches? This is true of course, but in the quest for building a reliable infrastructure you should aim to control updates to the foundations. This means that if you have a bunch of code that is working fine on version 10.5.0, nicely passing all of its tests and performing well, then a move to another version of Node should be something that is planned and tested. The only real pressure to move versions comes with the point releases such as 10.5.1 or 10.5.2, since they will contain security patches and bug fixes; a move to 10.6 or higher is certainly a ‘nice to have’, but if your code is working and your service is running, then you will definitely want to consider whether your time is better spent elsewhere.

Base OS

The second thing to note about the Node Docker image selection, is that we’ve used the alpine version of the image which uses Alpine Linux as the base operating system. This is the lightest of the Node images, and only provides the bare minimum of an operating system to get Node running–we’re most likely creating microservices, after all.

You’ve probably come across the alpine project but if you haven’t, take a look; it’s being used right across the Docker ecosystem to keep Docker images light.

It should be said as well that ‘light’ doesn’t just mean small for the sake of size–that’s all good of course, since it reduces the amount of data flying around your network. But in the case of a deployed service ‘light’ also means reducing the number of moving parts that can go wrong. If you start with something big like a Ubuntu base image you’re bringing in a bunch of unnecessary code and so increasing the possibility of something going wrong that wasn’t important in the first place. Imagine some nefarious outsider taking advantage of a security hole in Ubuntu, in a service that you didn’t even need!

(You may have come across the expression ‘reducing attack surface’; this is exactly what is being referred to.)

So keep it small, tight, and controlled…and most of all, secure.

Building Your Own Base Images - Don’t!

And it should probably go without saying that you don’t want to be building your own base images. The various Docker Node images, for example, are maintained by the Node project itself, so if anyone is going to know how to build a secure, fast and reliable image it’s them. What’s more, if anything does go wrong there is a whole community of people using the image and reporting issues; you’ll invariably find a solution very quickly.

A Development Environment

So we have chosen a Node image, and we have it running from the command line. Let’s press on with our development environment.

In order to be able to update files in our project directory we need to give our Node application ‘access’ to that directory. This is achieved with the ‘volume’ option on the Docker command. Try this:

docker run -it --rm -v ${PWD}:/usr/src/app node:10.5.0-alpine \
  /bin/sh -c "touch /usr/src/app/README.md"

This will:

create a directory inside your Docker container (at /usr/src/app), and make it refer to your current working directory outside your container (the ${PWD} part);
launch the Bash shell (rather than Node), to run the touch command which will create a README file.

The command should exit cleanly. Check your current directory to ensure that the file has been created:

$ ls -al
total 0
drwxr-xr-x 4 markbirbeck staff 136 1 Jul 13:26 .
drwxr-xr-x 10 markbirbeck staff 340 1 Jul 11:47 ..
-rw-r--r-- 1 markbirbeck staff 0 1 Jul 12:58 README.md

This is a laborious way to create a file, but we just wanted to check that our Docker container was able to ‘see’ our laptop project directory and that it could update files within it.

We now have two ways that we can work on our project: we can either fire up vi from inside the container and make edits which will immediately be mirrored to our working directory on our laptop; or we can use our familiar laptop tools–like Visual Studio Code, Sublime Text, and so on–to create and edit files outside the container, knowing that changes will be immediately mirrored to the /usr/src/app directory within the container.

Either way, we can now develop in pretty much the same way as we normally would on our laptop, but with an easy to manage Node environment, courtesy of Docker.

Opening Ports

One last thing. Let’s say we got started with Node by following the little intro on the Node site. You’ll see that it sets up a ‘hello world’ web server and suggests that the page can be viewed at http://localhost:3000. Go ahead and create that app.js file in your current directory…but there’s no point in running it since as things stand with our Docker development environment approach, this server won’t work.

However, just as we saw earlier that we can map directories between the host and the container we can also map ports. The first step is to add the -p option to our command like this:

docker run -it --rm -v ${PWD}:/usr/src/app -p 3000:3000 node:10.5.0-alpine \
  /bin/sh

We can now access port 3000 inside the container by making requests to port 3000 on our host machine, which satisfies the http://localhost:3000 part of the Node tutorial.

But there is one last minor tweak we’ll need to make; when the server launches it will listen on the IP address 127.0.0.1 which would be fine on our laptop, but is no good inside a container. We might use this address to prevent our server being reached from outside of our laptop, but in the case of a Docker container there is a network connection from our laptop to the container (think of them as separate machines), so keeping things ‘private’ inside the container will just mean that nothing is reachable.

All we need to do is change the file that was provided on the Node site, and modify the hostname variable from 127.0.0.1 to 0.0.0.0. This will tell the server to listen to all IP addresses within the container, not just localhost. We can still ensure that our server is not reachable from outside of our laptop if we want, by modifying the Docker command to this:

docker run -it --rm -v ${PWD}:/usr/src/app -p 127.0.0.1:3000:3000 \
  node:10.5.0-alpine /bin/sh

I.e., the mapping from host port to container port should only take place on 127.0.0.1 rather than on 0.0.0.0 (which is the default for a port mapping).

Whether you modify the port setting when you run the command or not, once the app.js file has this minor change then the server can be launched from inside the container. Change directory to where the app.js file is, and then launch it:

cd /usr/src/app
node app.js

Now you should be able to reach the ‘hello world’ page from the host machine by visiting http://localhost:3000.

Next Steps

Assuming all is well, we can now carry on with whatever project or tutorial we were following. Anywhere that the tutorial tells us to run something from the command-line we make sure to do it from inside the container by firing up the Bash shell. If the project requires that we expose a different port then just change the -p option (or add more mappings if necessary).

There are a lot more ways that we can improve our develoment environment; we can:

bring in Docker Compose to shorten our command lines;
add more directory mappings so that modules installed with npm install stay inside our container;
create test containers that include runners like Mocha or TAP;
launch local Nginx servers that will mirror our live deployments.

But all of these will build on the basic setup we have here. We’ll drill into these techniques in future posts.

Good luck with your projects!

Using Docker and Repo Linter to Capture Project Structure Rules

You Know, For Devs — Mon, 21 May 2018 16:44:01 +0000

One of the things that’s frustrating about having lots of open source projects is, well…having lots of open source projects. Some issues are obvious, such as how to keep projects moving along, responding to pull requests, fixing bugs and adding features. But there are plenty of less obvious things to deal with, such as how to keep a whole bunch of projects aligned once you’ve decided on some best practices.

StandardJS

Let’s say we’ve come across StandardJS (which we have) and we want to apply it to all of our projects (which we do); how do we go about this? Obviously we have to locally clone every project, run StandardJS to reformat the files, run tests on the results, commit the changes, and submit a pull request. And then when we realise that we forgot to add the StandardJS badge to each and every README file, we do the whole dance again.

Even if we put to one side (at least for now) the difficulty of making lots of changes to lots of projects, how do we ensure that any new project we create follows whatever new conventions we’ve adopted?

Occam’s Razor

Take something really simple like an AUTHORS file. In a Node project we can either put the names of contributors into the package.json file, or NPM will pick up the names of contributors from an AUTHORS file if one is present. Which approach is better? Well in my opinion the latter is better because the AUTHORS file is a convention that you’ll find in many different types of open source projects, not just Node. My personal rule is that if there is a language-independent convention that can be followed at no cost then I will always follow it, rather than using a language-specific equivalent. (Think Occam’s Razor.)

By self-created (Moscarlop) - Own work, CC BY-SA 3.0, Link

But regardless of which way we come down on tracking contributors, the point is that we’ve thought about a problem, we’ve made a decision, and the result is a simple rule that we can easily follow in all of our projects (regardless of language)–in my case it’s always have an AUTHORS file. If we can just remember this rule then we can avoid having to think about it each time we start a new project; after all, there’s nothing worse than the Groundhog Day moment(s) of going through exactly the same decision-making process all over again, just to arrive at the same conclusion.

All of which means we’ve moved the problem along a little; whereas before we were having to decide every time we started a new Node project whether to put contributor names in the package.json file or an AUTHORS file, now the problem is how do we remember what we decided last time? At least this problem has the advantage of being the same no matter what issue we are trying to solve…Which license should I use? Which version of Node should I put in my Dockerfile? And so on.

I’ve considered various ways to capture these kinds of rules, once they’ve been decided on. They could be written in a notebook and kept in the top drawer of your desk. But one criteria I’d like is to make the rules shareable and discussable. Which means that even if the only person discussing the rules was myself with myself, it would be handy to have a space where the advantages and disadvantages of using, say, StandardJS instead of ESLint directly, could be weighed up. (Not to mention whether or not to use semicolons.) So whilst some kind of shared document would be great, a Git repo would be even better.

Specifying Rules

Assuming this knowledge is captured in a repository with all the benefits of issue-tracking and comments, what form should the rules actually take?

They could just be written in prose, such as always use an AUTHORS file, and that would certainly be a great start. But even better would be to find a way to express the rules in such a way that they could be enforced by software. And it’s at this point in my research and Googling that I came across Repo Linter.

Repo Linter

Repo Linter contains a set of generic tests, such as check that a certain file exists or check that a set of files contain the specified text. These tests can then be used to create more specific rules such as does a README.md file exist? or do all of the source files have a copyright message at the top? Rules are then further combined into rulesets to create the exact collection of policies that you want to enforce for your projects.

To illustrate, a ruleset that contains two rules–one to check that there is a license and another to check that there is a README–might look like this:

{
  "rules": {
    "all": {
      "license-file-exists:file-existence": [
        "error", {"files": ["LICENSE*", "COPYING*"]}
      ],
      "readme-file-exists:file-existence": [
        "error", {"files": ["README*"]}
      ]
    }
  }
}

This is pretty powerful, but Repo Linter goes further and allows us to specify rules that are specific to certain languages (the rules above will apply to all languages). For example, to ensure that all of our Node projects include a package.json file we’d add this:

{
  "rules": {
    "all": {
      ...
    },
    "language=javascript": {
      "package-metadata-exists:file-existence": [
        "error", {"files": ["package.json"]}
      ]
    }
  }
}

So now we have a way to express our rules in a simple JSON file. All that’s left is to apply them against a repo.

The Answer is Docker (Now…What was the Question?)

The easiest way to get these rules into a reusable form is to create a Docker image that bundles both Repo Linter and a configuration file for some set of specific rules. I’ve created a basic Docker image that installs Repo Linter and its dependencies, and adds some simple rules to check for a license, AUTHORS and README files. There are also some instructions to help create new images based on this base, so that more specific rules can be added. The repo is markbirbeck/docker-repolinter and the Docker image it generates is in Docker Hub at markbirbeck/docker-repolinter.

You can also skip this and go straight to the repo I’ve set up to do exactly what I’ve described in this post–capture rules and have a place where they can be discussed. That repo is at markbirbeck/my-repolinter and it should be very straightforward to copy it and create a Docker image that captures your own policies.

Where Next?

In a future post I’ll look at how we might go about generating any files that we detect are missing.

Disposable Laptops With Docker Compose And NPM

You Know, For Devs — Sun, 23 Jul 2017 15:11:01 +0000

Switching laptops a lot more frequently than I would want has led over the years to my trying a variety of ways to keep track of the applications that I need to install to get up and running quickly. In this post I’ll look at where I’ve got to with this.

[If you want to look at the results of this exploration before you look at the ‘why’ then head over to docker-compose-run to see how to run cross-platform apps. Also, look at https://github.com/markbirbeck/mydesktop for an example of how to quickly install a collection of these cross-platform apps on any device.]

Just when you think you’ve got your laptop set up just how you like it, something happens that means you have to start again. Perhaps you upgraded your hard-drive to an SSD and had to reinstall everything. Or maybe your fan broke and you had to switch to another laptop whilst it was repaired. (Only to have the wireless network card break on the stand-in, requiring another swtich.)

Or maybe the laptop didn’t break at all, and you left it on an underground train after a cookery class–no doubt distracted by having too many bags of food to carry. The upside of this particular event is that losing an entire computer means you finally have a perfect excuse to buy Linux rather than Mac, as you’ve long been telling yourself you would.

All of these changes has led over the years to a variety of ways to document and track how a machine should get set up. My current interest is laptops but in the past I needed to address the same challenge with servers. From simple shell scripts in RightScale back in 2008, through Chef (and VirtualBox) in 2012, and finally to Docker today, I’ve used a myriad of tools to get servers deployed and running as quickly as possible (and to record exactly how I did it).

The general logic with servers has been to start with a clean slate, run some scripts to configure things how you’d like, and then let it go. Crucial to the approach is to not assume that the server will last forever, which means nothing gets placed on the machine that can’t be ‘recreated’ through the install scripts.

But for laptops it hasn’t been so straightforward. For a start, there isn’t an easy concept of ‘run these scripts to get a standard laptop’ because laptops tend to change a lot during normal use. What I’ve usually done–and I’ve seen many others do the same–is create a script that installs all the software I like on a new machine and then try to keep that script up-to-date ready for the fateful day when it will be used again.

One obvious problem with this is that you have to remember to update the script when you install new software…and then update it again when you remove software. Another issue is that the only time you ever run this script is when you have a brand new laptop, so you can’t be sure that it will always work (or conversely, the only time it will fail is when it’s crucial that it doesn’t). And yet another problem is that many applications don’t have easy installations, so your scripts end up not being a collection of actions but a bunch of comments that remind you of where to download a zip file from, or which special incantation needs to be run to apply a patch to a file to fix a buggy release.

But perhaps the most annoying issue is that my ‘Mac From Scratch’ script ends up being very different to my ‘Ubuntu From Scratch’ script–partly because there is different software on the different platforms, but also because, even when both platforms use the same software they each have different package managers.

Why would I need two sets of scripts? Well at the moment I have three old Mac’s lying around plus the Ubuntu laptop I’m writing this post on, and since I can’t afford downtime, in an emergency I’d like to be able to switch any of those machines into service as quickly and easily as possible.

So lately–and with this goal in mind–I’ve been taking a different approach, made possible by first Docker, and then Docker Compose…all bound together with NPM.

Dockerising The Desktop

The Docker integration on the Mac (and Windows, for that matter) has come on in leaps and bounds in the last year or so, so an obvious question now is why not run our desktop apps through Docker? ¹

This has a number of advantages. The most obvious is that suddenly you’re running exactly the same apps on any of your desktops. For example, you might use bitlbee to interact with all of your online chat services. Get this set up in Docker and you can run the exact same configuration on any machine you own, regardless of whether it’s Linux, OS X or Windows. More than that, as you tweak and poke and prod your configuration you can keep track of the changes in a Dockerfile in GitHub–or whatever version control system you prefer–rather than adding and removing comments in a Gist.

If you need more advantages, perhaps my favourite is that containerising your desktop keeps down the clutter that can happen when installing lots of dependencies. For example: I’ve never really liked Ruby, I’m afraid ², so I’d really rather not have loads of gems cluttering up my environment. Of course, if I’m running an application that is written in Ruby then I’ll need those gems, but if I move away from the app to use another I don’t want the app (and Ruby) to put up a fight when I come to uninstall it. By keeping all of the gems inside a container they’re all gone in one fell swoop.

And the icing on this already delicious cake is that if you decide to move an app to the cloud–and something like bitlbee is a prime candidate for that–then now that you have a Dockerfile it’s just a question of deployment.

Let’s Jekyll

To illustrate all of these ideas, let’s get a blog set up using Jekyll.

Docker

It so happens that the Jekyll development team have created a number of Docker images that contain everything we need to get going. The images are hosted on Dockerhub at jekyll/jekyll.

The general pattern with apps that are running in a Docker container is to provide a load of preamble and then the specific command that we want to run. The ‘specific command’ part that gets Jekyll to take the files in the current directory and build our blog looks like this:

jekyll build

With Docker we add the preamble at the front, so it will look something like this:

[preamble] jekyll build

The preamble will get Docker to launch the correct container before running the command that we add to the end–i.e., the jekyll build part– inside the container. We’ll look later at how we can refine this preamble, but for now let’s spell it out.

The Jekyll Docker image usage wiki page advises the following to run a command:

docker run \
  --rm \
  --label=jekyll \
  --volume=$(pwd):/srv/jekyll \
  -it \
  -p 127.0.0.1:4000:4000 \
  jekyll/jekyll \
  jekyll serve

This pattern is pretty much the same as we’ll use for all of our commands so it’s worth breaking apart:

Removing Containers On Close (`--rm`)

We tell Docker to remove the container after it has finished running because there is nothing ‘inside’ the container that we want to keep after it has exited, and we don’t want to have to periodically delete these finished containers to save space.

Labelling The Container (`--label`)

We give the container a label of jekyll to make it easier to keep track of.

Providing Access To Our Local Files (`--volume`)

We need to give the container access to the blog files that we have in the current directory. We don’t want to put the files inside the container, because then we’d have to work out how to share the container with our other computers as well as having to deal with its lifecycle. It’s much easier to treat the Docker container itself as disposable and then worry about giving it access to our data which we’ll store elsewhere. To share the blog posts on our different computers we can easily mirror our simple markdown files with Dropbox, Google Drive, or whatever.

The pattern we’ll see as we use Docker for more and more local apps is that some directory on our local machine–in this case the current working directory, obtained with $(pwd)–is made available inside the container, as if it was the directory /srv/jekyll.

Connecting To Our Console (`-it`)

We also need to tell Docker that we want to interact with the session whilst it’s running, which we can do using the -i and -t options. With Jekyll we’re mainly interested in any output messages, but there will be other apps–like email and chat clients–where we’ll actually interact fully with the running container.

Mapping The Ports (`-p`)

To get access to the running web server from a web browser on our local machine we need to give the -p option. This maps a port inside the container, to a port on our local machine.

Running the Command

So what happens when we run this command?

Well first, if you haven’t run it before, the Docker image jekyll/jekyll will be downloaded by Docker, and cached. Once the image has been downloaded it will be used to launch a container with all of the voumes and port mappings that we’ve given. And once all this is set up, our command (in this case jekyll serve) will be run. We’ll see Jekyll’s output messages in our console and then we can navigate to http://localhost:4000/ in a web browser.

The command we’ve just run is a bit of a mouthful so it might be tempting to create a shell script or alias to keep it shorter. However, there is another technique which is part of the Docker ecosystem, and gives us a number of advantages.

We’ll look at the benefits in a moment, but for now let’s look at how the approach works.

Docker Compose

The Docker ecosystem comes with a powerful tool called Docker Compose that you can think of as sitting ‘above’ Docker itself, since it’s capable of launching multiple Docker containers as a unit (hence ‘compose’). Docker Compose provides a convenient way to specify all of the different aspects of the various containers that you might want to run–volumes, network settings, which other containers to depend on, and so on–which makes it ideal, even when just launching one container.

So let’s see what a Docker Compose file that provides the same Jekyll ‘preamble’ might look like:

version: "2"
services:
  jekyll:
    image: jekyll/jekyll
    ports:
    - "4000:4000"
    volumes:
    - ${PWD}:/srv/jekyll

Docker Compose will default to wiring in a terminal for us so we don’t need to specify the -i and -t options we saw before. However, for some reason the default behaviour with docker-compose run is not to automatically delete any finished containers, or to map the exposed ports (both of which do happen when using docker-compose up), so we’ll need to add the --service-ports and --rm options.

If you create a file called docker-compose.yml in your blog directory with the YAML that we have above, you can then get the effect of the longer Docker command by doing this:

docker-compose run --rm --service-ports jekyll jekyll serve

In effect we’ve replaced the preamble of the long docker run ... command with docker-compose run --rm --service-ports jekyll, i.e., we’ve told Docker Compose to launch a Docker container using the options found under the key ‘jekyll’ in the docker-compose.yml file.

Benefits

We’ve seen how it works, so now what are the benefits?

An obvious one is that we’ve reduced the size of the preamble to something a little more manageable; we’ve taken this:

docker run \
  --rm \
  --label=jekyll \
  --volume=$(pwd):/srv/jekyll \
  -it \
  -p 127.0.0.1:4000:4000 \
  jekyll/jekyll \
  ...

down to this:

docker-compose run --rm --service-ports jekyll ...

But as we said above, we could have done that with an alias or shell script. The big advantage here is that we’ve captured the detail of the command–the name of the image to download and launch, the volume mapping to use, the ports to expose, and so on–in a platform-independent language that can be checked for validity and easily shared on different machines.

In fact, in the case of this Jekyll example the details of the blogging software you are using could actually be saved as part of your blog; you could save the docker-compose.yml file in the same repo as the markdown for the blog, and be up and running on any new machine within minutes, simply by running the docker-compose command.

This is not to be taken lightly. It’s as if we’ve installed the blogging software in the same directory as the blog itself, and committed it to source control; we’ve managed to get all the benefits of linking an app directly to the subdirectory that it acts upon, but with none of the disadvantages.

Standard Preamble

One last benefit is that by taking a lot of the parameters out into a configuration file we’ve effectively made our preamble consistent across many apps. This lends itself nicely to a simple alias or shell script that abbreviates things a little further; instead of running this:

docker-compose run --rm --service-ports jekyll jekyll serve

we could simply do something like this:

dcr jekyll serve

A simple alias to achieve this could be:

function dcr {
  docker-compose run --rm --service-ports "${1}" "${@}"
}

So instead of having one alias for each of our commands, we have a single alias that encapsulates the preamble and can be used for all commands.

Using A Global `docker-compose.yml` File

Having a local docker-compose.yml file in our blog is great, and with this shell function we can certainly get up and running pretty quickly. But what about our other applications? We were talking about how to get our entire laptop up-to-speed as quickly as possible and this doesn’t yet do that.

One way we could take this further would be to have a single central Docker Compose file that contains instructions for all of our applications. For example, we might have entries for Jekyll and Mutt:

version: "2"
services:
  mutt:
    image: fstab/mutt
    volumes:
    - ~/.mutt:/home/mutt

  jekyll:
    image: jekyll/jekyll
    ports:
    - 4000:4000
    volumes:
    - ${PWD}:/srv/jekyll

With a file like this it would then just be a question of placing it somewhere convenient and ensuring that it’s mirrored to our Git repo, before modifying the alias so that it finds this file. The -f option does this, like so:

function dcr {
  docker-compose --rm --service-ports -f ~/.dcr/docker-compose.yml run "${1}" "${@}"
}

This is a little cumbersome to maintain, and has the disadvantage of being tied to a particular operating system–although it would no doubt be pretty simple to convert.

But maybe we can go one better, by using the Node package manager tools to manage our apps?

Using NPM Modules

NPM gives us a couple of big benefits that can take our Docker Compose approach to another level. The first is that by treating our apps as a module we can easily keep track of the corresponding docker-compose.yml files.

The second advantage is that NPM has the ability to create symbolic links to applications when a module is installed; we can now get the same functionality that we’d get from aliases and functions, but in a way that will work across operating systems.

So to install our Jekyll Docker shortcut we might simply have to do this:

$ npm install -g dcr-jekyll

We now have a cross-platform way to manage installations of our cross-platform applications!

I’ve implemented this NPM approach at docker-compose-run on GitHub. This module provides the equivalent of the alias to run Docker Compose with the right parameters. It’s then used to create application shortcuts such as dcr-jekyll and dcr-mutt. For a list of applications available see the wiki page.

Reproducible Laptop

So now we’re able to run our apps, and we’re also able to use the same app across multiple platforms; the final part of the jigsaw is to keep track of the apps that we want to use on any of our devices. And here there is an interesting twist.

If we run any of our commands, such as dcr-jekyll serve, and the required Docker image is not in the cache, then Docker will download it.

But if we never run the command, it never gets downloaded.

Which means that we aren’t actually ‘installing’ all the commands we want on our brand new laptop; all we need to do is to ensure that our brand new laptop has Docker on it, as well as all of the docker-compose.yml files that drive our applications, and then when we run a command Docker will do what it’s good at and download the correct images. All of this means that we get our new laptop up to speed even faster, because we don’t install everything in one go.

A ‘mydesktop’ Repo

Since everything we need is now in the dcr-* style repos, then all we now need is a single package that brings the whole lot together…and that’s super simple; just create a package listing whatever dcr-* applications you need, and add bin definitions for the application name by which you want to run the command.

For example, if we want to have the dcr-jekyll and dcr-mutt applications available, but we want to run them by simply typing jekyll or mutt–rather than dcr-jekyll and dcr-mutt–then we’d have a package.json file like this:

{
  "name": "@markbirbeck/mydesktop",
  "version": "0.2.1",
  "description": "My desktop applications, saved as an NPM package.",
  .
  .
  .
  "bin": {
    "jekyll": "./node_modules/.bin/dcr-jekyll",
    "mutt": "./node_modules/.bin/dcr-mutt"
  },
  "dependencies": {
    "dcr-jekyll": "^0.3.0",
    "dcr-mutt": "^0.2.1"
  }
}

In short, whatever you put in the bin key is the application name you’ll use.

Now all that’s left to do is push this to GitHub–here’s my desktop if you want to take a look–and then you can install your entire saved desktop with this simple command (or your version of it):

npm install -g markbirbeck/mydesktop

We’re making use of the fact that NPM will install from GitHub by default when presented with an a/b style module name, which means we won’t clutter NPM with our own personal modules.

There’s nothing to stop you taking this approach further and creating separate app collections for development tools, communications tools, games, tools you use for particular clients (if you’re a freelancer), and so on. It would then just be a case of installing the correct combination for work and home machines.

Conclusion

It might feel like we’ve done a lot of work here, but actually we’ve simply brought together some state-of-the-art technologies to make management of our desktop machines simple, trackable and reproducible; we’ve used Docker to run the same application on different operating systems, we’ve used Docker Compose to capture the detail of how each application should run, and we’ve used Node’s package manager to ensure that we can install these valuable snippets into any environment.

In a future post we’ll look at how to ensure we can track our data and configuration files across devices.

Notes

When I started working on this idea I of course Googled around to see what other people had done in this space. I didn’t find anyone trying to turn the whole setup into a load of managed packages, or using Docker Compose to capture the logic, but I did find a few people using Docker to run apps that one would ordinarily install on your laptop. Far and away the best material I read was from Jessie Frazelle; no surprise there…she knows her Docker inside out. In particular her blog post Docker Containers on the Desktop is crammed full of really clever ideas, and there are also links to talks that she did on the subject. ↩

Each time I’ve worked on a Ruby project it has taken an age to get everything set up correctly, whether finding missing gems, getting compilers to work, or a myriad of other issues. I’ve usually been working with developers who’ve had their systems set up for ages and can’t quite remember how they got there. ↩

DEV Community: You Know, For Devs

What I Learnt From Reviewing 22 CVs

Looks Count More Than I Thought

Attention to Detail

Approaches v. Technologies

Approach From Technologies

Public Profile

Qualifications

Conclusion

Let TypeScript Inference Take The Strain

Why You Need a Pipeline Runner For Your Node Streams

Basic Node Streams

A Better Model: A Pipeline of Streams

Separating Pipeline Creation From Running

Creating a Runner

Conclusion

Getting Control Of Your .dockerignore Files

The Goal

Approach #1: Build From A Single Directory

Setting the Context

Ignoring With .dockerignore

Problems With Approach #1

Approach #2: A Richer .dockerignore File

Problems With Approach #2

Approach #3: Define Inclusions, Not Exclusions

Exclude Everything

Only Include Those Files Referenced In Dockerfile

Problems With Approach #3

Conclusion

Using _writev() to Create a Fast, Writable Stream for Elasticsearch

Creating an Elasticsearch Client

Writing Records

Index and Type

Buffering

Bulk Update API

Using an Array

_writev()

Buffer Size

Closing the Elasticsearch Client

Conclusion

Using Disqus with Netlify

Netlify

Using Netlify Snippets

Using the SSG

Your Development Workflow Just Got Better, With Docker Compose

Docker Compose

A Docker Compose Configuration File

Launching Our Development Container

Working Directory

Changing Launch Commands

Making a Clean Exit

Multiple Services

Don’t Repeat Yourself

Next Steps

Don’t Install Node Until You’ve Read This (Or, How to Run Node the Docker Way)

Docker First

Running Node

Interactive Containers

Disposable Containers

Bonus Points: No SSH

Versions

Controlled Updates

Base OS

Building Your Own Base Images - Don’t!

A Development Environment

Opening Ports

Next Steps

Using Docker and Repo Linter to Capture Project Structure Rules

StandardJS

Occam’s Razor

Specifying Rules

Repo Linter

The Answer is Docker (Now…What was the Question?)

Where Next?

Disposable Laptops With Docker Compose And NPM

Dockerising The Desktop

Let’s Jekyll

Docker

Removing Containers On Close (--rm)

Labelling The Container (--label)

Ignoring With `.dockerignore`

Approach #2: A Richer `.dockerignore` File

Only Include Those Files Referenced In `Dockerfile`

Removing Containers On Close (`--rm`)

Labelling The Container (`--label`)

Providing Access To Our Local Files (`--volume`)

Connecting To Our Console (`-it`)

Mapping The Ports (`-p`)

Using A Global `docker-compose.yml` File