DEV Community: Hendrik

Today I learned about "place-items"

Hendrik — Wed, 05 Oct 2022 15:26:11 +0000

Today I learned about the CSS property place-items, it allows you to easily position elements in flex and grid layouts. It also solves the eon old quest to "center something in a div".

To center the single child of a div simply add display: grid; and place-items: center;.

Place-items takes up to two parameters. The first to place the items vertically and the second to place them horizontally.

This concludes out TIL, see you next time.

.wrapper-that-centers {
  display: grid;
  place-items: center;
}

Trying my hands at Deno 1.0

Hendrik — Wed, 27 May 2020 11:13:19 +0000

Following the release of Deno v1.0 I got excited to try my hands at it. These are my first experiences writing a simple tool in Deno.

A super fast introduction to Deno:
Deno is the spiritual successor of Node trying to fix design mistakes that were made early on but recognized only late into the project. Deno supports TypeScript out of the box and relies on web-standards. In Deno you can import ES modules from any URL and use fetch like you would in the browser. To help unify the community on processes and workflows Deno provides a wide array of stdLibs and has build in solutions for bundling, testing and code formatting. You can read more in the Deno v1 release post.

To try my hands at Deno and collect some experience on how ready it is I decided to implement a simple cli. The goal: read all markdown files in the current directory, parse them to HTML, add a navigation and output them into a new folder. I also decided to use new.css as a class-less CSS library to style the pages.

You can go and checkout the finished repo or an example website created.

Organizing

First things first. A plan on how to implement the described tool. We will tackle it through the following steps:

Walk through all files in the directory, getting a list of markdown files.
Iterate over all those files and parse them into HTML.
Add a navigation and header to each page.
Output the files.

While following these steps we will see Denos features in action. Let's get started.

If you are a VSCode user I highly recommend the Deno extension for VSCode

Getting the files

First we need to walk through the files and find all the markdown ones. Lucky for us Denos fs stdLib includes a walk function that does just that.

The docs also tell us how to use this correctly:

async function printFilesNames() {
  for await (const entry of walk(".")) {
    console.log(entry.path);
  }
}

But we only want markdown files and maybe set some more options for this.

After some digging I found the available options in walks source. Which makes this entire thing feel quite immature. But equipped with that knowledge I arrived at a function to get all markdown files from a directory.

const findMarkdownFiles = async () => {
  const result = []
  for await (const entry of walk('.', walkOptions)) {
    result.push(entry)
  }
  return result
}

Feel free to correct my coding here (still new to generator functions).

On the way though I noticed that the docs were incorrectly not providing a path (e.g: ".") as the first argument to walk. Quickly opened a PR to update the docs so the next person won't face the same half hour of struggling.

Outputting the result we have our list of files that we want to work with, Step 1 ✅

Parse the markdown files to HTML

There are a TON of libraries on NPM that you can use to parse Markdown into HTML. Sadly they do not export ES Modules...

After spending some time digging through pika.dev which feel like as close as you can currently get to a central place of searching for ES Module dependencies, I was stuck. For some time I tried different libraries, importing them in different ways but usually they were missing some dependency. There is currently an initiative to support Node modules in Deno but it's still underway. And most NPM packages simply are not ES Modules.

In the end I found snarkdown which didn't have any active development for a year and only supports very basic markdown parsing. It will be good enough for this example project but no where near anything I would want for a real project.

Bringing it in via pika was a breeze and I also enjoyed playing around with the REPL feature on pika.

import snarkdown from 'https://cdn.pika.dev/snarkdown/^1.2.2'

const parseMarkdown = (mdFile: MDFile): HTMLFile => {
  const html = snarkdown(mdFile.content)
  return {
    originalPath: mdFile.path,
    originalName: mdFile.name,
    path: mdFile.path.replace(/md$/, 'html'),
    name: mdFile.name.replace(/\.md$/, ''),
    content: html,
  }
}

It's not sophisticated, it took waaaaay longer than I would want to find a module to use, but parsing is working. Step 2 ✅

Add a navigation

This is the part where native TS support becomes awesome. You might have seen in the code above that I created some types for the objects I am handing around. Thanks to that it is just super easy to know what exactly I have available later down the pipe.

On a side note a painful lesson I learned in an earlier project is that collecting all items is a necessity for supporting features like a navigation in a markdown to static website tool.

Equipped with that and some String Literals we can easily wrap the created HTML inside a whole HTML page with a nice title, navigation and inclusion of the new.css styles.

const addSurroundings = (headerCreator: (title: string) => string) => (
  htmlFile: HTMLFile
): ParsedFile => {
  const content = `<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>${htmlFile.name}</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/open-fonts@1.1.1/fonts/inter.min.css">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@exampledev/new.css@1.1.2/new.min.css">
</head>
<body>
  ${headerCreator(htmlFile.name)}${htmlFile.content}
</body>`
  return {
    path: htmlFile.path,
    content,
  }
}

And we have files with a navigation and styling, ready to be saved. A simple step, without dependencies. Greatly aided through TypeScript, Step 3 ✅

Saving new files

Here I started to get into the thick of things where in the end I searched for help on Denos Discord.

In theory Deno also has a stdLib to save files. It's also part of fs and called writeFileStr. The docs are also straight forward, just import it all from the mod.ts file. That is the current convention to export a module for Deno; create a mod.ts that exports everything.

Unfortunately just using an import as the docs say gave me an error at runtime. Reading through the docs I realized that:

All the following modules are exposed in mod.ts This feature is currently unstable. To enable it use deno run --unstable

Okay so I have to use --unstable, that's gonna make my little tool look trustworthy to users. But, let's give it a go.

After trying that I ran into the next error....

This time, even though I am importing all the modules from master there seems to currently be an incompatibility. Luckily the stdLibs come versioned and the version can be added to the URL to fix all dependencies to the same. There is an issue about this, if you want to dive deeper into the topic.

Long story short: after some friendly discussions on and great help from Denos Discord server I opted to fix the versions of my dependencies and import them from the sub modules to not force my users to use --unstable.

// We use:
import { ensureDir } from 'https://deno.land/std@0.51.0/fs/ensure_dir.ts'

// Instead of:
import { ensureDir } from 'https://deno.land/std/fs/mod.t'

It was an adventure that taught me a lot about Denos state and internals but finally, Step 4 ✅

And we can move onwards to using the tool.

Using it

With Deno installed you can try the tool right now.

deno run --allow-read --allow-write https://raw.githubusercontent.com/HoverBaum/md_website/v1.0.0/index.ts

Deno can simply run scripts from URLs which feels refreshingly easy to get started with. The above uses a version to make sure it keeps being the tool this post talks about.

You can also install the script and make it generally available on your machine using the following commands.

deno install --allow-read --allow-write --force --name mdw https://raw.githubusercontent.com/HoverBaum/md_website/v1.0.0/index.ts

This will install version 1.0.0. It also sets permissions that the tool needs to run. We need read and write access to read the Markdown files and write the parsed HTML. --force makes sure to replace previous installations and --name mdw aliases the cli to mdw in your terminal.

Conclusion

On this journey I go to explore many aspects of Deno, learn a lot and build a fun little tool. All in all I enjoyed working with Deno. But let's take a closer look at where things are at.

Standardisation and Modules

Deno is just starting out with v1.0 being about a week old at time of writing this. And Deno opted to not use a central repository but instead distributed hosting. We will need to see whether federations of Module repositories arise that can leverage the potential of this distributed architecture or if Pika (or similar) will rise to be the NPM for Deno.

The bigger issue I found in my small project was the unavailability of ES Modules to support my usecase. That might have been something specific for me though. Generally I am a huge fan of going with Webstandards as they are here to stay and I think this issue will continue to grow smaller, as everyone races to build the "awesome X for Deno" Module.

Community

Denos community is currently hyper-active, as everyone is racing to build the cool things and new standards for X in Deno. I experienced the GitHub issues to be active and the Discord server to be a welcoming place.

There are a couple of discussion points that have potential to split the community, like distributed dependencies but all in all I feel Denos community is a great and growing one.

Last words

Developing my first little tool in Deno was fun. The entire ecosystem does not yet feel production ready, even though it reached v1.0. But Deno is surely a technology you want to keep on your radar and evaluate.

Log to Elasticsearch using curl

Hendrik — Tue, 14 May 2019 11:27:10 +0000

Once you get a centralised logging solution like Elasticsearch setup you open up an incredible amount of possibilities. That is if you actually send logs to your central place. On a recent project, we wanted to

"just log our deploys from the Jenkins Server".

Thus we set out to find the "easiest" way to log to our Elasticsearch. We finally reached curl as our solution and today I will walk you through how to do that.

This post uses Elasticsearch version 7. We also focus on Mac and Linux terminals for commands. Some of them might work differently on Windows, I suggest using Git Bash to get around those issues.

Console basics

If like me, you find the command line highly fascinating for it's myriad of tools and how simple tools can be composed into bigger workflows, much like good software is, BUT you have no idea what you are actually doing (also like me), follow along as we dive into some tools we will use today. Should you be a firm terminal hero, feel free to jump this section.

curl

curl is a free and open source command line tool to transfer data over the internet. As such you can use it to make GET requests or send POST requests to servers. Let's look at a simple example you can try out:



curl https://jsonplaceholder.typicode.com/posts/1

This will fire a simple get request towards the JSON placeholder API and return the post with index 1. curl will log the result to the console for you. Through this logging, to the console, you can also chain curl with other commands, like grep the title of the post. But a deeper dive into chaining is a topic for another day.



curl https://jsonplaceholder.typicode.com/posts/1 | grep title

Let us instead focus on understanding curl and relevant options better. First, we will need to specify the method (POST) we want to use. For that we can use the -X (pay attention to the capital X) option and pass in the method we would like to use. Our placeholder API from before also allows us to post to it, so let us try:



curl -X POST https://jsonplaceholder.typicode.com/posts

Now we will want to send some data along. For this curl has -d to pass data along as well as -H to set a header. We want to set a header for the Content-Type so that we can tell receivers that we are sending JSON. Try the code below and also try it without the -H option to get a feeling for what we get by using it. You will notice that the placeholder API interprets out entire data as a key with no value specified. Setting the right Content-Type helps us to pass the data along in a way that the server can handle. You might run into an issue with ' on Windows.



curl -X POST https://jsonplaceholder.typicode.com/posts -H "Content-Type: application/json" -d '{"title": "My awesome title"}'

Read more about the used options in the curl manual -X, -H and -d

Timestamps and Environment variables

One thing that we wanted to add to all our logs are timestamps. Luckily our terminal comes with date which can print the current date in many formats. You can refer to the Ubuntu docs on date for documentation on date and this article on formatting dates for some inspiration. Generally, you call date with a single parameter that passes in what you want. date +%s for example gives you the seconds since Epoch.

For Elasticsearch however, we need date fields in milliseconds since Epoch. And our timestamp will be a field of type date so that we can filter by it. To achieve that we will add nanoseconds and then divide by 1000000. Because date behaves a bit differently on different System the command below will simply fill up the specificity between seconds and milliseconds with zeros on systems that do not support milliseconds on the date command (looking at you MacOS - Stackoverflow credit).



echo $(($(date +'%s * 1000 + %-N / 1000000')))

A quick foray on $(): This Syntax allows us to use the return value of previous commands as input for the next one. Check out the example command line inputs below to get a feeling for what that means.



export MY_ENV=date
echo $MY_ENV
# date
export MY_ENV2=$(date)
echo $MY_ENV2
# Thu  9 May 2019 12:01:14 CEST

In our concrete example the nested blocks return the following:



echo $(date +'%s * 1000 + %-N / 1000000')
# 1557396103 * 1000 + N / 1000000
echo $($(date +'%s * 1000 + %-N / 1000000'))
# command not found: 1557396188
echo $(($(date +'%s * 1000 + %-N / 1000000')))
# 1557396266000

Now that we have our timestamp we want to use it in our POST request to the server. For that, we will save it in an environment variable and reference that in our request.



NOW=$(($(date +'%s * 1000 + %-N / 1000000'))) && curl -X POST https://jsonplaceholder.typicode.com/posts -H "Content-Type: application/json" -d '{"timestamp": "'"${NOW}"'"}'

Let me unpack that for you: first, we assign the timestamp to an environment variable called NOW than we tell our shell to execute another command without losing context through && here we make our POST request. The && approach is not necessary on all platforms but we use it here to keep the command compatible with Linux and Mac. You would usually expect to now see ${NOW} wherever you want to insert the environment variables value. But to get our shell to properly escape here we need to wrap our value into wrapped single quotes.

A word on Operation System compatibility and escaping JSON values. Windows does not support single quotes ' in the command so you will have to instead use something like: "{\"title\": \"My awesome title\"}". you can also use that Syntax on Macs and Linux machines but I found the single quote approach nicer to read, even though it imposes the escaping for environment variables. Up to you.

Elasticsearch APIs

Elasticsearch provides great documentation on how to get started with understanding, creating and searching data.

Indexes and Documents

First, let's recap how Elasticsearch organizes data. The basic element in Elasticsearch is called a Document. You can think of it as a single event being logged, processed and organized through Elasticsearch. They are still organized in Types which are grouped in Indexes. Types, however, have been deprecated in Elasticsearch 6 (at the time of writing version 7 is current) which currently results in each Index only being allowed a single Type. Indexes, on the other hand, is a collection of similar documents where you can have as many Indexes as you want. You will need those to insert and query documents.

Local Elasticsearch

If you want to follow along examples from here run an Elasticsearch locally using Docker or replace the localhost domain with the one running your Elasticsearch. Run the command below for a Docker container with Elasticsearch and check out the Elasticsearch Docker documentation if you run into trouble.



docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.0.0

All following examples now refer to localhost as Elasticsearches domain so you can copy and paste them to try things out. Remember to update this if you want to use snippets in your environments.

Inserting Documents

You can post to any /index/type to add a new Document to the specific Type of an Index. If you want to get a pretty response, add ?pretty to the URL.

Say for example you wanted to have an Index that groups Documents related to deployments of your application. The object you initially want to log might contain the hash of your code being deployed and thus look like:



{
  "gitHash": "1234567"
}

To now add this Document to the deploys Index using _doc as the standard Type (remember deprecated but still needed) you can simply call the /deploys/_doc endpoint with your data.



curl -X POST "localhost:9200/deploys/_doc/?pretty" -H 'Content-Type: application/json' -d'
{
  "gitHash": "1234567"
}
'

If you are not running Elasticsearch locally but on a remote server replace localhost:9200 above with your domain.

Reading Documents

Now we can also get the document we just added by querying the /index/_search endpoint.



curl -X GET "localhost:9200/deploys/_search/?pretty" -H 'Content-Type: application/json' -d'
{
  "query": { "match_all": {} }
}
'

This will return us all documents within the Index. Check out the search API for more options.

Mappings

We briefly touched on Mappings before when we noted that Elasticsearch expects timestamps (or date in general) to be in milliseconds since Epoch. Through Mappings, we can add fields to an Index. With this, we can control the type that a field will get. While all fields default to "text" it is especially important for our timestamp that we set the field to be of type "date" before adding the first document to our Index. By doing so we ensure that we can later properly filter on our timestamp in Kibana.

Sadly this means we now need to delete our Index again (and any Kibana indexes we might have created to explore our documents before the Elasticsearch Indexes).



curl -X DELETE "localhost:9200/deploys"

Now we can add the Index back with the right mapping for our timestamp using:



curl -X PUT "localhost:9200/deploys" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}
'

You can refer to the docs on date datatypes for more information.

Logging using curl

Now that we have the fundamentals of curl and date down, as well as an understanding of Elasticsearches API, it is finally time to get logging.

We already locked at sending a message containing a timestamp via a POST request. With our knowledge of the Elasticsearch API, we can now log an event with a timestamp.



NOW=$(($(date +'%s * 1000 + %-N / 1000000'))) && \
curl -H "Content-Type: application/json" \
-X POST "http://localhost:9200/deploys/_doc" \
-d'
  { 
    "timestamp": "'"${NOW}"'"
  }
'

With the above, you should have a solid understanding of what is happening here. Below we will dive into a more complex example with once more an explanation of what is going on.

A practical example

Above I used an example of logging something containing a git hash related to a deployment. That, in fact, was my use case to dive into this. Let me present to you my solution to log the currently deployed git hash from our CI Server to Elasticsearch. The following command needs to be run from within a git repository.



HASH=$(git rev-parse --short HEAD) && \
NOW=$(($(date +'%s * 1000 + %-N / 1000000'))) && \
curl -H "Content-Type: application/json" \
-X POST "http://localhost:9200/deploys/_doc" \
-d '
  { 
    "gitHash" : "'"${HASH}"'",
    "timestamp": "'"${NOW}"'",
    "info": "some infos",
    "environment": "test"
  }
'

You can probably figure out what's going on here. Let me just go through it to make sure we are on the same page. First, we have three commands in total, let's examine them one at a time.

HASH=$(git rev-parse --short HEAD) assigns the short hash (first 7 symbols) of the git repository we are currently into an environment variables named HASH.

NOW=$(($(date +'%s * 1000 + %-N / 1000000'))) should look familiar, it gets the current timestamp in milliseconds and assigns it to a variable named NOW.

Finally, we bring it all together in one big curl command. Let me just go over the options for this one.

-H "Content-Type: application/json" sets a "Content-Type" header with "application/json" so that our Elasticsearch knows that it gets valid json.

-X POST "http://your.domain:9200/deploys/deploy" tells curl to fire a POST request and which endpoint to use.

-d '{ "gitHash" : "'"${HASH}"'", "timestamp": "'"${NOW}"'", "info": "some infos", "environment": "test"}' finally defines the JSON data we are going to send along with our request. Here we use the variables we created to fill in the git hash as well as a timestamp. Note that once again we use "'" for proper escaping.

Conclusion

Today we learned a lot about curl, date, and Elasticsearch. We not only took a look at how these tools can be used in isolation but also how to use them together to "just quickly log something".

I hope this sets you up to do great things and achieve quick wins empowered by your existing systems.

Utilizing ESLint rules to format JavaScrip in VS Code

Hendrik — Tue, 26 Feb 2019 09:08:19 +0000

Today we will take a quick look at how to configure VS Code to automatically format your code according to your ESLint rules every time you save a file.

This assumes that you already have some ESLint rules in place. If not the airbnb rules are a good place to get you started.

Once you have rules setup and are hacking away in VS Code, the editor will start to complain every time you forget that "you should indent x" or whatever extraordinary rule your company might use. Especially if you are regularly engaged with different companies using different styles this becomes quite cumbersome. Luckily there are two plugins that can help us get around this.

Go ahead and install ESLint and Prettier. The first is a plugin you probably have installed already to help you see errors in your code-style. The later is a tool generally used to format code.

With prettier you can achieve great things, even format your code on commit which can greatly help to streamline coding-style across the team. For today we will only use it every time we save a file in our editor. To achieve this we simply add two lines to our settings.json.

"editor.formatOnSave": true,
"prettier.eslintIntegration": true

You can find your settings.json by using Preferences: Open User Settings from your Command Palette (⇧+⌘+P | ⇧+CTRL+P).

And now every time you save a file Prettier will format it according to your ES Lint configuration.

Originally published on HoverBaum.net

Automate click-through testing with Puppeteer

Hendrik — Tue, 29 Jan 2019 15:28:09 +0000

Testing is one of the essential steps in software development. Yet, testing and especially manual testing that clicks through the application is a tedious job that we often shy away from. But it doesn't have to be. With great tooling coming out over the past couple of years we now have what it takes to automate click-through testing in our build process. In this post, we will look at why manual or click-through tests are important and how to turn this tedious and bothersome task into a fun one that drives our development forward.

Restless readers can jump straight into the repo accompanying this post.

Why we need manual QA

From a developer point of view; testing and QA, especially manual QA can seem like an unreasonable burden placed on us to hamper our development velocity. While the developer community as a whole, as well as most of the management community, are slowly coming to their senses and seeing that writing well-tested and working code that forms a great product for users is way more valuable than squeezing in one more feature, we still frown on manual QA as a roadblock.

Manual test runs often turn into roadblocks.

But manual QA really isn't the issue we face. It is the way we use it. Originally manual QA comes from the realization that someone really should click through the finished website to make sure that it really works, before shipping it to customers. You can unit test the hell out of things, users and the real world can always break things. Thus we try our best to emulate real-world scenarios and users using manual QA.

Somewhere in the DevOps movement we realized that shipping small pieces of code and doing so often highly reduces the number of bugs we ship, makes our companies more able to react to user needs and market requirements as well as creates a good working environment for ourselves. Too often a big pain point in achieving a high release velocity is manual QA. Because for whatever reason our organizations require a full manual QA run for each release. This tempts us to stock up on features so that QA can process as much as possible at once (if you read the Phoenix Project, this is our heat treat often). We totally ignore that this is exactly what makes it hard to track down which of the dozens of commits introduced a bug and that by doing so we usually block development for a few days just to make a single release happen. We invent complicated procedures like a week of code freeze to create a "stable testing environment" that in reality it often turns out to be anything but stable.

Programming needs QA as much as any other industry does.

Still, we need manual QA just like any other industry does. But it is time to shift the focus. When we look at other disciplines we will always see manual QA, it is simply reassuring to have a human control that everything is going right. They are more flexible and more like the real world users than machines will be for a long while. But do you see quality assurance happen on every single, finished piece leaving traditional production chains? Of course not, that would be such a waste of time! When we build cars we don't need to check every single one. We take out one every so often to ensure the process runs smoothly and trust that the rest work just the same.

That is the mode of operation we want to achieve in software development as well. we make a release and we make dozens of them (maybe even per day). Every once in a while someone manually verifies a release and with that, we trust that the rest work just as well as this one did because we have trust in our processes.

Building trust

But we don't currently have that trust. Often enough companies are scared by experiences of deployments gone horribly wrong. And as a sad result they introduce cumbersome procedures meant to catch errors before they happen when in reality these processes cause the opposite and slow the company down. It is on us to rebuild this trust and to alleviate manual QA of the burden they currently carry. We can achieve this by automating the process QA manually goes through.

To go from where we are to where we want to be, which is not blocked by manual QA runs and releasing multiple times a day, we need to invest into automated testing as well as into building the trust within the organization that our code does what it's supposed to.

Today we will take a great step forward on this road by automating a click-through test for a website.

A simple web application

For today's test run, we will look at the website below which is a simple counter application. We will be able to hit a button "Increment" to increase the count by one. Furthermore, we will provide a button to display a message that was not previously seen, just to add more dynamic behavior for us to test.

And here is the code for the website:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Testing Page</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body style="font-family: sans-serif">

  <h1>Testing page</h1>

  <!-- Counter interface -->
  <button data-test="button-increment" id="js-increment">Increment</button>
  <p>Current count: <span data-test="count-output" id="js-count"></span></p>

  <!-- Button to show a message -->
  <button data-test="button-display" id="js-display">Display Message</button>

  <script>

    // --- Counter ---
    let count = 0    
    const incrementBtn = document.getElementById('js-increment')
    const countSpan = document.getElementById('js-count')

    incrementBtn.addEventListener('click', () => {
      count += 1
      countSpan.innerHTML = count
    })

    countSpan.innerHTML = count

    // --- Display ---
    document.getElementById('js-display').addEventListener('click', () => {
      const messageContainer = document.createElement('p')
      messageContainer.innerHTML = 'A message here'
      messageContainer.setAttribute('data-test', 'display')
      document.body.appendChild(messageContainer)
    })
  </script>
</body>
</html>

As you can see we have HTML that displays the count along with some buttons. Then we added some in-line JavaScript at the bottom to implement the logic. We expect the page to:

Display when we visit it.
Increase the count displayed when we click the "Increment" button.
Display a message dialog when pressing the "Display Message" button.

Traditionally we would now manually click through this. We would host the website somewhere or locally run a web server to take a look. Then we would check that it opens and go through the requirements one by one. Let us make a computer do that for us.

The tools

We are going to use Jest, a framework for "Delightful JavaScript testing" as our testing framework. But you might as well use anything else that provides assertions. While we are going to use Jest here we will discuss things in a way so that we can also apply them to other tools.

Puppeteer will be our tool to automate the browser and interactions with it. Written by the Chromium Team Puppeteer provides an interface to the DevTools Protocol and can be used in combination with any browser supporting it. We will be using Puppeteer with the Chromium version that comes bundled with it.

For this post Jest was used in version 23.6.0 together with Puppeteer 1.11.0.

Apart from these, we will use live-server as a local web server. Using a local web server makes us independent of frameworks used. We will be able to test any website using this approach because no matter how you develop your site, be it React, Angular, Vue or some other flavor of the year framework, in the end, you will get a plain old website. And using this approach we can test them all.

I also decided that I would like ES6 module syntax in my code and included Babel to use import.

To get all of that into your project npm i --save jest puppeteer live-server.

Test environment Setup

Now that we have our toolbelt ready, it's time to dive into testing.

Earlier we said that before doing a manual test run we would host the website somewhere or run a local web server, we better do that in our test-suite before running our tests as well.

import puppeteer from 'puppeteer'
import { spawn } from 'child_process'

let browser
let serverProcess

// URL that the webserver will serve our website under.
const testUrl = 'http://localhost:8282'

beforeAll(async () => {
  // Wait until a local webserver is started to test against.
  await new Promise(resolve => {
    serverProcess = spawn('live-server', [`--port=${testUrl.split(':')[2]}`, '--no-browser'])
    // Better wait for a first response from the server to ensure the website is available.
    serverProcess.stdout.on('data', resolve)
  })

  browser = await puppeteer.launch({
    headless: false,
    slowMo: 100
  })
})

Here we remember the Puppeteer managed browser as well as a system process running our web server for all our tests by defining them as top-level variables. For now, we are telling Puppeteer to launch a browser that we can watch the tests being run in. We also slow this browser down using slowMo, later we will make this optional but it's always nice to see things happening when first trying them out.

Sadly we could not find a good web server solution that could be managed programmatically and live-server has an open issue about shutting down programmatically (see issue 196), thus we need to run it as a child process. It's basically like running the command live-server --port=8282 --no-browser in another terminal before doing your tests. We keep a reference so that we can later close the server.

beforeAll is a block that will execute once before any tests run. Thus it is the perfect place for our general setup. Here we use Puppeteer to launch a browser and wait for that to be done before using it to run our tests.

Of cause, we can't forget to shut all of this down once all our tests are done. For that Jest provides afterAll which will run once, after all, tests have finished. It is the perfect place to close our browser and web server.

afterAll(async () => {
  // Cleanup everything that is still running after all tests are through.
  serverProcess.kill()
  await browser.close()
})

While we start and stop the browser and web server for all tests we want to start with a clean, new tab for each individual test. We can achieve this using beforeEach and clean up in afterEach. We will also add a let page where we remember the browser as well so that our tests can access it.

beforeEach(async () => {
  // Get a new page for each test so that we start fresh.
  page = await browser.newPage()
})

afterEach(async () => {
  // Remember to close pages after tests.
  await page.close()
})

This will now open a new page for each test and store is always in the same variable named page that our tests have access to. Not that for your test-suite in your project you might or might not want to do it this way. For our website, it's nice to always start clean and we can achieve that by simply opening a new page for each test.

You might have noticed the extensive use of await within the code. We will continue to see that as the nature of all operations communicating with the browser are asynchronous. If you are unfamiliar with this Syntax, Mozilla has great docs on async and await.

Testing the site

Now it's finally time to write our first test!

We will wrap all our test for which we will use it blocks with the convention of "it should..." inside a describe block. As a quick recap: the idea behind this is to describe the part of an application tested and what it should do. With this we aim to get descriptive tests that precisely tell us what is broken, should they ever fail.

describe('Counter', () => {
  // "it" blocks go here.
})

As we would do in a manual test we will first make sure that we see the website. For that, we will check that a headline is present. We could also check to see if a specific text was in the headline but for us just having any headline will be good enough.

it('should have a headline', async () => {
  await page.goto(testUrl)
  await page.screenshot({ path: 'screens/basicRender.png' })
  const headlines = await page.$$('h1')

  expect(headlines.length).toBe(1)
})

This test navigates to the page that we want to test, finds all <h1> elements on it and expects to find exactly one h1 element on the page. We also decided to take a screenshot here. This can make it easier to check for what went wrong during a test or to debug your tests.

Congratulations, with your first test in place, it is time to run it. For that, we will follow nodes conventions and add a test script to our package.json. "test": "jest" will do the trick for us. Now you can run npm test and a browser will open and take a screenshot. Just make sure that you have a folder named screens that screenshots can be saved into.

Selecting Elements

The above verifies number 0 from the three things we want to check during our test. Next up is the counter itself and the displayed message.

✅ Display when we visit it.
Increase the count displayed when we click the "Increment" button.
Display a message dialog when pressing the "Display Message" button.

For the counter, we will want to check that it starts with 0 and then that it is incremented after we click a button. Checking the initial value is a simple matter of getting the contents of the <span> element that displays the count.

it('should initially have a count of 0', async () => {
  await page.goto(testUrl)
  const count = await page.$eval('[data-test="count-output"]', e => parseInt(e.innerHTML))
  await page.screenshot({ path: 'screens/initialCount.png' })

  expect(count).toBe(0)
})

Once again we visit the page and take a screenshot, but this time we also extract the initial value of the counter. Puppeteer provides many methods to interact with pages. Here we use $eval which takes an element selector that is used for a querySelect run on the page and as a second argument a function that gets the found element as it's input. The function passed to $eval is evaluated in the context of the page and its result becomes $evals return value. You can look up the full API in the official documentation.

Here we find an element identified by [data-test="count-output"] and then turn it's .innerHTML into an integer which we return. This exposes one crucial part of automated click-through testing. Our tests need to find the elements to interact with and run tests on. You could also solve this by searching for a certain text on the website but for sites with frequently changing text as well as for outputs we need a different way to select them.

A great approach is custom data-* attributes. Custom data attributes don't interfere with the rest of how web development works. They are here for exactly the case of an element needing some specific information that otherwise can't be provided.

The two alternatives would be classes and ids. While classes are meant to not be unique and thus are unusable to select exactly one thing, ids are meant for something else already. They are probably already used and are something you don't want to couple to your testing procedures, as it would block you from fully utilizing them for application development. Thus we use data attributes here to identify elements of interest for testing.

Interacting with Element

Once we can find elements on the page, we can also interact with them and for example, click a button to increment our counter.

it('should increment on click', async () => {
  await page.goto(testUrl)
  const incrementBtn = await page.$('[data-test="button-increment"]')
  const initialCount = await page.$eval('[data-test="count-output"]', e => parseInt(e.innerHTML))
  const expectedCount = initialCount + 1

  await incrementBtn.click()
  const newCount = await page.$eval('[data-test="count-output"]', e => parseInt(e.innerHTML))
  await page.screenshot({ path: 'screens/incrementedCount.png' })

  expect(newCount).toBe(expectedCount)
})

Here we get the initial count value and validate our expectation that clicking the "increment" button will increment the value by one. Using page.$ we can get an ElementHandle which provides us with certain convenience functions to interact with the selected element, like .click() to click it.

Within the repo for this blog post you can find another test for the counter that increments multiple times.

With that, we can check off another item from our testing list.

✅ Display when we visit it.
✅ Increase the count displayed when we click the "Increment" button.
Display a message dialog when pressing the "Display Message" button.

Dynamic DOM content

All we have left now, are elements that dynamically get added to the page and we can test those too, like the message in our example website. To do so we simply execute the action that will cause the element to be added to the page and then look for it.

it('should display a message', async () => {
  await page.goto(testUrl)
  await page.$eval('[data-test="button-display"]', e => e.click())
  await page.screenshot({ path: 'screens/messageDisplay.png' })
  const displays = await page.$$('[data-test="display"]')

  expect(displays.length).toBe(1)
})

This time we don't use Puppeteers build in support for click events but a piece of JavaScript evaluated in the context of our page to click the button e => e.click(). In the same way, you can pass any code to be evaluated in the context of the page.

That already gets us pretty far in testing websites. Don't forget that when testing your actual site you can probably take a lot of shortcuts by sending requests to your backend directly instead of doing everything all the time through your UI. Take a moment to reflect on what you want to test before you do so and do your best to de-couple tests from one another. One test failing shouldn't make all others fail as well. Ideally, you want one test to fail on its own to ease finding the issue. Following this, it's often good to setup (or seed) data and state for a test that could come from a backend by making calls to it directly (or using a mock backend). That way you can verify displaying a list even while creating items for it is currently failing its tests.

✅ Display when we visit it.
✅ Increase the count displayed when we click the "Increment" button.
✅ Display a message dialog when pressing the "Display Message" button.

Congratulations, all tests are green!

Making visualization optional

Earlier we hardcoded displaying a browser with a delay to watch our tests happen. Since we don't always want to do that, let's change it up to depend on an environment variable SHOW_BROWSER. Cases were we might not want to get the visual especially include pipelines for build and deploy of out application which usually runs on systems where we don't care about visualization but speed.

const browserConfig = process.env.SHOW_BROWSER ? {
  headless: false,
  slowMo: 100
} : {}
browser = await puppeteer.launch(browserConfig)

We will simply check whether or not the environment contains a variable named SHOW_BROWSERand depending on that show the browser or not. To now see the browser you could run env SHOW_BROWSER=t npm test to run tests with a browser. You can replace t with anything you want, as long as SHOW_BROWSER has any value at all. You could even turn this into an npm script "test:visual": "env SHOW_BROWSER=t npm test and run npm run test:visual to use it.

Limitations and good tests

As with every approach, this automated click-through testing isn't omnipotent, it comes with some limitations. For instance that Puppeteer basically only runs with Chrome. But as we discussed earlier automated click-through testing shouldn't replace manual QA runs but help them. Once we achieve fast development iterations which will give us fewer bugs, testing in one browser for each build will be just fine, as long as we do periodic checks in all browsers. Always assuming that developers check their code in multiple browsers anyways 😉

You might find it helpful to introduce timeouts in your tests. You will notice when to use them, as tests will fail when the default timeout isn't long enough simply hand your desired timeout as a second argument to it().

Wrapping up

Our goal is reached, we started testing a website using Jest and Puppeteer. We took a look at where automated click-through testing fits within the greater context of software development and then went on to write our first test cases. Check out the repo with our finished code.

Now go out and automate all your tests 🚀

Next episode: Automated testing using Cypress to visually debug automated tests (coming soon)

What are some great blogs to contribute to / collaborate with for webdevs?

Hendrik — Wed, 14 Nov 2018 09:58:36 +0000

Recently collaborations and contributions to and with blogs have motivated me a great deal to pick up my blogging game. I wrote an article for Contentful about API first CMS, produced a piece for Netlights make it new about polling in RxJS and then some.

Now I am on the look out for more collaboration / contribution programs out there on webdev related blogs, so my question to you today is:

Which webdev related blogs out there are looking for contributors?

If you worked with / for a blog, feel free to share you experience.
If you are looking for contributors to your blog, this is your time to advertise.
If you are searching for more places to post as well, this is your opportunity to get inspired.

Go based proxies for developing mobile websites on corporate WiFis

Hendrik — Tue, 02 Oct 2018 14:56:56 +0000

You might know this scenario:

We would really love to debug the web-app on an actual phone but they way our corporations WiFi is set up just won't allow it…

When networks become a show stopped for development.

If you do, stay tuned because in this blog post we will examine how we as developers can handle tightly secured WiFi Networks and still get all the connectivity we need.

We will set out by exploring a detailed, real world scenario and then explore a solution using hotspots and Go based proxies.

An exemplary situation

Recently we were developing a Capacitor based hybrid web application. To really debug that you run a webserver on your laptop and connect your phone via WiFi. Only trouble: on our corporations WiFi phones get isolated for security reasons.

Phones get isolated for security reasons!

Luckily the isolation problem can quite easily be solved by connecting your laptop with the internet via ethernet and using the Laptops WiFi to opening up a hotspot. Just make sure to secure that with a strong password!

Sadly, that cut our laptops off from the internal networks which we had to reach again by connecting our laptops to the companies VPN (jup, ethernet only went to the public internet). While the hotspot handed the internet connection through to the phone it doesn't do the same magic for the VPN connection. So, how do we now get this connection through to our phones?

We need the internal network because the phone needs to connect to the deployed test backend, available on the companies internal network that our laptops connect to using VPN.

Proxies to the rescue

Handing traffic around on a network is something we as web developers are familiar with. Often time you need an NGINX to hand requests from an entry URL to the right internal service running on a different machine (or to dockerize a web app). That is called a proxy.

As just mentioned we could setup an NGINX or similar on our laptops and use that as a proxy. And this does work. However it makes it more difficult to check the solution into version control. A better maintainable version is to write a simple proxy in your language of choice. We choose Go.

package main

import (
    "github.com/elazarl/goproxy"
    "log"
    "net/http"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true
    log.Fatal(http.ListenAndServe(":9090", proxy))
}

That is all the code you need to now run go run proxy.go and start a proxy on your machine.

Green light for our traffic thanks to proxies.

On your phone you now need to tell it to not only connect to your laptops hotspot but also use a proxy. For that you likely need your laptops IP within the hotspot (it will likely always assign itself the same one) as well as the port (here: 9090) to set that up.

With this you are now ready to finally do development on your phone.

Conclusion

Today we looked at how a simple proxy in Go can help us to develop in tightly secured corporate networks. With just a few lines of code and a bit of setup you can get going.

Now, go out and build something awesome ✨

Reflection

There is one obvious question here: Is this a good idea??!?

I am so glad that you asked. And in truth I am not sure. It surely feels like a hack and it is one. We are basically getting around a security measurement established for a good reason. I would encourage you to see the practices here as a short term solution and to pick up a discussion with you management and IT about a long term solution.

Any comments and ideas for better solutions are highly welcome and appreciated.

Dockerizing modern web apps

Hendrik — Wed, 19 Sep 2018 08:09:31 +0000

Most websites these days are Single Page Applications (SPA for short) where a single entry file handles all routes that a user might visit. Swept up in the ongoing trend of hosting in the cloud you might find yourself needing to “dockerize” your SPA. That is to say wrap it inside a Docker image and run it as a container.

These days everything is shipped in containers, even software.

In this post we will explore how we can get that done. We are going to build a simple SPA that just tells the user which route of our website they are currently visiting. That means you should be able to visit not only / but also any route you might think of, such as /unicorn or /rainbow. This SPA will be a super simple and hand made one but you can see it as representative for any complex React, Angular or Vue app you might want to deploy. Finally we are going to build our SPA into a Docker image and deploy that as a container.

We will run through all the basics of what we are doing. So whether you are seasoned with Docker and just can’t get that SPA to run on your cluster or you are a great web developer tasked to do this Docker thing, this post is for you.

The website

Our website will be super simple. Just a headline and a paragraph telling our user where they are by checking window.location. Below that we will offer links to navigate to a few routes.

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Simple SPA</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body style="font-family: sans-serif;">
  <h1>Welcome to a simple SPA</h1>

  <p>You are on: <span id="locationSpan"></span></p>

  <p>You could go to:</p>
  <ul>
    <li><a href="/">Home</a></li>
    <li><a href="/unicorn">Unicorn</a></li>
    <li><a href="/rainbow">Rainbow</a></li>
  </ul>

  <script>
    const span = document.querySelector('#locationSpan')
    span.innerHTML = window.location.pathname
  </script>
</body>
</html>

To test this locally you could initialize a package.json, install live-server and add a start script to your package.json "start": "live-server --entry-file=index.html".

npm init
npm i -D live-server
# Now add the script o your package.json before running:
npm start

Go ahead and click on a few of the links to move around or enter another path into your navigation bar.

Our super simply SPA telling a user where they are and allowing them to
navigate.

You might already notice that we are doing something to make our SPA work as we expect it to. We need to tell live-server to serve our index.html on all routes that it can't find a file for. We do this using --entry-file=index.html. Feel free to try running the ive-server without the — entry-file parameter and see what happens in that case.

Keep this flag in mind as we will need to do something equivalent for our dockerization.

See the website in action at dockerized-spa.now.sh. (Already dockerized and hosted on Now.sh.)

Naive Docker attempt

Docker is a system to create images which can then be run as containers. You can think of docker images as super lightweight Virtual Machines that can be run on many platforms (in this image containers are running VMs). The awesomeness of all this is that once you build a docker image and run it somewhere you will get the same thing running everywhere. Once we manage to build a Docker image locally that we can run as a container successfully we know that it will also run on AWS, GCP, Portainer or whatever else your company might be using.

Docker solves the infamous “it works on my machine” problem. Containers run the
same wherever you start them!

First you need a Dockerfile. Let us start out with a Naive-Dockerfile. Within it we will define the steps needed to create an image. In our case we just want an image that can serve websites and hold a copy of our Single Page Application.

FROM nginx

COPY index.html /usr/share/nginx/html

Here we base our image on the nginx image. NGINX is a simple and lightweight webserver that can serve our index.html. To enable this we copy our website into the folder that NGINX will be serving. Now that we have this, let’s build our image and run it.

Above we first create the Docker image using Docker build. Using the -f Flag we tell Docker which “Dockerfile” to use, which files holdes the configuration to build our image. The -t flag “tags” our Docker image. It gives it a name we can use to run it.

Using Docker run we then start the image as a container. By using -p we can specify a mapping for exposed ports, in this case that we want to reach the exposed port 80 at port 8888 on our local machines. So go ahead and open http://localhost:8888/ to checkout what we got.

Thats not how we imagined this would go…

With our “naive Docker” approach all but our entry route / display NGINXs 404 page.

docker rmi -f docker-spa

Lets clean up after ourselves using the Dockerrmi command and -f forcing it to remove our created image. Time to pull up our sleeves and get this fully working.

Empowering SPA capabilities

Remember how up above we needed to pass --entry-file=index.html to live-server in order for it to serve our index.html file for each route where it couldn’t find a file to serve? What we need now is the equivalent of this parameter for NGINX.

For that we will use an NGINX configuration and add it to our Docker image.

server {
    listen   80;
    listen   [::]:80 default ipv6only=on;

    root /usr/share/nginx/html;
    index index.html;

    server_name _; # all hostnames

    location / {
        try_files $uri /index.html;
    }
}

In the above config we tell NGINX to accept traffic on port 80 no matter the domain. Then we tell it to resolve paths ending in a slash as index.html and finally specify that for all routes it should check if there is a file and otherwise serve the index file.

After adding the above to our project we can now also COPY it into our Docker image to tell NGINX to use it.

FROM nginx

COPY nginx.config /etc/nginx/conf.d/default.conf
COPY index.html /usr/share/nginx/html

And once again you can test this locally using the same commands as above with our new Dockerfile. Remember to point your browser to http://localhost:8888/unicorn to see it in action.

docker rmi -f docker-spa

Clicking through the links you will see that now it does work. Each route you visit is now served by our Single Page Application we build at the beginning.

Bonus level — compiling your SPA inside Docker

Chances are your application isn’t just a single, static HTML file. In fact you probably have a quite modern and sophisticated toolchain involving TypeScript, Webpack, Parcel or similar to build your application. You could easily do this build step inside a Docker file.

# ---- Base Node ----
FROM node:alpine AS base
# Copy project file
COPY . .
# Build project
RUN npm run build

# ---- Prod ----
FROM nginx
# Copy needed files
COPY nginx.config /etc/nginx/conf.d/default.conf
COPY --from=base build /usr/share/nginx/html

The above uses a multistage Docker build. It first builds an image based on Node in which we run our build script and then builds an image as we did before but copying the compiled application form the
build image instead of our local file system where we run docker run from.

We can illustrate the usage of this with our simple SPA by adding a build script to our package.json.

"build": "rm -rf build && mkdir build && cp index.html build/index.html"

Go ahead and try it out.

docker rmi -f docker-spa

Take a look at all of this together in this repo illustrating how to host SPAs using Docker.

What a day

We built a Single Page Application, tested it locally, packaged it in a Docker image and finally enabled the Docker image to act as an SPA should, answering onall routes.

I hope you learned a thing or two today. Now, be brave, be bold, go out, apply your new found knowledge and host your application using Docker.

Repo with code

Live Demo on Now

How to add code highlighting to your Dev.to posts.

Hendrik — Mon, 10 Sep 2018 12:21:34 +0000

The simple truth of the matter is that:

const turorialFunction = (name) => {
  console.log(`Hello ${name}`)
}

does look way nicer than:

const turorialFunction = (name) => {
  console.log(`Hello ${name}`)
}

when writing a post here on Dev.to.

How to do it

Dev.tos posts are based on Markdown. Within Markdown we can use identation or so called Code Blocks to specify sections of code. The later ones are indicated using ```. Read more about this in this cheatsheet.

Using the three ` variant we can also specify a language for the code block. A lot of tooling build on top of Markdown utilized this characteristic to implement richer features. But the simplest of them is code highlighting. The above nicely colored code snippet is achieved by starting the code block with ```javascript.

The full example for the above would be:

```javascript
const turorialFunction = (name) => {
  console.log(`Hello ${name}`)
}
```

And if you are now wondering how the hell I got that to display:

<pre>
```javascript
const turorialFunction = (name) => {
  console.log(`Hello ${name}`)
}
```
</pre>

and the inline code is: <code>```</code>.

Here is where my explanations stop and your colorful posts start.

The list of supported languages is impressive, though not all encompassing (check comments).

Polling using RxJS

Hendrik — Mon, 10 Sep 2018 09:12:27 +0000

As observables are gaining more and more popularity in JavaScript we are looking to accomplish our everyday tasks using them and evaluating whether they are really worth all the hype. One task you might find yourself doing is polling the backend to know whether a longer running task has completed.

We will walk through an example of such a scenario and implement a solution using RxJS. On our way we will learn some basic operators for RxJS, and a few techniques as well as how to avoid a pitfall or two. At the end I will present a real-world example to show you how to implement what we learned in a specific scenario.

You should bring a basic understanding of Streams / Observables as well as solid foundations in JavaScript to enjoy this post. For the rest of this post I will treat Stream and Observable as interchangeable words for the same thing. While we will cover a lot of basic things they will mostly be RxJS specifics and less the basics about Streams. Should you be looking for a general introduction consider the gist title “The introduction to Reactive Programming you’ve been missing”.

Code for this post was tested using RxJS 6.2.0.

Scenario

Lets say we have a backend that exposes an endpoint /tasks/[taskId] which you can query to learn about the status of a specific task. It’s returning an object like such:

{
  // Whether the task is still running
  processing: boolean;
  // A unique ID for this task
  taskId: string;
}

Once we start polling we want to get the current state of this task twice a second and stop polling once processing === false.

Programmatic solution

To start out we are going to look at a programmatic solution for this problem.

    async pollUntilTaskFinished(taskId) {
      const fetchResponse = await fetch(`/tasks/${taskId}`)
      const responseObject = await fetchResponse.json()
      if (responseObject.processing) {
        setTimeout(() => pollUntilTaskFinished(taskId), 500)
      } else {
        pollingFinishedFor(taskId)
      }
    }

Here we simply invoke a new timeout every time the backend is still processing.

Using RxJS

Now we are going to achieve the same behavior using RxJS.

First of all we need something to emit an event every x time. RxJS provides two function for this:

interval
timer

While interval emits the first event after a given time and then continuously with the same interval, timer starts after a given time to emit events every x time. For our two updates per second we can start by using timer(0, 500). This will start firing events right of the bat and after that twice a second.

Let’s first see that in action by logging something to the console.

    import { timer } from 'rxjs'

    timer(0, 500)
      .subscribe(() => console.log('polling'))

You should see your console print “polling” twice a second now.

Take care to import from the correct package (rxjs or rxjs/operators). Sadly RxJS documentation might not be up to speed with the version you are using.

Next we want to turns these “ticks” into requests to our backend. We are going to use the same fetch from above but this time turn the promise into an Observable. Luckily RxJS provides convenient functions for this, namely from. Using this we can now create an Observable (or stream) representing a request to the backend on every tick and continue working with that.

    import { timer, from } from 'rxjs'
    import { map } from 'rxjs/operators'

    timer(0, 500)
      .pipe(from(fetch(`/tasks/${taskId}`)).pipe(map(response => response.json())))

.pipe is RxJS’s way to specify that a transform will now happen on the stream. By extracting operators into their own imports RxJS enables better treeshaking than an overloaded Observable implementation ever could, see this explanation for more context.

Pipe is RxJS’s wrapper around transforms that are applied to a Stream of event.

The result of this will be a stream of streams. Each emitted value will itself be an observable. To manage the mayhem we can pipe it through concatMap which will flatten all the Streams into a single one containing the nested values.

    import { timer, from } from 'rxjs'
    import { map, concatMap } from 'rxjs/operators'

    timer(0, 500)
      .pipe(concatMap(() => from(fetch(`/tasks/${taskId}`))
        .pipe(map(response => response.json())))
      )

Finish polling

Finally we really care about getting an event that tells us the backend finished processing, that our polling is done. We can achieve this by filtering for events where the backend is no longer processing and only taking the first one of those. By using take(1) we specify that we only care about a single (the first) event telling us that processing is finished. This will stop our polling once the backend is done processing the task.

    import { timer, from } from 'rxjs'
    import { map, concatMap, filter, take } from 'rxjs/operators'

    timer(0, 500)
      .pipe(concatMap(() => from(fetch(`/tasks/${taskId}`))
        .pipe(map(response => response.json())))
      )
      .pipe(filter(backendData => backendData.processing === false))
      .pipe(take(1))

Putting it all together

Now it’s time to put it all together and replace our function from up above using the new RxJS-based code. The final touch is to use subscribe at the end of our Stream to work with the single event our Stream emits.

    import { timer, from } from 'rxjs'
    import { map, concatMap, filter, take } from 'rxjs/operators'

    pollUntilTaskFinished(taskId) {
      timer(0, 500)
        .pipe(concatMap(() => from(fetch(`/tasks/${taskId}`))
          .pipe(map(response => response.json())))
        )
        .pipe(filter(backendData => backendData.processing === false))
        .pipe(take(1))
        .subscribe(() => pollingFinishedFor(taskId))
    }

You might not want to call a function once you are done though but use the output of your Observable to render your UI. Through the use of merge, which merges two streams together we can map our polling onto two states and use the output directly for our UI.

To achieve this we will merge our stream from above together with an initial value that we turn into a Stream using of.

    import { timer, from, merge, of } from 'rxjs'
    import { map, concatMap, filter, take } from 'rxjs/operators'

    const loadingEmoji = merge(
      of(true),
      timer(0, 500)
        .pipe(concatMap(() => from(fetch(`/tasks/${taskId}`))
          .pipe(map(response => response.json())))
        )
        .pipe(filter(backendData => backendData.processing === false))
      )
        .pipe(take(2))
        .pipe(map(processing => processing ? '⏳' : '✅'));

After we map the response from our backend onto the processing attribute using map, we can in turn map the result onto an emoji to display to our users.

A real world example

Theory is always nice but the real world usually poses a different challenge from a nicely written and contained tutorial. Let me present you with the solution to a problem we faced when building our knowledge about polling using RxJS.

The situation: We have an Angular application for which we use NGXS as a state manager. Similar to Redux it uses Actions to represent events changing the state.

As it turns out NGXS provides a stream of all Actions dispatched as an Observable we can hook into. Here is our final solution to poll the backend for processing states for each Document *that *gets added to the state and update the state once the backend is done processing.

    .actions$
      .pipe(ofActionSuccessful(AddDocument))
      .pipe(filter((action: AddDocument) => action.document.externalProcessingState === environment.documentStates.processing))
      .pipe(map((action: AddDocument) => action.document))
      .pipe(mergeMap((document: Document) => timer(environment.polling.startingOffset, environment.polling.interval)
         // Here we want a new stream per document add.
        .pipe(concatMap(() => from(this.backend.fetch(`/documents/${document.uuid}`))))
        .pipe(concatMap(response => from(response.json())))
        .pipe(filter((polledDocument: Document) => polledDocument.externalProcessingState !== environment.documentStates.processing))
        .pipe(take(1)))
      )
      .subscribe((polledDocument: Document) => {
                    this.store.dispatch(new AddDocument(polledDocument));
      });

A few notes:

environment is an Angular environment providing configuration for our application.
backend is a Service providing connection to our backend. It inserts a few required headers and such.
This uses TypeScript so polledDocument: Document describes a variable named “polledDocument” which follows the type “Document”.

A tricky thing here is that we need to create a new “polling Stream” per document getting added to our state. At first we tried pulling logic into a single level but that ended with us only being able to poll for a single document per page load as take(1) would block the Stream for all future pollings.

Wrapping up

Today we built our first polling logic using RxJS learning about this great library along the way. We also took a look at a real world example and saw how expressive it can make our code.

Now, go out and apply your newfound knowledge.

Other great resources

https://blog.strongbrew.io/rxjs-polling/

https://www.sitepoint.com/angular-rxjs-create-api-service-rest-backend/

https://www.learnrxjs.io/recipes/http-polling.html

Originally published on makeitnew.io on August 30 2018.