DEV Community: Quentin Ménoret

Browser extension - Deploy your extension

Quentin Ménoret — Mon, 22 Feb 2021 16:10:54 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This sixth blog post will focus on deploying your extension for Chrome and Firefox.

Packaging

In order to release an extension, you will need to provide a proper zip file to both the Chrome Web Store and the Firefox Addons Store. The zipped file should contain the code, the manifest.json and all assets you use (for instance the icon). One important thing to note: the manifest.json needs to be available at the root of the folder. Here is an example of a file structure you can zip:

And this is how to create such a zip file when the output of your build is in the built folder:

(cd built ; zip -r ../extension.zip .)

If your extension could run in the browser in development mode, there shouldn’t be any difficulties here.

Deploy for Chrome

First, you will need to create a developer account. Got to the Chrome Web Store. You will need to pay a 5$ fee to get started. Then you can create a new package.

Press the “New Item” button.

Select the Zip File you’ve built. Make sure to complete the description properly. An extension that is too light on description will get rejected. Add at least one screenshot of the extension behaviour (this is mandatory).

Now go to the Privacy tab.

Here, you will need to justify every permission you asked for. Make sure you did not require any permissions you did not need. Again, your extension will get rejected if you ask for too many permissions without justification.

Finally, at the bottom of the page, you need to certify that you don’t use or sell the user's data.

You should be ready to publish! Before it’s available, the extension will have to pass a review from Google. This can take hours or days depending on how many submissions they received. So make sure to provide as much information as possible for your submission so you don’t lose a couple days doing back and forth with them!

Deploy for Firefox

The process for Firefox is pretty similar. Go to the Firefox Addons store. Select “On this site” as the deployment means (you probably don’t want to handle this yourself), then upload your zip file.

The extension will pass an automated review at this point. Once it’s done, it will ask you if you use any kind of code obfuscation. Transpilation counts! For instance, as my extension is written using TypeScript, I have to say yes at this point. Using Webpack counts as well! If that’s the case for you too, you will need to upload your source code. Just zip it as well zip -r extension.zip extension/ and upload it.

Once you’ve selected your license, you should be good to go. After a review of your addon, it will be published on their store. Same as for Chrome, don’t leave anything to chance and provide as much information as possible so you won’t have to do back and forth with the review team.

We’re live!

And voilà, the extension should be ready to go live in both stores! 🎉
Congratulations if that was your first one!

Conclusion

We just learned how to properly deploy our extension for Chrome and Firefox, and how to do it the right way to avoid any rejections or delays. That's the end of this series! Let me know if anything is missing from your point of view and I'd be happy to add more content about Browser Extensions!

If you liked the series, feel free to follow me here if you want to check my next posts when they're out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Browser extension - Integrate your features securely

Quentin Ménoret — Tue, 16 Feb 2021 12:53:52 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This fifth blog post will focus on integrating your features securely into a website.

Short notice

For this part of the extension, we need to be able to import some external modules.

I won't explain how to setup webpack in this post, but if it's something you'd be interested in, drop a comment and I may write another post in the series about this.

What we are building

Here is a screenshot of what we'll have at the end of this post. We'll display safely the description of a Code Tour Step:

The challenge

In order to display a Code Tour to the user, here are the different steps that we implemented:

Find the list of tours
Get the content of each code tour
Redirect to the right page

Now that we are at the right place, with the content of the Code Tour, we need to load it into the page.

The content of a Code Tour is written using the Markdown language. There are ways to generate html out of Markdown, but we need to make sure that it's safe.

But let's first build a basic version!

The innocent version

First, let's add the text directly to the UI. Here is the code we had so far:

function forwardRequest(message) {
  return new Promise((resolve, reject) => {
    chrome.runtime.sendMessage(message, (response) => {
      if (!response) return reject(chrome.runtime.lastError);
      return resolve(response);
    });
  });
}

document.addEventListener("DOMContentLoaded", async () => {
  const urlParams = new URLSearchParams(window.location.search);
  const title = urlParams.get("code-tour-title");
  if (!title) return;

  const tour = await forwardRequest({ title });

  const step = urlParams.get("step");
  console.log(tour.steps[step]);
});

Now, instead of logging the state, let's add the description of the tour on the right line:

document.addEventListener("DOMContentLoaded", async () => {
  const urlParams = new URLSearchParams(window.location.search);
  const title = urlParams.get("code-tour-title");
  if (!title) return;

  const tour = await forwardRequest({ title });

  const step = urlParams.get("step");

  // We'll add the description on the right line
  const parent = document.querySelector(
    `#LC${tour.steps[step].line}.blob-code`
  );
  const section = document.createElement("div");

  const span = document.createElement("span");
  span.innerHTML = tour.steps[step].description;

  section.append(span);

  // A bit of style
  section.setAttribute(
    "style",
    `
    padding: 14px;
    margin: 14px;
    border: 1px lightgrey solid;
    background-color: white;
    border-radius: 1em;
    font-family: sans-serif;
    `
  );

  parent.append(section);
});

Transform Markdown to HTML

In order to transform the Markdown to HTML, we can use a generator such as showdown. It's really easy to use:

const showdown = require('showdown')
const converter = new showdown.Converter()
const htmlString = converter.makeHtml(yourMarkdownString)

Now we can use this as inner HTML for the section:

span.innerHTML = converter.makeHtml(tour.steps[step].description);

XSS injections with Markdown

Since our Markdown generation can write html, we can probably generate dangerous code as well. Consider the following Markdown code:

[XSS injection](javascript:alert('xss'))

Once you use a html generator (for instance showdown) with this code, you will get this html:

<p><a href="javascript:alert('xss')">XSS injection</a></p>

Try this in your browser, if you click it, it executes the JavaScript. Of course this is a very basic example, but there are a lot of more complex ways to exploit this. And since the Code Tours we load are untrusted code, we better protect our users!

Protecting from the XSS

There are a few libraries you can use to protect from xss. For instance the xss library on npm.

Using this, we are able to escape the dangerous bits of our HTML. Just use it this way:

filterXSS(converter.makeHtml(rawText))

Let's use it in our code:

span.innerHTML = filterXSS(converter.makeHtml(tour.steps[step].description));

Now our users are protected.

Conclusion

If there is one thing to remember from this post, it's this:

don't trust external inputs, even if it's text

As soon as you apply any kind of transformation to something you have no control over, there is a risk for it to be exploited. And trust me, it’s way worse when the code is in an extension that can be loaded on any website on your browser.

In the next post, we'll see how to deploy this extension on the different stores. Feel free to follow me here if you want to check the post when it's out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Browser extension - Manage your state in the Background script

Quentin Ménoret — Tue, 09 Feb 2021 15:16:14 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This fourth blog post will focus on keeping your state in a Background Script.

Our next feature

If you have followed the series until today, we have created an extension capable of requesting the content of the Code Tours. Now, we need to act on them.

What we'll build today is the ability to, from the list of tours, jump to the first step of any of them. And once we're there, we'll need to prepare to display the Code Tour.

Adding the link

Just so we're up to date, here is the state of the content script:

function forwardRequest(message) {
  return new Promise((resolve, reject) => {
    chrome.runtime.sendMessage(message, (response) => {
      if (!response) return reject(chrome.runtime.lastError)
      return resolve(response)
    })
  })
}

document.addEventListener("DOMContentLoaded", function(){
  Array.from(
      document.querySelectorAll('div[role=row] > div[role="rowheader"] > span > a').values(),
    ).map(
      async (parentElement) => {
        const title = parentElement.getAttribute('title')
        const href = parentElement.getAttribute('href')


        // Now we want to query the file content as a raw string.
        // In github, this means fetching the file using “raw” instead of “blob”
        const codeTourUrl = href.replace('blob', 'raw')

        // A Code Tour is a json object, we can use the fetch API to receive an object
        const content = await forwardRequest({ url: codeTourUrl })
        console.log(title, content)
  })
})

What we need to do now is, instead of logging the Code Tour content, to add a link to the right page. A basic Code Tour object looks like this:

{
  title: "The tour name",
  steps: [
    { 
      file: "manifest.json",
      line: 1,
      description: "The text that describes the step"
    }
  ]
}

We can definitely use it to generate the right link. For instance for this step, we want to go to:

https://github.com/<repository_owner>/<repository_name>/blob/main/manifest.json

For the sake of the exercise, let's consider that the base branch of a repository is always main.

Generating this URL is fairly easy. We take the URL we're at up to the repository name, then add blob/main/ and the filename.

In order to have the line focused on Github we can even add #L1 to the URL to focus the line 1.

function getLink(codeTour) {
  const currentRepo = /^\/([^/]+\/[^/]+)\//.exec(window.location.pathname)[1]
  return `/${currentRepo}/blob/main/${codeTour.steps[0].file}#L${codeTour.steps[0].line}`
}

Now let's display our link. Just replace the log in the code to add a link to the page:

const link = document.createElement("a");
link.setAttribute("href", getLink(content));
link.setAttribute("style", "padding: 5px;");
link.text = "Go!";
parentElement.parentNode.prepend(link);

It doesn't look pretty but it will work for now.

Content Scripts have the memories of goldfish

Now we're able to navigate to the first step of the tour. That's great. Unfortunately, as the page loads, the Content Script comes to life again... With no memory of why it's here!

We need it to know what to do.

The first thing we're going to do is to add a few query parameters to pass some information.

Let's add the current step and the name of the Code Tour.

function getLink(codeTour) {
  const currentRepo = /^\/([^/]+\/[^/]+)\//.exec(window.location.pathname)[1]
  return `/${currentRepo}/blob/main/${codeTour.steps[0].file}?step=0&code-tour-title=${codeTour.title}#L${codeTour.steps[0].line}`
}

This way, when the content script gets loaded, we know that we are currently playing a Code Tour, which one, and at which step we're at. For now we always return 0 as the step number but it'll be easy to make it generic later.

Keep the Code Tours in memory, in the Background Script

Now that we know which Code Tour we have to play, and which step we're at, let's save all the Code Tours in memory in the Background Script. Since the Background Script is already the one querying the data for the Content Script, we can just keep an index:


const tours = {}

chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    fetch(`https://github.com/${request.url}`)
         .then((response) => response.json())
         .then((codeTourContent) => {
           // Let's save the content of the code tour
           tours[codeTourContent.title] = codeTourContent
           sendResponse(codeTourContent)
         })
    return true
})

Retrieving the Code Tour from the Content Script

Now that we store the Code Tours, we need a way to be able to retrieve them from the Content Script. Let's do it this way:

if the message received by the Background Script contains a URL, it fetches the Code Tour
if it contains a title, it returns the cached version

We just need to add this line at the beginning of the function:

if (request.title) return sendResponse(tours[request.title])

Now, let query it from the content script:

function forwardRequest(message) {
  return new Promise((resolve, reject) => {
    chrome.runtime.sendMessage(message, (response) => {
      if (!response) return reject(chrome.runtime.lastError)
      return resolve(response)
    })
  })
}

document.addEventListener("DOMContentLoaded", async () => {
  const urlParams = new URLSearchParams(window.location.search)
  const title = urlParams.get('code-tour-title')
  if (!title) return

  const tour = await forwardRequest({ title })

  const step = urlParams.get('step')
  // Here we can retrieve the current step to be displayed
  console.log(tour.steps[step])
})

And voilà. We can now display the Code Tour, and continue linking to the next and previous steps.

Conclusion

We've just learned how to store the state of our extension in the Background Script so we can keep the Content Script aware of what it should do. That's a great milestone! With what we know now, you can definitely build a basic version of the Code Tour extension.

In the next post, we will have a look at how to make our Code Tour feature appear as though it was native to Github and how to securely inject the html needed. Feel free to follow me here if you want to check the next one when it's out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Browser extension - Secure http requests from content scripts

Quentin Ménoret — Fri, 05 Feb 2021 11:42:25 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This third blog post will focus on sending cross origin http requests from a Content script.

The problem

Remember that code tour extension I was mentioning just before? Let's try to implement the first bit. What we want is to be able to retrieve the content of code tours (json files) so we can process them. You’d think it should be as simple as using fetch with the right URL, right? Well, it’s a bit more complicated than that.

A naive approach

First, we need to find all code tours definition files. Once we’re on the .tours directory in Github, we can use a selector:

// Wait for the page to be ready
document.addEventListener("DOMContentLoaded", function(){
  Array.from(
      // Find all links to tour files
      document.querySelectorAll('div[role=row] > div[role="rowheader"] > span > a').values(),
    ).map(
      async (parentElement) => {
        const title = parentElement.getAttribute('title')
        const href = parentElement.getAttribute('href')


        // Now we want to query the file content as a raw string.
        // In Github, this means fetching the file using “raw” instead of “blob”
        const codeTourUrl = href.replace('blob', 'raw')

        // A code tour is a json object, we can use the fetch API to receive an object
        const content = await fetch(codeTourUrl).then((response) => response.json())
        console.log(title, content)
  })
})

Copy this code in the Content Script file you've created in this post:

Browser extension - Setup and test

Quentin Ménoret ・ Jan 29 '21

#javascript #typescript #extension

If you refresh the extension, it will get executed in any page you load. Go to the .tours folder of a project and open the console. You will see either one of these:

On Chrome, the list of Code Tours and their contents will be loaded
On Firefox, you will see an error and the http request will be denied

Forwarding the requests to the background

With this code, we should be able to retrieve all the code tours definition files. Unfortunately, Github will redirect us during this request. It won’t work on Firefox as this is a Cross Origin Request.

In general, you shouldn’t use fetch from your content script. The right way to process this will be to forward the query to the background script, so it can perform the request for you:

Let’s create a function allowing us to forward requests to the background script:

function forwardRequest(message) {
  return new Promise((resolve, reject) => {
    chrome.runtime.sendMessage(message, (response) => {
      if (!response) return reject(chrome.runtime.lastError)
      return resolve(response)
    })
  })
}

In the background script, we need to handle this request:

chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    // sendResponse can be used to send back a result to the content script
    fetch(`https://github.com/${request.url}`)
         .then((response) => response.json())
         .then((codeTourContent) => sendResponse(codeTourContent))
    // As we will reply asynchronously to the request, we need to tell chrome to wait for our response
    return true
})

⚠️ Be careful with how you forward requests to your background script. You should make sure you won’t trigger requests to random URLs as this is fully authenticated. In this case we should verify the URL (make sure it’s safe) and only access GET requests. As you can see here we enforce the usage of the Github domain, but you should probably include more checks. If possible, build the URL yourself in the background script to be surer to target the exact resource you need.

Now that we have this code, we can replace the call to fetch in the background script with a call to forwardRequest.

Array.from(
    document.querySelectorAll('div[role=row] > div[role="rowheader"] > span > a').values(),
  ).map(
    async (parentElement) => {
        const title = parentElement.getAttribute('title')
        const href = parentElement.getAttribute('href')
        const codeTourUrl = href.replace('blob', 'raw')

        // Now forward request will behave like fetch
        const content = await forwardRequest({ url: codeTourUrl })
        console.log(title, content)
})

This still won’t work on Firefox, as it will prevent the background script from making any requests to random hostnames. In order to fix this, you will need to ask the permission to query github.com and render.githubusercontent.com (github redirects you to this domain when querying a raw file) from the background script.

Just add this into your manifest file:

{
    "permissions": ["https://render.githubusercontent.com/*", "https://github.com/*"]
}

Reload the extension, and it works!

Conclusion

We just deep dived into one really important aspect of Browser Extension: securely retrieving data from content script. In the next post, we’ll use the data to build a feature! Feel free to follow me here if you want to check the next one when it's out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Browser extension - Setup and test

Quentin Ménoret — Fri, 29 Jan 2021 15:56:35 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This second blog post will focus on how to set up the environment to develop a Browser Extension.

The manifest file

All we’ve seen in the previous post must now be bundled together to be loaded into the browser. In order to do so, you will need to have a folder (let’s call it extension) containing the different scripts to be loaded, an icon for your extension, and a Manifest file to tell the browser what to load, and where to find it. The manifest for our extension looks like this:

{
  "name": "Code tours Github",
  "version": "0.0.1",
  "description": "Allows to run code tours in your browser",
  "manifest_version": 2,
  "minimum_chrome_version": "60",
  "background": {
    "scripts": ["background.js"]
  },
  "permissions": ["https://render.githubusercontent.com/*", "https://github.com/*"],
  "icons": {
    "128": "code-tour.png"
  },
  "content_scripts": [
    {
      "run_at": "document_start",
      "matches": ["https://github.com/*/*"],
      "js": ["github.js"]
    }
  ]
}

Let’s deep dive into the different properties.

Describing your extension

The properties name, description and version are here to describe your extension. The name will be displayed on the Chrome Web Store (or Firefox Addons store) and when you hover over the extension icon. The description will also be displayed in the Store by default. You should be sure to name your extension properly as a poor description is a cause for rejection (we'll see more about submitting the extension in a next blog post).

The version should only be incremented when you release a new version.

A nice logo!

The icon property should be the path to a nice image you want to show in the extension toolbar of the browser. It will also be shown in the Store page so make sure to have a decent resolution for it (128x128 will do).

Starting your scripts

The backgrounds and content_scripts sections list the different scripts you want to load. Just give it a relative path to the script from the manifest file. For the Content Scripts, you also need to explicitly state in which pages it should be included via the matches (or exclude_matches) properties.

Permissions

Depending on the actions you want to perform from your extension, you will need to require some permissions. You should list them in your manifest file. For instance, you could require:

bookmarks to be able to manage the bookmarks of the browser
nativeMessaging if you want to be able to start external programs
Any URL if you want too be able to query those with authentication (you can always do a simple GET without authentication, but if you want to query content where you need the cookies, you will need to declare it)

You could also dynamically require them from your background script, but I would recommend to put all the permissions that are required for your extension to work in the manifest file, so your user can’t reject them (either they accept, or won’t install the extension).

One important note: do not ask for more permissions than you need, you will have to justify all of them during the review process when you submit your extension, and this is yet another cause for rejection.

A minimal working extension

In order to demonstrate, just create a folder called extension with 3 files in it.

manifest.json:

{
  "name": "My extension",
  "version": "0.0.1",
  "description": "A small test!",
  "manifest_version": 2,
  "minimum_chrome_version": "60",
  "background": {
    "scripts": ["background.js"]
  },
  "content_scripts": [
    {
      "run_at": "document_start",
      "matches": ["http://*/*", "https://*/*"],
      "js": ["content_script.js"]
    }
  ]
}

content_script.js:

console.log('content script loaded')

background.js:

console.log('background script loaded')

Now let’s load it in the browser!

Loading the extension

Now that we have a folder with our content scripts, background, and manifest, we can load it into our Browser.

Chrome

For chrome, just go to chrome://extensions. First, activate the developer mode in the top right corner.

The select ”Load Unpacked” on the top left.

It will prompt a file picker. Select the extension folder (the one containing the manifest.json file). Your extension is loaded and can be reloaded from this same page.

Note that if you have a background script associated with your extension, you will have a “Inspect views: background page” link. This opens the dev tools linked to your background script, allowing you to check the logs.

Now every time you will edit a file and want to reload the extension, you can click the reload button:

Firefox

For Firefox, it’s as easy. Just go to the about:debugging page, click on “This Firefox”:

Then click “Load temporary addon”:

In the same way as for Chrome you will be prompted for a file. Select the manifest file (not the folder) and your extension will be loaded.

You will have access to a “Inspect button” granting you access to the devtools of the background page.

The result

In both cases, when inspecting the background script, you will see “background script loaded” appear in the console, and on every page you visit, you will see “content script loaded”, as if it was part of the website code.

Going further

In my case, I went with using TypeScript, which required me to transpile my code to generate what we described before, and use Webpack to generate the different script files. You can find the resulting code here.

To get started faster, you can find a lot of resources with ready to use repositories, such as chrome-extension-typescript-starter for TypeScript. It’s a good way to get started quickly.

Conclusion

We just deep dived into how to setup and test a browser extension. In the next post, we’ll start implementing some real features! Feel free to follow me here if you want to check the next one when it's out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Browser Extension - Extension architecture

Quentin Ménoret — Mon, 25 Jan 2021 13:18:25 +0000

I just published a new extension on Chrome and Firefox that allows anyone to run Code Tours from the Github UI. More information about Code Tours and the extension in this blog post.

Article No Longer Available

I thought it would be nice to write a series about how you could do exactly the same, step by step.

This first post will be about what an extension looks like in terms of architecture:
🛠 What are the main components? Let’s talk Background Script, Content Script and Injected Scripts
💡 Which one should you use for which purpose?
🤝 How to communicate between them?

The extension architecture

Background script

The heart of an extension is called a Background Script. This is simply a JavaScript script that runs in the browser as soon as the extension is loaded. It is not linked to any specific tab and can’t interact with their content. Its goal is to contain the state of your extension, and to implement all external calls or chrome API manipulation.

For instance, it’s from a Background Script that you would be able to focus a specific tab, or interact with the history or bookmarks of the browser. We won’t use such features for this extension, but we will need it to perform HTTP calls, and maintain the state of the application.

Content Script

A content script runs in the context of a single tab. The associated javascript code will get loaded when the tab opens, and stop as soon as it closes. You will be able to specify in which tabs your content scripts should be loaded (based on the URL for instance). We’ll see that in the next post.

A content script has access to most of the page context, including the DOM (you can add elements or alter the page from there). Note that because of security reasons, even though you have access to a window object, it won’t be the same instance as the one available to the code running in the website you are injecting to. This is meant to prevent you from accessing functions defined by another extension, or by the web page. If you need access to the window object, you would need to use injected scripts (the next part of this post!).

In order for the Content Script to communicate with a Background Script, you should use the Chrome Messaging API.

Sending a message is fairly easy and can be achieved with a single call:

chrome.runtime.sendMessage({message: "content"}, function(response) {
  console.log('message has been received: ', response);
});

Injected Script

An injected script is just a bit of code you inject directly into the page, using DOM manipulation (for instance using a <script> div tag). In this context, the script will have access to the same window object as the website code, but won’t have access to the extension API.

For the injected script to communicate with a Content Script, you will use the postMessage API:

window.postMessage({ message: 'from the injected script' }, "*");

A side note

None of these components are required to build a Browser extension. You could just have a content script, or a background script. But to have an injected script, you actually need a content script (since it is the one doing the injection).

There won’t be a need for an injected script in this tutorial, but you might need it if you were to expose a public API to the website (by creating methods on the window object for instance).

What it looks like

Conclusion

This was a short introduction to how a browser extension works. Next time, we’ll see how to bundle it all together so you can test the extension! Feel free to follow me here if you want to check the next post when it's out:

Quentin MénoretFollow

Engineering Manager @Prismic – Mostly writing about TypeScript / JavaScript

Photo by Ricardo Gomez Angel on Unsplash

Run Code Tours without leaving Github

Quentin Ménoret — Mon, 18 Jan 2021 16:23:53 +0000

tl;dr: I made this: https://github.com/doctolib/code-tours-github

Code tours

Code tours are an amazing tool to help people discover your codebase. It's simple and brilliant!

Basically, you will be able to walk people through the code, highlighting specific parts of the code with your explication. It's great to explain how a feature work, or how you apply certain patterns.

Here is what a Code Tour looks like:

Only in VSCode?

Unfortunately, this extension is built for VSCode. They recently added support for CodeSpace as well, but that's basically it. Since I'm not using VSCode, I felt like I had to do something about it!

Introducing the Code Tours Github extension

So I decided to reimplement part of the features directly in the Browser. This way, Code Tours are available to everyone!

Here is what a tour looks like:

Of course there is still a lot to do to have as many features as the original extension, but I'll keep working on it.

You can check the repository here, and install the extension for Chrome or Firefox.

Let me know what you think!

Testing Windows installers with Jest

Quentin Ménoret — Mon, 11 Jan 2021 16:47:09 +0000

At Doctolib, we've been building native software for a few years now, which we install directly on doctor's computers to improve their Doctolib experience.

Of course, we do write unit and integration tests to make sure that the software does what it is supposed to. But sometimes, it's not enough.

Once, we made a major change to a feature and for the new code to work properly, the updater needed to write a new line in the software's config file. Of course we manually tested that a fresh install would write this line. "That should be enough right?", we thought until the support calls started coming in.

Apparently, this tiny line was not written when the installer ran in update mode. Luckily we only had a few beta testers at that time, so that the error had only a low impact.

Today we can't afford to make a mistake like that again.

So what do we do now?

We release new versions of our native software weekly, and we need the updates to be thoroughly tested.

The risk of making a mistake with an installer is that it might completely corrupt your software. For example, an update could cause a crash on startup, which would stop users from even being able to subsequently update to a version with a fix. Imagine this happening on tens of thousands of computers at the same time. It would be impossible to fix, unless you call all 20k users one after another.

On the other hand, testing installers and executables is really hard. Most companies actually test those manually like we used to do. To automate this process, you cannot write unit tests: you have to completely install your software, validate that everything works properly and that all files are copied in the right place. On top, you need to do this for all supported operating systems and architectural varieties.

This is why we built several tools and processes to allow us to run end-to-end tests for installers and executables on our CI. In the rest of this post I will walk you through the process of creating such a safety net.

Setup

Stateless environment

Before you start, you will need to set up a stateless environment where you can run your tests. We chose Azure Devops because it allows us to run tests on Windows, Linux and Mac but there are other CI providers that offer the same service.

It is important that the environment is stateless because installers have the tendency to persist a lot of things which are a pain to cleanup after each run (files, registry, permissions…). If you don't isolate the test runs, you might have unexpected behaviours, or worse, false positives.

Just picture a case in which your installer needs to write a critical registry key. Unfortunately, your feature is broken. If your environment is not cleaning up the registry, the next run will be green, no matter how broken your code might be, since the registry key was written in a previous test run.

Headless testing

Most installers have a tendency to ask you stuff. Unfortunately, it's harder to simulate user inputs on an installer UI compared to a browser. So you'll need to skip that part.

With a standard NSIS installer (which is what we use at Doctolib) this means running the installer with the /S argument (silent mode - runs the installer without a UI). There are operations that can block the installer in silent mode, leaving you waiting forever. For those situations we came up with two solutions:

Use IfSilent to explicitly skip blocking operations.
Use registry keys instead of user inputs. Inside of your installer you can check for the presence of registry keys (which is never the case in production) and use that value in the tests instead of prompting the user.

Once your installer is able to run in non-interactive mode, you can start the actual testing.

File system and registry checks

Now we can talk about ensuring the installer works fine. Let's run it:

import { exec } from "child_process"
await promisify(exec)(`"${installerPath}" /S`)

What you want to assert is that your files get copied to the right place. This is very easy to do with Jest. Use snapshot testing:

try {
  // You can snapshot test the content of all the folders you 
  // install files in, such as your AppData folder
  const entries = await fsPromises.readdir(folder)
  expect(entries).toMatchSnapshot('entries in folder')
} catch (err) {
  expect('no folder').toMatchSnapshot('entries in folder')
}

You could also take a snapshot of the content of the registry if you save any important values there:

import { list } from 'regedit'
const values = await promisify(list)(yourKey)[yourKey]
expect(values).toMatchSnapshot()

Same thing for the content of any text/config files you write. And since values are sometimes dynamic, you'll want to use property matchers on the snapshot's file content after parsing:

const config = ini.parse(fs.readFileSync('./config.ini'))
expect(config).toMatchSnapshot({
  my_section: {
    my_value: jest.stringMatching(/expected_value/)
  }
})

Testing the binary architecture

When building native software for Windows, you are often building for either a 32-bit or 64-bit architecture. It is critical to get this right, if you don't want your process to crash.

If you were to ship a 64 bit exe for a 32 bit computer, or mixed architectures for your EXE and DLLs, your program would most likely not work, or even crash. This is why we built windows-binary-architecture. With this module you can easily assert the architecture of your binaries (exe or dll):

const arch = await getTargetArchitecture(filePath)
expect(arch).toBe('I386')

Ensuring binaries are signed

Unless you want your users to see those annoying SmartScreen messages, you will have to sign your binaries.
There are probably a lot of files to sign, and it's a tedious process to check them manually.

Don't worry, we got your back here too! We wrote a small JavaScript tool to assert that a binary is properly signed: authenticode. It is pretty easy to use:

const signature = await getAuthenticode(filePath)
expect(signature.Status).toBe(SignatureStatus.Valid)

Actually starting the thing

The last thing you might want to do is to actually start your process. This highly depends on the type and size of software you are building. The following test cases might not be worth it if your software takes 15 minutes to boot up.

But if you can afford it, there are three things you can easily get out of this:

Is the process crashing when starting?

return new Promise((resolve, reject) => {
  const process = spawn(yourProcessPath)
  process.on('error', () => reject())
  process.stdout.on('data', () => {
    // Maybe if the process starts writing on stdout
    // it means it is working? Depends on your software!
    resolve(process)
  })
})

Is the process writing anything to stdout / stderr that you should be worried about?

const process = spawn(yourProcessPath)
let stdout = ''
let stderr = ''
process.on('error', () => reject())
process.stdout.on('data', data => {
  stdout += data
})
process.stderr.on('data', data => {
  stderr += data
})
// You will need to implement custom logic to know when your process
// is "ready"
await processInitOver()
expect(stdout).toMatchSnapshot()
expect(stderr).toMatchSnapshot()

Is the process loading the DLLs you expect it to load?

It's pretty easy to test this using the listDlls executable:

listDllsOutput = (await exec(./Listdlls.exe ${processName} /accepteula`)).stdout
expect(listDllsOutput).toMatchSnapshot()

The Mac and Linux case

We focused a lot on Windows tests here but you can implement the same thing for Linux and Mac as well! If you want to achieve this, feel free to look at our jest-os-detection module, which allows you to run the same test suite on a different OS.

Conclusion

At Doctolib, we enforce that every single piece of software we build is tested. We extend this beyond pure feature testing (eg. unit or integration tests), and also test the output of our automated build processes. This includes installers and binaries.

These installer tests have protected us several times from serious issues that could have otherwise reached production and would have had significant consequences. A few months ago, we refactored our build pipelines, and almost published unsigned binaries. Our tests saved us.

If you like tests as much as we do, don't hesitate to subscribe to the docto-tech-life newsletter to receive our weekly selection of technical content!

My 2021 wishlist for DEV

Quentin Ménoret — Tue, 05 Jan 2021 07:24:47 +0000

I discovered DEV in 2020. I know, I'm late to the party. But I really love the platform. It's a great place to share content and discover or learn new things.

In the last few days, I started thinking: what are the things that I would add to make this platform even better? Here are the 2 that kept popping up in my mind!

Allow for alt/caption on cover image

I think a lot of people here use images from Unsplash or other sources as cover for their posts. I know I do. It's important in this case to give credit back to the people that produced the image.

Currently, everyone of my posts ends with the credit for the image. I'd love to be able to set it when I upload the image so it would be displayed naturally.

This would incentive people to do it (no way to forget about it!), and makes it more visible for readers.

It would also be awesome to have an option to set an alt text, for better accessibility!

Allow better reviews before publishing

I know that it's already possible to share a draft post before publishing it, but to be honest I am not using this feature. When I want a review, I either use a Google Doc, or a Github pull request. This way, people can make comment, or edits, related to a specific part of the post. It makes writing so much easier!

To be clear, it is fine as it is, I just copy and paste the markdown before publishing. But this can lead to some errors (you sometimes need to do your edits twice when you're as badly organised as I am).

Maybe a really nice option would be to be able to link a Github repository to your DEV account. Then every Markdown file that gets committed is automatically published. Or maybe just created as a draft. This way, no need to implement custom review process in DEV, we can just use PRs!

What are yours?

Of course I have a few other ideas: improving the post stats to get insights on post performance over time for example. But these two are the ones that I think about most!

I'd love to hear what are your thoughts and crazy idea about DEV!

Photo by Dariusz Sankowski on Unsplash

Mocks and Spies with Jest

Quentin Ménoret — Thu, 31 Dec 2020 16:53:32 +0000

When writing unit test, you often have to mock some functions. It can be to make your tests deterministic, or to assert that a specific function gets called. Let's imagine you are trying to assess that your function calls the right API using fetch.

async function getUser(id) {
  return fetch(`/users/${id}`)
}

Let's see the different options we have when trying to assert this.

Using jest.fn

The first way to achieve it is to use directly jest.fn and replace the fetch function.

describe('my test', () => {
  it('calls the right route', async () => {
    // Create an empty mock function which just returns nothing
    const mockedFetch = jest.fn()
    // Set the global fetch to be this function
    global.fetch = mockedFetch
    await getUser(id)
    expect(mockedFetch).toHaveBeenCalledWith('/users/12')
  })
}

This will work, but there are some drawbacks. The biggest one being: you will have to manually keep a reference to the actual fetch method, and put it back in place after the test. If this fails, it will impact all the other tests in your test suite.

You could do it this way:

describe('my test', () => {
  const realFetch = global.fetch
  beforeAll(() => {
    global.fetch = jest.fn()
  })

  afterAll(() => {
    global.fetch = realFetch
  })
}

Using jest.spyOn

A better approach in this case would be to use a spy. Spies have all the features of a mock function, but leave you with more flexibility. More importantly, Jest will handle the cleaning of mocks for you. Here is what the test look like using spies:

describe('my test', () => {
  it('calls the right route', async () => {
    jest.spyOn(global, 'fetch')
    await getUser(id)
    expect(global.fetch).toHaveBeenCalledWith('/users/12')
  })
}

There are two things to remember about spies:

You still need to tell Jest to forget about the mock between tests using mockClear, mockReset or mockRestore (more on that later)
By default it just spies on the function and does not prevent the original code to be executed.

If we wanted to fix these 2 behaviours, the test would look like this:

describe('my test', () => {
  beforeEach(() => {
    jest.restoreAllMocks()
  })

  it('calls the right route', async () => {
    jest.spyOn(global, 'fetch').mockReturnValue({})
    await getUser(id)
    expect(global.fetch).toHaveBeenCalledWith('/users/12')
  })
}

Here we prevented the actual call to fetch using mockReturnValue (mockImplementation can be used too), and we restore all existing mocks to their initial state before every test run.

Clear, reset and restore

When clearing mocks, you have 3 possible functions you can call:

mockClear - clearing a mock means clearing the history of calls that already have been stored in the mock. It can be useful if you want to start counting calls after some point in your test.
mockReset - reseting a mock returns the mock to a fresh state, just like if you just called spyOn on the function. All mocked implementation or return value will be forgotten. Of course it also implies everything mockClear implies.
mockRestore - Restoring the function actually removes the mock, and restore the original implementation.

All these functions can be used in two different ways:

Directly on a mocked function: myMockedFunction.mockClear
Globally to affect all existing mocks you've created: jest. clearAllMocks()

Using jest.mock

Another approach to mocking with Jest is to use jest.mock. It allows you to mock entirely a module. For instance:

// Here I am mocking the 'os' module entirely
// It now only exposes one function: hostname
// Which always returns 'my-computer'
jest.mock('os', () => {
  return { hostname: () => 'my-computer' }
})

As you can see, you can use the second parameter to provide a factory so that the import return something. If you don't importing the module will just return an empty object.

It is also possible to write this factory in a specific file. For instance if you were to import the file src/my-module.js, and wanted a specific factory for it in every test, you can create a file named src/__mocks__/my-module.js. Whatever this file exports if what will be imported when calling jest.mock('src/my-module') without a factory provided.

So what do I use now?

As much as possible, try to go with the spyOn version.

Using jest.fn directly have a few use cases, for instance when passing a mocked callback to a function.

jest.mock is powerful, but I mostly use it to prevent loading a specific module (like something that needs binaries extensions, or produces side effects). I also use it when I need to mock a lot of functions from an external module at once.

Photo by Tobias Tullius on Unsplash

Advent of code - Day 25

Quentin Ménoret — Sat, 26 Dec 2020 14:04:33 +0000

Are you participating in the Advent of code this year?

If you don't know what the advent of code is, it's a website where you'll find a daily challenge (every day it gets harder). It's a really fun event, you should participate!

I try to solve the exercises using either JavaScript or TypeScript and will share my solutions daily (with one day delay so no one can cheat!). I only share the solution for the second part.

And it's the last day of the event! For this one, Advent of Code was merciful. It only takes a few minutes to complete with a simple loop.

All in all this year challenges were great. A few were really complex and interesting, I really encourage you to try it out! But I'm honestly happy that it's over so I can relax and enjoy some time of. Completing all the challenges the day of the release takes time and commitment, I'm really proud I managed to keep up with the game!

Here is my solution for day #25:

function transform(subjectNumber: number, value: number) {
  return (value * subjectNumber) % 20201227
}

function getLoopSize(id: number) {
  let current = 7
  let iterations = 1
  while (current !== id) {
    current = transform(7, current)
    iterations++
  }
  return iterations
}

const card = getLoopSize(10943862)
console.log({ card })
const door = getLoopSize(12721030)
console.log({ door })

let resultFromDoor = 12721030
for (let i = 0; i < card - 1; i++) {
  resultFromDoor = transform(12721030, resultFromDoor)
}

console.log(resultFromDoor)

let resultFromCard = 10943862
for (let i = 0; i < door - 1; i++) {
  resultFromCard = transform(10943862, resultFromCard)
}

console.log(resultFromCard)

Feel free to share your solution in the comments!

Photo by Markus Spiske on Unsplash

Advent of code - Day 24

Quentin Ménoret — Fri, 25 Dec 2020 12:43:19 +0000

Are you participating in the Advent of code this year?

If you don't know what the advent of code is, it's a website where you'll find a daily challenge (every day it gets harder). It's a really fun event, you should participate!

I try to solve the exercises using either JavaScript or TypeScript and will share my solutions daily (with one day delay so no one can cheat!). I only share the solution for the second part.

Another game of life, but with hexagons this time! The trick here was to make sure you only process the black tiles and their adjacent tiles. Just drop all white tiles, they are useless! Once I figured that out, performance dropped to a ok level (this code runs in about 10 seconds).

Here is my solution for day #24:

interface Position {
  e: number
  se: number
  sw: number
  w: number
  nw: number
  ne: number
}

export function extract(str: string): (keyof Position)[] {
  if (str.length === 0) return []
  if (['e', 'w'].includes(str[0])) return [str[0] as keyof Position, ...extract(str.slice(1))]
  return [str.slice(0, 2) as keyof Position, ...extract(str.slice(2))]
}

function count(position: (keyof Position)[]): Position {
  return position.reduce(
    (acc, value) => {
      acc[value] += 1
      return acc
    },
    {
      e: 0,
      se: 0,
      sw: 0,
      w: 0,
      nw: 0,
      ne: 0,
    },
  )
}

interface Tile {
  position: Position
  state: State
}

enum State {
  Black,
  White,
}

const opposition: Record<keyof Position, keyof Position> = {
  e: 'w',
  w: 'e',
  nw: 'se',
  se: 'nw',
  sw: 'ne',
  ne: 'sw',
}

const complementaries: Record<keyof Position, (keyof Position)[]> = {
  e: ['se', 'ne'],
  w: ['sw', 'nw'],

  se: ['sw', 'e'],
  sw: ['se', 'w'],
  nw: ['ne', 'w'],
  ne: ['nw', 'e'],
}

function rationalize(position: Position): Position {
  for (let key of (Object.keys(opposition) as unknown) as (keyof Position)[]) {
    if (position[key] < 0) {
      return rationalize({
        ...position,
        [key]: 0,
        [opposition[key]]: position[opposition[key]] - position[key],
      })
    }
    if (position[key] > 0 && position[opposition[key]] > 0) {
      return rationalize({
        ...position,
        [key]: position[key] - 1,
        [opposition[key]]: position[opposition[key]] - 1,
      })
    }
  }
  for (let key of (Object.keys(complementaries) as unknown) as (keyof Position)[]) {
    if (complementaries[key].every((comp) => position[comp] > 0)) {
      return rationalize({
        ...position,
        [complementaries[key][0]]: position[complementaries[key][0]] - 1,
        [complementaries[key][1]]: position[complementaries[key][1]] - 1,
        [key]: position[key] + 1,
      })
    }
  }
  return position
}

function print(position: Position): string {
  return `${position.e},${position.se},${position.sw},${position.w},${position.nw},${position.ne}`
}

export function moveTo(newCenter: Position, tile: Position): Position {
  return rationalize(
    ((Object.keys(tile) as unknown) as (keyof Position)[]).reduce(
      (acc: Position, key) => {
        acc[key] = tile[key] - newCenter[key]
        return acc
      },
      {
        e: 0,
        se: 0,
        sw: 0,
        w: 0,
        nw: 0,
        ne: 0,
      },
    ),
  )
}

interface PotentialTile {
  adjacentBlacks: number
  position: Position
}

export function areAdjacent(tile: Tile, adjacent: Tile) {
  const newPosition = moveTo(tile.position, adjacent.position)
  const keys = (Object.keys(newPosition) as unknown) as (keyof Position)[]
  return keys.filter((key) => newPosition[key] === 1).length === 1 && keys.every((key) => newPosition[key] <= 1)
}

export function findAdjacent(tile: Tile): Position[] {
  return ((Object.keys(tile.position) as unknown) as (keyof Position)[]).map((key) => {
    return {
      ...tile.position,
      [key]: tile.position[key] + 1,
    }
  })
}

function runTurn(allTiles: Tile[]) {
  const potentialNewTiles: Record<string, PotentialTile> = {}
  const processed = new Set()

  const indexedTiles = allTiles.reduce((acc: Record<string, Tile>, v) => {
    acc[print(v.position)] = v
    return acc
  }, {})

  const existingTiles = allTiles
    .map((tile) => {
      processed.add(print(tile.position))

      const adjacents = findAdjacent(tile)
      adjacents.forEach((adjacentPosition) => {
        const actualPosition = rationalize(adjacentPosition)
        const printedPosition = print(actualPosition)
        if (processed.has(printedPosition)) return
        potentialNewTiles[printedPosition] = potentialNewTiles[printedPosition] || {
          position: actualPosition,
          adjacentBlacks: 0,
        }
        potentialNewTiles[printedPosition].adjacentBlacks += 1
      })

      const numberOfBlackAdjacents = adjacents
        .map((position) => indexedTiles[print(rationalize(position))])
        .filter(Boolean)
        .filter((adjacent) => adjacent.state === State.Black).length

      return {
        ...tile,
        state: numberOfBlackAdjacents === 1 || numberOfBlackAdjacents === 2 ? State.Black : State.White,
      }
    })
    .filter((tile) => tile.state === State.Black)

  const newTiles = Object.keys(potentialNewTiles)
    .filter((key) => !processed.has(key))
    .filter((key) => potentialNewTiles[key].adjacentBlacks === 2)
    .map((key) => {
      return {
        position: potentialNewTiles[key].position,
        state: State.Black,
      }
    })

  return [...existingTiles, ...newTiles]
}

const positions = input
  .split('\n')
  .map(extract)
  .map(count)
  .map(rationalize)
  .reduce((acc: Record<string, Tile>, value) => {
    acc[print(value)] = acc[print(value)] || { position: value, state: State.White }
    acc[print(value)].state = acc[print(value)].state === State.Black ? State.White : State.Black
    return acc
  }, {})

const tiles = Object.keys(positions)
  .map((key) => positions[key])
  .filter((tile) => tile.state === State.Black)
let currentTiles = tiles

for (let i = 0; i < 101; i++) {
  console.log(`Turn: ${i}: ${currentTiles.filter((t) => t.state === State.Black).length}`)
  currentTiles = runTurn(currentTiles)
}

Feel free to share your solution in the comments!

Photo by Markus Spiske on Unsplash