DEV Community: Carolyn Stransky

Broccoli, bots, and brains! The 3 projects where I added the Hacktoberfest topic.

Carolyn Stransky — Sat, 03 Oct 2020 21:57:25 +0000

For 2020, I decided to opt-in and add the Hacktoberfest topic to three of my personal repositories 🎃

The goal is to provide an easy-going (and hopefully sort of fun) experience for folks looking to get involved or make their first open-source contribution!

This post contains links to the projects and a short description of what they're all about.

Hopeful COVID-19 Bot

carolstran / hopeful-covid-bot

🤖 Little bot that tweets uplifting COVID-19 news

The code behind my little Twitter bot @hopefulcovidbot (I promise I will relaunch it this weekend). The best way to contribute is to add an uplifting news article for the bot to tweet out.

🤖 🖤 💉 🙌🏼 🧼

Seeing Sprouts

carolstran / seeing-sprouts

🥗 Browser extension that replaces images with brussels sprouts

A browser extension that replaces all images on a web page with photos of various veggies. There are step-by-step instructions in the README describing how to add support for more vegetables, nutrient-dense fruits, and other veggie-like things people enjoy!

🥗 🥔 🥑 🍆 📷

So you want to learn GraphQL?

carolstran / so-you-want-to-learn-graphql

✏️ 'Choose Your Own Adventure' but with GraphQL

Imagine 'Choose your own adventure' but for learning GraphQL. It's super messy because I always say I'm going to work on it and never do - so any support adding or sorting resources is appreciated. The frontend section and anything marked TODO are especially lacking. Check out the Contributors Guide for more details.

✏️ 🧠 💻 🗺 💫

Feel free to comment on this post or open an issue on the repositories if you have any questions!

Do you enjoy any of these projects or find them useful? If yes, please consider buying me a coffee so I can continue to make things like this ☕️

Community Bonding with the GraphQL Foundation for Google Season of Docs

Carolyn Stransky — Wed, 16 Sep 2020 11:44:03 +0000

Google Season of Docs is a 3-month program that connects experienced technical writers with an open-source community. I was accepted into the 2020 program to work with the GraphQL Foundation. This post is part of a series intended to give insight to technical writers considering future Season of Docs cohorts.

The first month of Season of Docs is known as the community bonding phase. According to Google, this period is intended to help technical writers prepare to contribute to the mentoring organization once the program is in full swing.

During this phase, I was employed full-time and dedicated about 10 hours per week to the GraphQL Foundation.

Some organizations will have clear expectations for what they’d like you to complete during this phase. Others, like the GraphQL Foundation, give you more freedom to decide how you want to get to know the community. I chose the following activities with the help of my assigned mentor, Ivan Goncharov.

Attempted to figure out the organization structure

Part of the community bonding phase is becoming familiar with existing practices and processes. I’ve been in the GraphQL ecosystem for a while - creating side projects, giving talks, and even using it at work sometimes. My previous involvement was actually a large part of why I was selected.

But getting involved with the Foundation has been really different than using the specification. There’s a hierarchy and internal politics. GraphQL's history (having been incubated at Facebook) also influences the existing setup. So figuring out how decisions are made and who controls which aspect of GraphQL is confusing.

I tried to decipher all of this through meetings with my mentor, watching recordings of the past working group meetings, and creating my own mind map of the Foundation structure. Once I’m more confident with it, I’ll share it in a future post.

Triaged issues and pull requests

Within the first week, I was made a member of the GraphQL organization on GitHub and given triage access to graphql.github.io (the repository that contains the base code for the GraphQL website and documentation).

For those who aren’t familiar with triaging, this can include:

Creating new labels to appropriately tag issues
Reviewing code on open pull requests
Reproducing and confirming any reported bugs
Closing anything that’s irrelevant or duplicates

If I ran into process gaps, I opened my own issues or flagged it to Ivan. Many of these issues and pull requests hadn’t been touched in years, so it took time to figure out why and how to move forward.

While my triage access was limited to the documentation repository, I started becoming familiar with the other repositories within the GraphQL organization. I focused on repositories that would be relevant to my project, like graphql-spec and an existing faq. Along the way, I’d note anything that might be useful for me in the future.

Helped with the Gatsby migration

Earlier this year, there were discussions about migrating the GraphQL website from custom scripts to Gatsby/Next.js. This migration was approved before I became involved with the Foundation and now it’s a priority to get this done.

Due to my limited availability during this phase, the help I offered was limited to reviewing the existing pull request by saihaj and planning the steps for the migration. We settled on merging it into its own gatsby branch and then open separate pull requests to fix the remaining issues. Which is part of what I’m currently working on!

Want to help us out with this migration? Check out and claim one of the remaining issues.

Aligned on expectations for my project

During this phase, I also had one meeting with my mentor to discuss the new Frequently Asked Questions (FAQ) page that I’ll build. We didn’t dig deep, but we did refine the project plan, agree on initial milestones, and figured out what potential blockers I may face.

What happens next?

Now that the bonding is over, it’s time to get to work on documentation development 🚀

If you’re interested, check out my blog post for the GraphQL Foundation.

Did you find this helpful or useful? If yes, please consider buying me a coffee so I can continue to write posts like this ☕️

Creating a digital CV in Markdown on GitHub

Carolyn Stransky — Tue, 08 Sep 2020 21:19:53 +0000

In early August, I announced on Twitter that I was looking for my next role. My tweet included a link to my digital CV and ever since I’ve received a lot of questions about how I built it. So here’s a short guide!

This guide is based on a Twitter thread that I saw back in 2017. Unfortunately, I can’t remember who posted it… but if it was you, please know that I really appreciate you!

⚠️ Prerequisites:

GitHub account
Comfort with Markdown

✅ Steps:

Create a new public repository on GitHub
Enable GitHub Pages for your repository
Select a theme (design template) for your CV
Write your CV in Markdown
Enjoy 💅🏼
Optional: Set up a custom domain
Show me your CV 💌

Create a new public repository on GitHub

Everything we need for our digital CV will live in a repository on GitHub.

To start, create a new repository without a template.

Choose a name (I called my cv) and include a description if you want. Make sure it's public.

This digital CV will be hosted with GitHub Pages, which requires your repository to be public.

And finally, don't initialize it with anything.

While you won't be importing an existing repository, it helps makes the process less complicated to start with a completely blank repository. You'll see why later!

Enable GitHub Pages for your repository

Your digital CV will be hosted with GitHub Pages. GitHub Pages allows you to host personal and project pages directly from a GitHub repository. It’s great because it’s free and the functionality is built-in to GitHub so there’s no need for an additional account or a complex build pipeline.

To enable GitHub Pages for your CV, go to your repository’s Settings page.

Scroll until you find the GitHub Pages section. There are two subsections here: Source and Theme Chooser.

Under Source, it should say, “GitHub Pages is currently disabled. You must first add content to your repository before you can publish a GitHub Pages site.” Because your repository is empty, you have no branches and you’re unable to select a source. That’s ok! This will be taken care of momentarily.

The subsection you want is Theme Chooser. Here, you’ll select a design template for your site. This will then create a gh-pages branch and publish your site with your selected theme from that branch.

All of the options are Jekyll themes. If you don’t know Jekyll, that’s ok. This guide will walk you through the necessary configuration. Honestly, I have no idea how Jekyll works beyond GitHub Pages templates.

Select a theme (design template) for your CV

CVs are supposed to give future employers an idea of who you are - so choose a theme that speaks to you! For this guide, we’ll continue with the “Minimal” theme as an example because that’s the one I use for my CV.

Once you select your theme, you’ll be prompted to edit a new index.md file.

Before you dive into that, there are a few things that happened in the background that you should know about…

A new gh-pages branch was created (as mentioned earlier).
This index.md file was created.
Jekyll theme was set to “Minimal”.

GitHub Pages confirms your theme selection by creating a file called _config.yml. By default, this file will only contain a theme field with your selection.

Here’s what the _config.yml will look like if you select the "Minimal" theme:

theme: jekyll-theme-minimal

For my CV, I also added title and description fields:

theme: jekyll-theme-minimal
title: 💻✒️📄🌟
description: Carolyn Stransky's CV

You can check out all of the configuration possibilities in the Jekyll documentation.

Write your CV in Markdown

Alright, back to the generated file. This is where all of the content for your CV will be.

First, you should rename that index.md file to README.md. This is optional, but having it as a README allows people to see your CV’s content immediately when they visit your repository. This is especially useful if you decide to pin your CV repository to your GitHub profile.

If there’s no index.md present, then your README will work automatically with GitHub Pages - no additional configuration required.

Then, you can write your CV ✍🏻

You’ll write and style your content using Markdown and then it will be rendered according to your selected theme.

For tips on how to showcase your skills effectively, I’d recommend Monica Lent’s article: 7 software developer resume tips to help you stand out. If you’re looking for inspiration on how to style your content in Markdown, you can also check out my digital CV.

Once you’re satisfied, commit your changes to the gh-pages branch.

Enjoy your new digital CV

It might take a few minutes to go live - but your fresh, new, digital CV should now be available on the interweb 🎉

In most cases, your page will be hosted at the following url:

<github-username>.github.io/<repository-name>

You can always confirm that your CV is published and what the url is by navigating back to the GitHub Pages section in your repository’s settings.

If your digital CV isn’t updating after pushing a commit and waiting a few minutes, try doing a hard refresh.

Optional: Set up a custom domain

Custom domains can help your CV stand out - or alternatively, use your digital CV as a personal website.

Admittedly I’ve never set up a custom domain for my CV, but there are step-by-step instructions for how to do this in the GitHub documentation.

If you’re not sure where to get domains, I like Namecheap.

Show me your CV

If you followed this guide and built a digital CV, I’d love to see it! Tweet me a link @carolstran 💖

Did you find this helpful or useful? If yes, please consider buying me a coffee so I can continue to write posts like this ☕️

Create a mock server for any REST API

Carolyn Stransky — Thu, 06 Aug 2020 15:38:41 +0000

If you make a REST API call from your codebase then there's a good chance that, at some point, you'll want to write a test for that integration. Instead of calling the real API, it might make sense to use a mock. But manually stubbing data can be time-consuming and doesn't lead to a high level of confidence.

That's where this tutorial comes in.

In this tutorial, you'll learn how to create a mock server for any REST API. This mock server will generate dynamic, type-based response data that you can use in your tests. You'll do this using the HTTP Mocking Toolkit (HMT), a tool that mocks HTTP APIs for use in automated and exploratory testing.

As an example, this tutorial uses the Studio Ghibli API, which catalogs the worlds created by Japanese anime studio Studio Ghibli. Why Studio Ghibli? Because they RULE!

✅ Steps:

Collect recordings of your API's traffic
Build an OpenAPI specification from your recordings
Mock server traffic using your specification

⚠️ Prerequisites:

Python 3.6+ installed (no previous Python knowledge is necessary, it's only for installing HMT)
Familiarity with REST APIs and how they work
Comfort using the command-line

💻 References:

There's a GitHub repository that contains a completed Studio Ghibli mock server. You can use it as a reference for this tutorial.

Initial setup

Before building the mock server, you first need to create a project directory. This will house your recordings and specifications (explained later in this tutorial).

Let's do this in the terminal and name it studio-ghibli-mock-server:

mkdir studio-ghibli-mock-server
cd studio-ghibli-mock-server

(Optional) Create a virtual environment

Using a virtual environment can help keep your packages organized and ensure that you have the correct Python version. For this example, you'll name yours .aichienv based on the location of the new Studio Ghibli theme park.

First, run the following script to set up a venv directory:

python3 -m venv .aichienv

Then, launch your virtual environment:

source .aichienv/bin/activate

Finally, check your Python version with python --version. It should be version 3.6 or higher.

Whenever you're done, you can terminate the virtual environment by running: deactivate

Install HMT

You can install HMT on the command line using pip, a package installer for Python:

pip install hmt

For HMT to install and run properly, it requires Python version 3.6+

Check your HMT version with hmt --version to make sure the command-line interface (CLI) is installed.

1. Collect recordings of your API's traffic

The first step towards creating a mock server is to collect recordings of your API's traffic. That way, our future mock server will know what types of data to mimic.

Using the record mode, the HMT CLI can be used to record HTTP API traffic in a format that you can later build into a specification.

Start HMT

To start an HMT server that will record your API traffic, use the hmt record command:

hmt record

This starts HMT as a reverse proxy on the default port of 8000.

To stop HMT without losing any of your data, type Ctrl + C or another kill command.

Keep this running. Then, in another terminal window, you can use HMT as a proxy with curl.

By default, HMT uses path routing to intercept HTTP API calls. Path routing appends the URL you wish to call (https://ghibliapi.herokuapp.com/) to the end of the URL of the recording server (http://localhost:8000/):

curl http://localhost:8000/https://ghibliapi.herokuapp.com/

HMT will automatically make an API call using the URL in the path - in this case, https://ghibliapi.herokuapp.com/ - and return the response from the called API.

Run API-specific curl commands

In order for HMT to later mock your API, it needs to understand what kinds of endpoints exist and the associated responses to expect. One way to feed it that information is by using curl.

There's no minimum or maximum number of calls you should make in order for HMT to accurately record your API traffic. You have to feel it out. A general rule is that you want to record every critical endpoint at least once.

With the Studio Ghibli API, there's five main endpoints: Films, People, Locations, Species, and Vehicles. You'll want to make sure to call each of those.

The example in the reference repository alternates between calling the root endpoint and calling a specific field within that endpoint. For thoroughness, here's exactly what was tested:

# Films
curl http://localhost:8000/https://ghibliapi.herokuapp.com/films

# Film - Castle in the Sky
curl http://localhost:8000/https://ghibliapi.herokuapp.com/films/2baf70d1-42bb-4437-b551-e5fed5a87abe

# People
curl http://localhost:8000/https://ghibliapi.herokuapp.com/people

# Person - Granny (My Neighbor Totoro)
curl http://localhost:8000/https://ghibliapi.herokuapp.com/people/08ffbce4-7f94-476a-95bc-76d3c3969c19

# Locations
curl http://localhost:8000/https://ghibliapi.herokuapp.com/locations

# Location - Irontown (Princess Mononoke)
curl http://localhost:8000/https://ghibliapi.herokuapp.com/locations/11014596-71b0-4b3e-b8c0-1c4b15f28b9a

# Species
curl http://localhost:8000/https://ghibliapi.herokuapp.com/species

# Species - Cats
curl http://localhost:8000/https://ghibliapi.herokuapp.com/species/603428ba-8a86-4b0b-a9f1-65df6abef3d3

# Vehicles
curl http://localhost:8000/https://ghibliapi.herokuapp.com/vehicles

# Vehicle - Sōsuke's Boat (Ponyo on the Cliff by the Sea)
curl http://localhost:8000/https://ghibliapi.herokuapp.com/vehicles/923d70c9-8f15-4972-ad53-0128b261d628

Check the recordings file

Running hmt record will automatically generate two directories: logs and specs.

By default, HMT records all traffic to the logs directory in a file called {hostname}-recordings.jsonl. {hostname} refers to your API's host URL and in this case, that's ghibliapi.herokuapp.com.

How it works is that HMT takes the recorded server traffic and serializes them as JSON objects in the http-types format. Then, this is written to a JSON Lines file.

This specific formatting and file type is required for HMT to be able to build an OpenAPI specification in the next step.

2. Build an OpenAPI specification from your recordings

Using the HMT CLI, you can build an OpenAPI specification that describes how your API works. This is built from that .jsonl file generated by the record command.

On the command-line, you'll run hmt build with the --input-file flag followed by the path to your recordings file:

hmt build --input-file logs/ghibliapi.herokuapp.com-recordings.jsonl

This uses the default build command. To see the other options, run hmt build --help or check out the HMT build documentation.

By default, HMT builds a new OpenAPI specification (called openapi.json) in the specs directory from earlier. Once you've generated the specification, you can use that to create a mock server.

3. Mock server traffic using your specification

Now you can use your openapi.json and a different HMT command to create a mock server.

All it takes is hmt mock followed by the path to the directory or file where your OpenAPI specification is (for this example, that's specs/):

hmt mock specs/

Keep this running. Then, in another terminal window, make another curl request:

curl http://localhost:8000/https://ghibliapi.herokuapp.com/people/08ffbce4-7f94-476a-95bc-76d3c3969c19

Yes, you read that right - it's almost identical to the command that you ran earlier after launching hmt record.

This is the same command we ran in the second step for Granny from My Neighbor Totoro. But our server won't return Granny's exact data. Instead, it'll return the same character response fields from the Studio Ghibli API with mock data for the values based on their types.

Here's an example of what this response could look like:

{
  "id": "Bradley Joseph",
  "name": "Jeremy Vasquez", 
  "gender": "Jodi Francis", 
  "age": "Karen Booth", 
  "eye_color": "Andre Mcclure", 
  "hair_color": "Brandon Freeman", 
  "films": [
    "Stephanie Campbell", 
    "Paul Young", 
    "Catherine Gonzalez DDS", 
    "Charles Dean", 
    "Jackson Smith", 
    "Alexander Manning", 
    "Anthony Reed"
  ], 
  "species": "Jasmine Davis",
  "url": "David Olson"
  }

Because this data is generated, there's a good chance that yours will be different if you're following along. Although the strings almost always seem to be random names 😅

There you have it, your very own mock server 🎉

Conclusion

This tutorial ran through the entire HMT flow: Collect, build, and mock. First, you collected data by recording server traffic from the Studio Ghibli API. Then, you built an OpenAPI specification based on that data. Finally, you used that specification to spin up a Studio Ghibli mock server.

If you want to see the code for the final mock server, check out the example repository on GitHub.

There are many customization options for your mock server - such as custom callback scripts, format options, and response types. You can find out more in the HMT documentation.

HMT is actively maintained by our team at Meeshkan. It's also open-source 🎉 If you run into any problems with the product or have questions, please open an issue on GitHub.

Finding and fixing bugs in your tests with fast-check

Carolyn Stransky — Mon, 27 Jul 2020 13:59:20 +0000

This post assumes that you have a solid understanding of JavaScript. You should also know the basics of property-based testing and the fast-check framework.

Recently, we published Property-based testing for JavaScript developers. In it, we introduced an issue to show the limitations of example-based testing. Throughout the rest of the guide, we provided examples of how to use fast-check and other property-based testing principles.

But we didn't offer a fix to that initial testing bug.

We've been told that this is unsatisfying, so consider this post our redemption.

💻 The tests featured in this post are available in the corresponding GitHub repository.

Summary of the bug in our test

If you're already familiar with the referenced issue, skip to the solution.

Imagine you have an input where users write in a price - but this input is type="text" rather than type="number". So you need to create a function (getNumber) that converts the input string into a number and an accompanying test:

const getNumber = inputString => {
  const numberFromInputString = Number(inputString)
  return numberFromInputString
}

test("turns input string into a number", () => {
  expect(getNumber("59.99")).toBe(59.99)
})

This passes 🎉

Now imagine your website also operates in Germany where the meaning of commas and decimals in numbers are switched (i.e. $1,200.99 in English would be $1.200,99 in German).

So you add another test case to address this:

test("turns input string into a number", () => {
  // English test case
  expect(getNumber("59.99")).toBe(59.99)
  // German test case
  expect(getNumber("19,95")).toBe(19.95)
})

But when you run the test, you hit an error:

expect(received).toBe(expected) // Object.is equality

Expected: 19.95
Received: NaN

Assuming that your website and input are equipped to handle various locales and localized strings, this would be a limitation of your tests - not a bug in your app.

There's something that can help though: Property-based testing.

Fixing this bug with fast-check

To solve this problem, let's use fast-check, a JavaScript framework for generative test cases. Our goal is to test that our function turns any valid input string into a number, regardless of locale.

First, we need to pick the property (or properties) to test for in this situation. There a couple of options:

It works with strings that aren't formatted.
It works with strings that are formatted using one of the accepted locales (in this case, that would be German).
A combination of the two.

We'll tackle the latter: A combination of formatted and non-formatted strings.

Spoiler alert: Final solution

Here's a peek at what our test will look like by the end of this post:

test("turns any valid input string into a number, regardless of locale", () => {
  fc.assert(
    fc.property(
      fc.float(),
      fc.option(fc.constantFrom("de-DE", "fr-FR")),
      (testFloat, locale) => {
        const fixedFloat = parseFloat(testFloat.toFixed(2))

        const floatString = locale
          ? new Intl.NumberFormat(locale).format(fixedFloat)
          : fixedFloat.toString()

        expect(getNumber(floatString)).toBe(fixedFloat)
      }
    )
  )
})

This is a lot all at once. So let's go through it step-by-step.

Setting up our test

To begin, we need to set up our test:

test("turns any valid input string into a number, regardless of locale", () => {
  fc.assert(
    fc.property(
      // Everything else will go here!
  )
})

Note: This test uses the assert and property fast-check functions. If you need a refresher, these are both covered in our JavaScript property-based testing guide.

Applying our properties

To test a combination of formatted and non-formatted strings, we need to pass two different arbitraries to property.

The first will be float to generate the floating-point numbers:

fc.float()

The second is also a combination of sorts. To start, we need to write a constantFrom function that takes in multiple, equally probable values. We'll use this for our German locale: de-DE.

fc.constantFrom(
  "de-DE"
  /* This is where you'd pass another locale */
)

Then, we'll wrap constantFrom in the option function. With option, the complete arbitrary will return either null (a stand-in for the English default) or our values from constantFrom.

fc.option(fc.constantFrom("de-DE"))

Prices in these locales only go to the second decimal place and we want to embody this in our test.

To do this, we'll take the testFloat value and use JavaScript's built-in toFixed method on it. The 2 value indicates how many decimal places we want.

This method returns a string, but we need a number for our final assertion. So we'll wrap it in another built-in function, parseFloat, and name this variable fixedFloat.

const fixedFloat = parseFloat(testFloat.toFixed(2))

Finally, we'll pass two arguments to the property callback: testFloat (representing each generated floating-point number) and locale (representing the generated locale from our option and constantFrom combination).

Altogether, our property function will look like this so far:

fc.property(
  fc.float(),
  fc.option(fc.constantFrom("de-DE")),
  (testFloat, locale) => {
    const fixedFloat = parseFloat(testFloat.toFixed(2))
    // More to come!
  }
)

Processing the strings

Because we're testing both formatted and non-formatted strings, we need a way to differentiate these two string types.

We'll create a variable called floatString that checks if there is a locale value.

If locale !== null, then we need to create a localized string based on the generated float.

To do this, we'll use the Intl.NumberFormat constructor and pass in our locale value. On top, we'll chain on the format method with our fixedFloat value from earlier.

new Intl.NumberFormat(locale).format(fixedFloat)

Note: This constructor should mimic existing functionality in your app.

If locale is null, then we'll use the built-in toString method on fixedFloat, which returns it as a string.

fixedFloat.toString()

The resulting floatString variable:

const floatString = locale
  ? new Intl.NumberFormat(locale).format(fixedFloat)
  : fixedFloat.toString()

Adding your assertion

At last, we need the most crucial part of any test: The assertion.

The following expect statement indicates that when you pass the floatString (formatted or not) to the getNumber function, you expect it to equal the fixedFloat number value:

expect(getNumber(floatString)).toBe(fixedFloat)

Putting it all together

After all that, you're back to that final result:

test("turns any valid input string into a number, regardless of locale", () => {
  fc.assert(
    fc.property(
      fc.float(),
      fc.option(fc.constantFrom("de-DE", "fr-FR")),
      (testFloat, locale) => {
        const fixedFloat = parseFloat(testFloat.toFixed(2))

        const floatString = locale
          ? new Intl.NumberFormat(locale).format(fixedFloat)
          : fixedFloat.toString()

        expect(getNumber(floatString)).toBe(fixedFloat)
      }
    )
  )
})

Reminder: You can check out the code in our accompanying GitHub repository.

Now your tests are more thorough and resilient 🎉

Special thanks

Nicolas Dubien, the creator of fast-check, was the first to flag this issue in the original guide. He also helped develop the solution. You're the best, Nicolas!

Have a better solution? Comment below or open a pull request.

Testing apps with third-party integrations

Carolyn Stransky — Fri, 24 Jul 2020 15:31:32 +0000

This post assumes that you're familiar with APIs and have a basic understanding of testing. It's also based on a presentation by Mike Solomon.

Whether you’re working with the latest technologies or fighting against legacy code, you'll likely need to make a call to an API. And even more likely, this will be an API that you don't maintain.

Reliance on external services is becoming more prevalent in our engineering ecosystem. As a result, we need cost-effective, secure, and reliable ways to test how our app interacts with these services.

In this post, we'll examine some strategies and available tools for testing apps that use third-party integrations.

TL;DR: If you don't have time to write tests but want to be sure your apps are covered, use an automated testing tool like Meeshkan.

Why should you test third-party integrations?

The way we build apps has changed a lot in the past two decades. At the beginning of the 2000s, when you ran into a problem or needed a specific service, you had to build it yourself.

Now, there's an API for almost everything. SendGrid for emails, Auth0 for authentication, Stripe for payments, and the list goes on. The Programmable Web API Directory has over 23,000 different APIs available for use. Even MySpace has its own set of realtime REST APIs.

But using a third-party integration that you didn't build and don't maintain is risky.

You need to be sure that the calls you're making to these APIs are reliable and serving the expected data. That's why you should test them.

There are a few different options for how to test these external services. Let’s start with the most common and walk-through some of the issues they can cause.

Three common (but misguided) ways to test APIs

1. Do nothing

Don't write tests - or comment them out when you write them and they don't work.

If this is you, don't be ashamed.

There's an entire economy around helping you solve issues that arise when you don't test your code. There are also products designed to test your apps when you understand the importance but don't want to manually write any (more on that later in this post).

2. Using the real APIs

We have another blog post that highlights the three primary reasons to not use the real APIs in tests. It covers security, integrity, and speed.

But there are also lots of unfortunate scenarios that can happen when you use the real APIs in a CI/CD pipeline.

During the research for his initial talk, Mike Solomon collected horror stories from developers who have used real APIs in their tests. Here are two of those.

“I want you back”
One company accidentally embedded a Twilio API key in their CI environment. This caused some poor person in Kansas to receive dozens of text messages a day saying “I want you back <3”.

OR 1=1
Another testing engineer created a fake username called OR 1=1 (a common SQL injection). This username got sent repeatedly to a production API and the organization was banned from using the service.

3. Reverse engineering the API

This option is the least dangerous, but the most time-consuming.

When you create a reverse-engineered version of the API you're testing, it comes at a cost. For instance, sinking more time into writing a test script than you did the feature you're testing. Or the stubbed API data ends up being longer than the code you're testing. It's also difficult to maintain unless you endlessly monitor the changelog of that external API.

None of these options are great, so let's look at more reliable strategies for testing your third-party integrations.

Better ways to test third-party integrations

It'd be cruel to leave you without a solution, so outlined below are two cost-effective, secure options for your tests.

Write your tests with a mocking library

A mock is a substitute for a service that mimics the original functionality. Or at least the functionality that you're aiming to test.

In this case, the mock acts as a replacement for your third-party integration. That way, you don't need to make a network call from your tests. This helps make your test resilient when the network is down or slow and enables you to specify and test corner cases. Plus mocks are often less complicated to access and maintain.

There are some libraries you can use to write your mocks. We recently published a post comparing two JavaScript mocking libraries - nock and unmock.

We’ve loved and used this approach ourselves (I mean, we built unmock). There are still issues with this approach though. Namely, that you have to manually write all the tests - even if the input values are generated.

This can be tedious and take a lot of time. So that’s where the second option comes in.

Use a service (like Meeshkan) to test your apps

Yes, this is a product plug - but with good reason.

We built Meeshkan for those of you reading this post and wondering who has time to write tests. Our automated testing service executes and reports on a collection of generated tests. These tests are designed to imitate any third-party integrations you're working with using schemas and a bit of NLP.

All you need to do is authorize our GitHub app, choose a repository test, and set up your base configuration.

Sound good? Request beta access today.

Property-based testing for JavaScript developers

Carolyn Stransky — Fri, 17 Jul 2020 17:00:25 +0000

Special thanks to Nicolas Dubien for sharing his generative JavaScript testing wisdom with me before I began writing this guide.

All experienced frontend developers know one thing to be true: Users are unpredictable. No matter how much user research you conduct or how thick the font-weight is on your input label, you can never be certain how users will interact with your product. That’s why, as the creators of the interface, we put in constraints. And to ensure that those constraints work properly, we write tests.

But there's a problem with traditional unit and integration tests.

They require us to manually think of and write every scenario that our tests will cover. Not only does this take a lot of time, but it also limits the test coverage to our imaginations. Whereas users, as we know, are unpredictable. So we need a way to test our software to withstand an unlimited number of potential user flows.

That’s where property-based testing comes in.

Within this guide, we’ll explain the must-knows of property-based testing in JavaScript. We'll walk through practical examples and you'll write your first test using the fast-check framework. Finally, we'll touch on what other property-based testing frameworks are out there.

What's in this guide

Property-based testing in JavaScript: What and why
Differences between property-based and example-based tests
What your existing test suite probably looks like (and is missing)
Choosing which properties to test for
Writing your first property-based test with fast-check
Available property-based testing frameworks
Conclusion

⚠️ Prerequisites:

A solid understanding of what unit tests are.
Familiarity with Jest or another JavaScript testing framework.
(Optional) NPM or Yarn installed if you want to follow along in your IDE.

🐍 Prefer Python? A similar guide is available to help you learn property-based testing in Python instead.

💻 References:

We've created a GitHub repository to accompany this guide. This repository includes all of the featured tests with instructions for how to execute them. It also provides more resources for learning property-based testing.

Property-based testing in JavaScript: What and why

Software testing as we know it today requires a lot of time and imagination. When you're writing traditional example-based tests, you're stuck trying to manually reproduce every action that a user might make.

Property-based testing is a different approach to writing tests designed to accomplish more in less time. This is because instead of manually creating the exact values to be tested, it's done automatically by the framework you're using. That way, you can run hundreds or even thousands of test cases in the same amount of time it takes you to write one expect statement.

As the developer writing the tests, what you have to do is:

Specify what type of values the framework should generate (i.e. integers or strings).
Assert those values on guarantees (or properties) that are true regardless of the exact value.

We'll cover how to choose which properties to test for later in this guide. But before going any further, let's talk about why you would want to integrate property-based testing into your workflow.

Nicolas Dubien, the creator of the fast-check framework we're exploring in this guide, wrote a post outlining the primary benefits of property-based testing.

To summarize his words, property-based testing enables developers to:

Cover the entire scope of possible inputs: Unless you specifically tell it to, property-based testing frameworks don't restrict the generated values. As a result, they test for the full spectrum of possible inputs.
Shrink the input when tests fail: Shrinking is a fundamental part of property-based testing. Each time a test fails, the framework will continue to reduce the input (i.e. removing characters in a string) to pinpoint the exact cause of the failure.
Reproduce and replay test runs: Whenever a test case is executed, a seed is created. This allows you to replay the test with the same values and reproduce the failing case.

In this guide, we'll focus on that first benefit: Covering the entire scope of possible inputs.

Differences between property-based and example-based tests

Even with the limitations mentioned, traditional example-based tests are likely to remain the norm in software testing. And that's ok because property-based tests aren't meant to replace example-based ones. These two test types can, and very likely will, co-exist in the same codebase.

While they may be based on different concepts, property-based and example-based tests have many similarities. This becomes evident when you do a side-by-side comparison of the steps necessary to write a given test:

Property-based	Example-based
1. Define data type matching a specification	1. Set up some example data
2. Perform some operations on the data	2. Perform some operations on the data
3. Assert properties about the result	3. Assert a prediction about the result

At its core, property-based testing is meant to provide an additional layer of confidence to your existing test suite and maybe reduce the number of boilerplate tests. So if you're looking to try out property-based testing but don't want to rewrite your entire test suite, don't worry.

What your existing test suite probably looks like (and is missing)

Because property-based tests are meant to fill the coverage gaps missed by traditional testing, it's important to understand how these example-based tests work and their downfalls.

Let's start with a definition: Example-based testing is when you test for a given argument and expect to get a known return value. This return value is known because you provided the exact value to the assertion. So when you run the function or test system, it then asserts the actual result against that return value you designated.

Enough theory, let's write a test.

Imagine you have an input where users write in a number indicating an item's price. This input, however, is type="text" rather than type="number" (trust me, it happens, I've seen it). So you need to create a function (getNumber) that converts the input string into a number using JavaScript's built-in Number() function.

It might look like this:

// getNumber.test.js
const getNumber = inputString => {
  const numberFromInputString = Number(inputString)
  return numberFromInputString
}

Now that you have your getNumber function, let's test it.

To test this using example-based testing, you need to provide the test function with manually created input and return values that you know will pass. For example, the string "35" should return the number 35 after passing through your getNumber function.

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
})

Note: To run this test on your IDE, you'll need to have Jest installed and configured.

And with that, you have a passing example-based test 🎉

Recognizing the limitations of example-based testing

There are many situations where an example-based test like this would work well and be enough to cover what you need.

But there can be downsides.

When you have to create every test case yourself, you're only able to test as many cases as you're willing to write. The less you write, the more likely it is that your tests will miss catching bugs in your code.

To show how this could be a problem, let's revisit your test for the getNumber function. It has two of the most common ways to write a price value (whole number and with a decimal):

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
})

Both of these test cases pass. So if you only tested these two values, you might believe that the getNumber function always returns the desired result.

That's not necessarily the case though. For instance, let's say your website with this price input also operates in Germany, where the meaning of commas and decimals in numbers are switched (i.e. $400,456.50 in English would be $400.456,50 in German).

So you add a third test case to address this:

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
  // Add a new test case:
  expect(getNumber("19,95")).toBe(19.95)
})

But when you run the test... you hit a Not-A-Number error:

expect(received).toBe(expected) // Object.is equality

Expected: 19.95
Received: NaN

Turns out the getNumber function doesn't work as expected when the input string contains a value or specific characters that Number() doesn't recognize. The same error occurs with inputs like twenty or $50. Maybe you already knew that, but maybe you would've never known that without a specific test case.

🐛🚨 This is one example of how property-based testing can be used to find bugs in your software. Once you realize that any string with a character that Number() doesn't recognize will return NaN - you might reconsider how you built that input. Adding the attribute type="number" to the input restricts the possible values that users can enter and, hopefully, helps reduce bugs.

Choosing which properties to test for

Issues like the one faced with the input type also help you write your property-based tests because then it's more clear what the property you're testing for actually is.

Let's dig into this. In property-based testing, a property is an aspect of the function being tested that's always true, regardless of the exact input.

If you look at the getNumber function from earlier, one property you'd test would be the string that is passed to getNumber. Regardless of whether that input value ends up being "59.99", "twenty", or "$50" - it will always be a string.

Some other examples of properties:

List length when testing the sort() method on an array. The length of the sorted list should always be the same as the original list, regardless of the specific list items.
Date when testing a method for the Date object like toDateString(). No matter the specifics entered, it will always be a date.

Writing your first property-based test with fast-check

To put property-based testing into practice, let's create an example test using fast-check, a JavaScript framework for generative test cases.

Let's use the getNumber function from earlier. As a reminder, here's what that looked like:

// getNumber.test.js
const getNumber = inputString => {
  const numberFromInputString = Number(inputString)
  return numberFromInputString
}

Now let's write a property-based test using fast-check. To limit the scope, you'll only generate input strings with floating-point numbers because values with decimals are more common in prices.

Structuring your tests

When getting started with fast-check, you first have to set up the base structure of your tests.

Initially, it'll look identical to any other Jest test. It starts with the test global method and its two arguments: A string for describing the test suite and a callback function for wrapping the actual test.

test("turns an input string into a number", () => {
  // Your property-based test will go here!
})

Introducing fast-check

Next, you'll import the framework and introduce your first fast-check function: assert. This function executes the test and accepts two arguments: The property that you're testing and any optional parameters. In this case, you'll use the property function to declare the property.

const fc = require("fast-check")

test("turns an input string into a number", () => {
  fc.assert(
    fc.property(/* Your specific property and expect statement will go here */)
  )
})

Testing your chosen properties

Finally, you'll add the details of the specific values you want to generate. There's an entire list of built-in arbitraries (aka generated datatypes) provided by fast-check. As mentioned previously, this test will cover input strings with floating-point numbers. There are multiple arbitraries for generating floating-point numbers, but this test will use float.

This float arbitrary will be passed as the first argument of the property function, followed by a callback wrapping the expect statement and any other logic necessary for executing the test.

In this test, testFloat represents each floating-point number generated by fast-check and it's then passed as an argument to the callback. The expect statement indicates that when you pass the testFloat as a string to your getNumber function, you expect it to return the same testFloat value as a number.

test("turns an input string into a number", () => {
  fc.assert(
    fc.property(fc.float(), testFloat => {
      expect(getNumber(`${testFloat}`)).toBe(testFloat)
    })
  )
})

And there you have it, your first property-based test 🎉

Examining the generated values

By default, the property check will be run against 100 generated inputs. For many arbitraries, you can also set a minimum or maximum number of generated inputs. At first, running hundreds of test cases might feel excessive - but these numbers are reasonable (and even considered low) in the property-based testing realm.

Going back to the example test, you can peek at the generated input values using fast-check's sample function. This function takes in an arbitrary or property and the number of values to extract. It then constructs an array containing the values that would be generated in your test.

fc.sample(fc.float(), 10)

If you wrap the previous function in a console.log() statement, you'll get something like this:

Note: Your numbers will likely be different - and that's ok. You also might not want to log a bunch of random numbers in your terminal - and that's ok too. You can take our word for it.

Available property-based testing frameworks

We opted to use the fast-check framework for this guide, but there are many other options out there to help you write property-based tests in a variety of programming languages.

JavaScript

Other languages

Hypothesis: Python (used in our Python guide)
FsCheck: .NET
jqwik: Java
PropCheck: Elixir
Proptest: Rust
PropEr: Erlang
RapidCheck: C++
QuickCheck: Haskell
QuickCheck ported to Rust: Rust
ScalaCheck: Scala

Conclusion

While it won't replace example-based tests, property-based testing can supply additional coverage where traditional tests fall short. One of the benefits of property-based testing is that it helps cover the entire scope of possible inputs for any given function. We explored that benefit throughout this guide by creating a getNumber function and writing a test that uses a generative floating-point number property.

This guide wasn't intended to be a series, but the possibility of future guides about shrinking, replaying tests, property-based testing in TypeScript, or our favorite fast-check features emerged during our research. If that sounds interesting to you, comment below or tweet at us!

At Meeshkan, we're working to improve how people test their products and APIs. So if you made it to the end of this guide and have thoughts, we want to hear from you. Reach out on Gitter or open an issue on GitHub to let us know what you think.

Functional on the frontend with fp-ts and pipe

Carolyn Stransky — Wed, 27 May 2020 16:18:04 +0000

This article is based on an internal presentation done by Mike Solomon, who will be further referred to as "my boss."

As a team, we decided to integrate functional programming practices into the codebase for our web application. More specifically, we're using fp-ts, a library for typed functional programming in TypeScript.

This article explains why we chose fp-ts and walks through a practical example using the pipe function.

Why we're going functional

Because my boss likes Haskell 🤷‍♀️

I'm joking (mostly). My boss does have an affinity for functional programming and he's more comfortable in this type of workflow. But even if the learning curve is steep for those of us who didn't know what monads are, we've realized something. Adopting functional programming practices has improved our web application.

Here are some of the reasons:

Productivity

Descriptive errors - When we see logs in the console, it's rarely Uncaught TypeError: Cannot Read Property 'name' of undefined or Object doesn't support property or method 'getPosts'. This helps for more efficient debugging.
Less code - Functional programming takes care of many patterns that would otherwise result in boilerplate code.
Limited options - With functional programming, you can only do things a certain number of ways.
Refactoring - With strong type safety, you refactor "against" the compiler. This means the red squiggles in your IDE guide the refactoring process and proposes helpful suggestions.

Correctness

Type safety - When you use a typed variable, you're defining a constraint on all possible values. This helps ensure that the inputs and outputs of our code work as expected.
Error routing - With functional programming, errors become first-class citizens and are propagated to error handlers based on rules.
Linear ordering - No more jumping between if this else that or getting stuck in a deep-nested JavaScript try/catch block.

Why we chose the `fp-ts` library

In theory, we could've switched out fp-ts for another functional programming library for TypeScript like Purify. Both libraries have similar syntax for common functional patterns like the Either class and the chain function. However, fp-ts has some additional classes that we use regularly like Reader and Semigroup.

If there were terms in that last paragraph that you didn't understand, don't worry! We'll cover those in a future post.

Working with our existing React codebase

Fortunately for us, the codebase we're working with is still fairly new. The repository was created a little over one month ago. The initial setup was done by two developers (myself included) with no functional programming experience. But, turns out, we were already applying functional programming principles to our React application.

Some examples:

Hooks as a functional way to manage state dependencies.
Function components instead of class components.
Arrow function expressions, which, when used without brackets, enforces a single flow of information.

But taking that next step into the functional programming world required us to restructure the way we think about and read code. To make it more tangible, the rest of this article will focus on one specific function from the fp-ts library: pipe.

Putting it into practice with `pipe`

The concept of piping goes well beyond the fp-ts library. According to The Linux Information Project, piping is defined as:

A form of redirection that is used to send the output of one program to another program for further processing.

Sounds intense and a bit abstract. Let's break it down.

Overall, a pipe is one big function of functions. It takes an initial value and then passes that as the argument(s) for the first internal function to use. Then it takes the result from that function and passes it to another internal function. And so on, potentially forever 🤪

Maybe it's better to explain with code.

Here's an example of piping written in vanilla JavaScript:

const examplePipe = (a, b, c) => c(b(a));

This examplePipe function takes in three parameters (a, b, and c). For examplePipe to work as expected, a should be a value that can be consumed by b. Then b should be a function that takes a as an argument. Finally, c should be another function that takes the result of b as an argument.

Let's put in some arguments:

examplePipe(1, (x) => x+1, (x) => x+5)

First, it takes an independent value: 1.

Then, 1 is passed to the next function: (x) => x+1. So because x is equal to 1, the result is 2.

Finally, this result (2) is passed to the last function: (x) => x+5. Because x is now equal to 2, the examplePipe will return 7.

And there you have it, our first pipe 🎉

This was a generic example of piping. Next, we'll go step-by-step to see how this would work in a web application. Throughout, we'll use the pipe function that's available through the fp-ts library.

Defining the initial value in a `pipe`

The most minimal pipe we can write is a pipe with a single object, like pipe(1). Here, the first value (1) isn't consumed by any functions in the pipe. This means that the result of pipe(1) is equal to 1.

As soon as a pipe grows to two values, it then enforces a contract - the second element of the pipe must be a function that can consume the first value. This first value can be anything: A number, a string, a class, a function, or even void.

This is common practice in functional programming. Instead of defining variables along the way, everything we need is defined at the start. "Priming the pipe" so to speak.

Let's start creating an example. We're going to define an exampleFunction that doesn't have any parameters and returns a pipe. To start, pipe contains an object with three values: projects (independent getProjects function), a users array, and a configuration object.

It should look like this:

const getProjects = () => ([]);

const exampleFunction = () => pipe(
    {
        projects: getProjects(),
        users: [5],
        configuration: {}
    }
);

This example assumes we have fp-ts installed and pipe imported.

Another nuance of pipe is the order (or lack of order) that we define our initial values. To show how this works, let's look at a real-world example.

In our web application, we often define our hooks within this first part of the pipe function. Alternatively, you could use const to define variables like so:

const useColorMode = useColorMode()
const useDisclosure = useDisclosure()

In this structure, useDisclosure will always be executed after useColorMode. This is because JavaScript code executes in order.

But with an object, there are no guarantees about the order of execution. JavaScript doesn't indicate which values in an object are created in memory first. This is true for any object, but it's especially useful in our pipe function.

Defining variables within the first object of pipe signals to anyone maintaining the code that the order of these variables is insignificant. This allows us to refactor with more confidence.

What's also nice about putting these values first is that it distinguishes what is independent in your function. So no matter what, you know that these values don't have any dependencies or rely on anything else. This can help with debugging and code readability.

First function in the `pipe`

The next part of the pipe is our first function. In this function, we can pass the values defined in the first object as an argument.

We do this in the following example with the valuesFromObjectAbove parameter:

const getProjects = () => ([]);

const exampleFunction = () => pipe(
    {
        projects: getProjects(),
        users: [5],
        configuration: {}
    },
    (valuesFromObjectAbove) => ({
        // Coming soon!
    })
);

Here, valuesFromObjectAbove represents projects, users, and configuration.

We can then use valuesFromObjectAbove to create new values. In this example, we're creating arrays of adminProjects and notAdminProjects using the projects value we defined in the first object:

const getProjects = () => ([]);

const exampleFunction = () => pipe(
    {
        projects: getProjects(),
        users: [5],
        configuration: {}
    },
    (valuesFromObjectAbove) => ({
        adminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === true),
        notAdminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === false)
    })
);

Now, we can see this grouping of independent values first, dependent ones second. Reading the code, we can deduce that adminProjects and notAdminProjects, by definition, depend on a value that was created earlier. This can help with debugging. For instance, if you insert a console.log() statement after the first object, you know that your log will only contain the independent values in the function.

Another round of functions

There are a few options available for what values are passed to our second function.

One option is to use a spread operator:

const getProjects = () => ([]);

const exampleFunction = () => pipe(
    {
        projects: getProjects(),
        users: [5],
        configuration: {}
    },
    (valuesFromObjectAbove) => ({
        ...valuesFromObjectAbove, // Look here!
        adminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === true),
        notAdminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === false)
    }),
    (valuesFromFunctionAbove) => ({
        ...
    })
);

By using the spread operator, we're saying that we want to pass down everything. This means that valuesFromFunctionAbove contains all of the values from the initial object (projects, users, configuration). And it also contains the values from the first function (adminProjects, notAdminProjects). Bonus: It's all type safe!

But let's say we delete the spread operator:

const getProjects = () => ([]);

const exampleFunction = () => pipe(
    {
        projects: getProjects(),
        users: [5],
        configuration: {}
    },
    (valuesFromObjectAbove) => ({
        // No spread operator
        adminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === true),
        notAdminProjects: valuesFromObjectAbove.projects.filter(a => a.admin === false)
    }),
    (valuesFromFunctionAbove) => ({
        ...
    })
);

Now, the second function only has access to adminProjects and notAdminProjects.

That is the power of pipe. We always know what's ready to use 💥

If organized appropriately, pipe can contain everything that we would need to create our React component. So those ... in the last two examples? That's where we could put in our JSX.

More with `fp-ts`

This article only scratched the surface of what the fp-ts library can bring to a web application. On our team, there are many more functions and patterns that we use (Either, chain, isLeft, isRight, Reader). If you'd be interested in learning about these, tweet at us or leave a comment and let us know!

In the meantime, check out the fp-ts documentation.

Six questions to answer before implementing a telemetry feature

Carolyn Stransky — Tue, 14 Apr 2020 13:44:04 +0000

In the first post of this series, I covered what telemetry features are and how developers feel about them. The general consensus was that the data needs to be anonymous, it should be clearly documented and it must be able to be switched off easily (or opt-in if possible).

While this sounds great, it's not always realistic for every product or organization. And it's definitely not that straightforward. There are still a few questions that you'll need to discuss together with your team.

This post outlines six of them:

Why are you collecting this data?
What kind of data will you track?
How will you collect the data?
Who has access to this data?
Opt-in or opt-out?
When will you delete data?

Why are you collecting this data?

Before anything else, you'll need to ask two important questions:

Is there a valid purpose for this data to be collected?
If yes, is there an alternative way to achieve the purpose without the data?

Write these answers down, regardless of whether you decide to implement the feature or not. If you decide to go the telemetry route, you might want to share these answers later in user documentation (more on this in a future post). Regardless, documenting these answers internally will help align your team and fuel future discussions.

What kind of data will you track?

Think about your purpose for collecting this data and let that inform the data you collect. If you're trying to measure usage, you could track how many people initialize your CLI and what commands they execute. Or you can prioritize bugs by tracking how many people run into particular errors and wherein the process these errors occur.

You should always question whether personal data is really necessary for what you need. For instance, while it might be interesting to see where in the world your users come from, tracking location is much more invasive. So unless you're trying to gauge something like where to add more customer support agents, you should reconsider.

"Telemetry helps to shape the product, but not at cost of user's privacy, we need data collection but make sure none of it contains any data points which can be used to profile a user," says developer Trishul Goel when discussing how they do telemetry at Cliqz.

A note about anonymous data

Some justify collecting personal data by saying, 'Well, as long as it's anonymous then it's fine.' But then you need to ask yourself another question, what does it mean for data to be anonymous?

"There's no such thing as anonymous usage data, especially if a third party is doing the collecting," says open-source developer Claus Due.

Typically, it's not enough to say that data is anonymous. Developers implementing telemetry features need to put in extra effort to make sure that people can't be identified from the data.

"If you're very confident it's anonymous, depends on the data, since a lot to anonymous data has actually been found to be able to identify people," says developer and designer Nick Colley. Nick also mentioned that he prefers to focus on data minimization because it's more "proactive and user-focused."

Other considerations around tracking data

What security measures do you have in place?

This could include (but isn't limited to) penetration testing, encryption, intrusion protection and vulnerability reporting.

What will you do if there is a problem?

Teams should consider having an incident response plan available, in case there are any issues.

How will you collect the data?

Or, alternatively, where is the data handled?

The answer will depend on whether you are building your telemetry feature in-house or using a third-party integration.

Homegrown

Developing your own telemetry feature will take more effort (time, money, maintenance) all around. But it also allows you to manage who has access to the data, what exactly is collected and how it is communicated to your users.

If the rest of your product is already open-source, your telemetry feature should be as well. A good example of this comes from Gatsby: gatsby-telemetry

Third-party

Using a third-party, like OpenTelemetry, will likely require less effort upfront. It also provides a built-in support system through bug reporting and community usage. But, with any outside source, you're giving up control of the data and increasing risk as more people are potentially able to access it. Resources like Mozilla's Outsourced Services Worksheet can help you evaluate the potential risk of a third-party service.

Who has access to this data?

It's recommended that data access only be granted to the people who need it to perform their jobs. The Security Access Controls Worksheet is another resource from Mozilla that can help you determine this.

Should this data be open?

If this usage data is truly anonymous, you could consider having it open and available to your users.

One way to do this is through public, accessible dashboards. You can use tools like Open MCT to showcase telemetry data. Another way is to make it available upon request or on an as-needed basis.

"I don't believe data should be open but it should be possible to get access with a reasonable process," says developer Florian Gilcher.

Opt-in or opt-out?

Users should always be able to decide if they're ok with having their data tracked. But when to present this option is the question.

This leads us to the most notorious debate in telemetry: Should users have to agree to tracking (opt-in) or should tracking be on by default (opt-out)?

Opt-in

This means asking for explicit consent from users before sending any data to your system. In our initial survey about telemetry in open source, many developers mentioned that telemetry features should be opt-in only. They also said that users should know what's being tracking and not have that information hidden in a blog post or setting somewhere.

One potential trade-off with an opt-in only approach is engagement. There's a lingering question of whether people will actually opt-in.

"I once build [an open-source project] with opt-in for telemetry data, bug reports and anonymous analytics," explains open-source maintainer Sebastian Golasch. "I kindly asked the users to enable this to help improve the project, but I must admit, less than 10% (judging from the download to telemetry ratio) actually did it."

Opt-out

Having telemetry settings on by default is much more common in modern products. These products give users the option to switch off tracking through an environment variable, command line argument/flag or a toggle in the product's GUI.

When taking an opt-out approach, it's advised to let your users know right away that these settings are on and they are being tracked. It's more transparent and also reduces the risk of ill feelings later.

"I do accept tracking if there is an explanation upfront. But I don't if I discover by accident that I am tracked," says Stephan Schuler.

When will you delete data?

The value of the data diminishes over time, so it's recommended to delete data when it's no longer relevant.

You should agree on a specific time limit for data (i.e. 3 months, 6 months, 1 year). You can choose to communicate that to your users through documentation or when alerting them that they are being tracked.

Next up: Telemetry in the wild

This is the second in an undetermined number of posts about telemetry in open-source software. Up next, I'll cover some telemetry feature examples in popular open-source products and what they are (and aren't) getting right.

If you have any requests for this series, please comment below!

Onboarding a junior developer to your team? Here's 12 tips.

Carolyn Stransky — Sat, 11 Apr 2020 17:26:23 +0000

A few weeks ago, my friend Nico König asked the Twittersphere about best practices when onboarding a new junior developer to your team:

// Detect dark theme var iframe = document.getElementById('tweet-1242770350645051394-995'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1242770350645051394&theme=dark" }

As someone who was recently a junior dev, I have a lot of opinions on this topic. So I decided to summarize them in this post (in no particular order).

💭 Distinguish between opinions and best practices

When you're talking over concepts with your junior, deliberately point out whether the things you're saying are your personal views or proven best practices.

We work in such an opinion-heavy industry. So sometimes it can feel impossible to differentiate feelings from facts - especially if you're new to programming. One tangible way to help with this is to provide resources describing why something is best practice.

🖥 Dedicate time early on for Git commands

On the very first day (or maybe second), walk through the Git commands that you use most frequently on the job with your junior. Nothing is too basic - particularly because many bootcamps or university programs don't teach Git. For example, the bootcamp I went to used Git for version control, but all of our tasks were done solo. So more collaborative Git workflows (like branching or pull requests) were new to me when I started my first developer job.

Ideally, you'd also take the time to explain what each command is actually doing. It took me nine months and one colleague sitting down with a pen and paper for me to finally figure out how a rebase works. Speaking of, teach them how to rebase properly!

📝 Have something for them to work on

Give your junior something to work on that is low pressure, but still impactful. If you're stuck on what kind of work falls into this category, think either nice-to-haves (style improvements, microinteractions, bugs that aren't critical, etc.) or general improvements (refactoring, accessibility, performance, etc).

Be sure to have a dedicated backlog of tasks ready for them to choose from. For most junior developers, this will be their very first software job. So it's unrealistic to expect them to "take initiative" and decide what they should work on without a curated task list.

✅ Give constructive code reviews

Code reviews are a major. At best, code reviews are constructive and serve as continuous learning opportunities. At worst, they can destroy trust and safety within a team.

As the reviewer, you should be pointing out things that are done well and if you have critiques, explaining why something is problematic rather than only saying ‘fix this’ or offering the solution.

👩‍💻 Practice pair programming

Set regular, dedicated times for pairing (like literally put it in the calendar). I'd recommend at least twice per week. You can have a set of tasks that you work on together or switch between individual tasks.

When pairing, make sure you're alternating between who's driving and who's navigating. Depending on your junior and their learning style, this could also be them observing you. Personally, I learn so much from watching and asking questions.

🧠 Pair on things that aren't code

It feels weird at first, but I promise that it’s helpful. Show them your workflow, how you prioritize tasks or approach learning something new. One of the best non-coding pairing sessions I've had was a colleague walking me through how he conducts a code review.

How you test your own code? Dive into a new project? Manage your time? Plan a company-wide presentation? All of this is valuable.

🙋‍♂️ Always be available for questions

Make your junior a top priority. When they have questions, stop what you're doing and help them. If you don't know the answer, help them find it - whether that's searching through StackOverflow or introducing them to someone who knows that domain better.

Jess Mitchell also made an especially excellent point about tone:

// Detect dark theme var iframe = document.getElementById('tweet-1242900048033259524-320'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1242900048033259524&theme=dark" }

💞 Get them a buddy or mentor

This should be someone (not their direct manager) that will guide your junior through their onboarding. Your junior should also feel comfortable talking openly with this person and discussing any problems they might be having.

Ideally, your junior and their buddy will meet at least once per week during the first 6 months.

📚 Learn their learning style

Get familiar with your junior’s learning style and try to cater the resources and materials you send them to that style. So if they like books, try to recommend books or written articles instead of videos courses. It's a small gesture, but it'll help them succeed and shows you care.

⏰ Respect their time

Set a good example by not sending emails at wild hours or pressing for tasks. Let them know what's expected from them in terms of working hours. An example of why this is important: In my first tech job, the lunch break didn't count as working time... but no one told me that even if I took a shorter lunch, I still couldn't leave earlier. Later, the fact that I left 30 minutes early every day was brought up as an argument not to keep me 🙄

On a related note, try to encourage wellness by recommending self-care tactics and being open about mental/physical health (if you feel comfortable and safe doing so).

📋 Provide regular feedback

Give regular feedback and updates about how they’re doing. If you're working in a country with a probation time, try to give weekly feedback during that time.

Your junior will likely assume they are doing worse than they really are. But if there is something that needs to be fixed, be sure to tell them enough in advance that there's time to make a change.

🌱 Show humility yourself

Admit when you don’t know things and search for answers together. Offer regular opportunities for them to give you feedback and maybe even in a variety of ways (sometimes face-to-face, sometimes written, etc).

Ultimately, their success is your success, so it's important to know how you're doing throughout this process as well.

Did you find this helpful or useful? If yes, please consider buying me a coffee so I can continue to write posts like this ☕️

From 1 to 10,000 test cases in under an hour: A beginner's guide to property-based testing

Carolyn Stransky — Mon, 06 Apr 2020 14:48:16 +0000

This guide was co-authored by Fredrik Fornwall.

Testing your software takes time... a lot of time. When you're writing tests, you're often stuck trying to manually reproduce every potential sequence of events. But what if you wanted to test hundreds (or thousands or even millions) of cases at once? We have an answer: Property-based testing.

Maybe you've written unit tests before, but this is the first time you've heard about property-based testing. Or maybe you've heard the term, but still don't really get what it's about. Either way, we've got you.

Throughout this guide, we'll cover the fundamentals of property-based testing. We'll walk you through practical examples and how to structure your tests. Finally, you'll learn how to use property-based testing to find bugs in your code and what existing libraries are out there.

What's in this guide

Traditional unit tests based on examples
- Limitations of example-based testing
Introduction to property-based testing
- An example using Hypothesis
- What can be a property?
- How does property-based testing differ from example-based?
Example properties and how to test for them
- Unexpected exceptions should never be thrown
- Values shouldn't change after encoding and then decoding
- A naive method should still give the same result
Finding bugs with property-based testing
Available libraries
Conclusion

⚠️ Prerequisites:

A general understanding of what unit tests are.
(Optional) Python 3+* if you want to follow along in your own IDE.

* This guide will use Python for code examples, but the concepts aren't limited to any specific programming language. So even if you don't know Python, we'd encourage you to read along anyway.

💻 References:
We've created a GitHub repository to go with this guide. All of the featured code examples exist there as unit tests and include instructions for how to execute them.

Traditional unit tests based on examples

Most often, software testing is done using example-based testing. This means you test that for a given argument, you get a known return value. This return value is known because, well, you provided that exact value as a sample. So when you run the function or test system, it then asserts the actual result against that sample return value.

Let's look at an example. Say you want to write a function called sort_this_list. This function will take a list as an argument and return the same list organized in ascending order. To do this, you'll use the built-in Python sorted function.

It might look like the following:

# test_sorted_list.py
def sort_this_list(input_list):
    sorted_list = sorted(input_list)
    return sorted_list

Now that you have your sort_this_list function, let's test it.

To test this using example-based testing, you need to (manually) provide the test function with return values that you know will be True. For example, the list [5, 3, 1, 4, 2] should return [1, 2, 3, 4, 5] after it's sorted.

# test_sorted_list.py
def test_sort_this_list():
    assert sort_this_list([5, 3, 1, 4, 2]) == [1, 2, 3, 4, 5] # True
    assert sort_this_list(['a', 'd', 'c', 'e', 'b']) == ['a', 'b', 'c', 'd', 'e'] # True

And with that, you have a passing example-based test 🎉

Limitations of example-based testing

While example-based tests work well in many situations and provide an (arguably) low barrier of entry to testing, they do have downsides. Particularly that you have to create every test case yourself - and you can only test as many cases as you're willing to write. The less you write, the more likely it is that your tests will miss catching bugs in your code.

To show why this could be a problem, let's look at the test for the sort_this_list function from the last section:

# test_sorted_list.py
def test_sort_this_list():
    assert sort_this_list([5, 3, 1, 4, 2]) == [1, 2, 3, 4, 5] # True
    assert sort_this_list(['a', 'd', 'c', 'e', 'b']) == ['a', 'b', 'c', 'd', 'e'] # True

Both of these assertions return True. So if you only tested these two values, you might believe that the sort_this_list function always returns the desired result.

But if you add a third potential return value:

# test_sorted_list.py
def test_sort_this_list():
    assert sort_this_list([5, 3, 1, 4, 2]) == [1, 2, 3, 4, 5] 
    assert sort_this_list(['a', 'd', 'c', 'e', 'b']) == ['a', 'b', 'c', 'd', 'e'] 
    # Add a new test case:
    assert sort_this_list(['a', 2, 'c', 3, 'b', 1]) == ['a', 'b', 'c', 1, 2, 3]

And then run the test... you'll hit an error:

TypeError: '<' not supported between instances of 'int' and 'str'

Turns out the sort_this_list function doesn't work as expected when the list contains both integers and strings. Maybe you already knew that, but maybe you would've never known that without a specific test case.

Even with these limitations, example-based testing will continue to be the norm in software testing. Throughout the rest of this guide, though, we'll explore another technique. One designed to compliment your existing (likely example-based) tests and improve the test coverage of your code.

Introduction to property-based testing

When thinking about the limitations of example-based testing, many questions come to mind. What if you want to test hundreds of cases? Or ones that you could never dream of coming up with yourself?

Property-based testing is a different approach here to help with that. With property-based testing, you don't generate the exact values manually. Instead, that is done by a computer automatically.

As the developer, what you have to do is:

Specify what value to generate.
Assert on guarantees (or properties) that are true regardless of the exact value.

An example using Hypothesis

To put property-based testing into practice, let's look at an example using Hypothesis, a Python library for generative test cases. We chose Hypothesis mostly because we're using Python - but also because the documentation is clear and thorough.

Let's use the sort_this_list function from earlier. As a reminder, here's what that looked like:

# test_sorted_list.py
def sort_this_list(input_list):
    sorted_list = sorted(input_list)
    return sorted_list

Now let's write a property-based test using Hypothesis. To limit the scope, you'll only test for lists of integers:

# test_sorted_list.py
import hypothesis.strategies as some
from hypothesis import given, settings

# Use the @given decorator to guide Hypothesis to the input value needed:
@given(input_list=some.lists(some.integers()))
# Use the @settings object to set the number of cases to run:
@settings(max_examples=10000)
def test_sort_this_list_properties(input_list):
    sorted_list = sort_this_list(input_list)

    # Regardless of input, sorting should never change the size:
    assert len(sorted_list) == len(input_list)

    # Regardless of input, sorting should never change the set of distinct elements:
    assert set(sorted_list) == set(input_list)

    # Regardless of input, each element in the sorted list should be
    # lower or equal to the value that comes after it:
    for i in range(len(sorted_list) - 1):
        assert sorted_list[i] <= sorted_list[i + 1]

Note: If you're following along on your machine, make sure to install Hypothesis and then you can run the tests using pytest.

And there you have it, your first property-based test 🎉

What's especially important here is the use of the @given function decorator:

@given(input_list=some.lists(some.integers()))

This specifies that you want a list of random integers as the input value and asserts on properties that are true regardless of the exact input.

Another significant feature of this test is the use of the @settings object:

@settings(max_examples=10000)

Here, it's using the max_examples setting to indicate the maximum number of satisfying test cases that will run before terminating. The default value is 100 and in this case, it's set to 10000.

At first, running tens of thousands of test cases might feel excessive - but these numbers are reasonable in the property-based testing realm. Even the Hypothesis documentation recommends setting this value well above the default or else it may miss uncommon bugs.

Going back to the example test, if you add a print(input_list) statement, you can peek at the 10,000 different generated input values:

[]
[92]
[66, 24, -25219, 94, -28953, 31131]
[-16316, -367479896]
[-7336253322929551029, -7336253322929551029, 27974, -24308, -64]
...

Note: Your values will likely be different from our example - and that's ok. You also might not want to print 10,000 example lists - and that's ok too. You can take our word for it.

The number of runs and specifics of the generated data can be configured. More on that later on.

What can be a property?

With this style of testing, a property is something that's true about the function being tested, regardless of the exact input.

Let's see this definition applied to assertion examples from the previous test_sort_this_list_properties function:

len(sorted_list) == len(input_list): The property tested here is the list length. The length of the sorted list is always the same as the original list (regardless of the specific list items).
sorted_list[i] <= sorted_list[i + 1]: This property was that each element of the sorted list is in ascending order. No matter the contents of the original list, this should be true.

How does property-based testing differ from example-based?

While they're from different concepts, property-based tests share many characteristics with example-based tests. This is illustrated in the following comparison of steps you'd take to write a given test:

Example based	Property based
1. Set up some example data	1. Define data type matching a specification
2. Perform some operations on the data	2. Perform some operations on the data
3. Assert something about the result	3. Assert properties about the result

There are several instances where it would be worthwhile to use property-based testing. But the same can be said for example-based testing. They can, and very likely will, co-exist in the same codebase.

So if you're stressed about having to rewrite your entire test suite to try out property-based testing, don't worry. We wouldn't recommend that.

Example properties and how to test for them

By now, you've written your first property-based test and many lists were sorted 🎉 But sorting lists isn't a representative example of how you'd use property-based testing in the real world. So we've gathered three example properties and in this section, we'll guide you through how they might be used to test software.

All of the examples will continue to use the Hypothesis testing library and its @given function decorator.

Unexpected exceptions should never be thrown

Something that was tested by default in the previous test_sorted_list_properties function was that the code didn't throw any exceptions. The fact that the code doesn't throw any exceptions (or more generally, only expected and documented exceptions, and that it never causes a segmentation fault) is a property. And this property can be a convenient one to test, especially if the code has a lot of internal assertions.

As an example, let's use the json.loads function from the Python standard library. Then, let's test that the json.loads function never throws any exceptions other than json.JSONDecodeError - regardless of input:

# test_json_decode.py
@given(some.text())
def test_json_loads(input_string):
    try:
        json.loads(input_string)
    except json.JSONDecodeError:
        return

When you run the test file, it passes 🎉 So the beliefs held up under testing!

Values shouldn't change after encoding and then decoding

A commonly tested property is called symmetry. Symmetry proves in certain operations that decoding an encoded value always results in the original value.

Let's apply it to base32-crockford, a Python library for the Base32 encoding format:

# test_base32_crockford.py
@given(some.integers(min_value=0))
def test_base32_crockford(input_int):
      assert base32_crockford.decode(base32_crockford.encode(input_int)) == input_int

Because this decoding scheme only works for non-negative integers, you need to specify the generation strategy of your input data. That's why, in this example, some.integers(min_value=0) is added to the @given decorator. It restricts Hypothesis to only generate integers with a minimum value of zero.

Once again, the test passes 🎉

A naive method should still give the same result

Sometimes, you can get the desired solution through a naive, unpractical way that isn't acceptable to use in production code. This might be due to the execution time being too slow, memory consumption being too high or it requiring specific dependencies that aren't acceptable to install in production.

For example, consider counting the number of set bits in an (arbitrary sized) integer, where you have an optimized solution from the pygmp2 library.

Let's compare this with a slower solution that converts the integer to a binary string (using the bin function in the Python standard library) and then counts the occurrences of the string "1" inside of it:

# test_gmpy_popcount.py
def count_bits_slow(input_int):
    return bin(input_int).count("1")

@given(some.integers(min_value=0))
@settings(max_examples=500)
def test_gmpy2_popcount(input_int):
    assert count_bits_slow(input_int) == gmpy2.popcount(input_int)

For illustrative purposes, this example specifies a @settings(max_examples=500) decorator to tweak the default number of input values to generate.

The test passes 🎉 - showing that the optimized, hard-to-follow code of gmpy2.popcount gives the same results as the slower but less complex count_bits_slow function.

Note: If this was the only reason to bring in gmpy2 as a dependency, it'd be wise to benchmark if the performance improvements of it really would outweigh the cost and weight of the dependency.

Finding bugs with property-based testing

We've gone over the concepts of property-based testing and seen various properties in action - this is great. But one of the selling points of property-based tests is that they're supposed to help us find more bugs. And we haven't found any bugs yet.

So let's go hunting.

For this example, let's use the json5 library for JSON5 serialization. It's a bit niche, sure, but it's also a younger project. This means that you're more likely to uncover a bug compared to a more established library.

The json5 library contains:

One property of JSON5 is that it is a superset of JSON.
Another property proving that deserializing a serialized string should give you back the original object.

Let's use those properties in a test:

# test_json5_decode.py
import json
from string import printable

import hypothesis.strategies as some
import json5
from hypothesis import example, given, settings

# Construct a generator of arbitrary objects to test serialization on:
some_object = some.recursive(
    some.none() | some.booleans() | some.floats(allow_nan=False) | some.text(printable),
    lambda children: some.lists(children, min_size=1)
    | some.dictionaries(some.text(printable), children, min_size=1),
)

@given(some_object)
def test_json5_loads(input_object):
    dumped_json_string = json.dumps(input_object)
    dumped_json5_string = json5.dumps(input_object)

    parsed_object_from_json = json5.loads(dumped_json_string)
    parsed_object_from_json5 = json5.loads(dumped_json5_string)

    assert parsed_object_from_json == input_object
    assert parsed_object_from_json5 == input_object

After creating a some_object generator of arbitrary objects, you can verify aspects of the previously mentioned properties: You serialize the input using both json and json5. Then, you deserialize those two objects back using the json5 library and assert that the original object was obtained.

But doing this, you'll run into a problem. At the json5.dumps(input_object) statement, you get an exception inside the internals of the json5 library:

    def _is_ident(k):
        k = str(k)
>       if not _is_id_start(k[0]) and k[0] not in (u'$', u'_'):
E       IndexError: string index out of range

Note: If you're following along and you want to discover the bug yourself, uncomment json5==0.9.3 and remove json5 from the requirements.txt file.

Besides showing the stack trace, as usual, you also get an informative message showing the failed hypothesis - otherwise known as the generated data that caused the test to fail:

# ------------- Hypothesis ------------- #
Falsifying example: test_json5_loads(
    input_object={'': None},
)

Using the {'': None} input data caused the issue that was promptly reported. Finding the issue led to fixing the bug. This fix has since been released in version 0.9.4 of the json5 library.

We want to add that the fix was merged and released within 20 minutes of reporting. Very impressive work by json5 maintainer @dpranke 👏

But what about the future, how can you be sure that the problem will never resurface?

Because the data currently generated contains the troublesome input ({'': None}), you want to ensure that this input is always used. This should be true even if someone tweaks the some_object generator or updates the version of Hypothesis used.

The following fix uses the @example decorator to add a hard-coded example to the generated input:

--- test_json5_decode_orig.py   2020-03-27 09:48:24.000000000 +0100
+++ test_json5_decode.py    2020-03-27 09:48:32.000000000 +0100
@@ -14,6 +14,7 @@

 @given(some_object)
 @settings(max_examples=500)
+@example({"": None})
 def test_json5_loads(input_object):
     dumped_json_string = json.dumps(input_object)
     dumped_json5_string = json5.dumps(input_object)

Bug found ✅ Bug fixed ✅ You're good to go 🎉

Available libraries

This guide uses the Hypothesis library for Python. But there's a lot of functionality that wasn't covered and, as previously mentioned, the documentation is nice. We'd recommend checking it out if you're a Python user.

If you're not using Python, no problem. There are several other libraries built for property-based testing, in a variety of languages:

fast-check: TypeScript
FsCheck: .NET
jqwik: Java
PropCheck: Elixir
PropEr: Erlang
RapidCheck: C++
QuickCheck: Haskell
QuickCheck ported to Rust: Rust

Conclusion

Example-based testing isn't going anywhere any time soon. But we hope that, after reading this guide, you're motivated to incorporate some property-based tests into your codebase.

Some lingering questions from us...

Why weren't you using property-based testing before?
After reading through this guide, would you be willing to try?
Are you interested in seeing another article expanding on the topic?

At Meeshkan, we're working to improve how people test their products. So no matter if you loved or loathed this guide, we want to hear from you. Leave us a comment below, tweet at us or reach out on Gitter to let us know what you think.

Understanding telemetry features in open-source software

Carolyn Stransky — Wed, 18 Mar 2020 15:36:50 +0000

Until recently, I was unaware that many of the open-source products I use regularly - like Visual Studio Code or Netlify - have telemetry features that track usage. Admittedly, I had no idea these kinds of features existed at all. But I wasn't surprised. What bothered me more was that I didn't know my data was being tracked.

So I began talking to other developers in my network. I learned that people fell somewhere on a spectrum - ranging from blissfully unaware (like me) to vigilant about data protection. Some even told me that they refuse to use products that have these features.

All of this got me thinking about the general attitude and ethics around telemetry features in open source technology. Can tracking data be ethical if that data is open and available? If telemetry is bad, how can open source maintainers best track usage to improve the project? I decided to dig deeper and thus, this series was born.

This first post will cover the what and why of telemetry features, insights on developer attitudes and what else you can expect from this series.

What is telemetry?
Why would I implement a telemetry feature?
How do developers feel about telemetry?
What's next?

What is telemetry?

Telemetry is a feature that enables data collection. Many products (open source or otherwise) implement telemetry features to track usage. This usage data can range from performance metrics to errors encountered by users.

Let's dive deeper into the term telemetry. If you look at a very scientific resource called Wikipedia, it defines telemetry as follows:

Telemetry is the collection of measurements or other data at remote or inaccessible points and their automatic transmission to receiving equipment or monitoring.

When you abstract this a bit, you can see how this would apply to software:

collection of measurements or other data: How and when people are using the product.
remote or inaccessible points: Other people's computers.
their automatic transmission: Actions done by the implemented telemetry features.
receiving equipment or monitoring: How the product is tracking this usage data (for instance, a dashboard or log).

To put it into context, you can check out Gatsby's telemetry package, which is entirely open source. Or if you're curious about working with a telemetry feature, take a look at this example of how CodeSandbox sends telemetry events.

Why would I implement a telemetry feature?

Firefox's source documentation describes this well. It says that their telemetry features are used to track "information about how Firefox performs in the wild." And this is exactly the benefit: Raw usage data.

Many teams choose to implement telemetry features to help them understand how to improve their product. There are other ways to source user feedback - like case studies, surveys or even GitHub issue templates. But telemetry features are different because they can offer continuous, unfiltered insight into a user's experiences.

It can be difficult to prioritize fixes or features when you aren't sure how the product is being used. And not every user will be motivated to file a bug report when they do run into issues. So for some, telemetry helps find those answers.

How do developers feel about telemetry?

Back in October 2019, I asked Twitter: Are you ok with open source projects having a telemetry feature that collects anonymous usage data? Assuming you can opt-out.

Over 440 developers participated in the poll, but the results varied.

// Detect dark theme var iframe = document.getElementById('tweet-1182571329456934913-331'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1182571329456934913&theme=dark" }

While 53% answered some form of yes, over half of them indicated that the data must be open and available. Twenty percent said no flat out.

Those who answered "Depends on what's tracked" and took the time to reply brought up a variety of issues. These included opting-in rather than out, what type of data is being tracked, whether the data is anonymous and many more that we'll dive into in a later post.

Developers being skeptical about telemetry is nothing new. Just look at this issue discussion from 2016 titled, .NET core should not SPY on users by default. But the attitude towards telemetry seems to be shifting. If there is some benefit that comes from sharing this usage data (less bugs, more descriptive errors, relevant features), people are more willing to do it.

The majority agreed that an important factor is the recipient - who is the one collecting the data? Dan Callahan pointed out specifically, "Happy to send telemetry to non-profits and independent developers. Otherwise, there needs to be an exchange of value." But we don't all maintain projects as independent developers or work for non-profits. So then the question becomes what value exchange would be enticing enough for people to share their personal data.

In cases involving larger organizations, some mentioned that how the information is communicated and documented is what matters.

No matter how you spin it, the general consensus is that the data needs to be anonymous, it should be clearly documented and it must be able to be switched off easily (or opt-in if possible).

What's next?

This is the first in an undetermined number of posts about telemetry in open-source software. Throughout this series, we'll discuss a variety of topics - from building a telemetry feature to documenting it after.

If you have any requests for this series, please comment below!

DEV Community: Carolyn Stransky

Broccoli, bots, and brains! The 3 projects where I added the Hacktoberfest topic.

Hopeful COVID-19 Bot

carolstran / hopeful-covid-bot

🤖 Little bot that tweets uplifting COVID-19 news

Seeing Sprouts

carolstran / seeing-sprouts

🥗 Browser extension that replaces images with brussels sprouts

So you want to learn GraphQL?

carolstran / so-you-want-to-learn-graphql

✏️ 'Choose Your Own Adventure' but with GraphQL

Community Bonding with the GraphQL Foundation for Google Season of Docs

Attempted to figure out the organization structure

Triaged issues and pull requests

Helped with the Gatsby migration

Aligned on expectations for my project

What happens next?

Creating a digital CV in Markdown on GitHub

Create a new public repository on GitHub

Enable GitHub Pages for your repository

Select a theme (design template) for your CV

Write your CV in Markdown

Enjoy your new digital CV

Optional: Set up a custom domain

Show me your CV

Create a mock server for any REST API

Initial setup

(Optional) Create a virtual environment

Install HMT

1. Collect recordings of your API's traffic

Start HMT

Run API-specific curl commands

Check the recordings file

2. Build an OpenAPI specification from your recordings

3. Mock server traffic using your specification

Conclusion

Finding and fixing bugs in your tests with fast-check

Summary of the bug in our test

Fixing this bug with fast-check

Spoiler alert: Final solution

Setting up our test

Applying our properties

Processing the strings

Adding your assertion

Putting it all together

Special thanks

Testing apps with third-party integrations

Why should you test third-party integrations?

Three common (but misguided) ways to test APIs

1. Do nothing

2. Using the real APIs

3. Reverse engineering the API

Better ways to test third-party integrations

Write your tests with a mocking library

Use a service (like Meeshkan) to test your apps

Property-based testing for JavaScript developers

What's in this guide

Property-based testing in JavaScript: What and why

Differences between property-based and example-based tests

What your existing test suite probably looks like (and is missing)

Recognizing the limitations of example-based testing

Choosing which properties to test for

Writing your first property-based test with fast-check

Structuring your tests

Introducing fast-check

Testing your chosen properties

Examining the generated values

Available property-based testing frameworks

JavaScript

Other languages

Conclusion

Functional on the frontend with fp-ts and pipe

In this article:

Why we're going functional

Why we chose the fp-ts library

Working with our existing React codebase

Putting it into practice with pipe

Defining the initial value in a pipe

First function in the pipe

Another round of functions

Why we chose the `fp-ts` library

Putting it into practice with `pipe`

Defining the initial value in a `pipe`

First function in the `pipe`

More with `fp-ts`