DEV Community: James

Making API calls from Terraform

James — Tue, 18 May 2021 11:37:03 +0000

Terraform is a powerful and intuitive tool for managing infrastructure as code. It has excellent community support with a wide range of available providers from their registry. That said, sometimes a provider may not be up-to-date with the API they interact with, meaning that we can't provision the infrastructure as we like without manual intervention.

One stopgap (until the provider releases an update) is to call the API yourself from Terraform.

The easiest way to do this is to call a script using null-resource and local-exec

resource "null_resource" "update-my-resource" {
  provisioner "local-exec" {
    command = "node modules/myProvider/do-update.js ${provider_module.my-resource.id}"

    environment = {
      CLIENTID     = var.client_id
      CLIENTSECRET = var.client_secret
    }
  }
}

We won't go into the details of the (Node.js) script referenced in the above example, but the script effectively invokes an API and manages the response. In the above example it takes an argument, the ID of a resource previously created.

Unfortunately when working with Terraform Cloud Node.js and other script runners aren't available without installation.

Thankfully we can achieve the same using curl and a few Terraform helpers.

The example below demonstrates a multiple step flow, in this case a client credential flow.

Once again, we can use null_resource and local-exec, but this time we'll call out to the API using curl. We'll use triggers to specify an arbitrary set of values, that if changed will cause the resource to be replaced. The path defined in filename will be used to store the content of the API call for the following step.

resource "null_resource" "get_token" {
  triggers = {
    filename = "${path.module}/_temp_token.txt"
  }
  provisioner "local-exec" {
    command = <<EOT
      curl -X POST https://${var.domain}/oauth/token \
        -H 'content-type: application/x-www-form-urlencoded' \
        -d 'audience=https://${var.domain}/api/&grant_type=client_credentials&client_id=${var.client_id}&client_secret=${var.client_secret}' \
        >${self.triggers.filename}
    EOT
  }
}

We can then use local_file to create a reference to the files content. It'll also ensure that the provisioner completes before Terraform reads the file.

data "local_file" "token" {
  filename = "${null_resource.get_token.triggers.filename}"
}

One we have the data from the first API call we can call the second. This example uses jsondecode to extract an access token needed for the subsequent call.

resource "null_resource" "update-resource" {
  provisioner "local-exec" {
    command = <<EOT
      curl -X PATCH https://${var.domain}/api/thing/${provider_module.my-resource.id} \
      -H "Authorization: Bearer ${jsondecode(data.local_file.token.content).access_token}" \
      -H "Content-Type: application/json" \
      -d '{"foo":"bar"}' 
    EOT
  }
}

This is a "hack" to get around what is hopefully a short-term limitation, but it's useful to know that these options are available.

Spike APIs with Postman

James — Tue, 04 May 2021 11:27:52 +0000

Postman is an incredibly useful tool for exploring, collaborating on, and testing APIs and API design. For me, its strengths lay in the ease and flexibility of spiking out solutions with existing or new APIs.

We can investigate or design an API with Collections, a logical grouping of sequential API calls. An example can be seen below.

Variables support the flows we want to investigate, providing shared state between APIs calls. The variable can be specified at a global environment level or scoped to the collection.

Below is an example of a Pre-request script, the script is associated with a particular API call and runs before the API is invoked. In this example we evaluate Dynamic variables and then set collection variables that'll be used in this API call.

const orgId = pm.variables.replaceIn('{{$randomUUID}}');
const orgName = pm.variables.replaceIn('{{$randomCompanyName}}');

pm.collectionVariables.set("_orgname", orgName);
pm.collectionVariables.set("_orgid", orgId);

These variables are interpolate using the {{ var }} syntax.

More often though, we need the result of the API for a later call in the collection. This can be achieved with Tests which in this scenario is acting as a post-request script rather than running any assertions. Tests come into their own when we integrate with CI pipelines beyond the design phase.

const d  = JSON.parse(responseBody);

if(d?.access_token){
    pm.collectionVariables.set("_token", d.access_token);
}

We can use the same variable interpolation to set the Authorisation bearer token.

Before Postman I would have relied on prototyping in the browser and/or NodeJS to spike out API flows and architectures. Postman provides a much faster workflow, with the added benefit of feeding into other pre and post design activities including collaborative design and documentation.

Utility Storming

James — Wed, 24 Mar 2021 11:39:54 +0000

This article attempts to describe "Utility Storming" a lightweight collaborative approach to brainstorming the non-functional requirements of technical recommendations. Encouraging transparent and methodical system design within and across teams.

In an effort to de-risk projects and improve predictability of delivery, technical design activities are common throughout the software industry.
These activities attempt to minimise and improve resilience of bad decision-making.

The approach I'm about to describe is an evolution of various lean design processes I've participated in over the last decade, inspired by the ATAM Utility Tree and Scenario brainstorming and SEI's Quality Attribute Workshop.

Exploring the requirements and trade-offs of a single or group of related technical recommendations. A Utility Storming session is best done at a whiteboard with post-it notes, a collaborative session specifically for those responsible for the architecture of a system or application, usually the software engineers themselves. If physical whiteboards are not possible tools such as Miro also work well.

It's best to start the session by ensuring the team understand the context and problem. A team member will have done some preliminary investigation before hand, and have sufficient understanding to propose and explain a recommendation to the group.
The facilitator of the session calls out the primary quality attribute (usually Security, Performance or Modifiability), writing that attribute onto the whiteboard. The attribute can be randomly selected when there is no obvious starting place.
The group is asked to write on post-it notes the likely outcomes (of the given recommendation) that are pertinent to that quality attribute. These believed outcomes can be positive or negative and are grouped as such on the board. The post-its are placed horizontally next to the attribute. Duplicate outcomes are removed and similar outcomes grouped as one.
Some of the outcomes suggested may effect other quality attributes. The group is asked to call out those attributes. Each of the raised attributes are added to the board, and the related post-its moved to become aligned with those attributes.
The facilitator moves onto the next attribute on the board and repeats steps 2–4 until no-more quality attributes are called. The quality attributes on the board represent the utility of the recommendation. It's useful at this point to remind the group of undiscussed quality attributes, if the group feel they have missed something the newly discovered attribute should be added to the board and the process repeated.
From the top, for each of the outcomes the facilitator asks how relevant the outcome is to the success of the recommendation and encourages the team to define acceptance criteria, where that criteria is demonstrable upon completion.

The whiteboard can then be digitised, if legible enough a photo is sufficient. If the recommendation is ratified the decision can be written up, perhaps as an Architecture Decision Record and/or backlog tickets, linking back to the digitised Utility Storm for future reference.

In my experience, this approach improves engagement with technical decision making, increasing awareness and ownership of the systems non-functional requirements. Knowledge is shared earlier making backlog refinement and sprint planning more productive, aiding the breakdown of tickets which in turn improves the predictability of delivery. It helps democratise software architecture.

Your mileage with "Utility-storming" may vary depending on the team and the recommendation. Like anything it requires practice but I'd recommend you give it a try.

Creating a Github comment from Netlify

James — Tue, 02 Mar 2021 16:58:10 +0000

Using octokit its possible to interact with Github from CI pipelines. The example below adds a comment to a PR.

const {
  env: { OWNER, ACCESS_TOKEN, REPOSITORY_URL, REVIEW_ID, PULL_REQUEST },
} = require('process');

function init() {
  const { Octokit } = require('@octokit/rest');

  const octokit = new Octokit({
    auth: ACCESS_TOKEN,
  });

  return {
    createComment: async (comment) => {
      const config = {
        owner: OWNER,
        repo: REPOSITORY_URL.split('/').pop(),
        issue_number: REVIEW_ID,
      };

      await octokit.issues.createComment({
        ...config,
        body: comment,
      });
    },
  };
}

And here's how you might use that in a Netlify plugin.

module.exports = {
  onSuccess: async () => {
    const { createComment } = initialiseGithub();
    await createComment('The Netlify build has succeeded');
  },
  onError: async () => {
    const { createComment } = initialiseGithub();
    await createComment('The Netlify build has failed');
  },
};

Send metrics to Datadog from Netlify builds

James — Tue, 02 Mar 2021 16:31:20 +0000

I previously wrote about gathering Lighthouse audit metrics within CI. These and other custom metrics can be sent to Datadog for analysis without needing the Datadog agent. The implementation below uses Dogapi to send custom metrics via the Datadog API.

const {
  env: { DATADOG_API_KEY, VERSION, ENVIRONMENT },
} = require('process');

function createMetricReporter({ service }) {
  const dogapi = require('dogapi');
  const defaultTags = [`env:${ENVIRONMENT}`, `releaseVersion:${VERSION}`, `service:${service}`];
  const metrics = [];

  dogapi.initialize({
    api_key: DATADOG_API_KEY,
  });

  return {
    gauge: (metric, points, tags = []) => {
      metrics.push({
        metric,
        points,
        tags: [...tags, ...defaultTags],
        type: 'gauge',
      });
    },
    send: () => {
      const sendPromise = new Promise((resolve) => {
        dogapi.metric.send_all(metrics, (error, result) => {
          if (error) {
            console.log(`Unable to send metrics to Datadog`, error);
          } else {
            console.log(`Successfully sent metrics to Datadog`, result);
          }

          resolve(!error);
        });
      });

      return sendPromise;
    },
  };
}

Below is a snippet of what the Netlify plugin might look like.

module.exports = {
  onSuccess: async ({ inputs: { service, url, budget } }) => {
    const metrics = createMeticsReporter({ service });
    const results = await lighthouse(url, budget);

    for (const result of results) {
      const tags = [`formFactor:${result.formFactor}`];
      metrics.gauge('lighthouse.accessibility', result.accessibility, tags);
      metrics.gauge('lighthouse.bestPractices', result.bestPractices, tags);
      metrics.gauge('lighthouse.performance', result.performance, tags);
      metrics.gauge('lighthouse.pwa', result.pwa, tags);
      metrics.gauge('lighthouse.seo', result.seo, tags);
    }

    await metrics.send();
  },
};

Programmatically audit with Lighthouse and performance budgets

James — Tue, 02 Mar 2021 16:09:13 +0000

Lighthouse is a fantastic tool for auditing websites. Not only can it be run in Chrome Devtools and on the CLI, but also programmatically! This is especially useful in CI pipelines where we can use Lighthouse to check metrics such as Core Web Vitals on local or ephemeral environments, in order to warn the engineer(s) that quality attributes have degraded before going to production.

const PORT = 8041;

function createAuditor(budget) {
  const flags = {
    port: PORT,
    disableStorageReset: true,
  };
  const optionsArray = [];
  const mobileOptions = {
    extends: 'lighthouse:default',
    settings: {
      budgets: budget && [budget],
    },
  };
  optionsArray.push(mobileOptions);

  return async function audit(url) {
    const lighthouse = require('lighthouse');
    const puppeteer = require('puppeteer');
    const browser = await puppeteer.launch({
      args: [`--remote-debugging-port=${PORT}`],
      headless: true,
    });
    const results = [];

    for (const options of optionsArray) {
      const runnerResult = await lighthouse(url, flags, options);
      const budgetReport = getBudgetReport(runnerResult.report);
      const { categories, finalUrl, configSettings } = runnerResult.lhr;

      results.push({
        url: finalUrl,
        formFactor: configSettings.formFactor,
        accessibility: categories.accessibility.score * 100,
        bestPractices: categories['best-practices'].score * 100,
        performance: categories.performance.score * 100,
        pwa: categories.pwa.score * 100,
        seo: categories.seo.score * 100,
        budgetReport,
      });
    }

    await browser.close();
    return results;
  };
}

function getBudgetReport(result) {
  const report = JSON.parse(result);
  const getOverBudget = (item) => item.countOverBudget || item.sizeOverBudget || item.overBudget;
  const perfBudget = report.audits['performance-budget'];
  const timingBudget = report.audits['timing-budget'];
  const budgetReport = [];

  if (perfBudget && perfBudget.details) {
    const perf = perfBudget.details.items.filter(getOverBudget);
    budgetReport.push(...perf);
  }

  if (timingBudget && timingBudget.details) {
    const timings = timingBudget.details.items.filter(getOverBudget);
    budgetReport.push(...timings);
  }

  return budgetReport;
}

The example implementation uses Puppeteer, as this would allow us to interact and navigate before beginning the audit. The audit method returns the results including metrics that have not met the budget requirements, allowing us to fail a build and/or report metrics.

Below is an example of the Lighthouse performance budget.

{
    "resourceCounts": [
        {
            "resourceType": "script",
            "budget": 15
        }
    ],
    "resourceSizes": [
        {
            "resourceType": "script",
            "budget": 180
        }
    ],
    "timings": [
        {
            "metric": "interactive",
            "budget": 4500
        },
        {
            "metric": "first-contentful-paint",
            "budget": 1300
        }
    ]
}

I've written about sending these metrics to Datadog in a following post.

Netlify build logging with Datadog

James — Tue, 02 Mar 2021 15:39:25 +0000

Netlify's build plugins allow us to to perform a variety of custom build/deploy related tasks. One such task might be to log deployment status in Datadog for analysis. Thankfully this is easy using agent-less logging.

Here's what a simple log plugin might look like.

const createLogger = require('./logger');

module.exports = {
  onSuccess: async ({ inputs: { service } }) => {
    const logger = createLogger({ service });
    logger.info('Deploy successful');
    await logger.send();
  },

  onError: async ({ inputs: { service } }) => {
    let logger = createLogger({ service });
    logger.info('Deploy failed');
    await logger.send();
  },
};

And below, a logger implementation using Winston.

const {
  env: { DATADOG_API_KEY, VERSON, ENVIRONMENT },
} = require('process');

function createLogger({ service }) {
  const { createLogger, format, transports } = require('winston');
  const queryString = require('query-string');
  const { combine, json, timestamp } = format;

  const query = queryString.stringify(
    {
      ddsource: `nodejs`,
      service,
      ddtags: [`env:${ENVIRONMENT)}`, `releaseVersion:${VERSON)}`],
    },
    { arrayFormat: 'comma' },
  );

  const httpTransportOptions = {
    host: 'http-intake.logs.datadoghq.com',
    path: `/v1/input/${DATADOG_API_KEY}?${query}`,
    format: combine(timestamp(), json()),
    ssl: true,
  };

  const logger = createLogger({
    level: 'info',
    exitOnError: false,
    transports: [new transports.Http(httpTransportOptions)],
  });

  return {
    info: (...args) => logger.info(...args),
    send: () => {
      const completePromise = new Promise((resolve) => {
        logger.on('finish', () => {
          clearTimeout(timeout);
          console.log(`Successfully sent logs to Datadog`);
          resolve(true);
        });
      });

      logger.end();
      return completePromise;
    },
  };
}

Typescript design token docs in Storybook

James — Wed, 03 Feb 2021 15:39:47 +0000

Storybook is great for documenting UI components but what about data structures such as a themes and design tokens? Here's a super quick way to add design token documentation into your Storybook.

Using MDX and ArgsTable

In designtoken.stories.mdx:

import { Meta, Story, Canvas, ArgsTable } from '@storybook/addon-docs/blocks';
import { Theme, Breakpoints } from './helpers';

<Meta title="Design Tokens/Theme" />

### Theme

<ArgsTable of={Theme} />

#### Breakpoint Device Types

<ArgsTable of={Breakpoints} />

<Canvas>
  <Story name="Theme">
    <span>See Docs tab</span>
  </Story>
</Canvas>

Where helper.tsx is:

import { ThemeType, theme } from '../DesignTokens';

export const Theme: (Theme: ThemeType) => void = () => {};

export const Breakpoints: (Breakpoints: typeof theme.Breakpoints) => void = () => {};

This super simple approach allows us describe important contracts throughout our design system implementation, without the need to manually write and maintain that documentation. Below is a screenshot of how that might look.

Spacing with Root EM

James — Thu, 10 Dec 2020 17:33:11 +0000

Consistent spacing across a website can make or break a design. When implemented correctly the visual quality of the page will really shine through.

I've pretty consistently been working with an 8px base spacing system for over a decade. It works, but using pixels as a unit of measurement can create a headache of complexity when building viewport/device responsive experiences.

Of course we've had the EM unit, but I've never liked it. The idea that the same component will look different depending on where it is placed seems fundamentally broken to me... And yes, the cascade in CSS is a fantastic tool, but I've always been against cascade as a doctrine, as "best practice". In my opinion, isolation through components will always be a better methodology to fall back on.

The REM unit give us the best of both worlds, pre-determined sizes that are consistent across implementations of a component, relative only to the root font size. But what about scale?

I can't suggest what scale would work for you when defining a spacing system, but it's important to have one. A codified scale will guide developers into using a consistent set of spaces, rather than simply random values. If you find yourself using custom values it probably means the scale is not good enough. But when the scale works, developers will work faster, and in my experience the page design falls into place almost effortlessly.

The scale I defined on a recent project is a simple array of values defined as TypeScript design tokens and shared using an Emotion theme. Linear increments of 0.25rem then switching to a fibonacci scale from 3rem, this gives plenty of options for internal component spacing and nudges, whilst still supporting the larger values needed for page layouts. Because I'm using REM, the actual pixel size will differ based on the base font-size, allowing me to holistically change the page layout based on viewport, page, and theme.

Below are a couple of screenshots from internal documentation (using Storybook). The first shows the scale using a 18px base unit for large viewports.

And the image below shows a 16px base for smaller viewports.

So far, so good. But it's hard to tell early in a project if and how this approach will pan out. I have worries about using an 18px base, but it's going to be easy to change if we want to. e.g. making 1rem equal 20px is literarily a one line change with a huge aesthetic impact. It's currently helping the team deliver faster and worry less about design, and that I think can only be a good thing.

Mocking components in Jest

James — Tue, 08 Dec 2020 18:23:44 +0000

Mocking React components and modules is something I find myself doing often in Jest. I’m instinctively weary of mocks, test doubles can create a false sense of security if misused, but they are incredibly important when dealing with third party libraries and "Inner-Source" packages, whilst avoiding the indirection and complexity of other techniques such as Dependency Injection. I don’t personally believe that mocking/patching in anyway compromises a clean architecture.

Below is a common mock in the codebase I'm working on. Next.js links will fail with an TypeError without a RouterContext. Either we add the router or mock the component, the latter seems easiest and prevents the test from knowing too much about what's needed underneath.

jest.mock('next/link', () => ({ children }: { children: JSX.Element }) => children);

Of course that'll work on any component import.

Unfortunately if you want to return some JSX in the mock the react/display-name ESlint rule will kick in. We can ignore the rule or perhaps define the mock like this:

jest.mock('../module', () => function Named() { return <div />; });

Oh, and named exports? The above examples show default exports. The example below takes Stripe's CardElement and return a no-op component.

jest.mock('@stripe/react-stripe-js', () => ({ CardElement: () => null }));

Typescript & reducer actions

James — Tue, 24 Nov 2020 18:41:55 +0000

I'm still learning Typescript. I like it, but sometimes it can feel like it's getting in the way. With React state reducers it seems all too easy to duplicate action types.

The following example tidies up the use of dispatch actions by taking an array of action names, creating a Union type from that array, and then creating an Action and Record type from that Union.

const actionsTypes = [
  'COMPONENT_STATE_VALID',
  'COMPONENT_STATE_LOADING',
  'COMPONENT_STATE_COMPLETE',
  ...
];

type ActionTypeUnion = typeof actionsTypes[number];
type Action = { type: ActionTypeUnion };
type Actions = Record<ActionTypeUnion, Action>;

const actions: Actions = actionsTypes.reduce(
  (acc, type) => ({ ...acc, [type]: { type } }),
  {},
);

This is pretty concise and gives us great type support when implementing an action e.g. dispatch(actions.COMPONENT_STATE_VALID)

Hey HSLA!

James — Thu, 19 Nov 2020 18:36:45 +0000

It's been interesting to see the industry slowly shift towards HSL(A) (Hue, Saturation, Light, Alpha). I've recently had the luxury of building a design system from scratch, and decided to use HSL. Given another year I may have chosen LCH instead, but for now HSL seems like a flexible way to manage colours whilst enjoying good browser support.

Given that we have a brand colour that is dark red.

const brandColour = [0, 100, 20]; // "hsl(0, 100%, 20%)"

I can easily "rotate" the hue and create new colours that match the brand. Or we can manipulate the saturation of the colour and lightness to derive various shades.

const contrastColour = [200, 100, 20]; // dark blue
const primaryHighlightColour = [200, 20, 90]; // light pink

This allows us to programmatically define the rules of the colour system. Reducing colour "bloat" and making holistic change easy and safe.

These colours can then be exposed, either as CSS custom properties or through a CSS-in-JS theme. Either way I'll not add the hsl() function to the variable itself.

--brandPrimary: 0, 100%, 20%

When we implement a colour we may need disabled states and other variations. It's useful to have a final bit of flexibility here. By storing the values rather than the function we can easily use the hsla() function instead.

button {
  background-color: hsl(var(--brandPrimary));
}
button:disabled {
  background-color: hsla(var(--brandPrimary), 50%);
}

It will be interesting to see how this approach evolves, but right now the design system I'm building uses just six core brand colours, from which all others are derived. This is simpler by an order-of-magnitude compared to systems I've worked on previously.