DEV Community: Stephen Charles Weiss

Configuring The AWS CLI

Stephen Charles Weiss — Tue, 28 Apr 2020 00:00:00 +0000

Prerequisites

Before we can configure the AWS CLI, we need to gather a few pieces of information first:

Access key ID
Secret Access Key ID

If you do not have these, you can generate them by going to the Identity and Access Management (IAM) service in your AWS Console.From there, select Access management > Users in the menu.

Select the user for which you’d like to configure the AWS CLI.

Select Security credentials and then under Access keys select “Create access key”.

This is the only time you’ll be able to view the Secret Access Key ID so be sure to write it down or download the CSV.

Now that we have these IDs, we’re ready to proceed.

Simple Use Case: One User

If you only have one user that you need to configure, you can use the default process:

$ aws configure
AWS Access Key ID [None]: <Your Access Key ID>
AWS Secret Access Key [None]: <Your Secret Access Key ID>
Default region name [None]: us-west-2
Default output format [None]: json

This will store the settings in the default profile which is used every time an AWS CLI command is run unless otherwise specified.

Command Line Options

aws configure can take three different options:

--region - referencing the AWS region to send data to. It defaults to the closest, however can be specified, e.g., us-east-1.
--output - specifies the format of the output. Options are json, yaml, text, and table.
--profile - adds a named profile to the AWS CLI configuration.

Multiple Users

Imagine we have two users we need to switch between on a single machine (maybe they’re in different regions): user1 and user2

We can configure them with the following:

$ aws configure --profile user1 --region us-east-1 --output json
$ aws configure --profile user2 --region us-west-2 --output json

(Not shown is adding the Access and Secret Access Key IDs.)

Now, when we need to run an AWS CLI command, we can do so like:

$ aws s3 ls

to use the default user or with a specified profile like:

$ aws s3 ls --profile user1
$ aws s3 ls --profile user2

AWS IAM In 30 Seconds

Stephen Charles Weiss — Tue, 28 Apr 2020 00:00:00 +0000

Preamble: I am exploring different areas of AWS and am finding the jargon overwhelming. To help myself, I’ll be writing up summaries of some of the different services I come across, what they are, some core concepts, and links to resources for future reference.

All entries in the series are tagged with aws in 30 seconds and can be found here.

Introduction to AWS Identity and Access Management | AWS Training

High level: Identity & Access Management (IAM) is a service provided by AWS to manage individual and group access to AWS resources. This is accomplished by authenticating and authorizing users based on IAM users, roles and/or groups.

Authentication

There are three different ways that a user can authenticate (a prerequisite to accessing any AWS resources):

AWS CLI
AWS SDKs
AWS Management Console.

Authorization

By default, all resources are private to a user’s account. To grant permissions IAM users and groups must be granted access via IAM policies.

Policies

An IAM Policy is a JSON document that records the effects, actions, resources and optional conditions for what API calls a policy holder may invoke.

Anything not explicitly listed is not permitted. Here’s a sample policy:

Roles

In addition to Groups, Policies, and Users, there are also IAM Roles.

Roles are similar to Users in that they can be used to determine which resources are accessible, however they are not uniquely associated with an individual and they have no standard login credentials.

They are used to delegate access to a user temporarily. When a user assumes a role, they give up their own permissions and take on the permissions of the role.

Utilizing IAM Roles avoids the need to modify a specific user’s permissions each time a change is required.

Wrap Up

Putting this into practice. Imagine setting up the permissions for an organization.

Create an IAM Group or groups
Create IAM Policy (or policies) and assign them to the group(s)
Create IAM Users for each individual and assign them to their respective IAM GroupIn this way, user permissions are efficiently managed.

If a user only needs temporary access to certain resources, IAM Roles are a good bet. They are also helpful in avoiding embedding credentials directly into an application.

Resources

Memoization: A Primer

Stephen Charles Weiss — Mon, 02 Mar 2020 00:00:00 +0000

Preamble

I’ve been spending more time with them lately as a way to remind myself of some of the parts of programming I don’t have a chance to interact with on a daily basis. As with most things, when I learn things, or want to remember them, I write them down.

I’ll likely write a few and post them on my website, all tagged with CS Fundamentals to make them easier to find.

Today’s topic is memoization. In this post, we’ll be looking at two classic problems to understand how to identify good candidates for a memoized approach. Later, we’ll explore different memoization techniques to understand how they provide value.

What Is Memoization?

Let’s begin with a definition. From Wikipedia:

In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again. Memoization has also been used in other contexts (and for purposes other than speed gains), such as in simple mutually recursive descent parsing.[1] Although related to caching, memoization refers to a specific case of this optimization, distinguishing it from forms of caching such as buffering or page replacement. In the context of some logic programming languages, memoization is also known as tabling.[2]

When Is Memoization Useful?

So, to memoize is to remember. What are we remembering? When is it useful?

To memoize a function is to remember the output given a set of inputs. It’s useful whenever we find ourselves doing the same calculation multiple times.¹

It’s an optimization strategy to reduce the time complexity (described with Big O notation) of an algorithm.

Drawbacks Of Memoization

Memoization is the trade off of space for speed. When a computer “remembers” it does so by storing that value in memory.

At some point, that memory could grow unwieldly. Not discussed here is when or how to start cleaning up the cache though two common strategies are Least Recently Used (LRU) and Least Frequently Used (LFU). This conversation on Stack Overflow provides a nice example of the differences between LRU and LFU.

Examples: Fibonacci And Factorials

For the rest of this post, we’ll be exploring memoization by examining what is required to find the n^th number in the Fibonacci sequence and calculating the n^th factorial

The Fibonacci sequence is defined as the value of the sum of the previous two numbers and the first two values in the sequence are 1. This can be written as:

function fibonacci(num) {
  let previousLow = 1
  let previousHigh = 1

  for (let i = 2; i < num; i += 1) {
    const tempHigh = previousHigh
    previousHigh = previousHigh + previousLow
    previousLow = tempHigh
  }
  return previousHigh
}

Calculating a factorial is accomplished by multiplying all non-negative numbers less than or equal to the desired number.

function factorial(num) {
  let result = 1
  while (num >= 0) {
    const value = num === 0 ? 1 : num // convention states that 0! is 1
    result *= value
    num -= 1
  }
  return result
}

These implementations actually have a lot going for them. They’re fairly efficient (from a time complexity perspective, they’re both O(n) and keep space requirements to a minimum by discarding past calculations).

Memoization, however, is a tactice of dynamic programming - which, simply stated, is the practice of reducing a problem into sub-problems and solving them recursively.

Those sub-problems are: the sum of the two previous numbers for the Fibonacci sequence, and multiplying a number by one less than the number for factorials.²

Using Dynamic Programming, we can refactor these functions into their recrusive forms. That can look like the following:

function fibonacci(num) {
  if (num === 0 || num === 1) return num
  return fibonacci(num - 1) + fibonacci(num - 2)
}

function factorial(num) {
  if (num === 0) return 1 // convention states that 0! is 1
  return num * factorial(num - 1)
}

In refactoring to a recursive solution, we traded simpler logic for potentially more function calls.

If we diagram the function calls to calculate the fifth number in the fibonacci sequence vs the fifth factorial, the differences will become apparent.

                                                    fibonacci(5)
                                       _____________/ \____________
                           fibonacci(4) fibfibonacci(3)
                        __/ \__ / \
            fibonacci(3) fibonacci(2) fibonacci(2) fibonacci(1)
           / \ / \ / \
    fibonacci(2) fibonacci(1) fibonacci(1) fibonacci(0) fibonacci(1) fibonacci(0)
   / \
fibonacci(1) fibonacci(0)

Similaraly, calculating the fourth factorial, you might wind up with:

    factorial(5)
    / \
   5 factorial(4)
               / \
              4 factorial (3)
                        / \
                       3 factorial (2)
                                 / \
                                2 factorial (1)
                                          / \
                                         1 factorial (0)

In calculating a fibonacci number, we make the same call multiple times.

function	# of calls
fib(5)	1
fib(4)	1
fib(3)	2
fib(2)	3
fib(1)	5
fib(0)	3

Said another way, the time complexity of calculating a fibonacci number with this algorithm is exponential. Each additional step (approximately) doubles the number of steps required to solve it. This would be written as O(2ⁿ) where n is the sequence we’re seeking.³

Wouldn’t it be nice if we could remember the value of fib(1) so that we don’t have to make the call again? Or even better, once we solve fib(3), we could eliminate an entire branch of the call tree!

That’s what memoization can offer. We’ll get to this in a moment.

What about factorial? We don’t have that same repetition and the time complexity reflects that too. It continues to grows linearlly with the number we’re seeking to calculate. That is, it’s still O(n).

The value of memoization here won’t be reducing calls the first time, but subsequent calls. Imagine calling factorial(3) or factorial(6) next? If the computer remembered its previous calculations, it could immediately retrieve the answer for factorial(3) (which was needed to calculate factorial(5)) and factorial(6) would be reduced to 6 * factorial(5) (where it can look up factorial(5) immediately too).

Now that we’ve seen examples of refactoring a problem into a recursive solution and identified areas where momoization might be useful, let’s talk about strategies!

Strategies For Memoization

Practically, to memoize a function means that each time a calculation is performed, its result is stored in a persistent state that future calls can access. Let’s call that state a cache.

We have options when it comes to creating the cache. Two we’ll look at are:

Refactor to create a cache and an internal function for calculating as necessary
Use a generalized memoization function which creates a closure with a cache present

The problem itself can provide a suggestion about the appropriate approach, but both take advantage of storing previously calculated results in a cache.

Aside On Cache Implementation

How you implement the cache is up to you, though its helpful to consider the characteristics of different data structures for picking. For example, dictionaries is a great choice because of the ability to look up values in constant time (as opposed to a list where you might have to iterate over each item).⁴

Internal Recursion With Cache

By internal recursion, I mean create a new function that can be accessed by our fibonacci function. This “internal” function holds all of the recursive logic, but the signature is modified to receive a cache that we define in our enclosing function.

This would be useful in the case of the Fibonacci sequence where our recursive approach created multiple redundant calls. It might look like (interactive repl here):

function fibonacci(num) {
  const cache = {}
  return calculateFibonacci(num, cache)
}

function calculateFibonacci(num, cache) {
  if (!cache[num]) {
    if (num === 0 || num === 1) {
      cache[num] = num
    } else {
      cache[num] =
        calculateFibonacci(num - 1, cache) + calculateFibonacci(num - 2, cache)
    }
  }
  return cache[num]
}

By creating the cache and passing it along to the new calculateFibonacci function, we can add a check first on the cache. If it’s present, we can skip right along to return the value.

Notice, however, that this provides no benefit for factorial because there’s no redundancy in the calls. We’ll need a different approach to gain benefits for that. Let’s look at that now.

Generalized Memoization Function

If the situation is such that we’ll be making the same computationally expensive call over and over, another opportunity is to memoize the results of the call itself so that if you see it again, the computer won’t need to crunch through it over and over.

function memoization(fn) {
  const cache = {}
  return (...args) => {
    const key = [...args]
    if (!cache[key]) {
      cache[key] = fn(...args)
    }
    return cache[key]
  }
}

It’s worth noting that this solution only helps us with the case where we’re calling the same function over and over.

For example. If we tweaked the factorial function to print each time it’s called, like so:

function factorial(num) {
  console.log(`factorial called with -->`, num)
  if (num === 0) return 1 // convention states that 0! is 1
  return num * factorial(num - 1)
}

And then calling it multiple times, such as:

console.log(memoizedFactorial(5))
console.log(memoizedFactorial(6))
console.log(memoizedFactorial(3))
console.log(memoizedFactorial(5))

We’d see a series of calls for each invocation except the final memoizedFactorial because we will have already stored the call of factorial with the argument(s) of 5 in the cache:

// factorial called with --> 5
// factorial called with --> 4
// factorial called with --> 3
// factorial called with --> 2
// factorial called with --> 1
// factorial called with --> 0
// 120
// factorial called with --> 6
// factorial called with --> 5
// factorial called with --> 4
// factorial called with --> 3
// factorial called with --> 2
// factorial called with --> 1
// factorial called with --> 0
// 720
// factorial called with --> 3
// factorial called with --> 2
// factorial called with --> 1
// factorial called with --> 0
// 6
// 120

Here’s a repl if you’d like to play with it for yourself. Included in the repl is Lodash’s memoize function, which you’ll see performs similarly, though, unsurprisingly, includes additional features and optimizations - more on these in the advanced usage section.⁵

Combined Approach

These two approaches (the internal and the general) can be combined to create even greater savings, though it requires a slightly more bespoke approach.

Because the cache is generated outside of the scope of the target function, we’ll need to pass it in as an argument. And then we’ll need to continue to forward it along all the way down.

Our generalized memoization function looks very similar to our previous one, though we’re now passing along the cache into the bound function:⁶

function memoize(fn) {
  const cache = {}
  return function() {
    const key = [...arguments]
    if (cache[key]) {
      return cache[key]
    } else {
      const val = fn.apply(this, [...arguments, cache])
      cache[key] = val
      return val
    }
  }
}

This allows us to memoize the internal calls and store them in the same cache - similarly to how we memoized the Fibonacci sequence earlier:

function factorial(num, cache) {
  if (num === 0) cache[num] = 1 // convention states that 0! is 1
  if (!cache[num]) {
    cache[num] = num * factorial(num - 1, cache)
  }
  return cache[num]
}

Now, each call - both the initial call and every recursive invocation will be stored in the cache on its first appearance. (repl for live testing.)

Advanced Usage: Multiple Arguments

One potential trip up when it comes to memoization can occur when the function takes multiple arguments. Our generalized memoization solution actually handles this case, but our internal one doesn’t.

Can you spot the difference?

It’s how the keys are generated.

Imagine we weren’t memoizing factorial, but add, where add is defined as:

function add = (a,b) => a + b;

In this case, we can’t just write:

function add = (a,b) => {
  if (!cache[a]) {
    cache[a] = a + b
  }
  return cache[a]
}

What happens when b changes but a stays the same? We’d retrieve the wrong result from the cache!

There are any number of ways to create a key, the only rule is that it needs to be unique to the given arguments of the function. If you’re dealing with strings, numbers, etc. (i.e., Primitive data types in Javascript other than a Symbol), you might create a concatenated string:

function add = (a,b) => {
  const key = [a,b].join('-')
  if (!cache[key]) {
    cache[key] = a + b
  }
  return cache[key]
}

If the data types include objects, or arrays, however, you might need a more custom resolver to create the key.⁵

Conclusion

That wraps up this primer on memoization. Hopefully you found it useful and learned a thing or two. If anything wasn’t clear (or worse, I am mistaken on any detail) - please let me know!

Resources

Footnotes

¹ This type of solution in which a larger problem is broken down into smaller pieces is called Dynamic Programming. Memoization is one tactic used to improve the performance of a solution using dynamic programming.
² Both of these sequences also have base cases which are important and which we’ll get to in a moment.
³ A more precise Big O calculation would actually be O(1.6ⁿ)
⁴ A POJO (plain-old Javascript object) is an easily accessible implementation of a Dictionary with constant time lookup. For an implementation that’s likely closer to other languages, look into Map which has the benefit of:
1. Being iterable,
2. Having guaranteed key order,
3. Built in getters/setters,
4. Can handle keys of any value (not just Strings or Symbols).
⁵ In reviewing the Lodash implementation of memoize, I was most impressed by how John-David Dalton and the community addressed the issue of multiple arguments with an optional second argument of a custom resolver. The documentation didn’t offer an example of how it might be used, but fortunately Justin Noel already wrote a post detailing it how to use a custom resolver with Lodash’s memoize. The crux is:
```
const _ = require('lodash')
const add = (a, b) => a + b
const memoizedAdd = _.memoize(add, (...args) => args.join('-'))
```
The resolver creates a unique key for each combination of arguments (a and b in this case) by joining them together with a -.
⁶ For demonstration purposes, this time I refactored slightly to use an anonymous function definition instead of an anonymous arrow function. Two differences emerge:
1. Function definitions have an arguments array which can be used to collect any arguments passed to a function. Function declarations do not have this, however they have the rest operator, which is why we used ...args in the previous example.
2. We need to bind the context when using the function definition. This is because context of this is set at invocation for function definitions, whereas it’s set at definition for arrow functions. The .apply in this example binds it to the context in which it’s defined so that later, when it’s invoked, it will still have access to the cache.

Global Themes, Variants, And Modes With Styled-Components

Stephen Charles Weiss — Mon, 10 Feb 2020 00:00:00 +0000

If I were to list my strengths, web design wouldn’t top the list. None the less, I like learning about it and know that I’ll continue to improve over time by chipping away.

One thing I’ve been struggling with since adding a dark mode to my site (I look forward to making this a much more pleasant experience soon), was how the mode was intended to interact with the global theme. And then what happens if you want variants for components?

I put together a CodeSandBox to play around with the concepts.

Because it’s such a simple single page, I can see how all of the pieces come together and interact. It’s also made clear why my coworker, Justin Connor, put together a library called styled-variants!

Dealing with variants can quickly become a hassle - particularly if you’re duplicating the logic between modes.

The full code is below.

import React from "react";
import "./styles.css";
import styled, { ThemeProvider } from "styled-components";

const theme = {
  light: {
    bg: "powderblue",
    fs: "72px",
    maxWidth: 1000
  },
  dark: {
    bg: "red",
    color: "white",
    fs: "72px",
    maxWidth: 1000
  }
};

const buttonVariant = {
  light: {
    primary: {
      boxShadowColor: "teal"
    },
    secondary: {
      boxShadowColor: "grey"
    }
  },
  dark: {
    primary: {
      boxShadowColor: "orange"
    },
    secondary: {
      boxShadowColor: "orange"
    }
  }
};

const Title = styled.h1`
  background-color: ${({ theme, mode }) => theme[mode].bg};
  font-size: ${props => props.theme[props.mode].fs};
  color: ${({ theme, mode }) => (mode === "dark" ? theme[mode].color : "")};
`;

const Button = styled.button`
  box-shadow: 0.25rem 0.25rem
    ${({ theme, mode, variant }) => theme[mode][variant].boxShadowColor};
`;

const Wrapper = styled.div`
  background-color: ${({ theme, mode }) => theme[mode].bg};
  padding: 1rem;
`;

function App() {
  const [mode, setMode] = React.useState("light");
  function handleClick() {
    const nextMode = mode === "light" ? "dark" : "light";
    return setMode(nextMode);
  }

  return (
    <ThemeProvider theme={theme}>
      <Wrapper className="App" mode={mode}>
        <Title mode={mode}>Hello CodeSandbox</Title>
        <h2>The current mode is: {mode}</h2>
        <Button
          mode={mode}
          variant={"primary"}
          theme={buttonVariant}
          onClick={handleClick}
        >
          Toggle Mode
        </Button>
      </Wrapper>
    </ThemeProvider>
  );
}

export default App;

Another Angle To Understand Global State With React: Next.js

Stephen Charles Weiss — Sun, 09 Feb 2020 00:00:00 +0000

Recently, I was playing around with Next.js (simply referred to as Next from here on) to see how a framework might make building React applications simpler.

When I got to the _app.js page a light bulb went off. The page illustrates how React applications manage global state.

First, let’s understand the problem we’re trying to solve: If you refresh a page running a React application, all of the state is blown away.

That means that React applications need to provide the appearance of navigating to different pages, updating the URL, creating a history, etc. without actually loading a brand new page. (Coincidentally, this helped me understand why a good router is so important.)

Now, coming back to Next, let’s examine the _app.js component. This is provided by default by Next, but can be overridden with a custom file as needed.

That’s what I’m doing here:

function MyApp({ Component, pageProps }) {
  return (
    <>
      <p>I'm on every page!</p>
      <Component {...pageProps} />
    </>
  )
}

export default MyApp

Yeah, it’s not much. I’m simply adding a paragraph tag before every <Component> is rendered.

The <Component> is the page that’s loaded. pageProps are any props passed along to the Component - it’s an empty object if not using getInitialProps.¹

The effect, however, is that when we navigate between pages, the state that exists in MyApp persists - it is not being rerendered.

To see this, here’s an example:

Notice in the video that the <p> doesn’t change or get reloaded even though the page contents change?

If we were to put our state management, global styles, etc. at this level then, they can be loaded once and persist even while different pages are loaded underneath.

Footnotes

¹ .getInitialProps is part of the Next API to asynchronously fetch data and then feed it to props. The big catch on using it is that it will disable Automatic Static Optimization.

Git Commit Archeology: Digging In With Grep And Pickaxe

Stephen Charles Weiss — Wed, 05 Feb 2020 00:00:00 +0000

In his talk “A Branch In Time” (which I wrote about previously), Tekin Süleyman discussed how an engineer can explore the git history to understand the context of changes (assuming her peers wrote meaningful commit messages).

Two tools that left me slackjawed when I saw them in action were grep and the pickaxe.¹

I’ve been playing with both over the past few weeks and wanted to jot down the how so that I wouldn’t forget.

Grep

If the team is writing useful commit messages, then grep’ing them will be a valuable exercise.²

Git’s grep option in the git log allows searching of the entire commit history.

From the manual:

--grep=<pattern>
   Limit the commits output to ones with log message that matches the specified pattern (regular expression). With more than one --grep=<pattern>, commits whose message matches any of the given patterns are chosen (but see --all-match).
   When --show-notes is in effect, the message from the notes is matched as if it were part of the log message.

For example, looking through the commit history on this site for

$ git log --grep "feat:"

I can see a full list of all of the commits which include feat in the subject or body.

This is particularly useful if used in conjunction with Conventional Commits or some other standard commit template.

As the manual suggests, you can also stack patterns to match.

$ git log --grep "feat:" --grep "major"

By default, the patterns are inclusive, however, they can be made exclusionary (that is all required) with the inclusion of the --all-match option:

$ git log --grep "feat:" --grep "major" --all-match

Pickaxe

I think of the Pickaxe like grep, but for the diff itself. The pickaxe actually comes from diffcore (the family of commands that are used to compare two files), but is available for use within the git log.

Specifically, the two standard ways to use the pickaxe are with the -S and the -G flags.

-S takes a string as an argument
-G takes a regular expression

While this sounds intimidating. In practice it’s not.

One key difference between the two options and noted in the manual is the scope of concern for the two options. Whereas the -G looks for changes in the pattern, the -S seems to be concerned only with the number of changes of the pattern.

From the manual:

To illustrate the difference between -S —pickaxe-regex and -G, consider a commit with the following diff in the same file:
+ return !regexec(regexp, two->ptr, 1, &regmatch, 0);
...
- hit = !regexec(regexp, mf2.ptr, 1, &regmatch, 0);
While git log -G”regexec(regexp” will show this commit, git log -S”regexec(regexp” —pickaxe-regex will not (because the number of occurrences of that string did not change).

Secret Weapon: Patch

While grep and the pickaxe are useful tools - they’re made even more useful with the --patch option.

By default, the returned results for a search with git log is… just that: the git log.

However, you can see the full diff if you include the --patch option.

This is another good opportunity to see the differences between --grep, -S and -G.

git log --grep="cachePublic: true" --patch

This returns nothing. There are no git commit messages (subject or body) with this string included in them.

What about the -S option though?

git log -S"cachePublic: true," --patch

This returns the full commit message (in this case, there wasn’t much of one) and the changed file:

commit e01fcf867905f4bcb3eb5ad8c961524547a782eb (origin/285/caching-netlify-builds, 285/caching-netlify-builds)
Author: Stephen <stephencweiss@gmail.com>
Date: Wed Dec 18 13:20:07 2019 -0600

    add netlify cache

diff --git a/gatsby-config.js b/gatsby-config.js
index 93d491e..c29b253 100644
--- a/gatsby-config.js
+++ b/gatsby-config.js
@@ -12,6 +12,12 @@ module.exports = {
   },
   plugins: [
     'gatsby-plugin-styled-components',
+ {
+ resolve: 'gatsby-plugin-netlify-cache',
+ options: {
+ cachePublic: true,
+ }
+ },
     {
       resolve: `gatsby-source-filesystem`,
       options: {

Finally, there’s the -G option:

git log -G"cachePublic: true," --patch

Like the -S option, this returns the full commit message and the changed file(s). However, in this case it also pulled back a second commit where the number of times cachPublic: true, was present didn’t change, but it was reformatted: ³

commit ee201712e254bfd300b77057344994c1c1bd7663
Author: Stephen <stephencweiss@gmail.com>
Date: Thu Dec 19 09:35:37 2019 -0600

    adding global styles with styled-components

diff --git a/gatsby-config.js b/gatsby-config.js
index bacedc4..96f74db 100644
--- a/gatsby-config.js
+++ b/gatsby-config.js
@@ -15,8 +15,8 @@ module.exports = {
     {
       resolve: 'gatsby-plugin-netlify-cache',
       options: {
- cachePublic: true,
- }
+ cachePublic: true,
+ },
     },
     {
       resolve: `gatsby-source-filesystem`,
@@ -100,21 +100,21 @@ module.exports = {
       resolve: `gatsby-plugin-feed`,
       options: {
         feeds: [
- {
- serialize: ({ query: { site, allMarkdownRemark } }) => {
- return allMarkdownRemark.edges.map(edge => {
- return Object.assign({}, edge.node.frontmatter, {
- date: edge.node.frontmatter.date,
- publish: edge.node.frontmatter.publish,
- updated: edge.node.frontmatter.updated,
- draft: edge.node.frontmatter.draft,
- url: site.siteMetadata.siteUrl + edge.node.fields.slug,
- guid: site.siteMetadata.siteUrl + edge.node.fields.slug,
- custom_elements: [{ 'content:encoded': edge.node.html }],
- })
- })
- },
- query: `
+ {
+ serialize: ({ query: { site, allMarkdownRemark } }) => {
+ return allMarkdownRemark.edges.map(edge => {
+ return Object.assign({}, edge.node.frontmatter, {
+ date: edge.node.frontmatter.date,
+ publish: edge.node.frontmatter.publish,
+ updated: edge.node.frontmatter.updated,
+ draft: edge.node.frontmatter.draft,
+ url: site.siteMetadata.siteUrl + edge.node.fields.slug,
+ guid: site.siteMetadata.siteUrl + edge.node.fields.slug,
+ custom_elements: [{ 'content:encoded': edge.node.html }],
+ })
+ })
+ },
+ query: `
                   {
                     allMarkdownRemark(
                       limit: 1000,
@@ -137,9 +137,9 @@ module.exports = {
                     }
                   }
                 `,
- output: '/rss.xml',
- title: 'Code-Comments RSS Feed',
- },
+ output: '/rss.xml',
+ title: 'Code-Comments RSS Feed',
+ },
         ],
       },
     },
@@ -210,14 +210,14 @@ module.exports = {
         ],
       },
     },
- `gatsby-plugin-offline`,
- `gatsby-plugin-react-helmet`,
     {
- resolve: `gatsby-plugin-typography`,
+ resolve: `gatsby-plugin-styled-components`,
       options: {
- pathToConfigModule: `src/utils/typography`,
+ displayName: true,
       },
     },
+ `gatsby-plugin-offline`,
+ `gatsby-plugin-react-helmet`,
     {
       resolve: `@gatsby-contrib/gatsby-plugin-elasticlunr-search`,
       options: {

commit e01fcf867905f4bcb3eb5ad8c961524547a782eb (origin/285/caching-netlify-builds, 285/caching-netlify-builds)
Author: Stephen <stephencweiss@gmail.com>
Date: Wed Dec 18 13:20:07 2019 -0600

    add netlify cache

diff --git a/gatsby-config.js b/gatsby-config.js
index 93d491e..c29b253 100644
--- a/gatsby-config.js
+++ b/gatsby-config.js
@@ -12,6 +12,12 @@ module.exports = {
   },
   plugins: [
     'gatsby-plugin-styled-components',
+ {
+ resolve: 'gatsby-plugin-netlify-cache',
+ options: {
+ cachePublic: true,
+ }
+ },
     {
       resolve: `gatsby-source-filesystem`,
       options: {

Conclusion

Writing good commit messages is hard(er than not). It requires time and effort. However, the total cost is not that high. Practices like Conventional Commits and templates can make the writing easier. Best of all, useful commit messages help you and your team nearly immediately. Even more so if you learn about the tools that make digging through them easy!

Footnotes

¹ See the manual entry for git log for a full list of what’s available. An interesting note is the difference between the -G and the -S options.
² It’s so easy to forget, but grep is an acronym for ”_g_lobally search a _r_egular _e_xpression and _p_rint”. Read more about the history of grep on Wikipedia.
³ Worth noting that if the grep had returned anything with the --patch, it would be the entire change set (which makes sense as it’s looking at the commit message, not at any particular modified file).

Building an XKCD Daily Digest

Stephen Charles Weiss — Sat, 25 Jan 2020 00:00:00 +0000

I love Randall Munroe’s XKCD, but going to his site every day is potentially problematic because it’s so easy to get sucked down the rabbit hole of clicking “random” over and over and over.

At the same time, he’s been writing comics for years and I haven’t kept up. I wondered if there wasn’t some way that I could pull five, just five images into my site every day for a digest of sorts.

Over time I would catch up, but I wouldn’t have to worry about fighting the gravitational pull that is XKCD.com.

So, I set about building this (which is live on this site at /xkcd)

Some of the decisions I made up front on how the page would look and interact:

Comics would have the Title and the Image, nothing else displayed (similar to XCKD’s native experience, however with fewer buttons)
The carousel would navigate through five, “randomly” chosen comics
To avoid the temptation of hitting the refresh button to get a different five, the randomness would be limited to the day of viewing.

The building process turned out to be quite informative and full of stumbling blocks.

Below, I highlight a few of them and the final result.

Getting Images

Step one to building this gallery was to actually be able to get the images. Here’s where I hit my first stumbling block.

The XKCD site has a page dedicated to programmatic access of comics.

Seems straightforward enough. I tested the end point in Postman.

Voila.

Unfortunately, this site is built on Gatsby, a statically site generator. One consequence of this is that I do not have access to a server in the traditional sense.

This is problematic only in that when I tried to make the same API call from the client, I received a CORS error.

// /pages/xkcd.jsx
function XKCDGallery() {
  fetch(`https://xkcd.com/info.0.json`)
    .then(res => res.json())
    .then(res => console.log(res))
    .catch(error => console.error(`oh no -> `, error))

  return (/* ... */)
}

Returned the console error:

> Access to fetch at 'https://xkcd.com/info.0.json' from origin 'http://localhost:8000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
> index.js:2177 oh no -> TypeError: Failed to fetch

I tried multiple different configurations on my request call. I tried JSONP. Nothing seemed to work.

Based on everything I read, in order for this to work, I’d need access to XKCD’s servers to white list my IP or allow cross origin requests.

Since that wasn’t going to happen, I needed an alternative solution. This is where Nicolas Marcora jumped in to save the day and pointed me to cors-anywhere.

A solution is to simply use the demo¹:

// /pages/xkcd.jsx
function XKCDGallery() {
  const proxyUrl = 'https://cors-anywhere.herokuapp.com'

  const proxiedRequest = (url, options = {}) =>
    fetch(`${proxyUrl}/${url}`, {
      ...options,
      headers: {
        ...options.headers,
        'X-Requested-With': 'wololo',
      },
    })
    .then(resp => resp.json())
    .then(res => console.log({comic: res}))
    .catch(error => console.error(`oh no --> `, error))

  const proxiedGet = url => proxiedRequest(url)

  proxiedGet('https://xkcd.com/info.0.json')

  return (/* ... */)
}

Whew! I now have results. I can proceed to the next step!

Predictable Results

One of the quirks I introduced in my design was that I wanted to be able to get the same images every time I hit the page for an entire day.

That meant I couldn’t just use Math.rand() to pull down five images. So, how could I do this?

Hashing!

When I first learned about hashing functions, my mind was blown. The quick summary is that they reliably transform data to always get to the same result. The variables are the input and the size of the hash. If you know that (and the hashing function), you can derive the answer.

Notably, this is not encryption. It can be reversed. There’s no concept of a public/private key.

A simple hashing function might look like this:

function hash(str) {
  let hash = 0
  if (str.length === 0) return
  for (let i = 0; i < str.length; i += 1) {
    let char = str.charCodeAt(i)
    hash = (hash << 5) + hash + char
    hash = hash & hash // Convert to 32bit integer
    hash = Math.abs(hash)
  }
  return hash
}

This is one I have kept around from a course I took a while back, but it turns out to be a fairly common one. In researching hashing functions I found this post on http://mediocredeveloper.com which walks through each step of a nearly identical function.

Armed with a hashing function, I now needed to bound it. If I didn’t, when I requested a comic, it might not exist.

Randall is prolific, but he’s not written an infinite number of comics.

In fact, we actually already discovered the total number of available comics. The proxied request above was for the most recent comic, but in the body of the response, we see the num property. This turns out to correspond to the sequential indicator Randall has chosen to identify his comics.

With that information, we can write another function to make sure that we stay between 1 and the maximum.

function boundedHash(str, max = 100) {
  let hashedVal = hash(str)
  return Math.max(hashedVal % max, 1)
}

A few quick notes about this implementation:

I’m okay with conflicts. This is a fun project and if I get a collision (i.e. two different hashes that end up occupying the same value), then I’ll see the comic twice.
The Math.max is an error guard as trying to find the comic with num = 0 will throw an error. I could swallow this later, however, again, for the purposes here, this is acceptable.

Encapsulating The Logic In A Hook

At this point, I have all the pieces I need to actually retrieve a number of images. Can I do it in a way that makes consuming them in a component relatively painless?

Normally best practice is to extract reusuable hooks and it’s very unlikely that I will reuse this hook elsewhere. However, I decided to extract it anyway. Why? Because it keeps my view component logic very simple.

// /hooks/useXkcd.jsx
import {useAsync} from 'react-use'
import { boundedHash } from '../utils/hashFn'
import { proxiedGet } from '../utils/proxiedrequest'
import { TODAY } from '../constants'

/**
 * Details re: XKCD's JSON: https://xkcd.com/json.html
 * */
export function useXkcd({ comicQty }) {

  async function latestComicNumber() {
    return await proxiedGet('http://xkcd.com/info.0.json')
      .then(res => res.json())
      .then(res => res.num)
  }

  async function fetchRequestedImages(comicQty) {
    const maxComic = await latestComicNumber()
    const images = []
    images.push(fetchComic()) // get the current comic
    for (let i = 0; i < comicQty; i += 1) {
      const img = fetchComic(boundedHash(i + TODAY, maxComic))
      images.push(img)
    }
    return images
  }

  async function fetchComic(comicId) {
    return proxiedGet(
      `http://xkcd.com/${comicId ? `${comicId}/` : ``}info.0.json`
    )
      .then(res => res.json())
      .then(res => {
        res.link = `http://xkcd.com/${comicId}` // Want to be sure to provide a way back to xkcd
        return res
      })
      .catch(error => console.error(`fetchComics --> `, error))
  }

  const {loading, error, value} = useAsync(() => {
    return fetchRequestedImages(comicQty).then(res => Promise.all(res))
  }, [])

  return {isLoading: loading, isError: error, xkcdComics: value}
}

What this hook does in a nut shell is:

Exposes a very simple surface area to the consumer: how many comics do you want back?
It handles all of the logic for fetching a collection that will match that request.
It wraps it all in a useAsync hook from react-use to make sure to communicate whether the data is ready to render or not.

How simple does that make my view component? Well, let’s look:

// /pages/xkcd.jsx
export function XKCDGallery(props) {
  const { isLoading, isError, xkcdComics } = useXkcd({ comicQty: 5 })

  if (isError) {
    return <div>eek! an error!</div>
  }

  return (
    <Layout location={props.location} title={title}>
      <Title> XKCD Daily Digest </Title>
      {isLoading ? (
        <CenteredLoader />
      ) : (
        <ImageCarousel images={xkcdComics} />
      )}
      {/* ... */}
    </Layout>
  )
}

Pretty nice! This takes us to the final stage of the exercise - displaying the data.

Displaying The Comics

This posed yet another problem for me. My first attempt was to use the pure-react-carousel library, however, the results I was getting were hardly ideal (all images were rendered on a single slide).

Sung M. Kim aka Dance2Die turned me onto nuka-carousel which turned out much better for me. It even has keyboard controls and by using a ref (thanks in no small part to Lee Warrick’s article React’s useEffect and useRefs Explained for Mortals I auto-focus on the carousel, so keyboard controls are available as soon as the comics load.

There is a lingering bug: the carousel doesn’t reserve enough space in the DOM initially for the first image. Once you move to the next image, this is resolved. This bug bothers me, but it’s also an opportunity to learn in the future. I’ll be back to fix it soon I hope! (If you know how to fix it, I'd be grateful to hear in the comments!)

Wrap Up


Current state of the daily digest!

Whew! That’s all I’ve got for today. It’s been a weird, wild ride but wow is it cool that I can wake up with an idea, spend a few hours thinking and tinkering, and with a little help from my friends build something that didn’t exist before.

Footnotes

¹ I’m using the demo proxy for now, however, I plan to migrate to my own version of the cors-anywhere soon using that as a serverless function - that will be a learning experience in its own right! For now, I’m grateful for Rob W, the developer behind cors-anywhere and his demo environment.

Node Version Management... Automatically

Stephen Charles Weiss — Fri, 24 Jan 2020 00:00:00 +0000

NVM, the Node Version Manager, is an invaluable tool when working on multiple node projects. Instead of manually managing the globally installed version of node, NVM can handle it for you.

Unfortunately, even with NVM, it’s easy to use the wrong node version when jumping between projects.

A colleague shared his solution to the problem with me recently:

//".bash\_profile"
function cd() {
    builtin cd "$@"
    nvm i 2> /dev/null
}

This function, when added to the .bash_profile (or .zshrc, etc.), will still perform the change directory as expected (that’s the first line: builtin cd "$@"), but it will also look for the presence of an .nvmrc file in the new directory.

Quick Aside: What Is The .NVMRC?

It’s worth taking a moment to discuss the .nvmrc file, it’s function, and what it may look like:

You can create a .nvmrc file containing a node version number (or any other string that nvm understands; see nvm —help for details) in the project root directory (or any parent directory). Afterwards, nvm use, nvm install, nvm exec, nvm run, and nvm which will use the version specified in the .nvmrc file if no version is supplied on the command line.

So, for example, you could have an .nvmrc for a specific version:

//.nvmrc
12

Or you could default to the latest LTS version:

//.nvmrc
lts/*

If there’s an .nvmrc included in the new directory, the function will execute /dev/null.


An example of this in use. The directory, `personal`, does not have an `.nvmrc` file, though `onething` does. Upon detecting it, `nvm` automatically switched to using the correct `node` version.

As noted in the description of the .nvmrc, it’s presence ensures that nvm i will use the version specified in the file.

The /dev/null is a specific form of redirection within Unix.

In this case it’s saying take the stderr (2), redirect (>) to /dev/null. The /dev/null is a “blackhole” on unix machines, effectively suppressing any errors generated by running this command.

Final Note

NVM manages node versions within each terminal session. This is critical because it ensures that you can run two node projects simultaneously even if the node versions are different.

Locate Files Quickly On Your Computer

Stephen Charles Weiss — Thu, 09 Jan 2020 00:00:00 +0000

I was looking to find all instances of Postgres the other day in order to check the version. A web search led me to this article.

Interestingly, while the search came up empty (I believe the file’s been deleted), I did learn about two new commands: locate and find.

Of the two, I will likely use find more often because locate works by building a database that it searches whereas find simply walks a file hierarchy looking for the search parameters.

The first time you run locate, you may be greeted by a message similar to this one:

WARNING: The locate database (/var/db/locate.database) does not exist.To create the database, run the following command:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist

Please be aware that the database can take some time to generate; oncethe database has been created, this message will no longer appear.

Find is simpler (though it may require super user permissions).

Find Example Usage

Here’s one example of searching your file system using the find utility:

In this case, I was interested in all locations for Postgres on my machine and I used the glob pattern */ to search.

$ sudo find /usr -wholename '*/bin/postgres*'
/usr/local/bin/postgres
/usr/local/Cellar/postgresql/12.1/bin/postgres

Ét voila! A new way to quickly locate files on a computer!

Resources

Manual pages posted below for posterity.

NAME
find -- walk a file hierarchy

SYNOPSIS
find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]

DESCRIPTION
The find utility recursively descends the directory tree for each path listed, evaluating an expression (composed of the ``primaries'' and ``operands'' listed below) in terms of each file in the tree.

The options are as follows:

-E Interpret regular expressions followed by -regex and -iregex primaries as extended (modern) regular expressions rather than basic regular expressions (BRE's). The re_format(7) manual page fully describes
both formats.

-H Cause the file information and file type (see stat(2)) returned for each symbolic link specified on the command line to be those of the file referenced by the link, not the link itself. If the referenced
file does not exist, the file information and type will be for the link itself. File information of all symbolic links not on the command line is that of the link itself.

-L Cause the file information and file type (see stat(2)) returned for each symbolic link to be those of the file referenced by the link, not the link itself. If the referenced file does not exist, the file
information and type will be for the link itself.

This option is equivalent to the deprecated -follow primary.

-P Cause the file information and file type (see stat(2)) returned for each symbolic link to be those of the link itself. This is the default.

-X Permit find to be safely used in conjunction with xargs(1). If a file name contains any of the delimiting characters used by xargs(1), a diagnostic message is displayed on standard error, and the file is
skipped. The delimiting characters include single (`` ' '') and double (`` " '') quotes, backslash (``\''), space, tab and newline characters.

However, you may wish to consider the -print0 primary in conjunction with ``xargs -0'' as an effective alternative.

-d Cause find to perform a depth-first traversal, i.e., directories are visited in post-order and all entries in a directory will be acted on before the directory itself. By default, find visits directories
in pre-order, i.e., before their contents. Note, the default is not a breadth-first traversal.

This option is equivalent to the -depth primary of IEEE Std 1003.1-2001 (``POSIX.1''). The -d option can be useful when find is used with cpio(1) to process files that are contained in directories with
unusual permissions. It ensures that you have write permission while you are placing files in a directory, then sets the directory's permissions as the last thing.

-f Specify a file hierarchy for find to traverse. File hierarchies may also be specified as the operands immediately following the options.

-s Cause find to traverse the file hierarchies in lexicographical order, i.e., alphabetical order within each directory. Note: `find -s' and `find | sort' may give different results.

-x Prevent find from descending into directories that have a device number different than that of the file from which the descent began.

This option is equivalent to the deprecated -xdev primary.

PRIMARIES
All primaries which take a numeric argument allow the number to be preceded by a plus sign (``+'') or a minus sign (``-''). A preceding plus sign means ``more than n'', a preceding minus sign means ``less than n''
and neither means ``exactly n''.

-Bmin n
True if the difference between the time of a file's inode creation and the time find was started, rounded up to the next full minute, is n minutes.

-Bnewer file
Same as -newerBm.

-Btime n[smhdw]
If no units are specified, this primary evaluates to true if the difference between the time of a file's inode creation and the time find was started, rounded up to the next full 24-hour period, is n
24-hour periods.

If units are specified, this primary evaluates to true if the difference between the time of a file's inode creation and the time find was started is exactly n units. Please refer to the -atime primary
description for information on supported time units.

-acl May be used in conjunction with other primaries to locate files with extended ACLs. See acl(3) for more information.

-amin n
True if the difference between the file last access time and the time find was started, rounded up to the next full minute, is n minutes.

-anewer file
Same as -neweram.

NAME
locate -- find filenames quickly

SYNOPSIS
locate [-0Scims] [-l limit] [-d database] pattern ...

DESCRIPTION
The locate program searches a database for all pathnames which match the specified pattern. The database is recomputed periodically (usually weekly or daily), and contains the pathnames of all files which are pub-
licly accessible.

Shell globbing and quoting characters (``*'', ``?'', ``\'', ``['' and ``]'') may be used in pattern, although they will have to be escaped from the shell. Preceding any character with a backslash (``\'') elimi-
nates any special meaning which it may have. The matching differs in that no characters must be matched explicitly, including slashes (``/'').

As a special case, a pattern containing no globbing characters (``foo'') is matched as though it were ``*foo*''.

Historically, locate only stored characters between 32 and 127. The current implementation store any character except newline (`\n') and NUL (`\0'). The 8-bit character support does not waste extra space for
plain ASCII file names. Characters less than 32 or greater than 127 are stored in 2 bytes.

The following options are available:

-0 Print pathnames separated by an ASCII NUL character (character code 0) instead of default NL (newline, character code 10).

-S Print some statistics about the database and exit.

-c Suppress normal output; instead print a count of matching file names.

-d database
Search in database instead of the default file name database. Multiple -d options are allowed. Each additional -d option adds the specified database to the list of databases to be searched.

The option database may be a colon-separated list of databases. A single colon is a reference to the default database.

$ locate -d $HOME/lib/mydb: foo

will first search string ``foo'' in $HOME/lib/mydb and then in /var/db/locate.database.

$ locate -d $HOME/lib/mydb::/cdrom/locate.database foo

will first search string ``foo'' in $HOME/lib/mydb and then in /var/db/locate.database and then in /cdrom/locate.database.

$ locate -d db1 -d db2 -d db3 pattern

is the same as

$ locate -d db1:db2:db3 pattern

or

$ locate -d db1:db2 -d db3 pattern

If - is given as the database name, standard input will be read instead. For example, you can compress your database and use:

$ zcat database.gz | locate -d - pattern

This might be useful on machines with a fast CPU and little RAM and slow I/O. Note: you can only use one pattern for stdin.

-i Ignore case distinctions in both the pattern and the database.

-l number Limit output to number of file names and exit.

-m Use mmap(2) instead of the stdio(3) library. This is the default behavior and is faster in most cases.

-s Use the stdio(3) library instead of mmap(2).

ENVIRONMENT
LOCATE_PATH path to the locate database if set and not empty, ignored if the -d option was specified.

FILES
/var/db/locate.database locate database
/usr/libexec/locate.updatedb Script to update the locate database
/System/Library/LaunchDaemons/com.apple.locate.plist Job that starts the database rebuild

SEE ALSO
find(1), whereis(1), which(1), fnmatch(3), locate.updatedb(8)

Woods, James A., "Finding Files Fast", ;login, 8:1, pp. 8-10, 1983.

Marty Cagan on Rewrites, Headroom, And 20%-Time

Stephen Charles Weiss — Tue, 07 Jan 2020 00:00:00 +0000

In 2007, Marty Cagan wrote a blog called “Engineering Wants to Rewrite!”.

It’s a good example of how product and engineering can work together to keep a business moving forward, even while “rebuilding the engine mid-flight.”

The key takeaways:

100% focus on new features can doom a company.
<20% focus on “headroom” is overly optimistic.
Engineers, conservative naturally, are wildly optimistic about rewrites. Plan accordingly.

Copying the post in full below for posterity:

Engineering Wants to Rewrite!

by Marty Cagan
Mar 29, 2007

Few words are more dreaded by product managers than being told by engineering: “No more new features! We need to stop and rewrite! Our code base is a mess, it can’t keep up with the number of users, it’s a house of cards, we can’t maintain it, the site is a dog!”

This situation has happened to too many companies, and continues to happen. It happened to eBay in 1999, and the company came far closer to collapsing than most people ever realized. It happened to Friendster a few years ago and opened the door for MySpace. It happened to Netscape during the browser wars with Microsoft, and everyone knows who won. The truth is that most companies never recover from this. It is a really bad situation to find yourself in.

When a company does get into this situation, everyone typically blames engineering. But in my experience, the harsh truth is that it’s usually the fault of product management. The reason is that for the past years the product managers have been pounding the engineering organization to deliver as many features as the engineering team possibly can produce. The result is that at some point, if you neglect the infrastructure, all software will reach the point where it can no longer support the functionality it needs to.

During this rewrite, you’re forced to stop forward progress as far as what the customers see. You might think that the rewrite will only take a few months (more about that below) but invariably it takes far longer, and you are forced to sit by and watch your customers leave you for your competitors.

If you haven’t yet reached this situation, here’s what you need to do to make sure you never do. You need to allocate a percentage of your engineering capacity to what at eBay we called “headroom”. The idea is to avoid slamming your head into ceilings. You do this by creating room – headroom – room for growth in the user base, growth in transactions, growth in functionality.

The deal with engineering goes like this. Product management takes 20% of the capacity right off the top and gives this to engineering to spend as they see fit – they might use it to rewrite, rearchitect or refactor problematic parts of the code base, or to swap out data base systems, improve system performance – whatever they believe is necessary to avoid ever having to come to the team and say “we need to stop and rewrite.” If you’re in really bad shape today, you might need to make this 30% or even more of the resources. I get nervous when I find teams that think they can get away with much less than 20%.

If you are currently in this situation, the truth is that your company may not survive this. But if you are to have a chance of pulling through, you’ll need to first do a realistic schedule and timeline for making the necessary changes that engineering identifies.

Most of the time, an experienced engineering team will come up with estimates that are on the slightly conservative side. The exception to this rule is this case of rewrites. Here the estimates are often wildly optimistic. You must make informed decisions in this situation, so you have to go through every line item on the schedule to make sure that the dates are realistic.

Second, if there’s any way humanly possible to break up the rewrite into chunks to be done incrementally, you should absolutely do so. Even though the rewrite might now stretch over two years instead of 9 months, if you can find a way to continue to make forward progress on user visible functionality, even if it’s only with 25% to 50% of the capacity, this is incredibly important.

Third, since you’ll only have very limited ability to deliver user visible functionality, you will need to pick the right features, and make sure you do define them right.

After eBay’s near-death experience, they made sure they wouldn’t put the company at risk again. They immediately began another rewrite, this time well in advance of issues. In fact, due to their very rapid growth, they ended up rewriting a third time, this time translating the entire site into a different programming language and architecture, and they did this massive multi-million line rewrite over several years, and most importantly, without impacting the user base and at the same time managing to deliver record amounts of new functionality. It’s the most impressive example of “rebuilding the engine mid-flight” that I know of.

But definitely the best strategy for dealing with this situation is to not get to this point. You need to pay your taxes and remember to dedicate at least 20% to headroom. If you haven’t had this discussion with your engineering counterpart, you should do so today.

GraphQL Fragments: Reuse and Variables

Stephen Charles Weiss — Mon, 06 Jan 2020 00:00:00 +0000

After thinking more about Lee Byron’s talk and some of the uses for fragments¹, I still had some questions. Specifically - how do you reuse them effectively?

It’s one thing to break down complicated queries into smaller fragments (as Lee did in his talk), but it’s another to actually reuse those query fragments across the app.

While researching, I also found the use of variables with fragments simple and elegant.

I’ll look at both below.

Reusing Fragments

At the end of the day, GraphQL fragments are template literals assigned to a variable. In Javascript, that means they can be passed around just like any other variable. Depending on the environment you’re in, the convention may be slightly different.

For example, the Apollo convention is to place fragments within an object named [something].fragment:

import gql from 'graphql-tag';

CommentsPage.fragments = {
  comment: gql`
    fragment CommentsPageComment on Comment {
      id
      postedBy {
        login
        html_url
      }
      createdAt
      content
    }
  `,
};

Then, to use this, would look like:

const SUBMIT_COMMENT_MUTATION = gql`
  mutation SubmitComment($repoFullName: String!, $commentContent: String!) {
    submitComment(repoFullName: $repoFullName, commentContent: $commentContent) {
      ...CommentsPageComment
    }
  }
  ${CommentsPage.fragments.comment}
`;

export const COMMENT_QUERY = gql`
  query Comment($repoName: String!) {
    # ...
    entry(repoFullName: $repoName) {
      # ...
      comments {
        ...CommentsPageComment
      }
      # ...
    }
  }
  ${CommentsPage.fragments.comment}
`;

The interesting thing to note about Apollo’s implementation is the use of the ...Name for the fragment and the inclusion of the gql template literal (${CommentsPage.fragments.comment}).

Gatsby’s approach is somewhat different and is a result of the architecture of a Gatsby site which allows defining GraphQL fragments in components, but only returns data to queries within Pages.

In the example from the Gatsby docs then, we:

Create a component with a GraphQL fragment (the fragment is exported and available as SiteInformation)
Import the component into a Page and use the fragment

The example (taken from the Gatsby site linked above):

"src/components/IndexPost.jsx"

import React from "react"
import { graphql } from "gatsby"

export default ( props ) => {
  return (...)
}

export const query = graphql`
  fragment SiteInformation on Site {
    siteMetadata {
      title
      siteDescription
    }
  }
`

And then used like so:

"src/pages/main.jsx"

import React from "react"
import { graphql } from "gatsby"
import IndexPost from "../components/IndexPost"

export default ({ data }) => {
  return (
    <div>
      <h1>{data.site.siteMetadata.title}</h1>
      <p>{data.site.siteMetadata.siteDescription}</p>

      {/*
        Or you can pass all the data from the fragment
        back to the component that defined it
      */}
      <IndexPost siteInformation={data.site.siteMetadata} />
    </div>
  )
}

export const query = graphql`
  query {
    site {
      ...SiteInformation
    }
  }
`

As of now, my questions that remain to be tested include:

Why does Apollo require the ${FragmentName}?

Can I export and use a fragment that’s defined as:

export const siteInfoFragment = graphql`
  fragment SiteNformation on Site {
    siteMetadata {
      title
      siteDescription
    }
  }
`

Right now, I’m still learning the theory of fragments and… I like them in theory. However, I need more experience with them and to date, my apps which use GraphQL simply haven’t called for the complexity. So, this is preliminary research and I’m excited to revisit when there’s an actual need.

Using Variables With Fragments

While researching how to pass fragments around, I also found the answer to a question I didn’t realize I had: how to use variables with fragments.

I found the answer in the GraphQL docs, which provide a nice example of how to use variables with fragments. Before looking at their implementation, let’s examine a more generic query to see how it compares:

query HeroComparison($first: Int = 3) {
  firstComparison: hero(episode: EMPIRE) {
    name
    friendsConnection(first: $first) {
      totalCount
      edges {
        node {
          name
        }
      }
    }
  }
  secondComparison: hero(episode: JEDI) {
    name
    friendsConnection(first: $first) {
      totalCount
      edges {
        node {
          name
        }
      }
    }
  }
}

In this example, the client would provide a value for the variable first (which defaults to 3), and that is then passed along to both the firstComparison and secondComparison.

Using Fragments, we can simplify this and reduce the duplicative code while still using variables as we would expect. Here’s what the GraphQL docs suggest:

query HeroComparison($first: Int = 3) {
  leftComparison: hero(episode: EMPIRE) {
    ...comparisonFields
  }
  rightComparison: hero(episode: JEDI) {
    ...comparisonFields
  }
}

fragment comparisonFields on Character {
  name
  friendsConnection(first: $first) {
    totalCount
    edges {
      node {
        name
      }
    }
  }
}

The key point is that, just as in the original query, the variable passed into the query is propagated all the way down and is accessible by the fragments.

Conclusion

When it comes to fragments, it turns out that the way to pass them around and reuse them or to use variables is exactly how I’d expect to do it. That’s pretty fantastic.

Footnotes

¹ I wrote about it here and here’s Lee’s original talk. [return]

Fixing My Embargo: Setting up a Cron Job

Stephen Charles Weiss — Fri, 27 Dec 2019 00:00:00 +0000

I wrote previously about setting up a simple embargo on my Gatsby site in the past. Even when I first implemented it, however, I knew that there’d be a better solution in the future.

As a reminder: Gatsby sites are static. In my case, the builds included all of the posts for my blog. Consequently, when I wrote a post and added it to my site - when the site built it was visible.

The purpose of the embargo was to create space between when I wrote a new post and when it would appear. The original solution solved this by simply hiding the posts on the front end. I wanted a better solution however. For that, I reached for an old standby: the cron job.

What Is A Cron Job

Cron is a utility that is used for scheduling commands or programs to execute at a specific time. Named for Chronos, the diety of time, these scheduled commands are referred to as “Cron Jobs”.

Common uses of cron jobs include backups, monitoring, and maintenance.

The syntax for cron jobs is a little peculiar, but there’s a logic to it that just takes some familiarity:

 ┏━━━━━━━━━━━━ minute (0 - 59)
 ┃ ┏━━━━━━━━━━ hour (0 - 23)
 ┃ ┃ ┏━━━━━━━━ day of month (1 - 31)
 ┃ ┃ ┃ ┏━━━━━━ month (1-12)
 ┃ ┃ ┃ ┃ ┏━━━━ day of week (0 - 6) or use names;
 ┃ ┃ ┃ ┃ ┃ 0 and 7 are Sunday, 1 is Monday,
 ┃ ┃ ┃ ┃ ┃ 2 is Tuesday, etc.
 ┃ ┃ ┃ ┃ ┃
 * * * * * <command to execute>

Some great examples of different schedules can be found on ostechnix.com.

Netlify Build Hooks

Now that we know a bit about cron jobs, we want to talk about which command to execute.

In this particular case, we’ll be using Netlify Build Hooks as I’m using Netlify to build and deploy my site.

Triggering a build requires only a POST request sent to the URL specified by a token.

To add a build hook, we log into Netlify, select the site and choose Settings.

Then, on the left, select “Build & Deploy” and scroll down to “Build Hooks”. Select “Add build hook”. From here, we can name the hook.

Once saved, Netlify will produce a token we can use to trigger a build. The token should be considered sensitive and should be kept private (similar to an API key).

Now that we have a build hook, we need to schedule the invocation of it. For that, we’ll use a cron job via GitHub Actions.

GitHub Actions

Recently out of a beta, GitHub Actions are a new feature from Github to automate software workflows.

To get the action set up, we will need to:

Add a workflow .yml
Add the secret

Adding GitHub Workflows

Github actions are managed by the presence of a .yml file in a directory in your project .yml.

Here’s the my nightly-build.yml file:

name: Daily Build
on:
  schedule:
    - cron: ‘0 8 * * *’ # 0 minute, 8th hour, every day of the month, every month, every day of the week (UTC)*
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Netlify Build Hook
        run: curl -s -X POST -d {} “https://api.netlify.com/build_hooks/${TOKEN}”
        env:
          TOKEN: ${{ secrets.NETLIFY_DAILY_CRON_HOOK }}

Remember that a cron job has two parts: a schedule and a command. In this yaml file, I’ve defined a schedule as the 0th minute of every 8th hour, every day of the month, every month, and every day of the week.¹ The job is to run the curl and send the POST to Netlify.

Two things to note:

Regarding yaml files generally: they’re space sensitive. My first attempt resulted in an error, “YAML mapping values are not allowed in this context”. Investigating it led me to this StackOverflow conversation and ultimately http://www.yamllint.com/ - a great tool for validating yaml files.
The curl is the same as what Netlify suggested - though I spefcified a TOKEN which takes a value secrets.NETLIFY_DAILY_CRON_HOOK. That secret hasn’t yet been defined. In fact, that’s the final step.

Adding A Secret

Though we’ve referenced a variable in our .yml file, we now need to define it.

To do so, in the repository, select settings and then “Secrets” on the left.

In this case, the secret is taken directly from Netlify as the string that follows /build_hooks/.

Conclusion

And with that, I have a new build process. Suddenly, my website builds every day at 8am which frees me to focus on the parts of writing and running this site that I enjoy and begin to automate the other stuff.

Footnotes

¹ Though it’s scheduled at 8am, it’s important to remember that the time is local to the computer running the command. In this case, that means wherever the Netlify servers are.

Resources

Additional resources that I found helpful include:

Scheduling Netlify Deploys With Github Actions | Voorhoede - This was a great tutorial that proved a great starting point to understanding these actions.
About GitHub Actions - GitHub Help
Automating your workflow with GitHub Actions - GitHub Help
Events that trigger workflows - GitHub Help
Automate your Netlify sites with Zapier | Netlify