DEV Community: Pete Corey

MongoDB Object Array Lookup Aggregation

Pete Corey — Wed, 29 Jan 2020 00:00:00 +0000

As part of an ongoing quest to speed up an application I’m working on, I found myself tasked with writing a fairly complicated MongoDB aggregation pipeline. I found no existing documentation on how to accomplish the task at hand, so I figured I should pay it forward and document my solution for future generations.

Widgets and Icons

Imagine we have two MongoDB collections. Our first collection holds information about widgets in our system:


db.getCollection('widgets').insert({
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text'
        }
    ]
});

Every widget has a name and a list of one or more info objects. Each info object has a text field and an associated icon referenced by an iconId.

Our icons collection holds some basic information about each icon:


db.getCollection('icons').insert({
    _id: 2,
    name: 'Icon',
    uri: 'https://...'
});

The goal is to write an aggregation that returns our widgets with the associated icon documents attached to each corresponding info object:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

Working Through the Pipeline

The aggregation that accomplishes this goal operates in six stages. Let’s work through each stage one by one. We’ll start by $unwinding our info array:


db.getCollection('widgets').aggregate([
    { $unwind: '$info' }
]);

This creates a new document for every widget/info pair:


{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
    }
}

Next, we’ll $lookup the icon associated with the given iconId:


db.getCollection('widgets').aggregate([
    ...
    {
        $lookup: {
            from: 'icons',
            localField: 'info.iconId',
            foreignField: '_id',
            as: 'info.icon'
        }
    }
]);

Our resulting document will now have a list of icons in the info.icon field:


{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
        icon: [
            {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        ]
    }
}

This is a step in the right direction, but we know that the info to icons relationship will always be a one to one relationship. We’ll always receive exactly one icon as a result of our $lookup.

Armed with this knowledge, we know we can $unwind on info.icon and safely turn our info.icon array into an object:


db.getCollection('widgets').aggregate([
    ...
    { $unwind: '$info.icon' }
]);



{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
        icon: {
            _id: 2,
            name: 'Icon',
            uri: 'https://...'
        }
    }
}

But now we need to roll our info back up into an array. We can accomplish this by $grouping our widgets together based on their _id. However, we need to be careful to preserve the original document to avoid clobbering the entire widget:


db.getCollection('widgets').aggregate([
    ...
    {
        $group: {
            _id: '$_id',
            root: { $mergeObjects: '$$ROOT' },
            info: { $push: '$info' }
        }
    }
]);

Our resulting document contains our info array and the original, pre-$group widget document in the root field:


{
    root: {
        _id: 1,
        name: 'Name',
        info: {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    },
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

The next step in our pipeline is to replace our root document with the root object merged with the actual root document. This will override the info object in root with our newly grouped together info array:


db.getCollection('widgets').aggregate([
    ...
    {
        $replaceRoot: {
            newRoot: {
                $mergeObjects: ['$root', '$$ROOT']
            }
        }
    }
]);

We’re getting close to our goal:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ],
    root: {
        _id: 1,
        name: 'Name',
        info: {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    }
}

An unfortunate side effect of this merger is that our resulting document still has a root object filled with superfluous data. As a final piece of housecleaning, let’s remove that field:


db.getCollection('widgets').aggregate([
    ...
    {
        $project: {
            root: 0
        }
    }
]);

And with that we’re left with our original goal:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

Success!

All Together

For posterity, here’s the entire aggregation pipeline in its entirety:


db.getCollection('widgets').aggregate([
    { $unwind: '$info' },
    {
        $lookup: {
            from: 'icons',
            localField: 'info.iconId',
            foreignField: '_id',
            as: 'info.icon'
        }
    },
    { $unwind: '$info.icon' },
    {
        $group: {
            _id: '$_id',
            root: { $mergeObjects: '$$ROOT' },
            info: { $push: '$info' }
        }
    },
    {
        $replaceRoot: {
            newRoot: {
                $mergeObjects: ['$root', '$$ROOT']
            }
        }
    },
    {
        $project: {
            root: 0
        }
    }
]);

I’ll be the first to say that I’m not a MongoDB expert, and I’m even less knowledgeable about building aggregation pipelines. There may be other, better ways of accomplishing this same task. If you know of a better, more efficient pipeline that gives the same results, please let me know!

Timing Streams in Node.js

Pete Corey — Sat, 11 Jan 2020 00:00:00 +0000

On a current client project, I was tasked with optimizing a very large, very slow, very CPU-bound stream-based pipeline. Before I even started to think about optimizing this pipeline, I needed an objective way to measure the execution time of each step of the pipeline.

Imagine the pipeline in question looks something like this:


pipeline(
    httpStream,
    decodeStream,
    parseStream,
    batchStream,
    processStream
);

We’re reading in a stream of JSON-encoded events (httpStream), making sure they’re appropriately decoded (decodeStream), JSON parsing each incoming event (parseStream), batching events together (batchStream), and finally processing each batch of events (processStream).

Ideally I’d like to measure any or all of these individual steps.

However, many of these stream implementations are out of our hands. We can’t easily reach in and add timing code. Thankfully, we can easily write a function that decorates a provided stream with a simple runtime calculation.

Let’s call our decorator function time:


const time = (stream, name) => {
    return stream;
};

Our time function accepts and returns the stream we’ll be decorating, along with a name that describes the provided stream. It should be noted that it’s assumed that stream implements the Readable interface.

What we’re trying to accomplish here is relatively simple. We want to measure the amount of time that elapses between data emission events on our stream. We can use console.time/console.timeEnd and an event listener to make short work of this task:


const time = (stream, name) => {
    let timing = false;
    stream.on('data', () => {
        if (timing) {
            console.timeEnd(name);
        }
        console.time(name);
        timing = true;
    });
    return stream;
};

Every time we receive a 'data' event on our stream, we log the duration since the last received 'data' event, and start a new timer. We’re using a timing flag to ensure that console.timeEnd isn’t called the first time we receive a 'data' event.

Notice that we’re also using the provided name as the label in our console.time/console.timeEnd calls. This keeps us from getting confused when we start measuring multiple stages of our pipeline.

This solution mostly works. Unfortunately, a data event isn’t fired when the stream starts processing its first chunk of data. This means that we’re missing a measurement for this first chunk of execution time. Thankfully, we can capture that missing metric by also listening for a 'resume' event, which is called when the stream starts processing its first chunk of data:


const time = (stream, name) => {
    stream.on('resume', () => {
        console.time(name);
    });
    stream.on('data', () => {
        console.timeEnd(name);
        console.time(name);
    });
    return stream;
};

Notice that we’re no longer concerned about wrapping our console.timeEnd call in a guard in our 'data' event listener. We know that the 'resume' event handler will always call console.time before we reach our 'data' event handler, so we have no need for the timing guard anymore.

We can use our time function by decorating any or all of the stages of our pipeline:


await pipeline(
    httpStream,
    decodeStream,
    parseStream,
    time(batchStream, 'batch'),
    time(processStream, 'process')
);

Now that our runtime durations are finding their way to the logs, we can either use them as-is, or take things a step further and aggregate them for more in-depth data analysis:

...
batch: 258.213ms
process: 512.493ms
batch: 239.112ms
process: 475.293ms
...

As a warning to the reader, I’ll be the first to admit that I’m no stream expert. That said, this utility function proved invaluable to me, so I thought I’d record what I learned and pass it along for posterity.

Stream on.

Random Seeds, Lodash, and ES6 Imports

Pete Corey — Wed, 01 Jan 2020 00:00:00 +0000

David Bau’s seedrandom Javascript library is an excellent tool for introducing deterministic random values into your Javascript project. After setting a fixed seed, Math.random will produce a stream of random values. Those same random values will be produced again, in order, the next time you run your program. This is very important when creating generative art or procedurally generated game content.

However, there’s a small problem when trying to combine seedrandom with a library like Lodash. Ideally, we’d like Lodash to respect our random seed, so methods like shuffle would always produce a deterministic shuffling. Unfortunately, with a setup like the one described below, this won’t be the case:


import _ from "lodash";
import seedrandom from "seedrandom";

seedrandom("seed", { global: true });

_.shuffle([1, 2, 3]); // Ignores our random seed.

The seedrandom library wholesale replaces Math.random with a new pseudo-random number generator. Because we’re importing lodash before we initialize seedrandom, Lodash defines all of its functions, shuffle included, to use the original reference to Math.random. We need to initialize seedrandom before importing Lodash.

Unfortunately, this won’t work:


import seedrandom from "seedrandom";
seedrandom("seed", { global: true });

import _ from "lodash";

Node.js requires all import statements to be at the top of a module. We can’t initialize seedrandom before importing Lodash.

Thankfully, a simple solution exists. We’ll make a new module called seed.js that simply imports seedrandom and then initializes it with our seed:


import seedrandom from "seedrandom";

seedrandom("seed", { global: true });

Next we can import our local "./seed.js" module before importing Lodash:


import "./seed.js";
import _ from "lodash";

_.shuffle([1, 2, 3]); // Produces deterministic shufflings!

And with that small change seedrandom, Lodash, and ES6-style imports all play nicely together. Our shuffle function will now product deterministic shufflings based on the seed we pass into seedrandom!

Count the Divisible Numbers

Pete Corey — Mon, 25 Nov 2019 00:00:00 +0000

Let’s try our hand at using a property test driven approach to solving a Codewars code kata. The kata we’ll be solving today is “Count the Divisible Numbers”. We’ll be solving this kata using Javascript, and using fast-check alongside Jest as our property-based testing framework.

The kata’s prompt is as follows:

Complete the [divisibleCount] function that takes 3 numbers x, y and k (where x ≤ y), and returns the number of integers within the range [x..y] (both ends included) that are divisible by k.

Writing Our Property Test

We could try to translate this prompt directly into a property test by generating three integers, x, y, and k, and verifying that the result of divisibleCount(x, y, k) matches our expected result, but we’d have to duplicate our implementation of divisibleCount to come up with that “expected result.” Who’s to say our test’s implementation wouldn’t be flawed?

We need a more obviously correct way of generating test cases.

Instead of generating three integers, x, y, and k, we’ll generate our starting point, x, the number we’re testing for divisibility, k, and the number of divisible numbers we expect in our range, n:


test("it works", () => {
  fc.assert(
    fc.property(fc.integer(), fc.integer(), fc.nat(), (x, k, n) => {
      // TODO ...
    })
  );
});

Armed with x, k, and n, we can compute the end of our range, y:


let y = x + n * k;

Next, we’ll pass x, our newly commuted y, and k into divisibleCount and assert that the result matches our expected value of n:


return n === divisibleCount(x, y, k);

Our final property test looks like this:


test("it works", () => {
  fc.assert(
    fc.property(fc.integer(), fc.integer(), fc.nat(), (x, k, n) => {
      let y = x + n * k;
      return n === divisibleCount(x, y, k);
    })
  );
});

Beautiful.

Our First Solution

Coming up with a solution to this problem is fairly straight-forward:


const divisibleCount = (x, y, k) => {
  return _.chain(y - x)
    .range()
    .map(n => x + n)
    .reject(n => n % k)
    .size()
    .value();
};

We generate an array of integers from x to y, reject those that aren’t divisible by k, and return the size of the resulting array.

Unfortunately, this simple solution doesn’t work as expected. Our property test reports a failing counterexample of [0, 0, 1] values for x, k, and n:


$ jest
 FAIL ./index.test.js
  ✕ it works (10ms)

  ● it works

    Property failed after 1 tests
    { seed: 1427202042, path: "0:0:0:1:0:0:0", endOnFailure: true }
    Counterexample: [0,0,1]
    Shrunk 6 time(s)
    Got error: Property failed by returning false

Looking at our solution, this makes sense. The result of n % 0 is undefined. Unfortunately, the kata doesn’t specify what the behavior of our solution should be when k equals 0, so we’re left to figure that out ourselves.

Let’s just set up a precondition in our test that k should never equal 0:


test("it works", () => {
  fc.assert(
    fc.property(fc.integer(), fc.integer(), fc.nat(), (x, k, n) => {
      fc.pre(k !== 0);
      let y = x + n * k;
      return n === divisibleCount(x, y, k);
    })
  );
});

Great!

Unfortunately, there’s another problem. Without putting an upper bound on the size of n * k, our solution will generate potentially massive arrays. This will quickly eat through the memory allocated to our process and result in a crash.

Let’s add some upper and lower bounds to our generated k and n values:


test("it works", () => {
  fc.assert(
    fc.property(fc.integer(), fc.integer(-100, 100), fc.nat(100), (x, k, n) => {
      fc.pre(k !== 0);
      let y = x + n * k;
      return n === divisibleCount(x, y, k);
    })
  );
});

Perfect. Our starting integer, x, can be any positive or negative integer, but our generated values of k are clamped between -100 and 100, and n ranges from 0 to 100. These values should be large enough to thoroughly test our solution, and small enough to prevent memory issues from arising.

Our Second Solution

In hindsight, our solution seems to be making inefficient use of both time and memory. If we consider the fact that our property test is computing y in terms of x, n, and k, it stands to reason that we should be able to compute n, our desired result, in terms of x, y, and k. If we can manage this, our solution will run in both constant time and constant space.

Let’s use some algebra and work our way backwards from calculating y to calculating n. If y = x + n * k, that means that y - x = n * k. Dividing by k gives us our equation for computing n: n = (y - x) / k.

Let’s replace our original divisibleCount solution with this equation:


const divisibleCount = (x, y, k) => (y - x) / k;

And rerun our test suite:


$ jest
 PASS ./index.test.js
  ✓ it works (8ms)

  Test Suites: 1 passed, 1 total
  Tests: 1 passed, 1 total

Wonderful!

Setting Apollo Context from HTTP Headers in a Meteor Application

Pete Corey — Wed, 13 Nov 2019 00:00:00 +0000

I was recently tasked with modifying an Apollo-based GraphQL endpoint served by a Meteor application to set a field in the resolvers’ context based on a HTTP header pulled out of the incoming GraphQL request.

This should be an easy, well-documented ask.

Unfortunately, Meteor’s Apollo integration seems to exist in an awkward, undocumented state. The createApolloServer function exposed by the meteor/apollo package doesn’t seem to have any real documentation anywhere I could find, and the Github repository linked to by the package doesn’t seem to relate to the code in question.

How can I access the current HTTP request when building my resolvers’ context?

With no documentation to guide me, I dove into the package’s source code on my machine to find answers. The source for the package in question lives in ~/.meteor/packages/apollo/<version>/os/src/main-server.js on my machine. The createApolloServer function accepts a customOptions object as its first argument. I quickly learned after digging through the source that if customOptions is a function, createApolloServer will call that function with the current request (req) as its only argument, and use the function call’s result as the value of customOptions:


const customOptionsObject =
  typeof customOptions === 'function'
    ? customOptions(req) 
    : customOptions;

This means we need to change our current call to createApolloServer from something like this:


import { createApolloServer } from "meteor/apollo";

createApolloServer({ schema }, { path });

To something like this:


createApolloServer(
  req => ({
    context: {
      someValue: req.headers['some-value']
    },
    schema
  }),
  { path }
);

This information is probably only pertinent to myself and an ever-shrinking number of people, but if you’re one of those people, I hope this helps.

The Collatz Sequence in J

Pete Corey — Wed, 06 Nov 2019 00:00:00 +0000

I’ve been playing around quite a bit with the Collatz Conjecture after watching the latest Coding Train video on the subject. In an effort to keep my J muscles from atrophying, I decided to use the language to implement a Collatz sequence generator.

At its heart, the Collatz sequence is a conditional expression. If the current number is even, the next number in the sequence is the current number divided by two. If the current number is odd, the next number in the sequence is one plus the current number multiplied by three:

We can write each of these branches with some straight-forward J code. Our even case is simply the divide (%) verb bonded (&) to 2:


   even =: %&2

And our odd case is a “monadic noun fork” of 1 plus (+) 3 bonded (&) to multiply (*):


   odd =: 1 + 3&*

We can tie together those two verbs and use agenda (@.) to pick which one to call based on whether the argument is even:


   next =: even`odd@.pick

Our pick verb is testing for even numbers by checking if 1: equals (=) 2 bonded to the residue verb (|).


   pick =: 1:=2&|

We can test our next verb by running it against numbers with known next values. After 3, we’d expect 10, and after 10, we’d expect 5:


   next 3
10
   next 10
5

Fantastic!

J has an amazing verb, power (^:), that can be used to find the “limit” of a provided verb by continuously reapplying that verb to its result until a repeated result is encountered. If we pass power boxed infinity (<_) as an argument, it’ll build up a list of all the intermediate results.

This is exactly what we want. To construct our Collatz sequence, we’ll find the limit of next for a given input, like 12:


   next^:(<_) 12

But wait, there’s a problem! A loop exists in the Collatz sequences between 4, 2, and 1. When we call next on 1, we’ll receive 4. Calling next on 4 returns 2, and calling next on 2 returns 1. Our next verb never converges to a single value.

To get over this hurdle, we’ll write one last verb, collatz, that checks if the argument is 1 before applying next:


   collatz =: next`]@.(1&=)

Armed with this new, converging collatz verb, we can try finding our limit again:


   collatz^:(<_) 12
12 6 3 10 5 16 8 4 2 1

Success! We’ve successfully implemented a Collatz sequence generator using the J programming langauge. Just for fun, let’s plot the sequence starting with 1000:


   require 'plot'
   plot collatz^:(<_) 1000

Rendering a React Application Across Multiple Containers

Pete Corey — Mon, 14 Oct 2019 00:00:00 +0000

A few of my recent articles have been embedding limited builds of Glorious Voice Leader directly into the page. At first, this presented an interesting challenge. How could I render a single React application across multiple container nodes, while maintaining shared state between all of them?

While the solution I came up with probably isn’t best practice, it works!

As a quick example, imagine you have a simple React component that manages a single piece of state. The user can change that state by pressing one of two buttons:


const App = () => {
  let [value, setValue] = useState("foo");
  return (
    <div>
      <button onClick={() => setValue("foo")}>
        Value is "{value}". Click to change to "foo"!
      </button>
      <button onClick={() => setValue("bar")}>
        Value is "{value}". Click to change to "bar"!
      </button>
    </div>
  );
};

Normally, we’d render our App component into a container in the DOM using ReactDOM.render:


ReactDOM.render(<App />, document.getElementById('root'));

But what if we want to render our buttons in two different div elements, spread across the page? Obviously, we could build out two different components, one for each button, and render these components in two different DOM containers:


const Foo = () => {
  let [value, setValue] = useState("foo");
  return (
    <button onClick={() => setValue("foo")}>
      Value is "{value}". Click to change to "foo"!
    </button>
  );
};

const Bar = () => {
  let [value, setValue] = useState("foo");
  return (
    <button onClick={() => setValue("bar")}>
      Value is "{value}". Click to change to "bar"!
    </button>
  );
};

ReactDOM.render(<Foo />, document.getElementById('foo'));
ReactDOM.render(<Bar />, document.getElementById('bar'));

But this solution has a problem. Our Foo and Bar components maintain their own versions of value, so a change in one component won’t affect the other.

Amazingly, it turns out that we can create an App component which maintains our shared state, render that component into our #root container, and within App we can make additional calls to ReactDOM.render to render our Foo and Bar components. When we call ReactDOM.render we can pass down our state value and setters for later use in Foo and Bar:


const App = () => {
  let [value, setValue] = useState("foo");
  return (
    <>
      {ReactDOM.render(
        <Foo value={value} setValue={setValue} />,
        document.getElementById("foo")
      )}
      {ReactDOM.render(
        <Bar value={value} setValue={setValue} />,
        document.getElementById("bar")
      )}
    </>
  );
};

Our Foo and Bar components can now use the value and setValue props provided to them instead of maintaining their own isolated state:


const Foo = ({ value, setValue }) => {
  return (
    <button onClick={() => setValue("foo")}>
      Value is "{value}". Click to change to "foo"!
    </button>
  );
};

const Bar = ({ value, setValue }) => {
  return (
    <button onClick={() => setValue("bar")}>
      Value is "{value}". Click to change to "bar"!
    </button>
  );
};

And everything works! Our App is “rendered” to our #root DOM element, though nothing actually appears there, and our Foo and Bar components are rendered into #foo and #bar respectively.

Honestly, I’m amazed this works at all. I can’t imagine this is an intended use case of React, but the fact that it’s still a possibility made my life much easier.

Happy hacking.

Animating a Canvas with Phoenix LiveView: An Update

Pete Corey — Tue, 01 Oct 2019 00:00:00 +0000

In my previous post on animating an HTML5 canvas using Phoenix LiveView, we used both a phx-hook attribute and a phx-update="ignore" attribute simultaneously on a single DOM element. The goal was to ignore DOM updates (phx-update="ignore"), while still receiving updated data from our server (phx-hook) via our data-particles attribute.

Unfortunately, the technique of using both phx-hook and phx-update="ignore" on a single component no longer works as of phoenix_live_view version 0.2.0. The "ignore" update rule causes our hook’s updated callback to not be called with updates. In hindsight, the previous behavior doesn’t even make sense, and the new behavior seems much more consistent with the metaphors in play.

Joxy pointed this issue out to me, and helped me come up with a workaround. The solution we landed on is to wrap our canvas component in another DOM element, like a div. We leave our phx-update="ignore" on our canvas to preserve our computed width and height attributes, but move our phx-hook and data attributes to the wrapping div:


<div
  phx-hook="canvas"
  data-particles="<%= Jason.encode!(@particles) %>"
>
  <canvas phx-update="ignore">
    Canvas is not supported!
  </canvas>
</div>

In the mounted callback of our canvas hook, we need to look to the first child of our div to find our canvas element:


mounted() {
  let canvas = this.el.firstElementChild;
  ...
}

Finally, we need to pass a reference to a Phoenix Socket directly into our LiveSocket constructor to be compatible with our new version of phoenix_live_view:


import { Socket } from "phoenix";
let liveSocket = new LiveSocket("/live", Socket, { hooks });

And that’s all there is to it! Our LiveView-powered confetti generator is back up and running with the addition of a small layer of markup.

For more information on this update, be sure to check out this issue I filed to try to get clarity on the situation. And I’d like to give a huge thanks to Joxy for doing all the hard work in putting this fix together!

Apollo Quirks: Polling After Refetching with New Variables

Pete Corey — Mon, 23 Sep 2019 00:00:00 +0000

While working on a recent client project, Estelle and I ran into a fun Apollo quirk. It turns out that an Apollo query with an active pollInterval won’t respect new variables provided by calls to refetch.

To demonstrate, imagine we’re rendering a paginated table filled with data pulled from the server:

const Table = () => {
    let { data } = useQuery(gql`
        query items($page: Int!) {
            items(page: $page) {
                pages
                results {
                    _id
                    result
                }
            }
        }
    `, {
        pollInterval: 5000
    });

    return (
        <>
            <table>
                {data.items.results.map(({ _id, result }) => (
                    <tr key={_id}>
                        <td>{result}</td>
                    </tr>
                ))}
            </table>
        </>
    );
};

The items in our table change over time, so we’re polling our query every five seconds.

We also want to give the user buttons to quickly navigate to a given page of results. Whenever a user presses the “Page 2” button, for example, we want to refetch our query with our variables set to { page: 2 }:

 const Table = () => {
- let { data } = useQuery(gql`
+ let { data, refetch } = useQuery(gql`
         query items($page: Int!) {
             items(page: $page) {
                 pages
                 results {
                     _id
                     result
                 }
             }
         }
     `, {
         pollInterval: 5000
     });

+ const onClick = page => {
+ refetch({ variables: { page } });
+ };

     return (
         <>
             <table>
                 {data.items.results.map(({ _id, result }) => (
                     <tr key={_id}>
                         <td>{result}</td>
                     </tr>
                 ))}
             </table>
+ {_.chain(data.items.pages)
+ .map(page => (
+ <Button onClick={() => onClick(page)}>
+ Page {page + 1}
+ </Button>
+ ))
+ .value()}
         </>
     );
 };

This works… for a few seconds. But then we’re unexpectedly brought back to the first page. What’s happening here?

It turns out that our polling query will always query the server with the variables it was given at the time polling was initialized. So in our case, even though the user advanced to page two, our polling query will fetch page one and render those results.

So how do we deal with this? This GitHub issue on the apollo-client project suggests calling stopPolling before changing the query’s variables, and startPolling to re-enable polling with those new variables.

In our case, that would look something like this:

 const Table = () => {
- let { data, refetch } = useQuery(gql`
+ let { data, refetch, startPolling, stopPolling } = useQuery(gql`
         query items($page: Int!) {
             items(page: $page) {
                 pages
                 results {
                     _id
                     result
                 }
             }
         }
     `, {
         pollInterval: 5000
     });

     const onClick = page => {
+ stopPolling();
         refetch({ variables: { page } });
+ startPolling(5000);
     };

     return (
         <>
             <table>
                 {data.items.results.map(({ _id, result }) => (
                     <tr key={_id}>
                         <td>{result}</td>
                     </tr>
                 ))}
             </table>
             {_.chain(data.items.pages)
                 .map(page => (
                 <Button onClick={() => onClick(page)}>
                     Page {page + 1}
                 </Button>
                 ))
                 .value()}
         </>
     );
 };

And it works! Now our polling queries will fetch from the server with the correctly updated variables. When a user navigates to page two, they’ll stay on page two!

My best guess for why this is happening, and why the stopPolling/startPolling solution works is that when polling is started, the value of variables is trapped in a closure. When refetch is called, it changes the reference to the options.variables object, but not the referenced object. This means the value of options.variables doesn’t change within polling interval.

Calling stopPolling and startPolling forces our polling interval to restart under a new closure with our new variables values.

Elixir Style Conditions in Javascript

Pete Corey — Mon, 16 Sep 2019 00:00:00 +0000

Elixir has a useful control flow structure called cond that lets you branch on arbitrary conditions. Unlike the more common switch control structure (case in Elixir), cond doesn’t match against a predetermined value. Instead, it evaluates each condition, in order, looking for the first one that evaluates to a truthy value (not nil or false).


numbers = [1, 2, 3]
result = cond do
  4 in numbers ->
    :four
  6 == Enum.sum(numbers) ->
    :sum
  true ->
    :default
end

This is all probably old hat to you.

As I mentioned, cond can be an incredibly useful control structure, and there are times when I’ve missed it while working in languages like Javascript that only have switch expressions.

A traditional Javascript implementation of the above (with a little help from Lodash) would look something like this:


let numbers = [1, 2, 3];
if (_.includes(numbers, 4)) {
  var result = "four";
} else if (6 === _.sum(numbers)) {
  var result = "sum";
} else {
  var result = "default";
}

However, I recently stumbled upon a trick that lets you implement a switch statement in Javascript that behaves very similarly to a cond expression in Elixir. The key is to switch on the value of true. The case expressions that evaluate to true will match, and their corresponding statements will be evaluated in order.


let numbers = [1, 2, 3];
switch (true) {
  case _.includes(numbers, 4):
    var result = "four";
    break;
  case 6 === _.sum(numbers):
    var result = "sum";
    break;
  default:
    var result = "default";
    break;
}

Whether or not this is any more useful or readable than a series of if/else blocks is debatable. That said, this is definitely an interesting example of perspective shifting and seeing old code in a new light. Hopefully you find it as interesting as I do.

Obverse and Under

Pete Corey — Fri, 13 Sep 2019 00:00:00 +0000

I previously wrote about plotting an “amazing graph” using the J programming language. The solution I landed on looked something like this:

require 'plot'
f =: ] - [: #. [: |. #:
'type dot' plot f"0 p: i. 10000

Our verb, f, is taking a very explicit approach by making judicious use of “capped” ([[:](https://www.jsoftware.com/help/dictionary/d502.htm)) verb trains. We’re essentially saying that f is (=:) the given number ([]](https://www.jsoftware.com/help/dictionary/d500.htm)) minus (-) the base two (#.) of the reverse (|.) of the antibase two (#:) of the given number.

Several members of the J community pointed out to me that this verb could be simplified with the help of the “under” (&.) conjunction. Let’s dig into what “under” is, and how we can use it.

Under What?

The best way to think about “under” (&.), as explained by the NuVoc page on “under”, is to think in terms of domains and transformations in and out of those domains.

Verb v defines a transformation of the argument(s) (x and) y into the v-domain. Next, verb u operates on the transformed argument(s). Lastly the result is transformed back from the v-domain to the original domain.

In our example, the domain of our input is base ten, but the transformation we want to apply (reversal) needs to happen in the base two domain. “Under” (&.) can be used to transform our input into base two (#:), apply our reversal (|.), and transform the result of that reversal back to our original base ten domain with the obverse, or opposite, of our base two verb, anti base (#.):

f =: ] - |. &. #:

Notice that we’re not explicitly stating how to transform the result of our reversal back into our original domain. J knows that the obverse of #: is #., and automatically applies it for us.

Out of the box, J comes with many obverse pairings. “Open” (>), for example, is the obverse of “box” (<), and visa versa. This pairing is especially useful when applying transformations to boxed values:

1+&.>1;2;3
┌─┬─┬─┐
│2│3│4│
└─┴─┴─┘

Check out a full listing of obverse pairs at the end of this Shades of J article.

Inferred Obverses

Even compound verbs built up of verbs with well-defined obverse pairings can be used with “under” (&.). J will correctly infer and apply the compound obverse without any intervention or instruction.

For example, if we wanted to unbox a list of values and then work with them in the “square root domain” (whatever that means), we could do something like this:

1+&.([:%:>)1;2;3
┌────────────────┐
│4 5.82843 7.4641│
└────────────────┘

J takes each value, opens it and finds its square root ([:%:>), adds one to the result, and then squares and boxes up ([:*:<) the incremented value.

Explicit Obverses

Even more interestingly, if an obverse pairing isn’t defined or inferable for a given verb, J lets us define our own pairing using the “obverse” (:.) verb.

As an example, imagine that we have a JSON string holding an array of values. We want to parse our string, perform some operation on those values, and then serialize the resulting list back into JSON.

We can use the dec_json and enc_json verbs provided by the convert/json package, and tell J that the obverse of dec_json is enc_json:

json =: dec_json :. enc_json

Running dec_json on a JSON array like '[1, 2, 3]' will return a list of boxed numbers, so we’ll want to open each of these boxes, perform our operation, and box the results back up. This sounds like another job for “under” (&.):

transform =: 1&+&.>

All together, we can perform our transform “under” (&.) the json domain:

transform &. json '[1, 2, 3]'
[2,3,4]

And our result is the JSON string '[2,3,4]'!

“Under” is definitely a very powerful conjunction, and I can see myself using it extensively in the future. Thanks to everyone in the J community who was kind enough to point it out and teach me something new!

Animating a Canvas with Phoenix LiveView

Pete Corey — Mon, 02 Sep 2019 00:00:00 +0000

Phoenix LiveView recently released a new feature called “hooks” that introduces Javascript interoperability into the LiveView lifecycle. Put simply, we can now run arbitrary Javascript every time a DOM node is changed by LiveView! LiveView hooks are a complete game changer , and open the doors to a whole new world of applications that can be built with this amazing technology.

As a proof of concept, let’s use LiveView hooks to animate an HTML5 canvas in real time using data provided by the server!

Getting Set Up

To keep this article short(er), we’ll skip the rigmarole of configuring your application to use LiveView. If you need help with this step, I highly recommend you check out Sophie DeBenedetto’s thorough walkthrough. Be sure to cross reference with the official documentation, as things are moving quickly in the LiveView world.

Moving forward, let’s assume that you have a bare-bones LiveView component attached to a route that looks something like this:

defmodule LiveCanvasWeb.PageLive do
  use Phoenix.LiveView

  def render(assigns) do
    ~L"""
    <canvas>
      Canvas is not supported!
    </canvas>
    """
  end

  def mount(_session, socket) do
    {:ok, socket}
  end
end

We’ll also assume that your assets/js/app.js file is creating a LiveView connection:

import LiveSocket from "phoenix_live_view";

let liveSocket = new LiveSocket("/live");

liveSocket.connect();

Now that we’re on the same page, let’s get started!

Generating Data to Animate

Before we start animating on the client, we should have some data to animate. We’ll start by storing a numeric value called i in our LiveView process’ assigns:

def mount(_session, socket) do
  {:ok, assign(socket, :i, 0)}
end

Next, we’ll increase i by instructing our LiveView process to send an :update message to itself after a delay of 16 milliseconds:

def mount(_session, socket) do
  Process.send_after(self(), :update, 16)
  {:ok, assign(socket, :i, 0)}
end

When we handle the :udpate message in our process, we’ll schedule another recursive call to :update and increment the value of i in our socket’s assigns:

def handle_info(:update, %{assigns: %{i: i}} = socket) do
  Process.send_after(self(), :update, 16)
  {:noreply, assign(socket, :i, i + 0.05)}
end

Our LiveView process now has an i value that’s slowly increasing by 0.05 approximately sixty times per second.

Now that we have some data to animate, let’s add a canvas to our LiveView’s template to hold our animation:

def render(assigns) do
  ~L"""
  <canvas data-i="<%= @i %>">
    Canvas is not supported!
  </canvas>
  """
end

Notice that we’re associating the value of i with our canvas by assigning it to a data attribute on the DOM element. Every time i changes in our process’ state, LiveView will update our canvas and set the value of data-i to the new value of i.

This is great, but to render an animation in our canvas, we need some way of executing client-side Javascript every time our canvas updates. Thankfully, LiveView’s new hook functionality lets us do exactly that!

Hooking Into LiveView

LiveView hooks lets us execute Javascript at various points in a DOM node’s lifecycle, such as when the node is first mounted, when it’s updated by LiveView, when it’s destroyed and removed from the DOM, and when it becomes disconnected or reconnected to our Phoenix server.

To hook into LiveView’s client-side lifecycle, we need to create a set of hooks and pass them into our LiveSocket constructor. Let’s create a hook that initializes our canvas’ rendering context when the element mounts, and renders a static circle every time the element updates:

let hooks = {
  canvas: {
    mounted() {
      let canvas = this.el;
      let context = canvas.getContext("2d");

      Object.assign(this, { canvas, context });
    },
    updated() {
      let { canvas, context } = this;

      let halfHeight = canvas.height / 2;
      let halfWidth = canvas.width / 2;
      let smallerHalf = Math.min(halfHeight, halfWidth);

      context.clearRect(0, 0, canvas.width, canvas.height);
      context.fillStyle = "rgba(128, 0, 255, 1)";
      context.beginPath();
      context.arc(
        halfWidth,
        halfHeight,
        smallerHalf / 16,
        0,
        2 * Math.PI
      );
      context.fill();
    }
  }
};

let liveSocket = new LiveSocket("/live", { hooks });

liveSocket.connect();

Notice that we’re storing a reference to our canvas and our newly created rendering context on this. When LiveView calls our lifecycle callbacks, this points to an instance of a ViewHook class. A ViewHook instance holds references to our provided lifecycle methods, a reference to the current DOM node in el, and various other pieces of data related to the current set of hooks. As long as we’re careful and we don’t overwrite these fields, we’re safe to store our own data in this.

Next, we need to instruct LiveView to attach this new set of canvas hooks to our canvas DOM element. We can do that with the phx-hook attribute:

<canvas
  data-i="<%= @i %>"
  phx-hook="canvas"
>
  Canvas is not supported!
</canvas>

When our page reloads, we should see our circle rendered gloriously in the center of our canvas.

Resizing the Canvas

On some displays, our glorious circle may appear to be fuzzy or distorted. This can be fixed by scaling our canvas to match the pixel density of our display. While we’re at it, we might want to resize our canvas to fill the entire available window space.

We can accomplish both of these in our mounted callback:

mounted() {
  let canvas = this.el;
  let context = canvas.getContext("2d");
  let ratio = getPixelRatio(context);

  resize(canvas, ratio);

  Object.assign(this, { canvas, context });
}

Where getPixelRatio is a helper function that determines the ratio of physical pixels in the current device’s screen to “CSS pixels” which are used within the rendering context of our canvas:

const getPixelRatio = context => {
  var backingStore =
    context.backingStorePixelRatio ||
    context.webkitBackingStorePixelRatio ||
    context.mozBackingStorePixelRatio ||
    context.msBackingStorePixelRatio ||
    context.oBackingStorePixelRatio ||
    context.backingStorePixelRatio ||
    1;

  return (window.devicePixelRatio || 1) / backingStore;
};

And resize is a helper function that modifies the canvas’ width and height attributes in order to resize our canvas to fit the current window, while fixing any pixel density issues we may be experiencing:

const resize = (canvas, ratio) => {
  canvas.width = window.innerWidth * ratio;
  canvas.height = window.innerHeight * ratio;
  canvas.style.width = `${window.innerWidth}px`;
  canvas.style.height = `${window.innerHeight}px`;
};

Unfortunately, our canvas doesn’t seem to be able to hold onto these changes. Subsequent calls to our updated callback seem to lose our resize changes, and the canvas reverts back to its original, blurry self. This is because when LiveView updates our canvas DOM node, it resets our width and height attributes. Not only does this revert our pixel density fix, it also forcefully clears the canvas’ rendering context.

LiveView has a quick fix for getting around this problem. By setting phx-update to "ignore" on our canvas element, we can instruct LiveView to leave our canvas element alone after its initial mount.

<canvas
  data-i="<%= @i %>"
  phx-hook="canvas" 
  phx-update="ignore"
>
  Canvas is not supported!
</canvas>

Now our circle should be rendered crisply in the center of our screen.

Animating Our Circle

We didn’t go all this way to render a static circle in our canvas. Let’s tie everything together and animate our circle based on the ever-changing values of i provided by the server!

The first thing we’ll need to do is update our updated callback to grab the current value of the data-i attribute:

let i = JSON.parse(canvas.dataset.i);

The value of canvas.dataset.i will reflect the contents of our data-i attribute. All data attributes are stored as strings, so a call to JSON.parse will convert a value of "0.05" to its numeric counterpart.

Next, we can update our rendering code to move our circle based on the value of i:

context.arc(
  halfWidth + (Math.cos(i) * smallerHalf) / 2,
  halfHeight + (Math.sin(i) * smallerHalf) / 2,
  smallerHalf / 16,
  0,
  2 * Math.PI
);

That’s it! With those two changes, our circle will rotate around the center of our canvas based entirely on real-time data provided by our server!

Requesting Animation Frames

Our solution works, but by forcing re-renders on the browser, we’re being bad net citizens. Our client may be forcing re-renders when its tab is out of focus, or it may be re-rendering more than sixty times per second, wasting CPU cycles.

Instead of telling the browser to re-render our canvas on every LiveView update, we should invert our control over rendering and request an animation frame from the browser on every update.

        <p>I find myself producing lots of ranging content from books and articles, like the one you're reading now, to open source software projects. If you like what I'm doing, <strong>nothing shows your support more than signing up for <a href="#newsletter">my mailing list</a></strong>.</p>

The process for this is straight forward. In our updated callback, we’ll wrap our rendering code in a lambda passed into requestAnimationFrame. We’ll save the resulting request reference to this.animationFrameRequest:

this.animationFrameRequest = requestAnimationFrame(() => {
  context.clearRect(0, 0, canvas.width, canvas.height);
  context.beginPath();
  context.arc(
    halfWidth + (Math.cos(i) * smallerHalf) / 2,
    halfHeight + (Math.sin(i) * smallerHalf) / 2,
    smallerHalf / 16,
    0,
    2 * Math.PI
  );
  context.fill();
});

It’s conceivable that our LiveView component may update multiple times before our browser is ready to re-render our canvas. In those situations, we’ll need to cancel any previously requested animation frames, and re-request a new frame. We can do this by placing a guard just above our call to requestAnimationFrame:

if (this.animationFrameRequest) {
  cancelAnimationFrame(this.animationFrameRequest);
}

With those two changes, our LiveView hooks will now politely request animation frames from the browser, resulting in a smoother experience for everyone involved.

Taking it Further

Using a canvas to animate a numeric value updated in real-time by a LiveView process running on the server demonstrates the huge potential power of LiveView hooks, but it’s not much to look at.

We can take things further by generating and animating a much larger set of data on the server. Check out this example project that simulates over two hundred simple particles, and renders them on the client at approximately sixty frames per second:

Is it a good idea to take this approach if your goal is to animate a bunch of particles on the client? Probably not. Is it amazing that LiveView gives us the tools to do this? Absolutely, yes! Be sure to check out the entire source for this example on Github!

Hooks have opened the doors to a world of new possibilities for LiveView-based applications. I hope this demonstration has given you a taste of those possibilities, and I hope you’re as eager as I am to explore what we can do with LiveView moving forward.