DEV Community: Robin van der Vleuten

A JSONPath engine that follows the spec and can't run code

Robin van der Vleuten — Mon, 20 Jul 2026 07:44:12 +0000

I needed to pull a few fields out of some JSON. Nothing exotic, a couple of nested values and one filtered list. So I did the ordinary thing, reached for a JSONPath library, wrote a query, and moved on.

A week later I swapped that library for an unrelated reason, and the same query came back with different results. Not wrong exactly, just different. My $..author matched one extra node, a slice counted from the other end, and a filter that worked before now threw. Welcome to JSONPath, where for the better part of two decades every implementation agreed on the dollar sign and improvised the rest.

The standard nobody had

Stefan Gössner's original write-up from 2007 sketched the idea and left the corners to the reader. Everyone filled them in differently. Does $..book[0] give you the first book in each array or the first book overall? Do you write a filter [?(@.price)] or [?@.price]? What does a negative index do at the edge of an array? Pick two libraries and you could get two answers to each.

That changed in February 2024, when the IETF published RFC 9535 and wrote the rules down: the selectors, the filter grammar, the meaning of a missing value, with a compliance test suite so you can check yourself against them. JSONPath went from folklore to specification.

So I wanted a library that actually followed it. Reading the spec, I noticed the part that makes JSONPath worth using is also the part that keeps turning up in security advisories.

The filter that runs a shell

Filters are the reason you pick JSONPath over a hand-written loop. $..book[?@.price < 10] reads like a sentence and does real work: keep the nodes where the comparison holds.

Evaluating that comparison is where libraries cut corners. jsonpath-plus, the most-downloaded option on npm, built its filters on generated code. That works right up until part of the query comes from somewhere you don't control. At that point a filter is no longer a filter, it is a program running with your process's permissions, and that is exactly how it went: CVE-2024-21534 was a remote code execution through a crafted JSONPath, with more to follow when the first patch left a gap.

There is a quieter cost too. Generating code at runtime needs 'unsafe-eval' in your Content Security Policy, and if you have already shipped a strict CSP, granting that back undoes most of the reason you shipped it.

A scout that stays on the path

padvinder is my answer to both. The name is Dutch for "pathfinder", and also the word for a scout, which fit something whose entire job is to go find a node and come back without disturbing anything on the way. It implements RFC 9535 in roughly 2.75KB min+gzip with no dependencies, and it clears all 456 valid-selector cases in the official suite.

import { query, find } from 'padvinder';

const data = { store: { book: [
  { title: 'Sayings of the Century', price: 8.95, category: 'reference' },
  { title: 'Sword of Honour', price: 12.99, category: 'fiction' },
  { title: 'Moby Dick', price: 8.99, category: 'fiction' },
] } };

find('$..book[?@.price < 10].title', data);
// => ['Sayings of the Century', 'Moby Dick']

find runs a query once. For a path you will reuse, query compiles it and returns a runner you can point at different data:

const fiction = query('$.store.book[?@.category == "fiction"].title');
fiction(data); // => ['Sword of Honour', 'Moby Dick']

Hand it a broken path and it throws a SyntaxError at compile time, not partway through your data. A path that matches nothing returns [].

The parts of the spec worth carrying around

Selectors are the dependable half. If you have written JSONPath they hold no surprises: $ for the root, .name or ['name'] for a child, [0] and [-1] for indexes, [1:3:2] for a stepped slice, * for every child, .. for a recursive descent, and [0, 2] to union a handful of selectors.

Filters are where RFC 9535 spends its pages, and a few of its rulings are worth keeping in your head:

Existence is not truthiness. A bare [?@.discount] keeps every child that has a discount, even one set to null or 0. It asks whether the path exists, not whether the value is truthy.
A missing path is "nothing", not an error. In a comparison, [?@.a.b == 1] just fails to match when a is absent instead of throwing, so filters stay safe to write against ragged data.
<strong>==</strong> is deep equality. [?@.tags == ["new", "sale"]] compares structure, and none of JavaScript's loose coercion tags along.
<strong>$</strong> is still the root inside a filter. A child can be measured against something elsewhere in the document: [?@.price > $.store.bicycle.price].

Five functions ship built in, taken straight from the RFC: length(), count(), value(), and the regex tests match() for a whole string and search() for a substring. For anything else you pass your own and call it with @ as the current node:

find('$..book[?discounted(@)].title', data, { discounted: b => b.price < 9 });

That registry is the single place executable code enters, and it enters from your side, by name. Whoever writes the query chooses which of your functions to call, never what it does.

The part I tried to borrow

padvinder started out leaning on xprsn. A JSONPath filter is an expression over the current node, so my first version handed each [?...] straight to xprsn and let it do the evaluating. The easy queries passed. The spec did not.

RFC 9535 filters only look like ordinary expressions. Their operands are queries as often as values, so @.a.b is a small path that runs and may match nothing at all. A bare @.discount is an existence test, not a truthiness check. Equality is structural. A missing singular query turns into a special "nothing" that compares equal to no value on earth. A general expression evaluator has no reason to think in any of those terms, and bending xprsn to fit would have meant loading it with JSONPath rules it has no business carrying.

So I stopped borrowing. The filter grammar moved into padvinder as its own recursive-descent parser written against the RFC, and the xprsn dependency came out with it. That is the real reason padvinder has zero runtime dependencies: the one piece it could have reused is the one piece the standard needed it to own.

Why it can't be turned into code

The safety is the same compile-to-closures technique I wrote up for xprsn, so here is only its shape. Every selector and every filter becomes a small function that already exists in the shipped source, and a parser wires them together. Query text never gets reassembled into JavaScript, which leaves nothing to inject into and nothing for a CSP to block.

The test suite keeps that honest by running under node --disallow-code-generation-from-strings, the flag that turns every eval-shaped call into a thrown error, exactly as a strict CSP does in the browser.

The prototype escapes are closed as well. __proto__, constructor, and prototype never match, in a path or a filter, so [?@.constructor] finds nothing rather than climbing toward Function. Matching reads own properties only, and a query never writes back to the data you gave it. A test snapshots the input and compares it afterward to prove that last one.

Is it the right tool?

If your paths are all written by you, committed to your repo, and never near a strict CSP, you do not need any of this, and a bigger library with a wider feature set is a reasonable pick. padvinder is built for the narrower spot: the query might come from a user or a config row, the page runs under a CSP you will not weaken, and you would rather the results match the standard than one project's reading of it. The 456-case suite is what I lean on there. It is not my definition of correct, it is the RFC's.

Getting started

npm install padvinder

The README has the full selector table, the filter grammar, and the safety notes. Use find when you want an answer now and query when you will ask the same question a lot.

Closing thoughts

The satisfying part of this one was not the size or even the safety. It was having a compliance suite play judge. Most of my small libraries are correct by my own definition, which is a comfortable and slightly circular place to sit. Here, 456 cases I did not write got a vote, and "I think this is right" became "the standard says this is right." That judge is also what ended the borrowing: you do not approximate your way through 456 cases, so the evaluator I had hoped to reuse had to go. I would take the trade on more projects if the specs existed for them.

If you have been running JSONPath over input you did not write yourself, I would like to hear how you drew the line around it.

Let users write rules without letting them write JavaScript

Robin van der Vleuten — Fri, 17 Jul 2026 12:16:26 +0000

You're building the kind of feature where the logic can't all live in your code. A pricing rule marketing wants to tweak without a deploy. A feature flag that should switch on when user.age > 18 and "admin" in user.roles. A filter your users configure themselves on a settings screen.

The condition is data, not code. So you store it as a string and evaluate it when you need it.

The obvious way to run a string like that is new Function or eval:

const rule = 'items[0].price * qty > 100';
const check = new Function('items', 'qty', `return ${rule}`);

check(order.items, order.qty); // works... on your machine

Then it meets production.

Where eval stops being an option

Two things break it.

The first is your Content Security Policy. Ship a strict one, the kind you actually want:

Content-Security-Policy: script-src 'self'

The browser now refuses to run anything built from a string. new Function and eval both need 'unsafe-eval', and adding that back gives up most of what you added the header for. I hit the same wall from the other side when I wrote about passing server data to the client without breaking CSP.

The second problem is trust. The moment a user, or a row in your database, can shape the expression, eval hands them the whole runtime. items.constructor.constructor("return process")() isn't a filter. It's a shell.

So you go looking for a library. expr-eval has been around for years but hasn't seen much attention lately. math.js is excellent and large, and it's aimed at math, not at "is this user an admin over eighteen." I wanted a small core that does one job: compile an expression, evaluate it against data, and let me add my own functions. Nothing fit that shape, so I pulled the piece I'd already written for a commercial project out into its own package.

Meet xprsn

xprsn evaluates expressions against data you hand it, without ever running them as JavaScript. It's about 1.2KB min+gzip, with zero dependencies. Let's see it in action.

import { compile, evaluate } from 'xprsn';

// One-shot:
evaluate('items[0].price * qty > 100', { items: [{ price: 60 }], qty: 2 });
// => true

// Compile once, evaluate many times:
const isAdmin = compile('user.age > 18 and "admin" in user.roles');
isAdmin({ user: { age: 30, roles: ['admin'] } }); // => true
isAdmin({ user: { age: 16, roles: [] } });        // => false

compile parses the expression once and hands back a function. Call it with different data as often as you like. evaluate is the shorthand for when you only need the answer once.

Custom functions come in as the last argument. They're the extension point I built the whole thing around:

evaluate('lower(name) == "robin"', { name: 'ROBIN' }, { lower: s => s.toLowerCase() });
// => true

Only the functions you pass in are callable. There's no ambient standard library to lock down, because there isn't one.

The syntax fits on a postcard

Enough to write real rules, and deliberately not enough to write a program:

Literals and collections. Numbers like 42 and 4.2, strings, true/false/null, arrays [1, 2, 3], and hashes { key: value }.
The usual operators. Arithmetic (+ - * / % **), comparison (== != < > <= >=), and logic in either spelling: and/&&, or/||, not/!.
Membership. "admin" in roles checks an array's items, a string's substrings, and an object's own keys.
Access and calls. user.name, items[0], items[i + 1], and methods on your data like name.toUpperCase().
Ternaries. a ? b : c, plus the a ?: b shorthand for "a if it's truthy, otherwise b."

One deliberate sharp edge: == is strict. 1 == "1" is false, because JavaScript's loose equality is a footgun you don't want to hand to whoever is editing these rules. To stay tiny, xprsn also leaves out string concatenation, regex matching, ranges, bitwise operators, and null-safe ?.. Most of those you can add back as custom functions, though. Pass a concat or a matches into the registry and call it like any built-in.

const funcs = {
  concat: (...parts) => parts.join(''),
  matches: (str, re) => new RegExp(re).test(str),
};

evaluate('concat(first, " ", last)', { first: 'Robin', last: 'van der Vleuten' }, funcs);
// => 'Robin van der Vleuten'

evaluate('matches(email, "^[^@]+@[^@]+$")', { email: 'robin@example.com' }, funcs);
// => true

The regex lives in your JavaScript, not in the expression string. The rule author only chooses which function to call by name, so adding matches back never reopens the door xprsn closed. Every bit of executable code still ships in your source, where a strict CSP is fine with it.

How it dodges eval entirely

Here's the part I find genuinely neat. xprsn never turns your expression text back into JavaScript. It compiles each expression into a tree of small closures that already exist in the shipped source.

Take price * qty. A naive evaluator would stitch together the string "price * qty" and eval it. xprsn parses it and returns a function that closes over two smaller functions:

// Conceptually, `price * qty` compiles to something like:
const left  = values => get(values, 'price');
const right = values => get(values, 'qty');
const expr  = values => left(values) * right(values);

Every literal becomes a closure that returns a constant. Every property read becomes a closure that reads one key. Operators combine the closures beneath them. Compilation is a precedence-climbing parser walking the tokens once and wiring these functions together. At no point does a string turn into code.

That's what makes it CSP-safe by construction rather than by configuration. There's no 'unsafe-eval' to grant, because nothing ever asks for it. The test suite proves it the hard way: it runs under node --disallow-code-generation-from-strings, which throws on any string-to-code call the same way a strict CSP does, and one test scans the source itself for eval, Function, and friends.

Safe to point at half-trusted input

CSP is only the outer wall. If rules can come from users, the evaluator itself has to refuse the classic escapes.

The famous one is the prototype chain: x.constructor.constructor is Function, which is eval wearing a hat. Every property read in xprsn, whether a.b, a[b], a method lookup, or a bare variable, goes through one guard that rejects __proto__, constructor, and prototype. There is no path from an expression to the Function constructor.

A few more guardrails fall out of the design:

Hash literals are null-prototype objects. So { "__proto__": something } is inert data. It cannot pollute Object.prototype.
in only sees own keys. Inherited properties stay invisible.
There are no assignment operators. An expression can read your data. It can't change it.

One honest limit: expressions can still call methods on the values you expose. Pass an object with a delete() method and data.delete() is a valid expression. The rule is simple: only hand xprsn data you're comfortable letting the expression touch.

When one line isn't enough

Expressions have no variables, on purpose. When a calculation needs intermediate results, you name the steps and feed each answer into the next:

const steps = [
  ['subtotal', 'price * qty'],
  ['discount', 'subtotal >= 100 ? subtotal * 0.1 : 0'],
  ['total',    'subtotal - discount + shipping'],
].map(([name, expr]) => [name, compile(expr)]);

function run(values) {
  const ctx = { ...values };
  for (const [name, fn] of steps) ctx[name] = fn(ctx);
  return ctx;
}

run({ price: 60, qty: 2, shipping: 5 });
// => { price: 60, qty: 2, shipping: 5, subtotal: 120, discount: 12, total: 113 }

Each step compiles once, and the steps are plain data. That's the quiet payoff: you can store a whole calculation in a database or a config file and let non-developers edit it, because none of it is code.

When to reach for it

User-defined filters and segments. Let people describe "orders over €100 from the last week" and store the string they wrote.
Feature-flag and targeting conditions. user.plan == "pro" or user.age > 65, edited in a dashboard instead of a deploy.
Pricing, discount, and scoring rules. The logic marketing changes weekly, kept out of your release cycle.
Low-code and config-driven UIs. Anywhere the shape of the logic lives in data and the client evaluates it under a strict CSP.

If you only need arithmetic, math.js is the more complete tool. If you never touch untrusted strings or a strict CSP, a hand-rolled function might be all you want. xprsn is for the overlap: dynamic expressions, half-trusted input, and headers you refuse to loosen.

How far does the closure trick go?

Once you have an evaluator that compiles to closures and never reaches for eval, you start wondering where the trick runs out. So I pushed on it.

A template is just text with expressions punched into it. But almost every runtime template engine compiles each template to a JavaScript function, which is the same new Function a strict CSP forbids. Handlebars without a precompile step, tempura, most of them hit the same wall xprsn was built to avoid.

So I built sjabloon on top of it. It's Dutch for "template." Same idea, one level up: it parses a template into a tree of closures, and every tag inside runs a real xprsn expression.

import { template, render } from 'sjabloon';

// Compile once, render many times:
const greet = template('Hello {{ user.name.toUpperCase() }}!');
greet({ user: { name: 'Robin' } }); // => 'Hello ROBIN!'

// Blocks, expressions, and custom functions:
render(
  `{{#each items as it, i}}{{ i + 1 }}. {{ it.name }}: {{ fmt(it.price * it.qty) }}
{{/each}}{{#if total >= 100 and "vip" in user.roles}}Free shipping!{{#else}}Shipping: {{ fmt(5) }}{{/if}}`,
  { items: [{ name: 'Koffie', price: 8, qty: 2 }], total: 120, user: { roles: ['vip'] } },
  { fmt: n => '€' + n.toFixed(2) }
);

The tags are what you'd guess: {{ expr }} interpolates and escapes HTML, {{{ expr }}} skips the escaping, {{#if}}/{{#else}} branches, {{#each list as item, i}} loops with an optional index, and {{! comment }} drops out of the output. Every expr is a full xprsn expression, custom functions included, so the same guards carry straight through. The whole engine is about 0.8KB, roughly 2KB with xprsn along for the ride, and it clears the same node --disallow-code-generation-from-strings test.

I want to be honest about when this is the wrong tool. If you can precompile templates at build time, Handlebars is faster and more capable. sjabloon earns its place only when templates arrive at runtime, like user-edited layouts, CMS fragments, or email bodies, and your CSP won't budge.

Getting started

npm install xprsn

The README has the full syntax table and the safety notes. Import compile and evaluate, hand them your expression and your data, and delete the new Function call you were nervous about.

Closing thoughts

I didn't set out to write an expression language. I had a real feature that needed one, reached for the existing options, and none of them fit the exact shape of tiny, extensible, and safe under a strict CSP. So the reusable core came out of the product and into a package small enough to read in one sitting. Small enough, it turned out, to build on.

That's usually the right moment to open-source something. Not when you imagine it might help someone one day, but when it has already earned its keep somewhere real. Have you had to run user-defined logic without handing over the keys to the runtime? I'd like to hear how you approached it.

AbortController is how fetch learned to clean up after itself

Robin van der Vleuten — Tue, 30 Jun 2026 15:05:14 +0000

Fetching data in a React component looks harmless at first.

You render a component, call an API in an effect, store the result in state, and move on with your day. It feels too small to deserve much ceremony.

Then the user navigates away before the request finishes.

Or they type quickly and your component starts fetching search results for r, re, rea, and react at almost the same time.

Or a slow response arrives after a newer one and quietly replaces the data with something stale.

The fetch worked. The component moved on. Those two facts do not always happen in the same order.

That is where AbortController earns its keep. It gives browser APIs like fetch() a cancellation signal, and React's useEffect gives us exactly one place to clean it up.

The fit is small enough that it is easy to miss.

The request that outlives the component

Start with the version most of us write first:

import { useEffect, useState } from 'react'

function UserProfile({ userId }) {
  const [user, setUser] = useState(null)

  useEffect(() => {
    fetch(`/api/users/${userId}`)
      .then((response) => response.json())
      .then((data) => setUser(data))
  }, [userId])

  if (!user) {
    return <p>Loading...</p>
  }

  return <h1>{user.name}</h1>
}

There is nothing exotic here. The component receives a userId, fetches the matching user, and renders the name.

The fast path is fine. The awkward bit starts when the component stops caring before the network does.

Maybe the user clicks back. Maybe userId changes. Maybe the request hangs for a few seconds on a train connection and resolves after a completely different screen has rendered.

React effects already have a cleanup mechanism for exactly this kind of thing. If an effect returns a function, React runs that function before the effect runs again and when the component unmounts.

Most of the time we use that for event listeners or timers. A network request deserves the same treatment.

Give fetch a signal

AbortController has two moving parts:

the controller, which can cancel the work
the signal, which gets passed to the thing doing the work

With fetch, that signal goes in the options object:

useEffect(() => {
  const controller = new AbortController()

  fetch(`/api/users/${userId}`, {
    signal: controller.signal,
  })
    .then((response) => response.json())
    .then((data) => setUser(data))

  return () => {
    controller.abort()
  }
}, [userId])

Now every render that starts a request also knows how to stop it.

When userId changes, React runs the cleanup for the previous effect. When the component unmounts, same thing.

There is one rough edge: aborting a fetch rejects the promise. If we do not handle that rejection, the console gets noisy.

So we catch the abort and ignore that specific error:

useEffect(() => {
  const controller = new AbortController()

  fetch(`/api/users/${userId}`, {
    signal: controller.signal,
  })
    .then((response) => response.json())
    .then((data) => setUser(data))
    .catch((error) => {
      if (error.name === 'AbortError') {
        return
      }

      throw error
    })

  return () => {
    controller.abort()
  }
}, [userId])

The useful shape is the same every time: create the controller inside the effect, pass its signal to fetch, abort it in the cleanup.

The hook is the point

If you only need this once, the effect above is fine. Ship it.

The second copy is where the noise starts to show:

loading state
response state
error state
abort handling
dependency changes
cleanup

That is a lot of plumbing to leave scattered through components.

React hooks are good at this kind of extraction. Every three lines of code do not deserve a custom hook, but repeated lifecycle code has a way of growing little differences in every file.

Let's move the fetch lifecycle into useAbortableFetch:

import { useEffect, useState } from 'react'

function useAbortableFetch(url) {
  const [data, setData] = useState(null)
  const [error, setError] = useState(null)
  const [loading, setLoading] = useState(false)

  useEffect(() => {
    const controller = new AbortController()

    setLoading(true)
    setError(null)

    fetch(url, { signal: controller.signal })
      .then((response) => {
        if (!response.ok) {
          throw new Error(`Request failed with ${response.status}`)
        }

        return response.json()
      })
      .then((json) => {
        setData(json)
      })
      .catch((error) => {
        if (error.name === 'AbortError') {
          return
        }

        setError(error)
      })
      .finally(() => {
        if (!controller.signal.aborted) {
          setLoading(false)
        }
      })

    return () => {
      controller.abort()
    }
  }, [url])

  return { data, error, loading }
}

Then the component gets boring again:

function UserProfile({ userId }) {
  const { data: user, error, loading } = useAbortableFetch(`/api/users/${userId}`)

  if (loading) {
    return <p>Loading...</p>
  }

  if (error) {
    return <p>Could not load the user.</p>
  }

  if (!user) {
    return null
  }

  return <h1>{user.name}</h1>
}

That trade works well.

The hook owns the awkward lifecycle. The component describes the screen.

It also gives the cleanup behavior a name. useAbortableFetch tells the next developer that cancellation is part of the contract. They do not have to inspect a chain of promises to discover whether the request cleans up after itself.

What happens when the URL changes quickly?

The hook gets more useful when the dependency changes often.

Think about search:

function UserSearch({ query }) {
  const { data: users, loading } = useAbortableFetch(
    `/api/users?query=${encodeURIComponent(query)}`
  )

  if (loading) {
    return <p>Searching...</p>
  }

  return (
    <ul>
      {users?.map((user) => (
        <li key={user.id}>{user.name}</li>
      ))}
    </ul>
  )
}

Every time query changes, the old request gets aborted before the new request starts.

That does not make the server do less work in every possible situation. The request may already have reached it.

It does stop the browser from continuing to care about a response the UI no longer needs.

This pairs nicely with debouncing. I wrote separately about debouncing a React input without adding a dependency, and the two ideas solve different halves of the same problem:

debounce before starting work you probably do not need
abort work that has already become irrelevant

They are not replacements for each other. They solve different timing problems.

A small note about stale state

AbortController cancels a request, but it does not magically solve every stale state problem.

If your effect does more work after the fetch, or if some other async function does not support abort signals, you still need to think about what happens when old work finishes late. Sometimes the right answer is a local ignore flag in the effect cleanup. The React docs show that pattern in their useEffect examples for data fetching.

For browser fetches, though, aborting is usually nicer than pretending the result did not arrive. We are telling the underlying operation that we are done with it instead of merely guarding the setState.

The contract is easier to reason about.

Why this belongs in a hook

This is the part I care about more than the API itself.

The browser gives us AbortController. React gives us effects and cleanup. The useful application code is the small hook that makes those two ideas meet in one place.

That is the same reason I like tiny hooks such as usePrevious. A ref and an effect are not complicated by themselves, but the hook turns the pattern into a reusable sentence: "give me the previous value."

This one says: "fetch this, and stop caring when the component does."

That beats copy-pasting another effect and hoping every cleanup branch survives the next refactor.

When I reach for it

I do not wrap every request in a custom hook. Sometimes a framework, router, or data library already owns the lifecycle, cache, retries, and invalidation. In that case, use the tool that is already solving the bigger problem.

But when a component does its own browser fetch, especially in smaller React codebases, AbortController is a good default.

Use it when:

the component can unmount before the request finishes
the request depends on props or state that change often
a search, filter, or tab switch can make old results irrelevant
you want loading and error state to be handled consistently

The hook is small enough to understand and boring enough to trust. That is exactly where I want this kind of code to live.

Running my DSMR parser backwards into a virtual smart meter

Robin van der Vleuten — Fri, 12 Jun 2026 14:58:22 +0000

A while back I wrote about how Erlang's parser tools saved my DSMR library: the story of binning a pile of regex and letting leex and yecc read the telegrams that Dutch and Belgian smart meters spit out every second. Messy text in, tidy Elixir structs out. I was pretty happy with it.

Then I got a little obsessed with running it the other way.

It started, as these things do, while I was writing tests. Every time I wanted to check the parser handled some edge case, I'd go digging for a captured telegram to paste in. And every time, the same nagging thought: I already have a complete description of what a telegram is: the grammar, the field types, the structs. Why am I copy-pasting strings around like it's 2009? Surely I could hand the library a struct and get a byte-perfect telegram back out.

That nagging thought turned into p1meter.dev: a virtual smart meter you can talk to over plain TCP, or just leave open in a tab and watch tick over.

The itch: a parser with nothing to parse

Here's the slightly absurd problem with building a smart meter library: to actually test it, you need a smart meter. Mine lives in a cupboard under the stairs, behind a P1 cable I have to physically go and plug into a laptop. Romantic, it is not.

Other people hit the same problem. Anyone building a Home Assistant integration, a P1 reader, or a parser like mine runs into the same wall: the only realistic test data is locked inside a box bolted to the wall.

A meter only ever emits telegrams. It never reads them, so my library only ever spoke half the protocol. Teach it the other half, struct to string, and suddenly I could generate as much realistic test data as I liked. No cupboard required.

First attempt: a Livebook

The first version wasn't a website. It wasn't really a project at all, just a Livebook. A Mix.install with dsmr and kino, a little GenServer coughing out a telegram every second over a TCP socket, and a VegaLite chart watching the usage curve climb.

And it worked! Which was sort of the problem. A Livebook is a wonderful place to prototype and a terrible place to leave something. You have to clone it, install Elixir, and run it yourself before it's any use to anyone else. This deserved a front door.

Besides, once I had a meter quietly humming away on the BEAM, I couldn't stop thinking about drawing it. Not graphing it. Drawing it. Something you could actually look at.

Running the parser in reverse

The whole thing rests on one small, satisfying trick: the same library that takes telegrams apart can put them back together. Forward is the day job. Backwards is the new party piece.

# Forward: the original dsmr job, text in, struct out.
{:ok, telegram} = DSMR.parse(raw_string)

# Backwards: struct in, text out.
telegram_string = DSMR.Telegram.to_string(telegram)

There's one detail that separates a telegram that's real from one that's merely plausible, and it's the checksum. Every telegram signs off with a ! and a CRC16 over everything before it. Get that wrong and a strict parser throws the whole packet straight in the bin, which is exactly the behaviour you want your fixtures to respect. So the simulator builds the struct, renders it once, runs the checksum over that text, and stamps it back in:

checksum =
  telegram
  |> DSMR.Telegram.to_string()
  |> String.split("!", parts: 2)
  |> List.first()
  |> Kernel.<>("!")
  |> DSMR.CRC16.checksum()

This is dogfooding in the most literal sense I can manage: the simulator's output gets fed straight back through the parser in the test suite. If a round-trip doesn't survive, something's broken in one half or the other. And once or twice, I'll admit, it was the half I'd already declared finished.

Making the numbers move

A meter that reports the same number forever is a deeply boring meter. I wanted the display to behave like an actual household over a day, so a GenServer keeps its own virtual clock and nudges every reading on each tick.

The usage curve is nothing clever, just a cosine wave. Low and sleepy overnight, climbing through the morning, peaking around dinner, with a dusting of random noise so it never looks suspiciously neat:

time_of_day = (hour * 60 + minute) / (24 * 60)
phase = (time_of_day - 0.75) * 2 * :math.pi()
wave = (:math.cos(phase) + 1) / 2
current_power = max(0.0, (0.2 + wave * 2.3) * noise_factor())

On top of that sit the little details that make it feel properly Dutch: a day/night tariff that flips the active register after 23:00, the odd burst of solar being pushed back to the grid on a sunny afternoon, and a gas reading that jumps once an hour rather than creeping up continuously, because that's exactly how the gas meter reports in real life.

None of this is physically rigorous. I'm not modelling anybody's actual fridge. It just has to be convincing enough that whatever you point at it can't tell it's talking to a fake.

One meter, two faces

This is the part where the BEAM quietly shows off.

There's one meter, generating one stream of telegrams, but two completely different crowds want it: people running nc in a terminal, and people watching a LiveView in a browser. Rather than wire the generator up to either of them, it just broadcasts over Phoenix.PubSub and gets on with its life:

Phoenix.PubSub.broadcast(
  LiveMeter.PubSub,
  "smart_meter:telegrams",
  {:smart_meter_telegram, telegram, telegram_string}
)

Everything else subscribes. The TCP server subscribes and fans each telegram out to its sockets. Every open web page subscribes and re-renders. The generator has no idea any of them exist. If I want a third audience tomorrow, I add a subscriber and nothing upstream has to change.

The TCP side runs on the same instinct. Each new connection gets handed its own little GenServer, monitored so a dropped socket tidies up after itself. In most runtimes a process per client would raise an eyebrow. Here it's just how you'd write it.

Pretending to be slow hardware

Here's a detail nobody asked for, which is precisely why it's my favourite.

A real P1 port doesn't hand you a telegram all in one go. It's a serial line, dribbling bytes out at 115200 baud, and firing the whole packet down the socket in a single send felt like cheating. So each line sleeps for roughly as long as the wire would have taken to carry it:

defp sleep({:baud, baud_rate}, line) do
  line
  |> byte_size()
  |> Kernel.*(10_000)      # about 10 bits per byte, in tenths of a millisecond
  |> ceil_div(baud_rate)
  |> max(1)
  |> Process.sleep()
end

It's a tiny, faintly ridiculous touch. But it means a client reading the stream watches the telegram arrive line by line, exactly the way it would off real hardware. That is the kind of thing that flushes out a bug in how you buffer input.

The star: a fake meter in pure LiveView

The TCP stream is the genuinely useful bit. But the part I had no business enjoying as much as I did is the one you can see.

It's a smart meter. A skeuomorphic one: CRT-green display, faint scanlines, little tariff lights, an LED blinking away at "1000 imp/kWh", an SVG barcode on the type plate, even a chunky physical-looking button. All of it rendered server-side in LiveView, updating every second, without a single line of custom JavaScript.

It's built entirely from HEEx function components, and it's fed by the very same PubSub message the nc crowd gets. When one lands, the LiveView turns the struct into display rows and re-renders:

def handle_info({:smart_meter_telegram, telegram, _string}, socket) do
  {:noreply, assign_snapshot(socket, Readings.from_telegram(telegram))}
end

LiveView works out what actually changed and pushes only that down the socket. The numbers climb, the tariff light switches over at night, the little connection dot goes green. The only client-side code anywhere near it is the LiveView runtime everyone already ships.

My favourite piece is the button. A real meter has one button to cycle through its readings, so naturally mine does too. It is just a phx-click bumping an index in the server-side state:

<button type="button" phx-click="cycle_reading"
        aria-label="Cycle through meter readings">
</button>

There is no client state or fetch layer to keep in sync. You press a styled <button>, the server moves to the next reading and sends back the difference. Building hardware-shaped UI in something this plain turned out to be an absurd amount of fun. The skeuomorphism is all CSS and SVG, and the behaviour is all server.

Try it

It's live, and the quickest way in is your terminal:

$ nc p1meter.dev 8080

A fresh DSMR 5.0 telegram every second, the same bytes a real P1 reader would hand you, checksum and all. Or open p1meter.dev and watch the meter tick, with copy-paste client examples for Node, Python and Go sitting right underneath.

It's intentionally small, and it's all open source over on GitHub. I've been collecting ideas for where to take it next: realistic daily patterns, configurable scenarios, the kind of nasty edge cases that exist purely to break parsers.

What's stuck with me, building this, is how little of it is really code I wrote. The parser was already there; I just ran it backwards. PubSub did the fan-out, OTP supervised the clients, LiveView did the live UI. My actual job came down to deciding what a believable meter should do, then getting out of the runtime's way.

So if you're working on anything P1- or DSMR-shaped, point it at the simulator and tell me what breaks. That's half the reason it's out there.

Use previous value through a React hook

Robin van der Vleuten — Thu, 13 Jan 2022 20:05:33 +0000

React Hooks were definitely a game changer when they got introduced in React v16.8.0. It took me some time to familiarize myself with their usage. But now that I have used them a lot, I can see how using them make React components easier to reason about and maintain.
By creating custom hooks that you reuse within components, you can break up complex logic to smaller understandable functions.

After developing many applications, I have gathered a library of React hook patterns used in almost any of these projects.

One of them is retrieving the previous value of the state for example. Written within a handful lines of code, it makes clever use of the useRef() hook. As I found out that not only you can use this hook to store the reference to a HTML element, but that it stores any value you pass it.
When combined with the useEffect() hook, you can create a small helper to store and retrieve the previous value of the variable you passed to it.

Let me show you the usePrevious() hook.

import React from 'react';

export default function usePrevious<T>(value: T): T {
  // Create a reference to hold the previous version of the value,
  // as it is basically a generic object whose `current` property can hold any value.
  const ref = React.useRef<T>();

  // Use the `useEffect` hook to run a callback...
  React.useEffect(() => {
    // ...to store the passed value on the ref's current property...
    ref.current = value;
  }, [value]); // ...whenever the value changes.

  // And return the currently stored value,
  // as this will run before the `useEffect` callback runs.
  return ref.current;
}

What to do with this hook?

When you combine the usePrevious hook with useState, you can run a callback whenever a value differs from the previous render cycle. I mainly use it in applications to compare boolean variables to their previous values, like the following.

import React from 'react';

const Application = () => {
  // Call some external API through a custom hook.
  const { data, loading } = useAsyncApi()
  // Pass its loading indicator to our created hook.
  const wasLoading = usePrevious<boolean>(loading);

  // Use the `useEffect` hook to run a callback...
  React.useEffect(() => {
    // ...to check if the API was loading but now it is completed...
    if (!loading && wasLoading) {
        // ...and run some arbitary code...
    }
  }, [loading, wasLoading]); // ...whenever one of these changes.

  return (
    <div>
      {/* ... */}
    </div>
  );
}

How will you use this hook in your applications? Let me know, as I am curious to find out any other use case you can come up with!

How to use Vim on Linux and macOS

Robin van der Vleuten — Mon, 07 Dec 2020 12:59:02 +0000

If you ever did try to edit files on Linux through the terminal, you'll probably have run into Vim. It is a text-based editor pre-installed on macOS and most Linux distributions.

Very handy, but a bit difficult to grasp for first-time users. Especially when you have opened up a file in the Vim editor by accident. It all bit us developers at least once in our career. As it's apparently one of the most sought-after answers on StackOverflow.

So I've summarized a couple of basic commands that will help you open a file, do some changes and then exit Vim.

How to open a file

First, you need to open a file that you would like to edit. You can use either vim or vi with the name of the file you would like to open. If the file does not exists in the current directory, it will be created for you.

vi file-to-open.txt

How to edit a file

When you have opened up a file, you get into normal (or master) mode. You will be able to read the contents, but you can't make any changes yet. From there, you can execute commands or switch to insert mode. Within this mode, all your keystrokes will result in changes to the text. To go back to the normal mode, you need to press the <ESC> key.

All other Vim commands start with a : character followed by a keyword, which you can only enter when you are in normal mode.

A good overview is provided by the following reference diagram.

Source: https://github.com/nilp0inter/Vim-for-Muggles

How to exit Vim

When you are done making changes, you need to make sure that you are in normal mode by pressing the <ESC> key. And then use the :q command (or one of its siblings) to exit the editor.

:q - just quit when you don't have any changes
:q! - quit and discard any changes you've made
:wq - quit and save all changes to the file

How to capitalize strings in JavaScript

Robin van der Vleuten — Fri, 13 Nov 2020 09:36:35 +0000

To capitalize a string in Javascript so the first character is in uppercase, we don't need to add another NPM dependency. We can use plain JavaScript or even CSS if it is solely for presentational purposes.

TLDR;

const chars = 'hello'
chars[0].toUpperCase() + chars.slice(1) // 'Hello'

Walk through all steps

Let's see how we can approach this through a couple of common JavaScript functions. First, you have to keep in mind that strings are characters. So another way to write a string, is to create an array of characters that we join together.

const chars = ['h', 'e', 'l', 'l', 'o'].join('') // 'hello'

Uppercase the first letter

You can access any character within this array through its index. So if we need the letter e from this array, we can use square brackets to access it at index 1 (as arrays always start counting their index at 0).

But since the introduction of ECMAScript 5, we can treat strings as an array-like object. And thus access any character from a string in a similar fashion.

// We get the first letter by accessing the character at index zero.
const chars = ['h', 'e', 'l', 'l', 'o']
chars[0] // 'h'

// We get the first letter by using the `charAt()` method with index zero.
const chars = 'hello'
chars[0] // 'h'

Now we have the first letter isolated from the rest of the string, we can utilize the String.prototype.toUpperCase() method to convert it to uppercase. This method does not convert the string itself, but returns a new string with all of its characters in uppercase. You can read more about the method at the MDN docs.

const chars = 'hello'
chars[0].toUpperCase() // 'H'

Slice the rest of the letters

Next we need to get the rest of string after the first character. As we gonna uppercase the first character and append the rest as is. To get a portion or a slice of an array, we can use the Array.prototype.slice method. This method gives us a slice between a start and end index. Read more about it at the MDN docs.

We already know that we do not want the first character (at index 0), so our slice starts at 1. Our word has 5 characters and as an array starts at 0, our slice ends at 4.

const chars = ['h', 'e', 'l', 'l', 'o']
chars.slice(1, 4) // ['e', 'l', 'l', 'o']

But this will not work if we do not know our string length upfront. So let's use the Array.prototype.length property to pass the length of our string to the slice method.

const chars = ['h', 'e', 'l', 'l', 'o']
chars.slice(1, chars.length) // ['e', 'l', 'l', 'o']

And as it is common to slice arrays till the end, we can even skip passing the length.

const chars = ['h', 'e', 'l', 'l', 'o']
chars.slice(1) // ['e', 'l', 'l', 'o']

Now to slice a string, we can use String.prototype.slice. Which is identical to the array's slice method. You can read more about it in the MDN docs as well.

const chars = 'hello'
chars.slice(1) // 'ello'

So let's now combine both the first uppercase character and the rest of the string.

const chars = 'hello'
chars.charAt(0).toUpperCase() + chars.slice(1) // 'Hello'

And that will gives us a capitalized string in JavaScript.

Just use CSS

Please remember though, that if its just for displaying a capitalized text on a web page, you can just use a CSS selector.

<style>
    .capitalize {
        text-transform: capitalize;
    }
</style>

<span class="capitalize">hello</span>

How to use React context

Robin van der Vleuten — Fri, 23 Oct 2020 12:47:26 +0000

React Context is a powerful feature introduced in React v16.3.0. It allows you to access values from the current context (e.g. the parent) inside your components' render methods, where you normally would be limited to accessing only its local state and props . This creates a very flexible system for sharing values between components. In this article, we'll cover how to use React Context to create a shared state, and how to consume that state in a child component.

This article assumes you are familiar with the basics of React, such as JSX, components, state, and props . If you're not familiar with these concepts yet, I would recommend reading the official React docs first.

What is React Context?

React Context is a new feature that allows you to share data between components. It's similar in concept to React's useState() hook, but it allows you to share data across multiple components at once. The most common use case for using the context API is sharing state across one or more child components. Let's look at an example:

import React from 'react'

const ParentComponent = () => {
    const [counter, setCounter] = React.useState(0)

    return (
        <ChildComponent
            counter={counter}
            onIncrement={() => setCounter(counter + 1)}
        />
    )
}

const ChildComponent = ({ counter, onIncrement }) => {
    return (
        <div>
            {counter}
            <button onClick={onIncrement}>increment</button>
        </div>
    )
}

Here, we have a <ParentComponent> that sets its own local state with an initial value of 0. The component is rendering a <ChildComponent />, which is receiving the current counter state as prop. It also updates the parent's counter state when the increment button is clicked.

This works great for simple cases, but it can quickly get out of hand if we want to share more state between components. Components start to become more and more coupled, which makes them harder to maintain and reason about. We can do better than this!

Instead of passing the counter prop into ChildComponent, we can use the React Context API to share the counter state between both components. Here's how:

import React from 'react'

const CounterContext = React.createContext(0)

const ParentComponent = () => {
    const [counter, setCounter] = React.useState(0)

    const increment = () => setCounter(counter + 1)

    return (
        <CounterContext.Provider value={{ counter, increment }}>
            <ChildComponent />
        </CounterContext.Provider>
    )
}

const ChildComponent = () => {
    const { counter, increment } = React.useContext(CounterContext)

    return (
        <div>
            {counter}
            <button onClick={() => increment()}>increment</button>
        </div>
    )
}

The ChildComponent is now consuming the counter state and increment action from the context instead of receiving them as props . This allows us to avoid coupling ChildComponent to ParentComponent.

How is this different from `useState()`?

The biggest difference between the context API and useState() is that the context API allows you to share state between multiple components at once. The useState() hook is scoped to a single component only.

Another difference is that the context API can be used to share any type of data, not just state. Like functions for example. We can even use context to share an entire Redux store (but don't do that please).

How does the context API work?

The Context object is a special type of React object that can be used to create and share data. It has two component: Provider and Consumer. The provider is used to create new context objects, while the consumer is used to access values from those contexts.

The context API allows you to provide multiple consumers with a single provider. For example:

import React from 'react'

const CounterContext = React.createContext(0)

const ParentComponent = () => {
    const [counter, setCounter] = React.useState(0)

    const increment = () => setCounter(counter + 1)

    return (
        <CounterContext.Provider value={{ counter, increment }}>
            <DisplayCounterComponent />
            <IncrementCounterComponent />
        </CounterContext.Provider>
    )
}

const DisplayCounterComponent = () => {
    const { increment } = React.useContext(CounterContext)

    return <div>{counter}</div>
}

const IncrementCounterComponent = () => {
    const { increment } = React.useContext(CounterContext)

    return (
        <div>
            <button onClick={() => increment()}>increment</button>
        </div>
    )
}

Here, we are using the CounterContext provider to share the context between multiple components. The DisplayCounterComponent is retrieving the counter state and displays it value, while IncrementCounterComponent is retrieving the increment action from the context instead.

Create a custom context provider

In the previous examples, we were using the CounterContext.Provider directly. But we can create our own CounterProvider component that will make it easier to reuse the context logic.

import React from 'react'

const CounterContext = React.createContext(0)

const CounterProvider = ({ children, initialCount = 0 }) => {
    const [counter, setCounter] = React.useState(initialCount)

    const increment = () => setCounter(counter + 1)

    return (
        <CounterContext.Provider value={{ counter, increment }}>
            {children}
        </CounterContext.Provider>
    )
}

The CounterProvider component receives an initial count from the parent, which it uses to create the context's initial state. It also receives a children prop, which we are passing through so any child component will be rendered.

Now let's see how we can use CounterProvider in our app.

import React from 'react'

const ParentComponent = () => {
    return (
        <CounterProvider initialCount={0}>
            <ChildComponent />
        </CounterProvider>
    )
}

const ChildComponent = () => {
    const { counter, increment } = React.useContext(CounterContext)

    return (
        <div>
            {counter}
            <button onClick={() => increment()}>increment</button>
        </div>
    )
}

The useContext hook is still used to retrieve our counter context within any child component. But we can create a custom useCounterContext hook for this for better reusability.

const useCounterContext = () => React.useContext(CounterContext)

Now we can use useCounterContext instead of useContext inside our components.

const ChildComponent = () => {
    const { counter, increment } = useCounterContext()

    return (
        <div>
            {counter}
            <button onClick={() => increment()}>increment</button>
        </div>
    )
}

Closing thoughts

The context API is a powerful tool for creating shared state between multiple components. It's a great way of avoiding coupling and your components together, which makes them easier to test and maintain.

The React docs have a great article on the context API as well. It's worth taking a look at if you are interested in learning more.

Trap focus in a React component

Robin van der Vleuten — Mon, 12 Oct 2020 14:39:42 +0000

Some actions on a web application requires shifting the focus of an user. We'll use the focus-trap package to trap focus within a DOM node or React component.

While developing interactive applications for the modern web, changes are that you'll need to create overlaying elements. When such elements are active, the rest of the page is usually visually hidden and the user is only allowed to interact with this element. We can use ARIA-attributes to help with assistive technologies, but that won't prevent keyboard users from tabbing out of the active element.

For example, setting aria-modal="true" on an active modal will tell assistive technologies that the elements below the current dialog are not available to interact with¹. This, however, will not prevent the user from tabbing out of your modal component. In this case we’ll have to resort to JavaScript and trap the keyboard focus, and keep the tabbed element in our modal overlay.

Trap focus with vanilla JavaScript

Trapping focus and staying accessible is quite complex and fragile functionality². So we're going to use the well written focus-trap to help implementing an accessible modal component.

Let's say that we have the following HTML markup containing the modal.

<div id="demo">
    <button id="show">show modal</button>

    <div id="modal">
        Modal with <a href="#">with</a> <a href="#">some</a>
        <a href="#">focusable</a> elements.

        <button id="hide">hide modal</button>
    </div>
</div>

With a script attached to the HTML page, we can trap the focus when the modal is shown.

import { createFocusTrap } from 'focus-trap'

const modal = document.getElementById('modal')

const focusTrap = createFocusTrap('#modal', {
    onActivate: function () {
        modal.className = 'trap is-visible'
    },
    onDeactivate: function () {
        modal.className = 'trap'
    },
})

document.getElementById('show').addEventListener('click', function () {
    focusTrap.activate()
})

document.getElementById('hide').addEventListener('click', function () {
    focusTrap.deactivate()
})

The focus-trap package provides many more options. Like deactivating when clicking outside of the component or trigger deactivating when the ESC key is pressed.

Trap focus with React

To achieve the focus trap with React, we can use the focus-trap-react package. It is a thin wrapper around the original package.

Let's convert our vanilla JavaScript example to a React component.

import React from 'react'
import ReactDOM from 'react-dom'
import FocusTrap from 'focus-trap-react'

const Demo = () => {
    const [showModal, setShowModal] = React.useState(false)

    return (
        <div>
            <button onClick={() => setShowModal(true)}>show modal</button>

            <FocusTrap active={showModal}>
                <div id="modal">
                    Modal with <a href="#">with</a> <a href="#">some</a>{' '}
                    <a href="#">focusable</a> elements.
                    <button onClick={() => setShowModal(false)}>
                        hide modal
                    </button>
                </div>
            </FocusTrap>
        </div>
    )
}

ReactDOM.render(<Demo />, document.getElementById('demo'))

You'll notice that we've wrapped the modal with an <FocusTrap /> component. Whenever the trap is active - as indicated by the showModal state - only the children within the trap will get focus.

Closing thoughts

It is quite complex to develop a modal or a dialog component that is accessible to any of your users. Especially when they are visually impaired and depend on the keyboard to navigate through your application. By adding the focus-trap package to your toolbelt, it becomes much easier to create keyboard-navigable components without leaving users behind.

Scroll a React component into view

Robin van der Vleuten — Sat, 04 Jul 2020 10:32:12 +0000

React has an easy way to access DOM APIs of HTML elements through references. We learn how React exposes HTML elements by scrolling an element into view on the click of a button.

People's behavior on websites did not change very much since the early days of the internet. But one thing that did change - since 1994 to be more precise - was that we learned to scroll longer pages of content. We are now used to websites where not all information may be visible at first sight.

But how do we grab a user's attention for something that isn't visible in the current part of the viewport it's currently looking at. We can utilize a very handy browser API for that, called Element.scrollIntoView(). Which does exactly what it says it does with a few nice options to modify its behavior.

Scroll to element with plain HTML

Before diving into the React implementation, let's try out the API on a simple HTML list with vanilla Javascript.

Let's say we have an article with a long text.

<article>
    <h1 id="title">
        An interesting article for Latin readers
    </h1>
    <p>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam sit amet luctus neque. Etiam eu quam lacinia, placerat sem ut, interdum risus. Quisque ut feugiat mauris. Aenean euismod fermentum facilisis. Donec ultricies maximus elit, sit amet convallis urna rhoncus vitae. Aliquam bibendum turpis et felis blandit commodo. Donec egestas neque non elit laoreet, faucibus tempor ante gravida.
    </p>
    <p>
        Duis in ante turpis. Phasellus dignissim tellus et nunc lacinia elementum. Sed venenatis tincidunt justo. Praesent sed purus facilisis, porttitor ligula in, mattis velit. Curabitur non pellentesque nunc. Duis elit urna, bibendum et purus nec, maximus imperdiet mauris. Cras euismod, leo id vehicula vulputate, nibh massa tincidunt justo, sit amet fringilla quam orci pellentesque enim.
    </p>
    <p>
        ...
    </p>
</article>

Whenever a user reached the end of the article, we would like to provide a button to scroll back to the top of the article. This can be achieved by adding a link with the id of the <h1> heading element on the end of the paragraph.

<article>
    ...
    <a href="#title">
        Back to the top
    </a>
</article>

Now when the user clicks the link, the browser will automatically jump back to the title element and the user is back on the top of the article. This is the basic way to scroll an element into view without using any Javascript at all.

Scroll to element with vanilla Javascript

To scroll to the element with Javascript, you can create a button which scrolls back the the top when a user clicks it.

<article>
    ...
    <button onclick="document.getElementById('title').scrollIntoView()">
        Back to the top
    </button>
</article>

By using an event listener on the button, whenever it is invoked we get the heading element by its title identifier and tell it to scroll into the browser's viewport.

For most use cases, this is sufficient. But sometimes you'll would like to have a nice animation while scrolling. Luckily you can pass additional options to the method to do exactly that.

const titleElement document.getElementById('title')
titleElement.scrollIntoView({ behavior: 'smooth' })

By setting the behavior option to smooth, the browser will gently scroll to the element instead of the instant jump.

Scroll to a React element

Now the next step is to figure out how we can apply this smooth scrolling behavior on a React component. We can still use the Element.scrollIntoView() method, but we need to grab the component's underlaying HTML element to access it.

First, let's convert our example to a React functional component.

import React from 'react'

const Article = () => {
  return (
      <article>
            <h1>
                A React article for Latin readers
            </h1>
            <p>
                ...
            </p>
            <p>
                ...
            </p>
            <button>
                Back to the top
            </button>
        </article>
    )
}

We could still give the <h1> element an id attribute. But to do it the React way, we'll give a reference instead with the useRef hook. You can read more about the useRef() hook in the official React documentation.

import React, { useRef } from 'react'

const Article = () => {
  const titleRef = useRef()

  return (
      <article>
            <h1 ref={titleRef}>
                A React article for Latin readers
            </h1>

            // Rest of the article's content...

            <button>
                Back to the top
            </button>
        </article>
    )
}

Now we need to handle the user clicking the button to scroll back to the top. We can use an onClick event handler for that. You can read more about event handling in the official React documentation.

import React, { useRef } from 'react'

const Article = () => {
  const titleRef = useRef()

  function handleBackClick() {
    // Scroll back to the title element...
  }

  return (
      <article>
            <h1 ref={titleRef}>
                A React article for Latin readers
            </h1>

            // Rest of the article's content...

            <button onClick={handleBackClick}>
                Back to the top
            </button>
        </article>
    )
}

Within the event handler, we now have access to the title element through its reference. And we can scroll to the title element like we did in the vanilla Javascript example.

const titleRef = useRef()

function handleBackClick() {
  titleRef.current.scrollIntoView({ behavior: 'smooth' })
}

By using useRef() in a React component, we have an entry to the underlaying HTML element. This gives us full access to all of the powerful DOM APIs.

Scroll to a React component

Now that we have seen how we can scroll to an element by using a reference. We can utilize that same method to scroll to a React component. By forwarding the reference to the root element of the component, we again have access to the HTML element from outside the component.

import React, { forwardRef, useRef } from 'react'

const Article = forwardRef(({ onBackClick }, ref) => {
  return (
      <article>
            <h1 ref={ref}>
                A React article for Latin readers
            </h1>

            // Rest of the article's content...

            <button onClick={onBackClick}>
                Back to the top
            </button>
        </article>
    )
})

// ...

Const AnotherComponent = () => {
    const articleRef = useRef()

    function handleBackClick() {
      articleRef.current.scrollIntoView({ behavior: 'smooth' })
    }

    return <Article ref={articleRef} onBackClick={handleBackClick} />
}

As you may see in the example, we've used the forwardRef() method, to allow other components to access HTML elements within our Article component by reference. You can read more about the forwardRef() method in the official React documentation.

Bonus: scroll to the first error in a Formik form

To apply what we've learned to a real-world use case. Let's imagine we have a large React form using the Formik library to handle submission and validation. For example the following newsletter signup form.

import React from 'react'
import { Formik } from 'formik'

const SignupForm = () => {
  return (
      <Formik
        initialValues={{ email: '' }}
        validate={values => {
        const errors = {}

        if (!values.email) {
              errors.email = 'Required'
            }

            return errors
      }}
      onSubmit={values => {
        // ...
      }}
      >
          {formik => (
            <form onSubmit={formik.handleSubmit}>
              <label htmlFor="email">Email Address</label>
              <input
                id="email"
                name="email"
                type="email"
                onChange={formik.handleChange}
                value={formik.values.email}
              />
              {formik.errors.email ? <div>{formik.errors.email}</div> : null}
              <button type="submit">Submit</button>
            </form>
      )}
    </Formik>
  )
}

When a user tries to submit the form, it will display an error saying that the email field is required. In this small form the user will notice this immediately. But when the form grows larger, it would be nice to scroll the error into the viewport so the user notices the error.

We can do this by creating a small helper component that we add to the form.

import React, { useEffect } from 'react'
import { useFormikContext } from 'formik'

const ErrorFocus = () => {
  // Get the context for the Formik form this component is rendered into.
  const { isSubmitting, isValidating, errors } = useFormikContext()

  useEffect(() => {
    // Get all keys of the error messages.
    const keys = Object.keys(errors)

    // Whenever there are errors and the form is submitting but finished validating.
    if (keys.length > 0 && isSubmitting && !isValidating) {
      // We grab the first input element that error by its name.
      const errorElement = document.querySelector(`input[name="${keys[0]}"]`)

      if (errorElement) {
          // When there is an input, scroll this input into view.
        errorElement.scrollIntoView({ behavior: "smooth" })
      }
    }
  }, [isSubmitting, isValidating, errors])

  // This component does not render anything by itself.
  return null
}

Now add this <ErrorFocus> component to our Formik form and the user is automatically scrolled to the first input that has a validation error.

import React from 'react'
import { Formik } from 'formik'
import ErrorFocus from './error-focus'

const SignupForm = () => {
  return (
      <Formik
        initialValues={{ email: '' }}
        validate={values => {
        const errors = {}

        if (!values.email) {
              errors.email = 'Required'
            }

            return errors
      }}
      onSubmit={values => {
        // ...
      }}
      >
          {formik => (
            <form onSubmit={formik.handleSubmit}>
              <label htmlFor="email">Email Address</label>
              <input
                id="email"
                name="email"
                type="email"
                onChange={formik.handleChange}
                value={formik.values.email}
              />
              {formik.errors.email ? <div>{formik.errors.email}</div> : null}
              <button type="submit">Submit</button>

              {/* The component itself does not render anything, but needs to be within the Formik context */}
              <ErrorFocus />
            </form>
      )}
    </Formik>
  )
}

Closing thoughts

By using useRef() and forwardRef() in your React applications, you will have a lot of powerful DOM APIs at your disposal. In this article we've only focussed on Element.scrollIntoView(), but there are many more cool and handy methods you can use.
Did you know that you can even animate elements through Javascript? The MDN web documentation will tell you more about this Element.animate() method.

Going epic with Redux Observable tests

Robin van der Vleuten — Tue, 09 Jun 2020 10:02:26 +0000

The last couple of React projects I've developed where built with a lot of help from the Redux Observable library. It is an excellent library to separate your business logic from your components, but the correct way to test is still something they need to find out. In this article, I am gonna share my ideas on this topic.

So what is this Redux Observable?

For those who aren't aware with any of this library, I recommend to you to check out the RxJS + Redux + React = Amazing! talk by Jay Phelps. It's a very inspiring talk on how Netflix uses some common JS patterns combined with the powers of RxJS to manage your business logic within your React application. They've extracted the core from Netflix and shared it as an open-source library on Github.

Their documentation is excellent and contains a lot of small running examples to help get you started. The whole library deserves an article on its own, but one important aspect is still a bit underexposed. As a matter of fact, they are still struggling with the best way™ themselves;

Testing async code that creates side effects isn't easy. We're still learning the best way to test Epics. If you have found the perfect way, do share!
Redux Observable docs

After struggling with the Observable tests on a couple of projects, I would like to give my two cents on the topic in this article.

What epic are we going to test?

To get a nice epic to show how you can test asynchronous business logic, I came up with the following;

export const authenticateUserEpic = (action$, store, { client }) => {
  // Only act on actions of a given type,
  // in this case "USER_AUTHENTICATE_REQUEST".
  return action$.ofType('USER_AUTHENTICATE_REQUEST')
    // Map the received action to a new action "observable".
    .switchMap(action => {
      // Authenticate with the dispatched credentials in the action,
      // using the injected client instance.
      return client.authenticate(action.username, action.password)
        .then(response => {
          if (!response.isSuccessful) {
            // Map the response to a "failed" action with the error.
            return {
              type: 'USER_AUTHENTICATE_FAILURE',
              error: 'Something went wrong while authenticating',
            };
          }

          return {
            // Map the response to a "successful" action with a JWT token.
            type: 'USER_AUTHENTICATE_SUCCESS',
            idToken: response.idToken,
          };
        });
    });
}

As you may have noticed is this epic about authenticating an user with the dispatched credentials. I can imagine I would dispatch such action like this;

export const authenticate = (username, password) {
  return { type: 'USER_AUTHENTICATE_REQUEST', username, password };
}

dispatch(authenticate('johndoe', 'mysupersecretpassword'));

You may also have noticed that I've injected the client dependency into my epic. You could get a client instance through a require or import statement. But by using dependency injection it makes the client way easier to mock and your epic way easier to test.

Creating the tests with Jest

Most of the React projects out there seem to be using Jest, so I'll just use it in the example test.

My approach to test the above epic, is to get the expected action when the epic receives the dispatched action. So a quick glanse on the epic tells us that we need two tests; one where we expect USER_AUTHENTICATE_SUCCESS with a JWT token and one where we expect USER_AUTHENTICATE_FAILURE with an error.
To define them as Jest tests, one would define them as follows;

describe('authenticateUserEpic', () => {
  it('should dispatch a JWT token when authenticating is successful', () => {
    // ...
  })

  it('should dispatch an error when authenticating has failed', () => {
    // ...
  })
});

So let us focus on the first test for now. We need pass the epic the dispatching action and get the resulting action when the RxJS Observable completes.
There are many ways to write such code, but the following works the best for me;

import { ActionsObservable } from 'redux-observable';
import authenticateUserEpic from './epics';

// ...

it('should dispatch a JWT token when authenticating is successful', async () => {
  // The response object we expect to receive from the server.
  const response = {
    isSuccessful: true,
    idToken: 'a-random-generated-jwt',
  };

  // Create a fake client instance which will return
  const client = { authenticate: jest.fn() };
  client.authenticate.mockReturnValue(Promise.resolve(response));

  // Create an Observable stream of the dispatching action.
  const action$ = ActonsObservable.of({
    type: 'USER_AUTHENTICATE_REQUEST',
    username: 'johndoe',
    password: 'mysupersecretpassword',
  });

  // Pass the Observable action to our action and inject the
  // mocked client instance.
  const epic$ = authenticateUserEpic(action$, store, { client });

  // Get the resulting actions by using async/await.
  const result = await epic$.toArray().toPromise();

  // Test if we've received the expected action as result.
  expect(result).toEqual([
    { type: 'USER_AUTHENTICATE_SUCCESS', idToken: 'a-random-generated-jwt' }
  ])
});

Not that hard right? You'll need to understand RxJS first. But after that, you will get a nice separation of concerns in your React applications.
To make the examples complete, the following test will handle the failed response;

it('should dispatch an error when authenticating has failed', async () => {
  // The response object we expect to receive from the server.
  const response = {
    isSuccessful: false,
  };

  // Create a fake client instance which will return
  const client = { authenticate: jest.fn() };
  client.authenticate.mockReturnValue(Promise.resolve(response));

  // Create an Observable stream of the dispatching action.
  const action$ = ActonsObservable.of({
    type: 'USER_AUTHENTICATE_REQUEST',
    username: 'johndoe',
    password: 'mysupersecretpassword',
  });

  // Pass the Observable action to our action and inject the
  // mocked client instance.
  const epic$ = authenticateUserEpic(action$, store, { client });

  // Get the resulting actions by using async/await.
  const result = await epic$.toArray().toPromise();

  // Test if we've received the expected action as result.
  expect(result).toEqual([
    { type: 'USER_AUTHENTICATE_FAILURE', error: 'Something went wrong while authenticating' }
  ])
});

Did I got some headaches along the way? I definitely got some questions before I had a basic understanding of RxJS! But luckily the Redux Observable community was very helpful. And now I have a very valuable new tool to structure my React applications around 👌

DEV Community: Robin van der Vleuten

A JSONPath engine that follows the spec and can't run code

The standard nobody had

The filter that runs a shell

A scout that stays on the path

The parts of the spec worth carrying around

The part I tried to borrow

Why it can't be turned into code

Is it the right tool?

Getting started

Closing thoughts

Let users write rules without letting them write JavaScript

Where eval stops being an option

Meet xprsn

The syntax fits on a postcard

How it dodges eval entirely

Safe to point at half-trusted input

When one line isn't enough

When to reach for it

How far does the closure trick go?

Getting started

Closing thoughts

AbortController is how fetch learned to clean up after itself

The request that outlives the component

Give fetch a signal

The hook is the point

What happens when the URL changes quickly?

A small note about stale state

Why this belongs in a hook

When I reach for it

Running my DSMR parser backwards into a virtual smart meter

The itch: a parser with nothing to parse

First attempt: a Livebook

Running the parser in reverse

Making the numbers move

One meter, two faces

Pretending to be slow hardware

The star: a fake meter in pure LiveView

Try it

Use previous value through a React hook

What to do with this hook?

How to use Vim on Linux and macOS

How to open a file

How to edit a file

How to exit Vim

How to capitalize strings in JavaScript

TLDR;

Walk through all steps

Uppercase the first letter

Slice the rest of the letters

Just use CSS

How to use React context

What is React Context?

How is this different from useState()?

How does the context API work?

Create a custom context provider

Closing thoughts

Trap focus in a React component

Trap focus with vanilla JavaScript

Trap focus with React

Closing thoughts

Scroll a React component into view

Scroll to element with plain HTML

Scroll to element with vanilla Javascript

Scroll to a React element

Scroll to a React component

Bonus: scroll to the first error in a Formik form

Closing thoughts

Going epic with Redux Observable tests

So what is this Redux Observable?

What epic are we going to test?

Creating the tests with Jest

How is this different from `useState()`?