DEV Community: jamestthompson3

Neovim Tip, Smarter Path

jamestthompson3 — Thu, 11 Nov 2021 00:00:00 +0000

Intro

Fuzzy finders can be a good tool for navigating files in a codebase, but they don't give you the same functionality as a properly set path option in vim. Key mappings like gf, [i, [d, and commands like :find depend on having your path set correctly to find the file you're searching for. Using the :find command also gives us the ability to tab complete a file name, a feature which fuzzy finders do not offer.

An poorly set path can cause a significant slow down when trying to tab complete a filename with :find, or jump to a file with gf, especially if you work with languages like javascript where projects contain the fathomless void known as node_modules.

In his excellent gist, romainl illustrates how a well set path can make navigating a codebase quick and easy, capturing only the files relevant to our work in a project. However, in his gist he sets the path manually, what I want to do is automatically set the path on a per-project basis. My answer to this, is to use fd.

The Code

I added the following code to my init.lua, but it can work in vimscript and in regular vim as well! This requires that you have the fd binary installed on your machine.

vim.o.path = table.concat(vim.fn.systemlist("fd . --type d"),",")

Since fd will respect your .gitignore file, the above code will set every directory currently tracked by git in your path, ensuring that you can access your project files without having to deal with slowdowns caused by vim looking inside of directories such as node_modules.

To speed things up when you open a directory in a path that is not currently tracked by git, you
can use the following snippet:

local setPath = function()
  if gitBranch() ~= "" then
    return ".," .. table.concat(vim.fn.systemlist("fd . --type d --hidden -E .git -E .yarn"),","):gsub("%./", "") .. "," .. table.concat(vim.fn.systemlist("fd --type f --max-depth 1"), ","):gsub("%./", "") -- grab both the dirs and the top level filesystem
  else
    return vim.o.path
  end
end



vim.o.path = setPath()

(Nvim) Lua for Javascripters: Module Exports

jamestthompson3 — Wed, 31 Mar 2021 00:00:00 +0000

Goals

This aims to be quick reference guide on lua module exports compared to JavaScript module exports.

Task

We want to export a series of module scoped functions in both JavaScript and Lua. This guide also show side by side comparisons on writing a file to a specific directory and deleting a file from the same directory.

In this quick tip, we will use CommonJS as the format for the JavaScript modules since it more closely resembles Lua modules in its construction.

To export modules in NodeJS we would write:

// filesystemUtils.js

const util = require("util")

const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);
const readDir = util.promisify(fs.readdir);
const deleteFile = util.promisify(fs.unlink);

function writeDataFile(name, data) {
  const dataDir = getDataDir();
  const filePath = `${dataDir}${name}.json`;
  return writeFile(filePath, JSON.stringify(data));
}

function deleteDataFile(name) {
  const dataDir = getDataDir();
  const filePath = `${dataDir}${name}.json`;
  return deleteFile(filePath);
}

module.exports = {
  writeDataFile,
  deleteDataFile
}

Where writeDataFile and deleteDataFile are the function being exported. We could them access them by requiring the filesystemUtils module:

const fs_utils = require("./filesystemUtils")

fs_utils.writeDataFile("test")
fs_utils.deleteDataFile("test")

In Lua, the syntax is quite similar:

-- filesystemUtils.lua

local M = {} -- initialize an empty table (or object in JS terms)

-- re-assign functions for convienience
local uv = vim.loop

-- assign the function to a key of the same name in the table
function M.deleteDataFile(name)
  local dataDir = getDataDir()
  local filePath = dataDir .. name .. ".json"
  return uv.fs_unlink(filePath)
end

function M.writeDataFile(name, data)
  local dataDir = getDataDir()
  local filePath = dataDir .. name .. ".json"
  return uv.fs_open(filePath, function (err, fd)
      if (err) then return end
      uv.fs_write(fd, data, function(err, bytes_written)
        if (err) then return end
        uv.fs_close(fd, function(err)
          if (err) then return end
        end)
      end)
  end)
end

return M -- This line exports the table

We can then use these functions in other code:

local fs_utils = require("filesystemUtils")

fs_utils.writeDataFile("test")
fs_utils.deleteDataFile("test")

Using modules in lua helps keep the global scope clean and makes it easier to refactor and reorganize code.

(Nvim) Lua for Javascripters: Spawning Processes

jamestthompson3 — Tue, 16 Mar 2021 00:00:00 +0000

Goals

This should be a quick reference guide for those familiar with NodeJS on how to execute the same async tasks in Lua using luv. This is aimed towards use cases inside Neovim, but is not limited to those cases.

Task

We want to spawn a child task to convert a markdown document into HTML using pandoc. This could be used as part of a publishing flow for a blog, for writing notes, or for implementing a markdown previewer.

// in Node we could do it like this:
const {spawn} = require('child_process');
const [sourceFile, destinationFile] = process.argv.slice(2)

const convert = spawn('pandoc', [sourceFile, '--from', 'gfm', '--to', 'html5', '-o', destinationFile, '-s', '--highlight-style', 'tango'])

convert.stderr.on('data', (data) => {
  console.error(data);
});

convert.on('close', (code) => {
  console.log(`child process exited with code ${code}`);
});

In Lua, the code looks very similar, but a bit more verbose:

-- The same as before, but this time we want to generate the file names 
-- based on the file we are currently editing instead of passing them as commandline args

function convert()
  -- cut off the `.md` part of the file
  local destinationFile = vim.fn.expand('%:t:r')
  -- the name of the file you're editing
  local sourceFile = api.nvim_buf_get_name(0)
  spawn('pandoc', {
    args = {sourceFile, '--from', 'gfm', '--to', 'html5', '-o', string.format('%s.html', destinationFile), '-s', '--highlight-style', 'tango'},
  }, 
  {stdout = function()end, stderr = function(data) print(data) end},
  function(code) -- we want to call this function when the process is done
    print('child process exited with code ' .. string.format('%d', code))
  end)
end

function spawn(cmd, opts, input, onexit)
  local handle, pid
  -- open an new pipe for stdout
  local stdout = vim.loop.new_pipe(false)
  -- open an new pipe for stderr
  local stderr = vim.loop.new_pipe(false)
  handle, pid = vim.loop.spawn(cmd, vim.tbl_extend("force", opts, {stdio = {stdout; stderr;}}), 
  function(code, signal)
    -- call the exit callback with the code and signal
    onexit(code, signal)
    -- stop reading data to stdout
    vim.loop.read_stop(stdout)
    -- stop reading data to stderr
    vim.loop.read_stop(stderr)
    -- safely shutdown child process
    safe_close(handle)
    -- safely shutdown stdout pipe
    safe_close(stdout)
    -- safely shutdown stderr pipe
    safe_close(stderr)
  end)
  -- read child process output to stdout
  vim.loop.read_start(stdout, input.stdout)
  -- read child process output to stderr
  vim.loop.read_start(stderr, input.stderr)
end

function safe_close(handle)
  if not vim.loop.is_closing(handle) then
    vim.loop.close(handle)
  end
end

One of the major differences is that in Lua you are responsible for cleaning up both the process handle and any pipes you have open to receive data from that handle. It then becomes useful to create a more generalized spawn function to handle all of these things under the hood, allowing you to just call spawn in a similar manner to the NodeJS API.

Peer Discovery over UDP

jamestthompson3 — Sun, 12 Jul 2020 00:00:00 +0000

What is Peer discovery

Peer discovery allows you to discover other computers on the same subnet, intranet, or through the internet. Discovering peers directly removes the necessity of a centralized server architecture, reducing the number of network jumps your packets require to share information with each other. Peer discovery can be used in: discovering microservices in the same docker network or kubernetes cluster, file sharing (like airdrop and bittorrent), and peer to peer gaming. Eliminating a centralized communication server can reduce operating costs, improve communication times between clients, and lead to more robust services since there is no single point of failure. Taking advantage of the benefits listed above requires a decentralized architecture.

Multicast Groups

Multicasting is one of the tools we can use in creating a decentralized system. Multicasting is the process where messages are sent to a group of participants on the network. Multicasting differs from Broadcasting by only sending data to a specified group of network nodes, whereas broadcasting sends data to all network nodes. Implementing multicasting incurs a distinct set of challenges compared to centralized architectures; consider the following listed by Microsoft’s article about IPV4 Multicasting:

Multicast traffic is sent to an ambiguous group destination.
Because group addresses represent different groups with different members, group addresses generally cannot be summarized in the IP multicast forwarding table.
The location of group members is not consistent, so the IP multicast forwarding table might need to be updated whenever a group member joins or leaves a multicast group. Multicast routing protocols update the IP multicast forwarding table.

Because the challenges like those listed above, reasoning about multicast traffic requires different mental model than a traditional client-server architecture. A critical concept in multicasting the multicast group. A multicast group can be compared to a chat application: membership is dynamic; members can leave and join at will, group members can be located anywhere on a multicast enabled network (compared to a server with a static IP address), a host can be a member of as many multicast groups as desired. A multicast group can have a well known address, for example 224.0.0.1 is the multicast address for all hosts in the subnet.

Members of multicast groups listening for incoming traffic will first bind their UDP socket to an available interface and join the multicast group. After joining the group, this member can receive datagram packets on the bound interface without the other members of the group knowing it’s specific IP address. A similar process goes for multicast group members sending data to the group. Senders will bind their UDP socket on an available interface and begin transmitting datagram packets to the multicast group address. Through the magic of multicasting, the sender does not require information other than the group address for their packets to reach group members who are listening for incoming data.

Diving into the code

To start multicasting over UDP requires only a few lines of code. For this post, we’ll create a small program which sends a username to members of a multicast group. First, we want to set up a listener for other peers sending data to the multicast group. To do this, we need to bind the UDP socket to an available interface and join the multicast group:

use std::net::{Ipv4Addr, SocketAddrV4, UdpSocket};

static MULTI_CAST_ADDR: Ipv4Addr = Ipv4Addr::new(224, 0, 0, 1);

pub fn listen() {
  let socket_address: SocketAddrV4 = SocketAddrV4::new(Ipv4Addr::new(0, 0, 0, 0), 9778);
  let bind_addr = Ipv4Addr::new(0, 0, 0, 0);
  let socket = UdpSocket::bind(socket_address)?;
   println!("Listening on: {}", socket.local_addr().unwrap());
   socket.join_multicast_v4(&MULTI_CAST_ADDR, &bind_addr)?;
}

Notice we create a new IP address struct with the values, 0, 0, 0, 0, which the equivalent of saying “Any available IP interface”.

import dgram from "dgram";

const MULTI_CAST_ADDR = "224.0.0.1";

function listen() {
  const server = dgram.createSocket("udp4");
  server.bind(9778, () => {
    server.addMembership(MULTI_CAST_ADDR);
  });
  server.on("listening", () => {
    const address = server.address();
    console.log(`Listening on: ${address.address}:${address.port}`);
  });
}

MULTI_CAST_ADDR points an IP address, 224.0.0.1, as stated earlier, this is the reserved multicast address for all systems on the current subnet. Since the code is listening for messages being sent to the multicast group, we need to join the group in addition to binding the socket on an available IP Interface. In contrast to a server listening to incoming HTTP connections, we not only bind our server to a local IP address and a port, but we also join a multicast group whose address is part of the subnet. Since we are binding the UDP server to a local address and port and joining the multicast group, it can receive data from a direct connection ( like HTTP ), and from the multicast group.

Now time for the logic for receiving the multicast group data. Luckily, whether the data coming from the multicast group or from a direct connection, the code the same.

pub fn listen() -> Result<()> {
    let socket_address: SocketAddrV4 = SocketAddrV4::new(Ipv4Addr::new(0, 0, 0, 0), 9778);
    let bind_addr = Ipv4Addr::new(0, 0, 0, 0);
    let socket = UdpSocket::bind(socket_address)?;
    println!("Listening on: {}", socket.local_addr().unwrap());
   socket.join_multicast_v4(&MULTI_CAST_ADDR, &bind_addr)?;
    loop {
        // set up message buffer with size of 120 bytes
        let mut buf = [0; 120];
        let (data, origin) = socket.recv_from(&mut buf)?;
        let buf = &mut buf[..data];
        let message = String::from_utf8(buf.to_vec()).unwrap();
        println!("server got: {} from {}", message, origin);
    }
}

function listen() {
  const server = dgram.createSocket("udp4");
  // Listen for incoming messages
  server.on("message", (msg, rinfo) => {
    console.log(`server got: ${msg} from ${rinfo.address}:${rinfo.port}`);
  });
  server.bind(9778, (a) => {
    server.addMembership(MULTI_CAST_ADDR);
  });
  server.on("listening", () => {
    const address = server.address();
    console.log(`Listening on: ${address.address}:${address.port}`);
  });
}

After setting up logic for listening to incoming messages on the multicast group address, our basic server is done! Now we can create the function that will send packets to the multicast address:

use std::net::{Ipv4Addr, SocketAddrV4, UdpSocket};

static MULTI_CAST_ADDR: Ipv4Addr = Ipv4Addr::new(224, 0, 0, 1);

pub fn cast() -> Result<()> {
    let socket_address: SocketAddrV4 = SocketAddrV4::new(Ipv4Addr::new(0, 0, 0, 0), 0);
    let socket = UdpSocket::bind(socket_address)?;
    socket.connect(SocketAddrV4::new(MULTI_CAST_ADDR, 9778))?;
    // Don't send messages to yourself.
    // In this case self discovery is for human developers, not machines.
    socket.set_multicast_loop_v4(false)?;
    let data = String::from("{\"username\": \"test\"}")
     loop {
        socket.send(data.as_bytes())?;
        thread::sleep(time::Duration::from_secs(2));
    }
    Ok(())

import dgram from "dgram";

const MULTI_CAST_ADDR = "224.0.0.1";

function cast() {
  const client = dgram.createSocket("udp4");
  setInterval(() => {
    const message = Buffer.from(JSON.stringify({ username: "hackerman1337" }));
    client.send(message, 9778, MULTI_CAST_ADDR);
  }, 2000);
}

Unlike the listen function, when we are sending data to the multicast address, we don’t need to join the multicast group. Since we are using UDP for peer discovery, we can fire and forget these messages from the cast function as there will be no response from the server.

To test our peer discovery functions, you need two computers connected to the same subnet, or two docker containers running in the same docker network, or a docker container and your computer. Note that while you don’t need to expose docker ports in order for the program running on your computer to discover the program running in the docker container, you will need to expose ports in order for your container to discover the host machine. We also need to combine our two functions so that we are both broadcasting our presence and listening for peers.

use std::thread;

fn main() {
    thread::spawn(||{
        listen();
    });
    cast();
}

import cluster from "cluster";

function main() {
  if (cluster.isMaster) {
    cluster.fork();
    listen();
  } else if (cluster.isWorker) {
    cast();
  }
}

main();

That’s it! If you run the program on two different computers on the same subnet, or two docker containers in the same docker network, you can observe the peers are able to discover each other’s username and IP Address. The final code output:

use std::net::{Ipv4Addr, SocketAddrV4, UdpSocket};
use std::thread;

static MULTI_CAST_ADDR: Ipv4Addr = Ipv4Addr::new(224, 0, 0, 1);

pub fn listen() {
    let socket_address: SocketAddrV4 = SocketAddrV4::new(Ipv4Addr::new(0, 0, 0, 0), 9778);
  let bind_addr = Ipv4Addr::new(0, 0, 0, 0);
  let socket = UdpSocket::bind(socket_address)?;
  println!("Listening on: {}", socket.local_addr().unwrap());
  socket.join_multicast_v4(&MULTI_CAST_ADDR, &bind_addr)?;
}

pub fn cast() -> Result<()> {
  let socket_address: SocketAddrV4 = SocketAddrV4::new(Ipv4Addr::new(0, 0, 0, 0), 0);
  let socket = UdpSocket::bind(socket_address)?;
  socket.connect(SocketAddrV4::new(MULTI_CAST_ADDR, 9778))?;
  // Don't send messages to yourself.
  // In this case self discovery is for human developers, not machines.
  socket.set_multicast_loop_v4(false)?;
  let data = String::from("{\"username\": \"test\"}")
   loop {
    socket.send(data.as_bytes())?;
    thread::sleep(time::Duration::from_secs(2));
  }
  Ok(())
}

fn main() {
    thread::spawn(||{
        listen();
    });
    cast();
}

import dgram from "dgram";
import cluster from "cluster";

const MULTI_CAST_ADDR = "224.0.0.1";

function listen() {
  const server = dgram.createSocket("udp4");
  server.on("message", (msg, rinfo) => {
    console.log(`server got: ${msg} from ${rinfo.address}:${rinfo.port}`);
  });
  server.bind(9778, (a) => {
    server.addMembership(MULTI_CAST_ADDR);
  });
  server.on("listening", () => {
    const address = server.address();
    console.log(`Listening on: ${address.address}:${address.port}`);
  });
}

function cast() {
  const client = dgram.createSocket("udp4");
  setInterval(() => {
    const message = Buffer.from("TEST");
    client.send(message, 9778, MULTI_CAST_ADDR);
  }, 2000);
}

function main() {
  if (cluster.isMaster) {
    cluster.fork();
    listen();
  } else if (cluster.isWorker) {
    cast();
  }
}

main();

Javascript Quick Tip -- Browser Notifications

jamestthompson3 — Sat, 16 May 2020 00:00:00 +0000

Let ‘Em Know

While notifications are one of the browser features that are often abused and lead to obnoxious spam, there are still use cases where notifications enhance the experience of your web application. Modern browsers (with the exception of iOS Safari), support two types of notifications: push, and web. This post discusses only web notifications, since they do not require registration on a server, or the use of a service worker (although they can be used with both). After a short intro on how to set up web notifications, we’ll dive into an example where web notifications are a useful addition to your web app.

Getting Started

To start using web notifications you must request permissions from the user. Important note: you cannot request notification permissions over insecure connections, which means you must be serving your application over HTTPS or from your localhost. Once you are serving you app over HTTPS or from your localhost, requesting permissions is straight forward:

<h1>Permission Spam</h1>
<p>Your current notification status is: <span id="permStatus"></span></p>

<script>
const status = document.getElementById('permStatus')
const currentPermission = Notification.permission
status.textContent = currentPermission
if (currentPermission === 'denied') {
  Notification.requestPermission().then(result => {
      status.textContent = result;
      })
}
</script>

The global Notification object exposes the permission property reflecting the current permission status and the requestPermission function which returns a promise resolving permission status given by the user after being prompted by the browser. In some browsers, such as Firefox 72 and onward, a popup requesting notifications won’t be displayed; the promise from requestPermission pends until the user clicks on the icon allowing notifications from their URL bar.

Note on above code

In the snippet for above, the browser requests permissions as soon as the page is loaded. Additionally, if the user hasn’t allowed notification permissions, they will be asked every time they visit the page. This is a horrible user experience. Nobody wants to be spammed with notifications, especially when no explanation of what type of events will trigger the notifications or how often they will be sent is given. Therefore, it is important that you never request notification permissions before giving an explanation of the notification behavior and after gathering user input for opting in.

For our examples, we will be using a Web Worker to manage the notification logic. This has the advantage of being off the main thread, which means that the logic is non-blocking and can be accessed via a global singleton for component based frameworks such as React, allowing all components to trigger notifications. One of the disadvantages of using a web worker for notifications is that the notifications are not triggered if the user closes the tab. To bypass that restriction, the example code will have to be executed in a Service Worker.

Example: Long Running Jobs

Some applications have long running jobs: processing an uploaded file, preparing data for download, executing a CI / CD pipeline, etc. Web notifications are a good way for users to “click and forget”, setting the job in motion and continue to other parts of the application or backgrounding the tab and continuing to browse without having to constantly check the status of their work. When the job has finished, they get a notification that lets them inspect the final output. Let’s get started!

<!-- index.html -->
<button onclick="notifyOnDone()">Notify me when this job finishes</button>
<span id="permErr" style="display:none; color:red;">You must allow notifications to subscribe to this job</span>

<script>
  async function notifyOnDone() {
    const currentPerms = Notification.permission
    if (currentPerms === 'denied' || currentPerms === 'default') {
      const result = await Notification.requestPermission()
        if (result === 'denied') {
          document.getElementById("permErr").style.display = "block"
        }
    }
    const notifierWorker = new Worker('notifier.js')
    notifierWorker.postMessage({jobId: 123})
  }
</script>

Since our examples don’t rely on the Push API, we’ll implement the checks via long polling in our web worker:

// notifier.js
onmessage = function(e) {
  switch (e.data.type) {
    case "JOB_START":
      watchForJobWithId(e.data.jobId);
      break;
    default:
      break;
  }
};

function watchForJobWithId(jobId) {
  // long polling
  const pollChanges = setInterval(() => {
    fetch("/jobs/completed")
      .then(res => res.json())
      .then(({ jobs }) => {
        const foundJob = jobs.find(job => job.id === jobId);
        if (foundJob) {
          const jobDoneNotificiation = new Notification(
            `Job finished with status: \n${foundJob.status}`,
            {
              body: foundJob.error ? `Error Code: ${foundJob.error}` : ""
            }
          );
        }
        clearInterval(pollChanges);
      });
  }, 30_000);
  window.onunload = () => pollChanges && clearInterval(pollChanges);
}

When users click the subscribe button, our worker polls the backend to see if the job has completed. If the job has finished, the worker triggers the notification with the status and any messages that are attached.

Duly Noted

Finding the line between utility and spam is tricky. When finding use cases for notifications, be sure to offer the users preferences for what type of notifications they want to receive and how often they want to receive them. Well designed notification experiences increase the usability of your app and keep users abreast of important changes, increasing their engagement and satisfaction.

Build a Better Web with Service Workers

jamestthompson3 — Sat, 07 Mar 2020 00:00:00 +0000

Working Hard, or Hardly Working?

The Service Worker API is a powerful tool for providing better offline experiences, push notifications, and background syncing for web applications. Like me, you may be familiar with service workers from the role they play in creating Progressive Web Apps (PWAs), or from seeing them registered as part of the build output of create-react-app. While this automatic output from build tools such as create-react-app is very useful in getting started with PWAs, using the Service Worker API is much more beneficial to your application when tailored to your use cases. Let’s dive into the Service Worker API to see how it provides a better user experience in some common use cases, starting with caching network responses.

A great place to start for any Web API is the MDN Page. In the case of the Service Worker API, MDN offers the following paragraph as an introduction:

Service workers essentially act as proxy servers that sit between web applications, the browser, and the network (when available). They are intended, among other things, to enable the creation of effective offline experiences, intercept network requests and take appropriate action based on whether the network is available, and update assets residing on the server. They will also allow access to push notifications and background sync APIs.

Since service workers act as a man in the middle between the browser, network, and your application, their potential is huge! However, this means that the potential for abuse is also huge, and to combat this abuse, browsers have built in security measures around the Service Worker API. Some of these security measures are: service workers are only registered over HTTPS or from your localhost, and only registered from the same origin as the current page. Working in Firefox, you enable service workers over HTTP in your developer tools console, and inspect and debug registered service workers in about:debugging.

Cache Money

Content Caching is a great starting point for utilizing service workers, and tools like create-react-app implement by default. Caching static assets is one of the first steps in creating a PWA. Let’s dive into caching by looking at the code for caching this blog for offline use! If you open up your devloper tools on the homepage of this blog, you’ll find an index.js file that has this content:

function registerSW() {
  if (location.hostname === "localhost") return;
  if ("serviceWorker" in navigator) {
    navigator.serviceWorker.register("/sw-prod.js").catch(e => {
      console.log("Registration fail: ", e);
    });
  }
}

registerSW();

This is the code for registering the service worker for this site. On its own, it’s not very compelling or informative, to really understand what’s going on under the hood, we need to take a look at the sw-prod.js file. Inspect this file by opening Firefox’s about:debugging in a new tab, selecting the service worker for this domain, and clicking inspect (or if it’s not running, clicking run, and then inspect).

What to cache

Now that you have the service worker code let’s break it down:

const CACHE_NAME = "posts-{someUUID}";
const PAGES = [
  "./ico.png",
  "./pandocoverride.css",
  "./style.css",
  "./space.png",
  "./reset.css",
  "./blog.css",
  "./index.html",
  "./vimloop.html",
  "./luanvim.html",
  "./vimtip-gitlens.html",
  "./frameworkpt2.html",
  "./frameworkpt1.html",
  "./frameworkintro.html",
  "./vimcandothat.html",
  "./datastructures.html",
  "./viiksetjs.html",
  "./proxies.html",
  "./rxjs-recompose.html",
  "./blogheader.js"
];

// install pages
self.addEventListener("install", installWorker);

async function installWorker(e) {
  await self.skipWaiting();
}
// ...continues below

Service Worker life cycle

This snippet handles the installation event, as well assigning a value for the cache name and the pages to be initially cached. The interesting part of this code is the function that gets called on the install event. Installation is the first event in the service worker lifecycle, it gets kicked off by the register function in index.js. In our installWorker function, we call the skipWaiting method on the service worker, which causes the waiting service worker (in this case the one we are trying to install) to become the active service worker. Let’s move on to the next section:

self.addEventListener("activate", activateServiceWorker);

async function activateServiceWorker(event) {
  await deleteOldCaches();
  await installCachedFiles();
  event.waitUntil(clients.claim()); // make the current sw the active sw in all cached pages
}

async function installCachedFiles() {
  const cache = await caches.open(CACHE_NAME);
  return cache.addAll(PAGES);
}

async function deleteOldCaches() {
  const keys = await caches.keys();
  const oldVersions = keys.filter(name => {
    if (/^posts-(\w{8}(-\w{4}){3}-\w{12}?)/.test(name)) {
      return true;
    } else {
      return false;
    }
  });
  return Promise.all(oldVersions.map(key => caches.delete(key)));
}
// ...continues below

After the service worker is installed, it activates, calling the activateServiceWorker function we registered on the activate event. When our service worker is activated, we want to delete the old cache and install the current version of the files denoted in the PAGES array in the new cache. The function, deleteOldCaches gets all of the cache keys (which are the old versions of CACHE_NAME), and checks them to see if they match the format we’ve given to our CACHE_NAME. We don’t simply clear all the keys in the cache because we may have several service workers running at a time, and clearing their data may have unintended consequences, so this service worker only removes data that it has placed in the cache itself.

Installing the new versions of the blog posts requires opening a cache with the key of CACHE_NAME, and using the addAll method to specify that we want to add everything in the PAGES array to be added. After deleting old versions of the cache and installing the latest posts, the clients.claim() call allows the current service worker to become the active service worker on all pages in its scope. Wrapping clients.claim() inside of event.waitUntil() is an important step because it prevents the browser from interrupting the claim process which could lead to the service worker becoming active on some pages but not on others within its scope. Let’s break down the final section:

self.addEventListener("fetch", event => {
  if (
    event.request.mode === "navigate" ||
    event.request.destination === "style" ||
    event.request.destination === "script" ||
    event.request.destination === "image"
  ) {
    event.respondWith(cacheResponse(event.request, event));
  }
});

async function cacheResponse(request, event) {
  const cache = await caches.open(CACHE_NAME);
  const match = await cache.match(request.url);
  if (match) {
    return match;
  }
  // Create promises for both the network response,
  // and a copy of the response that can be used in the cache.
  const fetchResponseP = fetch(request);
  const fetchResponseCloneP = fetchResponseP.then(r => r.clone());

  event.waitUntil(
    (async function() {
      await cache.put(request, await fetchResponseCloneP);
    })()
  );

  return fetchResponseP;
}

Going to the network

In this segment of the service worker code, we are adding an event listener for fetch events and responding with a cached response under certain conditions. The conditions we are checking before responding with cached response are: if the request is a navigation event (loading this blog post in your browser for example), or if the browser requests additional resources like CSS, Javascript, or images. To illustrate this flow, we’ll take a look at two different resources requested by this website, vimtip-gitlens.html and bb8_1.svg.

Since vimtip-gitlens.html is specified in the PAGES array we cached on service worker activation, we’ll look at it first. When you navigate to the post, the event.respondWith function is executed with the value from cacheResponse being evaluated. Using event.respondWith tells the service worker to intercept the request and respond with the argument passed to this function. Let’s step through the cacheResponse function: we first open the current cache (as denoted with the CACHE_NAME) and check to see if there are any entries for the URL of the incoming request. If we’ve cached this URL before, we return the cached response–avoiding a network call. If the service worker can’t find the URL from its cache, then makes a network request and places the response into the cache while returning the Promise containing the response back to our main application. Putting the response in the cache is wrapped inside event.waitUntil to ensure that the browser does not interrupt the service worker while updating.

Wrapping up

The final step in this example is making sure that the service worker always reflects the most up to date content. This means that every time we make a change to one of our cached assets, we update the CACHE_NAME in the service worker. For this blog, I created a bash script that helps ensure I don’t forget to update my service worker cache:

#!/bin/bash

UUID=$(cat /proc/sys/kernel/random/uuid)
rm -f sw-prod.js
cp sw.js sw-prod.js
sed -i "s/%VERSION%/$UUID/g" sw-prod.js
echo "Built version: ${UUID}"

I use the uuid functionality built into Linux to generate a unique name for my cache, then I copy my service worker code with a placeholder for CACHE_NAME to the sw-prod.js file. The final step is to replace my placeholder cache name with the unique identifier I generated.

Caching static files is a great way to start using the Service Worker API, and offers your users a better offline experience. Stay tuned for more service worker tips!

A Brief Look at Javascript Proxies

jamestthompson3 — Sat, 01 Feb 2020 00:00:00 +0000

Proxies

Javascript Proxies are a powerful concept in ES6. They give developers more control over JS objects, paving the way for declarative API design. I am going to show some examples of how use proxies to extend the functionality of traditional Javascript objects and why they are useful in understanding the flow of data inside your programs.

Microservice Client

The first example comes from a request management client that uses a JSON schema to validate incoming requests and route them to other services. The code for its implementation is here. Proxies are valuable in this context since our request client’s API reflects our schema, thus allowing the schema to serve as both validation and documentation. Another advantage to schema driven API definitions is that automated tools are able to generate and update the schemas based on microservice API docs, minimizing the work needed to stay in sync with other services to which we are making requests.

Designing The API

Since this a client for managing requests to different microservices, we want the API to reflect the supported HTTP methods and service name of each URL. A call to a posts service looks like this:

const mySchema = {
  posts: {
    methods: ["get", "post"],
    path: template("/"),
    bodyValidator: {
      id: "updateProduct",
      type: "object",
      properties: {
        title: { type: "string" },
        author: { type: "string" },
        content: { type: "string" }
      }
    },
    paramsValidator: {
      id: "sortBy",
      type: "object",
      properties: {
        id: { type: "string" }
      }
    }
  }
}
// The Manager class shown in the next example, request adaptors are not covered in this post
// but more information can be found in the repo link above.
const postsClient = new Manager('http://posts.some-service.net', axiosAdapter).validateWith(mySchema)
const getRequest = myclient.posts().get({sortBy: 'newest'})

getRequest.then(res => res.json()).then(console.log)

const postRequest = myclient.posts().post({title: 'why proxies are cool', author: 'taylor thompson', content: 'proxies are cool because they give you superpowers.'})

postRequest.then(res => res.json()).then(console.log)

The question then becomes, “how does our request client reflect the schema without manually typing out all the fields?”. This is where proxies come in to play. Let’s look at a stripped down implementation of the Manager class:

class Manager {
  constructor(baseURL, adaptor) {
    this.baseURL = baseURL
    this.adaptor = adaptor
    }
    validateWith(schema) {
      const self = this; // Added to prevent confusion with 'this' when nesting calls
      return accessInterceptor(function(_, schemaPath) {
        if (!(schemaPath in schema)) {
        // custom defined error
        throw new PathNotInSchemaError(`${schemaPath} not in schema.\n valid paths are\n [${Object.keys(schema)}]`)
        }
        return specifier => accessInterceptor(function(_, method){
          const supportedMethods = schema[schemaPath].methods
          if (!supportedMethods.includes(method)) {
            throw new MethodNotSupportedError(`${method} not supported in ${schemaPath}.\n supported methods are \n [${Object.values(supportedMethods)}]`)
            }
          })
       // continued below
       // ...
      })
      }
  }

function accessInterceptor(interceptor) {
  return new Proxy({}, {
    get(target, property) {
      return interceptor(target, property)
    }
   }
  )
}

Much of the heavy lifting in the Manager class is performed by the accessInterceptor function. It takes a function as an argument and returns a proxy. The interceptor function is called by the proxy whenever we try to access a property of the target object (which in this case is just an empty object). Using the interceptors on property access allows the object to dynamically take whatever shape we want, which in snippet above, is managerClassInstance.schemaPath.httpMethod. Because the first interceptor function checks whether or not the property attempting to be accessed is defined in the provided schema, if we were to try and access a non-existent property on the example schema like users, the Manager class instance throws a PathNotInSchemaError.

By using the accessInterceptor function, we are able to chain methods according to our schema. If our schema changes, the methods available on the Manager class instance also change. This enforces parity between our code and our documentation (in this case the schemas are self documenting).

Object Access Interception

Being able to introspect the activity of your data structures is useful. For example, if you want to trace all the get and set operations on an object, proxies are here to help:

const dataStore = {
  users: [{username: 'user123', id: '3a34cb03s'}, {username: 'tester456', id: '6kj77acv9'}],
  posts: [{title: 'some cool post', desc: 'a great post about javascript'}],
  //....
}

const handler = {
  get(...args){
    console.log('%cGETTING WITH: %o', 'color: purple;', ...args);
    return Reflect.get(...args)
  },
  set(...args){
    console.log('%cSETTING WITH: %o', 'color: blue;', ...args);
    return Reflect.set(...args)
  }
}

const dataStoreWithTracingEnabled = new Proxy(dataStore, handler)

In this example, when you assign or access property values to the dataStoreWithTracingEnabled variable, you see the target, property, value, and receiver in the console. If you set the new property onlineStatus by dataStoreWithTracingEnabled.onlineStatus = 'busy', you see the following in your console:

  - Our message:
  SETTING WITH:

  - the target (the datastore object):
{
  users: [{username: 'user123', id: '3a34cb03s'}, {username: 'tester456', id: '6kj77acv9'}],
  posts: [{title: 'some cool post', desc: 'a great post about javascript'}],
  onlineStatus: 'busy'
}

 - the property:
  onlineStatus

- the value:
  'busy'

- receiver:

Proxy
  <target>: Object {
    users: [{username: 'user123', id: '3a34cb03s'}, {username: 'tester456', id: '6kj77acv9'}],
    posts: [{title: 'some cool post', desc: 'a great post about javascript'}],
    onlineStatus: 'busy'
  },
  <handler>: Object {
    get: get(args),
    set: set(args)
  }

Intercepting Object properties is not only useful for introspection, but also for creating user friendly abstractions. Some interesting libraries that use proxies are: immer, uses proxies to produce immutable datastructures from an API that follows Javascript’s mutable Object methods, objecthistory which enables undo and redo for values assigned to objects, and echo, which uses proxies similarly to the above example, logging the evaluation of the code you type in the console. Some ideas where to use proxies in your current codebase: sending telemetry, better request logging, input sanitization, validating requests, and broadcasting state changes (a lightweight alternative to observables).

Additional Resources

For more information on how proxies work, mdn as always is a great reference, as well as the section meta programming with proxies from Dr. Axel Rauschmayer’s excellent book, Exploring ES6.

Neovim Tip - GitLens

jamestthompson3 — Tue, 28 Jan 2020 00:00:00 +0000

Intro

GitLens is a VSCode plugin that, among other things, allows you to see the time, commit author, and commit message of the current line. With a little help from to the neovim api and our shell, it we can recreate this functionality in a few lines of lua code.

The Code

-- in utils.lua
local M = {}
local api = vim.api
function M.blameVirtText()
  local ft = vim.fn.expand('%:h:t') -- get the current file extension
  if ft == '' then -- if we are in a scratch buffer or unknown filetype
    return
  end
  if ft == 'bin' then -- if we are in nvim's terminal window
    return
  end
  api.nvim_buf_clear_namespace(0, 2, 0, -1) -- clear out virtual text from namespace 2 (the namespace we will set later)
  local currFile = vim.fn.expand('%')
  local line = api.nvim_win_get_cursor(0)
  local blame = vim.fn.system(string.format('git blame -c -L %d,%d %s', line[1], line[1], currFile))
  local hash = vim.split(blame, '%s')[1]
  local cmd = string.format("git show %s ", hash).."--format='%an | %ar | %s'"
  if hash == '00000000' then
    text = 'Not Committed Yet'
  else
    text = vim.fn.system(cmd)
    text = vim.split(text, '\n')[1]
    if text:find("fatal") then -- if the call to git show fails
      text = 'Not Committed Yet'
    end
  end
  api.nvim_buf_set_virtual_text(0, 2, line[1] - 1, , {}) -- set virtual text for namespace 2 with the content from git and assign it to the higlight group 'GitLens'
end

function M.clearBlameVirtText() -- important for clearing out the text when our cursor moves
  api.nvim_buf_clear_namespace(0, 2, 0, -1)
end

return M

" in init.vim
lua vim.api.nvim_command [[autocmd CursorHold * lua require'utils'.blameVirtText()]]
lua vim.api.nvim_command [[autocmd CursorMoved * lua require'utils'.clearBlameVirtText()]]
lua vim.api.nvim_command [[autocmd CursorMovedI * lua require'utils'.clearBlameVirtText()]]

hi! link GitLens Comment

Using LibUV in Neovim

jamestthompson3 — Tue, 07 Jan 2020 00:00:00 +0000

Off The Main Loop: Async Actions within Neovim

Neovim embeds the libuv library in the editor and exposes lua (and to some extent vimscript through the jobstart function) bindings for interacting with the library’s API. This allows actions in the editor to happen asynchronously, keeping the main editor loop clear for user input. What this means practically for users is that actions which take a long time such as grepping through large projects, generating ctags, or linting can now be done in the background without blocking the user’s ability to keep editing text.

Let’s explore how to use the libuv lua bindings in Neovim to create useful asynchronous actions. I will cover two examples: using the pandoc program to convert a markdown file to html, and using ripgrep to search within a project. Both of these examples use the libuv bindings differently, but still illustrate well the capabilities of libuv inside Neovim.

First Example: Convert Markdown Files with Pandoc

Pandoc is a powerful tool for converting files into and different formats. I write many Markdown documents and it is useful to convert them into HTML for my blog. In order to do this in an asynchronous way, we will use vim.loop from Neovim’s lua library. I would recommend going through the documentation on vim.loop as you code along with this post. Let’s start by creating a basic lua function outline in our packagepath:

-- in ~/.config/nvim/lua/markdown.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.convertFile()
  local shortname = vim.fn.expand('%:t:r')
  local fullname = api.nvim_buf_get_name(0)
 -- loop logic goes here
end

return M

In this snippet, we first get the shortened name of the file (i.e. vimlooppost in vimlooppost.md), and the full name of the file (i.e. ~/blog/posts/vimlooppost.md). We will use both of these variables later when we spawn the pandoc process. In order to get a better understanding of what vim.loop.spawn does, let’s head over to the libuv documentation. Here we can see that it takes an options table, an onexit callback, and returns the processId and the handle of the spawned process. Using this information, let’s build out the rest of the convertFile function:


-- in ~/.config/nvim/lua/markdown.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.convertFile()
  local shortname = vim.fn.expand('%:t:r')
  local fullname = api.nvim_buf_get_name(0)
  handle = vim.loop.spawn('pandoc', {
    args = {fullname, '--to=html5', '-o', string.format('%s.html', shortname), '-s', '--highlight-style', 'tango', '-c', '--css=pandoc.css'}
  },
  function()
    print('DOCUMENT CONVERSION COMPLETE')
    handle:close()
  end
  )
end

return M

Pandoc receives as arguments the strings listed in the args table, and when the process finishes, we echo a success message and close the handle. Using this in our vimrc is as simple as:

" in ftplug/markdown.vim

nnoremap <leader>c :lua require'markdown'.convertFile()<CR>

Second Example: Async Grep

Much like in the first example, we will be relying on an external program to do the heavy lifting, ripgrep. Unlike in the first example, we don’t only want to kick off a background process, but also use the values generated in this process inside of Neovim. For that, we will need to pass a set of file descriptors to vim.loop.spawn:

-- in ~/.config/nvim/lua/tools.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.asyncGrep(term)
  local stdout = loop.new_pipe(false) -- create file descriptor for stdout
  local stderr = loop.new_pipe(false) -- create file descriptor for stdout
  handle = loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
  end
  )
  loop.read_start(stdout, onread) -- TODO implement onread handler
  loop.read_start(stderr, onread)
end
return M

Our function, asyncGrep will take a search term as an argument which it then passes to ripgrep in the loop.spawn call. After the process is spawned, we need to start reading the output into our file descriptors, which will then call the onread callback which we have yet to implement. Let’s implement the onread callback now:

-- in tools.lua
local results = {}
local function onread(err, data)
  if err then
    -- print('ERROR: ', err)
    -- TODO handle err
  end
  if data then
    table.insert(results, data)
  end
end

The onread callback takes data written to our file descriptor by the ripgrep process and appends a table called results. We now want to add functionality to our asyncGrep function that will allow us to use these results to set the quickfix list:

function M.asyncGrep(term)
  local stdout = loop.new_pipe(false) -- create file descriptor for stdout
  local stderr = loop.new_pipe(false) -- create file descriptor for stdout
  local function setQF()
    vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
    api.nvim_command('cwindow')
    local count = #results
    for i=0, count do results[i]=nil end -- clear the table for next search
  end
  handle = loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
    setQF()
  end
  )
  loop.read_start(stdout, onread) -- TODO implement onread handler
  loop.read_start(stderr, onread)
end

If you run this function, you will encounter this error message: lua/tools.lua:122: E5560: vimL function must not be called in a lua loop callback. In order for any vim functions to be called within a lua loop callback, they need to be wrapped in vim.schedule_wrap. Wrapping vim functions in vim.schedule_wrap is necessary since it schedules the callbacks to be invoked when it is safe, bridging the gap between the libuv event loop and the internal Neovim main loop. To learn more about vim.schedule_wrap, check out :h schedule_wrap. Let’s fix this error and see what our function looks like when it’s all put together:

-- in ~/.config/nvim/lua/tools.lua
local M = {}
local loop = vim.loop
local api = vim.api
local results = {}

local function onread(err, data)
  if err then
    -- print('ERROR: ', err)
    -- TODO handle err
  end
  if data then
    table.insert(results, data)
  end
end

function M.asyncGrep(term)
  local stdout = vim.loop.new_pipe(false)
  local stderr = vim.loop.new_pipe(false)
  local function setQF()
    vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
    api.nvim_command('cwindow')
    local count = #results
    for i=0, count do results[i]=nil end -- clear the table for the next search
  end
  handle = vim.loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  vim.schedule_wrap(function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
    setQF()
  end
  )
  )
  vim.loop.read_start(stdout, onread)
  vim.loop.read_start(stderr, onread)
end

return M

Let’s use this newly created async function in our vimrc:

" in init.vim
command! -nargs=+ -complete=dir -bar Grep lua require'tools'.asyncGrep(<q-args>)

Now we can call :Grep searchTerm and get results without blocking the main editor loop!

What’s Next?

Having libuv bindings in Neovim unlocks a lot of potential for extending the functionality of your editor. The ability to asynchronously spawn other process can be used for linting, file watching, formatting, and much more. Free yourself from main loop blockage and try experimenting with vim.loop.

From init.vim to init.lua

jamestthompson3 — Mon, 16 Dec 2019 00:00:00 +0000

Why Lua?

Neovim has an embedded lua 5.1 runtime which is used to create faster and more powerful extensions of your favorite editor. In the Neovim charter, it lists one of its goals as developing a first-class lua scripting alternative to VimL. One of the reasons for doing this is that VimL is a slow interpreted language with almost no optimizations. Much of the time spent in vim startup and in actions from plugins that can block the main loop in the editor is in parsing and executing vimscript. A great explanation of this can be found in Neovim lead maintainer, Justin M. Keyes’ talk, We can have nice things.

With the recent introduction of the built-in LSP client in the master branch written in lua, I became more interested in the possibilities lua has to offer and began trying to use lua in Neovim. I have never written lua before and have not seen very many guides on how to utilize the lua runtime in Neovim, so I want to illustrate the process of learning how to take advantage of the powerful scripting capabilities that are available in the Neovim runtime. Given that my experience is still very basic, these examples will also be quite small, but I hope that it can be a good jumping off point for those interested in using lua more in extending Neovim.

Getting Started

One of the first things I was confused about was how to use lua code inside of vim and vimscript. Luckily, the documentation in :h lua gives a few examples of how lua can be used in the editor. I recommend reading it for an in-depth explanation of how Neovim treats lua and the sourcing of lua files. Here’s a high-level overview of different approaches to executing lua code in your editor:

From the vim command line, you can run :lua <yourCodeHere>. This is useful for keybindings, commands, and other one-off execution cases.
Inside of a VimL file, you can demarcate lua code with the following code fencing:

lua << EOF
-- your lua code here
EOF

Inside of a VimL file you can use the lua keyword to execute commands similar to the first example. (i.e. lua <yourCodeHere>).

One important note here is that Neovim will look for lua code in the runtimepath you’ve set in your settings. Additionally, it will append your runtimepath with /lua/?.lua and /lua/?/init.lua so it is common practice to see a /lua sub-directory inside .nvim. For more detailed information about where Neovim looks for lua code, check out :h lua-require.

Your First Function

Porting your init.vim to lua can be a big undertaking, so it’s best to start small. For the first example, we’ll create a function which creates a scratch buffer.

This function will live in a file we’ll call tools, so create it in the lua directory in your nvim config: ~/.config/nvim/lua/tools.lua. Once we’ve created the file, we’ll fill it out with some boilerplate:

-- in tools.lua

local M = {}

function M.makeScratch()
end

return M

Using the table M here allows us to keep things out of the global scope and to use only what we need when calling the function from nvim. We’ll be using the neovim API to make scratch buffer, so let’s create a shorthand for it in our tools.lua file:

-- in tools.lua
local api = vim.api

local M = {}

function M.makeScratch()
end

return M

We can create a new buffer with the enew command, and the neovim API gives us a way to call nvim commands from lua:

-- in tools.lua
local api = vim.api

local M = {}

function M.makeScratch()
  api.nvim_command [[enew]] -- equivalent to :enew
end

return M

Next, we want to set some buffer options so that our scratch buffer isn’t listed in the buffer list and doesn’t have a swapfile created for it:

-- in tools.lua
local api = vim.api

local M = {}

function M.makeScratch()
  api.nvim_command [[enew]] -- equivalent to :enew
  vim.bo[0].buftype=nofile -- set the current buffer's (buffer 0) buftype to nofile
  vim.bo[0].bufhidden=hide
  vim.bo[0].swapfile=false
end

return M

That is all we need to create the scratch buffer! Now let’s use it in our init.vim:

" in init.vim

command! Scratch lua require'tools'.makeScratch()

Now a scratch buffer is created by running the command :Scratch.

You can port your init.vim to lua one function at a time, and if you get stuck, you can always use vim.api.nvim_command! When looking for help, make sure to check out :h api, and :h lua.

Using v:lua

The variable v:lua can be used to call lua functions from within vimscript. A great use case for this is accessing the LSP client’s omnifunc. If you wanted to use the LSP completion for Rust, you may have something like this in your configuration:

" in init.vim
lua << EOF
  local nvim_lsp = require 'nvim_lsp'
  nvim_lsp.rust_analyzer.setup({})
EOF


" in ftplugin/rust.vim

set omnifunc=v:lua.vim.lsp.omnifunc

Interop With vim.fn

It is useful to have access to vimscript functions from inside of lua, especially when interacting with autoloaded functions or functions provided by plugins. In this example, we will have an autocmd that will execute the vimscript function, tools#loadCscope when the VimEnter event happens.

" in autoload/tools.vim

function! tools#loadCscope() abort
  try
    silent cscope add cscope.out
  catch /^Vim\%((\a\+)\)\=:E/
  endtry
endfunction


-- in file that you source, such as init.lua

function sourceCScope()
  vim.fn['tools#loadCscope']() -- no arguments needed
end

function nvim_create_augroups(definitions)
  for group_name, definition in pairs(definitions) do
    vim.api.nvim_command('augroup '..group_name)
    vim.api.nvim_command('autocmd!')
    for _, def in ipairs(definition) do
      local command = table.concat(vim.tbl_flatten{'autocmd', def}, ' ')
      vim.api.nvim_command(command)
    end
    vim.api.nvim_command('augroup END')
  end
end

local autocmds = {
  startup = {
    {"VimEnter", "*", [[lua sourceCScope()]]};
  }
}

nvim_create_augroups(autocmds)

Vim does that already

jamestthompson3 — Sun, 25 Aug 2019 00:00:00 +0000

Power to Productivity

Vim is a powerful tool which helps you stay productive and spend less time doing the mechanical work of coding.For many, Vim seems archaic, under-featured, or even intimidating. I hope to illustrate that Vim none of those things, but is in fact a great editor for nearly every development environment.

I want to focus on three common tasks where Vim helps you be more productive without leaving your editor. These tasks are: navigation between files, and searching / replacing.No Vim experience is necessary for this post, but it definitely helps if you are at least familiar with the basics through vimtutor.

Navigation Between Files

One of the most common and time consuming activities we do as developers every day is jump between files.If you are using a more traditional text editor such as VSCode, Sublime Text, or Atom, chances are you spend more time that you would likescanning through your directory tree trying to find the right file. With Vim, the secret to file navigation begins with set path.

The path is where Vim searches for files when executing the various search commands. By default on Unix-like systems, it is .,/usr/include,,. Let’s break this down a bit. The first . indicates that Vim should include files relative to the current file’s directory.Directories that should be searched are separated by commas, so the next place Vim will look by default is the /usr/include/ directory.This directory typically contains headers so it can be useful if you are doing C and C++ programming. The final sequence of ,, instructs Vim to search in the current directory.Out of the box, we are able to search a large part of our project. What if we wanted to search downward recursively through our project?It is common to open your text editor in your project root ( often denoted by a vcs file such as a .git folder ).To ensure that Vim finds all of our project files when we search for them, we use * and **.The asterisks represent wildcards, with * matching 0 or more characters and ** matching only directories. By setting your path to set path=.,,, **, you can ensure that Vim will search all our project files.One caveat to this is that by default ** only searches 30 directories deep, so if you have an extremely nested directory structure, you can increase that limit, see :h starstar for more details.Now that we have our path correctly set, it’s time to navigate! Vim has a built in keybinding for jumping to the file under your cursor, and it is gf. We tell Vim which file endings it should add by using set suffixesadd.So for frontend web development, you might have something in your vimrc that looks like this: au! BufNewFile,BufRead *.js,*.jsx suffixesadd+=.js,.jsx. This allows us to jump to files with endings .js and .jsx.

In practice it looks like this:

import { someFunction } from "./utils";

Placing your cursor in normal mode on ./utils and pressing gf should takeyou to the utils.js file!

This is just the tip of the iceberg of what Vim can do when it comes to searching through your code, for a more in-depth take on this, I would recommend the blog post, death by a thousand files.I want to show one more useful command that benefits from having set our path to search downward recursively and that is find. With find, Vim will search for a file and then open it for editing if the file is found. We can also use* wildcard expansion and tab completion if we have the wildmenu setting on.

For example, typing: :find N*C*.jsx and pressing tab would expand to NavigationContainer.jsx. You could also type Navigation and tab to expand all the options that Vim finds in the current path. Just like that, you can fuzzy search through your project without any additional configuration or plugins.Sometimes, there are directories in the project which we would like to ignore, such as node_modules, target, and dist. Vim determines which files to ignore via the wildignore setting.A sample wildignore may look like this: set wildignore=*/dist*/*,*/target/*,*/builds/*,*/node_modules/*. Vim will no longer include these directories in it’s path.

Find and Replace

By default, Vim ships with vimgrep which allows you to search for strings within files. Vim also comes with the grep command which allows you to searchusing an external grep tool. I personally use ripgrep, a grep tool that is focused on speed and respects my .gitignore file by default.

To enable Vim to use ripgrep ( or any other grep program ) as a backend to the grep command, you need to set the grep program in your vimrc:

set grepprg=rg --smart-case --vimgrep

Now you have your search results loaded in the quickfix menu, and can leverage a Vim built-in plugin, cfilter. To add it, simply run :packadd cfilter and Vim will load the plugin for use. cfilter filters through entries in the quickfix list via regex.

If you have search results that look like this:

datastructures.html|623| 30: mixpanel.track("data structures loaded");
index.html|61| 26: <a href="./datastructures.html"
index.html|62| 19: >Data Structure Memes For Edgy JavaScript Teens</a
_site/index.js|64| 28: import { customStructure } from './datstructures'
_site/utils.js|64| 28: import { customStructure } from 'custom/datstructures'

If you want to only keep the search results from the JavaScript files, you run this command: :Cfilter! html. This keeps everything except matches that containhtml in the quickfix list. If you wanted only to keep only the third match, you run :Cfilter Memes and it keeps only matches containing Memes.

One more useful command when using the quickfix list is :cfdo. It allows you to execute whatever command you give it across the entire quickfix list.For example, if you wanted to search and replace you could do it like this:

search for the phrase, ‘data structures’:

:grep! 'data structures'

view results:

datastructures.html|279| 15: Data Structures
datastructures.html|574| 26: with these data structures, since the structure of our data is
datastructures.html|575| 43: irrelevant to react, custom data structures should be simple to
datastructures.html|623| 25: mixpanel.track("data structures loaded");

change ‘structures’ to singular ‘structure’ and save every file:

:cfdo s/ures/ure | update

The | update part of that last command tells Vim to save the changes after it executes the replace. If you wanted to check your changesbefore saving, you leave that part off. You still have to save every file manually if you choose this option.

Grep tips

Vim makes finding and replacing throughout a large codebase fast and easy. To make it even faster and easier, here are some things for your vimrc.

set grepprg=rg\ --smart-case\ --vimgrep " set grep program to ripgrep with smartcase flag. You can set it to anything you like.

" creates a :SearchProject command, makes it so you don't have to escape strings in ripgrep, will tab compete with directories in path.
command! -nargs=+ -complete=dir -bar SearchProject execute 'silent! grep!'.<q-args>.' | cwindow'

I hope that this post helps improve your Vim experience!

Data Structure Memes For Edgy JavaScript Teens

jamestthompson3 — Wed, 19 Jun 2019 17:01:10 +0000

Frontend developers increasingly face challenges surrounding complex state and data management. We encounter issues resulting from data management which has become too monolithic or too fragmented, our components update too often, or we spend much of our development time trying to discover how data is being passed through our entire application. Our components should consume only the minimum amount of information necessary to render their children. How can that be achieved in a way both easy to use and easy to comprehend throughout our application? I think the answer is in taking data management outside the realm of react and using react hooks to, well, hook into relevant data updates.

React has shipped with hooks since 16.8.0, and it this has caused developers to ditch class components in favor of functional components with hooks. You may have also considered ditching a library like redux by using hooks and react context. While initially seems like an excellent idea ( I re-wrote a large portion of our application at work this way ), you will find that hooks and context can cause unnecessary re-renders and increase the logical complexity of your codebase.

If you want to just skip to the code, here is the sandbox

Cache Me Outside: How 'Bout That?

Most react applications need to fetch data from a server and display it to the user of the application. Where to store that data in the application quick becomes a challenge as the application grows in size and scope. If you inspect a react application of nearly any size, you will probably find a combination of different solutions. It's popular to use a third-party library like redux or mobx, but sometimes this data is stored in local component state.

Trade offs need to be considered in each approach: using an external library can mean writing more code to update and consume our data in the application, leading to hard to follow logic; keeping application data in component state means that it disappears when the component is unmounted, forcing us to re-fetch the data or place the component higher up in the render tree ( often this is the pattern of "container" components ).

State and data management in many cases can and should be separated. The data available to consume in an application is not necessarily reflected in the current state of the components which consume that data. An example of this is storing data in redux. When we place data in a reducer from a server, we now have access to that data while we are connected to the redux store. A component that consumes that data may have several states and state transitions, but that does not change the availability of the data for consumption.

I think that we can move data management outside of react, giving us the benefits of:

0: Having a simple API for both writing and reasoning ( a problem with redux, sagas, mobx, etc. is boiler plate code and hard to follow logic ).

1: Allowing us to bind the UI to the data when necessary, but not having our data depend on our rendering library ( using react context means our data must follow the constraints of react )

2: Allowing for underlying changes to data only relevant to the current component to automatically trigger a request to the react scheduler to update.

Big OOF: Why Context Is Complex

Storing data inside of react context can lead to complexity and re-renders, which can both harm the performance of your application and decrease the codebase's maintainability. Nik Graf has an excellent talk concerning this, so if you'd rather here it from him, check it out. Digging into context, however, the problems quickly arise when looking for a solution for managing data needed by your application. Let's dig into some examples.

Using the following collection of data:

[
  {
      "name": "sam",
      "id": "1987ea87gde302",
      "likes": [
        { "id": 0, "item": "cars" },
        { "id": 1, "item": "dogs" },
        { "id": 2, "item": "Bruce Springsteen" },
        { "id": 3, "item": "mowing the lawn" }
      ],
      "dislikes": [
        { "id": 0, "item": "vegetables" },
        { "id": 1, "item": "income tax" },
        { "id": 2, "item": "existential crises" }
      ]
  },
  ...
]

If we wanted to store this in react context and pull it out with hooks, we would do something like this:

// Assuming the data structure above, a list of user objects is named userList
const UserContext = React.createContext(userList)

function UserListView() {
  const listOfUsers = React.useContext(UserContext)

  return listOfUsers.map(user => <p>{user.name}</p>)
}

This works great! Until you need to update that list of users, in which case you probably need to create a custom component that exposes methods for updating and retrieving values inside the context:

const UserContext = React.createContext([])
function UserContextHolder({children}) {
  const [users, setUsers] = React.useState([])
  return (
    <UserContext.Provider value={{users, setUsers}}>
      {children}
    </UserContext.Provider>
  )
}

Seems simple enough! However, this component will have to sit high enough in the react DOM tree that all components that consume from it can be its children. This means that any other children of this component will be forced to re-render whenever any values of this context are updated! Additionally, if we try and re-use this context to store something else related to our user list, such as a selected user, or a collection of selected users, we would again force all components which consume this information to be children of the context and force them to re-render anytime any of the data changes.

To illustrate this, imagine we have a UI that shows a list of our users and then a list of the likes and dislikes of a selected user. If we store all of this information in context, we would see a lot of render events when using this UI:

=== MAIN LIST ===
0: The context mounts and our user list is updated via an XHR request.

1: The default selected user is chosen from the user list and is set in the context

2: Every time a new selected user is chosen, the context is updated and the component is re-rendered

3: Updates from likes list
4: Updates from dislikes list

=== LIKES LIST ===
0: Selected user from Main List causes initial render
1: Selected user update
2: Updates to itself
3: Updates to dislikes list

=== DISLIKES LIST ===
0: Selected user from Main List causes initial render
1: Selected user update
2: Updates to itself
3: Updates to likes list

Notice how with context, even updates to irrelevant bits of the data cause re-renders. Our main list that just renders the users' names should not be forced to re-render when information about a specific user's likes and dislikes is updated. This model also assumes that the three lists are the only children of the context component, but in the real world, our applications tend to be a little more complex. For example, if we add button components for adding, deleting, and editing likes and dislikes, all of those components would also be re-rendered.

Imagine if we add properties to the user object--for example if we want to show if a user is online--or we have a recursive data structure, with each user having a list of friends who in turn are user objects. Since many changes to the user list could take place, we would increase the amount of re-renders of every component each time we add, remove, or modify a part of this list. Storing data in react context creates unnecessary links between components and forces us to wrap each component in useMemo to optimize rendering.

Don't Forget To Like And Subscribe: Using RXJS To Build Custom Data Structures

One of the convenient aspects of using react context is that you get updates for free! Anytime a context value is updated, all the components that consume it and their children call for a re-render. This behavior is fantastic when you think about having a truly data-driven UI, but not so fantastic when you consider the complexities introduced above. So how can we keep this auto-updating behavior while reducing component renders to only depend on data directly consumed by the component itself? Enter rxjs.

If you aren't familiar with rxjs or reactive programming, I recommend you check out Andre Staltz's gist covering some of the principles of reactive
programming. Rxjs subjects are a way for components to subscribe to data changes. They offer a clean API for receiving and interacting with updates to a data store. However, piping data directly from observables into components will not be compatible with future react updates since react updates on a pull based system, whereas observables are push based. Using the hooks related to updating state provided by react, we subscribe to changes in the data without directly pushing updates to our components but rather requesting an update from the react scheduler.

Using the user list defined above, we can construct a custom store to contain our data and expose methods for updating and subscribing to updates. By creating this data structure outside of react, we allow its methods to be accessed independently from our UI, giving us a powerful starting point for creating our own useful data management utilities.

Let's start by creating a basic structure for storing data and subscriptions:

import { Subject } from 'rxjs'

class DataStore {
  subjects = new Map()
  store = new Map()

  getSubscription = key => this.subjects.get(key)
  getValue = key => this.store.get(key)

  createSubscription = key => {
    const subject = this.subjects.get(key)
    const storeValue = this.store.get(key)
    if (subject && storeValue) return subject
    this.subjects.set(key, new Subject())
    this.store.set(key, undefined)
    return this.subjects.get(key)
  }

  setValue = (key, value) => {
    this.store.set(key, value)
    this.subjects.get(key).next(value)
  }

  removeSubscription = key => {
    const selectedSubscription = this.subjects.get(key)
    const selectedValue = this.store.get(key)
    if (selectedSubscription) {
      selectedSubscription.complete()
      this.subjects.delete(key)
    } else {
      throw new Error('Cannot find subscription %s', key)
    }
    if (selectedValue) {
      this.store.delete(key)
    } else {
      throw new Error('Cannot find store key %s', key)
    }
  }
}

Unlike our example using context, this structure is completely agnostic to the shape of our data giving it flexibility to be re-used across our entire application. A data structure like this allows us to store almost anything ( except duplicate keys ), which means that we could have many instances of the DataStore object, or we could have a monolithic store that contains all of our application's data ( though I'm not sure if that is the best idea ).

From the methods exposed by the DataStore object, we can make a whole host of utilities for consuming, updating, introspecting, and subscribing to our data. Let's see how they can be consumed directly within our react components.

First, instantiate an instance of the DataStore object outside of our component and create a subscription for our user list.

// in UserRender.jsx

const USER_STORE = new DataStore()
USER_STORE.createSubscription('userList')

Inside our component logic we can create methods for consuming the data from the store. Here is where we want to cause re-renders when our data changes.

// in UserRender.jsx

const USER_STORE = new DataStore()
USER_STORE.createSubscription('userList')

const fetchAndStoreUserList = () => {
  fetchUsers().then(users => USER_STORE.setValue('userList', users))
}

export function UserRender() {
  const [userList, setUserList] = React.useState([])
  USER_STORE.getSubscription('userList').subscribe(setUserList)
  React.useEffect(fetchAndStoreUserList, [])

  return userList.map(user => <p>{user.name}</p>)
}

This is how we pull the list of users out of the DataStore and into our component. This leverages react's useState function by allowing us to request an update from react instead of immediately pushing component updates from our subscription. Piping the output of our subscription into useState also allows react to batch renders, which comes in handy if the user list was being updated from a web-socket connection or any other method that rapidly triggers state updates.

At this point you are probably be thinking, "this looks nice, but won't I still have to re-render the main list when I call USER_STORE.setValue ?". The answer is yes. Even though we have moved the management of the application data outside of react, we're still tied to the update cycle called by useState as it's passed as a callback to USER_STORE.setValue. This is where hooks really start to shine!

Press F For Selects ?

If you've used redux, you've most likely encountered selectors. For those who aren't familiar with the subject, selectors allow us to isolate ( or select ) a part of our application data and only initiate renders when that part of the data changes. Using hooks, we wrap the functionality of the USER_STORE to use a selector which only updates the UserRender component when the list of users changes. This means that we update parts of the user data ( like a likes or dislikes list ) without having to re-render the components that don't consume that data directly. Creating hooks that take selectors as an argument also helps those of us transitioning from a redux heavy codebase and allows for the re-use of existing code.

export function useSelector(store, subscriptionKey, selector) {
  store.getSubscription(subscriptionKey).subscribe(selector)
}

The code for useSelector is simple, thanks to how we created the DataStore. We simply want to pass it the store we from which we want to read ( in this case USER_STORE ), the key for the subscription we are interested in ( userList ), and the selector function that will be called whenever a new value gets pushed to the stream. We can now reuse our redux selectors with our new data structure!

We want to serialize the keys of our user list and only update the UserRender component if those keys change. To do that, we need to first create our user list selector:

function memoUsers() {
  const cache = {}
  return function(updateUser) {
    return function(userList: User[]) {
      const key = JSON.stringify(userList.map(user => user.user))
      if (cache[key]) {
        // don't call to re-render
      } else {
        cache[key] = key
        updateUser(userList)
      }
    }
  }
}

Now, memoUsers can be passed to our useSelector hook and be used in place of our userSubscription.

// in UserRender.jsx

import { useSelector } from './hooks'

const USER_STORE = new DataStore()
USER_STORE.createSubscription('userList')

const fetchAndStoreUserList = () => {
  fetchUsers().then(users => USER_STORE.setValue('userList', users))
}


function memoUsers() {
  const cache = {}
  return function(updateUser) {
    return function(userList: User[]) {
      const key = JSON.stringify(userList.map(user => user.user))
      if (cache[key]) {
        // don't call to re-render
      } else {
        cache[key] = key
        updateUser(userList)
      }
    }
  }
}

const cache = memoUsers()

export function UserRender() {
  const [userList, setUserList] = React.useState([])
  const setCachedUserList = cache(setUserList)
  useSelector(USER_STORE, 'userList', setCachedUserList)
  React.useEffect(fetchAndStoreUserList, [])

  return userList.map(user => <p>{user.name}</p>)
}

The UserRender component now only updates if we have added or removed a user from the list or changed the selected user, and not when we change the properties of a particular user. The component itself is simple and the heavy lifting of application data is handled by our DataStore. We didn't need to create actions and reducers, or use higher order components.

You can do more to improve writing to your data store be extending the DataStore object. Extending DataStore should be on a per-use-case basis, as it would be an anti-pattern to add a new method to DataStore for every use case encountered. A better approach would be to create a new object that extends DataStore and adds the methods needed for a particular situation. The key here is that we maintain flexibility with these data structures, since the structure of our data is irrelevant to react, custom data structures should be simple to read and simple to write.

State Management !== Data Management: don't @ me ( or do, I would love to hear your feedback )

Taking data management outside of react gives us a base for controlling externally triggered component renders. It also allows us to develop patterns that are easily re-used across the application. It takes the burden of caching and manipulating away from react. With hooks, we can easily hook into our custom data structures which allows our components to only consume what they need and react only to updates that are relevant to them.

Focusing on how our components consume data across our application prevents brittle architecture by allowing each component to independently pick and choose how and when it reads and writes to a shared data structure. Unlike using context, we don't have to think about where in the react DOM tree our components are located, or wrapping components in useMemo to optimize render cycles.

Efficient data management boils down to simplicity. Can you reliably track the flow of data through you application, are you able to introspect your data, are your components forced to update when data they don't consume changes? These are questions that should be asked as you build your application. No one-size fits all solution exists for data management, but I hope you will consider simplifying and try some of the concepts talked about here.