DEV Community: benjaminadk

Build A Command Line Application With Node

benjaminadk — Fri, 31 Jan 2020 14:20:38 +0000

Source Code

benjaminadk / node-cli-tutorial

Build A Command Line Application Tutorial

Node CLI Tutorial

Based on Build A Command Line Application

View on GitHub

Working CLI Tool

Original Post

Introduction

JavaScipt can do a lot more than just work in the browser. With the help of Node it can be used to build powerful command line tools. I recently created my own CLI to speed up repetitive tasks at work. The project we build in this article be a small portion of that and will do a lot more than print "Hello World".

If you don't have Node installed on your local machine download it HERE.

Option 1 - Clone Source Code

Download the zip or better yet clone the source code to your local machine.

git clone https://github.com/benjaminadk/node-cli-tutorial.git

Option 2 - Build From Scratch

If you want to code along with me and build from scratch just continue reading.

The first thing needed is a folder to store your project code in.

mkdir node-cli
cd node-cli

Lets initialize this folder as an NPM project and create a file to write some code in. NPM stands for Node Package Manager, by the way. Check out NPM.

npm init -y
touch index.js

If any of the instructions above were unfamiliar or didn't work you might want to do some googling now and learn more about Node, NPM and the command line in general.

Now would be a good time to clue everyone in one what this application is actually going to do.

Drum roll please. 🥁

We are going to make thumbnail images. More specifically, we will be able to navigate to a directory full of images via the command line, invoke our command, and create a new directory full of thumbnails. Of course, tasks like this can be accomplished with Photoshop and other image manipulation software, but we are programmers and we want to make our own. In this use case we will be making 225x190 pixel thumbnails from 800x800 textile images and saving them as 40% quality JPEGs. To help make this super easy we will use a package called Jimp. While we are downloading stuff lets get Commander and rimraf as well. Commander is a helpful framework for command line apps and rimraf is a file system utility to help us delete a directory and its contents programmatically.

Make sure you are in the project directory aka the same level as package.json aka the root.

npm install jimp commander rimraf

To make sure everything is working correctly, add a little code to index.js.

index.js

console.log('Hello World')

And we're done! 😎. Just kidding. This is just to make sure Node is working. I try to write tutorials that beginners can follow. From inside our node-cli directory we can now run the following.

node ./index.js

Quick tip. index.js is recognized as a sort of default filename in Node. This works as well.

node .

You should see Hello World printed in the terminal.

This is how JavaScript files are executed by the Node interpreter. However, the goal of this exercise is to make a command line tool. We want to be able to type a command anywhere on our computer and have our code execute.

First add the following line to the top of index.js. Understanding this line isn't important right now. It uses a Shebang - #! - and instructs the system to use Node to interpret this file. This is required when building command line applications with NPM.

index.js

#!/usr/bin/env node // highlight-line

console.log('Hello World')

The package.json file also needs to be updated. The important lines are highlighted. This bin key is telling NPM that when we type make-thumbs on the command line we want to run index.js. I named it make-thumbs just to avoid any conflicts with node-cli, but this can be named anything really. The corresponds to the text typed into the command prompt to execute our app. Common commands include cd, ls, curl, mkdir, and tons more. When these commands are entered there is actually code running somewhere behind the scenes.

package.json

{
  "name": "node-cli",
  "version": "1.0.0",
  "description": "Command line tutorial",
  "main": "index.js",
  "bin": {// highlight-line
    "make-thumbs": "./index.js" // highlight-line
  }, // highlight-line
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": {
    "name": "benjaminadk",
    "email": "benjaminadk@gmail.com",
    "url": "https://github.com/benjaminadk"
  },
  "license": "ISC",
  "dependencies": {
    "commander": "4.1.0",
    "jimp": "0.9.3"
  }
}

Now type make-thumbs in the command line. It should throw and error something like what you see below.

There is one more step to wire the global command to work on our system. Make sure you are in the root of the project.

npm link

This should trigger the following output. NPM is working magic behind the scenes.

Try typing make-thumbs in the command line one more time.

Tada! 😁

Note that this link can be undone via npm unlink. On a Windows machine you can check "~\AppData\Roaming\npm to see that NPM has created a .cmd file corresponding to the command name. ~ refers to C:\Users\your-user-name aka the HOME directory. This information is not crucial but nice to know.

Now this project is setup and we can add some useful code.

index.js

#!/usr/bin/env node

const program = require('commander')

program
  .version('1.0.0')
  .name('make-thumbs')
  .description('An image resizer to make thumbnails')
  .option('-s,--source [folder]', 'Source images directory', 'images')
  .option(
    '-d,--destination [folder]',
    'Directory to be created for thumbnails',
    'thumbnails'
  )
  .parse(process.argv)

Commander is a great framework and helps to set up options and produces help menus automatically. Here I am assigning a version, name and description, as well as some options. Finally, we are parsing process.argv. These are the arguments provided to the command. With just this code we already have a working command line tool.

make-thumbs --help

The options allow the input of a directory of source images and a directory to save the new thumbnails. These are relative to the current working directory and not absolute paths. I have created aliases -s, normal options - --source, descriptions and defaults. Now the filesystem and image manipulation logic can be added.

I want to create a separate folder and file to hold some of this logic to keep things organized and modular.

mkdir lib
cd lib
touch index.js

I want to take advantage of Async/Await code so I am using promisify. These utilities help to read directories, make directories, remove directories and check if directories exist. Consult the Node documentation for more information on these. I have also including the Jimp logic to create a thumbnail to our specs. Consult their documentation more information.

lib/index.js

const jimp = require('jimp')
const rimraf = require('rimraf')

const fs = require('fs')
const { promisify } = require('util')

const thumbnail = async (src, dest) => {
  const image = await jimp.read(src)
  await image.resize(225, 190, jimp.RESIZE_BICUBIC)
  image.quality(40)
  await image.writeAsync(dest)
}

const directoryExists = filepath => {
  return fs.existsSync(filepath)
}

const readdir = promisify(fs.readdir)
const mkdir = promisify(fs.mkdir)
const rm = promisify(rimraf)

module.exports = {
  thumbnail,
  directoryExists,
  readdir,
  mkdir,
  rm
}

Here is the finished code for index.js with our utilities imported.

index.js

#!/usr/bin/env node

const program = require('commander')
const path = require('path')

const { thumbnail, directoryExists, readdir, mkdir, rm } = require('./lib')

program
  .version('1.0.0')
  .name('make-thumbs')
  .description('An image resizer to make thumbnails')
  .option('-s,--source [folder]', 'Source images directory', 'images')
  .option(
    '-d,--destination [folder]',
    'Directory to be created for thumbnails',
    'thumbnails'
  )
  .parse(process.argv)

const main = async () => {
  try {
    // Use current working dir vs __dirname where this code lives
    const cwd = process.cwd()

    // Use user input or default options
    const { source, destination } = program
    const srcPath = path.join(cwd, source)
    const destPath = path.join(cwd, destination)

    // Remove destination directory is it exists
    if (directoryExists(destPath)) {
      await rm(destPath)
    }

    // Create destination directory
    await mkdir(destPath)

    // Read source directory
    const imagesAll = await readdir(srcPath)

    // Create thumbnails
    for (let image of imagesAll) {
      const src = path.join(srcPath, image)
      const dest = path.join(destPath, image)
      console.log(`Creating thumbnail at: ${dest}`)
      thumbnail(src, dest)
    }

    console.log('Thumbnails created successfully!')
  } catch (error) {
    console.log('Error creating thumbnails.')
  }
}

main()

All of our logic is placed inside the main function which is executed at the bottom of the code. Within main there is a try/catch block. This is helpful to control any error messages that may occur. With this structure the message inside the catch block is all the user will see if there is an error. This can be customized to any message desired, including part or all of the actual error thrown. In development you can simply log the error here to troubleshoot.

One important aspect is the use of process.cwd(). This command line application works based on the directory level the user is located in. This is used to create the paths to the source and destination folders. The destination folder is deleted and created again if it exists. Then the contents of the source directory are read into memory. Finally, these files are looped over and a thumbnail is created for each image and saved into the destination folder. I added some logs to give a sense of the program working. These can be removed or even replaced with some sort of progress logic. It is all up to you!

To make sure everything is working I have included a test directory in the source files. To test functionality do the following.

cd test
make-thumbs

Everything should be working! I made a quick video to demonstate the application in action.

What is my Job Title

benjaminadk — Tue, 31 Dec 2019 04:01:46 +0000

So I have a degree in Software Development and had been working part-time designing and developing, mostly JavaScript SPA, websites for small businesses and individuals. I was working on a portfolio and hoping to land a full-time gig as a JavaScript(React/Node) Developer but I had trouble due to lack of professional experience. Funds started to get tight so I expanded my job search and ended up working for an eCommerce site. Here is information about the job.

WordPress w/Woocommerce - site not developed by me - I like JS, learning PHP now though
I do tweaks with CSS, configure settings/plugins report to a off-site dev with problems questions
I designed a system to acquire/store/manipulate/transfer data
I FTP or web scrape for data
I keep master spreadsheets in Excel - most time is spent in Excel
I take the raw incoming data and processes it into lots of (maybe 50) properties
lots of formulas to manipulate data - product linking, attributes for filtering/searching, too many product details - most thought provoking part of job - vlookups, index, match all day
keep product up to date, accurate - 150000 total products last I looked
download images optimize/size them with photoshop - automations for various views
upload image to image server

This wasn't exactly what I was expecting and I am the only tech person in a small office. I am hesitant to just put Data Entry, although that is what I do - for some reason my ego says my job is more technical than that - if its not I can probably accept that. I know I am not a web developer in this role - part of me wishes I went down the WP/PHP route after I found out how much they paid for site design. Maybe I am a Web Scraper, but that sounds kind of weird and doesn't really sum it all up.

Please chime in. Thanks in advance.

UPDATE

My boss is hilarious. He said I can call the position whatever I want and he has my back. We did entertain Ecommerce Data Management Specialist for a little bit.

Encode GIFs With Node

benjaminadk — Sun, 25 Aug 2019 22:07:57 +0000

When you want to convey as message, but an image is too simplistic and a video is too complex, a GIF can be the perfect middle ground. As a JavaScript developer, I recently wondered:

Could I write a program to create a GIF?
Could JavaScript even do this?

After a little research and a lot of trial and error, I found the answer to both question is yes. This article sums up what I found out.

The GIF Format

A good starting point is to research some of the history and structure of a GIF. It turns out the Graphics Interchange Format has was originally created by CompuServe back in the 1980s and was one of the first image formats used on the web. While the PNG format has pretty much replaced GIF for single images, GIF's ability to animate a series of images keeps the format relevant and supported today. In GIFs as we know them today, each image is allowed a maximum palette size of 256 colors. This limitation is why GIFs are more suited to illustrations rather than photography, even though they are used for both. GIF images are also compressed using the LZW algorithm, which provides lossless data compression. For more general information, Wikipedia is a great source, and for an in-depth breakdown of the entire specification, check out What's In a GIF.

My Use Case

I have been playing around with Electron a lot lately and I decided to attempt a desktop application that could record the user's screen and then turn the captured images into a GIF. The Electron environment combines the features of the browser, the the features of Node, and Electron's own APIs. Electron's desktopCapturer API makes capturing the user's screen a frame at a time and then saving those images to disk possible. Having these sequencial images is essential to this approach to GIF encoding. My project article GifIt goes into more detail on that subject, and the GifIt Source Code is available if you want to check out how I went about recording the desktop. At this point, my goal became to write my own library for GIF encoding.

Existing Libraries

The next step I took was to look into existing libraries on NPM and Github. There are a few options, and which one you use depends a lot of your use case and the available documentation. It looks like the original implementation in JavaScript was gif.js. I poked around the files and was happy to find that the LZWEncoder and NeuQuant algorithms had already been ported. I used these as building blocks for my library.

My Library

One thing I noticed about existing libraries was that GIFs took a long time to process and the size of the output files seemed really large. GIF Encoder 2 adds new features to help mitigate these downsides. The first thing I did was add an optional optimizer. I discoved that a lot of time was being spent reducing an image into its 256 color palette. This process involves looking at the color of every pixel in an image and was being done by the NeuQuant algoritm. I added the ability to reuse the palette from the previous image if the current and previous image were similar. Checking this adds overhead, but not nearly as much overhead as calculating a new color palette. I also added a second algorithm called Octree that uses a totally different method to calculate the color palette. This ended up resulting in smaller smaller file sizes.

Using Gif Encoder 2



npm install gif-encoder-2

Constructor

GIFEncoder(width, height, algorithm, useOptimizer, totalFrames)

Parameter	Type	Description	Required	Default
`width`	number	the width of images in pixels	yes	n/a
`height`	number	the height of images in pixels	yes	n/a
`algorithm`	string	`neuquant` or `octree`	no	`neuquant`
`useOptimizer`	boolean	enables/disables optimizer	no	false
`totalFrames`	number	total number of images	no	0



const encoder = new GIFEncoder(500, 500)
const encoder = new GIFEncoder(1200, 800, 'octree', false)
const encoder = new GIFEncoder(720, 480, 'neuquant', true, 20)

Methods

Method	Parameter	Description
`start`	n/a	Starts the encoder
`addFrame`	`Canvas Context`	Adds a frame to the GIF
`setDelay`	`number`	Number of milliseconds to display frame
`setFramesPerSecond`	`number`	Number of frames per second to display
`setQuality`	`number 1-30`	Neuquant quality
`setThreshold`	`number 0-100`	Optimizer threshold percentage
`setRepeat`	`number >= 0`	Number of loops GIF does
`finish`	n/a	Stops the encoder

Basic Example

This example creates a simple GIF and shows the basic way Gif Encoder 2 works.

Create an instance of GIFEncoder
Call any needed set methods
Start the encoder
Add frames as Canvas context
Get the output data and do something with it



const GIFEncoder = require('gif-encoder-2')
const { createCanvas } = require('canvas')
const { writeFile } = require('fs')
const path = require('path')

const size = 200
const half = size / 2

const canvas = createCanvas(size, size)
const ctx = canvas.getContext('2d')

function drawBackground() {
  ctx.fillStyle = '#ffffff'
  ctx.fillRect(0, 0, size, size)
}

const encoder = new GIFEncoder(size, size)
encoder.setDelay(500)
encoder.start()

drawBackground()
ctx.fillStyle = '#ff0000'
ctx.fillRect(0, 0, half, half)
encoder.addFrame(ctx)

drawBackground()
ctx.fillStyle = '#00ff00'
ctx.fillRect(half, 0, half, half)
encoder.addFrame(ctx)

drawBackground()
ctx.fillStyle = '#0000ff'
ctx.fillRect(half, half, half, half)
encoder.addFrame(ctx)

drawBackground()
ctx.fillStyle = '#ffff00'
ctx.fillRect(0, half, half, half)
encoder.addFrame(ctx)

encoder.finish()

const buffer = encoder.out.getData()

writeFile(path.join(__dirname, 'output', 'beginner.gif'), buffer, error => {
  // gif drawn or error
})

beginner.gif

Advanced Example

This example creates a reusable function that reads a directory of image files and turns them into a GIF. The encoder itself isn't as complicated as the surrounding code.

Note that setDelay can be called once (sets all frames to value) or once per frame (sets delay value for that frame).

Obviously, you can use any directory and filenames you want if you recreate the following example.

Read a directory of images (gets the path to each image)
Create an Image to find the dimensions
Create a write stream to an output gif file
Create an instance of the GIFEncoder
Pipe the encoder's read stream to the write stream
Call any needed set methods
Start the encoder
Draw each image to a Canvas
Add each context to encoder with addFrame
When GIF is done processing resolve1() is called and function is done
Use this function to compare the output of both NeuQuant and Octree algorithms



const GIFEncoder = require('gif-encoder-2')
const { createCanvas, Image } = require('canvas')
const { createWriteStream, readdir } = require('fs')
const { promisify } = require('util')
const path = require('path')

const readdirAsync = promisify(readdir)
const imagesFolder = path.join(__dirname, 'input')

async function createGif(algorithm) {
  return new Promise(async resolve1 => {
    const files = await readdirAsync(imagesFolder)

    const [width, height] = await new Promise(resolve2 => {
      const image = new Image()
      image.onload = () => resolve2([image.width, image.height])
      image.src = path.join(imagesFolder, files[0])
    })

    const dstPath = path.join(__dirname, 'output', `${algorithm}.gif`)

    const writeStream = createWriteStream(dstPath)

    writeStream.on('close', () => {
      resolve1()
    })

    const encoder = new GIFEncoder(width, height, algorithm)

    encoder.createReadStream().pipe(writeStream)
    encoder.start()
    encoder.setDelay(200)

    const canvas = createCanvas(width, height)
    const ctx = canvas.getContext('2d')

    for (const file of files) {
      await new Promise(resolve3 => {
        const image = new Image()
        image.onload = () => {
          ctx.drawImage(image, 0, 0)
          encoder.addFrame(ctx)
          resolve3()
        }
        image.src = path.join(imagesFolder, file)
      })
    }
  })
}



createGif('neuquant')
createGif('octree')

NeuQuant

Octree

Alternative Encoding Method

While Gif Encoder 2 is reliable and can encode GIFs faster than other existing libraries, I did find one alternative that works better but requires the FFmpeg stream processing library to be installed on the host machine. FFmpeg is a command line tool, but can be executed by Node using the child_process API. When I was creating GifIt I added the ability to adjust the duration of each frame in the GIF. Imagine a user wants to display a title page for 5 seconds before running through the rest of the frames or wants to cut the duration of certain frames by half. In order to accomadate these variable durations FFmpeg requires a text file describing the path and duration of each image. The duration is in seconds and the paths are relative.

example from FFmpeg Docs



file '/path/to/dog.png'
duration 5
file '/path/to/cat.png'
duration 1
file '/path/to/rat.png'
duration 3
file '/path/to/tapeworm.png'
duration 2
file '/path/to/tapeworm.png'

This is a simplifed version of the function I used in GifIt.

images is an object that contains the absolute path and duration of the frame
dstPath is the destination to save the output GIF file
cwd is the absolute path of the current working directory (image files must be here as well)
ffmpegPath is the absolute path to the FFmpeg executable on the host machine
the path to the last image is added twice to ensure thhe GIF loops correctly



import { execFile } from 'child_process'
import fs from 'fs'
import path from 'path'
import { promisify } from 'util'

const writeFile = promisify(fs.writeFile)

export const createGif = async (images, dstPath, cwd, ffmpegPath) => {
  return new Promise(resolve => {
    let str = ''
    images.forEach((image, i) => {
      str += `file ${path.basename(image.path)}\n`
      str += `duration ${image.duration}\n`
    })
    str += `file ${path.basename(images[images.length - 1].path)}`
    const txtPath = path.join(cwd, 'template.txt')
    writeFile(txtPath, str).then(() => {
      execFile(
        ffmpegPath,
        [
          '-f',
          'concat',
          '-i',
          'template.txt',
          '-lavfi',
          'palettegen=stats_mode=diff[pal],[0:v][pal]paletteuse=new=1:diff_mode=rectangle',
          dstPath
        ],
        { cwd },
        (error, stdout, stderr) => {
          if (error) {
            throw error
          } else {
            resolve()
          }
        }
      )
    })
  })
}

Best of luck creating your GIFs!!! Hit me up if you have any questions.

Basketball Stats Through D3 & React

benjaminadk — Tue, 25 Jun 2019 06:01:30 +0000

Make a Donut Chart to visualize the scoring totals of the 2018-19 Los Angeles Lakers basketball team.

Data

The first thing we need to create our data visualization is, not coincidentally, data. This well written article explains some of the legal and ethical ramifications of web scraping. This repository offers links to free public data. Dev.to itself has many articles on data, web scrapers, and visualizations. My two cents is that for simple data visualizations projects, good old Chrome Devtools on its own is more than enough to gather and shape data. Check out this over-simplified example.

Name	Age
LeBron James	34
Zion Williamson	18
Micheal Jordan	56

Given the above table here are the steps to massage the data:

Open Chrome Devtools
Isolate all table rows
Convert results from a NodeList to an Array and ditch the title row
Extract text from each table data cell and map the results to a new array of objects
Type c (the variable name) and press Enter and your new array will be displayed in the console
Right click the array and chose Store as Global Variable. You will see temp1 appear in the console.
Use the built in copy function to copy the temporary variable to the clipboard - copy(temp1)
Paste your data into a JavaScript or JSON file.
🤯



var a = document.querySelectorAll('tr') // 2
var b = Array.from(a).slice(1) // 3
var c = b.map(el => {
  // 4
  var name = el.children[0].innerText
  var age = el.children[1].innerText
  return { name, age }
})

c // 5
// right click array
copy(temp1) // 7

Note that every scenario is different and this example is simplified to help explain the process. Also, all of the logic above can be put into a single function to streamline the process. Remember you can create multi-line functions in the console by using Shift+Enter to create new lines. With this method we have what amounts to manual web scraping with JavaScript 101. Be sure to read a website's Terms of Service before going willy-nilly and harvesting data where you aren't supposed to.

Create a Donut Chart

Getting D3 and React to work together is not really that complicated. Generally, all that is needed is an entry point to the DOM and some logic that initializes the visualization when the page loads. To get started with our example project we want to have create-react-app installed. The first step is to create a new project. The first thing I like to do is clear out the src directory, leaving only App.js and index.js. Don't forget to remove any old import statements. Before we write any code we need to snag a couple dependencies.

1- Download D3 and Styled Components.



npm i d3 styled-components

2- Create a new file whatever-you-want.js, or even data.js in the src directory. The data used in the example is available in this gist.

3- Create some basic boilerplate that can be used for a variety of projects with this configuration - aka D3 + React + Styled Components. I encourage you to tweak whatever you see fit as like most developers I have my own quircks and patterns. Case in point, I am bothered by #000000 black so I use #333333, I like the font Raleway, etc. If you haven't used Hooks before, the useEffect hook with an empty [] dependency array is similar to componentDidMount in a React class component. The numbered comments correspond to upcoming steps and are the place to insert the code from those steps.



import React, { useRef, useEffect, useState } from 'react'
import * as d3 from 'd3'
import styled, { createGlobalStyle } from 'styled-components'
import data from './data'

const width = 1000
const height = 600
const black = '#333333'
const title = 'My Data Visualization'

// 4

// 7

export const GlobalStyle = createGlobalStyle`
@import url('https://fonts.googleapis.com/css?family=Raleway:400,600&display=swap');

body {
  font-family: 'Raleway', Arial, Helvetica, sans-serif;
  color: ${black};
  padding: 0;
  margin: 0;
}
`

export const Container = styled.div`
  display: grid;
  grid-template-rows: 30px 1fr;
  align-items: center;
  .title {
    font-size: 25px;
    font-weight: 600;
    padding-left: 20px;
  }
`

export const Visualization = styled.div`
  justify-self: center;
  width: ${width}px;
  height: ${height}px;
// 6
`

export default () => {
  const visualization = useRef(null)

  useEffect(() => {
    var svg = d3
      .select(visualization.current)
      .append('svg')
      .attr('width', width)
      .attr('height', height)
// 5

// 8
  }, [])

  return (
    <>
      <GlobalStyle/>
      <Container>
        <div className='title'>{title}</div>
        <Visualization ref={visualization} />
        {/*10*/}
      </Container>
    <>
  )
}

4- We need to establish a color scheme and some dimensions for our Donut Chart.

The radius of our pastry.



const radius = Math.min(width, height) / 2

It only makes sense to use a Lakers color theme.



var lakersColors = d3
  .scaleLinear()
  .domain([0, 1, 2, 3])
  .range(['#7E1DAF', '#C08BDA', '#FEEBBD', '#FDBB21'])

The D3 pie function will map our data into pie slices. It does this by adding fields such as startAngle and endAngle behind the scenes. We are using an optional sort function just to shuffle the order of the slices. Play around with this, pass it null or even leave it out to get different arrangements. Finally, we use the value function to tell D3 to use the points property to divide up the pie. Log the pie variable to the console to help conceptualize what the D3 pie function did to our data.



var pie = d3
  .pie()
  .sort((a, b) => {
    return a.name.length - b.name.length
  })
  .value(d => d.points)(data)

Now we need to create circular layouts using the arc function. The variable arc is for our Donut Chart and the outerArc will be used as a guide for labels later. getMidAngle is a helper function to be used at a later time also.



var arc = d3
  .arc()
  .outerRadius(radius * 0.7)
  .innerRadius(radius * 0.4)

var outerArc = d3
  .arc()
  .outerRadius(radius * 0.9)
  .innerRadius(radius * 0.9)

function getMidAngle(d) {
  return d.startAngle + (d.endAngle - d.startAngle) / 2
}

5- With a structure in place are almost to the point of seeing something on the screen.

Chain the following to our original svg variable declaration.



   .append('g')
   .attr('transform', `translate(${width / 2}, ${height / 2})`)

Now the magic happens when we feed our pie back to D3.



svg
  .selectAll('slices')
  .data(pie)
  .enter()
  .append('path')
  .attr('d', arc)
  .attr('fill', (d, i) => lakersColors(i % 4))
  .attr('stroke', black)
  .attr('stroke-width', 1)

Next we need to draw lines from each slice that will eventually point to a label. The well named centroid function returns an array with [x,y] coordinates to the center point of the pie slice (in this case d) within the arc. In the end we are returning an array of three coordinate arrays that correspond to the origin point, bend point, and termination point of each line the now appears on the screen. The midAngle helps determine which direction to point the tail of our line.



svg
  .selectAll('lines')
  .data(pie)
  .enter()
  .append('polyline')
  .attr('stroke', black)
  .attr('stroke-width', 1)
  .style('fill', 'none')
  .attr('points', d => {
    var posA = arc.centroid(d)
    var posB = outerArc.centroid(d)
    var posC = outerArc.centroid(d)
    var midAngle = getMidAngle(d)
    posC[0] = radius * 0.95 * (midAngle < Math.PI ? 1 : -1)
    return [posA, posB, posC]
  })

Now our lines are ready for labels. The label seems to look better by adding some symmetry by flip-flopping the order of name and points based on which side of the chart it appears on. Notice that the pie function moved our original data into a key named data. The top level keys of pie objects contain the angle measurements used in the getMidAngle function.



svg
  .selectAll('labels')
  .data(pie)
  .enter()
  .append('text')
  .text(d => {
    var midAngle = getMidAngle(d)
    return midAngle < Math.PI
      ? `${d.data.name} - ${d.data.points}`
      : `${d.data.points} - ${d.data.name}`
  })
  .attr('class', 'label')
  .attr('transform', d => {
    var pos = outerArc.centroid(d)
    var midAngle = getMidAngle(d)
    pos[0] = radius * 0.99 * (midAngle < Math.PI ? 1 : -1)
    return `translate(${pos})`
  })
  .style('text-anchor', d => {
    var midAngle = getMidAngle(d)
    return midAngle < Math.PI ? 'start' : 'end'
  })

6- To polish off our labels with some style we just need to add a couple lines of code to the Visualization styled component. Having used D3 to add a class attribute inside a React useEffect hook and then defining that class using Styled Components seems to check the boxes on integrating the libraries.



.label {
  font-size: 12px;
  font-weight: 600;
}

7- We are looking good but why not add a little more flavor to give the user an interactive feel. We can quickly grab the total amount of points scored using the sum function from D3.



var total = d3.sum(data, d => d.points)

8- The showTotal function will simply tack on a text node displaying our total. The text-anchor style property of middle should center the text within our Donut hole. The hideTotal function will come into play in a bit. Notice we are calling the showTotal function to make sure the text is showing when the page loads.



function showTotal() {
  svg
    .append('text')
    .text(`Total: ${total}`)
    .attr('class', 'total')
    .style('text-anchor', 'middle')
}

function hideTotal() {
  svg.selectAll('.total').remove()
}

showTotal()

We should tack on another class for total right next to our label class from step 6.



.total {
  font-size: 20px;
  font-weight: 600;
}

9- The numbered comment system is a getting a little gnarly at this point, but if you have made it this far you are smart enough to follow along. These next functions can go below hideTotal. These are listeners we will apply to each slice.



function onMouseOver(d, i) {
  hideTotal()
  setPlayer(d.data)
  d3.select(this)
    .attr('fill', d3.rgb(lakersColors(i % 4)).brighter(0.5))
    .attr('stroke-width', 2)
    .attr('transform', 'scale(1.1)')
}

function onMouseOut(d, i) {
  setPlayer(null)
  showTotal()
  d3.select(this)
    .attr('fill', lakersColors(i % 4))
    .attr('stroke-width', 1)
    .attr('transform', 'scale(1)')
}

When a slice is hovered the stroke and fill will be emphasized and a slight scale up will add a cool effect. The total points text will also be toggled so we can stick a tooltip with a little more information smack dab in the hole. First we need to create a piece of state, what would a React app be without it.



const [player, setPlayer] = useState(null)

A keen observer may have noticed the reference to this and wondered what was happening. The following listeners need to be tacked on to the end of the slices D3 chain.



   .attr('class', 'slice')
   .on('mouseover', onMouseOver)
   .on('mouseout', onMouseOut)

Since we are using a transform on the slice class let's control it through another couple lines in the Visualization styled component.



.slice {
  transition: transform 0.5s ease-in;
}

10- We can now create the tooltip to display the player state that changes as individual slices are moused over.



{
  player ? (
    <Tooltip>
      <div>
        <span className='label'>Name: </span>
        <span>{player.name}</span>
        <br />
        <span className='label'>Points: </span>
        <span>{player.points}</span>
        <br />
        <span className='label'>Percent: </span>
        <span>{Math.round((player.points / total) * 1000) / 10}%</span>
      </div>
    </Tooltip>
  ) : null
}

In terms of new information the user is only getting the percentage of the team's points the current player scored. However, with the centralized position combined with the movement a nice effect and a nice feeling of interactivity is created. A similar pattern could be used more effectively if there was more information to show or I was smarter. It seems the last thing needed is the Tooltip component, which goes with the other styled components.



export const Tooltip = styled.div`
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
  width: ${radius * 0.7}px;
  height: ${radius * 0.7}px;
  display: grid;
  align-items: center;
  justify-items: center;
  border-radius: 50%;
  margin-top: 10px;
  font-size: 12px;
  background: #ffffff;
  .label {
    font-weight: 600;
  }
`

Alas, our final code should look something like the following.



import React, { useRef, useEffect, useState } from 'react'
import * as d3 from 'd3'
import data from './data'
import styled, { createGlobalStyle } from 'styled-components'

/**
 * Constants
 */
const width = 1000
const height = 600
const radius = Math.min(width, height) / 2
const black = '#333333'
const title = 'Los Angeles Lakers Scoring 2018-19'

/**
 * D3 Helpers
 */

// total points
var total = d3.sum(data, d => d.points)

// lakers colors
var lakersColors = d3
  .scaleLinear()
  .domain([0, 1, 2, 3])
  .range(['#7E1DAF', '#C08BDA', '#FEEBBD', '#FDBB21'])

// pie transformation
var pie = d3
  .pie()
  .sort((a, b) => {
    return a.name.length - b.name.length
  })
  .value(d => d.points)(data)

// inner arc used for pie chart
var arc = d3
  .arc()
  .outerRadius(radius * 0.7)
  .innerRadius(radius * 0.4)

// outer arc used for labels
var outerArc = d3
  .arc()
  .outerRadius(radius * 0.9)
  .innerRadius(radius * 0.9)

// midAngle helper function
function getMidAngle(d) {
  return d.startAngle + (d.endAngle - d.startAngle) / 2
}
/**
 * Global Style Sheet
 */
export const GlobalStyle = createGlobalStyle`
@import url('https://fonts.googleapis.com/css?family=Raleway:400,600&display=swap');

body {
  font-family: 'Raleway', Arial, Helvetica, sans-serif;
  color: ${black};
  padding: 0;
  margin: 0;
}
`

/**
 * Styled Components
 */
export const Container = styled.div`
  display: grid;
  grid-template-rows: 30px 1fr;
  align-items: center;
  user-select: none;
  .title {
    font-size: 25px;
    font-weight: 600;
    padding-left: 20px;
  }
`

export const Visualization = styled.div`
  justify-self: center;
  width: ${width}px;
  height: ${height}px;
  .slice {
    transition: transform 0.5s ease-in;
  }
  .label {
    font-size: 12px;
    font-weight: 600;
  }
  .total {
    font-size: 20px;
    font-weight: 600;
  }
`

export const Tooltip = styled.div`
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
  width: ${radius * 0.7}px;
  height: ${radius * 0.7}px;
  display: grid;
  align-items: center;
  justify-items: center;
  border-radius: 50%;
  margin-top: 10px;
  font-size: 12px;
  background: #ffffff;
  .label {
    font-weight: 600;
  }
`

export default () => {
  const [player, setPlayer] = useState(null)

  const visualization = useRef(null)

  useEffect(() => {
    var svg = d3
      .select(visualization.current)
      .append('svg')
      .attr('width', width)
      .attr('height', height)
      .append('g')
      .attr('transform', `translate(${width / 2}, ${height / 2})`)

    svg
      .selectAll('slices')
      .data(pie)
      .enter()
      .append('path')
      .attr('d', arc)
      .attr('fill', (d, i) => lakersColors(i % 4))
      .attr('stroke', black)
      .attr('stroke-width', 1)
      .attr('class', 'slice')
      .on('mouseover', onMouseOver)
      .on('mouseout', onMouseOut)

    svg
      .selectAll('lines')
      .data(pie)
      .enter()
      .append('polyline')
      .attr('stroke', black)
      .attr('stroke-width', 1)
      .style('fill', 'none')
      .attr('points', d => {
        var posA = arc.centroid(d)
        var posB = outerArc.centroid(d)
        var posC = outerArc.centroid(d)
        var midAngle = getMidAngle(d)
        posC[0] = radius * 0.95 * (midAngle < Math.PI ? 1 : -1)
        return [posA, posB, posC]
      })

    svg
      .selectAll('labels')
      .data(pie)
      .enter()
      .append('text')
      .text(d => {
        var midAngle = getMidAngle(d)
        return midAngle < Math.PI
          ? `${d.data.name} - ${d.data.points}`
          : `${d.data.points} - ${d.data.name}`
      })
      .attr('class', 'label')
      .attr('transform', d => {
        var pos = outerArc.centroid(d)
        var midAngle = getMidAngle(d)
        pos[0] = radius * 0.99 * (midAngle < Math.PI ? 1 : -1)
        return `translate(${pos})`
      })
      .style('text-anchor', d => {
        var midAngle = getMidAngle(d)
        return midAngle < Math.PI ? 'start' : 'end'
      })

    function showTotal() {
      svg
        .append('text')
        .text(`Total: ${total}`)
        .attr('class', 'total')
        .style('text-anchor', 'middle')
    }

    function hideTotal() {
      svg.selectAll('.total').remove()
    }

    function onMouseOver(d, i) {
      hideTotal()
      setPlayer(d.data)
      d3.select(this)
        .attr('fill', d3.rgb(lakersColors(i % 4)).brighter(0.5))
        .attr('stroke-width', 2)
        .attr('transform', 'scale(1.1)')
    }

    function onMouseOut(d, i) {
      setPlayer(null)
      showTotal()
      d3.select(this)
        .attr('fill', lakersColors(i % 4))
        .attr('stroke-width', 1)
        .attr('transform', 'scale(1)')
    }

    showTotal()
  }, [])

  return (
    <>
      <GlobalStyle />
      <Container>
        <div className='title'>{title}</div>
        <Visualization ref={visualization} />
        {player ? (
          <Tooltip>
            <div>
              <span className='label'>Name: </span>
              <span>{player.name}</span>
              <br />
              <span className='label'>Points: </span>
              <span>{player.points}</span>
              <br />
              <span className='label'>Percent: </span>
              <span>{Math.round((player.points / total) * 1000) / 10}%</span>
            </div>
          </Tooltip>
        ) : null}
      </Container>
    </>
  )
}

NBA Player Salaries & Performance 2018-19 (Bubble Chart)

Inspiration for example Donut Chart

👩‍🎓 Open Source Exam Simulator

benjaminadk — Tue, 12 Mar 2019 17:25:57 +0000

Introduction ◽ Exam Simulator ◽ Exam Maker ◽ Moving Forward

Introduction

The last class I needed to pass before snagging my college degree entailed an Oracle Database Certification. The study material provided by the school was not enough to prepare students as evidenced by a course chatroom full of people complaining about failing miserably. I too failed my first attempt by 4 points 🤯. It turns out that the best way to study for these things is to find VCE files and a simulator to run them on.

The downside to VCE is that it often costs 💲💲💲 for the files, the simulator, or some sort of membership. Since I was majoring in Software Development I decided to make my own open source solution using JavaScript. As of today, everything is in alpha and I could use some feedback, exam creators or contributors to help move the project forward.

Exam Simulator

The simulator portion of the project is built with Electron and available for Windows and MacOS. I have not used/tested the MacOS version yet. Data for the simulator comes from JSON files stored on the user's local machine. I have built features including an exam timer, 4 question types (multiple choice, multiple answer, fill in & drag and drop list), exam review reports, saved exam sessions, exam validation and more.

Exam Maker

Even though exams can be created in any text editor I thought it would be a good idea to make online exam editor to create exam files and to act as a central repository for all exams. This site is a Next.js app with an Node.js backend and a Prisma database. Its up on Heroku on the free hobby-tier so it is sort of slow to boot. Users creating and sharing tests is what would make the app have value.

Moving Forward

Who knows. No one might ever use this thing 🤦‍♂️, but at least its fun to work on. Check out the following resources if you are interested. There is no shortage of bugs, performance optimizations to make and features to create.

Exam Simulator Download

Exam Simulator Docs

Exam Maker on Heroku

exam-simulator / simulator

👩‍🎓 JSON based exam simulator.

Exam Simulator JS

JSON based, open source and free exam simulator.

Documentation

Notes

This project is in alpha release. Stay tuned for lots of improvements and updates.

Donate

The app is free, but I won't turn down a donation. 🤑

View on GitHub

exam-simulator / maker-frontend

Create, edit and share exams for Exam Simulator. Built with Next.js.

Exam Maker Frontend

View on GitHub

exam-simulator / maker-backend

Apollo Express server for Exam Maker.

Exam Maker Backend

View on GitHub

Free Speech on Twitter

benjaminadk — Wed, 06 Mar 2019 02:08:37 +0000

Twitter boss Jack Dorsey and Vijaya Gadde, who oversees content moderation sit down of Joe Rogan podcast. Tim Cook is there as well. Many topics are discussed. Food for thought.

Color Harmonies in JavaScript

benjaminadk — Thu, 28 Feb 2019 05:25:57 +0000

Intro

I recently designed a color utility application using Electron. Electron is a JavaScript framework for building cross-platform desktop apps. I learned that when it comes to coding, color and math are joined at the hip. This article summarizes some of my takeaways.

benjaminadk / color-tool-remix

Color Tool Remix 🎨 A cross platform color utility app

Color Tool Remix

An HSL color picker, dropper, color analyzer and palette generator. The do it all color tool.

🧐 Looking for a contributor to help testing on macOS.

Installation
Usage
- Picker
- Dropper
- Palettes
- Generator
- Project Mode
- User Interface
- Options
Tech Stack
Changelog
Contributors
Donate

Installation

Latest Release

Platform	Auto Updates	Exexutable
Windows	yes	`electron-picker-remix-setup-[VERSION].exe`
macOS	no	`electron-picker-remix-setup-[VERSION].dmg`
Linux	N/A	N/A

System warnings are normal due to lack of Code Signing

Usage

Picker

Create a color quickly, via a variety of input methods

Color Bars
- Adjust Hue, Saturation, Lightness and Alpha Color Bars
Fine Tuning
- Adjust Hue, Saturation, Lightness and Alpha Input Values
Color String Parser
- Parses hsl, rgb, hex and css named colors
Random Color
- Generates a random color excluding white, blacks and greys
Dropper
- Select any pixel from the screen
Generators
- Mathematically generate colors from one base color

Dropper

Select a single pixel from…

View on GitHub

Formats

There is no shortage of color formats floating around. In CSS alone we can chose between 4 formats. Note that 3 of these formats also have an optional alpha, or opacity setting. To learn about the touching story of rebeccapurple, checkout this short article.

Format	Abbreviation	Example
Named Colors	n/a	`rebeccapurple`
Hexadecimal	HEX	`#663498`
Red, Green, Blue	RGB	`rgb(102,52,152)`
Hue, Saturation, Lightness	HSL	`hsl(270,49%,40%)`

In my day-to-day work I tend to favor HEX or RGB over the others. However, when designing Color Tool I used HSL as my base format.

HSL Benefits

I found that HSL was the most human-understandable color format. Why? Take a closer look at each component of this format. Hue is a number between 0 and 360, and represents degrees on a color wheel. In no other format does one value give us some much information about what the final color might look like. Red is at 0/360. Saturation is the amount of color vs grey, with 100% being all color and no grey and 0% being all grey and no color. Lightness is another scale between 0 and 100. 0 is totally black and 100 is totally white, therefore 50 is the happy medium.

Parsing HSL String

In order to work with HSL values we need to be able to take an initial string value, such as hsl(270, 49%, 40%), and break it down into individual H, S, L, and/or A pieces.



function parseHSL(str) {
  var hsl, h, s, l
  hsl = str.replace(/[^\d,]/g, '').split(',')   // strip non digits ('%')  
  h = Number(hsl[0])                            // convert to number
  s = Number(hsl[1])
  l = Number(hsl[2])
  return [h, s, l]                              // return parts
}

Color Harmonies

Color Harmonies are combinations of colors that relate to each other mathematically. Some of the more commonly used harmonies include Complementary, Split Complementary, Triadic, Tetradic and Analogous. Lets take it back to rebeccapurple to see what these look like.

Harmony	Colors	Wheel
complementary
split complementary
triadic
tetradic
analogous

Finding all of these values boils down to math. Using HSL is handy here. We are mostly concerned with the Hue in these calculations. When finding harmonies is these cases we will leave the Saturation and Lightness unchanged. To gain a visual understanding of where on the color wheel harmonies lie take a look at the small icons in the table. These are set patterns and if we know the Hue of our base color and that a circle has 360 degrees we can find our harmonies. For example, a colors complement is directly on the opposite side, or 180 degrees away.



const complementaryHue = (baseHue + 180) % 360

Split complementary produces two colors from the complement +- 30 degrees. At this point I start to think about a function that can work in real life. This function needs to take an actual color string e.g. hsl(34, 100%, 50%) and return an array with all colors in the harmony. We won't deal with other color formats that require conversion since that is another article itself. We already have our parseHSL function above so we are off to a good start. The key to this function is that we can look at the split complement as the baseHue + 150 and the baseHue + 210. In fact all the multiple color harmonies start at a base value(in this case 150) and make x more colors (in this case 1) at a set interval (in this case 60..gets us to 210😱).

So here we go.



const rp = 'hsl(270, 49%, 40%)'

function harmonize(color, start, end, interval) {
    const colors = [color]
    const [h, s, l] = parseHSL(color)

    for(let i = start; i <= end; i += interval) {
        const h1 = (h + i) % 360
        const c1 = `hsl(${h1}, ${s}%, ${l}%)`
        colors.push(c1)
    }

    return colors
}

const complement = harmonize(rp, 180, 180, 1)
const split = harmonize(rp, 150, 210, 60)
const triad = harmonize(rp, 120, 240, 120)
const tetrad = harmonize(rp, 90, 270, 90)
const analogous = harmonize(rp, 30, 90, 30)

Now that we armed with some code we can adapt it as an extra feature to spruce up a color picker app.

There are all kinds of interesting functionality combining math and color. A lot of it isn't as complicated as it might seem. I have been playing around with the app in the GIF and I am looking to attract contributors to help with testing and improving the macOS version and improve the product overall. Thanks.👌

How do I create thumbnails when I upload a video? aws lambda!

benjaminadk — Sat, 23 Feb 2019 10:09:54 +0000

Good question. 🤔

Introduction

I have been playing around with a YouTube clone I call FooTube. I had set up video uploads to be sent from the browser to an AWS S3 bucket, so the video file did not touch my node backend This made server-side video processing a non-starter. This put me in a dilemma because I wanted to generate 3 thumbnails for each video upload like the real YouTube does. I started thinking about creating a video player off-screen and using canvas to stream things around. While that might be possible, it didn't sound like fun, and thats not what I ended up doing.

The research began.

I discovered that YouTube uses deep neural networks to pick out thumbnails that display a subject or a face or something else that draws attention. They also capture a thumbnail for every second of video and use an algorithm to rank each one. This interesting article written by the YouTube Creator team from 2015 explains further. At this point I decided that just getting 3 thumbnail images would enough of a challenge for me - since I still had no clue what I was doing. 🤦‍♂️

Companion Video

Disclaimer

Please keep in mind this code is NOT meant to be a production ready solution, it is more an exploration or proof of concept. There are a lot of moving parts, and while I have managed to get this working in my local environment, I simply cannot guarantee it will work anywhere else! Sorry.

Lambda Functions

The first thing I found out was that I could use AWS Lambda to sort of outsource computations that might normally take place on a server. As a bonus, since I was already using S3, I could attach what amounts to an event listener to trigger my Lambda function when I uploaded a video file.

Creating a new Lambda function is straightforward. When prompted you want to chose create a function from scratch and come up with a decent name; createThumbail worked for me. Also, select the Node.js 8.10 runtime.

IAM Role Permissions

I had to create a new IAM role to execute this function. This can be done through a simple work flow in the IAM console. Name the role whatever you want but give it the AWSLambdaExecute permission. This will allow for PUT and GET access to S3 and full access to CloudWatch Logs. These are all the permissions we need to execute and monitor our createThumbnail Lambda function. I had to add the arn for this role to my bucket policy.



        {
            "Sid": "Stmt**************",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::**********:role/LambdaRole"
                ]
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::bucket/*"
        }

Triggers

Next we need to configure the trigger for our function. We want to listen to the bucket we are uploading videos to and watch for the PUT method since that is the method used to send the video. Optionally, you can set a prefix and/or suffix to narrow down the trigger. My function saves the thumbnails to this same bucket. In this case you might use a suffix of mp4 or webm (video formats). My videos were going to the user folder so I set a prefix of user/ since this would be at the beginning of any key.

Once your function is created and its trigger configured, these settings will show up in the S3 bucket referenced by said trigger. In fact they can be set from either S3 or Lambda consoles. Click the Properties tab then the Events box in the S3 console to view events associated with a bucket.

Getting Code to Lambda

There are a few ways to get code into our Lambda function. AWS provides a online code editor if your package size less than 3MB. You can also upload a package in the form of a zip file directly to Lambda or upload a zip file to S3 and then link that to your function. This zip format allows multiple files to be included in your bundle, including typical node_modules dependencies as well as executable files.

In fact, we are going to utilize a couple executable files to help process our video. ffmpeg is a command line tool to convert multimedia files and ffprobe is a stream analyzer. You might have these tools installed locally but we need to use static builds on Lambda. Download choices can be found here. I chose https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz. To unpack the compressed contents I used 7-Zip. Once unpacked we want to isolate the files ffmpeg and ffprobe, go figure.

Note that user,group and global all have read/execute permissions. I am on Windows and had a problem keeping these permissions. Lambda permissions are a little tricky, and global read is important for all files. On Windows the problem arose when I attempted the next step.

To get our executable files to Lambda we could but them into a directory with our index.js (the actual function script) then zip and upload that. There are a couple downsides to this. On Windows zipping the executable files in Windows Explorer stripped the permissions and caused errors when attempting to invoke the executable files my function. Also, every time I made a change in my script I had to re-upload a 40MB file. This is horribly slow and consumes data transfer credit. Not ideal for development and data transfer can cost 💲. The first part of the solution to this problem is to use a Lambda Layer.

Lambda Layers

A Lambda Layer can hold additional code in the form of libraries, custom runtimes or other dependencies. Once we establish a Layer it can be used in multiple functions and can be edited and saved in multiple versions. Very flexible.

First, we need to place our ffmpeg and ffprobe files into a folder called nodejs - the name is important. I ended up using Windows Subsystem for Linux and the zip command to compress the nodejs folder. This was the easiest way I found to preserve the proper permissions.

From the parent directory of our nodejs folder, I run:



zip -r ./layer.zip nodejs

The -r is to recursively zip the contents of nodejs into a new file called layer.zip.

From the Lambda console click the Layers tab and create a new layer. When you create your Layer make sure to set Node.js 8.10 as a compatable runtime. Now you can go back to the function configuration and add our new Layer to createThumbnail.

Finally, we get to the code. 😲

Disclaimer

If someone sees anything that could be better here please comment and let me know. It took me a while to cobble all these ideas together from various corners of the net and this is the first time I have used Lambda. What I'm saying is I am no expert, but finding an article like this when I started would have been helpful.

Code

Since we took the time to set up a Layer and our code has no other dependencies we can type our code directly into the inline editor. I made my local copy in VSCode just to have a my preferred editor settings, then copy and pasted.

First we need to require some of the stuff we need. The aws-sdk is available in the environment. child_process and fs are Node modules.



const AWS = require('aws-sdk')
const { spawnSync, spawn } = require('child_process')
const { createReadStream, createWriteStream } = require('fs')

spawn and spawnSync will allow us to run our executable files from within the Node environment as child processes.

The Lambda environment provides a /tmp directory to use as we wish. We will stream our image data from ffmpeg into /tmp and then read from there when we upload our thumbnails.

Now we can define some variables we will use later.



const s3 = new AWS.S3()
const ffprobePath = '/opt/nodejs/ffprobe'
const ffmpegPath = '/opt/nodejs/ffmpeg'
const allowedTypes = ['mov', 'mpg', 'mpeg', 'mp4', 'wmv', 'avi', 'webm']
const width = process.env.WIDTH
const height = process.env.HEIGHT

We create our S3 instance to interact with our bucket. Since we are using a Layer the paths to our executable files are located in the /opt/nodejs directory. We define an array of allowed types. Settings for width and height can be set as environment variables from the Lambda console. I used 200x112.

Our actual function is written in standard Node format and must be called handler. A custom name can be set in the console.



module.exports.handler = async (event, context) => {
  const srcKey = decodeURIComponent(event.Records[0].s3.object.key).replace(/\+/g, ' ')
  const bucket = event.Records[0].s3.bucket.name
  const target = s3.getSignedUrl('getObject', { Bucket: bucket, Key: srcKey, Expires: 1000 })
  let fileType = srcKey.match(/\.\w+$/)

  if (!fileType) {
    throw new Error(`invalid file type found for key: ${srcKey}`)
  }

  fileType = fileType[0].slice(1)

  if (allowedTypes.indexOf(fileType) === -1) {
    throw new Error(`filetype: ${fileType} is not an allowed type`)
  }

    // to be continued
}

We will make our function async so we can compose our asynchronous code in a way that appears synchronous. First we parse the srcKey from the event passed in from Lambda. This is the filename of our video without the bucket url. We also grab the bucket name. We can save our images to the same bucket as our video if we set our event listener up such that our function won't fire when they are uploaded. We then isolate the file extension and run some checks to make sure it is valid before continuing.



// inside handler function

  const ffprobe = spawnSync(ffprobePath, [
    '-v',
    'error',
    '-show_entries',
    'format=duration',
    '-of',
    'default=nw=1:nk=1',
    target
  ])

  const duration = Math.ceil(ffprobe.stdout.toString())

Here we use spawnSync to run ffprobe and get the duration of the video from the stdout. Use toString because the output is Buffered. By having the duration we can capture our thumbnails in a targeted way throughout the video. I thought taking a thumbnail at 25%, 50% and 75% was a reasonable way to go about getting 3. Of course, with the following functions you can take as many thumbnails as needed. ffprobe can also report much more data than duration, but that is all we are concerned with here.



  function createImage(seek) {
    return new Promise((resolve, reject) => {
      let tmpFile = createWriteStream(`/tmp/screenshot.jpg`)
      const ffmpeg = spawn(ffmpegPath, [
        '-ss',
        seek,     
        '-i',
        target,   
        '-vf',
        `thumbnail,scale=${width}:${height}`,
        '-qscale:v',
        '2',
        '-frames:v',
        '1',
        '-f',
        'image2',
        '-c:v',
        'mjpeg',
        'pipe:1'  
      ])

      ffmpeg.stdout.pipe(tmpFile)

      ffmpeg.on('close', function(code) {
        tmpFile.end()
        resolve()
      })

      ffmpeg.on('error', function(err) {
        console.log(err)
        reject()
      })
    })
  }

There is a lot going on here. The function takes a seek parameter. With this in place we can enter Math.round(duration * .25) for example. The -ss flag followed by time in seconds will seek the video to this spot before taking our thumbnail. We reference target which is our video file. We specify the dimensions we want to use, the quality, frames and format, then finally we pipe the output into a writeStream that is writing to the /tmp directory. All of this is wrapped in a Promise that resolves when this child_process closes.

Understanding exactly what each ffmpeg input does is mad confusing but the ffmpeg Documentation is decent and there are a lot of forum posts out there as well. The bottom line is we have a reusable function that lets us take a thumbnail whenever we want. It also works well in our async/await flow.



  function uploadToS3(x) {
    return new Promise((resolve, reject) => {
      let tmpFile = createReadStream(`/tmp/screenshot.jpg`)
      let dstKey = srcKey.replace(/\.\w+$/, `-${x}.jpg`).replace('/videos/', '/thumbnails/')

      var params = {
        Bucket: bucket,
        Key: dstKey,
        Body: tmpFile,
        ContentType: `image/jpg`
      }

      s3.upload(params, function(err, data) {
        if (err) {
          console.log(err)
          reject()
        }
        console.log(`successful upload to ${bucket}/${dstKey}`)
        resolve()
      })
    })
  }

Now we write a reusable function that will upload thumbnail images to an S3 bucket. Since I used prefix and suffix filters and I am uploading video files to /user/videos I can just replace videos with thumbnails and my function won't be triggered. You can put in any dstKey and bucket that you want. Again we are wrapping our function in a Promise to help with our async flow.

So our final code might look something like this:



process.env.PATH = process.env.PATH + ':' + process.env['LAMBDA_TASK_ROOT']

const AWS = require('aws-sdk')
const { spawn, spawnSync } = require('child_process')
const { createReadStream, createWriteStream } = require('fs')

const s3 = new AWS.S3()
const ffprobePath = '/opt/nodejs/ffprobe'
const ffmpegPath = '/opt/nodejs/ffmpeg'
const allowedTypes = ['mov', 'mpg', 'mpeg', 'mp4', 'wmv', 'avi', 'webm']
const width = process.env.WIDTH
const height = process.env.HEIGHT
}

module.exports.handler = async (event, context) => {
  const srcKey = decodeURIComponent(event.Records[0].s3.object.key).replace(/\+/g, ' ')
  const bucket = event.Records[0].s3.bucket.name
  const target = s3.getSignedUrl('getObject', { Bucket: bucket, Key: srcKey, Expires: 1000 })
  let fileType = srcKey.match(/\.\w+$/)

  if (!fileType) {
    throw new Error(`invalid file type found for key: ${srcKey}`)
  }

  fileType = fileType[0].slice(1)

  if (allowedTypes.indexOf(fileType) === -1) {
    throw new Error(`filetype: ${fileType} is not an allowed type`)
  }

  function createImage(seek) {
    return new Promise((resolve, reject) => {
      let tmpFile = createWriteStream(`/tmp/screenshot.jpg`)
      const ffmpeg = spawn(ffmpegPath, [
        '-ss',
        seek,
        '-i',
        target,
        '-vf',
        `thumbnail,scale=${width}:${height}`,
        '-qscale:v',
        '2',
        '-frames:v',
        '1',
        '-f',
        'image2',
        '-c:v',
        'mjpeg',
        'pipe:1'
      ])

      ffmpeg.stdout.pipe(tmpFile)

      ffmpeg.on('close', function(code) {
        tmpFile.end()
        resolve()
      })

      ffmpeg.on('error', function(err) {
        console.log(err)
        reject()
      })
    })
  }

  function uploadToS3(x) {
    return new Promise((resolve, reject) => {
      let tmpFile = createReadStream(`/tmp/screenshot.jpg`)
      let dstKey = srcKey.replace(/\.\w+$/, `-${x}.jpg`).replace('/videos/', '/thumbnails/')

      var params = {
        Bucket: bucket,
        Key: dstKey,
        Body: tmpFile,
        ContentType: `image/jpg`
      }

      s3.upload(params, function(err, data) {
        if (err) {
          console.log(err)
          reject()
        }
        console.log(`successful upload to ${bucket}/${dstKey}`)
        resolve()
      })
    })
  }

  const ffprobe = spawnSync(ffprobePath, [
    '-v',
    'error',
    '-show_entries',
    'format=duration',
    '-of',
    'default=nw=1:nk=1',
    target
  ])

  const duration = Math.ceil(ffprobe.stdout.toString())

  await createImage(duration * 0.25)
  await uploadToS3(1)
  await createImage(duration * .5)
  await uploadToS3(2)
  await createImage(duration * .75)
  await uploadToS3(3)

  return console.log(`processed ${bucket}/${srcKey} successfully`)
}

Tips

Lambda allows you to allocate a set amount of memory to your function. I am using 512MB and everything seems to be running well. My function is doing a couple more things that described here and uses around 400MB per invocation.
Utilize the CloudWatch logs and the monitoring graphs provided by AWS. My function averages about 12 seconds per invocation. Note that I have a ton of errors on this graph as I attempted to refactor things(all the green dots at the bottom).

This version of the code has no contact with the application from which the original video is uploaded. Solutions to this are to send a POST request from the Lambda function to your backend when the processing is complete. Another option I found is that adding 20 seconds delay to my video upload gives ample time for the thumbnails to be created. When uploading the video we know where its going so we know the url it will eventually have. Since we are building our thumbnail keys based on the original video key we know what those urls will be as well.



const videoUrl = 'https://s3-us-west-1.amazonaws.com/footube/user/videos/example.mp4'

const imageUrl = 'https://s3-us-west-1.amazonaws.com/footube/user/thumbnails/example-1.jpg'

Notice that I allow an extra 20 seconds for processing before I show the thumbnails.

ffmpeg can do much more. It can convert formats. It can even generate a preview GIF like what you see on YouTube when you hover a video thumbnail.

Resources

Articles I found helpful.

Conclusion

This article ended up way longer that I thought it would. I wanted to give a comprehensive view of how to set this thing up. If I left something out or got something wrong please let me know.

Parse user input for urls, timestamps & hashtags with RegEX 🧠

benjaminadk — Sun, 17 Feb 2019 16:23:02 +0000

Video code-along version of tutorial 📽

I used to avoid Regular Expressions aka RegEx at all costs. Aside from not understanding how to use it I didn't see much purpose for it in my code. I suppose, to be fair, RegEx isn't exactly beginner friendly. Nevertheless, I now find myself looking for opportunities to use it. The truth is that RegEx can save a lot of development time and is a powerful tool.

Recently, I have been focused on re-creating parts of YouTube, and I noticed something simple, yet cool, about video descriptions and comments. Users can enter urls, timestamps and hashtags and YouTube will parse the input and transform the text into links. Urls become external links, timestamps are links that seek the current video to a specific spot and hashtags become search terms to find related content.

There are some good tools and sites out there to learn about this. Sometimes just googling regex for <whatever> will bring up some good Stack Overflow. RegExr is really cool. You can create an account as save your expressions into a library of your own. Plus they break down each character and what it does, not to mention a database of community expressions. Regular Expressions Info has more detailed breakdowns of pretty much anything and everything related to RegEx.

Now this tutorial assumes you have already captured and stored the user input. That is the raw text we are parsing. From here we need to address a few things as we process the text into HTML.

Preserve the formatting of text - spacing, line breaks, etc
Fit the text into an HTML element
Parse text for urls, timestamps (HH:MM:SS format) and hashtags
Replace these with appropriate links, target and params if needed
Bonus: set the time of video, perform a search based on hashtag term

⚠ Disclaimer - all code examples will use React and/or JSX syntax and therefore JavaScript

Preserving the format is pretty easy. One option is HTML pre tag. pre is short for pre-formatted I think. 🤦‍♂️

<pre>{description}</pre>

Another option is to use the white-space CSS property set to pre. We might as well use pre-wrap. Otherwise long lines of text will overflow their container.

<div style={{whiteSpace: 'pre-wrap'}}>{description}</div>

Now we need to bust out the big guns 🔫. First we need to find, and somewhat understand the regular expressions involved. Here is a pretty standard expression to find http/s urls. It basically looks for http://anything, but it seems to do the trick. Note the g flag - matches all occurrences and the i flag that ignores case. It can also match ftp and file urls by using the OR operator in the first capture block.

const reUrl = /(\b(https?|ftp|file):\/\/[-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|])/gi

The timestamp expression isn't quite as bad. Note that (?:)? sets up capture groups that are optional. [0-5] makes sense because when dealing with HH:MM:SS you won't see 01:90:90, the highest minute or second can be 59. Anyways, this is set up to match MM:SS and :SS which is cool. This allow the user a little more flexibility in what they can use are time links.

const reTime = /\s(?:(?:([01]?\d):)?([0-5]?\d))?:([0-5]?\d)\s/g

Ok, lets get down to the function itself. We are going to leverage the replace method on the String prototype. String.prototype.replace can take RegEx as the first argument and a function as the second. This callback function can receive many arguments, but the first is the matched text itself. This means we can use the original urls/time/hash in our replacement string. The idea is to replace our matches with the appropriate HTML. To keep things simple, we'll start with urls. This process in commonly called the linkify process. Get it? 🧠

function linkify(text) {
    return text.replace(reUrl, url => `<a href="${url}" target="_blank">${url}</a>`)
}

I used an arrow function and returned a template string to save on space. Target set to _blank ensures that this page will be opened in a new window. Template strings you should probably know about by now.

Dealing with the timestamps is a little more advanced. We are going to need a helper function and some additional logic to make them useful. Assume we have a video player, like YouTube, for this example. We want to display the timestamp link in HH:MM:SS format but we need to convert that value to seconds so we can set a search parameter and have a value that we can send to our player - The HTML video element has a property called currentTime which gets/sets the time of the video in...seconds! We also need the value of the url to our player's page on our site.

function HHMMSStoSeconds(str) {
  var p = str.split(':')
  var s = 0
  var m = 1

  while (p.length > 0) {
    s += m * parseInt(p.pop(), 10)
    m *= 60
  }

  return s
}

function linkify(text) {
    const playerUrl = 'http://www.youtube.com/watch'
    return text.replace(reTime, time => {
        const seconds = HHMMSStoSeconds(time)
        return `<a href="${playerUrl}?time=${seconds}">{time}</a>`
    })
}

As a side note I really like the string to seconds function. Its been a while since i used a while loop. 🤓

Now when a user clicks a timestamp link we can implement some tricky logic in our React component to seek the video to the time specified in the link.


class Player extends React.Component {

    componentDidMount() {
        const params = new URLSearchParams(window.location.search)
        const time = params.get('time')
        if(time) {
            this.video.currentTime = time
        }
    }

    render() {
        return <video ref={el=>this.video = el} src={src} />
    }
}

This may look weird because we are used to routing libraries, but it works. Learn about URLSearchParams. Using a ref is also key here. There are a feature of React that gives us access to the underlying DOM Node and all the built in APIs that go with it. React Refs and HTML video/audio DOM... are helpful.

Hashtags work in a very similar way to timestamps. It is up to the developer to decide how to implement them into the UI. YouTube runs a search for anything related to hashtag term. The expression to match hashtags might look something like this.

const reHash = /(?:\s|^)?#[A-Za-z0-9\-\.\_]+(?:\s|$)/g

This one is actually almost understandable. But we can break it down as follows.

(?: // start of non-capture group
\s  // match space character
|   // logical OR
^   // beginning of string
)   // end non-capture group
?   // match 0 or 1 of preceding
#   // match # character
[]  // enclosed character set
A-Z // capital A through Z
a-z // lowercase a through z
0-9 // digits 0 through 9
\-  // \ is an escape character matches -
+   // requires 1 or more match of preceding token
$   // end of string

Now we can lump everything together into one big function. Of course everyone's needs are different but the following would be something like YouTube. This time I am passing a video object. This is just one way to do it. However, in my implementation I don't see much sense in making timestamp links if the time is greater than the duration of the video. Check out the if/else block, by returning the parameter to the callback function it is as if we ignore that specific match. Worthwhile.

import HHMMSStoSeconds from './above-this'

const reUrl = /(\b(https?):\/\/[-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|])/gi
const reTime = /\s(?:(?:([01]?\d):)?([0-5]?\d))?:([0-5]?\d)\s/g
const reHash = /(?:\s|^)?#[A-Za-z0-9\-\.\_]+(?:\s|$)/g
const frontend = 'https://www.youtube.com'

export default function linkify(video) {
  return (
    video.description
      .replace(reUrl, url => `<a href="${url}" target="_blank">${url}</a>`)
      .replace(reTime, time => {
        const secs = HHMMSStoSeconds(time)
        if (secs > video.duration) {
          return time
        } else {
          return `<a href="${frontend}/watch?id=${video.id}&t=${secs}">${time}</a>`
        }
      })
      .replace(
        reHash,
        hash => `<a href="${frontend}/search?term=${hash.replace('#', '').trim()}">${hash}</a>`
      )
  )
}

So if you actually made it this far you for sure learned something. I figured it took me a good part of a day to figure all this stuff out and I had to pull from all kinds of different websites and searches. Why not put it all down in the same place. Naturally, there are probably more efficient or more thorough RegEx out there. But these seem to work well for my use case.

Parser Tutorial

Clone Component Series

My YouTube Channel

Library that does all this for you

Learn By Copying Someone Else

benjaminadk — Thu, 14 Feb 2019 08:38:30 +0000

Are you learning React? Are you interested in Styled Components? If yes||yes, check out my new series, free on YouTube. Else, article.exit(0).

The idea is to isolate dynamic, but simple, components from popular websites and re-created them using React and Styled Components. We are not building a full stack app here and the target length of each series is about 1 hour. So these are easily digestible videos.

To get a better idea of what is up, the GIF below is the first component - the YouTube tag input. It appears when a user is uploading an video and the tags are used to help with search functions or video sorting later on. While most def not the flashiest UI ever created, it does pack a lot of functionality into a small space. What I like is that on first glance the user sees a regular input[type='text'] element. Upon closer inspection there is much more going on. Tags are being saved somewhere. The text input's placeholder only shows up when there are no tags. Commas, and the enter key save a tag. Certain characters and multiple spaces are not allowed in tags. Each tag has a little x that deletes it. Tags are cut off after they get too long. How is all this happening?

Okay. Lets dig a little deeper. One neat feature of the YouTube tags input is how individual tags are deleted. This is done with the backspace key. If any text, not in a tag, is present, backspace functions normally - erasing a single letter at a time. Now if there is no text the backspace key will turn that last tag into selected text. Then of course one more press will erase the entirety of the tag. This is cool because it gives the user a intermediate step in tag deletion without being over the top, like a confirm dialog or something - I mean we are just talking about a tag here. So if you are wondering how to do that...watch the video.

On the other hand, no one is making you do anything. But I will say that trying clone UI features from other websites is a worthwhile practice. Even if you aren't a React person, use your preferred UI library/framework, or even vanilla JavaScript if you are feeling it. I have found that when my mindset is that of a user I don't really think about the code behind the functionality. And when something is easy to use the complexity behind the scenes is overlooked.

Thanks for reading.

Something Someone Somewhere Uses... Sometimes

benjaminadk — Fri, 21 Dec 2018 14:07:55 +0000

🌌 Recently I found myself coding in a team and it was decided that all commit messages would begin with an emoji. Gitmoji defines what each emoji means. ✨ or :sparkles: is for a new feature and 💩 or :hanky: is for bad code that needs improvement...you get the idea. Turns out there are tons of these - I just counted 62. I admit, it adds an element of fun and repos look more stylish. Gitmoji says:

Using emojis on commit messages provides an easy way of identifying the purpose or intention of a commit with only looking at the emojis used.

Normal	Gitmoji

So coding begins on this project and the time comes to start making commits. I cruise over to Gitmoji and scroll around trying to find the best available emoji. This ends up taking a while as I learn the system. I also have a problem when I use the Windows 10 emoji tool (windows button + ;) to insert the emoji into the terminal. When the commit message gets to GitHub, the emoji looks like ࢊ aka nothing. So this is becoming a giant waste of time, seriously disturbing my flow, yo.

I was also unaware of gitmoji-cli at this time

Instead of running npm i -g gitmoji-cli, which I still haven't done, I spent the rest of the day making a VS Code extension and not working on my project. I had no idea what I was doing, but VSC is pretty amazing and has a tool to help scaffold an extension.

npm install -g yo generator-code
yo code

The yo installs Yeoman and generator-code is the VSC tool that generates all the files needed for an extension, ➕ a Hello World.

In any event, I found the docs and the api to be user friendly. Another 🤙 radical feature dude, is that VSC will automatically know that the workspace is for extension development (I believe this is based on ~.vscode/launch.json file the code generator makes). Instead of running an npm script or something, just press F5 and a whole new window opens along with a debugger. I feel like a theme is developing of me worshiping my code editor...idk...we spend a lot of time together.

After a few hours I had built a solution to my problem, integrated right into my favorite code editor. It works by having a big list of all the emojis and their definitions available in the command palette. VS Code's api is such that I didn't even have to write the logic for the search, only process the result. Less work, how great is that. Anyways, after an emoji is selected the prompt wants the commit message. All I really do is match the user's selected emoji to the list and roll it all together with a string literal....Then finally everything pops out in the integrated terminal (or creates one).

// pseudo code
const choice = '🚀 Deploying stuff'

const gitmoji = () => {
...
   if(choice.match('🚀')) return ':rocket:'
...
}
const commitText = 'Deploy app to Heroku'

send.to.terminal(`git commit -m ${gitmoji} ${commitText}`)

So the final string in the terminal might be something like:

 git commit -m ':rocket: Deploy app to Heroku'

Then you Hit enter.

Here is a gif that explains it better than I do:

The next step was publishing the extension to the marketplace. Not a big deal. There is a command for that. And 💥, my extension is available to the entire community. After a week or so its been installed 30 something times, not earth shattering, but for where I am at in my career I count anyone using anything I had a hand in as a success. I can only imagine what it feels like to work on a big app with millions of users. My brain would blow a gasket.

At the end of the day providing value to the user is the true purpose of writing software. Though most of the time I just enjoy coming up with ideas and solving problems, lost in my own little world.

So if u feel like giving it a try just search gitmoji commit in extensions.

Pro Tip - instead of going to the palette and selecting the Gitmoji Commit command, map the command straight to a keybinding. I use Ctl/Cmd + K G.

benjaminadk / emojigit

VS Code extension for composing Gitmoji commit messages

Gitmoji Commit

Stop wasting time looking up git commit message emojis. This extension helps you find the right emoji and helps compose your commit message, all without leaving VS Code. You can even supply your own custom emoji mapping to suit the needs of you and/or your team.

Content

Gitmoji Commit

Usage

Currently, Gitmoji Commit adds two commands to VS Code.

Create Commit Message

Automates creating a commit message by prompting user for the type of commit and commit message text. The type of commit selected determines the emoji. The emoji is inserted into the commit message as either unicode (e.g. ✨) or as GitHub emoji markdown colon syntax (e.g. :sparkles:).

When the time comes to type…

View on GitHub

DEV Community: benjaminadk

Build A Command Line Application With Node

Source Code

benjaminadk / node-cli-tutorial

Build A Command Line Application Tutorial

Node CLI Tutorial

Working CLI Tool

Introduction

Option 1 - Clone Source Code

Option 2 - Build From Scratch

What is my Job Title

UPDATE

Encode GIFs With Node

The GIF Format

My Use Case

Existing Libraries

My Library

Using Gif Encoder 2

Constructor

Methods

Basic Example

Advanced Example

Alternative Encoding Method

Basketball Stats Through D3 & React

Data

Create a Donut Chart

👩‍🎓 Open Source Exam Simulator

Introduction ◽ Exam Simulator ◽ Exam Maker ◽ Moving Forward

Introduction

Exam Simulator

Exam Maker

Moving Forward

exam-simulator / simulator

👩‍🎓 JSON based exam simulator.

Exam Simulator JS

Documentation

Notes

Donate

exam-simulator / maker-frontend

Create, edit and share exams for Exam Simulator. Built with Next.js.

Exam Maker Frontend

exam-simulator / maker-backend

Apollo Express server for Exam Maker.

Exam Maker Backend

Free Speech on Twitter

Color Harmonies in JavaScript

Intro

benjaminadk / color-tool-remix

Color Tool Remix 🎨 A cross platform color utility app

Color Tool Remix

Contents

Installation

Usage

Picker

Dropper

Formats

HSL Benefits

Parsing HSL String

Color Harmonies

How do I create thumbnails when I upload a video? aws lambda!

Introduction

Companion Video

Disclaimer

Lambda Functions

IAM Role Permissions

Triggers

Getting Code to Lambda

Lambda Layers

Disclaimer

Code

Tips

Resources

Conclusion

Parse user input for urls, timestamps & hashtags with RegEX 🧠

Learn By Copying Someone Else

Something Someone Somewhere Uses... Sometimes

benjaminadk / emojigit

VS Code extension for composing Gitmoji commit messages

Gitmoji Commit

Content