DEV Community: mzakzook

Destructuring JS

mzakzook — Sat, 29 Aug 2020 02:29:29 +0000

When working with JavaScript objects and/or arrays it can sometimes be helpful to extract properties/values and save them as individual variables. This can be achieved very efficiently with a process called destructuring.

Array Destructuring

If we are given a large array we can extract just the first three elements using a form of destructuring, shown below:

let array = [1, 2, 3, 4, 5, 6, 7, 8, 9];
let [a, b, c] = array;

console.log(a) => 1
console.log(b) => 2
console.log(c) => 3

If we wanted a to represent the first element, b to represent the second element and c to represent the remaining elements we could modify our earlier expression as follows:

let array = [1, 2, 3, 4, 5, 6, 7, 8, 9];
let [a, b, ...c] = array;

console.log(a) => 1
console.log(b) => 2
console.log(c) => [3, 4, 5, 6, 7, 8, 9]

We can also set default values for destructuring variables:

let array = [1, 2];
let [a = 5, b = 5, c = 5] = array;

console.log(a) => 1
console.log(b) => 2
console.log(c) => 5

The last couple array destructuring techniques I'll cover are assigning an array that is a return value and the process for skipping elements:

function returnArr() {
  return [1, 2, 3, 4, 5, 6];
}

let [a, ,b] = returnArr();

console.log(a) => 1
console.log(b) => 3

In the last example our function, returnArr, returns an array. We assign the first and third values of this array by using destructuring with a comma-separated empty value between our first and second variables (a & b).

Object Destructuring

Destructuring JavaScript objects is very similar to how we went about destructuring arrays. I have provided a basic example of object destructuring below:

let obj = {color: 'black/rust', size: 'large', breed: 'Rottweiler'};

let {color, size, breed} = obj;

console.log(color) => 'black/rust'
console.log(size) => 'large'
console.log(breed) => 'Rottweiler'

If you'd like to assign new variable names you can do so like this:

let obj = {color: 'black/rust', size: 'large', breed: 'Rottweiler'};

let {color: c, size: s, breed: b} = obj;

console.log(c) => 'black/rust'
console.log(s) => 'large'
console.log(b) => 'Rottweiler'

Object destructuring is very powerful when iterating over an array of many objects. It can be implemented into a function to minimize typing out long variable names. I have provided an example below:

let dogs = [
  {
    name: "Bruno",
    stature: {
      weight: "70lbs",
      size: "large"
    },
    age: 1,
    breed: "Mutt"
  },
  {
    name: "Bat",
    stature: {
      weight: "6lbs",
      size: "tiny"
    },
    age: 3,
    breed: "Pomeranian"
  },
  {
    name: "Kiwi",
    stature: {
      weight: "65lbs",
      size: "large"
    },
    age: 14,
    breed: "Chocolate Lab"
  },
  {
    name: "Ralph",
    stature: {
      weight: "90lbs",
      size: "extra large"
    },
    age: 9,
    breed: "Rottweiler"
  }
];

for (let {name: pup, stature: {size: largeness}} of dogs) {
  let a;
  largeness === 'extra large' ? a = 'an' : a = 'a';
  console.log(`${pup} is ${a} ${largeness} doggo`)
}

=>

'Bruno is a large doggo'
'Bat is a tiny doggo'
'Kiwi is a large doggo'
'Ralph is an extra large doggo'

Destructuring both makes it easier to write out code and can make it much easier to read through code that may include deeply nested variables. Hope this run-through helped!

Sources:

MDN Web Docs - Destructuring Assignment

StackOverflow

mzakzook — Sun, 23 Aug 2020 19:36:39 +0000

Recursion Refresh

Recursion is a fundamental concept in computer programming both because it provides an intuitive and clean way to write code, but also because it offers a glimpse into the way that code executes.

I've written a bit before about recursion before - first with a focus on dynamic programming and again later in my post about the Fibonacci sequence. As a refresher, a recursive program is one that calls itself. It is composed of a "base case" and a "recursive case". We make successive recursive calls until we reach the base case, at which point we return.

Here I will explore a different aspect of recursion: the stack.

The stack

We can think of the call stack as an underlying data structure that holds the local execution context for a program. The call stack has the same LIFO ("Last In First Out") properties of a stack data structure that we would use in our code (the opposite of a queue FIFO data structure). This makes sense when we think about how a recursive program executes: once we reach the base case, we want to return the output to the most recent recursive call, and build up our solution from there.

The call stack lives in memory, but unlike the heap, it is finite in size and is managed automatically by your OS. By default, the stack size on Mac OS is 8MB - you can see your current stack and memory limits by running:

ulimit -a

When I run this command on my MacBook Pro, I see the following:

// ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1392
virtual memory          (kbytes, -v) unlimited

StackOverflow

When most programmers hear "StackOverflow" in 2020, our minds jump to the Q&A forum that has helped us solve so many problems. But a stack overflow is a type of exception that occurs when a recursive program opens too many frames. In other words, if our program exceeds the memory limits allocated to the stack, we will see this exception. This is a safeguard against infinite recursion - if your program has a bug and never reaches the recursive case, it will encounter this exception.

Here is a sample program that encounters a stack overflow (in JS, a RangeError):

function helloWorld(i) {
  if (i == 1) {
    return "Hello";
  } else {
    return helloWorld(i);
  }
}

The bug here is that our recursive case doesn't modify the input variable, so we will never reach the base case. When I execute this program, I see the following:

> helloWorld(2)
Thrown:
RangeError: Maximum call stack size exceeded
    at helloWorld (repl:1:20)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)
    at helloWorld (repl:5:8)

Conclusion

Understanding the underlying data structures used to execute our code can help to inform our decisions as programmers. And in this case, it also gives us an understanding of the naming behind one of the most commonly used tools in a programmer's kit: StackOverflow.

Resources:

Recursion and stack
The JavaScript Call Stack - What It Is and Why It's Necessary - Charles Freeborn
Understanding Execution Context and Execution Stack in Javascript - Sukhjinder Arora
Stack Overflow logo
Memory : Stack vs Heap - Paul Gribble

Leetcode - Partition Labels

mzakzook — Sat, 15 Aug 2020 21:52:30 +0000

As I've prepared for interviews Leetcode has been a place I've turned to often to brush up on logic questions and try to expand the way I approach problems.

I recently worked on a problem called Partition Labels that I felt was a good exercise in dissecting a problem and creating a plan of attack.

The problem provides a string and asks us to break up the string into as many partitions as possible where any letter appears in no more than one partition. It then asks us to return the length of each partition. The string cannot be sorted or reordered in any way.

For example:

partitionLabel('abcd') => [1,1,1,1]
partitionLabel('abcda') => [5]

The first example abcd returns the array [1,1,1,1] because each letter becomes its own partition. The second example, abcda, returns the array [5] because the last a in the given string does not allow for the creation of any partitions. This is because a also appears at the beginning of the string.

Knowing that any repeating letter would need to be combined with the first occurrence of that letter and all letters between, my goal was to loop through the length of the string and create partitions where a letter appeared that had not appeared in the string since the last partition was created (using a continually modified comparison index). I then had to consider what I would do if a letter reoccured when its past instances had already been sectioned off into their own partition. My solution was to compare a given letter to each partition that had been created. If a partition included the given letter, I would remove that partition and any others created after that. I would also need to reset my comparison index to an earlier point in the string. With this strategy, once I complete my loop I am able to create one additional partition for the last slice of the string (from the comparison index through the end). Lastly I return the array of partitions mapped to the length of each element.

Here is my code (in JS):

var partitionLabels = function (S) {
  let result = [];
  let ind = 0;
  for (let i = 1; i < S.length; i++) {
    if (S.slice(ind, i).includes(S[i])) {
      continue;
    } else {
      let isGood = true;
      for (let y = 0; y < result.length; y++) {
        if (result[y].includes(S[i])) {
          result.splice(y);
          ind = result.join('').length;
          isGood = false;
        }
      }
      if (isGood) {
        result.push(S.slice(ind, i));
        ind = i;
      }
    }
  }
  result.push(S.slice(ind));
  return result.map(x => x.length);
};

First I created a result array to hold each partition that is created. Next I create my comparison index variable called ind (initially set to 0). I then create a for loop to iterate through each character of the input string (starting from the second letter because the first letter does not need to be compared to anything). If the targeted portion of the string (since the last partition was added) does include S[i], we simply continue; if not, we set a variable called isGood to true (indicating that the creation of a new partition is valid). We then check if any previous partitions include S[i]. If one does, we remove each partition starting from that one (inclusive). We then reset the comparison index, ind, to the length of the joined elements of result (because the length represents when the problematic partition began) and set isGood to false (indicating that a new partition is no longer valid). If S[i] is not found in an earlier partition isGood is unchanged and remains true so we push a new slice of S into result (starting from ind and ending at i) and reset ind to i. When we exit our for loop we still need to add one partition (from ind through the end of the string). Lastly we return result mapped to the length of each element.

I enjoyed working through this problem because it forced me to think about a practical solution before typing any code. Look forward to working through more problems similar to this one!

Heuristic Algorithms

mzakzook — Sun, 09 Aug 2020 16:31:50 +0000

Heuristic algorithms are solutions that optimize efficiency sometimes at the expense of accuracy. The idea is that if you have a computationally expensive problem (NP-hard), it's better to come up with a good approximation in a reasonable amount of time rather than find an exact solution after a very long time.

Traveling Salesman Problem

To illustrate, I will use a classic example of an NP-hard problem: the Traveling Salesman Problem. In this problem, the goal is to determine the shortest route for our traveling salesman to pass through a given set of cities.

To conceptualize why this problem is so expensive, let's think about examples: if our salesman has to visit 3 cities, how many possible routes are there to explore? For the first stop, we have 3 choices; for the second, we have 2; and for the third, we are only left with one city. So, we have a total of 6 possible routes. This might look familiar to some as an n! problem.

3 x 2 x 1 = 3!

With only 3 cities to visit, this problem is easy to visualize. However, if we jump up to 10 cities, we are suddenly at over 3 million solutions (3,628,800 to be exact). As we might imagine, the inputs for computer science problems tend to be much larger than 10, so this problem becomes inordinately expensive to solve very quickly.

So how do heuristic algorithms help us solve this problem? If we are able to estimate the shortest path without actually traveling all 3,628,800 routes, we can save a lot of time. One example of a heuristic algorithm that often works "well enough" for TSP is a greedy algorithm. At each stop, select the next closest city. With this, we are able to come up with an approximate solution for our problem in only 10 steps.

This YouTube video gives a good introduction to TSP, and even offers more optimal solutions than a simple greedy algorithm (like Simulated Annealing):

A* Search

Another common heuristic algorithm is A* search. If we are searching a graph for the shortest path from node A to node N, A* search will represent all of the possible first-steps from A in a tree and use a heuristic to select the "best" first step. The "best" node is the one with the smallest cost that brings us closest to the end goal. Of course, we won't travel every path from the start node to the end, but we can use an estimation of the distance (like absolute distance) to evaluate which node is best.

This tool is really useful for visualizing path traversal with A* and other algorithms: PathFinding by qiao.

Heuristic algorithm applications

Heuristic algorithms are used in many NP-hard problems, but they are especially common in Artificial Intelligence and anti-virus software. In both of these cases, there is a large "problem space" to traverse (a decision tree, or a computer's filesystem), and heuristic algorithms often provide a "good enough" solution.

Sources:

Breadth First vs Depth First Search

mzakzook — Mon, 03 Aug 2020 00:44:23 +0000

Search algorithms are something that I have not had to consider much when building full stack applications, but they are an important component of programming as they are frequently discussed in interviews. I am going to run through the differences between depth first and breadth first search and discuss when each is optimally used.

BFS

BFS stands for breadth first search. This method of search finds the shortest path to the target value, but is typically slower than depth first search. BFS utilizes a queue data structure to store values visited. It follows a 'first in first out' method and travels to each adjacent value before moving deeper. To demonstrate, I've created a sample "tree". A BFS search of the following tree will visit the nodes in alphabetical order:

      A
    /   \
   B     C
  / \   / \
 D   E F   G

DFS

DFS stands for depth first search. This method of search examines each value in a branch then moves to an adjacent branch. DFS utilizes a stack data structure to store values visited. After each value in a branch has been visited, all values are popped from the stack before moving to an adjacent branch. As with the example above, a DFS search of the following tree will visit the nodes in alphabetical order:

      A
    /   \
   B     E
  / \   / \
 C   D F   G

Which algorithm should I use?

BFS performs better when searching values that are closer to the source. DFS is a better option when searching values that may be further from the source and is typically the safer choice.

Because BFS searches each neighbor first, it is not an ideal solution for decision making trees. DFS is a better solution for decision making trees because once a value is targeted you will be interested in the paths that extend from that value.

BFS and DFS are the most elementary examples of search algorithms; they can become much more intricate and complex. Determining which algorithm is best depends on many factors, such as the ordering of the data (is it sorted? roughly sorted? in random order), and the structure of the data (is it hierarchical?). It is also important to recognize the constraints of the system running your search program: is it memory-sensitive, or CPU-sensitive? In short, there is no universal "best" search algorithm - but luckily, this isn't a new problem in the computer science community, so there are many resources to help make this decision.

Resources

Visualization of BFS and DFS on VisuAlgo.
Searching Algorithms on GeeksforGeeks.

Implementing Redux in React App

mzakzook — Mon, 27 Jul 2020 01:09:10 +0000

When developing a React frontend with data frequently being fetched from a backend/API, it can sometimes become challenging passing state through props from one component to the next. This practice can be very tedious and can tempt you to fetch data from a less-than-ideal component just to simplify the state issue.

Redux is a state manager that allows you to store and retrieve data from any component on your front-end. I'm going to go through the process of setting up Redux in your app.

First, you will need to install the redux library. Run both of the following commands:

yarn add redux
yarn add react-redux

Next, we will create a reducer. A reducer accepts two arguments: current state & an action (e.g. user logged in/out, data fetched, auth failed). The new updated state will be returned by the reducer, taking into account the action that was passed to it.

Here is an example of a reducer (I would create this reducer in a separate file; let's call this file 'auth-reducer'):

const INITIAL_STATE = {
  email: '',
  password: '',
  auth_token: '',
  errorFlag: false,
  spinner: false
};

export default (state = INITIAL_STATE, action) => {
  switch (action.type) {
    case 'EMAIL_CHANGED':
      return { ...state, email: action.payload };
    case 'PASSWORD_CHANGED':
      return { ...state, password: action.payload };
    case 'LOGIN_FAILED':
      return { ...state, errorFlag: true, password: '', spinner: false };
    case 'LOGIN_USER_SUCCESS':
      return { ...state, ...action.payload, spinner: false };
    case 'FIND_USER_SUCCESS':
      return { ...state, ...action.payload, spinner: false };
    case 'LOAD_SPINNER':
      return { ...state, spinner: true };
    case 'LOGOUT_USER_SUCCESS':
      return INITIAL_STATE;
    default:
      return state;
  }
};

Next, we'll create a Redux store in our app component. First we need to import several items.

import { createStore } from 'redux'
import { Provider } from 'react-redux'
import AuthReducer from './AuthReducer' -> (this is where we created our reducer in the last step)

Now we are ready to create our Redux store.

const store = createStore(AuthReducer);
render(
 <Provider store={store}>
   <App />
 </Provider>,
 document.getElementById(‘root’)
);

Notice that we created a store using the createStore function and passed it our AuthReducer. I also wrapped the 'App' component in a 'Provider' component which will allow all components inside App to access our store.

To access state inside a component you will create a function that assigns redux's state to the component's props. This is how I implemented my AuthReducer inside a login componenet:

const mapStateToProps = (state) => {
  return {
    email: state.auth.email,
    password: state.auth.password, 
    error: state.auth.errorFlag,
    spinner: state.auth.spinner,
    auth_token: state.auth.auth_token
  };
};

At the bottom of my component I also need to specify that I am using this functionality.

export default connect(mapStateToProps)(LoginForm);

Inside that component I could then access the password attribute in redux's state, by calling this.props.password.

To update redux's state you can either use mapDispatchToProps function or create individual actions that modify state. I will explain the process of creating individual actions.

If I wanted to change a user's email and/or password in state I could add the following functions:

export const emailChanged = (email) => {
  return {
    type: 'EMAIL_CHANGED',
    payload: email
  };
};

export const passwordChanged = (password) => {
  return {
    type: 'PASSWORD_CHANGED',
    payload: password
  };
};

Now, inside my auth component I would import these functions and call them by passing in a new email/password. I would also need to modify my final export statement to the following:

export default connect(mapStateToProps, { emailChanged, passwordChanged })(LoginForm);

This run through should allow you to quickly set up a React App with Redux. There are many powerful ways to utilize Redux to improve performance and organization so I hope you enjoy. Check out the links below for further questions.

Sources:
1. Redux Docs
2. React-Redux Docs
3. Quick and easy beginner’s guide to implementing Redux in a React app, by Paul Fitzgerald

Function Expressions & Declarations

mzakzook — Sat, 18 Jul 2020 22:00:49 +0000

There are subtle differences between function declarations and expressions which are rarely understood by JavaScript beginners. This article will review their differences and suggest when to use each. First, an example of each:

Function declaration:

function helloWorld() {
  console.log('Hello world!');
}

Function expression:

var helloWorld = function() {
   console.log('Hello world!');
};

Function declarations

First, we'll review the simpler/more straightforward of the two options: function declarations. They resemble function definitions in other languages. Declarations start with the function keyword, followed by the required function name (in the above, helloWorld), and the function body.

Declarations are ideal when writing a function that will be called throughout the global scope of the program. One handy feature of function declaration is that declared functions are "hoisted", meaning that the definition is moved to the top of the program scope before the program is executed. This essentially means that the function can be called above its own declaration. For example:

helloWorldDeclared(); // => Hello world!
helloWorldExpressed(); // => TypeError: helloWorldExpressed is not a function

var helloWorldExpressed = function() {
   console.log('Hello world!');
};

function helloWorldDeclared() {
  console.log('Hello world!');
}

Function expressions

While neither expressions nor declarations are better, function expressions do offer more possibilities. Here are some cases where expressions come in handy:

Conditional function creation

Conditionally declared functions are not consistently implemented across browsers, so for production-quality code, expressions are better.

Immediately invoked function expressions (IIFEs)

IIFEs are primarily used because they allow for private member variables. As the name implies, IIFEs are expressions which are invoked upon definition. By defining functions within IIFEs, we can create variables which are no longer accessible in the outer function scope. For example:

var HelloWorld = (function() {
    var greetee = "world";

    var HelloWorldInner = function() {
        this.getGreetee = function() {
            return greetee;
        };

        this.setGreetee = function(value) {
            name = greetee;    
        };    
    };

    return HelloWorldInner;
})();
var hw = new HelloWorld();
hw.greetee = "Max";
hw.getGreetee(); // => world

Callback functions

While functions defined via declaration can be passed into callback functions, expressions are sometimes preferable because the passed-in function isn't used outside of the callback. So the unnamed function expression can be passed directly into the callback call.

Anonymous function creation

While function declarations require that a name be assigned to the function, expressions do not have that requirement. If a function expression is assigned to a variable, it adopts the name of the variable.

Arrow expressions

Like lambda functions in other languages, arrow expressions offer a succinct format for function definition. For example:

var adder = (a, b) => a + b;
console.log(adder(1,2)); // => 3

Using the `Function` constructor

Using the Function constructor explicitly is seldom your best option; according to MDN docs, it "suffers from security and similar (but far less significant) performance issues to eval". While I don't recommend using the Function constructor I wanted to call it out as a lesser known option. Example:

const helloWorld = new Function('console.log("Hello world!")');
helloWorld(); // => Hello world!

Resources

Function expressions - MDN docs
Function declarations - MDN docs
Callback functions - MDN docs
Arrow functions - MDN docs
Example of IIFE - Coderwall

Count Number of Teams

mzakzook — Sun, 12 Jul 2020 16:30:13 +0000

I came across another LeetCode problem that I felt was worth exploring here. It's called 'Count Number of Teams'.

The problem tells us that there will be an input array of unique integers; each integer representing an individual's rating. We must return the number of 'teams' that can be formed from the input array while satisfying the following criteria:

Each team consists of three members
A team must be represented in ascending order consistent with each member's index in the input array (indexOfA < indexOfB < indexOfC)
A team's ratings must be ascending or descending while adhering to our 2nd bullet point (ratingA < ratingB < ratingC OR ratingA > ratingB > ratingC)

For example:

If given an input array of [2,5,3,4,1], our return would be 3. This is because we can construct three valid teams from this array. The first is (2, 3, 4) because 2 < 3 & 3 < 4 AND indexOf(2) < indexOf(3) & indexOf(3) < indexOf(4). Our second team is (5, 3, 1) because 5 > 3 & 3 > 1 AND indexOf(5) < indexOf(3) & indexOf(3) < indexOf(1). The last valid team is (5, 4, 1) because 5 > 4 & 4 > 1 AND indexOf(5) < indexOf(4) & indexOf(4) < indexOf(1).

I went about solving this problem with brute force. I constructed three for loops to check every possible combination of a team from the input array. If the team satisfies the ascending/descending requirement then I increment my result variable.

When we exit the outside for loop we should have counted every valid team. The last step is to return the counter, which I named result.

var numTeams = function(rating) {
    let result = 0;
    for (let i = 0; i < rating.length; i++) {
        for (let y = i + 1; y < rating.length; y++) {
            for (let z = y + 1; z < rating.length; z++) {
                if ((rating[i] < rating[y] && rating[y] < rating[z]) || (rating[i] > rating[y] && rating[y] > rating[z])) result += 1;
            }
        }
    }
    return result;
};

I find these types of problems to be very helpful in forcing me to consider how to dissect a problem and I look forward to working through more.

Group By Size

mzakzook — Mon, 06 Jul 2020 01:16:06 +0000

I worked on a fun LeetCode problem earlier this week and wanted to discuss the problem here.

The problem is called 'Group the People Given the Group Size They Belong To'.

The problem tells us that there should be a function that accepts an array of integers that represent the size of group that each element belongs to. We are asked to return one example of an acceptable grouping (with each element in the output represented by the index in the original array). For example:

An input of [1, 2, 4, 2, 4, 4, 4] could return [[0], [1, 3], [2, 4, 5, 6]. This is an acceptable return because when examining the input, index 0 is in a group of 1 so it should be grouped alone. Indexes 1 and 3 are both in a group of 2, and indexes 2, 4, 5 and 6 are all in a group of 4. This example was straightforward because there were not multiple groups of the same size. If we received an input of [2, 2, 2, 2, 2, 2, 2, 2] then you could return any combination of four pairs of indexes.

Here is my solution:

function groupThePeople(groupSizes) {
    let groups = {};
    let result = [];
    for (let i = 0; i < groupSizes.length; i++) {
        let tar = groupSizes[i];
        if (tar === 1) {
            result.push([i]);
            continue;
        }
        if (!groups[tar]) {
            groups[tar] = [];
            groups[tar].push(i);
        } else {
            groups[tar].push(i);
            if (groups[tar].length === tar) {
                result.push(groups[tar]);
                groups[tar] = [];
            }
        }
    }
    return result;
}

I went about solving this problem by utilizing an object to store groups as they are being filled. My logic was that if I created a key for each group size and used this object as a staging place, I could push a value array into my final result array once it was full. I then reset that object key to an empty array so that it would be ready for future indexes that also have the same group size.

My for loop iterates over each element of the input. If the element I am examining is 1, I know that I can immediately push the index into my result array and skip to the next index in my for loop.

The remaining code in my for loop carries out the logic to keep my staging object organized.

Once we exit the for loop I know that we have placed each index into an appropriate group, so we return the result array (this problem did guarantee that there would be at least one solution for each input, so I did not account for invalid inputs; if I did want it to notify of an invalid input I would flatten the result array before the return and confirm that it was the same length as the input array).\

This was one way of solving this problem but I'm sure there are many more!

Intro to Web Scraping w/ Puppeteer

mzakzook — Sat, 27 Jun 2020 21:18:54 +0000

I was recently challenged to learn how to perform web scraping and automated form-filling using Puppeteer and was very impressed with the simplicity and functionality of its implementation.

Puppeteer allows a user to do several things:

Scape webpages for content using HTML elements and CSS selectors to target information
Take screenshots
Create PDFs
Create CSVs
Automate simulated user interactions (click, keyboard input) to test webpage functionality

I will discuss the process of setting up Puppeteer and scraping paginated results of Craigslist listings to export to CSV (I'm using Craigslist because its HTML & CSS are easy to digest, but the logic demonstrated should work for just about any site). For more information about taking screenshots, creating PDFs, and automating user interactions (form-filling is a good place to start) check out the sources at the bottom of this post.

To get started with Puppeteer you'll want to create a directory with a JS file and install Puppeteer by running yarn add puppeteer.

Next you'll want to add the following to your JS file:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  await page.goto('https://sfbay.craigslist.org/d/arts-crafts/search/ara', {waitUntil: 'networkidle2'});
  await page.waitForSelector('#sortable-results > ul > li > p > a');
  await browser.close();
})();

We first open an async function and create a new instance of a Puppeteer browser. {headless: false} is an optional parameter that tells your program to open chrome to view your program run; you may omit this argument but your program will just run behind the scenes. Visualizing the execution of your program should help with debugging. Next we open a new page in the browser and navigate to a webpage (in this case Craigslist's arts & crafts). {waitUntil: 'networkidle2'} tells your program to wait until there are no more than two connections on your network to execute that action. Next we tell Puppeteer to wait until a specific selector is available on the page before resuming. This is especially important for SPAs, which may load HTML after a specific action is taken.

Now we'll run through the process of collection information and exporting to a CSV:

const puppeteer = require('puppeteer');
const createCsvWriter = require('csv-writer').createArrayCsvWriter;

(async () => {
  ...
  let listings = [];
  let moreItems = true;
  while (moreItems) {
    const tmp = await page.evaluate(() => {
      const itemLinks = '#sortable-results > ul > li > p > a';
      const itemPrices = '#sortable-results > ul > li > p > span.result-meta > span.result-price'
      const priceList = document.querySelectorAll(itemPrices)
      const itemList = document.querySelectorAll(itemLinks);
      const itemArr =  Array.from(itemList).map((itemLi) => { 
        return [itemLi.text, itemLi.href]});
      const priceArr = Array.from(priceList).map((pri) => { 
        return pri.textContent});
      for (let i = 0; i < itemArr.length ; i++) {
        itemArr[i].push(priceArr[i])
      }
      return itemArr
    });
    listings.push(...tmp)
    try {
      await page.click('#searchform > div > div.paginator.buttongroup > span.buttons > a.button.next');
      await page.waitForSelector('#sortable-results > ul > li > p > a');
    } catch (error) {
      moreItems = false;
    }
  }

  const csvWriter = createCsvWriter({
    header: [],
    path: './craigslist1.csv'
  });
  csvWriter.writeRecords(listings)
    .then(() => {
      console.log('...Done')
    })

  await browser.close();
})();

You'll notice there is one change to the top of our function - I've added the requirement for csv-writer; this will assist us later on. I've added our remaining code under ellipsis.

Our next line creates an array to contain our collected data, called listings. I then create a variable, moreItems, to represent if there are additional pages of results; its default value is true. Next we enter a while loop (for pagination) and create a variable, tmp, that will use Puppeteer's functionality to evaluate the webpage we have visited. For this CSV, I wanted to export an item's name, URL and price. I was able to access this information using two query selectors: itemLinks (which contains URLs and names) & itemPrices. I collect all of the results on the first page for each query selector, then convert the outputs to arrays that contain the specific information I desire. I then combine the two arrays (acting under the assumption that there will be the same length of each array). I then return the combined array before closing tmp and push tmp into listings.

Next I check if there are additional pages by using Puppeteer's click action to look for a 'Next' button. If the button is found I wait for the selector necessary to gather my results for the subsequent page and go back to the top of the while loop. If a 'Next' button is not found, I set moreItems to false and exit the while loop. Once we have exited the loop we should have all of the information we need and are ready to create our CSV. We added the requirement for csv-writer which allows us to accomplish this task very easily. Refer to the code provided (just make sure to provide the correct path for where you'd like the CSV to land; if you'd like it in another directory you can do that as well).

Once our program has collected all of the targeted data, we are able to close the browser. You should then be able to access the CSV file that was generated by navigating to the path that was specified in your program.

Web scraping seems to be a powerful tool to collect data that may not be available in an API and I look forward to exploring Puppeteer more!

Sources:

Fast JSON & Rails

mzakzook — Thu, 18 Jun 2020 15:38:20 +0000

When designing a backend it is important to consider how data will be constructed when a response is being prepared for a client. As any given schema becomes more complex and intertwined, the data that you will want provided to your frontend will likely necessitate information on more than one object or class per fetch.

Active Model Serializer was released about six years ago and accomplished that specific goal for Ruby & Rails: combine multiple objects/classes in a JSON response to be used for your HTTP requests. Data serialization "is the process of translating data structures or object state into a format that can be stored" Serialization - Wikipedia. While this tool was widely used and did accomplish its goal, new players have entered Ruby's serialization scene and have proven that we've come a long way over the last six years.

Netflix released its Fast JSON API about two and a half years ago, and over that time it has proven to be much faster than Active Model Serializer. According to Netflix's docs, these are the results from benchmark times for 250 records:

$ rspec
Active Model Serializer serialized 250 records in 138.71 ms
Fast JSON API serialized 250 records in 3.01 ms

In addition to being very fast, it is extremely user-friendly. To set up Fast JSON API, simply add the gem to your Gemfile and bundle install:

gem 'fast_jsonapi'
$ bundle install

You're then ready to create your serializers. This can be done from the terminal with the command:

rails g serializer Model-Name attribute attribute attribute

Once your serializers are generated, the next step is to add the appropriate 'belongs_to' and 'has_many' relationships (any modifications you would make to your models without Fast JSON API would be the same with it).

Each serializer can now be called from your controller actions and you can pass it a single instance of a class or many at once. For example:

ModelSerializer.new(model_instance)

I am a big fan of Postman and like to check how my data is formatted there, but feel free to confirm layout is as you intended by performing any valid request to your backend.

Because Fast JSON API recognizes the 'belongs_to'/'has_many' relationships, you are able to see full picture relationships with one fetch. If I created a database of dog owners and dogs (where a dog could only have one owner), one fetch for a specific dog owner could include an array of each of her/his dogs (with all of their data).

Fast JSON API has definitely made my life easier when linking backends and frontends and I hope it does for you too!

Rails Validations

mzakzook — Sun, 14 Jun 2020 00:39:06 +0000

When creating a backend for any project it is important to think about how incoming data will be collected and organized. If there are pieces of data that are essential for your app to function properly it is useful to use validations.

I will discuss the process of setting up validations in Rails and how to structure your app controllers to respond with appropriate data for a non-Rails frontend.

When constructing your Rails models you have the ability to add many different types of validations - including custom validations.

First I'm going to demonstrate how absent validations could negatively affect your code. Here's a schema for a Patient class:

create_table "patients", force: :cascade do |t|
  t.string "name"
  t.date "dob"
  t.string "email"
  t.datetime "created_at", precision: 6, null: false
  t.datetime "updated_at", precision: 6, null: false
end

If I wanted to add a new Patient to my database I could enter the rails console and perform 'Patient.create(...)' OR 'Patient.new(...)' combined with 'Patient.save'. Without validations, if I omit one or more of the three attributes of Patient, it will still save to my database. For example:

Patient.create(:name => "Max")

-> Patient Create (7.7ms)  INSERT INTO "patients" ("name", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["name", "Max"], ["created_at", "2020-06-13 22:37:39.727082"], ["updated_at", "2020-06-13 22:37:39.727082"]]
   (12.5ms)  COMMIT
 => #<Patient id: 6, name: "Max", dob: nil, email: nil, created_at: "2020-06-13 22:37:39", updated_at: "2020-06-13 22:37:39">

As you can see, a new Patient without a dob and an email still persisted to my database, with a Patient ID of 6. This could create problems down the line as it is important to have normalized data, which in turn creates a better user experience and leads to less bugs.

Preventing this issue from occurring is relatively simple. It requires that each model includes a 'validates' line for each attribute that needs to be checked, and that your controllers anticipate how to respond to improper POSTs and PATCHes.

The first step to fix the Patient example is to modify the model with validations:

class Patient < ApplicationRecord
  validates :name, presence: true
  validates :dob, presence: true
  validates :email, presence: true
  validates :email, uniqueness: true
end

Sidenote: There are many different types of built-in validations that can be used and you may also create custom validations. Refer to Active Record Validations for more info.

In my last code snippet I created validations to ensure that each of the attributes are present for any instance of Patient and I added an additional validation to check that the patient's email is unique.

Now when I try to perform the same console action as before I encounter an error:

Patient.create(:name => "Max")

Patient Exists? (0.4ms)  SELECT 1 AS one FROM "patients" WHERE "patients"."email" IS NULL LIMIT $1  [["LIMIT", 1]]
   (11.2ms)  ROLLBACK
 => #<Patient id: nil, name: "Max", dob: nil, email: nil, created_at: nil, updated_at: nil>

This time the patient instance did not persist to the database (Patient ID is returned as 'nil'). Our validations clearly worked!

When hitting a Rails API with a different frontend it is important to think about how data will be returned to a user. As you can see in our previous code snippet, an ActiveRecord instance was returned for our 'create' command, even though the instance was not persisted to the database. We would not want to send back information from this ActiveRecord instance because that could incorrectly indicate to a user that their data was saved. To prevent this we would write 'create' and 'update' controls that check if data was saved before sending it back to the user:

def create
  patient = Patient.new(patient_params)
  if patient.save
    render json: patient
  else
    render json: patient.errors.messages
  end
end 

def update
  patient = Patient.find(params[:id])
  patient.update(patient_params)
  if patient.valid?
    render json: patient
  else
    render json: patient.errors.messages
  end
end

Now when a user tries to create or update a record with invalid data, our backend will send an error (error messages can also be customized - Validation Errors).

This was a simple run-through on how to set up validations and modify controllers accordingly, but feel free to peruse the documentation for more advanced validation options!

Sources:

Active Record Validations

DEV Community: mzakzook

Destructuring JS

Array Destructuring

Object Destructuring

Sources:

StackOverflow

Recursion Refresh

The stack

StackOverflow

Conclusion

Resources:

Leetcode - Partition Labels

Heuristic Algorithms

Traveling Salesman Problem

A* Search

Heuristic algorithm applications

Breadth First vs Depth First Search

BFS

DFS

Which algorithm should I use?

Resources

Implementing Redux in React App

Function Expressions & Declarations

Function declaration:

Function expression:

Function declarations

Function expressions

Conditional function creation

Immediately invoked function expressions (IIFEs)

Callback functions

Anonymous function creation

Arrow expressions

Using the Function constructor

Resources

Count Number of Teams

Group By Size

Intro to Web Scraping w/ Puppeteer

Fast JSON & Rails

Rails Validations

Using the `Function` constructor