DEV Community: Jason F

Web Scraping With Puppeteer for Total Noobs: Part 3

Jason F — Thu, 02 Mar 2023 22:00:44 +0000

Hello and welcome to the third and final post of this series. If you missed the previous, please feel free to give them a quick read before proceeding. In this post I'll show you how I set up a GitHub Action to automatically scrape some weather data on a schedule.

Before we begin I must tell you that I changed the source of the weather data due to some complications with weather.com. I'm sure it's possible to get around the issues I was having, but for the sake of moving forward without the extra kerfuffle, I changed the source to one that I believe will not cause you any grief.

Note: This post assumes you're comfortable with creating and updating a GitHub repository

Catching up

In the previous post I shared with you the scrape function that we used to scrape weather.com and console.log the output. I'll go ahead and paste the updated scrape function, with the new source below:

async function scrape() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(
    "https://www.weathertab.com/en/d/united-states/texas/austin/"
  );

  const weatherData = await page.evaluate(() =>
    Array.from(document.querySelectorAll("tr.fct_day"), (e) => ({
      dayOfMonth: e.querySelector("td > div.text-left > .fct-day-of-month")
        .innerText,
      dayName: e.querySelector("td > div.text-left > .fct-day-of-week")
        .innerText,
      weatherIcon: e.querySelector("td.text-center > .fct_daily_icon")
        .classList[1],
      weatherIconPercent: e
        .querySelector("td.text-center")
        .getElementsByTagName("div")[1].innerText,
      highTempRange: e.querySelector(
        "td.text-center > .F > div > .label-danger"
      ).innerText,
      lowTempRange: e
        .querySelector("td.text-center > .F")
        .getElementsByTagName("div")[1].innerText,
    }))
  );

  await browser.close();
  return weatherData;
}

As you can see, the source is now https://www.weathertab.com/en/d/united-states/texas/austin/, of course you can change this to whatever city you fancy. The logic is practically the same (as in we're creating an array of data), however, since we switched sources, of course we have to target the elements in the page, which are without a doubt different than the weather.com page. Rather than a 10-day forecast, we're now getting the entire current month's forecast.

The data is a bit odd without some context. However, I'm sure you could develop some kind of map or object that would translate the data, such as the weather icon and weather icon percent to something meaningful.

Anyways, on with the show.

Preparing to write the data

In order for us to reach the end goal of writing this scraped data to a .json file in a GitHub repository, we'll need to add some final touches to the scraper.js file. Admittedly, I'm no expert when it comes interacting with the file system using node.js (or any language for that matter). However, I can fanagle my way around. I'll share with you what I've got, and if you're more knowledgeable than me on the subject, please feel free to fill in the blanks.

We want to write our scraped data to a file in our repository that we'll name weatherdata.json.

In order to do so, we'll have this line at the end of our scraper.js file:

// execute and persist data
scrape().then((data) => {
  // persist data
  fs.writeFileSync(path.resolve(pathToData), JSON.stringify(data, null, 2));
});

the writeFileSync method is part of the node.js filesystem module. You can loearn more about this method here. Essentially what we're doing with it is passing in the path to our file, weatherdata.json as the first parameter, and our scraped data as the second parameter. The writeFileSync method will create the file if it doesn't exist, or, overwrite the file with the new data if it does exist.

As I mentioned, our first parameter is the path to the weatherdata.json file, which is passed in to the writeFileSync like so: path.resolve(pathToData).

Since we are using ES Modules, __filename and __dirname are not readily available. I shamelessly cut and pasted the code below (from this source) in order to create the pathToData variable.

const __filename = fileURLToPath(import.meta.url);

const __dirname = path.dirname(__filename);

const pathToData = path.join(__dirname, "weatherdata.json");

Having the __dirname available helps us find the path to our weatherdata.json file.

That wraps up all of the changes to the scraper.js file, minus the import statements at the top. I'll share the whole file so that it may be easier to read.

import puppeteer from "puppeteer";
import fs from "fs";
import path from "path";
import { fileURLToPath } from "url";

const __filename = fileURLToPath(import.meta.url);

const __dirname = path.dirname(__filename);

const pathToData = path.join(__dirname, "weatherdata.json");

async function scrape() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(
    "https://www.weathertab.com/en/d/united-states/texas/austin/"
  );

  const weatherData = await page.evaluate(() =>
    Array.from(document.querySelectorAll("tr.fct_day"), (e) => ({
      dayOfMonth: e.querySelector("td > div.text-left > .fct-day-of-month")
        .innerText,
      dayName: e.querySelector("td > div.text-left > .fct-day-of-week")
        .innerText,
      weatherIcon: e.querySelector("td.text-center > .fct_daily_icon")
        .classList[1],
      weatherIconPercent: e
        .querySelector("td.text-center")
        .getElementsByTagName("div")[1].innerText,
      highTempRange: e.querySelector(
        "td.text-center > .F > div > .label-danger"
      ).innerText,
      lowTempRange: e
        .querySelector("td.text-center > .F")
        .getElementsByTagName("div")[1].innerText,
    }))
  );

  await browser.close();
  return weatherData;
}

// execute and persist data
scrape().then((data) => {
  // persist data
  fs.writeFileSync(path.resolve(pathToData), JSON.stringify(data, null, 2));
});

Setting Up the GitHub Action

The goal of our GitHub Action is to scrape our weather source on a weekly basis and save the latest data in a .json file.

If you've never created an action, I'll walk you through steps.

First, find the Actions tab in your GitHub repository.

Click to go in to the Actions page. Next, click the "New Workflow" button. This will open up a page of options, however, we just want to select the "Set up a workflow yourself" options.

This will drop you in to an editor. Feel free to name the yaml file whatever you'd like, or leave it as main.yml.

Go ahead and paste this in:

name: Resources
on:
  schedule:
    - cron: "0 13 * * 1"
  # workflow_dispatch:
permissions:
  contents: write
jobs:
  resources:
    name: Scrape
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: 16
      - run: npm ci

      - name: Fetch resources
        run: node ./scraper.js

      - name: Update resources
        uses: test-room-7/action-update-file@v1
        with:
          file-path: weatherdata.json
          commit-msg: Update resources
          github-token: ${{ secrets.GITHUB_TOKEN }}

After pasting it in, you can commit the new yml file by clicking the "Start Commit" button and entering the commit message, etc.

So, there are a couple of things to note about this action:

The action is run on a schedule using the cron property. The value that I have in place says that the action will be triggered every Monday at 1300 (1PM).
Make sure your JavaScript file with the scrape method is named scraper.js, or, if it's not, update the action and replace scraper.js with your file name.
We are using the test-room-7/action-update-file@v1 to assist us with updating the scraper.js file. Look them up on GitHub if you'd like to know more about that action.

If you'd rather run the action whenever you please, comment out or delete these two lines:

schedule:
    - cron: "0 13 * * 1"

and uncomment this line:

# workflow_dispatch:

Then you'll see a button somewhere in the actions area that will allow you to run the job whenever you'd like. This works well for testing out actions.

You can see my repository here

Conclusion

I have learned a whole heck of a lot in this series. I stepped out of my comfort zone and was able to get something working. There were some hurdles along the way, but at the end of the day, it's all about getting my hands dirty and figuring things out. I hope you learned something as well. If you found these posts interesting, boring, lacking, great, etc...please let me know. Thank you!

Web Scraping With Puppeteer for Total Noobs: Part 2

Jason F — Thu, 02 Feb 2023 02:29:53 +0000

Hello and welcome to the second post in this series on web scraping with Puppeteer. If you missed the first post you can check it out here. In this post we'll pick up where we left off and scrape some weather data from weather.com. The current goal is to scrape the 10-day forecast of Austin, Texas. Feel free to swap out Austin for your favorite city.

Picking up where we left off

Our scrape function that we created in the previous post looks like this:

async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}

const scrapedData = await scrape();
console.log(scrapedData);

Let's now add to the weatherData. In addition to the innerText of the h3 , we'll get the high temperature, the low temperature, and the precipitation percentage for the day.

Let's have a look at how we can do that:

const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
        highTemp: e.querySelector(".DetailsSummary--highTempValue--3PjlX")
          .innerText,
        lowTemp: e.querySelector(".DetailsSummary--lowTempValue--2tesQ")
          .innerText,
        precipitationPercentage: e.querySelector(
          ".DetailsSummary--precip--1a98O"
        ).innerText,
      })
    )

As you can see I am adding three new properties to the object that's returned in the Array.from mapping function. These properties are highTemp, lowTemp, and precipitationPercentage. I found the class names by inspecting the document in the browser. These values seem to work, but only time will tell if something will have to be updated.

Let's now run node scraper.js in the terminal and check out the results:

[
  {
    date: 'Tonight',
    highTemp: '--',
    lowTemp: '31°',
    precipitationPercentage: '84%'
  },
  {
    date: 'Thu 02',
    highTemp: '41°',
    lowTemp: '32°',
    precipitationPercentage: '53%'
  },
  {
    date: 'Fri 03',
    highTemp: '55°',
    lowTemp: '30°',
    precipitationPercentage: '6%'
  },
  {
    date: 'Sat 04',
    highTemp: '57°',
    lowTemp: '40°',
    precipitationPercentage: '7%'
  },
  {
    date: 'Sun 05',
    highTemp: '64°',
    lowTemp: '47°',
    precipitationPercentage: '9%'
  },
  {
    date: 'Mon 06',
    highTemp: '71°',
    lowTemp: '58°',
    precipitationPercentage: '14%'
  },
  {
    date: 'Tue 07',
    highTemp: '68°',
    lowTemp: '50°',
    precipitationPercentage: '54%'
  },
  {
    date: 'Wed 08',
    highTemp: '60°',
    lowTemp: '47°',
    precipitationPercentage: '40%'
  },
  {
    date: 'Thu 09',
    highTemp: '60°',
    lowTemp: '42°',
    precipitationPercentage: '52%'
  },
  {
    date: 'Fri 10',
    highTemp: '62°',
    lowTemp: '38°',
    precipitationPercentage: '17%'
  },
  {
    date: 'Sat 11',
    highTemp: '59°',
    lowTemp: '42°',
    precipitationPercentage: '11%'
  },
  {
    date: 'Sun 12',
    highTemp: '64°',
    lowTemp: '48°',
    precipitationPercentage: '15%'
  },
  {
    date: 'Mon 13',
    highTemp: '67°',
    lowTemp: '51°',
    precipitationPercentage: '24%'
  },
  {
    date: 'Tue 14',
    highTemp: '71°',
    lowTemp: '51°',
    precipitationPercentage: '24%'
  },
  {
    date: 'Wed 15',
    highTemp: '70°',
    lowTemp: '50°',
    precipitationPercentage: '21%'
  }
]

Very cool. We're getting the values I'd expect to get.

GitHub Repo

I've set up a GitHub repository for this project. You can find the link here. Feel free to fork/clone this repository and play around. If you're not too comfortable with using git, there's a plethora of resources out there. If you'd be interested in a tutorial for noobs, please let me know in the comment section.

Wrapping up

In this post we were able to scrape a bit more weather forecast data and return it in our scrape function. In the next post I'll show you how to create a GitHub Action that will run the scrape function once a day and save the scraped weather data in a .json file in the same GitHub repository.

Web Scraping With Puppeteer for Total Noobs: Part 1

Jason F — Tue, 31 Jan 2023 02:28:46 +0000

Web scraping is something I never thought I'd do. I'm primarily a UI developer, although my career started as a backend developer. I've never been asked at a job to perform any scraping tasks, and I'd never had a personal project that required me to scrape some data until recently. I'll share what I've learned, which is honestly probably just scratching the surface of what you can do with a technology like Puppeteer. In this post I'll walk you through installing and writing a Node.js script that will scrape a weather forecast website. The weather forecast site isn't my personal project that I mentioned earlier, it's more of a contrived example of how to get started with web scraping using Node.js and Puppeteer.

This post assumes that you have some knowledge of Node.js, NPM, async/await, and the command line.

This is intended to be a multi-series post/tutorial. We'll start off slow, and eventually have a project that will net us a 10 day weather forecast in json format.

Disclaimer: I'm no expert in Node.js and certainly not a Puppeteer expert, so, if you see anything that could be done in a more proficient manner, or that's just plain wrong, please let me know. Thanks.

Note: I'll be using npm in this post. Feel free to use your preferred package manager if you're comfortable doing so.

Project Setup

The website that we'll be scraping is weather.com. We'll work towards getting the 10-day forecast of Austin, Texas. You can most certainly swap out the city for your preferred city.

Let's go ahead and create a new directory for our project:
mkdir weather-scraper

Now navigate in to the directory:
cd weather-scraper

Let's initialize the project (I'll use the -y flag to skip all the questions):
npm init -y

Next, I'm going to go ahead and open my favorite editor and create a JavaScript file. I'll name mine scraper.js.

Before we get too far, let's add one line to the package.json file so that we can use the import declaration. Add this line:
"type": "module",

Your package.json should look something like this:

{
  "name": "weather-scraper",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^19.6.2"
  }
}

Let's now install Puppeteer.

npm i puppeteer

First Few Lines of Code

First things first, let's import puppeteer on line 1:

import puppeteer from 'puppeteer';

We'll create an async function called scrape and write out the first bit of code. The reason that this function is an async function will become quite clear in just a bit (we basically have to await everything).

async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();
}

We've created two variables, browser and page. The browser variable is created using puppeteer's launch method, which has a return type of Promise<Browser> . See this page for more info on the Browser type.

The page variable is created using the browser context's newPage method, which returns a type of Promise<Page>. See this page for more info on the Page class.

The Page class has a method that we'll use to navigate to the weather website we're trying to scrape, the goto method. This method takes in one parameter, the url, along with an optional options parameter which we won't use at this time. This method returns a type of Promise<HttpResonse | null>.

We'll add that next:
await page.goto('https://weather.com/weather/tenday/l/Austin+TX')

Our scraper.js file should look like this at this point:

import puppeteer from 'puppeteer';

async function scrape() {
    const browser = await puppeteer.launch({ dumpio: true });
    const page = await browser.newPage();

    await page.goto('https://weather.com/weather/tenday/l/Austin+TX')
}

Let the Scraping Begin

After the goto method, we can use the Page class's evaluate method to work some magic. Inside of the evaluate method is where we can write out function inside the page's context and have the result returned. Essentially, we'll write some code inside this method to get the data from the page that we want to get. I'm going to put the code below, then discuss:

async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}

The evaluate method is a higher-order function, meaning we can pass in another function as a parameter, which is exactly what we're doing. Inside the anonymous function that we pass in, we have access to the document object. If you go inspect the weather.com url that I shared, you should be able to find the DaypartDetails--DayPartDetail--2XOOV class.

We are using the Array.from method and passing in document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"), which will return a NodeList of all elements in the document with that class. The Array.from method has a second, optional parameter, which is a mapping function. We are using that mapping function to select all h3 elements inside of each element in the NodeList and assigning the value of it's innerText to a property that we call date;

Viewing Our Data

After the scrape function, add in these two lines and save:

const scrapedData = await scrape();
console.log(scrapedData);

Let's go to the terminal and run:
node scraper.js

We get results that look something like this:

[
  { date: 'Tonight' },
  { date: 'Tue 31' },
  { date: 'Wed 01' },
  { date: 'Thu 02' },
  { date: 'Fri 03' },
  { date: 'Sat 04' },
  { date: 'Sun 05' },
  { date: 'Mon 06' },
  { date: 'Tue 07' },
  { date: 'Wed 08' },
  { date: 'Thu 09' },
  { date: 'Fri 10' },
  { date: 'Sat 11' },
  { date: 'Sun 12' },
  { date: 'Mon 13' }
]

Very cool.

The entire scraper.js file looks like this:

import puppeteer from "puppeteer";

async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}

const scrapedData = await scrape();
console.log(scrapedData);

Wrapping Up

I implore you to play around with this. See if you're able to scrape other bits of data. In my next post I'll continue where I left off here. By the end of my web scraping posts I'll have shown you how to scrape, create a GitHub Action to automatically scrape, and have the GitHub Action save the data in to a .json file in the same repo.

I hope you found this post interesting. Thanks for sticking around. Until next time.

Create an array of all values in an object

Jason F — Mon, 23 Jan 2023 00:42:18 +0000

Suppose you want to create an array of all the values in an object. There is definitely more than one way to do that. We'll look at a few different approaches in this post.

`Object.values()` method

The Object.values static method was released in ECMASript 2017 aka ES8. This method accepts an object. It returns the objects enumberable property's values in an array.

Let's look at an example:

const obj = { a: 'hello', b: 'world', c: 'lorem', d: 'ipsum', e: 'dolor' };

const valuesArr = Object.values(obj);

console.log(valuesArr);

As you can see in the example, we passed in the obj object to the Object.values method. The returned value was assigned to valuesArr.

And here's the value of valuesArr:

[
  "hello",
  "world",
  "lorem",
  "ipsum",
  "dolor"
]

Let's take a look at different way we could do this.

`for...in` statement

Another way to create an array of all the values in an object is to use the for...in statement. The statement goes all the way back to ES5, so, it's been around for a bit. This statement iterates over the enumerable String properties of an obect with the exclusion of the symbol properties.

Using the same object from earlier, let's check out an example:

const obj = { a: 'hello', b: 'world', c: 'lorem', d: 'ipsum', e: 'dolor' };

const valuesArr = [];

for (const prop in obj) {
  valuesArr.push(obj[prop]);
}

console.log(valuesArr);

In the example we declare an empty array, valuesArr. We then use the for...in statement to iterate over the properties of our object, obj. The declaration, const prop in the for...in statement is actually the string property name of each property. Therefore, we can use that to access the value of each property like so: obj[prop]. We then push each value in to the valuesArr array.

And here are the values for valuesArr:

[
  "hello",
  "world",
  "lorem",
  "ipsum",
  "dolor"
]

Let's check out one more way to create an array of all the values in an object.

`Object.keys()` method

The Object.keys static method was has been around since ES5. This static method returns the objects enumberable property's keys in an array. We can use the Array.map method to help us get the value of each property. Let's jump in to an example:

const obj = { a: 'hello', b: 'world', c: 'lorem', d: 'ipsum', e: 'dolor' };

const valuesArr = Object.keys(obj).map(key => obj[key])

console.log(valuesArr);

In the example we are chaining on the Array.map method which receives the key of each property in our object, obj. We use the key to access the value of each property, just like we did in the for...in statement. The values are returned and assgned to valuesArr.

Let's have a look at the value of valuesArr:

[
  "hello",
  "world",
  "lorem",
  "ipsum",
  "dolor"
]

Conclusion

As I said in the intro, there's more than one way to create an array of all the values in an object in JavaScript. I didn't get in to the performance of these methods, but if that's something you'd be interested in, leave a comment below. If I didn't cover your preferred way of doing this, please share in a comment below.

Thank you for reading, until next time.

Adventures with JavaScript Arrays: Sorting an Array of Numbers

Jason F — Sun, 15 Jan 2023 18:41:01 +0000

Hello and welcome to the second Adventures with JavaScript Arrays post. If you haven't read the first post, you can find the it here. In this post we'll take a look at sorting an array of numbers.

Let's get started!

The sort method

The Array object in JavaScript contains a sort method that will sort the elements of an array in place. It does not return a new array. The sort method has one optional parameter, which is the compare function. If a compare function is not provided, the default sort logic will be applied. The default sort logic takes all undefined elements in the array, converts them to strings, and then sorts in ascending order. If the array has any undefined elements, they will be moved to the end of the array.

The default logic actually sounds pretty useful when you want to sort an array of strings in ascending order. However, if you want to sort an array of numbers, the default logic is probably not going to satisfy the requirement.

Let's have a look at an example:

const numbers = [1, 3, 100, 1000, 5];
numbers.sort();

If you're unfamiliar with the sort method, you may think that the value of numbers after the sort method is called to be

[1, 3, 5, 100, 1000]

However, the actual value is

[1, 3, 100, 1000, 5]

This is due to the default sort logic, the numbers are converted to strings, then compared. If we want to sort the numbers in numeric order, we'll have to supply a compare function.

Sort numbers using compare function

The compare function in the sort method is rather interesting.

sort((a, b) => { // do something } )

It has two parameters, the first is the first element to be compared (a in the example above), the second is the second element to be compared (b in the example above).

The compare function sort order is determined based on the returned value in comparison to the number zero. If a is less than b based on your compare function, a value of -1 is returned. If a is greater than b based on your compare function, a value of 1 is returned. Otherwise, the values are equal and a value of 0 is returned.

Per MDN:

That may sound confusing, so let's use an example. We'll sort the same array that we used earlier.

const numbers = [1, 3, 100, 1000, 5];
numbers.sort((a, b) => a - b)

The result of calling the sort method with the compare function is:
[ 1, 3, 5, 100, 1000 ]

By using the compare function we are telling JavaScript our ordering criterion. In our example, we are sorting in ascending order because the ordering criterion we defined is a - b.

If apply the logic of the comparator we'll see how we got to this result. Our unsorted array is [1, 3, 100, 1000, 5].

Let's start with the first two elements and compare their values 1 and 3.

1 - 3 = -2

The value of -2 is less than zero, therefore a is sorted before b. In this case, 1 is sorted before 3.

The same logic is applied to the rest of the array.

Sort Versatility

TheArray.sort is very versatile due to the ability to pass in a compare function. Strings and numbers can be sorted. More complex data structures can be sorted as well, as long as you provide a compare function that meets your requirements.

Conclusion

It is my hope that this post did not cause confusion. If it did, let's discuss in the comments section. I am having a great time revisiting some of the things in JavaScript I take for granted. I may do another array post next week, unsure yet. Thanks for reading!

Adventures with JavaScript Arrays: Random Numbers Array

Jason F — Tue, 10 Jan 2023 18:03:57 +0000

The JavaScript array is an ubiquitous data structure, thus I'd like to spend some time to get more familiar/reacquainted with the Array object. I'll share my knowledge with you in the form of blog posts.

In this post I'll create a function that returns an array of random whole numbers.

Let's get started!

function randomNumbersArray(length = 10) {
    // return array of random numbers between 1 and 100
    // if length parameter is not provided, length is set to 10
}

We know that we want to create an array of random whole numbers between 1 and 100 with either a specified length or 10. Let's first discuss the Array constructor.

Calling Array constructor with single number argument

With JavaScript's Array constructor you are able to call it with a single number argument. If you do this, an array will be created with the length of the number that is received. Each element will be undefined.

In the example above, I am calling the Array constructor with an argument of 10. As you can see, the length property is set to 10.

Array.from()

Array.from is a static method on the JavaScript Array object which will create a new array from an iterable/array-like object.

The first parameter of Array.from is an object. So we could do something like this:

We call Array.from with a string, which is an iterable object, and an array is created from that value.

Array.from has an optional second parameter which is a mapping function.

In the example above, we are using the mapping function to capitalize each letter in the first argument.

Putting it all together

Let's now take what we know about the JavaScript Array and create the randomNumbersArray function.

function randomNumbersArray(length = 10) {
  return Array.from(Array(length), () => Math.floor(Math.random() * 100) + 1);
}

Here's an example showing the output.

We are calling Array.from and passing in an array with a length of the value of the length argument that is created using the Array constructor as the first param, and we are passing in a mapping function that creates a random whole number between 1 and 100 for each element of the array as the second param.

We now have a function that creates an array, with a length of 10, unless a length argument is provided, of whole numbers between 1 and 100.

Conclusion

Of course there are multiple ways to create an array of random numbers in JavaScript. I thought this approach was interesting as it uses the Array object's constructor and the from static method. Thanks for reading! Until next time.

What The Facade?!

Jason F — Sun, 04 Dec 2022 22:17:40 +0000

What The Facade?!

Are your components hundreds of lines long? Are your services bare? Have you ever copy and pasted a function from one component in to another? If you answered yes to any of these, stick around. Today we'll be taking a look at how the Facade design pattern can help us write more maintainable code in an Angular application.

What is the Facade Design Pattern?

The Facade design pattern is a Gang of Four structural design pattern. The objective of the design pattern per the Gang of Four is to, "Provide a unified interface to a set of interfaces in a subsystem. Facade Pattern defines a higher-level interface that makes the subsystem easier to use." How does this translate to Angular? Imagine we're creating a todo list app and that we've already created the backend/API and we now need to make the UI. We could just make the calls to the API directly from our components, or, we could create a service that handles the API calls, handles any complex business logic, and just gives our component the data it needs to handle the view.

What are the benefits of the latter? Well, for starters, our component/s will be cleaner. They'll only need to interact with the service vs interacting with the API directly. Another benefit is that we can reuse the same methods over and over again, and only define them once. A third benefit is that if we need to swap out our API, or make any changes to the business logic, we only have to make those changes in one place.

Angular + Facade Pattern, A Perfect Match

Angular advocates for keeping your components lean and outsourcing your business logic to your services. Per the Dependency Injection section of the docs, "Angular distinguishes components from services to increase modularity and reusability. By separating a component's view-related features from other kinds of processing, you can make your component classes lean and efficient."

The Angular Style Guide reinforces this concept by stating, "Do limit logic in a component to only that required for the view. All other logic should be delegated to services."

We can now see that the Facade design pattern isn't some obscure pattern or buzzword you can throw around the office to sound like you know what you're doing. It appears that the Angular team is pretty much advocating for its use. Sure, you can go and create behemoth components and make services an afterthought. While the Angular docs do state that Angular doesn't enforce this separation of view and business logic, they've given us a framework that allows us to easily implement the Facade pattern.

Facade Pattern in Action

Let's use the Free Dictionary API and create a component that displays the definition of a word.

Here is the dictionary component template:

<h1>Coolest Dictionary Ever</h1>
<form #form="ngForm" (submit)="submit()">
  <label for="search-term">Search: </label>
  <input
    type="text"
    id="search-term"
    name="searchTermText"
    [(ngModel)]="searchTerm"
  />
  <button type="submit">Search</button>

  <div *ngIf="definitionArr">
    <h2>Definition:</h2>
    <ul>
      <li *ngFor="let definition of definitionArr">{{ definition }}</li>
    </ul>
  </div>
</form>

And here is the dictionary component class:

import { HttpClient } from '@angular/common/http';
import { Component, OnInit } from '@angular/core';
import { map } from 'rxjs/operators';

@Component({
  selector: 'app-dictionary',
  templateUrl: './dictionary.component.html',
  styleUrls: ['./dictionary.component.css'],
})
export class DictionaryComponent implements OnInit {
  constructor(private http: HttpClient) {}

  ngOnInit() {}

  searchTerm: string | null;
  definitionArr: string[] | null;

  submit() {
    if (this.searchTerm) {
      this.getDefinition();
    }
  }

  getDefinition() {
    this.http
      .get(`https://api.dictionaryapi.dev/api/v2/entries/en/${this.searchTerm}`)
      .pipe(
        map((result) =>
          result[0].meanings.map((meaning) => meaning.definitions[0].definition)
        )
      )
      .subscribe((result) => (this.definitionArr = result));
  }
}

As you can see, we've got a template-driven form. We display the definitions in the definitionArr array, if any exist.

We have two methods in the class, submit and getDefinition. The submit method is called when we submit the form by clicking the search button, which in turn calls getDefinition. The getDefinition method makes a call to the free dictionary API, then maps the result to the definitionArr property.

The code works, that means we can call it a day...right?

Let's ask a couple of questions:

What happens if we want to use the getDefinition elsewhere in the codebase?
How hard would it be to swap out the Free Dictionary API with a different API? Now imagine that we copy and pasted the getDefinition in a couple of other components and answer the question.

Let's Refactor

I'm going to create a dictionary service and move our business logic from the dictionary component to the dictionary service. Let's check it out:

We'll start with the dictionary component class:

import { Component, OnInit } from '@angular/core';
import { DictionaryService } from './dictionary.service';

@Component({
  selector: 'app-dictionary',
  templateUrl: './dictionary.component.html',
  styleUrls: ['./dictionary.component.css'],
})
export class DictionaryComponent implements OnInit {
  constructor(protected dictionaryService: DictionaryService) {}

  ngOnInit() {}

  searchTerm: string | null;

  submit() {
    if (this.searchTerm) {
      this.dictionaryService.getDefinition(this.searchTerm);
    }
  }
}

This looks a lot slimmer. We are no longer importing the HttpClient or the map operator. The getDefinition method is completely gone. We also see that the definitionsArr property is gone. We've injected the newly created DictionaryService and are using it in the submit method.

Next, let's look at the template:

<h1>Coolest Dictionary Ever</h1>
<form #form="ngForm" (submit)="submit()">
  <label for="search-term">Search: </label>
  <input
    type="text"
    id="search-term"
    name="searchTermText"
    [(ngModel)]="searchTerm"
  />
  <button type="submit">Search</button>

  <div>
    <h2>Definition:</h2>
    <ul>
      <li *ngFor="let definition of dictionaryService?.definitions$ | async">
        {{ definition }}
      </li>
    </ul>
  </div>
</form>

What's new here? Rather than iterating over the definitionsArr property, we're now iterating over the dictionary service's definitions$ observable using the async pipe.

Now let's check out the new DictionaryService:

import { HttpClient } from '@angular/common/http';
import { Injectable } from '@angular/core';
import { Subject } from 'rxjs';
import { map } from 'rxjs/operators';

@Injectable({
  providedIn: 'root',
})
export class DictionaryService {
  constructor(private http: HttpClient) {}

  private definitionsSubj = new Subject<string[]>();

  definitions$ = this.definitionsSubj.asObservable();

  getDefinition(searchTerm: string) {
    this.http
      .get(`https://api.dictionaryapi.dev/api/v2/entries/en/${searchTerm}`)
      .pipe(
        map((result) =>
          result[0].meanings.map((meaning) => meaning.definitions[0].definition)
        )
      )
      .subscribe((res) => this.definitionsSubj.next(res));
  }
}

We were able to use most of the logic from the component's getDefinition method. However, we're now calling next on the definitionsSubj Subject with the result. We've also created an Observable, definitions$. This is the Observable that we are using in the template file with the async pipe.

With this refactor we'd be able to use the getDefinition method anywhere in our app. Our dictionary component is cleaner. If we ever needed to swap out the Free Dictionary API with a different API, we'd only have to do that in one place. Our component should be easier to test. There's quite a few benefits that come along with the refactor.

Conclusion

If you're feeling like your components are too complex, maybe you've got the same method copy and pasted across several components, or maybe you're bringing in too many dependencies in your component which is making it harder to test and/or make changes to existing logic, it may be time to consider using the Facade pattern.

The example used today was just the tip of the iceberg. This pattern can be used to separate complex business logic from your component, make working with state management easier, handle communication with a backend service, and more.

If you want to check out the Stackblitz for this post, here's the link.

Thanks for reading.

Two years on Dev.to

Jason F — Thu, 07 Apr 2022 00:27:30 +0000

Wow, time flies. Today marks my two year mark on the platform. I appreciate this community. I've learned so much since joining. A big thank you to the Dev.to team for doing what you do!

What have I been up to? I started a new blog/portfolio and have wrote some posts over there. Check it out here.

I created my first NPM package

Jason F — Mon, 25 Oct 2021 02:04:29 +0000

Hello all! I am excited to announce that I have published my first NPM package. It's not super fancy or anything and is most certainly a work in progress. That being said, if anyone has time to check it out and provide feedback, here is the link. The library provides useful methods when working with dates in JavaScript/TypeScript. I wrote it in TypeScript, so you get type declarations right out of the box.

I'd appreciate any feedback. Also, if you want to contribute, just let me know :)

This has definitely been a learning experience for me. I live in my own little world as a front end developer. I install and use npm packages but never really considered that anyone can write one and publish it.

Looking for a project to contribute to this Hacktoberfest? I've got a few.

Jason F — Sun, 03 Oct 2021 18:38:41 +0000

Hello all! If you're looking for a project to contribute to, please feel free to check out my Hacktoberfest repo's below. All skill levels are welcome to contribute.

Projects

Here are my projects that are open for Hacktoberfest contributions.

News From The Wormhole

This project was built with React, Typescript, Tailwind, Vite, and the Spaceflight News API. This project lists news articles and blog posts that are related to space flight. You can find the repo link here, and can see the site here.

Motivational Cats

This project does not use a SPA framework. It uses plain ol' vanilla JS, Materialize CSS, and makes a couple of api calls (one to grab cat images, another to grab a motivational quote). This is this project's third Hacktoberfest! I kinda just let the Hacktoberfest contributions from the last two years drive the look and feel of the project and it's been fun to watch it change. It's in a broken state as of the time I'm writing this post. Please contribute, let's give this project another Hacktoberfest.
You can find the repo here and can see the site here.

Where is the ISS

This project shows you where the International Space Station is in real time. This project was made using Angular, Angular Material, and Leaflet. You can view the repo here, and see the site here.

Happy Hacking

Please feel free to work on any open issues in the projects I've listed. Also, if you find something you want to fix, or have an idea for a style change, or have an idea for an enhancement, then please create an issue in the project. I hope you all have a happy Hacktoberfest!

How I Set Up Wifi In Debian 11

Jason F — Tue, 14 Sep 2021 23:14:10 +0000

I just made the switch to Debian 11 last night. If you're new to Debian, you may be surprised to see that Wifi doesn't seem to work right out of the box. I'll walk you through the steps that I took to get Wifi up and running. Please note, I am by no means a Linux expert. I'm just a web developer who runs an older laptop and doesn't want to deal with Windows ever again.

If you just installed Debian 11 you may want to run through this article first (it won't help with the wifi, but some of these items are probably necessary/helpful).

Note: Your experience may vary based on your machine

Add non-free and contrib sources to your SourcesList

So...due to Debian's FOSS philosophy, you'll need to add some repo sources to your SourcesList. You can find the sources.list file here: /etc/apt/sources.list.

My sources.list file currently looks like this:



# Line commented out by installer because it failed to verify:
deb http://security.debian.org/debian-security bullseye-security main contrib
# Line commented out by installer because it failed to verify:
deb-src http://security.debian.org/debian-security bullseye-security main contrib

# bullseye-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
# A network mirror was not selected during install.  The following entries
# are provided as examples, but you should amend them as appropriate
# for your mirror of choice.
deb http://deb.debian.org/debian bullseye main contrib non-free
deb-src http://deb.debian.org/debian bullseye main contrib non-free

deb http://deb.debian.org/debian-security/ bullseye-security main contrib non-free
deb-src http://deb.debian.org/debian-security/ bullseye-security main contrib non-free

deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb-src http://deb.debian.org/debian bullseye-updates main contrib non-free
deb http://deb.debian.org/debian/ bullseye-updates main contrib

deb-src http://deb.debian.org/debian/ bullseye-updates main contrib

Notice that I've added the non-free and contrib repo sources. You'll want to do the same. You can learn more about the SourcesList here.

Note. I'm not sure which of these sources is the one that gave me access to the package I needed to get my wifi up and running, but am pretty sure it's the non-free

Figure out which wifi adapter you have

This is a super important step. You'll need to know which wifi adapter you have in order to know which package to install to use it. I accomplished this by using the lshw command. If you do not have 'lshw' installed you can easily install it by running sudo apt install lswh.

Once you have lshw installed, you can run sudo lshw -html > lshw.html. This will pipe out the results in to a nice html file for you.

Open the HTML file and look for an entry with a description of "network controller" or something similar. My entry looked like this:

I was able to see that I have the Wireless 7265 network controller from Intel. I did a quick internet search for "intel wireless 7265 debian 11" and found this link.
Note: You will need to search for your network controller and pray to your favorite deity that Debian has a package that will enable wifi for you

Install the necessary package

Now that you hopefully found the right package, it just needs to be installed. The iwlwifi page from Debian gave me a couple of command line instructions to run. Once installed, I was able to connect to my wifi. I did have to switch to the root user to run those commands. You can do that by running su - root.

Hope this helps

I hope at least one person out there finds this article helpful. Also, if you're a more experienced Linux user, please feel free to critique my post, give advice, etc..so long as it's helpful and not condescending. Thanks!

JavaScript Find vs Filter Array Methods.

Jason F — Sun, 06 Jun 2021 20:47:20 +0000

JavaScript's built in array methods are extremely useful. However, picking the right method for the job can be challenging. Today I'll describe use cases for the find and filter array methods.

Find method

The find method will return either the first element found that matches your criteria or undefined. For an in-depth definition of find, checkout this page on MDN.

Let's use the array below to try out the find method.

const persons = [
  { name: "Tom", age: 55, userId: 99000152, status: "full-time" },
  { name: "Miguel", age: 29, userId: 99000112, status: "full-time" },
  { name: "Jennifer", age: 42, userId: 97003211, status: "part-time" },
  { name: "Dean", age: 39, userId: 9800212, status: "part-time" },
  { name: "Jose", age: 33, userId: 99000972, status: "part-time" },
  { name: "Michelle", age: 23, userId: 89004444, status: "full-time" },
  { name: "Darren", age: 45, userId: 91000361, status: "full-time" },
  { name: "Gene", age: 24, userId: 93000345, status: "part-time" },
  { name: "Nancy", age: 35, userId: 99000111, status: "full-time" },
];

First, let's use the find method to find the person object with the name of Darren.

const found = persons.find((person) => person.name === "Darren");

If we were to log out the value of found, we would see the following:

{ name: 'Darren', age: 45, userId: 91000361, status: 'full-time' }

Perhaps we need to find the person whose id is 97003211. How could we accomplish that? With the find method of course.

const found = persons.find((person) => person.userId === 97003211);

So, the find method has one required argument, which is the current element in the array, in our case above we called that argument person. You can optionally use the index, and thisArg arguments.

As stated earlier, if the find method doesn't find what you are looking for, it will return undefined. Let's give that a shot.

Using the persons array again, let's search for someone whose age is over 100.

const found = persons.find((person) => person.age > 100);

If we log out the value of found now, we'll get undefined.

The find method is useful when you need to check if a particular value exists in an array.

Filter method

In the event that we needed to filter the array of persons based on a particular criteria, such as all persons over the age of 35, or, all persons whose name is longer than five characters, we could do that. The filter method is useful when you need to filter out all values that meet a certain criteria. This method always creates/returns a new array.

Let's say we need an array of that consists exclusively of all persons with a status of 'full-time'. We could do that like so:

const fullTimers = persons.filter((person) => person.status === "full-time");

If we log out the value of fillTimers we will get

[
  { name: 'Tom', age: 55, userId: 99000152, status: 'full-time' },
  { name: 'Miguel', age: 29, userId: 99000112, status: 'full-time' },
  { name: 'Michelle', age: 23, userId: 89004444, status: 'full-time' },
  { name: 'Darren', age: 45, userId: 91000361, status: 'full-time' },
  { name: 'Nancy', age: 35, userId: 99000111, status: 'full-time' }
]

The filter method, which you can read more in-depth about here, takes one required argument, which is the current element of the array. The optional args are the index that you're currently on, and the entire array.

Imagine now we want a list of everyone who isn't full-time. We could do that like so:

const nonFullTimers = persons.filter((person) => person.status !== "full-time");

And if we log this out, we'll see:

[
  { name: 'Jennifer', age: 42, userId: 97003211, status: 'part-time' },
  { name: 'Dean', age: 39, userId: 9800212, status: 'part-time' },
  { name: 'Jose', age: 33, userId: 99000972, status: 'part-time' },
  { name: 'Gene', age: 24, userId: 93000345, status: 'part-time' }
]

My Experience

There was a time where I was using the filter method when I could have been using find. For example, let's say I needed to find the person whose userId is 93000345. I would do something like this:

const found = persons.filter((person) => person.userId === 93000345);

The returned value of filter is an array as you can see below:

[ { name: 'Gene', age: 24, userId: 93000345, status: 'part-time' } ]

This creates some frustration because now if I needed to access the person's age for example, I would have to do something like this:

console.log(found[0].age);

If instead I used find to find the person whose userId is 93000345, which would look like this:

const found = persons.find((person) => person.userId === 93000345);

The value of found would be the person object, like so:

{ name: 'Gene', age: 24, userId: 93000345, status: 'part-time' }

And I could access the age prop like so:

console.log(found.age);

Conclusion

JavaScript array methods are awesome, but it can take time to learn about the different array methods and when to use each one. I hope you found this article useful. Thanks for reading.

DEV Community: Jason F

Web Scraping With Puppeteer for Total Noobs: Part 3

Catching up

Preparing to write the data

Setting Up the GitHub Action

Conclusion

Web Scraping With Puppeteer for Total Noobs: Part 2

Picking up where we left off

GitHub Repo

Wrapping up

Web Scraping With Puppeteer for Total Noobs: Part 1

Project Setup

First Few Lines of Code

Let the Scraping Begin

Viewing Our Data

Wrapping Up

Create an array of all values in an object

Object.values() method

for...in statement

Object.keys() method

Conclusion

Adventures with JavaScript Arrays: Sorting an Array of Numbers

The sort method

Sort numbers using compare function

Sort Versatility

Conclusion

Adventures with JavaScript Arrays: Random Numbers Array

Calling Array constructor with single number argument

Array.from()

Putting it all together

Conclusion

What The Facade?!

What The Facade?!

What is the Facade Design Pattern?

Angular + Facade Pattern, A Perfect Match

Facade Pattern in Action

Let's Refactor

Conclusion

Two years on Dev.to

I created my first NPM package

Looking for a project to contribute to this Hacktoberfest? I've got a few.

Projects

News From The Wormhole

Motivational Cats

Where is the ISS

Happy Hacking

How I Set Up Wifi In Debian 11

Add non-free and contrib sources to your SourcesList

Figure out which wifi adapter you have

Install the necessary package

Hope this helps

JavaScript Find vs Filter Array Methods.

Find method

Filter method

My Experience

Conclusion

`Object.values()` method

`for...in` statement

`Object.keys()` method