DEV Community: Matt Wing

📅 The Most Efficient Self-Taught Learning is Routine

Matt Wing — Tue, 26 Apr 2022 03:27:58 +0000

I love to learn, but I hate finding time for it.

We all only have 24 hours in a day. Between work, sleep, and everything else, you're probably juggling a lot already! The challenge for anyone doing self-taught learning is finding out how much time you can set aside consistently — fitting it into your regular routine, the cadence of your daily life.

Taking two hours to dive into something every other Saturday is a great supplement to your regular cadence. But if that's all you can do, you're gonna spend a good chunk of each time block just trying to remember where you left off.

If you can set aside even a half hour every weekday, or even just three days a week, you will feel the progress. And any big blocks of time you can snag will be infinitely more productive!

Fair warning that, best case, you'll be taking this time from your solo free time. Yes, that still sucks. But it's still better than depriving yourself of sleep, or neglecting your relationships.

It's common for people trying to learn new skills to be drained by the time they get time for it. I found a way around this in my schedule by doing it first thing in the morning, before I hop into any other work. This works for me because I'm already awake a couple hours before my job typically starts.

Everyone's schedule is different. As long as you're making the time and protecting it wherever it lands on your schedule, you're moving forward along your path. It doesn't have to be a huge long session each time, and it doesn't even have to be every day. It just needs to be consistent!

🔍 Parsing Schema Data with node-html-parser

Matt Wing — Wed, 29 Sep 2021 15:29:23 +0000

Did you know that there's a whole JSON object schema for providing machine-readable information about the contents of your website? Google uses the data in these objects to fill out search results and build rich snippets.

Here's a secret - it can power other stuff too. For example, I'm building a Node JS web app that includes the ability to plug a URL in and get a list of that recipe's ingredients.

Want to start parsing data yourself? Read on!

Challenges

Fetching the raw HTML
Making the Raw HTML Parse-able
Finding the right Schema object out of all of the ones on the page
Grabbing the right data out of that schema object

Fetching the raw HTML

First things first — we want to be able to fetch the HTML code of whatever link we end up pasting into our app.

There are a lot of ways to do this in Node JS. For this tutorial, we'll be using the native JavaScript fetch API.

With that in mind, here's how to make fetch happen:

// Use an async function so we can wait for the fetch to complete
async function getHtmlStringFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      // responseHtml is a huge string containing the entire web page HTML.
      // In the next section, we'll process it into something we can work with
    })
  );
}

Making the Raw HTML Parse-able

When we first fetch a URL and grab the response body, it's one enormous text string. There's HTML in there, but we can't really work with it yet. We need to plug this string into an HTML parser that will let us use DOM selectors to pick out the useful bits.

node-html-parser is my personal choice for this. It lets us use all the usual JavaScript DOM selector methods, and it's pretty fast too. Add it to your project with this terminal command:

yarn add node-html-parser

Then import the parse command from the package into the JS file where you'll be using it:

import { parse } from "node-html-parser";

Now we can take the response body string, plug it into our parser, and get to the real fun:

import { parse } from "node-html-parser";

async function getHtmlDocumentFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
            // parse the HTML string into a DOM-like object we can navigate
      const document = parse(responseHtml);
    })
  );
}

That's all we need to get the HTML into something we can sift through! The returned object has all the same methods as a typical document object, such as querySelector, getElementByID, and so on.

So, how do we work it to find the structured data we're looking for?

Finding the right Schema object(s)

The nice thing about working with structured data is that you can make some assumptions about the data you're processing, because it has to be structured in a way that web crawlers can understand to be useful.

The structured data Schema objects we're looking for are going to be found within ld+json script tags. Now that we've DOMified the HTML, we can run queries on it like this:

import { parse } from "node-html-parser";

async function getSchemaNodeListFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      const document = parse(responseHtml);
            // Create a NodeList of elements containing the page's structured data JSON. So close to useful!
            const structuredData = document.querySelectorAll('script[type="application/ld+json"]')
    })
  );
}

That will give us a NodeList of all the matching elements. That's close to perfect, but it's not a true array and could give us errors if we try to treat it like one (which we will soon). So let's turn it into an array:

import { parse } from "node-html-parser";

async function getSchemaArrayFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      const document = parse(responseHtml);
            // Create an ARRAY of elements containing the page's structured data JSON. Just one more step!
            const structuredData = Array.from(document.querySelectorAll('script[type="application/ld+json"]'));
    })
  );
}

Now we have an array of structured data nodes. In a way, we're back to square one with data that is so close to being useful. To make it useful, we need to grab the innerHTML of each node, which will come out as a big string. Then we can parse that into ✨real JSON!✨


import { parse } from "node-html-parser";

async function getJsonFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      const document = parse(responseHtml);
            const structuredData = Array.from(document.querySelectorAll('script[type="application/ld+json"]'));
        // Get an array containing the contents of each structured data element on the page. This is the ✨useful stuff✨
            const structuredDataJson = structuredData.map((node) => JSON.parse(node.innerHTML)).flat();
            // We also flatten the array with .flat() to handle how some sites structure their schema data. See epilogue for more info
        })
  );
}

Whoa, look at us. We've got the real, actual JSON object now. If you log structuredDataJson to your console, you'll see an array of structured data objects! Huzzah 🎉

But of course, we're not done yet! There is likely to be a ton of data you don't need in this array, in addition to whatever you're actually looking for.

Grabbing the right data out of that schema object

You're looking for some sort of specific data out of these objects. In my case, I'm looking for the list of ingredients within the Recipe object. So, now that we have actual JSON, we can view certain properties and use it to whittle our array down to a single, useful, piece of data:

import { parse } from "node-html-parser";

async function getIngredientsFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      const document = parse(responseHtml);
            const structuredData = Array.from(document.querySelectorAll('script[type="application/ld+json"]'));
            const structuredDataJson = structuredData.map((node) => JSON.parse(node.innerHTML)).flat();
            // Look for a Recipe schema and return its ingredients if it exists     
            const recipeData = structuredDataJson.find((schema) => schema["@type"] == "Recipe")
            if (recipeData) {
                return recipeData.recipeIngredient
              } else return null;
        })
  );
}

If one of the structured data objects is for a Recipe, we'll get the array of ingredients we're looking for. If not, the function will return null so we know it failed to find what we were looking for.

That's it! We've parsed the HTML into JSON into the actual thing we need 🎉

Conclusion

At this point, you have a function that takes a URL and returns an array of whatever information you're looking for. This general process can be used to do a whole lot of interesting stuff depending on what you're grabbing. Here's an example I put together to grab the ingredients within a recipe page.

Here are some of the most common schemas out there for inspiration. In my case, I'm parsing recipe ingredients so I can see if they're in my pantry, and add them to my shopping list if they're not.

How about you? If you end up using this process to parse website data in your web app, let me know what you're doing!

Epilogue: Handling Edge Cases with the flat() method

As mentioned earlier, structured data has to be readable by web crawlers to be useful, so we can make some assumptions about what it will look like. Still, we're ultimately trusting people to build their websites according to a certain convention, so you still might run into some issues across different websites and pages.

When I was testing my recipe parser, I ran into a few websites that structured their data in non-standard ways, which caused some trouble early on. The most common issue I found was that some sites would wrap their schema JSON within an array. This prevented my array.find() method from finding any of the data within the nested array.

In my production code, I handle this by flattening the parsed JSON to remove any nested arrays before I start looking for specific data. Here's what that looks like with the example code we've been using:

import { parse } from "node-html-parser";

async function getHtmlFromUrl(url) {
  return await fetch(url).then((response) =>
    response.text().then((responseHtml) => {
      const document = parse(responseHtml);
            const structuredData = Array.from(document.querySelectorAll('script[type="application/ld+json"]'));
            // Adding .flat() to the line below handles the most common edge cases I've found so far! 
            const structuredDataJson = structuredData.map((node) => JSON.parse(node.innerHTML)).flat();
            const recipeData = structuredDataJson.find((schema) => schema["@type"] == "Recipe")
            if (recipeData) {
                return recipeData.recipeIngredient
              } else return null;
        })
  );
}

Managing Side Projects to Account for Life

Matt Wing — Thu, 12 Mar 2020 05:25:15 +0000

Everyone loves a good side project! It's your chance to actually work with the shiny new technology you've read 12 tutorials about, to add some real code to your resume to show your proficiency with the shiniest new technologies, and re-learn just how difficult it can be to step outside your coding comfort zone.

It's common for coders to have a big barrel full of abandoned side projects. Each one is a milestone in a way. I personally have some concepts I've started from scratch on 3 or 4 times.

Much like the first few levels in World of Warcraft, the first few hours of a new project are full of excitement. Anything involving a package manager gives you those first moments where you watch things fly by in the console and feel like a master hacker as you sip on your beverage of choice. Then you plug in the example code and see it work like it did in the example! Just like that! Wow!

Eventually the joy ride ends as you come to terms with the vast gulf that spans between where your project is now, and its completed state. Invoking the magic words "mimimum viable product" can only do so much to reduce the scope of your dream when you're hoping to do something amazing. The rational part of your brain knows that spending an hour a day on this will get you to the finish line eventually.

That would be all we need for motivation if humans ran entirely on rational thought! Alas, most side projects do eventually fizzle out as we tire of using our spare time to bang our head against the wall in frustration. Even with a mild concussion, the dream that sparked the side project's creation never dies.

When you're ready to dive back into things, it's easy to feel overwhelmed by re-acquainting yourself with your old code. It's like loading up a game you almost beat six months ago and trying to remember what you were supposed to be doing.

How do we make the most of our fizzled-out projects to make the most of these bursts of renewed motivation? Here are a few suggestions from someone who has plenty of stalled projects to experiment with:

Document the heck out of them

Write up rough design documents. Map out what classes or functions you think you'll need to make things happen. You could even write tests for them that roughly describe how you expect them to work. These don't need to be detailed or fancy - they're just for you.

This does a few things for you:

Just like in "real" software projects, you'll be able to reconsider architecture and design decisions before you're halfway through implementing your first guess at how things should be. It's so much easier to draw a new diagram than it is to refactor your app from the ground up.
If (when?) your motivation for working on the project dries up, you can take a break and easily jump back in when you're ready. If you kept track of your progress too, you can just jump back in from where you left off.
If you want to start from scratch, your planning docs won't be tightly tied to specific code, so you can use a fancy new framework instead of the old framework you were excited about the last time you started things up.

I just did this in the aftermath of my most recent project hiatus. I wrote out a rough outline of the type of data the most complex objects would need to hold onto, some pseudocode for the most complicated functions, and a few basic unit tests. I found that it's easier for me to come up with unit testing ideas on paper than in the IDE - maybe because I don't get hung up on getting the syntax perfect as I'm trying to come up with them. What will you find easier to plan on paper? Only one way to find out!

Use what you've learned for next time

So... you didn't write up any design documents. That's okay! You moved fast, broke things, and got out of there. We've all done the same. Chances are good that you got stuck on a hard problem. In web development, running into one of those usually means one of three things:

The real problem is that you don't fully understand what's going on, which makes a small problem seem like a huge problem.
You're going to learn that someone solved that problem for you in a library - thanks, open source software!
You are a pioneer staking your claim on something that, truly, no one has done before. If you solve this, you can become the hero in the last scenario.

The first two situations are pretty common, at least in my experience. I've come back to projects before and found that the biggest problems had melted away.

The third scenario is why people say things like "the things worth doing in life are hard." If you run into a truly unique problem and keep puttering away at it, you'll become a master of that problem's domain before you know it.

No matter what stopped you in your tracks, your experience taught you something that no tutorial was going to show you. Maybe you learned the limits of the shiny new technology that looked so effortless in the demos. Maybe you found the edges of a library's helpfulness, or a bug you can report. No matter what happened, you learned something from it, so the next time you start a side project, you'll be that much more knowledgeable about how to make it work.

Like I said before, these are the practices I'm working on for myself in order to make the most of my personal programming time. What do you do to improve your chances of taking a side project to the finish line?