DEV Community: Wayne Smallman

Saving web pages with Vue and Node via Newspaper3k

Wayne Smallman — Tue, 01 Sep 2020 12:46:24 +0000

While the Under Cloud has an extension for Google Chrome that allows us to save a selection of text from a web page, what’s been lacking is the option to automate the saving of the entire page. Obvious though it is, saving a web page is no trivial task, and it’s something I’ve been 7 parts preparing for, 2 parts avoiding, and 1 part dreading for ages!

Yet, here we are — at last — and the Under Cloud now supports the saving of web pages via Newspaper3k, a versatile package written in Python. I am stretching the definition of now, since I’m still running tests in the staging environment, but it’s almost complete and should be on production within the week.

The documentation for Newspaper is sparse, and code samples were (are) few and far between. Worse, I had no idea how I would make Python talk to Node — the API is the obvious choice here, but I had no understanding of Python, the types of data it supported, or how I would get that data out of it.

I’m writing this from the perspective of someone on the other side of the learning curve, having walked the long route to get here, but — given the time constraints I’m up against — would have preferred a path less cluttered with obstacles. So this article is from present me for the attention of past me.

Alternatives to Newspaper3k

There are powerful services out there, such as DiffBot, but these are cost-prohibitive at this stage in the life of the Under Cloud, and — to be honest, and in spite of what I said a few paragraphs ago — I’d prefer to figure these things out myself first before delegating them, so I at least have a good technical understanding of what’s going on. However, there are some open source alternatives, such as BeautifulSoup.

Newspaper3k versus BeautifulSoup

I imagine some are wondering why I chose Newspaper3k instead of BeautifulSoup:

Newspaper appears to be focused on general purpose page scraping;
while BeautifulSoup — with its wealth of options for parsing the DOM — is geared more towards data science.

You need to know the specific parts of a web page to get the most from BeautifulSoup. I could be wrong, so I look forward to someone stepping in with more information!

Scraping a web page with Newspaper3k

I'm going to make a few assumptions:

you have an understanding of both Vue and Node;
and don’t need me to go through the whole process of installing and configuring either;
or instantiating a new project;
you have Python installed, along with the Newspaper3k package;
I’ll be providing concise examples of the code, rather than the complete versions.

As an aside, I don’t like scraping as a description of what we’re doing here, given the horrible connotations attached to it. Please don’t use this article to create nefarious garbage for the purposes of plagiarising the work of others.

Python

Although the Under Cloud is written in JavaScript (or ECMAScript, as it’s now known), the first thing I had to do was learn some Python to create the script that would act as a bridge between the backend written in Node and Newspaper written in Python:

import os
import sys
import json
from datetime import datetime

from newspaper import Article

# Here, the `url` value should be something like: https://www.bbc.co.uk/sport/football/53944598
url = sys.argv[1]

template_for_exceptions = "An exception of type {0} occurred. Arguments:\n{1!r}"

def get_web_page(url):

  try:

    if url and len(url) > 0:

      article = Article(url, keep_article_html = True)
      article.download()
      article.parse()

      dataForBookmarkAsJSON = json.dumps({
        'publicationDate': article.publish_date if article.publish_date is None else article.publish_date.strftime("%Y-%m-%d %H:%M:%S"),
        'title': article.title,
        'note': article.article_html,
        'authors': article.authors
      })

      try:

        sys.stdout.write(dataForBookmarkAsJSON)
        sys.stdout.flush()
        os._exit(0)

      except Exception as ex:

        message_for_exception = template_for_exceptions.format(type(ex).__name__, ex.args)
        print(message_for_exception)
        sys.exit(1)

  except Exception as ex:

    message_for_exception = template_for_exceptions.format(type(ex).__name__, ex.args)
    print(message_for_exception)
    sys.exit(1)

if __name__ == '__main__':
  get_web_page(url)

A few things to point out here, such as the article.publish_date variable, which is either a date string that I format, or is a null, that I handle when populating the JSON object. Yes, I could have done that upstream in Node, but I took the moment to learn a few things about and in Python.

Vue

At the frontend, I’m using a component with the following method:

getWebPage () {
  this.$axios.get(`/newspaper`, {
    params: {
      // Params.
    }
  }).then(function(response) {
    // Handle the response.
    }
  }).catch(function(error) {
    // Handle the error.
  })
}

Node

At the backend, I have the route:

router.get('/newspaper', async (req, res) => {
  const getNewspaper = await controllerNewspaper.getWebPage(data)
  res.json(getNewspaper)
})

… and in the controller, I have:

services.getWebPage = async (params) => {

  let { spawn } = require('child_process')
  let processForPython = spawn(process.env.PYTHON_VERSION, [
    `${process.env.PYTHON_PATH}/get_web_page.py`, 
    params.url
  ], {
    maxBuffer: 10240000
  })

  let dataForBookmarkStream = []

  return new Promise ((resolve, reject) => {
    processForPython.stdout.on('data', (response) => {
      dataForBookmarkStream.push(response)
    })

    processForPython.stderr.on('data', (error) => {
      reject({
        error: `An error occurred while attempting to parse the web page: ${error.toString()}`
      })
    })

    processForPython.on('exit', (code) => {
      switch (code) {
        case 0:
          if ( dataForBookmarkStream ) {
            if ( dataForBookmarkStream.length > 0 ) {
              try {
                try {
                  dataForBookmark = JSON.parse(dataForBookmarkStream.join().toString())
                } catch (exception) {
                  reject({
                    error: "JSON object supplied by Newspaper is invalid."
                  })
                }
                if (typeof dataForBookmark === 'object') {
                  const paramsForBookmark = new URLSearchParams()
                  paramsForBookmark.append('userID', params.userID)
                  // Additional parameters, using dataForBookmark...
                  instanceOfAxios.post('/assets', paramsForBookmark)
                  .then(function (response) {
                    resolve(response)
                  })
                  .catch(function (error) {
                    reject(error)
                  })
                }
              } catch (exception) {
                reject({
                  error: "An error occurred while attempting to save the web page."
                })
              }
            } else {
              reject()
            }
          } else {
            reject()
          }
          break
        case 1:
          reject({
            error: "Web page couldn't be saved."
          })
          break
      }
    })

  }).catch(error => {
    return {
      error: "Web page couldn't be saved."
    }
  })

}

Yeah, it’s a lot to take in, so let’s look at some specifics…

First, figure out what the version of Python is and create an equivalent environmental variable to process.env.PYTHON_VERSION.

Second, figure out what the path to Python is and create an equivalent environmental variable to process.env.PYTHON_PATH.

Then, feel free to tweak maxBuffer to fit. As an aside, I did attempt a version of the code using maxBuffer alone, but some web pages were too big, at which point the JSON object failed to parse and then everything went to crap.

Once the Python script is called, it begins to stream the JSON object to processForPython.stdout.on('data'), which I’m grabbing in chunks via the dataForBookmarkStream variable.

Assuming the process was a success, we hit the switch block in processForPython.on('exit') and exit when the code is 0. Here’s where we convert the encoded data in dataForBookmarkStream into something useful, using:

dataForBookmark = JSON.parse(dataForBookmarkStream.join().toString())

… before sending the data via the API to somewhere else in the application.

Do we have some Node and Python people shaking their collective heads wearing an avuncular expression with a hint of disappointment? If so, share and let’s learn what could be improved!

Our brains aren't hard drives, and how we remember things and make connections between them is personal — the Under Cloud is the missing link in the evolution of doing research.

Side projects as an employee KPI

Wayne Smallman — Wed, 05 Aug 2020 12:01:57 +0000

Working long hours — at the office, a coffee house, or at home — is one of the more obvious indicators that an employee is dedicated to their job, but is it indicative of how productive or how good they are?

It's possible the long hours are a sign that someone is struggling to accomplish in 5 hours what could or should have been done in 1 or 2, and if that's the case, the job is encroaching on their personal life, which in turn would have a commensurate negative affect on their work like — and that's when the downward cycle begins. Unchecked, these long hours would have negative consequences for both employer and and employee alike.

Gmail, Google Maps, Twitter, Slack, and Groupon started started as side projects, as did Under Cloud, the digital research assistant, which I've since spun out as a startup of sorts.

Side projects should be considered key performance indicator of an employee, something I imagine Google figured out a long time ago, giving rise to their famous 20% time set aside for their employees to focus on side projects. The powers at Google are no fools, and the fruits of that 20% me time become Google products, but that needn't be the case for everyone else.

"Should I tell my boss I have a side project?"

… is a question I see asked a lot while on a number of groups for entrepreneurs, developers, and designers to talk, share ideas, and learn from their peers. If it were me, I’d want to know if an employee had a side project, not because I’d feel threatened, but because I’d like to help, where possible. As an employer, I see side projects as evidence of some notable qualities:

proactive, forward thinking, and a willingness to look beyond their immediate environment;
inventiveness, and a mind for both identifying and fixing problems, or seeking out new and novel niches;
and a demonstration of a commitment to self improvement, and a desire to learn.

These are commendable qualities that are beneficial to the employer, too — more so if these side projects are work-related activities. It's feasible an employee could be working on a side project without me knowing — and that's not a problem in and of itself — but it's inconceivable that their out-of-hours dedication to their craft wouldn't be of benefit Octane to some extent, since they'd either be:

learning new or honing existing skills;
collaborating with others and improving their own project management and communication skills;
and perhaps learning to make more effective use of their time, and — through trial and error — create a productive work-life balance.

I'm sure there are other benefits, so what'd I not include, and what do you think of side projects?

Image by Free-Photos from Pixabay

Share functions between Vue and Node

Wayne Smallman — Fri, 24 Jul 2020 08:52:31 +0000

Vue and Node make a powerful and versatile team, but aside from the API — via Express for me — Vue doesn't talk to Node, or vice versa, and I had to fix that.

So there I was, building the front end, writing the code for a utilities mixin to store functions to handle common string and number operations. But then I realized that I had also written much the same code in a controller for Node.

What if I could use the same functions in Node and Vue?

So, first I created a file in Node that would contain the core functions I'd be using in the mixin in Vue:

// "utilities.js" in Node.

let utilities = {}

utilities.stripTags = (note, hellip = false) => {
  let removeHTML = note.replace(/(<([^>]+)>)/ig, "")
  let removeHTMLEntities = removeHTML.replace(/&#{0,1}[a-z0-9]+;/ig, "")
  let removeLineBreaks = removeHTMLEntities.replace(/\r?\n|\r/g, " ")
  let removeNonbreakingSpaces = removeLineBreaks.replace(/&nbsp;/ig, " ")
  let removeEmojies = removeNonbreakingSpaces.replace(/[\u1000-\uFFFF]+/g, " ")
  let string = removeEmojies
  return (hellip) ? string.concat('&hellip;') : string
}

module.exports = utilities

… then, I returned to the mixin:

// "utilities.js" in Vue.

'use strict'

// Core modules.
import CoreUtilities from 'path_to/utilities'

export const Utilities = {
  data () {
    return {
      // Reactive variables...
    }
  },
  methods: {
    stripTags (note, hellip = false) {
      // Utilizing a method in core utilities.
      return CoreUtilities.stripTags(note, hellip)
    }
  }
}

Now I have one definitive source of functions that both Vue and Node are able to access.

If anyone knows a more efficient method of replacing HTML, please share!

Image by Dyanap from Pixabay

Fantastic experts, and where to find them!

Wayne Smallman — Thu, 23 Jul 2020 14:23:17 +0000

A friend is encouraging me to start a podcast, and the immediate task is to build a list of at least 20 experts in the field of research — in the public or private sector — to discuss their methods, experiences, successes, and failures among other things.

If you could listen to anyone: who would it be; and what would you ask them?

As a bit of background, the purpose is to support the product I'm working on, the Under Cloud, with a range of resources from experts designed for the target audience of student and professional journalists.

How do you maintain momentum?

Wayne Smallman — Sun, 19 Jul 2020 09:28:08 +0000

If planning is a science, then timing is an art, and the more complex a plan is, the more critical time — and timing — becomes.

So it's vital I maintain momentum, or risk the slump, the sag, or the flop! Low places that are difficult to climb out from, and are places I hate to be.

This is a Sunday morning here in the north of England, and I've been flat out, cramming as much work as is feasible before I go for a 9-10k walk at 11am.

I run multiple threads of things, where each delivers something on short times scales — like improving the UX of a feature, refactoring some code to test a new thing I've learned and so on — but contribute to that greater whole.

Where the slump, the sag, and the flop come into things is when I have to reach out, ask for help, need authorization, require some essential piece of data or information, and I wait — then wait, and do some more waiting.

I mitigate against this by switching to some other thread, to keep the momentum going.

I get it, no-one but me works on Smallman Standard Time!

So, how do you maintain momentum?

Useful functions and commands for Bash

Wayne Smallman — Tue, 07 Jul 2020 12:28:07 +0000

I'll be honest, the command line was once a terrifying place. I remember when Mac OS X launched, and there it was, the Terminal — the name encapsulated the fear I had towards it.

But I persisted, and what was once fear is now a friendship (though not reciprocal).

Taming the Beast took a while, for sure, but I'm at a point where I'm able to do useful things in Bash, like create aliases and functions.

So here are two functions I use a lot, which I've decided to share in the hope someone finds them useful, too!

Before taking the plunge, be sure to create a copy of ".bash_profile" first!

nano ~/.bash_profile

Also, feel free to rename the functions.

Edit a file with a native application

I'm no command line hero, so when it comes to editing big files and making multi-line edits, I like to use something native:

wsef () {
    # edit file.
    # Usage: wsef file_name
    # Default application.
    APP="BBEdit"
    if [ "$1" ]; then
        open -a $APP $1
    else
        echo "A file name is required."
        return
    fi
}

If required, swap out "BBEdit" for something else.

Find a command

The up arrow key is perfect for the first ten previous commands, but then I need something else:

wsfc () {
    # find command.
    # Usage: wsfc term
    if [ "$1" ]; then
        history | grep -i $1
    else
        echo "A search term is required."
        return
    fi
}

Also, as an added a bonus:

export HISTTIMEFORMAT="%d/%m/%y %T "

... which adds a timestamp to the search results, or whenever you run the history command.

Extras

If you're a cat lover like me:

cat -n name_of_file

... and you'll see line numbers.

And for those who've got themselves stuck in vi as I have:

press "i"
make an edit, if you dare
press "esc"
write ":wq"
then press enter

Have fun!