DEV Community: Hackers And Slackers

Serving Assets via CDN with Google Cloud

Todd Birchard — Sat, 23 Apr 2022 16:37:03 +0000

Pardon my nostalgia, but it's hard not to miss the good old days of the world-wide-web. Websites consisted of the most straightforward architecture: single-node servers. It's hard to imagine the internet of the 90s, where blogs, forums, and obscure Flash sites were almost exclusively hosted on simple LAMP stacks instead of a myriad of cloud services that AWS has convinced us we need. We didn't need entire VPCs serving as ecosystems to countless microservices. Instead, each of us spun up individual servers. By no coincidence, the vast majority of such US-based servers were located in Texas.

Cheap energy costs aside, Texas is notably equidistant between the east and west coasts of the United States. Most US citizens live in coastal states; thus, we can expect the majority of traffic to any US-based website to originate on opposite sides of the country.

Connections to a server need to travel physical distances, and traveling distance takes time, more commonly known as latency. Therefore, as our user base grows and strays further and further from our node in Texas, the average user will experience longer load times when fetching assets from our site.

For all the magic of the internet, no amount of magic can defy physics to "solve" latency. That said, we can implement something indistinguishable from magic 😉.

CDNs In A Nutshell

The "not quite magic" solution to our latency problem is a Content-Delivery Network. We can't control where our users are geographically, but we can control where we serve chunks of our app from. For example, why should users in Asia wait to load images (or other static assets) from our app's server in Texas, when there are quality data centers in Singapore? CDNs are designed to mitigate this very scenario.

A CDN is a distributed network of machines that spans countries and continents to serve assets closer to the users requesting them. For example, a user in Europe might fetch Asset X from a node in the Netherlands. In contrast, a user in the Philippines might fetch Asset X from an edge node in Manila. Our hypothetical app still resides in Texas, yet neither user had to wait for Asset X to make its way halfway across the world via a transatlantic cable.

Visualizing the architecture of a CDN

CDNs are Worth the Trouble

Performance improvements offered by CDNs have shifted from "borderline excessive" to "absolutely critical" for the success of many (if not most) internet entities. Latency has a measurable impact on the bottom line - you're probably familiar with the Amazon study that equated 100ms in additional latency to a 1% loss in revenue.

Depending on your business model (if you have one), an equally (or arguably more significant side effect) is the indirect impact latency has on SEO. Tools like Google's Lighthouse have brought excruciating visibility into how serving large uncached assets affects site ranking. The results can be brutal:

The negative impact of poor performance

By Google's standards, the performance of Hackersandslackers.com is God-awful. Unbeknownst to me, I'd likely been committing atrocities against Google's page rank algorithms for years prior to the availability of Lighthouse. This is particularly unfortunate for a site where Google is the single traffic source.

Given the topic of this post, it should come as no surprise that these shortcomings are almost always related to CDN issues:

Serving large assets from a centralized location (as opposed to a distributed CDN).
Assigning proper cache policies to large assets to avoid reloads.
Serving massive JS/CSS bundles arbitrarily defined by build processes.

Large assets lacking efficient cache policies hindering page rank

What about Cloudflare? Netlify?

"Plug-and-play" CDN services such as these fill the demand for a common (and necessary) niche. Such services have, dare I say, "democratized" CDNs and the performance benefits they yield via an effective 1-2 punch:

Affordable pricing (ranging from free to a ceiling of perhaps $20/month).
Zero-effort implementation.

Indeed, any web app can live behind a Cloudflare CDN for zero dollars while requiring zero effort. This is good enough for most people, and I would implore those people to stop reading now and live a happier life by sticking with what's easy.

Unfortunately for me, these CDN services don't apply to sites heavy on user-generated content. Hackersandslackers.com is primarily generated by editors (posts containing images), with secondary content coming from users (comments, personal profiles, avatars). "Content" can be created by nearly anybody at any time. It's almost impossible to cache apps that are constantly changing by nature effectively.

Google Cloud CDN Architecture

Google Cloud follows its own unique service architecture that's admittedly a bit unintuitive. GCP's own documentation doesn't do much to help this problem, which is precisely what drove me to write this piece in the first place.

Setting up a proper Google Cloud CDN requires four separate GCP services:

Cloud DNS: Safely look past the obnoxious inclusion of the word "Cloud" here (Google marketing be damned). This "service" involves DNS configuration for a given domain, such as adding A/AAAA records. This allows us to serve assets from a custom domain (such as cdn.hackersandslackers.com , in my case).
Load Balancer: It feels a bit excessive, but GCP forces us to handle the routing of a frontend (our DNS) to a backend (storage bucket) via a load balancer.
Cloud Storage: Google's equivalent to an S3 bucket. This is where we'll be storing assets to serve our users.
Cloud CDN: When we configure our load balancer to point to a Cloud Storage bucket, we receive the option to enable "Cloud CDN." This is what tells Google to serve assets from edges in its global network and allows us to set caching policies.

The final product looks something like this:

Architecture of a CDN on Google Cloud

We'll tackle this one step at a time. Before we can configure our Load Balancer, we'll need our Cloud Storage backend and Cloud DNS frontends in place. We'll start with the former.

Create a GCS Bucket

While logged into the Google Cloud console, go to your bucket browser. Here we'll create a new storage bucket via the self-explanatory "Create Bucket" button:

Creating a new Google Cloud storage bucket

Give your bucket a name and you're good to go:

Configure your bucket & keep it simple

Configure Cloud DNS

I'll assume you have a domain in your possession from which to serve assets (this can be a subdomain, such as cdn.[YOUR_DOMAIN].com).

Navigate to the Cloud DNS console and create a new zone, as such:

Add a DNS zone to serve assets from a vanity domain

This will prompt you to "Create a DNS zone." This is straightforward:

Set Zone type is set to Public.
Enter a Zone name (this can be anything you please; it is for internal use only).
Set DNS name to the domain in your possession from which all assets will be hosted.

Create a public DNS zone for your CDN's domain

Your newly created DNS zone should be visible from the Cloud DNS console:

Your DNS Zone details

Google should provide you with 4 nameservers. Ensure your domain is pointed to these nameservers via whichever provider you purchased your domain from:

Point your domain NS records to the provided nameservers

Create & Configure a Load Balancer

Now that we've set up our " frontend" and " backend," we can turn our attention to the meat and potatoes of our architecture: the Load Balancer. Navigate to GCP's Load Balancing console:

Navigate to "Load Balancing" under "Network Services"

Here we'll Create a new Load Balancer :

Create a new load balancer

We'll be serving assets over the internet (HTTP/HTTPS), so go ahead and start configuration for HTTP(S) Load Balancing:

Create an HTTP(S) Load Balancer

The next step is self-explanatory; we want an Internet-facing , Classic HTTP(S) Load Balancer:

Specify internet-facing load balancer

The foundation of our load balancer is in place! Let's make our load balancer to do what load balancers do: distribute incoming traffic.

"Incoming Traffic," in this sense, is any request made to the domain associated with our Cloud DNS frontend. From a user's perspective, it's as though we're hosting images and such on a domain separate from our application. Behind the scenes, the domain isn't a host so much as a traffic cop.

Depending on where the user resides, they'll fetch assets from whichever "host" offers the best connection for them, where said host is a replica of Cloud Storage backend. Thankfully, only a few steps remain for us to achieve this sorcery.

I. Backend Configuration

We have a load balancer, but it doesn't do anything just yet. We need to tell the load balancer what the end "destination" is for incoming requests. Our options are:

Backend Service : Think servers, cloud-hosted applications, microservices, etc. Basically, things we don't want.
Backend Bucket : Indeed, this is how Google's product team chose to represent CDNs on their platform. As far as GCP is concerned, a CDN is just anther service to be served behind a load balancer (and yet it isn't, hence a separate option, but whatever). This is the option we want.

Specify Backend Bucket as your load balancer's backend

Now we're actually getting somewhere. This is perhaps the most important step of all: tuning the configuration of our load balancers "backend bucket." On this step, we want to:

Name our backend bucket. Yes, your bucket already has a name - this name is intended to represent the relationship said bucket has to your load balancer. Look, the process is really, really dumb. But the end result is great.
Select which GCS bucket to point to. Select the bucket you created earlier via the "browse" button.
ENABLE CLOUD CDN! This unsuspecting checkbox is responsible for the magic that hosts our bucket over a distributed network. It's essentially the entire reason we're here... CLICK IT.
Set cache policies. I could drone on about which Time To Live (TTL) settings might give you the best shot at convincing Google to rank your page higher. Instead, I implore you to mimic something similar to what I have below, and encourage you to experiment and make your own tweaks once you get comfortable:

Configure the cache settings for your CDN

II. Host & Path Rules

Leave Simple host and path rule selected. All we need to do is select the backend we created in the previous step from the Backend 1 dropdown:

Point your load balancer to your backend bucket

III. Frontend Configuration

Almost there, folks. We've set up a backend bucket to host assets, configured a distributed CDN from said backend bucket, and configured a load balancer to point to said CDN. All that remains is accepting incoming traffic for our load balancer to distribute.

The domain associated with our Cloud DNS will serve as our source of traffic. Go ahead and Add Frontend IP And Port :

Map an IP address to your backend bucket

This will expand a configuration menu for accepting traffic on a given hostname. We'll be creating configurations to accept traffic on via HTTP (port 80) and HTTPS (port 443).

Before we do anything, we need to reserve an IP address. Click the IP Address dropdown:

Create your first HTTP/HTTPS frontend configuration

This simple menu will have a single option to Create IP Address :

Reserve and assign a static IP address

Go ahead and wrap up this frontend config with the default values in place: Protocol set to HTTP, IP Version as IPv4, and Port left as 80. Click Done to wrap up the first of our two frontend configs:

Wrap up your HTTP frontend configuration

For our second config (HTTPS), we'll repeat the process with a few notable differences; Protocol should now be HTTPS, and Port set to 443, and most notably, specify an SSL Certificate. Google Cloud makes acquiring SSL certs easy via a dropdown.

Create A New Certificate here, and we're DONE:

OH GOD YES ITS FINALLY OVER. I know I told you everything was going to be okay, but deep down I wasn't sure we'd finally get here. Hell, I wouldn't read a tutorial this long... but apparently I'd write one.

Oh yeah, we haven't hit the Confirm and Finalize button yet.

IV. Confirm & Finalize

PRESS THE UPDATE BUTTON FOR GREAT JUSTICE:

Bask in the glory of what you've created

And NOW we've done it! Christ... it's a good thing Google Cloud can't design UIs or write documentation to save their lives. Otherwise, I'd be out of business.

Take a breath of fresh air. You've made it, and your CDN is in place. After you take a moment to regroup, you may be wondering how to make sure your CDN is actually doing what we think its doing. I'll point you in the right direction.

Monitoring Your CDN in Action

There are two ways to visualize our CDN in action. First is by monitoring our load balancer.

Load Balancer Traffic

"Monitoring" our load balancer gives us a cool visual of where incoming traffic is coming from, and how the traffic is being routed:

Navigate to the "Monitoring" tab of your load balancer

Using my own load balancer as an example, the below graphic shows incoming traffic requests broken out by continent:

Visualize how traffic being routed through your load balancer

There are some key takeaways from what we're seeing here. First, nearly all traffic is being served over HTTPS, even including requests which were attempted over HTTP. These requests are successfully being forced over a secure connection.

At the bottom of my chart, I see most assets are being served from my CDN's cache (seen as SERVED_FROM_CACHE). This is what we like to see, but perhaps there's room for improvement in my caching policies to ensure this is the case more often.

CDN Bandwidth & Hit Rate

We can also monitor our CDN itself. While staying in the "Networking services" tab of Google Cloud, our Cloud CDN can be found via the left-hand menu. We only have a single CDN, so let's check it out:

Navigate to the "Cloud CDN" tab and select your backend bucket

Smash that Monitoring button to check out how your CDN is performing over time with metrics such as bandwidth, hit rate, number of requests, and more:

Cloud CDN monitoring

Parting Thoughts

Well folks, it's been fun (by which I of course mean the opposite). So, was it worth the effort after-all? In my personal case, web.dev bumped my site's performance ranking to 80+, which is a substantial increase. Considering the load times this saves my readers (as well as the SEO bump for not sucking), the process and costs of serving assets via CDN are certainly worth it.

Was it worth writing this tutorial about a niche service on a relatively small cloud provider? Almost certainly not. Unless you enjoyed it, of course.

See you next time.

Async HTTP Requests with Aiohttp & Aiofiles

Todd Birchard — Wed, 02 Feb 2022 09:10:00 +0000

When building applications within the confines of a single-threaded, synchronous language, the limitations become very obvious very quickly. The first thing that comes to mind is writes: the very definition of an I/O-bound task. When writing data to files (or databases), each "write" action intentionally occupies a thread until the write is complete. This makes a lot of sense for ensuring data integrity in most systems. For example, if two operations simultaneously attempt to update a database record, which one is correct? Alternatively, if a script requires an HTTP request to succeed before continuing, how could we move on until we know the request succeeded?

HTTP requests are among the most common thread-blocking operations. When we write scripts that expect data from an external third party, we introduce a myriad of uncertainties that can only be answered by the request itself, such as response time latency, the nature of data we expect to receive, or if the request will succeed. Even when working with APIs we're confident in, no operation is sure to succeed until it's complete. Hence, we're "blocked."

As applications grow in complexity to support more simultaneous user interactions, software is moving away from the paradigm of being executed linearly. So while we might not be sure that a specific request succeeds or a database write is completed, this can be acceptable as long as we have ways to handle and mitigate these issues gracefully.

A Problem Worthy of Asynchronous Execution

How long do you suppose it would take a Python script to execute a few hundred HTTP requests, parse each response, and write the output to a single file? If you were to use requests in a simple for loop, you'd need to wait a fair amount of time for Python to execute each request, open a file, write to it, close it, and move on to the next.

Let's put asyncio's ability to improve script efficiency to an actual test. We'll execute two I/O-blocking actions per task for a few hundred URLs: executing and parsing an HTTP request and writing the desired result to a single file. The input for our experiment will be a ton of URLs, with the expected output to be metadata parsed from those URLs. Let's see how long it takes to do this for hundreds of URLs.

This site has roughly two hundred published posts of its own, which makes it a great guinea pig for this little experiment. I've created a CSV that contains the URLs to these posts, which will be our input. Here's a sneak peek below:

Sample Input

url
https://hackersandslackers.com/intro-to-asyncio-concurrency/
https://hackersandslackers.com/multiple-python-versions-ubuntu-20-04/
https://hackersandslackers.com/google-bigquery-python/
https://hackersandslackers.com/plotly-chart-studio/
https://hackersandslackers.com/deploy-serverless-golang-functions-with-netlify/
https://hackersandslackers.com/scrape-metadata-json-ld/
https://hackersandslackers.com/terraform-with-google-cloud/
https://hackersandslackers.com/deploy-golang-app-nginx/
https://hackersandslackers.com/4-ways-to-improve-your-plotly-graphs/
https://hackersandslackers.com/create-your-first-golang-app/
...etc

Input CSV

Sample Output

For each URL found in our input CSV, our script will fetch the URL, parse the page, and write some choice data to a single CSV. The result will resemble the below example:

title	description	primary_tag	url	published_at
Intro to Asynchronous Python with Asyncio	Execute multiple tasks concurrently in Python with Asyncio: Python`s built-in async library.	Python	https://hackersandslackers.com/intro-to-asyncio-concurrency/	2022-01-04
Deploy Serverless Golang Functions with Netlify	Write and deploy Golang Lambda Functions to your GatsbyJS site on Netlify.	JAMStack	https://hackersandslackers.com/deploy-serverless-golang-functions-with-netlify/	2020-08-02
SSH & SCP in Python with Paramiko	Automate remote server tasks by using the Paramiko & SCP Python libraries. Use Python to SSH into hosts; execute tasks; transfer files; etc.	Python	https://hackersandslackers.com/automate-ssh-scp-python-paramiko/	2020-01-03
Create Cloud-hosted Charts with Plotly Chart Studio	Use Pandas and Plotly to create cloud-hosted data visualizations on-demand in Python.	Plotly	https://hackersandslackers.com/plotly-chart-studio/	2020-09-03
Create Your First Golang App	Set up a local Golang environment and learn the basics to create and publish your first `Hello world` app.	Golang	https://hackersandslackers.com/create-your-first-golang-app/	2020-05-25
Creating Interactive Views in Django	Create interactive user experiences by writing Django views to handle dynamic content; submitting forms; and interacting with data.	Django	https://hackersandslackers.com/creating-django-views/	2020-04-23
Define Relationships Between SQLAlchemy Data Models	SQLAlchemy`s ORM easily defines data models with relationships such as one-to-one; one-to-many; and many-to-many relationships.	SQLAlchemy	https://hackersandslackers.com/sqlalchemy-data-models/	2019-07-11
...etc

Example of what our script will output

Tools For The Job

We're going to need three core Python libraries to pull this off:

Asyncio: Python's bread-and-butter library for running asynchronous IO-bound tasks. The library has somewhat built itself into the Python core language, introducing async/await keywords that denote when a function is run asynchronously and when to wait on such a function (respectively).
Aiohttp: When used on the client-side, similar to Python's requests library for making asynchronous requests. Alternatively, aiohttp can be used inversely: as an application webserver to handle incoming requests & serving responses, but that's a tale for another time.
Aiofiles: Makes writing to disk (such as creating and writing bytes to files) a non-blocking task, such that multiple writes can happen on the same thread without blocking one another - even when multiple tasks are bound to the same file.

$ pip install asyncio aiohttp aiofiles

Install the necessary libraries

BONUS: Dependencies to Optimize Speed

aiohttp can execute requests even faster by simply installing a few supplemental libraries. These libraries are cchardet (character encoding detection), aiodns (asynchronous DNS resolution), and brotlipy (lossless compression). I'd highly recommend installing these using the conveniently provided shortcut below (take it from me, I'm a stranger on the internet):

$ pip install aiohttp[speedups]

Install supplemental dependencies to speed up requests

Preparing an Asynchronous Script/Application

We're going to structure this script like any other Python script. Our main module, aiohttp_aiofiles_tutorial will handle all of our logic. config.py and main.py both live outside the main module, and offer our script some basic configuration and an entry point respectively:

/aiohttp-aiofiles-tutorial
├── /aiohttp_aiofiles_tutorial
│   ├── __init__.py
│   ├── fetcher.py
│   ├── loops.py
│   ├── tasks.py
│   ├── parser.py
│   └── /data # Source data
│   ├── __init__.py
│      ├── parser.py
│      ├── tests
│      └── urls.csv
├── /export # Destination for exported data
├── config.py
├── logger.py
├── main.py
├── pyproject.toml
├── Makefile
├── README.md
└── requirements.txt

Project structure of our async fetcher/writer

/export is simply an empty directory where we'll write our output file to.

The /data submodule contains the input CSV mentioned above, and some basic logic to parse it. Not much to phone home about, but if you're curious the source is available on the Github repo.

Kicking Things Off

With sleeves rolled high, we start with the obligatory script "entry point," main.py. This initiates the core function in /aiohttp_aiofiles_tutorial called init_script():

"""Script entry point."""
import asyncio

from aiohttp_aiofiles_tutorial import init_script

if __name__ == " __main__":
    asyncio.run(init_script())

main.py

This seems like we're running a single function/coroutine init_script() via asyncio.run(), which seems counter-intuitive at first glance. Isn't the point of asyncio to run multiple coroutines concurrently, you ask?

Indeed it is! init_script() is a coroutine that calls other coroutines. Some of these coroutines create tasks out of other coroutines, others execute them, etc. asyncio.run() creates an event loop that doesn't stop running until the target coroutine is done, including all the coroutines that the parent coroutines calls. So, if we keep things clean, asyncio.run() is a one-time call to initialize a script.

Initializing Our Script

Here's where the fun begins. We've established that the purpose of our script is to output a single CSV file, and that's where we'll start: by creating and opening an output file within the context of which our entire script will operate:

"""Make hundreds of requests concurrently and save responses to disk."""
import aiofiles
from config import EXPORT_FILEPATH

async def init_script():
    """Prepare output file & kickoff task creation/execution."""
    async with aiofiles.open(EXPORT_FILEPATH, mode="w+") as outfile:
        await outfile.write(
            "title,description,primary_tag,url,published_at\n"
        )
        # (The rest of our script logic will be executed here).
        # ...

aiohttp_aiofiles_tutorial/__init__.py

Our script begins by opening a file context with aiofiles. As long as our script operates inside the context of an open async file via async with aiofiles.open() as outfile:, we can write to this file constantly without worrying about opening and closing the file.

Compare this to the synchronous (default) implementation of handling file I/O in Python, with open() as outfile:. By using aiofiles, we can write data to the same file from multiple sources at virtually the same time.

EXPORT_FILEPATH happens to target a CSV ( /export/hackers_pages_metadata.csv ). Every CSV needs a row of headers; hence our one-off usage of await outfile.write() to write headers immediately after opening our CSV:

...
await outfile.write(
    "title,description,primary_tag,url,published_at\n"
)

Writing a single row to our CSV consisting of column headers

Moving Along

Below is the fully fleshed-out version of __init__.py that will ultimately put our script into action. The most notable addition is the introduction of the execute_fetcher_tasks() coroutine; we'll dissect this one piece at a time:

"""Make hundreds of requests concurrently and save responses to disk."""
import asyncio
import time

import aiofiles
from aiofiles.threadpool.text import AsyncTextIOWrapper as AsyncIOFile
from aiohttp import ClientSession
from config import EXPORT_FILEPATH, HTTP_HEADERS

from .data import urls_to_fetch # URLs parsed from a CSV
from .tasks import create_tasks # Creates one task per URL


async def init_script():
    """Prepare output file; begin task creation/execution."""
    async with aiofiles.open(EXPORT_FILEPATH, mode="w+") as outfile:
        await outfile.write(
            "title,description,primary_tag,url,published_at\n"
        )
        await execute_fetcher_tasks(outfile)
        await outfile.close()


async def execute_fetcher_tasks(outfile: AsyncIOFile):
    """
    Open async HTTP session & execute created tasks.

    :param AsyncIOFile outfile: Path of file to write to.
    """
    async with ClientSession(headers=HTTP_HEADERS) as session:
        task_list = await create_tasks(
            session,
            urls_to_fetch,
            outfile
        )
        await asyncio.gather(*task_list)

aiohttp_aiofiles_tutorial/__init__.py

execute_fetcher_tasks() is broken out mainly to organize our code. This coroutine accepts outfile as a parameter, which will serve as the destination for data we end up parsing. Taking this line-by-line:

async with ClientSession(headers=HTTP_HEADERS) as session: Unlike the Python requests library, aiohttp enables us to open a client-side session that creates a connection pool that allows for up to 100 active connections at a single time. Because we're going to make under 200 requests, the amount of time it will take to fetch all these URLs will be comparable to the time it takes Python to fetch two under normal circumstances.
create_tasks(): This function we're about to define and accepts three parameters. The first is the async ClientSession we just opened a line earlier. Next, we have the urls_to_fetch variable (imported earlier in our script). This is a simple Python list of strings, where each string is a URL parsed from our earlier "input" CSV. That logic is handled elsewhere via a simple function (and not important for the purpose of this tutorial). Lastly, our outfile is passed along, as we'll be writing to this file later. With these parameters, create_tasks() will create a task for each of our 174 URLs. Each of which will download the contents of the given URL to the target directory. This function returns the tasks but will not execute them until we give the word, which happens via...
asyncio.gather(*task_list): Asyncio's gather() method performs a collection of tasks inside the currently running event loop. Once this kicks off, the speed benefits of asynchronous I/O will become immediately apparent.

Creating Asyncio Tasks

If you recall, a Python Task wraps a function (coroutine) which we'll execute in the future. In addition, each task can be temporarily put on hold for other tasks. A predefined coroutine must be passed along with the proper parameters before execution to create a task.

I separated create_tasks() to return a list of Python Tasks, where each "task" will execute fetching one of our URLs:

"""Prepare tasks to be executed."""
import asyncio
from asyncio import Task
from typing import List

from aiofiles.threadpool.text import AsyncTextIOWrapper as AsyncIOFile
from aiohttp import ClientSession

from .fetcher import fetch_url_and_save_data


async def create_tasks(
    session: ClientSession, urls: List[str], outfile: AsyncIOFile
) -> List[Task]:
    """
    Create asyncio tasks to parse HTTP request responses.

    :param ClientSession session: Async HTTP requests session.
    :param List[str] urls: Resource URLs to fetch.
    :param AsyncIOFile outfile: Path of file to write to.

    :returns: List[Task]
    """
    task_list = []
    for i, url in enumerate(urls):
        task = asyncio.create_task(
            fetch_url_and_save_data(
                session,
                url,
                outfile,
                len(urls),
                i
            )
        )
        task_list.append(task)
    return task_list

aiohttp_aiofiles_tutorial/tasks.py

A few notable things about Asyncio Tasks:

We're defining "work is to be done" upfront. The creation of a Task doesn't execute code. Our script will essentially run the same function 174 times concurrently, with different parameters. It makes sense that we'd want to define these tasks upfront.
Defining tasks is quick and straightforward. In an instant, each URL from our CSV will have a corresponding Task created and added to task_list.
With our tasks prepared, there's only one thing left to do to kick them all off and get the party started. That's where the asyncio.gather(*task_list) line from __ init __ .py comes into play.

Asyncio's Task object is a class in itself with its attributes and methods, essentially providing a wrapper with ways to check task status, cancel tasks, and so forth.

Executing our Tasks

Back in create_tasks(), we created tasks that each individually execute a method called fetch_url_and_save_data() per task. This function does three things:

Make an async request to the given task's URL via aiohttp's session context (handled by async with session.get(url) as resp:)
Read the body of the response as a string.
Write the contents of the response body to a file by passing html to our last function, parse_html_page_metadata():

"""Fetch URLs, extract their contents, and write parsed data to file."""
from aiofiles.threadpool.text import AsyncTextIOWrapper as AsyncIOFile
from aiohttp import ClientError, ClientSession, InvalidURL
from logger import LOGGER

from .parser import parse_html_page_metadata


async def fetch_url_and_save_data(
    session: ClientSession,
    url: str,
    outfile: AsyncIOFile,
    total_count: int,
    i: int,
):
    """
    Fetch raw HTML from a URL prior to parsing.

    :param ClientSession session: Async HTTP requests session.
    :param str url: Target URL to be fetched.
    :param AsyncIOFile outfile: Path of file to write to.
    :param int total_count: Total number of URLs to fetch.
    :param int i: Current iteration of URL out of total URLs.
    """
    try:
        async with session.get(url) as resp:
            if resp.status != 200:
                pass
            html = await resp.text()
            page_metadata = await parse_html_page_metadata(
                html,
                url
            )
            await outfile.write(f"{page_metadata}\n")
            LOGGER.info(
                f"Fetched URL {i} of {total_count}: {page_metadata}"
            )
    except InvalidURL as e:
        LOGGER.error(
            f"Unable to fetch invalid URL `{url}`: {e}"
        )
    except ClientError as e:
        LOGGER.error(
            f"ClientError while fetching URL `{url}`: {e}"
        )
    except Exception as e:
        LOGGER.error(
            f"Unexpected error while fetching URL `{url}`: {e}"
        )

aiohttp_aiofiles_tutorial/fetcher.py

When fetching a URL via an aiohttp ClientSession, calling the .text() method on the response (await resp.text()) will return the response of a request as a string. This is not to be confused with .body(), which returns a bytes object (useful for pulling media files or anything besides a string).

If you're keeping track, we're now three "contexts" deep:

We started our script by opening an aiofiles.open() context, which will remain open until our script is complete. This allows us to write to our outfile from any task for the duration of our script.
After writing headers to our CSV file, we opened a persistent client request session with async with ClientSession() as session, which allows us to make requests en masse as long as the session is open.
In the snippet above, we've entered a third and final context: the response context for a single URL (via async with session.get(url) as resp). Unlike the other two contexts, we'll be entering and leaving this context 174 times (once per URL).

Inside each URL response context is where we finally start producing some output. This leaves us with our final bit of logic (await parse_html_page_metadata(html, url)) which parses each URL response and returns some scraped metadata from the page before writing said metadata to our outfile on the next line, await outfile.write(f"{page_metadata}\n").

Write Parsed Metadata to CSV

How are we planning to rip metadata out of HTML pages, you ask? With BeautifulSoup, of course! With the HTML of an HTTP response in hand, we use bs4 to parse each URL response and return values for each of the columns in our outfile: title , description , primary_tag , published at , and url.

These five values are returned as a comma-separated string, then written to our outfile CSV as a single row.

"""Parse metadata from raw HTML."""
from bs4 import BeautifulSoup
from bs4.builder import ParserRejectedMarkup
from logger import LOGGER


async def parse_html_page_metadata(html: str, url: str) -> str:
    """
    Extract page metadata from raw HTML into a CSV row.

    :param str html: Raw HTML source of a given fetched URL.
    :param str url: URL associated with the extracted HTML.

    :returns: str
    """
    try:
        soup = BeautifulSoup(html, "html.parser")
        title = soup.title.string.replace(",", ";")
        description = (
            soup.head.select_one("meta[name=description]")
            .get("content")
            .replace(",", ";")
            .replace('"', "`")
            .replace("'", "`")
        )
        primary_tag = (
            soup.head
            .select_one("meta[property='article:tag']")
            .get("content")
        )
        published_at = (
            soup.head
            .select_one(
                 "meta[property='article:published_time']"
            )
            .get("content")
            .split("T")[0]
        )
        if primary_tag is None:
            primary_tag = ""
        return f"{title}, {description}, {primary_tag}, {url}, {published_at}"
    except ParserRejectedMarkup as e:
        LOGGER.error(
            f"Failed to parse invalid html for {url}: {e}"
        )
    except ValueError as e:
        LOGGER.error(
            f"ValueError occurred when parsing html for {url}: {e}"
        )
    except Exception as e:
        LOGGER.error(
            f"Parsing failed when parsing html for {url}: {e}"
        )

aiohttp_aiofiles_tutorial/parser.py

Run the Jewels, Run the Script

Let's take this bad boy for a spin. I threw a timer into __init__.py to log the number of seconds that elapse for the duration of the script:

"""Make hundreds of requests concurrently and save responses to disk."""
import time
from time import perf_counter as timer

...

async def init_script():
    """Prepare output file & task creation/execution."""
    start_time = timer() # Add timer to function
    async with aiofiles.open(EXPORT_FILEPATH, mode="w+") as outfile:
        await outfile.write(
            "title,description,primary_tag,url,published_at\n"
        )
        await execute_fetcher_tasks(outfile)
        await outfile.close()
    LOGGER.success(
         f"Executed { __name__ } in {time.perf_counter() - start_time:0.2f} seconds."
     ) # Log time of execution

...

aiohttp_aiofiles_tutorial/__init__.py

Mash that mfing make run command if you're following along in the repo (or just punch in python3 main.py). Strap yourself in:

...
16:12:34 PM | INFO: Fetched URL 165 of 173: Setting up a MySQL Database on Ubuntu, Setting up MySQL the old-fashioned way: on a linux server, DevOps, https://hackersandslackers.com/set-up-mysql-database/, 2018-04-17 
16:12:34 PM | INFO: Fetched URL 164 of 173: Dropping Rows of Data Using Pandas, Square one of cleaning your Pandas Dataframes: dropping empty or problematic data., Data Analysis, https://hackersandslackers.com/pandas-dataframe-drop/, 2018-04-18 
16:12:34 PM | INFO: Fetched URL 167 of 173: Installing Django CMS on Ubuntu, Get the play-by-play on how to install DjangoCMS: the largest of three major CMS products for Python`s Django framework., Software, https://hackersandslackers.com/installing-django-cms/, 2017-11-19 
16:12:34 PM | INFO: Fetched URL 166 of 173: Starting a Python Web App with Flask & Heroku, Pairing Flask with zero-effort container deployments is a deadly path to addiction., Architecture, https://hackersandslackers.com/flask-app-heroku/, 2018-02-13 
16:12:34 PM | INFO: Fetched URL 171 of 173: Another 'Intro to Data Analysis in Python Using Pandas' Post, An introduction to Python`s quintessential data analysis library., Data Analysis, https://hackersandslackers.com/intro-python-pandas/, 2017-11-16 
16:12:34 PM | INFO: Fetched URL 172 of 173: Managing Python Environments With Virtualenv, Embrace core best-practices in Python by managing your Python packages using virtualenv and virtualenvwrapper., Software, https://hackersandslackers.com/python-virtualenv-virtualenvwrapper/, 2017-11-15 
16:12:34 PM | INFO: Fetched URL 170 of 173: Visualize Folder Structures with Python’s Treelib, Using Python`s treelib library to output the contents of local directories as visual tree representations., Data Engineering, https://hackersandslackers.com/python-tree-hierachies-treelib/, 2017-11-17 
16:12:34 PM | INFO: Fetched URL 169 of 173: Merge Sets of Data in Python Using Pandas, Perform SQL-like merges of data using Python`s Pandas., Data Analysis, https://hackersandslackers.com/merge-dataframes-with-pandas/, 2017-11-17 
16:12:34 PM | INFO: Fetched URL 168 of 173: Starting an ExpressJS App, Installation guide for ExpressJS with popular customization options., JavaScript, https://hackersandslackers.com/create-an-expressjs-app/, 2017-11-18 
16:12:34 PM | SUCCESS: Executed aiohttp_aiofiles_tutorial in 2.96 seconds.

The tail end of our log after fetching 174 pages in ~3 seconds

The higher end of our script's execution time is 3 seconds. A typical Python request takes 1-2 seconds to complete, so our speed optimization is in the range of hundreds of times faster for a sample size of data like this.

Writing async scripts in Python surely takes more effort, but not hundreds or thousands of times more effort. Even if isn't speed you're after, handling volume of larger-scale applications renders Asyncio absolutely critical. For example, if your chatbot or webserver is in the middle of handling a user's request, what happens when a second user attempts to interact with your app in the meantime? Often times the answer is nothing: User 1 gets what they want, and User 2 is stuck taking to a blocked thread.

Anyway, seeing is believing. Here's the source code for this tutorial:

hackersandslackers / aiohttp-aiofiles-tutorial

🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

Asynchronous HTTP Requests Tutorial

Make asynchronous HTTP requests and write to disk using asyncio, aiohttp, & aiofiles.

Source code demonstrating asynchronous Python for the corresponding Hackersandslackers post: https://hackersandslackers.com/async-requests-aiohttp-aiofiles/

Getting Started

Get up and running by cloning this repository and running make deploy:

$ git clone https://github.com/hackersandslackers/aiohttp-aiofiles-tutorial.git
$ cd aiohttp-aiofiles-tutorial
$ make deploy

Hackers and Slackers tutorials are free of charge. If you found this tutorial helpful, a small donation would be greatly appreciated to keep us in business. All proceeds go towards coffee, and all coffee goes towards more content.

View on GitHub

Source code for this tutorial

Intro to Asynchronous Python with Asyncio

Todd Birchard — Mon, 10 Jan 2022 14:22:00 +0000

It's fair to say that most Hackers and Slackers readers share one thing in common: we like writing stuff in Python. This does not make us unique; it's reflective of a well-known and easily explainable phenomenon as Data Scientists/Engineers enter (and more recently, leave) a space previously reserved for software engineering: multi-purpose programming languages. Despite how unique these disciplines are from one another, we share a common trait. To quote the Beatles, All we Need is Python™️.

And yet, each journey has a defining moment where it's clear the language has surpassed its 30th birthday. It's been three decades since Guido unleashed The Serpent to tackle the problems of 1991. This was the final year of the Cold War era: a different time in history and even more so computing. Most of the design decisions behind Python were sensible for their time, but a few of these decisions have become today's "quirks." The most controversial quirk is easily the topic of concurrency.

Indeed, I'm referring to the Global Interpreter Lock (GIL). I'll save you the pain of disparaging the GIL as others have done a much better job of doing this than I can (if you're interested in the nitty-gritty and have the time, I highly recommend a piece entitled The GIL and its effects on Python multithreading). The long-and-short of the GIL is that it restricts Python from utilizing multiple CPU cores effectively, leaving your 8-core laptop running Python scripts on a single core while others lay idle.

Concurrency is a complicated problem extending far beyond Python; most programming languages share a similar fate. But we're not here to lament about our circumstances; we're here to talk about asynchronous I/O.

Doing Multiple Things at Once in Python

Concurrency is a broad concept in programming that boils down to "doing a bunch of stuff at once." Code that is said to run concurrently generally takes one of two possible forms:

Tasks taking turns to be executed to minimize downtime of peers' tasks.
Tasks which truly tun run parallel to one another, simultaneously.

Python ships with two modules that handle either approach, Respectively:

Threading (single process): Python's threading module is limited to utilizing a single processor at a given time. The threading module enables the governance of tasks occupying a given thread. Task X runs until blocked by an external factor (such as awaiting a response to an HTTP request). In the meantime, Task Y is prioritized to execute it until Task X is ready to continue (hence "blocking I/O").
Multiprocessing (multiple processors): The multiprocessing module enables code to run in parallel. A script is initialized and run n times simultaneously across n CPUs. Expanding a single road into an 8-lane highway has obvious performance benefits until it comes time to consolidate the results of each task. Multiple processes cannot simultaneously write data to the same destination (databases, files, etc.) without creating locks that block one another. For scripts intended to produce an output, attempting to wrestle this problem is almost certainly an insurmountable endeavor.

Where Does Asyncio Fit In?

Asyncio is a third and generally preferred alternative to the approaches above. Despite being limited to a single thread, Asyncio can execute a high volume of operations much faster than Python's native single-threaded execution. To illustrate how this is possible, consider how human beings tend to "multitask." When people claim to be "multitasking," they generally get things done by juggling between tasks instead of doing multiple things simultaneously. Single-threaded programs are asynchronous in the same way: output is optimized by overlapping work.

While humans are notoriously awful at multitasking, machines can see significant performance benefits from this practice, as they are generally better suited to start and stop work without the extraneous overhead. This concept is the "secret sauce" of frameworks such as FastAPI, an asynchronous Python framework that self-proclaims performance to be "on par with NodeJS and Go" (I promise to write a fair share on FastAPI another time).

It took me a while to break my skepticism of how a single thread handling multiple tasks kind-of-at-the-same-time could deliver a performance benefit worth writing about. It wasn't until I stumbled upon the concept of event loops that things started making sense.

Event Loops

We start with a backlog of I/O "tasks." They can be HTTP requests, saving things to disk, or in our case, a mix of both of those. Synchronous Python workflows (read: standard Python) would execute each task one at a time, from start to finish. It's like waiting in line at the DMV, where there's only one line, and the lady hates all of you.

An event loop handles things differently to get through tasks quicker. Given many tasks, an event "loop" works by grabbing new tasks and delegating them to threads. The loop continuously checks the tasks in progress for downtime (or completion). When delegated tasks are "waiting" on an external factor (such as an HTTP request), the event loop fills the dead time by kicking off another task in the thread. If the event loop finds that an allocated task has been completed, the Task is removed from its thread, the output of that Task is collected, and the loop picks another task from the queue to occupy the thread:

Asynchronous I/O Event Loop

Unlike a DMV line, an event loop works somewhat similarly to a restaurant (stick with me here). Despite having many tables and a single kitchen, a server handles their "backlog" by rotating between tables (tasks) and a kitchen (thread). It's much more efficient to take multiple food orders in succession to be prepared in a kitchen than to take single orders and wait for them to be created/served before moving to the next customer.

Coroutines: Functions to Run Asynchronously

Asynchronous Python scripts don't define functions - they define coroutines. Coroutines (defined with async def, as opposed to def) can halt execution before completion, typically waiting on the completion of another coroutine. The snippet below demonstrates the simplest example of a coroutine:

"""Define a Coroutine function to be executed asynchronously."""
import asyncio

from logger import LOGGER

async def simple_coroutine(number: int):
    """
    Wait for a time delay & display number associated with coroutine.

    :param int number: Number to identify the current coroutine.
    """
    await asyncio.sleep(1)
    LOGGER.info(f"Coroutine {number} has finished executing.")

coroutines.py

simple_coroutine() halts its execution for 1 second before logging a message. Coroutines can't be invoked like regular functions; attempting to run simple_coroutine(1) won't work unless it's run inside an Asyncio event loop. Luckily, creating an event loop is easy:

import asyncio

from coroutines import simple_coroutine # Import our coroutine

asyncio.run(simple_coroutine(1))

Running a Coroutine

asyncio.run() creates an event loop, and runs the coroutine passed into it. This method of creating an event loop is best when your script has an entry point from which all logic originates. Alternatively, asyncio.gather() accepts any number of coroutines, if you'd prefer to simply execute a handful of coroutines:

import asyncio

from coroutines import simple_coroutine # Import our coroutine


asyncio.gather(
    simple_coroutine(1)
    simple_coroutine(2)
    simple_coroutine(3)
)

Running 3 coroutines inside an event loop

Running this script will execute all three coroutines and log the following:

1
2
3

Output of `asyncio.gather()` with three coroutines

How long would you suppose the above operation takes to complete? 3 seconds, perhaps? Or have we managed to optimize our code with sorcery?

You might be surprised to learn that running the above consistently executes in almost exactly one second (or occasionally 1.01 on a bad day). If we time our function's execution time using Python's built-in time.perf_counter(), we can see this first-hand:

import asyncio
import time

from coroutines import simple_coroutine # Import our coroutine

def async_gather_example()
    start_time = time.perf_counter()
    asyncio.gather(
        simple_coroutine(1)
        simple_coroutine(2)
        simple_coroutine(3)
    )
    print(f"Executed { __name__ } in {time.perf_counter() - start_time:0.2f} seconds.")


async_example()

Track execution time of executing 3 coroutines which sleep for 1 second

Sure enough, the script takes almost exactly 1 second:

Executed async_example in 1.01 seconds.

Output of `async_example()`

Our coroutine simple_coroutine() takes 1 second to execute on its own. What's impressive about the above is we've called this coroutine three times and got a runtime of nearly 1 second, whereas a synchronous Python script indeed would've taken 3 seconds. What's more, the overhead for executing these tasks was only less than .01 seconds, meaning the coroutines we completed at almost the same time.

Working with Tasks

Using asyncio.gather() in the above example, we side-stepped an essential data structure in Asyncio: the Task.

Coroutines are functions that can run asynchronously. It would be nice to "manage" these functions when running hundreds or thousands of such functions in specific ways. Knowing when a coroutine fails (and what to do with it), or simply checking in on which coroutine a loop is currently handling, especially when our event loop might take minutes or hours to execute or have the potential to fail.

Managing Tasks

In more complex workflows, Tasks offer a number of useful methods to help us manage Tasks being executed:

.set_name([name]) (and .get_name()): Gives the task a name for the purpose of having a human-readable way to identify which task is which.
.cancel(msg=[message]): Cancels a task in an event loop, allowing the loop to continue with other tasks. Useful for tasks that become unresponsive or are unlikely to complete.
.canceled(): Returns True if the task was canceled, or False otherwise.
.done(): Returns True if the task was completed successfully, or False otherwise.
.result(): Returns the end result of the task. canceled tasks will include the exception message for why the task was canceled, whereas done tasks will simply return done. A task that has yet to be invoked will return an InvalidStateError exception.
A bunch of other methods, all found in Asyncio's Task documentation.

Creating Tasks

Wrapping Coroutines with Asyncio Task is simple. Running asyncio.gather() earlier handled this for us, but this is simply a shortcut that deprives us of utilizing the upsides of Tasks, as the tasks are instantiated as generic objects and executed immediately. If we create our tasks beforehand, we can associate metadata to them and execute them in an event loop when we're ready.

We're going to create a new coroutine called create_tasks(), which is a coroutine that will:

Creates n Task instances of simple_coroutine().
Assigns each Task a name upon creation.
Returns all Tasks as a Python list, which can later be executed via an event loop:

"""Create multiple tasks from a Coroutine."""
import asyncio
from asyncio import Task
from typing import List

from logger import LOGGER

from asyncio_intro_part1.coroutines import simple_coroutine

async def create_tasks(num_tasks: int) -> List[Task]:
    """
    Create n number of asyncio tasks to be executed.

    :param int num_tasks: Number of tasks to create.

    :returns: List[Task]
    """
    task_list = []
    LOGGER.info(f"Creating {num_tasks} tasks to be executed...")
    for i in range(num_tasks):
        task = asyncio.create_task(
            simple_coroutine(i),
            name=f"Task #{i}"
        )
        task_list.append(task)
        LOGGER.info(f"Created Task: {task}")
    return task_list

tasks.py

Tasks in Action

With our create_tasks() method defined, it's time for the fun part: seeing tasks being created, executed, and completed. At the root of our project, we'll define one last function, async_tasks_example(), to demonstrate this:

...
from .tasks import create_tasks


async def async_tasks_example():
    """Create and inspect tasks to wrap simple functions."""
    task_list = await create_tasks(5)
    done, pending = await asyncio.wait(task_list)
    if done:
        LOGGER.success(
            f"{len(done)} tasks completed: {[task.get_name() for task in done]}."
        )
    if pending:
        LOGGER.warning(
            f"{len(done)} tasks pending: {[task.get_name() for task in pending]}."
        )

__init__.py

We first assign 5 tasks created via create_tasks() to the task_list variable. As this occurs, we see the proper logging we added in tasks.py :

17:00:53 PM | INFO: Creating 5 tasks to be executed... 
17:00:53 PM | INFO: Created Task: <Task pending name='Task #0' coro=<simple_coroutine() running at /Users/toddbirchard/Projects/asyncio-tutorial-part1/asyncio_intro_part1/coroutines.py:7>> 
17:00:53 PM | INFO: Created Task: <Task pending name='Task #1' coro=<simple_coroutine() running at /Users/toddbirchard/Projects/asyncio-tutorial-part1/asyncio_intro_part1/coroutines.py:7>> 
17:00:53 PM | INFO: Created Task: <Task pending name='Task #2' coro=<simple_coroutine() running at /Users/toddbirchard/Projects/asyncio-tutorial-part1/asyncio_intro_part1/coroutines.py:7>> 
17:00:53 PM | INFO: Created Task: <Task pending name='Task #3' coro=<simple_coroutine() running at /Users/toddbirchard/Projects/asyncio-tutorial-part1/asyncio_intro_part1/coroutines.py:7>> 
17:00:53 PM | INFO: Created Task: <Task pending name='Task #4' coro=<simple_coroutine() running at /Users/toddbirchard/Projects/asyncio-tutorial-part1/asyncio_intro_part1/coroutines.py:7>>

The output of creating 5 tasks in tasks.py

We subsequently execute these five tasks via asyncio.wait(task_list). asyncio.wait() attempts to complete all the tasks in task_list and returns a tuple of "done" and "pending" tasks. Thanks to some added logging and the presence of task names, we can confirm all tasks completed successfully:

17:00:54 PM | INFO: Coroutine 0 has finished executing. 
17:00:54 PM | INFO: Coroutine 1 has finished executing. 
17:00:54 PM | INFO: Coroutine 2 has finished executing. 
17:00:54 PM | INFO: Coroutine 3 has finished executing. 
17:00:54 PM | INFO: Coroutine 4 has finished executing. 
17:00:54 PM | SUCCESS: 5 tasks completed: ['Task #1', 'Task #4', 'Task #3', 'Task #0', 'Task #2'].

Executing tasks with asyncio.wait()

Did You Get All That?

I pray that this tutorial has been somewhat helpful as a starting point for Asyncio. Part of the reason I haven't posted new content in a while is my inability to break these concepts down into digestible pieces suitable for human consumption.

If you're stuck scratching your head, there's some good news. While I originally intended Asynchronous Python to be a single post, it quickly became evident that Asyncio is a long journey. Hooray for multi-part series' of dry, technical writing!

I'm aware I blew through the source code on this one, which is why I've uploaded a fully working version of this tutorial to Github below. It's yours to dig around in; here's to hoping the source code clears up any blind spots I might've missed:

hackersandslackers / asyncio-tutorial-part1

🐍🔁 Intro to concurrency in Python with Asyncio.

Intro to Asynchronous Python with Asyncio

Source code demonstrating asynchronous Python for the Hackersandslackers post: Introduction to asynchronous Python with Asyncio

Getting Started

Get up and running by cloning this repository and running make deploy:

$ git clone https://github.com/hackersandslackers/asyncio-tutorial-part1.git
$ cd asyncio-tutorial-part1
$ make deploy

View on GitHub

Create Cloud-hosted Charts with Plotly Chart Studio

Todd Birchard — Thu, 03 Sep 2020 11:28:00 +0000

Given the success of Plotly Dash and Plotly Express, it's easy to forget that Plotly's rise to success began with a product that was neither of these household names. Dash has cornered the interactive dashboard market, while Plotly Express has become the defacto Python library generating inline charts, particularly for Jupyter notebooks. What we don't get from either of these tools is a way to create data visualizations for any other scenario, such as creating a single chart to serve publicly as an image or interactive plot.

Plotly Chart Studio shines as a tool to instantly create cloud-hosted data visualizations. The syntax will appear familiar to those who have used Plotly Express (or any other Python data vis library), but Plotly Chart Studio stands alone how these charts are created: on a publicly accessible cloud. With a Chart Studio account, each chart you plot is saved to a public chart studio profile, like mine:

https://chart-studio.plotly.com/~toddbirchard#/

Each chart in my profile is publicly accessible as your choice of an embedded plot, an image, raw HTML, etc. Here's an example chart as an iFrame (sorry dev.to fam, but I don't think iFrames are supported here):

https://plotly.com/~toddbirchard/403/

Every chart created with Plotly studio automatically generates interactive plots along with static image varieties like PNGs. This opens the door to possibilities for us to create charts on demand, like creating dumb Discord bots, for example:

Serve charts on demand

This tutorial will demonstrate plot creation in Chart Studio with a fairly common real-life scenario. We'll be fetching stock price data from IEX Cloud, transforming the data in a Pandas DataFrame, and outputting a Candlestick chart. This same workflow can be applied to create any of Plotly's many chart types.

Getting Set Up

The two Plotly-related libraries we need are plotly and chart-studio. We'll install these along with the usual suspects for Python data analysis:

$ pip install requests pandas plotly chart-studio python-dotenv

Install requirements

Accounts & Configuration

You can create a Plotly Chart Studio account here; get yourself set up and grab an API key. I'm also using an API key to access data from IEX, which can be substituted for whichever API credentials you happen to need:

"""Configuration via environment variables."""
from os import environ, path
from dotenv import load_dotenv

# Load values from .env
basedir = path.abspath(path.dirname( __file__ ))
load_dotenv(path.join(basedir, '.env'))

# API
IEX_API_TOKEN = environ.get('IEX_API_TOKEN')
IEX_API_BASE_URL = 'https://cloud.iexapis.com/stable/stock/'

# Plotly
PLOTLY_API_KEY = environ.get('PLOTLY_API_KEY')
PLOTLY_USERNAME = environ.get('PLOTLY_USERNAME')

config.py

Save your secrets in a .env file as always. If you're not sure what this means, please stop coding and Google what environment variables are. Seriously, stop DMing me that my code is "broken" because you didn't create a .env file, or take 5 minutes to grasp what that even means.

What were we talking about, again? Right, charts.

Initializing Chart Studio

chart_studio is a standalone Python library to serve the sole purpose of saving charts to Plotly Chart Studio. We need to authenticate with Plotly once upfront before we chart anything, which we can do by importing the set_credentials_file function and passing our Plotly username and API key :

from chart_studio.tools import set_credentials_file
from config import (
    PLOTLY_USERNAME,
    PLOTLY_API_KEY
)

# Plotly Chart Studio authentication
set_credentials_file(
    username=PLOTLY_USERNAME,
    api_key=PLOTLY_API_KEY
)

Authenticating with Plotly Chart Studio

We'll come back to this later. Let's get some data.

Fetching & Preparing Data

Beautiful charts need beautiful data. We're going to get some beautiful data using two libraries you should hopefully be sick of by now: requests and Pandas.

Fetch Data from an API

Let's GET this bread! Below is a function called fetch_stock_data(), which constructs a GET request to our API. In my example, we're fetching 1-month's worth of stock price data for a given stock symbol:

"""Fetch data from third-party API."""
import requests
from config import IEX_API_BASE_URL, IEX_API_TOKEN


def fetch_stock_data(stock_symbol: str):
    """Fetch stock data from API"""
    params = {'token': IEX_API_TOKEN, 'includeToday': 'true'}
    url = f'{IEX_API_BASE_URL}/{stock_symbol}/chart/1m'
    req = requests.get(url, params=params)
    data = req.content
    return data

Fetch Stock Price Data

This is a straightforward GET request made to a REST API endpoint. The above effectively constructs and hits this URL:

https://cloud.iexapis.com/stable/stock/net/chart/1m?token=sk_54632753687697685697456&includeToday=true

IEX endpoint for 1-month price data

If everything goes correctly, req.content should spit out some raw JSON. Here's a glance at what that looks like:

[
    ...
    {
        "date": "2020-08-20",
        "open": 39.28,
        "close": 40.31,
        "high": 40.44,
        "low": 39.12,
        "volume": 3953482,
        "uOpen": 39.28,
        "uClose": 40.31,
        "uHigh": 40.44,
        "uLow": 39.12,
        "uVolume": 3953482,
        "change": 1.02,
        "changePercent": 2.5961,
        "label": "Aug 20",
        "changeOverTime": 0.059674
    },
    {
        "date": "2020-08-21",
        "open": 40.45,
        "close": 38.92,
        "high": 40.49,
        "low": 38.9,
        "volume": 3271742,
        "uOpen": 40.45,
        "uClose": 38.92,
        "uHigh": 40.49,
        "uLow": 38.9,
        "uVolume": 3271742,
        "change": -1.39,
        "changePercent": -3.4483,
        "label": "Aug 21",
        "changeOverTime": 0.023134
    }
    ...
]

API response

This response might seem daunting at first glance, but it's exactly what we're looking for. We just need a little help from an old friend...

Parsing with Pandas

It's time for some mo effin' Pandas! We're going to take the data we fetched from our API request and pass it into a new function called parse_data(), which will load our data into a Pandas DataFrame. Shoutout to the beautiful new Pandas method called read_json():

"""Parse raw API data into Pandas DataFrame."""
import pandas as pd


def parse_data(data) -> pd.DataFrame:
    """Parse JSON as Pandas DataFrame."""
    stock_df = pd.read_json(data)
    stock_df = stock_df.loc[stock_df['date'].dt.dayofweek < 5]
    stock_df.set_index(keys=stock_df['date'], inplace=True)
    return stock_df

Create Pandas DataFrame from raw JSON

It's important to note that we pass in req.content from our request - not req.json(). req.json() renders JSON as a Python dictionary, which you could load into Pandas with from_dict(), but whatever. Stick with me here.

Create a Chart with Plotly Studio

Let's handle this one part one step at a time. There are three things we need to do here:

Create a Plotly chart with the data from our Pandas DataFrame.
Adjust aspects of our chart's layout (such as title, color, etc.).
Saving our chart to Plotly's cloud.

Each step above is nice and easy. Here's step 1 :

"""Plotly chart creation."""
import plotly.graph_objects as go
import chart_studio.plotly as py
from chart_studio.tools import set_credentials_file
from pandas import DataFrame
from config import PLOTLY_API_KEY, PLOTLY_USERNAME


# Plotly Chart Studio authentication
set_credentials_file(
    username=PLOTLY_USERNAME,
    api_key=PLOTLY_API_KEY
)

def create_chart(stock_df: DataFrame, symbol: str) -> py.plot:
    """Create Plotly chart from Pandas DataFrame."""
    fig = go.Figure(data=[
        go.Candlestick(
            x=stock_df.index,
            open=stock_df['open'],
            high=stock_df['high'],
            low=stock_df['low'],
            close=stock_df['close'],
        )],
    )

Create chart from Pandas DataFrame

If you've ever used Plotly or Plotly Dash, this syntax should look familiar! go.Figure() is a Python class used to create charts/plots/figures/whatever. We're passing data as a keyword argument, which is actually just one of 3 notable arguments we could pass:

data : Accepts one or multiple "trace" types. A "trace" refers to the data on our plot; in other words, we could easily chart different types of data on the same plot, such as a combination of line chart data ( go.Line ) and bar chart data ( go.Bar ) on the same plot. In our case, we're sticking with a single candlestick trace, where we pass in columns of our Pandas DataFrame for each value go.Candlestick() expects.
layout : A plot's layout gives us the power to change anything and everything about our chart's appearance. This could be background color, trace color, plot title, margins, and so much more. Just look at all the ways we can customize our plot - it's almost ridiculous!
frames : Apparently, we could animate our charts by passing multiple frames of changing data into the frames keyword argument. That's just fuckin bonkers imo and not something we should worry about until we're older.

Anyway, on to step 2 : customizing our layout:

...

def create_chart(stock_df: DataFrame, symbol: str) -> py.plot:
    """Create Plotly chart from Pandas DataFrame."""
    fig = go.Figure(data=[
        go.Candlestick(
            x=stock_df.index,
            open=stock_df['open'],
            high=stock_df['high'],
            low=stock_df['low'],
            close=stock_df['close'],
        )],
        layout=go.Layout(
            title=f'30-day performance of {symbol.upper()}',
            xaxis={
              'type': 'date',
              'rangeslider': {
                  'visible': False
              },
            },
        )
    )

Modify plot layout

We pass layout into our figure immediately after passing data. As we've already established, there's a shit ton of possibilities we could get deep into here. I'm keeping things modest by setting a chart title and making sure the x-axis knows it's a range of dates.

We can finally save our hard work with step 3 : saving our plot to the cloud:

...

def create_chart(stock_df: DataFrame, symbol: str) -> py.plot:
    """Create Plotly chart from Pandas DataFrame."""
    fig = go.Figure(data=[
        go.Candlestick(
            x=stock_df.index,
            open=stock_df['open'],
            high=stock_df['high'],
            low=stock_df['low'],
            close=stock_df['close'],
        )],
        layout=go.Layout(
            title=f'30-day performance of {symbol.upper()}',
            xaxis={
              'type': 'date',
              'rangeslider': {
                  'visible': False
              },
            },
        )
    )
    chart = py.plot(
        fig,
        filename=symbol,
        auto_open=False,
        fileopt='overwrite',
        sharing='public'
    )
    return chart

Create chart

The options we're passing to py.plot() are pretty straightforward:

Setting the filename.
Handling file clashes by overwriting preexisting charts with the same name.
Making the chart available to the public

So, what does chart actually output? Here's what a print() would reveal:

https://plotly.com/~toddbirchard/466/

Output of print(chart)

It's a link to our chart! But wait, what if I don't want to view my chart in on Plotly's site? What if I want an image? Check out what chart[:-1] + '.png' gives you:

https://plotly.com/~toddbirchard/466.png

Which is a URL to...

https://plotly.com/~toddbirchard/466.png

WE DID IT!

Organizing Our Work

Throughout this tutorial, you've kinda just been watching me dump random code blocks one-by-one. If this confuses the shit out of you, it'll probably help to know that this is how I chose to structure my project:

/plotly-chartstudio-tutorial
├── /plotly_chartstudio_tutorial
│   ├── __init__.py
│   ├── api.py
│   ├── chart.py
│   ├── data.py
│   └── log.py
├── config.py
├── main.py
├── requirements.txt
└── .env

Project structure

"That doesn't tell me shit," you might exclaim. Luckily, that was just a set up to tell you that the source code for this tutorial is available for up on Github. Enjoy:

hackersandslackers / plotly-chartstudio-tutorial

📈 📊 Create Cloud-hosted Charts with Plotly Chart Studio.

Plotly Chart Studio Tutorial

Use Pandas and Plotly to create cloud-hosted data visualizations on demand in Python. Source for the accompanying tutorial: https://hackersandslackers.com/plotly-chart-studio/

Installation

Installation via requirements.txt:

$ git clone https://github.com/hackersandslackers/plotly-chartstudio-tutorial.git
$ cd plotly-chartstudio-tutorial
$ python3 -m venv myenv
$ source myenv/bin/activate
$ pip3 install -r requirements.txt
$ python3 main.py

Installation via Pipenv:

$ git clone https://github.com/hackersandslackers/plotly-chartstudio-tutorial.git
$ cd plotly-chartstudio-tutorial
$ pipenv shell
$ pipenv update
$ python3 main.py

Installation via Poetry:

$ git clone https://github.com/hackersandslackers/plotly-chartstudio-tutorial.git
$ cd plotly-chartstudio-tutorial
$ poetry shell
$ poetry update
$ poetry run

View on GitHub

Scrape Structured Data with Python and Extruct

Todd Birchard — Sat, 08 Aug 2020 14:57:00 +0000

Unless you're entirely oblivious to scraping data in Python (and probably ended up here by accident), you're well-aware that scraping data in Python library begins and ends with BeautifulSoup. BeautifulSoup is Python's scraping powerhouse: we first demonstrated this in a previous post where we put together a script to fetch site metadata (title, description, preview images, etc.) from any target URL. We were able to build a scraper which fetched a target site's <meta> tags (and various fallbacks) to create a fairly reliable tool to summarize the contents of any URL; which is precisely the logic used to generate link "previews" such as these:

Example of a preview link with data fetched via BeatifulSoup.

Perusing the various sites and entities we refer to as "the internet" has traditionally felt like navigating an unstandardized wild-west. There's never a guarantee that the website you're targeting adheres to any web standards (despite their own best interests). These situations lead us to write scripts with complicated fallbacks in case the owner of myhorriblewebsite.angelfire.com somehow managed to forget to give their page a <title>, and so forth. Search engines and other big players recognized this. The standardization of JSON-LD was born as a reliable format for site publishers to include machine-readable (and also quite human-readable) metadata to appease search engines and fight for relevancy.

This post is going to build upon the goal of scraping site metadata we previously explored with BeautifulSoup via a different method: by parsing JSON-LD metadata with Python's extruct library.

What's so great about JSON-LD, you might ask? Aside from dodging the hellish experience of transversing the DOM by hand, JSON-LD is a specification with notable advantages to old school HTML <meta> tags. The multitude of benefits can mostly be boiled down into two categories: data granularity and linked data.

Data Granularity

JSON-LD allows web pages to express an impressive amount of granular information about what each page is. For instance, here's the JSON-LD for one of my posts:

{
  "@context": "https://schema.org/",
  "@type": "Article",
  "author": {
    "@type": "Person",
    "name": "Todd Birchard",
    "image": "https://hackersandslackers-cdn.storage.googleapis.com/2020/04/todd@2x.jpg",
    "sameAs": ["https://toddbirchard.com", "https://twitter.com/ToddRBirchard"]
  },
  "keywords": "Golang, DevOps, Software Development",
  "headline": "Deploy a Golang Web Application Behind Nginx",
  "url": "https://hackersandslackers.com/deploy-golang-app-nginx/",
  "datePublished": "2020-06-01T07:30:00.000-04:00",
  "dateModified": "2020-06-01T09:03:55.000-04:00",
  "image": {
    "@type": "ImageObject",
    "url": "https://hackersandslackers-cdn.storage.googleapis.com/2020/05/golang-nginx-3.jpg",
    "width": "1000",
    "height": "523"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Hackers and Slackers",
    "founder": "Todd Birchard",
    "logo": {
      "@type": "ImageObject",
      "url": "https://hackersandslackers-cdn.storage.googleapis.com/2020/03/logo-blue-full.png",
      "width": 60,
      "height": 60
    }
  },
  "description": "Deploy a self-hosted Go web application using Nginx as a reverse proxy. ",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://hackersandslackers.com"
  }
}

Example JSON-LD for a Hackers and Slackers post.

There's significantly more information stored in the above snippet than all other meta tags on the same page combined. There are surely more supported attributes in JSON-LD than traditional meta tags, yet the representation of data in a JSON hierarchy makes it immediately clear how page metadata is related. It's immediately clear that we're looking at an object representing an article, written by an author, as part of an "organization."

Google's explanation of the benefits of structuring metadata goes something like this:

Structured data is a standardized format for providing information about a page and classifying the page content; for example, on a recipe page, what are the ingredients, the cooking time and temperature, the calories, and so on.

Type

The term "web page" is useless ambiguous, as web pages are documents that can provide information in any number of forms. Web pages might be articles, recipes, product pages, events, and far more. The official schema of possible page types includes over one thousand possibilities for what "type" or "subtype" a page might be considered to be. Knowing the "type" of a page reduces ambiguity, and declaring a page "type" allows us to attach type-specific metadata to pages as well! For instance, let's compare the attributes of an Episode type to an Article type:

Episode
Property	Description
`actor`	An actor, e.g. in tv, radio, movie, video games etc., or in an event. Actors can be associated with individual items or with a series, episode, clip. Supersedes actors.
`director`	A director of e.g. tv, radio, movie, video gaming etc. content, or of an event. Directors can be associated with individual items or with a series, episode, clip. Supersedes directors.
`episodeNumber`	Position of the episode within an ordered group of episodes.
`musicBy`	The composer of the soundtrack.
`partOfSeason`	The season to which this episode belongs.
`partOfSeries`	The series to which this episode or season belongs. Supersedes partOfTVSeries.
`productionCompany`	The production company or studio responsible for the item e.g. series, video game, episode etc.
`trailer`	The trailer of a movie or tv/radio series, season, episode, etc.

Article
Property	Description
`articleBody`	The actual body of the article.
`articleSection`	Articles may belong to one or more 'sections' in a magazine or newspaper, such as Sports, Lifestyle, etc.
`backstory`	For an Article, typically a NewsArticle, the backstory property provides a textual summary giving a brief explanation of why and how an article was created. In a journalistic setting this could include information about reporting process, methods, interviews, data sources, etc.
`pageEnd`	The page on which the work ends; for example "138" or "xvi".
`pageStart`	The page on which the work starts; for example "135" or "xiii".
`pagination`	Any description of pages that is not separated into pageStart and pageEnd; for example, "1-6, 9, 55" or "10-12, 46-49".
`speakable`	Indicates sections of a Web page that are particularly 'speakable' in the sense of being highlighted as being especially appropriate for text-to-speech conversion. Other sections of a page may also be usefully spoken in particular circumstances; the 'speakable' property serves to indicate the parts most likely to be generally useful for speech.
`wordCount`	The number of words in the text of the Article.

There are obviously data attributes of television shows which don't apply to news articles (such as actors, director, etc.), and vice versa. The level of specificity achievable is nearly unfathomable when we discover that types have subtypes. For instance, our article might be an opinion piece article, which has extended the Article type with even more attributes.

Who

All content has a creator, yet content-creators can take many forms. Authors, publishers, and organizations could simultaneously be considered the responsible party for any given content, as these properties are not mutually exclusive. For instance, here's how my author data is parsed on posts like this one:

Author
Property	Description
@type	Person
name	Todd Birchard
image	https://hackersandslackers-cdn.storage.googleapis.com/2020/04/todd@2x.jpg
sameAs	https://toddbirchard.com/
sameAs	https://twitter.com/ToddRBirchard

What makes this data especially interesting is the values listed under the sameAs attribute, which associates the "Todd Birchard" in question to the very same Todd Birchard of the website https://toddbirchard.com/, and Twitter account https://twitter.com/ToddRBirchard. This undoubtedly assists search engines in making associations between entities on the web. Still, a keen imagination may easily recognize the opportunity to leverage these strong associations to dox or harass strangers on the internet quite easily.

Scrape Something Together

Along with Extruct, we'll be installing our good friend requests to fetch pages for us:

$ pip3 install requests extruct

Install libraries

You already know the drill — pick a single URL for now and loot them for all they've got by returning .text from our request's response:

import requests

def get_html(url):
    """Get raw HTML from a URL."""
    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'GET',
        'Access-Control-Allow-Headers': 'Content-Type',
        'Access-Control-Max-Age': '3600',
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }
    req = requests.get(url, headers=headers)
    return req.text

Retrieve a page's HTML.

Simple stuff. Here's where extruct comes in; I'm tossing together a function called get_metadata, which will do precisely what you'd assume. We can pass raw the HTML we grabbed with get_html and pass it to our new function to pillage:

"""Fetch structured JSON-LD data from a given URL."""
from pprint import pprint
import requests
import extruct
from w3lib.html import get_base_url

def scrape(url):
    """Parse structured data from a target page."""
    html = get_html(url)
    metadata = get_metadata(html, url)
    pprint(metadata, indent=2, width=150)
    return metadata

...

def get_metadata(html: bytes, url: str):
    """Fetch JSON-LD structured data."""
    metadata = extruct.extract(
        html,
        base_url=get_base_url(url),
        syntaxes=['json-ld'],
        uniform=True
    )['json-ld']
    if bool(metadata) and isinstance(metadata, list):
        metadata = metadata[0]
    return metadata

Getting structured data with extruct.

Using extruct is as easy as passing raw HTML as a string and a site's "base URL" with extruct.extract(html, base_url=url). A "base URL" refers to a site's entry point (or homepage, whatever) for the targeted page. The page you're on right now is https://hackersandslackers.com/scrape-metadata-json-ld/. Thus the base URL, in this case, would be https://hackersandslackers.com/. There's a core library called w3lib that has a function to handle this exact task, hence our usage of base_url=get_base_url(html, url).

This is what our extract function returns, using one of my posts as an example:

{ '@context': 'https://schema.org/',
  '@type': 'Article',
  'author': { '@type': 'Person',
              'image': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/04/todd@2x.jpg',
              'name': 'Todd Birchard',
              'sameAs': ['https://toddbirchard.com', 'https://twitter.com/ToddRBirchard']},
  'dateModified': '2020-06-11T16:57:57.000-04:00',
  'datePublished': '2018-11-11T08:35:09.000-05:00',
  'description': "Use Python's BeautifulSoup library to assist in the honest act of systematically stealing data without permission.",
  'headline': 'Scraping Data on the Web with BeautifulSoup',
  'image': { '@type': 'ImageObject',
             'height': '523',
             'url': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/06/beautifulsoup-1-1.jpg',
             'width': '1000'},
  'keywords': 'Python, Data Engineering',
  'mainEntityOfPage': {'@id': 'https://hackersandslackers.com', '@type': 'WebPage'},
  'publisher': { '@type': 'Organization',
                 'founder': 'Todd Birchard',
                 'logo': { '@type': 'ImageObject',
                           'height': 60,
                           'url': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/03/logo-blue-full.png',
                           'width': 60},
                 'name': 'Hackers and Slackers'},
  'url': 'https://hackersandslackers.com/scraping-urls-with-beautifulsoup/'}

JSON-LD data for a Hackers and Slackers post.

One of the keyword arguments we passed to extruct was syntaxes, which is an optional argument where we specify which flavor of structured data we're after (apparently there's more than one). Possible options to pass are 'microdata', 'json-ld', 'opengraph', 'microformat', and 'rdfa'. If nothing is passed, extruct will attempt to fetch all of the above and return the results in a dictionary. This is why we follow up our extruct call by accessing the ['json-ld'] key.

Dealing with Inconsistent Results

You're might be wondering why we index [0] after getting our results from extruct. This is a symptom of structured data: where traditional <meta> tags are predictably 1-dimensional, the "structure" of structured data is flexible and determined by developers. This level of flexibility gives developers the power to do things like define multiple meta images as a site's share image as a list of dicts as opposed to a single dict. This means makes the output of any given site's data unpredictable, which poses problems for Python scripts which are unaware of whether they should searching a list index or accessing a dictionary value.

The way I handle this is by explicitly checking the Python type of data being returned before extracting it:

...

def render_json_ltd(url: str, html) -> Optional[dict]:
    """Fetch JSON-LD structured data."""
    metadata = extruct.extract(
        html,
        base_url=get_domain(url),
        syntaxes=['json-ld'],
        uniform=True
    )['json-ld']
    if bool(metadata) and isinstance(metadata, list):
        metadata = metadata[0]
    return metadata

Check the "type" of structured data

This uncertainly of returned data types occurs everywhere. In the example where a page may have multiple meta images, I might write a function like get_image() below, where I explicitly check the type of data being returned for a given attribute while transversing the data tree:

...

def get_image(parsed_metadata, _data: dict) -> Optional[str]:
    """Scrape parsed_metadata `share image`."""
    image = None
    if bool(_data):
        if isinstance(_data, list):
            image = _data[0]
        if image is not None and isinstance(_data, dict):
            image = image.get('image')
        if isinstance(_data, str):
            return image

Extract data depending on type

Put it to Work

A script to return fetch and return structured data from a site would look something like this:

"""Fetch structured JSON-LD data from a given URL."""
from pprint import pprint
import requests
import extruct
from w3lib.html import get_base_url


def scrape(url):
    """Parse structured data from a target page."""
    html = get_html(url)
    metadata = get_metadata(html, url)
    pprint(metadata, indent=2, width=150)
    return metadata


def get_html(url):
    """Get raw HTML from a URL."""
    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'GET',
        'Access-Control-Allow-Headers': 'Content-Type',
        'Access-Control-Max-Age': '3600',
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }
    req = requests.get(url, headers=headers)
    return req.text


def get_metadata(html: bytes, url: str):
    """Fetch JSON-LD structured data."""
    metadata = extruct.extract(
        html,
        base_url=get_base_url(url),
        syntaxes=['json-ld'],
        uniform=True
    )['json-ld']
    if bool(metadata) and isinstance(metadata, list):
        metadata = metadata[0]
    return metadata

Scrape a single page for structured data.

Testing our Scraper
Since we're grownups, it's best if we write a simple test or two for a script that could potentially be run on a massive scale. The bare minimum we could do is point our scraper to a site containing structured data and compare the output to the data we'd expect to see. Below is a small test written with Pytest to see that our scrape() function outputs data which matches a hardcoded copy of what I expect to get back:

"""Validate JSON-LD Scrape outcome."""
import pytest
from extruct_tutorial import scrape


@pytest.fixture
def url():
    """Target URL to scrape metadata."""
    return 'https://hackersandslackers.com/creating-django-views/'


@pytest.fixture
def expected_json():
    """Expected metadata to be returned."""
    return {'@context': 'https://schema.org/', '@type': 'Article',
            'author': {'@type': 'Person', 'name': 'Todd Birchard',
                       'image': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/04/todd@2x.jpg',
                       'sameAs': ['https://toddbirchard.com', 'https://twitter.com/ToddRBirchard']},
            'keywords': 'Django, Python, Software Development', 'headline': 'Creating Interactive Views in Django',
            'url': 'https://hackersandslackers.com/creating-django-views/',
            'datePublished': '2020-04-23T12:21:00.000-04:00', 'dateModified': '2020-05-02T13:31:33.000-04:00',
            'image': {'@type': 'ImageObject',
                      'url': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/04/django-views-1.jpg',
                      'width': '1000', 'height': '523'},
            'publisher': {'@type': 'Organization', 'name': 'Hackers and Slackers', 'founder': 'Todd Birchard',
                          'logo': {'@type': 'ImageObject',
                                   'url': 'https://hackersandslackers-cdn.storage.googleapis.com/2020/03/logo-blue-full.png',
                                   'width': 60, 'height': 60}},
            'description': 'Create interactive user experiences by writing Django views to handle dynamic content, submitting forms, and interacting with data.',
            'mainEntityOfPage': {'@type': 'WebPage', '@id': 'https://hackersandslackers.com'}}


def test_scrape(url, expected_json):
    """Match scrape's fetched metadata to known value."""
    metadata = scrape(url)
    assert metadata == expected_json

test_scrape.py

Build a Metadata Scraper

Of course, scrape() simply puts data on a silver platter for you - there's still the work of grabbing the values. To give you an example of a fully fleshed-out script to scrape metadata with extruct, I'll share with you my own personal treasure: the script I use to generate link previews:

"""Fetch structured JSON-LD data from a given URL."""
from typing import Optional, List
import requests
import extruct
from w3lib.html import get_base_url


def scrape(url: str) -> Optional[List[dict]]:
    """Parse structured data from a URL."""
    req = requests.get(url, headers=http_headers)
    base_url = get_base_url(req.content, url)
    json_ld = render_json_ltd(req.content, base_url)
    card = ["bookmark", {
                "type": "bookmark",
                "url": get_canonical(json_ld, html),
                "metadata": {
                    "url": get_canonical(json_ld),
                    "title": get_title(json_ld),
                    "description": get_description(json_ld),
                    "author": get_author(json_ld),
                    "publisher": get_publisher(json_ld),
                    "thumbnail": get_image(json_ld),
                    }
                }
            ]
    return card


def get_html(url):
    """Get raw HTML from a URL."""
    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'GET',
        'Access-Control-Allow-Headers': 'Content-Type',
        'Access-Control-Max-Age': '3600',
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }
    req = requests.get(url, headers=headers)
    return req.text


def get_metadata(html, url):
    """Fetch JSON-LD structured data."""
    metadata = extruct.extract(
        html,
        base_url=get_domain(url),
        syntaxes=['json-ld'],
        uniform=True
    )['json-ld']
    if bool(metadata) and isinstance(metadata, list):
        metadata = metadata[0]
    return metadata


def get_title(json_ld: dict) -> Optional[str]:
    """Fetch title via extruct."""
    title = None
    if bool(json_ld) and json_ld.get('headline'):
        if isinstance(json_ld.get('headline'), list):
            title = json_ld['headline'][0]
        elif isinstance(json_ld.get('headline'), str):
            title = json_ld.get('headline')
        if isinstance(title, str):
            return title.replace("'", "")
    if bool(json_ld) and json_ld.get('title'):
        if isinstance(json_ld.get('title'), list):
            title = json_ld['title'][0]
        elif isinstance(json_ld.get('title'), str):
            title = json_ld.get('title')
    return title


def get_image(json_ld: dict) -> Optional[str]:
    """Fetch share image via extruct."""
    image = None
    if bool(json_ld) and json_ld.get('image'):
        if isinstance(json_ld['image'], list):
            image = json_ld['image'][0]
            if isinstance(image, dict):
                image = image.get('url')
            if isinstance(image, str):
                return image
        elif isinstance(json_ld.get('image'), dict):
            image = json_ld['image'].get('url')
     return image


def get_description(json_ld: dict) -> Optional[str]:
    """Fetch description via extruct."""
    if bool(json_ld) and json_ld.get('description'):
        return json_ld['description']
    return None


def get_author(json_ld: dict, html: BeautifulSoup) -> Optional[str]:
    """Fetch author name via extruct with BeautifulSoup fallback."""
    author = None
    if bool(json_ld) and json_ld.get('author'):
        if isinstance(json_ld['author'], list):
            author = json_ld['author'][0].get('name')
        elif isinstance(json_ld['author'], dict):
            author = json_ld['author'].get('name')
    return author


def get_publisher(json_ld: dict) -> Optional[str]:
    """Fetch publisher name via extruct."""
    publisher = None
    if bool(json_ld) and json_ld.get('publisher'):
        if isinstance(json_ld['publisher'], list):
            publisher = json_ld['publisher'][0].get('name')
        elif isinstance(json_ld['publisher'], dict):
            publisher = json_ld['publisher'].get('name')
    return publisher


def get_canonical(json_ld: dict) -> Optional[str]:
    """Fetch canonical URL via extruct."""
    canonical = None
    if bool(json_ld) and json_ld.get('mainEntityOfPage'):
        if isinstance(json_ld['mainEntityOfPage'], dict):
            canonical = json_ld['mainEntityOfPage'].get('@id')
        elif isinstance(json_ld['mainEntityOfPage'], str):
            return json_ld['mainEntityOfPage']
    return canonical

Metadata scraper with extruct

One More For the Toolbox

Unless you're actually looking to create link previews like the one I included, using extruct as a standalone library without a more extensive plan or toolkit isn't going to deliver much to you other than an easy interface for getting better metadata from individual web pages. Instead, consider looking at the bigger picture of what a single page's metadata gives us. We now have effortless access to information that crawlers can use to move through sites, associate data with individuals, and ultimately create a picture of an entity's entire web presence, whether that entity is a person, organization, or whatever.

If you look closely, one of extruct's main dependencies is actually BeautifulSoup. You could argue that you may have been able to write this library yourself, and you might be right, but that isn't the point. Data mining behemoths aren't nuclear arsenals; they're collections of tools used in conjunction cleverly to wreak havoc upon the world as efficiently as possible. We're getting there.

This has been a quick little script, but if you're interested I've thrown the source up on Github here:

hackersandslackers / jsonld-scraper-tutorial

🌎 🖥 Supercharge your scraper to extract quality page metadata by parsing JSON-LD data via Python's extruct library.

Structured Data Scraping Tutorial

Supercharge your scraper to extract quality page metadata by parsing JSON-LD data via Python's extruct library.

This repository contains source code for the accompanying tutorial on Hackers and Slackers: https://hackersandslackers.com/scrape-metadata-json-ld/

Installation

Installation via requirements.txt:

$ git clone https://github.com/hackersandslackers/jsonld-scraper-tutorial.git
$ cd jsonld-scraper-tutorial
$ python3 -m venv myenv
$ source myenv/bin/activate
$ pip3 install -r requirements.txt
$ python3 main.py

Installation via Pipenv:

$ git clone https://github.com/hackersandslackers/jsonld-scraper-tutorial.git
$ cd jsonld-scraper-tutorial
$ pipenv shell
$ pipenv update
$ python3 main.py

Installation via Poetry:

$ git clone https://github.com/hackersandslackers/jsonld-scraper-tutorial.git
$ cd jsonld-scraper-tutorial
$ poetry shell
$ poetry update
$ poetry run

Usage

To change the URL targeted by this script, update the URL variable in config.py.

Hackers and Slackers tutorials are free of charge. If you found this tutorial helpful, a small donation would be greatly appreciated to keep us in business. All proceeds go towards coffee, and…

View on GitHub

Until next time.

Deploy Serverless Golang Functions with Netlify

Todd Birchard — Mon, 03 Aug 2020 13:09:00 +0000

The race to monetize static-site hype was over before it began — if you're a JAMStack developer, you're a Netlify customer. Surprisingly, the outcome of this unintentional vendor lock-in has been working out pretty well. JAMStack's paradigm of webhook-driven actions rewrites the narrative of static sites as dynamic entities. By either luck or foresight, Netlify is comfortably positioned to provide a home for static-sites as the only cloud host to focus solely on this model of development by providing features, like serverless functions.

Any market which finds itself dominated by a single vendor is bound to have some downsides. In Netlify's case, one such violation comes in the form of occasional poor documentation, especially regarding Golang function deployment. Netlify has dedicated precisely one page of documentation dedicated to writing serverless functions in Go with zero mention of how to actually deploy said functions. Compare this to Netlify's commitment to JavaScript function development, which includes a dedicated CLI, a build plugin, and documentation that covers useful details.

This tutorial assumes you have a basic understanding of Golang and GatsbyJS —this post is not a tutorial about learning Go from scratch. We will cover:

The basics of Go development in the context of Lambda functions.
Deploying Go functions as part of a GatsbyJS site hosted on Netlify.
Interacting with and monitoring deployed Go functions.

Creating a Go Serverless Function

We'll start by creating a new Go project on our GOPATH as we normally would:

$ mkdir ~/go/src/github.com/toddbirchard/netlify-serverless-tutorial
$ cd ~/go/src/github.com/toddbirchard/netlify-serverless-tutorial

Create a directory to store your function source.

Next, we initialize our module by running go mod init in the current folder, which should produce the following outcome:

module github.com/toddbirchard/netlify-serverless-tutorial

go 1.14

go.mod

Naming your Go module to match the URL of a Github repository is a common practice you should already abide by. Still, it's especially important to do so for our purposes. Netlify creates serverless functions by taking the Go source code we give and building executable binaries. To do so, Netlify needs to be aware of the dependencies our project might have. Instead of uploading our Go source with all its dependencies, Netlify intelligently looks to the dependencies found in our module's Github repository and pulls them itself.

Get started by pasting the minimum boilerplate into a new file called main.go :

package main

import (
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func Handler(request events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
    return &events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body: "Hello World!",
    }, nil
}

func main() {
    // Initiate AWS Lambda handler
    lambda.Start(Handler)
}

main.go

Oh snap, we've imported an external library! The aws-lambda-go library is essential for our function to work. Get and install the library by running go get in your project directory. This will update your go.mod file accordingly:

module github.com/toddbirchard/netlify-serverless-tutorial

go 1.14

require github.com/aws/aws-lambda-go v1.18.0

go.mod

Working with Request Data

Netlify Lambda functions take the form of HTTP endpoints, so odds are you're building something that takes the input of a request (whether it be a request's params or body) to produce a response. We can access information about an incoming request via aws-lambda-go's APIGatewayProxyRequest struct: Lambda's default parameter passed to our handler.

Let's see this in action by grabbing a query string parameter from an incoming request and outputting the result. We can easily modify our original handler to check an incoming request for a parameter called name (i.e., example.com/function?name=todd):

...

func Handler(request events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
    name := request.QueryStringParameters["name"]
    response := fmt.Sprintf("Hello %s!", name)

    return &events.APIGatewayProxyResponse{
        StatusCode: 200,
        Headers: map[string]string{"Content-Type": "text/html; charset=UTF-8"},
        Body: response,
    }, nil
}

...

main.go

APIGatewayProxyRequest has a variable called QueryStringParameters, which is a key/value map of query sting names and their values. The above attempts to extract the value of a query string called name to create a cheerful response for the user. If an incoming request were not to contain a query string called name, the value would return nil.

We could extract a lot more from a APIGatewayProxyRequest struct if we so chose:

Path : URL requested by the user which generated the request.
HTTPMethod : GET, POST, PUT, etc.
Headers & MultiValueHeaders : HTTP headers sent by the user's request.
QueryStringParameters & QueryStringParameters : Query strings with single or multiple values.
Body : Data (such as JSON) sent in the body of the request.

The output of a Lambda function is created by returning APIGatewayProxyResponse. Our response returns a simple 200 status code, along with a text response reading "Hello {name}!". We specify the content type of the response via our response's Headers variable, which accepts a map of headers such as Content-Type.

Testing our Function

I promised to spend minimal time on the actual Golang aspect of this post, but writing unit tests for Lambda functions is particularly tricky as it involves forming fake requests and evaluating their responses. aws-lambda-go has a recommended syntax for writing multiple tests against a single handler which I found quite useful:

package main_test

import (
    "github.com/aws/aws-lambda-go/events"
    "github.com/stretchr/testify/assert"
    main "github.com/toddbirchard/netlify-serverless-tutorial"
    "log"
    "testing"
)

func TestHandler(t *testing.T) {
    tests := []struct {
        request events.APIGatewayProxyRequest
        expect string
        err error
    } {
        {
            // Test name has value
            request: events.APIGatewayProxyRequest{QueryStringParameters: map[string]string{"name": "Paul"}},
            expect: "Hello Paul!",
            err: nil,
        },
        {
            // Test name is null
            request: events.APIGatewayProxyRequest{},
            expect: "Hello !",
            err: nil,
        },
    }

    for i, test := range tests {
        response, err := main.Handler(test.request)
        assert.IsType(t, test.err, err)
        assert.Equal(t, test.expect, response.Body)
        log.Printf("Test %d: %s", i, response.Body)
    }
}

main_test.go

TestHandler creates two hypothetical situations (aka tests) to run against our Handler function. The first test passes a query string parameter "Paul", while the second passes none at all. These scenarios are then evaluated in a loop, which outputs the following when tested with go test:

2020/08/01 03:17:47 Test 0: Hello Paul!
2020/08/01 03:17:47 Test 1: Hello !
PASS
ok github.com/toddbirchard/netlify-serverless-tutorial 0.139s

main_test.go output

The validity of each test is validated via assertions belonging to the testify module, which I recommend for unit tests like these. Be sure to go get this module if you chose to include it:

module github.com/toddbirchard/netlify-serverless-tutorial

go 1.14

require (
    github.com/aws/aws-lambda-go v1.18.0
    github.com/stretchr/testify v1.4.0
)

go.mod

We now have a Lambda function which is complete enough to deploy.... but how do we go about that?

Serverless Functions in GatsbyJS

The structure of a minimal GatsbyJS project looks something like this:

/my-gatsby-project
├── /src
├── /public
├── /static
├── gatsby-browser.js
├── gatsby-config.js
├── gatsby-node.js
└── package.json

GatsbyJS project structure.

All of the files and directories above are standard Gatsby stuff doing Gatsby things. Things become a little more interesting when we deploy a Gatsby site with Lambda functions, as our Lambda's source code and binary is going to live side-by-side with our Gatsby code. This takes the form of two directories: a /functions directory and a /go-src directory:

/my-gatsby-project
├── /functions
| └── helloworld
├── /go-src
| └── /helloworld
| └── main.go
├── /src
├── /public
├── /static
├── gatsby-browser.js
├── gatsby-config.js
├── gatsby-node.js
└── package.json

GatsbyJS project structure.

/go-src is where we dump the source code of our Golang functions ( .go files). We create subdirectories in go-src for each Lambda function we want to deploy. Go source code in each subdirectory will be compiled to a binary sharing the name of the directory in the /functions folder. In the above example, the /helloworld subdirectory compiles to a binary in functions/helloworld, which ultimately dictates the eventual URL of our endpoint as well, as such:

https://hackersandslackers.com/.netlify/functions/helloworld

URL of a deployed Golang Lambda function.

If you're wondering how the source of main.go got into our new /go-src/helloworld destination, the underwhelming answer is a simple copy-and-paste. No tricks there.

Building Netlify Lambda Binaries in GatsbyJS

Let's recap where we're at:

GatsbyJS project: Check ✅
Lambda source code: Check ✅
Binaries built from source upon project builds: Not check 🚫

When we deploy to Netlify, we want our project to find the source in /go-src and build the resulting binaries to the /functions directory. Netlify looks for a build command in netlify.toml, which would typically be gatsby build under normal circumstances. Since we're now building our site and functions, we're going to recruit the help of a Makefile to enable a more involved build process.

Make sure you have a netlify.toml file in your GatsbyJS project root where we can specify the build command of our Makefile, as well as define our functions folder:

[build]
  base = "/"
  command = "make build"
  publish = "/public"
  functions = "/functions"

[build.environment]
  GO_VERSION = "1.14.5"

netlify.toml

We've changed what you most likely had as command = "gatsby build" to command = "make build", which looks for a Makefile containing a build command to execute. Let's create that now.

Making a Makefile

If you're new to Makefiles, they're a standard fixture in projects that allow us to define one-liners that kick off a series of events, such as building a project. Our Makefile is going to look like this:

build:
   npm run-script build
   mkdir -p functions
   GOOS=linux
   GOARCH=amd64
   GO111MODULE=on
   GOBIN=${PWD}/functions go get ./...
   GOBIN=${PWD}/functions go install ./...

Makefile

Hopefully, you find this string of commands to be confusing as shit as I did (you might be overqualified for reading this post otherwise). We'll break down this build command line-by-line:

npm run-script build: This command runs a script in package.json labeled as "build," which is likely running gatsby build. This step is building your Gatsby site as you normally would for production, so nothing special there.
mkdir -p functions: Creates a directory called functions in your project's root directory, in case it doesn't exist. Since I don't recommend uploading your compiled binaries to Github (Netlify will rebuild them regardless), there's a good chance this folder won't exist upon committing. As a result, we need this command as a way to create a fresh functions folder on our Netlify server.
GOOS=linux & GOARCH=amd64: Golang's GOOS and GOARCH environment variables tell Go which target operating system (Linux) and architecture (AMD 64-bit) our binaries are intended to be built for. The values we're passing here align with the OS and architecture of Netlify's server, thus we're ensuring Go is targeting the correct infrastructure to compile binaries for.
GO111MODULE=on: Things get a bit interesting here as Go source we're building in /go-src is not actually on our GOPATH, nor will it be upon deployment to Netlify. GO111MODULE is a workaround for building Go projects outside of the designated GOPATH; this allows us to get dependencies of functions living in our Gatsby project and build the resulting binaries without worrying about being shackled to our (or Netlify's) GOPATH.
GOBIN=${PWD}/functions go install ./...: Lastly, we need to get and install all of our function's dependencies to build them into our final binary. This line is effectively looking for all go modules to build into binaries and builds them in the /functions folder.

Give it a Whirl

We're ready to test our function locally! Run make build in your Gatsby root folder to make sure things run smoothly. If you're successful, you should see an output like this towards the end of your build:

info Done building in 63.568185195 sec
mkdir -p functions
GOOS=linux
GOARCH=amd64
GO111MODULE=on
GOBIN=/Users/toddbirchard/projects/stockholm/functions go get ./...
GOBIN=/Users/toddbirchard/projects/stockholm/functions go install ./...

Output of make build

You should now see Go binaries in your local /functions directory! Add your /functions directory to your .gitignore file and commit that masterpiece.

See it in Action

To demonstrate a GatsbyJS site utilizing a Netlify function, I set up a demo for your pleasure here:

https://serverless-golang-tutorial.netlify.app

This is a simple GatsbyJS site that takes the output of our helloworld function and writes it to the page via some very simple frontend JavaScript. I busted some ass to put together a somewhat respectable demo for you, because copying is way easier than reading. Don't say I've never done anything for you:

hackersandslackers / netlify-functions-tutorial

Write and deploy Golang Lambda Functions to your GatsbyJS site on Netlify.

Netlify Function Golang Tutorial

An example GatsbyJS project demonstrating how to deploy Lambd functions with Netlify. Source code for the accompanying tutorial found here: https://hackersandslackers.com/deploy-serverless-golang-functions-with-netlify/

Demo here: https://serverless-golang-tutorial.netlify.app

View on GitHub

Why is this So Convoluted?

Compared to the workflow of deploying JavaScript functions on Netlify, Golang is an undocumented and seemingly half-supported nightmare. Until just this week, inspecting your Go Lambda functions on Netlify's UI would append a .js file extension to every function by default with the presumed assumption that everybody is picking JS over Go:

Netlify doesn't care about Golang™

This is generally in line with Netlify's actions of half-assing their Go documentation, and even releasing a CLI to assist in function development called netlify-lambda, which only supports JavaScript.

Other than feeling like a second-class citizen, these gaffs are more amusing than they are harmful. Once your functions are deployed, they work as you'd expect. All is well that ends well.

If you're looking for more resources on Go Lambda development, AWS has a Github repo of examples that might prove useful. Aside from that and this tutorial, I think you have everything you need 😉. Until next time.

Connecting Pandas to a Database with SQLAlchemy

Todd Birchard — Fri, 12 Jun 2020 13:53:00 +0000

Databases. You love them, you need them, but let's face it... you've already mastered working with them. There's only so much fun to be had in the business of opening database connections, pulling rows, and putting them back where they came from. Wouldn't it be great if we could skip the boring stuff and work with data?

Pandas and SQLAlchemy are a mach made in Python heaven. They're individually amongst Python's most frequently used libraries. Together they're greater than the sum of their parts, thanks to Pandas' built-in SQLAlchemy integration.

Create a SQLAlchemy Connection

As you might imagine, the first two libraries we need to install are Pandas and SQLAlchemy. We need to install a database connector as our third and final library, but the library you need depends on the type of database you'll be connecting to. If you're connecting to MySQL I recommend installing PyMySQL ( pip install pymysql ). If you're connecting to Postgres, go with Psycopg2 ( pip install psycopg2 ). The only time we'll use either of these libraries is when we establish a database connection with SQLAlchemy.

SQLAlchemy URIs

A URI (or connection string), is simply a string containing the information needed to connect to something like a database. Here's an example:

postgres+psycopg2://myuser:mypassword@hackersdb.example.com:5432/mydatabase

Postgres database connection URI

The first part of our string is postgres+psycop2, which is a combination of our target database type and our connector. If you're connecting to MySQL, replace this with mysql+pymysql. In case the rest of the URI isn't self-explanatory, here's a breakdown of each piece of this string:

[DB_FLAVOR]+[DB_PYTHON_LIBRARY]://[USERNAME]:[PASSWORD]@[DB_HOST]:[PORT]/[DB_NAME]

SQLAlchemy URI structure

SQLAlchemy Engines

An "engine" is an object used to connect to databases using the information in our URI. Once we create an engine, downloading and uploading data is as simple as passing this object to Pandas:

from os import environ
from sqlalchemy import create_engine

db_uri = environ.get('SQLALCHEMY_DATABASE_URI')
self.engine = create_engine(db_uri, echo=True)

Configure SQLAlchemy engine.

Besides accepting a URI, create_engine() can accept a few optional kwargs as well. I've decided to set echo=True, which will log every query our SQL database executes to the terminal. If your database requires SSL, you may need to utilize the connect_args parameter to pass a certificate.

Believe it or not, we're already done dealing with database setup! From here forward we're able to pull or upload data into Pandas via easy one-liners.

Create a SQL Table From a DataFrame

For our first trick, let's create a SQL table from data in a CSV. I downloaded a CSV containing NYC job data which I'll be using to demonstrate:

We're going to create a DataFrame from this CSV, as we've done a million times before:

import pandas as pd


jobs_df = pd.read_csv('data/nyc-jobs.csv')

Read and format data from CSV.

We now have a DataFrame ready to be saved as a SQL table! We can accomplish this with a single method built in to all DataFrames called to_sql(). As the name suggests, to_sql() allows us to upload our DataFrame to a SQL database as a SQL table. Let's see it in action:

from sqlalchemy.types import Integer, Text, String, DateTime

...

table_name = 'nyc_jobs'

jobs_df.to_sql(
    table_name,
    engine,
    if_exists='replace',
    index=False,
    chunksize=500,
    dtype={
        "job_id": Integer,
        "agency": Text,
        "business_title": Text,
        "job_category": Text,
        "salary_range_from": Integer,
        "salary_range_to": Integer,
        "salary_frequency": String(50),
        "work_location": Text,
        "division/work_unit": Text,
        "job_description": Text,
        "posting_date": DateTime,
        "posting_updated": DateTime
    }
)

Save DataFrame to SQL table.

There's quite a bit happening here! to_sql() attempts to create a table with the name nyc_jobs in the database associated with engine. These two positional arguments are technically the only required parameters we need to pass, but it's a very good idea to take advantage of Pandas' ability to be more specific in table creation:

if_exists: This argument specifies what to do in the situation where a database table with the name nyc_jobs already exists in the database. By default, Pandas will throw an error, which isn't very useful unless we only care about creating this table the first time. Passing replace to this argument will drop the existing table and replace it with the data & data types associated with the current DataFrame. append will keep the existing table the same, but append all rows in the DataFrame to the existing table.
schema: Accepts the name of the Postgres schema to save your table in.
index: When True , the resulting table will honor your DataFrame's index to create a column with the appropriate key in your database.
chunksize: Passing a number to this parameter will attempt to upload your data as a stream of "chunks" n rows at a time, as opposed to all at once. Passing a chunksize is useful for particularly large datasets which may be at risk of interruption during upload.
dtype: Passing a Python dictionary to dtype lets us explicitly set the datatypes of each column in our database, where each key is the column name and each value is the data type (I highly recommend doing this). You'll notice we import various data types from sqlalchemy.types , which we then associate with each column's name. If the target SQL table doesn't exist yet, passing these datatypes will ensure that each SQL column is created with the appropriate data constraint, as opposed to each column rendered simply as "text." If a target SQL table does exist, these data types must match the types of the existing table, or you'll receive a SQL error during the upload.

Since we set SQLAlchemy's echo parameter to True, I'm able to see exactly what my database does with this DataFrame:

2020-06-11 23:49:21,082 INFO sqlalchemy.engine.base.Engine SHOW VARIABLES LIKE 'sql_mode'
2020-06-11 23:49:21,082 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,396 INFO sqlalchemy.engine.base.Engine SHOW VARIABLES LIKE 'lower_case_table_names'
2020-06-11 23:49:21,396 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,432 INFO sqlalchemy.engine.base.Engine SELECT DATABASE()
2020-06-11 23:49:21,432 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,470 INFO sqlalchemy.engine.base.Engine show collation where "Charset" = 'utf8mb4' and "Collation" = 'utf8mb4_bin'
2020-06-11 23:49:21,470 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,502 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1
2020-06-11 23:49:21,502 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,523 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1
2020-06-11 23:49:21,523 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,537 INFO sqlalchemy.engine.base.Engine SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8mb4) COLLATE utf8mb4_bin AS anon_1
2020-06-11 23:49:21,537 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,587 INFO sqlalchemy.engine.base.Engine DESCRIBE "nyc_jobs"
2020-06-11 23:49:21,588 INFO sqlalchemy.engine.base.Engine {}
2020-06-11 23:49:21,654 INFO sqlalchemy.engine.base.Engine ROLLBACK
2020-06-11 23:49:21,691 INFO sqlalchemy.engine.base.Engine 
CREATE TABLE nyc_jobs (
    job_id INTEGER, 
    agency TEXT, 
    business_title TEXT, 
    job_category TEXT, 
    salary_range_from INTEGER, 
    salary_range_to INTEGER, 
    salary_frequency VARCHAR(50), 
    work_location TEXT, 
    division TEXT, 
    job_description TEXT, 
    created_at DATETIME, 
    updated_at DATETIME
)

SQLAlchemy creating a table from a Pandas DataFrame.

Just as we described, our database uses CREATE TABLE nyc_jobs to create a new SQL table, with all columns assigned appropriate data types.

Create DataFrame from SQL Table

Loading data from a database into a Pandas DataFrame is surprisingly easy. To load an entire table, use the read_sql_table() method:

table_df = pd.read_sql_table(
    table_name,
    con=engine
)

Convert SQL table to Pandas DataFrame

The first two parameters we pass are the same as last time: first is our table name, and then our SQLAlchemy engine. The above snippet is perhaps the quickest and simplest way to translate a SQL table into a Pandas DataFrame, with essentially no configuration needed! Interestingly, Pandas is still oblivious to the dtype of each column we've pulled despite having pulled from a database, as we can see with print(table_df.info()):

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3123 entries, 0 to 3122
Data columns (total 12 columns):
 # Column Non-Null Count Dtype 
--- ------ -------------- ----- 
 0 job_id 3123 non-null int64 
 1 agency 3123 non-null object
 2 business_title 3123 non-null object
 3 job_category 3121 non-null object
 4 salary_range_from 3123 non-null int64 
 5 salary_range_to 3123 non-null int64 
 6 salary_frequency 3123 non-null object
 7 work_location 3123 non-null object
 8 division 3123 non-null object
 9 job_description 3123 non-null object
 10 created_at 3123 non-null object
 11 updated_at 3123 non-null object
dtypes: int64(3), object(9)
memory usage: 292.9+ KB
None

SQLAlchemy creating a table from a Pandas DataFrame.

The read_sql_table() method can accept far more arguments than the two we passed. Here's an example where we read a SQL table and force some explicit things to happen:

...

table_df = pd.read_sql_table(
    "nyc_jobs",
    con=engine,
    schema='public',
    index_col='job_id',
    coerce_float=True,
    columns=[
        'job_id',
        'business_title',
        'job_category',
        'posting_date',
        'posting_updated'
    ],
    parse_dates=[
        'created_at',
        'updated_at'
    ],
    chunksize=500
)

Loading a SQL table with explicit values.

Some arguments should look familiar from when we ran to_sql() earlier. schema and chunksize have the same meanings as they did previously. We also have a few new arguments as well:

index_col: We can select any column of our SQL table to become an index in our Pandas DataFrame, regardless of whether or not the column is an index in SQL. We can pass the name of a single column as a string, or a list of strings representing the names of multiple columns.
coerce_float: When set to True , Pandas will look at columns containing numbers and attempt to convert these columns to floating point numbers. This attribute is set to True by default.
columns: Passing a list of column names to this attribute will create a DataFrame from only the columns we provide (similar to a SQL select on x columns).
parse_dates: When moving data into Pandas we need to explicitly state which columns should be considered DateTime columns.

Create DataFrames From Query Results

There will probably be times where you're just looking for a subset of data in a table as opposed to the entire table. In this scenario we can use read_sql(), which creates a DataFrame from the results of a SQL query you run on a table:

...

sql_df = pd.read_sql(
    "SELECT * FROM nyc_jobs",
    con=engine,
    parse_dates=[
        'created_at',
        'updated_at'
    ]
)

SQL query to Pandas DataFrame.

This time around our first parameter is a SQL query instead of the name of a table. We can modify this query to select only specific columns, rows which match criteria, or anything else you can do with SQL.

That's all folks! If you're interested, the source is up on Github here:

https://github.com/hackersandslackers/pandas-sqlalchemy-tutorial

Scraping Data on the Web with BeautifulSoup

Todd Birchard — Thu, 11 Jun 2020 22:40:09 +0000

Whether it be Kaggle, Google Cloud, or the federal government, there's plenty of reliable open-sourced data on the web. While there are plenty of reasons to hate being alive in our current chapter of humanity, open data is one of the few redeeming qualities of life on Earth today. But what is the opposite of "open" data, anyway?

Like anything free and easily accessible, the only data inherently worth anything is either harvested privately or stolen from sources that would prefer you didn't. This is the sort of data business models can be built around, as social media platforms such as LinkedIn have shown us as our personal information is bought and sold by data brokers. These companies attempted to sue individual programmers like ourselves for scraping the data they collected via the same means, and epically lost in a court of law:

https://www.forbes.com/sites/emmawoollacott/2019/09/10/linkedin-data-scraping-ruled-legal/

The topic of scraping data on the web tends to raise questions about the ethics and legality of scraping, to which I plea: don't hold back. If you aren't personally disgusted by the prospect of your life being transcribed, sold, and frequently leaked, the court system has ruled that you legally have a right to scrape data. The name of this publication is not People Who Play It Safe And Slackers. We're a home for those who fight to take power back, and we're going to scrape the shit out of you.

Tools for the Job

Web scraping in Python is dominated by three major libraries: BeautifulSoup, Scrapy, and Selenium. Each of these libraries intends to solve for very different use cases. Thus it's essential to understand what we're choosing and why.

BeautifulSoup is one of the most prolific Python libraries in existence, in some part having shaped the web as we know it. BeautifulSoup is a lightweight, easy-to-learn, and highly effective way to programmatically isolate information on a single webpage at a time. It's common to use BeautifulSoupin conjunction with the requests library, where requests will fetch a page, and BeautifulSoup will extract the resulting data.
Scrapy has an agenda much closer to mass pillaging than BeautifulSoup. Scrapy is a tool for building crawlers: these are absolute monstrosities unleashed upon the web like a swarm, loosely following links, and haste-fully grabbing data where data exists to be grabbed. Because Scrapy serves the purpose of mass-scraping, it is much easier to get in trouble with Scrapy.
Selenium isn't exclusively a scraping tool as much as an automation tool that can be used to scrape sites. Selenium is the nuclear option for attempting to navigate sites programmatically, and should be treated as such: there are much better options for simple data extraction.

We'll be using BeautifulSoup, which should genuinely be anybody's default choice until the circumstances ask for more. BeautifulSoup is more than enough to steal data.

Preparing Our Extraction

Before we steal any data, we need to set the stage. We'll start by installing our two libraries of choice:

$ pip3 install beautifulsoup4 requests

Install beautifulsoup and requests.

As mentioned before, requests will provide us with our target's HTML, and beautifulsoup4 will parse that data.

We need to recognize that a lot of sites have precautions to fend off scrapers from accessing their data. The first thing we can do to get around this is spoofing the headers we send along with our requests to make our scraper look like a legitimate browser:

import requests
from bs4 import BeautifulSoup

headers = {
    'Access-Control-Allow-Origin': '*',
    'Access-Control-Allow-Methods': 'GET',
    'Access-Control-Allow-Headers': 'Content-Type',
    'Access-Control-Max-Age': '3600',
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }

Set request headers.

This is only a first line of defense (or offensive, in our case). There are plenty of ways sites can still keep us at bay, but setting headers works shockingly well to fix most issues.

Now let's fetch a page and inspect it with BeautifulSoup:

import requests
from bs4 import BeautifulSoup

...
url = "http://example.com"
req = requests.get(url, headers)
soup = BeautifulSoup(req.content, 'html.parser')
print(soup.prettify())

Scrape example.com

We set things up by making a request to http://example.com. We then create a BeautifulSoup object which accepts the raw content of that response via req.content. The second parameter, 'html.parser', is our way of telling BeautifulSoup that this is an HTML document. There are other parsers available for parsing stuff like XML, if you're into that.

When we create a BeautifulSoup object from a page's HTML, our object contains the HTML structure of that page, which can now be easily parsed by all sorts of methods. First, let's see what our variable soup looks like by using print(soup.prettify()):

<html class="gr__example_com"><head>
    <title>Example Domain</title>
    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta property="og:site_name" content="Example dot com">
    <meta property="og:type" content="website">
    <meta property="og:title" content="Example">
    <meta property="og:description" content="An Example website.">
    <meta property="og:image" content="http://example.com/img/image.jpg">
    <meta name="twitter:title" content="Hackers and Slackers">
    <meta name="twitter:description" content="An Example website.">
    <meta name="twitter:url" content="http://example.com/">
    <meta name="twitter:image" content="http://example.com/img/image.jpg">
</head>

<body data-gr-c-s-loaded="true">
  <div>
    <h1>Example Domain</h1>
      <p>This domain is established to be used for illustrative examples in documents.</p>
      <p>You may use this domain in examples without prior coordination or asking for permission.</p>
    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
  </div>
</body>

</html>

HTML for example.com

Targeting HTML Elements

There are many methods available to us for pinpointing and grabbing the information we're trying to get out of a page. Finding the exact information we want out of a web page is a bit of an art form: effective scraping requires us to recognize patterns in document's HTML that we can take advantage of to ensure we only grab the pieces we need. This is especially the case when dealing with sites that actively try to prevent us from doing just that.

Understanding the tools we have at our disposal is the first step to developing a keen eye for what's possible. We'll start with the meat and potatoes.

Using find() & find_all()

The most straightforward way to finding information in our soup variable is by utilizing soup.find(...) or soup.find_all(...). These two methods work the same with one exception: find returns the first HTML element found, whereas find_all returns a list of all elements matching the criteria (even if only one element is found, find_all will return a list of a single item).

We can search for DOM elements in our soup variable by searching for certain criteria. Passing a positional argument to find_all will return all anchor tags on the site:

soup.find_all("a")
# <a href="http://example.com/elsie" class="boy" id="link1">Elsie</a>
# <a href="http://example.com/lacie" class="boy" id="link2">Lacie</a> 
# <a href="http://example.com/tillie" class="girl" id="link3">Tillie</a>

Find all <a> tags.

We can also find all anchor tags which have the class name "boy". Passing the class_ argument allows us to filter by class name. Note the underscore!

soup.find_all("a" class_="boy")
# <a href="http://example.com/elsie" class="boy" id="link1">Elsie</a>
# <a href="http://example.com/lacie" class="boy" id="link2">Lacie</a>

Find all <a> tags assigned a certain class.

If we wanted to get any element with the class name "boy" besides anchor tags, we can do that too:

soup.find_all(class_="boy")
# <a href="http://example.com/elsie" class="boy" id="link1">Elsie</a>
# <a href="http://example.com/lacie" class="boy" id="link2">Lacie</a>

Find all elements assigned a certain class.

We can search for elements by id in the same way we searched for classes. Remember that we should only expect a single element to be returned with an id, so we should use find here:

soup.find("a", id="link1")
# <a href="http://example.com/elsie" class="boy" id="link1">Elsie</a>

Find element by ID.

Often times we'll run into situations where elements don't have reliable class or id values. Luckily we can search for DOM elements with any attribute, including non-standard ones:

soup.find_all(attrs={"data-args": "bologna"})

Find elements by attribute.

CSS Selectors

Searching HTML using CSS selectors is one of the most powerful ways to find what you're looking for, especially for sites trying to make your life difficult. Using CSS selectors enables us to find and leverage highly-specific patterns in the target's DOM structure. This is the best way to ensure we're grabbing exactly the content we need. If you're rusty on CSS selectors, I highly recommend becoming reacquainted. Here are a few examples:

soup.select(".widget.author p")

Find elements via CSS selector syntax.

In this example, we're looking for an element that has a "widget" class, as well as an "author" class. Once we have that element, we go deeper to find any paragraph tags held within that widget. We could also modify this to get only the second paragraph tag inside the author widget:

soup.select(".widget.author p:nth-of-type(2)")

To understand why this is so powerful, imagine a site that intentionally has no identifying attributes on its tags to keep people like you from scraping their data. Even without names to select by, we could observe the DOM structure of the page and find a unique way to navigate to the element we want:

soup.select("body > div:first-of-type > div > ul li")

A specific pattern like this is likely unique to only a single collection of <li> tags on the page we're exploiting. The downside of this method is we're at the whim of the site owner, as their HTML structure could change.

Get Some Attributes

Chances are we'll almost always want the contents or the attributes of a tag, as opposed to the entirety of a tag's HTML. If we're scraping anchor tags, for instance, we probably just want the href value, as opposed to the entire tag. The .get method can be used here to retrieve values of attributes on a tag:

soup.find_all('a').get('href')

Get elements and extract attribute values.

The above finds the destination URLs for all <a> tags on a page. Another example can have us grab a site's logo image:

soup.find(id="logo").get('src')

Sometimes it's not attributes we're looking for, but just the text within a tag:

soup.find('p').get_text()

Get elements and extract text content.

Pesky Tags to Deal With

In our example of creating link previews, a good first source of information would obviously be the page's meta tags: specifically the og tags they've specified to openly provide the bite-sized information we're looking for. Grabbing these tags are a bit more difficult to deal with:

soup.find("meta", property="og:description").get('content')

Now that's ugly. Meta tags are an especially interesting case; they're all uselessly dubbed 'meta', thus we need a second identifier (in addition to the tag name) to specify which meta tag we care about. Only then can we bother to get the actual content of said tag.

Realizing Something Will Always Break

If we were to try the above selector on an HTML page that did not contain an og:description, our script would break unforgivingly. Not only do we miss this data, but we miss out on everything entirely - this means we always need to build in a plan B, and at the very least deal with a lack of tag altogether.

It's best to break out this logic one tag at a time. First, let's look at an example for a base scraper with all the knowledge we have so far:

import requests
from bs4 import BeautifulSoup

def scrape_page_metadata(url):
    """Scrape target URL for metadata."""
    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'GET',
        'Access-Control-Allow-Headers': 'Content-Type',
        'Access-Control-Max-Age': '3600',
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }
    pp = pprint.PrettyPrinter(indent=4)
    r = requests.get(url, headers=headers)
    html = BeautifulSoup(r.content, 'html.parser')
    metadata = {
        'title': get_title(html),
        'description': get_description(html),
        'image': get_image(html),
        'favicon': get_favicon(html, url),
        'sitename': get_site_name(html, url),
        'color': get_theme_color(html),
        'url': url
        }
    pp.pprint(metadata)
    return metadata

main.py

This function lays the foundation for snatching a given URL's metadata. The result we're looking for is a dictionary named metadata, which contains the data we manage to scrape successfully.

Each key in our dictionary has a corresponding function which attempts to scrape the corresponding information. Here's what we have for fetching a page's title , description , and social image values:

...

def get_title(html):
    """Scrape page title."""
    title = None
    if html.title.string:
        title = html.title.string
    elif html.find("meta", property="og:title"):
        title = html.find("meta", property="og:title").get('content')
    elif html.find("meta", property="twitter:title"):
        title = html.find("meta", property="twitter:title").get('content')
    elif html.find("h1"):
        title = html.find("h1").string
    return title

def get_description(html):
    """Scrape page description."""
    description = None
    if html.find("meta", property="description"):
        description = html.find("meta", property="description").get('content')
    elif html.find("meta", property="og:description"):
        description = html.find("meta", property="og:description").get('content')
    elif html.find("meta", property="twitter:description"):
        description = html.find("meta", property="twitter:description").get('content')
    elif html.find("p"):
        description = html.find("p").contents
    return description

def get_image(html):
    """Scrape share image."""
    image = None
    if html.find("meta", property="image"):
        image = html.find("meta", property="image").get('content')
    elif html.find("meta", property="og:image"):
        image = html.find("meta", property="og:image").get('content')
    elif html.find("meta", property="twitter:image"):
        image = html.find("meta", property="twitter:image").get('content')
    elif html.find("img", src=True):
        image = html.find_all("img").get('src')
    return image

main.py

get_title tries to get the <title> tag, which has a very low chance of failing. Just in case the target page actually is missing this tag, we fall back to Facebook and Twitter meta tags. If all of this still fails, we finally resort to trying to pull the first <h1> tag on the page (if we get to this point, we're probably scraping a garbage site).
get_description is nearly identical to our method for scraping page titles. The last resort is a desperate attempt to pull the first paragraph on the page.
get_image looks for the page's "share" image, which is used to generate link previews on social media platforms. Our last resort is to pull the first <img> tag containing a source image.

What Did We Build?

This simple script we just threw together is the basis for how most services generate "link previews": an embedded widget containing a synopsis of a site before clicking in (think Facebook, Slack, Discord, etc.). There are even some services which charge monthly fees of ~$10/month to provide the service we've just built. Instead of paying for something like that, feel free to take my source code and use it as you please:

"""Scrape metadata from target URL."""
import requests
from bs4 import BeautifulSoup
import pprint

def scrape_page_metadata(url):
    """Scrape target URL for metadata."""
    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'GET',
        'Access-Control-Allow-Headers': 'Content-Type',
        'Access-Control-Max-Age': '3600',
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
    }
    pp = pprint.PrettyPrinter(indent=4)
    r = requests.get(url, headers=headers)
    html = BeautifulSoup(r.content, 'html.parser')
    metadata = {
        'title': get_title(html),
        'description': get_description(html),
        'image': get_image(html),
        'favicon': get_favicon(html, url),
        'sitename': get_site_name(html, url),
        'color': get_theme_color(html),
        'url': url
        }
    pp.pprint(metadata)
    return metadata

def get_title(html):
    """Scrape page title."""
    title = None
    if html.title.string:
        title = html.title.string
    elif html.find("meta", property="og:title"):
        title = html.find("meta", property="og:title").get('content')
    elif html.find("meta", property="twitter:title"):
        title = html.find("meta", property="twitter:title").get('content')
    elif html.find("h1"):
        title = html.find("h1").string
    return title

def get_description(html):
    """Scrape page description."""
    description = None
    if html.find("meta", property="description"):
        description = html.find("meta", property="description").get('content')
    elif html.find("meta", property="og:description"):
        description = html.find("meta", property="og:description").get('content')
    elif html.find("meta", property="twitter:description"):
        description = html.find("meta", property="twitter:description").get('content')
    elif html.find("p"):
        description = html.find("p").contents
    return description

def get_image(html):
    """Scrape share image."""
    image = None
    if html.find("meta", property="image"):
        image = html.find("meta", property="image").get('content')
    elif html.find("meta", property="og:image"):
        image = html.find("meta", property="og:image").get('content')
    elif html.find("meta", property="twitter:image"):
        image = html.find("meta", property="twitter:image").get('content')
    elif html.find("img", src=True):
        image = html.find_all("img").get('src')
    return image

def get_site_name(html, url):
    """Scrape site name."""
    if html.find("meta", property="og:site_name"):
        site_name = html.find("meta", property="og:site_name").get('content')
    elif html.find("meta", property='twitter:title'):
        site_name = html.find("meta", property="twitter:title").get('content')
    else:
        site_name = url.split('//')[1]
        return site_name.split('/')[0].rsplit('.')[1].capitalize()
    return sitename

def get_favicon(html, url):
    """Scrape favicon."""
    if html.find("link", attrs={"rel": "icon"}):
        favicon = html.find("link", attrs={"rel": "icon"}).get('href')
    elif html.find("link", attrs={"rel": "shortcut icon"}):
        favicon = html.find("link", attrs={"rel": "shortcut icon"}).get('href')
    else:
        favicon = f'{url.rstrip("/")}/favicon.ico'
    return favicon

def get_theme_color(html):
    """Scrape brand color."""
    if html.find("meta", property="theme-color"):
        color = html.find("meta", property="theme-color").get('content')
        return color
    return None

main.py

I've uploaded the source code for this tutorial to Github, which contains instructions on how to download and run this script yourself. Enjoy, and join us next time when we up the ante with more nefarious scraping tactics!

https://github.com/hackersandslackers/beautifulsoup-tutorial

Understanding GatsbyJS: Create Your First Gatsby Theme

Todd Birchard — Wed, 10 Jun 2020 14:46:00 +0000

I’m no stranger to broadcasting my thoughts, opinions, and occasional lack of knowledge across the eternal internet. That said, I do pride myself on one thing as a shameless producer of mediocre content: I’ve never blogged about blogging, the state of blogs, or the act of creating blogs. Bloggers who blog about blogging carry the same lack of substance derived from rappers who rap about the act of rapping. Unfortunately for all of us, my untarnished record of blogging-about-blogging ends today.

We recently rewrote the blog theme for Hackers and Slackers in GatsbyJS : arguably the sexiest option for generating static sites on the JAMStack.

Why You're Probably Here

You're not here to learn what a JAMStack is, why it’s beneficial, or why you should think they’re cool. There's plenty of well-written documentation on the topic, and there are even more poorly written Medium articles that mostly copy & paste the former. Apologies for the grumpiness- I’ve been JAMing a bit too hard lately.

I'm here to shed light on implementing a stack that's worked well for me: Ghost as a CMS, GatsbyJS as a static site generator, and Netlify for deployment. This is an excellent combination of tools, but there's an absurdly frustrating lack of centralized documentation on how these pieces fit together. Each of these services has excelled at delivering its portion of the pipeline. We're here to put the pieces together.

This series is going to walk through how Gatsby generates static sites. To accomplish this, we're going to create our own Gatsby theme and walk through Gatsby's end-to-end build process. For the sake of this tutorial, we're going to assume you have basic knowledge of GraphQL and React.

Why Ghost as a Headless CMS?

Netlify has effectively cornered the market as the de-facto host of Gatsby sites, which leaves our choice of CMS to be the most significant decision we need to make in our stack. In my opinion, Ghost is an attractive CMS option because of its philosophy of simplicity. The team behind Ghost has shown a respectable amount of restraint when it comes to adding bloated features and data types, which becomes especially important when managing the schema of a static site generator.

When building a website with Gatsby, your site's structure is dictated by the relationships predetermined by your primary source of information. Our raw data implicitly makes fundamental decisions about our site's structure, such as what constitutes "page," or which attributes data models have, such as "tags." Ghost provides us with what we'd expect from a CMS originally intended for blogs: we have authors creating pages/posts which contain tags. It's what we need to build the structure of a static site.

CMS options like Strapi, Prismic, and Contentful are fantastic in what they're able to achieve by abstracting content types. Strapi doesn't even assume the relationship between pages and authors unless you explicitly create those content types and define a many-to-many relationship between them. While this is extremely powerful, I've found that the power to change the fundamental data structure of a site is more dangerous than beneficial. Sometimes we need to protect us from ourselves. This is where Ghost comes in: aside from being a good CMS, Ghost allows us to build a site structure first and extend on it later.

Starting a GatsbyJS Theme

First things first, we'll need to install the Gatsby CLI. The CLI allows us to create new Gatsby projects from the command line easily:

npm i gatsby-cli -g

Install Gatsby CLI globally.

The best way to get started with Gatsby is by cloning one of the many starter templates Gatsby has to offer and iterating on them to make the theme our own. Because we're using Ghost as our CMS, it makes the most sense to start with the Ghost starter template. Gatsby-CLI makes it easy to create new Gatsby projects from existing ones on GitHub, like so:

gatsby new my-gatsby-project https://github.com/TryGhost/gatsby-starter-ghost.git

Create new Gatsby project.

Running gatsby new is essentially the equivalent of running git clone and npm install within the resulting folder. The only difference is gatsby new will not retain a git remote, wheres git clone would.

We can already run our site locally to see what we've started:

$ cd my-gatsby-project
$ gatsby develop

Run Gatsby locally.

The gatsby develop command generates a static site in the directory of a Gatsby project in development mode. We can now preview our theme locally at http://localhost:8000:

Starter theme deployed to http://localhost:8000

Now that we have a working Gatsby theme, we can begin to dissect how Gatsby works. Let's start by dealing with all this placeholder content.

The Ghost Gatsby starter theme is configured to point to placeholder content by default. We can easily configure our theme to point to our own Ghost admin instead by changing the values in .ghost.json :

{
  "development": {
    "apiUrl": "https://gatsby.ghost.io",
    "contentApiKey": "9cc5c67c358edfdd81455149d0"
  },
  "production": {
    "apiUrl": "https://gatsby.ghost.io",
    "contentApiKey": "9cc5c67c358edfdd81455149d0"
  }
}

.ghost.json

The config asks for two values: an apiUrl and a contentApiKey. These are referring to values you'll find in your own Ghost admin by creating an integration on the integrations tab. Here's what mine looks like:

Ghost integration for sourcing content to Gatsby.

Creating any integration will provide us with a Content API Key and an API URL , which are the two things we need for our config:

{
  "development": {
    "apiUrl": "https://hackersandslackers.app",
    "contentApiKey": "8a1becd7267fd71108c327c0f6"
  },
  "production": {
    "apiUrl": "https://hackersandslackers.app",
    "contentApiKey": "8a1becd7267fd71108c327c0f6"
  }
}

.ghost.json

Save this file and confirm that http://localhost:8000 now serves your content (if you left gatsby develop running, the site should hot reload for you). Changes are that your content isn't going to immediately look great. This is what my abomination looks like:

gatsby-starter-ghost sourcing Hackers and Slackers content.

The content coming from my Ghost admin looks awful in a default theme, which shoudn't surprise us. We're going to need to make some changes to this theme.

Anatomy of a Gatsby Site

Navigating a Gatsby theme for the first time is probably a bit overwhelming. There’s a lot of Gatsby-specific things we’ve never seen before (obviously), which might be challenging to dissect at first glance. Let’s see what we’ve got:

/my-gatsby-project
├── /src
│ ├── /components
│ ├── /pages
│ ├── /styles
│ ├── /templates
│ └── /utils
├── /static
│ ├── /images
│ └── /fonts
├── /public
├── /node_modules
├── package.json
├── package-lock.json
├── .ghost.json
├── netlify.toml
├── gatsby-node.js
├── gatsby-config.js
└── gatsby-browser.js

/my-gatsby-project

Gatsby's purpose is to take raw data from sources (like our Ghost admin), use that data to inform our site's structure, and finally transform our data to produce a site mostly comprised of static HTML and CSS. All of these static pages, styles, and assets live in the public folder. You should never need to work within this folder, as it's output will change with every build.

Building to the Public Folder

The simplicity of static sites enables them to have speedy load times. Static pages don’t need to think about which widgets or navigation items to display each time a user loads a page. They don’t need to rely on frontend JavaScript to manipulate pages. Most impressive of all, this particular breed of static site doesn’t need to wait before loading the pages you’ll probably click on next. Because every static page has a finite number of links to other static pages, Gatsby can load pages before you click on them.

We’re tossing the word “static” around a lot here, which sounds kind of like we're dealing with the types of shitty sites we made using Dreamweaver in the '90s. Those were the days where changing a single link meant changing that same link manually on 100 other pages. Perhaps you're a bit more modern and picturing a workflow more along the lines of Jekyll and GitHub pages. The default method of deploying a production Gatsby site is by using the gatsby build command, which generates a site comprised of unintelligent markup and styles. That said, most Gatsby developers will hardly need to deploy using gatsby build at all.

The "A" in JAMstack stands for APIs. By setting up webhooks in our Ghost admin, we can trigger a gatsby build job every time we update content in our CMS. Most static sites are hosted on services like Netlify, which continuously listen to for changes to our content via webhooks and rebuild our website accordingly. Setting up such a webhook in Ghost is as easy as expanding on the integration we created earlier. Here's what I use to automatically trigger builds to Netlify upon content updates in Ghost:

Ghost admin -> Integrations -> gatsby

The reality of GatsbyJS and other site generators in the JAMStack is that they're hardly "static" at all. Even though the pages we serve to user clients are technically "static," a simple webhook has our Gatsby theme rebuilding itself over and over, remaking the contents of the public folder from scratch each time.

Templates, Pages, and Components

So, the end result of gatsby build or gatsby develop is to generate the files which make up our site and dump them into the public folder. After sourcing our own content and seeing the ugly result, it's clear that we're going to make some changes to our page layouts. The first logical move would be to make changes to the presentation layer, which is contained entirely in the src folder.

Gatsby's src folder contains the logic for generating the HTML and stylesheets which ultimately make up the pages that get built. Each JavaScript file living in src is essentially a React component. Each of these components output JSX as a result of their own GraphQL queries (or data passed in from other components' GraphQL queries). Most of the time we spend customizing our theme will occur in the src folder.

Let's first concentrate on customizing a page template.

Templates

A template is a repeating page structure that will be used by multiple pages on our site. A perfect example of when to use a template would be blog posts. Blogs typically have thousands of "posts" in the sense of content, but each of these posts likely utilizes a single "post" template. All sites follow these types of patterns, thus our templates are going to determine the vast majority of what people see on our site.

Here's a simple example of what a GatsbyJS blog post template looks like:

import React from 'react'
import PropTypes from 'prop-types'
import { graphql } from 'gatsby'
import Helmet from 'react-helmet'
import { Layout } from '../components/common'
import { MetaData } from '../components/common/meta'

const Post = ({ data }) => {
  const post = data.ghostPost

  return (
      <>
        <MetaData
          data={data}
          title={post.title}
          description={post.excerpt}
          type="article"
        />
        <Layout>
          <div className="container">
            <article className="content">
              { post.feature_image &&
                <figure className="post-feature-image">
                  <img src={ post.feature_image } alt={ post.title } />
                </figure>
              }
              <section className="post-full-content">
                <h1 className="content-title">{post.title}</h1>
                <main
                  className="content-body load-external-scripts"
                  dangerouslySetInnerHTML={{ __html: post.html }}
                />
              </section>
            </article>
          </div>
        </Layout>
      </>
  )
}

Post.propTypes = {
  data: PropTypes.shape({
    ghostPost: PropTypes.shape({
      title: PropTypes.string.isRequired,
      html: PropTypes.string.isRequired,
      feature_image: PropTypes.string,
    }).isRequired,
  }).isRequired,
  location: PropTypes.object.isRequired,
}

export default Post

export const postQuery = graphql`
  query($slug: String!) {
    ghostPost(slug: { eq: $slug }) {
      title
      html
      feature_image
    }
  }

/src/templates/post.js

Templates are comprised of three parts:

GraphQL Data : At the bottom of our template, we have a GraphQL query named postQuery. This query speaks to the Ghost admin to grab post-specific information for the current page: the title, HTML, and feature_image. Running this query allows us to use this data in our template as part of the data object being passed to Post. If we wanted our post to include information like the name of the author, we'd have to add that field to our query as well.
PropTypes : We need to type-check the results of our GraphQL query before we can utilize this data in our page. We associate each item of data with the data type we're expecting by setting PropTypes.
Template Structure : Post is the JSX which will ultimately output each post page. This is essentially a React Component that is accepting a parameter called "data," which is the data we grabbed in our GraphQL query, postQuery. Take note of how we build our template in JSX and include the data we decided was important to include, such as { post.title } or { post.feature_image }.

Components

A component is a reusable block of code typically shared by multiple pages, such as widgets or navigation items (a better term for these would be "partials"). Partials are reusable code intended to be shared by pages and templates. For example, I have a component called AuthorCard, which details the information of a single author:

import React from 'react'
import PropTypes from 'prop-types'
import { Link } from 'gatsby'

const AuthorCard = ({ author, headerClass }) => {
  const authorTwitterUrl = author.twitter ? `https://twitter.com/${author.twitter.replace(/^@/, ``)}` : null
  const authorFacebookUrl = author.facebook ? `https://www.facebook.com/${author.facebook.replace(/^\//, ``)}` : null
  const classes = headerClass ? `author-card info-card` : `author-card`

  return (
      <>
        <header className={classes}>
          <div className="author-card-image">
            {author.profile_image ? <img className="lazyload" data-src={author.profile_image} alt={author.name} /> : <FontAwesomeIcon icon="user-edit" size="sm" /> }
          </div>
          <div className="author-card-content">
            <Link to={`/author/${author.slug}`} className="author-card-name">{author.name}</Link>
            <div className="author-card-meta">
              {author.postCount && <span className="author-card-item">{author.postCount} Posts</span>}
              {author.location && <span className="author-card-item">{author.location}</span>}
              {author.website && <a className="author-card-item" href={author.website} target="_blank" rel="noopener noreferrer">Website</a>}
              {authorTwitterUrl && <a className="author-card-item" href={ authorTwitterUrl } target="_blank" rel="noopener noreferrer">Twitter</a>}
              {authorFacebookUrl && <a className="author-card-item" href={ authorFacebookUrl } target="_blank" rel="noopener noreferrer">Facebook</a>}
            </div>
            {author.bio && <p className="author-card-bio">{author.bio}</p>}
          </div>
        </header>
      </>
  )
}

AuthorCard.propTypes = {
  author: PropTypes.shape({
    name: PropTypes.string.isRequired,
    bio: PropTypes.string.isRequired,
    profile_image: PropTypes.string,
    website: PropTypes.string,
    twitter: PropTypes.string,
    facebook: PropTypes.string,
    postCount: PropTypes.number,
    location: PropTypes.string,
    slug: PropTypes.string,
  }).isRequired,
  headerClass: PropTypes.boolean,
}

export default AuthorCard

/src/components/authors/AuthorCard.js

Components and GraphQL

Component files are structured in the same way as templates, with a fundamental difference: components cannot create dynamic GraphQL queries_._

The structure of AuthorCard has the same fundamental structure as our Post template, but AuthorCard does not have its own GraphQL query. AuthorCard instead is able to accept a parameter while being defined; this means whichever page/template contains this partial can simply pass data from the parent page's GraphQL queries into child components.

In the AuthorCard example, one of our input parameters is called author which seems to contain all the author-related data we need! To do this, we can import our author card into our post template and include it in Post's JSX:

import { AuthorCard } from '../components/authors'
...

const Post = ({ data }) => {
  const post = data.ghostPost
  const author = data.ghostAuthor

  return (
    ...
    <AuthorCard author={author} />
    ...
  )
}

/src/templates/post.js

author is looking for data.ghostAuthor, which we get by expanding on our post's GraphQL query:

export const postQuery = graphql`
  query($slug: String!, $primaryAuthor: String!) {
    ...
    ghostAuthor(slug: {eq: $primaryAuthor}) {
      postCount
      location
      facebook
      cover_image
      bio
      name
      slug
      twitter
      website
      profile_image
    }
    ....
  }

/src/templates/post.js

The thinking here is that templates should pass contextual data on to their child components. There is sanity in keeping our GraphQL queries on the templates that utilize them, as opposed to letting templates and their children pull data independently of one another.

AuthorCard now has contextual information about the author.

Static Queries in Components

There are cases where components can execute GraphQL queries, but only under the circumstance that they pull data which is not contextual. In other words, these components can only run GraphQL queries that do not utilize variables. These queries are called Static Queries. It's best not to linger on this topic, but here's an example of where a static query is used for site-wide metadata in our Ghost Gatsby template:

const MetaDataQuery = props => (
  <StaticQuery
    query={graphql`
      query GhostSettingsMetaData {
        allGhostSettings {
          edges {
            node {
              title
              description
            }
          }
        }
      }
    `}
    render={data => <MetaData settings={data} {...props} />}
  />
)

export default MetaDataQuery

/src/components/common/meta/MetaData.js

Pages

The third and final type of layout in GatsbyJS are pages, not to be confused with templates. Where templates are reusable, Gatsby pages are pages that will only ever exist once on our site, such as an error page or transactional confirmation. The syntax for creating a page is identical to that of creating a template.

Every page we create will inevitably require some standard information. No matter what our page is for, it's going to need a title, some metadata, and a URL (obviously). Ghost provides us with a number of GraphQL Fragments to help us grab all properties of a page (or post) at once:

export const pageQuery = graphql`
  query GhostPageQuery($slug: String) {
    ghostPage(slug: {eq: $slug}) {
      ...GhostPageFields
    }
  }
`

/src/templates/page.js

Gatsby Configuration & Plugins

Cruising through the src folder gives us a pretty good idea of how to modify the structure the pages our site will serve. That's great, but where does the data feeding these pages actually come from? How do our components know the data source we're querying? Without any data model configuration on our side, our components already recognize things like ghostPage and ghostPost as data types.

Sourcing data to Gatsby happens in a magic file called gatsby-config.js. Gatsby is configured by installing and tweaking an entire ecosystem of Gatsby plugins, and some of those plugins tell Gatsby where to look for our data. If you're familiar with Webpack, gatsby-config is essentially identical to a Webpack configuration file. A few examples of what our theme already includes:

gatsby-plugin-feed : Generates a highly-configurable RSS feed for our site.
gatsby-plugin-advanced-sitemap : Serves an SEO-friendly sitemap.
gatsby-plugin-react-helmet : Provides a JSX element to easily set metadata per page.

These are all fine and necessary, but the plugins we want to pay special attention to are the ones with the prefix "gatsby-source-". Our config has a few of these by default: gatsby-source-filesystem , and gatsby-source-ghost :

...
{
  resolve: `gatsby-source-filesystem`,
  options: {
    path: path.join(__dirname, `src`, `images`),
    name: `images`,
  },
},
{
  resolve: `gatsby-source-ghost`,
  options: process.env.NODE_ENV === `development` ?
    ghostConfig.development :
    ghostConfig.production,
},
...

gatsby-config.js

gatsby-source-filesystem is a plugin that sources content from our local file structure. In the above example, it's being used to serve images from a local folder. If we wanted to, we could source our entire Gatsby site from locally saved Markdown files. Luckily, we aren't the types of savage barbarians who blog by building Jekyll sites. We're gentlemen, just as the Great Gatsby himself.

gatsby-source-ghost allows us to query content from Ghost sites. Simply installing this plugin gives us access to Ghost data models in our GraphQL queries. In terms of knowing which Ghost admin to source from, this is what we handled when we configured .ghost.json.

As it turns out, sourcing content to Gatsby is perhaps one of it's most alluring features. Our configuration is already pulling from two content sources, and adding a third source would be as easy as installing a plugin. We're not just talking about multiple CMS sources; Gatsby allows us to source content from Github repositories, Tweets, JIRA, or even databases directly. Each "source" plugin we install gives us opportunities to create powerful associations between our data, joining information from different sources without ever touching a database.

Data Sources in Action

You've probably noticed the prompt Gatsby gives after running gatsby develop, which encourages you to explore your site's data schema at http://localhost:8000/___graphql. This GraphQL playground is your best friend: the easiest way to understand the resulting schemas of sources you configure is via this interface. Here's what my schema looks like after installing gatsby-source-git and *@gatsby-contrib/gatsby-transformer-ipynb * to pull and parse Jupyter notebooks from a Github repo:

A GraphQL query pulling .ipynb files from Github.

Adding two plugins is the only configuration needed to build this query. Here's what we just achieved with minimal effort:

Gatsby recognized that files were added to our local file structure and provided us with information about said files (such as file name, extension, etc.). We can query all of these attributes.
Of the local files Gatsby found, our newly added plugin identified .ipynb files as Jupyter Notebooks. This allows us to query Jupyter-specific attributes of those files, in addition to the general information we already had accessible.
Gatsby also recognizes that these Jupyter files were sourced from Github, so we can pull repository-level metadata about where these pages were sourced from.

This is absolutely insane. It's difficult to express how powerful this is in words, so I won't even try. Let's move on.

Building Routes in Gatsby

We now know how Gatsby sources its data, and how Gatsby eventually creates pages from that data. The third and final piece of our puzzle is between these two layers. This piece of our site handles the URL patterns and routing of the pages we create, and it all happens in gatsby-node.js.

Before our site can be built, we need to know how it'll be structured. Blogs in particular share a few common patterns. They usually have paginated lists of posts, author profiles, and "tag" pages where posts sharing a particular can all be viewed at once. We happen to be in luck because our Ghost starter template handles all of these things for us. As briefly as possible, the chain of events happening in gatsby-node is like this:

Content sources are queried for only the information necessary to build routes to our pages
The queried data is split into a single segment per template type. For example, we extract the names of all the posts we'll publish by setting const posts = result.data.allGhostPost.edges. The same is done for author pages, static pages, etc.
With our data grouped 1-to-1 with the pages they create, we then loop through each group to call a createPage function. Let's use posts as an example. In this step, we're telling Gatsby to create a page using the post.js template for each "post" we pull in GraphQL. A part of this process is passing the URL structure of where each of these generated pages will live.

There's a lot to take in here. Luckily for us, our template already handles the heavy-lifting of creating a site structure for us. When the time comes to add a new static page or grouping of templates, following the format of what already exists in gatsby-node.js is relatively straightforward.

Page Contexts in gatsby-node

If there's one pitfall of working through the gatsby-node file, it would be the concept of "page contexts". Let's look at the createPage function I have for creating posts:

createPage({
    path: node.url,
    component: postTemplate,
    context: {
    // Data passed to context is available
    // in page queries as GraphQL variables.
    slug: node.slug,
    primaryAuthor: node.primary_author.slug,
    primaryTag: node.primary,
    seriesSlug: node.series,
    seriesTitle: node.name
    },
  })

gatsby-node.js

The first two parameters of createPage are simple: path determines the route of this instance of a page, and component refers to whichever React component in src we want to build the page with.

context is where things get interesting. Variables passed into a page context can be accessed by the target template in two ways. The first way is as a variable in the template's GraphQL query (this is how we see things like query($slug: String!) ). Passing variables to pages is useful if a page contains features that depend on knowing more about which instance of a page it is. For example, I pass primaryTag to posts as a way of querying other posts with the same primary tag to build a related posts widget.

We're getting way too deep here. I won't even mention the pageContext object, which gets passed into templates for purposes of things like pagination. Let's move on.

What About Frontend Javascript?

Client-side JS should be a last resort when building static sites, but there are times when it needs to happen. gatsby-browser allows us to execute client-side Javascript in response to browser events like onRouteUpdate(), which is triggered each time a user changes pages. This is how we can implement code syntax highlighting, for example.

The full list of browser events we can use to trigger scripts can be found here.

Is Gatsby THAT Great Tho?

It's right to be skeptical of any new technology. This is especially the case with JavaScript frameworks, the hype cycle of which has become a meme in itself . After writing over 4 thousand words attempting to explain the internals of Gatsby at a high-level, it's clear that Gatsby is architecturally complicated. For newer devs who might not have previous experience with React, GraphQL, or Webpack, I can only imagine how one can feel at the bottom of the mountain looking up.

For more experienced developers, Gatsby is totally that great. Gatsby improves on so many aspects of modern web development that it's difficult to summarize why Gatsby is so great to those for whom it is suited. Praising "faster load times" doesn't do justice to the efficient, painless ecosystem of GatsbyJS. This is a rare moment where I'd argue that a framework lives up to the hype, at the very least.

That said, we need to acknowledge the implications that things like Gatsby creates for developers as a whole. For those of us who've grown up with Javascript's nuances and frameworks, learning Gatsby is a manageable step forward. It's easy to neglect that this is only true because we've accrued a lifetime of related knowledge before this point. This knowledge puts us in a favorable position to learn one more thing. If we were to imagine being on the outside looking in, it feels like Gatsby is another layer of "things to know" in the comical collection of nonsense that is Javascript. While I'm an advocate of Gatsby, it's important to recognize that learning Gatsby is a privilege of circumstance. Most employed developers work for enterprises that can not (nor ever should) consider major changes to their technology stacks. It's unreasonable to think "this is the direction the world is going," because most people in the world are preoccupied with making the world work. And families, or whatever.

Anyway, Gatsby is excellent if you're in any position to pick it up. Ask yourself, are you:

Working for a young startup that uses Node?
A student school and have a lot of time?
A self-destructive personality that causes them to stay up until 4am every night to learn new frameworks just to post about them?

If you answered "yes" to any of these, then picking up Gatsby is definitely worth your time.

Building Java Projects with Gradle

Todd Birchard — Tue, 09 Jun 2020 14:14:00 +0000

I've had a few strongly worded opinions about Java as a language in the past. Be that as it may, choosing a programming language is a luxury that many people don’t have; as long as enterprises exist, there will always be a need for Java developers. According to the 2019 StackOverflow developer survey, about 40% of developers are actively using Java in some way.

For those who work mostly in more "modern" programming languages, coming to Java in 2019 has numerous pain points. Installing dependencies by manually downloading and dropping jar files into your Java path is a humbling look into the past where package managers were non-existent. Luckily for us, there are tools like Gradle to help us bridge the gap between older programming languages and the processes we're used to.

Java Build Tools

Build tools are useful to Java developers in that they automate many processes associated with packaging a build which would otherwise be manual. Build tools not only compile a project's source code, but also download dependencies and run tests. Java build tools aren't exactly a sexy part of anybody's stack - it can be argued that they simply accomplish the things that we would expect a modern programming language to have natively.

Three build tools have historically dominated the scene: Apache Ant (released 2000), Apache Maven (released around 2004), and Gradle (released 2007). We won't waste time comparing these tools since Gradle is objectively better than the other options. If you're curious as to why, I'll humor you:

Ease-of-use : Gradle build scripts are written in Groovy (Ant and Maven are configured via XML files... enough said?)
Performance : Gradle creates builds faster by only building necessary changes from build-to-build, reusing build outputs from previous builds, and leveraging a running daemon process.
Debugging : Part of Gradle's build process is outputting very useful HTML logs detailing anything that went wrong.
Extensible : Gradle has a vast ecosystem of available plugins which open plenty of opportunities including support for Java, C++, or Python.

Installing Gradle

If you haven't done so already, go ahead and install Gradle using homebrew

$ brew install gradle

Installing Gradle via homebrew will automatically configure your path and all that nonsense. Verify that Gradle was installed correctly and you should be good to go:

$ gradle -v

------------------------------------------------------------
Gradle 5.5.1
------------------------------------------------------------

Initiating a Java Project with Gradle

Starting a Java project under normal circumstances is notably obnoxious when compared to other programming languages. Java's philosophy around namespaces results in ridiculously complex folder structures for any Java project, even a simple hello world app. Perhaps the most undersold feature of Gradle is the ability to generate cookie-cutter projects to get started.

Create a new directory using mkdir myProject (or whatever you want your project to be). cd into that empty directory and run Gradle's init script:

$ gradle init

Gradle is going to prompt us with a few questions to get more context about what we're creating. The first question Gradle asks us to select a type of project:

Select type of project to generate:
  1: basic
  2: application
  3: library
  4: Gradle plugin
Enter selection (default: basic) [1..4] 2

Creating a basic project only initiates a bare minimum project, which isn't particularly useful. Creating an application , on the other hand, will initiate a barebones folder structure for the language of your choice.

Select application (2) for your type of project. This should prompt three more multiple-choice questions:

Select implementation language:
  1: C++
  2: Groovy
  3: Java
  4: Kotlin
Enter selection (default: Java) [1..4] 3

Select build script DSL:
  1: Groovy
  2: Kotlin
Enter selection (default: Groovy) [1..2] 1

Select test framework:
  1: JUnit 4
  2: TestNG
  3: Spock
  4: JUnit Jupiter
Enter selection (default: JUnit 4) [1..4] 1

Project name (default: java-gradle-tutorial): 

Source package (default: java-gradle-tutorial):

Since we selected application as our project type, Gradle lets us chose from one of four programming languages to build our project (selecting basic wouldn't have prompted us with this).

Next, we're able to select either Groovy or Kontlin as the language in which our build scripts will be written in. Most people choose Groovy as it's an easy language to pick up, especially in this context.

Lastly, we're asked to select a test framework to run our unit tests in. JUnit 4 is the most popular, and it's good enough for me.

Check out our folder structure now:

/java-gradle-tutorial
├── build.gradle
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── settings.gradle
└── src
    ├── main
    │ ├── java
    │ │ └── com
    │ │ └── hackersandslackers
    │ │ └── gradletutorial
    │ │ └── App.java
    │ └── resources
    └── test
        ├── java
        │ └── com
        │ └── hackersandslackers
        │ └── gradletutorial
        │ └── AppTest.java
        └── resources

/java-gradle-tutorial

Gradle has started our project by generating the barebones of our project as well as some Gradle-related configuration files. Not only does Gradle create our main module within /src, but it also creates the corresponding folder structure needed to test said module. If we were to create this project without Gradle, we would've had to create 10 directories!

Let's take a look at the files related to using Gradle in our project:

build.gradle : This is the file we'll use to configure our builds. This is where we specify things like dependencies to download, tasks to run, projects to import, etc. This is the file we'll do the most work in.
settings.gradle : Additional settings for our Gradle build.
gradlew : gradlew stands for "Gradle wrapper" and is the file we'll use to execute our builds (for example: $ ./gradlew build). When we initialize Gradle in a project we actually decouple our project's Gradle from our system's Gradle, which means we could hand off our source to somebody who doesn't have Gradle installed and they'd still be able to build our project.

Core Tasks for Java Projects

Before we get into any customization/configuration stuff, let's see what Gradle offers us out of the box! Typing ./gradlew tasks in your project directory lists which tasks you can run in your project, such as building or running your code. Because we initialized Gradle as a Java project, we have this specific list of tasks:

$ ./gradlew tasks

------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------

Application tasks
-----------------
run - Runs this project as a JVM application

Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles main classes.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
testClasses - Assembles test classes.

Build Setup tasks
-----------------
init - Initializes a new Gradle build.
wrapper - Generates Gradle wrapper files.

Distribution tasks
------------------
assembleDist - Assembles the main distributions
distTar - Bundles the project as a distribution.
distZip - Bundles the project as a distribution.
installDist - Installs the project as a distribution as-is.

Documentation tasks
-------------------
javadoc - Generates Javadoc API documentation for the main source code.

Verification tasks
------------------
check - Runs all checks.
test - Runs the unit tests.

No matter what you're building, you'll be using a handful of these tasks all the time. Here are a few of the most commonly used tasks.

./gradlew build will compile your project's code into a /build folder.
./gradlew run will run the compiled code in your build folder.
./gradlew clean will purge that build folder.
./gradlew test will execute unit tests without building or running your code again.

Each of these tasks can be chained together for convenience. For example, ./gradlew clean build run will create a new build of your Java project from scratch and then run said project.

These are just the core tasks that come standard with Java applications; we can script our own tasks as well. For that, we'll need to get deeper into configuring Gradle.

Configuring & Customizing Gradle

Gradle is much more useful to us when we can configure it to do things like download dependencies, import other projects, and execute specific parts of our code at runtime. All of this magic is contained in a build.gradle file, which is usually made up of 3 major parts:

Tasks are scripts Gradle can execute. The core tasks available to Java applications will suit most needs.
Dependencies are third-party .jar files to fetch from a repository and package with your project.
Plugins are Gradle plugins for additional functionality (we should have the java and application plugins active by default).
Projects are standalone applications being packaged together in a single build. Not all Gradle builds consist of multiple projects, but multi-project builds are a big plus of Gradle.

Before we even jump into those, we need to tell Gradle where the main class of our application lives.

Setting a Main Class

The first thing we should set in build.gradle is a variable named mainClassName. This is the path to our main Java class relative to the current file. Each time our build runs, Gradle will look for this class in src/java to start our application. It's important to note that mainClassName excepts the package name to be part of the path to our class, so at the very least we need our mainClassName to contain [PACKAGE_NAME].[CLASS_NAME]. Here's my main class name:

mainClassName="com.hackersandslackers.gradletutorial.App"

So my class is named Main and lives in a package called com.hackersandslackers.gradletutorial. If you recall the folder structure that Gradle created for us, the inside of src reflects this:

/src
└── main
    └── java
        └── com
            └── hackersandslackers
                └── gradletutorial
                    └── App.java

/src directory

Lastly, make sure you set your package name in your Main.java file:

package com.hackersandslackers.gradletutorial

public class App {
    public String getGreeting() {
        return "Hello world.";
    }

    public static void main(String[] args) {
        System.out.println(new App().getGreeting());
    }
}

Main.java

Gradle Plugins

Gradle provides us with a few "core" plugins out-of-the-box to build Java applications. Unsurprisingly, the two we have are named "Java" and "Application":

plugins {
    id 'java'
    id 'application'
}

apply plugin:'application'

build.gradle

To understand what these two plugins do, try deleting them from your build.gradle file and running the same ./gradlew tasks command we ran earlier:

$ ./gradlew tasks

------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------

Build Setup tasks
-----------------
init - Initializes a new Gradle build.
wrapper - Generates Gradle wrapper files.

All our tasks are gone! The java and application plugins actually contain basically all the useful things we gained from initializing Gradle in the first place. As it turns out, Gradle is designed to be modular to the point where Gradle alone is nothing but a wrapper designed to serve as a medium for plugins and custom logic. You can add those plugins back now.

Installing Java Dependencies

If you've built Java projects before you're already aware of what a painfully manual process this normally is. Java has no inherent PyPi or npm equivalent: people used to download .jar files and manually place them into their project directory like a bunch of absolute savages. Gradle does a decent job of making this process easier by handling it in build.gradle.

First, we need to set the remote repository we want to download Java packages from. People usually use either mavenCentral() or jcenter(), it doesn't really matter:

...

repositories {
    jcenter()
}

build.gradle

With our repositories block created, we can then specify which dependencies to download. Below I specify that I'd like my builds to include a MySQL connector, and use the JUnit testing library:

...

repositories {
    jcenter()
}

dependencies {
    implementation 'com.google.guava:guava:26.0-jre'
    compile group: 'mysql', name: 'mysql-connector-java', version: '5.1.13'

    testImplementation 'junit:junit:4.12'
}

build.gradle

Gradle Tasks

A major part of build.gradle is scripting custom tasks. Tasks are snippets that we can run directly from the command line in our project directory ( as in ./gradlew [TASK_NAME] ). Here's a generic task that prints something:

task('hello') {
    doLast {
        println "hello"
    }
}

build.gradle

The value in parenthesis is the name of our task. If our task has no specified name, the task will run during every Gradle build by default.

doLast is a built-in action which means that the code in this block will be executed last. If we wanted an action to occur first, we could use doFirst instead.

With that knowledge, here's a task which will run during every build and print "hello" followed by "world":

task {
    doFirst {
        println "hello"
    }
    doLast {
        println "world"
    }
}

build.gradle

We can also add some helpful metadata to our tasks for whoever is using the command line and wants to interact with our project. Here we add a group to our task and add some helpful description text:

tasks.register("hello") {
    group = 'Worthless tasks'
    description = 'An utterly useless task'

    doLast {
        println 'hello'
    }
}

build.gradle

Now when we run ./gradlew tasks in our project directory, they'll be able to see the following:

$ gradle tasks

Worthless tasks
-------------
hello - An utterly useless task

Multi-project Builds

The last thing worth looking at in Gradle is packaging multiple projects at once. In these types of setups, our top-level directory would contain multiple projects within it:

├── project1/
├── project2/
├── build.gradle
└── settings.gradle

With two project directories, we can now import each project into our top-level settings.gradle file:

rootProject.name = 'java-gradle-tutorial'

include 'project1', 'project2'

settings.gradle

Nested projects in multi-project builds can each have their own standalone Gradle configurations with unique tasks and dependencies!

Deploy a Golang Web Application Behind Nginx

Todd Birchard — Mon, 01 Jun 2020 13:00:00 +0000

We started last week strong with a foray into Golang, where we created a simple web app serving a "Hello world" route. For those of you who were enticed by this deviation from our regular programming, the next logical question you might have could be how to make this knowledge "useful" by making it accessible by other human beings.

We're going to build on our Golang momentum from last week to do just that: deploy a web application written in Go to a Linux host such as Ubuntu. We're going to cover everything from installing Go, creating a systemctl service, and configuring Nginx. All you need is a VPS.

Installation

In case you haven't done so already, make sure your VPS has Nginx installed.

$ apt update
$ apt upgrade -y
$ apt install nginx

Install Nginx

Installing Go on Linux

We're going to install Go via source. Pick the version of Go that suits your Linux distro's needs from the Go downloads page. We'll download this to our /tmp folder, build the source, and move the built source to where it belongs:

$ cd /tmp
$ wget https://dl.google.com/go/go1.14.3.linux-amd64.tar.gz
$ tar -xvf go1.11.linux-amd64.tar.gz
$ sudo mv go /usr/local

Build Go from source.

We've just unpacked the Go language and moved it to where Linux typically likes to keep its programming languages. This path is what Go refers to as the GOROOT , and its contents should look like this:

/usr/local/go
├── AUTHORS
├── CONTRIBUTING.md
├── CONTRIBUTORS
├── LICENSE
├── PATENTS
├── README.md
├── SECURITY.md
├── VERSION
├── /api
├── /bin
├── /doc
├── favicon.ico
├── /lib
├── /misc
├── /pkg
├── robots.txt
├── /src
└── /test

Contents of GOROOT

Add GOPATH and GOROOT to your Shell Script

We've installed and unpacked Go, but we haven't given our OS a way to recognize that we've done so just yet. We can accomplish this by modifying our shell script, which is typically called .profile :

$ vim ~/.profile

Edit ~/.profile

Here we'll add our GOROOT and GOPATH file paths. If you'll recall, GOROOT is where our OS looks for the Go programming language, and GOPATH is the working directory where we keep all Go projects and dependencies. I've chosen to set my GOPATH to /go , which is a directory we have yet to create:

export GOPATH=/go
export GOROOT=/usr/local/go
export PATH=$PATH:$GOPATH
export PATH=$PATH:$GOROOT/bin

~/.profile

Save your changes and activate the shell script:

$ source ~/.profile

Activate ~/.profile

We can now verify that everything's been installed:

$ go version
>> go version go1.14.3 linux/amd64

Verify Installation

Setting up our GOPATH & Project

We have to create our GOPATH manually, which is as simple as creating the following directories:

$ mkdir ~/go
$ mkdir ~/go/bin
$ mkdir ~/go/pkg
$ mkdir ~/go/src

Now we have a place to keep our Go projects! I'm going to pull down the "Hello world" project we created last week for convenience sake:

$ cd /go
$ go get github.com/hackersandslackers/golang-helloworld

Fetch a go project

I've chosen to clone the Github repo into my /src path as well, which leaves the structure of my GOPATH looking like this:

/go
├── /bin
│   └── golang-helloworld
├── /pkg
└── /src
    └── /github.com
        └── /hackersandslackers
            └── /golang-helloworld
                ├── README.md
                ├── go.mod
                ├── go.sum
                ├── golang-helloworld
                ├── main.go
                └── main_test.go

Contents of GOPATH

Create an Nginx Config

You may have done this a few times before, but whatever. We're going to set up an Ngnix reverse proxy to listen on the port our app will be running on, which happens to be port 9100 in our case. Of course, we need to make sure this port is enabled first:

$ sudo ufw allow 9100

Open a port

Create a configuration file in the Nginx /sites-available folder:

$ sudo vim /etc/nginx/sites-available/golang-helloworld.conf

Create Nginx config

We're going to drop the standard boilerplate for a reverse proxy here. The domain I happen to be using for this app is golanghelloworld.hackersandslackers.app. Replace this with the domain of your choice:

server {
   listen 80;
   listen [::]:80;

   server_name    golanghelloworld.hackersandslackers.app;

   location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:9100;
    }

    location ~ /.well-known {
        allow all;
    }

}

golang-helloworld.conf

We activate this configuration by creating a sym link from our file in sites-available to sites-enabled :

$ sudo ln -s /etc/nginx/sites-available/golang-helloworld.conf /etc/nginx/sites-enabled/golang-helloworld.conf

Symlink Nginx config

Finally, these changes are applied upon Nginx restart. If the below produces no output, you're in the clear:

$ sudo service nginx restart

Restart Nginx

SSL With Certbot

Adding SSL is arguably out of scope for what the point of this tutorial is, but whatever. Certbot makes adding SSL easy enough that we can blow through it in less than a minute.

Before we can actually install Certbot, we need to add the proper repositories:

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository universe
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update

Add Certbot PPA repository

Now we can install Certbot for real:

$ sudo apt-get install certbot python3-certbot-nginx

Install Certbot

The Certbot CLI is able to accept an --nginx flag, which scans the configurations we've set up on our machine:

$ certbot --nginx

Create certs based on your Nginx configs.

This will list every Nginx configuration you have on your machine and prompt you for which app you'd like to set up with SSL. My Ubuntu machine happens to host a bunch of sites. Feel free to check out any of them if you please :

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
1: broiestbro.com
2: www.broiestbro.com
3: consider.pizza
4: stockholm.ghostthemes.io
5: hackersandslackers.app
6: hackersandslackers.tools
7: django.hackersandslackers.app
8: flaskblueprints.hackersandslackers.app
9: flasklogin.hackersandslackers.app
10: flasksession.hackersandslackers.app
11: flasksqlalchemy.hackersandslackers.app
12: flaskwtf.hackersandslackers.app
13: plotlydashflask.hackersandslackers.app
14: www.hackersandslackers.tools
15: hustlers.club
16: www.hustlers.club
17: toddbirchard.app
18: golanghelloworld.hackersandslackers.app
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):

Certbot CLI

The configuration I'm looking for is #18. Selecting this will then prompt whether or not we'd like to redirect HTTP traffic to HTTPS, which is something you should definitely do (is there even a reason not to do this? Let me know in the COMMENTS BELOW and remember to smash that LIKE button).

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Certbot CLI

Select option 2.

Check out how Certbot has modified our original golang-helloworld.conf Nginx config:

server {

   server_name    golanghelloworld.hackersandslackers.app;

   location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:9100;
    }

    location ~ /.well-known {
        allow all;
    }

    listen [::]:443 ssl;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/golanghelloworld.hackersandslackers.app/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/golanghelloworld.hackersandslackers.app/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

}

server {

    if ($host = golanghelloworld.hackersandslackers.app) {
        return 301 https://$host$request_uri;
    }

   listen 80;
   listen [::]:80;

   server_name    golanghelloworld.hackersandslackers.app;
    return 404;

}

golang-helloworld.conf

That's what we like to see.

Create a Systemctl Service

If you've never messed around with Systemctl services before, a "service" is something we want to run continuously on our server (for example, Nginx is a service in itself). We're going to create a service for our Go app to make sure our app is always running, even if our server is restarted:

$ vim /etc/systemd/system/golanghelloworld.service

Add a service

The syntax for Linux services follows the .ini file format. There's a lot here which we'll dissect in a moment:

[Unit]
Description=golanghelloworld.hackersandslackers.app
ConditionPathExists=/go/src/github.com/hackersandslackers/golang-helloworld
After=network.target

[Service]
Type=simple
User=root
Group=root

WorkingDirectory=/go/src/github.com/hackersandslackers/golang-helloworld
ExecStart=/go/src/github.com/hackersandslackers/golang-helloworld/golang-helloworld

Restart=on-failure
RestartSec=10

ExecStartPre=/bin/mkdir -p /var/log/golang-helloworld
ExecStartPre=/bin/chown syslog:adm /var/log/golang-helloworld
ExecStartPre=/bin/chmod 775 /go/src/github.com/hackersandslackers/golang-helloworld/golang-helloworld

[Install]
WantedBy=multi-user.target

golanghelloworld.service

Here are the notable values being set above:

User & Group: Probably not the best idea in the word, but this tells our service to run our app as the Ubuntu root user. Feel free to change this to a different Linux user.
WorkingDirectory: The working directory that we'll be serving our app from.
ExecStart: The compiled binary file of our Go project.
Restart & RestartSec: These values are exceptionally useful for ensuring that our app is always up and available, even after crashing from unforeseen circumstances. These two values are telling our service to check if our app is running every 10 seconds. If the app happens to be down, our service will restart the app, hence the on-failure value for Restart.
ExecStartPre: Each of these lines contains a command to run prior to starting our app. We set three such commands: the first two ensure that our app logs correctly to a directory called /var/log/golang-helloworld. The third (and more important) command sets permissions on our Go binary file.

Save your service file. Below we register the changes we've made to our system's services, start our new service, and enable our service to be run upon system startup:

$ systemctl daemon-reload
$ service golanghelloworld start
$ service golanghelloworld enable

Start service

Let's check to see how things went:

$ service golanghelloworld status

Check service status

If you happen to be very lucky, you'll see a SUCCESS output like the following:

golanghelloworld.service - golanghelloworld.hackersandslackers.app
   Loaded: loaded (/etc/systemd/system/golanghelloworld.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-05-29 01:57:02 UTC; 4s ago
  Process: 21454 ExecStartPre=/bin/chmod 775 /go/src/github.com/hackersandslackers/golang-helloworld/golang-helloworld (code=exited, status=0/SUCCESS)
  Process: 21449 ExecStartPre=/bin/chmod 775 /var/log/golang-helloworld (code=exited, status=0/SUCCESS)
  Process: 21445 ExecStartPre=/bin/chown syslog:adm /var/log/golang-helloworld (code=exited, status=0/SUCCESS)
  Process: 21444 ExecStartPre=/bin/mkdir -p /var/log/golang-helloworld (code=exited, status=0/SUCCESS)
 Main PID: 21455 (golang-hellowor)
    Tasks: 6 (limit: 4915)
   CGroup: /system.slice/golanghelloworld.service
           └─21455 /go/src/github.com/hackersandslackers/golang-helloworld/golang-helloworld

Service status

Debugging Systemctl Services

The unfortunate truth about creating Linux services is that there are a lot of moving parts at play; I don't think I've ever gotten a new service to work on the first attempt without some sort of error. These errors range from permission errors, incorrect file paths, or port clashing. The good news is that each of these problems are individually simple to fix.

Your first line of defense to debugging services is journalctl to check the log output of any service:

$ journalctl -u golanghelloworld.service

Check logs for a service

An indispensable tool for debugging issues is the ability to grep for processes to see if they're running properly. We can search active processes by name or by port. If journalctl outputs an error that port 9100 is in use, you can find that process via the below:

$ ps aux | grep 9100

Search for process by port number

We can kill whichever process comes back with kill -9 [PID].

Alternatively, we can check if our app is running by grepping by process name:

$ ps aux | grep golang-helloworld

Search for process by name

If needed, we could kill the process with pkill -9 golang-helloworld.

If you need to make changes to your .service file, remember to run systemctl daemon-reload to pick up the changes, and service golanghelloworld restart to give it another go.

Get Out There

I have faith you'll work out the kinks and successfully get your Go app up and running. It took me a bit of time myself, but my shitty hello world is up and live in all its glory here.

If you run into issues that you can't seem to solve, feel free to reach out. We'll work it out together.

Create Your First Golang App

Todd Birchard — Tue, 26 May 2020 05:50:00 +0000

To be human is to be an unwilling passenger in a winding, aimless journey we call life. Each of us has felt the eternal solidarity of time break apart as we are thrust into existence to navigate the tribulations of existing, left only to wonder what the point of it all is. Just as we become complacent in our respective existential struggles, an event of unspeakable force shakes the foundations of our reality: we fall in love.

Falling in love is as exhausting as it is enchanting. Our lifespans only have the willing capacity to fall in love a finite number of times, if at all. This realization is responsible for making us wary of falling in love in the first place, as well as zealously defending the love we've found against potential intruders. I'm one to advocate for an opposite conclusion. As difficult, scary, or time-consuming as love may be, I argue that a life that has discovered love on multiple occasions is a life well-lived. If that makes me a slut, so be it: I am a slut for programming languages.

I've proudly maintained a long-term loving commitment to Python, as well as a complicated affair with JavaScript. Still, even this promiscuous lifestyle can leave one yearning for a kind of love that only a statically-typed language can deliver. As our relationships transition from scripting honeymoons to mature enterprise-level endeavors, it's natural to second-guess our choices. Why endure the overhead of a dynamically typed language when we end up annotating types with MyPy? Could there be validity in arguments that claim our language of choice is "slow?" And do we really think we can put up with baggage known as the GIL till death do us part?

If this all seems like a roundabout way to announce that I've been fooling around with Golang, that's because it absolutely is. I don't suspect that many people are willing or capable of leaving their comfort zones to partake in a journey of this magnitude. For the rest of you, I'd like to welcome you by my side for a short moment in our lives to explore the unexplored. Who knows, perhaps you'll even find love along the way.

Installation and Setup

Installing on OSX is simple thanks to Homebrew:

$ brew update
$ brew install golang

Install Golang via Homebrew

GOPATH vs GOROOT

Installing Golang via Homebrew automatically generates two directories critical to running Go:

GOROOT ( /usr/local/go ): The Go "root" directory contains Go's source code. Homebrew will automatically register this path for you; there's little reason to mess around in here unless you're a Go contributor or if you're attempting to run multiple versions of Go.
GOPATH ( /Users/toddbirchard/go ): Unlike most programming languages, Go takes an opinionated stance that all projects and dependencies of the language should exist in a single directory known as the GOPATH. Any time we develop a Go project or install a third-party module, the actions taken ultimately happen inside this directory.

To make sure OSX recognizes our GOPATH , we'll have to add it to our shell's startup script. Open your .bashrc , .zshrc , or whatever it is you use:

$ vim ~/.zshrc

Modify shell script

We're going to add the /go base directory, as well as the subdirectory /go/bin :

export GOPATH=/Users/toddbirchard/go
export PATH=$PATH:$GOPATH
export PATH=$PATH:$GOPATH/bin

Add GOPATH to your PATH

Save this and reload your shell script:

. ~/.zshrc

Active changes

Let's make sure everything went as planned:

$ go version
>> go version go1.14.2 darwin/amd64

Verify Installation

And as a last bit of due diligence, let's confirm that our GOPATH is being recognized correctly:

$ go env GOPATH
>> /Users/toddbirchard/go

Verify GOPATH

As an aside, go is Golang's CLI which is essential for compiling, formatting, and running Go code, as well as installing modules (we'll get to those in a sec). Try running go help to get acquainted.

Anatomy of GOPATH

Golang's GOPATH is a directory where all your Go code and project dependencies live. The go CLI actually has a built-in command $ go help gopath which explains this quite well:

The Go path is used to resolve import statements.

It is implemented by and documented in the go/build package.

The GOPATH environment variable lists places to look for Go code.

Similar to how Python looks for imported libraries in the Python path, Go searches the GOPATH for the same. A notable difference between Python and Go paths is that Go expects all your Go projects to live within the GOPATH, specifically /go/src. Contrast this with Python where projects can live anywhere.

The GOPATH directory is made up of 3 subdirectories:

/go
├── /bin
├── /pkg
└── /src

Structure of our GOPATH.

$ go help gopath explains the purpose of each of these directories.

src:

The src directory holds source code. The path below src determines the import path or executable name.

pkg:

The pkg directory holds installed package objects.

As in the Go tree, each target operating system and architecture pair has its own subdirectory of pkg (pkg/GOOS_GOARCH).

bin:

The bin directory holds compiled commands.

Each command is named for its source directory, but only the final element, not the entire path.

In short, your personal source code belongs in /src , installed third-party packages will live in /pkg , and third-party commands which extend the go CLI will live in /bin. To give an example, here's what my path looks like:

/go
├── /bin
│   ├── golint
│   └── tour
├── /pkg
│   ├── /darwin_amd64
│   │   ├── github.com
│   │   ├── go-pandas.a
│   │   ├── golangwebsite
│   │   └── hustlers
│   ├── /mod
│   │   ├── /cache
│   │   ├── /cloud.google.com
│   │   │   └── go@v0.57.0
│   │   ├── /github.com
│   │   │   ├── /google
│   │   │   │   └── go-cmp@v0.4.0
│   │   │   ├── /gorilla
│   │   │   │   └── mux@v1.7.4
│   │   │   ├── /mattn
│   │   │   │   └── go-runewidth@v0.0.7
│   │   │   ├── /olekukonko
│   │   │   │   └── tablewriter@v0.0.4
│   │   │   └── /rocketlaunchr
│   │   │       └── dataframe-go@v0.0.0-20200520082355-50e589cfde42
│   │   └── golang.org
│   └── /sumdb
│       └── sum.golang.org
└── /src
    ├── /golang-helloworld
    │   ├── README.md
    │   ├── go.mod
    │   ├── go.sum
    │   ├── golang-helloworld
    │   └── main.go
    ├── /golang.org
    │   └── x
    └── /hustlers
        ├── README.md
        ├── go.mod
        ├── go.sum
        ├── hustlers
        ├── main.go
        ├── main_test.go
        ├── static
        ├── templates
        └── vendor

My GOPATH.

We should be able to break this down quite easily.

/bin contains two Go commands I installed previously. golint is a third-party linter for Go, and "tour" is a local version of the official Tour of Go walkthrough to help Go newcomers learn their way around the language (I highly recommend completing this, btw).
/pkg contains the packages I've installed. Pay special attention to /pkg/mod, where you can see I've installed several packages from Github under the_/github.com_ directory. These packages include dataframe-go, which is a Go implementation of Pandas-like DataFrames, as well as mux, which is an HTTP router that we're going to use in our first hello-world project.
/src has three projects I've worked on already. golang-helloworld is the project we're about to create in this tutorial.

Golang Terminology

Before we get to coding, let's brush up on some basic Go vocabulary:

Packages: Go programs are made up of "packages," which mirror packaging concepts in other programming languages (think modules in Python or packages in Java). Every Golang program contains a package called main, which serves as the project's entry point.
Modules: Go modules are third-party libraries installed by Go. Modules are essentially projects which have been published for general use as dependencies in your projects.
Vendors: This is where things get interesting. While modules can be installed to the /pkg/mod directory for global use, source projects can contain their own versions of these modules to avoid clashing dependency versions between projects (this is not dissimilar to Python virtual environments). While not required, you can choose to keep module versions project-specific (we will do this in our example).

Creating a Hello World App

Enough chit-chat, let's make our first Go project. We start with creating our project's directory in the /go/src directory:

$ cd $GOPATH/src
$ mkdir golang-helloworld
$ cd golang-helloworld

While inside our new project directory, we're now going to initialize our project as a Go module. This means anybody will be able to install our Go code off Github if they so choose. I know I'm going to save my repo to github.com/hackersandslackers/golang-helloworld, so we run the following:

$ go mod init github.com/hackersandslackers/golang-helloworld
>> go: creating new go.mod: module github.com/hackersandslackers/golang-helloworld

Creating a Go module.

The moment this is done, a new file will appear in your directory called go.mod. Check out the contents using $ cat go.mod to see what this initializes with:

module github.com/hackersandslackers/golang-helloworld

go 1.14

go.mod

Pretty simple stuff so far! go.mod contains information about our module for others, such as the module name and Go version it is intended for. As we install dependencies for our project, these dependencies and their respective versions will be stored here.

main.go

As mentioned, every Go project's entry point is a file called main.go. We're going to create the most simple main.go file imaginable: a script which outputs "Hello, world.":

package main

import "fmt"

func main() {
    fmt.Println("Hello, world.")
}

main.go

Now remember: since Go is a compiled language, we need to build our project before we can run it. I know the extra effort is nearly unbearable, but things are about to pay off as you witness the fruits of your labor:

$ go build
$ go run main.go
>> Hello, world.

Build and run your module.

WE DID IT! We've just created our first "hello world" app in Go. Your project structure should now look something like this:

/golang-helloworld
├── go.mod
├── golang-helloworld
└── main.go

Our project so far.

The newly created golang-helloworld file is the compiled executable which is created each time we run $ go build. Each time we make changes to our source code, we should run $ go build again to rebuild this executable with our changes.

Bonus: Code Formatting

A nifty tool that comes out-of-the-box in Go is a code formatter to clean up any ugly indents and such in your source. Try messing up the indents in main.go and run the following:

$ go fmt
>> main.go

Formatting source code.

This should fix all the ugly formatting in the file names in outputs, in our case main.go.

Create a Web App

If we were to leave off with a stupid program that prints "Hello, world!", I'd be doing you a disservice. While we've set up Golang successfully, we haven't learned much about creating anything useful yet. It's time for us to kick things up a notch by making our app a web app which can be served from a browser.

Installing our First Dependency

To serve Go code via a web server, we're going to leverage the highly popular gorilla/mux module: a lightweight request router and dispatcher for matching incoming requests to their respective handler.

We're going to install this by running $ go get followed by $ go install:

$ go get -u github.com/gorilla/mux
$ go install github.com/gorilla/mux

Install a dependency.

go get installs the source for gorilla/mux to our /go/bin directory. The -u flag we pass is an "update" flag, which we use to grab the latest version just in case.

Let's see how go.mod was affected by running $ cat go.mod:

module github.com/hackersandslackers/golang-helloworld

go 1.14

require github.com/gorilla/mux v1.7.4

go.mod

As promised, our module dependency has now been added to go.mod along with the proper version number. Now we can import and use "github.com/gorilla/mux" to help us build a project!

We can also use $ go mod vendor to build this dependency in our /vendors folder to keep it local to our project.

Here's a wall of code which turns our hello world app into a web app:

package main

import (
    "github.com/gorilla/mux"
    "log"
    "net/http"
    "time"
    "io"
)

func handler(w http.ResponseWriter, r *http.Request) {
    io.WriteString(w, "Hello, world!\n")
}

// Route declaration
func router() *mux.Router {
    r := mux.NewRouter()
    r.HandleFunc("/", handler)
    return r
}

// Initiate web server
func main() {
    router := router()
    srv := &http.Server{
        Handler: router,
        Addr:    "127.0.0.1:9100",
        WriteTimeout: 15 * time.Second,
        ReadTimeout:  15 * time.Second,
    }

    log.Fatal(srv.ListenAndServe())
}

main.go

Our functions are main(), router(), and handler(), which get executed in that order.

main()

main() sets up an HTTP server to be served locally on port 9100 , with a couple read & write timeouts set as a form of best practice. Our server doesn't do much on its own without any routes to resolve. That's where our router() function comes in.

router()

We initialize a "router" by created variable r with r := mux.NewRouter(). From there we can set as many routes as we'd like with the following syntax:

r.HandleFunc([URL_ROUTE], [FUNCTION_TO_EXECUTE])

Setting a route with mux.

HandleFunc() is a built-in method to resolve URL routes. The first parameter is the target URL, and the second is the name of a function to be executed when a user requests said route. We only specify a single route in our example, but we could theoretically set as many as we'd like, for example:

r := mux.NewRouter()
r.HandleFunc("/", homeHandler)
r.HandleFunc("/about", aboutHandler)
r.HandleFunc("/contact", contactHandler)
return r

Example of setting multiple routes.

handler()

Mux handler functions always accept two parameters by default, which essentially resolve to output and input. w http.ResponseWriter expects a parameter named w with the type http.ResponseWriter, which is what we return to render something for the end-user. r *http.Request contains information about the user's request, saved to a parameter named r.

We're keeping things simple(ish) today, so we'll settle for our route to simple output a "hello world" message for our route:

io.WriteString(w, "Hello, world!\n")

Output a string.

Rebuild and run our project with $ go build and $ go run main.go. Now try visiting 127.0.0.1:9100 in your browser:

Our live Go app.

And there you have it, lovebirds.

Best Practices

Before I leave to give you and your new favorite Gopher some private time, there's some very low-hanging fruit worth picking in an intro tutorial. This won't last long.

"Exported" Functions (AKA: Public Versus Private)

Nearly every programming language has the concept of "private" versus "public" functions. Go has this concept as well regarding shared functions between packages. Functions that are "shared" are referred to as "exported functions" (sup JavaScript).

A name is exported if it begins with a capital letter. Our hello world example consisted solely of private functions (which makes sense, as we only had a single package). If we wanted to make our router() function accessible by other packages, we'd simply need to rename this to Router().

Type Declaration

Go expects that variables , incoming function parameters , and function return values to have declared types. In the below example, the function add() accepts two integers and adds them, which undoubtedly results in an integer:

func add(x int, y int) int {
    return x + y
}

Setting types for incoming function parameters.

Variables are set using the same syntax as function parameters, with the variable name coming first:

var x int = 0

Setting a single integer variable

There's also a shorthand way of setting multiple variables of the same type by separating variable names by commas. In this case, variables x, y, and z are set as integers with no assigned values:

var x, y, z int

Creating 3 variables each sharing the int type.

Short Assignment Statements

A very cool feature of Go is the := operator. The "short assignment" operator can be used to set multiple variables at once with implicit types. That means Go will resolve the type of each variable on its own based on the value assigned without the need for explicit type declaration. The below example creates three variables, where x and y are resolved as booleans, and z is resolved as a string:

func main() {
    x, y, z := true, false, "no!"
}

Implicitly set variable types with the := operator.

Constants

The last noteworthy nugget is the presence of constants in Go. While there's nothing unique about Go supporting constants, it's a breath of fresh air for Pythonistas who may be nostalgic about having the ability to do the following:

const Website = "hackersandslackers.com"

Happily Ever After?

Whether or not you hop on the Go train is a question of where your heart lies. While I'll continue using Python for the majority of what I do, it's nice to leave the Mrs. at home once in a while (inb4 this misogynistic analogy ruins me) to fool around building quick endpoints in a statically-typed language which isn't Java. Have I ever mentioned how much I hate Oracle? Like, how much I really hate them? No? Perhaps another time.

Anyway, get on with it then. The repository for what we've created today is up on Github here:

https://github.com/hackersandslackers/golang-helloworld