DEV Community: Mario Vega

Web Scraping with Puppeteer + Laravel Sail

Mario Vega — Wed, 22 Jun 2022 00:00:00 +0000

In my experience, the one predictable aspect of working with freelance clients is that assumptions are dangerous things to make.

To be clear, communication is crucial for any development work, freelance or not. Even in a job setting where you can expect technical literacy from your coworkers, clearly outlining expectations at the beginning of a new project can help differentiate you from the common programmer’s tendency to code first and ask questions later.

However, going over project requirements with a new freelance client warrants an extra degree of caution, for two reasons:

In the overwhelming majority of cases, the client knows nothing about development. This isn’t a bad thing, obviously — you wouldn’t have been hired otherwise! But, this does mean you have an additional responsibility to fully understand what you’re building, as there’s nobody to tell you if your technical assumptions are wrong.
The client knows their business better than you ever will. While this is, in many ways, a tremendous asset to you, it’s important to know that expertise has its own pitfalls. Namely, things may be obvious to your client that you, as someone with little experience in their field, wouldn’t guess in a million years. Because it’s obvious to them, they may not stop to think the same isn’t obvious to you.

Such a case happened to me recently, when a local sticker shop approached me about building a calculator to help them price out custom orders. On its face, the job seemed simple; they already had a price calculator that they had been using for just this purpose. At first, the only problem seemed to be that some of the numbers this calculator was producing weren’t anywhere close to correct. There were some configurations where buying 1,000,000 stickers was the same price as buying a 1,000 — a great deal for customers, but a potentially ruinous one for my client’s business.

Straightforward enough. Access the code, look into the database, find the error and fix it.

That’s when I asked about where the existing calculator was hosted. Apparently, this calculator had been built by a business associate who later became a competitor. This competitor had left the original calculator up as a favor to my client, but didn’t feel particularly inclined to give us access to either the code or the numbers behind it.

As you might imagine, this complicated matters somewhat. Not only did I need to build a new calculator from scratch, I also needed to somehow get all the relevant numbers from the original calculator, as well as figure out where these numbers went wrong.

Two of these processes — fixing the numbers and building the new calculator — were either too straightforward or too business-related to include in this dev post. Instead, this article focuses on the part that posed the most

Scraping in Sail

Before I could build a new calculator, I needed to get the numbers from the existing calculator. This calculator was publicly accessible, so I didn’t need to hack any servers, but getting the numbers involved actually using the calculator. As I needed to cover every height and width from 1 inch to 24 inches in increments of 0.25 inches, as well as every quantity from 50 to 10,000, the programmer’s tendency towards laziness would have never allowed me to derive these numbers by hand.

I needed to find a way to automate this process, and instinctively thought to use a web automation framework. Web automation, although primarily used for testing with Selenium as the most popular example, can also be used to navigate websites programmatically. If I could get such a framework to punch the relevant numbers into the calculator and return the results to my backend, that’d save me the hours it would have taken to use the calculator myself.

From previous jobs, both salaried and freelance, I have some experience automating websites with Node’s Puppeteer library, so I decided to use that. I believe it would have been possible to accomplish the same task using Laravel’s existing support for Selenium, including the Selenium instance that comes with Laravel Sail by default. However, hacking around Laravel’s testing-focused infrastructure to have Selenium act as a web scraping tool felt a bit too brittle for my liking. Furthermore, as I knew that at some point I’d need to evaluate JavaScript on the headless browser to read values from the calculator’s HTML, I figured it would be more efficient to write the entire scraper script in JavaScript, rather than the JavaScript-within-PHP approach that working within the Laravel context would have required.

This brought up my next issue — getting Puppeteer’s headless browser to work within the Docker image provided by Laravel Sail. The first step, as one might imagine, was to install Puppeteer: sail npm install puppeteer. Just to test things out, I wrote the most basic scraper script imaginable and ran that to see if everything was working as expected.

Reader, everything was not working as expected.

The first error I encountered referred to a browser not existing. I ended up having to go into node_modules/puppeteer and running npm install in that directory to install the necessary browser.

The second error read something along the lines of “libnss3 no such file”. This one was a bit more complicated, and I ended up following most of the instructions on this StackOverflow answer. From the answer, it seemed that I needed to install some dependencies into my Docker image to get things working as expected. However, as I didn’t have access to the Docker image, I first needed to publish Sail’s Docker config with sail artisan sail:publish. This allowed me to add a line to the Dockerfile below the existing image setup to add all the dependencies necessary for Puppeteer to work.

The third error was what truly threw a wrench in my plans. Even though my browser was now running as expected, I couldn’t get my scraper to start because the browser I had installed did not come with a sandbox. After doing a bit of research into what browser sandboxing was, I learned that I could circumvent this error by adding a --no-sandbox flag to the method launching Puppeteer in my script.

At last, my script was running! But, as mentioned, the sandbox error wasn’t in the rearview quite yet. Not running a sandbox in my browser meant that I couldn’t trust this scraper to run on any site I didn’t 100% trust. If you’ll remember, I didn’t have access to the code behind the existing calculator. Even though this site had been working for me without issue thus far, I couldn’t guarantee that it’d remain this way going forward.

Running a sandboxed browser within the Laravel image seemed fairly involved, if it was possible at all. Most answers I saw on the issue said simply to leave the --no-sandbox flag enabled, which I had already decided wasn’t a viable option in this case.

It was here that I decided to alter my approach. Until now, I had been trying to get the headless browser working in the same service as my Laravel application. What if I ran the headless browser in its own service instead? Within the docker-compose.yml that comes with Sail, there are multiple services included by default, all of which can connect to each other via a shared internal network. If I could get the headless browser running on this network as its own service, I could overcome two flaws of my current approach:

Using a dedicated service for my browser would almost certainly mean I could run my browser with the proper sandbox; and
I could remove some of the extraneous setup I had added to the Laravel Dockerfile, thus reducing the chances of those dependencies causing hiccups down the line.

It didn’t take long for me to find exactly what I needed: browserless/chrome. Adding this to my Dockerfile was even simpler than I had imagined:

chrome:
        image: 'browserless/chrome'
        ports: 
            - '${BROWSERLESS_CHROME_PORT:-3000}:3000'
        networks:
            - sail
        environment:
            - CONNECTION_TIMEOUT=-1

The only change I made that doesn’t follow the same pattern as the other services included with Sail by default is the CONNECTION_TIMEOUT=-1. This line was necessary to get my script working as intended because, by default, any connections to headless/chrome time out after 30 seconds, and my script ran… quite a bit longer than that.

Using a neat trick of docker-compose services, where any service in a network is accessible from any other service using that service’s name, connecting to my new headless/chrome instance was as simple as running:

const browser = await puppeteer.connect({
        browserWSEndpoint: "ws://chrome:3000",
});

Much better! Not only did this browser run with a sandbox, I could also return the included Dockerfile to its original state.

With the proper browser setup configured, writing the script itself was fairly straightforward. My scraping function ended up looking like this:

const getPrice = async (height, width, quantity, variant) => {
        await page.evaluate((_variant) => {
            document.documentElement
                .querySelector(`[value="${_variant}"]`)
                .click();
        }, variant);

        await page.evaluate(() => {
            document.documentElement
                .querySelector("input#product-custom-size")
                .click();
        });

        await page.select("select#cus_width", String(height));
        await page.select("select#cus_height", String(width));

        await page.evaluate(() => {
            document.documentElement
                .querySelector("input#product-custom-qty")
                .click();
        });

        await page.evaluate(() => {
            document.documentElement.querySelector("#Quantity").value = "";
        });

        await page.type("#Quantity", String(quantity));

        const price = await page.evaluate(
            () => document.documentElement.querySelector("#price").innerText
        );

        return {
            width,
            height,
            quantity,
            variant,
            price,
        };
    };

Basically, this script:

clicks on the select variant;
clicks on the “custom dimensions” and “custom quantity” buttons;
resets whatever values currently existed in these inputs (from previous iterations);
inputs the selected height, width and quantity; and
reads and saves the resultant price.

This script worked, with one major issue — at over 150,000 total entries, the resulting array was just too darn big. Trying to post this to my backend as-is resulted in in 413 - Payload Too Large error. To solve this, I decided to do two things:

Create two data structures: PriceSnapshot and PriceMeasurement. A PriceSnapshot represented one total run of the scraper, while a PriceMeasurement represented one measurement captured during the snapshot.
Begin the script by creating one PriceSnapshot, then including this PriceSnapshot with a new request to a PriceMeasurement POST route whenever I had 10 or more PriceMeasurements in the queue.

The whole script looked something like this:

require("dotenv").config();

const axios = require("axios").default;
const puppeteer = require("puppeteer");

let allPrices = [];
const CALCULATOR_URL = process.env.CALCULATOR_URL;

const VARIANTS = [
    "Gloss Laminated",
    "Gloss-Vinyl",
    "Clear Laminated",
    "Chrome",
];

(async () => {
    const { data: snapshotID } = await axios.post(
        "http://pricecalculator.test/api/price-snapshots",
        {
            url: CALCULATOR_URL,
        }
    );

    const browser = await puppeteer.connect({
        browserWSEndpoint: "ws://chrome:3000",
    });
    const page = await browser.newPage();
    await page.goto(CALCULATOR_URL);
    await page.waitForSelector("input#product-custom-size");

    const getPrice = async (height, width, quantity, variant) => {
        await page.evaluate((_variant) => {
            document.documentElement
                .querySelector(`[value="${_variant}"]`)
                .click();
        }, variant);

        await page.evaluate(() => {
            document.documentElement
                .querySelector("input#product-custom-size")
                .click();
        });

        await page.select("select#cus_width", String(height));
        await page.select("select#cus_height", String(width));

        await page.evaluate(() => {
            document.documentElement
                .querySelector("input#product-custom-qty")
                .click();
        });

        await page.evaluate(() => {
            document.documentElement.querySelector("#Quantity").value = "";
        });

        await page.type("#Quantity", String(quantity));

        const price = await page.evaluate(
            () => document.documentElement.querySelector("#price").innerText
        );

        return {
            width,
            height,
            quantity,
            variant,
            price,
        };
    };

    for (let variantIndex = 0; variantIndex < VARIANTS.length; variantIndex++) {
        for (
            let measurement = 1;
            measurement <= 24;
            measurement = measurement + 0.25
        ) {
            for (
                let quantity = 100;
                quantity <= 20000;
                quantity = quantity + 50
            ) {
                allPrices.push(
                    await getPrice(
                        measurement,
                        measurement,
                        quantity,
                        VARIANTS[variantIndex]
                    )
                );

                if (allPrices.length >= 10) {
                    await axios.post(
                        "http://pricecalculator.test/api/price-measurements",
                        {
                            prices: allPrices,
                            snapshotID,
                        }
                    );

                    allPrices = [];
                }
            }
        }
    }

    await browser.close();
})();

Notice how I used a similar trick to connect to my Laravel backend as I did to connect to the headless browser. Because I renamed my Laravel service from laravel.test to pricecalculator.test, I could connect to my app from my scraper by accessing pricecalculator.test.

With one VERY IMPORTANT CAVEAT! I wish this were documented in the Laravel docs, but for some reason my PR adding this detail was closed, so I’ll include it here.

If you change the service name in your docker-compose.yml file, as I did by changing to pricecalculator.test, you MUST also add a corresponding variable to your .env file:

APP_SERVICE=pricecalculator.test

And that’s it! If there’s interest, I can write a follow-up post about the Livewire integration, but that part was far more by-the-book. Hope this helps anyone who’s looking to include some custom services to their Laravel Sail setups.

I'm making an online course and I need your help! What platforms should I cover?

Mario Vega — Sun, 28 Jul 2019 20:45:41 +0000

Hello all!

Over the last couple of weeks, I've been teaching my brother how to program. This has been an interesting and powerful experience for me, as I've had the chance to reflect on how far I've come and how far technology has come since I started.

By this point, my brother has reached a stage of development that was, for me at least, profoundly disheartening. Like I did years ago, my brother has barrelled through Codeacademy's HTML, CSS & JavaScript classes, and he's started learning JavaScript in greater depth. Both he and I would say that he has a strong grasp of the building blocks of web technology. He knows his way around an HTML document, he can modify and add CSS classes as necessary, and he knows enough about JavaScript to target and change a given DOM element at will.

However, when it comes to applying those skills to an actual project, in many ways he hasn't learned a thing -- and that's not his fault. I distinctly remember how overwhelming it was to start poking around my first WordPress themes and plugins after I had taught myself the basics. There were variables and code snippets floating around everywhere. Even if I knew exactly what I wanted to do, finding the best place to implement my changes often took far longer than actually writing the code itself. Plugins and themes I would look at to try and improve my code seemed to pull the variables and functions they needed out of a magic hat. I had no clue whether the logic I was seeing belonged to PHP, WordPress, or the theme I was using, and I had no clue how to find out.

I knew that I had a solid understanding of the basics. I just didn't know why none of those skills seemed to help me when it came to actually getting things done. Without the platform-specific knowledge and without a broader understanding of how web pages were made and what functionality was available to me at any given moment, I felt powerless to actually implement what I wanted.

In my case, I overcame that barrier with stubborn persistence and lots of trial-and-error. I was fortunate in that, in addition to being a fairly quick learner, I had lots of experience with computers and a decent knack for Googling the right things to get the right answers. But watching my brother work through these same problems drove home for me how intimidating this process is to those starting out their dev hobbies and careers.

I know that I'll be able to walk my brother through these challenges. But what about everyone else? How many people give up on development because they couldn't apply the skills they learned to the platforms they were building with? I know I was in danger of this happening to me a couple times, and I had lots of innate advantages. What could I do to help create opportunities for those who don't share those advantages?

Soon, the answer became clear; I would make a class of my own. This class would be designed with one student in mind: a student like my brother, a student who knows the basics but doesn't know how to apply them yet. This class would walk through some of the most popular web platforms available and show how each platform uses the building blocks of the web -- HTML, CSS and JavaScript -- to create websites while simplifying the user experience of website owners.

So, my question to you is: what platforms would you like to see included in this class? At this point, I know for sure that I'm going to include sections on WordPress and Shopify, as those are the two most popular web platforms I know of that allow for developer extensions. I also know that I'm going to cover React in some way, shape or form, as it's a ubiquitous UI platform that is, in my experience, very intimidating for newbies.

But beyond that, what else would you like such a class to cover? Are there any platforms you use often but don't quite understand, or any major platforms that allow developer extensions I haven't thought of? Any concepts that are/were particularly confusing to you?

I've so far considered including sections on Twitch, Squarespace, and a static site builder like Jekyll or Gatsby. But ultimately I want this class to be useful to new developers, so I'm open to covering any and all platforms that devs might use as they begin to apply their skills to real-world scenarios.

Thanks so much for your help! I'll be keeping everyone posted on the class as it's being developed. So stick around -- you might even get some free stuff out of it :)

Snake Saga - Building a Game with Redux Generators

Mario Vega — Wed, 24 Jul 2019 04:33:43 +0000

In the process of interviewing for front-end jobs, I've taken to building shorter-term projects that I can complete in the space of a weekend, both to practice writing accessible HTML/CSS and to discover new features of JavaScript. One of the more interesting projects I took on recently involved building a game of Snake in React — and not only because it was the first "video game" I've built.

This project was particularly interesting to me because it introduced one of the most unique JavaScript features I've come across in the last year: generators. Specifically, because I needed to take action each time the snake moved, I did some research on the major side-effect libraries available in the Redux ecosystem.

My thinking was that the snake's movement was the "main event", and all potential actions arising out of its movement (eating fruit, losing the game, etc.) should be handled immediately after arriving at each new square. My strategy, then, was to write the post-movement logic into side effects that had access to all game information and could either update the game, stop it altogether, or allow it to continue if no noteworthy events had occurred.

I've used Redux Thunk in the past, and I believe I could have written my side effect logic in Redux Thunk without too many headaches. However, because the point of these side projects was to learn new skills, and because the generator model of Redux Saga seemed to offer more flexibility if I could overcome the initial learning curve, the library seemed a natural choice.

Plus, who doesn't like the idea of their code going on a saga? Picture tiny snakes sailing on a longboat with Viking hats and tell me that doesn't bring a smile to your face.

Wait, scratch that. Through writing the idea, I realize now that seafaring snakes are terrifying.

Before diving into things, if you just want to see the code, check out the project repo here: https://github.com/mariowhowrites/react-snake. Keep in mind that this was a weekend project and not a production assignment. Notably, there are some performance and styling issues I'd clean up if ever I were to ship this game — not to mention some tests I'd need to write.

Generators: A Quick Overview

The most straightforward explanation for generators that I've seen is that they're functions that your program can start and stop at will. Calling a normal function typically gives you no control over how and when the function runs. Your program simply runs the function and rolls with it until it either returns a value or throws an error. If the function triggers an infinite loop, your program is stuck waiting for an exit like the poor passengers of Mr Bones' Wild Ride (safe for work, Rollercoaster Tycoon content).

Generators work differently by giving execution control to the program itself. Put another way, think of generators as loops that your program can increment on its own time. Given the following code:

// the '*' marks this function as a generator
function* loopSayings() {
  yield "hello"
  yield "goodbye"
}

Callling loopSayings() for the first time would start the generator. In order to work with it further, you'd want to save the generator as a variable, such as const loopGenerator = loopSayings().

From there, your program can control the generator by calling loopGenerator.next(). Each time the method next() is called, the generator will advance to the following yield statement in the function.

Whenever a yield statement is encountered, the generator stops executing and returns an object with two properties:

value will return whatever's to the right of the yield statement where the generator stopped
done is a boolean value indicating whether the generator has reached the final yield statement or not. Further calls to next() after this point will give a value of undefined.

Therefore, after starting the generator for the first time, loopGenerator.next().value would return 'hello'. Calling loopGenerator.next().value again would return the value 'goodbye', at which point the done property would be true and all future next() calls would return undefined values.

Putting this all together, sample usage of a generator could look like this:

function* loopSayings() {
  yield "hello"
  yield "goodbye"
}

const loopGenerator = loopSayings() // starts the generator
console.log(loopGenerator.next().value) // 'hello'
console.log(loopGenerator.next().value) // 'goodbye'
console.log(loopGenerator.next().value) // undefined, generator has finished

Generators in Redux Saga

So now that we've got a basic understanding of how generators work, let's see how this pattern is applied within the Redux Saga library. Let's start from the obvious: Redux Saga is a library built on top of the Redux state management library, which itself is the most popular tool to manage complex state in React applications.

Specifically, Redux Saga works primarily as Redux middleware. For the uninitiated, middleware is essentially a fancy term for any logic that works in the middle of a given process.

For example, if we were building a web server, we could write middleware that determines whether a given user can access a specific resource. This middleware would happen in the middle of the request, after the user has made the request and before our server begins fetching the resource. If the user isn't able to access the given resource — if they're not logged in, for example, or if they're asking to access protected data that belongs to another user — this middleware can stop the request immediately, saving your application from potentially exposing sensitive information.

Applying this model to Redux, all middleware is run after receiving a request to update state, but before your reducers have actually updated to reflect the new state. This gives middleware the ability to change incoming state requests before they hit your reducers, offering a powerful method of customizing your Redux logic based on external events.

In the case of Redux Saga, because the library primarily deals with side effects, we won't be altering state requests directly. However, Redux Saga takes full advantage of the fact that middleware can see not only incoming actions, but also the present state of your reducers at the time that the action is dispatched. In the case of our Snake game, this setup allows us to combine the current board state with the action being dispatched to figure out what action should be taken.

Put another way, in my experience Redux Saga provides an excellent parallel to listeners or observers in other languages and frameworks. They observe external events and potentially trigger new actions in response to observed events.

Sagas in Practice

So far, this description has been pretty abstract — let's ground it with some actual Snake code. In my Snake implementation, I've set up the board as a square grid of blocks. In my Redux library, I keep track of which blocks represent walls, fruit, open spaces, and the snake itself. Once per tick, the snake moves forward one block and the new position is dispatched as a Redux action.

In my case, I wrote four different sagas to listen to various events occurring across the game board:

import { all } from "redux-saga/effects"

import watchPosition from "./watchPosition"
import watchFruitCollection from "./watchFruitCollection"
import { watchGameStart, watchGameEnd } from "./watchGameChange"

export default function* rootSaga() {
  yield all([
    watchPosition(),
    watchFruitCollection(),
    watchGameStart(),
    watchGameEnd(),
  ])
}

The all() method accepts a group of sagas and combines them into one middleware, which is attached to the Redux store shortly before loading the main application.

Let's look at the fruit collection saga, which fires whenever a new fruit has been collected:

import { takeEvery, put, select } from "redux-saga/effects"

import * as types from "../store/types"
import { makeFruit } from "../utils"

export default function* watchFruitCollection() {
  yield takeEvery(types.FRUIT_COLLECT, handleFruitCollection)
}

function* handleFruitCollection({ payload }) {
  const size = yield select(state => state.game.size)
  yield put({ type: types.FRUIT_ADD, payload: [makeFruit(size)] })
  yield put({ type: types.ADD_SCORE })
}

Notice that the saga itself contains only one line of code, starting with the takeEvery call. This function tells Redux Saga to "take" every action with the type FRUIT_COLLECT and pass the action to the handleFruitCollection method.

From there, because we know that the action is of type FRUIT_COLLECT, we know the snake has just collected a new fruit and we can dispatch actions accordingly. Namely, there are two actions that should be taken when a new fruit is collected:

The player score needs to be incremented by one.
A new fruit needs to be added to the game board.

To add a new fruit to the board, we first need to know how big our game board is so that we don't accidentally add a fruit where it shouldn't be — namely, in or beyond a wall. To get the board size, we first use the select function provided by Redux Saga to pull the size property from our game reducer. From there, we dispatch a new action FRUIT_ADD using a new fruit created by makeFruit, which returns a new fruit at a random valid position on the game board.

With that accomplished, the only thing left to do is to increment the current score. Instead of handling the state change within the saga, we dispatch a new action with type ADD_SCORE, which our game reducer will catch and use to update the player's score.

There are two important processes going on here:

All state modifications are relegated to reducers instead of being handled directly within the saga. This is an intentional design pattern — Redux Sagas are supposed to be side effects, not secondary reducers.
Our handler generator is not being called directly. Instead, the Redux Saga middleware is responsible for invoking our generators, which it does by walking through each saga until the done property from the generator returns true.

Why Use Generators At All?

Because the generator process is handled in a synchronous manner by our Redux Saga middleware, you might be wondering why generators are used in this case at all. Wouldn't it be faster and more direct to include all of our state update logic within the reducer itself? What's to stop us from incrementing the player score and adding a new fruit within the COLLECT_FRUIT reducer and skipping Redux Saga entirely?

Whether or not Redux Saga is a good idea for your application is mostly a matter of scale. For a simpler project, it may have made sense to write out all of our Redux data mutations within the reducer function itself. However, more complex applications often require more separation between cause and effect than you could get by grouping all of your logic in the same file. By separating all of the "side effects" of an update from the update itself, we can keep our reducers straightforward and add additional side effects without changing our reducer code and opening ourselves to state-related bugs.

For a better example of this, let's look at the watchPosition saga in the Snake app:

export default function* watchPosition() {
  yield takeEvery(types.CHANGE_POSITION, handlePositionChange)
}

const getState = state => ({
  fruitPositions: state.fruit.fruitPositions,
  snakeQueue: state.snake.snakeQueue,
  snake: state.snake.snake,
})

function* handlePositionChange({ payload: newPosition }) {
  const { fruitPositions, snakeQueue, snake } = yield select(getState)

  const gameIsOver = collidedWithSelf(snake, newPosition)

  if (gameIsOver) {
    yield put({ type: types.GAME_END })
    return
  }

  const fruitToRemove = findFruitToRemove(fruitPositions, newPosition)

  if (fruitToRemove >= 0) {
    yield put({ type: types.FRUIT_COLLECT, payload: fruitToRemove })
    yield put({ type: types.SNAKE_QUEUE, payload: newPosition })
  }

  if (snakeQueue.length >= 1) {
    yield put({ type: types.SNAKE_GROW })
  }
}

We see that watchPosition has a nearly identical structure to watchFruitCollection above. All actions of type CHANGE_POSITION are taken on a new saga led by the handlePositionChange generator.

From there, however, a more complex series of events takes place. Using helper methods, this generator checks on various game conditions, such as whether the snake has collided with itself or collected a fruit.

Would it make sense to handle the fruit collection logic within the position reducer? To me, the answer is no. By delegating all of the effect work to sagas, each of my reducer cases maxes out at around five lines of code. I can add as much functionality into this watchPosition generator as I want without needing to change the basic mechanics of how my snake moves across the board. And because put and select return simple JavaScript objects, all of this code can be easily tested by starting and iterating our sagas manually, much like we did with loopSayings in the intro to generators.

How My Coffee Maker Taught Me to Love Types

Mario Vega — Wed, 17 Jul 2019 20:11:59 +0000

TypeScript.

That one word had hung around the edges of my consciousness for months. VS Code would suggest the .ts extension for every new file I created, and every time my answer would be, "Not today." Looking up the implementations of JavaScript functions would bring me to TypeScript type definition files, which always looked a tad too intimidating to parse. I'd read quotes about the growing inevitability of typed JavaScript and think that that inevitability didn't apply to me. When had I ever run into a problem that typing would solve?

Little did I know that the evangelist that would convert me to the typed way would be my own coffee maker.

Like many work-inclined people I know, coffee fuels a large part of my working day. And like many developers I know, I follow my creative inspirations wherever they might take me -- often into the small hours of the night. At the end of my working hours, I typically set a pot of coffee on autobrew for the following morning so I can start my next day without missing a beat.

One day a couple weeks ago, I woke up to a calamity: coffee grounds everywhere. Coffee grounds in my coffee, coffee grounds on the counter, coffee grounds trailing down my cabinet doors and pooling on the kitchen floor below.

As it turns out, in my early-morning code-addled stupor, I had forgotten to put the filter back into my coffee after cleaning it. It was in that moment of cleaning coffee grounds that I realized why I had to start typing my JavaScript immediately.

Effectively, I realized that my coffee maker was like vanilla JavaScript. It was happy to let my delirious, inattentive self set my coffee up however I wanted. There were no checks to make sure that I had put in a new filter, no checks that I had put in new coffee grounds, no checks that I had put in water, no checks whatsoever to ensure that I would actually have coffee in the morning and not a waterfall of wasted grounds. Any problems that resulted from my coffee setup would have to be handled when it was time to brew, not while setting up and actually had the time and (some of) the attention span to fix things.

If my coffee machine had been strictly typed? Sure, maybe I would have been annoyed that my machine was bleeping at me to fix my improper setup when the only thing I wanted to do was go to sleep. But I would have fixed things all the same, and my diligence would have been rewarded with morning coffee.

There are few things more important to me than coffee, and properly-working code is one of them. So, while the jury's still out on when we'll see strong typing make its way into home appliances, I decided right then and there that I would start my journey into TypeScript.

Logging In With Laravel

Mario Vega — Wed, 05 Jun 2019 19:39:18 +0000

Welcome to the much-belated Part 2 of my Laravel Authentication series! As promised, in this chapter we'll be diving more into the nuts and bolts of the authentication process. We'll also look at the login process in an attempt to better understand how the authentication process works within a Laravel application.

If you haven't yet read Part 1 of the Authentication with Laravel series, you can find it on Medium here.

I'm going to start with the code that's added to a Laravel application once the developer uses the make:auth command. From there, we'll work our way from the top on down, looking at the code behind a typical login request before examining the code that makes such a login workflow possible.

Scaffolding Auth

First, let's look at code powering the make:auth command:

// Illuminate\Auth\Console\AuthMakeCommand

/**
* Execute the console command.
*
* @return void
*/
public function handle()
{
    $this->createDirectories();

    $this->exportViews();

    if (! $this->option('views')) {
        file_put_contents(
            app_path('Http/Controllers/HomeController.php'),
            $this->compileControllerStub()
        );

        file_put_contents(
            base_path('routes/web.php'),
            file_get_contents(__DIR__.'/stubs/make/routes.stub'),
            FILE_APPEND
        );
    }

    $this->info('Authentication scaffolding generated successfully.');
}

The important part for our purposes is the routes appended to our web.php file via file_put_contents. This is where Laravel adds the routes powering the default authentication experience.

As it turns out, the appended stub is only two lines long:

Auth::routes();

Route::get('/home', 'HomeController@index')->name('home');

In an additional convenience, the routes method is the only method defined directly on the Auth facade. Examining that method shows that the bulk of our authentication logic is contained in the Router class:

public static function routes(array $options = [])
{
    static::$app->make('router')->auth($options);
}

The corresponding Router function accepts an array of options specifying whether registration, reset and verify routes should be included. Using those options, the router adds a number of routes to our application:

// Illuminate\Routing\Router

/**
    * Register the typical authentication routes for an application.
    *
    * @param  array  $options
    * @return void
    */
public function auth(array $options = [])
{
    // Authentication Routes...
    $this->get('login', 'Auth\LoginController@showLoginForm')->name('login');
    $this->post('login', 'Auth\LoginController@login');
    $this->post('logout', 'Auth\LoginController@logout')->name('logout');

    // Registration Routes...
    if ($options['register'] ?? true) {
        $this->get('register', 'Auth\RegisterController@showRegistrationForm')->name('register');
        $this->post('register', 'Auth\RegisterController@register');
    }

    // Password Reset Routes...
    if ($options['reset'] ?? true) {
        $this->resetPassword();
    }

    // Email Verification Routes...
    if ($options['verify'] ?? false) {
        $this->emailVerification();
    }
}

Because we're trying to follow the journey of a typical login request, we'll be focused only on the routes that interface with Laravel once a login request has been sent. In particular, we are concerned with the POST login route, which redirects to the LoginController@login method.

Looking at the Login controller, we don't see much:

class LoginController extends Controller
{
    /*
    |--------------------------------------------------------------------------
    | Login Controller
    |--------------------------------------------------------------------------
    |
    | This controller handles authenticating users for the application and
    | redirecting them to your home screen. The controller uses a trait
    | to conveniently provide its functionality to your applications.
    |
    */

    use AuthenticatesUsers;

    /**
    * Where to redirect users after login.
    *
    * @var string
    */
    protected $redirectTo = '/home';

    /**
    * Create a new controller instance.
    *
    * @return void
    */
    public function __construct()
    {
        $this->middleware('guest')->except('logout');
    }
}

However, notice that there's an AuthenticatesUsers trait, which means any methods available in the trait will also be accessible in this controller. Let's look there and see if we can find a login method:

// Illuminate\Foundation\Auth\AuthenticatesUsers

/**
    * Handle a login request to the application.
    *
    * @param  \Illuminate\Http\Request  $request
    * @return \Illuminate\Http\RedirectResponse|\Illuminate\Http\Response|\Illuminate\Http\JsonResponse
    *
    * @throws \Illuminate\Validation\ValidationException
    */
public function login(Request $request)
{
    $this->validateLogin($request);

    // If the class is using the ThrottlesLogins trait, we can automatically throttle
    // the login attempts for this application. We'll key this by the username and
    // the IP address of the client making these requests into this application.
    if ($this->hasTooManyLoginAttempts($request)) {
        $this->fireLockoutEvent($request);

        return $this->sendLockoutResponse($request);
    }

    if ($this->attemptLogin($request)) {
        return $this->sendLoginResponse($request);
    }

    // If the login attempt was unsuccessful we will increment the number of attempts
    // to login and redirect the user back to the login form. Of course, when this
    // user surpasses their maximum number of attempts they will get locked out.
    $this->incrementLoginAttempts($request);

    return $this->sendFailedLoginResponse($request);
}

Fortunately, as with much of the Laravel source code, the top-level API utilizes descriptive method names to resemble the plain-language description of the workflow as closely as possible.

First, the login request is validated. We need to make sure that a username and password are included, and that both are strings:

protected function validateLogin(Request $request)
{
    $request->validate([
        $this->username() => 'required|string',
        'password' => 'required|string',
    ]);
}

From there, we move onto login throttling. As specified in the authentication docs, login throttling prevents a user from spamming your server with login requests if they've failed to login multiple times.

Here is the hasTooManyLoginAttempts method:

protected function hasTooManyLoginAttempts(Request $request)
{
    return $this->limiter()->tooManyAttempts(
        $this->throttleKey($request), $this->maxAttempts()
    );
}

We see that the brunt of the throttling logic is delegated to the class returned by limiter(), which is an instance of Laravel's Cache/RateLimiter. Before looking at that, let's inspect the two methods we can find in ThrottlesLogins: throttleKey and maxAttempts.

protected function throttleKey(Request $request)
{
    return Str::lower($request->input($this->username())).'|'.$request->ip();
}

We see that a throttle key is created by combining the username provided in the login request (by default, this is the user's email) with the IP address of the device from which the user is logging in.

Next, maxAttempts():

public function maxAttempts()
{
    return property_exists($this, 'maxAttempts') ? $this->maxAttempts : 5;
}

This one's fairly self-explanatory: unless the developer has defined a custom maxAttempts property, a default value of 5 is used. Keep in mind that you would define this property on the LoginController within your application, and not on the ThrottlesLogins trait itself. As the LoginController uses both the ThrottlesLogins and AuthenticatesUsers traits, any properties available on the controller are also available to its traits.

With this in mind, let's look at the tooManyAttempts method in Cache/RateLimiter:

public function tooManyAttempts($key, $maxAttempts)
{
    if ($this->attempts($key) >= $maxAttempts) {
        if ($this->cache->has($key.':timer')) {
            return true;
        }

        $this->resetAttempts($key);
    }

    return false;
}

We see that the RateLimiter class makes extensive use of Laravel's native cache functionality. It uses the throttleKey defined above to check the cache and see whether the user has logged in too many times. If they have, and there's still a timer on that particular key, tooManyAttempts returns true and the user is denied entry. Otherwise, if there's no timer on the user's throttle key, this means that their throttling period has expired. In this case, the RateLimiter resets the user's login attempts to give them a clean slate and avoid throttling based on stale attempts.

Of course, if true is returned from the tooManyAttempts method, the user isn't allowed to login, and Laravel responds accordingly. First, a new Lockout event is fired using Laravel's event system. After that, the user is sent a lockout response, specifying the number of seconds until another attempt is permitted:

protected function sendLockoutResponse(Request $request)
{
    $seconds = $this->limiter()->availableIn(
        $this->throttleKey($request)
    );

    throw ValidationException::withMessages([
        $this->username() => [Lang::get('auth.throttle', ['seconds' => $seconds])],
    ])->status(429);
}

Defensive Programming

By this point, having confirmed that the user has supplied login information and is not currently locked out, the authentication processes we've studied in the first chapter come into play. This logic is encapsulated at this point in the LoginController@login method:

if ($this->attemptLogin($request)) {
    return $this->sendLoginResponse($request);
}

Notice how the "happy path" is encapsulated within its own conditional logic: a good example of defensive programming.

In plain English, here's what the above means. The "happy path" for any piece of code is what happens when everything goes as planned. It's the golden brick road, the mythical destination where we as developers realize there's no place like a good user experience.

In our case, the "happy path" is a successful login: the user provides the correct credentials and is allowed access to our application. The sendLoginResponse method contains the necessary actions for the happy path to occur. It is thus important that this method is found only within a conditional. If, and only if, the return value of attemptLogin() is truthy, then our users can follow the happy path.

Let's look at this code with a bit more context:

if ($this->attemptLogin($request)) {
    return $this->sendLoginResponse($request);
}

$this->incrementLoginAttempts($request);

return $this->sendFailedLoginResponse($request);

If the users follow the happy path, the incrementLoginAttempts and sendFailedLoginResponse methods are never called, as execution stops on the sendLoginResponse method instead.

Therefore, if the user has reached the point where incrementLoginAttempts is being called, then we know that the login attempt has not been successful.

The crux of this strategy, and of defensive programming in general, is the assumption of failure. The happy path is guarded with a conditional, and defensive programming assumes that this condition returns falsy. The main benefit of this approach is that it's better to accidentally deny a successful login than to accidentally allow a fraudulent login.

There is little downside to accidentally denying valid logins: your users can always try again. By contrast, allowing illegitimate access because your program assumed the happy path can result in your website's accounts being compromised.

In this way, defensive programming is a lot like defensive driving: assuming the worst and planning accordingly is your best chance at success, no matter the outcome.

Attempting a Login

With that aside covered, let's look at the actual login attempt code:

protected function attemptLogin(Request $request)
{
    return $this->guard()->attempt(
        $this->credentials($request),
        $request->filled('remember')
    );
}

To perform our authentication check, we use the credentials provided by the request and the credentials() method:

protected function credentials(Request $request)
{
    return $request->only($this->username(), 'password');
}

By default, username() returns email, which means that the user's email and password are used in the auth check.

Here is the first major difference between the login auth process and the code we examined in the first chapter: while we were previously examining the TokenGuard, the default guard for login attempts is the SessionGuard. Let's take a look at the SessionGuard@attempt method:

public function attempt(array $credentials = [], $remember = false)
{
    $this->fireAttemptEvent($credentials, $remember);

    $this->lastAttempted = $user = $this->provider->retrieveByCredentials($credentials);

    if ($this->hasValidCredentials($user, $credentials)) {
        $this->login($user, $remember);

        return true;
    }

    $this->fireFailedEvent($user, $credentials);

    return false;
}

Notice defensive programming at work: only if the user has valid credentials are they logged in. Otherwise, a failed login event is fired and the user is denied access. As before, by assuming failure as the default, this method reduces the possibility of false positives and protects user data as a result.

Besides that, there are three major functions to look at here: retrieveByCredentials, hasValidCredentials, and login.

retrieveByCredentials is attached to the guard's provider. As we learned in the last chapter, providers are responsible for defining how your application defines a "user" within the context of authentication. Laravel provides two providers out of the box. Both attempt to fetch a user from your database, but they differ on how to create a user from a database row:

The eloquent driver casts your user into an Eloquent model.
The database driver casts your user into a GenericUser model, which is a simple class wrapper around a database row containing only the necessary methods for authentication.

The eloquent driver is default, so we'll look at that first:

// within Illuminate\Auth\EloquentUserProvider...

public function retrieveByCredentials(array $credentials)
{
    if (empty($credentials) ||
        (count($credentials) === 1 &&
        array_key_exists('password', $credentials))) {
        return;
    }

    // First we will add each credential element to the query as a where clause.
    // Then we can execute the query and, if we found a user, return it in a
    // Eloquent User "model" that will be utilized by the Guard instances.
    $query = $this->newModelQuery();

    foreach ($credentials as $key => $value) {
        if (Str::contains($key, 'password')) {
            continue;
        }

        if (is_array($value) || $value instanceof Arrayable) {
            $query->whereIn($key, $value);
        } else {
            $query->where($key, $value);
        }
    }

    return $query->first();
}

First, we ensure that we have enough information to fetch a user. As a user cannot be fetched by an unhashed password alone, the method returns null if only a password is provided.

Next, a new Eloquent query is created using whatever model you define alongside the eloquent driver. In the vast majority of cases, this will be the default User model that comes with all Laravel applications.

From there, for each value provided in the credentials array, a new WHERE clause is added to the query. The first database entry that matches the resulting query is returned as a User model.

Let's compare this against the retrieveByCredentials method in the DatabaseUserProvider:

// In Illuminate\Auth\DatabaseUserProvider

public function retrieveByCredentials(array $credentials)
{
    if (empty($credentials) ||
        (count($credentials) === 1 &&
        array_key_exists('password', $credentials))) {
        return;
    }

    // First we will add each credential element to the query as a where clause.
    // Then we can execute the query and, if we found a user, return it in a
    // generic "user" object that will be utilized by the Guard instances.
    $query = $this->conn->table($this->table);

    foreach ($credentials as $key => $value) {
        if (Str::contains($key, 'password')) {
            continue;
        }

        if (is_array($value) || $value instanceof Arrayable) {
            $query->whereIn($key, $value);
        } else {
            $query->where($key, $value);
        }
    }

    // Now we are ready to execute the query to see if we have an user matching
    // the given credentials. If not, we will just return nulls and indicate
    // that there are no matching users for these given credential arrays.
    $user = $query->first();

    return $this->getGenericUser($user);
}

As before, we abort if there are not enough credentials to successfully fetch a user. Unlike with the EloquentUserProvider, we establish a direct connection to our database instead of relying on Eloquent to manage the connection for us.

Once the connection has been established, the query itself is set up in a similar manner to EloquentUserProvider. The main difference is the return value of the query's first method. Instead of a User model, we are instead given the raw values returned from the database. Working with raw arrays can be clunky and error-prone, however, so the queried array is cast into a GenericUser model before being returned to the SessionGuard.

With any luck, the above explanation demonstrates how different providers use different data sources and methods to accomplish the same result. While the retrieveByCredentials method on both providers ultimately returns a user object, the two classes differ in how they choose to query the database and construct their user objects.

Validating Credentials

With our user in hand, it's time to check their password. This is handled by the SessionGuard's hasValidCredentials method:

protected function hasValidCredentials($user, $credentials)
{
    return ! is_null($user) && $this->provider->validateCredentials($user, $credentials);
}

If retrieveByCredentials didn't return a user, we can safely assume that the given credentials are invalid, and we return false. Otherwise, the responsibility for determining how to validate the credentials is passed off to the provider. In this case, the two providers implement the method in exactly the same way:

public function validateCredentials(UserContract $user, array $credentials)
{
    $plain = $credentials['password'];

    return $this->hasher->check($plain, $user->getAuthPassword());
}

Now, cryptography is an interesting and useful subject, but if we were to include an intro to cryptography in this column the reading time would span well over an hour. Suffice it to say for now that the hasher@check method returns either 'true' or 'false' depending on whether the provided password matches the password on record.

Mission Success: Logging In

At this point, if the user has passed the hasher check and reached the login method, the login attempt has been successful. From here, the SessionGuard transitions into doing the work necessary to log the user into your application.

/**
* Log a user into the application.
*
* @param  \Illuminate\Contracts\Auth\Authenticatable  $user
* @param  bool  $remember
* @return void
*/
public function login(AuthenticatableContract $user, $remember = false)
{
    $this->updateSession($user->getAuthIdentifier());

    // If the user should be permanently "remembered" by the application we will
    // queue a permanent cookie that contains the encrypted copy of the user
    // identifier. We will then decrypt this later to retrieve the users.
    if ($remember) {
        $this->ensureRememberTokenIsSet($user);

        $this->queueRecallerCookie($user);
    }

    // If we have an event dispatcher instance set we will fire an event so that
    // any listeners will hook into the authentication events and run actions
    // based on the login and logout events fired from the guard instances.
    $this->fireLoginEvent($user, $remember);

    $this->setUser($user);
}

First, because this is the SessionGuard, our application updates the user's session to reflect their successful login.

/**
* Update the session with the given ID.
*
* @param  string  $id
* @return void
*/
protected function updateSession($id)
{
    $this->session->put($this->getName(), $id);

    $this->session->migrate(true);
}

A new session value is added with the user's ID to let our application know that this user is logged in. From there, the session is renewed using migrate to force our app to refresh and acknowledge our changes.

I'm going to skip the logic behind the remember token process, as it's rather intricate, requires some knowledge of how cookies work, and ultimately not directly related to whether a user actually is who they claim to be.

Besides the event methods, which don't directly affect the login process, the only function remaining in the login method is setUser. The sole responsibility of this method is to save some information about the user and the login attempt in memory so that the user doesn't have to be loaded again should this class be called later on in the app lifecycle.

With this, the login function stops execution, effectively ending the login process. The true value is returned from the guard's attempt method, which then causes the the attemptLogin method in our LoginController to return true as well. At this point, the only thing left to do is to send the user a successful login response:

// Illuminate\Foundation\Auth\AuthenticatesUsers

/**
* Send the response after the user was authenticated.
*
* @param  \Illuminate\Http\Request  $request
* @return \Illuminate\Http\Response
*/
protected function sendLoginResponse(Request $request)
{
    $request->session()->regenerate();

    $this->clearLoginAttempts($request);

    return $this->authenticated($request, $this->guard()->user())
            ?: redirect()->intended($this->redirectPath());
}

Remember that our application's session was reset after we stored the user's ID. By refreshing the request's session, we guarantee that the request session is fully in sync with the application session. This means that calls to $request->session() that happen after our user has logged in will correctly state that the user is authenticated.

The last step depends on your application. If you've defined an authenticated method in your LoginController, that method will take over from here. This is good if you want to implement some custom logic that's more advanced than a simple redirect.

Otherwise, your user is redirected using the RedirectsUsers trait:

// Illuminate\Foundation\Auth\RedirectsUsers

/**
* Get the post register / login redirect path.
*
* @return string
*/
public function redirectPath()
{
    if (method_exists($this, 'redirectTo')) {
        return $this->redirectTo();
    }

    return property_exists($this, 'redirectTo') ? $this->redirectTo : '/home';
}

If there's a redirectTo method on your controller, this trait will call it and redirect users to whatever the return value is. This could be helpful if your application has different tiers of users that should be directed to different places - subscribers to /subscribers/ and regular users to /dashboard/, for instance.

Otherwise, if a redirectTo property exists, that will be the redirect target. In case neither a method nor a property exists, the default redirect path is to /home. Personally, I never end up using the /home route in my own applications, so I recommend that you define your custom redirectTo property to fit your application.

Wrapping Up

Hopefully this journey through the lifecycle of a Laravel login has been illustrative. We've taken the opportunity not only to examine how the authentication process works at a high level, but also how the different drivers and guards available within Laravel can customize individual steps of a process while creating the same end result.

Should you have any more requests for similar walkthroughs, please don't hesitate to reach out to me @mariowhowrites on Twitter, Medium, Practical Dev, or my own website.

Files and Decorators: Studying Storage in Laravel

Mario Vega — Tue, 04 Jun 2019 20:29:27 +0000

I've been working on a project in Laravel that requires substantial work with files — specifically, images. Uploading, downloading, copying, moving and deleting images is a crucial part of what makes this application tick.

At a certain point, I realized that I didn't know nearly as much as I wanted to about how my application processed images. I knew what was outlined in the documentation, sure, but not much else beyond that. I had heard of PHP's Flysystem package through my work with Laravel and Statamic, but I didn't know why this package was so popular or how it worked at a core level. This lack of knowledge left me in a tough position when the business logic I needed to write went beyond the functionality covered in Laravel's Storage API.

For this reason, I decided to do a bit of digging. I've still got to finish my series on Authentication and Tailwind CSS, so I'm going to leave an extended series on Filesystems for another time. The point of this writeup is primarily to share the notes I've taken on how the Storage API of Laravel works under the hood. Along the way, I hope to connect the sometimes-abstract world of data structures with actual implementations to show how you can use these patterns in your own applications.

Storage from the Top Down

At the most general level, my goal in studying the Storage API was to connect the dots between the framework-level business logic I write and the actual business of working with images. More specifically, I wanted to find out how Laravel was able to wrap abstract concepts such as directories, permissions, URLs, etc into an API that works across concrete implementations. How is it that I can write the same code to work with images stored on my computer and images stored in an Amazon data center halfway across the world?

To figure this out, I had two choices — work from the bottom up or the top down. While both approaches would have worked well for my purposes, I ultimately decided to work from the top down. This allowed me to keep track of the methods and classes I used in my own project and follow them down the the lowest level I feel comfortable exploring: native PHP functions. (One day I will summon the courage to study the C code behind the PHP interpreter, but today is not that day.)

So, let's look at some code! The first method you're likely familiar with is the Storage::disk method. The Storage facade defers to the FilesystemManager class, as we can see in the docblock for the facade:

/**
    * @method static \Illuminate\Contracts\Filesystem\Filesystem disk(string $name = null)
    *
    * @see \Illuminate\Filesystem\FilesystemManager
    */
class Storage extends Facade
{
    // facade here

And here's the disk method on the FilesystemManager class:

/**
    * Get a filesystem instance.
    *
    * @param  string  $name
    * @return \Illuminate\Contracts\Filesystem\Filesystem
    */
public function disk($name = null)
{
    $name = $name ?: $this->getDefaultDriver();

    return $this->disks[$name] = $this->get($name);
}

/**
    * Attempt to get the disk from the local cache.
    *
    * @param  string  $name
    * @return \Illuminate\Contracts\Filesystem\Filesystem
    */
protected function get($name)
{
    return $this->disks[$name] ?? $this->resolve($name);
}

If a disk has already been loaded before, it is found in the disks property, saving the need to load a disk twice. If the disk were not stored in memory using the disks property, the same disk would be loaded again every time you called the Storage facade, which would not be great from a performance standpoint.

Let's look at the resolve method:

/**
    * Resolve the given disk.
    *
    * @param  string  $name
    * @return \Illuminate\Contracts\Filesystem\Filesystem
    *
    * @throws \InvalidArgumentException
    */
protected function resolve($name)
{
    $config = $this->getConfig($name);

    if (isset($this->customCreators[$config['driver']])) {
        return $this->callCustomCreator($config);
    }

    $driverMethod = 'create'.ucfirst($config['driver']).'Driver';

    if (method_exists($this, $driverMethod)) {
        return $this->{$driverMethod}($config);
    } else {
        throw new InvalidArgumentException("Driver [{$config['driver']}] is not supported.");
    }
}

/**
    * Get the filesystem connection configuration.
    *
    * @param  string  $name
    * @return array
    */
protected function getConfig($name)
{
    return $this->app['config']["filesystems.disks.{$name}"];
}

The first thing that happens here is that we load the necessary configuration from our Laravel application setup. This is stored in a file called filesystems.php, in which a typical disk looks like this:

'disks' => [

    'public' => [
        'driver' => 'local',
    'root' => storage_path('app/public'),
    'url' => env('APP_URL') . '/storage',
    'visibility' => 'public',
    ],

    's3' => [
    'driver' => 's3',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'region' => env('AWS_DEFAULT_REGION'),
    'bucket' => env('AWS_BUCKET'),
    'url' => env('AWS_URL'),
    ],

],

Depending on the disk type, this configuration contains keys such as visibility, url, region, and so on.

First, Laravel checks to see whether a custom filesystem is registered under the driver name passed to Storage::disk. If so, the custom filesystem is used, skipping most of Laravel's default functionality.

If there isn't a custom drive, Laravel next checks to see whether it has a default driver available for the specified disk. If we're using the local driver, for example, the class will look for a createLocalDriver function, which exists and can be found here:

/**
    * Create an instance of the local driver.
    *
    * @param  array  $config
    * @return \Illuminate\Contracts\Filesystem\Filesystem
    */
public function createLocalDriver(array $config)
{
    $permissions = $config['permissions'] ?? [];

    $links = ($config['links'] ?? null) === 'skip'
        ? LocalAdapter::SKIP_LINKS
        : LocalAdapter::DISALLOW_LINKS;

    return $this->adapt($this->createFlysystem(new LocalAdapter(
        $config['root'], $config['lock'] ?? LOCK_EX, $links, $permissions
    ), $config));
}

The important logic for our purposes is the last statement with the return value.

I'm gonna copy it again, but with more indentation to make the hierarchy a bit clearer:

$this->adapt(
    $this->createFlysystem(
        new LocalAdapter(
            $config['root'], 
            $config['lock'] ?? LOCK_EX, 
            $links, 
            $permissions
        ), 
        $config
    )
);

We can see that there are three primary levels to this function. Working from the bottom up:

A LocalAdapter is created. This is the Flysystem class that handles the nitty-gritty of actually working with local files.
The created adapter is then passed into createFlysystem, which wraps the adapter in Flysystem's Filesystem class. This Filesystem class handles some of the more general filesystem work — normalizing path strings, reading and acting on config options, and so on.
The resulting filesystem is then handed off to the adapt method. This creates a Laravel-specific wrapper that allows Flysystem to integrate fully with the Laravel framework.

In essence, this function contains the primary hierarchy that comprises the Storage API. It's also worth noting here that this setup is an excellent example of the decorator pattern.

Decorators and Russian Nesting Dolls

The decorator pattern is kind of similar to Russian nesting dolls. A smaller base object is encapsulated in a slightly larger object, which retains all of the functionality of the smaller object and adds a few methods of its own.

Pictured: Russian nesting dolls. Like software development, with more colors and fewer edge cases.

Notice that only the bottommost level — the Flysystem adapter — is actually capable of manipulating files. Yet, by itself, the adapter isn't capable of interfacing with Laravel in an intuitive manner that developers could pick up without writing lots of boilerplate themselves. One potential solution to this would be to fork the Flysystem package and add the necessary functionality directly to Flysystem, but this approach comes with plenty of downsides. In addition to freezing Laravel's Flysystem fork out from any updates that might be added to Flysystem in the future, this approach would add lots of additional code to the Flysystem package that would make future changes very cumbersome.

This is a perfect place to deploy the decorator pattern. By wrapping the Flysystem package with classes providing additional methods and properties, the underlying package can be used as-is while still providing a fluid and intuitive API for Laravel developers.

To illustrate this, let's look at one of the most basic tasks you can do with an image: store it. Remembering that the top-down hierarchy is something like Laravel Filesystem -> Flysystem -> Flysystem Adapter, we'll first look at the put method in the Laravel class returned by the adapt method above, FilesystemAdapter:

// Illuminate\Filesystem\FilesystemAdapter

/**
    * Write the contents of a file.
    *
    * @param  string  $path
    * @param  string|resource  $contents
    * @param  mixed  $options
    * @return bool
    */
public function put($path, $contents, $options = [])
{
    $options = is_string($options)
                    ? ['visibility' => $options]
                    : (array) $options;

    // If the given contents is actually a file or uploaded file instance than we will
    // automatically store the file using a stream. This provides a convenient path
    // for the developer to store streams without managing them manually in code.
    if ($contents instanceof File ||
        $contents instanceof UploadedFile) {
        return $this->putFile($path, $contents, $options);
    }

    return is_resource($contents)
            ? $this->driver->putStream($path, $contents, $options)
            : $this->driver->put($path, $contents, $options);
}

The business logic stored in this example matches what we could potentially expect from Laravel classes: helpers to make sure developers travel along the "happy path" to productivity. Different branches are provided in case the method encounters a file or a stream, ensuring that put can be used with any type of resource.

Notice, however, that ultimately the work of storing the image is passed off to the driver->put method. The driver in this case represents the Flysystem Filesystem class, so let's look at the put method there:

// League\Flysystem\Filesystem

/**
    * Create a file or update if exists.
    *
    * @param string $path     The path to the file.
    * @param string $contents The file contents.
    * @param array  $config   An optional configuration array.
    *
    * @return bool True on success, false on failure.
    */
public function put($path, $contents, array $config = [])
{
    $path = Util::normalizePath($path);
    $config = $this->prepareConfig($config);

    if ( ! $this->getAdapter() instanceof CanOverwriteFiles && $this->has($path)) {
        return (bool) $this->getAdapter()->update($path, $contents, $config);
    }

    return (bool) $this->getAdapter()->write($path, $contents, $config);
}

Without diving too deep into the source code, we can see a couple things about the logic in this function. First, there's no mention of anything we might associate with Laravel — no branching paths, no Illuminate classes, etc. Keeping in mind that the Filesystem class is used across all different types of filesystems, from local to S3 and beyond, we see that the logic here is the kind of prep work that needs to be done for any path-based storage system. The path is normalized using Util::normalizePath, any necessary config for the adapter is formatted with prepareConfig, and then a method on the underlying adapter is called depending on whether the method should override existing content or not.

It's worth keeping in mind that any of the methods here can be called in Laravel by using the getDriver() method on a Storage API. So, if for some reason you wanted to call Flysystem's put directly instead of using Laravel's put method, you could do so by calling Storage::disk('your-disk')->getDriver()->put().

Finally, we see that this put method calls methods on the underlying Adapter instance by using getAdapter. As the adapter you're using will depending on which filesystem you're using, this method will look different if you're storing files locally vs storing them on AWS S3. We'll look at the LocalAdapter implementation for now:

// 

/**
* Write a new file.
*
* @param string $path
* @param string $contents
* @param Config $config   Config object
*
* @return array|false false on failure file meta data on success
*/
public function write($path, $contents, Config $config)
{
    $location = $this->applyPathPrefix($path);
    $this->ensureDirectory(dirname($location));

    if (($size = file_put_contents($location, $contents, $this->writeFlags)) === false) {
        return false;
    }

    $type = 'file';
    $result = compact('contents', 'type', 'size', 'path');

    if ($visibility = $config->get('visibility')) {
        $result['visibility'] = $visibility;
        $this->setVisibility($path, $visibility);
    }

    return $result;
}

/**
* Set the visibility for a file.
*
* @param string $path
* @param string $visibility
*
* @return array|false file meta data
*/
public function setVisibility($path, $visibility)
{
    $location = $this->applyPathPrefix($path);
    $type = is_dir($location) ? 'dir' : 'file';
    $success = chmod($location, $this->permissionMap[$type][$visibility]);

    if ($success === false) {
        return false;
    }

    return compact('path', 'visibility');
}

This is the point in the code where files are actually stored on your filesystem using PHP's native function file_put_contents. Here, the path and contents provided by the developer are combined with the writeFlags defined by Laravel when it created the LocalAdapter within the FilesystemManager.

Similarly, the visibility of the file is set using PHP's chmod function, which works in a nearly identical fashion to Linux's 'chmod' command.

Wrapping Up

By tracing the stack involved in storing images with Laravel, we've hopefully learned two things: how the Storage API works, and how the architecture of the Storage API follows the decorator paradigm. At each level of the API, the class does some preparatory work before passing responsibility off to the API below it. The end result is that your file is stored in the correct location within your application and the necessary information is returned for you as the developer to take action on — all with a single line of code.

Parsing Open Source: Tailwind CSS Part 2

Mario Vega — Thu, 18 Apr 2019 02:14:54 +0000

Welcome back! Firstly, I want to thank you for the warm reception you gave Chapter 1 of Parsing Open Source. I'm back with Chapter 2, where we'll finish off our analysis of Tailwind CSS. This chapter gets into the hairier parts of Tailwind CSS, so I'll be moving a bit slower than I did in the last chapter so we can make extra-sure we have a solid grasp of what's happening. I'll be building on what we established in Chapter 1, so it'd be a good idea to either read the first parts of that chapter or have some pre-existing knowledge of how PostCSS works.

This chapter is dedicated exclusively to two things: 1) a brief overview of my parsing process and 2) an extended look at the resolveConfig function, a deceptively short function that nonetheless encapsulates many of the patterns and abstractions that define functional programming.

Grokking Complex Code

You may be wondering how to start parsing code in the open source libraries you use. My strategies are admittedly pretty simple, but they've proven effective to me so far and they're what I used to write this chapter. I'm presenting them in brief here so that you can use them the next time you're struggling to understand some code.

Use the Tests, Luke

One of the biggest helps to me in writing this chapter was Tailwind's well-written tests. Good tests are sometimes better than documentation in helping to understand a library, as they provide examples of the codebase as the author intends for it to be used.

Because Tailwind's most intricate functions all have unique tests, parsing each individual function boiled down to running a single test over and over again. Here's my battle-tested workflow:

Isolate the test I want to run with Jest's CLI. If I'm testing the resolveConfig function, I run jest __tests__/resolveConfig.test.js on my command line from the project root.
Isolate one particular test that encapsulates the functionality I'm examining. Typically I pick the most complex test I can find in the test suite and change the line saying test('complex test here') to test.only('complex test here'). Adding the only method tells Jest to only run that particular test.
Throw console.log statements everywhere.

You think I'm joking, but I'm not. Much as I hate to admit it, Node's debugger is too clunky for me. Setting aside the time it takes to get it set up and working with Jest, you have to add a debugger statement to your code, then run the repl command once your code hits the right place. All of that, just to give you the same output as you get from a console.log? No thank you. Someone please let me know if I'm missing something, but until then console.log is bae.

If I'm feeling particularly ambitious, I'll write the output to a log file I create using Node's filesystem module. But most of the time, console.log will get me where I want to go.

Work From the Outside In

Ok, so we've got our console.logs ready to go. Now what?

This will likely sound obvious to the point of banality, but I work from the outside in. First I'll log a function's arguments at the moment it starts executing - what does everything look like at the beginning?

From there, find the first transformation that happens and console.log it. Then, compare the results to the results from the first log. What's changed between then and now? Upon understanding the first step, move to the second. Write down your results if it helps you, then repeat the same process at the next level. Take a moment to consider why the difference is as it is. Consider both what's happening at the code level and what the author's intention is in transforming the data in this way.

Eventually, with enough time and patience, what once looked daunting will seem manageable, even obvious.

Baby steps are the key here. I often comment out and rewrite code to get a console.log where I need it to go — adding brackets to turn single-line functions into multi-line functions, breaking chained functions into individual functions, and so on. Don't be afraid to change the code and see what happens; the undo shortcut is there for a reason. Worst comes to worst, delete the whole repo and clone it again from GitHub.

Those are my code parsing strategies in a nutshell. With those established, onwards to Tailwind!

ResolveConfig

The first function we left untouched in Chapter 1 is the resolveConfig function. To refresh quickly from last chapter, the resolveConfig function is responsible for merging the user-provided configuration with the default configuration to create a final configuration object. It's called near the beginning of the Tailwind process, before any PostCSS plugins have been applied, and it's responsible for creating the settings that the rest of Tailwind will abide by when creating its final CSS output.

Here is the code for that function:

    export default function resolveConfig(configs) {
      return defaults(
        {
          theme: resolveFunctionKeys(mergeExtensions(defaults({}, ...map(configs, 'theme')))),
          variants: defaults({}, ...map(configs, 'variants')),
        },
        ...configs
      )
    }

We should first note that the defaults function called here is imported from Lodash. How this function operates is crucial in the config resolution process, so let's go over it briefly. Basically, what defaults does is take the first object passed to it (also called the target) and fills it in with every other object in its parameter list, which are called sources. The function moves through the sources from left to right and, importantly, does not redefine a property if it already exists in the target.

Let's look at a simple example:

    const firstSource = {
        author: "Ursula Le Guin",
    }

    const secondSource = {
        author: "Dan Brown",
        actor: "Daniel Day-Lewis"
    }

    const finalTarget = defaults({}, firstSource, secondSource)

    // console.logging finalTarget would result in the following:
    {
        author: "Ursula Le Guin",
        actor: "Daniel Day-Lewis"
    }

Two steps happen when defaults is called:

The empty target object is filled with the first source object. Because there is no author in the target yet, the author is set to Ursula Le Guin.
The target is filled with the second source object. Because there is no actor in the target yet, the target's actor is set to Daniel Day-Lewis. However, because there is already an author set in the target from step 1, the target does not take on the second source's author property. Dan Brown is rightfully cast aside into the dustbin of authorship.

The reason why this is important will be clear in a moment. For now, looking back at the resolveConfig function we can see that defaults function is used to ensure that the theme and variants keys of our final config are resolved first. From there, all other configuration values are passed in from the configs passed to resolveConfig.

It is also important here to remember that resolveConfig accepts an array as its only parameter, in which the user config comes before the default config. This is important because, based on what we know about the defaults function, we now know that any properties defined in the user config will not be overwritten by properties in the default config. The user config and the default config can be understood as more intricate versions of firstSource and secondSource from our example above. Because our user config is our first source, nothing from our second source — the default config — will take precedence, instead deferring to the user's preferences.

Now, let's take another look at the resolveConfig function:

    export default function resolveConfig(configs) {
      return defaults(
        {
          theme: resolveFunctionKeys(mergeExtensions(defaults({}, ...map(configs, 'theme')))),
          variants: defaults({}, ...map(configs, 'variants')),
        },
        ...configs
      )
    }

What we want to focus on here is our target: the first argument to defaults that has theme and variant keys. Let's also use some more indentation to make things slightly easier to read:

    {
      theme: 
            resolveFunctionKeys(
                mergeExtensions(
                    defaults(
                        {}, 
                        ...map(configs, 'theme')
                    )
                )
            ),
      variants: 
            defaults(
                {}, 
                ...map(configs, 'variants')
            ),
    },

Let's look at what's happening in the theme property first, as it's a tad more complex. Knowing that the JavaScript engine will execute this function from the inside out, the first thing we need to look at is the defaults call.

That code looks like this:

    defaults({}, ...map(configs, 'theme'))

We see that an empty target object is filled with the theme key from each configuration. As before, the user configuration is filled first, then any keys left undefined by the users are filled in from the default config.

Using the strategies I outlined at the beginning of the chapter, I picked one test in the resolveConfig test suite to run repeatedly in my parsing process. That test looks like this:

    test.only('theme values in the extend section are lazily evaluated', () => {
      const userConfig = {
        theme: {
          colors: {
            red: 'red',
            green: 'green',
            blue: 'blue',
          },
          extend: {
            colors: {
              orange: 'orange',
            },
            borderColor: theme => ({
              foo: theme('colors.orange'),
              bar: theme('colors.red'),
            }),
          },
        },
      }

      const defaultConfig = {
        prefix: '-',
        important: false,
        separator: ':',
        theme: {
          colors: {
            cyan: 'cyan',
            magenta: 'magenta',
            yellow: 'yellow',
          },
          borderColor: theme => ({
            default: theme('colors.yellow', 'currentColor'),
            ...theme('colors'),
          }),
        },
        variants: {
          borderColor: ['responsive', 'hover', 'focus'],
        },
      }

      const result = resolveConfig([userConfig, defaultConfig])

    /* expected result not immediately relevant and thus left out for brevity */

When running the above test and examining the result of the first defaults function call, the result looks something like this:

    { 
        colors: { 
            red: 'red', 
            green: 'green', 
            blue: 'blue' 
        },
        extend: { 
            colors: { 
                orange: 'orange' 
            },
        borderColor: [Function: borderColor] 
        },
        borderColor: [Function: borderColor] 
    }

We see that any values defined in the user config override any values in the default config. Namely, the colors defined by default have been thrown out and replaced by the user-config colors. We also see that the extends key holds an extra color, orange, and an extra function that will define border colors.

Knowing that this result is then immediately passed to mergeExtensions, let's look at that function next:

    function mergeExtensions({ extend, ...theme }) {
      return mergeWith(theme, extend, (themeValue, extensions) => {
        if (!isFunction(themeValue) && !isFunction(extensions)) {
          return {
            ...themeValue,
            ...extensions,
          }
        }

        return resolveThemePath => {
          return {
            ...value(themeValue, resolveThemePath),
            ...value(extensions, resolveThemePath),
          }
        }
      })
    }

This function is trickier than it might appear at first, so let's take it line-by-line.

First, let's look at the function parameters. We see that an object is accepted as the sole parameter, and that this object is broken down into two key components. The extends key is pulled directly from the passed-in object, and all other keys on the object are combined using the rest operator ... into a single object called theme. So, taking our result above, the top-level color and borderColors keys would be combined into theme, while the extends key would be used as-is.

From there, another Lodash function is called: mergeWith. Personally, I'm not sold on the mergeWith name. I would likely call this method customMerge instead, as what it's doing is merging two objects together using a custom merge function. In other words, the function passed as the third argument to mergeWith is called on each key in the object passed in as the first argument.

In the context of our test object, this means that mergeWith will be called twice: once for colors and once for borderColors. For each key, the following process occurs:

Compare the key values in each object.
If neither value is a function, combine the values and return the result.
If either value is a function, return a function that calls both functions and returns the combined result.

Step 3 is a bit complicated, so we'll have to go over that in detail. For now, let's focus on Step 2, as there's a mechanic at play here that differs significantly from what we've seen before.

The difference has to do with the way the ellipsis operator ... is used in JavaScript. There are two primary uses of the spread operator, both of which we've seen already. The first, as we just saw in the function parameter, is used to condense multiple values into a single value. In the above example, colors and borderColors were combined into a theme object using an ellipsis. This use of the ellipsis in this manner is called rest syntax, and it creates one object out of multiple values.

The ellipsis is also used to perform an operation that is essentially the exact opposite of rest syntax. In this use, one object or array is expanded into multiple values. This syntax is called spread syntax, and we see it in use when creating the return objects in mergeExtensions.

There is one important detail to note here. In short, using spread syntax works exactly opposite to Lodash's defaults function: if the second source defines a key that is also present in the first source, the second source will override the first.

To use our previous example:

    const firstSource = {
        author: "Ursula Le Guin",
    }

    const secondSource = {
        author: "Dan Brown",
        actor: "Daniel Day-Lewis"
    }

    const finalTarget = { ...firstSource, ...secondSource }

    // console.logging finalTarget would result in the following:
    {
        author: "Dan Brown", // secondSource overrides firstSource!
        actor: "Daniel Day-Lewis"
    }

Sadly, Ursula Le Guin is pushed aside in this iteration to make room for a far less adept author. (I prefer Le Guin to Brown, if this hasn't been made clear.)

What this means in Tailwind context is that, given a key that exists in both the theme and the extends objects, the extends value will take precedence over the theme value.

In this way, the extends key can be useful in scenarios where you want to override one default value without replacing a given category entirely. For example, should you want to override the default red color without overriding all the default colors, to my understanding using the extends key would be a good way of doing so.

With a better understanding of how the rest and spread operators work, let's take another look at Step 3, which happens if either the theme or the extension is a function:

    function value(valueToResolve, ...args) {
      return isFunction(valueToResolve) ? valueToResolve(...args) : valueToResolve
    }

    mergeWith(theme, extend, (themeValue, extensions) => {
        // if themeValue or extensions is a function...
        return resolveThemePath => {
          return {
            ...value(themeValue, resolveThemePath),
            ...value(extensions, resolveThemePath),
          }
        }
      })
    }

There are some similarities to Step 2 here: both steps construct an object using the spread operators on both the theme and extension values. However, in this case, instead of creating the object and returning it directly, a function is returned whose sole responsibility is to create the object.

This function accepts the resolveThemePath and passes it into the value function, which then determines whether either themeValue or extensions is itself an function. If so, it calls that function with resolveThemePath. The results of the two value calls are then merged and returned.

I know: lots of functions. This logic encapsulates both the power and the frustration that often accompany functional programming. While the ability to pass functions around and load them with relevant data as necessary is undoubtedly one of JavaScript's most powerful features, it can be maddeningly difficult to figure out exactly what is happening at what point. Where is a function being called and when it is simply being created for use elsewhere?

Notably, in the code above, no functions are actually invoked when merging theme and extension functions during mergeExtensions. Instead, a function is returned that calls themeValue and extensions at the same time.

Let's look at what's returned from mergeExtensions when calling our previous test:

    { 
        colors: { 
            red: 'red', 
            green: 'green', 
            blue: 'blue', 
            orange: 'orange' 
        },
      borderColor: [Function] 
    }

We can see two primary differences from the previous result:

The colors keys from the theme and extensions objects have been merged.
The two borderColors functions from the last result have been combined into one.

Additionally, we see that the extends key no longer exists, as it has been merged into the theme.

We've almost worked our way through the logic governing how the theme is constructed. Let's examine the final function, resolveFunctionKeys:

    function resolveFunctionKeys(object) {
      const resolveObjectPath = (key, defaultValue) => {
        const val = get(object, key, defaultValue)
        return isFunction(val) ? val(resolveObjectPath) : val
      }

      return Object.keys(object).reduce((resolved, key) => {
        return {
          ...resolved,
          [key]: isFunction(object[key]) ? object[key](resolveObjectPath) : object[key],
        }
      }, {})
    }

We see that a function expression resolveObjectPath is defined — let's return to that in a moment, once we understand the context in which its used. Let's instead look at what happens with the result of mergeExtensions:

Object.keys is used to create an array of the object's keys. For our above result, we'd get an array like this: [colors, borderColors].
We loop through the array of keys using the reduce function. I'd definitely recommend doing some research on reduce if you're not familiar, because it's quite useful in a number of situations. For now, suffice it to say that reduce loops over an array in order to "build" a result. It's essentially a more flexible version of map.
For each key, we look at the assorted value. If it's a function, it's invoked using the resolveObjectPath function. If it's not a function, it's returned as-is.
The result is added to our "built" object. This "built" object is then passed along to the next key in the array.

In essence, this process converts the object from mergeExtensions into a raw JavaScript object, with all functions replaced by key-value pairs.

With this in mind, let's look at resolveObjectPath:

    function resolveFunctionKeys(object) {
      const resolveObjectPath = (key, defaultValue) => {
        const val = get(object, key, defaultValue)
        return isFunction(val) ? val(resolveObjectPath) : val
      }

      // rest of function here
    }

The resolveObjectPath function uses a pattern we've seen before: the use of function expression to embed state into a function. In this case, the function takes in a key and a defaultValue, and uses the object passed into the top-level resolveFunctionKeys function to get the value from the object, using the default value if the config doesn't contain the value in question. If the returned value is a function, the process is repeated with the new function, otherwise the value is returned as-is.

At this point, my head is starting to spin a bit. I've written the word "function" so many times it's starting to lose all meaning. So let's ground what we're doing in some actual usage: what happens when we pass a function into our config?

Let's go back to the test we've been working with, deleting parts that aren't relevant:

    test.only('theme values in the extend section are lazily evaluated', () => {
      const userConfig = {
        theme: {
          colors: {
            red: 'red',
            green: 'green',
            blue: 'blue',
          },
          extend: {
            colors: {
              orange: 'orange',
            },
            borderColor: theme => ({
              foo: theme('colors.orange'),
              bar: theme('colors.red'),
            }),
          },
        },
      }

      const defaultConfig = {
        theme: {
          colors: {
            cyan: 'cyan',
            magenta: 'magenta',
            yellow: 'yellow',
          },
          borderColor: theme => ({
            default: theme('colors.yellow', 'currentColor'),
            ...theme('colors'),
          }),
        },
      }

      const result = resolveConfig([userConfig, defaultConfig])

      expect(result).toEqual({
        theme: {
          colors: {
            orange: 'orange',
            red: 'red',
            green: 'green',
            blue: 'blue',
          },
          borderColor: {
            default: 'currentColor',
            foo: 'orange',
            bar: 'red',
            orange: 'orange',
            red: 'red',
            green: 'green',
            blue: 'blue',
          },
        },
      })
    })

The extra-important parts here are the two borderColor functions: the first in the extends key of the user config, and the second in the default config.

If we look at the result, we see that the results of both functions eventually make their way into the final borderColor property. In this case, foo and bar both resolve to the user-defined options of orange and red, respectively. However, because the default color references a yellow color that doesn't make it into the final config, the fallback default of currentColor is used instead.

Through this example, we get a better understanding of how functions work within the context of resolveConfig. Any functions within the theme key are passed in the final theme values after replacing defaults and merging extensions. Now, let's explore how exactly this process happens.

The first context in which we see functions come into play is within mergeExtensions. This is where the default functions and the extension functions are combined.

Let's rewrite this code in a more literal way, as if we were hard-coding the test case within Tailwind:

    function mergeExtensions() { // we are hard-coding arguments below rather than passing them in
        function userExtendsBorderColorFunction(theme) { // from user.theme.extend.borderColor
            return {
                foo: theme('colors.orange'),
            bar: theme('colors.red'),
            }
        }

        function defaultBorderColorFunction(theme) { // from default.theme.borderColor
            return {
              default: theme('colors.yellow', 'currentColor'),
              ...theme('colors'),
            }
        }

        return function(resolveThemePath) {
          return {
                ...defaultBorderColorFunction(...resolveThemePath),
            ...userExtendsBorderColorFunction(...resolveThemePath),
          }
        }
    }

With this more literal example, it is hopefully clearer what mergeExtensions does when it comes across a key with a function value. In this case, when mergeExtensions encounters the borderColor key and sees that its value is a function, it creates a new function that combines the default function with the function the user defined in the extends key. As before, any keys defined in the user config override keys found in the default config via spread syntax.

It bears repeating here that, as of now, neither userExtendsBorderColorFunction nor defaultBorderColorFunction have been called yet. This is an important distinction, as exactly when these functions are called is important. If our two functions were to be called within mergeExtensions, it is possible that they would be called using incorrect values. This is because, if the mergeExtensions function is still executing and has not yet finished its work, there are no guarantees that the config object has been populated with all of the user-defined extensions.

This is, incidentally, what is meant when the test is labeled: "theme values in the extend section are lazily evaluated". Laziness, which here means "waiting until other functions have finished" and not "binge-watching Parks and Recreation reruns on Netflix" ensures that when our functions finally do run, they work with the fully updated theme values.

So, we know that the function returned from our modified mergeExtensions key above is added to the borderColor key and combined with the other theme values in a unified theme object.

In a similar vein to the last code snippet, let's rewrite resolveFunctionKeys in a more literal way, substituting any abstracted values with literal values where possible:

    function resolveFunctionKeys(object) {
      const resolveObjectPath = (key, defaultValue) => {
        const val = get(object, key, defaultValue)
        return isFunction(val) ? val(resolveObjectPath) : val
      }

        return {
            borderColor: object.borderColor(resolveObjectPath)
        }
    }

I've removed the Object.keys and reduce from our modified function to simplify things a bit.

At this point, we can start connecting the dots regarding how Tailwind resolves functions using the extended configuration object. The mergeExtensions function finalizes all the static values (colors, padding, etc.) and sets up all functions to be run once all other values have been resolved. resolveFunctionKeys then takes those finalized values, creates a function that uses Lodash's get function to fetch keys out of the merged object, and returns them for the user to use in any theme functions.

Put another way, the resolveObjectPath function in resolveFunctionKeys is the actual function that is passed into the following theme key:

    borderColor: theme => ({ // theme === resolveObjectPath from resolveFunctionKeys
      foo: theme('colors.orange') // fetches colors.orange from merged theme object,
      bar: theme('colors.red', 'defaultColor') // fetches colors.red, with a default of defaultColor
    })

Because the theme config object is captured within resolveObjectPath using function expressions, it is automatically accessible to the end user within the passed-in function. All the user has to do is specify which key value they want, and optionally which default value to fall back upon if the key is not found.

Now, let's take another look at the resolveConfig function:

    export default function resolveConfig(configs) {
      return defaults(
        {
          theme: resolveFunctionKeys(mergeExtensions(defaults({}, ...map(configs, 'theme')))),
          variants: defaults({}, ...map(configs, 'variants')),
        },
        ...configs
      )
    }

With any luck, this function is beginning to make a bit more sense. Virtually all of the complicated logic involves combining the user-provided theme with the default theme. The variants are resolved shortly after via a simple Lodash defaults call. Once the theme and the variants have been resolved, all other keys defined in the configuration are added to the object via another defaults call, and the result is returned for use in the rest of Tailwind.

Wrapping Up Resolving Config

We've gone over quite a bit, written the word "function" quite a lot, and generally taken a whirlwind tour of functional programming, JavaScript-style.

At a high level, let's recap the steps that resolveConfig takes to merge the user's configuration with the default values:

Copies the user theme values into an object, with all functions left untouched.
Copies all default theme values into user theme values, not overriding any user settings.
Merges all values in the user's theme.extend property into the theme object. Static values are concatenated, while functions on the same property are rolled into a single function for later use.
Using the static values obtained from the last step, calls all functions created during the last step and merges the result to create a finalized theme property.
Resolves the variants property by combining the user config with the default config.
Resolves all other keys through the same user → default precedence.

We saw that this process is accomplished using the same techniques we've seen throughout Tailwind so far, namely: functional programming, function expressions, and Lodash. We also explored rest and spread syntax in greater depth, comparing them against Lodash's default function and observing how both are used in conjuction to resolve user themes against the default theme settings.

The next chapter in this series will cover the PostCSS plugin chain. As always, if you have questions on what we've covered so far or suggestions on what open source library I should parse next, let me know. I'm available in the comments section or on Twitter @mariowhowrites . Until next time!

Parsing Open Source: Tailwind CSS

Mario Vega — Fri, 05 Apr 2019 23:11:42 +0000

Perhaps no single tool entered my developer workflow immediately after release as quickly as Tailwind CSS. I've always been a huge CSS fan. My first foray into web development was a mission to alter a sidebar on my WordPress blog, back before I knew what CSS was or how it worked.

However, for as long as I've loved CSS as a language, I've struggled to scale CSS in my projects. No matter how I organized my stylesheets, I always reached a point in my code where I was too afraid to touch anything. My stylesheets were arranged as a house of cards, ready to fall apart at the first sign of customization.

It was for this reason that I adopted the utility-based approach of Tailwind CSS. However, it's always struck me as a bit of a black box: I edit a JavaScript file, add a line to my package.json, and boom! CSS. So for these chapters of Parsing Open Source, I'm digging through the inner workings of Tailwind CSS.

This first chapter will cover a top-level overview of the Tailwind CSS codebase. This includes both the specifics of Tailwind's architecture and how it interacts with PostCSS, the framework upon which Tailwind is built. A second chapter will examine more specific implementation details; the original draft of this column with all details included was both long and intimidating, which is the opposite of my intention in this series.

My goals for this series are twofold: one, to help demystify the process of examining open-source code; two, to help improve my own understanding of how large-scale JavaScript projects are organized and implemented.

Tailwind and PostCSS

Ultimately, Tailwind CSS is a PostCSS plugin. So in order to understand how TailwindCSS works, it's helpful to understand how PostCSS works.

PostCSS is a powerful library with a deceptively simply API. Essentially, it does two things:

Converts CSS files into JS.
Converts JS files into CSS.

Why would you want to turn CSS into JS and vice versa?

CSS is an immensely powerful language, but it lacks many scripting features that define Turing-complete languages. It doesn't have loops, recursion, etc., and doesn't offer an easy way to programmatically generate itself.

These features are all found in JavaScript. By converting CSS into JavaScript, developers can then modify that CSS, add new rules, and so on using all the programmatic power of a scripting language like JavaScript.

Then, once the JavaScript "stylesheets" has been customized to the developers' liking, PostCSS offers an API to turn that JavaScript back into a CSS file, ready to use on every website on the Internet.

Going into the specifics of how PostCSS accomplishes this is, to use a scientific term, "2deep4me". However, it is important to know the basics of how PostCSS handles the conversion to and from JavaScript, as these details are exposed in the PostCSS API used throughout Tailwind.

Basically, when PostCSS converts CSS into JavaScript, it chooses to store the stylesheet information in a data structure called an abstract syntax tree (AST). ASTs are one of those computer science concepts that sound much more complex than they actually are.

Before continuing, let's refresh ourselves quickly on some CSS terminology. Here's a diagram I found on the Internet going over the anatomy of a given CSS rule:

Source: https://ryanbroome.wordpress.com/2011/10/13/css-cascading-style-sheet/

As you can see, everything from the h1 selector to the closing bracket makes up one distinct CSS rule. Within the rule, there can be any number of declarations. In the case of the diagram above, there are two declarations. One declares the color to be the hex value #333, while the other declares the font size to be the value x-large.

If we were to think of this rule as a tree, we could say that the rule itself is the root, or the parent, while each declaration is a leaf, or a child. Here's an shoddy diagram I created to illustrate the point:

Stepping out a level, we could also apply this same line of thinking to the entire stylesheet. With the stylesheet as our parent, we could consider each rule within the stylesheet to be a child of the parent stylesheet.

Basically, what PostCSS does is convert CSS into a tree similar to the diagram above. Your stylesheet is the root node, each rule is a leaf of the document root, and each individual declaration is a leaf of the rule where it is defined. Once the whole tree is constructed, any PostCSS plugin can "walk" the stylesheet by looping over each rule before repeating the process to "walk" across the rule to each declaration. With a given rule or declaration in hand, plugins can make any necessary changes to the stylesheet by utilizing the PostCSS API.

With this understanding in mind, let's look at a sample PostCSS plugin, taken from Dockyard's tutorial on how to write a plugin:

    var postcss = require('postcss');
    module.exports = postcss.plugin('postcss-test-plugin', function() {
      return function(root) {
        root.walkRules(function(rule) {
          rule.walkDecls(/^overflow-?/, function(decl) {
            if (decl.value === 'scroll') {
              var hasTouch = rule.some(function(i) {
                return i.prop === '-webkit-overflow-scrolling';
              });
              if (!hasTouch) {
                rule.append({
                  prop: '-webkit-overflow-scrolling',
                  value: 'touch'
                });
              }
            }
          });
        });
      };
    });

Knowing what we know about how PostCSS works, we can say that this plugin does the following:

Accepts a spreadsheet as the root argument of the top-level function.
Walks through each rule of the spreadsheet.
Within each rule, walks through each declaration that matches the RegEx pattern /^overflow-?/. In other words, finds each declaration that begins with the phrase overflow-.
If the declaration has a value of scroll, checks to see whether any other declaration in the rule defines a property of -webkit-overflow-scrolling.
If not, adds such a declaration to the rule, and give it the value touch.

Hopefully this example offers a glimpse into the power of PostCSS. Editing CSS programmatically would be impossible if we were just working with CSS. Instead, by translating CSS into a JavaScript AST, we can then walk the tree and edit our stylesheets using the full suite of tools available in JavaScript.

If we want to get super technical, the approach used to navigate the tree in this example is depth-first traversal, as we are fully examining each individual declaration of a rule before moving onto the next rule. That's not strictly necessary to understand how Tailwind works, but I always like to pair theoretical concepts with real-world scenarios where possible so that the theory seems a little less abstract.

Now that we have a bit more knowledge as to the context in which TailwindCSS operates, let's start looking at some code!

The Tailwind API

There are two places I like to start when parsing open source repositories. The first is the public API — ie. what happens when a developer invokes the repository in their own project. The second is the test coverage — ie. what tests a given repo has written to ensure their code works as intended. In that spirit, looking at the Tailwind documentation as well as the tests, we can start with the following two code snippets. The first is taken from the Webpack setup instructions using a postcss.config.js file, while the second is taken from the sanity.test.js file included in the __tests__ directory of Tailwind's repo:

    var tailwindcss = require('tailwindcss');

    module.exports = {
      plugins: [
        // ...
        tailwindcss('./path/to/your/tailwind.js'),
        require('autoprefixer'),
        // ...
      ]
    }

    import tailwind from '../src/index'

    it('generates the right CSS', () => {
      const inputPath = path.resolve(`${__dirname}/fixtures/tailwind-input.css`)
      const input = fs.readFileSync(inputPath, 'utf8')

      return postcss([tailwind()])
        .process(input, { from: inputPath })
        .then(result => {
          const expected = fs.readFileSync(
            path.resolve(`${__dirname}/fixtures/tailwind-output.css`),
            'utf8'
          )

          expect(result.css).toBe(expected)
        })
    })

While the two code snippets ostensibly achieve the same goal, we can see that the two implementations differ considerably. These differences mostly boil down to the two different contexts in which these code snippets are designed to run. The Webpack example is meant to be used as one part of a comprehensive project, while the Jest code example is meant to fully simulate the interactions with PostCSS that would, in the first example, be handled by Webpack.

Let's focus on the similarities: both code examples invoke a tailwind function, although the function is called tailwindcss in the first example to match the name of the NPM package. We see that, although the Webpack example assumes that your project is using its own configuration file, a custom config is not strictly necessary to use Tailwind, as a fallback default is used instead.

Furthermore, although the CSS file is not defined in the postcss.config.js file, we know from looking at the documentation and at the webpack-starter project that both the real-world and test examples take in a CSS stylesheet as a required argument. In the Jest example, the CSS input file is fetched from a fixtures directory within the tests folder and loaded into JavaScript using the path and fs modules, which are native to NodeJS.

Inspecting the tailwind-input file, we see that it closely mirrors the example setup in the Tailwind documentation:

    @tailwind base;

    @tailwind components;

    @tailwind utilities;

    @responsive {
      .example {
        @apply .font-bold;
        color: theme('colors.red.500');
      }
    }

This file is run through PostCSS using the process method, which produces a string representation of a CSS file. This output is then compared against a tailwind-output.css file, which includes all of the default Tailwind styles plus the following example styles:

    .example {
      font-weight: 700;
      color: #f56565;
    }

    ...

    .sm\:example {
        font-weight: 700;
        color: #f56565;
      }

    ...

    .md\:example {
        font-weight: 700;
        color: #f56565;
      }

    /* other responsive classes below */

If the CSS returned from PostCSS's process function matches the output of this file, the test passes — which, as of publication, it does.

Implementing Tailwind

We now know that the main export of Tailwind is a PostCSS plugin. We also know that it is a function that takes a single argument: the (optional) path to a Tailwind config file. With that in mind, let's take a look at how the tailwind function is exported. We can find it in the src/index.js file within the TailwindCSS repo:

    const plugin = postcss.plugin('tailwind', config => {
      const plugins = []
      const resolvedConfigPath = resolveConfigPath(config)

      if (!_.isUndefined(resolvedConfigPath)) {
        plugins.push(registerConfigAsDependency(resolvedConfigPath))
      }

      return postcss([
        ...plugins,
        processTailwindFeatures(getConfigFunction(resolvedConfigPath || config)),
        perfectionist({
          cascade: true,
          colorShorthand: true,
          indentSize: 2,
          maxSelectorLength: 1,
          maxValueLength: false,
          trimLeadingZero: true,
          trimTrailingZeros: true,
          zeroLengthNoUnit: false,
        }),
      ])
    })

From a top-level perspective, we can see that the following things are happening within this plugin:

The configuration file is resolved from the path argument of tailwindcss().
The resolved config as added as a dependency. As far as I can tell, this is solely used for Webpack push notifications during the build process, but someone please let me know if it's used in some way I'm not aware of.
A PostCSS plugin is returned where the following steps happen:
- The dependency is registered.
- Tailwind features are processed using a configuration function built from the resolved configuration path.
- The resulting CSS is cleaned up using the perfectionist PostCSS plugin.

The resolveConfigPath function is fairly straightforward:

    function resolveConfigPath(filePath) {
      if (_.isObject(filePath)) {
        return undefined
      }

      if (!_.isUndefined(filePath)) {
        return path.resolve(filePath)
      }

      try {
        const defaultConfigPath = path.resolve(defaultConfigFile)
        fs.accessSync(defaultConfigPath)
        return defaultConfigPath
      } catch (err) {
        return undefined
      }
    }

Here we see some of the first usages of lodash, which is an immensely popular JavaScript utility library. Lodash is used throughout the Tailwind repository, and I often had the Lodash documentation open while writing this analysis to grok some of the more complicated logic.

This function allows for the following possible outcomes:

The filepath is an object — the config has already been loaded, so return nothing.
The filepath exists and is not an object — it is a string, so try and resolve it using NodeJS's path module.
The filepath does not exist — load the default configuration, but return nothing if the necessary file permissions do not allow access to the default config.

This function confirms our earlier conclusion; a configuration file is not necessary to run TailwindCSS, as it will use the default configuration if the path is undefined.

Let's briefly look at getConfigFunction, the other function defined directly within index.js:

    const getConfigFunction = config => () => {
      if (_.isUndefined(config) && !_.isObject(config)) {
        return resolveConfig([defaultConfig])
      }

      if (!_.isObject(config)) {
        delete require.cache[require.resolve(config)]
      }

      return resolveConfig([_.isObject(config) ? config : require(config), defaultConfig])
    }

This function covers the following possibilities:

The config is undefined and not an object — resolve config with the default.
The config is not an object — it is a string. Delete the cached version of the config, then resolve configuration with the passed-in configuration and the default config.

The one part of this function that might look a bit strange is the line beginning with delete require.cache. This method has to do with the way NodeJS's require function works. When you require something with NodeJS, the result is loaded and stored in a cache. When you require that file again, NodeJS looks to the cache first. If it finds the file you requested, it will load the file from cache rather than refetching the whole library again.

In most cases, this is the ideal behavior. If you use Lodash in 20 places in your code, for example, you don't want to load Lodash 20 times, as that would slow down your code significantly.

However, in this case, we are using require on our configuration file. Because our config can and likely will change, we want to ensure that the config we eventually load is the valid configuration at the time the code is run. Therefore, before loading the new cache, we must delete the old cache first.

I'm going to leave the details of the resolveConfig function for the next chapter, as it's a bit of a doozy. Suffice it to say for now that this function's primary responsibility is to merge any user-supplied configuration with the default configuration, overriding the default where necessary. Here's the first test from resolveConfig.test.js, which provides a basic example of how the function works:

    test('prefix key overrides default prefix', () => {
      const userConfig = {
        prefix: 'tw-',
      }

      const defaultConfig = {
        prefix: '',
        important: false,
        separator: ':',
        theme: {
          screens: {
            mobile: '400px',
          },
        },
        variants: {
          appearance: ['responsive'],
          borderCollapse: [],
          borderColors: ['responsive', 'hover', 'focus'],
        },
      }

      const result = resolveConfig([userConfig, defaultConfig])

      expect(result).toEqual({
        prefix: 'tw-',
        important: false,
        separator: ':',
        theme: {
          screens: {
            mobile: '400px',
          },
        },
        variants: {
          appearance: ['responsive'],
          borderCollapse: [],
          borderColors: ['responsive', 'hover', 'focus'],
        },
      })
    })

You can see that the user-supplied prefix key overrides the default prefix, but all other default values are preserved in the final result.

In the end, what the getConfigFunction returns is a function that will create the proper configuration file for Tailwind to use, based on a combination of user-provided and default settings.

By this point, we've covered the parts of Tailwind that create context in which the PostCSS plugin can exist. Now, with the processTailwindFeatures function, let's look at the "meat and potatoes" of the repository.

Processing Tailwind Features

The processTailwindFeatures function is where styles and configuration combine to create a stylesheet. Because the perfectionist plugin accepts a stylesheet as its input, we know that what is returned from processTailwindFeatures is a PostCSS plugin that returns a string containing our CSS rules.

Let's take a look at that function now:

    export default function(getConfig) {
      return function(css) {
        const config = getConfig()
        const processedPlugins = processPlugins([...corePlugins(config), ...config.plugins], config)

        return postcss([
          substituteTailwindAtRules(config, processedPlugins),
          evaluateTailwindFunctions(config),
          substituteVariantsAtRules(config, processedPlugins),
          substituteResponsiveAtRules(config),
          substituteScreenAtRules(config),
          substituteClassApplyAtRules(config, processedPlugins.utilities),
        ]).process(css, { from: _.get(css, 'source.input.file') })
      }
    }

At a glance, we can outline four major steps happening here:

Using the parameter passed to processTailwindFeatures (ie. getConfigFunction) the configuration file is retrieved.
With the config in hand, the core Tailwind plugins are combined with any user-defined plugins using the processPlugins function to create a PostCSS AST of our Tailwind styles.
That AST is then passed into a PostCSS plugin chain. Each step of that chain uses the config and the AST to incrementally create a fully-formatted CSS output, complete with responsive rules, variants, and components built with Tailwind's @apply directive.
Finally, the output of the PostCSS plugin chain is processed and returned as a CSS file using the process method.

We've already covered the basics of step #1, so we won't go over it again here except to remind ourselves that the return value of getConfig is an object containing our final configuration.

Step #2 is where things start to get interesting. There are two functions to consider here. corePlugins handles the loading of all the Tailwind default plugins, while processPlugins transforms all core and user-defined plugins into a PostCSS AST for use within the PostCSS plugin chain.

Let's look at corePlugins first:

    export default function({ corePlugins: corePluginConfig }) {
      return configurePlugins(corePluginConfig, {
        preflight,
        container,
        appearance,
        backgroundAttachment,
            // ... the rest of Tailwind core here
        zIndex,
      })
    }

We can see that corePlugins does two things:

It loads all core plugins from the plugins directory.
It applies the corePlugins property from our config to configure each core plugin using configurePlugins.

The configurePlugins method is also quite simple:

    export default function(pluginConfig, plugins) {
      return Object.keys(plugins)
        .filter(pluginName => {
          return pluginConfig[pluginName] !== false
        })
        .map(pluginName => {
          return plugins[pluginName]()
        })
    }

Basically, what this does is remove any core plugin that the user has specifically disallowed within their configuration. So, if I decided not to include any padding styles within my final Tailwind CSS file, I could add something like this to my configuration:

    {
        corePlugins: {
            padding: false
        }
    }

Keep in mind that the comparison is done using strict equality, ie. !== vs !=. Because undefined !== false, this means that no plugins will be excluded unless explicitly excluded in user config. By default, all plugins are included, as the configuration corePlugins property defaults to an empty object.

Next, we turn to the processPlugins function:

    export default function(plugins, config) {
      const pluginBaseStyles = []
      const pluginComponents = []
      const pluginUtilities = []
      const pluginVariantGenerators = {}

      const applyConfiguredPrefix = selector => {
        return prefixSelector(config.prefix, selector)
      }

      plugins.forEach(plugin => {
        plugin({
          postcss,
          config: (path, defaultValue) => _.get(config, path, defaultValue),
          e: escapeClassName,
          prefix: applyConfiguredPrefix,
          addUtilities: (utilities, options) => {
            const defaultOptions = { variants: [], respectPrefix: true, respectImportant: true }

            options = Array.isArray(options)
              ? Object.assign({}, defaultOptions, { variants: options })
              : _.defaults(options, defaultOptions)

            const styles = postcss.root({ nodes: parseStyles(utilities) })

            styles.walkRules(rule => {
              if (options.respectPrefix) {
                rule.selector = applyConfiguredPrefix(rule.selector)
              }

              if (options.respectImportant && _.get(config, 'important')) {
                rule.walkDecls(decl => (decl.important = true))
              }
            })

            pluginUtilities.push(wrapWithVariants(styles.nodes, options.variants))
          },
          addComponents: (components, options) => {
            options = Object.assign({ respectPrefix: true }, options)

            const styles = postcss.root({ nodes: parseStyles(components) })

            styles.walkRules(rule => {
              if (options.respectPrefix) {
                rule.selector = applyConfiguredPrefix(rule.selector)
              }
            })

            pluginComponents.push(...styles.nodes)
          },
          addBase: baseStyles => {
            pluginBaseStyles.push(...parseStyles(baseStyles))
          },
          addVariant: (name, generator) => {
            pluginVariantGenerators[name] = generateVariantFunction(generator)
          },
        })
      })

      return {
        base: pluginBaseStyles,
        components: pluginComponents,
        utilities: pluginUtilities,
        variantGenerators: pluginVariantGenerators,
      }
    }

Now, while this function might look like a doozy, it's actually not as bad as it looks. More importantly, there's a reason why everything is stacked together in one function instead of being split up into separate functions.

We'll get to the plugins.forEach loop in a moment, but to understand why this loop is structured as it is, let's take a quick look at the applyConfiguredPrefix function:

    const applyConfiguredPrefix = selector => {
        return prefixSelector(config.prefix, selector)
    }

There are two things to notice here that together help explain the following plugins.forEach loop. The first is that, to use the formal definition, applyConfiguredPrefix is a function expression, not a function declaration. Less formally, the function takes the form of

    const functionExpression = function() {
        // your function here
    }

And not the form of:

    function functionDeclaration() {
        // your function here
    }

If you're new to JavaScript, or coming from another programming language, this distinction might seem arbitrary and confusing. While I agree that the syntax could probably be a bit clearer, there is a specific reason for this distinction, and it has to do with the second thing we should notice about applyConfiguredPrefix. Specifically, we should note that, although the function uses config.prefix, the only argument that the function accepts is selector. Nowhere inside the function is config defined, yet we are able to use it just the same. Why is that?

The answer has to do with the way the JavaScript engine interprets JavaScript code when executing it. Essentially, two things happen in order:

All function declarations are "hoisted", making them available to the rest of your code. This means that you could declare a function at the end of your code and use it at the beginning of your code without a problem.
All remaining code is executed top-to-bottom, including function expressions.

What this means in context is that, because applyConfiguredPrefix is a function expression defined within processPlugins, any variables that are accessible to processPlugins by the time applyConfiguredPrefix is defined are also accessible within applyConfiguredPrefix. Because our config is passed into processPlugins as a parameter, it can be used without being specifically passed into applyConfiguredPrefix.

By contrast, had a function declaration been used instead, the function would have looked like this:

    function applyConfiguredPrefix(selector) { // because config is not passed in explicitly...
        return prefixSelector(config.prefix, selector) // this would have thrown an error!
    }

Because this function would have been "hoisted", we would not have had access to config unless we explicitly defined it as a parameter.

Confusing? I know it was for me when I started. This is one of those JavaScript features that, while powerful, can be a bit hard to parse even for experienced developers. I started my web development journey with PHP, and while the language does have its warts, I personally believe it handles this scenario a bit more directly. Such a function in PHP would have looked like:

    function applyConfiguredPrefix($selector) use ($config) {
        return prefixSelector($config->prefix, $selector);
    }

You can see specifically which variables this function depends on because they are defined in the use clause, which to me is far less confusing. But, I digress.

To see why this distinction between expressions and declarations is so important here, let's return to our plugins.forEach loop.

On a surface level, what's happening is that every plugin in Tailwind, whether defined in core or by the user, is invoked with the same parameter: an object with various methods that the plugin can use.

We see that virtually all of the methods defined on this parameter are function expressions, such as the config method:

    {
        // previous methods
        config: (path, defaultValue) => _.get(config, path, defaultValue),
        // ... the rest of the plugin methods
    }

Here, the colon indicates that this is a function expression. If it were a function declaration, it would instead be defined like this:

    {
        config(path, defaultValue) { 
            return _.get(config, path, defaultValue) // error: config is undefined
        },
    }

Because an expression is used instead of a declaration, config can be referenced just as it was in applyConfiguredPrefix.

Now, at this point you might be wondering: why go to all this trouble to avoid passing in another parameter? Wouldn't it be easier just to pass config into this method explicitly?

In this case, since we are simply reading from config and not editing it, this might be true. However, to see the true utility of function expressions, let's take a look at another one of the methods: addUtilities.

    const pluginUtilities = []
    // ... within plugin.forEach loop:
    {
        addUtilities: (utilities, options) => {
        const defaultOptions = { variants: [], respectPrefix: true, respectImportant: true }

        options = Array.isArray(options)
          ? Object.assign({}, defaultOptions, { variants: options })
          : _.defaults(options, defaultOptions)

        const styles = postcss.root({ nodes: parseStyles(utilities) })

        styles.walkRules(rule => {
          if (options.respectPrefix) {
            rule.selector = applyConfiguredPrefix(rule.selector)
          }

          if (options.respectImportant && _.get(config, 'important')) {
            rule.walkDecls(decl => (decl.important = true))
          }
        })

        pluginUtilities.push(wrapWithVariants(styles.nodes, options.variants))
      },
    }

Before parsing the rest of the method, let's look at the final line, where the method's results are pushed into pluginUtilities. Remember that the pluginUtilities array is defined before the plugin loop. Because addUtilities is a function expression that occurs after pluginUtilities is defined, it has access to the pluginUtilities array. Importantly, this means that it can also change the value of pluginUtilities.

Altering the array in this manner would not be possible if pluginUtilities was instead passed in as a parameter. Because all function declarations have their own scope, any changes made to the array within addUtilities would be discarded when the method stopped executing, leaving the original array unchanged.

Whew! With that out of the way, let's look at the function itself, shall we?

We see that the following actions are happening:

An object of default options is created.
We check the user-provided options passed into the method. Are the options an array?
1. If so, the options parameter is an array of supported variants. Create a new object with our default options, and replace the default variants with the user-provided variants array.
2. If not, the parameter is instead a full options object. Merge this object with the defaults using Lodash's defaults function.
Using PostCSS's root method, we create a PostCSS AST from the result of applying the parseStyles function to the provided utilities.
We walk over the rules of the newly-created PostCSS AST, applying prefixes and adding important declarations as necessary.
As mentioned before, we push the resulting AST onto the pluginUtilities array alongside any variants specified in the options object.

In summary, whatever utilities are passed to addUtilities are parsed with PostCSS and modified by the default options, as well as any options passed by the plugin itself.

To further contextualize this method, let's look at how it's used by one of the core plugins. We'll pick the display plugin, as it's a simple plugin defining widely-used CSS styles:

    export default function() {
      return function({ addUtilities, config }) {
        addUtilities(
          {
            '.block': {
              display: 'block',
            },
            '.inline-block': {
              display: 'inline-block',
            },
            '.inline': {
              display: 'inline',
            },
            '.flex': {
              display: 'flex',
            },
            '.inline-flex': {
              display: 'inline-flex',
            },
            '.table': {
              display: 'table',
            },
            '.table-row': {
              display: 'table-row',
            },
            '.table-cell': {
              display: 'table-cell',
            },
            '.hidden': {
              display: 'none',
            },
          },
          config('variants.display')
        )
      }
    }

The plugin itself doesn't contain much logic, instead delegating to the processPlugins methods to handle most of its functionality.

The addUtilities method is invoked with two arguments. The first is the object containing key/value pairs representing all styles that should be added as part of the plugin. The second is the options object, which in this case is pulled directly from the configuration key variants.display.

You might have noticed a contradiction in how I've described PostCSS versus how it's used in this case. When describing PostCSS originally, I said that it accepted a CSS stylesheet and converted that stylesheet into a JavaScript AST. However, we see here that the first argument passed to addUtilities, which is converted to an AST using PostCSS's process method, is not a stylesheet but an object. What gives? Am I snickering to myself, relishing in how my long-running deception has finally come full circle?

Fear not, dear reader. I would not lead you this far into the rabbit hole only to bamboozle you now. If I am snickering, it is only because as I write this, my cat has just tackled my unsuspecting mini Daschund like a safety pile-driving a wide receiver.

The answer lies within the parseStyles method, which eventually delegates to the parseObjectStyles function:

    import _ from 'lodash'
    import postcss from 'postcss'
    import postcssNested from 'postcss-nested'
    import postcssJs from 'postcss-js'

    export default function parseObjectStyles(styles) {
      if (!Array.isArray(styles)) {
        return parseObjectStyles([styles])
      }

      return _.flatMap(
        styles,
        style => postcss([postcssNested]).process(style, { parser: postcssJs }).root.nodes
      )
    }

In particular, the last line is what interests us. We've seen PostCSS's process method before, but what we haven't seen is the second argument, an options object which here specifies a custom parser: postcssJs. With this parser downloaded from NPM and configured in our processor, we can take a JavaScript object formatted like the object in the display plugin and turn it into an PostCSS AST as if it were a stylesheet.

When all is said and done, the processPlugins function returns an object containing four PostCSS ASTs:

base
components
utilities
variantGenerators

These ASTS are then used in the PostCSS plugin chain. The resulting ASTs are combined and compiled into a stylesheet, cleaned up by Perfectionist, and written to your project's CSS file, ready to help craft your beautiful and semantic websites.

Summing Up: Tailwind Patterns and Structures

We've covered a lot of ground in this analysis. We've hopefully learned something about Tailwind and PostCSS, and maybe we've learned a thing or two about JavaScript along the way.

There are a couple functions I've left off this analysis. In particular, resolveConfig and the entire processTailwindFunctions PostCSS plugin chain remain unparsed, as do some of the more complex Tailwind plugins included in core.

But even leaving those loose ends for the next chapter, we've still managed to uncover some of the more prevalent patterns used throughout TailwindCSS. Let's go through some of them now:

PostCSS

Though you probably knew already that TailwindCSS was a PostCSS plugin, it may have surprised you to find out how deeply PostCSS is integrated into the plugin. At virtually every depth, PostCSS functions are used to create and compose plugins, as well as parse and navigate ASTs. Accordingly, TailwindCSS makes heavy use of the tree structures created by PostCSS in order to figure out where and how to build its stylesheet output.

Functional Programming

One pattern we didn't discuss was the use of functional programming throughout TailwindCSS. You'll notice the codebase contained no classes and no inheritance, either class-based or prototypal. Instead, in keeping with the PostCSS API, which heavily emphasizes function-based code structures, virtually all of Tailwind's functionality is encapsulated within functions. Furthermore, the use of Lodash, a utility library emphasizing functional programming through the use of function chaining and common higher-order functions, solidifies TailwindCSS as adhering to a functional programming style.

It is important to note, however, that the functions used in Tailwind weren't exactly pure, for reasons we'll talk about in the next section.

Function Expressions

We noted a couple instances in which function expressions were used in place of function declarations. Function expressions are a good way of leveraging the power of functions while maintaining state at a high level. By binding top-level variables into localized functions, TailwindCSS is able to accomplish things such as the plugin processing loop, where many plugins are handled in a functional way without needing to resort to imperative and sometimes-clunky for loops.

As mentioned above, this does mean that Tailwind's functional programming is not "pure" in a strict sense. Pure functional programming means simply that functions only accept inputs and return outputs. Notably, pure functional programming does not allow for the use of "side effects", or modifying state that is not returned. We saw that the plugin processing loop breaks this rule, as the top-level arrays were defined outside the loop and modified in the functions defined within the loop.

In general this is not a bad thing, and one of the primary strengths of JavaScript is its ability to support multiple different styles of programming in one language. The primary drawback of nonpure functions is that the programmer needs to take extra care to ensure that state is not modified in unexpected ways. Assuming this is done, and everything I saw in the codebase assured me that it is, the debate about functional purity is more academic than consequential in a real-world sense.

Signing Off

That's it for this chapter of Parsing Open Source! Chapter 2 of TailwindCSS is forthcoming. I have plans to cover GatsbyJS and Laravel next, but do let me know if there are any open source libraries you'd like to see analyzed. I write primarily PHP and JavaScript, but I relish the opportunity to dive into new languages and repos. I'm here and on Twitter @mariowhowrites.

But don't request anything in Python. Indentation as syntax is evil and I won't stand for it. Don't @ me.

Want to Work as a Web Developer? Study Writing

Mario Vega — Tue, 02 Apr 2019 23:15:07 +0000

From a business perspective, March has been one of the worst months since I started freelancing. Projects falling through, clients not paying, clients ghosting after not paying, proposals rejected: you name it, March had it.

And I couldn't be happier.

That's not sarcasm, and I'm not a masochist. Looking at what happened over the last month, the most obvious takeaway is that analyzing my failures offers a blueprint for how to improve in the future. Better pre-project contracts, more vetting of clients (no more Craigslist gigs), a better defined personal brand/website, and so on.

But there's a second reason, adjacent but unrelated to my business woes, why March has been a good month for my mind and a bad month for my wallet.

Here's why I'm happy: on the technical side, I feel as if I'm doing the best work I've ever done. My APIs are tested and documented. My initial solutions to the problems I encounter are no longer wildly off the mark -- some even work on the first try. I finally feel as if I'm not running like a gerbil from framework to framework, throwing packages at my problems in hopes of stumbling upon a working codebase. And, for the first time in my career, I'm able to (occasionally) solve problems without needing to spend hours to research and compare potential solutions.

All of this is miles further than where I was even a year ago, and certainly further than when I first started building websites.

You may ask what any of this has to do with writing. Simple: until about four years ago, I studied writing and writing alone.

By the time I graduated college, the only coding I had done was in an introductory computer science course in high school, where I spent much of my time playing Warcraft III. I began my journey into web development when I couldn't figure out how to expand a sidebar on a WordPress site I ran. "There are people right now who are curing cancer," I told myself, "and you can't figure out how to move a sidebar on your website." Before then, my only experience writing code in a public context involved customizing my Myspace profile in middle school.

So my first major point is that you can succeed and thrive in tech without a background in computer science. But you might have guessed that already -- others here and elsewhere have told this story, many with longer odds and less technical background than myself.

However, while it may be provably possible to work in technology without making it the focus of your studies, I haven't seen as many arguments that it is preferable to do so. After all, starting a career in development requires a wide body of domain-specific knowledge. Wouldn't it be easier to start your career with that knowledge already in hand?

I believe this assumption is incorrect, and that skills commonly taught in the humanities transfer equally well to technology careers when compared to skills acquired in computer science. I have three primary reasons for believing so:

The humanities teach you how to learn -- and tech careers require lots of learning.
Good code and good writing ultimately look alike: they both strive for accessibility and concision.
While technology may be written with code, it is ultimately written for humans.

Technology as a Learning Process

In college, I studied two main subjects: history, specifically Latin American history, and writing, specifically screenwriting. Along the way, I took classes in everything from planetary geology to Assyriology to Russian science fiction. These subjects sounds entirely disparate, and in many ways they are. Starting a new class was again and again an introduction to a new world. Each had its own lexicon, its own luminaries, its own context that needed learning - and quickly.

In immersing myself in these new worlds of thought, I had no choice but to discover and refine my personal strategies for learning. Take notes often; write outlines whenever possible; ask questions of those who know more than you; do your homework. All of these skills have directly translated to my career in software development.

At this point in the conversation, I've had others say: wouldn't all of that learning not be necessary if you had studied computer science in college?

This claim does have a certain merit. As I mentioned before, there are numerous aspects of development that confound me to this day. I'm just now beginning to stumble my way through common algorithms and data structures. I learned how to provision my own servers reluctantly, and even after learning I default to platform-as-a-service solutions (Heroku, Zeit Now, etc.) whenever possible.

However, there are two main reasons why this tradeoff has been acceptable to me. First, from the limited anecdotal experience I've picked up via computer science grads, I've learned what is taught in the classroom doesn't always align with what's necessary to build software in real-world contexts.

I distinctly remember that one of the requirements for a computer science at my school was a course with the name "Build an Operating System". As illuminating as it might be to build an operating system, I can promise you that no website you build will require its own operating system.

Second, even if your comp sci education prepares you well for a job out of college, technology is a constantly-evolving field. Paradigms and frameworks change over time, new challenges arise, each opening up a range of new solutions. Perhaps most interestingly, the various disciplines and subfields of technology have a fascinating way of bleeding into each other. By imparting a generalized approach to learning, the humanities will give you the ability to keep pace with new tech as it enters the mainstream.

Good Code vs. Good Writing

As I grow in my ability to read and write code, I continually find new ways in which the arts of good code and good prose overlap. Both disciplines value structure, clarity and accessibility first and foremost.

There are many good resources that cover the basics of writing clear code, and I won't go over them in depth here. Instead, I'd like to point to some less-considered aspects of development that closely mirror the writing workflow.

One major part of the dev experience that's often left out of education is understanding/editing code written by someone other than yourself. The vast majority of learning experiences I've encountered offer students an empty page to craft and hone their solutions in an isolated environment. By contrast, when writing code for real-world scenarios, it's incredibly rare to build a solution without using any external code. Whether you're downloading code from a package repository or inheriting a legacy project that dozens of developers have worked on before you, your ability to work with existing code is oftentimes just as important, if not more so, than your ability to write new code.

Yet, because the skills required to parse and review external code are so infrequently taught, they end up never being used at all. I've worked with developers many years my senior who have asked me to review codebases for them -- not because they didn't know how to code but because the context switching required to parse someone else's code took too much time and effort to be worth it.

By contrast, because I had focused on the writing and editing process during my time in school, I was far more comfortable jumping into others' code from an early point in my career. And because good code resembles good writing in many ways, I've been able to make meaningful improvements to open-source code without any formalized development education.

Will I be the one to improve a codebase's performance 10x by my deft knowledge of algorithmic complexity? Likely not. But here's something I wished I had known from the start: performance isn't everything. I spent far too much time during my early days double-checking my code against every performance-squeezing resource I could find, worried that my lack of comp sci knowledge would lead me to write ineffective code and cripple my websites in the process.

This brings me to another parallel between the development and writing worlds: the tradeoff between complexity and accessibility. I've got a lot to say about this particular subject, and it's a topic I've planned to write about in a future column. But for now, here's a simple-ish metaphor to illustrate my point.

I spent a substantial portion of my education in writing workshops. In these workshops, a small group of writers would present our responses to a draft we'd received the week before. Without giving any context or explanation, we'd read our stories and receive constructive feedback from the other writers in the group. The "no context" part is important here -- without a window into the author's mind, we were forced to evaluate each story at face value.

This process was hugely helpful to my writing in several ways. The first was realizing that I had a tendency to depart from the real world as quickly as humanly possible. If the prompt was to describe a metropolitan bus ride, I'd find a way to have the bus make a wrong turn at Albuquerque and steer into an alternate dimension.

The second was noticing just how different that approach was from the methods of my fellow writers. With few exceptions, their stories were much more grounded. Their characters fell asleep and missed exits, mumbled awkward conversations into their phones to avoid eavesdropping passengers, and generally did the sorts of things one might expect people to do on buses.

The third thing I picked up on, however, is that the subject matter of a given story had very little to do with how easy it was to read and understand. While I'd often get comments on how weird my stories were, or guesses as to what sorts of drugs I must have ingested prior to writing them, other writers in my workshops were rarely in doubt as to what happened in the worlds I created. By contrast, other writers in my workshops would take greater risks than I when choosing how to order and present their stories. Sometimes these risks would pay off magnificently. I'd marvel at the way they could present an ordinary situation with and sometimes they'd leave everyone scratching their heads as to what, exactly, the story was trying to convey.

Bringing things back to development, I've noticed contrasts in coding styles that remind me of the divisions I saw in my writing workshops. Some developers value terseness above all else, and will avoid writing two lines of code where the opportunity exists to write one instead. Some developers live and breathe performance, and make sure that every line of code can scale exponentially from the very start. And some developers do neither of these things, and write code to solve problems and nothing more.

As much as I'd like to say that I'm in the third camp, the reality is that I draw from all three approaches. But I always strive to keep my code as accessible as my fiction -- and considerably more grounded. And without knowing how important it was to write something that others could understand and build upon, that lesson in sustainable development would have been a lot harder to internalize.

Writing Code for Humans

This is, again, a topic I have thought about at considerable length, and may or may not be expanded into a column of its own at some point. This present column has already run longer than anticipated, so I won't dive too far into extended and ham-fisted metaphors to make my point.

All code is meant for humans. Yes, computers are responsible for interpreting and acting on code, but that computer-interpreted code is ultimately written to advance some human end. The better you can define the relationship between code and the humans who benefit from it, the better positioned you will be to succeed in your career. And defining relationships between disparate topics is at the heart of the humanities.

If you can explain to freelance clients how the code you're writing can benefit them, you are more likely to win contracts than a developer who can code but cannot explain. If you can explain to your fellow developers why a given refactor is in the best interests of the team, you are more likely to have your changes approved and implemented than a developer who can code but cannot explain. If you can explain how the concepts you learn in bootcamps and tutorials apply to real-life scenarios, you are more likely to build audiences and gain employer interest than a developer who can code but cannot explain.

All of the tasks above are made easier if you know how to convert evidence into theses and actionable ideas. And to do that, you need to know how to write.

I haven't even begun to touch on the field of ethics in the technology sector, and how studying the humanities can equip you with a much more refined moral compass than studying the amoral world of computers. So please, if you're from a similar background to myself and you're wondering whether there's a place for you in technology, put those thoughts to rest for good. There isn't just a place for you -- there's a need for you.