DEV Community: Dylan Paulus

Easily Generate Mock Data with PostgreSQL

Dylan Paulus — Fri, 10 Nov 2023 16:51:59 +0000

Original Post: https://www.dylanpaulus.com/posts/easily-generate-mock-data-with-postgresql

When writing articles about PostgreSQL there are times I need to generate large swaths of data to test and demonstrate a topic (for example, these Timescale articles here and here).
It can be a pain to manually insert data or create a CSV and import it into PostgreSQL--which I always have to StackOverflow.
Lately, I ran into a much simpler solution to generating data: the generate_series function.
It's perfectly suited for integers and timestamp data types, streamlining various tasks such as populating test databases or setting up date ranges for reports.

Generating Mock Data

generate_series can be thought of as a for-loop or generator to produce a range of numbers or dates.

To generate a range of numbers, we call generate_series with a start and end value:

SELECT * FROM generate_series(1, 10);

We can specify a step value by providing generate_series as an optional third argument.

SELECT * FROM generate_series(1, 10, 2.5);

Inserting Mock Data

generate_series' power comes to fruition when we need to insert mock data into a table.
We can easily insert a range of numbers into a table by using generate_series in a SELECT statement and then inserting the results into a table.

For the rest of this article, we'll use users table that looks like this for our examples:

CREATE TABLE users (
 id serial PRIMARY KEY,
 name text NOT NULL,
 created_on timestamptz
);

Then, if we wanted to insert five users into the table we could run:

INSERT INTO users (name)
SELECT
    'Dylan'
FROM generate_series(1, 5);

This pattern can be used to insert any number of rows into a table.
If you're feeling brave, try inserting 1,000,000 users into the table by changing generate_series(1, 5) to generate_series(1, 1000000)!

Up to this point we are creating a lot of duplicate data.
What if we wanted to create a bunch of users with all distinct names?
We can access the current index of the loop (using as) to generate a unique name for each user.

That would look like this:

INSERT INTO users (name)
SELECT
    'Dylan number ' || i
FROM generate_series(1, 5) as i;

Creating Time-Series Data

generate_series is incredibly effective at generating time series data.
This is where I get the most use out of it because it can become extremely tedious to create a lot of data around time ranges.

Similar to the integer example, generate a range of timestamps by providing a start and end value followed by an optional step or interval.

For example, to generate a user for every hour in a week, we could run the following SQL:

INSERT INTO users (name, created_on)
SELECT
    'Dylan',
    time_hour
FROM generate_series(
    TIMESTAMPTZ '2023-11-01', 
    TIMESTAMPTZ '2023-11-07', 
    INTERVAL '1 hour'
) as time_hour;

This query creates 146 rows with a user named Dylan every hour between November 1st and November 7th, 2023.

You may be asking yourself, can we use multiple generate_series to populate multiple fields? Yes!
But, we need to use joins to combine the data from the two series.
To quickly create a grid of users and created_on dates we can take the cartesian product of two series, otherwise known as a cross join, by adding another generate_series to the FROM clause.

INSERT INTO users (name, created_on)
SELECT
    'Dylan number' || i,
    time_hour
FROM 
    generate_series(1, 5) as i,
    generate_series(
        TIMESTAMPTZ '2023-11-01', 
        TIMESTAMPTZ '2023-11-07', 
        INTERVAL '2 days'
    ) as time_hour;

In the INSERT statement above, we'll create four time_hour timestamps for every two days in the week for every iteration of i (there are five of them).
Resulting in 4 * 5 = 20 twenty rows created.
To better grasp what's happening here, it'll be easier to look at the SELECT output of the users table.

Using multiple generate_series functions combined with a cartesian product can be a powerful combination for creating unique, time-series data.

Conclusion

PostgreSQL's generate_series is a game-changer for developers like me, who frequently dive into the deep end of data generation for testing and optimizing queries. It's a powerhouse that can effortlessly create anything from a handful to millions of rows, transforming a potentially arduous task into a few simple lines of SQL.
So, when you need to mock up data, let generate_series be your go-to solution.

Happy data generating!

Unraveling Graph Structures: Exploring Adjacency Matrices

Dylan Paulus — Thu, 20 Jul 2023 15:59:40 +0000

Originally posted at dylanpaulus.com

In Postgres: The Graph Database You Didn't Know You Had we covered how we can store a DAG (Directed Acyclic Graph) in a relational database.
We looked at one representation of a graph data structure called an adjacency list--a table with two columns representing how nodes connect.
The columns store connections from the source or "from" node to a destination or "to" node.

Our Graph Adjacency List

Edges
Source	Destination
A	B
A	C

Using Python, a code representation would look like: ```python class Node: pass A = Node() B = Node() C = Node() adjacency_list = [ { "source": A, "destination": B }, { "source": A, "destination": C }, ] ``` An adjacency list is a simple and efficient way to store graph data, but there are many more ways to represent graphs. In this article, we'll explore one of these alternatives called the adjacency matrix. ## Adjacency Matrix An adjacency matrix is a table with a row and column for each node in the graph (or NxN matrix). To represent the edges we store a 1 in a column for a connection and a 0 for no connection. We will start out exploring how to represent DAG's as an adjacency matrix since we covered DAG's in the Postgres article. To store _direction_ in an adjacency matrix the x-axis or rows represent the "from" or "source" node. The y-axis represents the connection. Using the same graph as above, the adjacency matrix would look like this: Our Graph Adjacency Matrix

	B	C
A	1	1
B	0	0
C	0	0

```python adjacency_matrix = [ [0, 1, 1], [0, 0, 0], [0, 0, 0] ] ``` ### What about non-DAG graphs? For undirected graphs, graphs where all connections are bidirectional, the setup is the same to create the adjacency matrix. The only difference is our graph has no explicit _direction_. If `A` connects to `B`, then `B` also connects to `A`. Let's take our previous DAG graph, and remove the direction. Our Graph Adjacency Matrix

	A	B	C
A	0	1	1
B	1	0	0
C	1	0	0

A cool property emerges from undirected graphs. Can you spot a pattern in the table above? Draw a diagonal line from the top left of our matrix to the bottom right (sandwich square style) and fold the matrix along this line.

The connections mirror each other across this line. Our matrix is considered symmetric when this occurs.
Symmetric matrices appear in all undirected graphs, but not DAGs!

This is great, but why would we use an Adjacency Matrix?

Matrices are incredibly fast at checking if two nodes are connected

Instead of having to loop through all the edges in an adjacency list to find if two nodes are connected, an adjacency matrix acts
as a hash table. In our example graph above, if we wanted to check if A has an edge to B it would be as simple as checking the [B][A] position
in the matrix.

Note: Since we're working with DAG's and 2D arrays, the order we look up a connection is flipped: [Destination][Source].

A = 0
B = 1

has_connection = adjacency_matrix[B][A]

Removing and adding a connection between nodes is similar and run in constant time.
Need to connect two nodes? Modify the position in the matrix to be 1 (adjacency_matrix[destiation][source] = 1).
What about deleting an edge? Update the value to be 0 (adjacency_matrix[destiation][source] = 0)!

Why would I not use an Adjacency Matrix?

They're huge!

If we look at an adjacency matrix we need to store every bit of information about the graph, the connected nodes (1's), and not connected nodes(0's).
Compared to an adjacency list which stores only the connections; adjacency matrices take up a lot of space!

It's expensive to add/remove nodes

We talked about how adjacency matrices are fast at looking up, adding, and removing connections between nodes.
On the flip side, adjacency matrices are slow when adding or removing nodes from the graph.
To add a node we need to copy our matrix to a new matrix with an additional row and column(of size N+1 x N+1).
Then, populate the matrix with the connections to or from the new node.
To compare, adding a new node to an adjacency list is a matter of pushing a new node to the list of nodes.

End

There are many different ways to represent and visualize a graph.
A different representation may be better based on your personal use cases and requirements.
Need to quickly look up which nodes are connected or need to quickly add or remove edges? Then the adjacency matrix may be the best choice.
Are you constantly adding nodes to the graph and are limited in your memory footprint? The adjacency list is the representation for you!

Improve Your Prompts for LLMs: Simple and Effective Techniques

Dylan Paulus — Wed, 07 Jun 2023 16:17:19 +0000

Tools like ChatGPT and Bing have been skyrocketing in popularity.
ChatGPT took five days since launch to reach 1 million users.
Governments all over the world are trying to figure out what to do with this technology.
Sweet AI-generated images are showing up everywhere; like this dog I generated using Bing:

"Can you generate a picture of an astronaut dog jumping over a fire on the
moon?"

Though LLMs (Large Language Models) are still in their infancy, research has shown some pretty big productivity boosts in a range of different professions.
Using metrics from Github Copilot, an AI-based coding tool, Github has found that developers complete tasks in 55% less time when taking advantage of AI.

This is all to say that LLMs are a tool we'll need to learn to use and use well.

There are many options when choosing a LLM, with the most popular being ChatGPT and Bing.
OpenAI's ChatGPT is the most well-known, but if we want to use the latest features like ChatGPT-4 (an upgrade to ChatGPT-3) we'll need to pay $20 a month.
Microsoft has released new Bing that takes advantage of ChatGPT-4 behind the scenes and is free to use.
Finally, Google has Bard which is also free to use.
All offerings come with their unique pros and cons.

Regardless of which tool you use, the question remains: how do we get the most out of these tools?

Prompt Engineering

We interact with LLMs by giving them a prompt, or text input, generally in the form of a question or instruction.
LLMs then reason out the answer and respond.
Prompt Engineering is a term coined to describe the process of writing prompts to get the best results from LLMs.
Just like in everyday conversation, the words we choose to ask in a question can dramatically change the response we get.
For example, if I ask “What is the best way to get to the airport?” I’ll get a different response than asking “How do I get to the airport?“.

Throughout this article we’ll build on a prompt to create a meal plan for a single day; taking advantage of different Prompt Engineering techniques.
I’ll provide the prompt used, and the response I got back from the LLM—in my case I’ll be using ChatGPT-3.
Try it for yourself, but remember that the results will vary by the LLM being used.

Prompt

Please create me a low sugar meal plan for one day.

🤖 (LLM) Response

Low-Sugar Meal Plan for One Day
Breakfast

Scrambled eggs with vegetables (spinach, bell peppers, mushrooms)

Whole grain toast

A small serving of fresh berries (such as strawberries or blueberries)

Herbal tea or black coffee (unsweetened) Snack

Greek yogurt with a sprinkle of cinnamon

A handful of unsalted almonds Lunch

Grilled chicken breast

Mixed greens salad with cherry tomatoes, cucumber, and avocado

Olive oil and lemon juice dressing (no added sugar)

Steamed broccoli Snack

Sliced cucumber and carrot sticks with hummus Dinner

Baked salmon fillet with lemon and dill

Quinoa pilaf with mixed vegetables (broccoli, carrots, peas)

Steamed asparagus Dessert

Fresh fruit salad (such as watermelon, cantaloupe, and kiwi) Note: Remember to drink plenty of water throughout the day. Adjust portion sizes according to your specific dietary needs and consult with a healthcare professional or a registered dietitian for personalized advice.

Constraints

Just like people, LLMs need constraints and clarity to understand what's being asking.
Be specific when writing prompts, and have a desired outcome in mind.
The best mental model I've been told is to think of LLMs as an intern, brand new to the job.
They are eager to get stuff done, but need clear and direct instructions.
The best way to get the best results out of LLMs is to be specific.

Let's improve upon our prompt by adding some constraints.

Prompt

Please create me a low sugar meal plan for one day. The plan should be a
maximum 1,800 calories, and do not include breakfast. No gluten. High in
protein.

🤖 (LLM) Response

Low-Sugar, Gluten-Free, High-Protein Meal Plan for One Day
Lunch

Grilled chicken breast (4 oz)

Quinoa salad with mixed vegetables (cucumber, cherry tomatoes, bell peppers) dressed with lemon vinaigrette

Steamed broccoli (1 cup) Snack

Greek yogurt (6 oz) with sliced almonds (1 tbsp) and a sprinkle of cinnamon Dinner

Baked salmon fillet (4 oz) with garlic and herbs

Sautéed kale (1 cup) with garlic and olive oil

Roasted sweet potatoes (1 medium) with a sprinkle of paprika Snack

Celery sticks with almond butter (2 tbsp) Dessert

Mixed berries (1 cup) with a dollop of unsweetened whipped cream Note: Please note that portion sizes and specific calorie counts may vary depending on the size and brand of the ingredients used. Adjust the portion sizes and consult a healthcare professional or a registered dietitian for any specific dietary needs or concerns.

Role prompting, or "Act like a..."

Role Prompting or "Act like a ..." is a technique that can be used to provide more context and constraints to LLMs.
First we tell the LLM to assume a role or persona. Then we provide a prompt.
For example, if we had a question about woodworking we might say, "Act like a master woodworker. What is the best way to cut a dovetail joint?".
Role prompting is another great way to provide constraints to LLMs.

Let's improve upon our meal plan prompt by adding a role.

Prompt

Act as an expert nutritionist with 20 years of experience. Please create me a
low sugar meal plan for one day. The plan should be a maximum 1,800 calories,
and do not include breakfast. No gluten. High in protein.

🤖 (LLM) Response

Low Sugar Meal Plan
Maximum 1,800 Calories | No Gluten | High in Protein
Lunch

Grilled chicken breast (4 oz) seasoned with herbs and spices

Quinoa salad with mixed vegetables (such as bell peppers, cucumbers, and cherry tomatoes) dressed with lemon juice and olive oil

Steamed broccoli (1 cup) - Mixed greens salad with avocado slices and a light vinaigrette dressing

Sparkling water with a squeeze of fresh lemon Snack

Greek yogurt (6 oz) topped with mixed berries (such as strawberries and blueberries)

Handful of unsalted almonds Dinner

Baked salmon fillet (4 oz) marinated in a mix of lemon juice, garlic, and dill

Roasted Brussels sprouts with garlic and olive oil (1 cup)

Steamed asparagus (1 cup)

Quinoa (1/2 cup) cooked with vegetable broth

Herbal tea or water with a slice of lemon Evening Snack

Celery sticks and carrot sticks with hummus

Green tea or water This meal plan is designed to be low in added sugars and gluten-free while providing ample protein from lean sources like chicken, fish, and Greek yogurt. It incorporates a variety of colorful vegetables, healthy fats from avocado and olive oil, and whole grains like quinoa for additional fiber. Remember to drink plenty of water throughout the day to stay hydrated. Please note that this is a generalized meal plan, and it's always a good idea to consult with a registered dietitian or nutritionist who can create a personalized meal plan based on your specific dietary needs and health goals.

Few shot prompting

Few shot prompting is my favorite technique, and can really take the performance of LLMs to the next level.
First we provide the LLM with a few examples (or shots) of what we want, and then we ask the LLM to generate a similar response (furthering adding constraints to our prompt).
The term "shot" is synonymous with "example". In Pompt Engineering you will also find zero shot prompting and one shot prompting.

Zero Shot Prompting: No examples are provided
One Shot Prompting: One example is provided
Few Shot Prompting: Two or more examples are provided

The shots, or examples, can be given in the same prompt, or broken out into separate prompts.

For example, in our meal plan example we could ask the LLM in a single prompt:

Prompt

Use the following examples of a meal plan:
example 1:
Breakfast (364 calories)

1 serving Egg & Avocado Toast with Spinach & Salsa A.M. Snack (59 calories)

1 medium peach Lunch (584 calories)

1 serving Hot Turkey Pesto Sandwich P.M. Snack (200 calories)

1 cup edamame in pods Dinner (306 calories)

1 serving One-Pot Lentil & Vegetable Soup with Parmesan Daily Totals: 1,512 calories, 65g fat, 93g protein, 140g carbohydrate, 34g fiber, 1,538mg sodium To make it 1,200 calories: Omit the morning and afternoon snacks and reduce the pesto from 1 Tbsp. to 1 tsp. on the sandwich at lunch. To make it 2,000 calories: Add 1/2 cup unsalted roasted cashews to A.M. snack and increase to 1 1/2 cups edamame at P.M. snack. --- example 2: Breakfast (302 calories)

1 serving Copycat Spinach and Mushroom Egg Bites

1 slice whole-wheat bread

1 1/4 cups sliced strawberries A.M. Snack (315 calories)

1 medium banana

2 Tbsp. creamy natural peanut butter Lunch (306 calories)

1 serving One-Pot Lentil & Vegetable Soup with Parmesan P.M. Snack (247 calories)

32 unsalted dry-roasted almonds Dinner (349 calories)

1 serving Chicken, Quinoa & Sweet Potato Casserole Daily Totals: 1,518 calories, 67g fat, 75g protein, 158g carbohydrate, 31g fiber, 1,482mg sodium To make it 1,200 calories: Omit peanut butter at A.M. snack and reduce almonds to 18 at P.M. snack. To make it 2,000 calories: Increase to 2 slices bread and 1 1/2 cups strawberries at breakfast, increase peanut butter to 1/4 cup at A.M. snack, and increase almonds to 1/2 cup at P.M. snack. --- Act as an expert nutritionist with 20 years of experience. Please create me a low sugar meal plan for one day. The plan should be a maximum 1,800 calories, and do not include breakfast. No gluten. High in protein.

Or ask the LLM to read a few examples first before we ask our question.
To reduce the amount of time needed for the LLM to repond, we can ask it to confirm once it has read the example.

First Prompt

Read the following meal plan and respond with, "yes" once you read it.
Breakfast (364 calories)

1 serving Egg & Avocado Toast with Spinach & Salsa A.M. Snack (59 calories)

1 medium peach Lunch (584 calories)

1 serving Hot Turkey Pesto Sandwich P.M. Snack (200 calories)

1 cup edamame in pods Dinner (306 calories)

1 serving One-Pot Lentil & Vegetable Soup with Parmesan Daily Totals: 1,512 calories, 65g fat, 93g protein, 140g carbohydrate, 34g fiber, 1,538mg sodium To make it 1,200 calories: Omit the morning and afternoon snacks and reduce the pesto from 1 Tbsp. to 1 tsp. on the sandwich at lunch. To make it 2,000 calories: Add 1/2 cup unsalted roasted cashews to A.M. snack and increase to 1 1/2 cups edamame at P.M. snack.

Second Prompt

Read the following meal plan and respond with, "yes" once you read it.
Breakfast (302 calories)

1 serving Copycat Spinach and Mushroom Egg Bites

1 slice whole-wheat bread

1 1/4 cups sliced strawberries A.M. Snack (315 calories)

1 medium banana

2 Tbsp. creamy natural peanut butter Lunch (306 calories)

1 serving One-Pot Lentil & Vegetable Soup with Parmesan P.M. Snack (247 calories)

32 unsalted dry-roasted almonds Dinner (349 calories)

1 serving Chicken, Quinoa & Sweet Potato Casserole Daily Totals: 1,518 calories, 67g fat, 75g protein, 158g carbohydrate, 31g fiber, 1,482mg sodium To make it 1,200 calories: Omit peanut butter at A.M. snack and reduce almonds to 18 at P.M. snack. To make it 2,000 calories: Increase to 2 slices bread and 1 1/2 cups strawberries at breakfast, increase peanut butter to 1/4 cup at A.M. snack, and increase almonds to 1/2 cup at P.M. snack.

Third Prompt

Using the meal plan examples from before act as an expert nutritionist with 20 years of experience. Please create me a
low sugar meal plan for one day. The plan should be a maximum 1,800 calories,
and do not include breakfast. No gluten. High in protein.

🤖 (LLM) Response

As an expert nutritionist with 20 years of experience, I've created a low-sugar meal plan for one day that is gluten-free,
high in protein, and falls within the maximum 1,800 calorie range.
Please note that individual nutritional needs may vary, so it's always a good idea to consult with a healthcare professional or registered dietitian for personalized advice. Here's the meal plan:
Lunch (approximately 400-500 calories):
Grilled Chicken Salad with Avocado Dressing:

4-6 ounces of grilled chicken breast

Mixed salad greens (spinach, arugula, or your preferred greens)

Sliced cucumber, cherry tomatoes, and red onion

1/4 avocado, sliced

1-2 tablespoons of homemade avocado dressing (made with avocado, lemon juice, olive oil, salt, and pepper) Afternoon Snack (approximately 150 calories): Greek Yogurt Parfait:

1/2 cup of plain Greek yogurt

1/4 cup of mixed berries (such as blueberries and raspberries)

1 tablespoon of chopped nuts (almonds, walnuts, or your preference)

Optional: sprinkle of cinnamon or drizzle of honey (if desired) Dinner (approximately 500-600 calories): Baked Salmon with Quinoa and Roasted Vegetables:

4-6 ounces of baked or grilled salmon fillet

1/2 cup of cooked quinoa

Roasted vegetables of your choice (such as broccoli, bell peppers, and zucchini) seasoned with herbs and olive oil Evening Snack (approximately 150-200 calories): Celery with Almond Butter and Chia Seeds:

2-3 celery stalks

1-2 tablespoons of almond butter (make sure it's free of added sugars)

Sprinkle of chia seeds for added fiber and healthy fats Remember to drink plenty of water throughout the day and adjust portion sizes based on your individual needs and preferences. It's also essential to include regular physical activity and maintain a balanced overall diet for optimal health.

Conclusion

Compare the first meal plan to the final result. After taking advantage of the prompt engineering techniques we learned, we can see that the last meal plan is much more specific, detailed, and closer to the answer we would expect!

Using constraints, role prompting, and shot prompting, we can improve the quality of our prompts, impress our friends, and increase our productivity.

Faster and Easier Python Debugging with breakpoint() and PDB

Dylan Paulus — Wed, 10 May 2023 00:03:48 +0000

Debugging is an essential skill for any software developer.
It allows us to find and fix errors in our code, improve quality and performance, and learn from our mistakes.

One of the most common and powerful tools for debugging Python code is the Python Debugger (PDB).
PDB is a module that provides an interactive command-line interface for inspecting and manipulating the state of a running program.
With PDB, we can set breakpoints, step through the code line by line, examine variables and expressions, modify values, and more.

However, using PDB can also be cumbersome and tedious.
To invoke PDB, we need to import the pdb module and insert a pdb.set_trace() statement in our code where we want to pause the execution and enter the debugger.

import pdb

def add(a: int, b:int) -> int:
  pdb.set_trace()  # pause here and enter the debugger
  return a + b

print(add(1, 2))

This means we have to modify our source code every time we want to debug it and remember to remove or comment out the pdb statements when we are done.
Moreover, if we want to use a different debugger than PDB, such as IPython or PuDB, we have to change the import statement and the function call accordingly.

Fortunately, Python 3.7 introduced a new built-in function that simplifies this process: breakpoint().
The breakpoint() function is a convenient way to start a debugger at any point in our code without having to import anything or write debugger-specific code.
All we have to do is insert a breakpoint() statement where we want to pause the execution and enter the debugger!

In this article, we will learn what breakpoint() is, how to use it with PDB and other debuggers, and how to change its behavior with environment variables.

Using breakpoint()

When using PDB, we need to learn a couple commands to interact with the debugger.
There is a long list of commands PDB gives us, but we only need to know a few of them to get started.
For example, if we want to print values we use the p command.
If we want to execute the next line of code we use the n or next command.
To continue execution until the next breakpoint we use the c or continue command.
Finally, the l or list command shows us the code around the current line we are debugging.

Command	Description
`p [variable_name]`	Print the value of a expression.
`n` or `next`	Continue execution until the next line in the current function is reached or it returns.
`s` or `step`	Execute the current line, stop at the first possible occasion (either in a function that is called or in the current function).
`c` or `continue`	Continue execution, only stop when a breakpoint is encountered.
`l` or `list`	List source code for the current file.
`h` or `help`	Prints a list of commands we can use with PDB.

Let's see how breakpoint() works.
Below is a recursive function that calculates the factorial of a given number.
I've added a breakpoint() statement to the function so we can pause the execution and enter the debugger.

# main.py

def factorial(number: int) -> int:
  if number < 0:
    raise ValueError("`n` must be non-negative.")
  elif number <= 1:
    return 1
  else:
    breakpoint()  # pause here and enter the debugger
    return number * factorial(number - 1)


# factorial(5) is: 5 * 4 * 3 * 2 * 1 = 120
print(factorial(5))

When we run this code, the program will stop at the breakpoint() statement and launch the default debugger, which is PDB by default.
We can then use the PDB commands to inspect and manipulate the program state.

Play with PDB by clicking green "Run" button near the top right of the code editor. If you want to navigate back to the code, click "Show files" at the top left of the code editor and select "main.py".

We can see that breakpoint() pauses the execution at the line where it is called.
By using the commands we learned above, we can inspect the program state (p number to see the value of n) and step through the code line by line (s).

Changing the debugger behavior

One of the advantages of using breakpoint() over pdb.set_trace() is that we can switch to a different debugger at any time.
By setting the PYTHONBREAKPOINT environment variable we can tell Python which debugger we would like to use.
For example, if we want to use IPython as our debugger, we can run:

$ export PYTHONBREAKPOINT=IPython.core.debugger.set_trace
$ python3 main.py

Another advantage of using breakpoint() is that we can disable all breakpoints in our code by setting the PYTHONBREAKPOINT environment variable to 0.
This is useful when we want to run our code without entering the debugger; without having to remove or comment out all the breakpoint() statements.

$ export PYTHONBREAKPOINT=0
$ python3 main.py

End

breakpoint() is a much needed improvement over pdb.set_trace()!
It gives us the ability to quickly debug our code, and easily switch between different debuggers.

Postgres: The Graph Database You Didn't Know You Had

Dylan Paulus — Tue, 28 Mar 2023 18:43:09 +0000

PostgreSQL (Postgres), is a powerful relational database that can store a wide range of data types and data structures.
When it comes to storing graph data structures we might reach for a database marketed for that use case like Neo4J or Dgraph. Hold your horses!
While Postgres is not generally thought of when working with graph data structures, it is perfectly capable to store and query graph data efficiently.

Understanding Graph Data Structures

Before we introduce Postgres as a graph database we need to know what a graph data structure is.
A graph, or a graph data structure, is a collection of nodes and edges,
where each node represents an entity or "thing", and each edge represents the relationship between two nodes.

A visual example of a graph data structure.

To think about graphs in terms of code, we may write TypeScript that looks like this:

class Node {
  edges: Edge[] = [];
  data: string;
}

class Edge {
  previousNode: Node;
  nextNode?: Node;
}

Each node contains a list of its edges, and each edge contains a reference
to the next/previous node. As we'll see in SQL later, the nodes don't always
need to know about their edges.

Facebook is a popular social media platform that uses a graph to represent people and their relationships.
A person can have friends, and those friends also have their list of friends.
Each person is represented as a node, and each friendship would be represented as an edge.
Graphs are used to model a lot of different applications like your
npm dependencies,
workflows, transportation systems, manufacturing lines, and more!

Storing Graph Data Structures in Postgres

To store a graph in Postgres we only need to create two tables: nodes and edges.
The nodes table will store information about each entity, while the edges table will store information about the relationships between entities.

Let's start by creating a nodes table:

CREATE TABLE nodes (
  id SERIAL PRIMARY KEY,
  data VARCHAR(255)
);

The nodes table we defined here has two columns: id and data. The id column is an auto-incrementing integer that serves as the primary key for the table. The data column is a string that stores any extra data associated with the node.
For this example, we're keeping it simple and only storing a string column, but in real-world applications, this table could be anything and have any number of columns.

The most important table when creating a graph data structure is the edges table:

CREATE TABLE edges (
  previous_node INTEGER REFERENCES nodes(id),
  next_node INTEGER REFERENCES nodes(id),
  PRIMARY KEY (previous_node, next_node)
);

Here, we are creating two columns, previous_node and next_node, that represent our relationships between nodes.
Each of these columns represents a foreign key to a node. The important take away is an edges table references two rows in the same table.
An edge can only have one previous_node and next_node pairing, so we are using a composite primary key to ensure that each edge is unique and cannot reference itself.

With our tables created, we can now insert data into them.

INSERT INTO nodes (data) VALUES ('Bob');
INSERT INTO nodes (data) VALUES ('Hank');
INSERT INTO nodes (data) VALUES ('Jeff');

And then let's connect our nodes with edges:

INSERT INTO edges (previous_node, next_node) VALUES (1, 2);
INSERT INTO edges (previous_node, next_node) VALUES (1, 3);

nodes
id	data
1	Bob
2	Hank
3	Jeff

edges
previous_node	next_node
1	2
1	3

If we were to visualize our graph at this point, it would look like this:

Querying Graph Data Structures in Postgres

With our graph data structure created, we can now query it using SQL we know and love!

Want to know who Bob is friends with?

SELECT id, data
FROM nodes
JOIN edges ON nodes.id = edges.next_node
WHERE edges.previous_node = 1;

Find all `nodes` connected to the node with `id` 1 (Bob's id).

Looks like Bob is popular! But what if we want to know who Bob's friends are friends with?

Let's insert a few more nodes and edges to show this:

INSERT INTO nodes (data) VALUES ('Sally');
INSERT INTO nodes (data) VALUES ('Sue');
INSERT INTO nodes (data) VALUES ('Sam');

INSERT INTO edges (previous_node, next_node) VALUES (2, 4);
INSERT INTO edges (previous_node, next_node) VALUES (3, 4);
INSERT INTO edges (previous_node, next_node) VALUES (4, 5);

nodes
id	data
1	Bob
2	Hank
3	Jeff
4	Sally
5	Sue
6	Sam

edges
previous_node	next_node
1	2
1	3
2	4
3	4
4	5

To query for all Bob’s friends of friends we could extend the previous query to join the edges table again, but that would cause a maintenance nightmare having to join on each “level” in the graph.

Postgres has a built-in feature that allows us to query graph data without having to know exactly how many joins we need: recursive queries. Recursive queries allow us to traverse the graph starting from a specific node and following its edges until some determined endpoint.

Writing a recursive query to find all Bob’s friends and their friends we would write the following SQL:

WITH RECURSIVE friend_of_friend AS (
  SELECT edges.next_node
  FROM edges
  WHERE edges.previous_node = 1
  UNION
  SELECT edges.next_node
  FROM edges
  JOIN friend_of_friend ON edges.previous_node = friend_of_friend.next_node
)
SELECT nodes.data
FROM nodes
JOIN friend_of_friend ON nodes.id = friend_of_friend.next_node;

This can be confusing at first glance, so let's break it down.
A recursive query is made up of two parts: the base case and the recursive case.
The base case is where we want to start our query.
The recursive case is the "loop" that will continue to run until some endpoint is reached.

WITH RECURSIVE {name} AS (
  {base case}
  UNION
  {recursive case}
)

The basic SQL structure of a recursive query.

In our example we want to start our query with Bob's friends, so we find the edges where Bob (id: 1) is the previous_node.
Then in the recursive case we continually join the edges table to itself until we reach the end of Bob's graph (eg. when we reach friend_of_friend.next_node = NULL).
Finally outside our recursive query we bring it all together. We need to query the nodes that are associated with the edges from the recursive query so we can get each of Bob's friends' names.

data
Hank
Jeff
Sally

Conclusion

By using features built into Postgres, we can store and query graph data structures.
We used a similar approach in my previous job to dynamically generate work instructions on a manufacturing line.
Based on parameters given, and rules defined on each edge, we could generate the correct document by traversing a graph stored entirely in Postgres.
If you are already using Postgres for your relational data, you can integrate graph data structures into your existing database without adding extra systems!

Git Worktree: Enhance your Git Workflow

Dylan Paulus — Sat, 11 Mar 2023 00:09:35 +0000

Have you ever been in the middle of coding a feature, and out of nowhere you get a notification that production is down?
If you're using Git you now have a few options: commit your work, stash your work, or discard your work.
All of these come with their problems depending on the situation.
If you commit half-baked code you may want to come back later and rebase to clean up your commit history.
Stashing can cause problems, because what if you need to stash additional code later before switching back to your original feature work?
Maybe I'll throw away my current changes if they're small, but otherwise no thanks.

I recently stumbled upon Git worktree, and it has been an amazing boost in my productivity.

What is Git Worktree?

Git worktree allows us to checkout multiple branches in a git repository. This lets us to switch between different branches in a Git repository without losing our non-committed work.
Git does this by creating new, separate directories called a "linked worktree" that is associated with a single "main worktree" (the directory created using git clone or git init).

To checkout a new worktree we use the worktree add command: git worktree add <path> <branch_name>.
Worktrees are created as new directories; we give the add command a path to where we want the new directory created and which git branch we want to checkout.

Just like git checkout -b we can create brand new branches by adding the -b argument (git worktree add -b <branch_name> <path>).

We can see all the worktrees we've created using the git worktree list command.

With our different linked worktrees created, we can safely modify code and switch between isolated instances of our codebase. The changes done in one worktree do not affect the other worktrees.

untitled - asciinema

asciinema.org

What about if we want to commit and push our changes out to Github? No problem! Git worktree is a way to checkout multiple branches.
So once we're done making changes in our worktree we can git add, git commit, and git push origin just like if we are working on a different branch.
The changes would show up on Github under the <branch_name> we gave in git worktree add <path> <branch_name>.

Once we are done with any worktree we can call git worktree remove <path_to_worktree> to delete it.

End

I keep all worktrees in a separate directory ~/.worktree, outside of my general projects directory ~/projects, to reduce clutter and keep "main worktrees" explicit from "linked worktrees".
I am constantly switching between branches and code, and Git worktree has been a huge help to my workflow!

Advice for New Software Engineers

Dylan Paulus — Mon, 04 Jul 2022 20:06:19 +0000

A few months ago I had an opportunity to give a presentation to soon-to-be Computer Science (CS) graduates. We presented the software development process (SDLC), and how different teams at Schweitzer Engineering Laboratories adapt the development process to their needs. We ended the presentation with tips for what to expect entering the industry.
There is a big disconnect between what we learn while getting a CS degree and writing software that makes money. Not to say a CS degree isn't worth it; quite the opposite. I still use the algorithms, graph theory, data structures, and development processes taught in college. But, there is so much more to software engineering than code and theory. In this article we'll walk through the tips I would give myself leaving college, and the tips I gave to those soon-to-be CS graduates.

Don't worry about knowing everything

I've been programming a long time--I'm 1996 Space Jam websites old. I've contributed to many coding communities, built and maintained high-traffic applications with near-zero downtime, gave conference talks, released a course, and written a lot of these silly articles. If you were to pull me out of a crowd and ask me to invert a binary tree I would probably struggle. I've been rejected from many job interviews and called "not senior enough". The point is we all face rejection no matter how much we know, or how much experience we have. We can always be better. Don't fret about not knowing everything, and don't be afraid to say, "I don't know", but always follow up with, "But I'll find out". Throughout my four years getting a Computer Science degree I thought if I knew all the algorithms, and produced the fastest code I would be the best programmer. These things help, but after my first experience as a professional software engineer, I quickly learned there is so much more to software than Big O notation. It's not about knowing everything; it's being curious and able to figure it out.

Ask questions, have opinions

This piece of advice piggybacks "Don't worry about knowing everything", and adds some spice. Don't be afraid to ask questions. Ask the senior developer why they came to a certain solution, or how Map() is different than {} in JavaScript. This is a two-way street. When you become a senior software engineer ask the junior engineers questions. Remember, we can't know everything, but speak up when we do know something! Have opinions. Yeah, not every opinion will be correct, but this is how we learn. As a manager, I like when interns have opinions on architecture. This shows they care, they're engaged, and they're learning. Just don't be a jerk about it. Ask questions on the things you don't know, and have opinions on the things you do know!

Have a business-oriented mindset

We leave college with the wrong mindset of what companies look for. I thought software companies wanted the best software engineers because it makes them the best. Don't get me wrong, there is value in being good at writing software. Let's say we write a highly decoupled, blazing-fast, platform-agnostic, cleanest code application. If no one uses the application then all we did was create a waste of time and money pit. Great software engineers consider both technical and business needs. We need to think about how our decisions produce value. Switching frameworks may sound cool, but who gets the value out of that? If the old framework isn't maintained anymore, and we need to change to get security patch support then maybe it's worth switching frameworks. If we want to change frameworks, because we saw a post on Hacker News about it then maybe reconsider.

When I get asked how to get promoted and move up the software engineering ladder this is the biggest advice I can give. Consider how your skills produce value for your company. Same for writing a resume or interviewing. It's cool you know Haskell, but how does that transfer to producing value at the company you're being hired for? Maybe since you had to learn Haskell for a class you are a quick learner, and can onboard quickly?

Technology is a a means to an end. Our applications solve problems users have is the goal.

Everyone breaks production

You're going to break production. Everyone does it. Even that lead engineer you look up to who never does wrong has broke production at some point. Don't stress about if you're going to break production. Breathe in. Breathe out. Let it go.
A colleague, who did the talk with CS graduates with me, has a saying, "Optimize for mean to recovery." Find ways to reduce the blast radius once production is broken instead of worrying about breaking production in the first place. This could be:

Having battle buddies you can rely on when sh*t hits the fan. Find a support system who can jump in and help when production breaks
Learn everything about the environment. Ask questions not just about the systems you work on, but neighboring systems. How does each piece connect?
Improve your skills and learn different parts of the software stack. Learn how Databases work, maybe learn networking and DNS, and if you're working in containers maybe introduce yourself to Kubernetes

When interviewing for a company look for blameless cultures. This means when something eventually breaks in production the team doesn't start pointing fingers. Ask the interviewers how they deal with outages, and watch out for warning signs. Many buzzwords without any concrete plan/action can be a cause for concern.

Lastly, writing software is a team effort. When software breaks there are multiple chains in the process of breaking down. Maybe code reviews weren't as thorough as they could have been? Or maybe we're missing testing or test cases that would have caught the error? Did the Continuous Integration step run? Maybe the architecture is inherently complex? As engineers we like to point to a root cause (Fallacy of the single cause), but the Titanic didn't sink for one reason. The builders of the ship cut costs, the ship was moving too fast, and the ship's radio operator dismissed the warning (Pruitt, 2018). Find a team with a blameless culture and don't point fingers when systems fail. Iterate on failures to learn and get better. Software is a team pursuit.

References

Pruitt, S., 2018. _Why Did the Titanic Sink?_. [online] History. Available at: https://www.history.com/news/why-did-the-titanic-sink [Accessed 4 July 2022].

NPM: What are project dependencies?

Dylan Paulus — Fri, 01 Oct 2021 19:19:21 +0000

Code dependencies are like Lego's. We're able to pull in other people's code; combining and stacking different packages together to fulfill our goals. Using dependencies greatly reduces the complexity of developing software. We can take advantage of the hard work someone has already done to solve a problem so we can continue to build the projects we want. A development pipeline can have multiple kinds of code dependencies:

dependencies
developer dependencies (devDependencies)
peer dependencies (peerDependencies)

In JavaScript, we have a package.json file that holds metadata about our project. package.json can store things like our project name, the version of our project, and any dependencies our project has. Dependencies, devDependencies, and peerDependencies are properties that can be included in a package.json file.

{
    "dependencies": {
        ...
    },
    "devDependencies": {
        ...
    },
    "peerDependencies": {
        ...
    }
}

Production vs. Development

Depending on the instance where code will be used changes the type of dependency a package is. There are packages that our users will need to run our code. A user is someone not directly working in our code-base. This could mean a person interacting with an application we wrote, or a developer writing a completely separate library. In other words, this is a production environment. Alternatively, there are packages that a developer or system only needs while working in our code. For example linters, testing frameworks, build tools, etc. Packages that a user won't need, but a developer or build system will need.

Dependencies

Dependencies are packages our project uses in production. These get included with our code and are vital for making our application run. Whenever we install a dependency the package and any of its dependencies get downloaded onto our local hard drive. The more dependencies we add, the bigger our production code becomes. This is because each new dependency gets included in the production build of our code. Evaluate adding new dependencies unless they're needed!

Dependencies are installed using npm install X or yarn add X

Examples of dependencies: React, stylized-components, jQuery

Dev Dependencies

Packages needed in development, or while developing our code, are considered dev dependencies. These are programs, libraries, and tools that assist in our development workflow. Dev dependencies also get downloaded to your local hard drive when installed, but the user will never see these dependencies. So adding a lot of dev dependencies only affects the initial yarn or npm install completion time.

Dev Dependencies are installed using npm install --save-dev X or yarn add --dev X

Examples of Dev Dependencies: Jest, ESLint, Webpack

Peer Dependencies

Peer dependencies are similar to dependencies except for a few key features. First, when installing a peer dependency it doesn't get added to your node_modules/ directory on your local hard drive. Why is that? Well, peer dependencies are dependencies that are needed in production, but we expect the user of our code to provide the package. The package doesn't get included in our code. This is to reduce including multiples of the same dependency in production. If every React library included a version of React as a dependency, then in production our users would download React multiple times. Peer dependencies are a tool for library owners to optimize their project size.

Peer Dependencies are installed using yarn add --peer X

Examples of Peer Dependencies: React, Bootstrap

End

I recently released a course, Creating React Libraries from Scratch, where we walk through deploying a React library from yarn init to yarn publish. Creating React Libraries from Scratch includes content just like this and more!

To learn more click the image below!

CSS: The Important Stuff (Box Model)

Dylan Paulus — Sat, 24 Oct 2020 18:06:10 +0000

CSS is one of those technologies that has a low barrier to entry (good thing 🎉), but because of this sometimes how it works can seem like magic. It's easy to get started writing CSS, so we quickly dive head first into it. Sometimes we smash our face into the concrete with frustration. Why won't my text move over yonder? Where did that scroll bar come from? How do I center this junk?

This is a series I'm starting called, "CSS: The Important Stuff". The goal is to take a dive into the mechanics of CSS so we can get a better intuition when styling and positioning elements. In part one we'll take a look at Box Model; the underlying layout of the web.

What is the Box Model?

It comes with great danger to myself to tell you that the web is made of boxes. Yes, it's boxes all the way down. Just take a look at this website—open the Element Inspector in the browser's dev tools and add
* { border: 1px solid red; } . Look at all those fine boxes.

The browser uses boxes to layout and render our websites. Within each box there are four properties: content, padding, border, and margin. Boxes are like onions, they have layers. Each property defines the size of our box (ie. width and height), styling, and how content around the box flows.

Content is what we're usually thinking of when we create HTML elements. Text, images, video, etc. This is at the center of the box
Padding gives space around the edges of the content. Separating the content from the border
Borders are used to visually separate boxes from each other—as seen in our red border example above
Margin is non-stylistic spacing around the box. This is useful to separate boxes from each other and position boxes

Why does this matter?

Dimensions

There are different ways to determine the width and height of a box. Is the box's dimensions determined by all its layers (margin, border, padding, content)? Or is it determined by just the content?

The browser made a choice in box-sizing and the default is content-box . When we give our element a width and height, lets say .box { width: 100px; height: 100; } , the default behavior with content-box is to give just the content layer the width and height of 100px. But, the overall box still has additional layers with padding and border. Naturally we would assume if we gave the element of 100px that it would be 100px, but the browser needs to add the values of padding and border on top of the content to get its "rendered dimensions".

Given the styling on our moon icon below, what would you expect the width and height to be on the box?

.moon {
  box-sizing: content-box;
  width: 100px;
  height: 100px;
  padding: 10px;
  border: 1px solid #FFF;
}

That's easy it's 100px... wait, 122px, what? How did we get 122px? Remember when we gave the width and height of 100px it only applies to the content layer. We also applied a padding of 10px on each side of the box and a 1px border around it. If we were to calculate the width of the box, we start with 100px, add 10px for the left padding, add 10px for the right padding, add 1px for the left border, and finally 1px for the right border.

100 p x + 10 p x + 10 p x + 1 p x + 1 p x = 122 p x

Seems a little counter intuitive—if I set my box to have a 100px width then it should be 100px darn-it! Luckily, there are more ways for the browser to calculate a box's size. When I think of the size of a box I think of border-box . Think of the layers of a box, and when assigning box-sizing think of the value as being cascading. When an element has content-box that means the browser determines the size of a box starting from content and adding any child layers. Since there are no children to content width/height only applies to content. When using border-box, width/height is set for the border, padding, and content.

Let's take a look at the same moon icon, but apply border-box instead.

.moon {
  box-sizing: border-box;
  width: 100px;
  height: 100px;
  padding: 10px;
  border: 1px solid #FFF;
}

What we see is something quite different, the width and height of the moon is now a hard 100x100. This isn't magic though, the missing dimensions have to come from somewhere. Using border-box the content layer gets shrunk to make room for padding and border. With content-box or width was guaranteed to be 100px, but what is the width of our content now with border-box?

100 p x - 10 p x - 10 p x - 1 p x - 1 p x = 78 p x

To me border-box is easier to intuit and reason around, and becomes my default, but if the width and height of the content matters more than the visible box itself, use content-box.

References

Start with No

Dylan Paulus — Sun, 13 Sep 2020 00:00:00 +0000

Our default response to any idea that comes up should be: “Interesting. Maybe some day.” In other words, a very soft “no” that leaves all our options open. We don’t put it in a backlog. We give it space so we can learn whether it’s really important and what it might entail.

"Shape Up" - Ryan Singer {Basecamp}

Story time. At a previous employer I had a boss who I thought was pretty tough. Not to me, but our users. No matter the request his response was always the same, "No." Eventually, we implemented around 40% of the requests, but the constant rejection of these requests stuck with me. Now that I lead a team I notice myself using the same mindset. "No" isn't negative, and it's not dismissive. "No" is a way of protecting the team, the project, and the users.

No?

There is a natural progression to projects that looks something like this. First, the honeymoon phase. The pastures are green; the project has no technical debt and every feature is a great idea. As we add functionality our user base grows. If we're lucky enough to get traction we move to phase two, the monolith. Our application now handles a bunch of use-cases. It does your taxes, picks up your dog's poop, and reminds you to call your mom. Only power users can fully grasp the purpose of the application. The technical debt is increasing with every new feature added. Finally, we get to the last phase. The application has become so bloated that it's hard to develop new features. New users aren't willing to learn the ins-and-outs of the product. A new startup comes around with a trimmed down version of your application. It's so easy to use! Around this time users are jumping ship.

Saying "no" is a buffer in our development process. It gives us, the developer, time to think of an optimal solution. Could a feature being asked for be solved in a way that satisfies multiple feature requests? Is there some domain logic we have yet to understand? Does the feature really make sense in this application? All these questions help reduce the lifecycle described above. It reduces feature churn by keeping the application simple, and by solving issues the users actually have.

Be mindful of how features interacts within your application. Nothing comes without a cost--especially in software development.

No.

"No" is a way of protecting people--the users and the development team. There is a gradient of experience when it comes to using your application. Users will be complete beginners, and others will be power users. As a product owner we need to balance these two extremes. If the application is too power-user-centric, then it's harder for new users to grok. When the application is too hand-holding, power users become frustrated. Saying "no" to one extreme is a way of keeping the scales balanced. But, be mindful of what you say "no" to. Some features may be vital to the application's success. This is part of the art.

"No" gives time to find the optimal solution to a problem. Users ask for a solution to X and suggest implementing Y, but what they really need is K. Feature requests give us an opportunity to improve processes and workflows. Our user's are domain experts, but are prone to do things they way they've "always done them". We can come in and find ways to simplify. Ask the user, "do we really need this step?" or "Could the system handle that instead of a person?" Once code is written and users get accustomed to the feature, it's really hard to change. Do this step upfront.

"No" keeps the development team doing meaningful work. Spending a sprint or two implementing a feature nobody uses or finds frustrating is demoralizing. Think about the workflow before writing any code. Question the feature and the users. Does the application need this? Could we implement it in a simple way? Reduce code churn; reduce developer churn.

No!

Work needs to get done. At some point you need to say "yes", but instead of taking on every task, you get to be mindful on what to prioritize. You get to take on the tasks that add the most business value and delight users. Use "no" as a tool.

Original Article

Making React Fast: Tips on Optimizing ReactJS

Dylan Paulus — Tue, 04 Feb 2020 00:00:00 +0000

React does a fantastic job abstracting away the rendering of webpages. When state changes, our UI's reflect it. But, have you ever gotten done writing your application just to notice things seem off? Clicking a button displaying a popover stutters for a second, or animations don't run smoothly.
With React being just a view layer there is a single bottleneck causing "jank" (stuttering in the interaction)... Rendering. If our applications never had to render they'd all be blazing fast running at millions of frames per second (I had top scientists run the numbers)!
Fortunately we don't live in that world. Our applications are useful--they render and then re-render.

The majority of devices have screens refreshing 60 times a second. Our applications have a short time-frame to update, render, and interact with users. According to Google all work needs to be completed within 10ms. Yes you read that right, 10ms. A daunting task. React gives us a lot of tools to take on this challenge. Let's take a look at a few of these.

The DOM Tree

Before moving onto improving React we need to take a step back. We need to get a few foundational ideas out of the way. The universal data structure of the internet is the Tree. HTML is built off of it.
If we consider the following HTML:

<div>
  <span>Hello</span>
  <p>World</p>
</div>

Then the data structure representation would look like:

React takes the same DOM (Document Object Model.. ie. HTML) tree structure, and mirrors it in memory--called the Virtual DOM. Kind of like how a cache is to I/O, the Virtual DOM is to the DOM Tree. It's faster to perform CRUD operations (Create, Read, Update, Delete) in the Virtual DOM. The browser performs massive amounts of work when the DOM Tree is changed: painting, layout, calculations, etc. The Virtual DOM is only a representation of the DOM in memory, so things like painting and layout don't need to be calculated. It's easier to perform many changes to the Virtual DOM then commit the final result, than it is to perform each change to the browser's DOM tree.

When thinking to optimizing React, think of the Virtual and browser's DOM Tree. The less work each of these structures have to perform, the faster our applications behave!

What causes React to render?

There are two main reasons for components to render in React: when a component's state changes, or when its props change. Out of the box React chooses the naive approach in deciding what needs re-rendered. Re-render the element, and any of its children nodes, every time the props/state changes. For example in the figure below, say we change the state of node 2. All elements contained within the dashed line would also be re-rendered.

Don't take this as a bad thing! React cannot make any preconceived assumptions on how our application functions, so it chooses the most user friendly approach. It would instead be frustrating if we updated our application with a logged-in user prop, and the MenuBar component six layers deep didn't reflect this change.

shouldComponentUpdate()

Every (class-based) React component has a shouldComponentUpdate() method which returns a boolean. It does exactly what the name suggests, returns true if the component should render on prop/state changes, or false if it shouldn't. Remember how we said a component always updates on prop/state changes? By default each component's shouldComponentUpdate() looks like:

shouldComponentUpdate() {
  return true;
}

Yup, on every update the component re-renders! But, what if we only wanted a component to update when certain props change? React will first call shouldComponentUpdate(), then move to updating the component if true is returned. shouldComponentUpdate() takes two parameters: the next props and the next state. We could calculate any number of scenarios to optimize our application. Consider,

shouldComponentUpdate(nextProps, nextState) {
  if (!this.props.isOpen && nextProps.isOpen) {
    return true;
  }

  return this.state.textInput === nextState.textInput;
}

This is great for components with a few props and state, but as components grow so do the need to check every prop and state. There has to be an easier way, right? Stay tuned!

PureComponent/Memo

Let's start with memoization... what is it? Why do we care?

Think of when a function receives arguments, like add(1, 2). Given the same inputs we can assume that we'll receive the same output (from our add example, the output is always 3). Let's also assume we have bad code and add(a, b) is a function that performs a bunch of computation. On average it takes three seconds to complete. After running the add(1, 2) function once we already know it outputs 3, so why should we waste additional time computing the output? Memoization is the act of caching, or storing, the result of a function call and returning the cached result on future requests.

In the same way, memoization is also utilized within React to prevent having to compute expensive renders (computations) over and over again.

Remember our friend shouldComponentUpdate()? We can achieve the same effect with PureComponent. Generally, our class-based React components will look like:

class MyComponent extends React.Component {
  ....
}

But, for extra optimization goodness we can replace React.Component with React.PureComponent.

class MyComponent extends React.PureComponent {
  ....
}

These two classes differ in their implementation of shouldComponentUpdate(). React.Component's shouldComponentUpdate() will always return true unless we override it (ie. always re-render on update). React.PureComponent has its own implementation of shouldComponentUpdate() which automatically performs a shallow comparison of all the component's props and state. If any of the new props the component receives are changed, then return true. Otherwise, it will return false (ie. not triggering a re-render, and returning the previously calculated render of the component).

Up until this point we've only talked about class-based components. You may be asking yourself, "Fine, but what about function components?" Since our goal is to 'cache' the output of a function component, React gives us a handy utility for memoizing our function components... React.memo! This works similarly to React.PureComponent in class-based components. If the component receives new props/state it re-renders. Otherwise, return the computed output from before.

function MyComponent(props) {
  ...
}

export React.memo(MyComponent);

Warning: Don't get React.memo() confused with React.useMemo(). These are different, and used in different ways!

Hooks (useMemo, useCallback)

I won't specifically go over how hooks work in this section. For an introduction, check out the infinitely better documentation from the official website. Instead, we'll be checking out two hooks that help us improve the rendering speed of our applications.

useMemo() is useful for 'caching' the results of a computationally expensive expression which returns a value. For example,

function MyComponent(props) {
  const result = React.useMemo(() => JSON.parse(props.value), [props.value]);

  ...
}

We provide useMemo() a second argument, which says whenever the value changes (props.value), rerun the function provided in the first argument. This makes it so values only get recalculated when they change.

useCallback() instead returns a 'cached' version of a function.

function MyComponent(props) {
  const handleClick = React.useCallback((event) => {
    console.log(event);
  }, []);

  ...
}

Instead of rebuilding the function on every render, useCallback() calculates a function only when needed.

Extracting Components

Large components are hard to maintain. To make our components easier to manage we may break them up into smaller, easier to manage pieces. Great! We achieved clean code! But, splitting components into smaller chunks also makes it easier to optimize computational heavy code. Its hard to optimize certain aspects of a gigantic component. In React either the whole component re-renders, or it doesn't. If components are broken down, using the techniques mentioned above (memo, pureComponent, and shouldComponentUpdate) become really powerful. We can isolate the bottlenecks in our applications and optimize.

End

There is a tradeoff in optimization. Memoized components create more memory that needs to be stored (ie. the output being cached). Also, there are additional computations needed in figuring out when and when not to re-render. Don't prematurely optimize. Run performance tests when you can, and optimize the pieces of your code that really need it!

How Facebook Avoids Ad Blockers

Dylan Paulus — Sun, 24 Nov 2019 00:00:00 +0000

Many sites try to avoid our ad blockers through limiting content or just straight out pleaing. Other companies are looking at encrypting any kind of indicator that screams "AD!" through obfuscation and other fancy techniques--confusing ad blockers. Here we'll look at some interesting techniques Facebook deploys to ensure we view their ads.

What does a Facebook ad look like?

Facebook ads take the form of a card containing the text 'Sponsored' (as shown below).

At first glance blocking these ads seems simple. Look for elements with the text containing 'Sponsored'. Maybe even look at the class names to be more specific. Then, slap on a good-ol' display: none;. Hold up cowboy--it's a little bit more complex than that!

First, Facebook doesn't actually have 'Sponsored' in their HTML. At least, not together as we would think. In the DOM, 'Sponsored' is actually broken up character by character.

Instead of:

<span>Sponsored</span>

In Facebook it (kinda) looks like:

<span>
  <span>S</span>
  <span>p</span>
  <span>o</span>
  <span>n</span>
  <span>s</span>
  <span>o</span>
  <span>r</span>
  <span>e</span>
  <span>d</span>
</span>

Notice the 'kinda' above? Facebook actually doesn't put the text directly in the DOM. Instead, it's embeded in data attributes.

So really it looks like:

<span>
  <span data-content="S"></span>
  <span data-content="p"></span>
  <span data-content="o"></span>
  <span data-content="n"></span>
  <span data-content="s"></span>
  <span data-content="o"></span>
  <span data-content="r"></span>
  <span data-content="e"></span>
  <span data-content="d"></span>
</span>

Obfuscation

This kind of structure seems relatively easy to block. It can even be achieved in a few lines of CSS. We'll write a quick CSS ad blocker! It'll look for a sequence of 'Sponsored' as data attributes, then hide it.

span[data-content="S"]
  + span[data-content="p"]
  + span[data-content="o"]
  + span[data-content="n"]
  + span[data-content="s"]
  + span[data-content="o"]
  + span[data-content="r"]
  + span[data-content="e"]
  + span[data-content="d"] {
  display: none;
}

What's the catch? Well... I'm glad you asked!
To block against this simple ad blocker, Facebook deploys a few techniques.

First, I simplified the true structure of how the 'Sponsored' text is displayed. Remember how Facebook uses data attributes? Well, those span's are actually nested. This is to prevent the adjacency modifier seen in our CSS example +

<span>
  <span>
    <span data-content="S"></span>
  </span>
  <span>
    <span data-content="p"></span>
  </span>
  <span>
    <span data-content="o"></span>
  </span>
  <span>
    <span data-content="n"></span>
  </span>
  <span>
    <span data-content="s"></span>
  </span>
  <span>
    <span data-content="o"></span>
  </span>
  <span>
    <span data-content="r"></span>
  </span>
  <span>
    <span data-content="e"></span>
  </span>
  <span>
    <span data-content="d"></span>
  </span>
</span>

But, this doesn't completely ruin our CSS ad blocker. We can refer to the parent of the data attribute and drill down.

span
  span[data-content="S"]
  + span
  span[data-content="p"]
  + span
  span[data-content="o"]
  + span
  span[data-content="n"]
  + span
  span[data-content="s"]
  + span
  span[data-content="o"]
  + span
  span[data-content="r"]
  + span
  span[data-content="e"]
  + span
  span[data-content="d"] {
  display: none;
}

The next technique, this is the point where it gets particularly interesting, Facebook actually hides 'dummy' DOM nodes between the 'Sponsored' text. These values are entirely random characters, with a random number of DOM nodes between them. Invisible characters.

At this point our CSS ad blocker is completely broken. There is no way for us to possibly code every possible value in CSS. Ugh.

We could try playing around with ids or class names, but these are pretty generic. A lot of content within the Facebook code share similar classes, so there is no unique identifier in finding an ad.

End

We've looked at a few techniques Facebook deploys to prevent ad blockers from... blocking ads. Using data attributes instead of straight text in the DOM, nesting DOM elements to make DOM parsing more difficult, and adding random DOM nodes between characters to obfuscate the output. The next time an ad blocker doesn't completely remove all the ads on a site, try looking at the DOM. There may be some interesting techniques at play to keep the ads flowing through!

Cover Photo: Wojtek Witkowski on Unsplash