DEV Community: Engin Arslan

How much does it cost to build and run a profitable side project?

Engin Arslan — Thu, 26 Dec 2024 15:55:00 +0000

One of the things that I was curious about when starting to build my SaSS side project was the potential costs involved. I had no idea if it would be affordable to launch an app if it were to make no money. Obviously, the costs involved in launching anything online are generally much less than building anything in real life. However, I am usually reluctant to start spending money on a side project that I don't know would be profitable.

Luckily, my side project, Heuristica, became semi-successful by my standards, so I started to be able to spend more money on it. Here are some services I am paying to keep Heuristica running in the order that I started to pay for these services. It should give you a good idea of the costs of running a SaSS business as a micropreneur.

Domain (20$/year)

For me, the first cost involved in building any application is the domain name. However, I treat this as a given and don't really sweat about it. Many of my projects are built with the aspiration of becoming profitable, so I always use a purchased domain for my launch to look credible instead of free domains provided by hosting platforms.

LLM APIs (120$/month)

I built and launched Heuristica in May 2023. It was initially based solely on GPT from OpenAI, so using that API and bearing the costs was the only option. However, the costs related to OpenAI were negligible. I paid only a few dollars for the first few months.

The costs have scaled with an increase in usage. However, I find the costs acceptable since they scale proportionally to my revenue and usage. I currently offer multiple models on my website (from OpenAI, Claude, and TogetherAI). The total cost varies but is around 150$ a month.

Hosting (20$/month)

The second cost that I started to incur was due to Vercel. I am using Next.js to build Heuristica, so Vercel was the obvious choice for hosting. They are very generous with their free plan, and I didn't expect to pay anything for a long time. However, I noticed they had a very strict timeout policy on server endpoints, which dropped long-running requests after a few seconds. I started to get user complaints where they wouldn't get any responses for longer queries. It was obvious that I had to switch to a Pro plan. The good thing is that you can have multiple projects in your Vercel Pro Plan, which could be like a shared cost between multiple projects.

Analytics (15$/month)

After some time, I needed to start paying for my analytics provider, Plausible, due to the increased traffic I was getting. Plausible is a privacy-friendly analytics app that doesn't leverage cookies for tracking. The information it provides can be limited, but it is sufficient for my needs. I use it to see the amount and duration of traffic I am getting from different referrers and countries. Just like Vercel, it can also be a shared cost between multiple projects.

CDN (3$/month)

Heuristica is a tool for visual learners, so I wanted to display a lot of images and videos on my landing page. However, I quickly realized that hosting images on my website is not the brightest idea. With the traffic I was getting, I was rapidly depleting the bandwidth allocated to me on my Vercel account. So, I looked for an image hosting platform (CDN) and decided on Bunny. I am really happy with them; they only cost a few dollars a month.

Email (10$/month)

As Heuristica became more popular, I needed a professional-looking email address, so I signed up for two business email accounts on Gmail.

Authentication (0$/month)

I have been a developer for a while, but I still feel uncomfortable rolling out my authentication solution. When building Heuristica, I was delighted to have discovered Clerk. It made implementing authentication on my website a breeze. Heuristica is now a profitable project with around 3000 monthly active users, and I still don't pay anything for Clerk. Their free tier is incredibly generous.

Database (20$/month)

I am using Supabase as my cloud database provider. I am very happy with them, and just like Clerk, they also have an incredibly generous free tier. I could have kept using them for free, but I wanted to have the option of having daily database backups so as not to lose any customer data, so I recently switched to a paid account.

Error Logging (26$/month)

I initially used Bugsnag to be notified about production errors. However, I switched to Sentry because of its tighter integration with Vercel, which makes it easy to upload source maps on every deployment. However, their free tier is limited. I quickly ran out of my monthly quotas, which reduced my visibility until the end of the month. I could have handled the limitations by being more selective about the errors I care about, but I ended up paying for a monthly subscription.

Email Marketing (20$/month)

I always dreaded email marketing. It felt like something I was obliged to do despite not liking it at all. I think this was just because I had nothing valuable to say to my users/customers. After a while, I started to feel the urge to communicate with my newsletter subscribers to notify them of all the cool features that I have been building. Initially, I chose LemonSqueezy for sending email campaigns since it had a free tier. However, I quickly grew beyond their free tier limitations.

LemonSqueezy also lacked many essential features, such as customer segmentation, lists, or bulk user modification. I switched to Brevo because it has many good features at an affordable price point. In my experience, email marketing tools can be pretty expensive, so I am happy to have discovered Brevo.

SEO (100$/month)

I eventually made my peace with email marketing, but I still dread marketing in general. That's why I like the idea of SEO, which performs like passive marketing in some sense. I used to have a Keysearch membership, but I wasn't sure about the accuracy of some of the results I was getting on my keyword research. I ended up getting an Ahrefs membership, which costs a lot more but is supposed to offer much more accurate results.

Developer Experience (20$/month)

I only needed to try out Cursor for a few days to see how much of a productivity boost it provides. It helps me build features faster and completely removes some of the more mindless, boilerplate-y aspects of coding, so you have more mental capacity to focus on the important stuff. The return on investment is crazy with this one.

Transactional Emails (0$/month)

It took me a while to realize I needed to email my users about specific actions (like subscriptions) for a much better user experience. Initially, my customers didn't even know if they had successfully subscribed since I didn't send any confirmation emails. I ended up using Resend to send programmatic emails on completion of certain actions. It doesn't cost anything at my scale and offers an excellent developer experience.

Product Analytics/Metrics (0$/month)

Plausible provides analytics related to page views, view durations, visitor origin, etc. However, it doesn't offer fine-grained metrics around how the users interact with the product. I initially implemented custom database calls whenever an action takes place to collect usage metrics. However, I found out that it is not the collection of the metrics that is the problem; it is the visualization of those metrics. Amplitude is perfect for emitting events on user actions and creating dashboards around those events. Currently, I am still using the free tier of the tool, which only allows for visualizing a limited amount of metrics. However, I can see myself switching to a paid tier if I ever get more data-oriented with my production development.

Amplitude is super easy to setup so my suggestion would be to install this as soon as you can to be able to collect even more metrics.

Customer Support (0$/month)

I am using Crisp to have a chat bubble on my website. I can't recommend having this feature on your app enough. Most of the questions you receive will be just noise; however, some of the interactions you will have with your users and prospective customers will be priceless. A chat bubble is not only great for user support but also a powerful tool for user research and understanding customer intentions and sentiments. Crisp has a generous free tier that is currently sufficient for my purposes.

Total Costs (~400$/month)

All these costs add up to almost 400$ a month. These are also some other recurring costs I incur while running my side project. However, some other costs are specific to my platform's features, like integration with different APIs. These costs would be irrelevant to other projects.

If I were to cut costs, I would switch back to Keysearch from Ahrefs and downgrade Sentry to use the free tier. Anything else would be more difficult to get rid of.

When I started Heuristica, I was very frugal. However, as the platform became profitable and popular, my mindset shifted. Previously, my decisions were based on solving problems using the most cost-effective way possible (or for free). Now, I am always looking for opportunities to add value to the platform for a reasonable cost. Time is my most limited asset, and I am more than happy to spend money to solve problems instead of tackling them myself.

Using Notion API with JavaScript

Engin Arslan — Sat, 08 Oct 2022 22:43:37 +0000

Notion is an awesome organizational tool I have been using for a long time. It has so many capabilities that it slowly replaced many other tools I use daily. One of the ways that I use Notion is to create content for my personal website.

My personal website is built as a static website. This means that I have to provide all the information I want inside the website before deploying it on the web. Once it is deployed, it doesn’t change. This means I must redeploy the website anytime I want to update it.

This might sound cumbersome, but it isn’t. Building, updating and deploying a website means a couple more steps are involved, but it doesn’t take much more time. Additionally, having a static website has some benefits in terms of performance, cost-effectiveness and resilience. A static website is displayed quickly from anywhere worldwide, can be hosted freely and wouldn’t crash under heavy traffic. It happens to be a great architectural decision for certain kinds of websites.

There is a page on my website listing the books I like. Maintaining this kind of long and dynamic list using HTML can be cumbersome. You would probably need a visual editor accessible from anywhere to edit that content easily.

That’s why I have a database in Notion that I could easily populate with book information and import into my website. You can see what that database looks like here. Before discovering the Notion API, I used to export this database as a CSV and parse that CSV to populate the books list page. However, this process included too many steps, so I avoided doing it frequently.

However, Notion API makes this process super simple. In this post, I will show how I interact with the Notion API using JavaScript to populate my application with data from my Notion account.

Integrating with the Notion API

You can find the repository that includes the code from this post in my github repo.

This post assumes that you have a Notion account. Go to the https://www.notion.so/my-integrations page to create an API integration with Notion. An API integration essentially creates a way for you programmatically interact with a resource on Notion. That resource could be a page or a database. In this example, we will look into interacting with a database, but I will also have another post showing how to interact with pages.

We will create an internal integration in our workspace using read content capability.

Create an internal integration by giving it a name and some access capabilities. At a minimum, we need a read-content capability. Since we will be creating an API integration for our own usage, the capabilities we give to this app can be as generous as you would like. However, if we were to create a public integration, we would want to be more careful when deciding which access capabilities we might need. Too many and unneeded capabilities might make our users uneasy.

For this project, I created an integration called Book Reviews, but you can call yours whatever you want.

Create the integration and take note of the Internal Integration Token that is generated. This is a secret token that will allow us to control this integration. Now that we created the integration, go to a page in Notion where you would like to have this integration functional. For the sake of this example, choose a page that contains a full-page database. Then, click Add Connections to connect the integration to the page. If you don’t have a page available, you can duplicate mine: https://cuboid-hoof-835.notion.site/7c579be494fc4f87bc827c6653f52dfd?v=b03c21ff315740c8a951819476a82526

We need to note the ID of this resource (page/database), so we can refer to it in our code. You can find the ID in the URL. Here is what the URL looks like in my case https://www.notion.so/7c579be494fc4f87bc827c6653f52dfd?v=b03c21ff315740c8a951819476a82526. The ID is the part that is before ? and after the domain name, which is 7c579be494fc4f87bc827c6653f52dfd.

Fetching Data from Notion API

We can connect to the Notion API just like how you would connect to any API. For example, we can use the command line curl tool. However, Notion has a JavaScript client library that makes interacting with this library much easier. We will be using it in our example. I will be assuming you have Node.js and npm installed on your system.

Let’s create a new project by creating a folder and then creating a package.json file inside that folder.

mkdir book-reviews

We will run npm init -y to initialize an empty package.json inside this folder.

npm init -y

Now that we have a package.json file, we can start installing the packages that we will be using in this project.

npm install --save @notionhq/client@^2.2.1
npm install --save dotenv@^16.0.3

@notionhq/client is the library that will help us use the Notion API easier. dotenv is a library that makes it easier to use the secrets we store in environment variables. What secrets, you might ask. Remember the Internal Integration Token generated when we first created our integration? That will be the secret key we will use to communicate with the API. Let’s create a file called .env to store that secret and the database id.

NOTION_KEY=<Paste Your Internal Integration Token here>
NOTION_DATABASE_ID=<Pase your Notion Database ID here>

.env files are a way to manage secrets in application development. They are not committed to source control or anywhere that is public. To prevent it from going to git-based source control, you can ignore it using a .gitignore file alongside other things you might want to ignore, like node_modules.

.env
node_modules

We can now start writing our JavaScript logic to fetch the book list data from Notion! Create a file called index.js.

const dotenv = require('dotenv')

dotenv.config()

console.log(process.env.NOTION_KEY)
console.log(process.env.NOTION_DATABASE_ID)

Here we are importing the dotenv library and running the config method on it. This should load the variables stored inside the .env file into the environment. We are using console.log to print those variables to the screen to ensure they are working as expected.

We now have the pieces we need to start interacting with the Notion API and the database.

const Client = require('@notionhq/client').Client
const dotenv = require('dotenv')

dotenv.config()

const NOTION_CLIENT = new Client({ auth: process.env.NOTION_KEY })
const DATABASE_ID = process.env.NOTION_DATABASE_ID

async function getDatabaseData(client, databaseId) {
  const response = await client.databases.query({
    database_id: databaseId,
  })

  console.log(response)
}

getDatabaseData(NOTION_CLIENT, DATABASE_ID)

Here I am creating two constant variables used throughout the file: NOTION and DATABASE_ID.

We are also creating an async function called getDatabaseData that asynchronously connects to the given database_id using the Notion Client we instantiated. It takes the Notion Client library and the database id as arguments. It fetches the results and logs them to the screen.

async function getDatabaseData(client, databaseId) {
  const response = await client.databases.query({
    database_id: databaseId,
  })

  console.log(response)
}

Including some error handling is a good idea since many things can go wrong when working with a remote API. We will handle the errors using a try catch statement.

async function getDatabaseData(client, databaseId) {
  try {
    const response = await client.databases.query({
      database_id: databaseId,
    })

    console.log(response)
  } catch (error) {
    console.error(error)
  }
}

The database you are working with will be different than mine, but you might notice these values at the end of the data structure you receive. Note the has_more property.

next_cursor: '54bf7ed3-b876-45cc-b3ad-01eb8df0cb56',
has_more: true,
type: 'page',
page: {}

has_more being true means that the response you are receiving from the API is paginated. There is more data available than what we are currently receiving. The rest of the data is on the following pages*.* The Notion API returns only 100 results per query by default. The query results are contained inside response.results array. We will save the query results inside an array called results and log the length of that array (which will be logging a number that is 100 or less).

async function getDatabaseData(client, databaseId) {
  try {
    let results = []

    const response = await client.databases.query({
      database_id: databaseId,
    })

    results = [...results, ...response.results]
    console.log(results.length)
  } catch (error) {
    console.error(error)
  }
}

We will also update our code to get the results contained in the following pages.

async function getDatabaseData(client, databaseId) {
  try {
    let results = []

    const response = await client.databases.query({
      database_id: databaseId,
    })
    results = [...results, ...response.results]

    // while loop variables
    let hasMore = response.has_more
    let nextCursor = response.next_cursor

    // keep fetching while there are more results
    while (hasMore) {
      const response = await client.databases.query({
        database_id: databaseId,
        start_cursor: nextCursor,
      })
      results = [...results, ...response.results]
      hasMore = response.has_more
      nextCursor = response.next_cursor
    }

    console.log(results.length)
  } catch (error) {
    console.error(error)
  }
}

To fetch the next page, we need to provide a start_cursor value to the database query. start_cursor indicates the location in the list where we should be fetching the results. We are making it so that the start_cursor for the next fetch would equal the next_cursor of the current fetch results.

const response = await client.databases.query({
  database_id: databaseId,
  start_cursor: nextCursor,
})

We are doing this inside a while loop that will run until has_more is equal to false.

// while loop variables
let hasMore = response.has_more
let nextCursor = response.next_cursor

// keep fetching while there are more results
while (hasMore) {
  const response = await client.databases.query({
    database_id: databaseId,
    start_cursor: nextCursor,
  })
  results = [...results, ...response.results]
  hasMore = response.has_more
  nextCursor = response.next_cursor
}

With this logic in place, we should be able to fetch all the database data from Notion. Next, let’s format this data to use for our purposes.

Formatting the Data From Notion API

We will update the getDatabaseData function to return the results array instead of just logging it. We will use this function inside another function called main.

const Client = require('@notionhq/client').Client
const dotenv = require('dotenv')

dotenv.config()

const NOTION_CLIENT = new Client({ auth: process.env.NOTION_KEY })
const DATABASE_ID = process.env.NOTION_DATABASE_ID

async function getDatabaseData(client, databaseId) {
  try {
    let results = []

    const response = await client.databases.query({
      database_id: databaseId,
    })
    results = [...results, ...response.results]

    // while loop variables
    let hasMore = response.has_more
    let nextCursor = response.next_cursor

    // keep fetching while there are more results
    while (hasMore) {
      const response = await client.databases.query({
        database_id: databaseId,
        start_cursor: nextCursor,
      })
      results = [...results, ...response.results]
      hasMore = response.has_more
      nextCursor = response.next_cursor
    }

    return results
  } catch (error) {
    console.error(error)
  }
}

async function main() {
  const data = await getDatabaseData(NOTION_CLIENT, DATABASE_ID)

  console.log(data)
}

main()

We have simply added a return results statement inside the getDatabaseData function. We have additionally created the main function where we call the getDatabaseData function.

async function main() {
  const data = await getDatabaseData(NOTION_CLIENT, DATABASE_ID)

  console.log(data)
}

If we look at the received data, we will notice that the properties of the data we are most likely interested in are under the properties key.

properties: {
 review: [Object],
  url: [Object],
  quote: [Object],
  rating: [Object],
  priority: [Object],
  title: [Object]
},

These properties correspond to the columns of our Notion database. Let’s console.log them using JSON.stringify to see what they look like. Currently, we can only see that they are JavaScript objects ([Object]). Using JSON.stringify would help us investigate the internals of this structure.

async function main() {
  const data = await getDatabaseData(NOTION_CLIENT, DATABASE_ID)
  const firstDataItem = data[0]

  console.log(JSON.stringify(firstDataItem, null, 2))
}

You will notice that we are getting objects with many properties representing the data stored inside our database columns. Each piece of data contains meta information like id, type and rich_text. The rich_text property is particularly interesting, representing a text as an array of objects. The array might have more than one element if you use a different kind of styling for the words inside a text like bold, italic or code. My database entries are just using plain text, so I will not worry about this.

The only thing left to do for us here is to format the received Notion API data into something easier to consume. I will create a function called normalizeDataItem that would format a given data item.

function normalizeDataItem(item) {
  const { url, review, quote, rating, priority, title } = item.properties

  return {
    url: url.url,
    review: review.rich_text[0]?.plain_text ?? '',
    quote: quote.rich_text[0]?.plain_text ?? '',
    rating: rating.number,
    priority: priority.number,
    title: title.title[0]?.plain_text ?? '',
  }
}

We can run our results through this normalizeDataItem function to get a more usable data structure.

async function main() {
  const data = await getDatabaseData(NOTION_CLIENT, DATABASE_ID)
  const normalizedData = data.map((item) => normalizeDataItem(item))

  console.log(normalizedData)
}

We are pretty much done! It is probably not super fruitful to just console.log results. Let’s update our code to write the results into a JSON file. We will first import two built-in node libraries called path and fs at the top of our file.

const fs = require('fs')
const path = require('path')

We will then update our main function to write the returned results to a JSON file on disk.

async function main() {
  const data = await getDatabaseData(NOTION_CLIENT, DATABASE_ID)
  const normalizedData = data.map((item) => normalizeDataItem(item))

  const outputFilePath = path.join(__dirname, 'data.json')
  fs.writeFileSync(outputFilePath, JSON.stringify(normalizedData, null, 2))
}

And that’s pretty much it! You can now do whatever you want with this JSON file. In my case, I feed it into my Next.js app in build time to populate the website with the content from the data.

I hope you enjoyed this walkthrough. Formatting the page data from Notion is a bit more tricky. I will explore this in a separate blog post. Meanwhile, feel free to follow me up at: https://twitter.com/inspiratory and https://www.linkedin.com/in/enginarslan/.

A Practical Introduction to Developing Web3 Apps Using Solidity and React - Part 1

Engin Arslan — Wed, 16 Mar 2022 21:34:47 +0000

Developing a Smart Contract

Imagine building a project with someone. You are planning on monetizing this project by accepting payments in cryptocurrency. You would want to distribute the earnings fairly in between yourselves. Partnerly.co helps you do this. It is an app that helps you create a smart contract on the blockchain that can distribute the collected payments between two (or more) parties in a predetermined split ratio.

I will show you how I built Partnerly from scratch. In this part of this walkthrough, you will learn how to create the smart contract that can be deployed on the Ethereum blockchain using Solidity.

The code for the smart contract and the front end application is available at https://github.com/hibernationTheory/partnerly.

Disclaimer

I am not a Blockchain developer. In fact, I am just learning this stuff myself. However, I am a software developer and an educator who taught programming from scratch to thousands of students. I can't claim that you will become a Blockchain development expert by just following this walkthrough, but I am pretty confident that you will learn practical skills that will make this a valuable stepping stone in your journey.

Let's get started!

Getting Started

This walkthrough assumes you have a working knowledge of JavaScript, Node.js and React. We will be using the Solidity language to develop the smart contract, which is relatively similar to JavaScript. The user interface we will be building for this contract will use React, Next.js and Tailwind libraries.

I use Visual Studio Code (VS Code) for coding. VS Code has an extension that makes working with Solidity (sol) files easy, so I recommend you download that for a better development experience.

If you are yet to install Node.js, make sure to install it using nvm. nvm makes it easy to manage multiple Node versions on the same system.

We will first need to install a library called Hardhat. Hardhat is a local development tool that makes developing, testing and deploying smart contracts easy.

Let's create a folder for our smart contract and initialize npm to install packages from the node registry.

mkdir partnerly-contract
cd partnerly-contract
npm init -y
npm install --save-dev hardhat
npx hardhat

When you run npx hardhat you will be presented with some options. Select > Create a basic sample project and say yes to all the questions. This will set up a sample project with Hardhat, further dependencies will get installed.

Initializing the Contract

We will be developing our contract inside the contracts folder. Let's delete the sample contract in there and create a new file called Partnership.sol. Below is what the skeleton of our contract looks like. I will explain every line that is in there.

//SPDX-License-Identifier: Unlicense
pragma solidity ^0.8.4;

import "hardhat/console.sol";

contract Partnership {
    constructor() {
        console.log("Contract is deployed");
    }
}

The first line inside the contract is a comment that defines the licence for this contract. We will just leave it to be unlicensed.

//SPDX-License-Identifier: Unlicense

The second line is for the Solidity version you want for this contract. We will be using version 0.8.4.

pragma solidity ^0.8.4;

Hardhat created a config file called hardhat.config.js when we initiated this sample project. You should set the version declaration in there to match the contract version.

module.exports = {
  solidity: "0.8.4",
};

Next, we are importing another contract into our contract called hardhat/console.sol. This is similar to how you would import libraries when coding in Node.js to enhance the functionality of your programs. console.sol gives our contract the ability to call the console.log function, which will make debugging the contract easier as we are developing it. We will remove this function before deploying the contract to a live network since it costs some gas to execute the console.log statements on the Ethereum network. You can think of gas as the price you need to pay for things to get executed on the Ethereum blockchain.

Next, we define a contract called Partnership that console.log's the message Contract is deployed when it is deployed.

contract Partnership {
    constructor() {
        console.log("Contract is deployed");
    }
}

Notice the definition of a contract is pretty similar to the definition of a class in JavaScript. Just like a class, a contract has a constructor function that gets executed when the contract is initialized (deployed in our case). It potentially can have other attributes and methods that will help us define the contract's data and behaviour. But for now, let's just console.log a string to indicate that the contract is deployed.

Running the Contract

Using Hardhat, we can define a script that will simulate the deployment of a contract. Let's create a script called run.js inside the scripts folder.

const hre = require("hardhat");

async function main() {
  await hre.run("compile");

  const Contract = await hre.ethers.getContractFactory("Partnership");
  const contract = await Contract.deploy();

  await contract.deployed();
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

We can execute this script by running this statement inside the terminal:

 npx hardhat run scripts/run.js

We should now see our console.log message get executed. The contract gets deployed successfully.

This run.js file has a bunch of boilerplate that we will skim over. You won't need to adjust some of the statements in there.

const hre = require("hardhat");

The above statement imports the hardhat library into the run script. You might see some other examples where this line is omitted. That is because this dependency gets automatically injected when you call this script using npx hardhat.

It is the same with the first statement inside the main function. You might see it getting omitted in some examples. I have included in my examples for the sake of completeness.

async function main() {
  await hre.run("compiler");
}

The following statement executes the async main function and exits from the script once the execution is completed. It additionally logs errors if there are any.

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

What we need to care about are these three lines of code.

// We get the contract to deploy
const Contract = await hre.ethers.getContractFactory("Partnership");
const contract = await Contract.deploy();

await contract.deployed();

We first compile the smart contract with the given name and then deploy the contract using the deploy method. Then, we ensure that the contract is deployed using the deployed method. Note that all these operations are asynchronous, so we use the async-await pattern.

We can continue developing the contract now that we have a script to execute it.

Terms of the Contract

Let's think of what we would like to do with this contract.

We want to create a contract that requires two or more wallet addresses that can accept payments.
The contract will also require split ratios to be defined for each given address. 1 to 1 split ratio means an equal split. 2 to 1 split means that the first address will get double the amount of the second one.
The contract will store these given addresses and split ratios.
The contract should be able to receive and store payments.
When a withdrawal is requested, the stored amount should be transferred to the contract's parties in the designated split ratio.

At this point, I should mention that I am not a lawyer. In fact, I am just a simple programmer, and as a programmer, I write bugs all the time. This poses some problems when it comes to developing smart contracts. It is impossible to ensure that a contract (or a program, for that matter) doesn't have a bug, and once a contract is deployed, you can't change it. The implication is that it is possible to deploy a buggy program that you can't alter after the fact.

The code for the smart contract is available on the blockchain for anyone to see. This transparency is great since you can verify that the contract is working as intended by reading the code. However, it also means that any actor could be dissecting the contract to take advantage of its vulnerabilities.

Have a look at the above terms. Does it look airtight? When I started to code this, it looked pretty good to me until I realized the users could pass a split ratio equivalent to 0, which would cause the contract to fail (You can't divide a number with 0). This is only a single oversight I could catch, and I can't guarantee there aren't others!

So let's ensure we update our requirements to include that the split ratios should be bigger than 0.

The contract will also need split ratios to be defined for each given address. Each split ratio should be bigger than 0.

Passing and Storing Values inside the Smart Contract

You can store data (state) inside the contract by defining variables inside the contract body. Any value passed into these variables will be stored inside the contract permanently.

Let's define a contract that has some state variables.

//SPDX-License-Identifier: Unlicense
pragma solidity ^0.8.4;

import "hardhat/console.sol";

contract Partnership {
    string private deploymentMessage = "Contract is deployed";
    uint256 private partnerAmount = 2;
    address[] public addresses;

    constructor(address[] memory _addresses) {
        require(
            _addresses.length == partnerAmount,
            "You can't have more than 2 partners"
        );

        addresses = _addresses;

        console.log(deploymentMessage);
    }
}

Solidity is a typed language meaning that we need to define our variable types.

We can also define the visibility and accessibility of these variables by using the keywords public and private. A private variable is meant to be accessible only inside the contract. In contrast, Solidity creates a getter function for a public variable, meaning that you could read the value of this variable by calling an automagically generated method of the same name (partnerAmount() and addresses() in this case).

We have defined three variables at the top of this contract. A private string variable called deploymentMessage, a public uint256 (an unsigned 256 bit integer) variable called partnerAmount, and a public array of address type called addresses. The addresses variable doesn't have a value yet. It is going to be populated by the caller of this contract.

We have updated the constructor function to accept _addresses as an argument. It is conventional to prefix function arguments with an underscore (_) to differentiate them from the state variables.

We use a keyword called memory before the _addresses argument. The memory keyword instructs Solidity that this data doesn't need to be part of the permanent storage. The function arguments are always declared with the memory keyword.

Constructor function makes use of a built-in function called require that checks to see if the given conditional is true. If not, it reverts the operation (an operation is called transaction in Blockchain terms) with the given message.

In our case, we require the given address amount to be equal to the defined partnerAmount. If that is not the case, the transaction will not go through. If there aren't any errors, our contract will deploy, and the passed arguments will permanently get stored inside this smart contract.

If we tried running the run script as is, we would get an error since our script now expects an array of two addresses to be passed in. Let's update our run script to pass addresses inside this contract.

const hre = require("hardhat");

async function main() {
  await hre.run("compile");
  const [owner, person1] = await hre.ethers.getSigners();
  const addresses = [owner.address, person1.address];

  const Contract = await hre.ethers.getContractFactory("Partnership");
  const contract = await Contract.deploy(addresses);

  await contract.deployed();

  await console.log(await contract.partnerAmount());
  await console.log(await contract.addresses(0));
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

We use the hre.ethers.getSigners function to get objects representing Ethereum accounts. We create an array of addresses by using the addresses of these accounts by using their address property.

const [owner, person1] = await hre.ethers.getSigners();
const addresses = [owner.address, person1.address];

We then pass these addresses into our contract during the initialization. Our contract successfully deploys when provided with two addresses.

Another way of ensuring that our contract would only accept two addresses would have been to define the array to have a fixed size. That way, we wouldn't need the require statement since the condition would be enforced at a data type level.

address[2] public addresses;

We are also logging the partnerAmount and the first address inside the addresses array by using the getter functions that Solidity exposes since these are defined as public variables.

While the run script is helpful to us in understanding the runtime behaviour of our contract, it is not a reliable way to ensure that our contract behaves correctly for our use cases. To develop confidence in our contract, we should write some tests that cover its behaviour.

Create a file called Partnership.js inside the test folder. This is what it will look like:

describe("Partnership", () => {
  it("can be deployed by providing two addresses on initialization", async () => {
    const Contract = await hre.ethers.getContractFactory("Partnership");

    const [owner, person1] = await hre.ethers.getSigners();

    const addresses = [owner.address, person1.address];

    const contract = await Contract.deploy(addresses);
    await contract.deployed();
  });
});

We will call this file using npx hardhat test test/Partnership.js. You should see a nicely formatted terminal output that tells us that 1 test has passed. As you can see, this file is super similar to our run script. The only difference is that it includes a test suite that gets executed whenever this script is called. The describe function describes what this test suite is for. The test block that starts with the function name it defines a test block that tests one aspect of our contract. In this case, it just tests to see if the contract is deploying successfully.

Let's write a test to see if our contract is failing as expected when no addresses are provided. First, we will need to import an expect method to help us make assertations.

const { expect } = require("chai");

And then, we will add this test block right after the previous one.

it("can NOT be deployed when NOT providing addresses on initialization", async () => {
  const Contract = await hre.ethers.getContractFactory("Partnership");

  let error;
  try {
    const contract = await Contract.deploy();
    await contract.deployed();
  } catch (_error) {
    error = _error;
  } finally {
    expect(error).to.be.ok;
  }
});

You would get an error thrown when a contract that expects arguments is deployed without being provided with those arguments. Here we start by initializing an empty error variable.

let error;

And then, we use the try-catch block to see if the error is populated or not. The error gets populated because the deployment fails when addresses are not provided. We check for the error to exist (to be truthy) inside the finally block:

expect(error).to.be.ok;

Tests are a super important part of any software development process. They not only help you ascertain the behaviour of a program (or a smart contract), but they can also give you confidence that the existing behaviour of the contract doesn't break when you modify it. You can ensure this by running the test suite whenever you update the contract.

Tests are an essential part of the software development process. There is a software development paradigm called Test-Driven Development (TDD), where we first write a failing test for each feature. We then implement the bare minimum code that will make this test pass. This methodology forces us to be iterative with our development process. We will try to follow this methodology while developing our smart contract.

Smart Contract Continued

In the previous section, we wrote a contract that takes two addresses. But we would like our eventual contract to work with two or more addresses and equivalent amounts of split ratios.

Let's change our tests to encode this expectation. We will be deleting the tests from before and writing new ones. Here is the first test block we will have. It simply tests if the contract deploys without errors when provided with the required amount of addresses and split ratios.

it("can be deployed by providing at least two addresses and equal amounts of split ratios on initialization", async () => {
  const Contract = await hre.ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();
  const addresses = [owner.address, person1.address];

  const splitRatios = [1, 1];

  const contract = await Contract.deploy(addresses, splitRatios);
  await contract.deployed();
});

Our second test will check to see if our contract fails as expected when the address amount and the split ratio amount are different. In this test, we will have two addresses but three split ratios.

it("can NOT be deployed when the split ratios are not equivalent to the address amount", async () => {
  const Contract = await hre.ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();
  const addresses = [owner.address, person1.address];

  const splitRatios = [1, 1, 1];

  await expect(Contract.deploy(addresses, splitRatios)).to.be.revertedWith(
    "The address amount and the split ratio amount should be equal"
  );
});

We use a new method on the expect function called revertedWith that checks to see if the transactions are reverted with the expected message.

We will add one more test block to ensure that the contract fails to deploy when it is called with less than two addresses.

it("can NOT be deployed when the the address amount is less than two", async () => {
  const Contract = await hre.ethers.getContractFactory("Partnership");

  const [owner] = await hre.ethers.getSigners();
  const addresses = [owner.address];

  const splitRatios = [1];

  await expect(Contract.deploy(addresses, splitRatios)).to.be.revertedWith(
    "More than one address should be provided to establish a partnership"
  );
});

All these tests will currently fail when called. Let's update our contract to make them pass. Our new contract will look like this.

//SPDX-License-Identifier: Unlicense
pragma solidity ^0.8.4;

import "hardhat/console.sol";

contract Partnership {
    address[] public addresses;
    uint256[] public splitRatios;

    constructor(address[] memory _addresses, uint256[] memory _splitRatios) {
        require(
            _addresses.length > 1,
            "More than one address should be provided to establish a partnership"
        );

        require(
            _splitRatios.length == _addresses.length,
            "The address amount and the split ratio amount should be equal"
        );
        addresses = _addresses;
        splitRatios = _splitRatios;
        console.log("Contract is Deployed");
    }
}

We now have two require statements. The first one is to ensure that more than one address is passed into the contract, and the second is to ensure that an equal amount of split ratios is provided. We get the _addresses and _splitRatios and store them inside the contract.

If we want to update our run script, we can do so by including splitRatios as an argument.

const hre = require("hardhat");

async function main() {
  await hre.run("compile");
  const [owner, person1] = await hre.ethers.getSigners();
  const addresses = [owner.address, person1.address];
  const splitRatios = [1, 1];

  const Contract = await hre.ethers.getContractFactory("Partnership");
  const contract = await Contract.deploy(addresses, splitRatios);

  await contract.deployed();

  await console.log(await contract.addresses(0));
  await console.log(await contract.splitRatios(0));
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

We are now defining and passing split ratios to the contract as well inside the run script.

Requiring Valid Arguments

We also need to ensure that there are no split ratios that are less than or equal to 0. We should iterate through every split ratio to see if they are all above one. The test for this functionality would look like this.

it("can NOT be deployed when any of the split ratios is less than one", async () => {
  const Contract = await hre.ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();
  const addresses = [owner.address, person1.address];

  const splitRatios = [1, 0];

  await expect(Contract.deploy(addresses, splitRatios)).to.be.revertedWith(
    "Split ratio can not be 0 or less"
  );
});

As mentioned previously, this test will initially fail. Let's implement the code that would make it pass.

//SPDX-License-Identifier: Unlicense
pragma solidity ^0.8.4;

import "hardhat/console.sol";

contract Partnership {
    address[] public addresses;
    uint256[] public splitRatios;
    uint256 private splitRatiosTotal;

    constructor(address[] memory _addresses, uint256[] memory _splitRatios) {
        require(
            _addresses.length > 1,
            "More than one address should be provided to establish a partnership"
        );

        require(
            _splitRatios.length == _addresses.length,
            "The address amount and the split ratio amount should be equal"
        );
        addresses = _addresses;
        splitRatios = _splitRatios;
        splitRatiosTotal = getSplitRatiosTotal(splitRatios);

        console.log("Contract is Deployed");
    }

    function getSplitRatiosTotal(uint256[] memory _splitRatios)
        private
        pure
        returns (uint256)
    {
        uint256 total = 0;

        for (uint256 i = 0; i < _splitRatios.length; i++) {
            require(_splitRatios[i] > 0, "Split ratio can not be 0 or less");
            total += _splitRatios[i];
        }

        return total;
    }
}

We now added a new variable called splitRatiosTotal that is private. This means that it won't be accessible from outside of the contract.

uint256 private splitRatiosTotal;

We have defined a new function called getSplitRatiosTotal that goes through every item in a given uint256[] and checks to see if they are bigger than 0. It also calculates the sum of those items and stores it inside a state variable splitRatiosTotal. We will use this variable when we want to distribute the funds to the contract addresses.

function getSplitRatiosTotal(uint256[] memory _splitRatios)
    private
    pure
    returns (uint256)
{
    uint256 total = 0;

    for (uint256 i = 0; i < _splitRatios.length; i++) {
        require(_splitRatios[i] > 0, "Split ratio can not be 0 or less");
        total += _splitRatios[i];
    }

    return total;
}

We have defined this function as private and pure. A pure function means that the operation inside the function doesn't rely on any external state. It would return the same output for the same inputs on every execution. Pure functions are highly desirable in programming since their behaviour is very predictable.

Having a pure function in Solidity is also great since they don't cost any gas when executed. As mentioned previously, gas is the computational and monetary cost of doing operations on the Ethereum blockchain.

Receiving Payments

We can now store addresses and split ratios inside the smart contract. Let's update our contract so that it can also receive payments. Let's start by writing a failing test for this.

it("can receive transactions in Ether", async () => {
  const Contract = await ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();

  const addresses = [owner.address, person1.address];
  const splitRatios = [1, 1];

  const contract = await Contract.deploy(addresses, splitRatios);
  await contract.deployed();

  // send ether to contract
  await owner.sendTransaction({
    to: contract.address,
    value: ethers.utils.parseEther("1.0"),
  });
});

We are using the following function call to send Ether to the contract from the owner account.

// send ether to contract
await owner.sendTransaction({
  to: contract.address,
  value: ethers.utils.parseEther("1.0"),
});

Our test currently fails because the contract can't receive funds. Making it to receive funds is super easy to do so. We just need to add a single line of code into our contract.

receive() external payable {}

Run the test again. It should pass. The contract can now receive funds!

We will also add a function called getBalance that will get the current balance of the contract. We can test this function by checking if the balance increased by the expected amount after receiving a transaction.

it("has a balance that increases after receiving a transaction", async () => {
  const Contract = await ethers.getContractFactory("Partnership");

  const [owner, person1, person2] = await hre.ethers.getSigners();

  const addresses = [person1.address, person2.address];
  const splitRatios = [1, 1];

  const contract = await Contract.deploy(addresses, splitRatios);
  await contract.deployed();

  expect(await contract.getBalance()).to.equal(0);

  const value = ethers.utils.parseEther("1.0");

  // send ether to contract
  await owner.sendTransaction({
    to: contract.address,
    value,
  });

  expect(await contract.getBalance()).to.equal(value);
});

We first check to see if the contract balance is 0 to begin with.

expect(await contract.getBalance()).to.equal(0);

We then use the parseEther method to convert a single Ether's value to its Wei equivalent.

const value = ethers.utils.parseEther("1.0");

A Wei is the smallest unit of Ether. Think of it as what a penny is to a dollar. The similarity ends there, though, since a dollar is only 100 times more valuable than a penny, whereas an Ether is 10^18 times more valuable than a Wei!

We then execute a transaction with this Wei value and check to see if the balance has increased.

expect(await contract.getBalance()).to.equal(value);

Let's implement the getBalance method inside our contract so that the test would pass. This is again straightforward to do so.

function getBalance() public view returns (uint256) {
    return address(this).balance;
}

Note the usage of the view keyword in the function definition. A view function is a function that doesn't alter the state but merely reads it.

Our tests should start passing with the implementation of this function.

Sending Payments

We will be implementing a function called withdraw. When called, this function will initiate the transfer of the funds to the recorded addresses according to the provided split ratio.

We can create a new describe block called withdraw inside the existing one. Using describe function allows us to categorize related tests together. We will include any withdrawal-related test inside this new block.

describe("withdraw", () => {
  it("can be called if the contract balance is more than 0", async () => {
    const Contract = await ethers.getContractFactory("Partnership");

    const [owner, person1] = await hre.ethers.getSigners();

    const addresses = [owner.address, person1.address];
    const splitRatios = [1, 1];

    expect(addresses.length).to.equal(splitRatios.length);

    const contract = await Contract.deploy(addresses, splitRatios);
    await contract.deployed();

    // send ether to contract
    await owner.sendTransaction({
      to: contract.address,
      value: ethers.utils.parseEther("5.0"),
    });

    expect(await contract.getBalance()).to.not.equal(0);

    await contract.withdraw();
  });
});

We check to see if the contract balance is non-zero after the transfer.

expect(await contract.getBalance()).to.not.equal(0);

Then we call the withdraw function. The test fails because the withdraw function doesn't exist. Let's fix that. We will add a withdraw function inside the contract.

function withdraw() public {
    uint256 addressesLength = addresses.length;

    for (uint256 i = 0; i < addressesLength; i++) {
        addresses[i].transfer(
            (balance / splitRatiosTotal) * splitRatios[i]
        );
    }
}

This function doesn't quite work yet. The reason for that is that we should define the addresses as payable for us to invoke the transfer method on them. We need to update the state variable for addresses and the argument inside the constructor with the payable keyword.

address payable[] public addresses;

And inside the constructor:

constructor(
    address payable[] memory _addresses,
    uint256[] memory _splitRatios
) {
    // constructor body
}

With this change, the withdraw function should start working!

We would like to have the withdraw function fail if the balance is less than 0. Let's write the test for that.

it("can NOT be called if the contract balance is less than or equal to 0", async () => {
  const Contract = await ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();

  const addresses = [owner.address, person1.address];
  const splitRatios = [1, 1];

  expect(addresses.length).to.equal(splitRatios.length);

  const contract = await Contract.deploy(addresses, splitRatios);
  await contract.deployed();

  expect(await contract.getBalance()).to.equal(0);

  await expect(contract.withdraw()).to.be.revertedWith("Insufficient balance");
});

We first ensure that the balance of the contract is 0.

expect(await contract.getBalance()).to.equal(0);

Then we try to call the withdraw function. We should make it fail with the "Insufficient balance" error.

await expect(contract.withdraw()).to.be.revertedWith("Insufficient balance");

To implement this inside the contract, we should get the contract balance inside the withdraw function and ensure it is bigger than 0.

function withdraw() public {
    uint256 balance = getBalance();
    uint256 addressesLength = addresses.length;

    require(balance > 0, "Insufficient balance");

    for (uint256 i = 0; i < addressesLength; i++) {
        addresses[i].transfer(
            (balance / splitRatiosTotal) * splitRatios[i]
        );
    }
}

Ensuring that Split Ratio Total is Smaller Than the Balance

We should ensure that the total split ratio is smaller than the balance. This way, we won't get any numerical errors when dividing the balance with the split ratio. Here is the test for that looks like.

it("can NOT be called when the total split ratio is greater than the contract balance", async () => {
  const Contract = await ethers.getContractFactory("Partnership");

  const [owner, person1] = await hre.ethers.getSigners();

  const addresses = [owner.address, person1.address];
  const splitRatios = [10, 10];

  expect(addresses.length).to.equal(splitRatios.length);

  const contract = await Contract.deploy(addresses, splitRatios);
  await contract.deployed();

  expect(await contract.getBalance()).to.equal(0);

  // send ether to contract
  await owner.sendTransaction({
    to: contract.address,
    value: ethers.utils.parseEther("0.00000000000000001"),
  });

  expect(await contract.getBalance()).to.equal(10);

  await expect(contract.withdraw()).to.be.revertedWith(
    "Balance should be greater than the total split ratios"
  );
});

We are sending 10 Wei to the contract.

// send ether to contract
await owner.sendTransaction({
  to: contract.address,
  value: ethers.utils.parseEther("0.00000000000000001"),
});

expect(await contract.getBalance()).to.equal(10);

The total of the split ratios in this test is greater than 10 (it is 20). This causes the contract to fail with the "Balance should be greater than the total split ratios" message. We can implement this inside the contract by adding a simple require statement.

require(balance >=
  splitRatiosTotal, "Balance should be greater than the total split ratios");

Finalizing the Run Script

Now that we have finished the contract logic, let's update the run script to simulate how someone would interact with this contract.

const hre = require("hardhat");

async function main() {
  await hre.run("compile");
  const [owner, person1, person2] = await hre.ethers.getSigners();
  const addresses = [person1.address, person2.address];
  const splitRatios = [4, 1];

  const Contract = await hre.ethers.getContractFactory("Partnership");
  const provider = hre.ethers.provider;
  const contract = await Contract.deploy(addresses, splitRatios);

  await contract.deployed();

  let balance1 = await provider.getBalance(person1.address);
  console.log(`The balance for the person1 is: ${balance1}`);

  let balance2 = await provider.getBalance(person2.address);
  console.log(`The balance for the person2 is: ${balance2}`);

  let contractBalance = await contract.getBalance();
  console.log(`The balance in the contract is: ${contractBalance}`);

  await owner.sendTransaction({
    to: contract.address,
    value: ethers.utils.parseEther("5.0"),
  });

  console.log(
    `The balance in the contract after receiving funds is: ${await contract.getBalance()}`
  );

  await contract.withdraw();

  console.log(
    `The balance in the contract after withdrawal is: ${await contract.getBalance()}`
  );

  balance1 = await provider.getBalance(person1.address);
  console.log(`The new balance for the person1 is:${balance1}`);

  balance2 = await provider.getBalance(person2.address);
  console.log(`The balance for the person2 is: ${balance2}`);
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

Here, we simulate the contract deployment with two addresses and a 4 to 1 split ratio. We need to define a provider to get the balance from the Ethereum accounts.

const provider = hre.ethers.provider;

We can then call the getBalance method on the provider to see the contract balance.

let balance1 = await provider.getBalance(person1.address);

Hardhat initializes these accounts with a balance of 10000 ETH.

After the deployment, we get another user to send 5000 ETH into the contract. The balance of the contract changes accordingly.

await owner.sendTransaction({
  to: contract.address,
  value: ethers.utils.parseEther("5000.0"),
});

We then call the withdraw function on the contract. This distributes the funds inside the contract in between the addresses. We then check to ensure the balance of the accounts has changed accordingly.

await contract.withdraw();

It seems like things are working as expected!

Believe it or not, this is it! We now have a program that can permanently store data, securely receive transactions, store funds and transfer those funds to specified accounts (wallet addresses). And it took less than 100 lines to write it! Building something like this makes you truly appreciate the power of blockchains and smart contract platforms like Ethereum. The amount of code and complexity to achieve something similar using centralized solutions would have been substantial.

How can other people make use of this smart contract? We could upload the contract to a public repository like Github and have other people clone it from there. They would need to change the run script to handle the deployment of the contract on the Ethereum blockchain themselves. Obviously, this is not an ideal or feasible solution for those who don't have any familiarity with working with smart contracts.

We could do so much better than this! If you enjoyed this walkthrough, check out my ebook for learning how to build the front end application for this contract using React, Next.js, and React.

Building a Web API using GraphQL vs. REST

Engin Arslan — Thu, 06 Jan 2022 10:27:19 +0000

API is short for Application Programming Interface. APIs define how applications interact with each other. We might want to build a Web API to enable external applications like desktop or mobile clients to interact with our services.

When building a Web API for external clients, there are two popular options to choose from. These are REST and GraphQL APIs. In this post, I will try to clarify the differences between these two ways of building APIs to help us make the best decision for our projects.

REST (Representational State Transfer)

REST is short for Representational State Transfer. It is a set of architectural principles that describes how systems over a network (like the internet) should communicate.

An API that follows the REST principles is said to be a RESTful API or REST API in short. It is one of the predominant ways of building Web APIs. These REST principles guide us to build APIs that have a clean separation between the client and the server and that are easily discoverable so that the clients can navigate the API without relying on external documentation.

When building REST APIs, we define endpoints that point to the resources that are exposed by the server. We can think of resources as the entities that make up a business's data.

Imagine an online store that displays and sells a bunch of products. The REST API that powers the front-end of this store might have an endpoint like www.example.com/api/products.json, which exposes the Products resource (the products that this online store has) to the clients.

The client can interact with this resource using HTTP methods such as POST, GET, PUT, PATCH, and DELETE to perform operations like creating a resource, reading a resource, updating the resource, and deleting that resource.

Say a client wants to interact with the products resource to see what products this online store has. They can perform a GET request to see what products are available in this store, make a POST request if they want to create a product, use the PUT (or PATCH) method to update a product, and DELETE to delete a product.

Defining the API as a collection of resources and allowing interaction with these resources using the underlying communication protocol (HTTP for a Web API) is something that is prescribed by the REST principles.

Here are some of the advantages and disadvantages of REST APIs.

Advantages

REST principles help us build APIs with well-defined interfaces that are easy to discover and navigate. It also helps us achieve a clean separation between the server and the client.
REST leverages the HTTP protocol; this makes data caching relatively easy.

Disadvantages

REST is not a specification. It is a set of principles. As a result, it doesn’t provide strong guidance in how anyone should implement those principles. That can result in many divergent implementations that claim to be RESTful when they are really not.
When we query a resource on a REST API, we receive all the data related to that resource (unless some filtering logic is built into the endpoint). This might be acceptable if the API was built for a single client, but it can become problematic when serving multiple clients. There could be desktop computers, mobile phones, and smartwatches as a client to the same APIs. The data requirements for these clients could be wildly different. Serving them the same amount of data is not practical or efficient.
On a related note, since we serve all the fields to the client in every query, there isn’t an easy way of telling which fields are actually needed by the clients. This makes removing (deprecating) fields very difficult when building REST APIs.
Web applications can require a lot of different kinds of data to be displayed whenever a page is visited. For example, an online store product page can have information about the product, the shipping times, the inventory amount for that product, products related to this product, and completely different products that might be on sale. It is very common that we might have to call multiple endpoints to gather all the needed data for a given page. This can result in making a lot of separate calls to the API. This can result in efficiency and performance issues.

GraphQL

GraphQL is a more recent paradigm for building APIs, developed and open-sourced by Facebook (now Meta). GraphQL consists of a query language that describes how the clients should communicate with the server and a server runtime that handles those queries. It has been developed to address some of the shortcomings of REST APIs.

In a REST API, the client interacts with the server through resources exposed through multiple endpoints. Using GraphQL, the clients interact with the server using queries through a single endpoint. GraphQL queries are written in a human-readable language called gql (Graph Query Language). One of the strongest aspects of GraphQL is that the clients can craft their own queries to receive the exact data they need for their purposes. For example, a client for an online store can make a query like this:

{
  products {
    title
    inventory
    price
    relatedProducts {
      title
    }
  }
}

This query is fairly straightforward to interpret. It is essentially asking for the title, inventory amount, and price of all the products and the titles of all the products related to this product in a single HTTP call! The client will be getting the exact data that it is querying. Nothing less, nothing more. This makes GraphQL APIs extremely data efficient.

The shape of a result for a query would match the query. Here is an example of what it might look like:

{
  "data": {
    "products": [
      {
        "title": "The Best T-Shirt",
        "inventory": 32,
        "price": "80USD",
        "relatedProducts": [
          {
            "title": "The Second Best T-Shirt"
          },
          {
            "title": "Stylish Sunglasses"
          }
        ]
      },
      {
        "tile": "The Best Pencil Case",
        "inventory": 5,
        "price": "20USD",
        "relatedProducts": [
          {
            "title": "Red School Bag"
          }
        ]
      }
    ]
  }
}

You might wonder how the client knows what is available to request in the API. This is made possible by a type definition file called schema that defines all the fields and data types made available by the API. A client can query this schema file to see what is available for them to know exactly what data they can receive.

I have written more about building GraphQL servers in another post if you want to learn more about this approach.

Here are some of the advantages and disadvantages of GraphQL

Advantages

GraphQL gives the clients a lot of power and control by enabling them to query for the exact data they need.
This fine-grained access model also makes it easier to understand what fields are being used by the clients. It makes removing (deprecating) API features much easier.
GraphQL schema makes it easy for clients to understand what data is available and provides type safety.

Disadvantages

Certain things like caching and rate-limiting could be hard to implement when working with GraphQL.
A client might also become more complex and require a framework (such as Apollo Client) to work efficiently with a GraphQL API.

Summary

When building an API, choosing the right option for your project boils down to the requirements you have. GraphQL seems like a great choice if you are serving many different kinds of clients, and performance could become a concern, but it definitely comes with a certain level of complexity. There is no wrong or right decision here. It is essentially about what you and your clients need from the API.

It is important to note that the decision is not mutually exclusive. You could still have a REST API for certain actions like authentication and perform other interactions through GraphQL.

Learn How to Build a GraphQL API in Node.js Using Apollo Server

Engin Arslan — Thu, 06 Jan 2022 10:26:55 +0000

You might want to build an API to enable external applications like desktop or mobile clients to communicate with your services.

When building a Web API, you can choose from two popular options. These are REST and GraphQL APIs. Which option you decide to choose depends on various factors. I have previously written about the differences between REST and GraphQL APIs. This post will show how to build a GraphQL API in Node.js using Apollo Server.

You can find the working code for this post at this Codesandbox:

Apollo Server

Apollo Server is an open-source GraphQL server compatible with any GraphQL client. It is a pretty reliable choice for implementing a GraphQL server on your Node.js backend. It is easy to get started and rich with additional features if you want to customize it for your own needs.

GraphQL Schema

One of the best aspects of working with a GraphQL API is the flexibility that it provides on the client-side. When using a GraphQL API, clients can tailor their own custom queries to submit to the backend. This is a major departure from how the REST APIs work.

This is what a GraphQL query might look like:

{
  books {
    title
    author {
      name
      books {
        title
      }
    }
  }
}

Here we have a query that is for fetching all the books alongside their title and authors, getting the name of all those authors and all the books that those specific authors have written. This is a deeply nested query, and we could keep nesting it as well!

When we allow the clients to craft their own queries, they are empowered to fetch the exact amount of data they require. A mobile application can be built to query for fewer fields, whereas a desktop application can query for a lot more.

But how does a client know which data to request from the server? This is made possible by something called a schema.

GraphQL servers use a definition file called a schema to describe the existing types present in the backend so that the client application can know how they can interact with the API.

Schemas in Apollo Server

One of the major differentiators between GraphQL servers is how they require the schema to be implemented. Apollo Server requires the schema to be implemented using the spec-compliant human-readable schema definition language (SDL). Here is what SDL looks like:

type Book {
  title: String
}

type Author {
  name: String
  books: [Book]
}

As you can see, it is fairly easy to understand what types exist and what attributes (or fields) these types have by just looking at this schema written using SDL.

You might have seen other GraphQL server solutions where the schema is implemented by using a more programmatic approach. Here is an example of how schemas are implemented using the express-graphql library. (link: https://github.com/graphql/express-graphql)

new GraphQLObjectType({
  name: 'Book',
  fields: {
    title: {
      type: GraphQLString,
            // define a resolver here
    },
  },
});

These different approaches present a certain kind of tradeoff. SDL makes it easy for anyone to understand what is happening in the schema, while it might be harder to maintain when your schema becomes very large. When schema is programmatic, it might be easier to modularize, customize and scale the schema, but the readability can suffer.

Getting Started

Let’s create some mock data to explore building APIs using Apollo Server. For this example, we will be building a GraphQL API for an online store that has a bunch of products and collections that include those products. Our API should be able to fetch and update these products and collections.

We will have two files called products and collections to contain this data.

collections.json

[
  {
    "id": "c-01",
    "title": "Staff Favorites",
    "description": "Our staff favorites",
    "isPublished": true
  },
  {
    "id": "c-02",
    "title": "Best Selling",
    "description": "These are selling out fast!",
    "isPublished": true
  },
  {
    "id": "c-03",
    "title": "In Season",
    "description": "Discover what is in season",
    "isPublished": true
  }
]

products.json

[
  {
    "id": "random-id-00",
    "category": "apparel",
    "name": "The Best T-Shirt",
    "brand": "A&A",
    "inventory": 32,
    "price": {
      "amount": 100,
      "currency": "USD"
    },
    "collections": ["c-01"]
  },
  {
    "id": "random-id-01",
    "category": "stationery",
    "name": "The Best Pencil Case",
    "brand": "Pencils Forever",
    "inventory": 5,
    "price": {
      "amount": 25,
      "currency": "USD"
    },
    "collections": ["c-02", "c-03"]
  }
]

We have three collections and two products. This is enough to get started.

Setting up Apollo Server

You will need to be comfortable using JavaScript and have a recent version of Node.js (12+) to follow this introduction.

Let’s create a new folder and run npm init -y in this folder. This will create a package.json file that will keep a record of the project's dependencies. Initially, we will be installing apollo-server and graphql libraries.

npm install --save apollo-server@^3.5.0 graphql@^16.2.0

We will also install a library called nodemon that will automatically restart the server whenever there is a change. This will help us see the results of our updates much faster. This dependency has to do with the development environment, so we will install it using the --save-dev flag.

npm install --save-dev nodemon@2.0

We will also create an index.js file at the root of this project folder.

touch index.js

We will add a start script in our package.json file to call nodemon with our index.js file.

"scripts": {
  "start": "nodemon index.js",
  "test": "echo \"Error: no test specified\" && exit 1"
},

Let’s create a folder called data and place the collections.json and products.json files into that folder.

We can now start setting up our server in this index.js file.

const { ApolloServer } = require("apollo-server");

const server = new ApolloServer();

server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

We have imported the ApolloServer from the apollo-server package and trying to run it by calling its listen method. We can run this file by calling our start script.

npm start

At this point, we would get an error since ApolloServer requires you to have type definitions (schema) and a resolver object on instantiation. We already know what a schema is. A resolver object is an object that has a bunch of resolver functions. A Resolver function is a function that specifies what data should a single GraphQL field return on a query. We don’t have a schema or resolvers, so nothing works.

Let’s start by creating a schema.

Creating a Schema and GraphQL Types

First, we will import the gql function and then create a typeDefs variable to pass into the ApolloServer.

const { ApolloServer, gql } = require("apollo-server");

const typeDefs = gql``;

const server = new ApolloServer({
  typeDefs,
});

server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

We can now start declaring types for our GraphQL API inside the backticks for the gql function.

Remember the shape of our data for collections and products. We will start by creating the type definition for a collection.

type Collection {
  id: ID!
  title: String!
  description: String
  isPublished: Boolean!
}

This is a type definition for a collection object. Notice how readable it is. Our object has three properties, and we have created a corresponding type with three fields. Note that there doesn’t need to be a one-to-one mapping in between a data object and the corresponding type. The GraphQL type represents an interface for a user (client) to interact with. The client might or might not care about the underlying shape of the data. We should make sure to only surface information that the client would care about in a way that is easy to understand.

Int, Float, String, Boolean, and ID are the most basic types we can use when defining types in GraphQL.

Int: Represents whole numbers.
Float: Represents fractional numbers. (Like 3.14)
String: Represents textual data.
Boolean: Represents boolean data (Like true or false)
ID: Represents a unique identifier. GraphQL clients can use this ID for caching / performance optimization purposes. It is recommended that you don’t have this ID field be human-readable so that the clients wouldn’t be inclined to implement a logic on their side that relies on a pattern that might surface in the ID. In our example, we will leave the id fields to be human-readable, though.

We use String, Boolean, and ID types in our example for collections. Another thing to note is that the usage of the bang symbol (!). ! indicates that the field can not be null (empty). It has to have value.

Let’s create the type definition for a product.

type Product {
  id: ID!
  category: String!
  name: String!
  brand: String
  inventory: Int!
  price: Price
  collections: [Collection!]!
}

We are using several new types in the Product type definition for the following fields:

inventory: Int is used for the inventory field since the product inventory is defined using whole numbers.
collections: We are defining an array of Collection types as the return type of the collections field. The ! usage here suggests that the array can not contain a null value, and the field can not be equal to a null value. So the value can only be an empty array or an array with collection objects inside.
price: Here, we define a new object type called Price for the price field. An object type is a type that includes fields of its own. The definition of that object type will be as follows.

  type Price {
    amount: Int!
    currency: String!
  }

There is an enhancement we can make on the Product type. Notice how the category field is defined as a String. The categories in online stores tend to be equivalent to specific values like apparel, accessories, stationery, etc. So instead of defining the category **field to be any string, we can define it so that it would only be equivalent to certain values. The way to do that would be using an **enum type. Enum types are useful when defining a set of predefined values for the given field. Let’s create an **enum** type that has three category values.

enum Category {
  apparel
  accessories
  stationery
}

type Product {
  id: ID!
  category: Category!
  name: String!
  brand: String
  inventory: Int!
  price: Price
  collections: [Collection!]!
}

We are almost done with creating our schema! Finally, we need to define a special object type called Query that defines all the top/root-level queries we can run against our GraphQL API.

type Query {
  collections: [Collection!]!
  products: [Product!]!
}

Here is what the entire schema looks like at this point.

const typeDefs = gql`
  type Collection {
    id: ID!
    title: String!
    description: String
    isPublished: Boolean!
  }

  type Price {
    amount: Int!
    currency: String!
  }

  enum Category {
    apparel
    accessories
    stationery
  }

  type Product {
    id: ID!
    category: Category!
    name: String!
    brand: String
    inventory: Int!
    price: Price
    collections: [Collection!]!
  }

  type Query {
    collections: [Collection!]!
    products: [Product!]!
  }
`;

We can now pass this schema into our ApolloServer and have things start working!

const server = new ApolloServer({
  typeDefs,
});

server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

If we are to visit http://localhost:4000/ or wherever the API is hosted locally, we would land on an Apollo branded welcome page. Let’s click on the big button that reads Query Your Server.

Clicking on that button will take us to a GraphQL explorer interface. Using this interface, we can run GraphQL queries against our API. We can also explore the documentation of our API. Note that we didn’t explicitly write any documentation when building our API. It gets generated automatically using the data already available in the schema. That is a pretty awesome feature of GraphQL! This means that our documentation will always be up to date with our code.

Let’s run a query against our GraphQL API. Here is a query that would get the name of all the products

{
  products {
    name
  }
}

The result would be:

{
  "data": {
    "products": null
  }
}

We are getting null as a result since we didn’t define any resolvers that would specify what this field should return when queried. Under the hood, Apollo Server has created a default resolver that is returning a null result since this is a nullable field.

If we defined the Query object so that the products are not nullable then we should ideally receive an empty list as a result.

type Query {
  collections: [Collection!]
  products: [Product!]
}

However,, Apollo Server default resolver doesn’t take care of that situation, so we receive an error.

Creating Resolvers

A resolver is a function that defines what data a single field should return when queried.

The Query type has two fields called collections and products. Let’s create very simple resolvers for these fields that will return an empty array. We will provide this resolvers object (that contains the resolver functions) inside the ApolloServer function.

const resolvers = {
  Query: {
    collections: () => {
      return [];
    },
    products: () => {
      return [];
    },
  },
};

const server = new ApolloServer({
  typeDefs,
  resolvers,
});

server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

Now, if we are to run our previous query, we would get an empty array instead. The resolver function we have defined for products specifies how that query should be resolved.

{
  products {
    name
  }
}

Let’s create a proper resolver for these fields. We will first import the collections and products data into index.js. Then we will return this data from these queries instead of just returning an empty array. Here is what the implementation looks like.

const { ApolloServer, gql } = require("apollo-server");
const collectionsData = require("./data/collections.json");
const productsData = require("./data/products.json");

const typeDefs = gql`
  type Collection {
    id: ID!
    title: String!
    description: String
    isPublished: Boolean!
  }

  type Price {
    amount: Int!
    currency: String!
  }

  enum Category {
    apparel
    accessories
    stationery
  }

  type Product {
    id: ID!
    category: Category!
    name: String!
    brand: String
    inventory: Int!
    price: Price
    collections: [Collection!]!
  }

  type Query {
    collections: [Collection!]
    products: [Product!]
  }
`;

const resolvers = {
  Query: {
    collections: () => {
      return collectionsData;
    },
    products: () => {
      return productsData;
    },
  },
};

const server = new ApolloServer({
  typeDefs,
  resolvers,
});

server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

Now that we have defined the resolvers for the collections and products, we can query these fields for the data they represent. As I have mentioned at the beginning of this article, one of the strengths of GraphQL is the ability for the clients to create their own queries. We can even write a query that would ask for data from these two fields at the same time! This wouldn’t be possible to do in a REST API.

{
  collections {
    title
  }
  products {
    category
    name
    brand
    inventory
    price {
      amount
      currency
    }
  }
}

We are not including the collections field for the products in the above GraphQL query. That is because our existing resolver functions currently don’t know how to return the data for that particular field. If we tried to query that field, we would receive an error.

To fix this problem, we need to create another resolver function for the collections field of the Product type. This resolver function will need to make use of the resolver arguments.

const resolvers = {
  Query: {
    collections: () => {
      return collectionsData;
    },
    products: () => {
      return productsData;
    },
  },
  Product: {
    collections: (parent, args, context, info) => {
      const { collections } = parent;

      return collections.map((collectionId) => {
        return collectionsData.find((collection) => {
          return collection.id === collectionId;
        });
      });
    },
  },
};

Resolver Arguments

Any resolver function receives four arguments. These arguments are conventionally called parent, args, context, and info. Of course, you could choose different names for these arguments depending on your purposes.

For now, we will only take a look at the first two arguments.

parent

This argument refers to the return value of the resolver for the field’s parent. In our example, the parent of the field collections is a product. So this value would be equivalent to a product item.

args

We could have fields that accepts arguments (a parametrized field). The args argument captures the arguments provided by the client to query a parametrized field. We will look into this use case in a bit. For now, we only care about the parent argument.

Our resolver function for the collections field uses the parent argument to fetch the collections array of the parent product. We use the id data in this array to find and return the collection objects from the collectionsData.

Product: {
  collections: (parent, args, context, info) => {
    const { collections } = parent;

    return collections.map((collectionId) => {
      return collectionsData.find((collection) => {
        return collection.id === collectionId;
      });
    });
  },
},

Now, if we are to run a query that fetches fields of the collections field, we would be able to get the collection objects that are associated with each product.

{
  collections {
    title
  }
  products {
    category
    name
    brand
    inventory
    price {
      amount
      currency
    }
    collections {
      id
      title
    }
  }
}

Fields with Arguments

As mentioned earlier, we can define fields that would accept arguments in our schema. Let’s create a new field under Query type called productById that would get the product of a given ID. Here is what that would look like in our schema.

type Query {
  collections: [Collection!]
  products: [Product!]
  productById(id: ID!): Product
}

productById is a field that accepts an id argument and returns the product type that has the given id if it exists. Notice the return type for the field doesn’t have the ! symbol. This means that the returned value can be of type Product or null. That is because a product of a given id might not exist.

Let’s query this field using the GraphQL API Explorer.

query($id: ID!) {
  productById(id: $id) {
    name
  }
}

We need to define the parameters that we will pass into this query inside the variables section.

{
  "id": "random-id-00"
}

This is how that screen looks like.

We would be getting a null as a result of this query since we didn’t implement the resolver function for this field. Let’s do that.

We will be adding a new resolver function under Query called productById. It is going to fetch the given id from the provided args parameter and return the product with the matching id.

Query: {
    collections: () => {
      return collectionsData;
    },
    products: () => {
      return productsData;
    },
    productById: (_parent, args, _context, _info) => {
      const { id } = args;

      return productsData.find((product) => {
        return product.id === id;
      });
    },
  },

Notice the underscore (_) before the argument names that we are not making use of in our function. This is a coding convention to indicate that a named argument to a function is not being used. Now, our previous query should work and return the desired product!

There is a lot more to GraphQL then what I wrote about here but this should be a decent introduction to the subject. In production, we wouldn’t have any hardcoded product or category data in our servers as we did here. We would rather fetch this data from a database or from some other API. When working with data, you might want to use classes called data sources that manages how you interact with that data and helps with things like caching, deduplication, etc. You can learn more about data sources here.

If you wanted to deploy this API, you can use cloud services such as Heroku, Google Cloud, etc. More information on the deployment process can also be found in the Apollo Server documentation.

You can also see the full code in action at Codesandbox!

A Philosophy of Software Design: It is All About Complexity

Engin Arslan — Sun, 14 Nov 2021 13:22:45 +0000

It is All About Complexity

A Philosophy of Software Design by John Ousterhout is one of my favorite books on Software Development. It talks about how building software is all about managing complexity.

There are two ways we can deal with this complexity:

Write simple and obvious code to eliminate complexity.
Or encapsulate the complexity so that you don't have to deal with it.

A software system is complex if it is hard to understand or modify. Another measure of complexity is how much time you spend dealing with the complexity in the system. If there is complexity in the system that is isolated in an area that you don't have to interface with then this is equivalent to removing complexity.

Complex systems are:

hard to change
hard to understand
hard to predict

Since complexity can be managed by hiding it, it becomes important how we surface that complexity for other components in the system to interact with. One of the most important techniques for managing software complexity is by designing systems where you only need to deal with a small position of complexity at a time. This is called modular design. A module can be thought of as a service, component, class, or function. Essentially, it is any structure that encapsulates complexity.

Have Deep Modules

You should aim to have deep modules that hide a lot of complexity and that expose it through a minimal interface. A minimal interface reduces the complexity that a module imposes on the rest of the system.

It is not worth having shallow modules. It is likely that your abstraction is not hiding enough details and the surrounding system has too many touchpoints with the large surface area of a shallow module. You should try to find the simplest interface and build a somewhat general-purpose module that is deep.

An example of a deep module is the garbage collector in Python. It essentially has no interface and encapsulates a lot of complexity that arises from the task it performs.

Don't Mix Layers of Abstraction

A lot of complexity can also be managed by how you structure things. One thing to pay attention to is to have different layers for different abstractions and to not mix those abstractions. This would make it easier to think about a given layer without concerning yourself with the rest.

Application of this principle can manifest itself at the interface of a module. It suggests that the representations that are used internally by a module should be different from what is used by the interface.

Pass-through methods and variables break this rule. When you pass high-level information to a low-level method, you end up having all the intermediate layers become aware of their existence.

Pull Complexity Downwards

Another way of dealing with complexity is by pulling it downwards. If you have a module that introduces complexity to the system due to the functionality it provides, it is better to hide that functionality inside that module. Most modules have more users than developers, so it is better to hide the functionality to have the developers deal with it. One example of this could be the config parameters. They could obviously be useful and provide flexibility but they can also be misused and pass the responsibility of configuration to the users when as a developer you could have handled it dynamically. Ideally, each module should solve a problem completely and config parameters can indicate an incomplete solution.

Keep Things Together

By creating too many small modules, you could increase the surface area of the entire system which could increase the complexity. It could also make it harder to read the code since the context is spread around. Only break things up if it is going to make the overall system simpler.

Define Errors out of Existence

Exception handling code could be a source of a lot of complexity. For one, handling exceptions can create its own exceptions. Also, exception handling code doesn't get executed all that much by its very nature. This means it is hard to ensure that it actually works as intended and doesn't suffer from bugs. It could additionally be hard to test this kind of code too due to the exceptional circumstances it requires to be invoked. A piece of code that raises too many exceptions is shifting the responsibility of dealing with the complexity to the user. It would be best if you could just handle the errors silently without notifying the users whenever this makes sense.

Imagine a networking transport protocol such as TCP. You don't receive an exception when a package drops. The protocol just handles it automatically by resending it. This is the complexity hidden.

Another simple example of this can be how array/list access works in Python and JavaScript. This code below would throw an error in Python due to the list index being out of range.

myList = [1, 2, 3]
print(myList[3])

Whereas this operation in JavaScript just returns undefined. It doesn't throw an error and hence is less complex to deal with.

const myArray = [1, 2, 3];
console.log(myArray[3]);

Names and Consistency

Choosing appropriate names when coding is incredibly important in managing complexity. Names are abstractions. A wrong or misleading abstraction can increase complexity and can be a source of hard-to-fix bugs. Another thing to be careful about names is their consistent usage. Be consistent when naming similar things, and don't use that name for anything that is unrelated.

Consistency is important in general too. It helps reduce the complexity in the system by reducing the cognitive burden. Consistency plays on our strength in pattern recognition and helps us leverage what we already know.

That's why it is important to abide by existing conventions. Having a better idea of how to do things is not a sufficient excuse to introduce inconsistencies to the system. Your new convention should be markedly better than the existing one to replace it.

Comments

There are arguments against the usage of comments out there but they can still be valuable. One primary argument against comments is that the code should be descriptive enough to reveal what it does. But, if you have to dive deep into the code to understand its functionality, then it means the code doesn't have a good enough abstraction.

Comments are abstractions that can help you hide complexity. Comments can also contain information that might be impossible to capture in the code (like hardware quirks, etc.) Comments don't necessarily add a maintenance overhead either if they are high-level enough.

Summary

A Philosophy of Software Design (affiliate link) is primarily about one thing: complexity. Dealing with complexity is the most important challenge in software design. It is what makes systems hard to build and maintain and it often makes them slow as well. You should always evaluate your decisions from the standpoint of complexity to ensure the sustainable continuity of the overall system.

Reasons Why JavaScript is Awesome

Engin Arslan — Mon, 04 Oct 2021 17:44:09 +0000

Coding is one of the most valuable skills you can learn on your own. It can give you superpowers at your job, help you find a fulfilling career, and enrich your intellectual capabilities. You can use coding to:

Automate various aspects of your job, such as the creation of spreadsheets, presentations, email templates.
Build a coding-related career such as mobile app, game, web development, data analysis or science, machine learning.
Pursue interests such as art, science, entertainment.

Whatever your reason for learning it is, coding is awesome. Coding with JavaScript is even more awesome since it enables you to do all those things listed above and more. This is thanks to the wide adaptation of the JavaScript language and its 3rd-party packages/libraries. This post will introduce some of the exciting and awesome JavaScript packages with simple code examples to demonstrate the wide range of things you can do by coding in JavaScript.

Natural Language Processing

Natural Language Processing (NLP) is a field that concerns itself with the analysis of human language. Using NLP techniques, you can programmatically detect the language of a given text, analyze its sentiment to understand if it holds a positive or a negative opinion, or extract entities such as names, places, etc. There are numerous NLP packages in the JavaScript ecosystem that allow you to perform these kinds of operations.

Language Detection

Franc is a JavaScript package that allows you to detect the language of a given text. I have included a small program to illustrate how it works. You can provide the program with a paragraph of text, and it will use the franc library to try to detect its language. Note that the functionality doesn't work great when provided with a shorter text.

Named Entity Recognition

Named Entity Extraction identifies entities like names, locations, or phone numbers inside a given text. Compromise is a JavaScript package that we can use that allows us to not only extract entities in a text but also identify what types of entities they are. Here is a sample program that allows you to enter a text file into the input field, and it would extract and identify any recognizable entities in that text.

Note that even though having these libraries is excellent, you might need to use
an ML/AI-powered natural language service that would allow you to do all these
things and even more!

Data Visualizations

Creating data visualizations is extremely useful when you want to quickly grasp the information contained in the data. Luckily JavaScript has several excellent packages that would help you create awesome data visualizations. Which package you choose depends on your business and technical requirements like the kind and complexity of graphs you will be creating, if you need animations or not, etc.

Chart.js is an easy-to-use package that can help you quickly create various kinds of charts and visualizations from your data. Here is an example of a line graph of some random data that is drawn using Chart.js.

You can quickly create different kinds of visualizations by changing few lines
of code. Here is a pie chart using the same library:

Another strong option in this category that is important to mention is D3.js. D3.js is substantially harder to get started with than Chart.js, but it is great if you have complex and custom data visualization requirements. Take a look at the D3.js gallery to see the variety of things you can achieve with it.

Creating 3D Graphics

One of the fascinating things that you can do using JavaScript is to create 3D graphics that can run on a web browser. Creating these graphics used to require specialized software that was thousands of dollars to buy and required hours of computer power to generate. It is incredible how far technology has come and what can be achieved using JavaScript.

One awesome library that you might want to check out for creating 3D graphics is Three.js. Three.js is a widely adopted technology that is used to create all kinds of unique 3D content and experiences on the web. Here is an introductory 3D scene created using Three.js. It simply consists of a 2d plane and a 3d sphere and camera. It tries to capture a certain amount of realism by using interactive lighting, shadows, and textures.

If you would like to know how I created this scene above, you can look at this detailed walkthrough I have created for it.

This is a relatively simple example and does not do any justice to the fascinating things you could do using Three.js! Make sure to check out their demos on the Three.js homepage to see the full spectrum of things you could do with this library.

Also worth mentioning is that you can use JavaScript to build 3D content for AR and VR scenes as well.

Creative Coding and Generative Art

So far, we have seen libraries that have a specific utility value. They are great for a particular purpose. There is a subfield of programming called creative coding, where the goal is to create something expressive rather than functional.

p5.js is a great library for creating visuals and animation and doing artistic explorations using code since it makes it easy to create visualizations.

Here is a fractal tree created using p5.js.

p5.js is not only suitable for creative coding purposes. You can create all kinds of interactive and animated works with it. Here is a simple number-guessing game built using p5.js. You can press the number that you have on the screen to try to guess the number. Three wrong guesses, and you lose!

Note that even though you can build animated works and even games using p5.js, you might want to use a dedicated game framework such as Phaser.js to build anything more involving

These examples I have introduced here scratches the surface of the things that you could do with JavaScript! Using JavaScript, you can build mobile and desktop applications, perform machine learning/AI-related tasks, program internet of things (IoT) devices. There is simply no other language as accessible and powerful as JavaScript, which makes it an awesome choice to start learning coding with!

Web Scraping in JavaScript and Node.js using Puppeteer

Engin Arslan — Wed, 01 Sep 2021 21:16:01 +0000

Internet is a source for all kinds of useful (and useless) data. Most people manually access that data by using a web browser. You can visit a website using a web browser to do things like checking out social media, getting the latest news, or check stock/cryptocurrency prices.

Another way of accessing data is to use an API. API is short for Application Programming Interface. A Web API defines the way we can programmatically access and interact with a remote resource. This way, we can consume data on the web without using a web browser. For example, we can use an API of a money exchange to programmatically fetch the latest prices for a stock without visiting the website.

Web scraping is the act of extracting data from a website either by manual or automated means. Manually extracting data can be time-consuming, given the amount of data that is out there. Unfortunately, not every online resource has an API that you can interact with. For those cases, we can automate a browser to access a website programmatically.

We can control a browser programmatically by using JavaScript. Automating our interactions with the web by programming a browser enables us to build tools that can scrape data from websites, fill out forms for us, take screenshots or download files with a single command.

There are many libraries in the JavaScript ecosystem that would allow us to control a browser programmatically. The package that we will be using for this purpose is called Puppeteer. It is a well-maintained library that is developed by the teams at Google.

Puppeteer allows us to control a Chrome (or a Chromium) browser programmatically. When we control a browser without any graphical User Interface (UI), it is said to be running in a headless mode.

This post assumes that you are comfortable using the JavaScript async-await pattern that is used for writing asynchronous programs. JavaScript has a couple of patterns that are used for dealing with the asynchronous program flow, such as callback functions and Promises. async-await is an asynchronous programming structure that got introduced into JavaScript after Promises. It makes working with asynchronous code a lot easier. Using async-await, we can write code that almost reads like synchronous code. Using async-await makes working with Puppeteer much easier.

This post will also assume a basic knowledge of Node.js, HTML, CSS, and JavaScript DOM APIs. If you are not comfortable with any of these subjects make sure to check out my book Awesome Coding that teaches you these fundamentals and a lot more! You can find the source code for the program we write here at: https://github.com/hibernationTheory/awesome-coding/tree/master/sections/05-puppeteer/puppeteer-03-project-wiki

Prerequisite Skills

JavaScript
Node.js (Beginner Level)
HTML and CSS (Beginner Level)
JavaScript DOM APIs (Beginner Level)

Getting Started with Puppeteer

Let's install Puppeteer to start working with it. This post will assume that you have Node.js and npm installed on your machine. We will begin with creating a new folder for our project and running the npm init command in that folder to create a package.json file in there.

Now that we have the package.json file created. We can install the puppeteer library by running this command:

npm install --save puppeteer@3.0.4

This installation might take a while since it downloads a version of the Chromium browser compatible with this library.

After downloading the file we can create a file called main.js and start coding inside it.

Here is an example of a Puppeteer program that programmatically launches a headless browser to visit a website and then takes a screenshot of that site to save it onto the computer.

const puppeteer = require("puppeteer");

async function main() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto("https://example.com");
  const filePath = "example.png";
  await page.screenshot({ path: filePath });

  await browser.close();
}

main();

We start our code by importing the puppeteer library. After that, we define an async function called main and then call it at the end of our program. The main logic of our program resides inside the main function.

Inside the function body, we first launch a browser instance by calling puppeteer.launch(). Whenever we launch a browser, we should remember to close it to not to cause any memory leaks from our program. A memory leak means the program that is not working is still consuming the resources of the system. We close the browser by calling browser.close().

We launch a new page inside that browser by calling browser.newPage(). We then visit the [example.com](http://example.com) domain inside that page by using the page.goto method. We take a screenshot of the page by using the page.screenshot method and save that screenshot into the same folder that we have called the program from. We then ensure that we are closing the browser and exit the program.

Now that we know the basics of Puppeteer let's build a simple project to put our knowledge into use.

Using our Puppeteer knowledge, we will build a program that will fetch a random Wikipedia article every time it runs.

Let's look at how we would manually perform such a task to understand how we would automate it. In this case, we need to visit the website for Wikipedia (https://en.wikipedia.org) and click on the link named Random Article to take us to a random article page. On each article page, there is a title and an introductory paragraph.

We will need to follow the same steps with our Puppeteer program. We will visit the URL for random results and fetch the HTML elements with the title and the description. We would then need to display these results on the screen.

The URL for the Random Article page is https://en.wikipedia.org/wiki/Special:Random. We can get this value by right-clicking on this link and selecting Copy Link Address. We will start by writing a program that will visit this URL and take a screenshot.

const puppeteer = require("puppeteer");

async function main() {
  const browser = await puppeteer.launch();

  const page = await browser.newPage();
  const urlPath = "https://en.wikipedia.org/wiki/Special:Random";
  await page.goto(urlPath);
  const filePath = "example.png";
  await page.screenshot({ path: filePath });

  await browser.close();
}

main();

Everytime we run this program we are capturing a new screenshot from the visited URL.

We can inspect the HTML structure of an article page in a Chrome browser by clicking View > Developer > Inspect Elements. We would see that the title of the article is defined inside an h1 tag. This means that we can get the title data by running the code below inside the developer console when we are on an article page.

const title = document.querySelector("h1");
const titleText = title.innerText;

We can use Puppeteer to execute this code in the context of a webpage. We can use the page.evaluate function for this purpose. page.evaluate takes a callback function as an argument that gets evaluated in the current web page context. What we return from this callback function can be used in the Puppeteer application.

const puppeteer = require("puppeteer");

async function main() {
  const browser = await puppeteer.launch();

  const page = await browser.newPage();
  const urlPath = "https://en.wikipedia.org/wiki/Special:Random";
  await page.goto(urlPath);
  const filePath = "example.png";
  await page.screenshot({ path: filePath });

  const title = await page.evaluate(() => {
    const title = document.querySelector("h1");
    const titleText = title.innerText;

    return titleText;
  });

  console.log(title);

  await browser.close();
}

main();

Here we are capturing the value of the h1 tag in the webpage context and returning that value to the Puppeteer context.

const title = await page.evaluate(() => {
  const title = document.querySelector("h1");
  const titleText = title.innerText;

  return titleText;
});

page.evaluate can be a little unintuitive since its callback function can't refer to any value in the Puppeteer context. For example, we can't do something like the following example when using the page.evaluate function:

const selector = "h1";
const title = await page.evaluate(() => {
  const title = document.querySelector(selector);
  const titleText = title.innerText;

  return titleText;
});

console.log(title);

This program would throw an error. The selector variable doesn't exist inside the webpage context, so we can't refer to it from there. If we wanted to pass data to the webpage context, we could do so by providing it as an argument to the page.evaluate and its callback function.

const selector = "h1";
const title = await page.evaluate((selector) => {
  const title = document.querySelector(selector);
  const titleText = title.innerText;

  return titleText;
}, selector);

console.log(title);

In this example, we are passing the selector variable as the second argument to the page.evaluate function as well as an argument to the callback function.

In our program, let's get the first paragraph of the article as well. Looking at the HTML structure, it seems like the p element we are looking for is inside an element with the class value mw-parser-output. That element, in turn, is inside the element with the id value mw-content-text. We can select all p elements inside that container with this CSS selector: #mw-content-text .mw-parser-output p.

const [title, description] = await page.evaluate(() => {
  const title = document.querySelector("h1");
  const titleText = title.innerText;

  const description = document.querySelector(
    "#mw-content-text .mw-parser-output p"
  );
  const descriptionText = description.innerText;

  return [titleText, descriptionText];
});

We are now getting both the title and the first paragraph from the article page. We are returning them to the Puppeteer context as an array. We are using array destructuring to unpack these values. Let's also get the URL of the current page by using the window.location.href variable.

const [title, description, url] = await page.evaluate(() => {
  const title = document.querySelector("h1");
  const titleText = title.innerText;

  const description = document.querySelector(
    "#mw-content-text .mw-parser-output p"
  );
  const descriptionText = description.innerText;

  const url = window.location.href;

  return [titleText, descriptionText, url];
});

This is looking pretty great. We can format these values that we are capturing using a template literal and display them on the screen using console.log.

const puppeteer = require("puppeteer");

async function main() {
  const browser = await puppeteer.launch();

  const page = await browser.newPage();
  const urlPath = "https://en.wikipedia.org/wiki/Special:Random";
  await page.goto(urlPath);
  const filePath = "example.png";
  await page.screenshot({ path: filePath });

  const [title, description, url] = await page.evaluate(() => {
    const title = document.querySelector("h1");
    const titleText = title.innerText;

    const description = document.querySelector(
      "#mw-content-text .mw-parser-output p"
    );
    const descriptionText = description.innerText;

    const url = window.location.href;

    return [titleText, descriptionText, url];
  });

  console.log(`
Title: ${title}
Description: ${description}
Read More at: ${url}
`);

  await browser.close();
}

main();

This code works great so far, but I am noticing that the description text is sometimes empty. Looking at the article page, this seems to happen when the first p element has a class called mw-empty-elt. Let's update our code to check to see if the first element's class name is equivalent to mw-empty-elt. If so, we would use the second p element instead. We can use the document.querySelectorAll function to get an array of all HTML elements that match the given CSS selector.

const puppeteer = require("puppeteer");

async function main() {
  const browser = await puppeteer.launch();

  const page = await browser.newPage();
  const urlPath = "https://en.wikipedia.org/wiki/Special:Random";
  await page.goto(urlPath);
  const filePath = "example.png";
  await page.screenshot({ path: filePath });

  const [title, description, url] = await page.evaluate(() => {
    const title = document.querySelector("h1");
    const titleText = title.innerText;

    let descriptionParagraph;
    const descriptionParagraphs = document.querySelectorAll(
      "#mw-content-text .mw-parser-output p"
    );
    const firstDescriptionParagraph = descriptionParagraphs[0];
    if (firstDescriptionParagraph.className === "mw-empty-elt") {
      descriptionParagraph = descriptionParagraphs[1];
    } else {
      descriptionParagraph = descriptionParagraphs[0];
    }

    const descriptionText = descriptionParagraph.innerText;

    const url = window.location.href;

    return [titleText, descriptionText, url];
  });

  console.log(`
Title: ${title}
Description: ${description}
Read More at: ${url}
`);

  await browser.close();
}

main();

This program is now in a pretty good spot! We have added the logic to choose the second paragraph if the first one has the class name mw-empty-elt.

let descriptionParagraph;
const descriptionParagraphs = document.querySelectorAll(
  "#mw-content-text .mw-parser-output p"
);
const firstDescriptionParagraph = descriptionParagraphs[0];
if (firstDescriptionParagraph.className === "mw-empty-elt") {
  descriptionParagraph = descriptionParagraphs[1];
} else {
  descriptionParagraph = descriptionParagraphs[0];
}

const descriptionText = descriptionParagraph.innerText;

And that is pretty much it for this project! One thing to note is how we rely on specific ID and class names to be present on the webpage for our program to work. If the HTML and CSS structure of the website we are scraping is to be updated, we would also need to update our program.

Things to Keep in Mind with Web Scraping

Performing manual operations in a programmatic way gives us a lot of leverage. If we have a program that can access a single website, it can be a simple matter to scale it to access thousands of websites.

This can be problematic when interacting with the web. If we were to load thousands of pages from a single domain in a short amount of time, it could potentially overwhelm the servers hosting those pages. It can even be interpreted as an attack by the website. Our IP can temporarily get blocked from accessing their resources or even get banned. We need to be mindful when using websites programmatically. We might want to add artificial delays in between our operations to slow our program down. We also need to be careful about what data we can access programmatically. Some websites try to limit programmatic access to protect their data, or there can even be legal implications for accessing and storing certain kinds of data.

Summary

Automating a browser is not the only way to access data on a webpage. There are many web applications out there that expose an API to connect developers with their resources. An API is an Application Programming Interface that we can use to interface with a resource programmatically. Using APIs, developers can build applications on top of popular services such as Twitter, Facebook, Google, or Spotify.

In this post, we used Puppeteer in Node.js to scrape data from websites. We have used the JavaScript async-await structure to manage the asynchronous data flow. We have also used CSS selectors to grab data from the HTML structure inside a web page using DOM API methods such as document.querySelectorAll

Web scraping is the act of using programs like Puppeteer to access and harvest data from websites programmatically. There can be legal implications for web scraping, so you should do your own research before engaging in such an action.

Hope you enjoyed this post! Feel free to check out my book Awesome Coding that teaches coding using JavaScript and Node.js with cool and useful projects such as this!

Coding for Statistics: Develop an Intuition for Statistics Using Programming

Engin Arslan — Fri, 23 Jul 2021 22:15:36 +0000

I have always been fascinated by statistics. In my opinion, It is one of those subjects that is useful regardless of what your domain is. It is not only helpful as a tool to understand the world better but also captivating in its own right. Subjects like Bayesian Theorem can really go against our intuitive tendencies and reveal the shortcomings in our thinking.

If you don't know what Bayesian Theorem is, you should read up about it. In short, it tells us that the possibility of occurrence of an event is primarily dictated by the prevalence of that event in the general population. It sounds like an obvious statement when it is put that way, but we have a strong tendency to overlook this.

Say you are exhibiting some inexplicable medical symptoms, and search the internet to understand what that might mean. The internet lets you know, in a typical fashion, that some form of rare cancer might explain your symptoms. Now your inclination might be to believe that you might indeed have a high chance of having that disease since there is an overlap in symptoms. But, controlling for everything else, the probability of you having that problem is still incredibly low even though you have some of those symptoms. Remember that this is a rare disease. You still need to be very unlucky to actually have it. You having those symptoms at the moment doesn't increase the probability as much as your imagination might lead you to think, assuming those symptoms can have other, more common, explanations. This is not to say you shouldn't seek medical advice, but you can actually keep a cool head till you get an explanation and not stress over something you have read on the internet.

The Monty Hall Problem

The Monty Hall Problem, a problem inspired by a popular game show in the 60s, is a statistical puzzle that can be equally confusing.

"Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?" (from Wikipedia)

I don't know about you, but my intuitive answer to this question would be not to switch. I have already made a choice, and what the host does shouldn't have any implication over my initial choice. Why do I care if some other door turned out to be a dud when I didn't pick it anyway.

It turns out you should always switch to the other door. You have 1/3 of a chance of winning the car if you don't switch and 2/3 of a chance if you switch. You double your chances by switching! Don't worry if this doesn't make any sense. This is a famous statistical problem and has confused many people, including hundreds of PhDs and even prolific mathematicians. My first response to reading about the problem was to write a program to illustrate the result. If I can see the problem occurring with my own eyes, then I can develop an understanding of it. Programming brings abstract to reality. Here is how to go about it.

We will first create a function that sets up the game. We can simplify the game setup to be represented by an array that has two strings with the value goat and one string with the value car.

function getDoorSetup() {
  const setup = ["goat", "goat", "car"];
  return setup;
}

Remember that the order of these items should be random. We will pick an index number, and that will determine the result. If the value at the picked index of the array is the car, then we win; if not, we lose. To randomize the order of this array, I will be using a JavaScript library called lodash. lodash is a popular library with a bunch of utility functions that make working with JavaScript data types easier. It has a function called shuffle that takes an array and returns a new array with the order of the items changed randomly.

function getDoorSetup() {
  const setup = ["goat", "goat", "car"];
  return _.shuffle(setup);
}

Next, we want to have a function representing the choice we will make in the game. We will call this function getDoorPick. It will generate a number between 0 and 2. These numbers will correspond to the indices of the door setup array. We will use a lodash function called random that returns a random integer in between given numbers.

function getDoorPick() {
  const doorPick = _.random(0, 2);
  return doorPick;
}

Now we can create a function called playGame that will simulate playing this game. In this function, we will set up the game and pick a door number. If the door number we picked has the value car, we win; if not, we lose. We will return 1 from the function if we win, 0 if we lose.

function playGame() {
  const doorSetup = getDoorSetup();
  const doorPick = getDoorPick();

  if (doorSetup[doorPick] === "car") {
    return 1;
  }

  return 0;
}

This is a fairly simple setup. This is the point where we can start utilizing the power of programming. We can already simulate what happens when we play this game once. But we can also easily simulate what happens when we play this game 1000 times (or even millions of times).

let counter = 0;
let rounds = 100;
let wins = 0;

while (counter < rounds) {
  const result = playGame();
  wins = wins + result;
  counter = counter + 1;
}

console.log(`You have won ${wins} games`);

We have used a while loop that runs the amount that is defined with the rounds variable, which is 100. This simulation will run 100 times. We are using the wins variable to count the number of times we win the game. We console.log the result. Try running the program couple of times. You will notice that the number of times you win will hover around 33. This is 1/3 of 100. The result makes sense since there are 3 options; we have 1 in 3 chances of winning the game.

We now need a function that will return a door number in the doorSetup that will not be our pick, and it is not the number with the car either. I will call this function getGoatNotAtDoorNumber.

function getGoatNotAtDoorNumber(setup, doorNumber) {
  let goatDoorNumber;

  setup.forEach((item, itemNumber) => {
    if (item === "goat" && itemNumber !== doorNumber) {
      goatDoorNumber = itemNumber;
    }
  });

  return goatDoorNumber;
}

We will call this function with the doorSetup and doorPick to simulate the host revealing some other door with a goat behind it.

const revealedGoatPosition = getGoatNotAtDoorNumber(doorSetup, doorPick);

And we will create one more function called getFinalPick. This function will simulate the act of switching or not switching. If we choose not to switch, then we just stick to our original pick. If we switch, we should pick the door number that we didn't pick and the one that isn't revealed. Here is what that function looks like:

function getFinalPick(ourPick, revealedPick, changePick) {
  if (!changePick) {
    return ourPick;
  }

  const possibleDoorNumbers = [0, 1, 2];
  return possibleDoorNumbers.filter((doorNumber) => {
    if (doorNumber !== ourPick && doorNumber !== revealedPick) {
      return true;
    }
  })[0];
}

Using these two functions we can now have a more complete setup of the game.

function playGame() {
  const setup = getDoorSetup();
  const ourDoorPick = getDoorPick();
  const revealedGoatPosition = getGoatNotAtDoorNumber(setup, ourDoorPick);
  const switchDoors = false;
  const ourFinalPick = getFinalPick(
    ourDoorPick,
    revealedGoatPosition,
    switchDoors
  );

  if (setup[ourFinalPick] === "car") {
    return 1;
  }

  return 0;
}

Try changing the switchDoors variable from false to true. You will suddenly see that you are winning twice many games on average.

Isn't this amazing? We have created a computer simulation that can help us build a feel for the problem before developing an understanding of it.

Here is the code for you to tinker with:

Let's take a look at another statistical problem called Birthday Paradox.

Understanding the Birthday Paradox

The Birthday Paradox, according to Wikipedia, states that in a group of randomly chosen people, the probability of having a matching birthday is 50% for a group of 23 people. The counterintuitive part of this problem is that 23 people seems like a low number of people to generate such a high probability. But the math checks out. We can easily show it using programming again.

We can first generate a random number between 1 and 365 to represent a birthday. We can again use lodash to generate the random number.

function getRandomBirthday() {
  return _.random(1, 365);
}

Next, we can create a function called getRandomBirthdays to generate an array of the desired length consisting of these random numbers. We can think of this array representing the birthdays of a roomful of people.

function getRandomBirthdays(length) {
  const randomBirthdays = [];

  for (let i = 0; i < length; i++) {
    const randomBirthday = getRandomBirthday();
    randomBirthdays.push(randomBirthday);
  }

  return randomBirthdays;
}

Now the question is, what is the probability of finding a pair of numbers that match, which means a pair that essentially has the same birthday. We need a function that would tell us if there are matching numbers in our array.

function hasSameBirthday(arr) {
  const arrWithNoDuplicates = [...new Set(arr)];
  if (arrWithNoDuplicates.length !== arr.length) {
    return true;
  }

  return false;
}

There are probably a lot of ways of identifying if there is a repeating number in an array. Here, we are first converting the given array to a set. A Set is a collection that is similar to an array but can't have any duplicate entries. Then, we are converting the set back to an array on the same line.

const arrWithNoDuplicates = [...new Set(arr)];

This operation removes any duplicate entries from a given array. It is a neat trick that I sometimes use in my code. Then we compare the length of the resulting array to the original array. If there is a difference, that means the original array had some duplicate entries that got removed. This means the function should return true to indicate there are matching birthdays in the array. If not, we return false.

Now we can create our simulation to see how the probabilities work:

function simulate() {
  const rounds = 100;
  let counter = 0;
  let matches = 0;

  while (counter < rounds) {
    const arr = getRandomBirthdays(23);
    const hasDuplicate = hasSameBirthday(arr);
    if (hasDuplicate) {
      matches = matches + 1;
    }
    counter = counter + 1;
  }

  console.log(
    `There is ${(matches / rounds) * 100}% chance that there is match`
  );
}

simulate();

Running this code we would notice that the number hovers around 50% for 23 numbers. Decreasing that number by half reduces the probabilities almost by a third. We can play with other numbers to see how do the probabilities get affected by the input.

Conclusion

Statistics can be unintuitive and confusing. Programming allows us to cut through the complexity and let us develop a feel for these concepts. I can't claim to have a deep understanding of anything that I wrote about in this post, but thanks to the awesome power of coding, I know better to switch the door if given a chance. If you are curious about other awesome applications of programming you should check out Awesome Coding!

Programming for Art

Engin Arslan — Tue, 08 Jun 2021 20:36:26 +0000

Programming is not something that readily comes into mind when thinking about art, but many applications of programming allow for artistic expression. There is even a field of programming called creative coding where the goal is not to create something functional but rather expressive.

Even though programming for art has been a thing for a while, it used to be fairly inaccessible as it required deep knowledge in the field. Take Demoscene, for example. Demoscene is a computer art subculture that has been around since the 1980s. It involves creating audiovisual presentations, called demos, that are not only artistic creations but also technical achievements.

It can be challenging, even for a programmer, to understand how a demo is implemented. That is not to say other fields of art are devoid of technicalities. But, I can hope to create something that resembles a painting given a brush and a paint but creating a programmatic art given a computer and a programming environment can remain a monumental challenge for a non-expert.

Nowadays, we have modern, high-level programming languages and libraries that make programming a lot more accessible. Tasks like creating a website, building a game, or creating programmatic art have become a lot easier

If you are not too familiar with programming, there are two concepts to unpack there. High-level programming languages and programming libraries.

High-Level Programming Languages

According to Wikipedia, a high-level programming language is a programming language with strong abstraction from the details of the computer. Essentially, high-level refers to the level of abstraction. A high-level abstraction means a concept is presented in a way that is stripped away from the underlying details. Think of using a phone. At a high level, to use a phone, you would need to turn it on and press the buttons on the screen. At a low level, the battery in the phone needs to power the processor of the phone, and the touch screen needs to register the pressure, etc. Honestly, I don't even know how a phone works at a low level to give accurate examples here. The good thing is that I don't need to. I am still able to use a phone without knowing how it works. That is made possible by abstracting away the low-level details.

A high-level programming language provides similar flexibility. It allows you to program the computer without having to worry about low, machine-level details. This is one of the factors that makes programming so much easier than it used to be. With a low-level programming language, you need to worry about how the memory is allocated in your program, how it is garbage collected, etc... Essentially, we would need to tackle concerns that don't directly relate to the task at hand. High-level programming languages abstract away those details so you can focus on the primary goal and ignore the complex peripheral issues.

Another thing that makes programming easier is programming libraries.

Programming Libraries

A programming library (or package) is a bundle of code that is previously written for a specific purpose that is packaged and shared for reusability.

Imagine trying to build a first-person shooter game. If we had to write every piece of functionality we need by ourselves, we would have to write a lot of code to handle the 3D graphics. That means a lot of complex mathematical operations would need to be implemented in code. We would have to spend weeks or even months developing the foundational pieces before doing anything unique to our game. Luckily, creating and displaying 3D graphics on the computer is a solved problem. Many other smart people out there had to work on similar issues, and some of them choose to share their solutions to these problems in the form of programming libraries.

That is why getting started with programming is so much easier nowadays than it was before. You can simply install a library that would do the most complex tasks for you to focus on building things that will be unique to your application.

There are libraries out there that would allow you to build 3D graphics, data visualizations, e-commerce solutions, and even artificial intelligence applications. Nowadays, building a software application is mostly about figuring out how to use these libraries and bringing them together in ways that are unique to what you are trying to build.

A high-level programming language and programming language libraries can help you focus on creating art using programming instead of wrestling with technicalities.

One of the most suitable programming languages for this task is JavaScript. JavaScript is a modern, dynamic, and expressive programming language with vast popularity and a huge ecosystem. It has a lot of programming libraries that would help you to create all kinds of artistic works. One example of those libraries is p5.js. p5.js is a creative coding library that makes it easy to create visuals and animations. Here is an editable example below for a visual created by p5.js.

Programming Libraries for Art

It is hard to prescribe any programming libraries for programming art since what you choose to use is only limited by your creativity. Nevertheless, here are some programming libraries in JavaScript that can help you in programming artistic works.

Three.js

Three.js is a JavaScript library that allows you to create 3D visuals in the web browser. It has been around for a long time and is used to deliver a lot of impressive projects. If you want to create 3D, Virtual Reality, or Augmented Reality experiences on the web, then Three.js is the tool for the job. We could already create 3D graphics in the browser by using an API (application programming interface) called WebGL but Three.js abstracts away the complexity of this API to make working with 3D much easier.

ml5.js

Machine Learning is an exciting frontier in computer science that is ripe for artistic exploration. Unfortunately, working with machine learning can be quite challenging for someone who doesn't have in-depth knowledge in the field. ml5.js is a JavaScript programming library that makes it easy to work with machine learning by providing high-level abstractions.

Matter.js

Matter.js is a 2D physics engine that works in the web browser. It allows you to create realistic interactions on and in between shapes by simulating forces such as kinetic or gravity. You can combine it with other libraries such as p5.js to create visuals that can interact with each other in realistic ways.

These are just a few examples of all the awesome libraries in the JavaScript ecosystem that can help you create art and express your creativity. There are a lot more examples, such as Phaser.js for creating games, howler.js for working with audio, natural for natural language processing, etc.

This is not to say even a person without any coding knowledge can start programming as soon as they sit in front of a computer. But thanks to modern programming languages and libraries, programming for art is a lot more accessible than before for anyone willing to learn the basics of programming. I have created Coding for Visual Learners exactly for this purpose. It is not specifically a Programming for Art course, but it teaches programming in an artist-friendly way. Once you learn programming fundamentals, you can apply it to whatever you want, whether creating art, building web applications, or even programming robots!

How to use Youtube JavaScript API to Search for Videos with Node.js

Engin Arslan — Mon, 10 May 2021 14:10:17 +0000

It goes without saying that nowadays, Youtube is an amazing resource for any kind of information. From educational content to entertainment, from documentaries to live shows, Youtube has it all. But with this kind of scale, it is becoming extremely challenging to discover truly great content. It has some pretty decent filtering options, but this might not be enough for a sophisticated user. With all the algorithmic power of Youtube that is at your disposal, try to find the best programming talk from 2010 to 2012, sorted by most comments. It is not as easy as you would think.

That's why I have built Awesome JavaScript Talks as a weekend project. It simply is a collection of JavaScript conference talks that can be sorted in a way that would allow you to find the most relevant and interesting talks according to your needs. I don't have any grand ambitions for this project. It was simply a means for me to discover awesome talks that relate to JavaScript. But working on this project was an educational experience in interacting with the Youtube JavaScript API. That's why I have decided to share my learnings in a beginner-friendly tutorial.

In this tutorial, I will show you how to write a program that will search Youtube for public videos using JavaScript and Node.js and save that information on your computer. Here we go!

Getting a Youtube API Key

Getting an API key is my least favorite step when interacting with any API Service since the steps involved can vary a lot depending on the platform we use.

API Keys are used as authorization credentials in Youtube. Authorization gives us access to the resources on Youtube through the API.

There are two different kinds of credentials that we might need depending on the kind of service we are building on top of the Youtube API.

OAuth 2.0
API keys

If we wanted to build a program that is going to perform actions on behalf of a user we would need an OAuth 2.0 token. We won't be building such a program so API keys will suffice for our purposes.

We can first visit [https://console.developers.google.com/apis/credentials](<https://console.developers.google.com/apis/credentials>) and create a project by clicking on Create Project.

Google will prompt us to enter a project name. We can enter whatever you want. We don't have to choose a Location either. We then click Create.

Clicking Create Credentials creates the API key. We will see a dialog box that shows the API key that is created for us. We should treat this key just like any other secret, like a password. Please make a note of this key since we will be using it in our program.

Enabling Access to the Youtube Data API

Google has a lot of APIs that are available. The API that we are interested in is the Youtube Data API. We can go to the Library section through the sidebar and search for the Youtube Data API.

Click on the Enable button to activate this API for our project. We will be using the API key that we have created to access this service. This is pretty much it. We can now start coding!

Creating a new Node.js Project

Create a folder for the Node.js project and run npm init -y in there to initialize an empty package.json file. We will be using the package.json file to install libraries for this project.

mkdir project-youtube-api
cd project-youtube-api
npm init -y

We can now create two files in this folder. A main.js where we will write the functions that we will use to interact with the Youtube API and a config.js we will use to store the Youtube API key. We can create these files through the graphical interface of our operating system or the command line.

touch main.js
touch config.js

Inside the config.js, we will just have a variable to hold the API key and export that variable to be used from main.js.

const YOUTUBE_API_KEY = "";

module.exports = {
  YOUTUBE_API_KEY,
};

Make sure to assign your API key to the YOUTUBE_API_KEY variable. Next, we will make use of this key from inside the main.js.

const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

function main() {
  console.log("Ready to get Youtube data!");
}

main();

This file currently loads (imports) the YOUTUBE_API_KEY from the config.js file in the same folder. It then checks to see if the YOUTUBE_API_KEY exists. It throws an error if it doesn't.

If the YOUTUBE_API_KEY exists, we proceed to execute the main function. What we would like to do is to use this API key to authenticate with the Youtube API. We will interact with the Youtube API by using the node-fetch library.

Interacting with the Youtube API using Fetch

We will install the node-fetch package to be able to interact with the Youtube API.

npm install --save node-fetch@^2.6.1

We will make use of this library inside the main.js file.

const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function main(query) {
  console.log("Ready to get Youtube data!");
  const url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

main("JavaScript");

We are importing the node-fetch library on the first line with the require function.

const fetch = require("node-fetch");

We have changed the main function to be an async function since we would like to use the async-await pattern inside this function. Interacting with an API is an asynchronous process, and async-await will allow us to wait for the request to the API to resolve before proceeding to the next line.

This is the endpoint we are using to make a search for a given query using the Youtube API.

const url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;

There are two parameters in this URL right now. The YOUTUBE_API_KEY and the query to be searched for. When executed, this query returns an object with a field called items that have the results that we want. Let's take a look at the first item object to see what kind of properties it has.

console.log(data.items[0]);

The result would look something like this (the particular item might be different for you!)

{
  kind: 'youtube#searchResult',
  etag: 'HXpfXYuctt1KbbEEnnlYDhEiiVM',
  id: { kind: 'youtube#video', videoId: 'PkZNo7MFNFg' },
  snippet: {
    publishedAt: '2018-12-10T14:13:40Z',
    channelId: 'UC8butISFwT-Wl7EV0hUK0BQ',
    title: 'Learn JavaScript - Full Course for Beginners',
    description: 'This complete 134-part JavaScript tutorial for beginners will teach you everything you need to know to get started with the JavaScript programming language.',
    thumbnails: { default: [Object], medium: [Object], high: [Object] },
    channelTitle: 'freeCodeCamp.org',
    liveBroadcastContent: 'none',
    publishTime: '2018-12-10T14:13:40Z'
  }
}

This object has plenty of data that we can use for various purposes. Our script works, but it has a couple of problems. The items array is only returning 5 results by default. According to the totalResults field inside the pageInfo field of the returned object, we should have a million results instead! Let's address these issues.

Changing the Results Per Page in Youtube API

Solving the first issue is easy. Inside the pageInfo field, you can see a reference to a value called resultsPerPage, which is equivalent to 5 at the moment. This is the reason why we are getting 5 results. We need to add a parameter to the url variable to adjust the number of results per page. The API can't return a million results at once. It is paginated. This means the results are divided into separate sections (pages). We can make use of the maxResults parameter to change the number of results we are getting.

async function main(query, resultsPerPage) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

main("JavaScript", 25);

We have added a new parameter to the main function called resultsPerPage. We have also made the url variable use the let keyword so we can update its value. Additionally, we have added a conditional to check to see if a resultsPerPage value exists. If it does, we provide it as the value of the maxResults parameter of the URL.

let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
if (resultsPerPage) {
  url = `${url}&maxResults=${resultsPerPage}`;
}

This allows us to increase the number of results we are getting per page. But you will notice that the results are capped at 50 items. The rest of the results would be on the next page. Let's take a look at how to get those other results.

Working with Pagination in Youtube API

Notice how the returned object has a field called nextPageToken.

nextPageToken: 'CDIQAA',

The value of that field is an identifier for where we are at in the pagination. You can think of it as a page number. We can use that value in our API call to get the next page of results.

const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function getYoutubeResults(query, resultsPerPage, pageToken) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }
  if (pageToken) {
    url = `${url}&pageToken=${pageToken}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

async function main() {
  const data = await getYoutubeResults("JavaScript", 25);
  await getYoutubeResults("JavaScript", 25, data.nextPageToken);
}

main();

Here, we have changed the structure of our code a bit. I have renamed the main function to be getYoutubeResults. The sole responsibility of this function is to query the Youtube API using the given arguments. The new main function contains the logic for making specific queries using the getYoutubeResults function.

We have also made the getYoutubeResults function to accept a pageToken parameter as an argument. If this argument is provided, this function will use it when constructing the url variable.

if (pageToken) {
  url = `${url}&pageToken=${pageToken}`;
}

Now we can run this function, get the data that it returns, and use the nextPageToken field to rerun the function to get the next page of results!

async function main() {
  const data = await getYoutubeResults("JavaScript", 25);
  await getYoutubeResults("JavaScript", 25, data.nextPageToken);
}

This definitely works but feels a bit suboptimal. Surely there must be other ways to go to the next page than manually calling the getYoutubeResults over and over again.

We should have a way to collect these search results before we look at the pagination. Let's implement that first.

Collecting the Youtube Search Results

We will collect the items that are returned from the getYoutubeResults inside an array called videoData.

async function main() {
  const videoData = [];
  const data = await getYoutubeResults("JavaScript", 25);
  videoData.push(...data.items);

  console.log(videoData);
  console.log(`There are ${videoData.length} items in videoData`);

We are using the spread (...) operator to unpack the data.items array, and then we are pushing all those items inside the videoData array. We are then logging the videoData and the number of items in that array to the screen.

Now we have a mechanism to collect the results, let's build a loop that will collect data from all the pages.

async function main() {
  const videoData = [];

  let totalPages = 10;
  let nextPageToken = undefined;

  for (let counter = 0; counter < totalPages; counter = counter + 1) {
    const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
    videoData.push(...data.items);
    nextPageToken = data.nextPageToken;
  }

  console.log(videoData);
  console.log(`There are ${videoData.length} items in videoData`);
}

We introduced two new variables called totalPages and nextPageToken. totalPages is the number of pages that we would like to collect data from. nextPageToken is a variable to store the nextPageToken that is returned from the current page. We are using a for loop to loop through the pages.

for (let counter = 0; counter < totalPages; counter = counter + 1) {
  const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
  videoData.push(...data.items);
  nextPageToken = data.nextPageToken;
}

The loop currently collects data from 10 pages. We would see that we have 500 items stored in the videoData variable at the end of the loop. The last step would be to save this data on our computer.

Storing the Collected Data in a JSON file

We can save the collected data in a file. We generally use JSON data format to save JavaScript data. This format is human-readable, so you can just open the file and see how it looks like. JSON is also a file format that is natively supported in JavaScript. It is straightforward to read and write JSON files from JavaScript files. To be able to write data to the file system, we will first import the fs module.

const fs = require("fs");

The we will convert the videoData into JSON format using the JSON.stringify function.

const videoDataJSON = JSON.stringify(videoData);

We can then save this data into a file. Here, we are saving the file into a file called data.json in the same directory.

fs.writeFileSync("./data.json", videoDataJSON);

If we are to look at the saved data, we will notice it is actually not very readable. We can format this file a bit better by passing additional arguments into the JSON.stringify function.

const videoDataJSON = JSON.stringify(videoData, null, 2);

This should result in a much more readable code. Here is what the final result looks like.

const fs = require("fs");
const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function getYoutubeResults(query, resultsPerPage, pageToken) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }
  if (pageToken) {
    url = `${url}&pageToken=${pageToken}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

async function main() {
  const videoData = [];

  let totalPages = 10;
  let nextPageToken = undefined;

  for (let counter = 0; counter < totalPages; counter = counter + 1) {
    const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
    videoData.push(...data.items);
    nextPageToken = data.nextPageToken;
  }

  const videoDataJSON = JSON.stringify(videoData, null, 2);
  fs.writeFileSync("./data.json", videoDataJSON);
}

main();

Conclusion

This is it for now! There is a bit more to cover about the Youtube Data API, such as getting videos from the channels, and I will be covering that in another post. As always, be mindful of the usage quotas and limitations of the API that you are using! Feel free to connect with me at https://twitter.com/inspiratory and ask any questions you might have.

Resources

Here are some resources if you wanted to get more information about things that we have covered here:

Documentation for the Youtube Data API: https://developers.google.com/youtube/v3
Github Page for the Node-Fetch library: https://www.npmjs.com/package/node-fetch
How fetch works in general: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

Teaching Coding to Absolute Beginners

Engin Arslan — Tue, 13 Apr 2021 11:04:35 +0000

I have learned a lot of new skills by myself. I was into graphic design, so I taught myself Photoshop and Illustrator. Later, my interests have shifted to 3D modeling and animation, so I have learned 3ds Max and Zbrush. While working as a Visual Effects Artist, I have started to get interested in coding and taught myself programming with Python. When I have decided to become a full-time programmer, I got into web development, JavaScript, HTML, and CSS.

With every learning you do, there is additional learning that happens on how to learn and teach things. When you watch a course from a great teacher or listen to a talk by an exceptional speaker, you don't only receive what is being delivered. You inadvertently absorb how it is delivered as well. Learning coding by myself was an excellent education in how coding should be taught to absolute beginners. Self-teaching anything is an unintentional education in teaching, which is great because I have always wanted to become a teacher.

Luckily, my wish to become a teacher was granted when the college I went to in Canada asked me to become one of their part-time "professors." Colleges in Canada are similar to trade schools in some other countries. A lot of the teachers in these colleges are working professionals in their respective fields. Being a "professor" at a Canadian college doesn't require a pedagogical degree. Essentially, I was invited there to teach coding while learning how to teach coding myself.

I have spent five years teaching coding to absolute beginners in the Visual Effects (VFX) department. I have had a tough audience. They weren't there to learn to code as it didn't directly relate to the field they were studying for. Most of them were coming from an art background, so coding had bad connotations in their mind. I had some disadvantages as well. At one point, I changed careers to become a full-time programmer. I wasn't a VFX guy anymore. In fact, I didn't even care that much about VFX. But I was in love with coding. So I tried my best to get them to like it as well.

My learning journey and my experiences in teaching coding to absolute beginners made me incredibly opinionated in how it should be taught to beginners. Here are some of the principles that I abide by when I teach coding.

Progressive Disclosure

Not everything about a subject would be immediately relevant to you. In fact, you can go very far without knowing certain elementary things in a domain. For example, you could successfully drive a car for a long time without having any knowledge about how the engine works. It is not that the engine is irrelevant. It is just not immediately relevant. You can learn about it later.

Progressive Disclosure is an interaction design pattern where you gradually reveal an application's complexity. You might have also witnessed it in video games, where a tutorial will first try to get you comfortable with basic moves and gradually introduce more complex concepts. I deploy this in my teaching and always start with comprehensible, manageable, and applicable information at the level that the students are at.

Hide complexity

There can be things about any domain that you don't even have to learn at all. There is a chemistry to how molecules interact with each other when you are cooking, but you can be an excellent chef without knowing anything about it. I am a programmer, but I don't know some of the most basic things about how a computer works. And sincerely, I am not interested in it either. I would hide a lot of complexity that doesn't have an immediate application when I teach to a beginner. Any student with an inclination can explore the intricacies of a given domain on their own once they establish the basics in it.

Don't Oversimplify

Coding is an intellectual endeavor, and there are things about it that are absorbing and interesting. I believe in hiding complexity, but I can't entirely agree with oversimplifying things. Imagine learning a new language and not being taught a single line of poetry or an expression. Where is the excitement in that? When I was learning to code, I would dwell in forums like Hacker News to see how programmers or technologists would communicate. Not everything they said made sense to me, but I was fascinated by the richness of the domain that would surface in their conversations. I think there is a certain beauty in details, and it should not be completely hidden.

Keep it Relevant and Inclusive

When I teach, I try to have applicable examples that are relevant to my target audience. I personally find it painful when I come across a coding exercise that is not applicable or so niche that I can't even bother to think about what is being discussed. Using Star Wars characters when teaching might be fun if your audience is into that universe but can be a barrier when they haven't seen the movies. It can be hard to find something that is going to speak to everyone, though. Having a lot of different types of examples can be helpful to have good coverage. You are risking alienating your audience when you base your examples and explanation on one kind of thing.

One thing at a time

If I am learning statistics and being taught about it over an example of baseball, that means there are now two new things for me to worry about. There are much simpler examples to draw from and keep it to teaching one complex thing at a time.

Make it Actionable

It would be tough to learn how to play poker by just reading about it. At some point, you need to practice your learnings to get better at it. Coding is like that too. You can read all about it but what you learn needs to be enforced by exercises and examples. It is tough to ensure that you learned anything until you put it into practice. It is funny how your illusion of comprehension can get shattered so quickly when faced with reality. That is a good thing, though. When learning something new, you would want to identify the gaps in your knowledge as fast as possible so you can fill them quickly.

Don't make any assumptions

I don't particularly like making any assumptions about what the students might already know. As someone new to coding, they might not know what a code editor, terminal, or programming library is. I try to be mindful of this and remember to include short explanations whenever that is relevant.

Conclusion

I am a self-taught programmer, so I get learning how to code can be so challenging. It has been a while since my complete beginner days, but I still remember the frustration I would feel when I came across a resource that didn't respect some of these principles. I made sure to write Awesome Coding, my book that teaches coding from scratch, with these ideas in mind.

I derived these principles from my experiences, so I would love to hear if they overlap with the formal pedagogical methods out there. Feel free to let me know at hibernationtheory@gmail.com!