DEV Community: Keith Holliday

Basic Replication in Redis

Keith Holliday — Sun, 14 Aug 2022 21:17:31 +0000

Original post: https://koalatea.io/redis-replicate/

Intro

In this article, we will get a brief look on how to replicate in Redis. In production, you will often want to use a cluster or a manage instance. However, learning how to replicate manually will help with some ad hoc backups or debugging.

Getting Set Up

For this article, we will use docker and docker compose, so start by installing those.

Docker install: https://docs.docker.com/engine/install/
Docker Compose: https://docs.docker.com/compose/install/

Next create a new file called docker-compose.yml.

Add the following to lines to the top to declare our version and services:



version: "3.2"
services:

A warning, Redis still uses the old Master and Slave language, but this is old and should be changed.

With that in mind, let's add our master (main) service called redis. Everything here is standard except we pass REDIS_REPLICATION_MODE=master



redis:
    image: "redis:alpine"
    command: redis-server --requirepass some_pass
    ports:
     - "6379:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
     - REDIS_REPLICATION_MODE=master

Next, let's add a single slave (worker) server. Here we change the local mapping port to 6378, and set the replication mode to slave.



redis2:
  image: "redis:alpine"
  command: redis-server --requirepass some_pass
  ports:
    - "6378:6379"
  volumes:
    - $PWD/redis-data:/var/lib/redis
    - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
  environment:
    - REDIS_REPLICATION_MODE=slave

The full file looks like the following.



version: "3.2"
services:
 redis:
    image: "redis:alpine"
    command: redis-server --requirepass some_pass
    ports:
     - "6379:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
     - REDIS_REPLICATION_MODE=master

 redis2:
    image: "redis:alpine"
    command: redis-server --requirepass some_pass
    ports:
     - "6378:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
     - REDIS_REPLICATION_MODE=slave

Now, start the services using docker-compose up.

Connecting to the Redis Servers

Next we need to connect to our redis servers. For this, I use an open source GUI:

https://github.com/qishibo/AnotherRedisDesktopManager

Click "New Connection" and enter the information like the following. You can change the Connection Name to whatever you want. For this, I named it "Main". The password should be some_pass if you used the yml file above.

Do the same for the worker, but change the port to 6378 and change the connection name.

Replicating the Server

To start, expand the first server and add a new key.

Now, open the connection to the worker server and click the cli button to connect to the server.

First, set the master pass to some_pass (since we used the same password for both servers).



config set masterauth some_pass

Then, use the replicaof command to point to the master server.



replicaof redis 6379

Click the refresh icon, and you should see the master server key replicated. Add more keys to the master and the worker will continue to replicate.

To stop replication, you can use the following.



replicaof no one

Conclusion

That's it for the basics of replication. This helps if you want to quickly copy down some data for testing. There is much more to explore, but this should be a good start.

Getting Started with JUnit in Java

Keith Holliday — Fri, 29 Jul 2022 17:17:49 +0000

Original Post: https://koalatea.io/java-junit-getting-started/

Intro

Java has a nice ecosystem to get you started with testing. Libraries such as JUnit, Mokito, and Sprint test handle most of your day to day testing needs. In this article, we will learn how to get started with testing using JUnit.

Getting Set Up

Most IDEs or Code Editors will generate the initial test files for you when using Java, so this part is pretty straight forward. We will be using VSCode for this tutorial, but feel free to use any IDE such as Eclipse or JetBrains and create a new "gradle project".

Plugins

Open up a new window of VSCode and go to the plugins. Install the "Extension Pack for Java" and "Gradle for Java" plugins as show before.

New Project

Now, use the short cut "ctr+shift+P", "cmd+shift+P for mac, or go to Help -> Show All Commands to open the Command Menu. Then search for gradle and select the "Gradle: Create a Gradle Java Project..."

When the name is asked, type teststarter to follow this tutorial. After following the prompts, your project should look like the following. And JUnit will already be installed.

Creating the Test

In the file tree on the left of VSCode, go to src/test/java/teststarter, right click and create a new file called CalcTest.

The following code should be added for you. If not, copy the code below and paste it in the new file.

package teststarter;

public class CalcTest {
}

Now, let's create a function called testSum which will test our new Calc service we will create.

package teststarter;

public class CalcTest {
    void testSum() {
    }
}

Let's add the @Test annotation which is from JUnit. This will tell the compiler to run the function as a test.

package teststarter;

import org.junit.jupiter.api.Test;

public class CalcTest {
    @Test void testSum() {

    }
}

You may also need to import JUnit. Hover over the @Test and type the hotkey ctrl+. to get an auto suggest menu, and select to import JUnit.

Now, we can start to write our test. First, we will create an instance of our class to test. Note, this class doesn't exist yet.

Calc calc = new Calc();

Next, we need to defined our "actual" and "expected" to compare. The actual will call the sum function from our class. We will create this method in a minute. It will take a list of numbers and sum them up.

int actual = calc.sum(new int[] {1, 3, 4});

Then, add the expected value of 8.

int expected = 8;

Finally, we will add an assertEquals that is a method from JUnit to compare the two values. You will need to use the ctrl + . shortcut to import or copy the imports below.

assertEquals(expected, actual);

The final code should look like this.

package teststarter;

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.Test;

public class CalcTest {
    @Test void testSum() {
        Calc calc = new Calc();

        int expected = 8;
        int actual = calc.sum(new int[] {1, 3, 4});

        assertEquals(expected, actual);
    }
}

Creating the Class

We can create the class by using the left file browser tree, going to src/main/java/teststarter, right click, then create a new file. Call this Calc.java. Paste the following code which is a class with one method to sum a list of numbers.

package teststarter;

public class Calc {
    public int sum(int[] data) {
        int sum = 0;

        for (int i : data) {
            sum += i;
        }

        return sum;
    }
}

Run the Test

To end this tutorial, we can go back to our CalcTest.java and next to the testSum method, there should be a "play" button. Click that button and your test should run and pass.

Day to Day React: Api Integration

Keith Holliday — Thu, 21 Jul 2022 17:15:03 +0000

Intro

Recently while teaching, I started to compile a list of "day to day" activities that a React developer would do. One common activity is integrating a component with an api. This consists of making a request to an external api and displaying the data. As a bonus, I would add a element to interact and change parameters sending to this api.

With this in mind, this article will cover a simple step by step plan that can be repeated to practice this daily activity in preparation for working as a React Developer.

Getting the Api

The first step is to select an api and preview it's contents.

To select an api for practice, I'd recommend this list of public apis on Github.

https://github.com/public-api-lists/public-api-lists

Previewing the Api

The next step is to preview the api. For this example, we will use the Meowfacts api: https://github.com/wh-iterabb-it/meowfacts.

When previewing the api, we can make GET requests in our browser or use something like Hopscotch.io or Postman. (I use hopscotch since it is open source).

From the github page, there is one link: https://meowfacts.herokuapp.com/, so opening that in the browser, we can see the following.

And in Hopscotch, we make a GET request as well.

Setting up React

If you don't have a react app set up already, let's create one. In your normal work environment, this will be created for you usually.

npx create-react-app api-example

Now install and start the app.

cd api-example
npm start

Hardcode the Response

The order doesn't necessarily matter, but for the next step I like to add a hard coded response then mock up the UI around it. Pulling from Hopscotch, we can use the following as an example response.

{
  "data": [
    "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
  ]
}

Open up a component. If you are following along, open up App.js. Let's add the response as a hard coded variable inside the App component. I also removed some of the boilerplate html.

function App() {
  const catResponse = {
    data: [
      "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
    ]
  }

  return (
    <div className="App">

    </div>
  );
}

Now we can use the variable to render the response. For this we can just use a simple un ordered list element. We will also use a React code block {} to interpolate our variable into the html.

function App() {
  const catResponse = {
    data: [
      "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
    ]
  }

  return (
    <div className="App">
      <ul>
        <li>{catResponse.data[0]}</li>
      </ul>
    </div>
  );
}

This is good so far, but let's take it one step further. The data property is an array, so we can expect more facts to be in that array. Because of that, let's convert our html to use map to render our lists. Note that I leave the verbose syntax to be a little easier for newer js devs to read.

function App() {
  const catResponse = {
    data: [
      "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
    ]
  }

  return (
    <div className="App">
      <ul>
        {catResponse.data.map((fact) => {
          return (
            <li>{fact}</li>
          )
        })}
      </ul>
    </div>
  );
}

Convert The Response to State

Now that we are rendering the response, we want to convert this to a state variable. In the end, we will make a request and update our variable. To do this, we need to add in the useState hook.

The new line of code is this:

const [catResponse, setCatReaspon] = useState({})

Here we call React's useState function and pass an empty object as our initial state. UseState will return an array with the first item being the state and the second being a function to change the state. So catResponse will be empty.

To make this slightly better, we can use our hard coded response as the initial state. We can move the response out of the component and rename it initialState than pass it to the useState function. Then, when a user loads the page, they will see a fact while they wait for the api call. The final code looks like this.

import { useState } from 'react';
import './App.css';

const initialCatResponse = {
  data: [
    "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
  ]
}

function App() {
  const [catResponse, setCatResponse] = useState(initialCatResponse)

  return (
    <div className="App">
      <ul>
        {catResponse.data.map((fact) => {
          return (
            <li>{fact}</li>
          )
        })}
      </ul>
    </div>
  );
}

export default App;

Make the Api Call

We can now move on to making the api request. For this, we will use the fetch api. The following is the function we will use for making the GET request to MeowFacts. Read more on the fetch api here if you are not familiar: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch.

async function getMeowFacts() {
  const response = await fetch('https://meowfacts.herokuapp.com/')
  const data = await response.json()
  return data;
}

Update the State

And finally, we can now get the response and update our state. To do this, we will use the useEffect hook which allows for "side effects" in our app. It is commonly used on Component render to load api data. Some examples in w3schools are a nice explanation: https://www.w3schools.com/react/react_useeffect.asp.

We will use useEffect with no dependencies for now to do a one time load and call our getMeowFacts function. We also await the promise using then syntax as React recommends useEffect to be synchronous. Finally, we call the setCatReason method from useState to update our response. The following is the result.

useEffect(() => {
    getMeowFacts().then((meowFactData) => {
      setCatReason(meowFactData)
    })
  }, [])

The final code

The final code is the following which will be a familiar setup in your day to day React development.

import { useState, useEffect } from 'react';
import './App.css';

const initialCatResponse = {
  data: [
    "There are cats who have survived falls from over 32 stories (320 meters) onto concrete."
  ]
}

async function getMeowFacts() {
  const response = await fetch('https://meowfacts.herokuapp.com/')
  const data = await response.json()
  return data;
}

function App() {
  const [catResponse, setCatReason] = useState(initialCatResponse)

  useEffect(() => {
    getMeowFacts().then((meowFactData) => {
      setCatReason(meowFactData)
    })
  }, [])


  return (
    <div className="App">
      <h1>Meow Facts</h1>
      <ul>
        {catResponse.data.map((fact) => {
          return (
            <li>{fact}</li>
          )
        })}
      </ul>
    </div>
  );
}

export default App;

The web page looks pretty bare bones, but styling is a separate article.

How to create Auto Increment in Sql Server

Keith Holliday — Thu, 14 Jul 2022 04:17:00 +0000

originally posted here: https://koalatea.io/sql-server-auto-increment/

Intro

Recently I was migrating a postgresql database to sql server. One issue I ran into was migrating the Id columns and keeping the same values as before. SQL server does not let you change a column to identity and does not allow you to edit the identity column. Online resources suggested creating a new column or reinserting each row, however, these were not great options as I was using Azure DB migration tool for a large db and was not guaranteed to have sequential ids since the old db had deletes. My solution was to implement my own auto increment for the Id column.

Setup

The example is somewhat complicated, but let me set up a small example.

We start with an employee table.

CREATE TABLE employees (
    id int not null,
  first_name VARCHAR (50) NOT NULL,
  last_name VARCHAR (50) NOT NULL,
);

Notice that we use int for id as we can not map the old data to identity when using Azure migration tool.

Now, we can insert some existing data from the old database.

insert into employees (id, first_name, last_name) 
    values 
    (1, 'keith', 'holliday'),
    (2, 'jon', 'doe'),
    (4, 'jon', 'dane');

Notice that we do not have row 3. That is because in the old database, the row was deleted. I can not reset the Ids, because the old database has many relationships that depend on those ids.

Right now, if I insert, the id column will not auto increment. To resolve this, let's create a sequence for the table and set the next value as default for the id column.

CREATE SEQUENCE employees_ids 
    START WITH 5
    INCREMENT BY 1;

We created a sequence that will increment by 1 and starts at 5, which is one more than the current max id in the table. We now need to set the default value for the id column.

alter table employees 
add constraint employees_id
default (next value for employees_ids) for id;

The above creates a constraint that will crab the next value in our sequence on insert. We can test this out by adding a new employee.

insert into employees (first_name, last_name) 
    values 
    ('ash', 'ketchum');

id	first_name	last_name
1	keith	holliday
2	jon	doe
4	jon	dane
5	ash	ketchum

Finally, we can now make this our primary key.

ALTER TABLE employees 
   ADD CONSTRAINT PK_employees_id PRIMARY KEY (id);

Backing Up and Restoring Redis

Keith Holliday — Fri, 17 Sep 2021 05:09:28 +0000

Originally posted here: https://koalatea.io/redis-backup-restore/

Intro

Redis uses two methods for persistence, snapshotting and append only file. Both have different use cases and can be used separately or in conjunction. In this article, we will learn how to backup data and work with Redis persistence.

The Persistence Restore Flow

Redis makes it fairly simple to back up your data. The general method is as follows:

Turn on snashotting or AOF
Redis will write the data to a file (or two files)
Backup this file (maybe a cron job to copy this file to S3)
If your redis server fails, grab a back up, place in the correct directory, start redis
Redis will read the file and restore what is backed up

I mentioned the above because I am used to backing up databases such as psql where you usually import the database. However, redis just reads the file in correct directory.

Setting up Redis

Let's begin by setting up redis. We will use docker compose for this, so be sure to install docker for desktop.

Here is the file we will use:



version: "3.2"
services:
  redis:
    image: "redis:alpine"
    command: redis-server /usr/local/etc/redis/redis.conf --requirepass sOmE_sEcUrE_pAsS
    ports:
      - "6379:6379"
    volumes:
      - ./redis-data:/data
      - ./redis.conf:/usr/local/etc/redis/redis.conf
    environment:
      - REDIS_REPLICATION_MODE=master

Notice that we set up two volumes.



volumes:
    - ./redis-data:/data
    - ./redis.conf:/usr/local/etc/redis/redis.conf

The first line will connect to where redis puts its data directory. This is documented on the official redis image.

The second line will allow us to connect our config to the docker file. So, be sure to download a config from the website above and put the config in your current directory.

We also use the following command to tell redis to use our config. Again, this is documented in the official image.



command: redis-server /usr/local/etc/redis/redis.conf

Now we can start our redis instance by running



docker-compose up

Snapshotting

Let's begin by looking at snapshotting in Redis. We have a few configuration options.



save 60 1000
stop-writes-on-bgsave-error no
rdbcompression yes
dbfilename dump.rdb

You can find each of these and more in the configuration file. Configs can be found here: https://redis.io/topics/config. Here is the direct link to Redis 6 config: https://redis.io/topics/config. If you search for "SNAPSHOTTING" you will see the section below.

One of the main configs we will use is save 60 1000. Now, the redis config file does a great job of documenting the config, so we won't cover them all. But, let's run through an example of using the save. The above directive tells redis to save every 60 seconds if 1000 keys have changed. So, you can see that we may never have a snapshot if 1000 changes are not saved. This is important to remember.

Let's add the following save command. This will instruct redis to save every 10 seconds if 1 key has changed.



save 10 1

With this, let's restart the docker container.



docker-compose down
docker-compose up

Now, let's login to our docker instance and change a key. Click the CLI in docker desktop and you should see the cli screen below.

Next, type redis-cli to connect to your redis.

Then, let's see set a key value. This can be creating or updating a key.



set new-key "value-for-my-key"

Now, we wait about 10 seconds, and we should see the following logs.

This is redis saving our snapshot to the disk. You should also see the following dump.rdb example in the redis-data folder.

Append Only

Now that we know about snapshotting, we can move on to Append Only. Append only is defined by its name, and tells redis to append logs to a file. This can become a large file, but is a nice event log. This is a great pattern for you event sourcing nerds.

The following configs are available to us.



appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

We can find this in our redis config similarly to snapshotting. We can search for "APPEND".

To enable this mode, change the following line:



appendonly no



appendonly yes

Then restart your docker service. Then, open up your cli in docker like we did before. Next, add a new key, and feel free to run other coammnds as well.



set funtimes "all day"

You should now see a new file in your redis-data directory called appendonly.aof. This will contain some information regarding key changes for redis to use.

Note that it doesn't have anything before we turned on the setting. This is one of a few reasons why you might want to use snapshots and append only together.

Run manual backup

Let's take a slight detour to see how we can manually backup our data. If we want to manually trigger backups, say from a cron script, we have to commands we can use.

SAVE
BGSAVE

Ideally, you should only use BGSAVE so that redis runs the save in the background. To test this out, open up your redis cli and run the BGSAVE command. You should see save logs in your docker terminal.

Delete and Restore

We are now becoming familiar with backups. Next, we will move on to replicating a redis fail over.

Our first step to backup our data. For this, I will just copy my two db files over to my desktop, but feel free to do what you want.

Next, let's stop the docker server and delete the volume.

Restart the server, and check that our keys are indeed missing.



docker-compose up

scan 0

Next, drop in our db files and restart the server.

And now we see that our keys have been restored!

Getting Started with Redis in Nodejs

Keith Holliday — Tue, 24 Aug 2021 00:16:14 +0000

This was originally posted here: https://koalatea.io/node-js-redis-getting-started/

Intro

When building large scale applications, there comes a need for scaling. There are many places to start with scaling, but one place my be scaling your reads. Let's say that you have a read heavy application, like an ecommerce store or a comment system.
You may want to consider caching to address this concerns. Redis is a good place to start (and to end) when solving this problems.
In this article we will get started with Redis in Nodejs.

More about Redis and Caching

Caching is a large topic that is outlined here: https://github.com/donnemartin/system-design-primer#cache. We will simply introduce Redis here and in later articles, we will learn to implement this practices on large scale features.

Redis is use for a lot more than caching. For example, queues are implemented in Redis using bullqueue: https://optimalbits.github.io/bull/. I highly recommend checking out this package (and the bullq UI). This is a great start to scaling out services, not just microservices. You can
read about more use cases for redis here: https://redis.com/blog/5-industry-use-cases-for-redis-developers/, and I will write articles in the future about implementing this features later on.

Creating the project

Let's create the project as follows.

mkdir redis-example
cd redis-example
npm init -y
touch index.js
touch docker-compose.yml

Setting up Redis

For setting up Redis, I would recommend using a service for you in prod. Azure for example, has a great redis service that scales easily. However, you will want to learn redis and eventually how to scale it yourself. This will help with debugging cloud services or eventually, saving money and not using them.

We will start our intro to redis via using docker compose. Create a docker-compose.yml file and add the following.

version: "3.2"
services:
  redis:
    image: "redis:alpine"
    command: redis-server
    ports:
      - "6379:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
      - REDIS_REPLICATION_MODE=master

Ensure you have docker installed and run

docker-compose up

Installing Redis Modules

There are two modules I see often used in nodejs. I will tend towards
ioredis as it has built in support for promises and many other features in redis.

npm install ioredis

Writing the Code

Let's start by opening up the index.js file and importing our module. We will also connect to the redis server. By default, the module will assume we are using localhost on port 6379, which is what we setup in our docker compose file.

const Redis = require("ioredis")

const redis = new Redis()

Next, let's run some redis commands. We will start very basic with the set and get commands. As implied by the names, the set command will set a key and the get will retrieve the key.

async function main() {
    const setResult = await redis.set("foo", "bar")
    console.log(setResult)

    const getResult = await redis.get("foo")
    console.log(getResult)
}

Note, I usually create a main funciton to start a node file that will be an entry. We can call the above using a self invoking funciton. Eventually in later version of node we will not need this as we will be able to call await at the root level.

(async () => {
    await main()
})()

Here is the full file for context.

const Redis = require("ioredis")

const redis = new Redis()

async function main() {
    const setResult = await redis.set("foo", "bar")
    console.log(setResult)

    const getResult = await redis.get("foo")
    console.log(getResult)
}

(async () => {
    await main()
})()

A Redis GUI

Often over looked in the community is the use of a UI. Many are outdated or hard to run. One that I often use is patrikx3/redis-ui. Although a little
clunky, it usually does what I need. I will also suggest getting use to the redis-cli to help where GUIs cannot.

You can download the GUI here: https://github.com/patrikx3/redis-ui.

Once you have that downloaded, open up the app. Then go to Settings -> New Connection.

Type in the following to connect to local, then hit the "Add" button at the bottom.

Finally, click the bottom right, then select your localhost to connect.

Click Home and then you should see a screen like below, but with no keys on the left.

How to Add Commento (or other Comments) to Gatsby

Keith Holliday — Thu, 29 Apr 2021 17:13:01 +0000

Recently, I wanted to add comments to by Gatsby site. I found this open source sysetem (Commento), although I did sign up for the paid version to support open source :D.

Original post here: https://koalatea.io/gatsby-add-comments/

Anyway, I wanted to add this to my site. After signing up, I was given the follow code snippet.

<script defer src="https://cdn.commento.io/js/commento.js"></script>
<div id="commento"></div>

The system is supposed to automatically recognize the page and only pull comments for that. So, I just needed to add this to a page and it should work, however, that did not work for me.

This could be because of the caching I used on Gatsby, but I believe this is because of how React Handles it. To solve this, I created a Comment Component to manually inject the script.

import React, { useEffect } from 'react';

const Comments = () => {
    useEffect(() => {
        const script = document.createElement('script');
        script.src = 'https://cdn.commento.io/js/commento.js';
        script.async = true;

        const comments = document.getElementById('comments-container');
        if (comments) comments.appendChild(script);
    }, []);

    return (
        <>
            <div id="comments-container"></div>
            <div id="commento"></div>
        </>
    );
};

export default Comments;

I have a file called post.jsx which is used as a template to render post. In that file I add the following, and comments are added to each page.

<Comments />

Real Life Data Science: Exploring Clubhouse Data

Keith Holliday — Wed, 21 Apr 2021 16:34:56 +0000

Original Post here: https://koalatea.io/real-life-data-science-clubhouse/

Hello! And welcome to an article on real life data analysis. Recently, I found a clubhouse user data set on Kaggle. I though using this data would be a good opportunity to go over how review growth for your company. The data set is small, but we can learn some basic from it. Also, if the data is accurate, it is fun to see how a trendy company is doing in real life. Let's get started.

The data set can be fround here. https://www.kaggle.com/johntukey/clubhouse-dataset

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
import sqlite3

Load the Data

Our first step is to load the data. The data was given a a sqllite db, so we first connect then run a select query to load all users into a dataframe.

Sql = sqlite3.connect('./Clubhouse_Dataset_v1.db/Clubhouse_Dataset_v1.db')
df = pd.read_sql_query("SELECT * FROM user", Sql)

View at the Data

Next, we will take a look at the data to get an idea. We start by looking at the first few rows.

df.head()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	user_id	name	photo_url	username	twitter	instagram	num_followers	num_following	time_created	invited_by_user_profile
0	4	Rohan Seth	https://clubhouseprod.s3.amazonaws.com:443/4_b...	rohan	rohanseth	null	4187268	599	2020-03-17T07:51:28.085566+00:00	null
1	5	Paul Davison	https://clubhouseprod.s3.amazonaws.com:443/5_e...	paul	pdavison	null	3718334	1861	2020-03-17T14:36:19.468976+00:00	null
2	8	Johnny Appleseed	None	apple1	null	srt_tester_9	20	81	2020-03-19T19:47:00.323603+00:00	null
3	10	DK 🖍	https://clubhouseprod.s3.amazonaws.com:443/10_...	dk	dksf	null	49538	173	2020-03-19T23:38:52.574777+00:00	null
4	12	Jonathan Gheller	https://clubhouseprod.s3.amazonaws.com:443/12_...	jonathan	jgheller	null	21250	81	2020-03-20T02:30:22.188084+00:00	null

We look at the shape of the data and see that we have 1,300,515 rows. That's quite a lot of users. There are also 10 columns, so not a ton of detail about each user.

df.shape

(1300515, 10)

df.describe()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	user_id	num_followers	num_following
count	1.300515e+06	1.300515e+06	1.300515e+06
mean	7.303800e+05	2.603839e+02	1.154931e+02
std	4.276844e+05	1.674231e+04	1.945045e+02
min	4.000000e+00	0.000000e+00	0.000000e+00
25%	3.592120e+05	1.400000e+01	2.600000e+01
50%	7.258520e+05	3.700000e+01	6.100000e+01
75%	1.099280e+06	9.900000e+01	1.190000e+02
max	1.499999e+06	4.187268e+06	2.504000e+03

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1300515 entries, 0 to 1300514
Data columns (total 10 columns):
 #   Column                   Non-Null Count    Dtype 
---  ------                   --------------    ----- 
 0   user_id                  1300515 non-null  int64 
 1   name                     1300515 non-null  object
 2   photo_url                1236637 non-null  object
 3   username                 1300515 non-null  object
 4   twitter                  1300515 non-null  object
 5   instagram                1300515 non-null  object
 6   num_followers            1300515 non-null  int64 
 7   num_following            1300515 non-null  int64 
 8   time_created             1300515 non-null  object
 9   invited_by_user_profile  1300515 non-null  object
dtypes: int64(3), object(7)
memory usage: 99.2+ MB

View Data Problems

Our next step is to look at any data issues and convert. We also will create a few extra columns that will help use later on.

The first thing I noticed is that the "invited_by_user_profile" is tehnically a categirical variable, albeit a large one. You would want to see a unique count of the options here. The "time_created" is a datetime variable, but was recognized, so we will convert that too.

df['invited_by_user_profile'] = pd.Categorical(df['invited_by_user_profile'])
df['time_created'] = pd.to_datetime(df['time_created'])

From looking at the data above, I thought it would be good to create a few more categorical variables. It would be nice to see the distribution of users who are in twitter vs instragram. Also, we would like to see how has a photo or not and who has been invited or not.

We start with a social type column. I use the compare to "null" because the null value seemed to not be read equally. Basically, we are just saying, if you have dona twitter, we mark you as a instagram user otherwise a twitter.

Next, we do similar logic for if the user has a photo or has been invited.

df['social_type'] = np.where(df['twitter'] == "null", 'instagram', 'twitter')
df['has_photo'] = np.where(pd.isnull(df['photo_url']), 0, 1)
df['invited'] = np.where(df['invited_by_user_profile'] == "null", 0, 1)

df.head()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	user_id	name	photo_url	username	twitter	instagram	num_followers	num_following	time_created	invited_by_user_profile	social_type	has_photo
0	4	Rohan Seth	https://clubhouseprod.s3.amazonaws.com:443/4_b...	rohan	rohanseth	null	4187268	599	2020-03-17 07:51:28.085566+00:00	null	instagram	1
1	5	Paul Davison	https://clubhouseprod.s3.amazonaws.com:443/5_e...	paul	pdavison	null	3718334	1861	2020-03-17 14:36:19.468976+00:00	null	instagram	1
2	8	Johnny Appleseed	None	apple1	null	srt_tester_9	20	81	2020-03-19 19:47:00.323603+00:00	null	twitter	0
3	10	DK 🖍	https://clubhouseprod.s3.amazonaws.com:443/10_...	dk	dksf	null	49538	173	2020-03-19 23:38:52.574777+00:00	null	instagram	1
4	12	Jonathan Gheller	https://clubhouseprod.s3.amazonaws.com:443/12_...	jonathan	jgheller	null	21250	81	2020-03-20 02:30:22.188084+00:00	null	instagram	1

Univariate Analysis

In this section, we look at the distribution of our features or columns. In general, you can just loop through each column and plot their distribtuions. People tend to get fancy, but essentially we just want a boxplot for each numeric column and a bar plot for each cateogirical column.

There are definitely more plots to do depending on what model you will do in the future, but for generic reporting, these will do.

We first attempt to review the "invited_by_user_profile" column. I would like to null the top users who invite the most. The data was quite large, so we just create a table of the top 20. The top user is "null" which means there was no invite logged. Then we can see that user 12492 takes the lead.

# Found the trick here: https://stackoverflow.com/questions/47136436/python-pandas-convert-value-counts-output-to-dataframe
data = df['invited_by_user_profile'].value_counts()[:20].rename_axis('inviting_user').reset_index(name='count')
invitedby = pd.DataFrame(data)
invitedby.columns

invitedby.head(100)

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	inviting_user	count
0	null	3214
1	12492	593
2	402821	356
3	3272	306
4	881	301
5	50152	240
6	3547	203
7	104	200
8	14605	199
9	7557	168
10	27950	161
11	2710	158
12	846	152
13	5864	138
14	8406	137
15	42125	135
16	1940	133
17	5954	123
18	144372	118
19	1720	116

We now look at the distribution of our users based on social platform. We can see that twitter is the most popular by almost double.

fig = sns.catplot(x="social_type", kind="count", data=df, hue=None)
fig.set_xticklabels(rotation=90)
plt.show()

Photo usage is also quite high, which is to be expected. Most users have added a photo which is common on a social media platform.

fig = sns.catplot(x="has_photo", kind="count", data=df, hue=None)
fig.set_xticklabels(rotation=90)
plt.show()

Here, we can clearly see that the majority of users were invited. Clubhouse is a primarily invite only, which is a nice referal growth tactice. From the plot, you can't really see how many users weren't invited.

Remember from our table before that the "null" invite was the number of users not invited. We use the value_counts method to display those results again.

fig = sns.catplot(x="invited", kind="count", data=df, hue=None)
fig.set_xticklabels(rotation=90)
plt.show()

df['invited'].value_counts()

1    1297301
0       3214
Name: invited, dtype: int64

Now, we turn our attention to the numerical features. We loop through each of these and plot boxplots of each.

columns = ['num_followers', 'num_following']

figure = plt.figure(figsize=(20, 10))
figure.add_subplot(1, len(columns), 1)

for index, col in enumerate(columns):
    if index > 0:
        figure.add_subplot(1, len(columns), index + 1)
    sns.boxplot(y=col, data=df, boxprops={'facecolor': 'None'})

figure.tight_layout()
plt.show()

The plots above are hard to gather any information from. The box plots show a large amount of outliers. This might suggest that there is a small minority of users using the two features we describe.

From the left, that would be that there is a minority of users with the majority of followers.

On right right, that would suggest that a minority of users following a lot of users.

We would probably need to look at more data to get an idea of what's going on, but the left plot does help know where our distribtion of "popular" users are.

Bivariate or (Multi) Analysis

In this section, we would like to review two or more vairables at a time. This would give us an idea of the relationship between variables.

However, we don't have too much data. There is really only one question I have left which is the count of new users over time. This would tell us what the grow of Clubhouse is like.

df_copy = df.copy()
df_time = df_copy.set_index('time_created')
df_agg_day = df_time.resample('D').sum()
sns.lineplot(data=df_agg_day, x="time_created", y="user_id")

<AxesSubplot:xlabel='time_created', ylabel='user_id'>

From the plot above, we can see that the over all user groth surged in Jan 2021, but dipped since. There also seems to be a downward trend. This is common with a viral spike. Let's look at the recent dates so that we can determine if Clubhouse has stabilized.

df_reset = df_agg_day.reset_index()
recent = df_reset[df_reset['time_created'] > '02-01-2021']
sns.lineplot(data=recent, x="time_created", y="user_id")

<AxesSubplot:xlabel='time_created', ylabel='user_id'>

From Feb to March, the number of new users seems to be low. So either our data is missing information or Clubhouse seemst to be decreasing growth.

Note, that they could still have a stable usage data, this data is only on created accounts. Also, they data could be wrong or inauthentic.

Conclusion

I hope you have enjoyed this brief exploritory data analysis of Clubhouse data. I hope if gives you some insights into how to view your own users data in the future. Please leave comments and let me know if you have any questions. I'm happy to help you answer questions on other data sets as well.

Python Crash Course (Video)

Keith Holliday — Tue, 27 Oct 2020 21:27:15 +0000

Intro

Welcome to one of our Crash Course series! In this post, we will cover Python.
This course is meant for coders who have already learned a programming language,
but feel free to try if you are up for the challenge.

The idea here is there are some items common to almost every language:

Variables
Basic Data Type - string, number, list, dictionary
Operators
Control Flow - if/else
Loops
Functions

You could also argue classes, but there are many functional-first programming
languages, so we will keep with this list. After this list, you start to get into
more specific language features, like list comprehensions in Python, as well
as more intermediate and advanced features like the standard library and concurrency.

We will briefly cover each of the above topics before using Python. Let's get started!

Hello World

Let's start with the simple hello world script. We will introduce the print function
which is common to many languages.

print("Hello World")

Variables

In python, we can declare variables using the = assignment operator. You do not
need to specify a type.

name = "Keith"
print(name)

Data Types

Okay, not to get so more details going. Let's briefly look at each of the basic
data type in python. You should have seen many of these in your previous language.

## Strings
test = "Hello, Everoyne!"
print(test)

## Numbers
age = 18
print(age)

amount = 12.9
print(amount)

Lists/Arrays

Arrays are called lists in python. You can also have multiple types in a list.

## Creating
todo = ["step1", "step2"]
mixedList = [1, 2, "one", "two"]
print(todo)
print(mixedList)

## Acess/reading
print(todo[2])

## Updating
mixedList[1] = "a new item"
print(mixedList[1])

## Deleting
del todo[1]
print(todo)

## Helpful functions
print(len(todo))

combined = todo + mixedList
print(combined)

repeated = [4] * 12
print(repeated)

Dictionaries

Maps, Hashmaps, etc are called Dictionaries in Python. They work and much the same way as other languages. You can also
use multiple lines without following normal indention rules.

Notice that you need to use quotes on the key and value for Python Dictionaries.

## Creating
profile = {
  'name': 'Keith',
  'Age': 18,
}

## Accessing
print(profile['Name'])

## Updating
profile['Age'] = 21
print(profile)

## Deleting
del prfile['Name']

Operators

The basic operators are the same in Python as most languages. Let's take a review of the most common.

first = 10
second = 2

## Math
print(first + second)
print(first - second)
print(first * second)
print(first / second)

## Comparisons

# equal to
print(first == second)

# no equal
print(first != second)

# greater than
print(first > second)

# greater than or equal
print(first >= second)

# less than
print(first < second)

# less than or equal
print(first <= second)

Control Flow - if/else

For Python, there are no curly brackets. Python relies on
indentation for blocks and nesting. Let's see how this works for conditional flows (if, else if, and else).

age = 20

if age > 60:
  print("You are to old!")
elif age < 5:
  print("You are to young!")
else:
  print("Just right")

Notice that we don't need parenthesis either. We the format is if [condition]:.

Loops

Python has both a for and while loop. One special keuword in Python is the in keyword, which let's you use generators. This is very helpful when iteratoring of lists, but has many other advantages.

count = 0
while count < 10:
  print(count)
  count += 1

todoList = ["one", "two", "three"]
for item in todoList:
  print(item)

Functions

To create functions in Python, we use the def keyword followed by the name of the function, parenthesis and the arguments needed.

def getFullName(firstName, lastName):
  return firstName + " " + lastName

full = getFullName("Keith", "Holliday")
print(full)

Build an Election Texting bot with Twilio, Node, and Cron

Keith Holliday — Thu, 15 Oct 2020 00:00:00 +0000

Intro

To celebrate the upcoming election, we decided to learn Twilio and make a passive aggressive texting bot to remind ourselves and our friends to go vote. In this video, we walk you through how to create some text messages and automatically text them via the Twilio api. Help spread the word about voting by using the power of code. See http://vote.org to check your voter registration.

Video

Resources

Because Hacktoberfest is coming up, my brother and I celebrated Preptember by taking a journey on GitHub.

Keith Holliday — Thu, 24 Sep 2020 21:43:37 +0000

We made a contribution and want others to see how it works. You can view the whole journey on Youtube here: https://www.youtube.com/playlist?list=PLoD791kESPn8gLoK3mht3Uo4KXrAP5F7Q, or read on below to see each video. (Click the images to watch)

Finding a Project

Our journey began. The first step in our process was to find a good project to work on. We started by looking at Github trending repositories. We were looking for something simple, like a Javascript widget. It is always good to find something small to start, as big projects have many steps and require domain knowledge.

Using the Project

We found a project called React Select. This project was a simple select box, with tons of features. Not only did the project have many features, but the project was well documented and had a lot of CI/CD set up. This was a great project to start on as you can learn much of the open-source best practices.

Compiling the Project Locally

Next, our mission was to compile the project locally, make a change, and see that change we made. It was a small struggle to get started, as the linking setup wasn't clear. However, we decided to just start hacking away. This is a common strategy in programming. Sometimes, you just have to code and see what happens.

We achieved success and make a code change to the project. Now, we were on our way to contributing.

Finding an Issue & Reviewing Someone's Work

Afterward, we decided to explore the issues. There must be some bug that was easy enough for us to tackle. However, we decided to start browsing PRs to get some hints on how to make a modification, which led us to review someone's code and learn more about the project. This is a highly recommended strategy. Reviewing code and retracing the steps someone made to contribute will give you a head start on your contribution.

Making our Contribution

The final stage was set. Our journey was coming to an end. We read lots of code, struggled, and now we wanted to make our contribution. We decided to keep it simple by adding an example to the documentation.
We booted up the project, looked at some examples, and hacked away. After a short while, we made our first commit and proposed our PR. We were now contributors.

Original article here: https://koalatea.io/a-github-contribution-journey