DEV Community: Hannah (she/her)

Measuring Web Requests with Httping

Hannah (she/her) — Wed, 27 May 2020 02:07:24 +0000

I recently encountered a bug where an http request was taking up to 3 seconds. Once I developed a potential solution, I needed to test the request to see if it was faster with the new code changes. In order to test, I developed a baseline by curling the request without any code changes. Next, I updated the code with the hopeful solution and curled the request again to observe the change.

Below is an example. Note, the -v (short for -verbose) shows extra information, like the runtime, which shows the length of the request in seconds.

curl -v http://api.citybik.es/v2/networks

I found myself curling repeatedly to check the length of the request. I even recorded the times and averaged them. This was cumbersome. Fortunately, I discovered a useful program called httping that allows you to test the latency of a request by pinging it on a loop. When you exit the loop, an average runtime is provided for you.

httping http://api.citybik.es/v2/networks

It's worth checking out the man pages because there are a number other useful httping features. For example, the demo below enables color highlighting and highlights all requests above the provided threshold.

httping http://api.citybik.es/v2/networks -Y --threshold-red 325

Making Plans with Plantuml

Hannah (she/her) — Thu, 21 May 2020 01:53:10 +0000

When it comes to building software, diagrams serve as a useful planning tool. One drawback with diagrams is the time it takes to create them. I like using Plantuml to create diagrams because it allows me to generate diagrams through text, saving me time.

To get started, you just need to have Java and Graphviz installed.
Download the plantuml.jar file and keep it handy, you'll need it later.

Then, create a text file and use the Plantuml language to specify the information you would like to include in your diagram.

The contents of ranch-manager.txt is below:

@startuml

title Ranch Manager DB

' hide the spot
hide circle

' avoid problems with angled crows feet
skinparam linetype ortho

entity User {
  id : number
  --
  name : text
}

entity Plant {
  id : number
  --
  name : text
}

entity Animal {
  id : number
  --
  name : text
}

note left of User #pink
 <size:15><&heart>This is evolving.<&heart></size>
end note

User }|..|{ Plant
User }|..|{ Animal

@enduml

Next, run the Plantuml program that converts the text into a diagram and specify the text file you already created.

java -jar plantuml.jar ranch-manager.txt

And voila! The ranch-manager.png below is generated for you. No dragging, dropping, or color picking required.

Once you learn the language specifications, it's easy to generate a variety of diagrams through text.

Now your brain can put the extra time toward creative thinking and generating your next big idea.😻

SQL GROUP BY using JavaScript

Hannah (she/her) — Wed, 26 Jun 2019 05:07:40 +0000

This post explores how PostgreSQL's GROUP BY clause and JavaScript's reduce method both have the ability to help you group objects/rows of data based on properties. I'm assuming you are already familiar with both GROUP BY and reduce, and that you are also (mildly) interested in gaining a deeper understanding of each.

PostgreSQL's GROUP BY clause

According the PostgreSQL documentation, "the GROUP BY clause is used to group together those rows in a table that have the same values in all the columns listed...The effect is to combine each set of rows having common values into one group row that represents all rows in the group."

Here is a refresher on what that this looks like in PostreSQL (Source: PostgreSQL tutorial):



SELECT column_1, aggregate_function(column_2)
FROM tbl_name
GROUP BY column_1;

Let's say we have a cities table that includes a list of cities with the properties of name and state, and we want to aggregate those cities by state.

A list of United States cities and their states.

Name	State
San Antonio	TX
Atlanta	GA
Austin	TX
New York City	NY
Houston	TX

The following query will group together those rows in the table that have the same value in the state property.



SELECT state
FROM cities
GROUP BY state;

Below is a visual representation of the result of the query. You can see that GROUP BY in this case acts as SELECT DISTINCT. In other words, GROUP BY removed all the duplicate states in our table.

A list of all unique states in the cities table.

State
GA
NY
TX

Getting insights from the data

Aggregate functions operate on the groups created by GROUP BY to help you answer interesting questions about your data. Below lists some of the aggregate functions: (Source: PostgreSQL Documentation).

AVG() – return the average value.
COUNT() – return the number of values.
MAX() – return the maximum value.
MIN() – return the minimum value.
SUM() – return the sum of all or distinct values.

Let's answer some questions about our data using GROUP BY and aggregate functions!

How many cities are in each state?



SELECT state, COUNT(state)
FROM cities
GROUP BY state;

Each state along with the number of cities in that state.

State	Count
GA	1
NY	1
TX	3

Which state has the most cities?



SELECT state, COUNT(state)
FROM cities
GROUP BY state
ORDER BY count DESC
LIMIT 1

The state with the most cities.

State	Count
TX	3

JavaScript's reduce method

What if you're working in JavaScript and have an array of objects you need to group by a particular property? Well, let's extend the example above by assuming we have an array of staff location objects, and each object has the property of name and city and state. JavaScript's reduce method is one way to approach the problem.

According to the the MDN documentation, "the reduce() method executes a reducer function (that you provide) on each element of the array, resulting in a single output value.

You can write a function that takes 2 parameters: the array of objects and the properties that you would like to group the object by. The properties will represent the "bucket" that you put your staffLocations in based on their state.

The reduce method below takes the following arguments:

accumulator - This stores the return values created each time the callback function is invoked. This is returned when the method is complete (assuming the array passed in is not empty, in which case the initial value is returned).
object - This is the current object being manipulated in the array.
callback - This is the function you want to execute on each object in the array.
initialValue - The first time the reducer function runs, this will be the accumulator value. Below, the initialValue is {}.



const staffLocations = [
  { name: "Hannah", city: 'Houston', state: 'GA' },
  { name: "Ilhan",  city: 'Atlanta', state: 'GA' },
  { name: "Preet",  city: 'Houston', state: 'TX' },
  { name: "Adam",  city: 'Austin', state: 'TX' },
  { name: "Preston", city: 'New York City', state: 'NY' },
  { name: "Anna", city: 'Houston', state: 'TX' },
  { name: "Jakub",  city: 'Atlanta', state: 'GA' },
];

const groupBy = (objectArray, ...properties) => {
  return [...Object.values(objectArray.reduce((accumulator, object) => {
    const key = JSON.stringify(properties.map((x) => object[x] || null));

    if (!accumulator[key]) {
      accumulator[key] = [];
    }
    accumulator[key].push(object);
    return accumulator;
  }, {}))];
}

const groupedStaffLocations = groupBy(staffLocations, 'state');

groupedStaffLocations

groupedStaffLocations looks like:



[
  [
    { name: "Preet", city: "Houston", state: "TX" },
    { name: "Adam", city: "Austin", state: "TX" },
    { name: "Anna", city: "Houston", state: "TX" },
  ],
  [
    { name: "Hannah", city: "Houston", state: "GA" },
    { name: "Ilhan", city: "Atlanta", state: "GA" },
    { name: "Jakub", city: "Atlanta", state: "GA" },
  ],
  [
    { name: "Preston", city: "New York City", state: "NY" },
  ]
]

The callback steps include the following:

Read the values of the grouping properties, and store them in key. This symbolizes the group
If the accumulator doesn't have an existing group for the values in key, create a new group
Put the object in the group

Source: MDN: Reduce: Grouping objects by property

Getting insights from the data

After you reduce data to buckets of information with key value stores, you can map the same data to answer interesting questions, like the question we answered above: "Which state has the most cities?".



const groupedCities = groupBy(cities, 'state');

// sort by length of array
let sortedArr = groupedCities.sort((a, b) => b.length - a.length);
// get the state of the first array, which would have the greatest length
sortedArr[0][0]['state'];

// returns:
// "TX"

Multiple properties

This function also supports grouping by multiple properties, so it works like GROUP BY in SQL:



const cityGroupedStaffLocations = groupBy(staffLocations, 'state', 'city');

In this case, cityGroupedStaffLocations returns groups representing staff that live in the same city:



[
  [
    { name: 'Hannah', city: 'Houston', state: 'GA' },
  ],
  [
    { name: 'Ilhan', city: 'Atlanta', state: 'GA' },
    { name: 'Jakub', city: 'Atlanta', state: 'GA' },
  ],
  [
    { name: 'Preet', city: 'Houston', state: 'TX' },
    { name: 'Anna', city: 'Houston', state: 'TX' },
  ],
  [
    { name: 'Adam', city: 'Austin', state: 'TX' },
  ],
  [
    { name: 'Preston', city: 'New York City', state: 'NY' },
  ]
]

This can be easily paired up with map to get the number of staff in each city:



cityGroupedStaffLocations.map(cityStaff => ({location: `${cityStaff[0].city}, ${cityStaff[0].state}`, numberOfStaff: cityStaff.length}))

returning:



[
  { location: 'Houston, GA', numberOfStaff: 1 },
  { location: 'Atlanta, GA', numberOfStaff: 2 },
  { location: 'Houston, TX', numberOfStaff: 2 },
  { location: 'Austin, TX', numberOfStaff: 1 },
  { location: 'New York City, NY', numberOfStaff: 1 },
]

JSON.stringify???



const key = JSON.stringify(properties.flatMap((x) => object[x] || null));

When reading through the groupBy method, did you notice that the key was JSON? In order to ensure that multiple grouping properties can be passed into the function (state, name, or city), key had to be an array of the corresponding values. In JavaScript, Objects can only use strings and symbols as keys. Transforming the group (key) to JSON allows us to cheat JavaScript's lack of deep structural equality by using simple JSON string comparison. When the values in the group convert to the same JSON, they'll be considered part of the same group. While this probably hurts performance, it's the most succinct way I've found with vanilla JavaScript to use arrays as keys.

Interviewing TypeScript

Hannah (she/her) — Wed, 19 Jun 2019 23:33:36 +0000

Sometimes JavaScript tries to guess what you mean.

For example:

true + []

// returns "true"

You could say JavaScript is trying to be helpful by not asking any questions and just getting results, but sometimes those results come with a cost. Sometimes you may not find a bug that JavaScript's guesswork produced until your code is in production. TypeScript helps solve this problem by providing error messages in a more timely fashion

For example:

true + []

//error TS2365: Operator '+' cannot be applied to types 'true' and 'never[]'.

A scenario

Imagine that one day you find yourself complaining that you are frequently getting type errors in your JavaScript code once it's already in production. It's causing such a headache - can't we hire someone to take care of these JavaScript issues? Can't we get someone who can work hand-in-hand with our JavaScript code and catch type errors before runtime?

The interview process

You put a job post up on Hacker News and within 24 hours TypeScript's resume comes into your inbox. They have a decent cover letter, so you call them for a phone screen. This candidate stood out to you because they said they will deliver error messages in your text editor as you write your programs. You invite TypeScript in for an onsite.

During the onsite, TypeScript does a whole presentation about how your code will get type checked before it runs. But, you still have a lot of questions about how exactly everything works.

You say, "Hey, let's walk through this. I know that this is how my code gets compiled now..."

You grab a marker and write on the board:

High level overview of how JavaScript code runs:

JavaScript source code is parsed by a program called the compiler, which converts the code into an AST (abstract syntax tree), a data structure that ignores whitespace, comments, and spaces.
AST is converted to bytecode.
Bytecode gets fed into another program called runtime, and this is when code gets evaluated.

And TypeScript nods in agreement and says, "Yes, that how JavaScript code runs. When I get involved, there are some additional steps that allow the code to be type checked by the compiler".

TypeScript source code is compiled into a TypeScript AST
TypeScript AST is type checked
TypeScript AST is converted to JavaScript
JavaScript is converted to JavaScript AST
JavaScript AST is converted to bytecode.
Bytecode is evaluated at runtime.

TypeScript adds, "Please note, I improve your ability to catch type errors before runtime but will not change the results of running your code. Syntactically, all valid JavaScript is valid TypeScript. In other words, by using TypeScript, you aren't actually changing how your JavaScript code works."

TypeScript sometimes refers to themselves in third person. :D

Technical questions

Things are going great and everything seems to be moving along nicely. You decide to dig deeper and ask some tougher questions.

You ask, "Are you dynamically typed or statically typed?

To which TypeScript replies, "I'm not entirely one or the other. You can use type annotation to define types or you can simply leave them out and I'll do my best to infer the type. I'm at my best when I know the type of everything in the program at compile time, but I can still compile everything if I don't have all of that information.

You follow up, by asking, "If we migrate to you, do we have move everything to TypeScript at once in order for our code to work?"

TypeScript smirks, then replies, "You don't have to have full type coverage from the beginning, you can bring me in gradually. This ties into what I mentioned before: all JavaScript is valid Typescript."

After a short pause TypeScript says, "Look, I'm going to be upfront and tell you that it needs to be fixed when your code is type checked by the compiler. Some people might say I'm a bit of a complainer, but I just call things how I see 'em."

You are a bit put off by the arrogance, but you thank them for their transparency anyways. Next, you transition back to more "soft skills" by asking about how they communicate errors once they find them.

To which they reply, "Well, if you have a halfway decent text editor, you're going to see some of those red squiggly lines under places where you need to make changes. This goes back to how I type check at compile time, which gives you syntax and type errors at compile instead of at runtime."

Offer extended

TypeScript, you're hired!

Sources: Programming TypeScript, May 2019. Boris Cherny. Chapter 2. TypeScript: A 10_000 Foot View

Making Filth Finder Faster with Google Cloud

Hannah (she/her) — Wed, 22 May 2019 02:54:55 +0000

I've written a bit about Filth Finder, an app that surfaces health inspections from NYC restaurants near you. Most recently, I wrote about the API that served the restaurants to the frontend. While this implementation worked, it was pretty slow to load. In an effort to speed up the loading time, I rewrote the backend to pull the of index restaurants from Google Cloud Storage. Check out how much faster it loads.

If you're interested, here is another post about rewriting Filth Finder with React Hooks.

Why was it slow?

Previously, the app was hosted on Heroku as two different apps, each of which required 5-10 seconds to spin up since they were on the free plan. The frontend served up the React app, and the backend served up the restaurant listing API. Finally, these apps didn't start up at the same time, meaning the backend only would begin spinning up once the frontend had finished. The end result was a user would need to wait nearly 20 seconds to begin using the app if it hadn't been visited in a while.

Moving to the cloud

With the help of the Google Cloud Storage: Node.js Client, I now have a script that grabs all the restaurants from the NYC Open Data API and puts them in a Google Cloud Storage bucket. The city updates the health inspection info daily, so this script needs to run regularly to allow data from newly opened restaurants to be pulled. It takes about 45 minutes to load the restaurants.

Updating the Establishment model

According to Wikipedia, the haversine formula determines the great-circle distance between two points on a sphere given their longitudes and latitudes.

I also updated the Establishment model to fetch data from the newly created Google Cloud Storage bucket instead of the old backend. The haversine library allows the Establishment model to calculate the distance between the user (the latitude and longitude is passed in the through the params) and each of the nearly 27,000 restaurants. The restaurants are then returned in ascending order according to the distance between the user and the restaurants, then limited to 20 total results to keep the frontend zippy. The JSON file is really simple, and looks something like this:

[
   {
      "latitude" : 40.7178921762742,
      "camis" : "41631962",
      "longitude" : -74.0008766031009,
      "dba" : "POPEYES LOUISIANA KITCHEN"
   },
   {
      "latitude" : 40.7560475658859,
      "camis" : "50072138",
      "dba" : "HUI'S GARDEN",
      "longitude" : -73.8335953523179
   },
   {
      "longitude" : -73.8279669765882,
      "dba" : "JIN DAL LAE 8",
      "latitude" : 40.7648883680589,
      "camis" : "50087614"
   },
   // 26k+ more restaurants...
]

Firebase

In order to make the app load even faster, I followed this guide to take advantage of Google Cloud Firebase's free static hosting. This removes the need for Heroku completely, and Firebase static hosting doesn't require any spinning up.

Final notes

Now the app runs faster because the React frontend loads from Firebase immediately, and quickly fetches the restaurant list from Google Cloud Storage.

Google Cloud + Filth Finder = friends forever (or at least until something better comes along!).

Multi-Tenancy in Rails with Apartment

Hannah (she/her) — Sun, 08 Jul 2018 00:33:44 +0000

I'm working on a Rails app for nonprofit management that features multi-tenancy. This means that a single instance of the application will support multiple isolated users. Today, we'll discuss how to configure multi-tenancy using the Apartment library with sessions.

What is Multi-tenancy?

Multi-tenancy is a type of design architecture that allows an application to run multiple clients on one system. So, you can have multiple customers (customer == tenant == nonprofit administrative organization) logging into the same software. Slack, Discord, and Trello are examples of apps that feature multi-tenancy. Here are some primary characteristics of multi-tenant software:

Data isolation for each tenant
Customization for each tenant
Common infrastructure across all customers

Note: Multi-tenancy can be great, but sometimes you don’t need it. For example, if you are building social networking platform where the main idea is sharing data, you don’t need to separate data using multi-tenancy.

Getting Started with Apartment

Apartment helps you add multi-tenancy by making it easy for you to sequester data based on a tenant. To use Apartment, you need to add it to your Gemfile, then run the install task with bundle exec rails generate apartment:install. Now, you're ready to get started using Apartment.

Creating a Tenant

In order to select a tenant, we must first create a tenant. Since tenants map directly to Organizations in Theodora, it makes sense to have the tenant be created when the Organization is created. To do this, I simply used an after_create ActiveRecord callback which uses Apartment to create the tenant.

class Organization < ApplicationRecord
  has_many :organization_users
  has_many :users, through: :organization_users

  after_create :create_tenant

  def tenant_name
    "organization_#{self.id}"
  end

  private

  def create_tenant
    Apartment::Tenant.create(self.tenant_name)
  end
end

Configuring Apartment

When you install this gem, it creates config/initializers/apartment.rb file as the configuration file. There are a lot of knobs you can turn, but for our basic usage, we'll just need to touch the basics.

Setting up Multi-tenant Models

In Theodora, each tenant has its own schema but shares the same database. Some multi-tenancy applications use separate DBs for each account or use scoped access by separating data by a unique id. I used Apartment and Postgres to isolate my data using the schema separation approach. Using schemas for separation provides strong data isolation and allows for running migrations separately for each tenant, which means there is less data to migrate at one time.

The models created are Organization, User, and OrganizationUser, and they are all in the global schema ("public" in Postgres). These models are global because they're all concerned with tenancy (the tenant name is derived from the Organization name), or are concerned with who has access to a tenant. Once I've got multi-tenancy support complete, I'll be able to flesh out the rest of the application, which will be isolated to the selected tenant.

Example: A single grant manager may work at nonprofit known as an umbrella organization, where multiple nonprofits with overlapping missions will operate as one organization but may receive and report on separate grants. For example, one organization called "Center for Children" might have three local programs that run under three different statewide offices, such as Children's Advocacy Center (CAC), CASA (Court Appointed Special Advocate), Domestic Violence Hotline (DVH) all existing as one organization because their missions are all related to serving victims of abuse. The grant manager needs to be able to access all of the organizations seamlessly, without needing duplicate accounts.

You can see below that the models included in the global namespace are included below. You can also see how the configuration for schema separation is determined by tenant_name, which is derived from the Organization name.

Apartment.configure do |config|
  config.excluded_models = %w{Organization User OrganizationUser}
  config.tenant_names = -> { Organization.all.map(&:tenant_name) }
end

  def tenant_name
    "organization_#{self.id}"
  end

Configuring Middleware

When using Apartment, you'll typically use an Elevator. In Apartment, Elevators assist in configuring the selected tenant on
a per-request basis. Elevators are Rack middlewares that run before the HTTP request hits your Rails application. It is very common
to use a subdomain multi-tenancy approach, which Apartment has built-in support for. However, since I'll be hosting this on a free
Heroku server, I didn't want to have to tangle with getting a custom wildcard SSL certificate to run the app securely. So, I needed to
customize how the tenant is selected, and so I used Apartment::Elevators::Generic, which allows you to choose your own strategy.

Since the selected tenant naturally is part of the user's session with the application, I decided to use the request session for tracking
which tenant they are using. Below, I check to ensure the user is logged in with Warden, and that a tenant has been selected. Then, I
look up the tenant (an Organization), scoped to the current user to ensure they have access to select the tenant. Finally, if an authorized
Organization is found, I return the tenant name to use to Apartment. For the rest of the request, any ActiveRecord calls will be isolated to
the tenant returned from this Elevator. Since I'm using Postgres, this means that all queries sent to the database will be limited to the tenant's Postgres schema, which is like a folder for tables.

Rails.application.config.middleware.insert_after Warden::Manager, Apartment::Elevators::Generic, -> (request) { 
  if request.env['warden'].user && request.session[:selected_organization_id]
    org_id = request.session[:selected_organization_id]
    current_user = request.env['warden'].user

    if org_id.present?
      org = current_user.organizations.find_by(id: org_id)
      org.tenant_name if org
    end
  end
}

Setting the Tenant

Once a user logs in, they are prompted to select which Organization they want to manage. This is used by the previously mentioned Elevator,
which sets the tenant (and Postgres schema) on each request. Here, I simply take the ID that was POSTed to the server, and set it in the
session key that the Elevator is expecting.

class TenantSelectorController < ApplicationController
  def index
    @organizations = current_user.organizations
  end

  def create
    org = current_user.organizations.find(params[:id])
    session[:selected_organization_id] = org.id

    redirect_to root_path
  end
end

Theodora: Starting an Open Source Project

Hannah (she/her) — Fri, 29 Jun 2018 18:12:39 +0000

I'm building an open source rails app aimed at empowering nonprofits to efficiently manage government grants. It's called Theodora 👑. Large nonprofits often have a structure where one adminstrative organization distributes grant funds across multiple smaller nonprofits. This structure is (sort of) similar to how fast food 🍔 chain businesses are affiliates of a central corporate office.

This post will share the steps I took toward creating Theodora as an open source project. Here is an overview of each step I will discuss:

Determine the Goal
Determine the Stack
Determine the Architecture
Sketch the Idea
Create a Wireframe
Use Zenhub

Determine the Goal

Theodora will allow an adminstrative office to manage documents for each local nonprofit through a single app. Without this app, documents would have to be sent through email, resulting in multiple attachments. Sending and receiving information through email attachments creates room for human error (forgetting an attachment 😱, receiving attachments over multiple emails, etc.).

Theodora aims to solve this problem by providing one place to upload and submit all documents. In the future, I would like to add the ability to create document templates, so that users fill out forms instead of uploading Word or Excel documents.

Determine the Stack

Ruby on Rails is an opinionated framework that will allow me to set up the application quickly. There is a large community of Rails developers to learn from, as well a sea of libraries to choose from when building out features. For the MVP, there isn't a need to use a JavaScript framework for the frontend. As I build out more interactive features in the future, I plan to use React.

Determine the Architecture

Because each local nonprofit will need to log in to upload, edit, and submit documents, Theodora will be a multi-tenant Software as a Service (SaaS) application. Multi-tenancy means that multiple administrative organizations can employ the same application. This way, developers only need to maintain one code base while offering the same service multiple to organizations. For Theodora, administrative organizations will be the "tenants" and the local nonprofits will be the "users".

Sketch the Idea

To get started, I sketched out the Minimum Viable Product (MVP). My goal was to quickly draw out my ideas, so that I can easily build out a more detailed wireframe.

Create a Wireframe

Next, I created a wireframe using Balsamiq. I chose Balsamiq because I could use the free trial, and it was easy to use.

Note: If you want to export your wireframe to PDF and PNG, Balsamiq creates a low quality document. There are other tools, like Sketch,that allow you to quickly build web and mobile wireframes and also export high quality documents.

I used Balsamiq to formulate this plan:

Prompt user to log in either manually or through Google.

If the user clicks sign up, have user sign up and ensure they choose an organization. Signing up immediately logs in the user.

Once user is logged in, show them a custom organization page where they can upload documents. If all documents are uploaded, user can submit documents.

After user submits documents, show a confirmation message, as well a link to ask the administrator questions.

Use Zenhub

With a plan for the MVP, I could easily begin developing with a clear goal in mind. I created a repository for Theodora in Github and created a ZenHub Board to visualize the workflow and allow others folks to contribute to issues. I've already had one person submit a pull request!

Next Steps

At this point, it's time to start researching and coding. I hope this post will perhaps inspire others to collaborate on the project. Stay tuned for a new post on building out multi-tenancy in Rails, and feel free to reach out if you are interested in learning more.