DEV Community: Erin Fox

Extract, Transform and Load with React & Rails

Erin Fox — Mon, 20 Sep 2021 21:48:18 +0000

You might be thinking, WFT is ETL and have I been using it all this time?!

If you're an engineer, you probably have done some form of ETL. I never realized I was extracting, transforming and loading data throughout my career until researching it recently. I also, need to get better at it and the best way I know how is by researching, practice and writing about it.

I'm still working on learning it with more complicated data structures and data transformations, but I wanted to break it down to the beginning to make sure I understand it each step of the way. But with cats added.

What is ETL?

ETL = Extract, Transform, Load

ETL is a series of steps to move data from one location to another. When doing this, it transforms the data structure before it is loaded from its source to its new destination. In more words, it is a process you can use to help plan and execute the movement of data that you need.

Why use ETL?

I'm sure there are several answers to this question. For me, using it breaks down the steps of gathering and retrieving data. It also forces you to understand the shape of the data, what data you need, and how eventually you want it to look before rendering it in your app, browser or database.

A more fancier definition for why we use it: Data from different sources can be pulled together and restructured to a standardized format.

Let's walk through each step of extracting, transforming and loading data with React and Rails.

Extract - PLAN IT

Extract, is all about planning for the transforming. There are 3 steps or questions to ask yourself to find the answers needed in order to move on to the next step, transform.

Where is the data that I need?
- Get it locally from your routes file?
- From another endpoint or 3rd party API like the Spotify API?
  - For our example, we will use hardcoded code found in our controller.

  def cats
    render json: {
      cats: [ # Cat.all
        {
          name: "Maya",
          color: "calico",
          rating: "perfect",
          owners: [
            "Mark"
          ]
        },
        {
          name: "Sully",
          color: "seal bicolor",
          rating: "perfect",
          owners: [
            "Erin"
          ]
        }
      ]
    }
  end

What specific information do I need from that data?"
- Decide what data to extract
  - In our example, let's extract the colors of the cats. So we want to return only the colors.
What should I use to retrieve that data?
- Query/retrieve the data
- A fetch request with JS on the frontend?
- A Net::HTTP with Ruby?
  - For our example, we will use request which is a custom little thing we built internally. It is build off JS fetch.

This handleGet function is the main way we will extract and receive the data we need.

  async function handleGet() {
    const response = await request("/some-cool-route/cats") // built off js fetch, so this is a GET request
    console.log("handleGet", response)
  }

And our console log, would look like this:

Transform - DO IT

Now that we have learned where the data is (in a method within the controller), what part of it we need (the cat's colors) and how to retrieve the data (a fetch GET request using an internal tool we use) we can now start changing the data to the shape we want. We can restructure it, rename it, remove things we don't need and even add values.

What should the data structure look like?
- Since our example is small, we are looking only to return the cat's colors. We don't need the name, rating or owners.
  - We would want our transformed data to look like this if we were to console log it.

We can transform the cats array, to return only the cat colors by creating a function that takes the data (cat's array) and returns a new data structure (an array of cat colors).

With our data in a method in our controller, let's look at our react component that will render the page.

This is were we can create a transform function transformColors() that will return an array of each cat's color: ["calico", "seal bicolor"]

function transformColors(cats) {
  return cats.map(cat => cat.color)
}

Load / Render - SHOW IT

Where should the data then be loaded or rendered?
- Add it to the database or display it to the user
- In some situations, you may be adding this new array of cat colors to your database.
- I mostly work with rendering the data to the page with React components, so let's see it all play out that way.

Here is the react component rendering our new transformed data.

import React, { useState } from "react"
import { request } from "react-rb" // internal tool by CK

export default function Meow() {
  const [cats, setCats] = useState([])
  const [colors, setColors] = useState([])

  async function handleGet() {
// EXTRACT
    const response = await request("/some-cool-route/cats") // built off js fetch, so this is a GET request
    setCats(response.data.cats)

    const transformCatColors = transformColors(response.data.cats)
    setColors(transformCatColors)
  }

  return (
    <div className="m-8">
      <button onClick={handleGet}>Get cat colors 🐈 🐈‍⬛</button>
// LOAD/RENDER
      <div>{colors}</div>
    </div>
  )
}
// TRANSFORM 
function transformColors(cats) {
  return cats.map(cat => <div key={cat.color}>{cat.color}</div>)
}

Let's recap. We have Extracted the data from our controller using a fetch request. We then Transformed that data to return only the cat colors with our transformColors() function. And finally, we can Load/Render it to the page in our React component in the JSX.

Here's a gif it it all working! It is not pretty, but hopefully you get the idea.

Hopefully this small example helps explain ETL just a little but more!

A big shoutout to my coworker Mark M. for helping me grasp this concept even further and for setting up this awesome cat example.

NOTES: Merge or Rebase

Erin Fox — Tue, 15 Dec 2020 22:33:56 +0000

Recently, I learned that merging a main branch, into your local working branch is a thing 🤯. I was shocked that I just learned about this in my 3+ years of being an engineer. BUT it's never too late to learn. (And take notes for when you forget).

Before, I would always copy/paste my rebase notes from my Notion into my terminal, cross my fingers with selecting the correct "accepting incoming" changes. "blue is you" theories, hope the correct version is added, continued and then force pushed. Sounds scary... but it worked!

From what I understand, merging main into your branch and rebasing from master results in the same goal: The latest most up to date code on your branch with your new feature code.

Merging sounds better to me. At least, it makes more sense, visually. Gelling together all the code, vs dumping it on top, feels better. I needed to learn how to gel instead of dump.

So, let's gel.

Here are my merge and rebase notes. Be aware that these work for me, and may not do exactly what you're hoping for. Chat with your team and do more research when you're learning this.

MERGING

Merging takes the content of the main branch and integrates it with your branch. Only your branch is changing.

Let's do it.

$ git checkout [your-branch]
$ git merge main

~~another way~~

$ git merge main [your-branch]

REBASING

Rebasing takes the content of the main branch into a single "patch" and integrates onto your branch.

Apparently, rebasing incorrectly can unintentionally rewrite history. Which sounds really cool, but also like the climax of a bad time travel movie where everything goes bad because they talked to that person they weren't suppose to talk too and now everything is ruined.

For now, I'd like to not time travel until I understand it better.

It moves your entire branch on top of the main branch. It rewrites the project history by making new commits for each commit on your original branch.

Let's do it.

$ git pull origin main
$ git checkout [your-branch]
$ git rebase main

--resolve your conflicts-- when you are resolving conflicts *DO NOT COMMIT*

save ****the changes from the conflicts 

$ git add

and then... 

$ git rebase --continue

:wq // or however you have your editor setup 

-- repeat until no more conflicts --
*EVEN if there are no conflicts 

$ git push origin [your-branch] -f

Interactive Rebase

Now this just sound wild. Taking it a step further, interactive rebasing allows altering the commits as they are moved to the new branch. This gives you more control over your branch's commit history. I've heard you use this type of rebase when things are really harry.

Let's do it.

$ git checkout feature
$ git rebase -i master

This will then open the editor and list the commits that are about to be moved. 
This is where you can change the history, to make it look like how you need it.

Alright. So now we've learned there are 3 options:

Merge
Automation rebase ("regular")
Interactive rebase

Which one should you use? Great question. I'm still figuring that out, but it starts with understanding each of your options, and the situation that you're in. Do you need to rewrite history? Or will a merge be quicker and all you need?

I believe it's best to chat with your team to hear their philosophies and approaches so everyone is on the same page. Hopefully, all y'all are not time traveling at the same time causing madness and talking to that person you're not suppose to talk to that changes the future.

Nevertheless, Erin Fox Coded

Erin Fox — Fri, 02 Mar 2018 15:16:43 +0000

I began/continue to code because...

I love the feeling of FINALLY getting something to work! Even if it took hundreds of attempts, finally getting it and understand it is such an amazing feeling. That is what keeps me going.

I recently overcame...

getting my conference CFP's rejected. Even though none of them have been chosen, I still continue to enter them. You never know :)

I want to brag about...

the tech talk's I've been giving! Next one will be with GraphQL NYC.

My advice for allies to support women who code is....

show up and don't give up. There's too many times when I am the only female at a Meetup. We need to change that. If anyone needs a buddy to go with, I'm always in! Twitter= @erinfoox