DEV Community: Khalif Cooper

Building ETL Pipelines in Python: Part 1

Khalif Cooper — Sat, 21 Aug 2021 03:17:01 +0000

Data engineering refers to the development of software that performs three tasks:

Extract raw data from various sources.
Transform and clean up the data.
Load the data into a data warehouse, i.e., a database management system like, Google BigQuery or Amazon Redshift. Alternatively,load the data into MySQL, a relational database.

The process is complex involving numerous software tools and advanced technical expertise. For one thing is can be challenging and overwhelming to identify the starting point and resources from which to acquire the basic knowledge. No wonder the demand for data engineers in the technology sector has risen by an order of magnitude in the past years. As the need for data generation grows, the thirst for data engineers will continue to intensify.

This post, part 1 of a three-part tutorial shows you how to build a simple ETL[extract,transform,and load] pipeline with CSV files in Python, describes the extraction steps.

Software Requirements

You will need the following software:

glob, a package that is preinstalled that searches for files in relative paths.
For details, see this section for documentation.
pandas, a package used for data manipulation. To install, type on the command line, pip install pandas.
For details, see the pandas site.
virtualenvwrapper, a package that sets up virtual environments for Python. For details, see the related documentation.
This Kaggle dataset for the CSV data.

Here's an example of CSV data on car sales:

The Procedure:

Create a project called etl_car_sales with PyCharm.
Create a virtual environment with the command line mkvirtualenv etl_car_sales.
Install pandas and virtualenwrapper.
Extract the zip file and move the CSV files for car_sales to your etl_car_sales directory, like this:

Create an extract_from_csv function to read the CSV file in your main directory and then return a Dataframe with the pandas function read_csv.

Create an extract function to read the data from the extracted csv file and append the data to the Dataframe that is being return the extract_from_csv function.

Tip: To rename the columns, do the following:

Print out the extracted_data variable and the result would read like this:

Part 2 will describe how to transform data in the ETL pipeline. Stay Tuned.

Classes and Objects in Dart: Part 1

Khalif Cooper — Thu, 09 Apr 2020 15:49:19 +0000

Do you ever have trouble with OOP? Object Oriented Programming. Do you know what it entails? Do you want to be able to not only understand the concepts but, be able to implement them as well? If you answered yes to any of the following questions, then today is your day.

I will be creating a series of six parts going over objected oriented programming discussing classes, objects, constructors (abstraction, polymorphism, encapsulation and inheritance). Outside of classes, constructors, and objects; these four concepts make up what is called the four pillars or principles of object oriented programming.

Other objected oriented languages that you might use everyday include Java, C#, C++, PHP, JavaScript, Smalltalk, Python, Dart, Swift, etc.

Dart will be the language used in this series. If you want to follow along, you can use this awesome online dart editor.

Note: "class" is a keyword in dart and what follows after is the class name.

Think of a class as an initial blueprint or template. Lets' imagine for a minute, a car with nothing on it, a bare bones car, with no wheels, no windows, no color, nothing. The car itself is a class. A class tells you "What" it is and a class can only be declared once. So, for example, a type of a car could be a Limo, Sedan, Truck, Van, and a Taxi; but at the end of the day, every type of car, is still a car itself. They are all made from the initial car. So lets' see what a car class looks like:

  class Car {

   }

Now on to Objects! In the example above, we created an initial blueprint, "class" with the class name of "Car".

An object is also the name of a class, which is "Car" in our case. And if we wanted to create one copy of the "Car" class; it will create an object that will be an instance of a class. So, to put it all together, every time a copy of the initial class is created, an instance of an object and class will be created as well.

How? Lets' go back to our imagination again. You are at the dealership and you are not able to make up your mind what type of car you want. So you buy a Limo, Sedan, Truck, Van, and a Taxi. Every car that is made will use this original "Car" model class to create another one. All of those types of cars, will be considered objects that are "instances" or copies derived from the original "Car" class we made above. So in code that would look like this:

class Car {

   }

 void main(){
     Car limo = new Car();
     Car sedan = new Car();
     Car van = new Car();
     Car taxi = new Car();

     print(limo); //instance of 'Car'
    }

Note: In Dart, we use the "void" keyword, which is used when you create a method or function that doesn't return a value. The main method is required in dart to run any code. And In the main method we use the "new" keyword to create those instances. When we print out the limo instance, it tells us, we created an instance of the car class.

Now that is cleared up, lets' go back to our "Car" class. Inside of our "Car" class, we need to include some information or details as well as functionality about our car. For example, color, wheels, windows, make, how fast it goes, what sound does the car make etc..

class Car {
     double wheels = 4;
     double windows = 4;

    String mph(int speed){
     return "up to $speed mph";
     }

    void carEngineStart(){
       print("zzzrrrroooommmm");
       }


   }

 void main(){
     Car limo = new Car();
     Car sedan = new Car();
     Car van = new Car();
     Car taxi = new Car();

     print(limo.wheels); // 4
     print(limo.windows); // 4
     limo.carEngineStart();// zzzrrrroooommmm
     print(limo.mph(65));// up to 65 mph
    }

In the example, we created a void method that takes an integer and prints out the speed and string method that prints out car sound. Then in the main method, we called those methods on the instances.

Hope you learned about what OOP is in this article. And learned in summary, that when a class is first created, it creates a blueprint of an object. Also inside of a class, you can create methods and properties that can be initialize when you create multiple copies or instances of the original class. And when you will create instances of the class. You also create instances of the object as well.