DEV Community: Haji Rufai

Introduction to Python for Data Engineering

Haji Rufai — Thu, 01 Sep 2022 17:40:10 +0000

Yes hello! With increasing interest in data engineering expertise among organizations, we have seen a rise in the demand for data engineers. We have seen Python as one of the main pillars in data engineering.

Well, what is Python? Why is it preferred for data engineering? And finally, most importantly, the scope - and how to get started.

What is python?

Python is a 4 GL (fourth generation) dynamically typed programming language. Thus, it is high-level and hence easier to learn and understand.

Python has seen an increase on its use due to its ease of use and flexibility. You'll get a nice introduction to python here.

Why python for data engineering.

A data engineer's job entails interacting with various data formats. Python is the best choice in these situations. Its standard library facilitates simple management. One of the most popular data file types are csv files.
A data engineer is often required to use APIs to retrieve data from databases. The data in such cases is usually stored in JSON (JavaScript Object Notation) format, and Python has a library named JSON-JSON to handle such type of data.
The responsibility of a data engineer is not only to obtain data from different sources but also to process it. One of the most popular data process engines is Apache Spark which works with Python DataFrames and even offers an API, PySpark, to build scalable big data projects.
Directed acyclic graphs (DAGs) are used with data engineering tools like Apache Airflow, Apache NiFi, etc. DAGs are nothing more than task specification codes written in Python. Data engineers will therefore be better able to utilise these technologies by learning Python.
Finally, maybe not least, Python has got tonnes of libraries that a data engineer finds useful:

Some of the python libraries for data engineering.

1. Pandas

Pandas is the Python library popular among data analysts and data scientists. It is equally useful for data engineers, who use it for reading, writing, querying, and manipulating data. Pandas dataframes are extremely compatible with two popular data types: csv and json.

2. Psycopg2, pyodbc, sqlalchemy

When someone hears the word "database," they often picture information kept in the form of tables with different rows and columns. A relational database (RDB) is the name given to this kind of database.
There are many ways to communicate with these databases, and the majority of them rely on Structured Query Language (SQL). MyPostgreSQL is one such solution that is well-liked by data engineers, and Python has a number of libraries to connect to it, including pyodbc, Sqlalchemy, and psycopg2.

3. Scientific Python (SciPy)

As its name suggests, SciPy is a Python library that provides a number of functions for rapid mathematical operations. This library allows a data engineer to do mathematical computations on their data for more accurate analysis.

4. BeautifulSoup

This well-known library is used for web scraping and data mining. For the purpose of preparing their data, data engineers use this to extract information from websites and work with JSON/HTML data formats.

5. Petl

Petl is a Python package for extracting, modifying, and loading tabular data. Data engineers use this library for building ETL (Extract, Transform, and Load) pipelines.

Scope (And Lets get started)

Python is general-purpose programming language that is used in many field from web development, automation , networking, etc. you mention it.

A data engineer does not need to know every Python there is because each one is a large field on its own, which is another journey not on our roadmap, and thus we do not need to know in detail.

For example, python for web development (flask and Django), machine learning—well, a data engineer does not need to get deep into machine learning.

0. Getting started

Install Anaconda on your machine

Anaconda/download
No comprising here.

Anaconda is a free and open source distribution that consists of all the packages and web programs such as Jupyter that you'll need.

After installing anaconda you'll automatically have Jupyter notebook installed which is a great python IDE which saves your source files as .ipynb,

Jupyter runs on your browser:

Illustration

The above image shows on how Jupyter when opened will look on your machine. You can navigate to the folder where you want to create your .ipynb file

When you are on your desired directory, You then click on 'New' then select by clicking 'Python 3 (ipykernel)' to open your ipynb file.

It will open untitled ipynb file (notebook file) which will look like the above picture.
You can rename your notebook file by clicking on the 'untitled' as shown.

Start your stuff there! Oh yes, Press 'shift + Enter' to run your cell(the rectangular input field for your code).
All da best.

1. Python basics

Where to Learn

learnpython.org
It is a nice interactive website and beginner friendly for python language. There are several topics arranged in order, for each topic there is a coding exercise at the end to test you if you have mastered the topic.

The good part (not the lazy part) there is solution to all exercises!!

2. Data structures and algorithms

Learning data structure and algorithms is mandatory for a good data engineer and it will also sharpen you to a better programmer. This concept should be in your RAM!!

Where to Learn

Google/free/Udacity/data structures and algorithms
The comprehensive course will make you grasp in depth data structures and algorithms. The good part is that it is also taught in Python. And Yes it is free!

3. Python Statistics

A data engineer needs to have a base in mathematics of data and should have a ground on:

Descriptive and inferential statistics.
Probability distributions
Hypothesis testing

Where to learn:

Resource: Brief/Comprehensive/pdf

4. Python Developer

You have been coding for a while. Now you need to learn how to write clean code.

That is where Python Enhancement Proposal 8 (PEP-8) comes in place. It is a document written in 2001 by Guido van Rossum (the developer), Barry Warsaw, and Nick Coghlan.

The primary focus of PEP 8 is to improve the readability and consistency of Python code.

Code is read more often than it is written.

Guido van Rossum , the creator of Python programming language.

Where to learn

Official/Documenation
realpython.com/pep8
Yes you need to learn how to write clean code. You will need to how to proper document your functions, methods, etc

Micro Illustration

CONCLUSION

Without debate, we can conclude that python is the first choice programming language for a data engineer. Well, congrats till here!

REFERENCE:

https://explore-datascience.net/
https://www.projectpro.io/article/python-for-data-engineering/592
https://www.youtube.com/c/DataEngUncomplicated

Yes, well have you started your track yet?

101 DATA ENGINEERING

Haji Rufai — Sun, 21 Aug 2022 13:27:00 +0000

Hello there!! You may have heard or not about Data Engineering and Data Engineers. What is it? Who are these data engineers? What do they do? Are they paid well? I mean if yes! what stuff (technology and knowledge) do they have? Maybe I want to become one if its feasible - Then: ..

What is Data Engineering?

Data engineering is the profession of creating systems for massively scalable data collection, storing, and analysis.

That's it. Read it again if you are just interested.

Relationship and difference with Data Science

Data science and data engineering are both big data and data analytics fields that have gained momentum in recent years.

Data science is a multi-disciplinary field that involves extracting knowledge from data to solve problems.

Data engineers are those who are responsible for building the data pipeline that will move data from where it is stored to where it can be used e.g. by Data scientists.

What does a Data engineer do?

A data engineer is a person responsible for the data engineering aspects of building and maintaining data systems.

With a background in machine learning, analytics, and data sciences, the data engineer works to combine and manage data sets while providing its users with a wealth of data visualization tools.

Pay??

Hehe, data engineers are among the top paid tech professionals, more than software engineers and data scientists.

Furthermore, why is it better ? For a job posting, there are around 8x fewer applicants than software engineers.

Requirements?

No requirements but a degree in Mathematics, Computer Science, Engineering and related field is a plus.

Though the majority of those entering the field are software engineers, data analysts, and data scientists, the field is not limited to complete beginners.

What is the career path?

There are self-taught data engineers, but the most effective are those who follow a guideline on what and what to cover. There are online courses eg Udacity Data Engineering, edx.IBM Professional Data Engineering etc. most of which are premium (Ouch!).

That's not the end. You can follow a certain guideline and strictly stick with it, and each topic can be independently found on the internet.

Here are the base foundations that you need

1. Python

No compromising here. You need to get started and dig deep into Python. Also, keep in mind that it is Python, not any other programming language.

Where to learn

LearnPython.org
It is a nice interactive website and beginner friendly for python language. There are several topics arranged in order, for each topic there is a coding exercise at the end to test your if you have mastered the topic.

The good part (not the lazy part) there is solution to all exercises!!

2. Structured Query Language (SQL)

A data engineer interacts a lot more with databases than a back-end software engineer and a data scientist. To grasp SQL is mandatory for data engineers.

Where to learn

W3Schools/SQL
There is such a large SQL resource at w3school that you may be overwhelmed by it, especially if you are starting. But hey! It is very interactive and user-friendly and should be considered 50% of your journey of data engineering database conquest. The good part is that you can practice your SQL skills on their platform, which is very flexible. Take your time , take key notes as you learn. Learn how you learn best and improve on it. Cheers!

3. Big Data (Spark and Hadoop)

Let's talk about big data and the trend. The term big data is often used to describe large data sets and data warehouses used by businesses to store and analyze large amounts of data.

The technologies (framework) used are spark and Hadoop.

Where to learn

SkillUP by simplilearn
The above will give you the best dive in even if you are beginner. Yes ,It's free!

4. Cloud Computing

What is it?

Cloud computing is the use of computing resources (computers, storage, networking equipment, and applications) that are accessed through a web browser, over the Internet.

These resources are provided remotely - usually through the Internet - so that users do not have to be at the workplace to get their computing needs met.

There are several cloud computing providers. Some of them are:

Google Cloud Platform (GCP)
Amazon Web Service (AWS)
Microsoft Azure
Oracle Cloud

What you need to have

You need to master at least one cloud service and have a basic understanding of the others, as different companies use different cloud services.

My opinion master with AWS first as it is the largest cloud service provider.

Where to learn

Again Simplilearn-SkillUp Getting started with AWS fundamental will give you a headstart especially if you are a beginner.

It is a total of 4 hours of video divided into well-explained lessons, and it is free. From there, you are now robust enough to go to 999 Data Engineering.

How long will it take?

With the right plan and guidelines, giving 30 hrs/week for 6 months, you are ready for it!

Conclusion

Starting the data engineering journey can be easy. But maintaining the quest needs grit.

Have some patience as you begin your journey. For sure, you will face difficulties in grasping some concepts on the way (as you learn and do projects).

Another thing I want to add is

Difficulty is relative. With some concepts, you may find it easier and others harder compared to your peers. Push yourself and learn.

Congrats!

Have you started your data engineering journey yet?

DEV Community: Haji Rufai

Introduction to Python for Data Engineering

What is python?

Why python for data engineering.

Some of the python libraries for data engineering.

1. Pandas

2. Psycopg2, pyodbc, sqlalchemy

3. Scientific Python (SciPy)

4. BeautifulSoup

5. Petl

Scope (And Lets get started)

0. Getting started

Install Anaconda on your machine

Illustration

1. Python basics

Where to Learn

2. Data structures and algorithms

Where to Learn

3. Python Statistics

Where to learn:

4. Python Developer

Where to learn

Micro Illustration

CONCLUSION

REFERENCE:

101 DATA ENGINEERING

What is Data Engineering?

Relationship and difference with Data Science

What does a Data engineer do?

Pay??

Requirements?

What is the career path?

Here are the base foundations that you need

1. Python

Where to learn

2. Structured Query Language (SQL)

Where to learn

3. Big Data (Spark and Hadoop)

Where to learn

4. Cloud Computing

What is it?

What you need to have

Where to learn

How long will it take?

Conclusion