Data Engineering 102 ;Introduction to Python for Data Engineering

Kasamba Lumwagi — Mon, 29 Aug 2022 20:07:00 +0000

Commit yourself to learn the basics of python specifically the areas that are related to data Engineering.
You will need Python to be installed in the Operating system that you are using be it Linux, Mac OS, or the most common Windows. These are the topics that you should put more emphasis on:-

1.MATH EXPRESSIONS

Syntax Math Meaning
a+b a+b addition
a-b a-b) subtraction
a*b a\times b\ multiplication
a/b a\div b\, division (see note below)
a//b a\div b\ division - in python 2.2 & abv
a%b a mod b modulo
-a -a negation
abs(a) |a| absolute value
a**b a^{b} exponent
math.sqrt square root

2.Strings
Strings in python are surrounded by either single quotation marks, or double quotation marks.
e.g

'hello' is the same as "hello".

You can display a string literal with the print() function:

for more info :
Link

Variables Variables are containers for storing data values.

A variable is created the moment you first assign a value to it.

x = 5
y = "John"
print(x)
print(y)

for more info:
Link

4.Loops
Python provides three ways for executing the loops

(a)While Loop:
In python, while loop is used to execute a block of statements repeatedly until a given condition is satisfied. And when the condition becomes false, the line immediately after the loop in the program is executed.

while expression:
statement(s)

(b) For in Loop:
For loops are used for sequential traversal. For example: traversing a list or string or array etc. In Python, there is no C style for loop, i.e., for (i=0; i<n; i++). There is “for in” loop which is similar to for each loop in other languages. Let us learn how to use for in loop for sequential traversals

for iterator_var in sequence:
statements(s)

(c)Nested Loops:
Python programming language allows to use one loop inside another loop. Following section shows few examples to illustrate the concept.
Syntax:

while expression:
while expression:
statement(s)
statement(s)

5.Functions.
A function is a block of code which only runs when it is called.

The basic syntax is:
def my_function():
print("Hello from a function")

6.List, Tuples, Dictionary and sets

Its also important to learn how to connect databases with :-
1.BOTO3
2.Psycopg2
3.mysql

Together with the following essentials;-
1). JSON
2). JSONSCHEMA
3). datetime
4). Pandas
5). Numpy.

Data Engineering 101 : Introduction to Data Engineering

Kasamba Lumwagi — Sat, 20 Aug 2022 19:41:00 +0000

Before i thought data engineering and data science were just similar fields, well this changed this week data engineering entails the making of quality data available from various resources, maintain databases, build data pipelines, query data, data preprocessing, Feature Engineering, Apache hadoop and spark, Develop data workflows using Airflow etc while data science is about building ML algorithms, building data and ML models and deploy them, have statistical and mathematical knowledge and measure, optimize and improve results.

Week one down i've got the basic layout of the topics that we will be tackling finding it kinda easy for i now have a clear path on what it will take for me to become a data engineering. The following tools serve greatly in data engineering:
1.Cloud .
(AWS, Azure, GCP) master one but get a good grasp of all. AWS most preferably for its widely used. this will be used as a setup Development Environment to learn building Data Engineering Applications on GCP, AWS, Microsoft Azure.

2.Programming Language
I would recommend you to use python its faster and contains libraries and frameworks that are best suited for data engineering tasks .

3.SQL(Structured Query Language )
Makes it easy to manipulate databases.

4.A TEXT EDITOR
Visual Studio Code

5.Anaconda

Anaconda is a distribution of the Python and R programming languages for scientific computing (data science, machine learning applications, large-scale data processing, predictive analytics, etc.), that aims to simplify package management and deployment

6.Hadoop
It is an open-source framework that provides a distributed file system for big data sets. This allows users to process and transform big data sets into useful information using the MapReduce Programming Model of data processing.

7.Pyspark
PySpark is a data analytics tool created by Apache Spark Community for using Python along with Spark. It allows us to work with RDD (Resilient Distributed Dataset) and DataFrames in Python. PySpark has numerous features that make it such an amazing framework and when it comes to deal with the huge amount of data PySpark provides us fast and Real-time processing, flexibility, in-memory computation, and various other features. It is a Python library to use Spark which combines the simplicity of Python language with the efficiency of Spark.

THE TOPICS TO BE COVERED ARE-
1). Data Engineering
-What’s Data Engineering
-Why Data Engineering
-Data Engineers — ML Engineers — Data Scientists

2). Python for Data Engineering
-Basic Python with Project
-Advanced Python with Project
-Techniques and Optimization

Scripting and Automation -Shell Scripting -CRON -ETL

4). Relational Databases and SQL
-RDBMS
-Data Modeling
-Basic SQL
-Advanced SQL
-Big Query

5). NoSQL Data bases and Map Reduce
-Unstructured Data
-Advanced ETL
-Map-Reduce
-Data Warehouses
-Data API

6). Data Analysis
-Pandas
-Numpy
-Web Scraping
-Data Visualization

7). Data Processing Techniques
-Batch Processing : Apache Spark
-Stream Processing — Spart Streaming
-Build Data Pipelines
-Target Databases
-Machine learning Algorithms

8). Big Data
-Big data basics
-HDFS in detail
-Hadoop Yarn
-Sqoop Hadoop
-Hadoop Yarn
-Hive
-Pig
-Hbase

9). WorkFlows
-Introduction to Airflow
-Airflow hands on project

10). Infrastructure
-Docker
-Kubernetes
-Business Intelligence

11). Cloud Computing
-AWS
-Google Cloud Platform
-Microsoft Azure

THE FIRE KEEPS ON BURNING.

DEV Community: Kasamba Lumwagi

Data Engineering 102 ;Introduction to Python for Data Engineering

Data Engineering 101 : Introduction to Data Engineering