DEV Community: Amoskinuthia

Data Engineering Toolbox

Amoskinuthia — Fri, 16 Sep 2022 15:31:10 +0000

Data engineering toolbox

A data engineer is an IT professional whose responsibility is to ensure that data is available in the right place, secure, and in the required form for analysis. They are referred to as data engineers because their work revolves around designing systems and processes that to collect data from diverse sources and lead them to a central storage i.e data warehouse or a data lake. To do this a data engineer needs several tools and technologies. The tools and the skills vary depending on the amount of data to be handled or processed. A data engineer must work in conjunction with other departments in the company they work for to better understand the requirements of the data they need to work on. Mostly they work with executives, data analysts and data scientists. after understanding the kind of data the company needs the engineer will advise the company on the technology they need to deploy. In this post, I will discuss the various tools an engineer can have in their toolbox and their use cases and options.
Data engineering skill sets
To be able to design systems and solutions for data engineers must be equipped with software development technologies, having a development mindset enables them to use a wide variety of programming languages and even learn new ones easily to build data pipelines through which data passes, in this pipelines the data is transformed and put in the required form before being deposited in the data warehouse or data lake since they have a background. It is impossible to master all the available languages but a solid foundation is enough to learn a new technology on the go. Below is a listing of the technologies in data engineering in job listings in the year 2020 by Jeff Hale.

SQL
Structured query language(SQL) is a language used when communicating with relational databases.
Relational databases are databases that store related data i.e data organized in preset conditions and relationships where data is fed in tables with rows and columns. Some people have argued that SQL is not one language and they have divided it into data definition language(DDL) – This one deals with creating or modifying the database like creating and altering tables e.g using CREATE and ALTER commands, data manipulation language(DML) – this enables users to query data using commands like SELECT, UPDATE, DELETE, etc and data control language(DCL)- this enables access controls and security using commands like GRAND and REVOKE.
SQL has several that have some features different but advanced from the standard SQL, some of these dialects include ;
PL/SQL – procedural language/SQL
Transact- SQL
PostgreSQL
MySQL
This is a must-have tool for all professionals working with the data.

Python
This is the most used language in the field of data it is a general purpose, high-level, interpreted programming language. Being a higher-level language it is easier to learn since beginners will not have to learn or understand what happens under the hood when a program is run.
Being a general-purpose language it can be used for quite a variety of domains like web applications development, automation, data engineering, data science, AI, machine learning, software development, mobile applications and so much more.
Being an interpreted language it doesn’t need a compiler and uses an interpreter that reads the source code line by line while it is executing. The main reason why python has become the de facto language for data is because of its simple syntax and its rich third-party libraries that have been developed for data applications.
NOSQL
NoSQL refers to an approach to storing and accessing data that is unstructured unlike in relational databases, NoSQL data is modeled in other forms other than the tabular forms in relational databases, NoSQL is preferred where high scalability and availability are required especially in big data where data is continuously growing. Examples of NoSQL databases are MongoDB and Cassandra.
Cloud platforms
Due to the huge amount of data being generated each day data engineers must be well of the different cloud technologies available to store data in the cloud there are many cloud platform providers and mastering and or more is an invaluable tool for a data engineer. Amazon Web services(AWS), Microsoft Azure, and Google cloud platforms are the leading in the industry.

*Open frameworks *
There are several data engineering frameworks used to work on big data mastering the following will keep you equipped for data engineering roles;

Apache Spark
Hadoop
Kafka
MapReduce
Perhaps Hive
Apache Airflow
Apache Storm
Apache SAMOA(Scalable Advanced Massive Online Analysis).

In conclusion, data engineering is an ever-growing and evolving field and new tools are being invented daily to be efficient one has to keep learning to remain upto date. The goal is to develop efficient systems that are stable and reliable in collecting and maintaining data. A solid foundation of the above technologies will keep you ahead and you can always learn other tools on the go.

Data Engineering 101

Amoskinuthia — Sun, 21 Aug 2022 16:51:22 +0000

Data engineering is a very distinct role in the field of IT and is at the top of the ranks in the data roles stack since it lays the foundation for the rest of the professionals like; business analysts, data analysts and data scientists. Precisely data engineering is the process of preparing data for data analysis and general consumption. This done through complex processes that ensure that the needed datasets are collected, transported and stored in the right place and can be accessed by consumers in the right format. Therefore, a data engineer is responsible for deciding the data to be collected, how to collect it, how to transfer and store it securely in the right format.

Tools used in data engineering**

In the recent past big data has continued to grow immensely and it is projected to grow even further, recent projections show that by the year 2025 the world will be producing around 463 exabytes of data those are huge amounts of data, literally 18-digit amounts of data! This amount of data needs special skills and tools to be handled correctly, the commonly used tools and techniques include:

Programing using languages such as Python, Java, R, Scala etc. these languages are used to come up with tools such the data pipelines and for writing automation scripts.
Cloud computing technologies help in storing the data in large amounts as they can handle growing data. Examples of the providers include Microsoft Azure, Amazon web services and Google cloud platform.
Database management systems that include SQL and NOSQL systems like MySQL, mongo DB etc. these systems are used to store and manipulate data.
Machine learning – though machine learning is mostly used by data scientists, data engineers need the skills so that they understand the needs of the data scientists so that they can serve them better.

Path to becoming a data engineer

*A university degree *
A university degree in computer science, mathematics and statistics, physics or any other related field is important but not mandatory employers look at the skill set and the ability to solve problems.

*Certifications *
There are a number of certifications that test specific skills and give certificates, most of these certificates are taught by industry experts and they are an added advantage in the job market. Some examples of platforms offering certifications include:

Udacity
Coursera has a number of them
Edx
Udemy
Google
Microsoft
Aws

*Bootcamps and communities *
These add value since they are made of people practicing data engineering and other related skills.

After acquiring the necessary skills what’s next? One needs to brand as data engineer in order to secure a job this can be done by creating an online portfolio and include projects that one has undertaken, host them on personal websites or on GitHub.

Introduction to python for data engineering

Amoskinuthia — Sun, 24 Apr 2022 14:22:10 +0000

Do you want to build a website? an application? a video game
name it or manipulate data, python got you covered. Python is a multi-purpose high level programming language that can help you develop or do almost anything.
It is easily available for free and open source , it has gained popularity and it is being used by big organizations like Google, Disney and even NASA.
Python has evolved and now we have python 3 which is the latest version. Python is famous among data handlers that is data engineers, data scientists and data analysts, this because it has a wide variety of libraries and modules like Pandas, numpy, scikit-learn,seaborn, matplotlib etc that make data preparation,data processing and data analysis efficient. data engineers use python to create data pipelines and to write script to automate data cleaning amongst other processes
lets try it!
lets try writing our 1st python program and see how simple it is

print(`hello world!`)
output: hello world!

The print statement will be followed by a parentheses that enclose what we want displayed as our output
python syntax
whitespaces and indentation
Python uses whitespace and indentation to construct the code structure unlike other programming languages that use semicolons to separate statements.
comments
Comments help us document our code for future reference
we have several ways of writing comments in python

using #

`#this is a comment`

it is mostly used for single line comments

using ***

`*** this is a multiline comment***`

identifiers
Identifiers are used to identify variables, functions and other objects in python.
The name of an identifier should begin with a letter or an _
python is case sensitive and therefore care should be taken when naming them.
it is important to note that python key words should not be used as identifiers
keywords
Some keywords in python are listed below
False
class

None

True

while
raise
this are words which have a special meaning in python
you can use the code below to list all the keywords

import keyword 
print(keyword.kwlist)

Data types
strings
integers
floats
*control flow *
Boolean and comparisons
Booleans have two values that is true and false
if statements
with if statements the condition check and if its true the statement is executed otherwise they are not.

x=45
if x>5:
    print ("x is greater than 5")

else statements
The else statement can be used to execute statements if the if statements is false.
Same as if statements the code inside the statement needs to be indented.

x=4
if x==5:
 print("yes")
else:
 print("no")

while loops
we use while loops to repeat a block of code several times


i=1
while i<=5:
 print(i)
 i=i+1
print("finished!")

_break _
This is used to break a while loop if the conditions we want are matched

`i=0
while true:
 print(i)
 i=i+1
if i>=5:
 print("breaking")
break`