2022 Beginner Friendly Modern Data Engineering Career path With Learning Resources.

• Start by trying to understand the fundamentals, what data engineering is, some of the common task that data engineer performs at different companies, common data terminology and learn how to define your problems before you tackle them.

• Work with SQL to use databases for storing, reading and updating data.

• Learn the fundamentals of Python programming: working in notebooks; logic and functions; and data structures.

• Get to grips with everything about public cloud, You can use one for instance if you choose to use AWS, learn about a host of services offered by AWS and work hands-on with them to work with data and applications in the cloud.

• Learn how to connect large data sources in the cloud to create data lakes. Understand data analytics as it pertains to big data and data lakes.

• Learn how to build data pipelines. Learn how to get your data where you want it and when, using tools like Apache Hadoop and Apache Spark.

• Gain practical experience writing functions in Apache Spark to test quality metrics and learn how to document data lineage.

You can read more about this and more from the presentation we had for Data Science East Africa Data Engineering Bootcamp here.

Here are some of the resources i used when i was getting started in data engineering.

Python

1. https://docs.python.org/3/
2. https://www.pythontutorial.net/
3. https://www.w3schools.com/python/

SQL

1. https://www.w3schools.com/sql/default.asp
2. https://www.sqlservertutorial.net/

Amazon Web Services, AWS.

1. https://aws.amazon.com/

Microsoft Azure

1. https://learn.microsoft.com/en-us/azure/

Google Cloud Platform

1. https://cloud.google.com/

Apache Spark

1. https://spark.apache.org/docs/latest/

Please add any useful resources that you think might be important for an aspiring data engineer in the comment section.

Thank you for reading all through and all the best as you explore and build your world class data engineering career ✌️.

Comments

[Adit Modi]

Here is the advice that I wish I had known when I first began studying data engineering, and that I would advise everyone to be aware of.

1. Become proficient at programming

• Learn Python
• Learn the basics of Scala

1. Learn automation and scripting

• automation is crucial for data engineers
• Essential tools for automation

1. Understand your databases

• Start by learning SQL basics
• Learn how to model data
• Learn how to work with less structured data

1. Master data processing techniques

• Learn how to process big data in batches
• Learn how to process big data in streams
• Load the result in a target database

1. Schedule your workflows

• Workflow scheduling with Apache Airflow
• The ecosystem of tools

1. Study cloud computing

• The case for using a cloud platform
• Common services provided by cloud platforms

1. Internalize infrastructure

• When to use Docker
• When to use Kubernetes

1. Follow the trends

[Dennis Ngigi Karanja]

Thank you for this..been wanting to shift into tech , Data Engineer sanasana... asante