If you are a new reader I am Madiha Khalid Senior Data Engineer. I am working as a Professional Data Engineer and Solution Architect for the past six years. I bring here my passion to start writing about Modern Data Engineering Tools Stack along with Modern Data Architectures.
The purpose of this article is to convey my perspective and reflections on what I have observed during the preceding five months, as well as how the expectations for Data Engineering responsibilities trends vary from one company to the next in current times.
What Is the Data Engineering (DE) Role?
We saw that Data Engineer including myself works most of the time building reliable and scalable ETL pipelines. We spent an enroumous amount of time on data integration and making the pipeline reliable, providing refreshed data every day to our stakeholders. Building one source of a truth data warehouse is always a big challenge.
However, In the last couple of years, the Data Engineering position evolved a lot. There are countless tools available in the market. The data engineering role it became first and high demand job in the data world.
Why has DE become a high-demand job?
I want to fuel up my data engineering capabilities and start analyzing various Job posts with their requirements. I conclude that as Data Engineering position is in very high demand and highly paid job. Hence the expectation towards this role also increases. These days this is not enough if you write only Data Pipelines maybe in small Python scripts for data integration and Schedule these pipelines via some Orchestration tools like Airflow, Dagster etc. But
Data Engineering Tasks become extended Software Engineer Responsibilities skills with Data Integration, DevOps and Analytics Engineer sometimes.
I will try to elaborate more on this statement
Data Engineer with Software Engineering Skills:
Very strong Software Engineering Principles now become the first compulsory point for Data Engineers. Some companies required strong OOP programming, with unit testing while others want emphasis on Functional Programming concepts for specially for Apache Spark and Big Data Technologie Pipeline Development.
Data Engineer with DevOps Skills:
The Engineer should be responsible for all DevOps Tasks maintaining scalable infrastructure like Airflow, Kubernetes, Spark Cluster etc. Even though you could use Cloud Managed service there are still many things you need to optimize on daily ETL Pipeline optimization. Infrastructure as a code expertise AWS Cloudformation or Terrafoam or Terragrunt will add value to your profile.
Data Engineer with Data Integration Skills:
Should be responsible for building Data Integration for a Data warehouse or data lake with modern Cloud Datawarehouse (AWS Redshift, Snowflake, Databricks etc ) and Modern Data Lake concepts like Delta Lake.
Data Engineer with Analytics Engineer Skills:
The Data Engineer should be responsible for building a Data Model for the Analyst; if your team has no Analytics Engineer. Then the responsibility to create either a traditional datawarehouse using the Kimball Dimensional model or Databrick Medallion Architecture Layers technique to build Aggregated table. This could also be possible to write famous DBT (data build tool) jobs on Data Engineer’s shoulders.
Considering these skills and expertise, I believe this makes the Data Engineering position most challenging and high in demand. There are mixed requirements for these skills combination or all in one.
Future of Data Engineering Role:
The future of data engineering is proposing a no-ETL/ELT approach. As AWS announced for Zero-ETL. I am not sure how much we can able to achieve this is a reality due to many reasons like extensive data transformation, and data quality checks on the datawarehouse. I believe there are many self-service tools coming in the market which can help with the burden of Data Integration like Airbyte, Mage, Fivetran and Portable and so on. The tools are unlimited to help us out. But I believe No-ETL is still a bit far to achieve.
Conclusion:
The requirements are going crazy for Data Engineer’s responsibilities from company to company. Considering all these requirements Data Engineer position became in high demand but it comes with extra responsibilities.
PS: These thoughts are my own personal and the experience I observed during multiple interview processes.
This article is originally published at Medium
Top comments (0)