QN : Data Engineering on Azure

• Data Engineer is responsible for integrating, transforming and consolidating data from various structured and unstructured data systems into structures that are suitable for building analytics solutions

• Azure data engineer also helps ensure that data pipelines and data stores are high-performing, efficient, organized, and reliable, given a specific set of business requirements and constraints.

Types of Data

• Structured Data : Comes from table based source systems eg relational dbs or csv files. Primarily made up of rows and columns consistently throughout the file.

-Semi Unstructured Data : Data such as JSON which may require flattening before loading. Data has no table structure

• Unstructured data : data stored as key value pair.Has no relational db standard eg PDFS, Word documents and images

Data Operations

• Data Integration : Establishing links between operational and analytical services through data sources ensuring data is secure, reliable and accessible.

• Data Transformation : Transforming operational data into suitable structure for analysis often through ana ETL or ELT process. Here data is prepared for downstream processes.

• Data Consolidation : Combination of extracted data into consistent structure to support analytics and reporting.

Data Engineer uses common languages:

• SQL
• Python
• KQL - Kusto Query Language , Used for analyzing streaming and log data. Used in Microsoft Fabric Real Time Intelligence workload.
• Others dependent on organization

Keywords:

Operational data : transactional data generated ad stored by applications in relational or non relational dbs

Analytical data : data optimized for analysis and reporting often stored in a data warehouse

Streaming data : perpetual data sources that generate data values in real time to specific events eg IOT devices

Data Pipelines : Used to orchestrate activities that transfer and transform data. Primary way for ETL/ELT.

Data Lakes : Storage repository for native, raw data. It is optimized for scaling to massive volumes of data. Data comes from multiples sources. Data may be structured or semi or unstructured. Here, Store data untransformed.

Data warehouse : centralized repository of integrated data from one or more disparate sources.Data is optimized for analytical queries.Data is organized into relational tables organized into a schema.

Lakehouses : Combines the scalability of a data lake with the querying capabilities of a data warehouse.Stores data in delta lake format which supports ACID transactions, schema enforcement and support structured and unstructured data.

Apache Spark : parallel processing framework that takes advantage of in memory processing and distributed file storage.

The diagram above describes the flow of data from and enterprise data analytics solution

• Operational data is generated by applications and devices and stored in Azure data storage services such as Azure SQL Database, Azure Cosmos DB, and Microsoft Dataverse.Streaming data is captured in event broker services such as Azure Event Hubs.

• operational data is captured, ingested, and consolidated into analytical stores where it is modelled and visualized in reports and dashboards.

• Core Microsoft technologies used to implement data engineering workloads include:
  Microsoft Fabric, Azure Data Lake Storage Gen2, Azure Stream Analytics, Azure Data Factory, Azure Databricks

• Microsoft fabric is unified , end to end SaaS platform and brings together data engineering tools.