QN : Ingest Data with Dataflows Gen2 in Microsoft Fabric

• Dataflows are a type of cloud-based ETL (Extract, Transform, Load) tool for building and executing scalable data transformation processes.
• Dataflows offer a wide variety of transformations, and can be run manually, on a refresh schedule, or as part of a data pipeline orchestration
• A dataflow includes all of the transformations to reduce data prep time and then can be loaded into a new table, included in a data pipeline, or used as a data source by data analysts.
• Dataflows can be horizontally partitioned as well. Once you create a global dataflow, data analysts can use dataflows to create specialized semantic models for specific needs.
• Dataflows allow you to promote reusable ETL logic that prevents the need to create more connections to your data source.
• Benefits:

1. Extend data with consistent data, such as a standard date dimension table.
2. Allow self-service users access to a subset of data warehouse separately.
3. Optimize performance with dataflows, which enable extracting data once for reuse, reducing data refresh time for slower sources.
4. Simplify data source complexity by only exposing dataflows to larger analyst groups.
5. Ensure consistency and quality of data by enabling users to clean and transform data before loading it to a destination.
6. Simplify data integration by providing a low-code interface that ingests data from various sources.

Limitations:

1. Dataflows aren't a replacement for a data warehouse.
2. Row-level security isn't supported.
3. Fabric capacity workspace is required

• you can create a Dataflow Gen2 in the Data Factory workload or Power BI workspace, or directly in the lakehouse. Since our scenario is focused on data ingestion, let's look at the Data Factory workload experience. Dataflows Gen2 use Power Query Online to visualize transformations

- The combination of dataflows and pipelines is useful when you need to perform additional operations on the transformed data.