DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
The Wrath of Unicron - When Airflow Gets Scary

The Wrath of Unicron - When Airflow Gets Scary

5
Comments
4 min read
End to End Netflix data analytics and recommendation system project using Microsoft Azure tools

End to End Netflix data analytics and recommendation system project using Microsoft Azure tools

9
Comments
5 min read
Navigating the Data Engineering Landscape: From Raw Data to Insights

Navigating the Data Engineering Landscape: From Raw Data to Insights

5
Comments 1
7 min read
Machine learning 101

Machine learning 101

85
Comments
8 min read
How Starburst’s data engineering team builds resilient telemetry data pipelines

How Starburst’s data engineering team builds resilient telemetry data pipelines

1
Comments
7 min read
Building ETL/ELT Pipelines For Data Engineers.

Building ETL/ELT Pipelines For Data Engineers.

5
Comments 2
2 min read
Automating Talend Jobs Using Apache Airflow .

Automating Talend Jobs Using Apache Airflow .

7
Comments
3 min read
Data-aware Scheduling in Airflow: A Practical Guide with DAG Factory

Data-aware Scheduling in Airflow: A Practical Guide with DAG Factory

8
Comments
6 min read
Automating Data Pipeline Deployment on AWS with Terraform: Utilizing Lambda, Glue, Crawler, Redshift, and S3

Automating Data Pipeline Deployment on AWS with Terraform: Utilizing Lambda, Glue, Crawler, Redshift, and S3

Comments 1
8 min read
Push dbt beyond boundaries: Exploring a Fresh Approach to dbt Integration

Push dbt beyond boundaries: Exploring a Fresh Approach to dbt Integration

1
Comments
1 min read
There is no Data Engineering roadmap

There is no Data Engineering roadmap

2
Comments
5 min read
Workflow of Data Engineering Project on AWS

Workflow of Data Engineering Project on AWS

1
Comments
4 min read
Feature Engineering Has a Language Problem

Feature Engineering Has a Language Problem

1
Comments
15 min read
What is data engineering and a B.I architecture

What is data engineering and a B.I architecture

5
Comments
6 min read
How To Create Dataflow Job with Scio

How To Create Dataflow Job with Scio

2
Comments
8 min read
Using pyspark to stream data from coingecko API and visualise using dash

Using pyspark to stream data from coingecko API and visualise using dash

3
Comments
6 min read
AWS Redshift: Robust and Scalable Data Warehousing

AWS Redshift: Robust and Scalable Data Warehousing

3
Comments
6 min read
A mage on the Hero’s Journey: a fantasy epic on how a startup rose from the ashes

A mage on the Hero’s Journey: a fantasy epic on how a startup rose from the ashes

6
Comments
9 min read
Stream data processing with Mage

Stream data processing with Mage

6
Comments
8 min read
How to pivot data using Dynamic SQL in SQL Server

How to pivot data using Dynamic SQL in SQL Server

5
Comments 4
3 min read
How to clone tables in BigQuery

How to clone tables in BigQuery

2
Comments
1 min read
kafka: event driven microservices

kafka: event driven microservices

4
Comments
6 min read
Getting started with Apache Flink: A guide to stream processing

Getting started with Apache Flink: A guide to stream processing

33
Comments
8 min read
How to rotate data using Pivot & Unpivot operators

How to rotate data using Pivot & Unpivot operators

3
Comments 2
3 min read
Apply CDC From MySQL To Clickhouse on local environment

Apply CDC From MySQL To Clickhouse on local environment

7
Comments
6 min read
loading...