DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Data warehouse vs data lake

Data warehouse vs data lake

1
Comments
8 min read
SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

Comments
4 min read
Python Projects with SQL: Strategies for Effective Query Management

Python Projects with SQL: Strategies for Effective Query Management

11
Comments 2
9 min read
CAP Theorem

CAP Theorem

1
Comments
2 min read
NoSQL DATABASES

NoSQL DATABASES

1
Comments
1 min read
Apache Spark 101

Apache Spark 101

2
Comments
7 min read
PySpark: missing value

PySpark: missing value

Comments
2 min read
Cross-cluster replication for read-write separation

Cross-cluster replication for read-write separation

2
Comments
4 min read
"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

1
Comments
1 min read
Trino & Iceberg Made Easy: A Ready-to-Use Playground

Trino & Iceberg Made Easy: A Ready-to-Use Playground

11
Comments
3 min read
HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

Comments
3 min read
3 Reasons Data Engineers Should Embrace Apache Iceberg

3 Reasons Data Engineers Should Embrace Apache Iceberg

4
Comments
4 min read
Arrow Flight SQL in Apache Doris for 10X faster data transfer

Arrow Flight SQL in Apache Doris for 10X faster data transfer

6
Comments
10 min read
Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

1
Comments
5 min read
"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

1
Comments
1 min read
DAG no Airflow para invocar Google Cloud Function

DAG no Airflow para invocar Google Cloud Function

Comments
3 min read
"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

1
Comments
1 min read
How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

Comments 1
5 min read
Carregando dados com Apache HOP & Postgres

Carregando dados com Apache HOP & Postgres

2
Comments
3 min read
Demystifying:Azure Data Factory

Demystifying:Azure Data Factory

Comments
1 min read
Exploring Data Warehousing and ELT Tools

Exploring Data Warehousing and ELT Tools

1
Comments 1
2 min read
Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Comments
7 min read
"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

1
Comments
1 min read
Final project part 5

Final project part 5

Comments
3 min read
Few new things in Python which I learned last week.

Few new things in Python which I learned last week.

2
Comments 1
2 min read
"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

1
Comments
2 min read
"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

1
Comments
2 min read
"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

1
Comments
1 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part II

Data Ingestion in Snowflake with Google Cloud Storage - Part II

2
Comments
5 min read
Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Scheduling a BigQuery SQL script, using Apache Airflow, with an example

7
Comments 1
2 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part I

Data Ingestion in Snowflake with Google Cloud Storage - Part I

4
Comments
5 min read
Final project part 6

Final project part 6

Comments
3 min read
What is Kafka Connect?

What is Kafka Connect?

1
Comments
4 min read
Facilitating Real-Time Competitive Analysis

Facilitating Real-Time Competitive Analysis

Comments
3 min read
Learning Spark 2.0 Knowledge Dump

Learning Spark 2.0 Knowledge Dump

Comments
3 min read
ETL VS ELT (Data Pipeline)

ETL VS ELT (Data Pipeline)

Comments 1
1 min read
"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

1
Comments
2 min read
"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

1
Comments
1 min read
How to Transpose Columns in Each Group to a Single Row

How to Transpose Columns in Each Group to a Single Row

7
Comments
2 min read
"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

1
Comments
2 min read
"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

1
Comments
1 min read
The Importance of Data in Decision Making

The Importance of Data in Decision Making

4
Comments
2 min read
Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

1
Comments 1
4 min read
AI and Data Sets – Maximizing the Power of Data

AI and Data Sets – Maximizing the Power of Data

1
Comments
3 min read
"Day 43 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

"Day 43 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

1
Comments
2 min read
"Day 50 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Per & Comb- 5)

"Day 50 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Per & Comb- 5)

2
Comments
2 min read
Data Engineer Academy Review

Data Engineer Academy Review

4
Comments
2 min read
Top 10 Common Data Engineers and Scientists Pain Points in 2024

Top 10 Common Data Engineers and Scientists Pain Points in 2024

9
Comments
5 min read
"Day 59 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 5)

"Day 59 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 5)

1
Comments
1 min read
"Day 42 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -21)

"Day 42 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -21)

1
Comments
1 min read
Getting Started with SQL: An Overview of Its Role in Data Fields

Getting Started with SQL: An Overview of Its Role in Data Fields

Comments
3 min read
MWAA Plugins and Dependency Survival Guide

MWAA Plugins and Dependency Survival Guide

5
Comments
3 min read
Real Talk: My DE Academy Experience

Real Talk: My DE Academy Experience

Comments
2 min read
Column Transformation in Machine Learning

Column Transformation in Machine Learning

1
Comments
3 min read
RisingWave workshop

RisingWave workshop

1
Comments
5 min read
Visualization in dbt

Visualization in dbt

1
Comments
3 min read
Xavier's Insight: Overcoming Data Hoarding Disorder

Xavier's Insight: Overcoming Data Hoarding Disorder

5
Comments
3 min read
End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)

End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)

12
Comments
11 min read
The Pains of Data Ingestion

The Pains of Data Ingestion

16
Comments 3
6 min read
"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

1
Comments 1
2 min read
loading...