DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Converting .shp files to CSV with GeoPandas

Converting .shp files to CSV with GeoPandas

Comments
2 min read
SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

Comments
4 min read
Metadata for win — Apache Parquet

Metadata for win — Apache Parquet

Comments
5 min read
Data warehouse vs data lake

Data warehouse vs data lake

Comments
8 min read
Learninig: Creating Calculation Views

Learninig: Creating Calculation Views

Comments
2 min read
Apache Iceberg and Data Lakehouse Partitioning

Apache Iceberg and Data Lakehouse Partitioning

2
Comments 1
7 min read
3 Reasons Data Engineers Should Embrace Apache Iceberg

3 Reasons Data Engineers Should Embrace Apache Iceberg

1
Comments
4 min read
Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

Comments
5 min read
Usando Consultas de Percolação do Elasticsearch, Netflix Aperfeiçoa Buscas Reversas Eficientemente

Usando Consultas de Percolação do Elasticsearch, Netflix Aperfeiçoa Buscas Reversas Eficientemente

1
Comments
3 min read
Carregando dados com Apache HOP & Postgres

Carregando dados com Apache HOP & Postgres

Comments
3 min read
Apache Spark 101

Apache Spark 101

1
Comments
7 min read
Difference between Data Analysts, Data Scientists, and Data Engineers

Difference between Data Analysts, Data Scientists, and Data Engineers

Comments 1
1 min read
Cross-cluster replication for read-write separation

Cross-cluster replication for read-write separation

3
Comments
4 min read
What is Data Ethics?

What is Data Ethics?

Comments
8 min read
Trino & Iceberg Made Easy: A Ready-to-Use Playground

Trino & Iceberg Made Easy: A Ready-to-Use Playground

Comments
3 min read
Python Projects with SQL: Strategies for Effective Query Management

Python Projects with SQL: Strategies for Effective Query Management

11
Comments 1
9 min read
CAP Theorem

CAP Theorem

1
Comments
2 min read
NoSQL DATABASES

NoSQL DATABASES

1
Comments
1 min read
Learning: Nodes

Learning: Nodes

Comments
1 min read
PySpark: missing value

PySpark: missing value

Comments
2 min read
"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

1
Comments
1 min read
Data Engineer Academy Review

Data Engineer Academy Review

Comments
2 min read
HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

Comments
3 min read
Arrow Flight SQL in Apache Doris for 10X faster data transfer

Arrow Flight SQL in Apache Doris for 10X faster data transfer

4
Comments
10 min read
"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

1
Comments
1 min read
DAG no Airflow para invocar Google Cloud Function

DAG no Airflow para invocar Google Cloud Function

Comments
3 min read
Top 10 Common Data Engineers and Scientists Pain Points in 2024

Top 10 Common Data Engineers and Scientists Pain Points in 2024

9
Comments
5 min read
"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

1
Comments
1 min read
How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

Comments 1
5 min read
Demystifying:Azure Data Factory

Demystifying:Azure Data Factory

Comments
1 min read
Exploring Data Warehousing and ELT Tools

Exploring Data Warehousing and ELT Tools

1
Comments 1
2 min read
Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Comments
7 min read
"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

1
Comments
1 min read
Final project part 5

Final project part 5

Comments
3 min read
Few new things in Python which I learned last week.

Few new things in Python which I learned last week.

2
Comments 1
2 min read
"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

1
Comments
2 min read
"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

1
Comments
2 min read
"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

1
Comments
1 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part II

Data Ingestion in Snowflake with Google Cloud Storage - Part II

1
Comments
5 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part I

Data Ingestion in Snowflake with Google Cloud Storage - Part I

1
Comments
5 min read
Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Comments 1
2 min read
Final project part 6

Final project part 6

Comments
3 min read
What is Kafka Connect?

What is Kafka Connect?

1
Comments
4 min read
How to Convert Dates in One Group into An Interval

How to Convert Dates in One Group into An Interval

5
Comments
2 min read
Facilitating Real-Time Competitive Analysis

Facilitating Real-Time Competitive Analysis

Comments
3 min read
Learning Spark 2.0 Knowledge Dump

Learning Spark 2.0 Knowledge Dump

Comments
3 min read
"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

1
Comments
2 min read
Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

1
Comments 1
4 min read
ETL VS ELT (Data Pipeline)

ETL VS ELT (Data Pipeline)

Comments 1
1 min read
"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

1
Comments
2 min read
Desentrañando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

Desentrañando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

Comments
2 min read
"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

1
Comments
1 min read
Data Engineering - Practicing for fun

Data Engineering - Practicing for fun

2
Comments 1
1 min read
“Data has a Dream” — A Short comic about data mesh and how it can transform your company

“Data has a Dream” — A Short comic about data mesh and how it can transform your company

Comments
2 min read
How to Transpose Columns in Each Group to a Single Row

How to Transpose Columns in Each Group to a Single Row

7
Comments
2 min read
"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

1
Comments
2 min read
"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

1
Comments
1 min read
The Importance of Data in Decision Making

The Importance of Data in Decision Making

4
Comments
2 min read
Apache Doris 2.1.0: TPC-DS, Parallel Adaptive Scan, Local Shuffle, Arrow Flight-based HTTP Data API

Apache Doris 2.1.0: TPC-DS, Parallel Adaptive Scan, Local Shuffle, Arrow Flight-based HTTP Data API

Comments
29 min read
AI and Data Sets – Maximizing the Power of Data

AI and Data Sets – Maximizing the Power of Data

1
Comments
3 min read
loading...