DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Working with Dates and Times in SQL: Tips and Tricks

Working with Dates and Times in SQL: Tips and Tricks

Comments
3 min read
How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

1
Comments
6 min read
Bridging Backend and Data Engineering: Communicating Through Events

Bridging Backend and Data Engineering: Communicating Through Events

Comments
1 min read
Multi-tenant workload isolation in Apache Doris: a better balance between isolation and utilization

Multi-tenant workload isolation in Apache Doris: a better balance between isolation and utilization

Comments
9 min read
Data Mesh: An Executive Guide to Modern Data Architecture in Manufacturing

Data Mesh: An Executive Guide to Modern Data Architecture in Manufacturing

Comments
13 min read
SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

SQL Convertor for Easy Migration from Presto, Trino, ClickHouse, and Hive to Apache Doris

Comments
4 min read
Metadata for win — Apache Parquet

Metadata for win — Apache Parquet

Comments
5 min read
FastAPI for Data Applications: From Concept to Creation. Part I

FastAPI for Data Applications: From Concept to Creation. Part I

2
Comments
5 min read
Learninig: Creating Calculation Views

Learninig: Creating Calculation Views

Comments
2 min read
Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

Overcoming Prometheus's Single-Value Data Model Limitations - A New Approach by GreptimeDB

Comments
5 min read
Usando Consultas de Percolação do Elasticsearch, Netflix Aperfeiçoa Buscas Reversas Eficientemente

Usando Consultas de Percolação do Elasticsearch, Netflix Aperfeiçoa Buscas Reversas Eficientemente

1
Comments
3 min read
How to setup resources for k8s pod

How to setup resources for k8s pod

2
Comments
3 min read
Difference between Data Analysts, Data Scientists, and Data Engineers

Difference between Data Analysts, Data Scientists, and Data Engineers

Comments 1
1 min read
Cross-cluster replication for read-write separation

Cross-cluster replication for read-write separation

3
Comments
4 min read
What is Data Ethics?

What is Data Ethics?

Comments
8 min read
Converting .shp files to CSV with GeoPandas

Converting .shp files to CSV with GeoPandas

1
Comments
2 min read
Apache Iceberg and Data Lakehouse Partitioning

Apache Iceberg and Data Lakehouse Partitioning

3
Comments 1
7 min read
Data warehouse vs data lake

Data warehouse vs data lake

1
Comments
8 min read
Python Projects with SQL: Strategies for Effective Query Management

Python Projects with SQL: Strategies for Effective Query Management

13
Comments 2
9 min read
CAP Theorem

CAP Theorem

1
Comments
2 min read
NoSQL DATABASES

NoSQL DATABASES

1
Comments
1 min read
Apache Spark 101

Apache Spark 101

3
Comments
7 min read
Learning: Nodes

Learning: Nodes

Comments
1 min read
PySpark: missing value

PySpark: missing value

Comments
2 min read
"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

1
Comments
1 min read
Trino & Iceberg Made Easy: A Ready-to-Use Playground

Trino & Iceberg Made Easy: A Ready-to-Use Playground

1
Comments
3 min read
Data Engineer Academy Review

Data Engineer Academy Review

Comments
2 min read
HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

Comments
3 min read
3 Reasons Data Engineers Should Embrace Apache Iceberg

3 Reasons Data Engineers Should Embrace Apache Iceberg

2
Comments
4 min read
Arrow Flight SQL in Apache Doris for 10X faster data transfer

Arrow Flight SQL in Apache Doris for 10X faster data transfer

4
Comments
10 min read
"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

1
Comments
1 min read
DAG no Airflow para invocar Google Cloud Function

DAG no Airflow para invocar Google Cloud Function

Comments
3 min read
Top 10 Common Data Engineers and Scientists Pain Points in 2024

Top 10 Common Data Engineers and Scientists Pain Points in 2024

9
Comments
5 min read
"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

1
Comments
1 min read
How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

Comments 1
5 min read
Carregando dados com Apache HOP & Postgres

Carregando dados com Apache HOP & Postgres

Comments
3 min read
Demystifying:Azure Data Factory

Demystifying:Azure Data Factory

Comments
1 min read
Exploring Data Warehousing and ELT Tools

Exploring Data Warehousing and ELT Tools

1
Comments 1
2 min read
Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Comments
7 min read
"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

1
Comments
1 min read
Final project part 5

Final project part 5

Comments
3 min read
Few new things in Python which I learned last week.

Few new things in Python which I learned last week.

2
Comments 1
2 min read
"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

1
Comments
2 min read
"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

1
Comments
2 min read
"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! 🚀"

1
Comments
1 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part I

Data Ingestion in Snowflake with Google Cloud Storage - Part I

1
Comments
5 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part II

Data Ingestion in Snowflake with Google Cloud Storage - Part II

2
Comments
5 min read
Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Comments 1
2 min read
Final project part 6

Final project part 6

Comments
3 min read
What is Kafka Connect?

What is Kafka Connect?

1
Comments
4 min read
How to Convert Dates in One Group into An Interval

How to Convert Dates in One Group into An Interval

5
Comments
2 min read
Facilitating Real-Time Competitive Analysis

Facilitating Real-Time Competitive Analysis

Comments
3 min read
Learning Spark 2.0 Knowledge Dump

Learning Spark 2.0 Knowledge Dump

Comments
3 min read
"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

1
Comments
2 min read
Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

1
Comments 1
4 min read
ETL VS ELT (Data Pipeline)

ETL VS ELT (Data Pipeline)

Comments 1
1 min read
"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ⛵️ Today's Focus: Maths for Data Analysis ( Per & Com - 3)

1
Comments
2 min read
Desentrañando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

Desentrañando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

Comments
2 min read
"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

1
Comments
1 min read
Data Engineering - Practicing for fun

Data Engineering - Practicing for fun

2
Comments 1
1 min read
loading...