DEV Community

# dataengineering

Posts

šŸ‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

How to write memory efficient machine learning model prediction data pipelines in Python,With an Example

Comments
5 min read
Usando Consultas de PercolaĆ§Ć£o do Elasticsearch, Netflix AperfeiƧoa Buscas Reversas Eficientemente

Usando Consultas de PercolaĆ§Ć£o do Elasticsearch, Netflix AperfeiƧoa Buscas Reversas Eficientemente

1
Comments
3 min read
Carregando dados com Apache HOP & Postgres

Carregando dados com Apache HOP & Postgres

Comments
3 min read
Exploring Data Warehousing and ELT Tools

Exploring Data Warehousing and ELT Tools

Comments
2 min read
Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Scheduling a BigQuery SQL script, using Apache Airflow, with an example

Comments
2 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part II

Data Ingestion in Snowflake with Google Cloud Storage - Part II

Comments
5 min read
Data Ingestion in Snowflake with Google Cloud Storage - Part I

Data Ingestion in Snowflake with Google Cloud Storage - Part I

Comments
5 min read
Few new things in Python which I learned last week.

Few new things in Python which I learned last week.

1
Comments
2 min read
What is Kafka Connect?

What is Kafka Connect?

Comments
4 min read
Difference between Data Analysts, Data Scientists, and Data Engineers

Difference between Data Analysts, Data Scientists, and Data Engineers

Comments 1
1 min read
What is Data Ethics?

What is Data Ethics?

Comments
8 min read
Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

Bulletproof Your Analysis: Data Quality Checklists for Reliable Insights

1
Comments
4 min read
CAP Theorem

CAP Theorem

1
Comments
2 min read
NoSQL DATABASES

NoSQL DATABASES

1
Comments
1 min read
PySparkļ¼š missing value

PySparkļ¼š missing value

Comments
2 min read
"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

"Day 61 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Graph - 1)

1
Comments
1 min read
Data Engineer Academy Review

Data Engineer Academy Review

Comments
2 min read
HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

HOW TO ADD A DATA DISK TO A VIRTUAL MACHINE

Comments
3 min read
"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

"Day 58 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 4)

1
Comments
1 min read
DAG no Airflow para invocar Google Cloud Function

DAG no Airflow para invocar Google Cloud Function

Comments
3 min read
Top 10 Common Data Engineers and Scientists Pain Points in 2024

Top 10 Common Data Engineers and Scientists Pain Points in 2024

9
Comments
5 min read
"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

"Day 60 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 6)

1
Comments
1 min read
Demystifying:Azure Data Factory

Demystifying:Azure Data Factory

Comments
1 min read
Column Transformation in Machine Learning

Column Transformation in Machine Learning

1
Comments
3 min read
Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Data-driven customer acquisition: Machine Learning applied to Customer Lifetime Value

Comments
7 min read
"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

"Day 56 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 2)

1
Comments
1 min read
Final project part 5

Final project part 5

Comments
3 min read
"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

"Day 55 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 1)

1
Comments
2 min read
"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

"Day 54 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 9)

1
Comments
2 min read
"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! šŸš€"

"Completed Weeks 3 and 4 of the AI Engineering Journey!. Ready to tackle the next leg of the journey! šŸš€"

1
Comments
1 min read
Final project part 6

Final project part 6

Comments
3 min read
How to Convert Dates in One Group into An Interval

How to Convert Dates in One Group into An Interval

5
Comments
2 min read
Facilitating Real-Time Competitive Analysis

Facilitating Real-Time Competitive Analysis

Comments
3 min read
Learning Spark 2.0 Knowledge Dump

Learning Spark 2.0 Knowledge Dump

Comments
3 min read
"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

"Day 53 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Perm & Comb - 8)

1
Comments
2 min read
ETL VS ELT (Data Pipeline)

ETL VS ELT (Data Pipeline)

Comments 1
1 min read
Variant in Apache Doris 2.1.0: a new data type 8 times faster than JSON for semi-structured data analysis

Variant in Apache Doris 2.1.0: a new data type 8 times faster than JSON for semi-structured data analysis

Comments
12 min read
"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ā›µļø Today's Focus: Maths for Data Analysis ( Per & Com - 3)

"Day 48 of My Learning Journey: Setting Sail into Data Excellence! ā›µļø Today's Focus: Maths for Data Analysis ( Per & Com - 3)

1
Comments
2 min read
DesentraƱando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

DesentraƱando el Proceso ETL: La Columna Vertebral de la Ciencia de Datos

Comments
2 min read
"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

"Day 47 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( P & C- 2)

1
Comments
1 min read
Data Engineering - Practicing for fun

Data Engineering - Practicing for fun

2
Comments 1
1 min read
ā€œData has a Dreamā€ ā€” A Short comic about data mesh and how it can transform your company

ā€œData has a Dreamā€ ā€” A Short comic about data mesh and how it can transform your company

Comments
2 min read
How to Transpose Columns in Each Group to a Single Row

How to Transpose Columns in Each Group to a Single Row

7
Comments
2 min read
"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

"Day 44 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

1
Comments
2 min read
"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

"Day 45 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -24)

1
Comments
1 min read
The Importance of Data in Decision Making

The Importance of Data in Decision Making

4
Comments
2 min read
Apache Doris 2.1.0: TPC-DS, Parallel Adaptive Scan, Local Shuffle, Arrow Flight-based HTTP Data API

Apache Doris 2.1.0: TPC-DS, Parallel Adaptive Scan, Local Shuffle, Arrow Flight-based HTTP Data API

Comments
29 min read
AI and Data Sets ā€“ Maximizing the Power of Data

AI and Data Sets ā€“ Maximizing the Power of Data

1
Comments
3 min read
"Day 43 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

"Day 43 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -22)

1
Comments
2 min read
"Day 50 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Per & Comb- 5)

"Day 50 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis ( Per & Comb- 5)

2
Comments
2 min read
"Day 59 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 5)

"Day 59 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Maths for Data Analysis (Probability - 5)

1
Comments
1 min read
"Day 42 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -21)

"Day 42 of My Learning Journey: Setting Sail into Data Excellence! Today's Focus: Mathematics for Data Analysis (Stats Day -21)

1
Comments
1 min read
Getting Started with SQL: An Overview of Its Role in Data Fields

Getting Started with SQL: An Overview of Its Role in Data Fields

Comments
3 min read
MWAA Plugins and Dependency Survival Guide

MWAA Plugins and Dependency Survival Guide

2
Comments
3 min read
Real Talk: My DE Academy Experience

Real Talk: My DE Academy Experience

Comments
2 min read
A deep dive into the concept and world of Apache Iceberg Catalogs

A deep dive into the concept and world of Apache Iceberg Catalogs

Comments
8 min read
šŸ“¢ About job offers, innovation & data strategy šŸ”­

šŸ“¢ About job offers, innovation & data strategy šŸ”­

Comments 3
3 min read
RisingWave workshop

RisingWave workshop

1
Comments
5 min read
Visualization in dbt

Visualization in dbt

1
Comments
3 min read
Xavier's Insight: Overcoming Data Hoarding Disorder

Xavier's Insight: Overcoming Data Hoarding Disorder

5
Comments
3 min read
loading...