DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
BigQuery Schema Generation Made Easier with PyPI’s bigquery-schema-generator

BigQuery Schema Generation Made Easier with PyPI’s bigquery-schema-generator

5
Comments 2
2 min read
Embrace simple tech stacks and code generation in DevOps and data engineering

Embrace simple tech stacks and code generation in DevOps and data engineering

2
Comments
6 min read
Apache Doris for log and time series data analysis in NetEase, why not Elasticsearch and InfluxDB?

Apache Doris for log and time series data analysis in NetEase, why not Elasticsearch and InfluxDB?

1
Comments
9 min read
MapReduce Vs Tez

MapReduce Vs Tez

6
Comments
2 min read
Azure Synapse Analytics Security: Data Protection

Azure Synapse Analytics Security: Data Protection

2
Comments
6 min read
Leveraging PySpark.Pandas for Efficient Data Pipelines

Leveraging PySpark.Pandas for Efficient Data Pipelines

Comments
3 min read
Why Apache Doris is the Best Open Source Alternative to Rockset

Why Apache Doris is the Best Open Source Alternative to Rockset

3
Comments
3 min read
Apache Spark-Structured Streaming :: Cab Aggregator Use-case

Apache Spark-Structured Streaming :: Cab Aggregator Use-case

1
Comments
4 min read
Introduction to Apache Hadoop & MapReduce

Introduction to Apache Hadoop & MapReduce

5
Comments
3 min read
Analytics don't want duplicated data, so get it exactly-once with Flink/Kafka

Analytics don't want duplicated data, so get it exactly-once with Flink/Kafka

Comments
3 min read
Metadata for win — Apache Parquet

Metadata for win — Apache Parquet

Comments
5 min read
Remove unwanted partition data in Azure Synapse (SQL DW)

Remove unwanted partition data in Azure Synapse (SQL DW)

1
Comments
6 min read
Replacing Saas ETL with Python dlt: A painless experience for Yummy.eu

Replacing Saas ETL with Python dlt: A painless experience for Yummy.eu

2
Comments
3 min read
Simplifying SDMX Data Integration with Python

Simplifying SDMX Data Integration with Python

2
Comments
3 min read
Unlocking the Power of Large Language Models (LLMs): Your Ultimate Guide

Unlocking the Power of Large Language Models (LLMs): Your Ultimate Guide

6
Comments
1 min read
Clustering vs Partitioning your Apache Iceberg Tables

Clustering vs Partitioning your Apache Iceberg Tables

6
Comments
10 min read
From Messy Data to Super Mario Pipeline: My First Adventure in Data Engineering

From Messy Data to Super Mario Pipeline: My First Adventure in Data Engineering

1
Comments
12 min read
The Data Professions

The Data Professions

1
Comments
3 min read
Database generated events: LiveSync’s database connector vs CDC

Database generated events: LiveSync’s database connector vs CDC

4
Comments
5 min read
MySQL: Using and Enhancing `DATETIME` and `TIMESTAMP`

MySQL: Using and Enhancing `DATETIME` and `TIMESTAMP`

1
Comments
3 min read
Analyzing Svenskalag Data using DBT and DuckDB

Analyzing Svenskalag Data using DBT and DuckDB

1
Comments
4 min read
How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

12
Comments
6 min read
Working with Dates and Times in SQL: Tips and Tricks

Working with Dates and Times in SQL: Tips and Tricks

Comments
3 min read
FastAPI for Data Applications: From Concept to Creation. Part I

FastAPI for Data Applications: From Concept to Creation. Part I

3
Comments
5 min read
Bridging Backend and Data Engineering: Communicating Through Events

Bridging Backend and Data Engineering: Communicating Through Events

Comments
1 min read
loading...