DEV Community

# dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Reading CSVs with varying column counts that pandas cannot read using DuckDB
nk_maker profile

Reading CSVs with varying column counts that pandas cannot read using DuckDB

#dataengineering #duckdb #csv #programming
1
Comments
3 min read
Working with Apache to automate collection of Weather data for Kenya’s major Agricultural Areas
amolo_hillary profile

Working with Apache to automate collection of Weather data for Kenya’s major Agricultural Areas

#dataengineering #dataanalytics #apacheairflow #luxdev
Comments
5 min read
Building a Data Career: The Skills That Truly Matter
datalane profile

Building a Data Career: The Skills That Truly Matter

#datascience #dataengineering #career #sql
10
Comments
5 min read
You Can't Trust COUNT and SUM: Scalable Data Validation with Merkle Trees
andrey_s profile

You Can't Trust COUNT and SUM: Scalable Data Validation with Merkle Trees

#dataengineering #dataquality #datamonitoring #sql
2
Comments 1
8 min read
Unable to emit metadata to DataHub GMS with Airflow - a solution
ivicak profile

Unable to emit metadata to DataHub GMS with Airflow - a solution

#airflow #dataengineering #datahub #data
Comments
4 min read
Snowflake RBAC 101 – Episode 2: Role Hierarchies & Least Privilege
qvfagundes profile

Snowflake RBAC 101 – Episode 2: Role Hierarchies & Least Privilege

#dataengineering #dataprivacy #datawarehouse #snowflake
Comments
1 min read
Lightweight ETL with AWS Glue Python Shell, DuckDB, and PyIceberg
datapenguin profile

Lightweight ETL with AWS Glue Python Shell, DuckDB, and PyIceberg

#aws #iceberg #dataengineering #duckdb
5
Comments 1
7 min read
PyIceberg on AWS Lambda: Comparing GlueCatalog and REST Catalog Access Methods
datapenguin profile

PyIceberg on AWS Lambda: Comparing GlueCatalog and REST Catalog Access Methods

#aws #iceberg #dataengineering
2
Comments
3 min read
The Rise of Real-Time Data: Why Batch Might Be Fading
milcah03 profile

The Rise of Real-Time Data: Why Batch Might Be Fading

#ai #kafka #airflow #dataengineering
10
Comments
3 min read
đź“š A Complete Guide to Data Science Courses: How to Choose, What to Learn, and Where to Begin
virat_kohli_b281b19511fc2 profile

đź“š A Complete Guide to Data Science Courses: How to Choose, What to Learn, and Where to Begin

#datascience #dataengineering #database #machinelearning
Comments
5 min read
Engineering with SOLID, DRY, KISS, YAGNI and GRASP
andrey_s profile

Engineering with SOLID, DRY, KISS, YAGNI and GRASP

#dataengineering #softwareengineering #softwaredevelopment #software
1
Comments
16 min read
Three Formats Walk into a Lakehouse: Iceberg, Delta and Hudi in a Local Setup You Can Run on Your Laptop
olgabraginskaya profile

Three Formats Walk into a Lakehouse: Iceberg, Delta and Hudi in a Local Setup You Can Run on Your Laptop

#dataengineering #data #tutorial #learning
10
Comments 4
16 min read
Which is Best for Real Time Dashboards: Airbyte, Fivetran, or Estuary
techsourabh profile

Which is Best for Real Time Dashboards: Airbyte, Fivetran, or Estuary

#dataengineering #etl #elt
1
Comments
6 min read
Personal Picks: Data Product News (July 9, 2025)
sagara profile

Personal Picks: Data Product News (July 9, 2025)

#dataengineering #lakehouse #snowflake
Comments
6 min read
Key Concepts Every Data Engineer Should Master
georgecodes_ profile

Key Concepts Every Data Engineer Should Master

#datascience #dataengineering #olap #partitioning
4
Comments
6 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.