DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Building a Real-Time Healthcare Data Pipeline with Apache Spark: From SQS to Parquet (Part 2)

Building a Real-Time Healthcare Data Pipeline with Apache Spark: From SQS to Parquet (Part 2)

Comments
8 min read
Virtual Private Database (VPD) | DBMS_RLS | fine-grained access control (FGAC) | mrcaption49

Virtual Private Database (VPD) | DBMS_RLS | fine-grained access control (FGAC) | mrcaption49

5
Comments
5 min read
Kafka Internal Architecture and Mechanisms

Kafka Internal Architecture and Mechanisms

Comments
14 min read
DBMS_SCHEDULER with Practical example | mrcaption49

DBMS_SCHEDULER with Practical example | mrcaption49

5
Comments
4 min read
🖼️ PixelSink: Hunt Hidden Data Inside Images

🖼️ PixelSink: Hunt Hidden Data Inside Images

1
Comments
1 min read
SQL Server 2025 - What’s New and How to Visualize the Schema

SQL Server 2025 - What’s New and How to Visualize the Schema

14
Comments 1
7 min read
Apache Iceberg Table Optimization #9: Managing Large-Scale Optimizations — Parallelism, Checkpointing, and Fail Recovery

Apache Iceberg Table Optimization #9: Managing Large-Scale Optimizations — Parallelism, Checkpointing, and Fail Recovery

Comments
3 min read
Apache Iceberg Table Optimization #8: Hidden Pitfalls — Compaction and Partition Evolution in Apache Iceberg

Apache Iceberg Table Optimization #8: Hidden Pitfalls — Compaction and Partition Evolution in Apache Iceberg

Comments
3 min read
Apache Iceberg Table Optimization #10: The Endgame — Building an Autonomous Optimization Pipeline for Apache Iceberg

Apache Iceberg Table Optimization #10: The Endgame — Building an Autonomous Optimization Pipeline for Apache Iceberg

Comments
3 min read
🧪 Virtual Environments for Data Engineers — 2025 Edition

🧪 Virtual Environments for Data Engineers — 2025 Edition

Comments
1 min read
Apache Iceberg Table Optimization #2: The Basics of Compaction — Bin Packing Your Data for Efficiency

Apache Iceberg Table Optimization #2: The Basics of Compaction — Bin Packing Your Data for Efficiency

Comments
3 min read
Big Data Fundamentals: big data tutorial

Big Data Fundamentals: big data tutorial

1
Comments
5 min read
Big Data Fundamentals: big data tutorial

Big Data Fundamentals: big data tutorial

1
Comments
5 min read
Apache Iceberg Table Optimization #5: Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

Apache Iceberg Table Optimization #5: Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

Comments
3 min read
Apache Iceberg Table Optimization #7: Using Iceberg Metadata Tables to Determine When Compaction Is Needed

Apache Iceberg Table Optimization #7: Using Iceberg Metadata Tables to Determine When Compaction Is Needed

Comments
3 min read
Apache Iceberg Table Optimization #4: Smarter Data Layout — Sorting and Clustering Iceberg Tables

Apache Iceberg Table Optimization #4: Smarter Data Layout — Sorting and Clustering Iceberg Tables

1
Comments
3 min read
Big Data Fundamentals: big data tutorial

Big Data Fundamentals: big data tutorial

1
Comments
5 min read
Apache Iceberg Table Optimization #1: The Cost of Neglect — How Apache Iceberg Tables Degrade Without Optimization

Apache Iceberg Table Optimization #1: The Cost of Neglect — How Apache Iceberg Tables Degrade Without Optimization

Comments
3 min read
Apache Iceberg Table Optimization #3: Optimizing Compaction for Streaming Workloads in Apache Iceberg

Apache Iceberg Table Optimization #3: Optimizing Compaction for Streaming Workloads in Apache Iceberg

Comments
3 min read
Database Design Errors to Avoid & How To Fix Them

Database Design Errors to Avoid & How To Fix Them

11
Comments 1
5 min read
Cross-Platform Multi-Channel Attribution in Marketing: Balancing Costs and Results Across Devices

Cross-Platform Multi-Channel Attribution in Marketing: Balancing Costs and Results Across Devices

1
Comments
5 min read
🛒 Real-Life Data Lakehouse Use Case: Revolutionizing Retail Analytics

🛒 Real-Life Data Lakehouse Use Case: Revolutionizing Retail Analytics

1
Comments
2 min read
Data and analytics reimagined with Terraform and DevOps principles

Data and analytics reimagined with Terraform and DevOps principles

Comments
3 min read
Top Trends and Applications in Data Engineering and AI for the Modern Enterprise

Top Trends and Applications in Data Engineering and AI for the Modern Enterprise

Comments 1
5 min read
A Real-Time Earthquake Monitoring Pipeline with Kafka, MySQL, PostgreSQL, and Grafana

A Real-Time Earthquake Monitoring Pipeline with Kafka, MySQL, PostgreSQL, and Grafana

3
Comments
4 min read
loading...