DEV Community

# bigdata

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
3 Data Observability Tools

3 Data Observability Tools

Comments
3 min read
Spark AI - Bringing Chat GPT to Data Engineering

Spark AI - Bringing Chat GPT to Data Engineering

12
Comments 1
5 min read
Why Are There So Many Snapshot Tables in BI Systems?

Why Are There So Many Snapshot Tables in BI Systems?

5
Comments
9 min read
Why does wide table prevail?

Why does wide table prevail?

5
Comments
13 min read
Open-source SPL: The Breaker of Closed Database Computing System

Open-source SPL: The Breaker of Closed Database Computing System

Comments 1
8 min read
Bulk load to Elastic Search with PySpark

Bulk load to Elastic Search with PySpark

6
Comments
2 min read
Routable computing engine implements front-end database

Routable computing engine implements front-end database

Comments
5 min read
How does the in-memory database bring memory’s advantage into play?

How does the in-memory database bring memory’s advantage into play?

Comments
12 min read
How to clone tables in BigQuery

How to clone tables in BigQuery

2
Comments
1 min read
Why ETL Becomes ELT or Even LET?

Why ETL Becomes ELT or Even LET?

Comments
8 min read
Processing EventHub Captured Messages in Avro Files Using Databricks

Processing EventHub Captured Messages in Avro Files Using Databricks

2
Comments
2 min read
HTAP: Learning from Xiaohongshu

HTAP: Learning from Xiaohongshu

1
Comments
5 min read
HTAP database cannot handle HTAP requirements

HTAP database cannot handle HTAP requirements

Comments
13 min read
Integrating Apache Age with Other Big Data Tools and Frameworks

Integrating Apache Age with Other Big Data Tools and Frameworks

2
Comments 1
2 min read
The current Lakehouse is like a false proposition

The current Lakehouse is like a false proposition

Comments
11 min read
How to make the columnar storage data warehouse more efficient

How to make the columnar storage data warehouse more efficient

Comments
11 min read
Simplest pyspark tutorial

Simplest pyspark tutorial

2
Comments
7 min read
Making Debezium 2.x Support Confluent Schema Registry

Making Debezium 2.x Support Confluent Schema Registry

3
Comments 3
3 min read
Performance Enhancement: Conversion Funnel Analysis

Performance Enhancement: Conversion Funnel Analysis

Comments
9 min read
Boost Your Testing Strategy: The Coolest Methods to Prioritize A/B Tests Like a Pro! 🎲📊😎

Boost Your Testing Strategy: The Coolest Methods to Prioritize A/B Tests Like a Pro! 🎲📊😎

3
Comments
4 min read
A Comprehensive Comparison of JuiceFS and HDFS for Cloud-Based Big Data Storage

A Comprehensive Comparison of JuiceFS and HDFS for Cloud-Based Big Data Storage

2
Comments
11 min read
How to use docker to compile Apache Doris

How to use docker to compile Apache Doris

2
Comments
3 min read
The Secret to Rapid Scaling: How Scraping Helped These Startups Go From Zero to $1.2+ Trillion

The Secret to Rapid Scaling: How Scraping Helped These Startups Go From Zero to $1.2+ Trillion

7
Comments 1
6 min read
Mastering Large-Scale Data Processing: Building a Data Pipeline with ApacheAGE for Efficient Ingestion, Processing, and Analysis

Mastering Large-Scale Data Processing: Building a Data Pipeline with ApacheAGE for Efficient Ingestion, Processing, and Analysis

2
Comments
2 min read
How we mastered dbt: A true story

How we mastered dbt: A true story

7
Comments
14 min read
Exploration of Spark Executor Memory

Exploration of Spark Executor Memory

2
Comments
9 min read
GETTING STARTED WITH SENTIMENT ANALYSIS.

GETTING STARTED WITH SENTIMENT ANALYSIS.

2
Comments
4 min read
Lightweight HTTP API for Big Data on S3

Lightweight HTTP API for Big Data on S3

3
Comments
3 min read
How to cope with high-concurrency account query?

How to cope with high-concurrency account query?

Comments
6 min read
Don't Break the Bank on SQL Queries: BigQuery On-Demand vs Flat-Rate prices. Which Saves You More? 💰😎

Don't Break the Bank on SQL Queries: BigQuery On-Demand vs Flat-Rate prices. Which Saves You More? 💰😎

5
Comments 3
5 min read
Read before-The Ultimate Guide to AWS IoT Core: What it is, How it helps, and Real-World use Cases. Mini-Project-Intro

Read before-The Ultimate Guide to AWS IoT Core: What it is, How it helps, and Real-World use Cases. Mini-Project-Intro

9
Comments
3 min read
"Features of Data Lake Federated Analysis"_Apache Doris Summit 2022 31:03

"Features of Data Lake Federated Analysis"_Apache Doris Summit 2022

2
Comments
1 min read
Tencent Data Engineer: Why We Go from ClickHouse to Apache Doris?

Tencent Data Engineer: Why We Go from ClickHouse to Apache Doris?

1
Comments
11 min read
ClickHouse is fast, esProc SPL is faster

ClickHouse is fast, esProc SPL is faster

1
Comments
10 min read
EXPLORATORY DATA ANALYSIS ULTIMATE GUIDE.

EXPLORATORY DATA ANALYSIS ULTIMATE GUIDE.

1
Comments
3 min read
Apache Doris be common problem positioning and processing

Apache Doris be common problem positioning and processing

2
Comments
3 min read
Importando Funções Python do Repos para o Notebook do Databricks

Importando Funções Python do Repos para o Notebook do Databricks

Comments
3 min read
How To Deal With a Database With Billions of Records

How To Deal With a Database With Billions of Records

2
Comments
6 min read
Amazon Redshift: What, Why, and How

Amazon Redshift: What, Why, and How

2
Comments 1
5 min read
Hadoop/Spark is too heavy, esProc SPL is light

Hadoop/Spark is too heavy, esProc SPL is light

Comments
12 min read
What Is Deep Learning? Deep Learning Algorithms Take Center Stage

What Is Deep Learning? Deep Learning Algorithms Take Center Stage

1
Comments
4 min read
How working/install Pig with Notebooks?

How working/install Pig with Notebooks?

3
Comments
4 min read
Why we use Terraform for BigQuery

Why we use Terraform for BigQuery

11
Comments
6 min read
#011 Databricks explained for busy engineers | Databricks quick start | Databricks Data Security

#011 Databricks explained for busy engineers | Databricks quick start | Databricks Data Security

2
Comments
2 min read
Apache Kafka — The Big Data Messaging tool

Apache Kafka — The Big Data Messaging tool

12
Comments 1
10 min read
DataWarehouse and BigQuery

DataWarehouse and BigQuery

1
Comments
4 min read
How working/install Spark with Notebooks?

How working/install Spark with Notebooks?

3
Comments
3 min read
Nesting Columns like a Pro: A Guide to Mastering Nested Structs in PySpark

Nesting Columns like a Pro: A Guide to Mastering Nested Structs in PySpark

3
Comments
4 min read
Type of data in hadoop

Type of data in hadoop

2
Comments
2 min read
The impasse of SQL performance optimizing

The impasse of SQL performance optimizing

1
Comments
9 min read
Data Pipeline: From ETL to EL plus T

Data Pipeline: From ETL to EL plus T

Comments
4 min read
Design considerations for large data import

Design considerations for large data import

1
Comments
3 min read
Playing Window Function in Postgres

Playing Window Function in Postgres

Comments
4 min read
Read Hierarchical Data Format file

Read Hierarchical Data Format file

Comments
1 min read
Explaining Pagination in ElasticSearch

Explaining Pagination in ElasticSearch

7
Comments
5 min read
Java serialization with Avro

Java serialization with Avro

6
Comments
10 min read
Real Time Data Infra Stack

Real Time Data Infra Stack

4
Comments
6 min read
Example of applying CDC to JSON files with PySpark

Example of applying CDC to JSON files with PySpark

4
Comments 1
7 min read
To study Apache Kafka Architecture in details, and how to install, deploy configure Apache kafka.

To study Apache Kafka Architecture in details, and how to install, deploy configure Apache kafka.

4
Comments
3 min read
How to create Stored Procedure in MySQL

How to create Stored Procedure in MySQL

2
Comments
1 min read
loading...