DEV Community

mmllllzcn
mmllllzcn

Posted on

TIL: How GBase Database Achieves High Compression in Columnar Storage

#gbase #database #数据库 #todayilearned

Keywords: GBase Database, Columnar Database, Data Compression, Column Store, Dictionary Encoding, MPP Database, OLAP, Database Storage, LZ4, ZSTD

Today I learned that the impressive compression ratios achieved by modern analytical databases aren't the result of a single compression algorithm. Instead, they come from combining multiple storage and encoding techniques that work particularly well for analytical workloads.

Using GBase Database as an example, let's look at why a columnar database can often achieve compression ratios around 1:20–1:30.

1. Columnar Storage Groups Similar Data Together

The biggest advantage comes from storing data by column instead of by row.

In a row-store database, each record contains many different data types:

OrderID | Region | Amount | Date
Enter fullscreen mode Exit fullscreen mode

Because numbers, dates, and strings are mixed together, compression algorithms have fewer repeated patterns to exploit.

A column-store organizes data differently:

OrderID
10248
10249
10250
...

Region
APAC
APAC
EMEA
...

Amount
120
135
98
...
Enter fullscreen mode Exit fullscreen mode

Since each column contains values of the same type, repeated patterns become much easier to compress.

2. Dictionary Encoding Replaces Repeated Values

Many business columns contain only a small number of distinct values.

Examples include:

  • region

  • status

  • country

  • payment_type

Instead of storing the same string repeatedly, the database builds a dictionary:

APAC → 1
EMEA → 2
NA → 3
Enter fullscreen mode Exit fullscreen mode

Each row stores only the integer identifier, dramatically reducing storage requirements.

Dictionary encoding is one of the primary reasons columnar databases achieve much better compression than traditional row-store systems.

3. Sorted Data Makes Compression Even Better

Analytical databases often sort data before applying compression.

Sorting places similar values next to each other:

APAC
APAC
APAC
APAC
EMEA
EMEA
NA
NA
Enter fullscreen mode Exit fullscreen mode

Long runs of similar values are highly compressible.

After sorting, general-purpose compression algorithms such as LZ4 or ZSTD can reduce the remaining storage even further.

Each technique contributes part of the overall compression ratio.

Why Compression Matters

Higher compression provides benefits beyond simply saving disk space.

A smaller dataset means:

  • Less disk I/O

  • Faster table scans

  • Reduced backup size

  • Lower network traffic

  • Improved cache efficiency

For analytical databases, these improvements often translate directly into better query performance.

A Practical Example

GBase Database (GBase 8a MPP Cluster) combines columnar storage, dictionary encoding, and distributed MPP execution for large-scale analytical workloads.

Depending on the characteristics of the data, compression ratios around 1:20–1:30 are achievable.

For example, a dataset occupying 10 TB in its original form may require only a few hundred gigabytes after compression, significantly reducing storage, backup, and data transfer costs. Actual compression ratios will vary based on data distribution, cardinality, and workload characteristics.

TIL Takeaway

Today I learned that high compression ratios are not the result of a single algorithm.

They are achieved by combining:

  • Columnar storage

  • Dictionary encoding

  • Data sorting

  • Efficient compression algorithms such as LZ4 or ZSTD

When evaluating an analytical database like GBase Database (GBase 8a MPP Cluster), don't focus only on the compression ratio itself. Understanding how that compression is achieved provides a much clearer picture of both storage efficiency and query performance.

Top comments (0)