gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

[AI Collaboration Note] gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

This article documents an optimization strategy adopted during the development of a high-performance database middleware (hypool) to address gRPC transmission efficiency and type limitations. The solution, proposed with the assistance of AI, draws inspiration from low-level database storage principles to resolve two major challenges faced when transmitting large database result sets via traditional gRPC.

---

1. Background and Technical Challenges

When designing APIs for database middleware, we typically need to return multi-row query results. Using traditional gRPC definitions can lead to the following performance and implementation issues.

1.1 Payload Redundancy (Key Repetition)

The most intuitive Protobuf definition is to map each row of data to a Map or Object:

message Row {
    map<string, string> data = 1;
}
message Response {
    repeated Row rows = 1;
}

Problem Analysis:
This structure leads to severe payload bloating. Assuming a query result has 10,000 records containing fields customer_id, created_at, and status, the field names (Keys) are transmitted repeatedly 10,000 times, consuming significant bandwidth resources.

1.2 Protobuf NULL Value Limitations

The design philosophy of Protobuf (proto3) treats scalar types as non-nullable.

• If a string field is NULL, it is serialized as an empty string "".
• The Client cannot distinguish between an "empty value" and a "NULL value in the original database".

Although Wrapper types like google.protobuf.StringValue can solve this, they add extra message nesting levels and processing overhead.

---

2. Solution: Database-Like Low-Level Architecture

Addressing these challenges, the AI suggested breaking away from traditional API object thinking and referencing Database Columnar Storage or ODBC/JDBC driver implementations.

The core optimization strategy consists of two parts:

A. Flattening

Primarily used to solve the Key-Value redundancy problem.

Instead of transmitting an "Array of Objects", we flatten the data structure:

1. Header (Metadata): Transmit field definitions (columns) only once.
2. Body (Values): Flatten all data values into a massive one-dimensional array (values).

This design completely removes Key transmission from each row, significantly reducing payload size.

B. Bitset (Bitmap) Mechanism

Primarily used to solve the NULL value marking problem.

Since string is used to transmit "Values", we introduce an extra binary field (bytes) to specifically record "State":

• Use a Bitset to mark whether each value is NULL.
• 1 bit corresponds to 1 value.
  • Bit = 1: Indicates the value is NULL.
  • Bit = 0: Indicates the value is valid.

Space Efficiency:
Every 8 field values consume only 1 byte of extra space. For 1,000 rows x 8 columns, only about 1 KB of overhead is needed to accurately record all NULL states.

---

3. Implementation Essentials

This section demonstrates key code for "Compression" on the Server side and "Restoration" on the Client side.

3.1 Protobuf Definition (proto/query.proto)

message QueryResponse {
    repeated string values = 3;   // Flattened values
    repeated Column columns = 4;  // Field definitions
    bytes null_bitmap = 7;        // NULL marker bitstream
    int32 row_count = 6;
}

3.2 Server Side: Encoding and Compression

The Server's task is to iterate through database results once, simultaneously completing "Value Flattening" and "Bitmap Generation".

// $result['rows'] is the 2D array returned by the database
$values = [];
$packedBytes = "";
$currentByte = 0;
$bitIndex = 0;

foreach ($result['rows'] as $row) {
    foreach ($row as $value) {
        if ($value === null) {
            $values[] = "";             // Placeholder for empty string
            $currentByte |= (1 << $bitIndex); // Mark as NULL (Set bit to 1)
        } else {
            $values[] = (string)$value; // Store actual value
            // Non-NULL, Bit remains 0
        }

        // Write a byte every 8 bits
        if (++$bitIndex === 8) {
            $packedBytes .= chr($currentByte);
            $currentByte = 0;
            $bitIndex = 0;
        }
    }
}
// Write remaining bits
if ($bitIndex > 0) {
    $packedBytes .= chr($currentByte);
}

3.3 Client Side: Decoding and Restoration

Upon receiving data, the Client needs to slice it according to the number of columns and refer to the null_bitmap to restore NULLs.

$fetchedRows = [];
$columns = $response->getColumns();
$colCount = count($columns);
$values = $response->getValues();     // Get flattened array
$bitmap = $response->getNullBitmap(); // Get Bitmap string

$rowCount = $response->getRowCount();

for ($r = 0; $r < $rowCount; $r++) {
    $row = [];
    for ($c = 0; $c < $colCount; $c++) {
        // Calculate absolute index in the 1D array
        $flatIndex = ($r * $colCount) + $c;

        // [Core] Bitset Lookup
        // 1. Find corresponding Byte position
        $bytePos = (int)($flatIndex / 8); 
        // 2. Find Bit position within the Byte
        $bitPos = $flatIndex % 8;

        // 3. Check if the Bit is 1
        $isNull = (ord($bitmap[$bytePos]) >> $bitPos) & 1;

        if ($isNull) {
            $row[] = null; // Accurately restore NULL
        } else {
            $row[] = $values[$flatIndex];
        }
    }
    $fetchedRows[] = $row;
}

Through the symmetric logic above, we achieve compression and restoration of data with extremely low computational cost.

---

4. Optimization Benefits Analysis

Adopting this architecture yields the following specific benefits:

1. Extreme Transmission Efficiency

   • Through the flattening design, payload size grows linearly with data volume, unaffected by field name length. In large data query scenarios, bandwidth savings are extremely significant.

2. Precise Type Restoration

   • The Client can accurately restore database NULL states by reading the null_bitmap, solving the limitations of gRPC default types.

3. Parsing Performance Improvement

   • For PHP and other languages, processing flat one-dimensional arrays (Indexed Arrays) generally offers better CPU Cache hit rates and lower memory fragmentation compared to processing massive complex nested objects.

5. Conclusion

This optimization case demonstrates the importance of moderately introducing low-level system design thinking in modern distributed systems. Through collaboration with AI, we moved beyond the framework of simple API design, utilizing Bitwise Operations and Data Structure Optimization to solve inherent limitations of gRPC/Protobuf in database application scenarios at a very low cost.