Optimizing Parquet I/O in C11: Enhancing Performance Across ISAs and Compilers Beyond Arrow C++

Introduction

In the world of data processing, Parquet has emerged as a cornerstone format for efficient columnar storage. However, the tools to read and write Parquet files often fall short of leveraging modern hardware capabilities, leading to performance bottlenecks. Enter Carquet, a lightweight C11 Parquet reader/writer designed to address these limitations. With a binary size of just 500 KB and no dependency on Arrow, Carquet demonstrates that meticulous SIMD optimization and cross-ISA compatibility can significantly outperform established solutions like Arrow C++. On an M3 processor, Carquet writes compressed files 4.6–5.9x faster and reads them 1.9x faster than Arrow C++, setting a new benchmark for performance.

The core problem Carquet solves is twofold: first, existing Parquet tools often fail to fully exploit modern CPU features, such as SIMD instructions, leading to suboptimal performance; second, they lack compatibility across diverse instruction set architectures (ISAs) and compilers, limiting their usability in heterogeneous computing environments. Carquet tackles these challenges by implementing 30 SIMD kernels across SSE4.2, AVX2, AVX-512, NEON, and SVE, all within a single binary that dynamically selects the optimal kernel at runtime. This approach ensures that Carquet maximizes performance on any supported hardware, from x86 to ARM.

Key Technical Innovations

• SIMD Optimization Across ISAs: Carquet’s performance gains stem from its extensive use of SIMD instructions tailored to each ISA. For example, AVX-512 kernels process 512-bit vectors in a single cycle, significantly reducing the number of instructions required for operations like bit-unpacking and floating-point statistics. However, this optimization is non-trivial; each ISA has unique capabilities and limitations. For instance, NEON on ARM lacks native support for certain operations, requiring custom implementations to achieve parity with x86.
• Compiler Behavior and ISA Quirks: One of the hardest lessons learned was the unpredictability of compiler behavior. For example, MSVC emits FMA/BMI instructions even in code targeting AVX2, which can lead to runtime errors on CPUs lacking these extensions. To mitigate this, Carquet employs a runtime dispatch mechanism that verifies not just the ISA but also the specific instructions supported by the CPU. Additionally, differences in ISA behavior, such as how x86 and ARM handle min(x, NaN), required a NaN contract to ensure consistent behavior across platforms.
• Bit-Unpacking and Buffer Safety: Bit-unpacking, a critical operation in Parquet decoding, demands precise handling to avoid buffer overruns. Carquet implements 32 distinct shuffle operations, one for each bit width, and rigorously tests each against a scalar reference to ensure correctness. This approach prevents undefined behavior, such as reading past the buffer, which could corrupt data or crash the application.
• Compiler Auto-Vectorization vs. Handcrafted SIMD: Interestingly, Carquet’s development revealed that compilers are not always inferior to handcrafted SIMD code. For simple operations like min/max, the compiler’s auto-vectorized scalar loop outperformed the NEON implementation. This highlights the importance of profiling and benchmarking to identify where manual optimization is truly beneficial.

The Stakes and Timeliness

Without advancements like Carquet, the data processing ecosystem risks stagnation. Larger, less efficient tools fail to fully leverage modern hardware, hindering performance and scalability in data-intensive applications. As data volumes grow exponentially and hardware continues to evolve, the need for lightweight, high-performance tools like Carquet is more critical than ever. By ensuring efficiency, scalability, and accessibility across diverse computing environments, Carquet paves the way for the next generation of data processing solutions.

For those interested in exploring Carquet further, the project is available on GitHub, where you can delve into its implementation details and contribute to its ongoing development.

Technical Approach

Carquet’s design philosophy centers on maximizing SIMD utilization across diverse ISAs while maintaining a lightweight footprint and cross-compiler compatibility. The core challenge was to achieve performance parity or superiority over Arrow C++ without inheriting its complexity or dependencies. Here’s how Carquet accomplishes this:

1. SIMD Optimization Across Multiple ISAs

Carquet implements 30 SIMD kernels targeting SSE4.2, AVX2, AVX-512, NEON, and SVE within a single binary. This approach ensures runtime adaptability to the host CPU’s capabilities. For example:

• AVX-512 kernels process 512-bit vectors in a single cycle, reducing instruction overhead for operations like bit-unpacking. This directly translates to higher throughput by minimizing pipeline stalls and memory latency.
• NEON kernels on ARM leverage parallel lane execution for floating-point statistics, though they required workarounds for missing intrinsics (e.g., custom implementations for operations absent in ARM’s SIMD instruction set).

The runtime dispatch mechanism selects the optimal kernel based on CPUID flags, but this alone wasn’t sufficient. For instance, MSVC’s behavior of emitting FMA/BMI instructions in AVX2-targeted code forced Carquet to explicitly verify instruction support beyond the targeted ISA. Failure to do so would trigger illegal instruction faults on CPUs lacking these extensions.

2. Handling ISA Quirks and Compiler Behavior

Cross-ISA compatibility required addressing fundamental differences in how ISAs handle edge cases:

• NaN Handling in min(x, NaN): x86 and ARM diverge in their treatment of NaNs. Carquet introduced a NaN contract enforced across all ISAs, ensuring consistent behavior. Without this, floating-point statistics would produce platform-dependent results, breaking reproducibility.
• Bit-Unpacking Safety: Bit-unpacking required 32 shuffle operations (one per bit width) to avoid buffer overruns. Each implementation was tested against a scalar reference to prevent undefined behavior. For example, a misaligned read could corrupt adjacent memory, leading to silent data corruption—a risk mitigated by rigorous bounds checking.

3. Compiler Auto-Vectorization vs. Handcrafted SIMD

Profiling revealed that compiler auto-vectorization sometimes outperformed handcrafted SIMD. For instance, GCC’s auto-vectorized scalar loop for min/max matched Carquet’s NEON code. This highlights a critical trade-off:

• When to Use Handcrafted SIMD: If the compiler’s auto-vectorization fails to exploit data locality or instruction-level parallelism (e.g., in complex bit-unpacking), handcrafted SIMD is necessary. However, for simple operations, relying on the compiler avoids maintenance overhead and potential bugs.
• Rule of Thumb: If the operation involves irregular memory access patterns or ISA-specific optimizations (e.g., AVX-512 mask registers), handcrafted SIMD is optimal. Otherwise, benchmark both approaches.

4. Compiler-Specific Pitfalls

Carquet’s worst bug stemmed from MSVC’s miscompilation of a 64-bit bit-width loop on x86, leading to silent file corruption. This occurred because MSVC incorrectly optimized a loop involving 64-bit integer arithmetic, causing out-of-bounds writes. The fix required:

• Disabling Aggressive Optimizations: For critical loops, Carquet now uses pragmas to limit optimization levels, trading some performance for reliability.
• Validation Framework: All outputs are validated against a reference implementation to catch silent corruption early. This adds overhead during development but is non-negotiable for production use.

5. Performance Gains and Trade-offs

Carquet’s 4.6–5.9x write speedup over Arrow C++ on M3 processors stems from:

• Reduced Instruction Count: SIMD kernels collapse multiple scalar operations into single vector instructions, reducing pipeline stalls.
• Lower Memory Footprint: Carquet’s 500 KB size minimizes cache eviction compared to Arrow’s larger binary, improving cache residency.

However, this comes with trade-offs:

• Maintenance Overhead: Supporting five ISAs requires continuous testing across diverse hardware. For example, SVE’s scalable vector length demands dynamic code generation, adding complexity.
• Breaking Point: Carquet’s approach fails if a future ISA introduces non-deterministic behavior (e.g., speculative execution vulnerabilities) that cannot be abstracted away via runtime dispatch.

Conclusion

Carquet’s success lies in its granular optimization and pragmatic trade-offs. By prioritizing SIMD utilization, addressing ISA quirks, and navigating compiler behavior, it achieves superior performance without sacrificing portability. Developers should adopt Carquet when X (high-throughput Parquet I/O on modern CPUs) is required, but must accept Y (increased maintenance complexity for cross-ISA support). The project’s impact extends beyond Parquet, demonstrating a blueprint for high-performance, lightweight data processing tools in an era of exploding data volumes.

Performance Benchmarks: Carquet vs. Arrow C++

Carquet’s performance gains over Arrow C++ are rooted in its meticulous SIMD optimization and cross-ISA compatibility. Below, we dissect the benchmarks, explaining the causal mechanisms behind its speed and efficiency advantages.

Write Speed: 4.6–5.9x Faster on M3 Processors

Carquet’s write performance is driven by:

• SIMD Utilization: AVX-512 kernels process 512-bit vectors in a single cycle, reducing instruction overhead. This minimizes pipeline stalls and memory latency, directly translating to higher throughput. For example, bit-unpacking operations leverage 32 shuffle operations per bit width, ensuring precise buffer handling and eliminating redundant memory accesses.
• Lightweight Footprint: At 500 KB, Carquet’s binary size reduces cache eviction compared to Arrow’s larger footprint. This improves cache residency, allowing the CPU to sustain higher write speeds without thrashing.
• Compiler-Aware Optimization: Carquet avoids pitfalls like MSVC’s aggressive optimization, which caused silent file corruption in 64-bit bit-width loops. By disabling such optimizations for critical loops, Carquet ensures correctness without sacrificing speed.

Read Speed: 1.9x Faster on M3 Processors

Carquet’s read performance benefits from:

• Runtime Dispatch Mechanism: Dynamically selecting the optimal SIMD kernel (e.g., NEON on ARM, AVX2 on x86) ensures maximal hardware utilization. For instance, NEON kernels use parallel lane execution for floating-point statistics, compensating for missing intrinsics.
• NaN Contract Enforcement: By standardizing NaN handling across ISAs, Carquet avoids inconsistencies like x86 vs. ARM behavior in min(x, NaN). This eliminates edge-case errors that could degrade performance.
• Buffer Safety: Bit-unpacking operations are rigorously tested against scalar references to prevent buffer overruns. This avoids undefined behavior and memory corruption, which could otherwise introduce latency spikes.

Memory Usage: 500 KB vs. Arrow’s Larger Binary

Carquet’s minimal memory footprint is achieved by:

• Modular Design: Implementing only essential Parquet functionality without dependencies like Arrow reduces binary size. This directly lowers memory pressure, improving performance on memory-constrained systems.
• Efficient SIMD Kernels: By consolidating 30 SIMD kernels into a single binary, Carquet avoids code bloat. Each kernel is tailored to its ISA, ensuring no redundant instructions are included.

Scalability Across ISAs

Carquet’s cross-ISA compatibility is enabled by:

• Runtime Instruction Verification: Explicitly checking for supported instructions (e.g., FMA/BMI) prevents illegal instruction faults. This ensures Carquet runs reliably on diverse hardware, from SSE4.2 to SVE.
• ISA-Specific Optimizations: For example, AVX-512 mask registers are used for efficient predicate operations, while NEON kernels workaround missing intrinsics. This tailoring maximizes performance on each architecture.

Trade-offs and Limitations

Carquet’s performance comes with trade-offs:

• Maintenance Complexity: Supporting multiple ISAs increases development and testing overhead. For example, handling ISA quirks like NaN behavior requires a NaN contract, adding complexity to the codebase.
• Compiler Dependency: While Carquet outperforms Arrow, it relies on compiler auto-vectorization for simple operations. If a future compiler version degrades auto-vectorization, performance could suffer.
• Future ISA Compatibility: Carquet’s runtime dispatch assumes deterministic ISA behavior. Non-deterministic behavior in future ISAs could break compatibility, requiring updates.

Professional Judgment

Carquet’s performance dominance over Arrow C++ is a result of its laser focus on SIMD optimization, cross-ISA compatibility, and lightweight design. However, its success hinges on rigorous testing, compiler awareness, and proactive handling of ISA quirks. If your use case prioritizes speed, efficiency, and portability across diverse hardware, Carquet is the optimal choice. Conversely, if you require a feature-rich ecosystem and can tolerate performance trade-offs, Arrow C++ remains a viable alternative.

For developers, the rule is clear: If you need maximal performance with minimal footprint and cross-ISA support, use Carquet. Otherwise, default to Arrow C++ for broader functionality.

Conclusion and Future Work

Carquet has demonstrated that meticulous SIMD optimization and cross-ISA compatibility can revolutionize Parquet I/O performance. By achieving 4.6–5.9x faster write speeds and 1.9x faster read speeds than Arrow C++ on M3 processors, Carquet proves that lightweight, specialized tools can outperform larger, more generalized frameworks. Its 500 KB footprint and runtime dispatch mechanism ensure efficiency and portability across diverse hardware environments, addressing the stagnation in data processing tools that fail to fully leverage modern CPU capabilities.

The project’s success hinges on its technical innovations, particularly in SIMD optimization. By implementing 30 kernels across SSE4.2, AVX2, AVX-512, NEON, and SVE, Carquet maximizes hardware utilization while maintaining a single binary. However, this approach revealed critical challenges, such as compiler quirks (e.g., MSVC emitting FMA/BMI instructions in AVX2 code) and ISA-specific behaviors (e.g., x86 vs. ARM NaN handling). These issues were resolved through rigorous runtime instruction verification, a NaN contract, and precise bit-unpacking shuffle operations, ensuring correctness and performance.

Looking ahead, Carquet’s potential impact on the data processing ecosystem is significant. Its lightweight design and high performance make it an ideal candidate for integration into larger systems, particularly in environments where resource efficiency is critical. Future work should focus on:

• Expanding Features: Adding support for additional Parquet features (e.g., nested data types, advanced compression codecs) to broaden its applicability.
• Integration with Ecosystems: Developing bindings for popular languages (e.g., Python, Rust) to facilitate adoption in diverse workflows.
• Community Involvement: Encouraging contributions to address edge cases, improve testing frameworks, and ensure compatibility with emerging ISAs.
• Benchmarking and Profiling: Continuously profiling against competitors like Arrow C++ to identify areas for further optimization and maintain performance leadership.

However, Carquet’s success is not without trade-offs. Supporting multiple ISAs increases maintenance complexity, and reliance on compiler auto-vectorization introduces risks if future compilers degrade this feature. For example, MSVC’s miscompilation of a 64-bit bit-width loop highlights the need for robust validation frameworks to catch silent data corruption. Practitioners should adopt Carquet when maximal performance and minimal footprint are priorities, but opt for Arrow C++ when broader functionality is required.

In conclusion, Carquet represents a paradigm shift in Parquet I/O optimization, proving that specialized, hardware-aware tools can outperform generalized frameworks. Its achievements underscore the importance of SIMD optimization, cross-ISA compatibility, and compiler awareness in modern data processing. As data volumes and hardware complexity grow, tools like Carquet will be essential to ensure efficiency, scalability, and accessibility in the data processing ecosystem.