What is Apache arrow and when to use it?

What is Apache arrow(Bookish definition)? 🤔

Apache Arrow defines a language-independent columnar and in memory format, it also supports modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

Is it a database? 💾

No, Apache Arrow is not a database. Instead, it's a cross-language development platform for in-memory data. Arrow's primary role is to act as a "universal data language" allowing different systems and programming languages to work with large datasets more efficiently and quickly.

Then what is Apache arrow...

My use-case for apache arrow arrived, when I wanted to create a data streaming pipeline of a very large dataset with almost zero latency for ML related processing.

• Why not sql databases(PG, mysql etc..): My primary usecase is to have very low streaming latency and the transactional nature and disk-based storage of many SQL databases can introduce significant latency
• Why not a database plus kafka or redis streams: While this setup is good for production applications with distributed system scaling. But for developing a data streaming pipeline which will be used by a processing service for data analysis, setting up kafka and configuring it for required throughput becomes a little complex or overkill. Plus I didn't wanted data to be transferred over network between my compute nodes. So eventually I had to setup kafka on each system, thus giving me no advantage of its distributed architecture.
• Why Apache arrow: Apache arrow is in-memory storage and lazily loads data when iterated to it, making latency very small, and its table format storage also allows me to use simple filters. Plus it can interact with GPU memory as well for data processing. And it doesn't have any serialisation/de-serialisation overheads, as compared to redis which sores data in-memory as bytes and for my usecase it required serialisation/de-serialisation overheads.

Conclusion 👋

In conclusion, Apache Arrow emerges as an ideal solution for specific use cases where low-latency and efficient in-memory data processing are paramount. It's particularly well-suited for scenarios, where the goal is to create a high-performance data streaming pipeline for machine learning and data analysis purposes.