DEV Community

Florian Bernard
Florian Bernard

Posted on

2

Kotlin DataFrame ❤️ Arrow

Kotlin DataFrame v0.14 comes with improvements for reading Apache Arrow format, especially loading a DataFrame from any ArrowReader.
This improvement can be used to easily load results from analytical databases (such as DuckDB, ClickHouse) directly into Kotlin DataFrame.

Here are two examples of integrations that allow for smooth data import into Kotlin DataFrames using Apache Arrow.

DuckDB

DuckDB is an Analytics database that can be embedded for use in a Kotlin notebook. DuckDB facilitates reading query results as an Arrow Stream, enabling straightforward loading into a Kotlin DataFrame.

Here is a basic notebook which uses DuckDB for querying data from a remote parquet file and then importing the results into a Kotlin dataFrame using Arrow Stream

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

ClickHouse

ClickHouse is a high-performance, column-oriented SQL database management system (DBMS) designed for online analytical processing (OLAP).
ClickHouse allows using Arrow Stream as an output format.

The next notebook uses the ClickHouse client for querying data in Arrow stream format and loads it in a kotlin dataFrame.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

Conclusion

Loading Arrow data into Kotlin DataFrame is both straightforward and efficient, harnessing the full power of Kotlin for data analysis. This integration not only simplifies the process but also enhances performance, making Kotlin DataFrame a powerful tool for handling and analyzing large datasets.

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay