Polars + Delta Lake: Azure Function vs. Laptop on Small Data

Sebastian Walter — Sun, 05 Jan 2025 14:26:13 +0000

Have you ever wondered how the performance of Polars + Deltalake on Azure compares to a consumer grade laptop?
No? Well, I have. If I have sparked your curiosity, read on.

Here are the contenders

EliteBook 840 G10, AMD Ryzen 7840U, 8 cores, 16 threads, 64 GB RAM
Azure Function running on a Linux B3 SKU app service plan (4 cores, 7 GB RAM)
- with standard ADLS2 storage
- with premium ADLS2 storage

See Pricing for a full list of available app service plans.

Test Setup

The test measures three scenarios

create delta table
write to delta table
read from delta table

The code is executed via REST API endpoints:

polars_azure_create: https://function-hekori-learning-002.azurewebsites.net/api/polars/azure/create
polars_azure_read: https://function-hekori-learning-002.azurewebsites.net/api/polars/azure/read
polars_azure_write: https://function-hekori-learning-002.azurewebsites.net/api/polars/azure/write
polars_local_create: http://localhost:7071/api/polars/local/create
polars_local_read: http://localhost:7071/api/polars/local/read
polars_local_write: http://localhost:7071/api/polars/local/write

On the HP EliteBook I used func start to launch https://localhost:7071.
To publish to Azure I, followed the instructions from https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python
to set up the necessary development environment. This allowed me to publish the function via
func azure functionapp publish function-hekori-learning-002.

I used terraform to set up the Azure resources in the North Europe region.

Here is a code snippet showing the code executed when visiting https://function-hekori-learning-002.azurewebsites.net/api/polars/azure/read


@app.route(route="polars/azure/read", auth_level=func.AuthLevel.ANONYMOUS)
def polars_azure_read(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Reading from delta table')

    tic = time.time()
    df = pl.read_delta(AZURE_STORAGE_PATH, storage_options=storage_options
                       )

    df = df.sql(
        "select sum(value) as sum, avg(value) as mean, count() as count, name from self group by name order by sum asc"
    )

    toc = time.time()

    logging.info(f"Elapsed time {toc - tic:.2f} seconds")

    return func.HttpResponse(
        "Success from polars." + str(df) + '\n' + "Elapsed time " + str(toc - tic) + " seconds",
        status_code=200
    )

Test Results

As one can see the HP EliteBook is roughly one order of magnitude faster in all scenarios.

Interpretation

This is my personal interpretation

The Azure Function timings are sufficient for synchronous tasks. E.g., to be used in POST requests where the client expects a response in < 2 seconds.
If you have small data and want the best performance, you should consider running Polars on bare metal or virtual machine with low IO latency.

Please note that the delta table has a small size of 3 commits and 2 parquet files. I.e., the runtime effectively measure the overhead of the file access from the compute unit.

If you ❤️ this article and want to see more benchmark results with larger datasets for out of core processing give this article a 👏
and subscribe 🔔 to my channel 👇👇👇.

DEV Community: Sebastian Walter

Polars + Delta Lake: Azure Function vs. Laptop on Small Data

Test Setup

Test Results

Interpretation