Save Pandas objects to HDF5

What is HDF5?

HDF stands for "Hierarchical Data Format" and it was designed to store enormous amounts of data. Originally was developed at the National Center for Supercomputing Applications and now it's supported by The HDF Group, a non-profit corporation.

Why use HDF5?

• At its core HDF5 is binary file type specification.

• It has the ability to store many datasets, user-defined metadata, optimized I/O, and the ability to query its contents.

• Many programming languages have tools to work with HDF.

• HDF allows datasets to live in a nested tree structure. In effect, HDF5 is a file system within a file. The 'folders' inside this filesystems are called groups, and sometimes nodes or keys (or at least these terms are used indistinctively).

Toolbox

There are at least three Python packages which can handle HDF5 files: h5py, pytables, and pandas.

Also, there are a few tools to visualize them: HDFViewer (Java), HDFCompass (Python) and ViTables (Python). They can be found at the Ubuntu repositories, but often they don't work as expected.

Fortunately, ViTables is available in the conda-forge package channel and works flawlessly.

Example #1: Dump a DataFrame

import pandas as pd

# Create an example DataFrame
data = {'A': [1,2,3], 'B': [4,5,6]}
df = pd.DataFrame.from_records(data)

with pd.HDFStore('test.h5', mode='w') as f:
    f.put(key='/new_dataset', df)

Example #2: Write metadata

Maybe one of the most interesting aspects of HDF is the ability to store metadata*.

meta = { 'date': '21/06/2019', 'author': 'epassaro'}

with pd.HDFStore('test.h5', mode='a') as f:
    f.get_storer('/new_dataset').attrs.metadata = meta

* the good old FITS format can do this as well!

Example #3: Write metadata to root ("/")

meta = { 'date': '21/06/2019', 'author': 'epassaro'}

with pd.HDFStore('test.h5', mode='a') as f:
    f.root._v_attrs['author'] = 'epassaro'