DEV Community: Christopher Ariza

Type-Hinting DataFrames for Static Analysis and Runtime Validation

Christopher Ariza — Mon, 13 Nov 2023 18:58:22 +0000

Since the advent of type hints in Python 3.5, statically typing a DataFrame has generally been limited to specifying just the type:

def process(f: DataFrame) -> Series: ...

This is inadequate, as it ignores the types contained within the container. A DataFrame might have string column labels and three columns of integer, string, and floating-point values; these characteristics define the type. A function argument with such type hints provides developers, static analyzers, and runtime checkers with all the information needed to understand the expectations of the interface. StaticFrame 2 now permits this:

from typing import Any
from static_frame import Frame, Index, TSeriesAny

def process(f: Frame[   # type of the container
        Any,            # type of the index labels
        Index[np.str_], # type of the column labels
        np.int_,        # type of the first column
        np.str_,        # type of the second column
        np.float64,     # type of the third column
        ]) -> TSeriesAny: ...

All core StaticFrame containers now support generic specifications. While statically checkable, a new decorator, @CallGuard.check, permits runtime validation of these type hints on function interfaces. Further, using Annotated generics, the new Require class defines a family of powerful runtime validators, permitting per-column or per-row data checks. Finally, each container exposes a new via_type_clinic interface to derive and validate type hints. Together, these tools offer a cohesive approach to type-hinting and validating DataFrames.

Requirements of a Generic DataFrame

Python's built-in generic types (e.g.,tuple or dict) require specification of component types (e.g., tuple[int, str, bool] or dict[str, int]). Defining component types permits more accurate static analysis. While the same is true for DataFrames, there have been few attempts to define comprehensive type hints for DataFrames.

Pandas, even with the pandas-stubs package, does not permit specifying the types of a DataFrame's components. The Pandas DataFrame, permitting extensive in-place mutation, may not be sensible to type statically. Fortunately, immutable DataFrames are available in StaticFrame.

Further, Python's tools for defining generics, until recently, have not been well-suited for DataFrames. That a DataFrame has a variable number of heterogeneous columnar types poses a challenge for generic specification. Typing such a structure became easier with the new TypeVarTuple, introduced in Python 3.11 (and back-ported in the typing_extensions package).

A TypeVarTuple permits defining generics that accept a variable number of types. (See PEP 646 for details.) With this new type variable, StaticFrame can define a generic Frame with a TypeVar for the index, a TypeVar for the columns, and a TypeVarTuple for zero or more columnar types.

A generic Series is defined with a TypeVar for the index and a TypeVar for the values. The StaticFrame Index and IndexHierarchy are also generic, the latter again taking advantage of TypeVarTuple to define a variable number of component Index for each depth level.

StaticFrame uses NumPy types to define the columnar types of a Frame, or the values of a Series or Index. This permits narrowly specifying sized numerical types, such as np.uint8 or np.complex128; or broadly specifying categories of types, such as np.integer or np.inexact. As StaticFrame supports all NumPy types, the correspondence is direct.

Interfaces Defined with Generic DataFrames

Extending the example above, the function interface below shows a Frame with three columns transformed into a dictionary of Series. With so much more information provided by component type hints, the function's purpose is almost obvious.

from typing import Any
from static_frame import Frame, Series, Index, IndexYearMonth

def process(f: Frame[
        Any,
        Index[np.str_],
        np.int_,
        np.str_,
        np.float64,
        ]) -> dict[
                int,
                Series[                 # type of the container
                        IndexYearMonth, # type of the index labels
                        np.float64,     # type of the values
                        ],
                ]: ...

This function processes a signal table from an Open Source Asset Pricing (OSAP) dataset (Firm Level Characteristics / Individual / Predictors). Each table has three columns: security identifier (labeled "permno"), year and month (labeled "yyyymm"), and the signal (with a name specific to the signal).

The function ignores the index of the provided Frame (typed as Any) and creates groups defined by the first column "permno" np.int_ values. A dictionary keyed by "permno" is returned, where each value is a Series of np.float64 values for that "permno"; the index is an IndexYearMonth created from the np.str_ "yyyymm" column. (StaticFrame uses NumPy datetime64 values to define unit-typed indices: IndexYearMonth stores datetime64[M] labels.)

Rather than returning a dict, the function below returns a Series with a hierarchical index. The IndexHierarchy generic specifies a component Index for each depth level; here, the outer depth is an Index[np.int_] (derived from the "permno" column), the inner depth an IndexYearMonth (derived from the "yyyymm" column).

from typing import Any
from static_frame import Frame, Series, Index, IndexYearMonth, IndexHierarchy

def process(f: Frame[
        Any,
        Index[np.str_],
        np.int_,
        np.str_,
        np.float64,
        ]) -> Series[                    # type of the container
                IndexHierarchy[          # type of the index labels
                        Index[np.int_],  # type of index depth 0
                        IndexYearMonth], # type of index depth 1
                np.float64,              # type of the values
                ]: ...

Rich type hints provide a self-documenting interface that makes functionality explicit. Even better, these type hints can be used for static analysis with Pyright (now) and MyPy (pending full TypeVarTuple support). For example, calling this function with a Frame of two columns of np.float64 will fail a static analysis type check or deliver a warning in an editor.

Runtime Type Validation

Static type checking may not be enough: runtime evaluation provides even stronger constraints, particularly for dynamic or incompletely (or incorrectly) type-hinted values.

Building on a new runtime type checker named TypeClinic, StaticFrame 2 introduces @CallGuard.check, a decorator for runtime validation of type-hinted interfaces. All StaticFrame and NumPy generics are supported, and most built-in Python types are supported, even when deeply nested. The function below adds the @CallGuard.check decorator.

from typing import Any
from static_frame import Frame, Series, Index, IndexYearMonth, IndexHierarchy, CallGuard

@CallGuard.check
def process(f: Frame[
        Any,
        Index[np.str_],
        np.int_,
        np.str_,
        np.float64,
        ]) -> Series[
                IndexHierarchy[Index[np.int_], IndexYearMonth],
                np.float64,
                ]: ...

Now decorated with @CallGuard.check, if the function above is called with an unlabelled Frame of two columns of np.float64, a ClinicError exception will be raised, illustrating that, where three columns were expected, two were provided, and where string column labels were expected, integer labels were provided. (To issue warnings instead of raising exceptions, use the @CallGuard.warn decorator.)

ClinicError:
In args of (f: Frame[Any, Index[str_], int64, str_, float64]) -> Series[IndexHierarchy[Index[int64], IndexYearMonth], float64]
└── Frame[Any, Index[str_], int64, str_, float64]
    └── Expected Frame has 3 dtype, provided Frame has 2 dtype
In args of (f: Frame[Any, Index[str_], int64, str_, float64]) -> Series[IndexHierarchy[Index[int64], IndexYearMonth], float64]
└── Frame[Any, Index[str_], int64, str_, float64]
    └── Index[str_]
        └── Expected str_, provided int64 invalid

Runtime Data Validation

Other characteristics can be validated at runtime. For example, the shape or name attributes, or the sequence of labels on the index or columns. The StaticFrame Require class provides a family of configurable validators.

Require.Name: Validate the name attribute of the container.
Require.Len: Validate the length of the container.
Require.Shape: Validate the shape attribute of the container.
Require.LabelsOrder: Validate the ordering of the labels.
Require.LabelsMatch: Validate inclusion of labels independent of order.
Require.Apply: Apply a Boolean-returning function to the container.

Aligning with a growing trend, these objects are provided within type hints as one or more additional arguments to an Annotated generic. (See PEP 593 for details.) The type referenced by the first Annotated argument is the target of subsequent-argument validators. For example, if a Index[np.str_] type hint is replaced with an Annotated[Index[np.str_], Require.Len(20)] type hint, the runtime length validation is applied to the index associated with the first argument.

Extending the example of processing an OSAP signal table, we might validate our expectation of column labels. The Require.LabelsOrder validator can define a sequence of labels, optionally using ... for contiguous regions of zero or more unspecified labels. To specify that the first two columns of the table are labeled "permno" and "yyyymm", while the third label is variable (depending on the signal), the following Require.LabelsOrder can be defined within an Annotated generic:

from typing import Any, Annotated
from static_frame import Frame, Series, Index, IndexYearMonth, IndexHierarchy, CallGuard, Require

@CallGuard.check
def process(f: Frame[
        Any,
        Annotated[
                Index[np.str_],
                Require.LabelsOrder('permno', 'yyyymm', ...),
                ],
        np.int_,
        np.str_,
        np.float64,
        ]) -> Series[
                IndexHierarchy[Index[np.int_], IndexYearMonth],
                np.float64,
                ]: ...

If the interface expects a small collection of OSAP signal tables, we can validate the third column with the Require.LabelsMatch validator. This validator can specify required labels, sets of labels (from which at least one must match), and regular expression patterns. If tables from only three files are expected (i.e., "Mom12m.csv", "Mom6m.csv", and "LRreversal.csv"), we can validate the labels of the third column by defining Require.LabelsMatch with a set:

@CallGuard.check
def process(f: Frame[
        Any,
        Annotated[
                Index[np.str_],
                Require.LabelsOrder('permno', 'yyyymm', ...),
                Require.LabelsMatch({'Mom12m', 'Mom6m', 'LRreversal'}),
                ],
        np.int_,
        np.str_,
        np.float64,
        ]) -> Series[
                IndexHierarchy[Index[np.int_], IndexYearMonth],
                np.float64,
                ]: ...

Both Require.LabelsOrder and Require.LabelsMatch can associate functions with label specifiers to validate data values. If the validator is applied to column labels, a Series of column values will be provided to the function; if the validator is applied to index labels, a Series of row values will be provided to the function.

Similar to the usage of Annotated, the label specifier is replaced with a list, where the first item is the label specifier, and the remaining items are row- or column-processing functions that return a Boolean.

To extend the example above, we might validate that all "permno" values are greater than zero and that all signal values ("Mom12m", "Mom6m", "LRreversal") are greater than or equal to -1.

from typing import Any, Annotated
from static_frame import Frame, Series, Index, IndexYearMonth, IndexHierarchy, CallGuard, Require

@CallGuard.check
def process(f: Frame[
        Any,
        Annotated[
                Index[np.str_],
                Require.LabelsOrder(
                        ['permno', lambda s: (s > 0).all()],
                        'yyyymm',
                        ...,
                        ),
                Require.LabelsMatch(
                        [{'Mom12m', 'Mom6m', 'LRreversal'}, lambda s: (s >= -1).all()],
                        ),
                ],
        np.int_,
        np.str_,
        np.float64,
        ]) -> Series[
                IndexHierarchy[Index[np.int_], IndexYearMonth],
                np.float64,
                ]: ...

If validation fails, @CallGuard.check will raise an exception. For example, if the above function is called with a Frame that has an unexpected third-column label, the following exception will be raised:

ClinicError:
In args of (f: Frame[Any, Annotated[Index[str_], LabelsOrder(['permno', <lambda>], 'yyyymm', ...), LabelsMatch([{'Mom12m', 'LRreversal', 'Mom6m'}, <lambda>])], int64, str_, float64]) -> Series[IndexHierarchy[Index[int64], IndexYearMonth], float64]
└── Frame[Any, Annotated[Index[str_], LabelsOrder(['permno', <lambda>], 'yyyymm', ...), LabelsMatch([{'Mom12m', 'LRreversal', 'Mom6m'}, <lambda>])], int64, str_, float64]
    └── Annotated[Index[str_], LabelsOrder(['permno', <lambda>], 'yyyymm', ...), LabelsMatch([{'Mom12m', 'LRreversal', 'Mom6m'}, <lambda>])]
        └── LabelsMatch([{'Mom12m', 'LRreversal', 'Mom6m'}, <lambda>])
            └── Expected label to match frozenset({'Mom12m', 'LRreversal', 'Mom6m'}), no provided match

The Expressive Power of `TypeVarTuple`

As shown above, TypeVarTuple permits specifying Frame with zero or more heterogeneous columnar types. For example, we can provide type hints for a Frame of two float or six mixed types:

>>> from typing import Any
>>> from static_frame import Frame, Index

>>> f1: sf.Frame[Any, Any, np.float64, np.float64]
>>> f2: sf.Frame[Any, Any, np.bool_, np.float64, np.int8, np.int8, np.str_, np.datetime64]

While this accommodates diverse DataFrames, type-hinting wide DataFrames, such as those with hundreds of columns, would be unwieldy. Python 3.11 introduces a new syntax to provide a variable range of types in TypeVarTuple generics: star expressions of tuple generic aliases. For example, to type-hint a Frame with a date index, string column labels, and any configuration of columnar types, we can star-unpack a tuple of zero or more All:

>>> from typing import Any
>>> from static_frame import Frame, Index

>>> f: sf.Frame[Index[np.datetime64], Index[np.str_], *tuple[All, ...]]

The tuple star expression can go anywhere in a list of types, but there can be only one. For example, the type hint below defines a Frame that must start with Boolean and string columns but has a flexible specification for any number of subsequent np.float64 columns.

>>> from typing import Any
>>> from static_frame import Frame

>>> f: sf.Frame[Any, Any, np.bool_, np.str_, *tuple[np.float64, ...]]

Utilities for Type Hinting

Working with such detailed type hints can be challenging. To aid users, StaticFrame provides convenient utilities for runtime type hinting and checking. All StaticFrame 2 containers now feature a via_type_clinic interface, permitting access to TypeClinic functionality.

First, utilities are provided to translate a container, such as a complete Frame, into a type hint. The string representation of the via_type_clinic interface provides a string representation of the container's type hint; alternatively, the to_hint() method returns a complete generic alias object.

>>> import static_frame as sf
>>> f = sf.Frame.from_records(([3, '192004', 0.3], [3, '192005', -0.4]), columns=('permno', 'yyyymm', 'Mom3m'))

>>> f.via_type_clinic
Frame[Index[int64], Index[str_], int64, str_, float64]

>>> f.via_type_clinic.to_hint()
static_frame.core.frame.Frame[static_frame.core.index.Index[numpy.int64], static_frame.core.index.Index[numpy.str_], numpy.int64, numpy.str_, numpy.float64]

Second, utilities are provided for runtime-type-hint testing. The via_type_clinic.check() function permits validating the container against a provided type hint.

>>> f.via_type_clinic.check(sf.Frame[sf.Index[np.str_], sf.TIndexAny, *tuple[tp.Any, ...]])
ClinicError:
In Frame[Index[str_], Index[Any], Unpack[Tuple[Any, ...]]]
└── Index[str_]
    └── Expected str_, provided int64 invalid

To support gradual typing, StaticFrame defines several generic aliases configured with Any for every component type. For example, TFrameAny can be used for any Frame, and TSeriesAny for any Series. As expected, TFrameAny will validate the Frame created above.

>>> f.via_type_clinic.check(sf.TFrameAny)

Conclusion

Better type hinting for DataFrames is overdue. With modern Python typing tools and a DataFrame built on an immutable data model, StaticFrame 2 meets this need, providing powerful resources for engineers prioritizing maintainability and verifiability.

Memoizing DataFrame Functions: Using Hashable DataFrames and Message Digests to Optimize Repeated Calculations

Christopher Ariza — Wed, 01 Mar 2023 02:28:45 +0000

A well-known technique for improving the run-time performance of a software system (in Python or any language) is function memoization. Memoization is a type of caching applied to a single function. If a function is called multiple times with the same arguments, repeating the calculation can be avoided by storing the results in a mapping (or on disk), keyed by the arguments. Upon subsequent calls, if the arguments are found, the stored result is returned.

This opportunity comes with tradeoffs. Memoization reduces time at the cost of space: previously calculated results must be stored either in memory or on disk. Additionally, the function memoized must be pure: the output must be determined exclusively by its inputs. Finally, not all types of function arguments are suitable. With in-memory memoization, where results are stored in a mapping, arguments must be hashable and immutable. With disk-based memoization, where results are stored in a file, arguments must be reducible to a unique file name; a message digest derived from a cryptographic hash function is optimal for this purpose.

Another challenge of memoization is cache invalidation: to avoid excessive cache growth, caches must be dropped. The Python standard library provides an in-memory solution with the functools.lru_cache() decorator. This decorator implements memoization with a "least recently used" (LRU) cache invalidation strategy: after reaching a maximum count, caches that have least-recently been used are dropped.

For Python programmers using Pandas DataFrames as function arguments, there are further challenges. As mutable containers, Pandas DataFrame and Series are not hashable. The functools.lru_cache() will fail if an argument is a Pandas DataFrame.

>>> import functools
>>> @functools.lru_cache
... def cube(v):
...     return v ** 3
...
>>> import pandas as pd
>>> df = pd.DataFrame(np.arange(1_000_000).reshape(1000, 1000))
>>> cube(df)
Traceback (most recent call last):
TypeError: unhashable type: 'DataFrame'

StaticFrame is an alternative DataFrame library that offers efficient solutions to this problem, both for in-memory and disk-based memoization.

Hash Functions and Hash Collisions

Before demonstrating DataFrame memoization with StaticFrame, it is important to distinguish different types of hash functions.

A hashing function converts a variable-sized value into a smaller, (generally) fixed-sized value. A hash collision is when different inputs hash to the same result. For some applications, hash collisions are acceptable. Cryptographic hash functions aim to eliminate collisions.

In Python, the built-in hash() function converts hashable objects into an integer. Arbitrary types can provide support by implementing the magic method __hash__(). Importantly, the results of hash() are not collision resistant:

>>> hash('')
0
>>> hash(0)
0
>>> hash(False)
0

Python dictionaries use hash() to transform dictionary keys into storage positions in a low-level C array. Collisions are expected, and if found, are resolved with equality comparisons using __eq__(). Thus, for an arbitrary type to be hashable, it needs to implement both __hash__() and __eq__().

Cryptographic hashing functions are unlike hash(): they are designed to avoid collisions. Python implements a collection of cryptographic hashing functions in the hashlib library. These functions consume byte data and return, with the hexdigest() method, a message digest as a string.

>>> import hashlib
>>> hashlib.sha256(b'').hexdigest()
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
>>> hashlib.sha256(b'0').hexdigest()
'5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9'
>>> hashlib.sha256(b'False').hexdigest()
'60a33e6cf5151f2d52eddae9685cfa270426aa89d8dbc7dfb854606f1d1a40fe'

In-Memory Memoization

To memoize functions that take DataFrames as arguments, an immutable and hashable DataFrame is required. StaticFrame offers the FrameHE for this purpose, where "HE" stands for "hash, equals," the two required implementations for Python hashability. While the StaticFrame Frame is immutable, it is not hashable.

The FrameHE.__hash__() method returns the hash() of the labels of the index and columns. While this will collide with any other FrameHE with the same labels but different values, using just the labels defers the more expensive full-value comparison to __eq__().

The implementation of FrameHE.__eq__() simply delegates to Frame.equals(), a method that always returns a single Boolean. This contrasts with Frame.__eq__(), which returns an element-wise comparison in a Boolean Frame.

>>> f = sf.FrameHE(np.arange(1_000_000).reshape(1000, 1000))
>>> hash(f)
8397108298071051538
>>> f == f * 2
False

With a FrameHE as an argument, the cube() function, decorated with functools.lru_cache(), can be used. If lacking a FrameHE, the to_frame_he() method can be used to efficiently create a FrameHE from other StaticFrame containers. As underlying NumPy array data is immutable and sharable among containers, this is a light-weight, no-copy operation. If coming from a Pandas DataFrame, FrameHE.from_pandas() can be used.

In the example below, cube() is called with the FrameHE created above. The IPython %time utility shows that, after being called once, subsequent calls with the same argument are three orders of magnitude faster (from ms to µs).

>>> %time cube(f)
CPU times: user 8.24 ms, sys: 99 µs, total: 8.34 ms
>>> %time cube(f)
CPU times: user 5 µs, sys: 4 µs, total: 9 µs

While helpful for in-memory memoization, FrameHE instances can also be members of sets, offering a novel approach to collecting unique containers.

Creating a Message Digest from a DataFrame

While in-memory memoization offers optimal performance, caches consume system memory and do not persist beyond the life of the process. If function results are large, or caches should persist, disk-based memoization is an alternative.

In this scenario, mutability and hashability of arguments is irrelevant. Instead, cached results can be retrieved from a file with a name derived from the arguments. Applying a cryptographic hash function on the arguments is ideal for this purpose.

As such hash functions generally take byte data as input, a Frame and all of its components must be converted to a byte representation. A common approach is to serialize the Frame as JSON (or some other string representation), which can then be converted to bytes. As underlying NumPy array data is already stored in bytes, converting that data to strings is inefficient. Further, as JSON does not support the full range of NumPy types, the JSON input might also be insufficiently distinct, leading to collisions.

StaticFrame offers via_hashlib() to meet this need, providing an efficient way to provide byte input to cryptographic hash functions found in the Python hashlib module. An example using SHA-256 is given below.

>>> f.via_hashlib(include_name=False).sha256().hexdigest()
'b931bd5662bb75949404f3735acf652cf177c5236e9d20342851417325dd026c'

First, via_hashlib() is called with options to determine which container components should be included in the input bytes. As the default name attribute, None, is not byte encodable, it is excluded. Second, the hash function constructor sha256() is called, returning an instance loaded with the appropriate input bytes. Third, the hexdigest() method is called to return the message digest as a string. Alternative cryptographic hash function constructors, such as sha3_256, shake_256, and blake2b are available.

To create the input bytes, StaticFrame concatenates all underlying byte data (both values and labels), optionally including container metadata (such as name and __class__.__name__ attributes). This same byte representation is available with the via_hashlib().to_bytes() method. If necessary, this can be combined with other byte data to create a hash digest based on multiple components.

>>> len(f.via_hashlib(include_name=False).to_bytes())
8016017

StaticFrame's built-in support for creating message digests is shown to be more efficient than two common approaches with Pandas. The first approach uses the Pandas utility function pd.hash_pandas_object() to derive per-column integer hashes. This routine uses a bespoke digest algorithm that makes no claim of cryptographic collision resistance. For comparison here, those per-column integer hashes are used as input to a hashlib message digest function. The second approach provides a JSON representation of the entire DataFrame as input to a hashlib message digest function. While this may be more collision resistant than pd.hash_pandas_object(), it is often slower. The following chart displays performance characteristics of these two approaches compared to via_hashlib(). Over a range of DataFrame shapes and type mixtures, via_hashlib() outperforms all except one.

Disk-Based Memoization

Given a means to convert a DataFrame into a hash digest, a disk-based caching routine can be implemented. The decorator below does this for the narrow case of a function that takes and returns a single Frame. In this routine, a file name is derived from a message digest of the argument, prefixed by the name of the function. If the file name does not exist, the decorated function is called and the result is written. If the file name does exist, it is loaded and returned. Here, the StaticFrame NPZ file format is used. As demonstrated in a recent PyCon talk, storing a Frame as an NPZ is often much faster than Parquet and related formats, and provides complete round-trip serialization.

>>> def disk_cache(func):
...     def wrapped(arg):
...         fn = '.'.join(func.__name__, arg.via_hashlib(include_name=False).sha256().hexdigest(), 'npz')
...         fp = Path('/tmp') / fn
...         if not fp.exists():
...             func(arg).to_npz(fp)
...         return sf.Frame.from_npz(fp)
...     return wrapped

To demonstrate this decorator, it can be applied to a function that iterates over windows of ten rows, sums the columns, and then concatenates the results into a single Frame.

>>> @disk_cache
... def windowed_sum(v):
...     return sf.Frame.from_concat(v.iter_window_items(size=10).apply_iter(lambda l, f: f.sum().rename(l)))

After first usage, performance is reduced to less than twenty percent of the original run time. While loading a disk-based cache is slower than retrieving an in-memory cache, the benefit of avoiding repeated calculations is gained without consuming memory and with the opportunity of persistent caches.

>>> %time windowed_sum(f)
CPU times: user 596 ms, sys: 15.6 ms, total: 612 ms
>>> %time windowed_sum(f)
CPU times: user 77.3 ms, sys: 24.4 ms, total: 102 ms

The via_hashlib() interfaces can be used in other situations as a digital signature or checksum of all characteristics of a DataFrame.

Conclusion

If pure functions are called multiple times with the same arguments, memoization can vastly improve performance. While functions that input and output DataFrames require special handling, StaticFrame offers convenient tools to implement both in-memory and disk-based memoization. While care must be taken to ensure that caches are properly invalidated and collisions are avoided, great performance benefits can be realized when repeated work is eliminated.

The Performance Advantage of No-Copy DataFrame Operations

Christopher Ariza — Mon, 21 Nov 2022 23:00:00 +0000

A NumPy array is a Python object that stores data in a contiguous C-array buffer. The excellent performance of these arrays comes not only from this compact representation, but also from the ability of arrays to share "views" of that buffer among many arrays. NumPy makes frequent use of "no-copy" array operations, producing derived arrays without copying underling data buffers. By taking full advantage of NumPy's efficiency, the StaticFrame DataFrame library offers orders-of-magnitude better performance than Pandas for many common operations.

No-Copy Operations with NumPy Arrays

The phrase "no-copy" describes an operation on a container (here, an array or a DataFrame) where a new instance is created, but underlying data is referenced, not copied. While some new memory is allocated for the instance, the size is generally insignificant compared to a potentially very large amount of underlying data.

NumPy makes no-copy operations the primary way of working with arrays. When you slice a NumPy array, you get a new array that shares the data from which it was sliced. Slicing an array is a no-copy operation. Extraordinary performance is gained by not having to copy already-allocated contiguous buffers, but instead just storing offsets and strides into that data.

For example, the difference between slicing an array of 100,000 integers (~0.1 µs), and slicing and then copying the same array (~10 µs), is two orders of magnitude.

>>> import numpy as np
>>> data = np.arange(100_000)
>>> %timeit data[:50_000]
123 ns ± 0.565 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> %timeit data[:50_000].copy()
13.1 µs ± 48.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

We can illustrate how this works by examining two attributes of NumPy arrays. The flags attribute displays details of how the array's memory is being referenced. The base attribute, if set, provides a handle to the array that actually holds the buffer this array references.

In the example below, we create an array, take a slice, and look at the flags of the slice. We see that, for the slice, OWNDATA is False, and that the base of the slice is the original array (they have the same object identifier).

>>> a1 = np.arange(12)
>>> a1
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])

>>> a2 = a1[:6]
>>> a2.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

>>> id(a1), id(a2.base)
(140506320732848, 140506320732848)

These derived arrays are "views" of the original array. Views can only be taken under certain conditions: reshaping, transposing, or slicing.

For example, after reshaping the initial 1D array into a 2D array, OWNDATA is False, showing that it still references the original array's data.

>>> a3 = a1.reshape(3,4)
>>> a3
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a3.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

>>> id(a3.base), id(a1)
(140506320732848, 140506320732848)

Both horizontal and vertical slices of this 2D array similarly result in arrays that simply reference the original array's data. Again, OWNDATA is False, and the base of the slice is the original array.

>>> a4 = a3[:, 2]
>>> a4
array([ 2,  6, 10])

>>> a4.flags
  C_CONTIGUOUS : False
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

>>> id(a1), id(a4.base)
(140506320732848, 140506320732848)

While creating light-weight views of shared memory buffers offers significant performance advantages, there is a risk: mutating any one of those arrays will mutate all of them. As shown below, the assignment of -1 into our most-derived array is reflected in every associated array.

>>> a4[0] = -1
>>> a4
array([-1,  6, 10])
>>> a3
array([[ 0,  1, -1,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a2
array([ 0,  1, -1,  3,  4,  5])
>>> a1
array([ 0,  1, -1,  3,  4,  5,  6,  7,  8,  9, 10, 11])

Side-effects like this should concern you. Passing around views of shared buffers to clients that can mutate those buffers can lead to serious flaws. There are two solutions to this problem.

One option is for the caller to make explicit "defensive" copies every time a new array is created. This removes the performance advantage of sharing views but ensures that mutating an array does not lead to unexpected side effects.

Another option, requiring no sacrifice in performance, is to make the array immutable. By doing so, views of arrays can be shared without concern of mutation causing unexpected side effects.

A NumPy array can easily be made immutable by setting the writeable flag to False on the flags interface. After setting this value, the flags display shows WRITEABLE as False and attempting to mutate this array results in an exception.

>>> a1.flags.writeable = False
>>> a1.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

>>> a1[0] = -1
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: assignment destination is read-only

The best performance is possible, with no risk of side-effects, by embracing immutable views of NumPy arrays.

The Advantages of No-Copy DataFrame Operations

This insight, that an immutable-array-based data model offers the best performance with the minimum risk, was foundational to the creation of the StaticFrame DataFrame library. As StaticFrame (like Pandas) manages data stored in NumPy arrays, embracing the usage of array views (without having to make defensive copies) offers significant performance advantages. Without an immutable data model, Pandas cannot make such use of array views.

StaticFrame is not yet always faster than Pandas: Pandas has very performant operations for joins and other specialized transformations. But when leveraging no-copy array operations, StaticFrame can be a lot faster.

To compare performance, we will use the FrameFixtures library to create two DataFrames of 10,000 rows by 1,000 columns of heterogeneous types. For both we can convert the StaticFrame Frame into a Pandas DataFrame.

>>> import static_frame as sf
>>> import pandas as pd
>>> sf.__version__, pd.__version__
('0.9.21', '1.5.1')

>>> import frame_fixtures as ff
>>> f1 = ff.parse('s(10_000,1000)|v(int,int,str,float)')
>>> df1 = f1.to_pandas()
>>> f2 = ff.parse('s(10_000,1000)|v(int,bool,bool,float)')
>>> df2 = f2.to_pandas()

A simple example of the advantage of a no-copy operation is renaming an axis. With Pandas, all underlying data is defensively copied. With StaticFrame, all underlying data is re-used; only lightweight outer containers have to be created. StaticFrame (~0.01 ms) is almost four orders of magnitude faster than Pandas (~100 ms).

>>> %timeit f1.rename(index='foo')
35.8 µs ± 496 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
>>> %timeit df1.rename_axis('foo')
167 ms ± 4.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Given a DataFrame, it is often necessary to make a column into the index. When Pandas does this, it has to copy the column data to the index, as well as copy all the underlying data. StaticFrame can re-use a view of the column in the index, as well as re-use all of the underlying data. StaticFrame (~1 ms) is two orders of magnitude faster than Pandas (~100 ms).

>>> %timeit f1.set_index(0)
1.25 ms ± 23.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> %timeit df1.set_index(0, drop=False)
166 ms ± 3.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Extracting a subset of columns from a DataFrame is another common operation. For StaticFrame, this is a no-copy operation: the returned DataFrame simply holds views to the column data in the original DataFrame. StaticFrame (~10 µs) can do this an order of magnitude faster than Pandas (~100 µs).

>>> %timeit f1[[10, 50, 100, 500]]
25.4 µs ± 471 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
>>> %timeit df1[[10, 50, 100, 500]]
729 µs ± 4.14 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

It is common to concatenate two or more DataFrames. If they have the same index, and we concatenate them horizontally, StaticFrame can re-use all the underlying data of the inputs, making this form of concatenation a no-copy operation. StaticFrame (~1 ms) can do this two orders of magnitude faster than Pandas (~100 ms).

>>> %timeit sf.Frame.from_concat((f1, f2), axis=1, columns=sf.IndexAutoFactory)
1.16 ms ± 50.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> %timeit pd.concat((df1, df2), axis=1)
102 ms ± 14.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Conclusion

NumPy is designed to take advantage of sharing views of data. Because Pandas permits in-place mutation, it cannot make optimal use of NumPy views. As StaticFrame is built on an immutable data model, side-effect mutation risk is eliminated and no-copy operations are embraced, providing a significant performance advantage.

One Fill Value Is Not Enough: Preserving Columnar Types When Reindexing DataFrames

Christopher Ariza — Tue, 28 Jun 2022 15:58:36 +0000

When working with DataFrames, reindexing is common. When a DataFrame is reindexed, an old index (and its associated values) is conformed to a new index, potentially reordering, contracting, or expanding the rows or columns. When a reindex expands a DataFrame, new values are needed to fill the newly created rows or columns: these are "fill values."

When reindexing with Pandas, only a single value, via the fill_value parameter, is permitted. If that fill_value is a type incompatible with the type of one or more columns, the column will be re-cast into a different, likely undesirable, type.

For example, given a DataFrame with three columns typed object, integer, and Boolean, reindexing the index, by default, fills new rows with NaN, a float type that forces the integer column to be converted to float and the Boolean column to be converted to object.

>>> df = pd.DataFrame.from_records((('a', 1, True), 
    ('b', 2, False)), columns=tuple('xyz'))
>>> df
   x  y      z
0  a  1   True
1  b  2  False

>>> df.dtypes.tolist()
[dtype('O'), dtype('int64'), dtype('bool')]

>>> df.reindex((1, 0, 2))
     x    y      z
1    b  2.0  False
0    a  1.0   True
2  NaN  NaN    NaN

>>> df.reindex((1, 0, 2)).dtypes.tolist()
[dtype('O'), dtype('float64'), dtype('O')]

Columnar type degradation is often detrimental. The pre-existing columnar type was probably appropriate for the data; having that type unnecessarily changed simply due to reindexing is often unexpected. Going from one C-level NumPy type to another, such as from int to float, might be tolerable. But when going from C-level NumPy types to arrays of Python objects (object dtypes), performance will be degraded. When reindexing with Pandas, there is no way to avoid this problem.

StaticFrame is an immutable DataFrame library that offers solutions to such problems. In StaticFrame, alternative fill value representations can be used to preserve columnar types in reindexing, shifting, and many other operations that require fill_value arguments. For operations on heterogeneously typed columnar data, one fill value is simply not enough.

StaticFrame supports providing fill_value as a single element, as a row-length list of values, as a mapping by column label, or as a FillValueAuto, a novel object to define type-to-value mappings.

All examples use Pandas 1.4.3 and StaticFrame 0.9.6. Imports use the following convention:

>>> import pandas as pd
>>> import static_frame as sf

We can reproduce Pandas behavior in StaticFrame by reindexing the same DataFrame with a single fill value, NaN. This results in the same columnar types as Pandas. Notice that StaticFrame, by default, displays the dtype for each column, making columnar type degradation easily apparent.

>>> f = sf.Frame.from_records((('a', 1, True), 
    ('b', 2, False)), columns=tuple('xyz'))
>>> f
<Frame>
<Index> x     y       z      <<U1>
<Index>
0       a     1       True
1       b     2       False
<int64> <<U1> <int64> <bool>

>>> f.reindex((1, 0, 2), fill_value=np.nan)
<Frame>
<Index> x        y         z        <<U1>
<Index>
1       b        2.0       False
0       a        1.0       True
2       nan      nan       nan
<int64> <object> <float64> <object>

One way to avoid type degredation in reindexing is to provide a fill value per column. With StaticFrame, fill values can be provide with a list, providing one value per column:

>>> f.reindex((1, 0, 2), fill_value=['', 0, False])
<Frame>
<Index> x     y       z      <<U1>
<Index>
1       b     2       False
0       a     1       True
2             0       False
<int64> <<U1> <int64> <bool>

Alternatively, a dictionary can be used to provide a mapping of column label to fill value. If a label is not provided, the default (NaN) will be provided.

>>> f.reindex((1, 0, 2), fill_value={'z':False, 'x':''})
<Frame>
<Index> x     y         z      <<U1>
<Index>
1       b     2.0       False
0       a     1.0       True
2             nan       False
<int64> <<U1> <float64> <bool>

The previous examples all require an explicit value per column, providing maximum specificity. In many cases (and in particular for larger DataFrames), a more general way of specifying fill values is necessary.

One option might be to map a fill value based on specific NumPy dtypes. Such an approach is rejected, as NumPy dtypes define a variable "itemsize" in bytes, leading to a very large number of possible NumPy dtypes. It is more likely that the same fill value would be used for families of dtypes independent of itemsize; for example, all sizes of integers (int8, int16, int32, and int64).

To identify size-independent type families, we can use dtype "kind". NumPy dtypes have a "kind" attribute independent of dtype itemsize: for example, int8, int16, int32, and int64 dtypes are all labeled kind "i". As shown below, there are eleven dtype kinds, each with a one-character label:

Kind Label	Type
b	bool
i	int
u	uint
f	float
c	complex
m	timedelta
M	datetime
O	object
S	bytes
U	str
V	void

Specifying a fill value per dtype kind provides a convenient way to avoid columnar type coercions while not requiring a cumbersome specification per column. To do this, StaticFrame introduces a new object: FileValueAuto.

Using the class FillValueAuto as a fill value provides type-coercion-free defaults for all dtype kinds. If a different mapping is desired, a FillValueAuto instance can be created, specifying a fill value per dtype kind.

Returning to the previous reindexing example, we see the convenience of using the FillValueAuto class and that all columnar types are preserved:

>>> f
<Frame>
<Index> x     y       z      <<U1>
<Index>
0       a     1       True
1       b     2       False
<int64> <<U1> <int64> <bool>

>>> f.reindex((1, 0, 2), fill_value=sf.FillValueAuto)
<Frame>
<Index> x     y       z      <<U1>
<Index>
1       b     2       False
0       a     1       True
2             0       False
<int64> <<U1> <int64> <bool>

If we need to deviate from the supplied FillValueAuto defaults, an instance can be created, specifying fill values per dtype kind. The key-word arguments of the initializer are the single-character dtype kind labels.

>>> f.reindex((1, 0, 2), 
    fill_value=sf.FillValueAuto(U='x', i=-1, b=None))
<Frame>
<Index> x     y       z        <<U1>
<Index>
1       b     2       False
0       a     1       True
2       x     -1      None
<int64> <<U1> <int64> <object>

In StaticFrame, the same multitude of fill value types are accepted nearly everywhere fill values are needed. For example, in shifting data, fill values must be provided; but when shifting an entire DataFrame of heterogenous types, one fill value is not enough. As shown below, the default fill_value, NaN, forces all columnar types to either object or float.

>>> f = sf.Frame.from_records((('a', 1, True, 'p', 23.2), 
    ('b', 2, False, 'q', 85.1), ('c', 3, True, 'r', 1.23)), 
    columns=tuple('abcde'))

>>> f.shift(2)
<Frame>
<Index> a        b         c        d        e         <<U1>
<Index>
0       nan      nan       nan      nan      nan
1       nan      nan       nan      nan      nan
2       a        1.0       True     p        23.2
<int64> <object> <float64> <object> <object> <float64>

As before, using a FillValueAuto instance permits a general fill value specification that completely avoids columnar type degradation.

>>> f.shift(2, 
    fill_value=sf.FillValueAuto(U='', b=False, f=0, i=0))
<Frame>
<Index> a     b       c      d     e         <<U1>
<Index>
0             0       False        0.0
1             0       False        0.0
2       a     1       True   p     23.2
<int64> <<U1> <int64> <bool> <<U1> <float64>

A fill value is also needed in many applications of binary operators. In general, binary operations on labelled data force operands to reindex to a union index, potentially introducing missing values. If the missing value is only NaN, the resulting columnar types might be recast.

For example, given two DataFrames, each with a float and an integer column, a binary operation will introduce NaN for reindexed values, coercing the integer column to floats. This can be avoided in StaticFrame by using FillValueAuto.

As binary operators do not accept arguments, StaticFrame provides the via_fill_value interface to permit specification of a fill value to be used if reindexing is required in binary operations. This is similar to functionality provided by Pandas DataFrame.multiply() and related methods. With StaticFrame's via_fill_value, we can continue to use expressions of arbitrary binary operators.

When multiplying two DataFrames, each with a column of floats and a column of integers, the introduction of NaNs due to reindexing forces all values to floats.

>>> f1 = sf.Frame.from_records(((10.2, 20), (2.4, 4)), index=('a', 'b'))
>>> f2 = sf.Frame.from_records(((3.4, 1), (8.2, 0)), index=('b', 'c'))

>>> f1 * f2
<Frame>
<Index> 0         1         <int64>
<Index>
a       nan       nan
b       8.16      4.0
c       nan       nan
<<U1>   <float64> <float64>

By using via_fill_value and FillValueAuto, we can preserve columnar types, even when reindexing is required, and continue to use binary operators in expressions.

>>> f1.via_fill_value(sf.FillValueAuto) * f2
<Frame>
<Index> 0         1       <int64>
<Index>
a       nan       0
b       8.16      4
c       nan       0
<<U1>   <float64> <int64>

Examples with just a few columns, as used above, do not fully demonstrate the power of FillValueAuto: when dealing with heterogeneously typed DataFrames of hundreds or thousands of columns, the generality of specification provides a concise and powerful tool.

The cost of inadvertent type coercion caused by reindexing or other transformations can lead to bugs or degraded performance. StaticFrame's flexible fill value types, as well as the new FillValueAuto, provide solutions to these practical problems.

Ten Reasons to Use StaticFrame Instead of Pandas

Christopher Ariza — Tue, 30 Jun 2020 15:42:02 +0000

If you work with data in Python, you probably use Pandas. Pandas provides nearly instant gratification: sophisticated data processing routines can be implemented in a few lines of code. However, if you have used Pandas on large projects over many years, you may have had some challenges. Complex Pandas applications can produce Python code that is hard to maintain and error-prone. This happens because Pandas provides many ways to do the same thing, has inconsistent interfaces, and broadly supports in-place mutation. For those coming from Pandas, StaticFrame offers a more consistent interface and reduces opportunities for error. This article demonstrates ten reasons you might use StaticFrame instead of Pandas.

Why StaticFrame

After years of using Pandas to develop back-end financial systems, it became clear to me that Pandas was not the right tool for the job. Pandas's handling of labeled data and missing values, with performance close to NumPy, certainly accelerated my productivity. And yet, the numerous inconsistencies in Pandas's API led to hard-to-maintain code. Further, Pandas's support for in-place mutation led to serious opportunities for error. So in May of 2017 I began implementing a library more suitable for critical production systems.

Now, after years of development and refinement, we are seeing excellent results in our production systems by replacing Pandas with StaticFrame. Libraries and applications written with StaticFrame are easier to maintain and test. And we often see StaticFrame out-perform Pandas in large-scale, real-world use cases, even though, for many isolated operations, StaticFrame is not yet as fast as Pandas.

What follows are ten reasons to favor using StaticFrame over Pandas. As the primary author of StaticFrame, I am certainly biased in this presentation. However, having worked with Pandas since 2013, I hope to have some perspective to share.

All examples use Pandas 1.0.3 and StaticFrame 0.6.20. Imports use the following convention:

>>> import pandas as pd
>>> import static_frame as sf

No. 1: Consistent and Discoverable Interfaces

An application programming interface (API) can be consistent in where functions are located, how functions are named, and the name and types of arguments those functions accept. StaticFrame deviates from Pandas's API to support greater consistency in all of these areas.

To create a sf.Series or a sf.Frame, you need constructors. Pandas places its pd.DataFrame constructors in two places: on the root namespace (pd, as commonly imported) and on the pd.DataFrame class.

For example, JSON data is loaded from a function on the pd namespace, while record data (an iterable of Python sequences) is loaded from the pd.DataFrame class.

>>> pd.read_json('[{"name":"muon", "mass":0.106}, {"name":"tau", "mass":1.777}]')
   name   mass
0  muon  0.106
1   tau  1.777
>>> pd.DataFrame.from_records([{"name":"muon", "mass":0.106}, {"name":"tau", "mass":1.777}])
   name   mass
0  muon  0.106
1   tau  1.777

Even though Pandas has specialized constructors, the default pd.DataFrame constructor accepts a staggering diversity of inputs, including many of the same inputs as pd.DataFrame.from_records().

>>> pd.DataFrame([{"name":"muon", "mass":0.106}, 
{"name":"tau", "mass":1.777}])
   name   mass
0  muon  0.106
1   tau  1.777

For the user, there is little benefit to this diversity and redundancy. StaticFrame places all constructors on the class they construct, and as much as possible, narrowly focuses their functionality. As they are easier to maintain, explicit, specialized constructors are common in StaticFrame. For example, sf.Frame.from_json() and sf.Frame.from_dict_records():

>>> sf.Frame.from_json('[{"name":"muon", "mass":0.106}, {"name":"tau", "mass":1.777}]')
<Frame>
<Index> name  mass      <<U4>
<Index>
0       muon  0.106
1       tau   1.777
<int64> <<U4> <float64>
>>> sf.Frame.from_dict_records([{"name":"muon", "mass":0.106}, {"name":"tau", "mass":1.777}])
<Frame>
<Index> name  mass      <<U4>
<Index>
0       muon  0.106
1       tau   1.777
<int64> <<U4> <float64>

Being explicit leads to lots of constructors. To help you find what you are looking for, StaticFrame containers expose an interface attribute that provides the entire public interface of the calling class or instance as a sf.Frame. We can filter this table to show only constructors by using a sf.Frame.loc[] selection.

>>> sf.Frame.interface.loc[sf.Frame.interface['group'] == 'Constructor']
<Frame: Frame>
<Index>                              cls_name group       doc                  <<U18>
<Index: signature>
__init__(data, *, index, columns,... Frame    Constructor
from_arrow(value, *, index_depth,... Frame    Constructor Convert an Arrow ...
from_concat(frames, *, axis, unio... Frame    Constructor Concatenate multi...
from_concat_items(items, *, axis,... Frame    Constructor Produce a Frame w...
from_csv(fp, *, index_depth, inde... Frame    Constructor Specialized versi...
from_delimited(fp, *, delimiter, ... Frame    Constructor Create a Frame fr...
from_dict(mapping, *, index, fill... Frame    Constructor Create a Frame fr...
from_dict_records(records, *, ind... Frame    Constructor Frame constructor...
from_dict_records_items(items, *,... Frame    Constructor Frame constructor...
from_element(element, *, index, c... Frame    Constructor Create a Frame fr...
from_element_iloc_items(items, *,... Frame    Constructor Given an iterable...
from_element_loc_items(items, *, ... Frame    Constructor This function is ...
from_elements(elements, *, index,... Frame    Constructor Create a Frame fr...
from_hdf5(fp, *, label, index_dep... Frame    Constructor Load Frame from t...
from_items(pairs, *, index, fill_... Frame    Constructor Frame constructor...
from_json(json_data, *, dtypes, n... Frame    Constructor Frame constructor...
from_json_url(url, *, dtypes, nam... Frame    Constructor Frame constructor...
from_pandas(value, *, index_const... Frame    Constructor Given a Pandas Da...
from_parquet(fp, *, index_depth, ... Frame    Constructor Realize a Frame f...
from_records(records, *, index, c... Frame    Constructor Construct a Frame...
from_records_items(items, *, colu... Frame    Constructor Frame constructor...
from_series(series, *, name, colu... Frame    Constructor Frame constructor...
from_sql(query, *, connection, in... Frame    Constructor Frame constructor...
from_sqlite(fp, *, label, index_d... Frame    Constructor Load Frame from t...
from_structured_array(array, *, i... Frame    Constructor Convert a NumPy s...
from_tsv(fp, *, index_depth, inde... Frame    Constructor Specialized versi...
from_xlsx(fp, *, label, index_dep... Frame    Constructor Load Frame from t...
<<U94>                               <<U5>    <<U17>      <<U83>

No. 2: Consistent and Colorful Display

Pandas displays its containers in diverse ways. For example, a pd.Series is shown with its name and type, while a pd.DataFrame shows neither of those attributes. If you display a pd.Index or pd.MultiIndex, you get a third approach: a string suitable for eval() which is inscrutable when large.

>>> df = pd.DataFrame.from_records([{'symbol':'c', 'mass':1.3}, {'symbol':'s', 'mass':0.1}], index=('charm', 'strange'))
>>> df
        symbol  mass
charm        c   1.3
strange      s   0.1

>>> df['mass']
charm      1.3
strange    0.1
Name: mass, dtype: float64

>>> df.index
Index(['charm', 'strange'], dtype='object')

StaticFrame offers a consistent, configurable display for all containers. The display of sf.Series, sf.Frame, sf.Index, and sf.IndexHierarchy all share a common implementation and design. A priority of that design is to always make explicit container classes and underlying array types.

>>> f = sf.Frame.from_dict_records_items((('charm', {'symbol':'c', 'mass':1.3}), ('strange', {'symbol':'s', 'mass':0.1})))
>>> f
<Frame>
<Index> symbol mass      <<U6>
<Index>
charm   c      1.3
strange s      0.1
<<U7>   <<U1>  <float64>

>>> f['mass']
<Series: mass>
<Index>
charm          1.3
strange        0.1
<<U7>          <float64>

>>> f.columns
<Index>
symbol
mass
<<U6>

As much time is spent visually exploring the contents of these containers, StaticFrame offers numerous display configuration options, all exposed through the sf.DisplayConfig class. For persistent changes, sf.DisplayConfig instances can be passed to sf.DisplayActive.set(); for one-off changes, sf.DisplayConfig instances can be passed to the container's display() method.

While pd.set_option() can similarly be used to set Pandas display characteristics, StaticFrame provides more extensive options for making types discoverable. As shown in the following terminal animation, specific types can be colored or type annotations can be removed entirely.

No. 3: Immutable Data: Efficient Memory Management without Defensive Copies

Pandas displays inconsistent behavior in regard to ownership of data inputs and data exposed from within containers. In some cases, it is possible to mutate NumPy arrays "behind-the-back" of Pandas, exposing opportunities for undesirable side-effects and coding errors.

For example, if we supply a 2D array to a pd.DataFrame, the original reference to the array can be used to "remotely" change the values within the pd.DataFrame. In this case, the pd.DataFrame does not protect access to its data, serving only as a wrapper of a shared, mutable array.

>>> a1 = np.array([[0.106, -1], [1.777, -1]])
>>> df = pd.DataFrame(a1, index=('muon', 'tau'), columns=('mass', 'charge'))
>>> df
       mass  charge
muon  0.106    -1.0
tau   1.777    -1.0

>>> a1[0, 0] = np.nan # Mutating the original array.

>>> df # Mutation reflected in the DataFrame created from that array.
       mass  charge
muon    NaN    -1.0
tau   1.777    -1.0

Similarly, sometimes NumPy arrays exposed from the values attribute of a pd.Series or a pd.DataFrame can be mutated, changing the values within the DataFrame.

>>> a2 = df['charge'].values
>>> a2
array([-1., -1.])

>>> a2[1] = np.nan # Mutating the array from .values.

>>> df # Mutation is reflected in the DataFrame.
       mass  charge
muon    NaN    -1.0
tau   1.777     NaN

With StaticFrame, there is no vulnerability of "behind the back" mutation: as StaticFrame manages immutable NumPy arrays, references are only held to immutable arrays. If a mutable array is given at initialization, an immutable copy will be made. Immutable arrays cannot be mutated from containers or from direct access to underlying arrays.

>>> a1 = np.array([[0.106, -1], [1.777, -1]])
>>> f = sf.Frame(a1, index=('muon', 'tau'), columns=('mass', 'charge'))

>>> a1[0, 0] = np.nan # Mutating the original array has no affect on the Frame

>>> f
<Frame>
<Index> mass      charge    <<U6>
<Index>
muon    0.106     -1.0
tau     1.777     -1.0
<<U4>   <float64> <float64>

>>> f['charge'].values[1] = np.nan # An immutable array cannot be mutated
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: assignment destination is read-only

While immutable data reduces opportunities for error, it also offers performance advantages. For example, when replacing column labels with sf.Frame.relabel(), underlying data is not copied. Instead, references to the same immutable arrays are shared between the old and new containers. Such "no-copy" operations are thus fast and light-weight. This is in contrast to what happens when doing the same thing in Pandas: the corresponding Pandas method, df.DataFrame.rename(), is forced to make a defensive copy of all underlying data.

>>> f.relabel(columns=lambda x: x.upper()) # Underlying arrays are not copied
<Frame>
<Index> MASS      CHARGE    <<U6>
<Index>
muon    0.106     -1.0
tau     1.777     -1.0
<<U4>   <float64> <float64>

No. 4: Assignment is a Function

While Pandas permits in-place assignment, sometimes such operations cannot provide an appropriate derived type, resulting in undesirable behavior. For example, a float assigned into an integer pd.Series will have its floating-point components truncated without warning or error.

>>> s = pd.Series((-1, -1), index=('tau', 'down'))
>>> s
tau    -1
down   -1
dtype: int64

>>> s['down'] = -0.333 # Assigning a float.

>>> s # The -0.333 value was truncated to 0
tau    -1
down    0
dtype: int64

With StaticFrame's immutable data model, assignment is a function that returns a new container. This permits evaluating types to insure that the resultant array can completely contain the assigned value.

>>> s = sf.Series((-1, -1), index=('tau', 'down'))
>>> s
<Series>
<Index>
tau      -1
down     -1
<<U4>    <int64>

>>> s.assign['down'](-0.333) # The float is assigned without truncation
<Series>
<Index>
tau      -1.0
down     -0.333
<<U4>    <float64>

StaticFrame uses a special assign interface for performing assignment function calls. On a sf.Frame, this interface exposes a sf.Frame.assign.loc[] interface that can be used to select the target of assignment. Following this selection, the value to be assigned is passed through a function call.

>>> f = sf.Frame.from_dict_records_items((('charm', {'charge':0.666, 'mass':1.3}), ('strange', {'charge':-0.333, 'mass':0.1})))
>>> f
<Frame>
<Index> charge    mass      <<U6>
<Index>
charm   0.666     1.3
strange -0.333    0.1
<<U7>   <float64> <float64>

>>> f.assign.loc['charm', 'charge'](Fraction(2, 3)) # Assigning to a loc-style selection
<Frame>
<Index> charge   mass      <<U6>
<Index>
charm   2/3      1.3
strange -0.333   0.1
<<U7>   <object> <float64>

No. 5: Iterators are for Iterating and Function Application

Pandas has separate functions for iteration and function application. For iteration on a pd.DataFrame there is pd.DataFrame.iteritems(), pd.DataFrame.iterrows(), pd.DataFrame.itertuples(), and pd.DataFrame.groupby(); for function application on a pd.DataFrame there is pd.DataFrame.apply() and pd.DataFrame.applymap().

But since function application requires iteration, it is sensible for function application to be built on iteration. StaticFrame organizes iteration and function application by providing families of iterators (such as Frame.iter_array() or Frame.iter_group_items()) that, with a chained call to apply(), can also be used for function application. Functions for applying mapping types (such as map_any() and map_fill()) are also available on iterators. This means that once you know how you want to iterate, function application is a just a method away.

For example, we can create a sf.Frame with sf.Frame.from_records():

>>> f = sf.Frame.from_records(((0.106, -1.0, 'lepton'), (1.777, -1.0, 'lepton'), (1.3, 0.666, 'quark'), (0.1, -0.333, 'quark')), columns=('mass', 'charge', 'type'), index=('muon', 'tau', 'charm', 'strange'))
>>> f
<Frame>
<Index> mass      charge    type   <<U6>
<Index>
muon    0.106     -1.0      lepton
tau     1.777     -1.0      lepton
charm   1.3       0.666     quark
strange 0.1       -0.333    quark

We can iterate over a columns values with sf.Series.iter_element(). We can use the same iterator to do function application by using the apply() method found on the object returned from sf.Series.iter_element(). The same interface is found on both sf.Series and sf.Frame.

>>> tuple(f['type'].iter_element())
('lepton', 'lepton', 'quark', 'quark')

>>> f['type'].iter_element().apply(lambda e: e.upper())
<Series>
<Index>
muon     LEPTON
tau      LEPTON
charm    QUARK
strange  QUARK
<<U7>    <<U6>

>>> f[['mass', 'charge']].iter_element().apply(lambda e: format(e, '.2e'))
<Frame>
<Index> mass     charge    <<U6>
<Index>
muon    1.06e-01 -1.00e+00
tau     1.78e+00 -1.00e+00
charm   1.30e+00 6.66e-01
strange 1.00e-01 -3.33e-01
<<U7>   <object> <object>

For row or column iteration on a sf.Frame, a family of methods allows specifying the type of container to be used for the iterated rows or columns, i.e, with an array, with a NamedTuple, or with a sf.Series (iter_array(), iter_tuple(), iter_series(), respectively). These methods take an axis argument to determine whether iteration is by row or by column, and similarly expose an apply() method for function application. To apply a function to columns, we can do the following.

>>> f[['mass', 'charge']].iter_array(axis=0).apply(np.sum)
<Series>
<Index>
mass     3.283
charge   -1.667
<<U6>    <float64>

Applying a function to a row instead of a column simply requires changing the axis argument.

>>> f.iter_series(axis=1).apply(lambda s: s['mass'] > 1 and s['type'] == 'quark')
<Series>
<Index>
muon     False
tau      False
charm    True
strange  False
<<U7>    <bool>

Group-by operations are just another form of iteration, with an identical interface for iteration and function application.

>>> f.iter_group('type').apply(lambda f: f['mass'].mean())
<Series>
<Index>
lepton   0.9415
quark    0.7000000000000001
<<U6>    <float64>

No. 6: Strict, Grow-Only Frames

An efficient use of a pd.DataFrame is to load initial data, then produce derived data by adding additional columns. This approach leverages the columnar organization of types and underlying arrays: adding new columns does not require re-allocating old columns.

StaticFrame makes this approach less vulnerable to error by offering a strict, grow-only version of a sf.Frame called a sf.FrameGO. For example, once a sf.FrameGO is created, new columns can be added while existing columns cannot be overwritten or mutated in-place.

>>> f = sf.FrameGO.from_records(((0.106, -1.0, 'lepton'), (1.777, -1.0, 'lepton'), (1.3, 0.666, 'quark'), (0.1, -0.333, 'quark')), columns=('mass', 'charge', 'type'), index=('muon', 'tau', 'charm', 'strange'))
>>> f['positive'] = f['charge'] > 0
>>> f
<FrameGO>
<IndexGO> mass      charge    type   positive <<U8>
<Index>
muon      0.106     -1.0      lepton False
tau       1.777     -1.0      lepton False
charm     1.3       0.666     quark  True
strange   0.1       -0.333    quark  False

This limited form of mutation meets a practical need. Further, converting back and forth from a sf.Frame to a sf.FrameGO (using Frame.to_frame_go() and FrameGO.to_frame()) is a no-copy operation: underlying immutable arrays can be shared between the two containers.

No. 7: Dates are not Nanoseconds

Pandas models all date or timestamp values as NumPy datetime64[ns] (nanosecond) arrays, regardless of if nanosecond-level resolution is practical or appropriate. This creates a "Y2262 problem" for Pandas: dates beyond 2262-04-11 cannot be expressed. While I can create a pd.DatetimeIndex up to 2262-04-11, one day further and Pandas raises an error.

>>> pd.date_range('1980', '2262-04-11')
DatetimeIndex(['1980-01-01', '1980-01-02', '1980-01-03', '1980-01-04',
               '1980-01-05', '1980-01-06', '1980-01-07', '1980-01-08',
               '1980-01-09', '1980-01-10',
               ...
               '2262-04-02', '2262-04-03', '2262-04-04', '2262-04-05',
               '2262-04-06', '2262-04-07', '2262-04-08', '2262-04-09',
               '2262-04-10', '2262-04-11'],
              dtype='datetime64[ns]', length=103100, freq='D')
>>> pd.date_range('1980', '2262-04-12')
Traceback (most recent call last):
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2262-04-12 00:00:00

As indices are often used for date-time values far less granular than nanoseconds (such as dates, months, or years), StaticFrame offers the full range of NumPy typed datetime64 indices. This permits exact date-time type specification, and avoids the limits of nanosecond-based units.

While not possible with Pandas, creating an index of years or dates extending to 3000 is simple with StaticFrame.

>>> sf.IndexYear.from_year_range(1980, 3000).tail()
<IndexYear>
2996
2997
2998
2999
3000
<datetime64[Y]>

>>> sf.IndexDate.from_year_range(1980, 3000).tail()
<IndexDate>
3000-12-27
3000-12-28
3000-12-29
3000-12-30
3000-12-31
<datetime64[D]>

No. 8: Consistent Interfaces for Hierarchical Indices

Hierarchical indices permit fitting many dimensions into one. Using hierarchical indices, n-dimensional data can be encoded into a single sf.Series or sf.Frame.

A key feature of hierarchical indices is partial selection at arbitrary depths, whereby a selection can be composed from the intersection of selections at each depth level. Pandas offers numerous ways to express those inner depth selections.

One way is by overloading pd.DataFrame.loc[]. When using Pandas's hierarchical index (pd.MultiIndex), the meaning of positional arguments in a pd.DataFrame.loc[] selection becomes dynamic. It is this that makes Pandas code using hierarchical indices hard to maintain. We can see this by creating a pd.DataFrame and setting a pd.MultiIndex.

>>> df = pd.DataFrame.from_records([('muon', 0.106, -1.0, 'lepton'), ('tau', 1.777, -1.0, 'lepton'), ('charm', 1.3, 0.666, 'quark'), ('strange', 0.1, -0.333, 'quark')], columns=('name', 'mass', 'charge', 'type'))

>>> df.set_index(['type', 'name'], inplace=True)
>>> df
                 mass  charge
type   name
lepton muon     0.106  -1.000
       tau      1.777  -1.000
quark  charm    1.300   0.666
       strange  0.100  -0.333

Similar to 2D arrays in NumPy, when two arguments are given to pd.DataFrame.loc[], the first argument is a row selector, the second argument is a column selector.

>>> df.loc['lepton', 'mass'] # Selects "lepton" from row, "mass" from columns
name
muon    0.106
tau     1.777
Name: mass, dtype: float64

Yet, in violation of that expectation, sometimes Pandas will not use the second argument as a column selection, but instead as a row selection in an inner depth of the pd.MultiIndex.

>>> df.loc['lepton', 'tau'] # Selects lepton and tau from rows
mass      1.777
charge   -1.000
Name: (lepton, tau), dtype: float64

To handle this ambiguity, Pandas offers two alternatives. If a row and a column selection is required, the expected behavior can be restored by wrapping the hierarchical row selection within a pd.IndexSlice[] selection modifier. Or, if an inner-depth selection is desired without using a pd.IndexSlice[], the pd.DataFrame.xs() method can be used.

>>> df.loc[pd.IndexSlice['lepton', 'tau'], 'charge']
-1.0
>>> df.xs(level=1, key='tau')
         mass  charge
type
lepton  1.777    -1.0

This inconsistency in the meaning of the positional arguments given to pd.DataFrame.loc[] is unnecessary and makes Pandas code harder to maintain: what is intended from the usage of pd.DataFrame.loc[] becomes ambiguous without a pd.IndexSlice[]. Further, providing multiple ways to solve this problem is also a shortcoming, as it is preferable to have one obvious way to do things in Python.

StaticFrame's sf.IndexHierarchy offers more consistent behavior. We will create an equivalent sf.Frame and set a sf.IndexHierarchy.

>>> f = sf.Frame.from_records((('muon', 0.106, -1.0, 'lepton'), ('tau', 1.777, -1.0, 'lepton'), ('charm', 1.3, 0.666, 'quark'), ('strange', 0.1, -0.333, 'quark')), columns=('name', 'mass', 'charge', 'type'))

>>> f = f.set_index_hierarchy(('type', 'name'), drop=True)
>>> f
<Frame>
<Index>                                    mass      charge    <<U6>
<IndexHierarchy: ('type', 'name')>
lepton                             muon    0.106     -1.0
lepton                             tau     1.777     -1.0
quark                              charm   1.3       0.666
quark                              strange 0.1       -0.333
<<U6>                              <<U7>   <float64> <float64>

Unlike Pandas, StaticFrame is consistent in what positional sf.Frame.loc[] arguments mean: the first argument is always a row selector, the second argument is always a column selector. For selection within a sf.IndexHierarchy, the sf.HLoc[] selection modifier is required to specify selection at arbitrary depths within the hierarchy. There is one obvious way to select inner depths. This approach makes StaticFrame code easier to understand and maintain.

>>> f.loc[sf.HLoc['lepton']]
<Frame>
<Index>                                  mass      charge    <<U6>
<IndexHierarchy: ('type', 'name')>
lepton                             muon  0.106     -1.0
lepton                             tau   1.777     -1.0
<<U6>                              <<U4> <float64> <float64>

>>> f.loc[sf.HLoc[:, ['muon', 'strange']], 'mass']
<Series: mass>
<IndexHierarchy: ('type', 'name')>
lepton                             muon    0.106
quark                              strange 0.1
<<U6>                              <<U7>   <float64>

No. 9: Indices are Always Unique

It is natural to think index and column labels on a pd.DataFrame are unique identifiers: their interfaces suggest that they are like Python dictionaries, where keys are always unique. Pandas indices, however, are not constrained to unique values. Creating an index on a pd.Frame with duplicates means that, for some single-label selections, a pd.Series will be returned, but for other single-label selections, a pd.DataFrame will be returned.

>>> df = pd.DataFrame.from_records([('muon', 0.106, -1.0, 'lepton'), ('tau', 1.777, -1.0, 'lepton'), ('charm', 1.3, 0.666, 'quark'), ('strange', 0.1, -0.333, 'quark')], columns=('name', 'mass', 'charge', 'type'))
>>> df.set_index('charge', inplace=True) # Creating an index with duplicated labels
>>> df
           name   mass    type
charge
-1.000     muon  0.106  lepton
-1.000      tau  1.777  lepton
 0.666    charm  1.300   quark
-0.333  strange  0.100   quark

>>> df.loc[-1.0] # Selecting a non-unique label results in a pd.DataFrame
        name   mass    type
charge
-1.0    muon  0.106  lepton
-1.0     tau  1.777  lepton

>>> df.loc[0.666] # Selecting a unique label results in a pd.Series
name    charm
mass      1.3
type    quark
Name: 0.666, dtype: object

Pandas support of non-unique indices makes client code more complicated by having to handle selections that sometimes return a pd.Series and other times return a pd.DataFrame. Further, uniqueness of indices is often a simple and effective check of data coherency.

Some Pandas interfaces, such as pd.concat() and pd.DataFrame.set_index(), provide an optional check of uniqueness with a parameter named verify_integrity. Surprisingly, by default Pandas disables verify_integrity.

>>> df.set_index('type', verify_integrity=True)
Traceback (most recent call last):
ValueError: Index has duplicate keys: Index(['lepton', 'quark'], dtype='object', name='type')

In StaticFrame, indices are always unique. Attempting to set a non-unique index will raise an exception. This constraint eliminates opportunities for mistakenly introducing duplicates in indices.

>>> f = sf.Frame.from_records((('muon', 0.106, -1.0, 'lepton'), ('tau', 1.777, -1.0, 'lepton'), ('charm', 1.3, 0.666, 'quark'), ('strange', 0.1, -0.333, 'quark')), columns=('name', 'mass', 'charge', 'type'))
>>> f
<Frame>
<Index> name    mass      charge    type   <<U6>
<Index>
0       muon    0.106     -1.0      lepton
1       tau     1.777     -1.0      lepton
2       charm   1.3       0.666     quark
3       strange 0.1       -0.333    quark
<int64> <<U7>   <float64> <float64> <<U6>
>>> f.set_index('type')
Traceback (most recent call last):
static_frame.core.exception.ErrorInitIndex: labels (4) have non-unique values (2)

No. 10: There and Back Again to Pandas

StaticFrame is designed to work in environments side-by-side with Pandas. Going back and forth is made possible with specialized constructors and exporters, such as Frame.from_pandas() or Series.to_pandas().

>>> df = pd.DataFrame.from_records([('muon', 0.106, -1.0, 'lepton'), ('tau', 1.777, -1.0, 'lepton'), ('charm', 1.3, 0.666, 'quark'), ('strange', 0.1, -0.333, 'quark')], columns=('name', 'mass', 'charge', 'type'))
>>> df
      name   mass  charge    type
0     muon  0.106  -1.000  lepton
1      tau  1.777  -1.000  lepton
2    charm  1.300   0.666   quark
3  strange  0.100  -0.333   quark

>>> sf.Frame.from_pandas(df)
<Frame>
<Index> name     mass      charge    type     <object>
<Index>
0       muon     0.106     -1.0      lepton
1       tau      1.777     -1.0      lepton
2       charm    1.3       0.666     quark
3       strange  0.1       -0.333    quark
<int64> <object> <float64> <float64> <object>

Conclusion

The concept of a "data frame" object came long before Pandas 0.1 release in 2009: the first implementation of a data frame may have been as early as 1991 in the S language, a predecessor of R. Today, the data frame finds realization in a wide variety of languages and implementations. Pandas will continue to provide an excellent resource to a broad community of users. However, for situations where correctness and code maintainability are critical, StaticFrame offers an alternative designed to be more consistent and reduce opportunities for error.

For more information about StaticFrame, see the documentation (http://static-frame.readthedocs.io) or project site (https://github.com/InvestmentSystems/static-frame).

DEV Community: Christopher Ariza

Type-Hinting DataFrames for Static Analysis and Runtime Validation

Requirements of a Generic DataFrame

Interfaces Defined with Generic DataFrames

Runtime Type Validation

Runtime Data Validation

The Expressive Power of TypeVarTuple

Utilities for Type Hinting

Conclusion

Memoizing DataFrame Functions: Using Hashable DataFrames and Message Digests to Optimize Repeated Calculations

Hash Functions and Hash Collisions

In-Memory Memoization

Creating a Message Digest from a DataFrame

Disk-Based Memoization

Conclusion

The Performance Advantage of No-Copy DataFrame Operations

No-Copy Operations with NumPy Arrays

The Advantages of No-Copy DataFrame Operations

Conclusion

One Fill Value Is Not Enough: Preserving Columnar Types When Reindexing DataFrames

Ten Reasons to Use StaticFrame Instead of Pandas

Why StaticFrame

No. 1: Consistent and Discoverable Interfaces

No. 2: Consistent and Colorful Display

No. 3: Immutable Data: Efficient Memory Management without Defensive Copies

No. 4: Assignment is a Function

No. 5: Iterators are for Iterating and Function Application

No. 6: Strict, Grow-Only Frames

No. 7: Dates are not Nanoseconds

No. 8: Consistent Interfaces for Hierarchical Indices

No. 9: Indices are Always Unique

No. 10: There and Back Again to Pandas

Conclusion

The Expressive Power of `TypeVarTuple`