Pandas' Classy Styler

I knew it could be faster, but I could not find a way of doing it... Hang on, let me start at the beginning.

I have been working on a largish data set that gets pivoted down to something quite small. About 20 rows by 30 columns. Quite a bit of the data was empty and the figures were just integers. I had to display it in a table, with one row and some rows showing up in yellow (the totals or running totals), and some cells (matching a non-visible criteria), in red.

I thought this was easy. You pivot the table, get the Styler, apply formatting, once for the yellow rows and once for the red cells, and then you render the whole thing to HTML. Easy... but apparently slow.

It must have been my apply statements that were slow (commenting them out made the render run much faster). I managed to reduce the runtime down a bit by supplying a subset. Not a big win, but something.

Next I figured the computation of the red cells were too slow, so I did it before hand, having an exact same-shaped DataFrame indicating which to format as red. The apply for this part could then simply return the full DataFrame, pretty much as is (no computation required). This did not lead to much of a performance improvement either.

I kept on bumping my head. I looked at the output, and say style a lot. I knew what I had to do... I had to look at the actual code.

I started off my looking at the template. This part shows that there are attributes and class. Well, the class was there, but not what I needed it for.

pandas/pandas/io/formats/templates/html.tpl (v0.23.x)

...
<tbody>
{%- block before_rows %}{%- endblock before_rows %}
{%- for r in body %}
{%- block tr scoped %}
<tr>
{%- for c in r %}
{%- if c.is_visible != False %}
<{{ c.type }} id="T_{{ uuid }}{{ c.id }}" class="{{ c.class }}" {{ c.attributes|join(" ") }}>{{ c.display_value }}</{{ c.type }}>
{%- endif %}
{%- endfor %}
</tr>
{%- endblock tr %}
{%- endfor %}
{%- block after_rows %}{%- endblock after_rows %}
</tbody>
...


And so it I. I eventually tracked down a part where there is some mention of a class... But I wasn't getting a class produced, just styling. Class must be better than styling surely, so how do I do that? I kept on looking at the code and come to the conclusion that nobody uses the class part of this code. Wait... what...? Nobody uses this? Nobody uses this. Nobody. Uses. This.

What is This? Well, it is seemed to be a same-shaped list of lists that tells the styler which class to use, if any. If only there was a way of getting my pre-calculated styles into that variable before it being rendered.

One monkey-patch later, this thing produces cleaner HTML and rendered much faster.