Automatic Reporting in Python - Part 3: Packaging It Up

As outlined in my previous posts (Part I and Part II available on this fine website), the goal of this project is to make an automatic reporting tool.

In this series of guides, the outcome I'm shooting for is a single HTML page that allows me to interrogate and compare the output of machine learning models.

At the conclusion of the previous tutorial, the reporting tool was actually showing some genuine use! It could accept a number of .csv summaries of machine learning summary files and output a single .html page that presented the information is a form that was... functional.

There three main features that I'd like to add to the tool for now:

1. The report looks incredibly dull. Readability counts! We need to improve the a e s t h e t i c s of the report.
2. It's hard to dig into the tables - currently it's just 100 rows presented with no tools to search. Some basic search functionality would be grouse.
3. The datasets that we want to run the report with are currently hard-coded into the script. This needs to be split out to make the tool slightly more flexible.

A brief comment

Compared to the previous posts, this post is much more of an exploratory, learning experience - this post represents a neophyte's attempt to build a working tool, rather than a beautiful depiction of all that is possible. If you can see a much better approach for anything listed in this post, please feel free to share it with myself and all the other readers!

But without further ado, let's take a crack at improving the aesthetics.

---

Step Seven - Improving the Aesthetics

Those of you with an understanding of HTML pages might know what comes next: Cascading Style Sheets, known more commonly as CSS.

Taking your first steps down a path

As indicated above, the challenge in the context of this tutorial is that this is an enormous field that I personally am not actually particularly well-versed in, having only dabbled and hacked in this space. However, I am familiar with learning new things.

So, if this is your first real introduction to CSS, let me take you down the same path I would recommend in learning any new tech:

• Do some background reading on the fundamentals of CSS. Hit up Wikipedia. If you're keen, hit up the CSS standard! Never be afraid to Google "simple introduction to [topic]."
• As you read and explore, make note of potential good resources of future information. I would encourage everyone to take a look for an "Awesome List" relevant to your topic - and in case, the Awesome CSS List is here.
• With a basic understanding of what the hell is going on under your belt, play around to your heart's content with local files and implement as much as you can in local scratch files. In this case, create small (or large?) HTML pages and figure out how to structure the CSS neatly. I find this really helps to get to know some of the practical realities and challenges of working with the language before I start diving into frameworks.

Being a bit more mercenary

For self improvement, I find nothing beats spending the time working up solutions from scratch. Of course, if you're trying to hack together a solution for a business need, it may not behoove you to spend hours and days coming up with an elegant CSS framework from the beginning.

Instead, we may like to quickly jump off of someone else's CSS framework. Fortunately, there are a number of these available, with a great number of them focusing on being 'minimal'. A quick Google search should get you started down this path.

Integration

How is the CSS file to be integrated our report? There's a few options that are typically at play:

• Download the CSS file and keep it as external file. The advantage is that we have our .html and .css files neatly separated, and we have full control over both; the far more significant disadvantage is that now if we want to move our report around, we have to drag a bunch of .css files around with it.
• Use a Content Delivery Network (CDN) copy of the CSS file. Most frameworks will offer a CDN link for their file: this is essentially a link to an efficient, readily available copy of the data. The advantage is that you can get a CSS up and going in your page just by dropping a single link in the <head> section of your .html, no mussin' and fussin' with local files. The disadvantage is that you don't have control of the file, and an internet connection is required.
• A slightly more complex option is to have local copies of the CSS file, and then write them into the .html file. This could probably be done relatively easily and sustainibly if we got clever with our templating. The advantage is that we'd have our report in a single file, and it wouldn't require an internet connection to use; the disadvantage is that it is going to require a bit more effort to get set up. (This is commonly used approach when creating standalone versions of interactive pages. Write a Jupyter Notebook to .html, inspect the file, and you'll find all the CSS and JavaScript magic packaged up in the <head> section.)

At this early stage of prototyping, I prefer to use CDNs if possible. The advantage of being able to swap CSS frameworks just by changing a single line of code and not having to bother with local files is worth the cost of not being able to play with and edit the framework. Optimisation (in the form of being able to automatically integrate the CSS into the .html file) can come a little later.

To start with, I'm going to use Milligram, a lightweight little framework. To get started using the CDN method, I simply follow the provided instructions to integrate into the CDN. Under our templates/report.html file, I'll add the requisite links into the <head> section:

report.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>{{ title }}</title>
    <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Roboto:300,300italic,700,700italic">
    <link rel="stylesheet" href="//cdn.rawgit.com/necolas/normalize.css/master/normalize.css">
    <link rel="stylesheet" href="//cdn.rawgit.com/milligram/milligram/master/dist/milligram.min.css">
</head>
<!-- body section continues below... -->

And all of a sudden, our plain, early 90's looking webpage has been transformed into something a bit more pleasing to the eye:

But we note there's something still not quite right here - primarily, why does the page (and the table in particular) always take up the whole width of the window? Why doesn't this look right on mobile?

It turns out just adding a bunch of .css files isn't quite enough. We need to make sure the layout of our .html pages match what's expected by the .css layout.

HTML layouts

Like most topics in this space, the layout of your HTML page is a reasonably intuitive concept, while simultaneously being a problem that you spend years diving into. What makes it a bit more challenging is that despite there being a number of somewhat fragmentary explanations, I've struggled to find a simple and/or holistic to the field (although this explanation is currently my favourite gentle introduction, and the Mozilla guide appears to be quite thorough).

For brevity, I'm going to leave most of the further reading to you, the reader (sorry!), and instead focus on what Milligram expects.

If we inspect the code of the Milligram page, we'll see that within the <body>, we can see the HTML of the site is structured roughly as:

<body>
    <main class="wrapper">
        <header class="header">
            <section class="container">
        <section class="container">

Now, based on some of the readings in the above links, and assuming that this is the structure that Milligram is expecting, we can apply the same structure to our own report, giving us something like the following:

report.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <!-- Note the addition of the viewport! --> 
    <meta name="viewport" content="width=device-width, initial-scale=1.0, minimal-ui">
    <title>{{ title }}</title>
    <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Roboto:300,300italic,700,700italic">
    <link rel="stylesheet" href="//cdn.rawgit.com/necolas/normalize.css/master/normalize.css">
    <link rel="stylesheet" href="//cdn.rawgit.com/milligram/milligram/master/dist/milligram.min.css">
</head>
<body>
    <main class="wrapper">
        <header class="header">
            <section class="container">
                <h1>{{ title }}</h1>
                <p>This report was automatically generated.</p>
            </section>
        </header>
        {% for section in sections %}
        {{ section }}
        {% endfor %}
    </main>
</body>
</html>

summary_section.html

<section class="container" id="summary">
    <h2>Quick summary</h2>
    <h3>Accuracy</h3>
    {% for model_results in model_results_list %}
    <p><em>{{ model_results.model_name }}</em> analysed <em>{{ model_results.number_of_images }} image(s)</em>, achieving an
        accuracy of <em>{{ "{:.2%}".format(model_results.accuracy) }}.</em></p>
    {% endfor %}
    <h3>Trouble spots</h3>
    {% for model_results in model_results_list %}
    <p><em>{{ model_results.model_name }}</em> misidentified <em>{{ model_results.number_misidentified }} image(s)</em>.</p>
    {% endfor %}
    <p><em>{{ number_misidentified }}</em> misidentified image(s) were common to all models.</p>
</section>

table_section.html

<section class="container" id="{{ model }}">
    <h2>{{ model }} - Model Results</h2>
    <p>Results for each image as predicted by model <i>'{{ model }}'</i>, as captured in file <i>'{{ dataset }}'</i>.</p>
    {{ table }}
</section>

We had already structured this report to be a collection of largely independent collection of sections - we were even using this terminology! - so it's not a huge drama to add the <section> tags to the system.

Proof that this works

Run autoreporting.py and inspect the report - try it both at full-screen and simulating a mobile screen.

Progress! That's the benefit of using a well-made responsive layout.

GitHub status

Oooft. That was a lot of background reading for not a huge amount of code. All the same, the project should look like this.

---

Step Eight - Making Our Tables Interactive

Okay. So now we have our tables, and they look pretty good - the challenge is that they're not interactive. For instance, it would be wonderful to have the functionality to filter by a category, or drill down to a specific image.

Now, as with exploring CSS, we have a couple of options. Certainly, we can explore the option of creating all of this ourselves - there are plenty of examples around, and they're not terribly difficult. But, if we're being pragmatic (or pressed by business needs!) we can probably find a pre-built package of what we need.

After a bit of googling, I came across DataTables - this is a plugin for JQuery, a very common JavaScript framework. DataTables looks like it covers most of the functionality we need, and provides a wealth of extensions and plugins for any of the functionality we don't have. All in all, a promising candidate.

Implementing DataTables

Fortunately, it turns out that implementing DataTables is relatively straight forward. From the front page of the DataTables site, we can see that the general principles are that we need to:

• Although not spelled out explicitly - DataTables is a JQuery plugin - so first we need load the Jquery .js file.
• We'll then load the DataTables .js and .css files.
• Finally, we call the DataTables function, pointing it at the HTML id of the table we want to add the functionality to.

That's all relatively straightforward, with just some very solvable wrinkles:

1. Our tables don't have id tags to refer to.
2. We need a way to call the DataTables function and point it at the id of the table, in a way that fits with our templating system.

Let's address these one by one.

Importing the relevant files

Before we get to our wrinkles, let's hit our basics. As we imported our files from CDNs previously, we'll do the same for JQuery.

Let's update the <head> section of templates/report.html and add the links:

report.html

<!-- More above! -->
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{{ title }}</title>
    <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Roboto:300,300italic,700,700italic">
    <link rel="stylesheet" href="//cdn.rawgit.com/necolas/normalize.css/master/normalize.css">
    <link rel="stylesheet" href="//cdn.rawgit.com/milligram/milligram/master/dist/milligram.min.css">
    <link rel="stylesheet" href="//cdn.datatables.net/1.10.19/css/jquery.dataTables.min.css">
    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
    <script src="//cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js"></script>
</head>
<!-- More below! -->

Adding id tags to the tables

We need to add an id tag to the <table> objects in our report. How can we do this?

Well, let's work backwards from the templates/table_section.html file. Within that file, we note that we insert our fully-formed HTML tables via the {{ table }} insert.

The {{ table }} insert is generated in autoreporting.py when we call the get_results_df_as_html method of ModelResults class. This method takes the pandas DataFrame and converts it into a string of HTML using the DataFrame.to_html function.

If we inspect the docs for that function, we see that there's an optional argument table_id. Ah yeah, cool man! If we pass the model name as that argument, the HTML table will be generated with the id that we want. The ModelResults class already has model_name as an attribute, so we can include that:

class ModelResults:

    # ...

    def get_results_df_as_html(self):
        """
        Return the results DataFrame as an HTML object.
        :return: String of HTML.
        """
        html = self.df_results.to_html(table_id=self.model_name)
        return html

You can run autoreporting.py and inspect the tables that are generated to confirm that they do indeed have the model name as an id.

Easy! Just a matter of tracing it back from the end result of HTML to the actual source within the code.

Calling the DataTables function

We need to call the DataTables function listed above, pointing at the appropriate id as we've just generated. The actual code to call the function is pretty straightforward. The question is: where do we put it?

This challenge has a couple of bounding constraints on it.

• We need to call the function to generate the DataTable using the model name - something like this:

$(document).ready( function () {
    $('#VGG19').DataTable();
} );

• Traditionally, JavaScript is placed either in the <head> section, although this is a somewhat controversial discussion. Note that this is tradition - it will actually run anywhere.

This is a challenge because of the structuring of our template. We want to place the JavaScript call for DataTables in <head>, which is in templates/report.html. Right now, when report.html is rendered under the main() function in autoreporting.py, it doesn't know anything about the model names: the render() call only has arguments for title, the overall title of the report, and sections, a list of pre-rendered strings of HTML ready to be inserted into the document. We just need to modify autoreporting.py to pass in the model names and tweak report.html accordingly. We tweak our files thusly:

autoreporting.py

def main():

    # ...

    # Production and write the report to file
    f.write(base_template.render(
        title=title,
        sections=sections,
        model_results_list=[vgg19_results, mobilenet_results]
    ))

report.html

<head>
    <!-- Lots of calls above... -->
    <script>
        {% for model_results in model_results_list %}
            $(document).ready(function() {
                $('#{{ model_results.model_name }}').DataTable();
            } );
        {% endfor %}
    </script>
</head>

Bingo-bango: when we render report.html, the calls to render the DataTable functionality for each existing table is included. Nice!

And now, if we run autoreporting.py and inspect the output, we get something like this:

We can now order our tables, search for categories, filter by image names - we have some rich functionality available, with more available via extensions.

This is a huge advantage for a report! Imagine if you only wanted to check out the common themes of incorrect image categorisations, or rapidly narrow down on a specific image.

GitHub status

Your repo should look a little something like this.

---

Step Nine - Packaging it up

AKA Step the Last.

The good news: we have the functionality we want and need. We can take a .csv file or two and punch out an interactive report.

The bad news: we've hardcoded it to two files, VGG19_results.csv and MobileNet_results.csv, which limits the functionality.

The final step for this exploration is therefore to turn this hard-coded script into a tool that can be called from the command-line. We want to be able to call the report and an arbitrary number of .csv files and have the report spat-out. So if we called our script and specified the relevant .csv files, we'd get a report successfully written to the /outputs folder - looking a little like this on the command line:

$ python autoreporting.py VGG19_results.csv MobileNet_results.csv
Successfully wrote "report.html" to folder "outputs".

This can be accomplished by utilising command line arguments - or, to put it rather simply, the commands that follow the call to Python. (In the example above, the first argument is autoreporting.py, our script. The second and third command line arguments are VGG19_results.csv and MobileNet_results.csv, respectively.) We have a couple of main ways we can approach this:

• We can crunch the arguments manually, using sys.argv. There's absolutely nothing wrong with this approach, sys.argv is really quite simple to use.
• We can use a Parser like argparse, primarily to assist in generating help and error messages.

Because I've not used argparse before, I'm interested in giving it a go and testing it for these purposes.

Implementing argparse

Most everything we want to work with in argparse is handled within the main() call within autoreporting.py. To make this work, we're going to:

• Define and parse the arguments we're interested in (specifically, filepaths to the results .csv files), using argparse;
• Convert these filepaths into ModelResults objects that we can use to generate our reports;
• Adapt our existing code to output reports using these ModelResults objects.

So, first of all, we need to make sure that argparse is imported. It's been part of the standard library since Python 3.2.

autoreporting.py

import argparse
# ...

Next, within main(), we'll define the parser - we're saying how we want the command line arguments to be interpreted. This code is adapted pretty quick smart from the demo in the argparse docs. We actually only have one argument, by how argparse defines it - just the filepaths to our results .csv files. The key thing to note is that we set the nargs argument to "+", indicating that we can have a undefined number of arguments of this kind, but we do need at least one.

When we call parser.parse_args(), all the arguments are neatly returned as a Namespace object that makes the inputs very easy to access, as we'll see in the following steps.

# ...

def main():
    """
    Entry point for the script.
    Render a template and write it to file.
    :return:
    """
    # Define and parse our arguments
    parser = argparse.ArgumentParser(description="Convert results .csv files into an interactive report.")
    parser.add_argument(
        "results_filepaths",
        nargs="+",
        help="Path(s) to results file(s) with filename(s) '<model_name>_results.csv'."
    )
    args = parser.parse_args()

From args, the Namespace object, we can pull out the filepaths and use them to generate ModelResults objects.

args.result_filepaths holds a list of our filepaths, which we have indicated should point at filenames in the format <model_name>_results.csv. We use the os.path module to manipulate this filepath, extract the model name, and generate the ModelResults object, adding it into a list called model_results as we go.

This filename manipulation can look a little tricky, but inspect the doccies of os.path and you'll see it's mostly clever string manipulation. os.path is full of very, very useful functions that can save you a lot of time with common path manipulations, and help your code to work cross-platform!

    # Create the model_results list, which holds the relevant information
    model_results = []
    for results_filepath in args.results_filepaths:
        results_root_name = os.path.splitext(os.path.basename(results_filepath))[0]
        model_name = results_root_name.split("_results")[0]
        model_results.append(
            ModelResults(model_name, results_filepath))

The logic for the set intersection - how we figure out which images are common across all results files - has be changed to account for the fact that we now have an arbitrary number of ModelResults objects in a list.

To make this work, we quickly extract the misidentified_images property of each object using a list comprehension, and then calculate the intersection of sets based on this resulting list. (Note that we have to use a leading asterix (*) when we call set.intersection() so that each member of the list gets passed in as an individual argument).

    # Create some more content to be published as part of this analysis
    title = "Model Report"
    misidentified_images = [set(results.misidentified_images) for results in model_results]
    number_misidentified = len(set.intersection(*misidentified_images))

Everything below this point is relatively consistent with our previous version, but now we're taking advantage of the fact that we have our ModelResults objects already packed up into the model_results list.

    # Produce our section blocks
    sections = list()
    sections.append(summary_section_template.render(
        model_results_list=model_results,
        number_misidentified=number_misidentified
    ))
    for model_result in model_results:
        sections.append(table_section_template.render(
            model=model_result.model_name,
            dataset=model_result.dataset,
            table=model_result.get_results_df_as_html())
        )

    # Produce and write the report to file
    with open("outputs/report.html", "w") as f:
        f.write(base_template.render(
            title=title,
            sections=sections,
            model_results_list=model_results
        ))
    print('Successfully wrote "report.html" to folder "outputs".')

Oooft! With all of the explanations, this looks a little complex. However, when you compare this code to the previous commit, you'll see there's not a great deal that's actually significantly different here - we've really kept the core principles the same and just played with the packaging a bit.

GitHub status

Your project should look a little like this.

---

The End?

At the very start of the first post, I indicated that the goal of this project was to create an automatic HTML reporting tool, where the outcome was a single stand-alone HTML file, with info and interactivity.

Well, it's done! We've got a tool that can accept an arbitrary number of standard results files, and spit out a report that crunches them into an interactive format.

Take a breather, push your chair away from your desk, and pat yourself on the back. We've done what we set out to do!

Does that mean we're done? That depends, really.

What comes next?

At this point, we have a tool that works for a very narrow use-case, and assumes perfect inputs and perfect operation from the user. Now, if you're using a tool like this just for yourself, and the inputs to the tool are quite consistent, then that could in fact be perfectly satisfactory - so no more work to be done, you've got something that's fit for purpose.

But of course, there are any number of ways we can work to extend and harden this tool. As I was writing this tutorial, I made notes on some of these. Running losely from simpler to more complex, here are a few notes and ideas:

• Could we add a default command-line argument with argparse that allows us to specify the title of the report?
• Our .csv inputs need to be named in a perfectly consistent format. How can we restructure our inputs so that we can define the model name and not have it read from the filename?
• How can we make this a command-line script that can be run anywhere on our machine - not just in the folder the script is in? If our data is generated and stored elsewhere, it would certainly be more useful to be able to call autoreport on the terminal, rather than trace back to where the script is stored, for instance.
• In its current form, we're analysing images - can we add functionality to show the images we're analysing? This would be great for generating hypotheses for why a model failed.
• Our reports need an internet connection each time they're open. As part of the templating process, could we pull down the JavaScript and CSS files and embed them into our files?

This is a tiny sliver of the possible extensions - and this is not even to mention that there's plenty of refactoring and tidying to be done across the project. The job of improving your work is never done!

A thankyou and a call to action

This has been the first tutorial of this scope I've ever written. I've had a lot of fun doing so, and to paraphrase Sigur Rós, this has been a good beginning.

But I'm really keen to hear what parts of this you, the tutorial-reader, enjoyed and what parts were challenging or obscure. Feel free to drop a comment or send me a message on what worked and what didn't.

See you next time!

Comments

[Leif-Erik]

Hi, thanks for a great guide.

I am currently trying to get the argparse up and running in the script, but when I run it does not find the files. Is there any hardcoding of the filepath to the datafolder in the script that should have included?

[goyder]

Howdy! Glad you've found it useful.

If you could provide a full dump of the error code, I can better take a guess at the issue.

However, looking through the code, I now realise it assumes that an outputs folder is available in the folder. You could either:

• Manually create an outputs folder (not a thorough fix)
• Modify the code to check if the target folder exists, and create it if it doesn't, before writing to it.

[Leif-Erik]

Yeah it is really useful. I am a python noob, so it was like jumping out to swim in the deep end. But I have learned a lot by just playing around with it, trying to find out what everything does. Are you planning on doing a guide on how to include images/figures in a html report?

Below is the error I get.

Success When I have the files in the same folder as the script

PS C:\Users\lpe061\Dropbox\Python\Automatic Reporting> python Autoreport.py LAS318_results.csv LAS319_results.csv LAS320_results.csv
Successfully wrote "report.html" to folder "outputs".

When the files are in the "datasets" folder

PS C:\Users\lpe061\Dropbox\Python\Automatic Reporting> python Autoreport.py LAS318_results.csv LAS319_results.csv LAS320_results.csv
Traceback (most recent call last):
File "Autoreport.py", line 239, in
main()
File "Autoreport.py", line 202, in main
SampleResults(sample_name, results_filepath))
File "Autoreport.py", line 29, in init
self.df_results = csv_to_df(filepath) # Filesystem access
File "Autoreport.py", line 167, in csv_to_df
df = pd.read_csv(filepath, index_col=0)
File "C:\Users\lpe061\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 678, in parser_f
return read(filepath_or_buffer, kwds)
File "C:\Users\lpe061\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 440, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\lpe061\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 787, in __init_
self.make_engine(self.engine)
File "C:\Users\lpe061\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1014, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\lpe061\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1708, in __init_
self.reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 384, in pandas._libs.parsers.TextReader.cinit_
File "pandas_libs\parsers.pyx", line 695, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: File b'LAS318_results.csv' does not exist
PS C:\Users\lpe061\Dropbox\Python\Automatic Reporting>

[goyder]

Hey Leif,

Thanks for the comparison of functional/non-functional calls - gives me a lot more context on where the issue might be.

You'll need to provide a relative path to the datasets (relative to the script) to get it to run if you have have your datasets under the folder - i.e. run python Autoreport.py datasets/LAS318_results.csv datasets/LAS319_results.csv datasets/LAS320_results.csv. In its current form, the script won't search any subfolders.

(It's curious to me that the FileNotFoundError is returning a byte string, indicated the b in front of the filepath... that makes me raise an eyebrow.)

I hadn't planned on adding any additional examples around how to publish images in the reports, but in previous projects I've added similar features. The key questions that I worked through were:

• How do I present the images in the report? Do I present them all, just provide links, or have some kind of interactive tool?
• Where do I store the images for access? Do I keep a local copy of all the images (and if so, how do I manage moving the report around)? Should I push all the images to be hosted somewhere else - or even inside the .html file somehow?

These questions are a little outside of Python and more into web dev, which I am definitely not an expert in. They're all very interesting options to explore, however.