DEV Community: Agriya Khetarpal

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 7 and 8

Agriya Khetarpal — Mon, 31 Jul 2023 00:00:00 +0000

Hi there! This is the fifth blog in a series of seven blogs that document my journey of my Google Summer of Code project under the NumFOCUS umbrella organisation where I aim to improve the documentation infrastructure for PyBaMM, a Python package for working with physics-based battery models.

A summary of the tasks I completed

I am continuing to work on adding quality-of-life improvements to the documentation and also to refine it, i.e., to consolidate all its parts in one place so that it is more readable and maintainable for core users and developers respectively.

With the main requirements of my project and all of the stretch goals associated with it complete, my project is nearing its twilight phase. Therefore, we have decided to create a template for PyBaMM-based projects that will be available for the battery community to make the process of packaging and distributing code much easier, more details below:

Algolia docsearch (pybamm-team/PyBaMM #3159)
Generate teams (pybamm-team/pybamm.org #31)
Accessibility improvements (pybamm-team/pybamm.org #32)
Generate teams in the CI (pybamm-team/pybamm.org #34)

Some other, shorter PRs that I opened for minor fixes and improvements:

Configure versions.json and remove the version switcher dropdown (pybamm-team/PyBaMM #3108)
Ensure setup-python outputs are always included in nox cache keys (pybamm-team/PyBaMM #3157)
Restructure notebook galleries (pybamm-team/PyBaMM #3167)
Enable downloadable PDF, EPUB, and zipped HTML formats on Read the Docs (pybamm-team/PyBaMM #3182)
Add docstrings for nox sessions (pybamm-team/PyBaMM #3183)

The website is at the verge of its completion at the time of writing, and will be showcased officially to the battery community soon!

Algolia DocSearch v3

We recently applied for Algolia's search-as-you-type service that is available for eligible open-source projects and technical blogs. It replaces the older, slower search available via the PyData Sphinx Theme and Sphinx.

Try it out on the latest PyBaMM documentation website!

// Detect dark theme var iframe = document.getElementById('tweet-1684222979008507904-738'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1684222979008507904&theme=dark" }

Generating a page highlighting the team members and contributors

Truly, recognising the work of everyone who contributed to PyBaMM is the insignia of legends.

I added a teams page on the PyBaMM website at https://pybamm.org/teams/. It lists the maintainers, emeritus maintainers, maintainer trainees, and contributors one after the other. The scripts/generate_teams.py file, also linked against the target make teams, when run, gets data from the GitHub API and adds details about the teams in their respective HTML pages.

I also added a simple workflow to accompany this script; it runs this script set according to a CRON job weekly with GitHub Actions, automatically creates a pull request if there are changes, and asks for reviews from the four maintainers of the repository.

Note: PyBaMM follows the all-contributors specification to recognise contributions beyond code (bug reports, documentation, fiscal sponsorships, et cetera). To see the full list of contributors, see the Contributors section in the PyBaMM repository README.

Setting up a new project: `pybamm-cookiecutter`

The mentors and I have decided to implement a standardised template for projects that use PyBaMM. This template is intended to be used with projects that use PyBaMM and serve as an educational resource as well.

Once completed, this template can be used with standard templating engines such as cookiecutter and cruft, and possibly copier in the future. This way, people can distribute Jupyter notebooks as examples, utility functions and custom source code that build upon PyBaMM as a dependency, scripts for code releases for a preprint or a paper that uses PyBaMM in a collaborative academic setting, and much more.

The build-backends that are to be supported are hatch, flit. Support for the backends setuptools621 and setuptools is also planned. The repository will remain opinionated and will also support adding custom tools such as the PyBaMM's favourite pre-commit hooks, such as ruff, black, and nbQA.

Optionally, users of the template will also be able to load data through python-dotenv and a .env file in the root directory that will containing the path to data-related files, which will serve the purpose of keeping code and data separate.

Last but surely not the least, this will provide a unified method for adding third-party parameter sets through a combination of a parameters/ folder and entry points in pyproject.toml.

Please feel free to comment on the issue thread in the pybamm-cookiecutter repository and share your thoughts. Keep watching this space for more updates!

Stay tuned

I will be posting these blogs fortnightly as a part of my GSoC project. Thoughts, comments, constructive criticism, and insights will be greatly appreciated. See you soon on the next one! For updates, please follow me on my social media handles:

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 5 and 6

Agriya Khetarpal — Thu, 13 Jul 2023 22:12:58 +0000

Hi there! This is the fourth blog in a series of seven blogs that document my journey of my Google Summer of Code project under the NumFOCUS umbrella organisation where I aim to improve the documentation infrastructure for PyBaMM, a Python package for working with physics-based battery models.

A summary of the tasks I completed

Use nbsphinx to embed Jupyter notebooks (pybamm-team/PyBaMM #3043) was now merged.
Display when pages were last updated in the docs (pybamm-team/PyBaMM #3089)
Enable multithreaded documentation builds (pybamm-team/PyBaMM #3097)
Add an option to download Jupyter notebooks from readthedocs (pybamm-team/PyBaMM #3076)
Add floating window tooltips for cross-references in the docs (pybamm-team/PyBaMM #3083)
Generate inheritance diagrams for models and submodels in the documentation (pybamm-team/PyBaMM #3074)
Cache nox environments and refactor unit tests in CI (#3079)

PyBaMM also recently migrated from the DSL-based tox to the Python-based (infrastrucure-as-code) nox for running the test suite, thanks to my brilliant GSoC workmate Arjun. Many improvements to the testing infrastructure are planned, and I took on the task for one of them (more to come in the future!). I like this article titled Why I Like Nox by the PSF Fellow Hynek Schlawack, that somewhat echoes the sentiment of why we decided to switch to it.

Some PRs in the pybamm.org website repository that I opened, which will be completed in the coming weeks:

Generate teams (pybamm-team/pybamm.org #31)
Accessibility improvements (pybamm-team/pybamm.org #32)

The website is at the verge of its completion and will be showcased to the battery community and ecosystem soon!

Gorgeous galleries for notebooks

In an attempt to incorporate the notebooks in the documentation, I had previously started adding the notebooks to the documentation. All the notebooks have been ported now and can be viewed on the Example Notebooks page in the PyBaMM documentation.

I also configured the sphinx-gallery extension to generate thumbnails for each notebook. See it in action below:

All the notebooks can be accessed through Google Colab as before. The example scripts will be ported in the coming weeks in order to finalise the thumbnails.

Download the notebooks at ease

The nbsphinx extension also includes support for prologues and epilogues, i.e., text that can be embedded at the Sphinx builder runtime at either the top or the bottom of a particular page—in this case, the notebooks pages. I added an nbsphinx_prolog configuration value in docs/source/conf.py (the Sphinx configuration file) in this manner:

nbsphinx_prolog = r"""

{% set readthedocs_download_url =
'https://docs.pybamm.org/en/latest/' %}

{% set doc_path = env.doc2path(env.docname, base=None) %}

.. raw:: html

    <div class="admonition tip">
        <p class="admonition-title">
            Tip
        </p>
        <p>
            You may
            <a href="{{ readthedocs_download_url | e }}{{ doc_path | e }}"
            target="_blank" download>
            download this notebook</a> and run it offline.
        </p>
    </div>

"""

The combination of raw HTML and Jinja templating, indubitably, allows us to do many useful things with Sphinx. Users can use the aforementioned Open in Colab button as well to run the notebooks in the browser.

Here is an image of the admonition note in action:

It can be tried on any of the notebooks on the Example Notebooks page in the documentation.

Version control history for the documentation pages

...so that one can know when they messed things up, or when an archaic and unkempt API description was last taken care of. Essentially the git blame for documentation hosted on Read the Docs.

I added the sphinx-last-updated-by-git extension and configured it to be in the end of the footer for each page using the html_theme_options configuration value. The doctests did create some errors because GitHub Actions does not clone the PyBaMM repository to its full depth; it only pulls the latest 50 commits in its checkout step. A similar issue occurred on Read the Docs as well, which was ultimately fixed by adding a custom build command for the post-checkout step. This way, a complete rendition of the Git version control history is downloaded while hosting.

Speeding up the documentation builds

Now that the nbsphinx PR was merged, the sphinx-build commands for the documentation were between 1.5x–2x slower, which subsequently became a menace when building the documentation locally and while running the doctests. The reason behind this was that nbsphinx was re-building all the notebooks pages and uses Pandoc under the hood to convert cell-based Markdown to .rst formats. We had decided not to use the other notebooks rendering solution, i.e., a rival project called MyST-NB, because it was not compatible with sphinx-gallery. Its advantage would have been that it has native caching by default and would not slow down the documentation builds by a significant amount.

In a short PR, eventually enabled multithreading for building the documentation using the -j auto flag in SPHINXOPTS, and the results bestowed, to my delight, much faster build times (~40%)! The time came down from the previous ~3.5 minutes to ~2 minutes. Please see the mentioned issue above for more details.

Traversing the maze of PyBaMM's many, many battery models with inheritance diagrams

PyBaMM comes with an extensive library of models and submodels, such as the Doyle-Fuller-Newmann model, the Single Particle Model, lead-acid models, submodels for oxygen diffusion, particle cracking models, and many more. It is often difficult for inexperienced users who might still be learning the ropes of how to use PyBaMM effectively. To tackle this issue, I wrote a simple Sphinx extension and added it to the docs/sphinxext folder. It relies on appending and/or inserting particular lines of code through the sphinx.ext.autodoc extension, which provides functionality to include documentation for classes and methods by extracting their docstrings.

I used a dropdown from the sphinx-design extension to make the inheritance diagram collapsible upon request and through the click of a button. The extension uses graphviz and sphinx.ext.inheritance_diagram as dependencies to create the diagram and can be configured in conf.py. Currently, the extension creates these plots in .PNG format for the models and the submodels.

Here is an example for the Many Particle Model (MPM):

Floating-window tooltips for hyperlinks in the documentation

It is also necessary to sometimes have a quick look at a particular parameter for a method or a class, but one does not want to search for it and go to the results. To enhance the productivity for PyBaMM users and developers, I added and configured a Sphinx extension sphinx-hoverxref, which adds tooltips that can be viewed upon hovering on a particular link within the same tab. See it in action below in the Interpolant class in PyBaMM, which is used heavily in the expression tree and its operators:

Tasks to focus on in the coming weeks

Implement a search-as-you-type feature for the hosted documentation for relatively faster search results in comparison to the default JavaScript-based search engine
Make the documentation more maintainable using sphinx.ext.autosummary

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 3 and 4

Agriya Khetarpal — Thu, 13 Jul 2023 22:11:08 +0000

Hi there! This is the third blog in a series of seven blogs that document my journey of my Google Summer of Code project under the NumFOCUS umbrella organisation where I aim to improve the documentation infrastructure for PyBaMM, a Python package for physics-based battery models.

A summary of the tasks I completed

With my GSoC project now in full swing and the website almost complete, I ventured into the foray of improving the documentation infrastructure for PyBaMM as well. You may read the API documentation and the user guide for getting started with PyBaMM at https://pybamm.readthedocs.io.

In these weeks, I opened many issues and PRs, listed as follows. These involved working not only with the documentation but other infrastructure-related quality of life improvements too!

Restructure pages on the navigation bar and reformat contents (pybamm-team/pybamm.org #22)
Reformat news to be single-page only (pybamm-team/pybamm.org #24)
Add inline-tabs for Sphinx docs (pybamm-team/PyBaMM #3022)
Use sphinxcontrib-bibtex for citing references in docs (pybamm-team/PyBaMM #3034)
Use nbsphinx to embed Jupyter notebooks (pybamm-team/PyBaMM #3043) — this is currently not merged at the time of writing, though I am glad that I was able to make it work, since this was also one of the stretch goals of my project!

Other small contributions, related to repository infrastructure and maintenance:

Use Python 3.11 for generating coverage in CI (pybamm-team/PyBaMM #3016)
Remove backport importlib_metadata from required dependencies (pybamm-team/PyBaMM #3050)
[Bug]: Conversion to scalars for arrays with ndim>0: NumPy 1.25 deprecation (Python 3.9–3.11)(pybamm-team/PyBaMM #3052) and its corresponding PR: Extract single element from np.ndarray when converting to scalar (pybamm-team/PyBaMM #3055)

Inline-tabs for Sphinx

This was a short PR to add the sphinx-inline-tabs extension. It removes redundancy for code blocks and those sections across the documentation and makes them much neater than before to the users' eyes:

You may view other examples of the extension in action on the PyBaMM Installation guide(s).

Keeping in-line textual citations in the documentation instead of hard-coding them

PyBaMM has a Citations class which reads information from a CITATIONS.bib file to register citations when users run a particular simulation and use certain models and solvers. Similarly, PyBaMM also offers a variety of common parameter sets referenced from values in papers over the past two decades of research and beyond. They were hard-coded in the docstrings for the model classes and the parameters.rst file, which made the citations prone to mistakes, spelling errors, and even validation errors in case a reference is not formatted correctly.

With the sphinxcontrib-bibtex extension; I was able to parse the CITATIONS.bib file to extract references from it, and then use the directives provided, such as :footcite:t: and .. footbibliography:: to display all references on a page programmatically. This serves the purpose of automation: I also added BibTeX references to some missing papers in this PR. An example of how the textual citation references look like is shown below.

More examples can be viewed at the Parameter Sets page in the PyBaMM documentation.

Galleries are always pretty, whether for Dadaist art... or for Jupyter notebooks

Insipired by the scikit-learn documentation, we decided to embed a gallery for PyBaMM's myriads of Jupyter notebooks that contain examples, beginner tutorials, advanced examples, model demonstrations, and more. This was particularly challenging because Sphinx still does not support relative links in the table of contents (the .. toctree:: directive), so the only options were to either use a cumbersome nbsphinx-link extension along with nbsphinx, or to move all the example notebooks from the examples/ folder in the root directory to the docs/source/examples/ directory. PyBaMM also uses Sphinx-based doctests, which also started failing due to errors in how the Markdown contents inside .ipynb notebooks are formatted—fixing them also took some time, since some notebooks had widgets, which caused a frozen module error with Python 3.11 and the doctests continued to run indefinitely in a PR for five hours and counting!

There were also other problems, such as the MathJax rendering for the model equations going haywire in many a location in likewise many a notebook when building the documentation—which were fixed via an agnostic syntax style for how equations are displayed, rather than KaTeX.

Watch out for the next blog to see the result and the forthcoming troubles (of course)!

Tasks to focus on in the coming weeks

Further improvements to the documentation infrastructure in general
Getting the revamped PyBaMM website (https://pybamm.org) ready to be shown to the world
Embedding the PyBaMM example scripts in the documentation

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 1 and 2

Agriya Khetarpal — Thu, 13 Jul 2023 22:08:03 +0000

Hi there! This is the second blog in a series of seven blogs that document my journey of my Google Summer of Code project under the NumFOCUS umbrella organisation where I aim to improve the documentation infrastructure for PyBaMM, a Python package for working with physics-based battery models.

A summary of the tasks I completed

Use Lychee to check for broken links (#3)
Add pre-commit support (#4)
Improvements to pre-commit (#5)
Netlify deployment infrastructure (#8)
Add news pages from PyBaMM blogspot (#10)
Custom PyBaMM 404 page (#12)
Favicon for PyBaMM (#14)
Donations and sponsors (#16)
Twitter feed, institutional partners, and about PyBaMM (#19)

I have started a summary issue to keep track of my progress, which I will be editing on a regular basis. Please feel free to suggest an improvement in the summary issue thread and I will be glad to look into it!

Building blocks: the initial infrastructure for the new pybamm.org website

We started the project by creating a repository for the website in the pybamm-team organisation. To set up the website itself, we planned to use the Scientific Python Hugo theme; also used by NumPy and SciPy, and by some other Python projects in the Scientific Python community.

To host the website, we preferred Netlify to GitHub Pages owing to the fact that it would be easier to see collaborative previews on changes made via pull requests. The development version of the website can be accessed at https://pybamm-developer-preview.netlify.app/. Please feel free to open an issue if you see something broken!

The war against broken links: validating and checking them with a workflow

I also added my favourite link checker (Lychee in a GitHub Action to check for broken links and. It is incredibly fast and written in Rust — which is also the reason it is one of my favourites! It also serves as a personal reminder to learn Rust some day. The workflow currently runs as a CRON job at 0300 hours UTC scheduled daily. The link checker also has features such as excluding selected links or status codes, adding user agents, and an option to enable caching.

`pre-commit` hooks are cool!

I added pre-commit hooks for developers to use, such as ruff—an extremely fast Python linter written in Rust, black as an autoformatter, codespell to fix common spelling errors and some other pre-commit hooks to remove trailing whitespaces and lint Markdown files. This way, developers working on the website locally can run

pre-commit install
pre-commit run --all-files

to fix all the files with the pre-commit hooks before committing, or let git do it for them.

In addition, we enabled support for the pre-commit.ci bot to automatically perform these operations wherever possible on pull requests and commits.

For every piece of art, there is another that does it better

I ported over many pages and items from the previous PyBaMM website, redesigning and editing them as neccessary:

The Donations page for enlisting sponsorships and ways to contribute PyBaMM fiscally
Blogs by the PyBaMM team, in a newsroom page (https://pybamm.org/news/)
A custom 404 page
A favicon for the website, which is essentially the PyBaMM logo
A Twitter feed for PyBaMM, and
The home page and other pages, which contain training material and institutional partners

Tasks to focus on in the coming weeks

Improving the structure for the newly-set-up PyBaMM website and reordering certain pages
Adding over the remaining pages, and
Making sure the website adheres to the W3C Web Content Accessibility Guidelines specification

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Community Bonding

Agriya Khetarpal — Mon, 29 May 2023 00:00:00 +0000

Introduction

Hi there! This is the first blog in a series of seven blogs that I will be writing to document my journey as I cruise through the coding period of the Google Summer of Code (GSoC) 2023 open-source programme. I will be working with PyBaMM under the NumFOCUS umbrella for the next twelve weeks to improve its documentation infrastructure and to revamp its website available at pybamm.org.

About PyBaMM

PyBaMM (Python Battery Mathematical Modelling) is an open-source Python package for running battery simulations with a mission to accelerate and partake in cutting-edge battery modelling research. It comes with a library of popular battery models and parameter sets, specialised tools for visualisations and interpretability, and a suite of incredibly fast and robust solvers to solve electrochemical differential algebraic equations.

The PyBaMM logo in standard Python blue and yellow colours displaying a pack of batteries

PyBaMM is supported by institutional partners such as the Faraday Institution and is sponsored by NumFOCUS since 2022, in a mission to advance open science by means of industry-wide and academic collaborative practices.

This is a long-size project offered by PyBaMM, and I will be working over the course of 350 hours.

My journey so far

This section describes how I came to be associated with PyBaMM since the beginning to the end of the Community Bonding period.

Interests

I was scourging through issues while contributing to Hacktoberfest last year, and PyBaMM caught my eye–an open-source project in Python for scientific research! I had always loved my readings of electrochemistry in high school: I quickly submitted a pull request then to drop support for Python 3.7.

Contributing

I followed with more pull requests for short fixes in the documentation, a pull request to introduce helper functions in order to autogenerate docstrings, a newer link checker in the CI, and more. I do not possess an understanding of battery physics; however, I continued pushing changes wherever I could. Most of my contributions were for quality-of-life improvements, and I followed suit by joining the PyBaMM Slack organisation looking for chances to help out and learn as much as I can. My fear of large codebases has subsided ever since and I can say that I am familiar with PyBaMM's workflows and how they all integrate together with each other on pull requests, releases, nightly runs, benchmarks, and more.

GSoC 2023: Community Bonding

In almost as if a recognition for my efforts, I have been selected as a GSoC student this year and I am elated to be mentored by two of the PyBaMM maintainers Valentin Sulzer and Saransh Chopra.

The Community Bonding period ended on 28 May, 2023, culminating with an invitation to join PyBaMM's GitHub organisation as a member.

Footnotes

I am eager to start working as soon as possible and get my hands dirty on some code! I would also love to build a substantial sense of camaraderie with my mentors and the PyBaMM developers over the course of the next twelve weeks.

References

A link to my GSoC proposal: [PyBaMM] – Documentation (GSoC 2023), and my GSoC project on the Google Summer of Code website

DEV Community: Agriya Khetarpal

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 7 and 8

A summary of the tasks I completed

Algolia DocSearch v3

Generating a page highlighting the team members and contributors

Setting up a new project: pybamm-cookiecutter

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 5 and 6

A summary of the tasks I completed

Gorgeous galleries for notebooks

Download the notebooks at ease

Version control history for the documentation pages

Speeding up the documentation builds

Traversing the maze of PyBaMM's many, many battery models with inheritance diagrams

Floating-window tooltips for hyperlinks in the documentation

Tasks to focus on in the coming weeks

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 3 and 4

A summary of the tasks I completed

Inline-tabs for Sphinx

Keeping in-line textual citations in the documentation instead of hard-coding them

Galleries are always pretty, whether for Dadaist art... or for Jupyter notebooks

Tasks to focus on in the coming weeks

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Weeks 1 and 2

A summary of the tasks I completed

Building blocks: the initial infrastructure for the new pybamm.org website

The war against broken links: validating and checking them with a workflow

pre-commit hooks are cool!

For every piece of art, there is another that does it better

Tasks to focus on in the coming weeks

Stay tuned

GSoC 2023 @ PyBaMM, NumFOCUS: Community Bonding

Introduction

About PyBaMM

The PyBaMM logo in standard Python blue and yellow colours displaying a pack of batteries

My journey so far

Interests

Contributing

GSoC 2023: Community Bonding

Footnotes

References

Stay tuned

Setting up a new project: `pybamm-cookiecutter`

`pre-commit` hooks are cool!