DEV Community

Cover image for Silence Kedro Logs
Waylon Walker
Waylon Walker

Posted on β€’ Edited on β€’ Originally published at waylonwalker.com

3 1

Silence Kedro Logs

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output.

Silence a Python log

First, how does one silence a python log? Python loggers can be retrieved by the logging module's getLogger function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless it's essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways.

python logging levels

Level Numeric value
CRITICAL 50
ERROR 40
WARNING 30
INFO 20
DEBUG 10
NOTSET 0

Get or Create a logger

Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in.

logger = logging.getLogger('kedro')
Enter fullscreen mode Exit fullscreen mode

πŸ”₯ If a logger doesn't exist under the passed in name, it will create one for you.

Set Level

Once we get the logger, we need to silence it by setting the log level. Typically it's not appropriate to completely turn off loggers as you would still want information in the case of a complete failure. If you are building a cli such as one that prints out the pipelines to the console, you may not want to see logs that happen during regular operation as this would make it more challenging to integrate with other shell applications.

logger.setLevel(logging.ERROR)
Enter fullscreen mode Exit fullscreen mode

⚠ Be sure to leave some logging left. After the point of error, you are not going to get a clean output anyways. So let the user see what happened.

It is possible to set the log level before kedro even registers the logger, if there is no logger currently setup under getLogger, it will create one.

Silent all kedro loggers

As of kedro==0.17.3 this function covers every logger issued by kedro. I generated this list of known_kedro_loggers by looking through their codebase and filling in a few others I found by running it.

def silent_loggers() -> None:
    """All logs need to be silent in order for a clean kedro diff output."""
    known_kedro_loggers = [
        "ProfileTimeTransformer",
        "hooks_handler",
        "kedro.__init__",
        "kedro",
        "kedro.config",
        "kedro.config.config",
        "kedro.extras.decorators.memory_profiler",
        "kedro.framework.cli",
        "kedro.framework.session.session",
        "kedro.framework.session.store",
        "kedro.framework.session",
        "kedro.io.cached_dataset",
        "kedro.io.data_catalog",
        "kedro.io",
        "kedro.journal",
        "kedro.pipeline",
        "kedro.pipeline.decorators",
        "kedro.pipeline.node",
        "kedro.pipeline.pipeline",
        "kedro.runner",
        "kedro.runner.runner",
        "kedro.versioning.journal",
        "py4",
    ]
    for logger in [
        *known_kedro_loggers,
        *list(logging.root.manager.loggerDict.keys()),  # type: ignore
    ]:
        logging.getLogger(logger).setLevel(logging.ERROR)
Enter fullscreen mode Exit fullscreen mode

This function comes right from a plugin I am currently working on kedro-diff. Check it out, give it a star, and watch it for release.

https://waylonwalker.com/what-is-kedro/

Not familiar with kedro, check out this article to see what it's all about.

Master the log

Python logs can seem super confusing at first, understanding how to get a logger and set its level are the first steps to mastering it.


Check Out These Related Posts

https://waylonwalker.com/if_name_main/

https://waylonwalker.com/install-micromamba/

https://waylonwalker.com/kedro172_replit/

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

πŸ‘‹ Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay