Python logger

Context

There are too many posts defining how to configure python logs and some are too noise others are incomplete.

An example with json logger at the end

So here’s my version of it that just works and suits my purposes well

Example with plain text output

logger = logging.getLogger(__name__)
# Change this level according to your needs
logger.setLevel(level=os.getenv("LOG_LEVEL", "INFO"))
logger.propagate = False
stdout = logging.StreamHandler(stream=sys.stdout)
# Default level for root logger is NOTSET which means all messages will be logged
# I prefer having this as is to avoid two places controlling what gets to output
# Leaving this to logger.setLevel to define what should be printed is more flexible undless devs are not allowed to control this (for log volume purposes, for example)
# https://docs.python.org/3/library/logging.config.html
# stdout.setLevel(logging.DEBUG)
fmt = logging.Formatter(
    "%(name)s: %(asctime)s | %(levelname)s | %(filename)s:%(lineno)s | %(process)d >>> %(message)s"
)
stdout.setFormatter(fmt=fmt)
logger.addHandler(stdout)


# YOUR CODE BELOW

# Database connection
import psycopg2
from psycopg2.extras import RealDictCursor

logger.info(
    f"Connecting to database '{rds_host}' on port '{rds_port}' as user '{rds_user}' to database '{rds_database}'"
)
logger.debug("Debug message when creating connection to database")

connection = psycopg2.connect(
    host=rds_host,
    port=rds_port,
    user=rds_user,
    password=rds_password,
    database=rds_database,
    cursor_factory=RealDictCursor,
)

Sample output for log level set as INFO :

__main__: 2024-06-25 06:15:12,171 | INFO | main.py:39 | 61869 >>> Connecting to database 'mydbinstance.cjzjxjzjzjzj.us-west-2.rds.amazonaws.com' on port '5432' as user 'postgres' to database 'postgres'

Sample output for log level set as DEBUG :

__main__: 2024-06-25 06:17:38,664 | INFO | main.py:39 | 62437 >>> Connecting to database 'mydbinstance.cjzjxjzjzjzj.us-west-2.rds.amazonaws.com' on port '5432' as user 'postgres' to database 'postgres'
__main__: 2024-06-25 06:17:38,664 | DEBUG | main.py:42 | 62437 >>> Debug message when creating connection to database

Example with json output

Using python-json-logger dependency

logger = logging.getLogger(__name__)
# Change this level according to your needs
logger.setLevel(level=os.getenv("LOG_LEVEL", "INFO"))
logger.propagate = False
log_handler = logging.StreamHandler(stream=sys.stdout)
# Default level for root logger is NOTSET which means all messages will be logged
# https://docs.python.org/3/library/logging.config.html
# stdout.setLevel(logging.DEBUG)
fmt = "%(name)s: %(asctime)s | %(levelname)s | %(filename)s:%(lineno)s | %(process)d >>> %(message)s"
formatter = jsonlogger.JsonFormatter(fmt=fmt)
log_handler.setFormatter(fmt=formatter)
logger.addHandler(log_handler)

{"name": "__main__", "asctime": "2024-07-01 15:28:07,119", "levelname": "INFO", "filename": "main.py", "lineno": 42, "process": 60945, "message": "Connecting to database 'mydbinstance.cjzjxjzjzjzj.us-west-2.rds.amazonaws.com' on port '5432' as user 'postgres' to database 'postgres'"}
{"name": "__main__", "asctime": "2024-07-01 15:28:07,119", "levelname": "INFO", "filename": "main.py", "lineno": 207, "process": 60945, "message": "Starting execution..."}