Dynamically Adjust Your Kubernetes Pod Log Levels with Ease

Changing pod log levels at runtime is essential for monitoring and troubleshooting your Kubernetes applications. By adjusting the log level of a pod, you can control the amount and type of information that is logged, making it easier to pinpoint issues and identify root causes.

For example, during normal operations, you may want to log only essential information to conserve storage space and minimize clutter. However, when a problem arises, you may need to increase the log level to capture more detailed information about the issue.

Traditionally, changing log levels required redeploying pods, which can be time-consuming and disrupt application availability. With the ability to change log levels at runtime, you can make adjustments without downtime, reducing the impact on users and maintaining service availability.

Overall, dynamic log level management helps streamline troubleshooting and improves operational efficiency, enabling you to better monitor and optimize your Kubernetes applications.

Proposed solution

The Pod-Log-Level library is designed to work with k8s.io/klog/v2, and it allows you to dynamically change the log level of your pods without causing any downtime or interruptions.

To use the library, you first need to deploy the LogSetting CRD into your Kubernetes cluster:

kubectl apply -f https://raw.githubusercontent.com/gianlucam76/pod-log-level/main/config/crd/bases/open.projectsveltos.io_logsettings.yaml

Then, you can import the library and register it with your application using the provided code snippet

        import "github.com/gianlucam76/pod-log-level/lib"
...
    lib.RegisterForLogSettings(ctx,
        "<YOUR POD NAMESPACE>", "<YOUR POD IDENTIFIER>", <logr.Logger>,
        <cluster *rest.Config>)

Make sure to specify the correct pod namespace and identifier and ensure that the associated ServiceAccount has the necessary permissions to get, list, and watch LogSetting CRD.

Once you have registered the library, you can use the CLI that comes with it to change the log level manually or use the API to change the default log levels for your pods.

For instance, to set log level for a specific pod to info:

./bin/helper log-level set --namespace=projectsveltos --identifier=SveltosManager --info

You can also see all current settings

./bin/helper log-level show                                                             
+---------------------+----------------------+--------------+
| COMPONENT NAMESPACE | COMPONENT IDENTIFIER |  VERBOSITY   |
+---------------------+----------------------+--------------+
| projectsveltos      | SveltosManager       | LogLevelInfo |
+---------------------+----------------------+--------------+

By using the Pod-Log-Level library, you can streamline troubleshooting and optimize your Kubernetes applications more efficiently, reducing the impact on users and maintaining service availability.

For more info, please refer to README.