DEV Community: Blitzcode

W1203: logging-fstring-interpolation (Solved)

Blitzcode — Sun, 21 Jan 2024 07:45:59 +0000

A few days back, I was implementing a feature, and when I was done and happy with my work, I decided to commit the code. We have enabled pre-commit hooks in our repository to maintain some coding formats and reduce the burden on the folks reviewing the code. One of the code checkers that run as part of the pre-commit check is Pylint. Pylint gave me the warning 'logging-fstring-interpolation (W1203)' and the pre-commit check kept failing. Even though I knew what needed to be done to fix the warning, being me, I could not just leave it be. I wanted to know the why part of it.

Other Pylint warnings related to the logging module are:

logging-not-lazy / W1201
logging-format-interpolation / W1202

Pylint

A little introduction about pylint. Pylint is a static code analyzer, it analyses your code without actually running it. Pylint looks for potential errors, gives suggestions on coding standards that your code is not adhering to, potential places where refactoring might help, and also warnings about smelly code.

Sometimes there are false positives, but you can inspect and decide for yourself whether it flagged the line incorrectly or not. If you identify it as a false positive, you can disable the pylint check for that particular line.

Overall, I think it takes out a lot of pain from the reviewer to point out code smells and enforce coding standards. I would say this module has saved hours of review time.

logging-fstring-interpolation warning

Python has multiple ways of string interpolation. I have explained the details of it in a previous blog - String Interpolation in Python

Even though we are using the latest string interpolation mechanism in python, the Pylint checker is giving us a warning that there is something wrong. The official documentation for this warning says:

Use another type of string formatting instead. You can use % formatting but leave interpolation to the logging function by passing the parameters as arguments.

To understand what was happening behind the hood, I wrote this code.

Here, I have an Employee class with an __str__ magic method. The logging module is configured to log into the console and the logging level is set to DEBUG. The statements of interest are all debug statements, and I also added some headings to understand what section we are executing now.

Running this code with the logging level set to DEBUG. I expect to see the str method called for all the cases because the log level is DEBUG is the lowest and all the log messages will be printed.

The response is as expected. The string evaluation of the object takes place and then the log statement is evaluated.

What if the log level is set to INFO, but the log statements are at the DEBUG level?

I expect the string evaluation to be deferred until the logger determines whether this log statement is above the log level set, i.e., all the statements with log level as INFO and higher will be printed whereas all the statements with log level as DEBUG will be ignored. If the log statements with DEBUG are ignored, then there is no need to call the str method of the objects.

Making only 1 change on line #5, I executed the code.

The response was not something I was expecting.

Upon inspection, it was clear the evaluation of the message objects was happening irrespective of whether the statement was executed or discarded.

So irrespective of what type of string interpolation I use, the order of events is as follows:

evaluate all the arguments to the log.debug() - this calls the str method,
Call the log.debug(),
Determine whether to log it or discard it.

This seemed very counterintuitive because then the logging statement when turned off will still consume time and memory. I decided to peek into the logging library. Looking at the function signature it was clear, there is nothing wrong with the library. But I am not using the library in the intended way.

The intended way to use the logger function is to pass the string template as the first argument and the arguments to be substituted as *args. This way, the string interpolation happens within the logger library and not when I call the logger method.

The correct way to invoke the logging statement where the string interpolation is deferred until it is necessary.

Added two more lines of code and repeated the experiment. In this case, I am passing the %-formatted string template separately and the argument to substitute it with separately.

Running this code again, the result is as expected, the string evaluation is not happening because the log level is set to INFO.

After changing the log level to debug on line #5, the log statement is getting printed as expected.

So, the order of events will be:

Call the log.debug(),
Determine whether to log it or discard it.
Discards the log because the log level is below the level set.
Evaluate the message object by substituting the arguments.

Now that the internal workings and the correct usage is clear, let us understand the impact of this small change on our program.

Impact

You might think this string interpolation is harmless, but if the logging statement is in a function that is called frequently and the logging level is set to be discarded during normal operation, you are wasting away a lot of execution cycles. To understand the impact, I looped the logging statements 100,000 times, collected the run time, and tabulated it.

The x-axis defines different approaches and the y-axis gives the time in milliseconds for each approach.

When we are logging and the log message evaluation is expensive, using the approach of deferred evaluation can save a lot of time. When you want to turn off the debug logs and run in optimized mode. Otherwise, all the approaches are yielding similar run time.

Conclusion

Hope this blog helps someone who is scratching their head trying to figure out why the Pylint-type checker is giving this weird warning. Pylint is trying to tell you to check your syntax because you could be committing a mistake by not using deferred evaluation. This warning message is generated when you are trying to use f-string substitution in the logging statement.

Instead of simply disabling the warning, now you know the reason behind it and how to handle it the correct way.

Share with me your thoughts on this in the comments! Or leave a like if you learned something new.

Technical Debt - The good, bad, and the ugly.

Blitzcode — Wed, 02 Aug 2023 07:30:00 +0000

Let me start with some scenarios to illustrate the concept of technical debt before providing a formal definition and discussing how to eliminate it.

Scenario #1

You are part of a startup trying to ship the latest updates to the customers and your product owner is concerned if you take two days extra to document the code and add more test cases, the initial traction you have built with the customers will die down. You are accumulating technical debt. If a new teammate joins, this person will not have any references to learn from. He might modify code and may reintroduce bugs that were fixed before.

Scenario #2

You are part of a highly unstructured software team. The development team is involved in customer interaction and bug fixes, the team lead juggles multiple roles in the team - as the software architect, scrum master, and technical team lead. Again by not creating sustainable processes and by investing too much into one resource, you are creating a dependency. If this person leaves, or changes team, the whole knowledge is lost.

Scenario #3

If you have been part of the organization for too long and sometime in the past you and your team worked on the tool. But eventually, you have moved on and the tool is not actively maintained, and then after a year, the customer wants to add features to it. You are the only one around, but you also do not remember the requirements at that time. Also, since the tool is not maintained for a year now, the libraries are all stale. It will take as much time for you as it would take a new person to add a feature to this tool.

Technical Debt - Definition

From all the examples above, you might have gotten a fair understanding of the word "technical debt"(also known as design debt or code debt. Technical debt is the result of the software development team expediting the delivery of a certain feature or a project without proper documentation, software architecture practices, or employing resources to develop automation testing.

Is technical debt bad?

Technical debt is like monetary debt. It can be good or bad, depending on how it is used. If you use technical debt to create a quick and dirty proof of concept, it can be a valuable tool. However, if you use technical debt to create a system that is not scalable or maintainable, it can be a serious problem. Sooner or later, you will have to rewrite the entire system because it is not scalable, or, did not discuss the requirements properly.

Like borrowed money, if you do not pay it back, it accumulates interest. Similarly, if you remember to maintain the code, refactor it when necessary, and document and write test cases, then you are keeping the yin and the yang together.

How can you eliminate technical debt?

You may not be able to completely get rid of technical debt, but you and your team can reduce the ramifications of it by:

Allocating resources and prioritizing code documentation,
Paying attention to testing and test case development,
Refactoring the code when necessary, and,
Updating libraries

I would love to hear how your team is tackling technical debt (design debt or code debt). Do let me know in the comments!

Learning Path: MySQL Day 1

Blitzcode — Thu, 20 Jul 2023 09:19:50 +0000

Technically, this is not the day one. In the last few days, I took the time to create a curriculum of what I needed to learn. I explored some of the content available on the internet in the form of YouTube videos and paid courses to which I could subscribe to. The problem was these learnings will not be complete. They will only cover limited topics and I had to do my own search to gain more knowledge.

Instead of identifying a single course, I decided to create a curriculum. I can learn the topics from material available on the internet.

These are the topics I learned on day 1 of this journey.

Connecting and disconnecting from the Server.
Executing a query on the MySQL console.
Creating a database
Creating a table
Loading data into a table.

Satisfactory start to the journey.

Learning Path: MySQL - Day 0

Blitzcode — Wed, 05 Jul 2023 05:40:19 +0000

After 10 years of being a programmer, I decided to revisit my basics of databases and learn MySQL again to brush up on my basics. I will post about my success and failures and all the material I have referred to as I go through them.

What marks success at the end of the journey?

The end goal of this journey is to take the MySQL certification and as a side goal, I want to achieve some consistency in my daily schedule. After the covid pandemic things have been going south and I get more and more demotivated to work. So, this is my way to do something to feel accomplished.

If anyone of you is interested to tag along and do a peer learning journey, ping or comment!

Software Defined Networking

Blitzcode — Fri, 19 May 2023 10:55:28 +0000

_Disclaimer: None of the links are affiliates. _

Introduction

Before I delve into what Software-defined networking is and how I stumbled upon this. Let me explain the problem statement and all of the solutions I tried before I finally arrived at this solution.

I am hoping there is another person scratching their head trying to figure out how to go about this and hope this guide saves them a lot of time.

Problem statement

I have a piece of hardware (you can assume, an Arduino board and a motor driver circuit connected to the Arduino) connected to my laptop and I have a UI created in React that will allow me to control the pins of my Arduino. Let's say, I want to control the PWM output of my Arduino board, which will, in turn, modify the speed at which the motor is running.

I have this react UI deployed on a digitalocean droplet and I wanted the UI to control the Arduino. I have written a thin flask layer between the Arduino and the React which will run on a Gunicorn server in my localhost.

The solutions discussed are very much applicable to any of the Python web frameworks - FastAPI, flask, Django, and others.
Also, to other cloud providers like Azure, AWS, and Heroku.
Even if you want to replace the frontend with another low-code platform, that is also fine.

Solution 1

This was the first thing I tried. Put up an Nginx server and tried to ping from the droplet, but it failed fast. The IP address of my laptop is not visible to the public. I did not want to compromise the security, so I didn't pursue it further.

Normally, if I wanted to run a react app with a Flask backend this is what I would do. Have my Flask server running on a digital ocean droplet and have the react application also be on digital ocean and both are on the same VPC (Virtual Private Cloud). Nginx will be front-facing the incoming traffic on the digital ocean. But here, I don't need to expose my API server to the internet since they are on the same VPC.

Solution 2

From the first failure, I understood I wanted to bring my laptop into the same network as my digital ocean droplet. This will make my laptop visible to the droplet without causing security issues to my laptop. Naturally, VPN was my first choice.

The architecture I created was to have another droplet that will act as a VPN Access server. The droplet running my react UI and the droplet running the VPN access server will be on the same VPC. I will have to install a VPN client so that I can bring my laptop into the same network as the React UI.

I had a guide from the digital ocean on how to set this up. Attaching some reference links:
vpn-access-server-setup
how-to-configure-the-openvpn-access-server

The benefits of using such a network architecture are:

Quickly extend your private networking to remote users and other sites.
Create hub-and-spoke network topology, site-to-site, user-to-cloud, and various other secure VPN connections.
Provide secure, remote access to applications deployed on the private cloud.

I did not test this out, but I can already see the pain of doing all this configuration. But I do not want to go through this for a prototype.

While this experiment was not a failure, I was still searching if there was another more robust solution than this one.

Solution 3

During a brainstorming session with a colleague, he suggested why don't we use a Zero Config VPN instead of setting up a VPN Server. He suggested the name ZeroTier which actually is providing a software-defined network solution. (I will write more once I have the solution finalized)

I went back to the drawing board. What ZeroTier does is, it creates a private network and gives me a network id. If I have the Zerotier client installed on my system, my system (droplet or laptop, or mobile phone) can communicate with other resources in the network. All this, while staying private.

The Zerotier dashboard is much easier to manage, and I get up to 25 nodes on the free tier.

Test Procedure and Result

Step 1: Create an account on Zerotier
Step 2: Create a network.
Step 3: Install the client on your laptop, on the droplet, or on your mobile phone.
Step 4: Open the Zerotier client application and click "Join a Network". key in the 16-character network id when prompted.

Step 5: Go back to the Zerotier dashboard and authorize the new connection.
Step 6: Voila, you are done!

Step 7: But not just yet. In my case, my flask application was listening on port 8000 and I had to go to Windows firewall settings as an administrator and create an inbound rule to allow port 8000 within the private network.

Step 8: You can call the rest API from any device if they have joined the network!

If you want more detailed documentation to setup Zerotier, check out their docs here

Conclusion

Going with the third approach makes sense to me for the following reason:

Less overhead on configuring the network.
Security - my PC is connected to digital ocean within a private network.

Are there other solutions other than the ones I could come up with? What are the limitations or flaws of my final solution? Any other tools I can use other than Zerotier?

Do let me know in the comments or drop a message on LinkedIn!