DEV Community: Thuwarakesh Murallie

How to Work With Large Pandas Dataframes and Limited Memory.

Thuwarakesh Murallie — Fri, 20 Jan 2023 02:36:48 +0000

Few Pandas users take advantage of this amazing tweak.

The default configuration when you read a dataframe is suitable for smaller datasets. But for larger datasets, you'd soon need more memory.

This post describes compressing your dataframes to fit in your memory without losing any information. Do this and tap into the full potential of your computer.

A Little Pandas Hack to Handle Large Datasets with Limited Memory

GitHub Actions + Black = Automatically Clean Python Code

Thuwarakesh Murallie — Tue, 17 Jan 2023 02:22:23 +0000

A codebase that is easy to understand and navigate allows programmers to work more efficiently and effectively. Thi sis why having a clean and well-formatted codebase is crucial for the success of programmers. Yet, ensuring that every team member follows a common standard can be difficult, which is where GitHub Actions comes in.

GitHub Actions is a powerful tool. And you can use it to automatically format code whenever changes are made to a repository. By configuring Black and other formatting libraries in GitHub Actions, you can ensure that your codebase is always consistent and easy to read. Additionally, if any formatting tasks fail, you can set up email notifications to alert you of the issue.

This post provides a step-by-step guide on how to set up automated code formatting in GitHub Actions, so you never have to worry about poorly formatted code being committed by team members again.

If you're a Python dev, I'm sure you'll use this on a daily basis.

Maintain Clean Python Code With Black and GitHub Actions

Are No-Code Platforms a Threat to Developers?

Thuwarakesh Murallie — Sat, 19 Nov 2022 11:41:52 +0000

The world is simplified. The work of developers is to be.

Ever since we started using machines, there has been programming. The algorithm for the Analytical Engine is considered the first computer programming language. The English mathematician Ada Lovelace created it in 1883 for Charles Babbage's mechanical general-purpose computer.

After several evolutionary steps, modern programming languages such as Python, JavaScript, and Java rose in popularity in the early nineties. These languages are high-level programming languages. Meaning it's very close to our human languages, but computers can still understand them.

And it never stopped there. The next leap in programming is visual programming/no-code development.

Visual programming lets users create graphical elements to build application logic. Several no-code online platforms offer this, and there's hope that this will be the primary mode of programming in the future.

Today you can find visual programming tools for almost any need. You have Microsoft Power Apps for general software development. If you need to develop data science apps, KNIME is an excellent choice.

Beyond visual programming, you have no-code options for famous use cases. For instance, using terraform, you can create and manage an entire cloud architecture only with a configuration file. No-code involved.

Yet, what do they mean to developers of the present and future?

Will no-code platforms reduce the need for trained developers? Or will developers have a better tool in their toolkit? Let's find out.

What are no-code platforms?

No-code platforms are a relatively new kind of software allowing users to create complex applications without coding knowledge.

These platforms typically provide a visual drag-and-drop interface, making them easy to use even for those with no technical background. While no-code platforms were initially designed for simple applications, they have become increasingly influential in recent years and can now support sophisticated workflows and business processes.

In many cases, no-code platforms can completely replace traditional coding frameworks, making them a popular choice for businesses that want to quickly develop and deploy new applications without incurring the cost and complexity of working with code.

When do no-code platforms do well over code-based development?

No-code sure lets you build apps and websites without coding knowledge. Yet, there are still many instances where code-based development is the better option.

As of writing this post, most no-code platforms are suitable for faster prototyping, yet not for production systems.

When planning the app for the first time, it's essential to visualize your ideas as soon as possible. Of course, you can have some Figma wireframes. But the next step would be to put up a minimum viable product (MVP.)

At this point, you still don't know if you're application has any potential in the future. You don't want to hire developers to get an MVP. It would only shoot up your initial investment.

The best way to get users to test your idea is by creating an app using a no-code platform. You'd only be spending a few hours, if not days. Then your users have a working product at hand for testing.

Some no-code platforms are great for production systems as well. For instance, you can use KNIME to build your pipelines faster when working on data science projects. But KNIME is capable of running production-grade applications as well.

Therefore the critical factor to consider is the complexity of the project. A no-code platform can likely suffice if you're trying to build something straightforward, such as a basic attendance app. But, if you're looking to create something more complex, such as a custom web app, then code-based development will be necessary to give you the flexibility and control you need.

Drawbacks of no-code development

Like anything, no-code development has its pros and cons. Here are five potential drawbacks to keep in mind:

Lack of Flexibility: No-code development platforms can be inflexible, making it challenging to customize your project to your specific needs. Think about the switching cost. After developing so much, and at some point in the future, when you realize you can't create a new feature that all your competitors have, you have to spend a massive chunk of your budget switching to a different technology.

Limited Functionality: While no-code development platforms have come a long way, they still offer limited functionality compared to traditional coding languages. You can get the basics for sure. You may even be able to connect it to your other systems. But you still have to wait for a very long time every time a new software concept sets the trend in the industry. That's because you can't develop them for yourself.

Difficulty scaling: As your project grows, managing everything within a no-code development platform can become challenging. You may not be able to install extra servers and database instances as you'd do in a traditional development environment. And most no-code platforms charge per user. Thus when you scale, the cost becomes massive.

Are no-code platforms a threat to programmers?

By now, you'd have guessed that the answer is no. But no code development can replace programmers in the future.

Even though no-code development platforms help a lot, they are not a direct substitute for more production-grade applications. You can create more scalable, efficient, and cost-effective applications with programming.

But with the rate of technological growth, this would soon change. In the future, you could directly translate the diagrams from a business meeting to an app. And with minor tweaks, they could become production ready.

Thus, it's primarily a question of what skillset developers should develop to stay ahead of the curve.

Skills developers need to work with no-code platforms.

No-code platforms have been gaining popularity in recent years, as they offer a way for non-technical users to create custom applications and solutions without writing code. However, working with no-code platforms still requires some technical skills. Here are five skills that developers need to work with no-code media:

Ability to explain things visually: Programmers do not care much about diagrams. They care about them to understand the concepts but do not explain what we build. Business Analysts usually do this.

Cloud computing experience: Many no-code platforms are cloud-based, so developers need to have some experience working with cloud services such as Amazon Web Services (AWS) or Microsoft Azure.

Negotiation skills: This is another skill that developers long considered not relevant. But they do now. When diagrams become the code, the business team might not understand the challenges. Besides, these challenges are new to the developers too.

These are not new requirements. Articulating skills and negotiation skills are just as crucial for any job. But they will be more significant as no-code development platforms take over the stage.

Conclusion

No code platforms are a leap in the programming evolution. They are going to let everyone create software products with little idea-to-app time.\
Yet, although the initial benefits are high, no-code solutions can be costly to scale up. In some instances, you may not be able to develop the feature you need.\
But a code-based platform gives the flexibility to develop anything you want. Also, you don't have to wait till the no-code platform introduces the feature. You can set it for yourself.\
Thus developers will be in the driving seat for a very long time. But what skills they need is going to be different. As a developer, you might need excellent skills to articulate ideas with diagrams and negotiate with business stakeholders.

Is Your Python For-loop Slow? Use NumPy Instead

Thuwarakesh Murallie — Wed, 28 Sep 2022 00:12:19 +0000

Speed is always a concern for developers — especially for data-savvy work.

As developers, the easiest way to scale up things is using a for-loop. But there are drawbacks to using them for large numerical computations.

This post compares how for-loops perform in such workloads with vectorized alternatives.

Is Your Python For-loop Slow? Use NumPy Instead

5 Unusual Ways Bias Can Sneak into Your Models

Thuwarakesh Murallie — Sun, 18 Sep 2022 00:58:22 +0000

These "Generally Good Practices" have their downsides.

We should be using more AI solutions by now. But there's this bias issue to consider!

We've seen AI models perform differently for underrepresented groups. These issues have been debated heavily in recent years. In search of why bias arises, we found that there are more ways than a human trainer's intention could cause bias.

Yet, when other people's lives and jobs are concerned, the innocence of a creator is not excused. Customer backlashes, public opinion, and badmouthing could harm your reputation, and it may be tough to recover from them.

Thus it's critical to understand AI bias. You can't manage something you don't understand.

Here are five situations where bias can sneak into your models.

When Your Machine Learning Model Performance Starts to Fade ...

Thuwarakesh Murallie — Tue, 13 Sep 2022 14:22:46 +0000

Would you raft with no paddles in a river no one had traveled before?

Hoping everything will be okay after deploying an ML model is also like that.

ML models in production degrade in performance over time. Finding them before they happen and making a course correction is vital.

Data scientists often use these strategies to recover their failing models.

3 Tested Techniques to Recover Your Failing Models

Painless Task Scheduling Using Python

Thuwarakesh Murallie — Tue, 16 Aug 2022 02:15:42 +0000

Running tasks on a schedule is not a rare use case. Pretty much every programmer does it.

The standard technique used for scheduling is using Cron jobs. I have no objection to using Cron and still think it's a stable way.

But what if you want a scheduler all in Python? What if you also want it easier to configure?

That's the focus of this post.

Scheduling tasks in Python

Task scheduling in Python is made easy with the python package called 'schedule.' You can install it from the PyPI repository.

$ pip install schedule

If you're using poetry instead of virtualenv

poetry add schedule

The schedule uses a familiar builder pattern. It lets you construct schedules almost as you'd speak in a natural language. That is, if you want to run a function (say send_email) once every hour, you could do it like:

schedule.every().hour.do(send_email)

That's very close to a normal conversation, isn't it? Here's a complete script to send emails in a schedule, but once a day instead of every hour.

import time
import schedule

def send_email():
    # All your email sending logics goes here
    print("Sending email...")

schedule.every().day.at("14:45").do(send_email)

while True:
    schedule.run_pending()
    time.sleep(1)

The above code will call the send_email function at 2.45 PM per your system time.

Besides the ease of scheduling, the package offers a range of other features.

Scheduling with a decorator

I have a personal preference for using decorators whenever possible. It's more clean and elegant.

The 'schedule' package supports decorators out of the box. Here's our example of sending emails, this time using decorators.

import time
from schedule import repeat, every, run_pending

@repeat(every(10).seconds)
@repeat(every(5).seconds)
def send_email():
    # All your email sending logics goes here
    print("Sending email...")

while True:
    run_pending()
    time.sleep(1)

In the example above, you can also see that we've done two scheduled for the same task. The first one will call the function every 10 seconds. The second one will call it every 5 seconds.

Run scheduled tasks with parameters

Scheduling alone is not very useful. We often want more control over the execution of the function. We do it by passing arguments.

@repeat(every(10).seconds)
@repeat(every(5).seconds, email="thuwarakesh@abc.com")
def send_email(email="default@mydomain.abc"):
    # All your email sending logics goes here
    print(f"Sending email...: to {email}")

As we see, our send email function takes an optional argument, email. When scheduling, we can pass in the parameters as keyword arguments.

These are just a few examples of using the library in Python. Please refer to the official documentation for more valuable examples.

Scheduling tasks with cron tab--- the conventional way.

Crontab is the most popular technique for running scheduled tasks. It's famous for a couple of reasons.

One it's provided by the OS. Crontab is for Linux. Even Windows users can use Crontab using WSL.

The other reason is that Crontab is independent of the tasks' programming language. You can also use the same technique to run a node js script.

First, we need to modify our code to run on demand instead of on a schedule because Crontab will take care of the scheduling.

import argparse

def send_email(email="default@mydomain.abc"):
    # All your email sending logics goes here
    print(f"Sending email...: to {email}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-e", "--email", help="Email to send")

    args = parser.parse_args()

    if args.email:
        send_email(args.email)
    else:
        send_email()

I've used the argparse module in the above version to accept command line arguments. You can also use Typer to create more advanced CLIs in Python.

We can run the above script in our terminal as follows:

$ python send_mail.py -e thuwarakesh@abc.com
# OR without any arguments
$ python send_email.py

We can start editing the Crontab with the following command.

crontab -e

The command prompt will ask you to choose an editor upon your first edit. I've chosen vim as it's comfortable for me. But you can choose anything.

I've added the following line at the end of the file.

45 14 * * * /<ENV_PATH>/env/bin/python /<PROJECT PATH>/send_email.py -e thuwarakesh@abc.com

This configuration tells Crontab to run the send_email.py script at 2.45PM every day. Save it and close the file. Your script should run as expected.

If you're not clear about the crontab configurations, my best advice is to use Crontab.guru.

Crontab or Python Schedule?

Both tools are excellent for scheduling tasks. But depending on the circumstances, one works better than the other.

The plus point of Python's schedule is its flexibility and friendliness. Anyone can understand the schedule just by reading it. You don't even have to refer to the documentation.

Although crontab configurations are not that hard to understand, beginners will need a little effort to understand them.

Everything is in one place, which makes the schedule library straightforward. In Crontab, you'd have to edit the configuration file outside your developing module. If someone else wants to use your application, you'd have to instruct them.

Also, if I want to create a new schedule, All I have to do is to annotate the function with another. @repeat. Also, in Crontab, you'd add another configuration line. But not as swift as you'd do in a Python script.

These pros make Python's schedule library suitable for most use cases. But take a look at the library's documentation page.

This library is designed to be a simple solution for simple scheduling problems. You should probably look somewhere else if you need to remember schedule between restarts, sub-second precision execution, multiple threads, time zones, workdays or holidays support. --- schedule library docs.

Final Thoughts

Running tasks on a schedule is a frequent need in software development. I often find it useful to run the little productive hacks I do on the go.

I've been using Crontab for several years, thinking that it's the only way to do the job. But the schedule Python library made it super simple.

It may not be the perfect tool for every instance. But not everything we do daily needs more sophisticated techniques.

Now, I've converted many such periodic executions to Python schedules.

Did you like what you read? Consider subscribing to my email newsletter because I post more like this frequently.

Thanks for reading, friend! Say Hi to me on LinkedIn, Twitter, and Medium.

How to Create File System Triggers in Python

Thuwarakesh Murallie — Fri, 12 Aug 2022 04:46:30 +0000

Imagine you work with a client system where they upload files to an FTP folder. We’ve got to process the file as soon as it appears in the folder and push it to a database.

A real-time dashboard is accessing the database. Therefore, we must update the database without any delays.
You could run a periodic task and check for folder content. But let’s assume the quicker you update the database, the better for the user. The cost of the little delay when using a periodic task is high. Shorter periods might need more resources as your tasks run more often.

We need to build a filesystem trigger to accomplish the task.

Monitor new file creations in a folder.

We can use a Python script that actively listens to file system events in a folder.

We can start by installing a Python package called Watchdog. It’s available through the PyPI repository.

pip install watchdog

If you're using Poetry instead of Virtualenv

poetry add watchdog.

Here’s an example to start with. The following Python script will watch for file changes in the current directory. It’ll log all the changes when they happen.

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler


class FileCreateHandler(FileSystemEventHandler):
    def on_created(self, event):
        print("Created: " + event.src_path)

if __name__ == "__main__":

    event_handler = FileCreateHandler()

    # Create an observer.
    observer = Observer()

    # Attach the observer to the event handler.
    observer.schedule(event_handler, ".", recursive=True)

    # Start the observer.
    observer.start()

    try:
        while observer.is_alive():
            observer.join(1)
    finally:
        observer.stop()
        observer.join()

The important part of the above code is the FileProcessor class. But before getting in there, we should create an observer object to attach an event handler.

We can attach an event handler to the observer using the schedule method. In the example above, we’ve attached it to watch events in the current and all its downstream directories.

If you run this code and create a new file in the current directory, we could see the Python script printing the event on the terminal.

We’ve used the current folder in the schedule method of the observer object. You could also use any path of your choice. You could also choose to get it from the command line argument.

Here’s a modification to the same code that converts our script into a CLI. Now you can pass the path to monitor using a command line argument.

...
import argparse

...


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Watchdog script to watch for new CSV files and load them into the database"
    )
    parser.add_argument("path", help="Path to watch for new CSV files")

    args = parser.parse_args()
    path = args.path

    # Create an observer.
    observer = Observer()

    observer.schedule(event_handler, path, recursive=True)

    ...

You can now run your script like the following in your terminal.

python <YourScript>.py /somewhare/in/your/computer

In my previous post, you can learn more about creating a command line interface using Python.

Process file changes in the handler class

In our example, we’ve created an event handler by subclassing the ‘FileSystemEventHandler’ class. All event handlers should be like this.

The parent class has placeholders for several methods for file system events. We’ve used the ‘on_create’ method to handle all new file creations. Likewise, you can also use on_deleted, on_modified, on_moved, and on_any_event methods to handle other types of events.

Let’s update the on_create to process the file and insert values into a database. Feel free to skip this section if it’s irrelevant to your use case.

import os

import pandas as pd
from sqlalchemy import create_engine
from watchdog.events import FileSystemEventHandler
...


class FileCreateHandler(FileSystemEventHandler):
    def on_created(self, event):
        # Create SQLAlchemy engine for postgres database using environment variables
        engine = create_engine(os.environ["DATABASE_URL"])

        with engine.connect() as con:
            # Read the CSV data
            df = pd.read_csv(event.src_path)

            # Do the required transformations
            df["date"] = pd.to_datetime(df["date"])

            # Write the data to the database
            df.to_sql("weather", con, if_exists="append", index=False)

...

This code might look very familiar if you’ve worked with Sqlalchemy and Pandas. What’s worth noting is how we get the path of the newly created file.

Each event trigger in the ‘FileSystemEventHandler’ class references the path in its event argument. We can access it with the src-path tag as shown in the code.

If you run the code and let the Python script listen to the changes, it’ll also immediately push those changes to the database.

Serve your app in the background.
By now, you’d have noticed that our app runs on a live terminal. But it’s not wise to do this in production. Anything to the terminal session can impact the app.

The best way to run such services in the background is through a system service. Windows users can use the tool NSSM (Non-sucking Service Manager.) It’d be pretty straightforward if you skim through their documentation.

But in this post, I’ll cover the Linux system's use of systemctl.

You can create a new system service by creating a file with the following content in ‘/etc/systemd/system’ folder. You can name it anything with an extension of ‘.service’

[Unit]
Description="Process Data"

[Service]
Restart=always
WorkingDirectory=<PATH_TO_PROJECT_DIRECTORY>
ExecStart=<PATH_TO_PYTHON_EXECUTABLE> <YOUR_SCRIPT>.py

[Install]
WantedBy=multi-user target

Once done, you can run the following commands on the terminal to activate the service.

# To make your new service available to systemctl utility.
$ systemctl daemon-reload

# To start the service 
$ systemctl start <YOUR_SERVICE_FILE_NAME>.service

This will start the process. Now, as new files are being created on our FTP destination, this service will process and upload them to the database. And our real-time database will get fresh data without any delays.

You can check if your service is running properly with the following command.

systemctl status <YOUR_SERVICE_FILE_NAME>.service

Final thoughts

Processing file system change is rare these days as the world moves towards more robust integration between systems. But it doesn’t mean filesystem triggers have no usage.

There are many instances where we need to monitor new file creation or modifications. Take, for example, log stream processing. You could use the technique described here to process a new log line and push it to a data warehouse.

I’ve been using it for a long time now. And I don’t see the need for it to be reduced yet.

Did you like what you read? Consider subscribing to my email newsletter because I post more like this frequently.

Thanks for reading, friend! Say Hi to me on LinkedIn, Twitter, and Medium.

How To Write Readable, Elegant Regex Patterns In Python

Thuwarakesh Murallie — Wed, 10 Aug 2022 16:30:00 +0000

Regex is, without a doubt, the most helpful text processing tool ever invented. It helps us find patterns rather than exact words or phrases in a text. And regex engines are noticeably faster too.

Yet, the difficult part is to define a pattern. Experienced programmers may define it on the go. But most developers will have to spend time googling and reading through documentation.

Regardless of experience, everyone finds reading a pattern others defined difficult.

This is the problem PRegEx solves.

PRegEx is a Python library that makes regex patterns more elegant and readable. It’s now one of my favorite libraries for cleaner python code.

You can install it from the PyPI repository.

pip install pregex

If you're using Poetry instead of Virtualenv

poetry add pregex

Start writing more readable regex.

Here’s an example of grasping how cool PRegEx is.

It’s widespread to need to extract (US) zip codes from addresses. It’s not difficult if the addresses are standardized. Otherwise, we need to use some clever techniques to extract them.

United States zip codes are usually five-digit numbers. Also, some zipcodes may have an extension of four digits separated by a hyphen.

For instance, 88310 is a postal code in New Mexico. Some prefer to use also the geographic segment with an extension like 88310–7241.

Here’s the typical approach (using the re module) to find patterns of this kind.

import re

pattern = r"\d{5}(-\d{4})?"

address = "730 S White Sands Blvd, Alamogordo, NM 88310, United States"

zip_code = re.search(pattern, address).group()

print(zip_code)

# 88310

The steps may seem straightforward. However, if you’re to explain how you defined the pattern to a novice programmer, you’ll have to do an hour-long lecture.

I’m not going to explain it either. Because we have PRegEx. Here’s the PRegEx version of it.

from pregex.classes import AnyDigit
from pregex.quantifiers import Exactly, Optional

pattern = Exactly(AnyDigit(), 5) + Optional("-" + Exactly(AnyDigit(), 4))

address1 = "730 S White Sands Blvd, Alamogordo, NM 88310, United States"
address2 = "730 S White Sands Blvd, Alamogordo, NM 88310-7421, United States"

pattern.get_matches(address1)
# ['88310']

pattern.get_matches(address2)
# ['88310-7421']

As you can see, this code is both simple to define and understand.

The pattern has two segments. The first segment should have exactly five digits, and the second one is optional. Also, the second segment, if available, should have a hyphen and four numbers.

Understand the submodules to create more exciting regex patterns.

Here we used a couple of submodules of the PRegEx library — classes and quantifiers. The ‘classes’ submodule determines what to match and the quantifier submodule help specifying how many repetitions to perform.

You could use other classes such as AnyButDigit to match non-numeric values or AnyLowercaseLetter with lower case strings. To create more complex regex patterns, you could also use different quantifiers such as OneOrMore, AtLeast, AtMost, or Indefinite.

Here’s another example with more exciting matches. We need to find out email addresses in a text. That’s simple. But we’re also interested in capturing the domains of email addresses in addition to matching the pattern.

from pregex.classes import AnyButWhitespace
from pregex.groups import Capture
from pregex.quantifiers import OneOrMore, AtLeastAtMost


pattern = (
    OneOrMore(AnyButWhitespace())
    + "@"
    + Capture(
        OneOrMore(AnyButWhitespace()) + "." + AtLeastAtMost(AnyButWhitespace(), 2, 3)
    )
)

text = """My names is Alice. I live in Wonderland. You can mail me: alice@wonderland.com. 
In case if I couldn't reply, please main my friend the White Rabbit: whiterabbit@wonderland.com.
But for more serious issues, you should main Tony Stark at tony@stark.org.
"""

# Get everything you captured. 
pattern.get_captures(text)
# [('wonderland.com',), ('wonderland.com',), ('stark.org',)]

# Get all your matches. 
pattern.get_matches(text)
# ['alice@wonderland.com', 'whiterabbit@wonderland.com', 'tony@stark.org']

We’ve used the Capture class from the ‘groups’ submodule in the above example. It allows us to collect segments within a match so that you don’t have to do any post-processing to extract them.

Another submodule you’d often need is the operator module. It helps you concatenate patterns or select either of a set of options.

Here’s a slightly modified version of the same example above.

from pregex.classes import AnyButWhitespace
from pregex.groups import Capture
from pregex.operators import Either
from pregex.quantifiers import OneOrMore


pattern = (
    OneOrMore(AnyButWhitespace())
    + "@"
    + Capture(OneOrMore(AnyButWhitespace()) + Either(".com", ".org"))
)

text = """My names is Alice. I live in Wonderland. You can mail me: alice@wonderland.com. 
In case if I couldn't reply, please main my friend the White Rabbit: whiterabbit@wonderland.com.
But for more serious issues, you should main Tony Stark at tony@stark.org.
Please don't message [thanos@wierdland.err](https://thuwarakesh.medium.com/subscribe)
"""

pattern.get_captures(text)
# [('wonderland.com',), ('wonderland.com',), ('stark.org',)]

In the above example, we’ve restricted the top-level domain to either ‘.com’ or ‘.org. We’ve used the ‘Either’ class from the operator submodule to build this pattern. As you can see, it didn’t match with thanos@wierdland.err as its top-level domain is ‘.err,’ not ‘.com’ or ‘.org.’

Final thoughts

Defining regex may not be a massive task for experienced developers. But even for them, reading and understanding a pattern created by someone else is difficult. For beginners, both can be daunting.

Besides, regex is an excellent tool for text mining. Any developer or data scientist will almost certainly come across regex usage.

If you’re a Python programmer, PRegEx has the complex parts covered.

Did you like what you read? Consider subscribing to my email newsletter because I post more like this frequently.

Thanks for reading, friend! Say Hi to me on LinkedIn, Twitter, and Medium.

How to Easily Build Command Line Tools in Python?

Thuwarakesh Murallie — Fri, 05 Aug 2022 04:15:00 +0000

It's so much easy to develop a command line tool than a graphical UI. But it's easier in Python now than it was before.

Typer is a handy tool that helps convert our ordinary functions into command line utilities.

A common need in most CLIs is to accept arguments from the user. Typer makes it super easy to do this. Further, Typer also has the flexibility to style your output.

All this happens in two magical lines of code!

Here's a simple python function that accepts a name and says hello. Two lines of Typer code have changed it to a CLI.

If you're not using Typer, you can use the standard module, argparser to create CLIs. But, Typer is so much easier than argparser.

Create your first command line tool in Python.

Before you begin with the rest of the guide, please install typer on your local computer or your virtual environment. The following code will help.

pip install typer[all]

import typer

def main(name: str):
   print(f"Hello {name}")

if __name__ == "__main__":
   typer.run(main)

Say you've named your Python script app.py. If you run the python script on your terminal with an argument, you'd see it printing hello to your input value.

(env) $ python app.py thuwa
Hello thuwa

(env) $ python app.py
Usage: app.py [OPTIONS] NAME
Try 'app.py --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Missing argument 'NAME'.                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

In the above example, you could also see it failing when you try to run the script without an argument. But try adjusting your code and give the name parameter a default value. This makes the command line argument 'name' an optional one.

def main(name: str = "world"):
    print(f"Hello {name}")

(env) $ python app.py
Hello world

Create multiple command line functions in the same Python script.

Often you'd want to have more than one function on your Python script. In such cases, you can choose which parts appear as command line utilities.

For instance, the following script has three Python functions. One is to say hi, and the other is to say bye. We also have a third function to tell 'Run Away.'

We want to convert the first two into command line functions and leave the last one.

The following script does it.

import typer

app = typer.Typer()

@app.command()
def say_hi(name: str):
    print(f"Hello {name}")

@app.command()
def say_bye(name: str):
    print(f"Bye {name}!")

if __name__ == "__main__":
    app()

This script may look a bit more complicated than the previous simple example. But it is not.

In this example, we create a Typer app and manually add commands to it. We add commands by annotating functions with @app.command.

In the main app, we called our app itself.

Now run the script with --help at the end and see the beautiful help document printing on the screen.

(env) $ python app.py --help

 Usage: app.py [OPTIONS] COMMAND [ARGS]...

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                              │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.       │
│ --help                        Show this message and exit.                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ say-bye                                                                                                              │
│ say-hi                                                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

In the help menu, you can see the two functions we've annotated with @app.command. We can use these with arguments in our command line.

We can also see that the runaway function is not a CLI, although it's there in the script. If we try to run this command, it'll fail as shown below.

env) $ python app.py say-hi thuwa
Hello thuwa
(env) $ python app.py say-bye thuwa
Bye thuwa!
(env) $ python app.py run-away thuwa
Usage: app.py [OPTIONS] COMMAND [ARGS]...
Try 'app.py --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'run-away'.                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Running shell commands in your CLI.

I've wanted to write a find and replace command. It should go inside every file in the current directory and replace values.

You can do it in Linux using the following command.

find ./ -type f -exec sed -i -e 's/<FIND VALUE>/<REPLACE VALUE>/g' {} \;

Yet, the above function is hard to memorize and reuse. Let's create a convenient Python wrapper around it.

We can run this shell command inside Python.

import typer
import subprocess

def replace(old: str, new: str):
    """Find and replace 'old' with 'new' in every file in the current directory"""

    command = [
        "find",
        "./",
        "-type",
        "f",
        "-exec",
        "sed",
        "-i",
        f"s/{old}/{new}/g",
        "{}",
        ";",
    ]

    subprocess.call(command)

if __name__ == "__main__":
    typer.run(replace)

This may not be a very Pythonic way to find and replace values in a folder. But it explains well how to run shell commands inside your CLI wrapper.

Did you like what you read? Consider subscribing to my email newsletter because I post more like this frequently.

Thanks for reading, friend! Say Hi to me on LinkedIn, Twitter, and Medium.

A Brief Guide To Manage Configurations Using TOML Files

Thuwarakesh Murallie — Thu, 04 Aug 2022 14:37:00 +0000

How'd you change the behavior of a software project based on a set of parameters?

You could use environment variables. But what if you want complex structures in the parameters?

Can you use JSON? Yes, you can. JSON files don't allow comments. How'd you describe your parameters to readers?

The answer is to use a TOML configuration file.

Why should we use a config file, and why TOML?

Config files are great ways to extract the project parameters. When you share your code with others, they know exactly where to make changes to tweak the behavior of the software.

For instance, say yours is a website codebase that supports several themes. Others who set up your code can go to the config file and change the theme variable to a different one. It's more convenient than going to the codebase and editing the theme in various files.

But to let your users know what themes are available, you need to have comments on the config files. You can say your website supports 'dark', 'light,' 'pop,' and 'grayscale' themes. JSON files fall short, as they don't allow comments.

It's a misconception that environment files are config files. They are not.

While it's also possible to store values in the environment file, its purpose is to hide secrets from other developers.

Say you have database credentials and API keys. Publishing your code with these values to a public cloud like GitHub can be harmful. Hence we use env variables to separate them.

Config files are to store values that you can happily share with other developers. The themes option we discussed is a good example.

Why TOML?

TOML is an excellent option because it's both super straightforward, and most popular platforms accept it.

For instance, the popular headless CMS platform Netlify uses TOML files to upload build configurations. We can configure the node environment, the build command, the output directory, etc., in this config file.

Another good example is the GitLab Runner. Gitlab runner is part of the Gitlab CI/CD pipeline. It allows you to run jobs in a pipeline during continuous deployment. How do you configure it? You use a TOML file.

In a TOML file, you can tell the runner how many concurrent jobs to execute, the log level, listening port, and many other options.

TOML also files also have syntax highlighting support from many code editors. For instance, on VSCode, I've installed the "Even Better TOML" extension.

TOML file is a great feature of a modern Python project structure.

What is a TOML file?

TOML (Tom's Obvious Minimal Language) is a configuration file format that is easy to read and write. It's minimal, and even people with no programming experience can easily understand it.

TOML supports many data structures, such as key-value pairs, arrays, and tables. Parser libraries can convert them to native data structures in different programming languages.

Most mainstream programming languages can parse it. As of this post, over 40 languages have TOML parsers. This list includes JavaScript, Java, C#, PHP, and C++.

This post focuses on TOML file usage in Python.

A TOML file can live anywhere in a project. The ideal location is the project's root folder because we want to edit the configuration for the project. If we only care about a module, we can move it to a subfolder.

TOML files have an extension, .toml. Here's an example config file.

# This is a TOML document

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T07:32:00-08:00

[database]
enabled = true
ports = [ 8000, 8001, 8002 ]
data = [ ["delta", "phi"], [3.14] ]
temp_targets = { cpu = 79.5, case = 72.0 }

[servers]

[servers.alpha]
ip = "10.0.0.1"
role = "frontend"

[servers.beta]
ip = "10.0.0.2"
role = "backend"

There's a lot to learn from this example.

First, the file starts with a comment. You can create a comment anywhere in the config file by adding a # sign.

Next is value assignment. It's pretty straightforward, as you can see. You can name a key and assign a value to it using the = sign.

The values can be of many types. In the example, the title is assigned to a string. But you can see the DOB is a DateTime. The Ports key is a list, data is a list of lists, and targets are another set of key-value pairs.

An important aspect of TOML file is the headers([owners], [database], etc) . In TOML, these headers are called tables. It depicts a hashtable which is a collection of key-value pairs.

Parser libraries will use this information to create high-level keys in their native types. More on this later.

Tables can also be multi-level. As you can see in the above example, alpha and beta are two hashtables created inside the server hashtable.

Python parser libraries will convert these into nested dictionaries.

TOML supports many other data types that parser libraries convert to native varieties of the programming language.

We're going to use a Python TOML parser library called tomli.

How to read TOML files in Python.

The upcoming Python version, 3.11, has a standard library called tomlib. This module helps read TOML files from a string.

Yet, for all other Python versions (3.6+), we can use an external package called tomli. We can install tomli from PyPI repository as follows.

pip install tomli

First, we need to import the module. Thankfully, both the standard module in 3.11 and the external package follow the same API. Thus we can import them conditionally as follows:

try:
    import tomllib
except ModuleNotFoundError:
    import tomli as tomllib

Now, we can convert any TOML string into Python native variables.

toml_str = """
# This is a TOML document

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T07:32:00-08:00

[database]
enabled = true
ports = [ 8000, 8001, 8002 ]
data = [ ["delta", "phi"], [3.14] ]
temp_targets = { cpu = 79.5, case = 72.0 }

[servers]

[servers.alpha]
ip = "10.0.0.1"
role = "frontend"

[servers.beta]
ip = "10.0.0.2"
role = "backend"
"""

data = tomllib.loads(toml_str)

data
# {'title': 'TOML Example', 'owner': {'name': 'Tom Preston-Werner', 'dob': datetime.datetime(1979, 5, 27, 7, 32, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600)))}, 'database': {'enabled': True, 'ports': [8000, 8001, 8002], 'data': [['delta', 'phi'], [3.14]], 'temp_targets': {'cpu': 79.5, 'case': 72.0}}, 'servers': {'alpha': {'ip': '10.0.0.1', 'role': 'frontend'}, 'beta': {'ip': '10.0.0.2', 'role': 'backend'}}}

In the converted example, we can see our hashtables (or the headers) are the top level-keys. Secondary headings are converted to nested dictionaries.

We can also see the parser library did an amazing job converting TOML configurations to native data types.

We can also read TOM files from a file. We need to open it as a binary file and load it with tomllib.

import tomli as tomllib

with open("./config.toml", "rb") as f:
    data = tomllib.load(f)

data

# {'title': 'TOML Example', 'owner': {'name': 'Tom Preston-Werner', 'dob': datetime.datetime(1979, 5, 27, 7, 32, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600)))}, 'database': {'enabled': True, 'ports': [8000, 8001, 8002], 'data': [['delta', 'phi'], [3.14]], 'temp_targets': {'cpu': 79.5, 'case': 72.0}}, 'servers': {'alpha': {'ip': '10.0.0.1', 'role': 'frontend'}, 'beta': {'ip': '10.0.0.2', 'role': 'backend'}}}

Why TOML over YAML?

YAML is another widely-used, feature-rich config file format. Both file formats have pretty straightforward syntax. Yet, the notable difference in the YAML file is the significance of whitespaces.

In YAML, you specify a list by typing the list items in separate lines with an indentation. You can create nested lists with more indentation.

This can confuse a non-technical reader.

In TOML, however, whitespace plays no role. You create lists with a square bracket.

If you think your config file users are somewhat technical, you can choose one based on other factors. For instance, you can think of the platform on which you will deploy your project. If it's Netlify or Gitlab, you can choose a TOML file.

Final thoughts

Abstracting config parameters out transforms a project from good to great. Whenever you need a new deployment, you change the configurations in one place, and your app responds.

A configuration file is what you need to do this. This post introduced TOML files and why they are fantastic. You could also use the environment file, a JSON file, or any text document. But TOML files have significant advantages over these other formats.

We've discussed using the Python library tomli to read from TOML files. Going forward, we might not need an external library to work with TOML. Python 3.11 ships with a standard module called tomllib.

Lastly, we discussed how we compare TOML with YAML. YAML is also a popular library to store configurations. While YAML can be beneficial for complex configuration options, TOML is more beginner friendly.

Did you like what you read? Consider subscribing to my email newsletter because I post more like this frequently.

Thanks for reading, friend! Say Hi to me on LinkedIn, Twitter, and Medium.

How to Create Web UIs for Python APIs and ML Models

Thuwarakesh Murallie — Fri, 01 Jul 2022 06:15:58 +0000

An experienced data scientist would agree. Building Machine learning models are highly iterative.

The critical challenge for a data scientist is to prototype the model as fast as possible. What they need is a quicker way to build apps around their work.

Only then can they share their model with others for evaluation and collect feedback.

Yet, building an app requires a lot of HTML, CSS, and JavaScript knowledge. Also, hosting them is a different challenge.

But with tools like this, Data scientists can build apps at no time only using Python. They can also deploy the app on Hugginface spaces and share it with others.

This article walks you through the steps you need to build web UI's around your ML models.

Create Web UIs for Python APIs and ML Models