DEV Community: keonakhon

Monitor all the files changed on the Linux server with Python watchdog

keonakhon — Wed, 04 Oct 2023 16:44:09 +0000

Watchdog is a Python library used to monitor files and directories. On the Linux server, it depends on inotify (inode notify), which is a kernel subsystem of Linux used to monitor changes to the filesystem and report those changes.

I had tried to use Watchdog to monitor files changed since the root level "/" on macOS, and it worked fine. However, on the Linux server, I encountered an error.

[Errno 2] No such file or directory

The reason for this error to occur is that it couldn't catch up with the Process ID in the /proc directory, as the files or directories were created and disappeared instantly; it's too fast.

Here are the details of the directories and files inside /proc: there's a directory named 666 in the first ls command run and 667 in the second ls command run.

Those are the numbers of Process IDs we saw in the ps command.

Make sure you have already installed watchdog library.

pip3 install watchdog

The provided code below is going to monitor all files with .txt, .js, .py, and .php extensions that were created or updated. It will monitor in every directory inside "/". I have to ignore /proc, as well as all symlink directories or files in order to prevent the error and improve performance, respectively.

import time
from datetime import datetime
from os import listdir
from os.path import isdir, join, islink
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler


# Scan for newly created files
class MyHandler(PatternMatchingEventHandler):
    def __init__(self):
        super().__init__()
        patterns = ['*.txt', '*.js', '*.py', '*.php', '*.html', '*.css']
        PatternMatchingEventHandler.__init__(self,
                                             patterns=patterns,
                                             ignore_directories=True)

    def on_created(self, event):
        current_time = datetime.now()
        if not event.is_directory and event.event_type == 'created':
            print(f"[created] {current_time} - {event.src_path}")

    def on_modified(self, event):
        current_time = datetime.now()
        if not event.is_directory and event.event_type == 'modified':
            print(f"[modified] {current_time} - {event.src_path}")


def main():
    try:
        root_path = "/"
        directories = []
        for entry in listdir(root_path):
            entry_path = join(root_path, entry)
            if isdir(entry_path) and not islink(entry_path) and entry_path != '/proc':
                directories.append(entry_path)

        observers = []

        observer = Observer()
        for directory_to_monitor in directories:
            event_handler = MyHandler()
            observer.schedule(event_handler,
                                path=directory_to_monitor,
                                recursive=True)
            observers.append(observer)

        observer.start()

        try:
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            for obs in observers:
                obs.stop()
        for obs in observers:
            obs.join()
    except Exception as error:
        print(error)


if __name__ == '__main__':
    main()

If you encounter an error related to inotify limits, please follow this Stack Overflow anwser.

Lastly, you might see duplicate messages of the same file on both events (created and modified) because it depends on the command you used. It's not actually an issue with watchdog or my code. For more details, please refer to this Stack Overflow anwser and watchdog's repo.

P.S. It might not be an error in your case, but I hope this is helpful.

Looking for qualify software with affordable price? contact: funwarn.dev@gmail.com

How I Made My Linux Software Keep Running

keonakhon — Sun, 01 Oct 2023 11:50:13 +0000

We successfully built our Linux software or package in the previous article. In this article, I'll show you how I made it run on an Ubuntu Server. This can be applied to any Linux OS since they all have the same base.

I will start the package to keep it running by using Systemd, which is at the core of almost all Linux packages. You might be familiar with the command systemctl, which is a part of the Systemd command that we use for starting our database and Nginx. Additionally, we will also update our Python source code. Lastly, we will install our package inside a venv (virtual environment) and make it available outside of the venv.

Systemd Unit File

Create a Systemd unit file named time-tracking-pkg.service, it is a configuration file used by Systemd to define how a service should be managed. Place it into /etc/systemd/system/.

Description=Time Tracking Service
After=syslog.target network.target

[Service]
User=root
ExecStart=/usr/local/bin/time-tracking-pkg
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Start Systemd

Run the following commands:

Enable the service to start automatically at boot. This is useful for services that need to run continuously.

sudo systemctl enable time-tracking-pkg.service

Start our package to keep it running in the background as a daemon or service. Adding its extension is not necessary.

sudo systemctl start time-tracking-pkg.service

Check whether it's currently active or not.

sudo systemctl status time-tracking-pkg

Now that the service is up and running, in order to track it, I'm going to adjust our Python code and add a socket. Additionally, I will add an argument to make it run only its specified task.

import time
import socket
import argparse


localhost_network = "127.0.0.1"
socket_port = 52000


# Sender Data to socket
def send_data():
    try:
        # Create a socket object
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

        # Connect to Receiver
        s.connect((localhost_network, socket_port))

        current_time = time.strftime("%Y-%m-%d %H:%M:%S")

        s.send(current_time.encode())

        # Close the connection
        s.close()
    except Exception:
        pass


# Receive Data from socket
def receive_data():
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

        # Set the SO_REUSEADDR option
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

        # Bind and listen
        s.bind((localhost_network, socket_port))
        s.listen()

        # Accept connections
        while True:
            (clientConnected, _) = s.accept()
            try:
                data_from_client = clientConnected.recv(1024)
                data_to_print = data_from_client.decode()
                print(data_to_print)
            finally:
                # Close the connection
                clientConnected.close()
    except socket.error:
        pass
    except KeyboardInterrupt:
        pass
    except Exception:
        pass
    finally:
        s.close()


def main_func():
    try:
        parser = argparse.ArgumentParser(allow_abbrev=False)

        # Define start argument
        parser.add_argument(
            '--start',
            action='store_true',
            help="Use this argument on systemd"
        )

        args = parser.parse_args()

        # Condition when user pass argument start
        if args.start:
            while True:
                send_data()
                time.sleep(5)
        else:
            receive_data()
    except KeyboardInterrupt:
        pass


if __name__ == '__main__':
    main_func()

I used argparse to add an argument named "--start" for calling a function that will initiate sending the datetime data over a socket.

After that, we will compile our project as we did in the previous article. Then, we are going to update our Systemd script to start our service with this argument.
From:

ExecStart=/usr/local/bin/time-tracking-pkg

To be:

ExecStart=/usr/local/bin/time-tracking-pkg --start

After updating, we are going to reload our service with the following commands:

sudo systemctl reenable time-tracking-pkg
sudo systemctl stop time-tracking-pkg
sudo systemctl start time-tracking-pkg

or we can also run these commands.

sudo systemctl reenable time-tracking-pkg
sudo systemctl daemon-reload
sudo systemctl restart time-tracking-pkg

Now, we can still use time-tracking-pkg command to see the current time.

time-tracking-pkg

Check The Working Process

Open a new tab to access the server and run the following command.

ps aux | grep time-tracking-pkg

You will see two working processes and their Process IDs:

/usr/local/bin/time-tracking-pkg
/usr/local/bin/time-tracking-pkg --start

As you can see, even though we didn't run time-tracking-pkg --start, we still see it because Systemd runs it in the background. You can think of it as another program/process.

Now, terminate the tab that is running time-tracking-pkg and then re-run it again; you will notice that its process ID changes. Meanwhile, --start is still running, and its Process ID remains the same.

Virtual Environment (venv)

We finally have the service up and running. However, there's a problem with our distribution package setup file. You won't be able to install it on Debian, and in some cases, even further, some of the package versions on the server are incompatible with our package, which would result in errors.

To prevent this from happening, we're going to use a venv. Please ensure that you have python3.XX-venv installed on your Ubuntu Server. If you haven't, please run apt install python3.<enter your python3 version>-venv.

python3 -m venv "time-tracking"

Before activating the venv, I'm going to stop our service and uninstall the package.

systemctl stop time-tracking-pkg
pip3 uninstall -y time-tracking-pkg

The directory named time-tracking will be created at the current directory level where we are currently located. We are going to activate the venv that we just created.

source time-tracking/bin/activate

Even though we can still run the command normally within the venv, we still need to make the command available outside of the venv. As you can see in the image below, once we exit the venv, we can no longer run the command.

So, I'm going to create a symlink (symbolic link) or a shortcut of the binary file of our package from inside the venv to /usr/local/bin/.

ln -s /root/time-tracking/bin/time-tracking-pkg /usr/local/bin/time-tracking-pkg

As you can see, now we can run the command even when we're outside of the venv. However, we still need to start our service again.

Try to run time-tracking-pkg.

Finally, we have our service up and running within Systemd and venv. Additionally, You can check the source code in my repository.

Thank you for reading up to this point. Please feel free to leave a comment.

Looking for qualify software with affordable price? contact: funwarn.dev@gmail.com

How I Made My First Linux Distribution Package

keonakhon — Tue, 22 Aug 2023 19:26:32 +0000

I built Linux software or distribution packages using Python. I accomplished this with the assistance of wheel, which compiles my source code into a single installation file compatible with any Linux system.

We'll begin with a simple Python code named main.py, which will print out the current time every 5 seconds in an infinite loop.

Ensure that you have a directory structure like the following (this is necessary for proper functioning): a root directory, followed by the package name directory and its module files.

time-tracking-package/
--time_tracking/
----main_file.py
--setup.py

Our main_file.py code:

import time


def main_func():
    try:
        while True:
            current_time = time.strftime("%Y-%m-%d %H:%M:%S")
            print("Current time:", current_time)
            time.sleep(5)
    except KeyboardInterrupt:
        pass


if __name__ == '__main__':
    main_func()

Next, we'll add the Python configuration file setup.py to compile the entire project into an installable file. This file will be used for installation on the Linux server:

from setuptools import setup


setup(
    name='time-tracking-pkg',
    version='1.0.0',
    install_requires=[
        'wheel',
    ],
    entry_points={
        'console_scripts': [
            'time-tracking-pkg=time_tracking.main_file:main_func'
        ]
    },
)

name is the package name in pip3 list. In case you use underscore "_", the package name will still convert it to dash "-" anyway.
version of the package.
install_requires the requires package which will be install during the setup.
console_scripts is where we define the command of the package(time-tracking-pkg) and its source(time_tracking). The package has to be identical to the package name, you can use either "-" or "_" on the command unlike package name.

Compile our project by:

python3 setup.py bdist_wheel

We'll get 3 newly created directories: dist/, build/ and time_tracking_pkg.egg-info/.

We'll use the file time_tracking_pkg-1.0.0-py3-none-any.whl located in dist/ to upload it to the Linux Server(Ubuntu). Also make sure that you have python3 and pip3 on your Ubuntu Server.

After uploading to Ubuntu Server, use this command to install the package into our server:

pip3 install time_tracking_pkg-1.0.0-py3-none-any.whl

Once it done, we'll use pip3 list to check if it install correctly.

Finally! We can run the command and get its result:

time-tracking-pkg

To see its location, use:

pip3 show -f time-tracking-pkg

Conclusion:

You can also add another package in this project, which will be included in the setup file. But please keep in mind that both package will be remove when you uninstall with:

pip3 uninstall time-tracking-pkg

In the next article I'm going to show you how I made it run on the server and we will adjust the code in order to check how it works.

I hope this article is helpful to you. I struggled when I attempted to develop it for my work, which is why I decided to write about it here.

Looking for qualify software with affordable price? contact: funwarn.dev@gmail.com