DEV Community: Javed Shaikh

What is fork() system call and how to fork using Python

Javed Shaikh — Tue, 26 Jan 2021 02:35:22 +0000

Before going into fork, lets understand what is process. A process in computer term is a program being executed currently by a computer. Each process is unique and can be identified by its PID or process ID.

Note: All the examples and demo codes shown below were tried on Ubuntu 20.04 LTS and Python v3.8.5.

Visit my Github page for all the demo code snippets https://github.com/jaqsparow/fork-demos

What we will learn in this post?

What is fork 💡
How to call fork in python 📗
How to get process id or PID in Python 📙
How to identify parent and child process 📕
Examples with code snippets 💻

Introduction : What is fork system call?

Fork is one of the most important concept in Unix and Linux operating system. In short note, fork is nothing but cloning a process. That means fork will create a new process with exact copy of calling process. So when a program encounters a fork() system call, it will create another process with same copy of memory. So here comes the concept of parent and child process.

The main or first process which calls fork and creates a new process is called parent process. The new process created by fork is known as child process.

How to identify parent and child process?

Since both processes child and parent have exact same copy of memory, then comes the question how can we identify which of them is parent and which one is child. As I mentioned above, each process has unique ID known as process ID or PID which can be used to differentiate among processes.

To identify parent and child process, we need to check the return code of fork system call.

Return code of fork()

Return code of fork system call determines the parent or child process.
When the parent process calls fork, fork returns ** PID of child process just created** to parent process and **0 **to child process. So basically if return code from fork call is zero, then its child process and if its a positive value, then it must be the parent process.

ZERO If return code is 0, then it must be the child process
A positive value , If return code is positive value (or child's PID), then its parent process
*Negative *, If return code is negative, then child process creation is failed and unsuccessful

How to fork using Python?

Python's os module provides a function fork() to create a child process. To know the PID of any process, use function getpid() from os module

import os
os.fork()

Now lets do some demos to understand whats going on

DEMO 1: To check process ID of any process

In below example we are just checking how getpid() can be used to get the PID of current process.

import os

def demo1():
    print('I am the only process, My PID:',os.getpid())
demo1()

Here is the output:

DEMO 2: Create one child process using fork()

In below example we are printing process ID before and after fork() call. That means before fork, we have one process and after call we got another new process with total of 2 processes.

Lets check the output of below snippets

import os

def demo2():
    print('Before calling fork(),PID: ', os.getpid())
    os.fork()
    print('After calling fork(), PID: ', os.getpid())

demo2()

Here goes output: -

shaikh@ubuntu:~/Jupyter/fork demos$ python3 demo2.py
Before calling fork(),PID:  6837
After calling fork(), PID:  6837
After calling fork(), PID:  6838
shaikh@ubuntu:~/Jupyter/fork demos$

As displayed above, before fork() we only had one process with PID 6837 and after fork we have a new process with PID 6838.

Demo 3: To identify parent and child

Lets see, how we can identify parent and child programmatically. As mentioned in the last section, that if return code from fork is zero, then its child process and if its a positive value then its parent process. Lets check the same here

import os

def demo3():
    print('Before calling fork(),PID: ', os.getpid())
    rc = os.fork()
    if rc == 0:
        print('I am child, PID: ', os.getpid())
        os._exit(0)
    elif rc > 0:
        print('I am parent,PID:',os.getpid())
    else:
        print('Child process creation failed!!')

demo3()

Output

shaikh@ubuntu:~/Jupyter/fork demos$ python3 demo3.py
Before calling fork(),PID:  7316
I am parent,PID: 7316
I am child, PID:  7317

Lets understand what happened above. Before fork we had only one process with PID 7316 and the moment it called fork(), we got another process. Each of those processes have different copy of return code rc. The parent has rc with positive value (PID of child process) and child has the rc equals to zero

Demo 4: Lets create two child process

In below example, we are calling fork() twice.

import os

def demo4():
    #No fork, only one process
    print('Before any fork,  PID:',os.getpid())
    #First fork
    os.fork()
    print('After first fork, PID:',os.getpid())
    #Second fork
    os.fork()
    print('After second fork,PID:',os.getpid())

demo4()

Here goes the output:

shaikh@ubuntu:~/Jupyter/fork demos$ python3 demo4.py
Before any fork,  PID: 7471
After first fork, PID: 7471
After first fork, PID: 7472
After second fork,PID: 7471
After second fork,PID: 7473
After second fork,PID: 7472
After second fork,PID: 7474

Before first fork there was only one process
After first fork, total processes are two
After second call, total processes are four

If we call fork 3 times, guess the number of processes that will be created in the comment section 🙂

Demo 5: Lets do it for fun

Below example will show that after fork call both parent and child will have different copies of variable num

import os

def demo5():
    num = 0
    rc = os.fork()
    if rc > 0:
        num = os.getpid()
    print ('num: ',num)    
demo5()

And guess the output 🙂

shaikh@ubuntu:~/Jupyter/fork demos$ python3 demo5.py
num:  7825
num:  0

So in above code, only parent process can go inside if statement because it has the positive response code which is PID of child process. Since rc for child is ZERO, it will still have the original copy of num

Conclusion

Hope this was fun and interesting learning 🙂. fork is very common way to create a child process in any Linux operating system. It is being used to create multiple processes and the most common use case is web server that forks a new process on each http request.

Be careful when you use fork and make sure you have exited the process successfully after completion of a task else there will be high memory and cpu usage and may create memory leak situation which is called fork bomb

All the demos are in my GitHub page. Click here to visit

References

Files, Folders and Python

Javed Shaikh — Wed, 13 Jan 2021 04:09:28 +0000

In this post we are going to see how we can use Python to find the size of any file and folder. We will check how many files we have inside a folder and how many of them are empty files and how to delete those empty files. We can also check how old a file is and when was the last time file was updated and accessed. We are going to use Python's powerful module os for this. So lets start..

How to get current working directory and how to change it?

First of all, lets import os module.

import os

To get the current working directory lets use getcwd() function

path = os.getcwd()
print(path)

Output: C:\Users\jaqsp\Desktop\other

How to change working directory?

Lets change the current working directory with chdir()

os.chdir('C:\\Users\\jaqsp\\Desktop')

Lets check the current directory once again

print(os.getcwd())

Output C:\Users\jaqsp\Desktop

How to get list of all files and folders inside a directory ?

To get the list of files and folders inside a directory, we can use function scandir. This function returns an object of os.DireEntry. This object represents file path and other file attributes such as

name : file or directory name
path : full path of the file
is_dir() : This function returns True if the entry is a directory
is_file() : This function returns True if the entry is a file

So lets use scandir() function as shown below

files = []
dirs = []
path = 'C:\\Users\\jaqsp\\Desktop\\other'
with os.scandir(path) as it:
    for file in it:
        if os.DirEntry.is_file(file):
            files.append(file.name)
        else:
            dirs.append(file.name)
print('Directories: ',dirs)
print('Files :',files)

Output : Directories: ['.ipynb_checkpoints', 'expense', 'git', 'qual-id']

files : ['exp.json', 'files.ipynb', 'Google spreadsheet.ipynb', 'gspread.ipynb', 'my_screenshot.png', 'OS.ipynb']

In the above code, we have created two empty lists files and dirs and then used scandir() function with input path.
We have used is_file() and is_dir() to identify if the entry returned is file or a directory. Finally we appended the list using append function

How to get file size ?

There are two ways to get the file size i.e. using os.path.getsize() function or using os.stat() function. The size returned by each function is in bytes
In below example we will get size of file exp.json

size = os.path.getsize('exp.json')
print(size)

Output : 2313

As shown the file size is 2313 bytes.

How to get file create, update and access date, owner name and size of file

We will use os.stat(file) function to get all the details about a file. This function returns below along with other details. We are not mentioning all the return attributes. Please check this link for all the attributes here

st_uid : This attribute tells the user id of file owner
st_gid : This is group id of file owner
st_size: This is size of the file in bytes
st_atime: Last accessed time in no of seconds
st_ctime: File creation time in seconds
st_mtime: last update/modified time in seconds

Now lets do an example

stat = os.stat('exp.json')
print(stat)

os.stat_result(st_mode=33206, st_ino=24206847997197434, st_dev=4237007564, st_nlink=1, st_uid=0, st_gid=0, st_size=2313, st_atime=1603293698, st_mtime=1602796599, st_ctime=1602796598)

As shown above file size is 2313 bytes and file creation/modification times are in seconds

Lets convert the time in seconds to readable string

import time
time.ctime(stat.st_ctime)

Output: 'Fri Oct 16 02:46:38 2020'
We have used ctime function to convert the time in seconds to string readable format

How to get folder size and empty files

import os
from os.path import join, getsize
path = 'C:\\Users\\jaqsp\\Desktop\\other'
folder_size = 0
empty_files = []
all_files = []
for paths,dirs,files in os.walk(path):
    for file in files:
        filename = os.path.join(paths, file)
        file_size = os.path.getsize(filename)
        all_files.append(filename)
        if file_size == 0:
            empty_files.append(filename)
        folder_size += file_size
print('current folder size is',folder_size)
print('total files present: ',len(all_files))
print('Total empty files: ',len(empty_files))
#Print all the empty files
print('Below are empty files found')
for i in range(len(empty_files)):
    print(empty_files[i])

Output:

current folder size is 578231
total files present:  230
Total empty files:  4
Below are empty files found
C:\Users\jaqsp\Desktop\other\qual-id\qual_id__init__.py
C:\Users\jaqsp\Desktop\other\qual-id\test__init__.py
C:\Users\jaqsp\Desktop\other\qual-id\test\categories__init__.py
C:\Users\jaqsp\Desktop\other\qual-id\test\utils__init__.py

In the above example we have used Python's walk(path) function to list all the files inside the directory tree by walking either top-down or bottom-up
Then we used join() to join the path and file and return full path of the file and finally we use getsize() function to retrieve size of the file. Folder size in above example is 578231 bytes (564KB)

How to remove a file and directory?

Deleting or removing a file or directory in Python is super easy but make sure you want to delete it as it cannot be recovered once deleted.

To remove or delete a file, Python's os module provides function remove(path). If the input path is directory, it will through an exeception as this function is only to remove a file not a folder.

os.remove('abc.txt')

We can use rmdir(path) function to remove or delete a directory

os.rmdir('some_folder')

Conclusion

Hope this post will help to understand how we can use Python's OS module to do daily activities related to files and directories. We learned how to get current directoy, how to change the directory and how to get different file attributes and also how we can delete a file and directory. Visit my GitHub page for code samples and don't forget to check my other posts.

References

https://docs.python.org/3/library/os.html

Check my other posts

How to manage background processes in Linux

Javed Shaikh — Wed, 06 Jan 2021 03:36:51 +0000

If you are new to Linux operating system and you start a command or process using terminal/session, you must have noticed that you need to wait till a process or command get finished before starting another command. This is because when you run a command using session or terminal, the process starts in the foreground by default. What should you do to run another command without opening a new terminal?
In this post I am going to show how you can manage background and foreground process in Linux.

But before that lets understand what is foreground process and what is background process.

Foreground process is the process or job that is currently running on the terminal. So there will be only one foreground process per terminal.You need to wait till the current foreground process finishes before starting a new foreground process.
**Example: **Any command or process you start in the current session

Background process is the process or job running on the background and doesn't require interaction from the user. There can be more than one background process obviously.
Example: most common example is your web server.

For this example I am going to start command VMSTAT 1

shaikh@shaikhu-com:~$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3045156 167484 2449380    0    0    80    56  469 1138 19  5 76  0  0
 1  0      0 3044644 167492 2450152    0    0     0    92  299  604  2  1 98  0  0

vmstat is command that displays real time memory usage and cpu utillization. If you want to know more about it visit my previous post here

In the above command
Things to note here is that above command will print CPU stats every five second on the terminal until you interrupt it. If you want to terminate it just press CTL + C or if you want to pause or stop, press CTL + Z

bg: Move a job to background

If you want to move a job that is already started to background so that you can access the terminal press CTL + Z ** and then **bg

lets start a job in the foreground using below command. Here we are writing CPU stats on a text file. As you can see , we cannot start a new command as this is running on the foreground.

shaikh@shaikhu-com:~$ vmstat 5 >> vmstat.txt

Lets do CTL + Z to pause this job and then do bg

shaikh@shaikhu-com:~$ vmstat 5 >> vmstat.txt
^Z
[1]+  Stopped                 vmstat 5 >> vmstat.txt
shaikh@shaikhu-com:~$ bg
[1]+ vmstat 5 >> vmstat.txt &
shaikh@shaikhu-com:~$

Now the job running in the background and we got the terminal

How to start command and run it in the background

To start a command and run it in the background use & as shown below

command &

example:

shaikh@shaikhucom:~$ sleep 400 &
[2] 11122
shaikh@shaikhucom:~$

jobs : command to check the job status

This command displays all the jobs running in the current terminal

shaikh@shaikhucom:~$ jobs
[1]-  Running                 vmstat 5 >> vmstat.txt &
[2]+  Running                 sleep 400 &
shaikh@shaikhucom:~$

Here number withing bracket is [n] is job id or job number and + indicates most recent command or job whereas - indicates previous job. If you want to see the process id use -l option

shaikh@shaikhucom:~$ jobs -l
[1]- 10216 Running                 vmstat 5 >> vmstat.txt &
[2]+ 11122 Running                 sleep 400 &
shaikh@shaikhucom:~$

The number 10216 and 11122 are process id

Different options for jobs

-l : lists process IDs in addition to the normal information

-n : lists only processes that have changed status since the last

notification

-p : lists process IDs only

-r : restrict output to running jobs

-s : restrict output to stopped jobs

kill %n: to kill a job with job id n

Kill command is used to kill a job. Note that % will be used to indicate job id or job number

haikh@shaikhu-com:~$ jobs
[1]-  Running                 vmstat 5 >> vmstat.txt &
[2]+  Running                 sleep 400 &
shaikh@shaikhucom:~$ kill %2
shaikh@shaikhucom:~$ jobs
[1]+  Running                 vmstat 5 >> vmstat.txt &
shaikh@shaikhucom:~$

As you can see now we have only one job running in the background and notice the + sign which indicates the job id 1 becomes the most recent job 🙂

fg : command to move a job to foreground

Use **fg **command to move a job to foreground. By default it will bring most recent background job to foreground

shaikh@shaikhucom:~$ jobs
[1]   Running                 vmstat 5 >> vmstat.txt &
[2]-  Running                 sleep 400 &
[3]+  Running                 sleep 500 &
shaikh@shaikhucom:~$ fg
sleep 500
[2]   Done                    sleep 400
shaikh@shaikhucom:~$

If you have more than one jobs running in the background then use %n to move a specific job to the foreground

shaikh@shaikhucom:~$ fg %2

ps command to see all the processes

Use ps command to see active processes.
Use below options

ps ax : to see all the process currently active in the system (Use my previous post to check the commands for real time process). It will be a very long list, so use less/more parameters

ps T: list all processes running on current terminal

shaikh@shaikhucom:~$ ps T
    PID TTY      STAT   TIME COMMAND
   5786 pts/0    Ss     0:00 bash
  10216 pts/0    S      0:00 vmstat 5
  12983 pts/0    R+     0:00 ps T
shaikh@shaikhucom:~$

Conclusion

So in this post we learned different commands to manage background and foreground process. We learned

- bg and fg to move a job to the background and foreground ##### respectively.

- jobs command to list all the jobs active in the current terminal.

- kill command to kill a job

- ps command to see list of all the active and running processes

We also learned how to start a job in the background using &. If you want to know real time memory and cpu usage and control all the processes running on your system please check my last post https://shaikhu.com/how-to-monitor-cpu-utilization-in-linux-ckgcp9gex07g8pas1ees0hz7t

How to monitor CPU utilization in Linux

Javed Shaikh — Fri, 01 Jan 2021 14:14:09 +0000

Monitoring CPU utilization is one of the very important tasks of a back-end system engineer. CPU utilization is nothing but the total works or tasks being processed by your central processing unit or CPU. Its always a good practice for every developer to know the common utilities to monitor cpu utilization of the system they are working.

In windows we use Task manager to check the statistical performance of CPU, memory ,disk network.. But what about Linux operating system. Lets check it out.

We will go through these utilities one by one.

top
htop
nmon
vmstat

top

top is one of the oldest command or utility to monitor system performance. It is a built in utility for any Linux operating system. It will display information like no of tasks, no of users, cpu utilization, statistical data about memory usage and list of active processes.

Just input top and enter

shaikh@shaikhu-com:~$ top

here is the output of the top command

top - 11:23:33 up  3:11,  2 users,  load average: 0.86, 0.69, 1.01
Tasks: 317 total,   2 running, 315 sleeping,   0 stopped,   0 zombie
%Cpu(s): 34.6 us,  9.5 sy,  0.0 ni, 55.0 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st
MiB Mem :   7814.9 total,    938.9 free,   2650.3 used,   4225.7 buff/cache
MiB Swap:   3906.0 total,   3906.0 free,      0.0 used.   3753.6 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  7427 shaikh    20   0 9309320 483092 306628 S  65.7   6.0  24:12.65 chrome    
   1935 shaikh    20   0 4556040 288016 108556 S  17.6   3.6  10:12.54 gnome-shell  
   1655 shaikh    20   0  843820  76204  41576 S  10.8   1.0   5:44.07 Xorg 
   2831 shaikh    20   0 1574816 502964 363492 S   8.2   6.3   9:03.78 chrome  
   1572 shaikh     9 -11 2539652  19556  15264 S   7.2   0.2   2:08.94 pulseaudio

Now there are few quick commands available for top utility like press below keys for respective output on top command screen

k to kill a process. It will ask for process id or pid to kill
p to display processes based on cpu usage
m to display list of processes based on memory usage
q to quit

htop

htop is another realtime cpu performance monitoring tool and is almost similar to top command but with expanded screen. It is more user friendly and will give better viewing experience than top. You may need to install this utility using below command for ubuntu os

shaikh@shaikhu-com:~$ sudo snap install htop

Lets take a look

shaikh@shaikhu-com:~$ htop

Here is the output of htop

nmon

nmon is system admin tool to monitor performance of a Linux operating system. This is developed by IBM's Nigel Griffiths and the name nmon is a short hand for Nigel's Monitor. It has nice colorful screen with many different statistical views.

First lets install this utility using below command

shaikh@shaikhu-com:~$ sudo apt-get install nmon

lets start it, type nmon and enter

shaikh@shaikhu-com:~$ nmon

You will see below menu screen for nmon

As shown above you have to select which realtime statistical data you want to see and use the keys accordingly
Lets type m and c to check realtime cpu and memory usage

vmstat

This command can be used to display information about system processes, memory, swap, disk and cpu usage

Lets type vmstat and enter

shaikh@shaikhu-com:~$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  10240 645752 313692 3789356    0    0    55    66  493  109 18  5 76  0  0

If you want to display vmstat for every 2 seconds, use below command

shaikh@shaikhu-com:~$ vmstat 2

The above command will display system status every 2 seconds unless you interrupt it.
Lets take another example

shaikh@shaikhu-com:~$ vmstat 5 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  10240 512428 315012 3869748    0    0    52    65  487  154 18  5 77  0  0
 0  0  10240 512428 315020 3869588    0    0     0    45  396  738  2  1 98  0  0
 0  0  10240 481684 315028 3900856    0    0     0     7  726 1153  3  1 96  0  0
 0  0  10240 467320 315036 3914636    0    0     0    37  743 1730  6  2 92  0  0
 1  0  10240 486756 315052 3894072    0    0     0    90  694 1586  5  2 93  0  0
 0  0  10240 494316 315064 3887496    0    0     0    12  592 1264  3  1 95  0  0
 0  0  10240 500900 315072 3879552    0    0     0     9  623 1461  5  1 94  0  0
 0  0  10240 505200 315072 3874556    0    0     0    30  893 2205  8  2 90  0  0
 0  0  10240 474204 315080 3906004    0    0     0   156  971 2396  8  3 89  0  0
 0  0  10240 469416 315088 3910568    0    0     0    14  920 2261  8  2 89  0  0

The above command will display system status every 5 seconds for 10 times

Conclusion

Knowing and practicing above commands can help troubleshoot any process and performance related issue.

I know there are many more utilities out there including iostat,sar,mpstat.. and what not 😊 but I have been using these four for most of the times to keep an eye on the system. Out of these, nmon and top are my favorite utility to check the system usage data and manage the processes accordingly. Let me know which utility you have been using to monitor system health.

How to schedule and manage tasks using crontab

Javed Shaikh — Sat, 26 Dec 2020 03:19:32 +0000

Before going into details about crontab, lets suppose you are running an online store and you want to send an email about latest deals on every Friday at 9AM MST. So how you can achieve this. Of course you can write your own job scheduler of n number of lines of codes if you have that much of time or you can simply use crontab available in all Unix and Linux operating system to schedule a task.

What is Crontab

Crontab is short form of cron table. The cron is a utility available on all Linux and Unix operating system that runs a task or process at a given date and time. So crontab is actually a table that contains script or commands along with date and time to be run at.

How to view the crontab or cron table

On Ubuntu you can use crontab -l to view the current table

shaikh@shaikhu-com:~$ crontab -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

Use crontab -e to edit and manage cron table

How to set date and time of job run

Below is the format of crontab that is supported by all Linux systems.

# * * * * * command to be executed
# | | | | |
# | | | | |
# | | | | |
# | | | | |_______________Day of the week (0 - 6)(Sunday to Saturday)
# | | | |_______________Month (1 - 12)
# | | |_______________Day of the Month(1 - 31)
# | |_______________Hour(0 - 23)
# |_______________Minute(0 - 59)

Lets get into some examples for better understanding

Example 1: Lets get into the first example we discussed at the beginning i.e.
Run a job to send an email to all subscriber at 9AM every Friday

0  9  *  *  5  /usr/bin/python3 sendEmail.py

Example 2: In below example, we are running a job every 15 minutes( Note / operator)

*/15  *  *  *  *  doSomething.sh

Example 3: Here we are running a job every 1st and 5th hour ( Note the comma)

*  1,5  *  *  *  doSomething.sh

Example 4: Below job runs on 1st January every year at 7AM

0  7  *  1  *  happyNewYear.sh

What if you want to run a job at random minutes every hour

As of now we have seen how crontab can be used to schedule a job at a specific date and time. But what if we want to run a job at a random time or say random minute every hour. Well we can achieve this as well by writing a sleep command as shown below.

0  *  *  *  *  sleep $(($RANDOM%60))m;sh test.sh

Lets understand what we did above. As per crontab rules above command set will be run at every hour. At the start of every hour, the cron will encounter the two commands and the first command is to sleep for random minutes. So this cron will delay for this random minutes and then it will execute the second command which is our job once it wakes up from sleep 😃

If you want to do some more stuff than just sleep, we can write a shell script for the same instead of sleep.

As shown below you can run a python script from shell script after delaying n no of minutes.

Crontab:

0  *  *  *  *  sh test.sh

test.sh

#!/usr/bin/sh
#test.sh

TIME=$((RANDOM%60))
sleep "${TIME}m"
#Do some stuff
#Do some more stuff
/usr/bin/python3 /mybots/newsbot.py

So basically you are running a python code from a shell script once after delaying RANDOM minutes and after doing some more code stuffs:)

Conclusion

Cronjob is a very helpful utility table that is being used to schedule a task. Rules are very simple and easy to remember. You just have to follow the correct format to run a task or script. Hope this help you schedule your script/jobs :).

How to access Google Sheets using Python and gspread

Javed Shaikh — Mon, 21 Dec 2020 03:44:40 +0000

Hello everyone, In this post I am going to show how we can access and update Google spreadsheet using Python and an excellent library gspread. With just few lines of codes we can create a new or use existing sheet and do the operation such as add,
update or delete. Another cool thing is you can convert this spreadsheet into pandas dataframe using just one line. So lets get started..

Prerequisite

Google developer account to create api credentials
Google service account to access spreadsheet programmatically
gspread - Python library
oauth2client - Python library

Create Google service account

**Step 1: **First you need to signup for a Google developer account if you have not already. Visit Google developer signup page at https://console.developers.google.com/ to create your account.

Step 2: Create service account
Since we will be accessing spreadsheet using programs, we need to create service account and credentials for it. So login to your developer account and create new project by clicking at the top left corner new project icon.

As shown below a new window will appear. Provide your project name and click create button

Now once project is created, it will take you to the project page (If not click on small bell icon on the top right and go to the project page). It should look like below screen print

Now we need to enable APIs that we want to use. For this tutorial we need to enable api for Google drive and Google sheet. We need google drive api because your spreadsheet is going to be saved on your google drive.
So click menu icon on top left corner and then go to 'APIs and Services' and then click Library and it will take you to the API Library page like below

Now search for Google Drive api and click enable

Once enabled it will take you to Google drive api page. Now click create credentials

To create credentials follow the steps as shown in below screenshot and click "what credentials do I need?"

In the next page you need to provide service account name, service account id and role as editor since we want to do add and update operation on Google sheet programmatically.

When you click continue in above step, a JSON file with authentication details will be created.

Save this file in your system.

At this point we have created the service account successfully. Now lets enable Google Sheet api using the same steps we used to enable Google drive api.

Congratulations! we have successfully created our service account to access Google spreadsheet using Python. Now lets create a sample spreadsheet so that we can use it for our Python program to access. Lets go to https://docs.google.com/spreadsheets and create a sample sheet. I have create a sheet named Student for this example tutorial.

To use this sheet with our python code we need to share the file with Google api client email ID. Open a spreadsheet and click Share button on top right corner of the spreadsheet. You can find the client email from the JSON file we just downloaded in the previous example.

Install Python libraries

Install these libraries so that we can add and update spreadsheet using python

pip install gspread
pip install oauth2client

Though these two libraries are enough for us to for this tutorial. However we can install pandas to view our data using Pandas's Dataframe

pip install pandas as pd

Lets start coding

At this point we have created Google account and then enabled Google drive api and Google sheet api and created credentials and saved in our local drive as JSON file. We also created a sample spreadsheet and shared the sheet with client email provided in the JSON file. We also installed required Python libraries in the above step. So now lets get started to code 🙂

import gspread
from oauth2client.service_account import ServiceAccountCredentials
import pandas as pd

With above code we imported the libraries we installed in the last step. Now before accessing the spreadsheet we need to authenticate using JSON file we downloaded

#Authenticate Google service account
gp = gspread.service_account(filename='Testapp-9f185f872dd9.json')

We need to provide the correct path of JSON file. Since I have copied this file to my project directory, so just name is enough.

Now lets open the spreadsheet 'Student' we created in one of the steps.

#Open Google spreadsheet
gsheet = gp.open('Student')

Since spreadsheet is opened, now we need to select the worksheet we want to access using below line

#Select worksheet
wsheet = gsheet.worksheet("Sheet1")

As shown above we have selected 'Sheet1' which is the first worksheet.

Retrieve all records from a worksheet

Now to retrieve the records or cell values we can use one of below functions

get_all_values() fetches all values from a worksheet as a list of lists.
get_all_records() fetches all values from a worksheet as a list of dictionaries.

Lets try one by one on Jupyter Notebook

wsheet.get_all_values()

Output of above statement on Jupyter notebook. As shown all records are fetched as list of dictionaries

[['Student', 'Course', 'Grade', 'Score'],
 ['John', 'Data Science', 'A', '480'],
 ['Amit', 'Cyber Security', 'B', '440'],
 ['Ronit', 'Data Science', 'A', '475'],
 ['Maya', 'Business Analytics', 'A', '481']]

Similarly for get_all_values()

wsheet.get_all_records()

Output:

[{'Student': 'John', 'Course': 'Data Science', 'Grade': 'A', 'Score': 480},
 {'Student': 'Amit', 'Course': 'Cyber Security', 'Grade': 'B', 'Score': 440},
 {'Student': 'Ronit', 'Course': 'Data Science', 'Grade': 'A', 'Score': 475},
 {'Student': 'Maya',
  'Course': 'Business Analytics',
  'Grade': 'A',
  'Score': 481}]

Using Pandas DataFrame. First we extracted all values from worksheet, then set the columns for dataframe which is nothing but first list of all_rows.

all_rows = wsheet.get_all_values()
columns = all_rows.pop(0)
df = pd.DataFrame(all_rows,columns=columns)

Here is the output of df

    Student Course  Grade   Score
0   John    Data Science    A   480
1   Amit    Cyber Security  B   440
2   Ronit   Data Science    A   475
3   Maya    Business Analytics  A   481

Retrieve cell values from a worksheet

To get a specific cell value use acell function. Below snippet will print "John"

cell_A2 = wsheet.acell('A2').value
print(cell_A2)

To get the value using coordinates use *cell * function as shown below. This will also print "John"

cell = wsheet.cell(2,1).value
print(cell)

Creating another worksheet

To create another sheet on the same spreadsheet use add_worksheet function

another_sheet = gsheet.add_worksheet(title="AnotherSheet", rows="10", cols="10")

This will create another sheet named "AnotherSheet" with 10 rows and 10 columns

Deleting a worksheet

Use function **del_worksheet **to delete a worksheet. So below instruction will delete "AnotherSheet" we created just now in the last step

gsheet.del_worksheet(another_sheet)

Updating cell value

To update cell values we can use **update **function like below
Below operation will replace "John" with "David"

wsheet.update('A2', 'David')

We can perform same operation using coordinates

worksheet.update_cell(1, 2, 'David')

Insert a new row

With append_row, we can append a new row at the end. As shown below, we are adding a new row for new student.

wsheet.append_row(["Gina","Computer Science",'B',447])

Conclusion

As shown in above examples, gspread ** is an excellent and super easy to use library to access Google spreadsheet. Hopefully examples and steps shown above will help you build your apps using Google spreadsheet.
I have been using *gspread * for one of my app to track my daily expenses using sms. I am going to post about this in my next story on how to build an expense tracker using **gspread,Twilio **and **Flask and then deploying it on Heroku 🙂

How to keep your computer awake using Python

Javed Shaikh — Wed, 16 Dec 2020 16:09:47 +0000

This sounds very funny but how can you let your team know that you are always online during work from home and doing your important stuffs 🙂. Here I will show how we can write a Python script with just few lines of code to keep our computer awake all the time.
In this post, we are going to write a CLI script to keep our computer awake.

What this script will do?

This script will take number of minutes as input and will keep right clicking every minute until it reaches the input time. We can add additional code to shutdown or restart the system once it reaches the countdown. It will also log the current system time in a text file and takes screenshot of the system before shutting down 🙂.

*So what this CLI script will do?
*

Takes number of minutes (stop_time) as input (No. of minutes computer will awake) In a loop perform these after initiating a counter to 0
- Sleep for 60 seconds
- Perform mouse click to awake computer
- Increase a counter every minute from 0 to stop_time
Save the logs of system
Take screenshot
Shutdown, Restart or do nothing if no action is provided by user

What we are going to learn with this script

Well with this script you can pretend to be working and will always be online 😁 but in real you are going to learn few interesting things like

How to do mouse operation such as right click using Python 📙
How to write a file in Python 📗
How to check current system time
How to take screenshot 📙
How to shutdown or restart computer using Python 📕
How to write CLI script in Python

Start coding

First of all lets import few libraries that we are going to use.

import argparse
import pyautogui
import time
import os

Why we are using these libraries?

argparse : Since we are writing CLI script, we need to pass arguments using this library
pyautogui : For mouse operation and screenshot of the system
time : to capture current system time
os : to shutdown and restart the system

Now lets write a new function named timer which will take two inputs i.e. input time or stop time and action to be performed. Please note that the input time is a integer variable which is number of minutes to keep our computer active

def timer(stop_time, action):
    counter = 0

Above we have defined a new function timer and initialized a counter to 0
Next we are going to start a loop until counter reaches the stop_time.

    while stop_time > counter:
        counter = counter + 1

Above code will execute and end within a moment but we want to keep system active for stop_time. To achieve that we are going to sleep for a minute and then do right click for each iteration

        #Lets sleep for 60 seconds only
        time.sleep(60)
        pyautogui.click(button='right')

Lets understand what we did above..
sleep(n) **is a special python function to delay or pause further execution of codes for **n no of seconds. Here we are delaying execution for 60 seconds in each iteration.
After 60 seconds we are doing mouse operation ('Right click') using Python library pyautogui

Next we are going to capture current system time using below instruction

    #Save current date and time
    timestamp = time.strftime('%d-%b-%Y_%H-%M-%S', time.localtime())

We used strftime function which convert local time to any readable format you want. Visit this link for more details on the format

Now lets write the log in a text file named "shutdown_logs.txt". Please note we are using mode a+ so that we can append the log in the same file for each script run. You need to update the location of the text file accordingly

#Create a text log when system shuts down
f = open('C:\\Users\\jaq\\Snapshot\\shutdown_logs.txt','a+')

Next we are going to write the log in the text file and close the file

   f.write('Shutting down at {}'.format(timestamp))
   f.close()

Take screenshot before shutting down

#Take screenshot
image_location = 'C:\\Users\\jaq\\Snapshot\\image_' + timestamp + '.jpeg'
pyautogui.screenshot(image_location)

In the last part we are going to issue shutdown or restart command based on user inputs

#Shutdown system or Restart
if action in ['s', 'S']:
    os.system("shutdown -s -f -t 0")
if action in ['r', 'R']:
    os.system("Shutdown -r -f -t 0")

If you have noticed 'action' is nothing but the function parameter in the very first line of code. Depending on the action parameter, we are issuing command to shutdown or restart using system() function of OS module.

With this , we are done with timer() function. Now we need to call it from our main function. So lets defined our function as below.

if __name__ == "__main__":

    parser = argparse.ArgumentParser()
    parser.add_argument("-t", "--time", help="Enter time in minutes")
    parser.add_argument("-a", "--action", help="want to shutdown?")

Lets understand what we did above. We are using argparse module to create our user friendly command line interface or CLI. In the first line we are creating **ArgumentParser **object and then in the next two lines we are adding two arguments for time and action input.

Next we will be calling timer function which we defined earlier. We will call the timer function only if we have the input time argument.

    args = parser.parse_args()
    if args.time:
        timer(int(args.time), args.action)

That's it, we completed writing a CLI script in Python to keep our computer awake.

Conclusion

If you want to take a look at complete code, its available in my Github repository here https://github.com/jaqsparow/keep-system-active
Though it sounds funny that we wrote a script to keep our system active or pretend that we are always online :), however in this post, we learned some important Python modules to do mouse operation, to write a file, to take screenshot and to shutdown or restart computer all with just few lines of code. We have also covered beautiful python CLI module argparse which is easy and efficient way of writing command line interface scripts.

complete code in Github

Visit this Github page for the complete script click here

How to get email alert when your website is down using shell and Python?

Javed Shaikh — Fri, 11 Dec 2020 03:21:12 +0000

There are many ways to monitor and set alerts for your web server using third party apps but they don't come with free, be it Pingdom or Amazon's CloudWatch. In this post I am going to show how we can write and setup our own alert tool with just few lines of code.
However before going further into codes let us see what are the commands and utilities available in Linux to monitor web server health.

All the below commands are very basic and available on all Linux flavors. I am running these commands on Ubuntu 20.04 LTS for below demos

What we will learn in this article?

📗 How to check if a website is up and running or down?
📙 How to write a basic shell script?
📗 How to call Python program from shell script?
💡 How to send an email using Python?

How to check if web server is up and running?

There are many commands and utilities available to monitor if your website or app is up and running fine. I have listed few of them. Lets checkout

👉 curl
👉 wget
👉 ping

Curl

curl is a very powerful utility to send and receive data over many supported protocols such as HTTP,FTP, SMTP...
We are going to use curl with I options to fetch the headers from a website. Based on the header we can identify if server is down or up. Lets try it

*USAGE: * curl -I SERVER IP

shaikh@ubuntu:~/scripts$ curl -I "https://hashnode.com"
HTTP/2 200 
date: Thu, 29 Oct 2020 17:35:18 GMT
content-type: text/html; charset=utf-8
cache-control: public, s-maxage=1800, max-age=0
etag: W/"4436a-AFOlSSR9igz5hRyOL+SdGTPJfxk"
vary: Accept-Encoding
x-frame-options: Deny
x-powered-by: Next.js
cf-cache-status: HIT
age: 1020
cf-request-id: 061706b33c0000051308a22000000001

As shown above curl -I returns header and the very first line (HTTP/2 200 ) tells us that webserver for hashnode is up and running. Note that server is up and running as long as the response code is 200,301,302,308. In above example response code is 200 , so its up.

wget

This is another utility in linux used as network down loader. We will use option -S to get the headers from a website.
USAGE:

wget -s SERVER : This will fetch and download the headers and pages of a website

wget -S --spider SERVER :This will work exact similar way as above except that it will behave as web spider and wont download the pages.

shaikh@ubuntu:~/scripts$ wget -S --spider http://shaikhu.com
URL transformed to HTTPS due to an HSTS policy
Spider mode enabled. Check if remote file exists.
--2020-10-29 10:47:55--  https://shaikhu.com/
Resolving shaikhu.com (shaikhu.com)... 192.241.200.144
Connecting to shaikhu.com (shaikhu.com)|192.241.200.144|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  Alt-Svc: h3-29=":443"; ma=2592000
  Content-Length: 206
  Content-Type: text/plain; charset=utf-8
  Date: Thu, 29 Oct 2020 17:47:55 GMT

As shown above the returned response code is 302 for redirection, so our server is up.

Ping

Ping is most common utility to check the web server's health. It sends ICMP ECHO_REQUEST to network host to get the up information. We are going to use -C count option to send n number of request packets.

USAGE

ping -c COUNT SERVER
Above command will send COUNT no of request packets to SERVER

shaikh@ubuntu:~/scripts$ ping -c 3 shaikhu.com
PING hashnode.network (192.241.200.144) 56(84) bytes of data.
64 bytes from 192.241.200.144 (192.241.200.144): icmp_seq=1 ttl=55 time=38.2 ms
64 bytes from 192.241.200.144 (192.241.200.144): icmp_seq=2 ttl=55 time=34.0 ms
64 bytes from 192.241.200.144 (192.241.200.144): icmp_seq=3 ttl=55 time=36.0 ms

--- hashnode.network ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 33.954/36.039/38.178/1.724 ms
shaikh@ubuntu:~/scripts$

As shown above we sent 3 packets and received 3 packets as well, so that means our website is up and running

How to write a basic shell script?

Now that we know which commands and utility to use to know the status of our website, so lets get started. We are going to issue the command

Step 1 : Create a new file for shell script

Lets create a new file named alert.sh and save it to your home folder. We can use any of the above command to know the status of a website. I am going to use wget's web spider option to get the details. However you can use any of them.

Before writing any script, note the exact location of bash

shaikh@ubuntu:~/scripts$ which bash
/usr/bin/bash

Now lets start writing shell script alert.sh

#!/usr/bin/bash
MYHOST="https://YOUR WEBSITE ADDRESS"
if wget --spider -S $MYHOST 2>&1 | grep -w "200\|301\|302\|308" > /dev/null
then
echo "server is up"
else
#Server is down"
/usr/bin/python3 send_alert.py
fi

This is a very simple script to monitor your web server and send an email notification if its down. Lets understand what we did line by line.

In the very first line we are providing location of bash which was found in the previous step above
then we have saved our website address in a variable named MYHOST
From line number 3 we are executing the wget command in an if else statement. If the return code from wget is other than 200,301,302 and 308 then control will move to else block considering server is down or unreachable.
In this step you can change the command to ping or curl as you like

In the else block we are executing a Python file named send_alert.py to send an email notification. You can ignore this step if you are sending email using MAILX of linux package within shell script. However I dont want to install any third party library and want to use existing Python library to send the email.

How to send an email using Python?

Writing an email using Python is very simple. No third party library is needed to send an email. Here we are using Python's inbuilt library smtplib to send an email.
As shown rest of the code is pretty simple. We have defined FROM_EMAIL,Password, TO_EMAIL,header and finally body

Lets see what our send_alert looks like

import smtplib

FROM = 'FROM EMAIL ADDRESS'
PSWD = 'YOUR PASSWORD'

TO = 'TO EMAIL ADDRESS'

#Subject and header
SUB = 'Subject: Alert for shaikhu.com \n'
header = 'To:' + TO + '\n' + 'From: ' + FROM + '\n' + SUB

#Message
body = """
MAY DAY! MAY DAY!! MAY DAY!!!
Your webserver for shaikhu.com is DOWN."""
message = header + body
#Define function to send email
def sendalert(msg):

    smtpObj = smtplib.SMTP('smtp.mail.yahoo.com', 587)
    smtpObj.ehlo()

    smtpObj.starttls()
    smtpObj.login(FROM, PSWD)

    smtpObj.sendmail(FROM,TO,msg)
    smtpObj.quit()
if __name__ == '__main__':
    sendalert(message)

Congratulations!!🥳. We have just wrote our own script to monitor our server and send us an email notification if it is down. Now all we have to do is put this script on a scheduler or cron table so that the script will run every n no of minutes and keep monitoring our website.

Add the job to scheduler or crontab

If you want to know how to setup job scheduler using crontab then please do check my article https://shaikhu.com/how-to-schedule-and-manage-tasks-using-crontab

*/5  *  *  *  *  sh alert.sh

As per above cron table, our alert.sh will be executed every 5 minutes. That means alert.sh will check our website health every 5 minutes and will send me an email if the website is down.

This is how the email from my script looks like.

Conclusion

Though there are many websites are providing website alert service but not all of them are free. As seen in this article with just few lines of code, you can build your own alert app that will keep monitoring your website every moment and will send an email notifications when its down.

We have learned some important commands and codes such as how to check website is down or up and how to send email using Python and also how to write a basic shell script.

Hope this tutorial helped you learn something new today. Let me know your thoughts in the comment section 😃.

References

https://docs.python.org/3/library/smtplib.html#smtplib.SMTP.sendmail

How to auto change desktop wallpaper every minute using Python

Javed Shaikh — Sat, 07 Nov 2020 16:06:32 +0000

This post was originally posted on https://shaikh.com
This post has been updated to include changes to support Linux operating system as well. So you can use this code to change desktop background in both linux and windows computers 🙂.

In this fun tutorial we are going to download a random wallpaper from **pexels **and set it to our desktop background every minute using Python. While it sounds like fun project, but we are going to learn some important concepts like how to make http request, how to download a file and save it to your local machine and most importantly how to set the wallpaper using Windows's system function.

Visit my GitHub page for complete codes at https://github.com/jaqsparow/change-desktop-wallpaper

Note: All these code snippets are tested on windows 10 and Ubuntu 20.04 LTS, with Python v3.9.0

What we will build?

We are going to build a CLI app using Python to change desktop wallpaper every given number of minutes. Every wallpaper downloaded from internet will be unique and our app will change the wallpaper based on the time we set.

What we will learn in this post?

How to generate random number? 📗
How to make http request using Python? 📕
How to download an image to your local machine?📙
How to set wallpaper using Python 💡
How to call windows's C++ system function from Python💡

Get API key to download wallpaper

First of all, we need an api key to make HTTP request and download image. For this tutorial, I am going to use pexels. Follow these steps to get the api key

Step 1: Register for a free account

Visit this link and then click "Join" link. Provide your name and email details in the text provided and create an account
https://www.pexels.com/join-consumer/

Step 2: Get started

Once you registered and confirmed your email visit https://www.pexels.com/api/ and click "Get Started" and it will take you to new page for applications. Click new application and provide your application details.

After above step you should be provided with an api key which we are going to use in the next step. So save this api key somewhere in your machine.

Note - Do not abuse the Pexels api. They are providing 200 requests per hour with a free account. So respect 👏

Install requests module

To make an http request, we need a module requests . so lets install this module

pip install requests

Now we have everything setup and lets go to next step to start coding 💻.

Start coding..

First we need to import all the modules we need to make this app. So lets get started.

import argparse
import os
import time
import random
import requests
import ctypes
import platform

Why we need these modules

*argparse: * Since this is a CLI app, we need to pass arguments to our app using this module
*os: * We need this module to find the current working directory and full path of our wallpaper
*time: * This module will be used to call our function every given minutes and also to use sleep() method
random: To generate random number , so we can make random request url every time
*requests: * We are using this module to make http request and to download the wallpaper
ctypes: ** This module is to call foreign function in Python. It provides C compatible data types and allows to call DLL and shared libraries. We need this to call **SystemParametersInfoW to change our wallpaper
*platform: * We are using this inbuilt module to know the operating system where this code will run. Based on that we will call system function to change the wallpaper

Generate a random number

Lets create a function named get_wallpaper() and generate a random number inside this function, which we will use to make random http request URL.

def get_wallpaper():
    #Random number
    num = random.randint(1,99)

Make http request

Now lets save our api key inside the function as shown below. Replace your api keys in the field and also set the url for pexel request as shown below

    #Set your api key here
    payload = {'Authorization': '<Your api key goes here>'}
    #Search query
    query = 'flower'
    #URL for PEXELS
    url = 'https://api.pexels.com/v1/search?per_page=1&page=' + str(num) + '&query=' + query

With above codes, we are creating a random URL with page parameter and query =flower 🙂. Change the query depending on what type of wallpaper you want to download like nature, mountain, airplane or anything you want. I am downloading wallpaper with query flower
So we have the url, now lets make the http request to that url with get method

    res= requests.get(url, headers=payload)

In above code, we are making http request to Pexels with api key in the headers. Also note that we have saved the response in res variable which we are going to use in our next step.
Now its time to download the image. Lets do that using below code

    #Get the url of the image from the response
    if res.status_code == 200:
        img_url = res.json().get('photos')[0]['src']['original']
        #Make request to get the image
        img = requests.get(img_url)
        #Write and save the imgae locally with name temp.jpg
        with open('temp.jpg', 'wb') as f:
            f.write(img.content)
    else:
        print('error in making http request')

Lets understand the above code. In the very first line we are checking the status code from our http request in the last step. Please note that for every http request, we will get the response with status_code. If the status code is 200, that means the request was successful and a good response object should be returned.
We are converting the response content into json object so that we can use it parse the image url.

In the second line we got the image_url after parsing the JSON response object.
We are making another http request to download the image and finally we are saving the image in our local system using write method. Please note that we are saving the file using wb mode so that a new file will be downloaded and replaced the old one. We are saving the wallpaper as "temp.jpg"

Finally we are printing 'error message' if the status code returned is not 200

Set wallpaper using Python

In this section we are going to create a new function set_wallpaper() as follow

def set_wallpaper():
    get_wallpaper()
    #Check the operating system
    system_name = platform.system().lower()
    path = ''
    if system_name =='linux':
        path = os.getcwd()+'/temp.jpg'
        command = "gsettings set org.gnome.desktop.background picture-uri file:" + path
        os.system(command)
    elif system_name == 'windows':
        path = os.getcwd()+'\\temp.jpg'
        ctypes.windll.user32.SystemParametersInfoW(20,0,path,0)

Lets take a look what we did above. In the first line inside set_wallpaper() we are calling the function get_wallpaper() to download the image and save it to temp.jpg file. In the next line, we are assigning path variable with the complete path of the image that we just downloaded.

Please note we are using platform.system() function to identify the operating system and based on system we are running respecting command to change the desktop wallpaper.

How to call Windows' system function?

The next line shown below is interesting. With the help of ctypes module, we are calling Windows's system C++ function SystemParametersInfoW() to change the wallpaper. Lets see different parameters of this function.

ctypes.windll.user32.SystemParametersInfoW(20,0,path,0)

Syntax

BOOL SystemParametersInfoW(
  UINT  uiAction,
  UINT  uiParam,
  PVOID pvParam,
  UINT  fWinIni
);

uiAction: ** this is systemwide parameter (such as desktop/icon/menu..)to be set or retrieved. > We are using SPI_SETDESKWALLPAPER or 0x0014 to set this parameter. Please note 0x0014 is nothing but **20 in decimal which are sending in above code.
uiParam: This is second parameter which depends system parameter of windows. We are sending 0 as for most of the request. as suggested by the usage document.
pvParam: This parameter can be used to send the argument for action set in the first parameter. With this we are sending full path of the image
fWinIni: This parameter can be used to set or update user profile or broadcast the change to all the top level windows. In this case we don't want to broadcast anything , so we are sending 0

Return code

If the above function execution succeeds, the return value will be nonzero else it will be zero

With this we are done with the coding to change and set the wallpaper. Now its time to call this function every minute

Call function every minute to change the wallpaper

Here we will be calling from the main function based on the user input. Since this is CLI app, we are going to ask user for what would be the frequency to change the wallpaper? every minute, or every 5 minutes or any other value?

if __name__ == "__main__":

    parser = argparse.ArgumentParser()
    parser.add_argument("-t", "--time", help="Enter time in minutes")

    args = parser.parse_args()
    minute = int(args.time)
    while(True):
        time.sleep(minute*60)
        set_wallpaper()

With the above code we are setting up the entry point of our app. First two lines are to setup CLI arguments where we are setting up input argument -t which will take number minutes we will be calling set_wallpaper(). We have used sleep() method to delay execution and call set_wallpaper() every n number of minutes as per user input.

Ok. All done now lets run the app 🙂

How to run the app?

Run in the CMD as shown below.
To run every minute

C:\Users\jaqsp>wallpaper.py -t 1

Or if you want to change wallpaper every 5 minutes

C:\Users\jaqsp>wallpaper.py -t 5

Conclusion

Hope this post will help you understand some important concepts like how we made http requests and downloaded an image to our local system. Also how we used Windows's system function or(linux's gnome setting) and called it from Python program to change the wallpaper. There are many other parameters available for this system function that we can use to perform some other interesting cases. Visit the link below to know more about this function.

Visit my GitHub repository to see complete codes. Thanks for reading:)

References

How to setup alarm for CPU usage using IFTTT ?

Javed Shaikh — Fri, 06 Nov 2020 21:33:57 +0000

Monitoring CPU and memory usage are one of the top todo checklist for a backend engineer. Sometimes you wont even notice when your server is down due to high CPU usage unless you login and manually check the system. In this tutorial, we are going to setup alarm for CPU Usage by most easy way possible(at least I think so) using a simple shell script and IFTTT web hooks.

What we are going to learn?

📗 What is IFTTT and how to create an applet?
📙 How to check CPU usage of a system?
📗 How to make web request to IFTTT from your server?
📙 How to write a script to check CPU usage and schedule it on crontab?

What is IFTTT?

IFTTT stands for If This, Then That. It is simple web based service that lets you connect your device, server or apps. It is used to create conditional statements or actions that triggered by the changes in your apps, devices or servers. So it helps automate your tasks for your web apps and devices. Lets get started.

First we need to create a free account and login to IFTTT

To use IFTTT, we need to create an applet (Free account can create up to 3 applets)

How to create applet to set alarm?

Go to the top right and click Create link. You should see similar to below page. Now click Add button on "If this add" link

On next page, You need to select a service. In this case we are going to use IFTTT web hook. So search for web hook and select it to set it up. Click "Receive a web request" and set the event name.

As shown below we are setting the event name as "high_cpu_usage" and then we will click "Create Trigger" to create the trigger.

So far we have created the trigger event, next we are going to create action service like what action needs to be taken if the above event is triggered. I am going to set EMAIL service so that an email will be generated when CPU usage will be high.

So click "Then That Add" and choose EMAIL service and set the subject and body of the email that will be sent to your account. Then Review and finish the setup.

How to make web request to IFTTT?

At this moment we are done with setting up our IFTTT applet. Now we need to know the complete web URL of our applet so that we will make web request to notify cpu usage is high.
To do that go to your account and lick my service, then webhooks and then 'documentation' page on the top right corner. You should see something like below.

We need to make a small change to add event name to the above URL.
So our correct URL to make web request should look like this and this is how are going to make a web request to IFTTT.

curl -X POST https://maker.ifttt.com/trigger/high_cpu_usage/with/key/boJOWPwAx6x2qjAWr3CUym

I have deleted webhook created for this tutorial. So please don't use the above key, Its not going to work 😃.

How to check CPU Usage?

There are many ways to check CPU usage of a server. I have already posted a complete article for CPU Utilization and we are not going to cover in detail here. Please go and check my article if you are interested https://shaikhu.com/how-to-monitor-cpu-utilization-in-linux

For this tutorial, we are going to use VMSTAT command. This command will tell current CPU and memory usage of the system.
Below command will tell CPU usage every second for 5 times. So with this command we can check CPU usage for last 5 seconds. We are going to use this entry in our next shell script to check the cpu usage.

VMSTAT 1 5

Shell Script to monitor CPU Usage

Since we got the idea how to check CPU Usage lets start writing a simple script to monitor CPU usage.
Here is how our script looks like.

#!/usr/bin/bash
#Set Threshold for CPU Usage
THRESHOLD=90.0
#CPU usage for last 5 seconds
cpu_usage=$(vmstat 1 5 | awk '{if(NR > 2) {sum+=$15}} END {print 100-sum/5}')
if [ $(echo "$cpu_usage > $THRESHOLD" | bc) -eq 1 ]
then
#SET ALARM using IFTTT webhook
curl -X POST https://maker.ifttt.com/trigger/high_cpu_usage/with/key/dhFYYNRL4to_1k5O1NOUd3
fi

Lets understand the above script.

First line is the bash interpreter location.
In the second line we are defining a constant for CPU threshold value.
In the next line we are executing VMSTAT command and saving CPU usage in a variable named cpu_usage
In the next line no 4, we are comparing CPU usage value with the threshold value set in the second line.
If the CPU usage is higher than the threshold, we are making a web request to IFTTT with the use of CURL command, That's it! Lets save this script as cpu_alert.sh in your system.

Run this CPU usage script every 5 minutes.

With the help of crontab we are going to schedule this shell script ,so that script will be running every 5 minutes to monitor our CPU health.

*/5  *  *  *  *  sh cpu_alert.sh

With this setting in crontab, our script will run every 5 minutes and check CPU usage. If you want to know more about how to set crontab and schedule jobs, then check my post How to use crontab .

If the cpu usage is higher than Threshold, it will make a web request to IFTTT and IFTTT will send us an email alert 😃. Now how easy that was!!!

Output of above work

This is how IFTTT email looks like when CPU usage reaches the threshold limit.

Conclusion

IFTTT is a very easy and also powerful web based service that can be used to connect your apps and devices. As shown in this tutorial, with just few lines of script, we have build a powerful tool to monitor our system's CPU usage. Same logic can be used and extended for memory usage as well.
The amazing thing is that we are not writing codes to send an email. We are just making a web request conditionally to IFTTT and IFTTT is sending us the email alert.

Hope this tutorial will help you build some amazing apps using IFTTT. Let me know your thoughts in the comment section and dont forget to check my other How to tutorials listed below. Thanks!

Flask or bottle

Javed Shaikh — Thu, 05 Nov 2020 17:54:50 +0000

Both Flask and Bottle are lightweight micro web framework for Python. I never used bottle but I am going to give it a try?

Which one you prefer and why?

How to trigger a job or program from anywhere with a SMS

Javed Shaikh — Thu, 05 Nov 2020 16:49:52 +0000

Its holiday season every where and we don't want to open laptop and system all the time when travelling and enjoying our personal time. In this post, I am going to show how we can run some important jobs or programs with just one SMS when we are on the go.

Introduction

In this tutorial we are going to learn how we can trigger a job or program with just one SMS. For this tutorial we are going to use Twilio API for sending and receiving SMS. With just few lines of codes, we will implement a very basic app to receive sms and trigger jobs based on the sms. So lets check out what we are going to learn in this tutorial.

What we will learn in this article?

📗 How to setup programmable SMS api on Twilio?
📙 How to write a basic Flask app?
📗 How to setup ngrok to access your app outside localhost?
📙 How to run a shell command from Python?

### How to setup programmable SMS api on Twilio?

In the very first step, we are going to need a mobile number and API where we can send a SMS and run a script. For this purpose, we are going to use Twilio's beautiful and easy to use API to setup our SMS app.

Get your free $10 credit to use Twilio SMS api

Twilio is providing $10 credit for new user signup and interesting thing about Twilio trial account is that you don't need to have a credit card to try its free service. Please click here and visit Twilio to create a free account and get $10 credit to use its SMS api during trial period.

Once your free account is created, login to your account and create a project from top right corner of your dashboard.
Then go to All products & services from the left tab and click ** # Phone numbers*. You should see screen similar to below. Click **Buy a Number* and get a new number where we will send a SMS.

Now that we have a number registered, lets go to next step and setup a Flask app.

How to write a basic Flask app?

As I mentioned we are going to use a very basic Flask app. Before writing our Flask app, lets install virtual environment so that our app will be running independently with the packages installed within the environment.

Step 1: Create virtual environment

Lets install virtual environment.

pip3 install virtualenv

Now create a virtual environment within a new directory smsapp

shaikh@ubuntu:~/scripts/smsapp$ virtualenv venv

Activate the virtual environment

shaikh@ubuntu:~/scripts/smsapp$ source venv/bin/activate
(venv) shaikh@ubuntu:~/scripts/smsapp$

Step 2: Install Flask

Install Flask app using below entry

pip3 install flask

Lets write very basic flask app with below lines of code.

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
  app.run()

Run this app with python3 app.py as shown below and check the result at http://localhost:5000

You should see the output "Hello World!" in your localhost at port 5000. As shown above you can check the output at http://localhost:5000

Step 3: How to install and access app using ngrok

Now that we have installed our flask app and able to run it in our localhost but we cannot access our app outside our local network. To access this app over internet lets install ngrok which will allow us to expose our app over the internet.

Install this with below command (for Ubuntu)

snap install ngrok

Activate ngrok to expose our app to the internet.

python3 app.py &
ngrok http 5000

This is how it will look once we run above commands

We need to use the ngrok http URL from above output. Lets checkout the app at this url on any device over the internet.

How to run a shell command from Python?

To run a shell command from Python, we will use OS module's special function popen().

os.popen(command)

So this is how our Flask app looks like if we want to run a command using popen()

from flask import Flask
import os
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'
@app.route('/command',methods=['POST'])
def command():
    command = "/usr/bin/python3 ~/scripts/send_alert.py"
    os.popen(command)
    return 'Command executed'

if __name__ == "__main__":
  app.run()

Please note that we have created a new route /command to execute the command to run another python script send_alert.py. You can execute any command you want. For this example I am running a script to send email alert.

So this is how our Flask app will trigger a new process and execute the command. Now we are going to configure Twilio so that when we send sms to its number, Twilio should trigger a HTTP request to our /Command route.

Lets head over to Twilio again and click the mobile number you purchased in previous steps and then update the webhook for "A message comes in" as shown below

Now when we send a sms to Twilio number, It will make HTTP Post request to above link and when it does, our command() function is going to be executed 😃

Now if you want to run multiple jobs based on SMS body (like JOB1,JOB2) we need to install Twilio library so that we can parse SMS and run the jobs accordingly.
Here is the app.py looks like with that feature.

from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse

app = Flask(__name__)

@app.route('/command', methods=['POST'])
def command():
    number = request.form['From']
    message_body = request.form['Body']
    action = message_body.split()[0]

    response = 'command ' + action + ' executed'

   if action.lower()== 'job1':
        command = "/usr/bin/python3 ~/scripts/job1.py"
        os.popen(command)

   elif action.lower()== 'job2':
        command = "/usr/bin/python3 ~/scripts/job2.py"
        os.popen(command)
   else:
        response = "Invalid request

    resp = MessagingResponse()
    resp.message(response)
    return str(resp)

if __name__ == '__main__':
    app.run()

As shown above, with the help of Twilio library we are able to parse text message and run jobs based on incoming message content. Now if we send a text 'JOB1' to the Twilio mobile number, Twilio will make a HTTP POST request to our flask app and execute our job1 script. Once job1 script is executed a return sms will be received on your mobile number that 'command executed' else you will receive 'Invalid request' in case you send some invalid job requests:)
In above Flask app, you can replace the jobs or program names and add any number of programs or commands or even shell scripts if you want to execute them.

Output:

Here is the output of above script and this is how our app should work.

Conclusion

Congratulations on building your SMS app to run a job with just a SMS🥳.. In this article we learned some interesting concepts like how to run a shell command from Python script and how to integrate third party sms api to our FLASK app.

Hope this will help you build some amazing sms apps. Let me know your thoughts in the comment section and dont forget to read my other interesting posts in below links.

Thanks for reading this article. Hit like if you want more such posts 😊

Visit my GitHub page for complete code at GitHub

References

Note: This article was originally published at https://shaikhu.com/how-to-trigger-a-job-or-program-from-anywhere-with-a-sms

DEV Community: Javed Shaikh

What is fork() system call and how to fork using Python

What we will learn in this post?

Introduction : What is fork system call?

How to identify parent and child process?

Return code of fork()

How to fork using Python?

DEMO 1: To check process ID of any process

DEMO 2: Create one child process using fork()

Demo 3: To identify parent and child

Demo 4: Lets create two child process

Demo 5: Lets do it for fun

Conclusion

References

Similar posts

Files, Folders and Python

How to get current working directory and how to change it?

How to change working directory?

How to get list of all files and folders inside a directory ?

How to get file size ?

How to get file create, update and access date, owner name and size of file

How to get folder size and empty files

How to remove a file and directory?

Conclusion

References

Check my other posts

How to manage background processes in Linux

bg: Move a job to background

How to start command and run it in the background

jobs : command to check the job status

-l : lists process IDs in addition to the normal information

-n : lists only processes that have changed status since the last

notification

-p : lists process IDs only

-r : restrict output to running jobs

-s : restrict output to stopped jobs

kill %n: to kill a job with job id n

fg : command to move a job to foreground

ps command to see all the processes

ps ax : to see all the process currently active in the system (Use my previous post to check the commands for real time process). It will be a very long list, so use less/more parameters

ps T: list all processes running on current terminal

Conclusion

- bg and fg to move a job to the background and foreground ##### respectively.

- jobs command to list all the jobs active in the current terminal.

- kill command to kill a job

- ps command to see list of all the active and running processes

We also learned how to start a job in the background using &. If you want to know real time memory and cpu usage and control all the processes running on your system please check my last post https://shaikhu.com/how-to-monitor-cpu-utilization-in-linux-ckgcp9gex07g8pas1ees0hz7t

How to monitor CPU utilization in Linux

top

htop

nmon

vmstat

Conclusion

How to schedule and manage tasks using crontab

What is Crontab

How to view the crontab or cron table

How to set date and time of job run

What if you want to run a job at random minutes every hour

Conclusion

How to access Google Sheets using Python and gspread

Prerequisite

Create Google service account

Install Python libraries

Lets start coding

Retrieve all records from a worksheet

Retrieve cell values from a worksheet

Creating another worksheet

Deleting a worksheet

Updating cell value

Insert a new row

Conclusion

How to keep your computer awake using Python

What this script will do?

What we are going to learn with this script

Start coding

Conclusion

complete code in Github

How to get email alert when your website is down using shell and Python?

What we will learn in this article?

How to check if web server is up and running?