DEV Community: petercour

Python as terminal commands

petercour — Thu, 25 Jul 2019 20:30:04 +0000

Python is often used when creating scripts. But did you know you can run it straight from the command line?

No, I don't mean running a .py file from the command, but running a program input.

Whats Python?

Python is a programming language.

The code below won't make sense if you don't know the Python programming language. You should be familiar with the idea of for loops.

Python code as command

You can run Python on a single line with command line as input, like this:

python -c "for i in range(1,10): print(i)"

This then outputs the numbers.
Now you can use the alias command to create custom commands that run this python command

alias forloop='python -c "for i in range(1,10): print(i)"'

Then in your terminal

# forloop

You can store these commands in your .bashrc file.

Is this a good idea? from a security point of view probably not.
See Inject code in Python process.

Deep Learning, Object detection with Python

petercour — Thu, 25 Jul 2019 18:01:56 +0000

Luminoth is a Deep Learning toolkit for Computer Vision. It uses tensorflow underneath, so install tensorflow or tensorflow-gpu (if you have a gpu).

pip install luminoth

One of these then

pip install tensorflow
pip install tensorflow-gpu

One of the things you'll need is a faster computer (and probably a GPU). In my case, running scripts like this sometimes freezes my computer.

Download an image, like this one

Then run the command

lumi predict bicycling-1160860_960_720.jpg

Then confirm you want to download

Checkpoint not found. Check remote repository? [y/N]: y
Retrieving remote index... done.
2 new remote checkpoints added.
Checkpoint not present locally. Want to download it? [y/N]: y
Downloading checkpoint...  [####################################]  100%
Importing checkpoint... done.
Checkpoint imported successfully.

Results in this json output:

{"objects": [{"prob": 0.9997, "bbox": [294.0, 231.0, 468.0, 536.0], "label": "person"}, {"prob": 0.9971, "bbox": [494.0, 289.0, 578.0, 439.0], "label": "person"}, {"prob": 0.997, "bbox": [727.0, 303.0, 800.0, 465.0], "label": "person"}, {"prob": 0.9965, "bbox": [555.0, 315.0, 652.0, 560.0], "label": "person"}, {"prob": 0.9934, "bbox": [569.0, 425.0, 636.0, 600.0], "label": "bicycle"}, {"prob": 0.9932, "bbox": [326.0, 410.0, 426.0, 582.0], "label": "bicycle"}, {"prob": 0.9334, "bbox": [744.0, 380.0, 784.0, 482.0], "label": "bicycle"}, {"prob": 0.8723, "bbox": [506.0, 360.0, 565.0, 480.0], "label": "bicycle"}, {"prob": 0.8142, "bbox": [848.0, 319.0, 858.0, 342.0], "label": "person"}], "file": "bicycling-1160860_960_720.jpg"}

You can get the list of checkpoints

lumi checkpoint list

you can run the progrma

#!/usr/bin/python
from luminoth import Detector, read_image, vis_objects

image = read_image('traffic-image.png')
detector = Detector(checkpoint='traffic')
objects = detector.predict(image)

print(objects)

vis_objects(image, objects).save('traffic-out.png')

then outputs the objects like above.

Related links:

Listing files, dirs and subdirs with Python

petercour — Thu, 25 Jul 2019 15:29:54 +0000

If you know Python programming, there's a lot of things you can do. You can use the os module to list all files in the directory (including subdirectories).

If you use Linux or Mac, you can even list every file on your computer (search from /). Or you could browse from your home directory.

If you want all files from the current directory and subdirs, use '.'.

#!/usr/bin/python3
import os
path = '.'
for dirpath, dirname, filenames in os.walk(path):
    for filename in filenames:
        print(os.path.join(dirpath,filename))

This lists all files in the directory, including all hidden files.
For readability you can change it to:

#!/usr/bin/python3
...
    file_and_path = os.path.join(dirpath,filename)
    print(file_and_path)

File interaction

You can show only specific files and interact with them. For instance, you can show every image on your computer

#!/usr/bin/python3
import os
import cv2
import time

path = '/home/linux/Desktop'
for dirpath, dirname, filenames in os.walk(path):
    for filename in filenames:
        file_and_path = os.path.join(dirpath,filename)
        if ".jpg" in file_and_path or ".jpeg" in file_and_path:
            print(file_and_path)

            try:
                # show image
                image = cv2.imread(file_and_path)
                cv2.imshow(file_and_path, image)
                cv2.waitKey(2000)
                cv2.destroyAllWindows()
                #time.sleep(1)
            except:
                print('Error loading image ' + file_and_path)

Related links:

Inject code in Python process

petercour — Thu, 25 Jul 2019 14:46:06 +0000

Python is widely used for software development. By universities, companies and many more. So is Python an utopia that will solve all computer problems?

Did you know you can inject code into a running Python process? You can use Pyrasite for that. This needs Python 2.4 or higher.

You can watch a demo here: demo

It comes with several programs, including a shell

$ pyrasite-shell
Usage: pyrasite-shell <PID>

So you can inject a payload

$ pyrasite <PID> payload.py

You can find some example payloads here.

So if you have the process id , you can inject anything into the running program.

There's also a GUI. It lets you easily monitor, analyze, introspect, and alter running Python programs.

Related links:

What is kmeans?

petercour — Wed, 24 Jul 2019 16:13:39 +0000

kmeans is a clustering algorithm for Machine Learning purposes. A cluster is another word for group.

So what do I mean with kmeans?

Starting situation

The goal of the algorithm is to find groups in datapoints. In the context of machine learning, you measure data.

For instance, you could measure height and salary. Then you would have 2d vectors (height,salary) that you could plot on a 2d plot.

There are K groups, or formally, K clusters. There must be at least 2 groups, because 1 group doesn't need the algorithm to run.

End goal

Based on the previous input (the data points). Then the algorithm will find K groups, in this case K=3.

So now if you have a new data point, the group that's closest is the type. Which K is best? that's a bit of a searching quest.

The code

The program below uses Machine Learning and the modules sklearn, matplotlib to create the plots and run the kmeans algorithm. It creates the images shown in this article.

#!/usr/bin/python3
from sklearn.datasets import make_blobs
from matplotlib import pyplot as plt
from sklearn.cluster import KMeans

X, y = make_blobs(n_samples=200,
                  n_features=2,
                  centers=4,
                  cluster_std=1,
                  center_box=(-10.0, 10.0),
                  shuffle=True,
                  random_state=1)

plt.figure(figsize=(6, 4), dpi=144)
plt.xticks(())
plt.yticks(())
plt.scatter(X[:, 0],X[:, 1], s=20, marker='o')
plt.show()

n_cluster = 3
k_means = KMeans(n_clusters=n_cluster)
k_means.fit(X)
print("kmean: k={}, cost={}".format(n_cluster, int(k_means.score(X))))

labels = k_means.labels_
centers = k_means.cluster_centers_

markers = ['o', '^', '*']
colors = ['r', 'b', 'y']

plt.figure(figsize=(6, 4), dpi=144)
plt.xticks(())
plt.yticks(())

for c in range(n_cluster):
    cluster = X[labels == c]
    plt.scatter(cluster[:, 0], cluster[:, 1], marker=markers[c], s=20, c=colors[c])

plt.scatter(centers[:, 0], centers[:, 1], marker='o', c='white', alpha=0.9, s=300)
for i, c in enumerate(centers):
    plt.scatter(c[0], c[1], marker='$%d$' % i, s=50, c=colors[i])

plt.show()

Related links:

Translate with the snake (Python)

petercour — Wed, 24 Jul 2019 15:20:16 +0000

There are some sites that offer language translation. You may know Google translate which is one of them.

Did you know you can use those sites from Python code? Welcome to the jungle!

To do so you need the modules requests and BeautifulSoup. These modules are used to make the web request and fetch data. The example code below translate a string from English to French.

#!/usr/bin/python3
import requests
from bs4 import BeautifulSoup

def getHTMLText(url):
    try:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        return r.text
    except:
        print("Get HTML Failed!")
        return 0

def google_translate(to_translate, from_language="en", to_language="fr"):
    base_url = "https://translate.google.com/m?hl={}&sl={}&ie=UTF-8&q={}"
    url = base_url.format(to_language, from_language, to_translate)

    html = getHTMLText(url)
    if html:
        soup = BeautifulSoup(html, "html.parser")

    try:
        result = soup.find_all("div", {"class":"t0"})[0].text
    except:
        print("Translation Failed!")
        result = ""

    return result

if __name__ == '__main__':
    print(google_translate("Hello World!"))

The important part here is the last line, a function call to google_translate with the text to translate as parameter.

If you run it:

python3 example.py
Bonjour le monde!

Related links:

Read data with pandas

petercour — Wed, 24 Jul 2019 15:05:31 +0000

In Python there's a module that helps you parse data. Data can be in many forms (files, tables, excel, sql, json). There exists so many data sources for historic reasons.

That module to work with data is named the pandas module.

You may know you can use the Pandas module for data analysis. But did you know there are many ways to read data?

pd.read_csv(filename)
pd.read_table(filename)
pd.read_excel(filename)
pd.read_sql(query, connection_object)
pd.read_json(json_string)
pd.read_html(url)
pd.read_clipboard()

Yes, you can read data from many sources. These methods allow you to quickly grab your data

#!/usr/bin/python3 
import pandas as pd
data = pd.read_csv('yourfile.csv', header=None)

If you are using MySQL as source

#!/usr/bin/python3
db = MySQLDatabase(DATABASE_HOST, DATABASE_USER, DATABASE_PASSWORD, DATABASE_NAME)
db_work_view = db.get_work_view()
connection = db_work_view._db_connection
df_people = pd.read_sql('select * from people', connection)

Reading an excel file

df = pd.read_excel('File.xlsx', sheetname='Sheet1')

Well you get the idea. Pandas allows you to quickly fetch data from different data sources. It includes most of the existing data sources.

Related links:

Bulk converts files to QR codes

petercour — Wed, 24 Jul 2019 14:46:37 +0000

What's the craziest way to take backups? The cloud? floppy disks? QR codes?

QR codes started with the Japanese automobile industry. They are graphical codes you can use to store data.

You can quack QR codes with duckduckgo. Type "QR hello world" for it to generate a QR code containing "hello world". But did you know you can do it from code?

QR codes backup with Python

We'll convert files in bulk to QR codes with Python programming. This guide is for Linux computers.

Install zbar-tools and qrcode module

sudo apt install zbar-tools

Then you can convert all (.py) files in a directory to qr codes.

#!/usr/bin/python3
import qrcode
import glob

files = glob.glob("*.py")
for filename in files:
    data = ""
    with open(filename, 'r') as file:
            data = file.read()

    img = qrcode.make(data)
    img.save('qr-' + filename + '.png')

There's a limit to the file size, about 8 kilobytes. But for small files this actually works.

This creates a QR code when running the program for each python file in the directory (*.py).

You can decode a QR code with the command:

zbarimg qr-qrfs.py.png

Then you can print all your files on paper. If they files fit into the QR code that is.

Related links:

Python pip, what's installed?

petercour — Wed, 24 Jul 2019 14:26:45 +0000

You probably use pip thousands of time if you are programming Python. pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

When creating a Python project, you do so in a virtual environment (or maybe you don't?).

pip install your_module

Then you install packages for your project. So how do you know what all the installed modules are?

Easy, there's a command

pip list

So if you want to know which version of a module you have, combine with grep.

λ  ~  pip list | grep 'tensorflow'
tensorflow (1.14.0)
tensorflow-estimator (1.14.0)

If you are using Ubuntu Linux or similar, you may have the odd situation in which you have both pip, pip3, python and python3 installed. Did the Ubuntu creators have some evil plan?

In that case, just add 3 after the pip command. For other operating systems, you can just use Python and pip.

Related links:

Call DLL functions from Python

petercour — Tue, 23 Jul 2019 18:26:16 +0000

Do you hav experience with C programming?

You can call C functions from Python, with ctypes. What is ctypes?

ctypes is a foreign function library for Python. It provides C compatible data types, and allowscalling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

This how to do it in Python 2.x (if you still have that):

#!/usr/bin/python
from ctypes import *

libc = cdll.LoadLibrary("/lib/x86_64-linux-gnu/libc.so.6")
printf = libc.printf
printf("hello world\n")

For Python 3.x programs (yes the difference is one character)

#!/usr/bin/python3
from ctypes import *

libc = cdll.LoadLibrary("/lib/x86_64-linux-gnu/libc.so.6")
printf = libc.printf
printf(b"hello world\n")

Make sure the path to your shared library (libc.so.6) is right.

Chances are it's on another location. On Windows or Mac its different path and name. Otherwise it's very basic, and this should work for any C library.

Related links:

Generate QR codes in Python

petercour — Tue, 23 Jul 2019 16:43:32 +0000

QR code a type of matrix barcode first designed in 1994 for the automotive industry in Japan. These days it's used in many places like banking, cryptocurrency, advertisements and more.

Did you know you can generate these codes with Python? You should know the Python programming language. But even without to much practice, you can generate them.

Install the qrcode module

pip install qrcode

Then you can load it in Python

#!/usr/bin/python3
import qrcode
img = qrcode.make('hello world')
img.save('test.png')

This will output the QR code as an image (test.png).

So it's that easy. How much data can you store on a QR code?

According to stackoverflow

QR codes have three parameters: 
Datatype, size (number of 'pixels') and error correction level. 
How much information can be stored there also depends on these   
parameters.

Numeric only Max. 7,089 characters
Alphanumeric Max. 4,296 characters
Binary/byte Max. 2,953 characters (8-bit bytes)

You may be able to zip your data and bypass this limit to some extend.

Related links:

Fuzzywuzzy and Python

petercour — Tue, 23 Jul 2019 13:05:13 +0000

What's fuzzywuzzy?

It's a string matching module. A string is variable that can store (and modify) text. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package.

Fuzzy string matching like a boss.
How to get started? First you should know Python programming.

pip install fuzzywuzzy

Then you can use it like this:

#!/usr/bin/python3
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

r = fuzz.ratio("this is a test", "this is a test!")
print(r)

r = fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
print(r)

This outputs the ratio:

97
91

You can run this from the interpreter:

>>> fuzz.ratio("this is a test", "this is a test!")
    97

Another example of fuzzywuzzy:

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio("this is a test","a test this is")
50

Related links: