DEV Community: Piotr Maliński

Creating a local 2.5 Gigabit Ethernet network with a switch, USB-C, and PCIe cards

Piotr Maliński — Fri, 03 Nov 2023 17:17:03 +0000

If you want to make a local fast network to share large files between devices, PCs you can try with a 2.5GBit switch and USB-C/PCIe 2.5GBit network cards on the cheap from Aliexpress or other stores.

Sharing ISO over 1Gbit Ethernet

Sharing ISO over 2.5Gbit Ethernet

Starting new web service business from IT point of view — ChatPilot case study

Piotr Maliński — Tue, 14 Jun 2022 17:47:09 +0000

A company wants to start a new web service - a messaging aggregator for customer support. How do you go from an idea to a Python/Vue/Socket.IO web application? This is a story of ChatPilot.io told by it dev team.
Communication aggregator

ChatPilot is a communication aggregator for customer support. You run a business, you have your Facebook fanpage, Instagram profile, maybe even a WhatsApp number - and customers contact you through all those channels. To manage all of such communication companies tend to use aggregators. Those apps aggregate messaging platforms like Facebook Messenger or WhatsApp into a (usually) web application where business employees can manage business-customers communication way more efficiently.

There are apps that aggregate communication, there are even companies providing API/services for making of such applications. The market for such applications is already quite established and the services aren't cheap compared to other online services. If you make your aggregator it quickly will face a lot of matured applications with established brand and customer base. So how and what to do?

The answer would be to make an application that provides value over existing applications. You can't just "make an application", it has to be targeted from start at providing unique value and being very agile at adapting to early customer feedback or business opportunities. Easier said than done...

The team

I work for Social WiFi which offers a captive portal service for open WiFi networks. From the code side it's composed of serveral microservices and most of them run Flask and offer some sort of API or handle Celery queues. Few like the dashboard are made in Ember.js. The project is deployed to Google Cloud. Initial version of this service, more than 5 years ago, was written as a monolithic Django application that with time started having problems including things like geographic scaling ;)

ChatPilot was planned as an additional service in the company portfolio while also having some synergy with the captive portal service. For this project we created a 2-man team: CTO - planning, devops, code review and rubber ducky. Fullstack (Python/Fronted) developer - that would be me - to make things done. We also had a sort of a Product Owner position, which was mostly focused on UX/UI planning and review.

Social WiFi dashboard made with Ember.js

Research

Before we even started codding the business part of the company alongside our CTO did the research - looked at existing aggregators, also contacted some of our customers that used such apps and after a while they had some idea how the market looks like. One of discoveries was that one of our partners had customers using and older aggregator that wasn't that good, mostly because messages were slow to show up in the app or didn't showed up at all. Low quality/broken application is something you can compete against, yet it will be unlikely all of your competition has such obvious problems.

To kickstart many applications you can use a service provider that does all of the crude work for you and exposes nice API to work with. The downside is that it's not free and you become vendor locked to given service provider. In some cases using third party service to bypass some crude part of the application may be worth it - either permanently or for an early version to quickly deliver value built on top of said crude data.

We did initially looked at one of communication service provider and business part of the company even talked with them and setup a sandbox for development. However it quickly turned out their pricing won't be feasible for an early version and low scale customers. So a decision was made to hold them as a future option for bigger volume customers. Such "service for developers" would allow us to quickly handle multiple channels but at a price. We had to develop our own support for Messenger and Instagram to be able to launch and scale first.

Choosing the technology

The goal is to have nearly live communication between the app and communication channels like WhatsApp or Messenger. The app must receive and display incoming messages as they arrive - with as little delay as possible. So how do you get incoming messages? You could pull on a REST API endpoint but that's not very efficient to constantly hit your backend with HTTP requests. Websockets would be the go-to choice although browser support and handling still differs a bit so either we would have to handle it or find a library that handles that for us. Such library does exist and is called socket.io.

With socket.io you can create a client in SPA JS web page that connects to a socket.io server written in for example Python. Business agent opens the dashboard, it connects him to the server and awaits on events - like "new message", "new conversation" but also allows him to send events to the server like send reply. You can see the basic flow on socket.io original example.

Before starting ChatPilot we used Ember.js for our SPA dashboard but for this project we looked at other options, like Angular and Vue. Even though Ember worked well for us we wanted to check if more popular platforms offer some advantages. With Ember we could start developing an application quicker and we would be familiar with all the solutions, libraries for it. With a new platform we would be going in fresh without experience or knowledge of all the little details.

So Ember or Vue? We feared a bit that Ember lower popularity would with time limit the availability of handy third party solutions and that switching to a more popular framework would allows us to hire developers in the future that know it already. We did some "hello world" type of simple projects in Vue, we checked how socket.io integration looks like and how/if other developers did similar projects to ours. It looked well so we decided to pick vue over ember for this project.

Flask/JSONApi microservices
Vue.js dashboard
Tailwind CSS and Tailwind UI
Socket.io server
Google Cloud
Kubernetes and Docker
Agile development

Prototypes

Project started with a prototype - to showcase the technology and see what are the risks for the project. We started with a very basic Vue project that had to offer a very simple set of features, added progressively:

Connect to SIO server and display fixed messages the server would send back on connect event
Send message, make server send a fixed response.
Use socket.io room feature allowing two separate connections to "talk" to each other (see each other messages)

First prototype with fixed messages

Such incremental work on the prototype allowed us to validate Socket.io as well as vue.js as both were new technologies for us. We did use Vue 3 which is rather new and not every third party vue library supports it. We did encounter this problem few times, having to look for vue3 forks (like vue-3-socket.io) and alike. In the end we dropped vue-3-socket.io in favor of direct socket.io client usage for extra flexibility (like authentication and so on).

Initial version of the layout, with the help of Tailwind CSS and Tailwind UI

From prototype to production

Using a framework for the first time and making a basic prototype on start won't result with everything being codded and designed optimally or even "good" from the start. We used Ember but we never used Vue so the change was a learning curve. We didn't used all of popular Vue extensions from start as well. Early prototype didn't need routing so no vue-router, we didn't even used any storage layer like Vuex - but moved to use Pinia later on.

We started with a simple prototype by working on the list of basic features. As things moved from prototype to intended production feature we had to refactor the codebase quite a lot - reimplementing a placeholder solution with a real one. Like at start messages were stored by the server in a Python list which then was replaced with a database. From the frontend/prototype side of things nothing changed (aside of not loosing messages after server restart ;)) yet the project moved forward. Same change happened in Vue - we used JS arrays but move to Pinia for storage handling and moving such logic from vue components.

Current version of the dashboard

Doing a very dumb early prototype allowed us to start validating the technology but also ideas for the application not to mention early UX/UI work. As this wasn't production level yet we were also allowed to do breaking changes, create and delete data as needed. We started with prototyping basic features using often "dumb" placeholder solutions and when that worked for a showcase we proceed with refactoring it into permanent solutions. This created cycles where business saw some new features or a working prototype, then for some time noting changed from the outside when we refactored/reimplemented things to then to be able to start implementing features that would not be possible with the dummy placeholder solutions. This has been communicated to the business - so that so "lack of changes" doesn't mean nothing is happening.

Even when having "permanent" solutions those were solutions designed for the early prototype so a lot of things changed with time. For example as we started with simple message model with time we had to start differentiating between customer and agent messages to then handle other special message types. You could start adding optional columns to a table but that's not very clean. So with the help of SQLAlchemy models we setup polymorphic relationships between base message model and all message-type models that inherit from it. That way you can do easy timeline/date range select of all messages as well as having unique models for each message type.

The project managed to move to production, got some early customers while the development continues. Things can't break any more (the message model migration had to be planed and had Vue dashboard compatible with both versions) but still we can move forward with the project. As we learn new software stack and as new things becomes needed or would benefit us we implement and refactor and that seems the way to go with such early prototyping/production approach.

First simple statistics

Groups management

Programmatically creating video clips and animated GIFs in Python

Piotr Maliński — Sun, 08 May 2022 18:12:34 +0000

Video editing is an art of it own but sometimes you may have to generate a short video programmatically - like a marketing or manual type of content that uses data from your application at the time of creation. This can be done in Python although it won't be that easy.

MoviePy and Gizeh can be used to create animated GIFs or video clips with text, images, animated vector graphics and alike. Let see what they can do.

Gizeh is a wrapper around Cairo library and tries to make working with it easier. This library can be used to create vector drawings. To use it you have to have Cairo installed on your system.

MoviePy is a video editing library that has a similar function set as a desktop video editor application and thus allows creating video editing flows programmatically.

Code commentary and examples on:

https://rk.edu.pl/en/programmatically-creating-video-clips-and-animated-gifs-in-python/

Generating realistic landscape images with Nvidia Canvas showcase app

Piotr Maliński — Fri, 06 May 2022 23:09:42 +0000

Nvidia Canvas is a showcase application that allows you to generate realistic landscape images with AI. Using options like grass, hill, ocean you can tell the AI to generate given feature on selected spot of the image. Lets take a look.

You can download Nvidia Canvas for free but to use it you need a PC with Nvidia RTX GPU.

The UI is quite simple although unique to this application:

You start from selecting an image from the right column. You can add your own as well. You should use landscape images, on others you will get weird abstract results. I used Windows XP wallpaper.

When you select an image you likely won’t see it on the preview. The app has two panels — drawing box on the left and preview box on the right. Using materials from the right column you draw where given landscape feature should be on the image and the AI will try to draw it there.

To have a base start you should try drawing where original landscape features are on the base image — so I’ve market the area where the Windows XP wallpaper had grass and where it had sky. This generated something similar to the original image. Next I started experimenting with adding water, ocean, hills and so on. Note that you need somewhat realistic angles and size of features like hills or mountains for the AI to generate something that looks like them and not weird artifacts.

More examples on:

https://rk.edu.pl/en/generating-realistic-landscape-images-with-nvidia-canvas-showcase-app/

Scripting LCD on Wio Terminal with ArduPy

Piotr Maliński — Thu, 05 May 2022 13:09:40 +0000

One of Wio Terminal key features is the LCD display. It can be used to provide a graphical interface for your device. In this tutorial I'll go over using the LCD in MicroPython with ArduPy.

To get ArduPy running on Wio Terminal we have to flash it with latest ArduPy build. It's described on wiki and in short it's like so:

Enter bootloader mode by pressing the power button down two times quickly - this will mount Wio Terminal like if it was a flash drive
Copy the UF2 ArduPy file - the device will reconnect shortly in ArduPy mode

Optionally you can install aip on your local machine. This tool can help managing Wio Terminal, especially for ArduPy.

Now for the LCD there is a reference of the module: https://wiki.seeedstudio.com/ArduPy-LCD/ - which we can use to draw text, geometric shapes or even simple graphics (assuming we handle low memory limitations of the microcontroller):

Code and more details on:

https://rk.edu.pl/en/scripting-lcd-on-wio-terminal-with-ardupy/

Overview of PYNQ project offering FPGA capabilities to Python and data engineers.

Piotr Maliński — Thu, 05 May 2022 12:54:53 +0000

FPGA is an chip that can be configured via a hardware programming language to make nearly any digital circuit. Hardware engineer can use them while Python developer can’t. But there is a distinct group of Xilinx FPGA based devices that can be used by software developers and data engineers, even with Python. This is possible thanks to PYNQ for Xilinx Zynq chips. In this article I'll showcase this platform and it capabilities.

Introduction

FPGA is configured by HDL - hardware description language like Verilog. When as on a microcontroller given pin is for example SPI on a FPGA it can be whatever it was programmed to be - wherever it's SPI, I2C or even USB, HDMI, Ethernet and alike. Aside of such I/O FPGA can do various type of processing and logic.

FPGA isn't something we can use as easily as Raspberry Pi, Micro:bit and similar boards. But there is an option to use Xilinx Zynq based chips that contain an FPGA.

Xilinx created an architecture called Zynq. It combines an FPGA with ARM cores and I/O into one product. The ARM part is called Processing System (PS) while the FPGA is called Programmable Logic (PL). Such concept isn't limited only to this company but Xilinx went further and created PYNQ (pink).

PYNQ is a hardware-software stack that allows using an FPGA via Python and Jupyter notebooks running on the chip itself. It doesn't replace Verilog, it doesn't allow you to create designs for the FPGA but it allows interfacing and using designs made by hardware engineers.

PYNQ from the software developer side is aimed mostly at edge data processing and analysis (video, image), data processing function acceleration and machine learning. Depending on design it can compete with edge NPU chips or CPU/GPU platforms like Nvidia Jetson (in terms of power efficiency and or flexibility. One of example usage cases are drones and autonomous robots - processing images and video streams to detect obstacles, collision detection, controlling drone flight and more... but there are simpler usage cases too ;)

PYNQ

PYNQ is an open source project made by Xilinx for their Zynq based boards. As the project website states you can use it to create high performance applications with:

parallel hardware execution
high frame-rate video processing
hardware accelerated algorithms
real-time signal processing
high bandwidth IO
low latency control

PYNQ big picture is quite simple. A Linux distribution is running on the ARM cores, it has Jupyter web server and configuration for the FPGA so that when the chip boots the FPGA is ready to work.

PYNQ overlay is a hardware design of the FPGA - it implements the logic on the FPGA - with Verilog and Vivado designer application. Hardware engineers can make various overlays and software engineers can use them in their applications. In-between we have Linux Kernel drivers and system libraries connect them together.

For PYNQ boards and some basic PYNQ-Z2 board operations visit my tutorial at:

https://rk.edu.pl/en/overview-pynq-project-offering-fpga-capabilities-python-and-data-engineers/

Quick start into GUI applications with PyQt5 and PySide2

Piotr Maliński — Thu, 05 May 2022 12:43:37 +0000

Qt is a powerful framework for developing cross platform GUI applications. KDE Linux desktop environment uses it as well as many Open Source and commercial applications.

In this article I'll showcase how to use Qt in Python to make GUI applications (the very basics of it).

PyQt and PySide

For Python developers since long time PyQt binding were available. As of now there is also Qt for Python project that provides PySide2 bindings.

There is a very good comparison on machinekoder.com. In short PyQt bindings existed for much longer and are distributed on GPL or Commercial license. PySide2 is distributed on LGPL or Qt commercial license - which may make a difference in some legal cases. In terms of making simple apps there shouldn't be much differences aside of imports. For more complex ones there may be some.

You may find a lot of tutorials for PyQt due to it age. PyQt4 is the former main version while now we use PyQt5. Some code did change (like signals and slots syntax) but overall the workflow should be the same. If you pick up some older PyQt4 books keep that in mind (and the book may use Python 2 instead of 3).

There are also books and resources for PyQt5, like Create Simple GUI Applications with Python and PyQt5 while also having some online resources on learnpyqt.com.

PyQt and PySide can be installed as any other Python packages via PyPi. You should also install Qt development tools. Qt Designer is the desired application for now. It allows creating user interface of the app in a visual manner. On Linux it can be in a separate package while for macOS or Windows it will likely be with the official Qt development package (macOS package managers like homebrew may also have split packages).

A note on desktop applications

Compared to web development, common in the Python community, the GUI application development is less popular and it can get quite complex much quicker. Before developing an app check if a web variant isn't a better solution (no need to build, distribute, install etc.).

When creating an app with a GUI be sure to make the GUI follow look and feel of other similar apps - a.k.a. you should try not to reinvent styles or some widget placements unless you are making an app with non-standard UI. For each operating system or desktop platform there are Human interface guidelines that describe how applications should look and behave to meet user experience standards.

Qt does follow those guidelines and also tries to make widgets follow current style of the system - if you run your app on Windows or on macOS it will look like other apps of that particular OS as well as some widgets may look different or be placed in different spots of the window.

Kivy uses it own style and will not follow OS style and usability guidelines – everything is in your hands. Tkinter will follow the style to some extent although the widgets won’t look "as good" as in native apps.

Python standard library versus Qt

In Qt class reference you will find functionalities that are also covered by Python standard library or popular third party packages. For example you can find QFile, QDir classes that represent files and directories and with other related classes implement a lot of filesystem operations. You can even run a SQL query via QSqlQuery and more.

With such overlap one can wonder which solution should be used - Python or PyQt/PySide bindings to C++ classes of Qt? Overall I would advise using Qt path as much as possible. A lot of classes and widgets is inter-compatible - meaning one class can accept an instance of another, like accepting a QFile object or use QSqlQuery to populate a grid and more.

If the Qt usage seems overly complicated and you know you aren't loosing any of that "glue" then sure, you can also use a Python way of doing things.

Also note that a lot of Qt classes takes operating system differences into account which may not be handled as well by non-Qt Python package. Like if you want to print a file Qt has you covered while when trying pure Python solutions you would likely need more time and testing to see if it's cross-platform (like you even have QPrintDialog in Qt...).

QtDesigner is used to create the user interface of your app. It can cooperate with other Qt tools to create translation files for i18n-enabled apps or manage assets like images and other files used by the app.

Now it's time for some code...

For a full code commentary on making a simple app in PyQt and PySide you can check my tutorial at:

https://rk.edu.pl/en/quick-start-gui-applications-pyqt5-and-pyside2/

Designing a sentiment analysis application for TripAdvisor reviews in Python

Piotr Maliński — Wed, 04 May 2022 20:17:04 +0000

Sentiment analysis allows us to quantify subjectivity and polarity of text - of a review, comment and alike.

Subjectivity scores a phrase between fact and opinion while polarity scores negative to positive context.

In this article I'll show you how to use it to score TripAdvisor reviews highlighting real business value it can bring.

To do such analysis on English (and French, German with plugins) text we can use textblob package. It's as simple as:

from textblob import TextBlob

TextBlob("This is very good!").sentiment
TextBlob("This is very bad!").sentiment
TextBlob("This isn't that bad!").sentiment

TextBlob "sentiment" property returns a tuple containing polarity and subjectivity. Polarity is a number from [-1.0, 1.0] range where 1 is positive, -1 is negative. subjectivity is a number from [0.0, 1.0] where 0 is fact and 1 is opinion.

>>> TextBlob("This is very good!").sentiment Sentiment(polarity=1.0, subjectivity=0.7800000000000001)

Polarity 1 so it was rated as very positive and with high subjectivity it was treated as a more of an opinion than a fact.

>>> TextBlob("This is very bad!").sentiment Sentiment(polarity=-1.0, subjectivity=0.8666666666666667)

In this case we got -1 so very bad indeed.

>>> TextBlob("This isn't that bad!").sentiment Sentiment(polarity=-0.8749999999999998, subjectivity=0.6666666666666666)

With no context TextBlob still rated it negatively. What actually "isn't that bad" means? Better than expected? Worse than expected? Hard to tell and to get a more useful quantification we need sentences with enough context.

Tripadvisor reviews

A review is a good piece of text to try out TextBlob on. Beforehand I've fetched a set of reviews for few venues on TripAdvisor using third party scrappers. This gave me a list of reviews text I could feed into the library and see what results I get. To visualize polarity I assigned red/green colors to -1/1 and intensity for in-between values:

The first marked review is negative and the user left a 1-star review. The last one is also 1-star but the polarity is big higher. Even though a customer wasn't happy he used less negative language in his review. What's interesting is the middle market review. It's a 5-star review yet it has a low polarity - customer was happy, but found things about which he wasn't - such things could be a value feedback for the venue while it could be lost among positive reviews (with 4,5 average you look closely on the negative ones while positive review may get lost among other positive ones).

The code is on Github:

https://github.com/riklaunim/sentiment-analysis-example

Full article with code commentary:

https://rk.edu.pl/en/designing-sentiment-analysis-application-tripadvisor-reviews-python/

Streaming live video feed on a website with a MJPEG server

Piotr Maliński — Wed, 04 May 2022 19:50:56 +0000

Generic IP cameras often offer a simple MJPEG streaming that allows embedding a live view from the camera on a website. However when that's not available or you want to stream something different (desktop view, machine vision camera or some pre-recorded video clip) then you need your own MJPEG server.

There is a lot of example on the web, and by now MJPEG is not the best choice in terms of bandwidth/quality but it's simple...

So let's take a look at a simple server from Github: https://github.com/bootrino/maryjane

This simple server will stream contents of a JPEG file. To make it a video we have to use some other app to constantly update said file. This can be done with ffmpeg to stream a video clip or your desktop. Or some other source, like a machine vision camera and it SDK allowing capturing and saving frames.

Examples and more info on my blog:

https://rk.edu.pl/en/streaming-live-video-feed-on-a-website-with-a-mjpeg-server/