DEV Community: Ziga Brencic

🐍 Python: Standard library

Ziga Brencic — Mon, 11 Jan 2021 19:31:28 +0000

🐍 Python: Standard library

Python, like all languages ships with an extended amount of modules/packages you can use. I'm always surprised how much is built into the library.

Why is it so important to know what's available to you? Well for starters if you know what's at your disposal, you can reuse a lot of tools others wrote. Plus if you stick with the implementations form the standard library your code will be more robust, less external library dependencies, which is always a good practice in software development.

Here's the thing python standard is one of the most stable parts of the ecosystem out there. Core language developers are really, really careful with the changes they make. Changes are advertised in advance, tested, ... Which can't be said for all other libraries out there.

Now to be clear. Use a library that's not part of the standard if it fits your need, but think twice, because every crap you bring into your codebase will make your life harder in the future.

Here you can find the list of all python 3 modules. In the next few paragraphs, we'll cover a few that are not too specialised.

How to use a module?

For example, module collections.

Module full of container like data types like: OrderedDict and deque (list-like container with fast appends and pops on either end).

To import part of a module:

from collections import OrderedDict

d = OrderedDict()

Or import the entire module:

import collections

d = collections.OrderedDict()

IO

One large aspect that has great support in python is IO writing to the disk and reading from it.

json: Module for reading .JSON files. More about json in the upcoming post.
csv: Write and read tabular data to and from delimited files.
gzip: Interfaces for gzip compression and decompression using file objects.
hashlib: Secure hash and message digest algorithms.
pathlib: Object-oriented filesystem paths. Supper handy to check if file paths exist, ...
sqlite3: A DB-API implementation using SQLite 3.x.
aifc: Read and write audio files in AIFF or AIFC format.
tarfile: Read and write tar-format archive files.
io: Core tools for working with streams of data (binary file readers ...).
zipfile: Read and write ZIP-format archive files.
xml: Package containing XML processing modules
copy: Shallow and deep copy operations. To perform deep copies of objects.
types: Names of built-in types. Use them to "fix" data structures.

There are also some caching options:

from functools import lru_cache

lru_cache is a Last Recently Used cache, so there is no expiration time for the items in it.

There's also shelve: python object persistence package, which allows you to store objects in key value like manner for the cycle of your program. For example, to create a shelve do:

with shelve.open('spam') as db:
    db['eggs'] = 'eggs'

Then you can call objects from the db anywhere in the scope of with. This was if the resource has already been loaded from the disk to the memory you can read the resource from the cache which is much faster than disk access.

Parallelization, background processes

threading: Thread-based parallelism.
multiprocessing: Process-based parallelism.
queue: The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads.
asyncio: Asynchronous I/O. Execute code forward and let the processes run until it runs

GUI

There's tkinter. An interface to Tcl/Tk for building really simple graphical user interfaces. I'm not a big fan of it.

Reliability

There are a few built-in modules in python standard that can help you make your codebase more reliable, faster, readable and modular:

warning: Issue warning messages.
logging: Flexible event logging system for applications. Especially to store error logs.
cProfile: For profiling python programs. To measure how long certain portions of the code take to execute. Don't mindlessly optimise. Write the code. Identify the slow stuff and optimise that.
pydoc: Documentation generator and online help system.
venv: Creation of virtual environments. Use this to ensure that others can run your code by creating a virtual environment, fast. No extra dependencies needed.

Time

datetime: Basic date and time types.
zoneinfo: IANA time zone support.
time: Time access and conversions.
calendar: Module for a calendar like operations.

Networking

Networking tools are really well supported in python:

mailbox: Manipulate mailboxes in various formats.
webbrowser: Easy-to-use controller for Web browsers.
smtpd: A SMTP server implementation in Python.
smtplib: SMTP protocol client
socketserver: A framework for network servers.
ssl: TLS/SSL wrapper for socket objects.
email: Package supporting the parsing, manipulating, and generating email messages.
socket: Low-level networking interface.
ftplib: FTP protocol client.
html: Helpers for manipulating HTML.
http: HTTP status codes and messages.
urllib: URL handling module.

Math and numerics

There's quite a few things you can do mathematics wise with pure python before you bring in the crap like scipy and numpy. To some extent, those libraries are great but hide too much implementation details. Which is fucking dangers. So use the basics from python whenever you can.

statistics: Mathematical statistics functions.
math: Math functions exp, sin, ...
cmath: Mathematical functions for complex numbers.
random: Generate pseudo-random numbers with various common distributions.

Command line/OS interface

Having a nice interface with the underlying OS is always nice. bash is painful to write so if control from python is on the table, sign me up.

subprocess: Subprocess management.
os: Module with os.system() option that allows us to run shell commands among other functionality.
sys: Access system-specific parameters and functions.
sysconfig: Python's configuration information.
syslog: An interface to the Unix syslog library routines.
pipes: Interface to shell pipelines.
argparse: Command-line option and argument parsing library.

Other cool modules:

difflib: Helpers for computing differences between objects. Can come in handy if you want to track changes.
importlib: The implementation of the import machinery. This way you can import python modules, packages, folders, ... programmatically and not by typing boilerplate code.
pprint: Data pretty printer.
're`: Regular expression operations.

That's it. In the upcoming post, we'll explore some of the features in-depth.

🐍 Python series:

🐍 Python: Crash course

Ziga Brencic — Sat, 09 Jan 2021 15:31:55 +0000

🐍 Python series:

Python is a scripting language that gained in popularity in recent years.

Official site here.

How language came to be?

Python was created in the late 1980s by Guido van Rossum as a Christmas project. Heh like this newsletter :P

The language was then officially released for the first time to the public in 1991.

Python is a scripting language that is interpreted, rather than compiled into the machine code, which is one of the major reasons why python is by design slower than C++.

Sure there's the option of converting python to "C" with Cython to achieve speedups. But pure C and C++ will always be faster than the snake 🐍 .

Python versions

There are two major versions of python out there at the moment:

python 2 which you should use only if you have to work with legacy code.
python 3 with the latest version 3.9.1. You can find the docs here.

Side note. macOS still ships with python2 but is finally switching from python2 so if you run:

> python2
WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within Terminal.

How to start python?

To access python3 from the command line simply run:

> python3

This will open the following command line interpreter:

> python3
Python 3.9.0 (default, Nov 21 2020, 14:01:50)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

in which you can run commands.

Hello world

To run the classical hello world type into the terminal:

>>> print("Hello, world!")

and press enter. That's it.

That's one way of using python, but I'm personally not a big fan of it. The alternative is to write scripts, which is a way to go if you indent to write more then five lines of code.

Run a script from the command line

You can write scripts and run them directly from the terminal.

Create a file called hello_world.py and write to it the following line:

print("Hello world!")

Close the file.

Now run in the command line:

> python3 hello_world.py

You should see:

> python3 hello_world.py 
Hello, world!

Before we wrap up, there's something else I want to cover. Above example has one slight issue. It works fine in our simple case, but the moment our software grows into multiple scripts such "direct" commands become an issue.

If we imported the hello_world.py file to another python program, the print statement would be executed. But we want to execute the print command only if we ran the program directly python3 hello_world.py and not when we import our script.

To solve the issue, we can use something called __main__.

`main`

To achieve the desired behaviour we correct our hello_world.py example into:

if __name__ == "__main__":
    print("Hello, world!")

Where __name__ will be equal to "__main__" only if we run our hello_world.py script directly. If we import our script to another python program __name__ will be different.

Now once we run the script python will check if global variable __name__ is set to "__main__" and then execute our code.

Why you code?

Ziga Brencic — Sun, 22 Dec 2019 19:44:50 +0000

How would be your answer the question:

Why you code?

Let me know in the comments below and read on the story.

Motives

I'm not a software developer not by training nor experience. I never worked as a professional software developer. Software is a tool for me, like math. I'm a physicist, after all, 🙂 I love doing math and writing code, but I prefer to stay in the physics lane for the majority of my time. So to no surprise, the conversations like the one below occur over and over:

Colleges: Why do you code?
Me: Because it gives me an edge.
C: Huh OK.

Start

I started coding (simple web site) in high school. As I started, I was really frustrated by coding. I would spend days banging my head against the wall. Most of the time, I had no clue what was going on. But I forged on because my studies required me to do some basic coding. I kind of despised coding. Physics and math was more fun.

But it wasn't until my 2nd year at uni before I took a deep dive into coding. I realised that every physicist that was an excellent coder was perceived as an expert in the community. Those mini experts new something way better they anyone else. If you needed help with coding, you went to them. I figured that the coding skills could be my secret weapon.

Competition

This will depend on your field of expertise. Still, no matter the profession, there are several things that the majority of people find hard and confusing. The harder or more boring the concepts are fewer people know then. As a result, you face less competition. There's a good analogy:

A lot of people go to the mountains, but a few well-trained enthusiasts climb the highest peaks of the world. Higher we go thinner the air. There's far less competition at an altitude of 8km.

Use this to your advantage. Figure out what others despise. Most probably they don't like certain areas because they don't understand them. Dig into those, and your skills will be sought by others for the rest of your career.

Old technology

Physicists, write a lot of (shitty) code. OK, a few can code really really well. I would claim that I'm somewhere in between the two poles. As long as the code works (most of the time), we don't bother staying up to date with all new fancy tools and concepts. Yes, some are still convinced that FORTRAN is easy. Some still send emails via vim.

So to no surprise, the first language I really learned was C in 2013. Even the website we used for the course looked like it was written in the eighties. I found my sanity and at least switched from Windows to Unix in 2013. So I started my path by going down the C rabbit hole. Then came across something called C++. Then the dependency hell, build systems... you know how the rest of the story goes.

Frustration is the way

Every time I tried something in software, I asked myself. Why is this so hard? It's not rocket science nor theoretical physics. I should be simple right? Yet on the contrary, when I decided to build my first proper web site, I read online that:

I needed to sign up for AWS account create an EC2 server that runs on ubuntu. SSH to the server through bash shell. Then install Nginx, SSL and uwsgi. Then my application should use on the back-end the python Django engine with PostgreSQL database. On front-end I should use react/vue/angular with some HTML, JavaScript, CSS. The application should be running in docker containers so it can scale based on the web site traffic.

I knew that python, CSS, HTML, JS were languages heard of git and bash. The rest was foreign to me. In a few years, I mastered enough of the above concepts that I'm capable of running a simple personal web site. Though I still have nightmares from time to time about the first experience with Django.

After coding for a while, I came to realise that:

Experienced developers fail a few times faster than junior ones.

After coding for years, I still get stuck and frustrated every day. But these days I'm somehow cable of getting over the frustration in a matter of minutes instead of days 🙂 I guess that comes with experience so hang in there and embrace failure. 😉

Why?

When I was picking up the software development skills, it took me years before I started seeing progress. The hard truth is:

You learn how to write good software through years of practice.

There are no shortcuts. If you want to get good, you'll need to practice the craft. Nothing new here. But make sure you know the answer to the question:

Why you code?

For me, software development is all about being able to:

Write fast code as fast as possible.

As long as the code get's the job done in the reasonable amount of time I'm happy with it. There are cases where I have to take care of documentation, tests, share code with others then the why slightly changes.

For you the answer to the why will probably be something completely different. Nothing wrong with that. The why should dictate how you improve your craft.

As a result of my needs, these days, my proficiency lies in python and C++. My work is mostly oriented around big data processing and simulations on clusters. Still, I come across other languages like CSS, HTML, JS, latex, (bash) every day. A server crosses my desk from time to time. For me, quick and dirty solutions usually do the job. Though I sure kick my self later for writing the bad code.

Parameters object in C++

Ziga Brencic — Thu, 12 Dec 2019 19:56:57 +0000

How to consistently use a set of parameters throughout the larger codebase.

I've done my fair share of data processing. And one thing that most big data software implementations I worked with didn't tackle correctly was parameter handling.

The process usually goes like this. You write a class and hard code:

unsigned int num_of_users = 1401;

Before you know it, the variable num_of_users is used in ten places all over the code. Then you update num_of_users to 1402. But forget to update the variable at some of the places in the code. The code works but gives inconsistent results. After hours of searching for the needle in the haystack, you figure out what you forgot to update. OK, problem solved. Two days later, you're pulling out your hair because another variable is causing trouble.

Parameter object

In software I try to stick to the paradigma:

If you copy-paste your code/parameters, you are doing it wrong.

To avoid hardcoding and parameter pasting in a large codebase, I create an object that holds the values of all my parameters. Store, all parameters in one object, then access them whenever they are needed:

class Parameters {
pubic:
    unsigned int num_of_users;

    // rest of parameters

    Parameters() {
        this->num_of_users = 1501;
    }
};

Then with initialization:

Parameters var;

we get access to all parameters via:

var.num_of_users;

Passing Parameters object around

Let's say we want to have one instance of our parameter object and then pass the pointer to that object around. Smart pointers come in handy here:

auto var = std::make_shared<Parameters>();

We can then pass the var smart pointer into any part of the code/module that needs access to those parameters. For example, we need to pass the parameter to two processing modules:

ProcessingModuleA processing_module_a(var);
ProcessingModuleB processing_module_b(var);

Where ProcessingModuleA is defined as:

class ProcessingModuleA{
public:

    // rest of the module code

    ProcessingModuleA(
        const std::shared_ptr<Parameters> &var){

        var.num_of_users; // accesible  

        // rest of the module code
    }
};

Loading parameters

Another handy thing is if we can load parameters from the outside of our program. A JSON file, for example. So we can compile our C++ code once and then control the processing output via a JSON file. Advantages:

The considerable development process speed up. Since we don't have to recompile the code at each parameter change.
We can store the parameters we used to produce specific data set and make the data set reproducible. As long as we know what code version was used ;)

Now when we want to use the parameters.JSON file as the input file the implementation becames a bit more technical. Example class implementation below:

#include <iostream>
#include <string>
#include "json.hpp"

class Parameters {
pubic:
    unsigned int num_of_users;

    // rest of parameters

    Parameters(
        const std::string &parameters_location) {

        json parameters = read_json_file(parameters_location, "parameters.JSON");

        try {
            this->num_of_users = parameters.at("num_of_users").get<unsinged int>();
        }
        catch(std::range_error &e) {
            std::cout << e.what() << std::endl;
            throw;
        }
        catch (...) {
            std::cout << "Unknown exception occurred" << std::endl;
            throw;
        }
    }
};

You should use try and catch block to ensure that you catch any errors that occur while reading the JSON file. On what read_json_file function does check the article JSON in C++: Read & Write

Benefits for the development process

Once the code is wired correctly, the processing becomes a breeze. Sure it takes a few minutes to set up. But once up and running, the loading of program parameters from the external JSON file gives us a tremendous amount of speed up in the data analysis. As a result, we can get insights faster. Insights are what the whole big data and analytics are about anyway.

Is Web Browser local storage available?

Ziga Brencic — Tue, 10 Dec 2019 15:20:17 +0000

Modern web browsers allow us to store data in the client's browser: not larger chunks but smaller amounts when it's convenient.

Interface

To store the data to the localStorage from JS simply do:

localStorage.setItem('myCat', 'Tom');

or any other object (list, int, dict). Then to retrieve it:

let cat = localStorage.getItem('myCat');

and to remove it:

localStorage.removeItem('myCat');

or to clear all items in the local storage that belong to the current domain, we are on:

localStorage.clear();

There's another storage object sessionStorage. Same interface as local storage though session storage get's purged once the user leaves the site. Friendly note: I would avoid storing any sensitive data (personal or security) into the local storage.

Local storage status

Here's the thing. The local storage might not be available to us, or it might be full. So before we use the local storage in our web site code, we should check that we can.

Is local storage quota exceeded

We get depending on the browser certain amount of space per web page domain:

Mobile:
- Chrome, Firefox: 10MB
- Safari, iOS WebView: 5MB
- Android Browser: 2MB
Desktop:
- Chrome, Firefox, IE: 10MB
- Safari: 5MB

If we fill this space up with our web page, we won't be able to add any new elements to localStorage.

Is local storage available

Users can disable the use of local storage in the web browser. Most won't, but there are enough geeks out there to cause us trouble. So we need to be able to check if the use of local storage is allowed.

Side note: In Safari incognito mode, the localStorage is blocked and will throw a quota exceeded error. So we might want to check if the browser is in incognito mode to be on the safe side.

Get Local storage status

All right, let's write a function that will allow us to check if local storage is full and accessible.

The function below tries to set the test object into the local storage and then remove it. If an error occurs, we catch it with the try catch block and then analyze what happened.

Function get_local_storage_status() will give us the status of localStorage :

full
unavailable
available

Full code:

function get_local_storage_status() {
    let test = "test";
    try {
        // try setting an item
        localStorage.setItem("test", test);
        localStorage.removeItem("test");
    }
    catch(e)
    {   
        // browser specific checks if local storage was exceeded
        if (e.name === "QUATA_EXCEEDED_ERR" // Chrome
            || e.name === "NS_ERROR_DOM_QUATA_REACHED" //Firefox/Safari
        ) {
            // local storage is full
            return "full";
        } else {
            try{
                if(localStorage.remainingSpace === 0) {// IE
                    // local storage is full
                    return "full";
                }
            }catch (e) {
                // localStorage.remainingSpace doesn't exist
            }

            // local storage might not be available
            return "unavailable";
        }
    }   
    return "available";
}

We can then run the function with:

get_local_storage_status();

Or display the status in the console:

console.log(get_local_storage_status())

A convenient test to see if our script works is to disable local storage, for example in Firefox by:

Typing: about:config in your address bar and hit enter. You'll see browser settings.
Search for dom.storage.enabled and change status to false. Double mouse click on the setting should do the change.

Add your script to an example website and check in the browser console for the status. If you managed to set up the Firefox settings correctly the console.log(get_local_storage_status()) should give you "unavailable" status.

When is local storage convenient

All right, we are equipped with the local storage. Now, where would we want to use it?

Let's say that the user internet connection drops down, and the user wants to leave the web site. But some data will be lost if the user leaves the web site in the offline mode. Then local storage could come to our aid. We store data in the local storage until user reconnects to the internet with our website again. Again don't store any sensitive data in the local storage.

Another use case of local storage would be to optimize the page load time by storing some page parts into the local storage. On the next load, just part of the page would have to be loaded.

There's one more edge case we didn't cover here. The user's web browser doesn't support HTML5 and localStorage at all. In that case well ...

Don't do things globally in C++

Ziga Brencic — Fri, 06 Dec 2019 10:01:23 +0000

Don't expect other developers to have a computer size brain.

To often I come across a codebase where objects magically apeare in the code. Then I would ask the developer that wrote the code where the object is coming from and they would go:

Developer: Oh yeah that's a special object we defined globaly.
Me: Sigh. 🤦‍♂️

Sure when you writing code you are lazy and global objects my seem like a viable solution, but on the long run they never are. If you use global objects your code most probably won't be thread safe so forget about multithreading. Oh and let's not forget that the code will be painful to read for any future developer.

Code organisation

Organise your program from top to bottom. Code that gets called last should be at the end of the file. There are few advantages of this. Code becomes readable like an article. You don't have to specify class/function definition before their implementation. Or in other words there's no need for forward declarations.

As rule of thumb I try to follow these guidelines:

code is split into self contained blocks,
every code block can have multiple inputs, but only a signle output,
don't use global object's ever,
if global variable would be convinient use a class,
don't hard code anything.

Hard coding is pain in the ass for mentainence and code reading. If data is needed in multiple places in the code pass it around. One more thought. Don't over use classes. There are a lot of cases where a function with multiple inputs and a single output will do just fine.

Global objects

Don't do it. Maybe you think you are smart enough to keep all the global stuff in your head but once your codebase grows to few thousand lines of code (it does pretty fast ;) ) you'll loose track. Sure there maybe a one percent of cases where global stuff makes sense but unless you're doing the computing on scale of CERN or Google you most probably won't come across them.

Global objects never help with code readability. They just increase number of WTF per minute when another developer reads the code.

Localy/Class global

It's true sometimes its handy to have objects/variables available in multiple functions without the need to pass them around every time. In that case you should write a C++ class. The use private to make sure that variables needed in the class stays "global" only in the class. Other parts of the code shouldn't have access to those variables.

For example let's create a C++ class named MyObject:

class MyObject{
private:
    unsigned int hidden_counter;
    double x;

public:
    void print_next(){ // public function
        ++hidden_counter;
        std::cout << x*hidden_counter << std::endl;
    }

    MyObject(double x){ // object constructor
        this->hidden_counter = 0;
        this->x = x;
    }
}

Now to work with the object you can initialize it with:

MyObject  var(3.5);

To then use the public functions do:

var.print_next(); // prints to screen: 1*x = 3.5;
var.print_next(); // prints to screen: 2*x = 7.0;

Advantage of this approach is that we created an environment where hidden_counter is global in scope of the class but isolated from the rest of the program that shouldn't modify it.

If we now write var.hidden_counter = 3; the C++ compiler will complain that we are trying to modify a private variable of class MyObject.

Defensive programing

Another concept worth mentioning. Idea of defensive programming is that our code also specifies how it shouldn't be used. For example by making hidden_counter private inside class MyObject we made sure that hidden_counter can't be modified outside of the MyObject methods. If we would have made hidden_counter public we woldn't prevent the code user from missing the code of MyObject.

To wrapp up. All the time invested into thinking how code should be structured/implemented is worth way more then you imagine. More readable code will decrease the likelihood of you and other coders pulling out your hair. Though don't get stuck in the code planing ;)

Analytics with vanilla JS: page view duration

Ziga Brencic — Wed, 04 Dec 2019 19:22:45 +0000

Third post in series: Analytics with vanilla JS

Page view duration is essential in determining whether our users read the content of our article or not.

To determine the time of user page visit, we need to detect two events:

page view start time: t_page_opened
page view end time: t_page_closed

Request page, close page

We first cover the case of page view duration, which is the easiest to measure.

Let's say the user clicks a link or types the address of our web page to the web browser. The user will land on our web page. The page files will be sent over from the server that hosts the web site.

In this case, when the page is loaded, we can detect onload Java Script event and determine that as the start of page visit with:

window.onload = function () {
    t_page_opened = new Date();
};

Now let's say that the user leaves the web page by closing the browser tab or browser window. We can detect that via beforeunload event by adding the event listener:

window.addEventListener("beforeunload", leftWebSite);

leftWebSite function then get's the time stamp when user left the page:

function leftWebSite() {
    const t_page_closed = new Date();

    const data = JSON.stringify({
        "page_opened": t_page_opened,
        "page_closed": t_page_closed
    });
    post_data(data, "define_URL");
}

and sends the t_page_opened and t_page_closed to the a prediefined URL with post_data function:

function post_data(data, url) {
    let xhr = new XMLHttpRequest();
    xhr.open("POST", url, true);
    xhr.setRequestHeader("Content-Type", "application/json");
    xhr.onreadystatechange = function () {
        if (xhr.readyState === 4 && xhr.status === 200) {
            console.log(xhr.responseText);
        }
    };
    xhr.send(data);
}

Job done, right? Far from it.

If we want to have a proper estimate of the read times, we need to be really careful about potential corner cases for page opening and page closing.

Wait? Page view start and page view end should be determinable in a few lines of code. Not quite.

User leaves the web site

What if the user left the page via a link, history forward or history back button, lost internet connection, refreshed the web page. In those cases, beforeunload won't get consistently triggered (from browser to browser).

Let's list the cases we need to cover. When the user leaves the page via:

Browser tab/window closing: detected via beforeunload. Solved.
internal or external link: detectable via onclick event
internet connection lost:
- connection reestablished during page view: no issue
- user revisits a page in the future: fetch view data from user browser localStorage if the user didn't clean it
- the user never comes back to our site: data lost
page refresh: can be detected with the help of localStorage
page left via history back, forward button: using pagehide event, since page JS won't be re-loaded and beforeunload event won't be triggered.

Keep in mind that the solutions that use localStorage won't work if the user:

disabled the use of local storage,
uses incognito browser mode.

User opens the web site

To detect if the user opened the web site is slightly easier. There are just three cases to handle:

We already handled the first case with onload. To handle the page left via history back, the forward button, we again use pagehide event.

Tackling edge cases

In the upcoming articles we'll cover the code needed for:

detecting: page refresh, history button clicks, internal-external links
detection of incognito mode
detection of internet connection loss

But even after we handle all of those cases, we won't ensure the correct estimate of reading time. What if the user opens the web page, leaves it open and goes to the toilet. Or scrolls through the article really fast. Well, we could monitor page scrolls. This way, we could eliminate a few more edge cases that could corrupt our read time estimates and analysis. Stay tuned.

JSON in C++

Ziga Brencic — Mon, 02 Dec 2019 07:53:49 +0000

Third article in series: Modern C++

Modern C++ series:

One thing in python that's super useful are dictionaries and JSON. They allow flexible data structures. What about JSON structures? They are not possible in a statically typed language like C++. We'll modern C++ can handle JSON believe it or not :)
C++ itself doesn't provide JSON, but there are a few options that bring JSON type data structure to C++.

I personally use C++ JSON library nlohman. There are other libraries, though, so take your pick. I like nlohman JSON lib because it's a single header file library (drop header file into the project). Or it can also be used as a dependency. It depends if you want to ship the JSON lib with your software or allow the user to add it as a dependency.

What can the JSON library do? Well, quite a lot, but the syntax and usage can be quite confusing, so I'll provide a few examples to clear things out.

Note: I'm assuming ' using nlohmann::json; to write less syntax.

To create a JSON object, we simply do:



json data;

and then add elements to it:



// vector
data["M_final"] = std::vector<double>(N);

// a double
data["beta"] = 10.0;

or any other structure from STL library. Some STL structures are not supported though most are. More on adding custom structures to JSON later.

There are a lot more options on how to construct the json object. For example, we can actually write the JSON format in the program, but I prefer to load JSON data structures from an external file. json object creation inside the codebase:



json data = R"(
    {
        "happy": true,
        "pi": 3.141
    }
)"_json;

For more construction options, look into the docs. The library has a ton of other functionality STL like access, for example, but really read the docs if you are interested in more.

Read & write JSON files

How do we load JSON file into the JSON data object from C++? We'll quite easyly. If we specify:



std::string path = "path to json file";
std::string file_name = "json_file_name";

we then load the file with few lines of code:



json data;
std::ifstream file(path + "/" + file_name + ".json");
file >> data;

Side note. You need to include #include <iostream> and #include <fstream> libraries. But they are in STL so no need for messy installs.

Then to store to the JSON file:



std::ofstream file(path + "/" + file_name + ".json");
file << std::setw(4) << data << std::endl;

Custom JSON serializers

So let's say we want to store a custom data structure to a JSON file. For that, we need to define a serializer. JSON serializer tells our library how it should convert a C++ object into a format that can be written into a JSON file. Or in other words, it tells the software how to load the JSON file into appropriate C++ data structure.

For example, let's say we want to be able to store std::complex<double> to JSON. Then we can expand the std namespace with two methods to_json and from_json:



namespace std {

    /// std::complex<double>
    void to_json(json &j, const std::complex<double> &complex_number) {
        j = json{{"re", complex_number.real()},
                 {"im", complex_number.imag()}};
    }

    void from_json(const json &j, std::complex<double> &complex_number) {
        complex_number.real(j.at("re").get<double>());
        complex_number.imag(j.at("im").get<double>());
    }
}

Or to add a structure of type person:



struct person_t {
    std::string name;
    std::string address;
    int age;
};

void to_json(json &j, const person_t &p) {
    j = json{{"name",  p.name},
             {"address", p.address},
             {"age",   p.age}};
}

void from_json(const json &j, person_t &p) {
    j.at("name").get_to(p.name);
    j.at("address").get_to(p.address);
    j.at("age").get_to(p.age);
}

By providing to_json and from_json we allow the program to automatically map our data structure to the JSON format. Now we can store a person structure to the JSON file via:



person_t p;
// define person: name, address, age
json data;
data["person"] = p;

If we stored json data now into a JSON file, the content would be:



"person": {
    "name": "John",
    "address": "web",
    "age": "31"
}

That's it. OK when reading back from JSON file we actually need to provide the data structure type before we can get the C++ function. For example to retrieve our person object:



json data; // contains person object but in string format
auto p = data.at("person").get<person_t>();

I know that's not as pretty as in python, but if you store the type of the data structures when you create the JSON file, you can use a little trick. You can write an extra wrapper around json library that allows you to make calls of type:



enhanced_json data;
person_p p = data.at("person");

In the above case, you handle data loading based on the data type in the JSON file inside the enhanced_json object using the at function. To get the enhanced_json object, you need a few if statements and auto.

Sure all the above software will add overhead and make your code slower. So if you need fast data storage for large amounts of data, this automatic serialization is probably not a good idea. I would argue that JSON is not a good idea if you need to read and store a lot of data. But if you need C++ to do a lot of data crunching and need a bit of storage. Then JSON interface can tremendously speed up your workflow.

JSON doesn't make C++ totally python

In the next article I'll dive into the use of C++ JSON data structures to speed up the analysis workflow. Until then, if you know any more tricks that make C++ more pythonic, let me know.

Analytics with vanilla JS: page views

Ziga Brencic — Fri, 29 Nov 2019 18:46:01 +0000

How to get basic page view statistics?

Second article in the series Analytics with Vanilla JS. Motivation here.

Today we'll look into the impelentation of vanila JS analytics tool that analyses page views.

For the sake of example we need some simple HTML code for our tracker (file example_page.html). You can add anything you want to the HTML file:



<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <script src="js/page_view_tracker.js"></script>
</head>
<body>
    <a href="https://www.google.com" class="external">Leave page by going to Google</a>
</body>
</html>

The rest of the code will be in page_view_tracker.js. First, let's define the function that will allow us to POST all the gathered data as a string to a specific URL:



function post_data(data, url) {
    let xhr = new XMLHttpRequest();
    xhr.open("POST", url, true);
    xhr.setRequestHeader("Content-Type", "application/json");
    xhr.onreadystatechange = function () {
        if (xhr.readyState === 4 && xhr.status === 200) {
            console.log(xhr.responseText);
        }
    };
    xhr.send(data);
}

data in the string is in JSON format. The server you'll be sending data to can be whatever you prefer: node.js, Django, flask, ... There's even an option to post into Google Docs spreadsheets if you want to avoid the backend.

Data is posted with the following command:



post_data(JSON.stringify(data), "http://0.0.0.0:5000/analytics");

where we defined data object as:



const data = {
    "current_page_name": current_page_name
};

Now let's add the rest of the data.

Tracking

Number of views per page: this one is easy. Every time a user visits our website, the post_data function will be triggered, so we need to add current_page_name to our data object. It's defined with:



let current_page_name = window.location.href;

In principle, we could get the URL of the current page from the request on the backend by I prefer to have all the data in the JSON object.

User origin: We want to know from what website the user came from. This information is important because it allows us to track sources of our web site traffic. Are we getting:

direct traffic (users entering the URL to browser),
traffic via referrals (links to our site), or
via organic search (user finds us via a Search engine like Google, Bing, Baidu ...).

In all browsers except the Internet Explorer, the following will give us the source from which user came:



let page_source = document.referrer;

If traffic is dirrect or user used Internet Explorer page_source will be empty so we set:



if (page_source === "") {
    // could be direct traffic or Internet explorer
    page_source = "empty";
}

Now we can detect what web browser the user has with something like this, but that doesn't help us to determine the source from which the user came. If you know a workaround, please let me know how to get user origin in IE.

Device screen: We want to know what devices the majority of our users are using. We get device screen size via:



let screen_width = window.screen.width;
let screen_height = window.screen.height;

and screen size that we can draw on with:



let screen_available_width = window.screen.availWidth;
let screen_available_height = window.screen.availHeight;

Browser type, language, time zone: To get the browser type we do:



let browser_type = navigator.userAgent;

the language:



let language = navigator.language;

and the time zone:



let time_zone_offset = Intl.DateTimeFormat().resolvedOptions().timeZone;

Tracking parameters: You can enhance your analytics if you publish URL-s with added parameters. For example, you can use the Urchin Tracking Module or UTM, a format used by Google to track your unique URLs:



http://www.example.com/?utm_source=JohnDoe&utm_medium=mail

By adding parameters to links you share, you can segment the traffic way better during the analysis process. For example: What was published by you, what was shared by others, social media source, ...

Page performance: We want to know how long does it take for our web page to load. For that, we need to understand a bit about web browser events:

1.) First, the browser sends the request to the server to get page files.
2.) Page files are sent to our device.
3.) Then the browser needs to render the web page.
4.) Once the web page is rendered, onload/load event is triggered.
5.) The user views the page.
6.) The onload/onunload event happens when the user closes the web page.

The page loading and rendering should happen in a matter of ms. If it doesn't, our user either has a really crapy internet, or we are sending over to many files. Either way, it's good to track that. According to Mozilla docs we can obtain the data about page loading from:



let performance_data = window.performance.timing;

Then get:



let page_load_time = performance_data.loadEventEnd - performance_data.navigationStart;
let request_response_time = performance_data.responseEnd - performance_data.requestStart;
let render_time = performance_data.domComplete - performance_data.domLoading;

We need to trigger page performance monitoring code after the page is loaded. Full code snipet for page perfromance:



window.addEventListener("load", function () {
    let performance_data = window.performance.timing;

    // calculate request response time: network latency
    let request_response_time = ...

    // calculate page render time
    let render_time = ...


    // page load time: wait until load event is finished with setTimeout
    setTimeout(function () {
        let page_load_time = ...

        // Post data to the server
        ...
    }, 0);
});

setTimeOut is needed because we need to wait for the load event to finish before we can measure the page load time.

Stay tuned

The full code can be found on my blog at page views. There you'll find the HTML, JS, python files you need to run the whole thing.

If you have any ideas what else we could track or how let me know in the comment section below.

I'm not a very proficient JavaScript developer, so there is probably a better way to do some of the parts. Any comments and solutions are welcome. Stay tuned for more. Next week we'll look into page view duration tracking. Why an entire article for that? Well, there are a few edge cases with web page closing that can complicate things.

C++ the python way

Ziga Brencic — Thu, 28 Nov 2019 16:56:20 +0000

Second article in series: Intro to Modern C++

First article from Intro to Modern C++ series: here

I keep hearing. But C++ is so ugly to write. It's not like python. Sure. But the C++ community adapted, and they introduced a few cool tricks that make C++ more pythonic. To make it clear, all of the syntax mentioned here won't work in older versions of C++.

You have element based for loops in C++:

std::vector<double> x = {1, 2, 3};
for(auto &element: x){
    std::cout << element << std::endl;
}

python version would be:

x = [1, 2, 3]
for element in x:
    print(x)

Or for a map: key, value. In modern C++:

std::map<std::string, double> dict = {{"a", 2}, {"b", 4}};
for (auto const&[key, value] : dict)
    std::cout << key << " " << value << std::endl;

And in python:

dict = {"a": 2, "b": 4}
for key, value in dict.items():
    print(key + " " + value)

OK true there's nothing similar to python's enumerate in C++ but you can always do:

std::vector<double> x = {1, 2, 3};
unsigned int i = 0;
for(auto &element: x){
    std::cout << i << " " << element << std::endl;
    ++i;
}

For now that's the best implementation I came up with in C++ that is "similar" to python enumerate:

x = [1, 2, 3]
for i, element in enumerate(x):
    print( i + " " + element)

So can C++ be completely pythonic?

It’s quite interesting how the languages evolved over time. python was written with the aim to provide a simpler functionality than C++. But now python concepts are migrating into the C++ codebase. Though C++ can never replace python and python can never replace C++. But they can work really well hand in hand if we know how to leverage their strengths. Sure, in python we need to write less code then in C++. But honestly, if I have to type 10 more characters to get 15 times of code execution speed up, I’ll go with C++.

Next week I’ll show you how to bring JSON into the C++ world.

Intro to modern C++

Ziga Brencic — Wed, 27 Nov 2019 16:25:49 +0000

Cover image source

I turn to C++ when I have to do some sort of massive data crunching. In any other case I turn to python since I can write numeric code in it faster. But there are cases where C++ is the only way to get out alive. We'll mention some advance concepts here but mostly to make you as a novice aware that they exist. Because you always don't know what you still don't know and I'll try to bridge this gap a bit.

C++ not the C way

In natural sciences for some reason people practice and teach the young the C++ the C way. Maybe they are stuck in the past or were just lazy to learn the cool features of modern C++ (C++ 17). But this C is C++ mantra is producing an army of coders that are not leveraging modern tools.

I keep seeing code that uses double x[100]; for arrays instead of std::vector<double> x(100);. Or even worse: double *x = new double[100];. Hack when I started coding I was doing the same thing. Then someone told me. Hey there's this thing cold STL (C++ Standard Template Library) that has vectors that are smart. If you try to access an element that doesn't exist it will raise a warning. Cool that's worth exploring.

If you use std::vector<double> instead of a double array you have some pretty cool features at your disposal. You can enlarge the vector as the program gets executed, resize it, clean it up, ... To access elements you can use x.at(i) instead of x[i] sure the [i] is faster but it doesn't check like .at(i) if you are accessing elements that are not in the vector.

IDE is your friend

Use something like CLion it's the best Interactive Development Environment out there. It will tell you what you screwed up, remind you that you're maybe making some redundant copies of data, ... For the geeks. You can use vim mode in it 😉

Don't use raw `*` pointers. Ever !!

But they are fast right. Yes sure raw pointers are the most efficient way to blow your head off 😀 General rule of thumb is that you don't use pointers at all. OK except for the reference & which you use when you don't want unnecessary copying. For example when passing data to a function:

void my_calculator(const std::vector<double> &x){
// implementation
}

const will make sure that the function can't modify x that you passed in. If you wouldn't pass in &x but just x the data would be copied for no reason.

If you really have to use pointers then smart pointers. std::shared_ptr or std::unique_ptr are the answer. Sure there might be some small percentage of cases where the library you are using forces you to use raw pointers (BTW that's bad library design). In that case keep in mind that smart pointer is just a smart envelope around the raw pointer so you can always get the raw pointer 'out'. Example of packaging an arbitrary C++ object into a smart pointer would be:

std::shared_ptr variable_name = std::make_shared<Object>();

Instead of the raw way that you shouldn't use ever:

double *x = new double [100];

Now why the hack is something like double * so dangerous? Well if you delete the raw pointer to your data you will have pretty hard time getting back to it and cleaning it up. So before you realise your RAM is full and your PC can't perform any new calculations. In fancy words you cause a memory leak. By using smart pointers the smart pointers implementation sort of prevents you from doing that.

Code readability over performance

I always recommend that you write code with readability in mind first. So other people (yes your future self in 2 years included) can understand, read and use the code afterwards. I propose that you:

write your programs in English even if you don't have to. They become more readable.
split larger sections of code into files: ~300 lines max per file
don't use too much libraries per project
use descriptive variables not something like stRG2 or sss. The variable has to tell the reader descriptively what it represents. Good practice would be num_of_simulation_steps for example.

There a thousand more guidelines but in general the above should give you the basis. For more you can check code style guides from google or someone else. But be careful at the end of the day you want to write code that produces good results so strike for healthy balance on spending time writing clean code and one that works. Your code readability will improve as you gain programming experience. In a sense programming is learning tricks and the person with most tricks is the 'best coder'.

Don't go down the optimisation rabbit hole

Don't try to optimise the code for performance from the start. In most of cases that's usually a bad idea. Simply because before you write your whole program you don't know what the slow parts are. Maybe some function is slow but you don't really care because you run it once in your entire program. On the other had you might be executing specific line of code millions of times per program execution. That's the part that should be optimised. So general rule of thumb. Write your program. Then figure out where most of the program execution time is spent and optimise that part.

Key takeaways

So what you should take from here:

use CLion editor (it's free for students and academics for non commercial use) and cmake build system
file that contains int main(){} should be named main.cpp that's usually good practice 😀
use descriptive names for variables and objects
don't do stuff globally, in 99% of the cases it's bad design
use STL (standard template library)
don't optimize your code for performance from the get go. Compiler is usually smarter then you in most cases :) Let it do it's job.
use STL libraries for example: #include<cmath> from STL instead of #include<math.h>
Don't use global name spaces using namespace std sure you write a bit more code but you keep the option open to introduce functions with same names
strive for code readability over performance. At the end what costs you the most is the time that coders spent on your code base.

If you find any bugs/mistakes shoot me a massage. Stay tuned for the upcoming articles. This is just the beginning of the series 😉

Analytics with vanilla JS: motivation

Ziga Brencic — Tue, 26 Nov 2019 21:41:16 +0000

How to implement a feedback loop into the product?

A useful analytics tool will provide a direct feedback loop on how to improve our product. Every product (business/web page/blog, ...) want's to track different things. For example, user behavior, web site visits... For some of that, you can use Google Analytics or another commercial tool. But what if you want to track something extra? Or you don't like that Google charges you for data exports. Or you simply don't want to share user data with a 3rd party. All of a sudden, you end up with some hybrid custom solution. But why bother with tracking in the first place?

Users are lazy. They won't take the time to tell you what's wrong with your product. They'll just leave and take their $$$ elsewhere. So how can you improve your product? By catching as much user data as possible and analyzing it without endangering the user's privacy.

I had to start somewhere, so I decided to build my own web page analytics tool. If I used every tool on the market, I would end up with a blob of JavaScript:

But I wanted something with a minimal amount of code and zero libraries. So pure JS, not 1000 libraries. Partially to reduce complexity but mostly to learn.

When I headed out to build my own analytics tool, a quick search online didn't help much. Sadly most of the knowledge on analytics tool building is in the hands of big companies like Facebook, Amazone, Netflix, Google... Then at some point, I came across a simple example by accident. Then built forward from thereon. My process?

Google => Implement => Test => Brainstorm => Google => Implement => ...

User privacy: Depending on where you provide your web page (EU, for example), you have to notify the users about tracking and allow them to disable it. Track only the data you need for the development of your product. Don't track for advertising and selling data. Be better then Facebook ;) Encrypt the tracking data, so even if it gets stolen, it is a bit harder to decode.

Build products for the user benefit. Not to exploit and sell their privacy.

Requirements

I first though how the user interacts with our page:

Each user starts a user session when he/she requests the web page files from the server for the first time. Once the first request is processed user get's back the first web page view. Then the user clicks a link on the web page and gets back from the server another page view. On each page view, we want to track what the user does: mouse clicks, keypresses, ...

With that in mind, I came up with a few requirements for the implementation:

an alternative to third party tracking tools
only pure/vanilla JS code (no external JS libraries)
support for new web browsers
track users that are not logged in
be able to track:
- number of views per page
- what web site user came from
- how long did the page take to load
- type of user device and screen size
- browser type
- user language
- time zone
- page view durations
- events during user page visit: clicks, keypresses, ...
- user sessions: what pages did the user visit sequentially

With the above list in mind, I started building things.

Stay tuned

In the upcoming article, I'll focus on counting page visits and monitoring the necessary quantities. If you have any ideas, what else we could track let me know in the comment section below.

DEV Community: Ziga Brencic

🐍 Python: Standard library

🐍 Python: Standard library

How to use a module?

IO

Parallelization, background processes

GUI

Reliability

Time

Networking

Math and numerics

Command line/OS interface

Other cool modules:

🐍 Python: Crash course

How language came to be?

Python versions

How to start python?

Hello world

Run a script from the command line

__main__

Why you code?

Motives

Start

Competition

Old technology

Frustration is the way

Why?

Parameters object in C++

Parameter object

Passing Parameters object around

Loading parameters

Benefits for the development process

Is Web Browser local storage available?

Interface

Local storage status

Is local storage quota exceeded

Is local storage available

Get Local storage status

When is local storage convenient

Don't do things globally in C++

Code organisation

Global objects

Localy/Class global

Defensive programing

Analytics with vanilla JS: page view duration

Request page, close page

User leaves the web site

User opens the web site

Tackling edge cases

JSON in C++

Read & write JSON files

Custom JSON serializers

JSON doesn't make C++ totally python

Analytics with vanilla JS: page views

Tracking

Stay tuned

C++ the python way

So can C++ be completely pythonic?

Intro to modern C++

C++ not the C way

IDE is your friend

Don't use raw * pointers. Ever !!

Code readability over performance

Don't go down the optimisation rabbit hole

Key takeaways

Analytics with vanilla JS: motivation

Requirements

Stay tuned

`main`

Don't use raw `*` pointers. Ever !!