DEV Community: Tomaž Vinko

CI/CD from Zero To Hero 2/N

Tomaž Vinko — Mon, 03 Jan 2022 14:29:20 +0000

This is the second tutorial in CI/CD series. Please refer to first post for introduction.

In this tutorial, we are going to explain what are we CI/CD-ing.
System is hosted on github.

I'm not going into application logic details, because the scope of the tutorial series is to create CI/CD pipeline.
So let's just briefly go through some technical stuff here.

TrackingService
This is Microservice API that receives and validates requests. GET method of TrackingServiceController is pretty much self explanatory:

/// <summary>
/// Validates account and publish event if account is valid
/// </summary>
/// <param name="accountId">Account to validate</param>
/// <param name="data">Published event message data</param>
/// <returns></returns>
[HttpGet]
[Route("{accountId}")]
public IActionResult Get(int accountId, string data)
{
    var accountStatusData = new AccountStatusData(accountId);
    // Validates account 
 _trackingServiceAccountUnit.CheckAccountStatus(accountStatusData);

   // Account has passed validation
   if (accountStatusData.HttpCode == HTTP_CODES.SUCCESS_ACCOUNT_ACTIVE)
   {
       if (!_mqttService.Connect())
       {
           accountStatusData.HttpCode = HTTP_CODES.ERROR_MQTT_CONN;
           accountStatusData.HttpMessage = "MQTT Connection error";
       }
       else
       {
           _mqttService.Publish(string.Format("{0}/{1}", _mqttService.RootTopic, accountId),
                         JsonConvert.SerializeObject(new PropagatedData(accountId, data)),
                         MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE,
                         false);
        }
    }
    return new ObjectResult(accountStatusData.HttpMessage) { StatusCode = (int)accountStatusData.HttpCode };
}

WebApi accepts two parameters. The first is the account id that has to be validated and the second is some arbitrary data.
Account can have three states: active (exists in DB with active status), non-active (exists in DB but with non-active status), and invalid (does not exist in DB)
If the account is valid, then event data is forwarded and published by the MQTT broker on accountId topic

TrackingServiceCLI
Tracking Service CLI is the command-line client which subscribes to MQTT topics to receive events from TrackingService WebAPI. It can be controlled from the command line. You can see it in action here.

TrackingServiceTest
Code unit and integration tests. You can see details here

A particularly interesting method is Performance_Test_500_Requests which tests performance - 500 cycles must be completed in under one second. One cycle represents delivering request through MQTT to subscribing client.

The platform is written in C# and uses SQL Server database. You don't have to have NET Core and SQL Server installed on your local machine because CI/CD pipeline will be executed and compiled by GitHub action.

In the next tutorial, we are going to explain GitHub’s actions. You can fork the repository because you are going to create CI/CD pipeline step by step from scratch.

CI/CD from Zero To Hero 1/N

Tomaž Vinko — Tue, 21 Dec 2021 20:46:33 +0000

INTRO

Grab your coffee, beer or whatever keeps you awake, because this series of articles will be a quite long journey. We are going to create a fully functional CI/CD system from the scratch.

Lets start with some official documentation
Continuous integration (CI) is an automation process for developers. Your code changes are regularly built, tested, and merged into a shared repository.

Continuous delivery (CD) is on the other hand a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, without doing so manually.

Those definitions might be hard to understand if you're new to CI/CD. In practice it looks like this:

You made some code changes locally
Push those changes to your code repository (github...)
Code repository receives your changes and executes automated actions that compile and test the code
In the final phase, the deployment to your environments (testing, staging, production...) is made. Those deployments can be on cloud (AWS, Azure, Google...) or on premises

In practice, those cycles might be more complicated. But the point is that the whole procedure is fully automated - from code commit to deploying.

Our CI/CD pipeline

Through the next tutorials, we are going to implement following pipeline:

GitHub Action will be triggered on new code commit
Action will prepare the environment for pulling your repository
It will compile your system and run tests inside this environment
Deploy system to Kubernetes cluster on your local machine

CI/CD stack

As you might guess, there are many tools that help you reach those goals. Our stack will use following technologies:

Kubernetes

Open-source container orchestration system that enables you to automate containerized application deployment, scaling, and management. You can install it from here

Minikube

Kubernetes cluster contains master and worker nodes running on different machines. Deploying those kind of system would be overkill for your daily development work.
Minikube enables you to run a single-node Kubernetes cluster right on your development PC.

GitHub

We will trigger code compilation and testing with the GitHub Actions. You will need a GitHub account

ngrok

GitHub Actions will deploy system on Kubernetes cluster running on your development machine. You probably don't have your PC accessible from outside.
ngrok is a free tunneling system to expose your local services externally.
You can create a free account from official ngrok site

Helm

Package manager for Kubernetes.

WHAT ARE WE CI/CD-ing?

Note that focus won't be application per se as we're deep diving into CI/CD, but let's describe it anyway.
You could use CI/CD template described in this tutorial for any kind of application, regardless of programming language (with a minor adoption)

Our project will be Tracking Service Platform made of three parts:

API Microservice

API receives events which contain two-piece information: user account and any arbitrary data.
When the event arrives at our almighty API, it checks the account status that can be active, disabled or non-existing.
If account is active (found in database with active status) then some interesting things happen - it broadcasts it to subscribers.

Mosquitto

Broadcasting events to subscribers lead us to need for some kind of publisher/subscriber system.
The middleman between API and clients is Mosquitto - message broker which implements MQTT protocol.

Clients

Those can be different types of clients that receive published events. In our case, it's a console application.

The code is hosted on GitHub where each CI/CD step will be explained in the next tutorials.

Flex Tables In Vertica

Tomaž Vinko — Fri, 16 Jul 2021 11:56:20 +0000

Vertica Analytics Platform

Column-oriented Vertica Analytics Platform was designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses and other query-intensive applications.

This Wikipedia citation was enough for me to start exploring the Vertica database.

After my journey into the realm of Vertica, I found great support for machine learning, efficient columnar storage, great performance and Flex Tables among others.

I found the latter one very interesting, so let's dive into it.

What are Flex Tables

Flex Tables are nothing more than flexible tables :)
They are different kinds of database tables, designed for loading and querying unstructured data.
Flex tables can contain only unstructured, raw data, or both unstructured and columnar data.
You can create a flex table with or without a schema or real columns. Hybrid tables consist of both unstructured and real columns.

Let's get our hands dirty

If you don't have Vertica already, grab free version

I run queries in DbVisualizer, but you can also use other Vertica compatible tools.

We'll import 1 million records from the CSV file into Flex Table.
First, download and extract the SalesRecords.zip archive.

Then create flex table:

CREATE FLEX TABLE csv_hello_world();

We created two tables with the previous command.
First one is flex table csv_hello_world and the second one is auto generated normal table csv_hello_world_keys.

csv_hello_world_keys table contains three columns:

key_name: The name of the virtual column (key).
frequency: The number of times the key occurs in the map.
data_type_guess: The type guess that the helper functions determine for each distinct key in the map data.

We'll need to use COMPUTE_FLEXTABLE_KEYS to calculate this information for the flex keys table columns:

SELECT compute_flextable_keys('csv_hello_world');

Let's fire query command to see the content:

SELECT * 
FROM csv_hello_world_keys;

As expected, the table is empty. We didn't import our CSV yet, so the function doesn't have any information for type guessing.

Import CSV with following statement:

COPY csv_hello_world 
FROM '/home/dbadmin/docker/data/SalesRecords.csv' PARSER fcsvparser();

Change file path to your local path where you extracted the archive. I'm running Vertica in docker so this is mapped to my host machine.

Now run COMPUTE_FLEXTABLE_KEYS command again:

SELECT compute_flextable_keys('csv_hello_world');

and check csv_hello_world_keys table for updated keys:

SELECT * 
FROM csv_hello_world_keys;

Voila, we have columns information now:

If we query our csv_hello_world table, we'll see two columns:

identity which is just auto increment field
raw binary column that stores loaded data

This doesn't help us much. But if we query virtual columns explicitly, we'll get the values:

SELECT 
        "Region",
        "Country",
        "Item Type",
        "Sales Channel",
        "Order Priority",
        "Order Date",
        "Order ID",
        "Ship Date",
        "Units Sold",
        "Unit Price",
        "Unit Cost",
        "Total Revenue",
        "Total Cost",
        "Total Profit"
FROM csv_hello_world;

Conclusion

Flex tables are just one of the many interesting Vertica features. I encourage you to dive deeper, especially the machine learning part.
SQL script for this tutorial can be found here.

Control .NET runtime from native code

Tomaž Vinko — Thu, 15 Jul 2021 06:06:06 +0000

Introduction

You can find many examples how to P/Invoke C++ code from C#, but there aren't so many the other way around - calling C# functions from C++.

There are many advantages of hosting .NET runtime from C++, but the main two I deal with in real life are:

It's not mandatory for the customer to have installed .NET Core. NET Core can be shipped within your application so you can control the version
Second goodie is that you can load heavy work to C++ and not so performance critical to C#. So you have two benefits - power of C++ and coding speed of C#. Of course you can achieve the same with P/Invoke but I like it another way, because I have more control over .NET runtime. But hey, that's just me :)

Example is taken from the official dotnet github repository, I just made a few adoptions.

Code is located here, so let's dive right into it.

C# code

This is managed C# function that we're going to call from C++:



public static int Hello(IntPtr arg, int argLength)
{
    if (argLength < System.Runtime.InteropServices.Marshal.SizeOf(typeof(LibArgs)))
    {
        return 1;
    }

    LibArgs libArgs = Marshal.PtrToStructure<LibArgs>(arg);
    Console.WriteLine($"Hello, world! from {nameof(Lib)} [count: {s_CallCount++}]");
    PrintLibArgs(libArgs);
    return 0;
}

There's nothing fancy about it. It just prints some text (Hello world of course).
But if you look closely, you can see strange parameter type IntPtr.
You don't see many of those declarations when coding strictly inside the .NET environment. But as soon as you start with languages interoperability, this is your only channel for passing arguments back and forth.
Official explanation is a little boring: A platform-specific type that is used to represent a pointer or a handle.
You can think of it as the starting position of the memory chunk where the argument is located.

Some more details for bravest ones: it's internally represented as void* but exposed as an integer. You can use it whenever you need to store an unmanaged pointer and don't want to use unsafe code.
You don't have to know what void* pointer is atm, but that would be a must when you'll dive deeper into the magic world of languages interoperability by yourself.

Ok, let's move forward. We have some sanity checks at the start. The size of LibArgs (structure is explained in following steps) must match the size passed as second argument.

But the winning line of our function is



LibArgs libArgs = Marshal.PtrToStructure<LibArgs>(arg);

Marshal.PtrToStructure copies (marshals) data from unmanaged block of memory to managed (C#) object.
This is the most crucial part in the whole story. There are some caveats when writing functions that are shared between languages.
You'll earn some of the hardest memory bugs if you screw up here.

Let's take a closer look at the LibArgs structure definition:



[StructLayout(LayoutKind.Sequential)]
public struct LibArgs
{
    public IntPtr Message;
    public int Number;
}

This is fundamental type in C# (and other modern languages). They are simple at the first sight, but can become complicated very fast.
If you stay just inside the .NET environment, you don't have to worry how fields are laid out in memory.

Let's say that you have a few byte fields, int and short fields and you calculate that struct size is 12 bytes.
No, no, wrong. The fact is that compiler add padding bytes to align data within a struct. The reason is that some CPUs like data to be aligned on address boundaries.
Holy s**t, you created the application that crashes randomly.

But don't worry (or you maybe should with complicated structs), because this padding can be controlled explicitly with StructLayout attribute:



[StructLayout(LayoutKind.Sequential)]

We are using Sequential enum (this is also the default value).
Compiler will assign structure sequentionaly as listed in the definition.
With Explicit enum for example, we could specify the size of each field.

Armed with knowledge till now, it's not hard to figure out what one more interesting line in our application does:



Marshal.PtrToStringUni(libArgs.Message)

Let's move now to C++ side.

C++ code

There are few steps before starting with C# interaction.

Load HostFxr

First step is to load HostFxr and get pointers to a few functions that we're gonna need.
Those are

hostfxr_initialize_for_runtime_config: initializes a host context and prepares for initialization of the .NET Core runtime
hostfxr_get_runtime_delegate: gets a delegate for runtime functionality
hostfxr_close: closes a host context ```cpp

bool load_hostfxr()
{
// Pre-allocate a large buffer for the path to hostfxr
char_t buffer[MAX_PATH];
size_t buffer_size = sizeof(buffer) / sizeof(char_t);
int rc = get_hostfxr_path(buffer, &buffer_size, nullptr);
if (rc != 0)
return false;

// Load hostfxr and get desired exports
void *lib = load_library(buffer);
init_fptr = (hostfxr_initialize_for_runtime_config_fn)get_export(lib, "hostfxr_initialize_for_runtime_config");
get_delegate_fptr = (hostfxr_get_runtime_delegate_fn)get_export(lib, "hostfxr_get_runtime_delegate");
close_fptr = (hostfxr_close_fn)get_export(lib, "hostfxr_close");

return (init_fptr && get_delegate_fptr && close_fptr);

}

It's worth to mention that the function *get_hostfxr_path* locates the right hostfxr which is then loaded with *load_library* call.
You can also load it from your custom location. But let's just keep things simple for now.

###Initialize & start the .NET Core runtime
Now we need to get a runtime delegate that allows loading a managed assembly.
```cpp


const string_t config_path = root_path + STR("DotNetLib.runtimeconfig.json");
    load_assembly_and_get_function_pointer_fn load_assembly_and_get_function_pointer = nullptr;
    load_assembly_and_get_function_pointer = get_dotnet_load_assembly(config_path.c_str());
    assert(load_assembly_and_get_function_pointer != nullptr && "Failure: get_dotnet_load_assembly()");

Load managed assembly and get function pointer to a managed method

Now it's time to load our managed library and get a function pointer to the managed Hello method.
To do this, we need few information:

assembly path
type name
method name ```cpp

const string_t dotnetlib_path = root_path + STR("DotNetLib.dll");
const char_t dotnet_type = STR("DotNetLib.Lib, DotNetLib");
const char_t *dotnet_type_method = STR("Hello");
//
// Function pointer to managed delegate
component_entry_point_fn hello = nullptr;
int rc = load_assembly_and_get_function_pointer(
dotnetlib_path.c_str(),
dotnet_type,
dotnet_type_method,
nullptr /*delegate_type_name/,
nullptr,
(void**)&hello);
//
assert(rc == 0 && hello != nullptr && "Failure: load_assembly_and_get_function_pointer()");

### Run managed code
Finally! It's time to fire our Hello World managed function.
```cpp


for (int i = 0; i < 3; ++i)
{
    // <SnippetCallManaged>
    lib_args args
    {
        STR("from host!"),
        i
    };

    hello(&args, sizeof(args));
    // </SnippetCallManaged>
}

And here is the result of our hard work

Conclusion

Although the result of this tutorial isn't much - black screen with Hello World text, you can just imagine the possibilities that opens up.

You can also check my Languages Interoperability project that seamlessly combines different programming languages.

The Power Of Power BI

Tomaž Vinko — Tue, 06 Jul 2021 05:44:17 +0000

Power BI is a tool for data modeling and visualisation.
It can connect to various sources like SQL Server, Vertica, Business Central etc.

But what most impressed me is the web data source. It's basically a web scraper that can extract tables from web pages. This is a very powerful feature which allows you to bring data from web pages that you can visualise and model along with your data warehouse source.

Let’s see Power BI in action on a stock market data example.
We are going to import tables from this page

If you don’t have it already, download free Power BI desktop version

Launch Power BI, click on Get data and find Web data source

Enter url https://www.barchart.com/stocks

Select table and click Load

Visualise and model data

Languages Interoperability

Tomaž Vinko — Mon, 22 Mar 2021 08:28:38 +0000

    * Introduction to Visual Microservices

    * Environment Setup

    * Executing Algonia project

    * Project description

    * Conclusion

Introduction to Visual Microservices

Modern world challenges

The requirements and expectations of the modern world drive us to the point where specialists from different fields must collaborate and work in tandem.

Software engineers are focused on developing backends and UI engineers on designing applications. Data scientists and analysts are gathering and analyzing data. System integrators are implementing solutions in end customer environments.

The specialists mostly use different and incompatible technologies as there is no one that rules them all.

Python is the most sought-after among data professionals because it provides an environment for easily setting up foundations for Machine and Deep learning. Software engineers may prefer C# which has great support for developing multithreaded applications. On the other hand, C/C++ and Rust are very good at solving performance bottlenecks.

Algonia vision

Algonia's vision is to overcome those technology barriers and create tool for smooth collaboration between different technical profiles.

The first step onward to accomplish this goal is ensuring interoperability between programming languages.

Data scientists must be able to seamlessly share their algorithms and models with Software Engineers. On the other hand Software Engineers must provide data sources to data scientists.
UI engineers must be able to flawlessly integrate developed designs with backends, without much of concern about technology powering the backend.

Another problem is that complex systems are hard to maintain. Software is a living creature, constantly developed and driven by new requirements that can easily break existing codebase. Sometimes new features just cannot be added due to current codebase foundations.

Lets see how Algonia deals with those problems.

Language interoperability

Everyone who has ever worked on a system that involves a mix of different languages and technologies can witness that project can easily turn into a nightmare. Calling functions from one language to another can easily lead to improper memory releasing causing hard-to-find bugs, memory leaks and crashes.
Algonia’s main advantage is a solution where engineers can easily pack their code into drag'n droppable visual boxes. Once packed, they become a part of the existing ecosystem. They can be connected to other algorithms in any order, regardless of programming language.
Many programming languages are supported and new ones are adding constantly.

Easy maintenance

Visual programming languages offer easy maintenance and there is no exception here. Visual boxes can be re-connected, deleted from and added to visual workflows.

Performance

If you used some of the visual programming tools already, then you know that performance was sacrificed for that benefit.
Algonia is lightning fast and low memory usage on-premise environment that executes visual workflows. It is by far the most powerful visual programming language ever created, with more raw processing horsepower than anything available at any price.

For non developers

System integrators, end users and other non-developers can create new functionality by purchasing visual boxes from the marketplace and connecting them into existing workflows.

Setting up Algonia Environment

An introductory example will be an application for displaying charts of the historical stock prices.

Architecture, demonstrated on practical Stock Prices project could serve for other projects, even larger ones. Of course you can adopt it also to your needs.

C# is going to be used for fetching data from api, Python for creating charts and JS/HTML for designing user interface.

Prerequisites

Make sure you have met following requirements:

Windows or MacOS
Git
VS Code
Python 36, 37 or 38
NetCore 3.1
Alpha Vantage api key. It's for free. We’ll use this API for getting Stock prices.

Environment setup

Clone StockPrices project

Open command prompt and type command
git clone https://github.com/tvinko/algonia.git
Go to the repository folder after cloning.

Install Python Packages

We’ll use virtualenv to install Python packages. There are many benefits of using virtualenv for package management. It really shines in an environment with many projects that don’t share the same Python installation. This way we solve conflicting requirements.

Run virtualenv --version command to check if you have installed virtualenv.
If you don’t have it then install it with command pip install virtualenv
Next step is to create a virtual environment with the name venv_38. Run the python --version command to check your python default version.

Then run the command according to your default Python version
virtualenv venv_38 --python=python3.8
virtualenv venv_37 --python=python3.7
virtualenv venv_38 --python=python3.6

This command will create a new folder in your project root:

Activate virtual environment with .\venv_38\Scripts\activate
(Windows) or
source venv_38/bin/activate (macos) command
Install packages from requirements: pip install -r requirements.txt

Now the packages are installed in your virtual, isolated python environment.

Download Algonia Bundle

Download Algonia Bundle from github repo

Windows
Use the latest release and extract algonia-bundle-win64.zip to your repository folder.
Your folder structure should now look like this:

MacOS
Download algonia-bundle-macos.zip, extract it and drag algonia-designer to your Applications folder.

Project structure description

First and foremost, the weapon of choice here is VSCode because of its versatility. It is a source code editor that can be used with a variety of programming languages and supports multiple platforms such as Linux, Windows, and macOS.

algonia-bundle-win64 / algonia-bundle-macos

Bundle contains a development tool (IDE) for gluing your algorithms (Algonia Visual Designer) and an environment for executing visual scripts (Algonia Engine).
Algonia Visual Designer contains (among other things) bundled .NET Core SDK, so it doesn't have to be installed on production environment. However, it must be installed on development machine.

When you ship your project to customers, you’ll create a package. This is a lightweight executing environment containing only Algonia Engine, without Algonia Designer.

Using your own .NET Core version and creating Algonia Engine packages will be explained in the following tutorials.

algos

Your team’s projects. Stock Prices contains three projects. They are located in algos directory of repository:

Python project for graphs imaging algos/py/graphs/plot_graph.py
NetCore project for fetching data from api algos/netcore/StockPrices/StockPrices.csproj
HTML/Js for user interface algos/html/assets

Those projects will be glued together in Algonia Visual Designer and executed by Algonia Engine.

templates

The Algonia Visual Designer projects are called visual templates - drawing canvases for connecting your projects. Templates folder contains those Algonia Visual Designer projects.

venv_36/venv_37/venv_38

Python packages, required for the Stock Prices project.

requirements.txt

Python requirements file

sync.bat / sync.sh

Scripts for synchronizing your teams and 3rd party projects with Algonia Designer. Use sync.bat on Windows and sync.sh on MacOS systems.
Algonia Designer "glues" your project results (.dll, *.so., *.py, *.js,.html and similar files).
When you compile code in your preferred IDE (VisualStudio Code, Visual Studio...) the library is normally created in the bin folder of your project which is out of Algonia Designer reach.
Those scripts synchronize your libraries with Algonia Designer.
The rule of thumb is: run sync script when you make changes outside of Algonia Designer (in VS Code or your preferred IDE)

Executing Algonia project

Compile StockPrices project (C#)

Open algonia.code-workspace in VSCode.
Click on CTRL+Shift+B (Win) or Shift+Command+B (MacOS) to get a list of build tasks and select StockPricesTest project.

Sync libraries and Algonia Designer

Synchronize .Net Core library with Algonia Designer by executing sync.bat/sync.sh script. Script will also synchronize our Python and HTML files.

Set python virtual environment packages

You need to set the location to virtualenv packages so that Algonia Engine will add it to it’s packages path.

Open algonia-designer.
On Windows navigate to the algonia-bundle-win64 folder and click on algonia-designer executable to launch Algonia Designer.

On MacOS go to Applications and open algonia-designer. On the first run you’ll get following warning:

Click on Cancel, navigate to Security & Privacy -> General and click Open Anyway

Click on the Project menu tab and select Open. Navigate to the templates/StockPrices folder of your cloned repository and select StockPrices.algo to open the Algonia project.

Packages are located in the site-packages directory.
On Mac this is normally in your cloned repository under StockPrices/venv_38/lib/python3.8/site-packages and on Windows venv_38\Lib\site-packages

Click on the Settings tab, select Open and enter full path to the packages:

Set Alpha Vantage api key

Alpha Vantage API is used to fetch Stock prices.

Select PRICES_OBJECT visual box:

Click on the Add args in the properties window:

Enter your API key in the value field of API_KEY property

Click on the Save button

Click Project and Save to save the project. You can also use shortcut CTRL+S.

Set Python version if needed

Project is currently set for the Python 3.8 environment. If you’re using Python 3.6 or 3.7 then the version must be set on Python visual boxes.
Select GUI visual box and change Python version:

Select SAVE_GRAPH visual box and repeat procedure.
Click on the Project menu item and click on Save to save the project.

Run project

Press F5 to run the application and open main application window

Click on the Draw Graph button

You can change the value from MSFT to let's say ETH and see the result.

Project description

In the main project window you’ll see green boxes, some of them connected and some of them just floating around:

Those boxes are wrappers around our C# and Python projects.

For example, SET_SYMBOL visual box calls C# function ChangeSymbol from StockPrices project (algos\netcore\StockPrices):

And the GET_PRICES visual box calls function GetPrices from the same solution.

SAVE_GRAPH visual box for instance calls plotGraph method from Python project (algos\py\graphs\plot_graph.py):

Confused? How can “visual box” just call the function from our, or someone else’s project? Create an object instance?
Don’t worry, it’s going to be explained in the next sections.

But I hope that you briefly got some clue at this point: projects created with different technologies and programming languages can be glued together inside Algonia Designer. You can connect them in any meaningful order you want.

Now, let’s rewind to the beginning and take a look at the first row of the boxes:

Start box

Start box is the entry point to our application. Applications can have multiple entry points running in parallel, but there is no need for this in our Stock Prices project.

PRICES_OBJECT

Next, PRICES_OBJECT box, creates an object from the StockPrices class defined in algos/netcore/StockPrices/StockPrices.csproj project.

Select the box and click on the Add code button in it’s properties window:

This is ordinal C# code, with some extensions:

In header section we reference C# dlls and write using directives
Code is written in the code section. Tags containing attribute run with value true are entry points to visual boxes

Rest of the code should look familiar to you. Except for the two functions, get_arg and set_result.
Those two functions are part of the Algonia SDK and their purpose is to control visual boxes and workflows. There are few more functions for that purpose and will be explained later.

get_arg
this function receives the argument key and returns it’s value. We can set those arguments programmatically or through Algonia Designer user interface. In fact you already set argument on Algonia Designer UI when you set your API key:

set_result
this function takes an object and saves it into the current Visual Box. Each Visual Box can store some result that is accessible to other Visual Boxes.

GUI Box

This is a Python visual box containing a user interface. It uses the Python Eel package for making Electron-like offline HTML/JS GUI apps. I’m not going into Eel details as this would be out of scope for our tutorial, but I strongly recommend that you check it out here

The wrapper code inside Visual Box is:

Let’s just go through a few interesting points.
def registerGui(self)
Entry point to Visual Box is registerGui function, set in it’s properties window:

This function starts gui with main.html. This is Eel’s way of starting UI:

def getPricesRequest(symbol)

Algonia SDK functions here are:
algonia_helpers.set_element_arg
sets the value of the visual box’s argument. First parameter is the name of the visual box that argument is set and the second parameter is it’s value (besides storing results, visual boxes can also store arguments).

This function is exposed to Javascript with eel.expose attribute.
It receives a Stock symbol parameter from html input and it’s called by jQuery (we’ll get to HTML and jQuery part later).
The call to set_element_arg stores this symbol as an argument.

algonia_helpers.exec executes Visual Box and all chained boxes connected into the same workflow. The parameter of this function is the visual box name.

Lets just recap what we just did:

We set the SET_SYMBOL argument to the current Stock symbol that we got from the user interface and then executed it.
After SET_SYMBOL finishes, GET_PRICES is automatically executed (because those two boxes are visually connected).

def registerExecNodes(self)
There is another Algonia system function, or more precise system callback:

All Visual Boxes names, that are executed dynamically with the algonia_helpers.exec method, must be registered and returned as string.

def saveGraphReuest()
This function returns graph image, encoded as base64 string and will be explained in following steps

Lets go to the next row of Visual Boxes

SET_SYMBOL box

There is a new algonia SDK function here that gets the result of Visual Box - get_result_raw.
It takes the name of the box as a parameter.

Remember, Visual boxes can see each other’s results. StockPrices object is stored in a PRICES_OBJECT box.
Now we are going to use this object and call the ChangeSymbol function.

get_element_arg gets the box’s current argument. We set it in previously described eel exposed Python function:

GET_PRICES box

This box calls the GetPrices method. There is nothing new here, as we already described all Algonia SDK functions contained in this box:

Lets recap what we did:

getPricesRequest Python function is called from Javascript
This function set the symbol name of the SET_SYMBOL box argument and executed it
SET_SYMBOL called our C# StockPrices ChangeSymbol method.
After the SET_SYMBOL, GET_PRICES is executed, because those two boxes are directly connected with virtual wire. GET_PRICES call our GetPrices method defined in StockPrices object

SAVE_GRAPH box

Let's move to the last Visual Box. If you open the code, you can see following Python script:

Visual Box reads result from GET_PRICES (prices in CSV format):

Interesting part is call to function plotGraph, which is defined in our plot_graph Python project and returns an image in base64 string format:

The image is then shown on UI.

User Interface

All six Visual Boxes from Algonia Designer canvas were described, but there is one topic left - User Interface.

It is located in libs\algonia\html\assets\main.html file.
The Javascript part that communicates with Visual Boxes is:

STEP 1

Draw Graph button on the user interface executes eel.getPricesRequest function and passes currently selected symbol from html input (ETH, MSFT...)
This call is handled by Eel in Python GUI box that executes SET_SYMBOL visual box:

STEP 2

Python code executes callback to javascript onGetPricesCompleted function:

This function then calls saveGraphRequest in GUI box:

Here we execute the SAVE_GRAPH box to get a base64 encoded image string and call the javascript function onSaveGraphCompleted.

STEP 3

onSaveGraphCompleted function shows a graph image in divGraph div

Conclusion

We just went through all steps of our Stock Prices example. In real life we should handle exceptions and errors more conveniently. We could also simplify workflow by reconnecting Visual Boxes differently. But the purpose of the introduction tutorial was to show different ways of Visual Boxes execution.

Main Algonia purpose is technology interoperability, collaboration foundations between different specialists. In our example we used just C# and Python, but of course there are many more choices.

Algonia also forces loosely coupled and modular architecture from the start. You can easily rearrange visual boxes, adding new and deleting existing ones.
Full blown code including all object oriented goodies can be written right inside Visual Boxes. But the biggest strength of Visual Boxes is that they act like wrappers to your and 3rd party libraries. Libraries are developed independently but their functions are wrapped inside boxes.
Visual Boxes hold their object state - there are not just fire’n’forget function executions.

And last but not least, system integrators and non-developers can maintain systems on their own by adding or removing Visual Boxes purchased from Algonia Marketplace.

We are just entering the next era of programming - Visual Microservices.

Visual Microservices

Tomaž Vinko — Tue, 22 Dec 2020 12:13:31 +0000

Inheriting dysfunctional code can be nightmare

I frequently struggle with fixing bugs and debugging code that I inherited. The problem is that the code is poorly written,
without programming design principles and many unnecessary duplicate parts of code.

This made me to start brainstorm what would be an ideal solution. I started thinking outside the box, beyond classic object oriented concepts.

I came up with the idea that reusable parts of code would be in the form of drag'n droppable visual boxes of algorithms that I could connect in any order.

I could then run those visual boxes from my code by executing first box (some kind of entry point). Complete chain of connected boxes would be executed and last box will return some computed result back to my main application.

That will give me simple overview what application does. The other advantage is reusability. I could just connect visual boxes in different order and create different application logic. Without recompiling my codebase. Maintenance would be much easier this way instead of doing changes in spaghetti code.

There are no Object Oriented Desing Patterns in this direction yet. Or I'm not aware of them. I'm just going to call it Visual Microservices for now.
Visual because they are represented in form of visual boxes.
Microservices because each box represent some isolated, loosely coupled service.

Visual programming

Of course Visual Programming is nothing new. But curently there aren't solutions there yet, which would solve real life demands. The problems are mostly interoperability and poor performance.

Some of the benefits that must be sacrified by using existing Visual Programming tools:

I don't want that tool use more resources (memory, CPU) than needed. Just firing up an almost empty project created in visual tool takes 50MB of memory. Really??!
I want to be fast. Let's say that we have two or more connected Visual Microservices. I don't want that transition from one microservice to another takes longer then actual microservice execution.
I want to include Visual Microservices written in different programming languages. I like coding in C# but the truth is that in some areas, especially in ML, Python has better support. I want to effortlessly include my C# and Python libraries in Visual Microservices. My ideal tool has to support as much as possible different languages, because there is no tool to rule them all.
Everyone that tells you that a complete solution can be created just with a visual programming tool is a big lie. Yes you can do small projects, but as long as you have complex real world project, your hands are tied. It's nothing worse than when tool dictate which portion must be used. It might happen that in middle of project you realize it's weaknesses and you must start project from scratch with different technology. So I want that my tool could be easily integrated with my code
I want to have debugging capabilities. Not just "Console.log()", but a full blown debugger with breakpoints and run time variables inspection.
I want to share a state between Visual Microservices. I want to share objects - with all their states, properties and functions

My attempt to create "ideal" Visual Tool

As the wish list grows, I decided to make a tool that will overcome all these obstacles. I decided to write my Visual Tool in C because I wanted top-notch performance, portability and support for hosting many different programming languages on the top of it.

At the moment tool supports :

C# and Python languages
Visual Studio and web-pdb integration for debugging Visual Microservices
it can be referenced as library from 3rd party code. This gives me ability to seamlessly experience to include my Visual Workflows into projects

In short, tool is hybrid between visual and ordinary programming. Existing or new algorithms can be easily represented and combined visually. For that reason I also created marketplace for exchanging those algorithms.

I'm going to demonstrate it in the next posts. In meantime, you can check it here

Happy Coding :)

DEV Community: Tomaž Vinko

CI/CD from Zero To Hero 2/N

CI/CD from Zero To Hero 1/N

INTRO

Our CI/CD pipeline

CI/CD stack

Kubernetes

Minikube

GitHub

ngrok

Helm

WHAT ARE WE CI/CD-ing?

API Microservice

Mosquitto

Clients

Flex Tables In Vertica

Vertica Analytics Platform

What are Flex Tables

Let's get our hands dirty

Conclusion

Control .NET runtime from native code

Introduction

C# code

C++ code

Load HostFxr

Load managed assembly and get function pointer to a managed method

Conclusion

The Power Of Power BI

Languages Interoperability

Table Of Contents

Introduction to Visual Microservices

Modern world challenges

Algonia vision

Language interoperability

Easy maintenance

Performance

For non developers

Setting up Algonia Environment

Prerequisites

Environment setup

Clone StockPrices project

Install Python Packages

Download Algonia Bundle

Project structure description

algonia-bundle-win64 / algonia-bundle-macos

algos

templates

venv_36/venv_37/venv_38

requirements.txt

sync.bat / sync.sh

Executing Algonia project

Compile StockPrices project (C#)

Sync libraries and Algonia Designer

Set python virtual environment packages

Set Alpha Vantage api key

Set Python version if needed

Run project

Project description

Start box

PRICES_OBJECT

GUI Box

SET_SYMBOL box

GET_PRICES box

SAVE_GRAPH box

User Interface

STEP 1

STEP 2

STEP 3

Conclusion

Visual Microservices

Inheriting dysfunctional code can be nightmare

Visual programming

My attempt to create "ideal" Visual Tool