* Introduction to Visual Microservices
* Environment Setup
* Executing Algonia project
* Project description
The requirements and expectations of the modern world drive us to the point where specialists from different fields must collaborate and work in tandem.
Software engineers are focused on developing backends and UI engineers on designing applications. Data scientists and analysts are gathering and analyzing data. System integrators are implementing solutions in end customer environments.
The specialists mostly use different and incompatible technologies as there is no one that rules them all.
Python is the most sought-after among data professionals because it provides an environment for easily setting up foundations for Machine and Deep learning. Software engineers may prefer C# which has great support for developing multithreaded applications. On the other hand, C/C++ and Rust are very good at solving performance bottlenecks.
Algonia's vision is to overcome those technology barriers and create tool for smooth collaboration between different technical profiles.
The first step onward to accomplish this goal is ensuring interoperability between programming languages.
Data scientists must be able to seamlessly share their algorithms and models with Software Engineers. On the other hand Software Engineers must provide data sources to data scientists.
UI engineers must be able to flawlessly integrate developed designs with backends, without much of concern about technology powering the backend.
Another problem is that complex systems are hard to maintain. Software is a living creature, constantly developed and driven by new requirements that can easily break existing codebase. Sometimes new features just cannot be added due to current codebase foundations.
Lets see how Algonia deals with those problems.
Everyone who has ever worked on a system that involves a mix of different languages and technologies can witness that project can easily turn into a nightmare. Calling functions from one language to another can easily lead to improper memory releasing causing hard-to-find bugs, memory leaks and crashes.
Algonia’s main advantage is a solution where engineers can easily pack their code into drag'n droppable visual boxes. Once packed, they become a part of the existing ecosystem. They can be connected to other algorithms in any order, regardless of programming language.
Many programming languages are supported and new ones are adding constantly.
Visual programming languages offer easy maintenance and there is no exception here. Visual boxes can be re-connected, deleted from and added to visual workflows.
If you used some of the visual programming tools already, then you know that performance was sacrificed for that benefit.
Algonia is lightning fast and low memory usage on-premise environment that executes visual workflows. It is by far the most powerful visual programming language ever created, with more raw processing horsepower than anything available at any price.
System integrators, end users and other non-developers can create new functionality by purchasing visual boxes from the marketplace and connecting them into existing workflows.
An introductory example will be an application for displaying charts of the historical stock prices.
Architecture, demonstrated on practical Stock Prices project could serve for other projects, even larger ones. Of course you can adopt it also to your needs.
C# is going to be used for fetching data from api, Python for creating charts and JS/HTML for designing user interface.
Make sure you have met following requirements:
- Windows or MacOS
- VS Code
- Python 36, 37 or 38
- NetCore 3.1
- Alpha Vantage api key. It's for free. We’ll use this API for getting Stock prices.
Open command prompt and type command
git clone https://github.com/tvinko/algonia.git
Go to the repository folder after cloning.
We’ll use virtualenv to install Python packages. There are many benefits of using virtualenv for package management. It really shines in an environment with many projects that don’t share the same Python installation. This way we solve conflicting requirements.
Run virtualenv --version command to check if you have installed virtualenv.
If you don’t have it then install it with command pip install virtualenv
Next step is to create a virtual environment with the name venv_38. Run the python --version command to check your python default version.
Then run the command according to your default Python version
virtualenv venv_38 --python=python3.8
virtualenv venv_37 --python=python3.7
virtualenv venv_38 --python=python3.6
Activate virtual environment with .\venv_38\Scripts\activate
source venv_38/bin/activate (macos) command
Install packages from requirements: pip install -r requirements.txt
Now the packages are installed in your virtual, isolated python environment.
Download Algonia Bundle from github repo
Use the latest release and extract algonia-bundle-win64.zip to your repository folder.
Your folder structure should now look like this:
Download algonia-bundle-macos.zip, extract it and drag algonia-designer to your Applications folder.
First and foremost, the weapon of choice here is VSCode because of its versatility. It is a source code editor that can be used with a variety of programming languages and supports multiple platforms such as Linux, Windows, and macOS.
Bundle contains a development tool (IDE) for gluing your algorithms (Algonia Visual Designer) and an environment for executing visual scripts (Algonia Engine).
Algonia Visual Designer contains (among other things) bundled .NET Core SDK, so it doesn't have to be installed on production environment. However, it must be installed on development machine.
When you ship your project to customers, you’ll create a package. This is a lightweight executing environment containing only Algonia Engine, without Algonia Designer.
Using your own .NET Core version and creating Algonia Engine packages will be explained in the following tutorials.
Your team’s projects. Stock Prices contains three projects. They are located in algos directory of repository:
- Python project for graphs imaging algos/py/graphs/plot_graph.py
- NetCore project for fetching data from api algos/netcore/StockPrices/StockPrices.csproj
- HTML/Js for user interface algos/html/assets
Those projects will be glued together in Algonia Visual Designer and executed by Algonia Engine.
The Algonia Visual Designer projects are called visual templates - drawing canvases for connecting your projects. Templates folder contains those Algonia Visual Designer projects.
Python packages, required for the Stock Prices project.
Python requirements file
Scripts for synchronizing your teams and 3rd party projects with Algonia Designer. Use sync.bat on Windows and sync.sh on MacOS systems.
Algonia Designer "glues" your project results (.dll, *.so., *.py, *.js,.html and similar files).
When you compile code in your preferred IDE (VisualStudio Code, Visual Studio...) the library is normally created in the bin folder of your project which is out of Algonia Designer reach.
Those scripts synchronize your libraries with Algonia Designer.
The rule of thumb is: run sync script when you make changes outside of Algonia Designer (in VS Code or your preferred IDE)
Open algonia.code-workspace in VSCode.
Click on CTRL+Shift+B (Win) or Shift+Command+B (MacOS) to get a list of build tasks and select StockPricesTest project.
Synchronize .Net Core library with Algonia Designer by executing sync.bat/sync.sh script. Script will also synchronize our Python and HTML files.
You need to set the location to virtualenv packages so that Algonia Engine will add it to it’s packages path.
On Windows navigate to the algonia-bundle-win64 folder and click on algonia-designer executable to launch Algonia Designer.
Click on Cancel, navigate to Security & Privacy -> General and click Open Anyway
Click on the Project menu tab and select Open. Navigate to the templates/StockPrices folder of your cloned repository and select StockPrices.algo to open the Algonia project.
Packages are located in the site-packages directory.
On Mac this is normally in your cloned repository under StockPrices/venv_38/lib/python3.8/site-packages and on Windows venv_38\Lib\site-packages
Alpha Vantage API is used to fetch Stock prices.
Click Project and Save to save the project. You can also use shortcut CTRL+S.
Select SAVE_GRAPH visual box and repeat procedure.
Click on the Project menu item and click on Save to save the project.
You can change the value from MSFT to let's say ETH and see the result.
And the GET_PRICES visual box calls function GetPrices from the same solution.
Confused? How can “visual box” just call the function from our, or someone else’s project? Create an object instance?
Don’t worry, it’s going to be explained in the next sections.
But I hope that you briefly got some clue at this point: projects created with different technologies and programming languages can be glued together inside Algonia Designer. You can connect them in any meaningful order you want.
Start box is the entry point to our application. Applications can have multiple entry points running in parallel, but there is no need for this in our Stock Prices project.
Next, PRICES_OBJECT box, creates an object from the StockPrices class defined in algos/netcore/StockPrices/StockPrices.csproj project.
Select the box and click on the Add code button in it’s properties window:
This is ordinal C# code, with some extensions:
- In header section we reference C# dlls and write using directives
- Code is written in the code section. Tags containing attribute run with value true are entry points to visual boxes
Rest of the code should look familiar to you. Except for the two functions, get_arg and set_result.
Those two functions are part of the Algonia SDK and their purpose is to control visual boxes and workflows. There are few more functions for that purpose and will be explained later.
this function receives the argument key and returns it’s value. We can set those arguments programmatically or through Algonia Designer user interface. In fact you already set argument on Algonia Designer UI when you set your API key:
this function takes an object and saves it into the current Visual Box. Each Visual Box can store some result that is accessible to other Visual Boxes.
This is a Python visual box containing a user interface. It uses the Python Eel package for making Electron-like offline HTML/JS GUI apps. I’m not going into Eel details as this would be out of scope for our tutorial, but I strongly recommend that you check it out here
Let’s just go through a few interesting points.
Entry point to Visual Box is registerGui function, set in it’s properties window:
Algonia SDK functions here are:
sets the value of the visual box’s argument. First parameter is the name of the visual box that argument is set and the second parameter is it’s value (besides storing results, visual boxes can also store arguments).
It receives a Stock symbol parameter from html input and it’s called by jQuery (we’ll get to HTML and jQuery part later).
The call to set_element_arg stores this symbol as an argument.
algonia_helpers.exec executes Visual Box and all chained boxes connected into the same workflow. The parameter of this function is the visual box name.
Lets just recap what we just did:
- We set the SET_SYMBOL argument to the current Stock symbol that we got from the user interface and then executed it.
- After SET_SYMBOL finishes, GET_PRICES is automatically executed (because those two boxes are visually connected).
All Visual Boxes names, that are executed dynamically with the algonia_helpers.exec method, must be registered and returned as string.
This function returns graph image, encoded as base64 string and will be explained in following steps
There is a new algonia SDK function here that gets the result of Visual Box - get_result_raw.
It takes the name of the box as a parameter.
Remember, Visual boxes can see each other’s results. StockPrices object is stored in a PRICES_OBJECT box.
Now we are going to use this object and call the ChangeSymbol function.
get_element_arg gets the box’s current argument. We set it in previously described eel exposed Python function:
Lets recap what we did:
- This function set the symbol name of the SET_SYMBOL box argument and executed it
- SET_SYMBOL called our C# StockPrices ChangeSymbol method.
- After the SET_SYMBOL, GET_PRICES is executed, because those two boxes are directly connected with virtual wire. GET_PRICES call our GetPrices method defined in StockPrices object
Let's move to the last Visual Box. If you open the code, you can see following Python script:
Interesting part is call to function plotGraph, which is defined in our plot_graph Python project and returns an image in base64 string format:
The image is then shown on UI.
All six Visual Boxes from Algonia Designer canvas were described, but there is one topic left - User Interface.
It is located in libs\algonia\html\assets\main.html file.
Draw Graph button on the user interface executes eel.getPricesRequest function and passes currently selected symbol from html input (ETH, MSFT...)
This call is handled by Eel in Python GUI box that executes SET_SYMBOL visual box:
We just went through all steps of our Stock Prices example. In real life we should handle exceptions and errors more conveniently. We could also simplify workflow by reconnecting Visual Boxes differently. But the purpose of the introduction tutorial was to show different ways of Visual Boxes execution.
Main Algonia purpose is technology interoperability, collaboration foundations between different specialists. In our example we used just C# and Python, but of course there are many more choices.
Algonia also forces loosely coupled and modular architecture from the start. You can easily rearrange visual boxes, adding new and deleting existing ones.
Full blown code including all object oriented goodies can be written right inside Visual Boxes. But the biggest strength of Visual Boxes is that they act like wrappers to your and 3rd party libraries. Libraries are developed independently but their functions are wrapped inside boxes.
Visual Boxes hold their object state - there are not just fire’n’forget function executions.
And last but not least, system integrators and non-developers can maintain systems on their own by adding or removing Visual Boxes purchased from Algonia Marketplace.
We are just entering the next era of programming - Visual Microservices.