DEV Community: Lorenzo (Mec-iS)

ML in Rust: smartcore

Lorenzo (Mec-iS) — Wed, 09 Nov 2022 23:28:53 +0000

Hi,

Today we have released version 0.3 of smartcore: a comprehensive library for machine learning and numerical computing. The library provides a set of tools for linear algebra, numerical computing, optimization, and enables a generic, powerful yet still efficient approach to machine learning.

We provide an easy and familiar API for ML practitioners to have an entrypoint into Rust but also a compact implementation of most popular machine learning algorithms to work with environments running Wasm32 and Wasi.

Easiest way to get the feeling are the Jupyter Notebooks but also the documentation.

Feel free to contribute or ask questions!
Cheers,
Lorenzo

Dagger and CUE, the new kids in the block

Lorenzo (Mec-iS) — Wed, 27 Apr 2022 14:37:55 +0000

Intro

Some of the nice guys that gave Docker to the world have moved on into a new project for automating CI/CD called dagger.io: this new product starts from nice assumptions and all the experience from the previous project, absolutely interesting indeed. What we are going to peak here is the language at the base of Dagger: CUE, to which I have taken a peak recently.

What is CUE

From the CUE language homepage:

CUE is an open source language, with a rich set of APIs and tooling, for defining, generating, and validating all kinds of data: configuration, APIs, database schemas, code, … you name it.

CUE is a superset of JSON specializing in "Validate, define, and use dynamic and text-based data". This means that all the serialization fans can find this useful: JSON follower, YAML prayers, ProtoBuf believers can give it a try to the language and see what is can do to make more solid their configuration files.

Install GO and CUE

CUE is a CLI tool written in Golang and can be installed as a library for your Golang installation as stated in the minimal documentation:

# install as a library
$ go install cuelang.org/go/cmd/cue@latest
# install as CLI
$ git clone git@github.com:cue-lang/cue.git && cd cue && go get cuelang.org/go/cue

(ensure that your $HOME/go/bin directory is in your path).

Try your installation:

$ cue --help

The declarative power

The main point of CUE is to provide powerful schema definition and serialization to different format via a declarative language, let's see an example with mixed data (string, numbers, collections):

// save this content to a namefile.cue file

// you can declare a generic structure to define your map 

nameyourmap: data: "point.json":{ x: 4.5, y: 2.34 }
nameyourmap: yaml_string: """
one: this_is_yaml
two: this_is_also_yaml
"""
nameyourmap:this_is_list: [1, 2, 3.5]

This is a generic CUE file that can be serialized into text formats:

# generate text file from code
$ cue export namefile.cue
{
    "nameyourmap": {
        "data": {
            "point.json": {
                "x": 4.5,
                "y": 2.34
            }
        }
    }
}

Or:

$ cue export namefile.cue
{
    "nameyourmap": {
        "data": {
            "point.json": {
                "x": 4.5,
                "y": 2.34
            }
        },
        "yaml_string": "one: this_is_yaml\ntwo: this_is_also_yaml",
        "list": [
            1,
            2,
            3.5
        ]
    }
}

This is already great as we can easily write scripts to create large configuration files, inject values or even entire files in the position where it is needed.
We can also see three of the basic characteristics of CUE syntax:

inline, easy to read
multiple declarations add new fields
fields cannot be accidentally over-written (try to add nameyourmap:this_is_list: [1, 2, 3.5, 4.5] to the end of the file and you will see an error)

Marshal/Unmarshal formats

In the previous example we had two critical points, at lines 4 and 5 we defined two structure for the fields point.json and yaml_string. How is it possible to have two different formats in the same declaration? That is one of the best feature up to now, as strings can be imported from different formats leveraging the right functions:

// save this content to a test1.cue file

// you can declare a generic structure to define your map 

nameyourmap: data: "point.json":{ x: 4.5, y: 2.34 }
nameyourmap: yaml_structure: yaml.Unmarshal("""
one: this_is_yaml
two: this_is_also_yaml
""")
nameyourmap:this_is_list: [1, 2, 3.5]

Let's run the code:

$ cue export test1.cue
{
    "nameyourmap": {
        "data": {
            "point.json": {
                "x": 4.5,
                "y": 2.34
            }
        },
        "yaml_structure": {   <<<
            "one": "this_is_yaml",
            "two": "this_is_also_yaml"
        },
        "this_is_list": [
            1,
            2,
            3.5
        ]
    }
}

As you can see now the YAML field is not a string anymore but a map defined from the YAML string.

That is all for now. CUE has other superpowers:

Data Validation
Schema Definition
Code Generation and Extraction
Querying
Scripting

You can find some explanations in the documentation.

Dagger is built on CUE to provide building CI/CD pipelines quickly and run them anywhere.

If you liked this post please ❤️ and I will consider going on with this walk-through.

A hidden realm new on Youtube

Lorenzo (Mec-iS) — Sat, 01 May 2021 00:18:42 +0000

I have opened a Art&Science channel on Youtube in which I try to provide mini-experiences based on microscopy and data science. Level of geeking, off-the-counter.
Read video's description to enter the realm.

https://youtu.be/PczEN0HohaU

Please subscribe and like if you want more.

An intuition pump: APIs as cognitive tool — Part 1

Lorenzo (Mec-iS) — Fri, 17 Apr 2020 15:42:11 +0000

For a proper background to enjoy this post take a look to these links about:

Intuition pump: for an overview about how philosophical tales and thought experiments are powerful tools for a productive and generative idea about

intentional stance (briefly, the semantic layer with which we explain causal relations, the collections of meanings and their relations as result of cognitive processes and experiences).

Two Black-Boxes experiment (Dennett, 2010), a thought experiment about two connected machines embedding the same contextualised knowledge of the world and how they can be reverse-engineered to result in collections of meanings (e.g. abstractions that enrich somebody’s intentional stance).

Hydra Ecosystem, a set of tools to design-deploy semantically linked Web APIs. In early development by myself and other contributors in this organisation. We will try in Part 2 to explain how a client-server system works in terms of the concepts built from the Two Black-Boxes Experiment.

If you are quite confident of your understanding of what a Web API is, you can skip to paragraph II.

I. APIs (Application Programming Interfaces) are powerful layers between programs that allow “wiring” of different modules (maybe created by different contributors) with simple programmatic function calls. Integration of software is eased on by just calling functions and forgetting about the underlying implementation; APIs are like membranes with well-documented receptors, that accept some specific proteins and produce expected results. They are bridges over implementation intricacies: a program can ask for an operation and expect the right value to return, without knowing thing about the complexity required to compute the value. Every layer and module of a program is kept together by a multiplicity of APIs; those, at their simplest description, map a name to an operation. Call “ADD 1 3” to receive “4”, ADD is the name followed by inputs, usually what ADD is supposed to do is well documented so the outcome is an expected output ready to be immediatly used by the caller.

Extending this concept to Web APIs using a Client-Server paradigm implies having two different specialised systems that ask-answer computational questions; the only difference is in the fact that the name to be called is a URL (not “ADD” but “example.com/add” ) and there are vastly complicated protocols to allow exhange of inputs and outputs (most of current usage of Web APIs sound more like, for example, “RETURN DATUM x FOR USER y” as, in particular, REST paradigm is about providing data and not about performing operations; for operations like “ADD x y” there are other paradigms that apply like RPC).

II. A system that uses Web API is probably a geographically and semantically distributed artifact, it is the task of the developer that design programs to take advantage of a call to a faraway service. Hopefully these URIs and their inputs and outputs are well documented. This is obviously the foundation for Anything as a Service (I will avoid using an acronym here) as we know it. The developer, according to his/her intentional stance, provides the context for creating a sequence of operations that become an effective process; in the larger picture, the process is part of a product and/or an endeavour.

Leveraging Semantic Web technologies (see Linked Data), developers try to add a layer of abstraction to their Web APIs to allow more powerful applications. In particular W3C Hydra allows semantic annotations of Web APIs using Linked Data standards. What does it mean and how we can describe the level-up that these technologies allow? How the role of software and data exchange is improved by the addition of context (via meaning/semantics) to processes? How, if possible, can APIs be programmatic intuition pumps for software designers/developers in the scope of a particular domain?

III. We can try to build a framework to answer the questions on the lines above by creating a parallel interpretation between the Client-Server system we created at Hydra Ecosystem and the ideal system as described by the Two Black-Boxes Experiment (TBBE); this allows to have a biunivocal perspective on what intuition pumps are, but also what Semantic Web APIs can achieve.

Roughly (please take a glimpse to the linked paper), TBBE delivers a narration from the point of view of some rational investigators analyzing a system made up of two boxes (A and B) with completely hidden mechanism, that are connected by a cable. The first box has two buttons (alpha and beta), the second box has three bulbs (Red, Green and Amber). We can here finds out the initial affinity with APIs, both boxes are themselves (sub-)systems that are accessible via interfaces: the investogators can use the buttons (interface) in box A, box A can use the cable to send a signal to box B, box B returns a value via the bulbs (their interfaces are highly non-uniform, very bad non-standard situation; exactly the opposite of a well-designed software).

Investigators and boxes are all parts of a client-server system in which the machinery of each part is completely inscrutable to the other (a very badly documented set of APIs). Moving on with the investigation it appears evident that the system has a very basic behaviour: pressing alpha triggers the light R on, pressing beta triggers the light G on, light A seems never triggered. These seem, tought, a very limited set of experiments and the lack of documentation does not allow to know the functioning of the single parts.

The investigators move on into disassembling each part. Briefly, all the details in the paper, they find out each box is a very evoluted expert system; each expert system has been created by engineers with quite different backgrounds, but both rely on the same assessments over some true/false statements about some contextualised knowledge. Eventually what the full system is doing in reality: box A was stating a truth (button alpha) or a falsehood (button beta) and box B was confirming a truth (bulb R) or a falsehood (bulb G) or a malformed statement (bulb A). The correlation between buttons and bulbs was not the result of a direct wiring of circuits but, instead, the “agreement” about some statements based on the common reality that the two engineers share of the “world”. The TBBE is full of nuances about how the investigation is carried on and it is very intriguing in establishing boundaries about what is known and what this knowledge implies. I leave to reading further discoveries.

My take on this great thought experiment: TBBE is, generically, about analytically tackling the behaviour (syntactic engine) of a system and trying to establish an intentional stance in the mind of an investigator. A syntactic engine is the manifestation of patterns (simply, the observed behaviour) of a system (i.e. a non-random sequence of 0 or 1, a sequence of characters, etc.); the investigators do not accept the first superficial obervation of patterns, they decide to go with finding out the real flow of information between the boxes by opening them, and they discover a very high complexity in data and encoding. Initially their map of meaning (stance) is very limited and the syntatic engine they witness is so simple that it doesn’t really deserve explanation (one button, one bulb); but, as the investigation goes on and data and patterns start to be highly explanatory of a complex behaviour, their stance evolves to mirror what they observe to be an two-component expert system and all the meanings that the components embed (the shared “world” of the engineers that far away from each other created the two black boxes).

Every cognitive process tries to observe something that intially looks gibberish (highly probable, noise) by analysing recurrences of patterns (highly improbable) to assign relations into maps of meanings (Predictive Coding tries to explain in more detail this concurrent bottom-up and top-down process of adjusted predictions between “syntactic engines” produced by senso-motorial data and high-level representation). What we are initially observing is a syntactic engine made of data streams, there is no “cognition happening” in there and our initial stance is confused. As much the analytical processing goes on the stance improves, and we can better “make sense” of the data.

As we can imagine using a clean and well-documented API is the a great way of entering a specific domain. If you are developing software for any purpose or starting from a specialised background, APIs are gateways to cross-domain awareness; they are great tools of organising knowledge so by consequence to learn principles and make users more familiar with specific domains. Are APIs programmatic intuition generators? Maybe not, but I like the idea of looking at APIs as a valuable entrypoint to many scientific domains.

On the cognitive side, this experiment can be interpreted as a good hint for the strict relationship between meaning and context. Intelligence (information) emerges from data and patterns (from a syntactic engine); assigning meanings to patterns is the domain of a stance, these meanings once “discovered” are strictly bound to a context (the network of relations in which meanings are defined). The intentional stance “store” meanings and relations among those meanings (context); generation of contextualised meanings is a big part of our cognitive activities. In the domain of software engineering, the best way for delivering meanings is to work with clean and well-documented APIs.

Intuition pumps are well-established thinking tools in Philosophy. Is it possible to use APIs functions as building blocks to construct inutition pumps for specific domain. e.g. Which level of understanding of mathematics does reading a math library’s API provides? Does a software engineer mind represent intentionality in terms of APIs? Is this useful to acquire actual knowledge of the domain?
These questions are important as is vital to understand how we experience knowledge and learn that is the field of Cognitive Sciences.

If you are an API designer/developer and you want to know more how these concepts describe the tools developed at Hydra Ecosystem (also, for donations), please proceed to Part 2 of this blogpost.

Ford on in this world of Interfaces!

Memory views: Handling strings

Lorenzo (Mec-iS) — Mon, 26 Aug 2019 15:12:22 +0000

Brace yourself as this post is going to be a little less straightforward than the last ones as we are going to talk about how to read/write UTF-8 strings from/to WebAssembly (Wasm) memory views!

Unicode (a resume)

feel free to skip to the next paragraph if you think you are already familiar with

Even if Rust-compiled Wasm and Python are very different languages in many points, they are probably at the opposite of the spectrum for the most popular programming languages, they both take advantage of Unicode Format for Network Interchange; fortunately nowadays an accepted and well-supported standard.

Some basic generics about Unicode:

... it encodes a character as a sequence of one to four bytes [1]

... only the shortest encoding for any given code point is considered well-formed; you can’t spend four bytes encoding a code point that would fit in three [1]

Every group of code points (for UTF-8, a code point can be as small as 1 byte, up to 4) in the string encodes a character (in other words UTF-8 is an 8-bit variable width encoding for Unicode); usually there is a base code point plus code points afterwards that add details to the base. Unicode version 12.1 allows the recording of 137,994 code points that covers all characters for a large amount of languages.

UTF-8 supports any Unicode character, which pragmatically means any natural language (Coptic, Sinhala, Phonecian, Cherokee etc), as well as many non-spoken languages (Music notation, mathematical symbols, APL) and emoticons [source]

More information about this wonderful Internet-omnipresent tool at Unicode Consortium.

Strings

What we need to know for this experiment is that we can represent strings of characters as lists of 8-bit integers that stand for Unicode (or group of) code points. To pass strings from/to Python and Wasm using python-ext-wasmer we need to know:

In Python strings are immutable sequences of characters; open the Python IDLE and try:

# a Unicode string containing non-ASCII chars
>>> st = 'ABW@☀CC☯'

# same string as bytestring encoded in UTF-8 (hex)
>>> bytes(st, 'UTF-8')
b'ABS@\xe2\x98\x80CC\xe2\x98\xaf'

# UTF-8 (decimal) represention of the same bytestring
>>> [b for b in bytes(st, 'UTF-8')]
# elements for Unicode code points as Python integers
[65, 66, 83, 64, 226, 152, 128, 67, 67, 226, 152, 175]

The black sun with rays (U+2600) code point is represented in UTF-8 encoding with: 3 hexadecimal values or equivalently 3 positive decimal values:

#
# same code point, same encoding (different formats)
#
\xe2\x98\x80 ----> ☀ as UTF-8 (hex)
226 152 128 ----> ☀ as UTF-8 (decimal)

Any sequence of Unicode points can be represented as List[int] in Python; in the particular case, if we want to use UTF-8 to inter-operate with code in Rust we need to use on its side a Vec<u8> (more details here); finally at Wasm-level we have a linear memory array of cells that index its elements:

# pseudo-code
array_memory[0] = 226
array_memory[1] = 152
array_memory[2] = 128

It is now evident how we can store the decimal UTF-8 points in Wasm linear memory.

Wasmer memory views

We can think at every Wasm Instance as a mini-sandbox with its own abstractions over memory. Memory is represented as "views", basically tables that manage references between values and their position in linear memory.

Details: Wasmer runtime core provides these very powerful abstractions that allow to leverage Wasm memory; in this case we want to take a look to python-ext-wasm Memory Views; the interface methods are generated from a macro for every primitive type.

Passing a string

Let's enter the main part of the experiment: how can we use all these powerful tools to make Python talk fluently to Rust/Wasm and receive back?
Briefly:

we are going to tell from Python what to write to Instance's memory using Wasmer views
we are going to call a Wasm function (compiled from Rust) reading from Instance's exported functions
we are going to pass the result back to Python

All this process is demonstrated by this function:

def test_reverse(instance, func, bytestr):

    # return a Wasmer Memory View for 8-bit integers
    mem_view = allocate_bytes(instance)

    # write the UTF-8 (decimal) code points to the view
    mem_view = write_to_memory(bytestr, mem_view, offset=0)

    # call the exported function by reading the offset of the
    # view for the length of the bytes string (b'...')
    result = func(0, len(bytestr))

    # read the result value at a given position in linear memory
    return address_to_utf8(mem_view, result, len(bytestr))

The methods used are loaded from this mini Python-to-Wasmer API.

The result is easily showed by this chunk of code:

Reversing b'Test sTRing' >>>
b'gniRTs tseT'

Thanks for your attention.

Sources:
[1] O'Relly "Programming Rust" ISBN-10: 9781491927281, pg. 392-393
[2] kunststube.net/encoding

Rust-scales Python: Flask Experiment

Lorenzo (Mec-iS) — Wed, 31 Jul 2019 17:33:34 +0000

Going on with this series of brief demos about how to embed Wasmer binaries into Python modules via python-ext-wasmer (see previous episodes 1 and 2).

I wondered what it would look like a Web application that leverages Wasm binaries; let's use Flask micro-framework!

We start the usual way, using the previously compiled Rust functions. For this experiment we pick:

The two versions of Fibonacci, one as Python function, the other as Wasm-compiled Rust;
The two versions for a static computation of Convex Hull.

This time we wrap the pre-compiled binaries into a Flask app and serve them through two endpoints:

As you can see, the loading of the binaries is practically at no-cost in terms of service performance as it is done at app creation. This allows to load as much binaries as we need to perform the processing via the endpoints. It could possibly be a good solution for a RPC API that accepts arbitrary parameters.

Let's try to run it:

[UPDATED]

platform linux -- Python 3.6.8, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/mec-is/rustcode/wasm-python-101
collected 4 items                                                                                 

pywasm/examples/flask_basic/tests.py 2019-08-18 16:17.29 Time it                        func=fibo_wasm time=0.07568788528442383
.2019-08-18 16:17.35 Time it                        func=fibo_py time=0.06508588790893555
.2019-08-18 16:17.41 Time it                        func=convexhull_wasm time=0.10294783115386963
.2019-08-18 16:17.48 Time it                        func=convexhull_py time=0.4820021390914917

Not bad for a first run!

Rust-scales Python: Some examples (1)

Lorenzo (Mec-iS) — Sun, 07 Jul 2019 16:38:30 +0000

Please read the first part of this post before going on with this one.

Introduction

In the first post of this series I displayed the basic commands needed to leverage python-ext-wasm. Wasmer.io is a project aiming to provide a streamlined solution to allow compilation and usage of "Universal Binaries" via WebAssembly; python-ext-wasm is a library that provides binding from Python to compiled binaries so that in future any Python developer can take advantage of all the incredible WebAssembly features by keeping writing Python code.

Environment and assumptions

As described in the previous post we work with a diverse setup without leaving behind ergonomics and ease-of-use. The basics components for starting experimenting are Rust (and cargo) and Python. I have spotted some functions that are going to be our targets for the sake of exercise with the workflow and also provide some basic measures of how useful the Wasmer.io approach is and will be. These functions are written in Rust:

a simple addition function:

pub extern fn simple_add(a: i32, b: i32) -> i32 { a + b}

a Fibonacci series implementation, one of the many you may have used while trying to exercise your coding skills (credits here):

#[no_mangle]
pub extern fn fibo(n: i32) -> i32 {
if n < 0 {
    panic!("{} is negative!", n);
} else if n == 0 {
    panic!("zero is not a right argument to fibonacci()!");
} else if n == 1 {
    return 1;
}

let mut sum = 0;
let mut last = 0;
let mut curr = 1;
for _i in 1..n {
    sum = last + curr;
    last = curr;
    curr = sum;
}
sum
}

The function used to compute Convex Hull over a set of Cartesian points, as taken from Rust geo

use geo::{Polygon, LineString};
use geo::convexhull::ConvexHull;
///
/// Constructive operations: Convex hull in Rust Geo
///
#[no_mangle]
pub extern fn rust_geo_convex_hull() -> () {
// An L shape
let coords = vec![
    (0.0, 0.0),
    (4.0, 0.0),
    (4.0, 1.0),
    (1.0, 1.0),
    (1.0, 4.0),
    (0.0, 4.0),
    (0.0, 0.0)];
// conversions to geo types are provided from several kinds of coordinate sequences
let poly = Polygon::new(coords.into(), vec![]);

// uses the QuickHull algorithm to calculate the polygon's convex hull
let hull = poly.convex_hull();

I picked these functions so that is possible to compare them with their own functionally equivalent counterparts in Python that you can find coded here:

the most basic addition function in Python
a Fibonacci implementation taken from SO that uses a while loop (as the Rust implementation above using a for-loop)
the Convex Hull implementation as provided by shapely, one of the most popular Python geolibrary (pip install shapely[vectorized]). The same input has been passed as in its Rust counterpart.

Running the tests

If you have already your Python virtual environment up with the requirements installed, and you follow the simple instructions in the previous post, you may be able to clone the repo and try on your own:

in the root directory, do your cargo build --target wasm32-unknown-unknown to compile your universal binary with the Rust functions above;
in timing directory run python run_timing_collection.py. This is a series of timeit calls that time both implementations for each function

This is the outcome as printed by my machine listing the Python function timing and the Wasm function timing over some thousands of iterations:

Modules exported from Rust: 
["loop_str", "rust_geo_convex_hull", "fibo", "simple_add"]
py add 1.2614150420049555
t_wasm add 8.43558448299882
py fibo 44.35585713999899
t_wasm fibo 1.240821371000493
py shapely convex hull 47.863506109999435
t_wasm rust-geo convex hull 2.2532036840057117

You should also find some data in timing/data/timing.csv

Disclaimer: I do not assume that this is a proper benchmark but more an exercise, so I do not expect the results to be accurate but only to give a general indication of trends.

Conclusion

I would be glad to receive more data from different setups. If you want you can just drop a comment with the results printed out by your machine; just copy/paste the last line in your timing.csv to help with some data.

Thanks to all the OpenSource maintainers that made this short experiment possible.

Rust-scales Python: Basic Experiment

Lorenzo (Mec-iS) — Thu, 18 Apr 2019 19:33:31 +0000

Some bits

Python bindings from/to C and C++ are probably one of the hottest topic for an intermediate/advanced Python engineer trying to get some handle on profiling and optimisation, and the inner capabilities of Python’s Virtual Machine. These low-level system languages are probably the fellows to walk with towards understanding of Python’s stack and the wider C-family of language.

The latest most notable addition to this family is probably Rust, a Mozilla-backed language that provides a stable API since 2015: there is a lot of enthusiasm around its adoption and in this article I try an experiment that may suggest one reason for; beside the many other advantages you can already read about in many other articles around. Main reasons I find motivating to learn Rust are:

it is multi-paradigm by design:

(C-like structures but also Functional tools beside Static-typing and more) the most popular languages around are designed taking one paradigm as reference, and have tried to cope by adding support for other paradigms via additions to the respective standard libraries. This unfortunately ends up in a mix that makes a lot of engineers unsatisfied; the outcome of a single-paradigm design are usually limitations in possibilities of expression.

it is designed considering user-experience for the entire workflow/life-cycle:

(from installation to release, through testing, documentation, packages managing) most of the languages have grown in their core capabilities before reaching the critical mass to justify the development of automation/productivity tools; this realised into gaps, conflicts and quality differences between tools used at different steps of the developing experience.

it provides the best level of integration with WebAssembly (Wasm) at the moment and possibly in perspective:

WebAssembly has reached a stable interface and it is now a standard; it will be the platform on-top of which Web software is going to be built. Rust provides straightforward access to Wasm, as simple as we are going to show in this experiment.

Let’s see how simple it is to start a project, create a function in Rust, compile the library in WebAssembly and finally call this function from Python (yea! no work on C-bindings with all the memory-safety pitfalls and compiling flags! Python will call native WASM modules).

Simple that matters

You can try this kind of experiment by installing:

rustup: tool-chains manager to navigate the manifold of Rust’s ecosystem
cargo: the package manager
rustc: the Compiler, working along with the rest of a stable Rust toolchain for your machine.

rustup install all of that for you!

Let’s start:

.1. Add the Wasm tool-chain:

 rustup target add wasm32-unknown-unknown

.2. Create a new project and change directory:

cargo new medium-xp && cd medium-xp

.3. Create a lib.rs file in src directory. All code goes in src, lib.rs is where Rust keep the root library code.

.4. Cargo.toml is your package configuration file, add this section to the Cargo.toml file:

[lib]
crate-type = [“cdylib”]

This is to define the project as a dynamic library

.5. main.rs is your entry-point module, we are not going to use it this time as we are creating a library, not a binary. In lib.rs add this code for our function:

#[no_mangle]
pub extern fn simple_add(a: i32, b: i32) -> i32 { a + b}

.6. Compile the library for Wasm:

cargo build --release --target wasm32-unknown-unknown

Now you will find your library in target/wasm32-unknown-unknown/release compiled as WASM module!

.7. On the Python side:

install wasmer extension for Python (tested on Python 3.6):

 python3.6 -m pip install wasmer

create a run-wasm-from-python.py file with this script:

from wasmer import Instance
path = '<relative path to your Wasm file>'
with open(path, ‘rb’) as bytecode:
    wasm_bytes = bytecode.read()
    instance = Instance(wasm_bytes)
    result = instance.exports.simple_add(12, 12)
    print('Modules exported from Rust: ')
    print(instance.exports)  # this will print function's name
    print('call simple_add(12, 12): ')
    print(result)  # 24

run this script with: python3.6 run-wasm-from-python.py

If you have done everything correctly…

Modules exported from Rust: 
["simple_add"]
call simple_add(12, 12): 
24

This demonstrates how it is technically possible to make Python inter-operating with compiled Rust using Wasm. It is incredibly straightforward thanks to the powerful design of Rust and the incredible work done at every step of the tool-chain. As you could see everything runs smoothly with high-levels of productivity involved. I leave to You the considerations about advantages and disadvantages of trying to optimise heavy loads of computing with Rust while keeping the dynamic approach of Python at the higher level. This was possible only recently as all the interfaces and tools involved have stabilised, thanks to the great work done by Open Source developers all over the World.

References

Full repository on Github
Definitely check out this Reddit
Wasmer here

In the same series

New Episode ---> some-examples-1
New episode ---> Flask-experiment

Personal History Of Computing

Lorenzo (Mec-iS) — Sun, 14 Apr 2019 17:18:45 +0000

Philosophical foundations for digital programming

Some Introductory words

"After the tool, responding only to his hand; after the machines, covering complex tasks and operations but subject to his will; here he is to delegate automata to take care of managing and thinking in place of himself, on the basis of apparently rational criteria.“ — H.J. Martin, 1988

I — On contemporary society

“If you want peace, prepare to set yourself free”
“Si vis pacem para remissionem”
“Se vuoi la pace, prepara la liberazione” — A. Capitini

As young reckless troglodytes, in this new Neolithic of competences, we struggle searching for a feeble fire, donated by some other superior ancestors. They are together with a kind of a new machine-mind demiurgos (crafter): Information [1]. Total immersion into media tracks our decisions; three centuries after political individualism single action reaches critic quantity leveraging electronics. The flames of valuable minorities (like myself or any other individual) can reach people, and be carried on. We are again fording on, carrying weak flames on thin branches from camp to camp, struggling to remember, to have a clear representation of human cognition.

This constant intellectual (Latin — inter+lego: collect, gather [2]) process, of camping and leaving from/to representations or from/to frameworks, typical of digitally-high-exposed jobs (communication jobs, finance, coding, programming,engineering, automation ...) proposes new tones of colours for a “nomadic” approach to frameworks and “theories”. Mostly because immobility can spot up dogmatism and because obsolescence is an almost omnipresent threat: to avoid which, we try to build tools to put up this logical ‘architrave’ (a cornerstone for our Archive?[11] )[3], that is for sure a mathematical, physical, algorithmic and, finally, complex tool. Let’s iterate and see what can we say by defining the words “theories” “systems” and “frameworks” in the context of highly interconnected digital networks. This can help in building a caliper to fairly measure reality with the right amount of subjectivity: an Existentialism for Computability (what we are going to define as being-search). What are the strings pulled on logical thinking and the Self by knowledge networks (networks of digital media)? We would like to argument that is the cybernetic yoking of digital networks themselves, with their new languages, artificial or natural. We draw shapes for these structures leveraging, among others, the concept of authenticity, finalised to highlight individual possibilities inside the networks that provide shape translate Information.

Let’s take this starting datum: for every frame into the monitor of a nowadays computer (1920x1080 pixels) we capture the same amount of Information transmitted in months by the whole of the telegraphic network in its early times (1920x1080x32 bits can contain approximately 1,300,000 messages). This unit of measure (bit) has both logical foundation, as measure of a decision-searching process, and physical, as expression of a set of signals on a wire (the first assumption is evident if considering how telephone numbers could address a receiver in early age of telephone). As we are, accustomed to the tools of natural language-based (socio-economical) background, we face, hidden, issues processing masses of digital information. Some general misconceptions of the tooling required is a missing epistemological update; fortunately new theories and frameworks come to the rescue, they are just hard to find, and to carry on the struggle to update them consistently. We are drifting from the lack of knowledge source to the over-abundance of them. This implies a constant adjustment of our filtering abilities inside “frameworks” with the aim of understanding The Medium, the Internet. As a race driver is conscious of the movement of his/her engine’s pistons, we all need epistemological structure to interact and stack Information together with digital machines, we feel the “breathing” of the network and the rising or lowering currents. The way to epistemological structure implies “philosophic harmonisation of logic” worldwide, based on these main components: Mathematics and Humanism in the same feedback loop. This is what has been happening with global software products. What is the state of this comprehension? We need to unravel it in a way that saves diversity while dancing on a common layer, we need peculiarity of existence, this is the outcome of searching over an unprecedented number of sources of knowledge. Ultimately, frameworks, as most of tools, incorporate knowledge to ease usage or even provide black-boxed usage. Is there any difference in this kind of usages?

The sensation of third-party-user-experience referred to cognitive inputs (estrangement), multiplied to the incredible set of digital media we use, is the central knot of the quippus [4] describing the necessity of moving from framework to framework. The movement/process itself becomes Science (Systems Thinking [5]), the importance of meta-skills (reading, writing, learning, possibly very fast) surges. Every one of these experiences needs representation (reading) and binary decision-making (writing) finely tuned on the medium delivering the experience, to allow understanding. The feedback loop dances between man and medium/machine, and the third space in-between, the interface. Reading a proper representation that allows interactions with the medium via the interface (somebody called this loop Cyberspace or, in some quite weird times in the past Will-Representation [6]) is a matter of knowing codes, languages, and metalanguages on which anything, society economics communication, nowadays happens to run. We call this way of solving the problem (aka managing the fire) ‘medium-driven approach to the being-search’, as differentiating itself to a theory- (or proof-) driven approach to Knowledge. Some day-by-day practical implementations of this are research, “code literacy” or more in general pattern-recognition. This is now happening not by observing the manifest behaviour of units (that sounds theory driven isn’t it?), but instead by studying the tools they are massively using (a more “theoristic” approach) and how they are leveraged by the units, that is possibly one way to have some gist of the concept of Complexity. The preferred vehicles of this flow among computing models, media, societies are scientific discourses [8] that, in this case, are hybridisation of natural and artificial languages (algorithms); for which we need to absorb the tools of Cybernetics and Computer Science. As a starter, what we are trying to do here is to characterise what “framework” “method” and “technique” actually mean in this new context.

II — A nomadic approach: continuous differential adjustment

“And the reason that such complexity is not usually seen in human artifacts (tools or machines) is just that in building these we tend in effect to use programs that are specially chosen to give only behaviour simple enough for us to be able to see that it will achieve the purpose we want.” — S. Wolfram, 2002

Chain production (Fordism-Taylorism) had been the way to mass development until current times. Since not so recent time industries have developed fully automated facilities, or even entire segments of industry; they define an output exclusively through machines controlled by few technicians. This movement from the tool to the robot had surely wide opened attractive scenarios for businesses in the services industry, through reallocation of workforce and the improvements in automated tasks and calculations. The goods produced by these facilities have been conquering the markets and found place into our lives; with shapes and materials but, above all, because of the huge immaterial knowledge they carry. How they influence the psychology and the sociology of people? What is the impact into the personal sphere of individuals? Which level of adaptation is necessary to use them consistently? These software-intensive objects (software in this case intended as the immaterial added value embodied in the products) bring a peculiar epistemological style compared to Tayloristic goods, that makes them the top vehicles for a new metabolism of knowledge. This is a trend and getting even more evident for 3D-printed products and eventually future SPIMEs (cloud-based blueprints deployed over automated industrial feedback loops [7]). SPIMEs are ultimately “fully artificial being-search”, while human beings are nowadays, in the context of cognition, “human-machine being-search”.

The process of “to be“[9], being-search for us, is adapting the shapes we use in logical thinking, aiming to a “continuous analysis of different types of transformation” (M. Foucault, 1978, in our interpretation we assume “transformation” in the meaning defined by Ashby [10]); probably the most important group of transformations is framework/paradigm shift, this kind appears to happen much more often in cognitive system behaving in highly-interconnected networks (what somebody calls "liquidity" of opinion). Going beyond the fixed “shape that programs the empirical world by the imposition of his teleology”. We assume here, this is a process of leaving the way of interpreting shift as discontinuity,

“… with the aim of removing the terms of totalitarian history from epistemé, using instead differential analysis.” — M. Foucault

This kind of approach, in the end, is typical of fairly acknowledged media. Modern approaches to the experiment (a “Proof-based epistemology” or complete Formal Logic that witnessed the first cracks appearing with Gödel) was an imposition of teleology (a mechanical system of actions and reactions). In need for new “stable” shapes we are experiencing in our representation abstract and concrete objects, their designers impose a representation of reality using, according to Foucault and Wolfram, a kind of unilateral schema on the empirical world. We are witnessing increase in changes of representations with increasing popularity of Making and Coding. How programming concepts made a switch of paradigm possible? How this paradigm switch touch our cognition? How this ledger of meta-tools (programming languages, protocols, data pipes) that is the Internet is radically changing the way people does Science? Some points: less stability, more decision-making in selecting sources, more powerful frameworks (collection of shapes and/or transformation) but more prone to evolution and/or obsolescence. The eventually probably the biggest issue we are facing: how can we handle being in societies with high-degree and frequency of framework/paradigm shift?

Computational sciences (see computational Linguistics or computational Biology, Genetics, ...) have nature of ‘search for code’ more than search for theories? This kind of search has been involving mainly mathematical and philosophical works, to reach a terrible conceptual breakthrough (summed by Hofstadter, 1976), realised cognitively by widely distributed Internet. Keyword of this unveiling track is the word medium. Medium as a vehicle, a tool, a machine/medium (a system part of a teleological loop): an inter-braiding of human machine and interface: a way of dealing between increasing heights in the morphology of Knowledge. This is an overview about what we can say up to now about the being-search; picking up the first part of our sentence above: “human-machine“ or “machine-mind“, it is up to us, digital-intensive workers, to give a practical example of what it means a proper (truthful, authentic) collaboration among humans and computational devices [12] through languages, frameworks, paradigms for software.

[1] https://www.scientificamerican.com/article/claude-e-shannon-founder/
[2] https://en.wiktionary.org/wiki/lego#Latin
[3] https://en.wiktionary.org/wiki/architrave#English
[4] https://en.wiktionary.org/wiki/khipu#Quechua
[5] https://en.wikiversity.org/wiki/Systems_Thinking
[6] http://critique-of-pure-reason.com/schopenhauers-key-concepts-1-representation-vostellung/
[7] https://en.wikipedia.org/wiki/Spime#Concept
[8] http://www.massey.ac.nz/~alock/theory/foucault.htm
[9] as starting point: https://en.wikipedia.org/wiki/Being_and_Time
[10] https://en.wikipedia.org/wiki/W._Ross_Ashby#Publications
[11] https://pdfs.semanticscholar.org/24d2/3b944b269356f88dcefd5536b02c4636bc3f.pdf
[12] https://en.wikipedia.org/wiki/Garry_Kasparov#Chess_and_computers

Copyright @ Lorenzo github.com/Mec-iS 2007-2019
This is re-edited, corrected and completed edition of the blog post published here

Python Hydra: Making Summer Great by Hacking Web APIs

Lorenzo (Mec-iS) — Fri, 12 Apr 2019 15:08:25 +0000

This is a blog from 2017, original of this post can be found here. I thought to propose a reedited version here, with some updates and additions. The activities mentioned in this post are the first steps that brought to the foundation of Hydra Ecosystem, our brand-new Open Source Organisation.

With the same enthusiasm this year as well, my adventure as Mentor for Google's summer programme (Summer of Code or GSOC) has happily started and been completed. A great opportunity is offered to students world-wide to understand the Open Source ecosystem and experience real-job activities on the field of real-world OS organizations, by contributing to software repositories while been funded by Google Open Source. This year I managed to organize a super-powered team, putting together efforts from:

Python Software Foundation's mentorship programme, very active in facilitating access for smaller OS organizations;
W3C Hydra developing group that is designing a very interesting specification to make Web APIs (in particular REST) to communicate automatically (see Hypermedia for a general introduction);
Two kind students that had the great feeling for understanding the importance of what we are doing, and decided to submit proposals to realize the project: Chris Andrew and Akshay Dahiya.

Hydra

Hydra is a tool defined in a draft registered at W3C and the specification is in active development by Markus Lanthaler and the members of the team (video presentation here) in the working group.

Hydra is a client-server framework (based on W3C's RDF) used to describe and publish Web APIs to network agents. Through modeling a high-level data-structure ("ApiDocumentation" or shorter "ApiDoc", written in JSON-LD), leveraging the semantic tools provided by the specification, developers can make their data accessible to any Hydra-aware client around.

Accomplishments

The Hydra client-server system that makes possible the implementation of this framework can now be set up easily by adding your own custom ApiDoc to a server software codenamed hydrus (Hydra Universal Server), that has been developed by the students this summer. Hydrus let developers to run the system through a two-steps procedure: writing once their ApiDoc and start-up the server with or without pre-loaded data (complete instructions here).

ApiDoc can be written once at system design phase and amended later if needed; clients asks for the ApiDoc when handshaking with the server and this way they are automatically able to operate the interface, implementing the business logic that drives them. Once the system is set up server-side with the right documentation and data, any Hydra-aware client can connect and start operating with the interface (clients can be found here and here, here an example of ApiDocumentation).

The next idea that has been implemented during this summer is a visualization of all the work done by Hydra in the background of a complex network. That is why we came up with developing a graphic interface that simulates a real-world scenario and displays every API interactions going on. We called this browser application hydra-flock-demo. The flight of a flock of commercial drones is simulated to demonstrate how Hydra documentation can make the built-in clients in every drone to communicate changes of state to a central server; every component in the simulation is both a Hydra-aware client and a hydrus server instance.

Unexpected outcome

The ones described above are quite cool achievements to demonstrate how a Python-Hydra implementation can easily bring automation in a network of Web interfaces. As spill-overs we have been experiencing great insights into designing and implementing with Python functional programming tools a quite usable small-scale DBMS for storing semantic triples using PostgreSQL, see the design here and data model here. Furthermore, some nice options came up about how to index this data using an HexaStore for example, all the experience has been really fruitful since we started with the preparation in February. More insights have arose from understanding smart devices' operational procedures and how to make them send/receive signals and perform message-passing operations efficiently, the protocol for drones' communications is super-simple and clear, meant to be efficient in terms of frequency of messages and size of payloads.

All this material is going to be object of live-demo events to bring these tools to reach a wider audience. You can read students blog posts describing in more detail this adventure here and here.

Potential growth

In the scenario of interconnected devices (IoT) and linked knowledge-bases, Hydra and its ecosystem of tools can lead to higher level of integration and automation between agents. The possibility of driving a network of APIs by adjusting and improving their documentation is incredibly interesting for data but also for cloud services.

Thanks again to Python Software Foundation, W3C-Hydra and Google.

Where to start

We are developing an ApiDoc from scratch in this repository. That is the easiest way of having a grasp about what Hydra is about.

Update

After two year of dedication these activities brought to the foundation of Hydra Ecosystem.

References

M. Lanthaler, "Full-on Hypermedia APIs with Hydra," presented at the API Strategy & Practice Conference.
C. Pautasso, E. Wilde, and R. Alarcon (editors), "REST: Advanced Research Topics and Practical Applications," Springer New York, 2014.
L. Richardson, M. Amundsen, and S. Ruby, "RESTful Web APIs," O'Reilly Media, 2013.
M. Lanthaler, "Building Next-Generation Web APIs with JSON-LD and Hydra," presented at the Symfony Live Portland 2013 [slides] [video].
M. Lanthaler and C. Gütl, "Hydra: A Vocabulary for Hypermedia-Driven Web APIs," in Proceedings of the 6th Workshop on Linked Data on the Web (LDOW2013) at the 22nd International World Wide Web Conference (WWW2013), 2013 [slides].
M. Lanthaler, "Creating 3rd Generation Web APIs with Hydra," in Proceedings of the 22nd International World Wide Web Conference (WWW2013), 2013, pp. 35–37 [slides].
M. Lanthaler and C. Gütl, "Model Your Application Domain, Not Your JSON Structures," in Proceedings of the 4th International Workshop on RESTful Design (WS-REST 2013) at the 22nd International World Wide Web Conference (WWW2013), 2013, pp. 1415–1420 [slides].

Deep Learning is officially "a critical component of computing"

Lorenzo (Mec-iS) — Thu, 28 Mar 2019 11:35:56 +0000

according to Turing Award committee

https://www.bbc.co.uk/news/amp/technology-47721129

Being a Mentor at Google Summer of Code

Lorenzo (Mec-iS) — Wed, 27 Mar 2019 11:36:33 +0000

Hi,

I am Lorenzo (@Mec-iS), administrator and mentor at hydraecosystem.org and contributor for Python Software Foundation Mentorship Program.

We are an organisation at its third year developing for Google Open Source Program, in particular at Google Summer Of Code (GSOC), take a little time to read about as it is a great driver for Open Source initiatives!

We are looking for mentors to help with code-reviewing students' Python contributions on Github and/or review/write our documentation. For an overview you can also take a look to our blog. If you are interested we can provide support for becoming a good mentor and loads of knowledge about REST Web APIs and Python.

Cheers,

Lorenzo

PS. You can join via Slack