DEV Community: Dandy Vica

Getting network DNS addresses for a Windows host in Rust

Dandy Vica — Thu, 13 Jul 2023 16:45:21 +0000

As you probably know, Windows is seriously considering and investing in Rust. The Windows Rust crate is one example of this commitment.
It allows you to call most if not all Windows APIs from Rust natively.

One caveat though: this crate is full of unsafe code.

In one of my future crate, I needed to get the DNS servers of a host. In Linux, it's quite straightforward, looking into the /etc/resolv.conf file.

But in Windows, it's more complicated. One Windows API is meant to get the details of each network interface in the system. it's called
GetAdaptersAddresses.

Let's explain how I achieved this.

First create a binary crate:

$ cargo new dnssrv

Then, add the following to the Cargo.toml file:

[dependencies.windows]
version = "0.48"
features = [
    "Win32_Foundation",
    "Win32_NetworkManagement_IpHelper",
    "Win32_NetworkManagement_Ndis",
    "Win32_Networking_WinSock",
]

The Rust equivalent API to call is:

pub unsafe fn GetAdaptersAddresses(
    family: u32,
    flags: GET_ADAPTERS_ADDRESSES_FLAGS,
    reserved: Option<*const c_void>,
    adapteraddresses: Option<*mut IP_ADAPTER_ADDRESSES_LH>,
    sizepointer: *mut u32
) -> u32

All the different parameters are explained in the Microsoft Windows API link.

To use the API and Windows types, add the following:

use windows::Win32::{
    Foundation::{ERROR_BUFFER_OVERFLOW, ERROR_SUCCESS},
    NetworkManagement::IpHelper::{
        GetAdaptersAddresses, GAA_FLAG_INCLUDE_PREFIX,
        IP_ADAPTER_ADDRESSES_LH,
    },
    Networking::WinSock::AF_INET,
};

The trick is how to convert raw pointer to structures. It's actually quite simple using the as_mut_ptr() function on a vector.

As explained in the Windows API page, first call the API with a low buffer length:

// first call with a empty buffer
let family = AF_INET.0 as u32;  // look for IPV4 addresses
let mut buflen = 0u32;
let mut rc =
    unsafe { GetAdaptersAddresses(family, GAA_FLAG_INCLUDE_PREFIX, None, None, &mut buflen) };

This should return a ERROR_BUFFER_OVERFLOW return code, with the true value needed for the buffer:

// second with the actual buffer size large enough to hold data
if rc == ERROR_BUFFER_OVERFLOW.0 {
    let mut addr = vec![0u8; buflen as usize];  // beware though of the conversion between u32 and usize
    let ptr = addr.as_mut_ptr() as *mut IP_ADAPTER_ADDRESSES_LH;

    rc = unsafe {
        GetAdaptersAddresses(
            family,
            GAA_FLAG_INCLUDE_PREFIX,
            None,
            Some(ptr),
            &mut buflen,
        )
    };

After that it's just a matter of moving through the linked list of adapters, and for each adapter, the link list of DNS ip addresses:

    // second with the actual buffer size large enough to hold data
    if rc == ERROR_SUCCESS.0 {
        // loop through adapters and grab DNS addresses
        let mut p = ptr;

        while !p.is_null() {
            unsafe {
                let mut p_dns = (*p).FirstDnsServerAddress;

                // loop through DNS addresses for this adapter
                while !p_dns.is_null() {
                    let sockaddr = (*p_dns).Address.lpSockaddr;
                    println!("found DNS server: {:?} for adapter '{}'", (*sockaddr).sa_data, (*p).Description.display());

                    p_dns = (*p_dns).Next;
                }

                p = (*p).Next;
            }
        }
    } else {
       println!("error {} calling GetAdaptersAddresses()", rc);
    }

You can grab the full code here: https://github.com/dandyvica/get_dns_ip

Hope this helps !

Photo by Nathan Fertig on Unsplash

Rust procedural macros step by step tutorial

Dandy Vica — Sat, 04 Dec 2021 16:06:33 +0000

Rust procedural macros are one of the most exciting feature of the language. They enable you to inject code at compile time, but differently from the method used for generics by monomorphization. Using very specific crates, you can build new code totally from scratch.

I decided to write this article to share my experience, because event though the different resources are more and more widespread, it's not really straightforward at first sight.

Let's see how it works.

Building a procedural derive macro

The operating principle of the procedural macros is quite simple: take a piece of code, called an input TokenStream, convert it to an abstract syntax tree (ast) which represents the internal structure of that piece for the compiler, build a new TokenStream from what you've got at input (using the syn::parse() method), and inject it in the compiler as an output piece of code.

Using a procedural derive macro

A derive macro is used by declaring the

#[derive()]

attribute, like for example the well-known:

#[derive(Debug)]

Building a procedural derive macro

Suppose you want to create a WhoAmI derive macro, to just print out the name of the structure under the derive statement:

#[derive(WhoAmI)]
struct Point {
    x: f64,
    y: f64
}

What you need to do:

create a brand new lib crate (procedural macros must be defined in their own crate, otherwise if you try to use the macro in the same one, you face the following error: can't use a procedural macro from the same crate that defines it)

$ cargo new --lib whoami

add the required dependencies to Cargo.toml and flags:

[lib]
proc-macro = true

[dependencies]
syn = { version = "1.0.82", features = ["full", "extra-traits"] }
quote = "1.0.10"

define a new regular fn Rust fonction like this one in lib.rs:

use proc_macro::TokenStream; // no need to import a specific crate for TokenStream
use syn::parse;

// Generate a compile error to output struct name
#[proc_macro_derive(WhoAmI)]
pub fn whatever_you_want(tokens: TokenStream) -> TokenStream {
    // convert the input tokens into an ast, specially from a derive
    let ast: syn::DeriveInput = syn::parse(tokens).unwrap();

    panic!("My struct name is: <{}>", ast.ident.to_string());

    TokenStream::new()
}

As you can't use the regular Rust macros to print out some information on stdout (like println!()), the only way is to panic with an output message, to stop the compiler and tell that guy to output the message for you. Not really convenient to debug, nor easy to fully understand the nuts and bolts of a procedural macro !

Now, in order to use that awesome macro (not really handy because it won't compile):

you have to define a new crate:

$ cargo new thisisme

add our macro crate as a dependency:

[dependencies]
# provided both crates are on the same directory level, otherwise replace by your crate's path
whoami = { path = "../whoami" }

replace main.rs source code with:

// import our crate
use whoami::WhoAmI;

#[derive(WhoAmI)]
struct Point {
    x: f64,
    y: f64
}

fn main() {
    println!("Hello, world!");
}

and compile the whole project:

error: proc-macro derive panicked
 --> src/main.rs:3:10
  |
3 | #[derive(WhoAmI)]
  |          ^^^^^^
  |
  = help: message: My struct name is: <Point>

Your can watch the compiler spitting the error message with defined in the procedural macro.

Using the proc-macro2 crate for debugging and understanding procedural macros

The previous method is unwieldy to say the least, and not meant to make you understand how to really leverage from
procedural macros, because you can't really debug the macro (although it can change in the future).

That's why the proc-macro2 exists: you can use its methods, along with its syn::parse2() counterpart, in unit tests or regular binaries. You can then directly output the code generated to stdout or save it into a "*.rs" file to check its content.

Let's create a procedural macro artefact which auto-magically defines a function which calculates the summation of all fields, for the Point structure.

create a new binary crate

$ cargo new fields_sum

add the dependencies:

syn = { version = "1.0.82", features = ["full", "extra-traits"] }
quote = "1.0.10"
proc-macro2 = "1.0.32"

Add the following code in the main.rs file:

// necessary for the TokenStream::from_str() implementation
use std::str::FromStr;

use proc_macro2::TokenStream;
use quote::{format_ident, quote};
use syn::ItemStruct;

fn main() {
    // struct sample
    let s = "struct Point { x : u16 , y : u16 }";

    // create a new token stream from our string
    let tokens = TokenStream::from_str(s).unwrap();

    // build the AST: note the syn::parse2() method rather than the syn::parse() one
    // which is meant for "real" procedural macros
    let ast: ItemStruct = syn::parse2(tokens).unwrap();

    // save our struct type for future use
    let struct_type = ast.ident.to_string();
    assert_eq!(struct_type, "Point");

    // we have 2 fields
    assert_eq!(ast.fields.len(), 2);

    // syn::Fields is implementing the Iterator trait, so we can iterate through the fields
    let mut iter = ast.fields.iter();

    // this is x
    let x_field = iter.next().unwrap();
    assert_eq!(x_field.ident.as_ref().unwrap(), "x");

    // this is y
    let y_field = iter.next().unwrap();
    assert_eq!(y_field.ident.as_ref().unwrap(), "y");

    // now the most tricky part: use the quote!() macro to generate code, aka a new
    // TokenStream

    // first, build our function name: point_summation
    let function_name = format_ident!("{}_summation", struct_type.to_lowercase());

    // and our argument type. If we don't use the format ident macro, the function prototype
    // will be: pub fn point_summation (pt : "Point")
    let argument_type = format_ident!("{}", struct_type);

    // same for x and y
    let x = format_ident!("{}", x_field.ident.as_ref().unwrap());
    let y = format_ident!("{}", y_field.ident.as_ref().unwrap());

    // the quote!() macro is returning a new TokenStream. This TokenStream is returned to
    // the compiler in a "real" procedural macro
    let summation_fn = quote! {
        pub fn #function_name(pt: &#argument_type) -> u16 {
            pt.#x + pt.#y
        }
    };

    // output our function as Rust code
    println!("{}", summation_fn);
}

Now running our crate gives:

pub fn point_summation (pt : & Point) -> u16 { pt . x + pt . y }

So far, so good.

Combining TokenStreams

The previous example is straightforward because we knew in advance the number of fields in the struct.
What if we don't know it beforehand ? Well we can use a special construct of quote!() to generate the summation on all fields:

// create the list of tokens
// tokens type is: impl Iterator<Item = TokenStream>
let tokens = fields.iter().map(|i| quote!(pt.#i));

// the trick is made by: 0 #(+ #tokens)*
// which repeats the + sign on all tokens
let summation_fn = quote! {
    pub fn #function_name(pt: &#argument_type) -> u16 {
        0 #(+ #tokens)*
    }
};

Result is:

pub fn point_summation (pt : & Point) -> u16 { 0 + pt . x + pt . y + pt . z + pt . t }

Hope this help !

Photo by Stéphane Mingot on Unsplash

Go regular expressions

Dandy Vica — Sat, 10 Jul 2021 09:14:17 +0000

This time, I wanted to tackle the regular expression package in Go. In one of my recent projects, I had to use this library. But I have to confess it's not straightforward at first sight. I hope this article will unravel this feature.

To use the library (https://golang.org/pkg/regexp/), just add:

import (
    ...
    "regexp"
    ...
)

to your code.

Following is a collection of tips for solving a specific problem. Throughout the following examples, I used the assert Go package to make it easier to understand.

Compiling the regular expression

Before anything, you need to compile your regular expression before using any package function. You can use Compile() or MustCompile(). The difference between them is that the latter panics whenever an error is found compiling the regex but the former doesn't. If your code doesn't involve a regex provided at runtime, it's safer to use MustCompile() as you know immedialty whether your regex syntax is correct.

Note that the Posix() versions require a Posix regex rather than a Perl-like regular expression.

Is my string matching the regexp ?

To just check whether a string is matching your regex, you can use two different functions:

// don't worry, fake phone number
phone := "202-555-0147"
phoneRE := `\d{3}\-\d{3}\-\d{4}`

// returns: true <nil>
matched, err := regexp.MatchString(phoneRE, phone)
assert.True(matched)
assert.Nil(err)

// or compile and test
re := regexp.MustCompile(phoneRE)
assert.True(re.MatchString(phone))

// but this returns true as well!
assert.True(re.MatchString("202-555-0147mkljhfQDHMFJ"))

// but not this
re = regexp.MustCompile(`\d{3}\-\d{3}\-\d{4}$`)
assert.False(re.MatchString("202-555-0147mkljhfQDHMFJ"))

Beware that the string 202-555-0147mkljhfQDHMFJ is matched using the first regex. The explanation is given is the Compile() definition and by the term leftmost:

When matching against text, the regexp returns a match that begins as early as possible in the input (leftmost), and among those it chooses the one that a backtracking search would have found first. This so-called leftmost-first matching is the same semantics that Perl, Python, and other implementations use, although this package implements it without the expense of backtracking.

You can influence this behavior with the Longest() function.

It's wiser to use MustCompile() or Compile() if you have to reuse your regex to just compile it once.

Using capture groups

Capture groups are called Submatches in the regexp package. You get access to capture groups using on FindStringSubmatch() function:

// use capture groups
phoneRECaps := `(\d{3})\-(\d{3})\-(\d{4})$`
re = regexp.MustCompile(phoneRECaps)

// caps is a slice of strings, where caps[0] matches the whole match
// caps[1] == "202" etc
matches := re.FindStringSubmatch(phone)

// print out: there're 3 capture groups
assert.Equal(re.NumSubexp(), 3)
assert.Equal(matches[0], "202-555-0147")
assert.Equal(matches[1], "202")
assert.Equal(matches[2], "555")
assert.Equal(matches[3], "0147")
assert.ElementsMatch(matches, []string{"202-555-0147", "202", "555", "0147"})

Using named capture groups

To fully benefit from the Python-like named capture groups, you can't have a direct access to the value of the submatch for a particular name. You only have an indirect and unwieldy access: first get all names, the get the corresponding index for that name and then fetch the capture group string:

// use named capture groups
phoneRENamedCaps := `(?P<area>\d{3})\-(?P<exchange>\d{3})\-(?P<line>\d{4})$`
re = regexp.MustCompile(phoneRENamedCaps)

// print out: [ area exchange line], not that the first element is the empty string
names := re.SubexpNames()
assert.ElementsMatch(names, []string{"", "area", "exchange", "line"})

// // indirect access to names
matches = re.FindStringSubmatch(phone)
assert.Len(matches, 4)
capName := names[1]; nameIndex := re.SubexpIndex(capName); assert.Equal(matches[nameIndex], "202")
capName = names[2]; nameIndex = re.SubexpIndex(capName); assert.Equal(matches[nameIndex], "555")
capName = names[3]; nameIndex = re.SubexpIndex(capName); assert.Equal(matches[nameIndex], "0147")

Splitting a string

It might be useful sometimes to split a string delimited with characters matching a regexp:

csv := "a;b;c;;;;d;e;f;;;g"

split1 := regexp.MustCompile(";").Split(csv, -1)
split2 := regexp.MustCompile(";*").Split(csv, -1)

assert.Len(split1, 12)
assert.ElementsMatch(split1, []string{"a", "b", "c", "", "", "", "d", "e", "f", "", "", "g"})

assert.Len(split2, 7)
assert.ElementsMatch(split2, []string{"a", "b", "c", "d", "e", "f", "g"})

Replacing strings

You can replace strings by providing a template made of references to a matched capture group. You can use $1 (or ${1}) to refer to the first submatch, $2 for the second etc:

csv = "a;b;c;;;;d;e;f;;;g"

split := regexp.MustCompile("(;+)")

// prints: "a;b;c;d;e;f;g"
assert.Equal(split.ReplaceAllString(csv, ";"), "a;b;c;d;e;f;g")

digits := "0123456789"
digitsRe := regexp.MustCompile(strings.Repeat(`(\d)`,10))
assert.Equal(digitsRe.ReplaceAllString(digits, "$10$9$8$7$6$5$4$3$2$1"), "9876543210")

You can use names instead:

// using names rather than indexes
digitsRe = regexp.MustCompile(`(?P<zero>\d)(?P<one>\d)(?P<two>\d)(?P<three>\d)(?P<four>\d)(?P<five>\d)(?P<six>\d)(?P<seven>\d)(?P<eight>\d)(?P<nine>\d)`)
assert.Equal(
    digitsRe.ReplaceAllString(digits, "nine=${nine}, eight=${eight}, seven=${seven}, six=${six}, five=${five}, four=${four}, three=${three}, two=${two}, one=${one}, zero=${zero}"),
    "nine=9, eight=8, seven=7, six=6, five=5, four=4, three=3, two=2, one=1, zero=0",
)

Hope this helps !

Photo by Mick Haupt on Unsplash

Implementing SHA2 (256/512) algorithm with Rust const generics

Dandy Vica — Tue, 30 Mar 2021 16:35:30 +0000

The const generics feature is landing in stable Rust from version 1.51. This feature has been longly-awaited, and you can get an idea of its possibilities here: Rust 1.51 release notes

I wanted to give it a try and thought that SHA2 message digest algorithms were a good test bed.

SHA2 algorithm

I'm not going to dig into the details of the SHA message digest algorithm. It's a NIST standard (National Institute of Standards and Technology) labelled FIPS-180. You can get the detailed description in the NIST US government web site: FIPS-180-4

In a nutshell, following is the general algorithm:

initialize the hash value and constants
pad the massage to be a multiple of 64 or 128 (i.e. 512 or 1024 bits)
for each block of data:
- prepare the message schedule using scrambling functions
- for each of the 64 or 80 rounds, compute intermediate hash value
output hash value

I'm only dealing here with the hash of byte-oriented messages.

What can be noticed are the similarities between SHA224/256 and SHA384/512 in terms of functions, initial hash values,
constants, padding:

Tables	SHA224/256	SHA384/512
block size	64 bytes	128 bytes
padding	56 bytes boundary	112 bytes boundary
Initialization vector	8 4-byte values	8 8-byte values
K-constants	64 4-byte values	80 8-byte values
rounds	64	80
functions	see below	see below

Rust implementation

Taking into account all those common features, we can define a SHA hash as:

// hash is either 256 or 512 bits but always 8 u32 or u64 integers
// T is either u32 or u64
pub struct Hash<T, const BLOCKSIZE: usize, const ROUNDS: usize> {
    pub k_constants: [T; ROUNDS],   // ROUNDS = 64 or 80
    pub hash: [T; 8],
    pub scramble_funcs: ScramblePool<T>, // scrambling functions sigma etc
    pub block: [u8; BLOCKSIZE], // BLOCKSIZE = 64 or 128
}

Scrambling functions

This terminology is not found in any NIST paper but I found it useful to name those functions. These are a combination of logical or, and, xor, bit rotation or bit shifting. But they are very similar:

Ch(x,y,z) = (x & y) ^ (!x & z)
Maj(x,y,z) = (x & y) ^ (x & z) ^ (y & z)
Σ0 and Σ1 are the same, but with different shift values
σ0 and σ1 are the same, but with different shift values

Using const generics, we can define those function generically, for u32 or u64 types but also for shifting values:

    // SIGMA(X) = RotR(X,A) ⊕ RotR(X,B) ⊕ RotR(X,C)
    #[allow(non_snake_case)]
    pub fn SIGMA<const A: u8, const B: u8, const C: u8>(x: T) -> T
    where
        T: BitAnd<Output = T>,
        T: BitXor<Output = T>,
        T: Shifter<T>,
        T: Copy,
    {
        x.right_rotate(A) ^ x.right_rotate(B) ^ x.right_rotate(C)
    }

    // sigma(X) = RotR(X,A) ⊕ RotR(X,B) ⊕ X >> C
    #[allow(non_snake_case)]
    pub fn sigma<const A: u8, const B: u8, const C: u8>(x: T) -> T
    where
        T: Shr<Output = T>,
        T: BitXor<Output = T>,
        T: Shifter<T>,
        T: Copy,
    {
        x.right_rotate(A) ^ x.right_rotate(B) ^ x.right_shift(C)
    }

where right_rotate() and right_shift() are trait functions implemented for u32 and u64.

The scrambling functions are then gathered into a generic structure:

// Rust doesn't allow yet utf-8 symbol for variables
#[allow(non_snake_case)]
pub struct ScramblePool<T> {
    pub ch: FnScramble3<T>,
    pub maj: FnScramble3<T>,
    pub sigma0: FnScramble1<T>,
    pub sigma1: FnScramble1<T>,
    pub SIGMA0: FnScramble1<T>,
    pub SIGMA1: FnScramble1<T>,
}

SHA2 algorithm

Because, of all this genericity, we can implement the SHA2 algorithm, bring either SHA256 or SHA512, totally generically. You can browse the code here: https://github.com/dandyvica/sha

Monomorphization

Depending of which algorithm you want to use, it's easy to define an alias for a specific type:

pub type Sha256 = Hash<u32, 64, 64>;
pub type Sha512 = Hash<u64, 128, 80>;

and implement the generic Hash for both u32 and u64:

// example for u32
impl Hash<u32, 64, 64> {
    pub fn new() -> Self {
        let iv: [u32; 8] = [
            0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab,
            0x5be0cd19,
        ];
        Hash {
            hash: iv,
            k_constants: Self::k_constants(),
            scramble_funcs: ScramblePool::<u32> {
                ch: Scramble::<u32>::Ch,
                maj: Scramble::<u32>::Maj,
                sigma0: Scramble::<u32>::sigma::<7, 18, 3>,
                sigma1: Scramble::<u32>::sigma::<17, 19, 10>,
                SIGMA0: Scramble::<u32>::SIGMA::<2, 13, 22>,
                SIGMA1: Scramble::<u32>::SIGMA::<6, 11, 25>,
            },
            block: [0u8; 64],
        }
    }

    pub fn k_constants() -> [u32; 64] {
        [
            0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4,
            0xab1c5ed5, 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe,
            0x9bdc06a7, 0xc19bf174, 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f,
            0x4a7484aa, 0x5cb0a9dc, 0x76f988da, 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
            0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc,
            0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, 0xa2bfe8a1, 0xa81a664b,
            0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, 0x19a4c116,
            0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
            0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7,
            0xc67178f2,
        ]
    }
}

Testing with SHA2 test vectors

The NIST site give a list of files to be tested, for integration test. When running cargo t, those tests are also run and sha digests verified.

I didn't implement SHA224 or SHA384 but is left as an exercice 😀

Hope this help !

Photo by Conscious Design on Unsplash

Rust error management: lessons learned

Dandy Vica — Wed, 23 Dec 2020 18:10:38 +0000

As I improved my knowledge in managing user defined errors in Rust, I decide to share my experience building a CLI application.

Rust error management is powerful, using the Result construction. But it can be sometimes confusing, specially when associated with the ? operator. Using it greatly simplifies the error management in a Rust application, but also introduces a kind of magic, which makes the whole process difficult to untangle.

I decided to build my own error management, to better understand how it works. I didn't resort to custom error crates like anyhow or failure because I already started using mine. Implementing those could have led to a lot of breaking changes I simply couldn't afford.

The Result enum

This enum is described here: https://doc.rust-lang.org/std/result/ and introduces a result which can either be Ok or yield an error with the Err variant. This enum is generic and can be (should be) used whenever it's possible.

Here is a very simple but contrived example:

// returns the quotient a/b if a is divisible by b or an error message
fn divide(a: u32, b: u32) -> Result<u32, String> {
    if a % b == 0 {
        Ok(a / b)
    } else {
        Err(format!("{} is not divisible by {}", a, b))
    }
}

fn main() {
    assert!(divide(5, 2).is_err());
    assert_eq!(&divide(5, 2).unwrap_err(), "5 is not divisible by 2");

    assert!(divide(6, 2).is_ok());
    assert_eq!(divide(6, 2).unwrap(), 3);    
}

Note the different methods to get the core value or the error variant: unwrap() or unwrap_err().

The I/O Result

All I/O Rust standard library functions use a specific error type called Result which seems to be confusing at first sight. This is because the standard library is using an alias:

pub type Result<T> = result::Result<T, Error>;

to lighten the I/O methods' prototypes. The Error defined above is specific to I/O methods which gives an indication of the underlying error, and is actually found in std::io::Error.

Here is an exemple:

fn main() {
    // open a non-existing file
    let f = std::fs::File::open("/foo");
    assert!(f.is_err());

    // analyze error
    let error = f.unwrap_err();
    assert_eq!(error.kind(), std::io::ErrorKind::NotFound);
    println!("error is: {}", error);
}

One problem we can stumble upon here is the lack of context in the last message when printed out:

error is: No such file or directory (os error 2)

The error message doesn't include the file name being tried to be opened.

The ? operator

The ? operator allows you to greatly simplify the error management when calling nested functions or methods. It's a simple and convenient sugar for a match expression:

// opens a file and reads the file or return an I/O error
use std::fs::File;
use std::io::{BufRead, BufReader};

// use std::io::Result on purpose to show it's an alias defined in std::io.
fn read_file(name: &str) -> std::io::Result<()> {
    let file = File::open(name)?;
    let buffer = BufReader::new(file);

    for line in buffer.lines() {
        println!("{}", line?);
    }

    Ok(())
}

fn main() {
    // read ok
    let result = read_file("/var/log/syslog");
    assert!(result.is_ok());

    // read error
    let error = read_file("/foo");
    assert!(error.is_err());
}

Without the ? operator, it's much more verbose (only the first line is desugared here):

let file = File::open(name);
let file = match file {
    Ok(file) => file,
    Err(e) => return Err(e),
};

Nested calls

The power of the ? operator is that you can use it for all functions returning the same Result:

// read several files but use the same return result
fn read_files() -> std::io::Result<()> {
    read_file("/var/log/syslog")?;
    read_file("/var/log/kern.log")?;

    // only read by root, so returns an error for a non-root user
    read_file("/var/log/boot.log")?;
    Ok(())
}

Returning errors from main()

You can also use the Result as a return for the main() function.

fn main() -> std::io::Result<()> {
    read_files()?;
    Ok(())
}

In case of an error, it displays the error message on the console when run:

Error: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }

Problem with several error types

Until now, everything is fine and dandy. But things get worse when not all results are of the same type. Suppose you get a file with a list of string regexes and you want to create a vector of Regex structs from the regex awesome crate:

// read a file and create a vector of Regex structs
use regex::Regex;

fn get_regexes(name: &str) -> std::io::Result<Vec<Regex>> {
    let file = File::open(name)?;
    let buffer = BufReader::new(file);
    let v = Vec::new();

    for line in buffer.lines() {
        let expr = line?;
        let re = Regex::new(&expr)?;
    }
    Ok(v)
}

This doesn't compile with the pretty cryptic message (at first sight):

the trait `std::convert::From<regex::Error>` is not implemented for `std::io::Error`

When you have several error types coming from serde, parse etc this could be a nightmare with lots of compliing errors.

The solution is to implement the convertion methods and define an enum for storing all error types. You can also add the context by using the map_err method which converts all errors to your own defined error.

Using your own

Suppose you want to manage, for your CLI project, I/O, regex and serde error (you name it) in addition to your own. You'd like also to add some context, which is blatantly missing.

This is what I've done in one of my projects:

define a custom error enum:

#[derive(Debug, PartialEq)]
pub enum AppCustomErrorKind {
    SeekPosBeyondEof,
    UnsupportedPatternType,
    FileNotUsable,
    FilePathNotAbsolute,
    UnsupportedSearchOption,
    OsStringConversionError,
    PhantomCloneError,
}

define an enum with all possible errors:

#[derive(Debug)]
pub enum InternalError {
    Io(io::Error),
    Regex(regex::Error),
    Parse(num::ParseIntError),
    Yaml(serde_yaml::Error),
    Json(serde_json::Error),
    SystemTime(std::time::SystemTimeError),
    Utf8(std::str::Utf8Error),
    Custom(AppCustomErrorKind),
}

define the application error I used, adding a context:

#[derive(Debug)]
pub struct AppError {
    pub error_kind: InternalError,
    pub msg: String,
}

impl AppError {
    /// A simple and convenient creation of a new application error
    pub fn new_custom(kind: AppCustomErrorKind, msg: &str) -> Self {
        AppError {
            error_kind: InternalError::Custom(kind),
            msg: msg.to_string(),
        }
    }

    /// Convert from an internal error
    pub fn from_error<T: Into<InternalError>>(err: T, msg: &str) -> Self {
        AppError {
            error_kind: err.into(),
            msg: msg.to_string(),
        }
    }
}

implement fmt:Display for all internal errors
implement std::convert::From for all internal errors
define a convenient alias:

pub type AppResult<T> = Result<T, AppError>;

finally implement a context!() macro to bring the context to the error

You can browse the whole source here: https://github.com/dandyvica/clf/blob/master/src/misc/error.rs

Now, using the map_err() method on a Result, it's easy to add additional information:

let file = std::fs::File::open(&file_name).map_err(|e| context!(e, "unable to read configuration file: {:?}", &file_name))?;

A final word

The error management is sometimes left behind and tackled in the end of a project. My advice is to start thinking of it right from the beginning, because it often leads to a zillion of changes afterwards.

Hope this helps !

Photo by Sigmund on unsplah

How to use Latex for awesome letters

Dandy Vica — Tue, 13 Oct 2020 18:02:42 +0000

If you don't know Lateχ (pronounced Lay-teck, because the last letter is the Greek letter Chi χ), you definitely have to. For professional looking documents, letters, resumes, articles, it's one of the best document production system on earth. To get an idea of its power: https://en.wikipedia.org/wiki/LaTeX

It has nothing to do traditional WYSIWYG software. It more relates to a traditional compiler, where source code language is Lateχ and compilation output is generally a PDF file (but can also be a DVI or PostScript one). Lateχ is extensible using packages, for almost everything. It is very popular among scientists because of its rich and powerful science symbols library.

The number of Lateχ documents, tutorials, references etc is literally staggering. Just google them to get the full power of it. My goal is not to add another one, but rather help you to design a good-looking one page letter.

Installing Latex on Linux

$ sudo apt-get install texlive-full

This is a full blown installation, maybe bloated (around 5GB is installed), with all world languages and lots of packages you'll probably never use. But this is much simpler to deal with IMHO.

Choosing an editor

On Linux, TexStudio is my favorite. Intuitive and already configured, its fast and efficient. It sports syntax color highlighting, shortcuts for most common definitions, wizards, etc. Available for a host of distributions, just download the relevant package and install it: https://www.texstudio.org/

Designing a letter

Every Lateχ document is starting with a preamble:

% define a new A4 document with 12pt font size
\documentclass[a4paper,12pt]{article}

For english-spoken fellows, you can change a4paper with letterpaper which is the default page size.

After that, a list of packages needed to get the desired effects are loaded:

% define a new A4 document with 12pt font size
\documentclass[a4paper,12pt]{article}

% for mobile phone etc symbols
\usepackage{marvosym}

% set margins with no header and no footer
\usepackage[left=1.9cm,top=2cm,bottom=2.5cm,right=1.9cm,nohead,nofoot]{geometry}

% if fancy tables are needed
\usepackage{array}

% if colors are needed
\usepackage{color}

% Latex can't use TrueType fonts out of the box
% to use TrueType fonts, use this package. 
% Caveat: need to compile with XeLatex
\usepackage{fontspec}
\setmainfont{Linux Libertine O}

% no page numbers
\pagestyle{empty}

% for images and image files list of search paths
\usepackage{graphicx}
\graphicspath{{.}{../../../Photos/}}

% no paragraph identation (or change it to whatever is needed)
\setlength{\parindent}{0pt}

% space between paragraphs
\setlength{\parskip}{11pt}

% interline space
\linespread{1.125}

% PDF metadata into the final file
\usepackage[
            pdfauthor={Conan Doyle},
            pdftitle={Adventures of Sherlock Holmes}
            ]{hyperref}

Now it's to time to start typing your text, because up to now, all tags are only for Lateχ internal use. Everything should stick between those 2 Latex tags:

\begin{document}
\end{document}

Here is the whole text:

%% ------------------------------------------------------------
%% document start
%% ------------------------------------------------------------
\begin{document}

%% ------------------------------------------------------------
%% header table
%% ------------------------------------------------------------
\begin{tabular}{p{10cm} p{5cm}}

    \begin{minipage}[t]{0.40\textwidth}
        \begin{flushleft}
            {\Large \textbf{John H. Watson}}\\[10pt]
            221B Baker Street\\
            London\\[8pt]
            \Mobilefone ~ +41 06 00 00 00 00\\
            \Email ~ john.watson@outlook.com\\[20pt]
        \end{flushleft}
    \end{minipage}

    &

    \begin{minipage}[t]{0.40\textwidth}
        \begin{flushright}
            {\Large \textbf{Professor Moriarty}}\\[10pt]
            6649 N Blue Gum St\\
            London\\
        \end{flushright}
    \end{minipage}  

\end{tabular}



\begin{flushleft}
London, 1895

Object: A scandal in Bohemia
\end{flushleft}



\vspace{1.5cm}

%% ------------------------------------------------------------
%% Letter body
%% ------------------------------------------------------------
Dear Sir,
\vspace{1.5cm}

To Sherlock Holmes she is always the woman. I have seldom heard
him mention her under any other name. In his eyes she eclipses and
predominates the whole of her sex. It was not that he felt any emotion
akin to love for Irene Adler. All emotions, and that one particularly,
were abhorrent to his cold, precise, but admirably balanced mind. He
was, I take it, the most perfect reasoning and observing machine that
the world has seen; but, as a lover, he would have placed himself in a
false position. He never spoke of the softer passions, save with a gibe
and a sneer. 

They were admirable things for the observer—excellent
for drawing the veil from men’s motives and actions. But for the
trained reasoner to admit such intrusions into his own delicate and
finely adjusted temperament was to introduce a distracting factor which
might throw a doubt upon all his mental results. Grit in a sensitive
instrument, or a crack in one of his own high-power lenses, would not
be more disturbing than a strong emotion in a nature such as his. And
yet there was but one woman to him, and that woman was the late Irene
Adler, of dubious and questionable memory.

I had seen little of Holmes lately. My marriage had drifted us away
from each other. My own complete happiness, and the home-centred
interests which rise up around the man who first finds himself master
of his own establishment, were sufficient to absorb all my attention;
while Holmes, who loathed every form of society with his whole
Bohemian soul, remained in our lodgings in Baker Street, buried among
his old books, and alternating from week to week between cocaine and
ambition, the drowsiness of the drug, and the fierce energy of his
own keen nature.

\vspace{1cm}
\hspace{10cm} Sherlock Holmes

\end{document}

Several points to explain:

tables are not the most easy to read in Lateχ source code. This is quite verbose, and not really intuitive. But once you got the whole idea, this is much simpler
units for spacing are so numerous that there's a complete article on Wikipedia: https://en.wikibooks.org/wiki/LaTeX/Lengths
to separate 2 paragraphs, just add one blank line between 2 blocks of text
for text, just type your text without thinking of spacing, alignment, carriage return etc. Just type it, the Lateχ compiler will manage and calculate justification with correct word hyphenation (the way of word is cut to prevent overflow)
hspace and vspace are, as you already guessed, to keep horizontal or vertical spaces between text

Beware that to get the full power of TrueType fonts, you need to compile with XeTeχ. This is done by selecting XeLaTeχ in the combo list in Options -> Configure -> Production.

The result is a very neat, professional and good looking letter !

Hope this helps.

Photo by Andrew Buchanan on Unsplash

Use mitmproxy as a personal firewall

Dandy Vica — Sat, 12 Sep 2020 18:29:05 +0000

mitmproxy is a very versatile and powerful tool aiming at acting exactly like what its name implies: act as a man in the middle by deciphering TLS connections between a browser and a target website.

By defining its own Certificate Authority, and because you explicitly tells your browser to trust this brand new authority, you can allow mitmproxy to snoop the traffic between your browser and the remote website, even when this connection is made through TLS. Furthermore, using its powerful addon system, you can even tamper the stream of data and for example save resources automatically, deny some to be downloaded, modify headers, delete cookies etc. Those scripts are simple Python scripts, which use the dedicated mitmproxy Python API.

This gave me the idea to develop a straightforward method to implement a tiny personal firewall, to get rid of all those pesky ads resources, like those digital marketing services. Using one of the 3 mitmproxy commands, you can get an idea of how many of those resources are downloaded when you head to any website, as you can using the Chrome developer menus.

Beware that when started, mitmproxy can see and intercept all the https traffic between your browser and the remote site. You should therefore take all necessary security actions to prevent anyone other than you to use your configuration.

So let's elaborate a little bit.

First steps

first, download the mitmproxy executables for your platform. I personally use Linux, so I've just downloaded the archive and extracted the 3 executables into a dedicated directory. For a difference between those, go to https://docs.mitmproxy.org/stable/
next, start your proxy with no parameters for the moment:

./mitmdump

which by default is listening to the 8080 TCP port

start your Chrome browser and tells it to use mitmproxy:

google-chrome --proxy-server="http://127.0.0.1:8080"
head to http://mitm.it and follow the instructions. For Linux, I just clicked Other and the CA certifcate, generated by mitmproxy when it first runs, is downloaded:

mitmproxy-ca-cert.pem
now, type chrome://settings/certificates, click on Authorities, Import and select the CA file you've been downloading. The CA certificate is now installed.
restart Chrome with the same command line arguments, and voilà !

Note: I didn't try to install the CA file for other browsers, but it seems a little bit trickier.

Tampering data

Addons are simple Python scripts used to just get or even modify https traffic. A list of addon samples is located here: https://docs.mitmproxy.org/stable/addons-examples/

This sample addon is used to block resources, depending on an URL regex list. It can be enhanced for sure:

""" 
Block URLs matching a regex, by just returning an HTTP 404 code. As addons can be called with an argument,
the file containing the URLs is hardcoded, but could be extracted from an environment variable for example.

Unfortunately in Python, contrary to Rust, you can't define a regex set and try to match any regex for a string.

"""
import re
from mitmproxy import http
from mitmproxy import ctx

class BlockResource:
    def __init__(self):
        # define a new list for holding all compiled regexes. Compilation is done once when the addon
        # is loaded
        self.urls = []

        # read the configuration file having all string regexes
        for re_url in open('urls.txt'):
            self.urls.append(re.compile(re_url.strip()))

        # log how many URLS we have read
        ctx.log.info(f"{len(self.urls)} urls read")

    def response(self, flow):
        # test if the request URL is matching any of the regexes
        if any(re.search(url, flow.request.url) for url in self.urls):
            ctx.log.info(f"found match for {flow.request.url}")
            flow.response = http.HTTPResponse.make(404)

addons = [
    BlockResource()
]

And a sample for urls.txt:

scorecardresearch\.com
geo\.yahoo\.com
adversting\.
ads\.
\.taboola\.
\.doubleclick\.
\.xiti\.
criteo
\.pubmatic\.
\.chartbeat\.net
\.twimg\.com
\.keywee\.co
\.\w+adserver\.com
outbrain
\.bouncex\.

Now, just start mitmproxy with the -s parameter:

./mitmdump -s ./block-urls.py

For sure, for lots of URL regexes, this might be sub-optimal, because Python has to try to match all regexes up to the one which matches.

Kudos to the mitmproxy team for this awesome piece of code !

Hope this helps!

Photo by Blaz Erzetic on Unsplash

Linux commands for sysadmins (but not only, part 1)

Dandy Vica — Sun, 31 May 2020 15:15:05 +0000

Whether you're a Linux or an OS/X user, you often ends up starting a terminal and typing some shell commands.

Following is a list of some commands I often use, and they're by no means exhaustive. They are given as examples, and a second list will include other topics like benchmarking, network, or performance. Some require the root password or at least be part of the sudoers.

Be warned these are mainly Linux commands, used from a Linux Mint 19.3 machine. My advice is to man each one (when available) to list all the numerous possibilities for a specific command. man is always your friend and you can often discover the missing parameter you need to solve your problem.

Dealing with hardware

list your hardware configuration (the most comprehensive): inxi -Fxz
disks

what?	command
list all hard disks	lsblk -o NAME,TYPE,SIZE,MODEL,SERIAL,VENDOR -d
get information on a disk	sudo hdparm -i /dev/sda
list partitions	sudo sfdisk -l /dev/sda
mounted disks	df -T -x squashfs -x tmpfs -x devtmpfs
sanitize or partition	dcfldd if=/dev/urandom od=/dev/sdX
copy a disk or partition to a file image	dcfldd if=/dev/sdX of=image.img hash=sha1 conv=sync,noerror
list usb devices	lsusb
list PCI devices	lspci

Files

Listing files with ls

list	command
sort by size (largest first)	ls -lS
sort by time (newest first)	ls -lt
reverse sorting order	add the -r flag
recursive	ls -lR
sort by extension	ls -lX

Using the tree command

If not installed: sudo apt-get install tree

list	command
only directories	tree -d
full path with files	tree /usr -f
on the current filesystem only	tree . -x
only subdirectories	tree . -L 1
only files with pattern	tree . -P ".jpg"*
without indentation (like find)	tree . -i -f
output as a JSON object	tree . -J
output as an XML object	tree . -X

Finding files

find is a very feature rich command and not easy to master at first. man find is your friend!

find	command
files matching a pattern	find . -name '.jpg'*
files matching a pattern (case insensitive)	find . -iname '.jpg'*
files only on current mounted filesystems	find . -mount -name ".jpg"*
owned by userID	find . -user 1000
whose permissions are 644	find . -perm 644
greater than 50M	find / -size +50M
modified more than 1 day	find . -mtime +1
only directories	find . -type d
only (regular) files	find . -type f
executable files	find . -executable -type f
using a regex	find . -regextype posix-extended -regex "(..jpg

Open files

lsof is a very powerful and versatile command.

find	command
open files for a list of users	lsof -u johndoe,1000
open files by a list of processes	lsof -p 1999,2050
on a specific fiessystem	lsof /dev/sdb1

Photo by Barn Images on Unsplash

Different ways of reading files in Rust

Dandy Vica — Sun, 26 Apr 2020 16:37:57 +0000

This time, I'd like to share my experience on the different ways to read files in Rust. Different methods can be used, each one having its own drawbacks.

Reading ASCII files

Basically, there're 3 ways of reading ASCII files in Rust, and an additional possibly more harmful.

1.loading the entire file in a String.

This is done using the std::fs::read_to_string() method. If you're familiar with Python or Ruby, this method is as convenient as Python's read() function or Ruby's File.read() methods. Combined with the power of generics, you can easily build a vector of structs, matching the data type in a file:

// Loads an entire file of ip addresses as a Vector of Result<Ipv4Addr> structs
fn read_all<T: FromStr>(file_name: &str) -> Vec<Result<T, <T as FromStr>::Err>> {
    std::fs::read_to_string(file_name)
        .expect("file not found!")
        .lines()
        .map(|x| x.parse())
        .collect()
}

let addr = read_all::<Ipv4Addr>("ipv4.txt");

2.using the lines() iterator.

This is another easy method for reading a file line by line, using the lines() iterator. This iterator operates on a BufReader created from a File object. So a BufReader structure needs to be created for this to be used.

This example function calls a closure on each line:

// Calls *func()* on each line
fn read_iter(file_name: &str, func: fn(&str)) {
    let file = File::open(file_name).expect("file not found!");
    let reader = BufReader::new(file);

    for line in reader.lines() {
        func(&line.unwrap());
    }
}

This method is useful for small files but not really appropriate for very large files because each iteration incurs a String::new() allocation.

3.using the read_line() function.

The read_line() function can make use of the same String buffer, without reallocating on each iteration. But due to the way Rust iterators work, we can't build a standard iterator here. We have to use a mere loop construct, and stop it when the read_line() function returns Ok(0), which means EOF:

// Reuse the same String buffer
fn read_line(file_name: &str, func: fn(&str)) -> Result<(), std::io::Error> {
    // open target file
    let file = File::open(&file_name)?;

    // uses a reader buffer
    let mut reader = BufReader::new(file);
    let mut line = String::new();

    loop {
        match reader.read_line(&mut line) {
            Ok(bytes_read) => {
                // EOF: save last file address to restart from this address for next run
                if bytes_read == 0 {
                    break;
                }

                func(&line);

                // do not accumulate data
                line.clear();
            }
            Err(err) => {
                return Err(err);
            }
        };
    }

    Ok(())
}

Don't forget to clear the buffer after you've got the data, otherwise buffer will grow unexpectedly.

One can also use the split() iterator, which incurs the same drawback than lines(): it allocates a
Vec<u8> on each iteration.

// Reuse the same Vec<u8> buffer
fn read_split(file_name: &str, func: fn(&[u8])) -> Result<(), std::io::Error> {
    // open target file
    let file = File::open(&file_name)?;

    // uses a reader buffer
    let reader = BufReader::new(file);

    // use a for loop construct
    for line in reader.split(0x10) {
        func(&line?);
    }
    Ok(())
}

4.use mmap()api

For an explantion of this system call, have a look to https://en.wikipedia.org/wiki/Mmap.
As it's not included in the standard Rust library, you can use the memmap crate:

// Maps the file to memory
fn read_mmap(file_name: &str) -> Result<(), std::io::Error> {
    // open target file
    let file = File::open(&file_name)?;

    // create a memmap struct. After that, mmap variable maps directly file contents
    let mmap = unsafe { Mmap::map(&file)? };

    // use from_utf8() to convert to an UTF8 Rust string
    for s in mmap.split(|x| *x == 0x10) {
        println!("{:?}", std::str::from_utf8(&s).unwrap());
    };

    Ok(())
}

Beware this is not a foolproof process, as if the file is changed, you can could get a SIGBUS error.

Reading binary files

Reading a binary file is not really different from an ASCII file. But you should be aware of any endianess issues, and use the byteorder crate, although not really related to the Rust read methods per se.

This is an example of a PNG file header reading, to get image width & height:

// Read PNG file image width and height
fn read_png(file_name: &str) -> Result<(), std::io::Error> {
    const BUFFER_SIZE: usize = 256;

    // open target file
    let mut file = File::open(&file_name)?;

    // we'll use this buffer to get data
    let mut buffer = [0; BUFFER_SIZE];

    // reader PNG header of 8 bytes
    let _ = file.by_ref().take(8).read(&mut buffer)?;
    assert_eq!(&buffer[1..4], "PNG".as_bytes());

    // read IHDR chunk
    let chunk_size = file.read_u32::<BigEndian>().unwrap();
    let _ = file.by_ref().take(4).read(&mut buffer)?;
    assert_eq!(&buffer[0..4], "IHDR".as_bytes());

    let image_width = file.read_u32::<BigEndian>().unwrap();
    let image_height = file.read_u32::<BigEndian>().unwrap();
    println!("image is W={} x H={}", image_width, image_height);

    Ok(())
}

Hope this helps !

Photo by Maarten van den Heuvel on Unsplash

Rust standard iterators

Dandy Vica — Wed, 11 Mar 2020 21:00:49 +0000

Today, I'll tackle Rust standard iterators, which can be browsed here: Rust Iterators

In you ever read this page, you've probably stumbled upon a list of methods along with some examples, but even after reading this full page, you may have wondered: how to use all these methods ?

Because the examples are based, for the most part, on a vector of integers (which by the way implement the Copy trait), it's pretty easy to use with such an iterable. But once you have a more complicated iterator like a vector of structs, this might be more tedious to use.

That's why I wrote this article.

As the base for my examples, I've use the Mendeleiev periodic list of elements available as a CSV file here: https://gist.github.com/GoodmanSciences/c2dd862cd38f21b0ad36b8f96b4bf1ee (beware to fix lines which contain spaces betweens fields, as Rust might crash when loading the CSV).

I've create a simple function to load the data into a struct, whose members map (snake case though) the CSV column names:

pub fn load_as_vector() -> Result<Vec<Element>, Box<dyn Error>> {
    // load CSV
    let reader = std::fs::File::open("elements.csv").unwrap();
    let mut rdr = csv::Reader::from_reader(reader);

    // create a vector of structs
    let mut v: Vec<Element> = Vec::new();

    for result in rdr.deserialize() {
        let record: Element = result?;
        v.push(record);
    }

    Ok(v)
}

The vector is loaded using this dedicated method (you can find the whole code here: https://github.com/dandyvica/articles/tree/master/combinators)

let v = element::load_as_vector()?;

I didn't respect the alphabetic order of all the methods, I rather adopted an incremental approach, with the simplest ones in the beginning of the article. I tried to express what the method is representing for the vector data, instead of writing its definition which you can get anyway. I also unwrap() the results when possible, because a lot of iterators methods usually return an Option type.

I also didn't cover all the methods, by lack of time. If you get ideas on how to add other examples for this missing methods, feel free to reach out ! In addition, the following examples are probably not optimized and some better combination of iterators should exist.

Beware I'm by no means a chemical engineer, I used this material just to illustrate my article.

Iterator methods

count(): there're 118 elements in the Mendeleiev table

let n = v.iter().count();
assert_eq!(n, 118);

last(): Oganesson is the last element

let last_element = v.iter().last().unwrap();
assert_eq!(last_element.element, "Oganesson");

nth(): Uranium is the 92th element (vector is indexing from 0), but there's not a 119th element

let uranium = v.iter().nth(91).unwrap();
assert_eq!(uranium.element, "Uranium");
assert!(v.iter().nth(118).is_none());

map():: Carbon is the 6th element

let mut mapped = v
    .iter()
    .map(|x| (x.atomic_number, x.element.as_ref(), x.symbol.as_ref()));
assert_eq!(mapped.nth(5).unwrap(), (6u8, "Carbon", "C"));

collect(): create a vector of elements' names

let names: Vec<_> = v.iter().map(|x| &x.element).collect();
assert_eq!(names[0..2], ["Hydrogen", "Helium"]);

take(): the 2 first elements are Hydrogen and Helium

let first_2: Vec<_> = v.iter().take(2).map(|x| x.element.clone()).collect();
assert_eq!(first_2, ["Hydrogen", "Helium"]);

take_while(): there're 10 elements with less than 10 neutrons

let less_than_10e = v.iter().take_while(|x| x.number_of_neutrons <= 10);
assert_eq!(less_than_10e.count(), 10);

any(): there's at least one element with more than 50 electrons

assert!(v.iter().any(|x| x.number_of_neutrons > 50));

all(): all elements have their symbol composed by 1 or 2 letters (e.g.: C or Na)

assert!(v.iter().all(|x| x.symbol.len() == 1 || x.symbol.len() == 2));

cycle(): when cycling through elements, Lithium is the 120th element

assert_eq!(v.iter().cycle().nth(120).unwrap().element, "Lithium");

find(): Helium is the first element whose name ends with ium

let helium = v.iter().find(|x| x.element.ends_with("ium")).unwrap();
assert_eq!(helium.element, "Helium");

filter() and for_each(): there're 11 gases

let gases: Vec<_> = v.iter().filter(|x| x.phase == "gas").collect();
assert_eq!(gases.iter().count(), 11);
v.iter()
    .filter(|x| x.phase == "gas")
    .for_each(|x| println!("{:?}", x.element));
// gives:
// "Hydrogen"
// "Helium"
// "Nitrogen"
// "Oxygen"
// "Fluorine"
// "Neon"
// "Chlorine"
// "Argon"
// "Krypton"
// "Xenon"
// "Radon"

filter_map(): there're 37 radioactive elements

let radioactives: Vec<_> = v
    .iter()
    .filter_map(|x| x.radioactive.as_ref())
    .filter(|x| **x == YN::yes)
    .collect();
assert_eq!(radioactives.iter().count(), 37);

enumerate(): the last element index is 117

let (i, _) = v.iter().enumerate().last().unwrap();
assert_eq!(i, 117);

skip_while(): the first non-gas is Lithium

let first_non_gas = v.iter().skip_while(|x| x.phase == "gas" ).next().unwrap();
assert_eq!(first_non_gas.element, "Lithium");

zip(): Uranium mass number is 238

let neutrons = v.iter().map(|x| x.number_of_neutrons);
let protons = v.iter().map(|x| x.number_of_protons);
let mass_numbers: Vec<_> = neutrons.zip(protons).map(|(x, y)| x + y).collect();
assert_eq!(mass_numbers[91], 238);

chain(): when listing gases and solids, Lithium is the first element, Radon the last

let all_gases = v.iter().filter(|x| x.phase == "gas");
let all_solids = v.iter().filter(|x| x.phase == "solid");
let gases_and_solids: Vec<_> = all_solids.chain(all_gases).collect();
assert_eq!(gases_and_solids.iter().nth(0).unwrap().element, "Lithium");
assert_eq!(gases_and_solids.iter().last().unwrap().element, "Radon");

position(): searches for the Potassium element

let potassium = v.iter().position(|x| x.element == "Potassium").unwrap();
assert_eq!(v[potassium].symbol, "K");

rposition(): Radon is the last gas

let last_gas = v.iter().rposition(|x| x.phase == "gas").unwrap();
assert_eq!(v[last_gas].element, "Radon");

max_by(): the heaviest non-artificial element is Uranium

use std::cmp::Ordering;
let cmp = |x: &Element, y: &Element| -> Ordering {
    if x.atomic_mass < y.atomic_mass {
        Ordering::Less
    } else if x.atomic_mass > y.atomic_mass {
        Ordering::Greater
    } else {
        Ordering::Equal
    }
};

let heaviest = v
    .iter()
    .filter(|x| x.phase != "artificial")
    .max_by(|x, y| cmp(x, y))
    .unwrap();
assert_eq!(heaviest.symbol, "U");

rev(): the last element when reversing the vector is Hydrogen

let hydrogen = v.iter().rev().last().unwrap();
assert_eq!(hydrogen.symbol, "H");

max_by_key(): the longuest element's name is Rutherfordium

let longuest = v.iter().max_by_key(|x| x.element.len()).unwrap();
assert_eq!(longuest.element, "Rutherfordium");

max(): Carbon was the first element discovered, Tennessine the last

//use std::cmp::Ordering;

impl Ord for Element {
    fn cmp(&self, other: &Self) -> Ordering {
        if self.year < other.year {
            Ordering::Less
        } else if self.year > other.year {
            Ordering::Greater
        } else {
            Ordering::Equal
        }
    }
}

impl PartialOrd for Element {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        Some(self.cmp(other))
    }
}

impl PartialEq for Element {
    fn eq(&self, other: &Self) -> bool {
        self.year == other.year
    }
}

impl Eq for Element {}

let first_discovered = v.iter().min().unwrap();
assert_eq!(first_discovered.element, "Carbon");
let last_discovered = v.iter().max().unwrap();
assert_eq!(last_discovered.element, "Tennessine");

Hope this helps ! Feel free to comment.

Photo by Bill Oxford on Unsplash

Understanding Rust modules

Dandy Vica — Sun, 02 Feb 2020 19:20:03 +0000

If you ever fought against Rust modules, you probably already googled for terms such as "rust modules" or "rust modules structure". And you probably stumbled upon the Rust book. But after reading the articles, you're maybe in a situation were you still don't grab the whole idea, and according to the new article called Path Clarity:

... why it's so confusing to many: while there are simple and consistent rules defining the module system, their consequences can feel inconsistent, counterintuitive and mysterious.

For modules defined inside a source .rs file, that's OK. But if you're a seasoned developer, you're used to break down your code into different source files.

I wrote this article, with a simple example, to better understand the Rust module system. I'm referring to the last version of Rust called Rust 2018.

So let's create a simple project structure using cargo, with a hierarchical structure

$ cargo new rust_modules --bin

Let's create a simple module hierarchical structure based on maths:

// main.rs
mod math {
    pub mod arithmetic {
        pub fn add(x: i32, y: i32) -> i32 {
            x + y
        }
        pub fn mul(x: i32, y: i32) -> i32 {
            x * y
        }
    }

    pub mod trigonometry {
        pub mod ordinary {
            pub fn sin(x: f32) -> f32 {
                x.sin()
            }
            pub fn cos(x: f32) -> f32 {
                x.cos()
            }
        }

        pub mod hyperbolic {
            pub fn sinh(x: f32) -> f32 {
                (x.exp() - (-x).exp()) / 2f32
            }
            pub fn cosh(x: f32) -> f32 {
                (x.exp() + (-x).exp()) / 2f32
            }
        }
    }
}

fn main() {
    use math::arithmetic::{add, mul};
    use math::trigonometry::hyperbolic::{cosh, sinh};
    use math::trigonometry::ordinary::{cos, sin};
}

This code compiles because it's located in the same source file. Things are getting more complicated if you want to split the module into different source files:

// src/math.rs or src/math/mod.rs
pub mod arithmetic {
    pub fn add(x: i32, y: i32) -> i32 {
        x + y
    }
    pub fn mul(x: i32, y: i32) -> i32 {
        x * y
    }
}

pub mod trigonometry {
    pub mod ordinary {
        pub fn sin(x: f32) -> f32 {
            x.sin()
        }
        pub fn cos(x: f32) -> f32 {
            x.cos()
        }
    }

    pub mod hyperbolic {
        pub fn sinh(x: f32) -> f32 {
            (x.exp() - (-x).exp()) / 2f32
        }
        pub fn cosh(x: f32) -> f32 {
            (x.exp() + (-x).exp()) / 2f32
        }
    }
}

You can put the internal of the math module into a single file in src/math.rs (filename has the same name of the module without extension) or in src/math/mod.rs. Then the
main.rs becomes:

// math module internals is either in src/math.rs or src/math/mod.rs
mod math;

fn main() {
    use math::arithmetic::{add, mul};
    use math::trigonometry::hyperbolic::{cosh, sinh};
    use math::trigonometry::ordinary::{cos, sin};
}

You can even use the new path attribute to move your source code wherever you want:

// the source code containing the math module is moved to /tmp/math.rs
#[path="/tmp/math.rs"]
mod math;

fn main() {
    use math::arithmetic::{add, mul};
    use math::trigonometry::hyperbolic::{cosh, sinh};
    use math::trigonometry::ordinary::{cos, sin};
}

So using the mod.rs technique, you could breakdown the math module into several source files:

├── main.rs
└── math
    ├── arithmetic.rs
    ├── mod.rs
    └── trigonometry
        ├── hyperbolic.rs
        ├── mod.rs
        └── ordinary.rs

with the following mod.rs files:

// src/math/mod.rs
pub mod arithmetic;
pub mod trigonometry;

// src/math/trigonometry/mod.rs
pub mod hyperbolic;
pub mod ordinary;

I should say this is not the most straightforward module strategy, but once you get acquainted to it, it feels more natural. So the key takeaways are:

the mod & use keywords are like import or require in popular languages
a module named mymod reflects either a filename similar to mymod.rs or mymod/mod.rs

Photo by Timo Volz on Unsplash

A remote program execution in Go

Dandy Vica — Tue, 17 Dec 2019 20:47:54 +0000

This time, I'll be tackling the Go programming language. Even though its syntax is not the most inspiring and it's missing some modern features like generics (but this is well requested by the Go community and it'll be probably part of the Go 2 umbrella project), I was ready to give a try. Being backed by Google is really a competitive advantage.

I was curious of how long it takes to write a decent WEB API project, as a complete beginner in Go. Of course being proficient at C/C++ and intermediate in Rust and D, helps.

The project

As I'm currently dabbling in Nagios those days, I discovered how the Nagios Remote Plugin Execution works. But also how terse the source code was.

So I decided to experiment a Remote Execution Program in Go, by developing a REST simple WEB API mock-up. The interesting fact in Go resides in its speed and in the fact it's generating a standalone executable, which is a key element when distributing your app.

The general idea is this: execute a remote program or script in the remote server running the app, by sending a HTTP GET command with a path and optionally a query string. It's easy to test using wget or even postman. On the server side, the list of available commands is read from a TOML file and result is delivered to the client as a JSON string.

Of course, this could be done using Python Flask or Ruby Sinatra but the point here was to experiment Go and see how far I can land.

The server side

Basically, the server app will:

read the TOML file and marshal its data into a Go structure
verify if commands defined in the TOML file are found locally
define a route to handle GET request
start the WEB server on port specified in the TOML file
whenever a GET request is received, exec and return output data to the client, along with the command exit code

Managing TOML data

First of all, the TOML file could look like this:

# RPE configuration file sample
[general]
logfile = "/tmp/rpe.log"
var_prefix = "$"

# network specifics
[network]
address = "127.0.0.1"
port = 8080

[[command]]
name = "list_files"
path = "/bin/ls"
args = ["-l", "$DIR"]

[[command]]
name = "find_files"
path = "/usr/bin/find"
args = ["$ROOT", "-name", "$PATTERN"]

The [[command]] section defines the command path and filename, its arguments with potential variable arguments and a name to refer to. That name will be used as the GET route, and its query string as variable arguments. The Go TOML package is used to marshal data to a Go structure, with field names mapping TOML data and sections:

// Global configuration
type tomlConfig struct {
    General generalConfig
    Network networkConfig
    Command []commandDetails
}

// General section 
type generalConfig struct {
    Logfile string
    Var_Prefix string
}

// Topic/Item configuration
type networkConfig struct {
    Port    int16
    Address string
}

// Commands as defined in the TOML file
type commandDetails struct {
    Name    string
    Path    string
    Args    []string
}

Marshalling TOML data is straightforward:

// now marshall toml data
if err := config.loadConfig(tomlFileName); err != nil {
    fmt.Println(err)
    os.Exit(2)
}

WEB framework

It seems mux is the de facto standard as a HTTP WEB framework. It works similarly to what's usual in Flask or Sinatra, binding routes to method handlers:

// http loop and receive commands
router := mux.NewRouter()

// only handle GET requests
router.HandleFunc("/{command}", CommandHandler).Methods("GET")

// build address:port and listen HTTP incoming requests
addr := fmt.Sprintf("%s:%d", config.Network.Address, config.Network.Port)
http.ListenAndServe(addr, router)

It merely binds the CommandHandler Go function, only for GET requests and for the variable URL path {command}. The handler is then responsible for getting command characteristics from the TOML configuration, and if found, it spawns the command and return the results as JSON.

Using the app

Just use wget like this:

$ wget -O - http://127.0.0.1:8080/find_files\?ROOT\=/home\&PATTERN\=\*.go 2>/dev/null

which gives a list of *.go files from the /home directory.

Possible refinements

It's easy to add other security features:

use https
use authentication
use host filter to allow connexion only from specific hosts
implement a timer to kill a lengthy command
write unit tests

Conclusion

I was really surprised at how fast I learned Go. Being a C expert, it took only 2 days googling Go documentation and stackoverflow posts to be productive.

Of course, it's not as powerful as Rust, because it lacks modern features like generics, sum types or macros. And I found some constructs somewhat weird (like initializing a map) and error management is rather old-style.

Being a garbage collected language, it's not meant to be a true system's programming language, but rather targeted at devops. I think it has a great potential and a bright future when Go 2 will be rolled out with generics and additional bells and whistles. Its standard library is large and covers lots of features.

You can get the whole source code at: https://github.com/dandyvica/rpe

Hope this helps !

Photo by Rowan Heuvel on Unsplash