DEV Community: Paweł bbkr Pabian

How to omit enum name in variant

Paweł bbkr Pabian — Tue, 12 Aug 2025 08:55:36 +0000

In my Rust journey I still discover some less known gems. Today's hero is use. In most programming languages it is responsible both for reaching to some code and bringing it into the scope. In Rust this functionality is divided between mod and use - mod makes something visible, use shortens the call by creating local bindings.

That brings us to enums. A lot of my code looked like this:

enum SomeLongAndDescriptiveName {
    Foo,
    Bar
}

fn main() {
    let a = SomeLongAndDescriptiveName::Foo;
    match a {
        SomeLongAndDescriptiveName::Foo => (),
        SomeLongAndDescriptiveName::Bar => (),
    }
}

In order to shorten it use can be used to locally bind enum members:

enum SomeLongAndDescriptiveName {
    Foo,
    Bar
}
use SomeLongAndDescriptiveName::*;

fn main() {
    let a = Foo;
    match a {
        Foo => (),
        Bar => (),
    }
}

That instantaneously made my code much more readable. Especially when using alternatives Foo|Bar syntax in match or joined if let statements from Rust 1.88.

Note 1: When using enum from some other module if you need both enum type and enum variants it looks like this:

mod other;
use other::SomeLongAndDescriptiveName::{self, *};

fn flip ( x: SomeLongAndDescriptiveName ) -> SomeLongAndDescriptiveName { // enum type
    match x {
        Foo => Bar,    // enum variant
        Bar => Foo
    }
}

Note 2: You can rename everything if needed, including variants:

mod other;
use other::SomeLongAndDescriptiveName::{self as E, Foo as F, Bar as B};

fn flip ( x: E ) -> E {
    match x {
        F => B,
        B => F
    }
}

So if you come to Rust from some other language with "classic" use semantic don't be afraid to use use more frequently for convenience/readability binding.

Guard state transitions with proto methods

Paweł bbkr Pabian — Mon, 10 Mar 2025 20:04:47 +0000

Problem

When implementing stateful object it is very important to enforce proper transition between states. Take a look at this happy-path code:

class File {

    has IO::Path $.path is required;
    has IO::Handle $!handle;

    method open () {
        $!handle = $!path.open();
        say 'opened';
    }

    method lines-count ( --> Int ) {
        return $!handle.lines.Int;
    }

    method close () {
        $!handle.close();
        $!handle = Nil;
        say 'closed';
    }

}

my $file = File.new( path => '/var/log/messages'.IO );
$file.open();
say $file.lines-count;
$file.close();

We implemented some abstraction over file, that requires its methods to be called in certain order. But what if someone misuses our class and for example calls close() without prior open()?

my $file = File.new( path => '/var/log/messages'.IO );
$file.close();

Invocant of method 'close' must be an object instance of type
'IO::Handle', not a type object of type 'IO::Handle'.  Did you forget a
'.new'?

User gets very confusing error message.

Naive fix

First thing that comes to mind to improve user experience is to inject some state guards with descriptive messages:

class File {

    method open () {
        die 'Cannot call open if already opened' if $!handle.defined;
        ...
    }

    method lines-count () {
        die 'Cannot count lines if not opened' unless $!handle.defined;
        ...
    }

    method close () {
        die 'Cannot call close if not opened' unless $!handle.defined;
        ...
    }
}

There is nothing wrong with this approach. Go for it if your code is simple and call it a day.

Troubles ahead

But few months later you may need to extend this code and implement remote functionality over your File abstraction that should transparently download it before original open() and remove after original close().

class RemoteFile is File {

    has $.url;

    method open () {
        HTTP::Tiny::mirror( $.url, $.path );
        say 'downloaded';
        callsame();
    }

    method close () {
        callsame();
        $.path.unlink();
        say 'removed';
    }

}

my $file = RemoteFile.new(
    path => '/tmp/example.html'.IO,
    url => 'http://example.com'
);
$file.open();
say $file.lines-count;
$file.close();

Suddenly you are back at square one. You need to inject another state guard in your top open() method, because double open() misuse won't be detected early enough.

my $file = RemoteFile.new(
    path => '/var/log/messages'.IO,
    url => 'http://example.com'
);
$file.open();
$file.open();

Will download file twice before detecting misuse in parent class.

downloaded
opened
downloaded
Cannot call open if already opened

Proto methods for the rescue!

Proto methods are used to "formally declare signature commonalities between multi candidates". But less known fact is that they can have body on their own that will be executed before their candidates.

enum State <Opened Closed>;

role FileStates {
    has State $!state;

    proto method open () {
        die 'Cannot call open if already opened' if $!state ~~ State::Opened;
        {*}
        $!state = State::Opened;
    }

    proto method lines-count ( --> Int ) {
        die 'Cannot count lines if not opened' unless $!state ~~ State::Opened;
        {*}
    }

    proto method close () {
        die 'Cannot call close if already closed' if $!state ~~ State::Closed;
        die 'Cannot call close if not opened' unless $!state ~~ State::Opened;
        {*}
        $!state = State::Closed;
    }

}

Here we defined proto methods to encapsulate state guards in a Role. Each proto method checks if it was called when object instance was in in expected State, calls its multi candidate using {*} placeholder and optionally saves new State afterwards.

When composed in base File class it reduces line noise because we no longer have to implement state guards scattered across every method. All we have to do to take advantage of proto declarations is to change our methods to multi variant ones:

class File does FileStates { # compose Role here

    has IO::Path $.path is required;
    has IO::Handle $!handle;

    multi method open () { # change method to multivariant
        $!handle = $!path.open();
        say 'opened';
    }

    multi method lines-count ( --> Int ) {
        return $!handle.lines.Int;
    }

    multi method close () {
        $!handle.close();
        $!handle = Nil;
        say 'closed';
    }

}

my $file = File.new( path => '/var/log/messages'.IO );
$file.open();
$file.open(); # test bad method call order

opened
Cannot call open if already opened

Yay! Clean and tidy.

Child classes are automatically protected by proto method in parent class. All we have to do is to also change them to multi variant ones:

class RemoteFile is File {

    has $.url;

    multi method open () { # change method to multivariant
        HTTP::Tiny::mirror( $.url, $.path );
        say 'downloaded';
        callsame();
    }

    ...
}

my $file = RemoteFile.new(
    path => '/var/log/messages'.IO,
    url => 'http://example.com'
);
$file.open();
$file.open(); # test bad method call order

downloaded
opened
Cannot call open if already opened

Misuse was detected immediately by proto, without second download.

It's a win-win

By using proto methods we implemented isolated, easy to read and reusable state transition guards. We can go further and define/visualize our state machine using Graph module for more complex cases.

I recommend also looking at Event Sourcing article if your state changes chain needs to be traced as immutable log of events.

Can we implement state guards to detect misuse in compile time?

To be honest I don't think it is possible in Raku. It can be done in Rust using it's ownership transfers and borrow checker, but that is subject for another blog post :)

SSH port forwarding from within Rust code

Paweł bbkr Pabian — Mon, 06 Jan 2025 20:54:09 +0000

Reminder

In this series I demonstrate how to create on demand SSH tunnel to connect to service in another network directly inside your code. I will be referring to theory described in the first post of the series, so please make sure you read it first.

Preparation

We will be using tokio async framework and russh crates. Add them to your project:

cargo add tokio --features=full
cargo add russh
cargo add russh_keys
cargo add async_trait

cargo build

If you get compilation error

error: could not compile `p521` (lib)
thread 'opt cgu.0' has overflowed its stack

then you should increase stack size during build, here is how to increase it from default 2MB to 16MB:

RUST_MIN_STACK=16777216 cargo build

No abstractions or error handling

I want this code to be transparent in Steve Mould style. That means I will purposely use instant failures everywhere and no abstractions to focus only on working principle.

Boilerplate

Let's start by setting necessary imports and variables.

use std::sync::Arc;
use tokio::net::TcpListener;
use russh_keys::{self, ssh_key, key::PrivateKeyWithHashAlg};
use russh::client;
use async_trait::async_trait;

#[tokio::main]
async fn main() {

    let service_host: &str = "service";
    let service_port: u16 = 7878;
    let jump_host: &str = "jump";
    let jump_port: u16 = 22;
    let jump_user: &str = "me";
    let jump_private_key_file: &str = "/home/me/.ssh/jump";
    let jump_private_key_password = Some("s3cret!");
    let local_host: &str = "127.0.0.1";
    let local_port: u16 = 8080;

As mentioned in previous post - jump_private_key_password should never be stored in code directly, this is for demonstration purposes only. You can have passwordless SSH key, in such case set it to None.

Connect to jump host

Russh does not provide SSH client directly, instead it provides russh::client::Handler trait that we must implement in our own struct.

struct Client {}

#[async_trait]
impl client::Handler for Client {
    type Error = russh::Error;
    async fn check_server_key (&mut self, _server_public_key: &ssh_key::PublicKey) -> Result<bool, Self::Error> {
        Ok(true)
    }
}

let ssh_client = Client{};

By default russh client will not connect to any host. Implementation of host authenticity check through check_server_key method is mandatory part of using this module, which may be tricky for newcomers. Jump host key will be available as instance of russh_keys::key::PublicKey and you are expected to return Ok(true) for successful verification or Ok(false) if you do not want to proceed.

Our SSH connection needs config. Default values are sane, so you can go with that:

let ssh_config = client::Config::default();

But if you want to replace some of them remember that you can still use remaining default values like so:

let ssh_config = client::Config {
    inactivity_timeout: Some(Duration::from_secs(5)),
    ..<_>::default()
};

Because russh assumes config is meant to be passed to multiple clients on different threads we also must wrap it in Arc thingy:

let ssh_config = Arc::new(ssh_config);

Third component required to connect is jump private key:

let jump_private_key = russh_keys::load_secret_key(
    jump_private_key_file, jump_private_key_password
).expect("Cannot load SSH private key");

let jump_private_key = PrivateKeyWithHashAlg::new(
    Arc::new(jump_private_key), None
).expect("Cannot determine SSH key algorithm");

Note that this changed in v0.49.0 version of russh. Previously key and hashing algorithm were in one instance of key::PrivateKey, now key itself must be decorated with hashing algorithm separately.

Phew! After implementing client::Handler trait, setting up SSH config using client::Config struct and creating instance of private key we are ready to connect to jump host:

let mut ssh_session = client::connect(
    ssh_config,
    (jump_host, jump_port),
    ssh_client
).await.expect("Cannot connect to SSH host");

ssh_session.authenticate_publickey(
    jump_user, ssh_key_secret
).await.expect("Cannot authenticate using SSH key");

Note that connect method requires tokio::net::ToSocketAddrs as second parameter, so it must be one tuple of two parameters instead of two standalone parameters.

Open local listener

let local_listener = TcpListener::bind(
    (local_host, local_port)
).await.expect("Cannot bind local port");

Bind method also requires tokio::net::ToSocketAddrs as second parameter, so it must be one tuple of two parameters instead of two standalone parameters.

There is one cool trick here - if you do not know free local port up front you can provide 0 and system will assign one for you.

let local_listener = TcpListener::bind(
    (local_host, 0)
).await.expect("Cannot bind local port");
let local_port = local_listener.local_addr().unwrap().port();

Open channel from jump to service

tokio::spawn(
        async move {
            let (mut local_socket, _) = local_listener.accept().await.expect("Cannot process local client");

            let ssh_channel = ssh_session.channel_open_direct_tcpip(
                service_host, service_port as u32,
                local_host, local_port as u32
            ).await.expect("Cannot open SSH forwarding channel");
            let mut ssh_stream = ssh_channel.into_stream();
            ...
        }
);

When we receive local connection we must ask jump host to create SSH channel that will pass TCP packets to service host. But we cannot wait for local connection in blocking manner so we spawn tokio thread for that purpose. This is simplified example that will wait for one connection and therefore allow us to use SSH forwarding once.

Note that channel_open_direct_tcpip does not use tokio::net::ToSocketAddrs but separate parameters. It is because ToSocketAddrs trait is for resolvable addresses while from local machine perspective service_host is not resolvable. Also port numbers must be of u32 type.

We immediately consume the channel to produce a bidirectionnal stream capable of sending and receiving ChannelMsg::Data as AsyncRead + AsyncWrite.

Create bidirectional data flow between local connection and SSH channel

tokio::spawn(
        async move {
            let (mut local_socket, _) = ...
            let mut ssh_stream = ...

            tokio::io::copy_bidirectional(
                &mut local_socket, &mut ssh_stream
            ).await.expect("Copy error between local socket and SSH stream");
        }
);

Last part is the easiest one due to copy_bidirectional method provided by tokio framework, which can automatically couple two ends as long as both implement AsyncRead + AsyncWrite traits.

Done

To recap here is whole code with marked spot where you can implement your own logic that requires connection to inaccessible service host but can now be achieved by connecting to local host that forwards it to service host through jump host.

use russh::client;
use russh_keys::{self, ssh_key, key::PrivateKeyWithHashAlg};
use tokio::net::TcpListener;
use std::sync::Arc;
use async_trait::async_trait;


#[tokio::main]
async fn main() {

    let service_host: &str = "service";
    let service_port: u16 = 7878;
    let jump_host: &str = "jump";
    let jump_port: u16 = 22;
    let jump_user: &str = "me";
    let jump_private_key_file: &str = "/home/me/.ssh/jump";
    let jump_private_key_password = Some("s3cret!");
    let local_host: &str = "127.0.0.1";
    let local_port: u16 = 8080;

    struct Client {}
    #[async_trait]
    impl client::Handler for Client {

        type Error = russh::Error;

        async fn check_server_key (&mut self, _server_public_key: &ssh_key::PublicKey) -> Result<bool, Self::Error> {
            Ok(true)
        }

    }
    let ssh_client = Client{};

    let ssh_config = client::Config::default();
    let ssh_config = Arc::new(ssh_config);

    let jump_private_key = russh_keys::load_secret_key(
        jump_private_key_file, jump_private_key_password
    ).expect("Cannot load SSH private key");
    let jump_private_key = PrivateKeyWithHashAlg::new(
        Arc::new(jump_private_key) ,None
    ).expect("Cannot determine SSH key algorithm");

    let mut ssh_session = client::connect(
        ssh_config, (jump_host, jump_port), ssh_client
    ).await.expect("Cannot connect to SSH host");

    ssh_session.authenticate_publickey(
        jump_user, jump_private_key
    ).await.expect("Cannot authenticate using SSH key");

    let local_listener = TcpListener::bind(
        (local_host, local_port)
    ).await.expect("Cannot bind local port");

    tokio::spawn(
        async move {
            let (mut local_socket, _) = local_listener.accept().await.expect("Cannot process local client");

            let ssh_channel = ssh_session.channel_open_direct_tcpip(
                service_host, service_port as u32,
                local_host, local_port as u32
            ).await.expect("Cannot open SSH forwarding channel");
            let mut ssh_stream = ssh_channel.into_stream();

            tokio::io::copy_bidirectional(
                &mut local_socket, &mut ssh_stream
            ).await.expect("Copy error between local socket and SSH stream");
        }
    );

    println!(
        "Connect to {}:{} to actually connect to {}:{}.",
        local_host, local_port,
        service_host, service_port
    );

    // your logic that requires service host goes here

}

At this point you should understand mechanics behind ssh forwarding. As I mentioned before my spaghetti code was oversimplified on purpose to focus on working principle, however for production use I strongly recommend creating Session abstraction as in russh examples section.

I'm still pretty new to Rust so if you think something could/should be better implemented then let me know in the comments.

SSH port forwarding from within Raku code

Paweł bbkr Pabian — Mon, 06 Jan 2025 20:53:31 +0000

Reminder

In this series I demonstrate how to create on demand SSH tunnel to connect to service in another network directly inside your code. I will be referring to theory described in the previous post, so please make sure you read it first.

Preparation

We will be using SSH::LibSSH module that connects to system libssh C library through built-in NativeCall mechanism. Install Raku module by invoking zef install SSH::LibSSH. As for libssh it is very likely you have it already installed if you use any major Linux distribution.

Boilerplate

Let's start by including required module and setting necessary variables.

use SSH::LibSSH;

my Str $service-host = 'service';
my Int $service-port = 7878;
my Str $jump-host = 'jump';
my Int $jump-port = 22;
my Str $jump-user = 'me';
my Str $jump-private-key-file = '/home/me/.ssh/jump';
my Str $jump-private-key-password = 's3cret!';
my Str $local-host = '127.0.0.1';
my Int $local-port = 8080;

Note is that $service-host is not resolvable or reachable from local network but must be resolvable and reachable from jump host.

Raku is gradually typed, so you can skip type annotations if you like but they are super useful to detect errors in compile time.

As mentioned in previous post - $jump-private-key-password should never be stored in code directly, this is for demonstration purposes only. You can have passwordless SSH key, in such case just skip all key-related variables from following code.

Connect to jump host

my $jump-connection = await SSH::LibSSH.connect(
    host => $jump-host, port => $jump-port, timeout => 4,
    user => $jump-user,
    private-key-file => $jump-private-key-file,
    private-key-file-password => $jump-private-key-password
);

To confirm jump host authenticity you can provide 3 additional parameters: on-server-unknown, on-server-known-changed and on-server-found-other. Their value should be a subroutine accepting one $handler parameter. Jump host hash will be available at $handler.hash and you have $handler.accept-this-time() or $handler.decline() methods to decide if you want to proceed. Default implementations are provided.

Open local listener

There is one cool trick here - if you do not know free local port up front you can provide 0 and system will assign one for you.

my $local-server = IO::Socket::Async.listen($local-host, $local-port);
my $local-listener = $local-server.tap(
    ... # TODO
);
await $local-listener.socket-port;

First we should open asynchronous socket that acts as a Supply of incoming connections, then tap it. Tap is just a subscription to a Supply and we will implement its guts in the next step.

But beware! Having local port listener does not mean we can immediately connect to this port, because those actions are asynchronous. That is what await $local-listener.socket-port part is for. It is a Promise that will be kept when port is actually opened. Promises are thread safe thingies that can be kept or broken once per their lifetime and can carry more info than just boolean state: if you used local port 0 that is how you will receive system assigned local port number - just check value kept by this Promise as $local-listener.socket-port.result.

Open channel from jump to service

my $local-listener = $local-server.tap(
    my $local-listener = $local-server.tap(
    -> $local-connection {
        my $jump-channel = await $jump-connection.forward(
            $service-host, $service-port,
            $local-host, $local-port
        );
        ...
    }
);

Back to tap implementation. It expects closure with one parameter, which will be incoming connection to local. When we receive such connection we must ask jump host to create SSH channel that will pass TCP packets to service host.

Create bidirectional data flow between local connection and SSH channel

my $local-listener = $local-server.tap(
    my $local-listener = $local-server.tap(
    -> $local-connection {
        my $jump-channel = ...
        react {
            whenever $local-connection.Supply(:bin) {
                $jump-channel.write($_);
                LAST $jump-channel.close();
            }
            whenever $jump-channel.Supply(:bin) {
                $local-connection.write($_);
                LAST $local-connection.close();
            }
        }
    }
);

To establish bidirectional data exchange we can use reactive programming. Both local connection and jump channel have Supplies that will give us incoming data. All we need to do it pass data from local connection to jump channel and from jump channel to local connection.

LAST phaser is used to gracefully close corresponding side when one of the Supplies closes. In happy path it is local connection that will be closing SSH channel.

To capture any errors you can use QUIT phasers.

Done

use SSH::LibSSH;

my Str $service-host = 'service';
my Int $service-port = 7878;
my Str $jump-host = 'jump';
my Int $jump-port = 22;
my Str $jump-user = 'me';
my Str $jump-private-key-file = '/home/me/.ssh/jump';
my Str $jump-private-key-password = 's3cret!';
my Str $local-host = '127.0.0.1';
my Int $local-port = 8080;

my $jump-connection = await SSH::LibSSH.connect(
    host => $jump-host, port => $jump-port, timeout => 4,
    user => $jump-user,
    private-key-file => $jump-private-key-file,
    private-key-file-password => $jump-private-key-password
);

my $local-server = IO::Socket::Async.listen($local-host, $local-port);
my $local-listener = $local-server.tap(
    -> $local-connection {
        my $jump-channel = await $jump-connection.forward(
            $service-host, $service-port,
            $local-host, $local-port
        );
        react {
            whenever $local-connection.Supply(:bin) {
                $jump-channel.write($_);
                LAST $jump-channel.close();
            }
            whenever $jump-channel.Supply(:bin) {
                $local-connection.write($_);
                LAST $local-connection.close();
            }
        }
    }
);
await $local-listener.socket-port;

printf "Connect to %s:%d to actually connect to %s:%d.\n",
    $local-host, $local-listener.socket-port.result,
    $service-host, $service-port;

# your logic that requires service host goes here

# gracefully close jump SSH channel and connection when done
$local-listener.close();
$jump-connection.close();

When using this flow make sure you truly closed local connection first before closing local listener and jump connection. Common mistake is forgetting for example that HTTP modules with keep-alive capability will keep connection open in background.

Alternatives

SSH::LibSSH::Tunnel module implements exactly the same solution but in different manner - it uses separate thread and single react block wrapping whole logic under the hood.

SSH port forwarding from within code

Paweł bbkr Pabian — Mon, 06 Jan 2025 20:52:43 +0000

Intro

Port forwarding is a common technique used to access some service behind jump host from your local machine. It is a daily bread of many administrative tasks. In this post I'll show how to create such connections on demand from within Raku or Rust code (without relying on ssh -L system command).

Theory

Before jumping to implementation let me go briefly through operating principle.

There is service host that exposes something on port 7878. However you cannot access it directly from local machine because it is in different network.

Luckily there is jump host in that network that exposes SSH service on port 22, which you can connect to (1).

It is time to open any free port on your local machine (2), let's say 8080, and start listening for incoming connections. Note that those connections can be made from different thread of the same program, which is case we are interested in here.

Whenever connection arrives to local machine (3) we ask jump host using SSH connection (1) to create SSH channel (4) that will pass TCP packets with source port 8080 to jump machine to port 7878.

To complete puzzle we must pass traffic (5) between connection made to our local port 8080 (3) and SSH channel (4) in both ways using existing SSH connection (1).

Now to connect to service:7878 you actually connect to local:8080 and your data goes like this.

Important! You need another set of yellow arrows (4) and (5) for every forwarded connection you want to make, you cannot push packets from two connections through single SSH channel. Channel only knows that it should pass TCP packet with source port 8080, it has no knowledge that it came from specific connection (3) on local machine.

SSH key based authentication setup

In pretty much every case where SSH port forwarding is used SSH keys are preferred over interactive user/password login. If your local to jump connection is not already configured please perform following steps.

Generate new key on local machine:

ssh-keygen -t ed25519 -C "me@example.com"

When asked for location choose some meaningful file name and remember the path, in my case it will be /home/me/.ssh/jump.

Enter key password and remember it - in my case it will be s3cret!.

Append content of generated public key /home/me/.ssh/jump.pub to ./ssh/authorized_keys file on jump host in your home directory there. Create file if needed. You can do it manually or you can use ssh-copy-id -i /home/me/.ssh/jump me@jump if you can already login to jump using login/password method and have ssh-copy-id installed on your local machine.

Test your setup by invoking ssh -i /home/me/.ssh/jump me@jump, enter your key password and if it connects you are good to go.

Security concerns

You should never store SSH key password directly in your code. I'll do it here purely for demonstration purposes, but in real life scenario I recommend storing and fetching it from remote vault. Just to have manageable "lockdown" procedure in case of security breach.
You should always confirm your jump host identity in case of man-in-the-middle attack, when someone redirects your DNS to a fake jump host. I'll explain later how to do it in Raku and Rust.

Let's start coding

Jump to:

Dynamically pairing tokio spawns

Paweł bbkr Pabian — Fri, 07 Jun 2024 15:01:56 +0000

Hi everyone!

While learning Rust I stumbled upon problem not covered by popular online tutorials. When they talk about tokio and MPSC (multi-producer single-consumer) channels they usually connect spawned threads in some fixed way. However in my project I have to match dynamically asynchronous producers and consumers in various configurations. So let me share useful pattern I've discovered in my Rust journey.

Let's say we have a restaurant:

$ cargo init restaurant
    Creating binary (application) package
$ cd factory
$ cargo add tokio --features=full
    Updating crates.io index
      Adding tokio v1.38.0 to dependencies
      ...

As a manager we can assign different cooking stands to asynchronously prepare different types of food (don't worry about undefined values for now):

async fn cooking_stand (food: char) {
    loop {
        somewhere.send(food.clone()).await;
    }
}

Food should be delivered to tables awaiting it.

async fn table (number: u8) {
    loop {
        let food = somehow.recv().await;
        println!("Got {} at table {}", food, number);
    }
}

Now we can organize our restaurant:

#[tokio::main]
async fn main () {

    // cooking stands
    tokio::spawn(cooking_stand('🥗')); // salad
    tokio::spawn(cooking_stand('🍔')); // burger
    ...
    // tables for guests
    tokio::spawn(table(1));
    tokio::spawn(table(2));
    ...
    // keep our restaurant open for 1s
    sleep(Duration::from_millis(1000)).await;
}

Problem

For simplicity let's assume we accept orders through application. So restaurant manager (main thread) knows for example that table 1 is waiting for 🥗 and table 3 is waiting for 🍔. But how to actually fullfil those orders?

Naive approach

cooking_stand -> 🥗🥗🥗🥗🥗 ->         -> table 1
cooking_stand -> 🍕🍕🍕🍕🍕 -> manager -> table 2
cooking_stand -> 🍔🍔🍔🍔🍔 ->         -> table 3

If we force manager to do the job he can wait for 🥗 cooking stand to prepare salad and then pass it to table 1. Then wait for 🍔 cooking stand to prepare burger and carry it to table 3. This is obviously flawed design:

Cooking stands produce food whether it is needed or not.
If cooking stand is slow then manager will be waiting for food to be prepared.
Manager should not do the heavy lifting because it affects his responsiveness.

We need waiters

Fortunately tokio gives perfect tool for the job - oneshot channels. Those channels are designed and optimized to pass single value one time.

let (waiter_rx, waiter_tx) = oneshot::channel::<char>();

To make waiter deliver 🥗 to table 1 first we need to modify our cooking stands:

async fn cooking_stand (
    product: char,
    mut waiters: tokio::sync::mpsc::Receiver<oneshot::Sender<char>>
) {
    while let Some(waiter) = waiters.recv().await {
        waiter.send(product.clone());
    }
}

Where tokio::sync::mpsc::Receiver<oneshot::Sender<char>> is a queue of waiters. Yes, you read it right. You can send oneshot channels through other channels. When waiter arrives at cooking stand then cooking stand prepares food and gives it to waiter for being delivered to table. Let's do the same for tables, but they should get receiving part of specific waiter who will bring food to them:

async fn table (
    number: u8,
    mut waiters: tokio::sync::mpsc::Receiver<oneshot::Receiver<char>>
) {
    while let Some(waiter) = waiters.recv().await {
        let food = waiter.await.unwrap();
        println!("Got {} at table {}", food, number);
    }
}

When waiter is assigned to table customer waits for this waiter to deliver food produced by food stand. And to complete puzzle let's modify our main function. Manager, instead of doing the heavy lifting himself, can hire waiters and assign them to pairs of cooking stands and tables to fullfill food orders.

#[tokio::main]
async fn main () {

    // used by manager to send waiters to cooking stands
    let (stand_salad_tx, stand_salad_rx) = mpsc::channel::<oneshot::Sender<char>>(100);
    let (stand_pizza_tx, stand_pizza_rx) = mpsc::channel::<oneshot::Sender<char>>(100);
    let (stand_burger_tx, stand_burger_rx) = mpsc::channel::<oneshot::Sender<char>>(100);

    // set up cooking stands
    tokio::spawn(cooking_stand('🥗', stand_salad_rx));
    tokio::spawn(cooking_stand('🍕', stand_pizza_rx));
    tokio::spawn(cooking_stand('🍔', stand_burger_rx));

    // used by manager to send waiters to tables
    let mut tables: Vec<tokio::sync::mpsc::Sender<oneshot::Receiver<char>>> = Vec::new();

    // set up tables
    for number in 1..=4 {
        let (table_tx, table_rx) = mpsc::channel::<oneshot::Receiver<char>>(100);
        tables.push(table_tx);
        tokio::spawn(table(number, table_rx));
    }
}

Let's check if it works by adding following code at the end of our main:

    // create waiter
    let (waiter_tx, waiter_rx) = oneshot::channel::<char>();
    // send him for food to salad stand
    stand_salad_tx.send(waiter_tx).await;
    // send him to deliver food to table `1`
    tables.get(0).unwrap().send(waiter_rx).await;
    // manager can go back to doing his stuff

    // keep our restaurant open for 1s
    sleep(Duration::from_millis(1000)).await;

When ran it produces following output:

Got 🥗 at table 1

Yay!

Conclusions

This pattern of sending two halves of oneshot channels through regular channels to tokio spawns can be used to implement all kind of traffic control. Passing messages with given ratio, with throttling, etc.

Is it efficient? Very! I was surprised how well oneshot channels are optimized. Single core of my Ryzen 6800U processor was able to create over 5_000_000 oneshot channels and send them to corresponding spawns per second. That's crazy fast.
How to scale it? There will be situations when manager may encounter overfill of channels (in tokio all channels are capped) and will not be able to immediately send oneshot channel. In that cases you may for example increase amount of producers/consumers by issuing more spawns. Like add another burger stand and send oneshots to them in round robin order. Everything depends on what your spawns are actually doing.
What about error handling? You must have oneshot channel behavior in mind: If the Receiver is closed before receiving a message which has already been sent, the message will remain in the channel until the receiver is dropped, at which point the message will be dropped immediately. So even if two halves of oneshot channels were sent to corresponding spawns it still does not mean it's purpose will be fullfilled. Error handling in this case depends on which scenario you implement and how you need to react on delivery issues.

Thanks for reading

This is my first Rust post and I'm still discovering its features. If you think something could/should be better implemented then let me know in comments.

Dev.to has big spam issue

Paweł bbkr Pabian — Thu, 19 Oct 2023 10:20:31 +0000

Let's open Latest posts from last hour:

Omani Palms and Date Farms: A Sweet Escape
Does My Business Really Need Dallas Commercial Photography?
Electrifying Workouts at Kratos Studios EMS Gym in Salem NH (with #security tag)
Bully Sticks for Dogs (with #programming tag)
Dinno's Carpet Cleaning & Pest Control
Industrial Ethylene Oxide Sterilizer-Lodha International
USDA REAP Grants: Powering Rural America with Clean Energy
Unlocking Your Personal Car Driver Options: A DriverSab Guide
Bitcoin BTC price predication Qtfcrypto Technical Analysis

Nearly 40% spam makes this site unpleasant to check on daily basis. Spam issue was already mentioned by moderators in this post, but I have a feeling that it is getting progressively worse.

Ideas?

First posts from new accounts should have captcha.
Posting through API should not be available for new accounts until they make 3 posts from web interface.
Limited exposure, posts from new accounts should be shown for small percent of users and shown to rests if not flagged by control group within 2 hours.

UTF-8 series wrap up

Paweł bbkr Pabian — Sun, 24 Sep 2023 21:31:07 +0000

What a long, strange trip it's been. Idea to "quickly blog about some Unicode basics" grew into 17 posts monster series :)

Special thanks go to:

Unicode Consortium. You are the hero everyone needs. Oh, if you read this - can you please explain why ł does not decompose? It still scratches my brain :)
Raku language authors. Because of the excellent out-of-the-box UTF-8 support it was excessively used to illustrate many examples in this series.
Readers, for all the comments and reactions.

SeeYa

UTF-8 in MySQL

Paweł bbkr Pabian — Sun, 24 Sep 2023 21:10:06 +0000

This series is supposed to be focused on technical aspects of Unicode and I do not plan to analyze UTF support in various technologies. However for MySQL I want to make an exception, because I've seen countless examples of misunderstanding its concepts and falling into traps.

Character Set vs Collation

CREATE TABLE `foo` (
    `bar` VARCHAR(32)
) Engine=InnoDB
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_0900_ai_ci;

Character set tells how data will be stored in binary form.
Collation tells how data will be compared. It was explained in this post of the series.

Later I will explain what those cryptic names mean.

Property inheritance

Character set and/or collation can be specified on 7 (yes, seven!) different levels.

Server

MySQL > SELECT @@character_set_server, @@collation_server;
+------------------------+--------------------+
| @@character_set_server | @@collation_server |
+------------------------+--------------------+
| utf8mb4                | utf8mb4_0900_ai_ci |
+------------------------+--------------------+

Those are your global settings that will be used when creating databases. So:

CREATE DATABASE `test`;

Is the same as:

CREATE DATABASE `test` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Those server settings are copied when database is created, so changing server settings later will not affect your databases.

Database

CREATE DATABASE `test` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

It is just another level of default, this time applied to created tables. So:

CREATE TABLE `foo` (
    `bar` VARCHAR(32)
) Engine=InnoDB;

Is the same as:

CREATE TABLE `foo` (
    `bar` VARCHAR(32)
) Engine=InnoDB
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_0900_ai_ci;

And just like server settings those are also copied when tables are created. Altering database with ALTER DATABASE test CHARACTER SET xxx COLLATE yyy will not alter tables in this database.

You can check currently used database character set and collation either from variables:

MySQL [test]> SELECT @@character_set_database, @@collation_database;
+--------------------------+----------------------+
| @@character_set_database | @@collation_database |
+--------------------------+----------------------+
| utf8mb4                  | utf8mb4_0900_ai_ci   |
+--------------------------+----------------------+

Or from information schema:

MySQL [test]> SELECT `default_character_set_name`, `default_collation_name`
    FROM `information_schema`.`schemata`
    WHERE `schema_name` = 'test';
+----------------------------+------------------------+
| DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME |
+----------------------------+------------------------+
| utf8mb4                    | utf8mb4_0900_ai_ci     |
+----------------------------+------------------------+

Note the inconsistency - DEFAULT_CHARACTER_SET_NAME in information schema, but character_set_database in variable and CHARACTER SET in create.

Table

CREATE TABLE `foo` (
    `bar` VARCHAR(32)
) Engine=InnoDB
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_0900_ai_ci;

It is - you guessed it - another level of defaults applied to columns. So:

CREATE TABLE `foo` (
    `bar` VARCHAR(32)
) Engine = InnoDB;

Is the same as:

CREATE TABLE `foo` (
    `bar` VARCHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci
) Engine = InnoDB;

And just like database settings those are also copied when columns are created. Altering table with ALTER TABLE foo CHARACTER SET xxx COLLATE yyy will not alter columns in this table.

However this time tool is available for convenient conversion - ALTER TABLE foo CONVERT TO CHARACTER SET xxx COLLATE yyy will alter both table defaults and columns in this table.

You can check table collation in information schema:

MySQL [test]> SELECT `table_collation`
    FROM `information_schema`.`tables`
    WHERE `table_schema` = 'test'
        AND `table_name` = 'foo';
+--------------------+
| TABLE_COLLATION    |
+--------------------+
| utf8mb4_0900_ai_ci |
+--------------------+

Note another inconsistency - this time TABLE_COLLATION implies character set, which is not given explicitly. Also it inconsistent with database level naming, being a default but missing DEFAULT_ prefix.

If you want to retrieve implied character set there is another information schema resource to do so:

MySQL > SELECT `character_set_name`
    FROM `information_schema`.`character_sets`
    WHERE `default_collate_name` = 'utf8mb4_0900_ai_ci';
+--------------------+
| CHARACTER_SET_NAME |
+--------------------+
| utf8mb4            |
+--------------------+

Column

Finally, this is the "true" thing. That is how data will be stored and sorted. Server, database and table levels were only the defaults used for column creation.

You can check column character set and collation from information schema:

MySQL [test]> SELECT `character_set_name`, `collation_name`
    FROM `information_schema`.`columns`
    WHERE `table_schema` = 'test'
        AND `table_name` = 'foo'
        AND `column_name` = 'bar';
+--------------------+--------------------+
| CHARACTER_SET_NAME | COLLATION_NAME     |
+--------------------+--------------------+
| utf8mb4            | utf8mb4_0900_ai_ci |
+--------------------+--------------------+

Yes, you can have different character sets and collations within single table:

CREATE TABLE `foo` (
    `bar` VARCHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
    `baz` VARCHAR(32) CHARACTER SET latin1 COLLATE latin1_general_ci
) Engine = InnoDB;

I will give examples when it may be useful once all those cryptic names are explained.

My advice is: always provide character set and collation when creating databases, tables and columns. I've seen this too many times - developers adding tables without checking which character set and collation will be inherited from database. Or adding columns without checking which character set and collation will be inherited from table. Being more explicit = having less headache later.

System

MySQL > SELECT @@character_set_system;
+------------------------+
| @@character_set_system |
+------------------------+
| utf8mb3                |
+------------------------+

This is global character set for metadata. It tells what characters you can use in schema names:

CREATE TABLE `łąka` (
    `bąki` int unsigned
) Engine = InnoDB

It is not part of inheritance chain Server -> Database -> Table -> Column.

Connection

MySQL > SELECT @@character_set_connection, @@collation_connection;
+----------------------------+------------------------+
| @@character_set_connection | @@collation_connection |
+----------------------------+------------------------+
| utf8mb4                    | utf8mb4_general_ci     |
+----------------------------+------------------------+

Those are wire protocol information. Character set tells meaning of transferred data, for example 0xF0 0x9F 0x98 0x8A sent or received means 😊. Collation will be used for comparing/sorting data not derived from any column, for example bare SELECT 'A' = 'a'.

Connection and Column character set may not be aligned, but it will fail if Connection wire protocol cannot transfer code points encoded in Columns. Best practice is to always use utf8mb4.

Query

SELECT *
FROM `foo`
ORDER BY `bar` COLLATE utf8mb4_estonian_ci;

You can override default column collation for ordering / grouping within SELECT query. This is useful when different alphabets sorts the same characters differently.

MySQL > CREATE TABLE `collation_test` (`data` text) Engine = InnoDB;

MySQL > INSERT INTO `collation_test` (`data`)
    VALUES ("A"), ("Ä"), ("Z");

MySQL > SELECT *
    FROM `collation_test`
    ORDER BY `data` COLLATE utf8mb4_sv_0900_as_cs;
+------+
| data |
+------+
| A    |
| Z    |
| Ä    |
+------+

MySQL > SELECT *
    FROM `collation_test`
    ORDER BY `data` COLLATE utf8mb4_es_0900_as_cs;
+------+
| data |
+------+
| A    |
| Ä    |
| Z    |
+------+

Character set `utf8` vs `utf8mb4`

MySQL cheated in the past. They added character set utf8 but it was capable only of handling up to 3 byte code points.

MySQL [test]> CREATE TABLE `foo` ( `bar` CHAR(1) )
    Engine = InnoDB
    CHARACTER SET = utf8;

MySQL [test]> INSERT INTO `foo` (`bar`) VALUES ('😊');
ERROR 1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x8A' for column 'bar' at row 1

They did it however in good faith - back then 4 byte code points were not used. Indexes are constructed in such a way, that they must assume maximum byte length of a string. Maximum supported index byte length was 767 bytes, which allowed to index columns up to CHAR(255) - because 255*3=765 was fitting into index. For 4 byte code points maximum indexable column would be only CHAR(191).

Later MySQL added utf8mb4 character set capable of storing proper 4 byte code points. Legacy utf8 was aliased as utf8mb3. Default maximum supported index byte length was also extended in MySQL 8 to 3072 bytes, allowing to index columns up to VARCHAR(768).

Today MySQL tries to fix this technical debt, and if you specify character set as utf8 you will get following warning: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.

But how to index longer UTF-8 columns? Common trick is to use hash indexing:

CREATE TABLE `foo` (
    `bar` varchar(1000),
    `bar_hash` CHAR(32),
    KEY (`bar_hash`)
) ENGINE = InnoDB
    CHARACTER SET = utf8mb4
    COLLATE = utf8mb4_0900_ai_ci;

CREATE TRIGGER `foo_insert`
BEFORE INSERT ON `foo`
FOR EACH ROW SET NEW.`bar_hash` = MD5( WEIGHT_STRING( NEW.`bar` ) );

CREATE TRIGGER `foo_update`
BEFORE UPDATE ON `foo`
FOR EACH ROW SET NEW.`bar_hash` = MD5( WEIGHT_STRING( NEW.`bar` ) );

Function WEIGHT_STRING is super useful, because it converts text to format used by collation. Function MD5 reduces too long texts always to 32 bytes HEX representation.

Now you can for example create UNIQUE KEY on column bar_hash or use it in query:

SELECT *
FROM `foo`
WHERE `bar_hash` = MD5( WEIGHT_STRING( 'looked up text' ) );

Collation `utf8mb4_0900_ai_ci`

MySQL 8 did huge cleanup in collation naming. utf8mb4_0900_ai_ci means that it is collation of 4 byte UTF-8 done by Unicode 9.0 standard in accent (diacritic) insensitive and case insensitive manner.

It does not mean that database cannot store characters from Unicode version 10 onward. As I explained previously UTF-8 is designed in such a way, that storage is independent from versioning. Just comparison rules from Unicode version 9.0 will be used. That pretty much means recent ones, because almost nothing new was declared in this aspect later.

Accent / case insensitivity is up to you to decide. Basically you have 3 options:

utf8mb4_0900_ai_ci - Accent and case insensitive, 'a' = 'A' = 'ą' = 'Ą'.
utf8mb4_0900_as_ci - Accent sensitive but case insensitive, 'a' <> 'ą' but still 'a' = 'A' and 'ą' = 'Ą'.
utf8mb4_0900_as_cs - Accent and case sensitive, 'a' <> 'A' <> 'ą' <> 'Ą'.

Remember that you can mix them. For example unique column for login may have collation utf8mb4_0900_ai_ci so Józef, józef and jozef are treated as the same user. While column hobby may have collation utf8mb4_0900_as_ci because baki (fuel tanks) and bąki (bumble bees) are not the same.

You can list all utf8mb4 related collations by following query:

SHOW COLLATION WHERE Charset = 'utf8mb4';

Best practice is to stick with utf8mb4_0900_* set and avoid alphabet specific collations in columns. For example if you know your user is from Poland you can always use more friendly collation in query, ignoring column one:

SELECT `name`
FROM `products`
ORDER BY `name` COLLATE utf8mb4_pl_0900_ai_ci

Also avoid legacy collations like utf8mb4_general_ci, use only those with *_0900_* within name.

Triggers, Procedures, Functions

Things are weird for triggers, because they inherit character set and collation from... definer's connection. I won't go much into details here because it rarely bites the developer. Just remember to also drop / create them if you are migrating from old databases to new character set and collation. For full description of consequences read MySQL 5.1.21 change log.

Coming up next: Series wrap up.

UTF-8 Byte Order Mark

Paweł bbkr Pabian — Mon, 18 Sep 2023 12:04:18 +0000

In previous post of this series I explained that UTF is a multi byte encoding that also has few variants: UTF-8, UTF-16 and UTF-32. To make things more complicated in UTF-16 and UTF-32 there are two ways to send bytes of single code point - in big endian or little endian order.

BTW: Endianness term is not related to Indians. It comes form Gulliver's Travels book. There was a law in Lilliputians world that forced citizens to break boiled eggs from little end. Those who rebelled and were breaking eggs from big end were called "big endians".

What is Byte Order Mark?

To notify which byte order is in processed file or data stream a special sequence of bytes at the beginning was introduced, called Byte Order Mark. Or BOM for short.

For example UTF-16 can start with 0xFE 0xFF for big endian and 0xFF 0xFE for little endian order. And UTF-32 can start with 0x00 0x00 0xFE 0xFF for big endian and 0xFF 0xFE 0x00 0x00 for little one.

Impact on UTF-8

Here things gets weird. UTF-8 is constructed in such a way, that it has only one meaningful byte order, because first byte describes how many bytes will follow to get code point value.

However BOM specification has magic sequence for UTF-8, which is 0xEF 0xBB 0xBF. It only indicates encoding type, therefore has no big endian / little endian variants.

Implications

BOM idea may sound weird today, because UTF-8 became prevalent and dominant. But remember that we are talking about year 2000, when things were not that obvious.

Spec claims that if a protocol always uses UTF-8 or has some other way to indicate what encoding is being used, then it should not use BOM. So for example BOM should not appear in *.xml files:

<?xml version="1.0" encoding="UTF-8"?>
<tag>...

Or in MIME *.eml files:

--3e6ea2aa592cb31d47cefca38727f872
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"

Because those specify encoding internally. Unfortunately this is sometimes ignored, so if something broke your parser and you cannot find obvious error - look if file has UTF-8 BOM:

$ raku -e 'say "file.txt".IO.open( :bin ).read( 3 ) ~~ Buf.new(0xEF, 0xBB, 0xBF)'

True

Security issues

But what if BOM is not aligned with internal/assumed encoding? Let's create following file:

$ raku -e '
spurt "file.txt",
    Buf.new( 0xFE, 0xFF, 0x3c, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x3e )
'

Now you upload this to some service. This service has validator that respects BOM and should strip all HTML tags. Validator sees nonsense but perfectly legal content that passes validation:

Later this service opens and displays uploaded file, but it ignores BOM and assumes UTF-8:

Oooops! If you trusted validator and displayed this file without proper HTML escaping then you have JavaScript injection. This happened because 㱳捲楰琾 in UTF-16 suggested by BOM has the same byte sequence as <script> in assumed UTF-8.

Conclusions

You should still be aware of existence of Byte Order Mark, even if it makes zero sense in UTF-8 dominated world today.

Coming up next: UTF-8 in MySQL.

Fun with UTF-8: Homoglyphs

Paweł bbkr Pabian — Fri, 15 Sep 2023 10:06:22 +0000

ꓧ𐐬𝗆𐐬𝗀ⅼУрႹ ⅰѕ 𝗌е𝗍 𝗈ſ ဝո𝖾 𝗈г ꝳо𝗋е ɡ𝗋аρႹ𝖾ⅿе𝗌 𝗍Ⴙа𝗍 Ⴙ𝖺ѕ 𝗂ꝱ𝖾ꝴ𝗍𝗂𐐽а𝗅 о𝗋 ѵ𝖾г𝗒 𝗌Ꭵⅿі𝗅аꝵ ⅼꝏ𝗄 𝗍ᴏ 𝗌იო𝖾 о𝗍ꜧ𝖾𝗋 𐑈е𝗍 ဝſ ɡꝵ𝖺рႹеოеѕ. Like in previous sentence, that does not use a single ASCII letter:

ꓧ - LISU LETTER XA
𐐬 - DESERET SMALL LETTER LONG O
𝗆 - MATHEMATICAL SANS-SERIF SMALL M
𐐬 - DESERET SMALL LETTER LONG O
𝗀 - MATHEMATICAL SANS-SERIF SMALL G
ⅼ - SMALL ROMAN NUMERAL FIFTY
У - CYRILLIC CAPITAL LETTER U
р - CYRILLIC SMALL LETTER ER
Ⴙ - GEORGIAN CAPITAL LETTER CHIN
...

Homoglyphs are not Unicode specific, but it was ability to write in many scripts using single UTF encoding that made them popular.

Similarity is conditional

It is font dependent. Two sets of graphemes looking very similar (or even identical) in one font may not look that similar in another. For example т - CYRILLIC SMALL LETTER TE looks like ASCII T, but in cursive fonts (those that resembles handwriting connected letters) looks like m.

Similarity is subjective

For many people unfamiliar with given alphabets Ǧ and Ğ may look exactly the same. But if someone is using those letters on daily basis he will notice immediately that first one has CARON and the other has BREVE on top.

They are not limited to single grapheme

For example ထ - MYANMAR LETTER THA looks like two ASCII o letters. And the other way - ASCII rn looks like single ASCII letter m.

Applications?

Fun. 𐐑ǃkǝ pɹoducǃng weird looking bᴝt ɹeadɐble ʇext.
Trolling. Programmer's classic is to replace in someone's code ; with ; - GREEK QUESTION MARK - and watch some funny debugging attempts. More advanced version is to modify keybinding. For example on macOS create ~/Library/KeyBindings/DefaultKeyBinding.dict with following content:

{
    ";" = (insertText:,";");
}

And observe how Python suddenly became someone's favorite language of choice :P

Just promise you won't troll stressed out junior dev before the end of sprint.

Phishing. This is "Fun with UTF-8" sub series, but unfortunately this application is anything but fun. Homoglyphs are massively used to spoof company names, bypass anti-spam filters and create fake domains. For example can you spot difference between Paypal and ꓑayраl?

Common way to detect those is to check Script Unicode property, more on those in this post. Single word using more than one script should be considered suspicious:

$ raku -e '"Paypal".comb.classify( *.uniprop("Script") ).say'
{Latin => [P a y p a l]} # real

$ raku -e '"ꓑayраl".comb.classify( *.uniprop("Script") ).say'
{Cyrillic => [р а], Latin => [a y l], Lisu => [ꓑ]} # fake

Raku note: Method comb without param extracts list of characters. Those characters are classified by classify method. Classification key is output of uniprop method for given character.

Tools

I'm maintaining HomoGlypher library/package which allows to handle common homoglyph operations:

Unwind. From ASCII text create list of all possible homoglyphied text variants. This is useful for example in checking if some domain is spoofed.
Collapse - From homoglyphied text recover all possible ASCII text variants. Useful for normalization of text before passing it to content filters.
Randomize - From ASCII text create single homoglyphied text with given replacement probability.
Tokenize. Create regular expression token that will match homoglyphied text equivalent to given ASCII text. I think this may be the only homoglyph related library in the existence having this feature :)

Huge list of mappings is provided, so you won't have to dig through Unicode blocks on your own to find possible similarities between graphemes.

Give it a try. And if you know other homoglyph libraries please leave a note in the comments for future readers.

UTF-8 regular expressions

Paweł bbkr Pabian — Thu, 07 Sep 2023 07:26:53 +0000

For many, many years Perl language has been top choice for text processing tasks. As a result it established informal standard of regular expressions. Today almost every big language uses either PCRE (Perl Compatible Regular Expressions) library directly or implements own regular expression engine heavily inspired and mostly compatible with Perl one.

Raku language was meant to be direct continuation of Perl (former name was Perl 6). Its regular expression engine was redesigned from scratch. However with modernized syntax and new features came lack of backward compatibility.

Let's compare them side by side to have general understanding of what is currently available in most languages (I will call those regular expression "Perl" ones) and what may be adopted to languages if Raku manages to establish new standard. There is a lot to cover here, so comments will be divided into Unicode specific aspects and separate section that clarifies technical differences.

Literal text

$ perl -E 'use utf8; say "Żółw 🐢" =~ /Ż..w 🐢/'
1

$ raku -e 'say "Żółw 🐢" ~~ /Ż..w \s "🐢"/'
｢Żółw 🐢｣

Unicode:

Perl and Raku matches text in Unicode aware manner, respecting multi-byte code points. Non-ASCII characters are allowed in regular expression body. Yay, good start!
Raku treats non-alphabetic symbols (like Emojis) as meta characters and requires them to be quoted.
Perl needs use utf8 pragma to indicate source code is in UTF-8, similar declaration is a common requirement in a lot of other languages. Raku source code is UTF-8 by default.

Technical:

White space handling is flipped. Perl treats white spaces in regular expression literally and can ignore them with //x modifier. Raku ignores white spaces by default and can treat them literally with m:s// or m:sigspace// modifier. So you can write /Ż..w \s 🐢/x in Perl to get Raku behavior or m:s/Ż..w "🐢"/ in Raku to get Perl behavior.
In Raku modifiers were moved to the beginning of regular expression for better readability.
Perl returns boolean as match result and matched text is available in $& variable while Raku returns Match object.

Predefined character classes

$ perl -E 'use utf8; "1꧕ żółtych róż" =~ /\d{2} \w+ [[:alpha:]]+/; say $&'
1꧕ żółtych róż

$ raku -e 'say "1꧕ żółtych róż" ~~ /\d**2 \s \w+ \s <.alpha>+/'
｢1꧕ żółtych róż｣

Unicode:

Both Perl and Raku support similar set of long and short classes (Perl long/short, Raku long/short) that includes non-ASCII characters.
Raku also supports small set of predefined tokens.
Careful what you wish for!

Very common mistake is to write regular expression in Unicode aware language without realizing what given character classes matches. Or blindly copy-pasting old regular expressions into Unicode aware code. For example \d matches digit. Javanese digit five ꧕ is a digit and will be matched in ^\d{5}\z American short zip code regular expression, probably causing weird side effects and errors. If you need only ASCII digits you must be explicit about it - [0-9] in Perl or <[0..9]> in Raku.

Technical:

Character classes are handled very differently. In Perl predefined POSIX [:classes:] are only usable within class group []. While in Raku they are written as <tokens>, which is super consistent with built-in Grammars. More on that later.

Code point properties

I recommend reading this post in series before continuing...

$ perl -E 'use utf8; "Cool😎" =~ /\p{Lu}\P{Uppercase_Letter}+\p{Block=Emoticons}/; say $&'
Cool😎

$ raku -e 'say "Cool😎" ~~ /<:Lu><:!Uppercase_Letter>+ <:Block("Emoticons")>/'
｢Cool😎｣

Unicode:

Both Perl and Raku support code point properties.
Binary properties can be tested without value using both long or short names (for example Uppercase_Letter or Lu).
Specific value of property can be checked by providing parameter (for example value of property named Block should be equal to Emoticons).
Perl mixes Unicode properties, POSIX properties and internal properties under common \p{} test. They also have variants, \p{PosixDigit} matches 0-9 while \p{XPosixDigit} matches all Unicode digits. One way to look at it is that property is a property, no matter who defined it. But I personally dislike it because it provides duplicated, overlapping functionality and makes regular expressions less portable. I really wish there was separate test dedicated for Unicode properties only.

Technical:

Perl uses \p{Foo} for property and \P{Foo} for negated property while Raku uses token-ish form <:Foo> for property and <:!Foo> for negated property.
Property value parameter is different. Perl uses Foo=Bar syntax, which is compact but kind of weird due to unquoted value - even Perl itself does not compare strings like that. While Raku decided on Foo('Bar') method call style, aligned with the rest of the Raku and commonly used in other languages.
Perl treats string properties called without value as matching if they return any value indicating that the property applies. While Raku only matches if value matches:

$ perl -E 'use utf8; say "4" =~ /\p{Digit}/;'
1

$ raku -e 'say "4" ~~ /<:Digit>/'
Nil # oops, not explicit enough

$ raku -e 'say "4" ~~ /<:Digit("Decimal")>/'
｢4｣ # because property "Digit" of "4" is "Decimal"

Raku has nasty trap here. One may think that "if I need Digit property of any kind I can just request any defined value":

$ raku -e 'say "4" ~~ /<:Digit(Any:D)>/'
｢4｣ # success?

This is very far from being correct, because some properties returns defined strings indicating that they do not apply:

$ raku -e 'say "A" ~~ /<:Digit(Any:D)>/'
｢A｣ # wrong

$ raku -e 'say "A".uniprop("Digit")'
None # literal string 'None' matching Any:D value

Hint:

If you are mixing tests for General_Category, Script and Block properties in a single regular expression I strongly recommend using full property names. For example can you tell what 'A' ~~ /<:Latin>/ test means? Yes, it tests Script, not the Block, because A is in Block named Basic Latin. Being explicit greatly improves regular expression understanding, for example in Perl:

$ perl -E '
    use utf8;
    "A" =~ /\p{General_Category=Uppercase_Letter}/;
    "A" =~ /\p{Block=Basic Latin}/;
    "A" =~ /\p{Script=Latin}/;
'

Warning, in Raku explicit General_Category test currently only accepts short forms.

Property arithmetic

One of the features that looks useless but really shines when combined with Unicode properties. Let's assume you got text about animal life expectancy stats: แฮมสเตอร์ ๔, แมว ๑๖ (stats: hamster 4, cat 16) and must extract Thai words from it, skipping numbers.

One way to solve it is to manually enumerate all Thai letters:

$ perl -E '
    use utf8;
    my $text = "stats: แฮมสเตอร์ ๔, แมว ๑๖";
    say for $text =~ /[กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯะาำเแโใไๅๆ]+/g;
'

แฮมสเตอร  # hamster
แมว       # cat

That works but will cause a lot of head scratching if someone unfamiliar with Thai alphabet encounters this regular expression. You can try to be more explicit and provide range:

$ perl -E '
    use utf8;
    my $text = "stats: แฮมสเตอร์ ๔, แมว ๑๖";
    say for $text =~ /[\N{THAI CHARACTER KO KAI}-\N{THAI CHARACTER MAIYAMOK}]+/g;
'

แฮมสเตอร
แมว

Which also works, but still requires knowledge about Thai alphabet and introduces new risk that provided range may not be continuous series of code points exclusively from this alphabet. For example Polish alphabet starts with a, ends with ź, but there are actually 280 code points between them containing a lot of other stuff.

That is the perfect application for extended character class:

$ perl -E '
    use utf8;
    my $text = "stats: แฮมสเตอร์ ๔, แมว ๑๖";
    say for $text =~ /(?[ \p{Thai} & \p{Letter} ])+/g;
'

แฮมสเตอร
แมว

Extended class is wrapped in (?[ ]) and allows to perform classes arithmetic, in this case & indicates intersection between Thai script and Letter general category. You can make intersections &, unions +, subtraction - and XOR ^ logic. No Thai alphabet knowledge is needed to extract Thai words!

Well, kind of... Full Thai word for hamster is
หนูแฮมสเตอร์ (thehamster). You may already noticed that none of previous solution extracted last character ร์ properly. And our code actually splits this word:

$ perl -E '
    use utf8;
    my $text = "stats: หนูแฮมสเตอร์ ๔, แมว ๑๖";
    say for $text =~ /(?[ \p{Thai} & \p{Letter} ])+/g;
'

หน        # the
แฮมสเตอร  # hamster
แมว

This is because นู and ร์ are actually two characters written one above other forming grapheme cluster, let's analyze them:

$ raku -e '.say for "นู".uninames;'
THAI CHARACTER NO NU
THAI CHARACTER SARA UU

$ raku -e '.say for "นู".uniprops;'
Lo # Letter_Other
Mn # Nonspacing_Mark

That solves our mystery. Those missing Thai characters are not letters but non spacing marks. But hey, we have property arithmetic. Let's fix that quickly:

$ perl -E '
    use utf8;
    my $text = "stats: หนูแฮมสเตอร์ ๔, แมว ๑๖";
    say for $text =~ /(?[ \p{Thai} & ( \p{Letter}  + \p{Nonspacing_Mark} ) ])+/g;
'

หนูแฮมสเตอร์
แมว

So now we have intersection of Thai script with union of Letter and Nonspacing_Mark general category. Everything encapsulated in neat, self-documenting, extended character class. Lovely!

In Raku word things are not that mature yet. Character class arithmetic only supports union and subtraction. For example let's find stuff that looks like model numbers (at least 2 characters long):

$ raku -e '
    say "Production of AR-15 riffle..."~~ /
        <:Uppercase_Letter + :Digit("Decimal") + :Dash_Punctuation> ** 2..*
    /
'

｢AR-15｣

Syntax for extended class is <:A + :B>, no grouping inside.

Grapheme clusters

$ perl -E 'use utf8; "หนูแฮมสเตอร์" =~ /\p{Letter}+/; say $&;'
หน # the

$ raku -e 'say "หนูแฮมสเตอร์" ~~ /<:Letter>+/'
｢หนูแฮมสเตอร์｣ # thehamster, unharmed :)

This time point goes to Raku, which handles grapheme clusters properly.

Perl has predefined \X class, which represents "what appears to be a single character, but may be represented internally by more than one", so pretty much everything. Because it cannot be intersected in extended class to get cluster of specific property it is next to useless.

Diacritics

Matching with ignoring combining code points is Raku-only feature.

$ raku -e 'say "👋🏾Cześć" ~~ m:ignoremark/ "👋" Czesc /'
｢👋🏾Cześć｣

In Perl it is possible through decomposing using Unicode::Normalize module, filtering out combining code points and matching preprocessed text. But Perl regular expression engine does not support that out of the box.

Variable case length

There is perfect example in German language - sharp s, also named Eszett.

It looks like this ß and basically is equal to ss. So weiße and weisse both mean white. It had no uppercase form, SS was always used. I wrote "was", because in 2017 uppercase form of ß was officially added to German alphabet as ẞ, causing some backward-compatibility havoc:

$ raku -e 'say "ß".uc'
SS # still translates to SS, backward compatibility

$ raku -e 'say "ẞ".lc'
ß # this does not translate to ss, because it never did

So we have intransitive case change, that also changes length - lower case ẞ is ß which is synonym for lower case ss. Both Perl and Raku handles this correctly:

$ raku -e 'say "WEIẞE" ~~ m:ignorecase/ weisse /'
｢WEIẞE｣

$ perl -E 'use utf8; say "WEIẞE" =~ /weisse/i;'
1

Pick your poison

We had two regular expression engines flexing muscles to prove being Unicode handling champion. Perl dominates with Unicode properties and property arithmetic. Raku fights back with grapheme clusters and diacritic insensitive matching.

Coming up next: Optional fun with homoglyphs. And Byte Order Mark. I promise next posts will be shorter and easier.

DEV Community: Paweł bbkr Pabian

How to omit enum name in variant

Guard state transitions with proto methods

Problem

Naive fix

Troubles ahead

Proto methods for the rescue!

It's a win-win

Can we implement state guards to detect misuse in compile time?

SSH port forwarding from within Rust code

Reminder

Preparation

No abstractions or error handling

Boilerplate

Connect to jump host

Open local listener

Open channel from jump to service

Create bidirectional data flow between local connection and SSH channel

Done

SSH port forwarding from within Raku code

Reminder

Preparation

Boilerplate

Connect to jump host

Open local listener

Open channel from jump to service

Create bidirectional data flow between local connection and SSH channel

Done

Alternatives

SSH port forwarding from within code

Intro

Theory

SSH key based authentication setup

Security concerns

Let's start coding

Dynamically pairing tokio spawns

Problem

Naive approach

We need waiters

Conclusions

Thanks for reading

Dev.to has big spam issue

UTF-8 series wrap up

UTF-8 in MySQL

Character Set vs Collation

Property inheritance

Character set utf8 vs utf8mb4

Collation utf8mb4_0900_ai_ci

Triggers, Procedures, Functions

UTF-8 Byte Order Mark

Fun with UTF-8: Homoglyphs

UTF-8 regular expressions

Character set `utf8` vs `utf8mb4`

Collation `utf8mb4_0900_ai_ci`