DEV Community: Sunshine

Dart and Rust: the async story 🔃

shekohex — Mon, 22 Jun 2020 00:04:28 +0000

In the last blog post: Dart Meets Rust: a match made in heaven we show how to connect both languages and build more secure, performant Flutter Applications.
In this blog post, we will talk about how to use Multithreaded async Rust code with async Dart and how we could build applications that have more than one runtime working together in a smooth way like how the sōzu works.

The Problem 🎯

Running more than async loops is not always easy to get it right, you know that Dart VM has it is own async runtime and in Rust, we could plug-in any one of the popular runtimes, here is a few for reference Tokio, async-std, and recently smol which is now built-in the async-std as it is low-level runtime.

The issue is that Dart as a language is designed to be single-threaded, that's not a problem in itself, but we will see later how would that be an issue when using Rust.
well, you could run more than Isolate which is like a small Dart VM instance they can work together and communicate by message passing, here is a video where Andrew from Flutter team talk about Dart Isolates and the Event Loop, I highly recommend watching it before continuing if you have no idea about Dart isolates (PS: it is less than 6 min 😅)

Let's Scrape the web 🌍

For making this post more exciting let's build a simple web scraping library in Rust and use it in Flutter Application.
using our experience from the last blog post on how to connect both languages, we will use the same idea here too.

Create the Rust crate first:

$ cargo new --lib native/scrap # yes, it is a scrap lol

cool, now we need a way to send async web requests, so let's use reqwest

[dependencies]
# we will use `rustls-tls` here since openssl is an issue when cross-compiling for Android/iOS
reqwest = { version = "0.10", default-features = false, features = ["rustls-tls"] }

Now let's write some rusty code 😀

// lib.rs
use std::{error, fmt, io};

/// A useless Error just for the Demo
#[derive(Copy, Clone, Debug)]
pub struct ScrapError;

impl fmt::Display for ScrapError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "Error While Scrapping this page.")
    }
}

impl error::Error for ScrapError {}

impl From<reqwest::Error> for ScrapError {
    fn from(_: reqwest::Error) -> Self {
        Self
    }
}

impl From<io::Error> for ScrapError {
    fn from(_: io::Error) -> Self {
        Self
    }
}

/// Load a page and return its HTML body as a `String`
pub async fn load_page(url: &str) -> Result<String, ScrapError> {
    Ok(reqwest::get(url).await?.text().await?)
}

so as you see it is some simple code with sh**ty error handling 😂

Next, Let's Write our FFI binding to the scrap crate:

$ cargo new --lib native/scrap-ffi

in our Cargo.toml

[lib]
name = "scrap_ffi"
crate-type = ["cdylib", "staticlib"]

[dependencies]
scrap = { path = "../scrap" }

Now, it gets dirty because we need to consider a few things here

Setting up the Runtime 🔌

We will choose Tokio here as our runtime, it does not matter which runtime we use, but in our case, since reqwest works well with Tokio we going to use it.
also lazy_static it will be handy in a second.

tokio = { version = "0.2", features = ["rt-threaded"] }
lazy_static = "1.4"

Write some code to set up the runtime

// lib.rs
use tokio::runtime::{Builder, Runtime};
use lazy_static::lazy_static;
use std::io;

lazy_static! {
    static ref RUNTIME: io::Result<Runtime> = Builder::new()
        .threaded_scheduler()
        .enable_all()
        .core_threads(4)
        .thread_name("flutterust")
        .build();
}

/// Simple Macro to help getting the value of the runtime.
macro_rules! runtime {
    () => {
        match RUNTIME.as_ref() {
            Ok(rt) => rt,
            Err(_) => {
                return 0;
            }
        }
    };
}

This setup a Tokio Runtime with 4 threads and naming them flutterust for each thread.

Error Handling 🧰

Handling Errors between FFI is hard, so we will use a helper crate that will make our life easier.
add ffi_helpers to the dependencies

ffi_helpers = "0.2"

it provides useful functions and macros that could help us to handle errors, exposing last_error_length and error_message_utf8 so in Dart's side we could get readable error messages.

macro_rules! error {
    ($result:expr) => {
        error!($result, 0);
    };
    ($result:expr, $error:expr) => {
        match $result {
            Ok(value) => value,
            Err(e) => {
                ffi_helpers::update_last_error(e);
                return $error;
            }
        }
    };
}

macro_rules! cstr {
    ($ptr:expr) => {
        cstr!($ptr, 0);
    };
    ($ptr:expr, $error:expr) => {{
        null_pointer_check!($ptr);
        error!(unsafe { CStr::from_ptr($ptr).to_str() }, $error)
    }};
}


#[no_mangle]
pub unsafe extern "C" fn last_error_length() -> i32 {
    ffi_helpers::error_handling::last_error_length()
}

#[no_mangle]
pub unsafe extern "C" fn error_message_utf8(buf: *mut raw::c_char, length: i32) -> i32 {
    ffi_helpers::error_handling::error_message_utf8(buf, length)
}

Exposing `load_page` function 🔑

Now comes the issue, how we would expose async function? well C/Dart has no idea how Rust's async works.

One way could be writing a normal non-async function as a wrapper around the async function and call block_on to get the result out from the async fn, as you see the function name says it all, we will block that thread only to run that async task, so what the benefit in the first place?

There is another way, what about using callbacks?!
Good, that would be an option and use javascript Promise style to handle the async tasks by passing two callbacks one called on the success and the other one called on err.

Theoretically, that would work, but

As I said before, Dart is designed to be single-threaded so you can't just call the callback from other threads it basically will break Dart semantics (you could read more here).

So what would be the solution then?
What if I told you that we could communicate between Dart and Rust using an Isolate 😦.

Every new Isolate get a SendPort and ReceivePort these are used to communicate with the Isolate and you only need SendPort to send messages to the Isolate, also every SendPort have what is called a NativePort this the underlying number of that port.

Just a small note, the term of Port here is not related at all to a Networking Port or something like that, it is just numbers used as Keys for a Map implemented internally in Dart VM, think of them as a Handle to that Isolate

What if we used these numbers to send our async fn result on them from any thread we want, is that possible? YES

Enter: Allo Isolate 📞

allo-isolate is a crate we built at Sunshine to help us Running Multithreaded Rust along with Dart VM (in isolate) 🌀
It only needs a Port of that isolate and you could send any message on it from your Rust code.

Under the hood, it uses the Dart API Dart_PostCObject function and it can convert most (almost all?) of Rust types to Dart types, see the docs for more information.

Allo (pronounced Hello, without the H) usually used in communication over the phone in some languages like Arabic :).

Back to our example, we now know how to expose the load_page function lets use allo-isolate

allo-isolate = "0.1"

use it

use allo_isolate::Isolate;

...

#[no_mangle]
pub extern "C" fn load_page(port: i64, url: *const raw::c_char) -> i32 {
    // get a ref to the runtime
    let rt = runtime!();
    let url = cstr!(url);
    rt.spawn(async move {
        // load the page and get the result back
        let result = scrap::load_page(url).await;
        // make a ref to an isolate using it's port
        let isolate = Isolate::new(port);
        // and sent it the `Rust's` result
        // no need to convert anything :)
        isolate.post(result);
    });
    1
}

Great, we are ready to go, after generating our binding.h and building everything for iOS and Android we need also to write the Dart side FFI... ugh not again 🤦‍♂️

What if I told you that you could now generate your Dart FFI binding from a C Header File?

dart-bindgen: a new way to write Dart FFI

dart-bindgen is a tool for generating Dart FFI bindings to C Header file, it comes as a CLI and a Library.

and we going to use it to help us generate the FFI, just add it to your build-dependencies

[build-dependencies]
cbindgen = "0.14.2" # Rust -> C header file
dart-bindgen = "0.1" # C header file -> Dart FFI

and in our build.rs

use dart_bindgen::{config::*, Codegen};

fn main() {
    ...
    let config = DynamicLibraryConfig {
        ios: DynamicLibraryCreationMode::Executable.into(),
        android: DynamicLibraryCreationMode::open("libscrap_ffi.so").into(),
        ..Default::default()
    };
    // load the c header file, with config and lib name
    let codegen = Codegen::builder()
        .with_src_header("binding.h")
        .with_lib_name("libscrap")
        .with_config(config)
        .build()
        .unwrap();
    // generate the dart code and get the bindings back
    let bindings = codegen.generate().unwrap();
    // write the bindings to your dart package
    // and start using it to write your own high level abstraction.
    bindings
        .write_to_file("../../packages/scrap_ffi/lib/ffi.dart")
        .unwrap();
}

So far so good, let's write our higher-level abstraction over the ffi.dart file.

// packages/scrap_ffi/lib/scrap.dart

import 'dart:async';
import 'dart:ffi';
import 'package:ffi/ffi.dart';
// isolate package help us creating isolate and getting the port back easily.
import 'package:isolate/ports.dart';

import 'ffi.dart' as native;

class Scrap {
  // this only should be called once at the start up.
  static setup() {
    // give rust `allo-isolate` package a ref to the `NativeApi.postCObject` function.
    native.store_dart_post_cobject(NativeApi.postCObject);
    print("Scrap Setup Done");
  }

  Future<String> loadPage(String url) {
    var urlPointer = Utf8.toUtf8(url);
    final completer = Completer<String>();
    // Create a SendPort that accepts only one message.
    final sendPort = singleCompletePort(completer);
    final res = native.load_page(
      sendPort.nativePort,
      urlPointer,
    );
    if (res != 1) {
      _throwError();
    }
    return completer.future;
  }

  void _throwError() {
    final length = native.last_error_length();
    final Pointer<Utf8> message = allocate(count: length);
    native.error_message_utf8(message, length);
    final error = Utf8.fromUtf8(message);
    print(error);
    throw error;
  }
}

in our flutter application, we could use that package as so:

...
class _MyHomePageState extends State<MyHomePage> {

  ...
  Scrap scrap;
  @override
  void initState() {
    super.initState();
    scrap = Scrap();
    Scrap.setup();
  }  

  // somewhere in your app
  final html = await scrap.loadPage('https://www.rust-lang.org/');

That's it, Fire up Android Emulator or iOS Simulator and Run:

$ cargo make # build all rust packages for iOS and Android

Then,

$ flutter run # 🔥

All the Code is available in flutterust which is only clone and Hack 😀.

Stay tuned for other Hacks in Dart and Rust, you could follow me on twitter and github

Thank you 💚.

Convenient substrate identity solution

David Craven — Sat, 06 Jun 2020 15:08:04 +0000

For our sunshine-dao we wanted a convenient easy and secure identity solution, that doesn't require centralized key management.

The requirements were that we wanted the users to have per device keys they didn't have to copy around and that they should be able to associate their github account, so that users can bring their reputation to our bounty management platform.

I already heard about keybase and had briefly used it. When I set out to build substrate-identity I played around with it and read their technical documentation. A really interesting one for me was local-key-security. I had written a quick and dirty keystore before for the sunshine-flutter, but that had a lot of interesting things in it. I decided to reimplement our keystore based on the keybase keystore. A particularly interesting feature is that it will allow password changes on one device to reencrypt the keys using the public xor of the old password with the new one.

Setup your account

Set your device key to //Alice:

cli-identity key set --suri //Alice
Please enter a new password (8+ characters):

Your device id is 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY
Your user id is 0

Add a paper backup for your account:

cli-identity device paperkey
Generating a new paper key.
Here is your secret paper key phrase:

mandate robust earth scan shrimp second pipe pitch eternal snap glare tooth
bean crucial river bar crash nice sorry laundry oppose filter aunt swear

Write it down and keep somewhere safe.

and list your device keys:

cli-identity device list
5Fe8Da8o2TQY6heaopRA9Zs2dpiJ2GFtvWThnd89uxYEXK1q
5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY

Prove your online identity

Add a new online identity:

cli-identity id prove dvc94ch@github
Claiming dvc94ch@github...
Please *publicly* post the following Gist, and name it 'substrate-identity-proof.md'.

### substrate identity proof

I hereby claim:

  * I am dvc94ch on github.
  * I am 0 on the substrate chain with genesis hash mzyTJZVm7IXDUBeZwyWk6rG1YGIt8BQnNzrshKJCalYI.
  * I have a public key 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY valid at block with hash mfBFseDYNX31Poqei8A/9teYmxJIj4PFROoKLKEPaStE.

To claim this, I am signing this object:

{"block":[124,17,108,120,54,13,95,125,79,162,167,162,240,15,253,181,230,38,196,146,35,224,241,81,58,130,139,40,67,218,74,209],"body":{"Ownership":[{"Github":["dvc94ch"]}]},"ctime":1591448931056,"expire_in":18446744073709551615,"genesis":[207,36,201,101,89,187,33,112,212,5,230,112,201,105,58,172,109,88,24,139,124,5,9,205,206,187,33,40,144,154,149,130],"prev":null,"public":"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY","seqno":1,"uid":0}

with the key 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, yielding the signature:

mAU6Gon1dqctnS/zPKHd9gWFvEJBqADvgYQy0OFuamA5CwVQk7papR0xBq8DijRqSXVGpJtNFmy7aYJk5cGLxv4c

And finally, I am proving ownership of the github account by posting this as a gist.

and list your identities:

cli-identity id list
Your user id is 0
dvc94ch@github https://gist.github.com/da8bbf9c69976a3d750e2c433126852b

Receive payments to your public identity

Transfer a balance from //Bob to dvc94ch@github:

cli-identity --path /tmp/bob key set --suri //Bob
Please enter a new password (8+ characters):

Your device id is 5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty
Your user id is 1

make sure ipfs is running so that //Bob can fetch //Alice's identity:

cli-identity run

finally make the transfer:

cli-identity --path /tmp/bob wallet transfer dvc94ch@github 10000
transfered 10000 to 5Fe8Da8o2TQY6heaopRA9Zs2dpiJ2GFtvWThnd89uxYEXK1q

Dart Meets Rust: a match made in heaven ✨

shekohex — Sat, 06 Jun 2020 13:38:25 +0000

A small piece of Dart

Dart is a client-optimized language for fast apps on any platform, it make it easy to build the UI of your application and it is quite nice language to work with, it the language used by Flutter Framework, Flutter is Google’s UI toolkit for building beautiful, natively compiled applications for mobile, web, and desktop from a single codebase.

Enter Rust

Rust is blazingly fast and memory-efficient, with no runtime or garbage collector, it can power performance-critical services, run on embedded devices, and easily integrate with other languages.

We are using both Rust and Dart (in Flutter) at Sunshine to enable open-source grant initiatives to easily operate in an on-chain ecosystem.

Almost all of our Code is written in Rust, that's why we needed to think about using the same code and the same logic in our client-side application, but How?

Well, let's see what options we have here

Using Flutter Platform Channels

Flutter Platform channels is a flexible system that allows you to call platform-specific APIs whether available in Kotlin or Java code on Android, or in Swift or Objective-C code on iOS.
this way, we will have to first bind our rust code to Java (for Android), Swift (for iOS), and WASM for the Web, but that would be an over complicated, and maybe that could result a performance issues in the future. Here is a simple graph to get an idea of how it looks like:

but as you could see, there is a lot of overhead involved here and data serialization/deserialization is very costly at runtime, so what else we could do?

FFI, break boundaries

as Wikipedia says: A foreign function interface (FFI) is a mechanism by which a program written in one programming language can call routines or make use of services written in another.

hmmm, interesting let's see what we could do, dose Dart support FFI?
Yes!, actually FFI introduced in Dart 2.5 quite recently at the end of last year, so it is still under active development, but quite stable.

After Playing around with FFI Examples with Dart, I started to work on flutterust a simple template to show how to use Flutter/Dart with Rust through FFI.

The simple idea here is that we build our rust code for all supported targets then build a Flutter Package that uses these targets.

And Here is the benefits of using the FFI Method here

No Swift/Kotlin wrappers
No message passing
No async/await on Dart
Write once, use everywhere
No garbage collection
No need to export aar bundles or .framework's

So, it would be like this:

that is so cool, here is a simple example

Learning How to count!

we are going to use the same flutter hello world example, but instead of doing the logic (incrementing the counter) in the Dart side, we going to do it in the Rust side.

Our Project Sturcutre:

.
├── android
├── ios
├── lib                     <- The Flutter App Code
├── native                  <- Containes all the Rust Code
│   ├── adder
│   └── adder-ffi
├── packages                <- Containes all the Dart Packages that bind to the Rust Code
│   └── adder_ffi
├── target                  <- The compiled rust code for every arch
│   ├── aarch64-apple-ios
│   ├── aarch64-linux-android
│   ├── armv7-linux-androideabi
│   ├── debug
│   ├── i686-linux-android
│   ├── universal
│   ├── x86_64-apple-ios
│   └── x86_64-linux-android
└── test

The Rust Side

Start by creating a Cargo Workspace, so we add a simple Cargo.toml to the root of our Flutter app

[workspace]
members = ["native/*"]

[profile.release]
lto = true
codegen-units = 1
debug = true # turn it off if you want.

Create our simple adder package

$ cargo new --lib native/adder

and let's write some code

pub fn add(a: i64, b: i64) -> i64 {
    a.wrapping_add(b)
}

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(super::add(2, 2), 4);
    }
}

boring, isn't it? 🥱

let's show the world our new add function :)

$ cargo new --lib native/adder-ffi

and don't forget to change it's type in the native/adder-ffi/Cargo.toml

[lib]
name = "adder_ffi"
crate-type = ["cdylib", "staticlib"]

[dependencies]
adder = { path = "../adder" }

// lib.rs

#[no_mangle]
pub extern "C" fn add(a: i64, b: i64) -> i64 {
    adder::add(a, b)
}

Nice, but how to compile our code for the mobile?
Well, it is a bit complicated. We could use cargo directly and it would of course work, but we need to configure a lot of other things, so we will relay on other tools that would do it for us like cargo-lipo and cargo-ndk.

After Compiling our rust code to all of these platforms:

aarch64-apple-ios
aarch64-linux-android
armv7-linux-androideabi
i686-linux-android
x86_64-apple-ios
x86_64-linux-android

we are ready to go to next step, that we will copy our compiled code to specific locations

start first by generating a flutter plugin named after our rust crate:

$ flutter create --template=plugin packages/adder

target/universal/debug/libadder_ffi.a -> packages/adder/ios/libadder_ffi.a
target/aarch64-linux-android/debug/libadder_ffi.so -> packages/adder/android/src/main/jniLibs/arm64-v8a/libadder_ffi.so
...
...other android libs

Are we ready yet? well, technicllay yes, but Xcode has another thing to do like writing a C Header file for our FFI for iOS, if you developing on a macOS you should do these steps here other than that you are ready to go to the next step, writing a Flutter Package to our rust lib.

The Dart Side

so back to Dart, in our generated flutter plugin, we will define how our rust function look like (the type definition) in dart code

import 'dart:ffi';

// For C/Rust
typedef add_func = Int64 Function(Int64 a, Int64 b);
// For Dart
typedef Add = int Function(int a, int b);

and we need a function that loads our rust lib depending on the platform like iOS/Android or Linux/macOS or whatever it is.

import 'dart:io' show Platform;


DynamicLibrary load({String basePath = ''}) {
  if (Platform.isAndroid || Platform.isLinux) {
    return DynamicLibrary.open('${basePath}libadder_ffi.so');
  } else if (Platform.isIOS) {
    // iOS is statically linked, so it is the same as the current process
    return DynamicLibrary.process();
  } else if (Platform.isMacOS) {
    return DynamicLibrary.open('${basePath}libadder_ffi.dylib');
  } else if (Platform.isWindows) {
    return DynamicLibrary.open('${basePath}libadder_ffi.dll');
  } else {
    throw NotSupportedPlatform('${Platform.operatingSystem} is not supported!');
  }
}

class NotSupportedPlatform implements Exception {
  NotSupportedPlatform(String s);
}

and finally create a simple Class that holds our ffi function

class Adder {
  static DynamicLibrary _lib;

  Adder() {
    if (_lib != null) return;
    // for debugging and tests
    if (Platform.isWindows || Platform.isLinux || Platform.isMacOS) {
      _lib = load(basePath: '../../../target/debug/');
    } else {
      _lib = load();
    }
  }
}

and here is the add method

int add(int a, int b) {
    // get a function pointer to the symbol called `add`
    final addPointer = _lib.lookup<NativeFunction<add_func>>('add');
    // and use it as a function
    final sum = addPointer.asFunction<Add>();
    return sum(a, b);
}

so far so good, lets use it in our Flutter app

in the pubspec.yaml of the app, add our adder package under dependencies

adder:
    path: packages/adder_ffi

and in lib/main.dart change the logic of the _incrementCounter method to use our rust logic

import 'package:adder/adder.dart';

// in the `MyHomePage` add 
final adder = Adder();

// and latter in `_MyHomePageState` replace
...
 void _incrementCounter() {
    setState(() {
      _counter = widget.adder.add(_counter, 1);
    });
  }
...

and fire up the Flutter App on Android Emulator or iOS Simulator and Test it 🔥.

phew ..

but we found it is so boring to do that, and especially when it comes to using other build systems like Xcode and Android NDK toolchain and linking everything together 🤦‍♂️. That's why we tried to automate everything, but we need something that is easy to use, cross platform, and CI friendly.

Cargo-make to rescue 🚀

cargo-make is a cross-platform task runner and build tool built in Rust, it is really an Amazing tool to use, it helps you write your workflow in a simple set of tasks, and it has a lot of other cool features like it is easy to add inline scripts in it and a lot more.
you could see how we using it at sunshine-flutter.

That's it, I hope it helped to understand how Dart FFI and Rust works together.

Next Up, How to handle async Rust and Dart FFI
I will leave this to a next blog post, pretty soon :)

For now, you could see that I start hacking on the scrap package that created to demonstrate how we could integrate async Rust with Dart.

Other Intersting Rust + Mobile FFI Development

Introducing Sunshine

Amar Singh — Sat, 30 May 2020 02:30:30 +0000

Hello world! We are building a bounty platform with Substrate called Sunshine. Our project is supported by a grant from the Web3 Foundation ❤

The primary goal of Sunshine is to provide censorship resistant infrastructure for foundations to post bounties, accept applications, and supervise ongoing contracts.

We hope this infrastructure will make it easy for developers to form teams, pursue bounties, and disband after or even during the completion of a bounty. Our platform is built for people so we plan to support all of the expected unexpected things that happen.

Our motivation is empathy for open source developers, especially those in places with less reliable legal infrastructure.

The most basic idea is an expression for a team's ownership on-chain.

team: Vec<(AccountId, Shares)>

Shares is generic in this example, maybe it's a u32, but, more importantly, each tuple in the vector serves as a measure of ownership relative to other members of the team. The payout for each milestone of the grant can enforce this ownership structure to restrict withdrawals beyond a certain size for each team member (based on the member's proportion of ownership).

Of course, financial relationships are often much more nuanced than this 2D ownership structure. We'll be studying VC mechanisms for hints at how to express more complex ownership agreements.

Sunshine Mobile Client

Our first significant deliverable will be the Sunshine Mobile Client. We intend to finish this work by the end of June 2020.
The Sunshine Mobile Client will be in Rust and it will share a sled db between

ipfs-embed node
substrate-subxt light client

The Sunshine Mobile Client constructs transactions to form coherent calls to the nodes on the Sunshine Network. Each of these nodes will run sunshine-node instances.

The Sunshine Mobile Client also communicates directly with our Flutter interface via native Dart-Rust FFI (so no dependency on polkadot-js).

If any of that sounds interesting, check out our github, the rust-ipfs github, or the previous links to see the code. There is also a list of interesting runtime features here.

In Conclusion

By enforcing the terms of agreement for bounty contracts on-chain, Sunshine aspires to encourage power structures that decrease dependence on individuals and/or single-purpose entities.

We hope our work enables collaborative funding models that respect and appreciate the shared interests of people that operate in the same domain at different corporations.

The long-term vision is a system that uses comprehensive stakeholder signal to efficiently allocate resources to the projects that add the most value to an ecosystem.

Chasing this vision will involve designing and building configurable group governance, resilient to capture. It's an ideal and we're excited to work on it!

If any of this sounds interesting and you want to get involved, feel free to pop into our gitter.