DEV Community: Roberto Huertas

Send the logs of your Shuttle-powered backend to Datadog

Roberto Huertas — Fri, 29 Sep 2023 22:00:00 +0000

Learn how to send logs to Datadog from your Shuttle powered backend.

Some words about observability

As we all know, being able to ‘see’ what’s going on in our services can be critical in many ways. We can easily find bugs or identify undesired behaviors, and it’s certainly an invaluable tool at our disposal.

Observability, in software, refers to the ability to understand the state of a system and its behavior by collecting, analyzing, and presenting data about its various components and interactions. This enables engineers to diagnose and resolve issues and make informed decisions about system health and performance.

Observability is critical for ensuring the reliability, scalability, and performance of modern systems, and is becoming increasingly important as software continues to play a larger role in our daily lives.

Fortunately, in the Rust ecosystem, we have Tokio Tracing which is a powerful framework for instrumenting Rust programs to collect structured, event-based diagnostic information. It provides a convenient and flexible API for collecting and viewing traces of events in your application and you can easily add context and structure to your traces, making it easier to identify bottlenecks and debug issues.

Shuttle logs

A few months ago, I wrote a post about Shuttle, where I explained how ridiculously easy it is to deploy a Rust backend to the cloud by using their CLI tool.

Shuttle is still in beta, and although its observability features are not really polished yet, they offer support for Tokio Tracing and a way to view logs by using their CLI tool.

By simply running cargo shuttle logs --follow, you will be able to see something like this:

This is great for simple applications, but what if you want to send your logs to a more powerful tool like Datadog? Well, in this post, I’ll show you how to do it.

Datadog

Datadog is a monitoring and observability platform that provides a single pane of glass for your infrastructure and applications. It is a cloud-based service that allows you to collect, aggregate and analyze your data, and it is extremely powerful.

As a disclaimer, I must say that I’m currently working at Datadog, so I’m a bit biased, but I’m also a huge fan of the product and I think it’s a great tool for developers 😅.

Most of the time, the easiest way to send anything to the Datadog platform is by using the Datadog Agent, but in this case, as we cannot install it in any way, we will use a small library I created for the occasion called dd-tracing-layer, which happens to be using the Datadog HTTP API under the hood to send logs to the Datadog platform.

How to use tracing with Shuttle

If we check the Shuttle documentation, we can read this:

Shuttle will record anything your application writes to stdout, e.g. a tracing or log crate configured to write to stdout, or simply println!. By default, Shuttle will set up a global tracing subscriber behind the scenes.

// [...]
use tracing::info;

#[shuttle_runtime::main]
async fn axum(#[shuttle_shared_db::Postgres] pool: PgPool) -> ShuttleAxum { 
    info!("Running database migration");
    pool.execute(include_str!("../schema.sql"))
        .await
        .map_err(CustomError::new)?;

    // [...]
}

So, as you can see, it seems that the Shuttle macro is already instantiating and initializing a tracing subscriber for us.

This is pretty convenient for most of the simple cases, but unfortunately, it’s not enough for our purposes.

Ideally, if we had access to the underlying infrastructure, we could probably install the Datadog Agent and configure it to send our logs directly to Datadog, or even use AWS Lambda functions or Azure Event Hub + Azure Functions in case we were facing some specific cloud scenarios.

You can check the Datadog docs for log collection and integrations if you want to learn more.

Those solutions are generally great because they allow us to remove the burden of sending our logs to Datadog from our application, thus becoming the responsibility of the platform itself.

If we could do something like that with Shuttle, it would be great. But, as we just mentioned, in the case of Shuttle, we don’t have access to the underlying infrastructure, so we need to find a way to send our logs to Datadog from our application.

And that’s what we are going to try to do in this post.

Getting access to the subscriber

So, the basic idea is to add a new tracing layer to the subscriber which will be responsible for sending our logs to Datadog.

But for that, we’ll need to get access to the subscriber instance prior to its initialization, and it turns out that Shuttle provides a way to do that just by disabling the default features on shuttle-runtime crate.

shuttle-runtime = { version = "*" default-features = false }

Creating our project

As a walkthrough, we are going to create a new Shuttle project from scratch.

The idea is to build a simple REST API using Axum and send our logs to Datadog using the dd-tracing-layer crate.

Although I’m going to describe all the steps you need to take to make this work, you can see the final state of the project in this GitHub repository.

Feel free to use it as a reference.

Initializing the project

First of all, we need to create a new Shuttle project. You can do that by using the Shuttle CLI:

cargo shuttle init --template axum

Follow the instructions and you should have a new project ready to go. I called mine shuttle-datadog-logs, but use the name you want.

Adding some dependencies

In our example, we are going to be using Shuttle Secrets, Tokio Tracing and dd-tracing-layer.

Make sure you have the following dependencies in your Cargo.toml file:

[dependencies]
axum = "0.6"
shuttle-axum = "0.27.0"
shuttle-runtime = { version = "0.27.0", default-features = false }
tokio = "1"
# tracing
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json", "time"] }
dd-tracing-layer = "0.1"
# secrets
shuttle-secrets = "0.27.0"

Instrumenting a little bit the default project

Now that we have our dependencies ready, we can start instrumenting our project a little bit.

Note that we have added the #[instrument] macro to the hello_world function and added a tracing::info! and a tracing::debug! log to it. We have also added an info log to the axum function.

// [...]
use tracing::instrument;

#[instrument]
async fn hello_world() -> &'static str {
    tracing::info!("Saying hello");
    tracing::debug!("Saying hello for debug level only");
    "Hello, world!"
}

#[shuttle_runtime::main]
async fn axum() -> shuttle_axum::ShuttleAxum {
    let router = Router::new().route("/", get(hello_world));
    tracing::info!("Starting axum service");
    Ok(router.into())
}

At this point, if you try to run the project locally by using the shuttle run command, you should see none of our logs.

That’s ok, as we haven’t initialized a tracing subscriber yet.

Adding our tracing subscriber

The first thing we’re going to do is to add a tracing subscriber to our application.

Then we will add several layers to it:

EnvFilter layer to set the tracing level according to a variable’s value.
Format layer to format the logs. We will be using JSON format.
Datadog Tracing layer to send our logs to Datadog.

Apart from that, we’re also going to add support for Shuttle Secrets.

Let’s do it! Make sure your axum function looks like this:

use axum::{routing::get, Router};
use dd_tracing_layer::{DatadogOptions, Region};
use shuttle_secrets::SecretStore;
use tracing::instrument;
use tracing_subscriber::prelude::*;

// version of our app to be sent to Datadog
const VERSION: &'static str = "version:0.1.0";

// [...]

#[shuttle_runtime::main]
async fn axum(#[shuttle_secrets::Secrets] secret_store: SecretStore) -> shuttle_axum::ShuttleAxum {
    // getting the Datadog Key from the secrets
    let dd_api_key = secret_store
        .get("DD_API_KEY")
        .expect("DD_API_KEY not found");

    // getting the Datadog tags from the secrets
    let tags = secret_store
        .get("DD_TAGS")
        .map(|tags| format!("{},{}", tags, VERSION))
        .unwrap_or(VERSION.to_string());

    // getting the log level from the secrets and defaulting to info
    let log_level = secret_store.get("LOG_LEVEL").unwrap_or("INFO".to_string());

    // datadog tracing layer
    let dd_layer = dd_tracing_layer::create(
        DatadogOptions::new(
            // first parameter is the name of the service
            "shuttle-datadog-logs",
            // this is the Datadog API Key
            dd_api_key,
        )
        // this is the default, so it can be omitted
        .with_region(Region::US1)
        // adding some optional tags
        .with_tags(tags),
    );

    // filter layer
    let filter_layer =
        tracing_subscriber::EnvFilter::try_new(log_level).expect("failed to set log level");

    // format layer
    let fmt_layer = tracing_subscriber::fmt::layer()
        .with_ansi(true)
        .with_timer(tracing_subscriber::fmt::time::UtcTime::rfc_3339())
        .json()
        .flatten_event(true)
        .with_target(true)
        .with_span_list(true);

    // starting the tracing subscriber
    tracing_subscriber::registry()
        .with(filter_layer)
        .with(fmt_layer)
        .with(dd_layer)
        .init();

    // starting the server
    let router = Router::new().route("/", get(hello_world));
    tracing::info!("Starting axum service");
    Ok(router. Into())
}

There are many things going on in this code, so take your time to go through it.

Secrets

Before running our project, there's still a thing we have to deal with: secrets.

As you can see in the code above, we are using the Shuttle Secrets crate to get the Datadog API key, the tags and the log level.

Shuttle Secrets relies on having a Secrets.toml file in the root of our project containing all the secrets, and it also supports having a Secrets.dev.toml file for local development. You can learn more about this convention in the Shuttle Secrets documentation.

So, let's create two files in the root of our project:

Secrets.dev.toml

DD_API_KEY = "21695c1b35156511441c0d3ace5943f4"
DD_TAGS = "env:dev,service:shutle-datadog-logs"
# setting info as the default log level, but debug for our project
LOG_LEVEL = "INFO,shuttle_datadog_logs=DEBUG"

and Secrets.toml

DD_API_KEY = "21695c1b35156511441c0d3ace5943f4"
DD_TAGS = "env:prod,service:shutle-datadog-logs"
LOG_LEVEL = "INFO"

Remember to add these files to your .gitignore file!

Running the project

Now, run cargo shuttle run and go to http://localhost:8000 in your browser to see our "Hello, world!" message.

Alternatively, you can also use curl to test the endpoint:

curl -i http://localhost:8080

You should be able to see the logs in your terminal now.

But remember... this endpoint was instrumented! So, if everything went well, we should be able to see the logs in Datadog.

Let’s check it out! 👀

It works! 🎉

Conclusion

As you can see, it’s pretty easy to send your logs to Datadog from your Shuttle powered backend.

Again, you can see the full code in this GitHub repository.

I hope you’ve enjoyed it! 😁

Send the logs of your Shuttle-powered backend to Datadog (outdated)

Roberto Huertas — Tue, 31 Jan 2023 23:00:00 +0000

IMPORTANT: This post is OUTDATED. Visit the reviewed version.

Learn how to send logs to Datadog from your Shuttle powered backend.

Some words about observability

Shuttle logs

A few weeks ago, I wrote a post about Shuttle, where I explained how ridiculously easy it is to deploy a Rust backend to the cloud by using their CLI tool.

Shuttle is still in alpha, and although its observability features are not really polished yet, they offer support for Tokio Tracing and a way to view logs by using their CLI tool.

By simply running cargo shuttle logs --follow, you will be able to see something like this:

This is great for simple applications, but what if you want to send your logs to a more powerful tool like Datadog? Well, in this post, I’ll show you how to do it.

Datadog

As a disclaimer, I must say that I’m currently working at Datadog, so I’m a bit biased, but I’m also a huge fan of the product and I think it’s a great tool for developers 😅.

How to use tracing with Shuttle

If we check the Shuttle documentation, we can read this:

When you build an app with the #[shuttle_service::main] macro, a global subscriber will be created and installed behind the scenes. This means you can skip this step when implementing tracing in your application, all you have to do is add tracing as a dependency in your Cargo.toml, and you’re good to go!

// [...]
use tracing::info;

#[shuttle_service::main]
async fn axum(#[shuttle_shared_db::Postgres] pool: PgPool) -> ShuttleAxum { 
    info!("Running database migration");
    pool.execute(include_str!("../schema.sql"))
        .await
        .map_err(CustomError::new)?;

    // [...]
}

So, as you can see, it seems that the Shuttle macro is already instantiating and initializing a tracing subscriber for us.

This is pretty convenient for most of the simple cases , but unfortunately, it’s not enough for our purposes.

You can check the Datadog docs for log collection and integrations if you want to learn more.

Those solutions are generally great because they allow us to remove the burden of sending our logs to Datadog from our application, thus becoming the responsibility of the platform itself.

If we could do something like that with Shuttle, it would be great. Probably, it would be still nice to be able to customize the subscriber before its initialization in order to filter out some logs or add some context to them, but at least we wouldn’t have to worry about sending our logs to Datadog from our application.

But, as we just mentioned, in the case of Shuttle, we don’t have access to the underlying infrastructure, so we need to find a way to send our logs to Datadog from our application, and that’s what we are going to try to do in this post.

Getting access to the subscriber

So, the basic idea is to add a new tracing layer to the subscriber which will be responsible for sending our logs to Datadog.

But for that, we’ll need to get access to the subscriber instance prior to its initialization, and, at the moment of writing this post, there doesn’t seem to be a way to make that possible out of the box.

So… it seems that our only solution is to get rid of the #[shuttle_service::main] macro and go macro-less!

Walking on the wild side

Fear not! It’s not as bad as it sounds. Of course, this is a workaround, and hopefully the Shuttle team will provide a better solution for this in the future, but for now, this is the only solution we have.

As a walkthrough, we are going to create a new Shuttle project from scratch.

The idea is to build a simple REST API using Axum and send our logs to Datadog using the dd-tracing-layer crate.

This backend will also have a simple PostgreSQL database, which will be managed by using the Shuttle Shared Database plugin and will also serve some static files.

Although I’m going to describe all the steps you need to take to make this work, you can see the final state of the project in this GitHub repository. Feel free to use it as a reference.

Creating a new project

First of all, we need to create a new Shuttle project. You can do that by using the Shuttle CLI:

cargo shuttle init --axum

Follow the instructions and you should have a new project ready to go. I called mine shuttle-datadog-logs but use the name you want.

Replacing the Shuttle macro

Once you have your project ready, you need to remove the #[shuttle_service::main] macro.

Find the axum function in the src/lib.rs file and remove the #[shuttle_service::main] macro from it.

In case you are trying to instrument an already existing project and you’re using some of the Shuttle features/plugins, you’ll need to remove the macros associated to them in the axum function, too.

So make sure you get rid of these macros if already present:

#[shuttle_shared_db::Postgres]
#[shuttle_secrets::Secrets]
#[shuttle_static_folder::StaticFolder]

Don’t worry, I’ll explain later how to inject those dependencies.

Adding some dependencies

In our example, we are going to be using these Shuttle features:

So, make sure you have the following dependencies in your Cargo.toml file:

# secrets
shuttle-secrets = "0.9.0"
# database
shuttle-shared-db = { version = "0.9.0", features = ["postgres"] }
sqlx = { version = "0.6.2", features = ["runtime-tokio-native-tls","postgres"] }
# static files
shuttle-static-folder = "0.9.0"
tower-http = { version = "0.3", features = ["fs"] }

Note that these are optional dependencies and depending on the kind of project you’re working on you may not need them at all. I just added them to show you how to inject them in the axum function in case you need them.

Aside from that, as we’re going to deal with tracing, we’ll need to add the following dependencies:

tracing = "0.1"
dd-tracing-layer = "0.1"

Instrumenting a little bit the default project

Now that we have our dependencies ready, we can start instrumenting our project a little bit.

Note that we have added the #[instrument] macro to the hello_world function and added a tracing::info! log to it.

use sync_wrapper::SyncWrapper;
use tracing::instrument;

#[instrument]
async fn hello_world() -> &'static str {
    tracing::info!("Saying hello");
    "Hello, world!"
}

async fn axum(
    _pool: PgPool,
    _secret_store: SecretStore,
    _static_folder: PathBuf,
) -> shuttle_service::ShuttleAxum {
    let router = Router::new().route("/hello", get(hello_world));
    let sync_wrapper = SyncWrapper::new(router);
    Ok(sync_wrapper)
}

At this point, if you try to run the project locally by using the shuttle run command, you should see the following output:

Providing the Shuttle entrypoint

As we have removed the #[shuttle_service::main] macro, we need to provide our own entrypoint.

We will be adding two new functions to our src/lib.rs file:

_create_service: This is the entrypoint for our application. It will be called by the Shuttle runtime and it needs to be called exactly like this.
main: This is the function that will be called by the _create_service function. It will be the one that will start our application and instantiate all the dependencies we may need in our axum function (db, secrets, static files folder, etc.).

Just copy and paste the following code in the src/lib.rs file:

async fn main(
    factory: &mut dyn shuttle_service::Factory,
    runtime: &shuttle_service::Runtime,
    logger: shuttle_service::Logger,
) -> Result<Box<dyn shuttle_service::Service>, shuttle_service::Error> {
    todo!("Instantiate dependencies and start the axum service");
}

#[no_mangle]
pub extern "C" fn _create_service() -> *mut shuttle_service::Bootstrapper {
    use shuttle_service::Context;
    let bootstrapper = shuttle_service::Bootstrapper::new(
        |factory, runtime, logger| Box::pin(main(factory, runtime, logger)),
        |srv, addr, runtime| {
            runtime.spawn(async move {
                srv.bind(addr)
                    .await
                    .context("failed to bind service")
                    .map_err(Into::into)
            })
        },
        shuttle_service::Runtime::new().unwrap(),
    );
    let boxed = Box::new(bootstrapper);
    Box::into_raw(boxed)
}

Now, we’re going to implement the main function.

Adding our tracing subscriber

The first thing we’re going to do is to add a tracing subscriber to our application.

Then we will add the Datadog Tracing layer to our subscriber along with the Shuttle Logger layer that we’ll get from the main function arguments.

Let’s do it! Add this code to the main function:

use shuttle_service::tracing_subscriber::prelude::*;
use dd_tracing_layer::{DatadogOptions, Region};

// datadog tracing layer
let dd_layer = dd_tracing_layer::create(
    DatadogOptions::new(
        // first parameter is the name of the service
        "shuttle-datadog-logs",
        // this is the Datadog API Key
        "21695c1b35156511441c0d3ace5943f4",
    )
    // this is the default, so it can be omitted
    .with_region(Region::US1)
    // adding some optional tags
    .with_tags("env:dev, version:0.1.0"),
);

// feel free to change this implementation
// this will set the tracing level according to the RUST_LOG env variable
// and will use INFO if it's not set
// NOTE that in production, this env var will be set to DEBUG by Shuttle
// so you could skip the env var part and just use the level that you want
let filter_layer =
    shuttle_service::tracing_subscriber::EnvFilter::try_from_default_env()
        .or_else(|_| shuttle_service::tracing_subscriber::EnvFilter::try_new("INFO"))
        .unwrap();

// starting the tracing subscriber
runtime
    .spawn_blocking(move || {
        shuttle_service::tracing_subscriber::registry()
            .with(filter_layer)
            // note that we added the datadog layer here
            .with(dd_layer)
            // and the shuttle logger layer
            .with(logger)
            .init();
    })
    .await
    .map_err(|e| {
        if e.is_panic() {
            let mes = e
                .into_panic()
                .downcast_ref::<&str>()
                .map(|x| x.to_string())
                .unwrap_or_else(|| "panicked setting logger".to_string());
            shuttle_service::Error::BuildPanic(mes)
        } else {
            shuttle_service::Error::Custom(
                shuttle_service::error::CustomError::new(e).context("failed to set logger"),
            )
        }
    })?;

Instantiating the dependencies

Now that we have our tracing subscriber ready, we can start instantiating our dependencies.

Let’s add this piece of code to the main function. As you can see, it’s pretty straightforward:

// shared database
let pool = shuttle_shared_db::Postgres::new()
    .build(factory, runtime)
    .await?;

// shuttle secret store
let secret_store = shuttle_secrets::Secrets::new()
    .build(factory, runtime)
    .await?;

// shuttle static folder support
let static_folder = shuttle_static_folder::StaticFolder::new()
    .build(factory, runtime)
    .await?;

Starting the axum service

Finally, let’s start our axum service.

This is the final part of the main function:

// starting the axum service and injecting the dependencies
runtime
    .spawn(async {
        axum(pool, secret_store, static_folder)
            .await
            .map(|ok| Box::new(ok) as Box<dyn shuttle_service::Service>)
    })
    .await
    .map_err(|e| {
        if e.is_panic() {
            let mes = e
                .into_panic()
                .downcast_ref::<&str>()
                .map(|x| x.to_string())
                .unwrap_or_else(|| "panicked calling main".to_string());
            shuttle_service::Error::BuildPanic(mes)
        } else {
            shuttle_service::Error::Custom(
                shuttle_service::error::CustomError::new(e).context("failed to call main"),
            )
        }
    })?

Adding a static folder

Before we run cargo shuttle run, as we added the Shuttle Static Folder plugin, we will need to add a static folder to our project. If not, the project will fail to start.

Let’s create a static folder in the root of our project and add an index.html file to it.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Shuttle Datadog logs</title>
  </head>
  <body>
    Welcome to the Shuttle Datadog logs app!
  </body>
</html>

Then, we can add a new endpoint in our axum function to serve this html.

This should be the current implementation:

use axum::{
    http::StatusCode,
    response::IntoResponse,
    routing::{get, get_service},
    Router,
};
use tower_http::services::ServeDir;

#[instrument]
async fn handle_error(error: std::io::Error) -> impl IntoResponse {
    tracing::error!(?error, "Error serving static file");
    (StatusCode::INTERNAL_SERVER_ERROR, "Something went wrong...")
}

async fn axum(
    pool: PgPool,
    _secret_store: SecretStore,
    static_folder: PathBuf,
) -> shuttle_service::ShuttleAxum {
    let serve_dir = get_service(ServeDir::new(static_folder)).handle_error(handle_error);

    let router = Router::new()
        .route("/hello", get(hello_world))
        .route("/", serve_dir);
    let sync_wrapper = SyncWrapper::new(router);
    Ok(sync_wrapper)
}

Running the REST API

Run cargo shuttle run and go to http://localhost:8080 in your browser to see the static folder in action.

Now, let’s browse to http://localhost:8080/hello to see the hello_world endpoint in action.

Alternatively, you can also use curl to test the endpoint:

curl -i http://localhost:8080/hello

But remember that this endpoint was instrumented! So, if everything went well, we should be able to see the logs in Datadog.

Let’s check it out! 👀

It works! 🎉

Using Secrets to store the Datadog API key

Wait a minute… we are using a hardcoded API key in our code. This is not a good practice. We should use a secret to store the API key.

Let’s add a new secret to our project. Create a file called Secrets.toml in the root of the project and add the following content:

# Secrets.toml file
DD_API_KEY="YOUR-DATADOG-API-KEY"

IMPORTANT : Don’t forget to add it to your .gitignore file.

Now, we will have to go to the main function and change the order in which the dependencies were instantiated.

Basically, we need to instantiate the secret store before the dd-tracing-layer is set, so we can get the DD-API-KEY secret from there and pass it to the dd-tracing-layer.

// shuttle secret store
let secret_store = shuttle_secrets::Secrets::new()
    .build(factory, runtime)
    .await?;

// getting the datadog api key from the secret store
let dd_api_key = secret_store
    .get("DD_API_KEY")
    .expect("DD_API_KEY not found");

// datadog tracing layer
let dd_layer = dd_tracing_layer::create(
    DatadogOptions::new(
        // first parameter is the name of the service
        "shuttle-datadog-logs",
        // this is the Datadog API Key
        dd_api_key,
    )
    // this is the default, so it can be omitted
    .with_region(Region::US1)
    // adding some optional tags
    .with_tags("env:dev, version:0.1.0"),
);

// ... rest of the code

You can test that everything works as expected by running the project again and checking the logs in Datadog.

Secrets per environment

Let’s try to do one more thing. Wouldn’t it be nice to be able to send different tags to Datadog in case we were running from different environments?

For example, we could add an env tag to the logs in dev and a prod tag in production. Something simple, for the sake of this exercise, and add different secrets for each environment.

Although, this is not supported yet, we can implement something similar.

Let’s change our Secrets.toml file to look like this:

DD_API_KEY="my-datadog-api-key"
DD_TAGS="env:production"
LOG_LEVEL="INFO"

# development environment
DD_TAGS_DEV="env:dev"
LOG_LEVEL_DEV="DEBUG"

Probably you’ve already guessed the idea. We are going to use the _DEV suffix to define the dev environment.

Add this small function to your lib.rs file:

fn get_secret(secret_store: &SecretStore, key: &str) -> Option<String> {
    let final_key = if cfg!(debug_assertions) {
        format!("{}_DEV", key)
    } else {
        key.to_string()
    };

    secret_store.get(&final_key)
}

Finally, we can use this function to get the DD_TAGS and LOG_LEVEL secrets:

// we are adding a version constant to the file
const VERSION: &'static str = "version:0.1.0";

// then we're getting the secrets in the main function
let tags = get_secret(&secret_store, "DD_TAGS")
    .map(|tags| format!("{},{}", tags, VERSION))
    .unwrap_or(VERSION.to_string());

let log_level = get_secret(&secret_store, "LOG_LEVEL").unwrap_or("INFO".to_string());

// NOTE: As we're defaulting to INFO, we're also getting rid of the environment variable support,
// as Shuttle will ingore env vars in production anyway.
let filter_layer = shuttle_service::tracing_subscriber::EnvFilter::try_new(log_level)
    .expect("failed to set log level");

That’s it! It’s not ideal, but it works! 😄

Adding a database

And finally, let’s add a database to our project. We will be using PostgreSQL for this.

Let’s keep things simple, though. We will just add a messages table to our database and a new endpoint to get one message from that table. Feel free to expand this example to your needs.

Cool! 😊 Let’s create a file called db.sql in the root of the project and add the following content:

-- schema
CREATE TABLE IF NOT EXISTS messages (
  id serial PRIMARY KEY,
  message text NOT NULL
);

-- add a message
INSERT INTO messages (id, message) VALUES (1, 'hello world from Database!') ON CONFLICT (id) DO NOTHING;

Now, let’s initialize the database in our axum function:

pool. Execute(include_str!("../db.sql"))
    .await
    .map_err(CustomError::new)?;

Next, we’ll declare a new endpoint and share the pool as state:

let router = Router::new()  
    // here we add the new endpoint
    .route("/message", get(message))
    .route("/hello", get(hello_world))
    .route("/", serve_dir)
    // set the state so we can get access to our database from the endpoints
    .with_state(pool);

Finally, we’ll create the message endpoint:

#[instrument(skip(db))]
async fn message(State(db): State<PgPool>) -> Result<String, (StatusCode, String)> {
    tracing::info!("Getting a message from the database");
    let row: (String,) = sqlx::query_as("SELECT message FROM messages LIMIT 1")
        .fetch_one(&db)
        .await
        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
    let msg = row.0;
    tracing::info!(?msg, "Got message from database");
    Ok(msg)
}

Done!

Let’s run the project again and browse to http://localhost:8000/message to see the message from the database:

And now, let’s check the logs in Datadog:

Conclusion

Well, the length of this post is getting a bit out of hand, so I’ll stop here. 😅

We’ve not only covered how to add Datadog support to our project, but we’ve also shown how to work with databases, secrets, and static folders with Shuttle.

For the moment, it requires a little bit of manual work, but I’m sure that Shuttle will improve this in the future and make this process even easier.

Again, you can see the full code in this GitHub repository

I hope you’ve enjoyed it! 😁

Build and deploy a Rust backend with Shuttle

Roberto Huertas — Sun, 08 Jan 2023 23:00:00 +0000

Learn how to build and deploy a Rust backend with Shuttle.

Value proposition

What if I told you that you could write a Rust backend and deploy it for free in the cloud?

What if I told you that you could get a Postgres database for free, too?

What if I told you that you wouldn’t have to worry about setting the whole infrastructure up? Nor even have to deal with connection strings?

Finally, what if I told you that you could use some of the most popular web frameworks available in the Rust ecosystem?

To be honest, I couldn’t believe it myself when I first learnt about Shuttle, but I can say that they live up to their promise!

And on top of all that, their code is totally open-source!

How to use it

Check out this Axum’s Hello World example:

use axum::{routing::get, Router};
use sync_wrapper::SyncWrapper;

#[shuttle_service::main]
async fn axum() -> shuttle_service::ShuttleAxum {
    let router = build_router();
    let sync_wrapper = SyncWrapper::new(router);
    Ok(sync_wrapper)
}

fn build_router() -> Router {
    Router::new().route("/", get(hello_world))
}

async fn hello_world() -> &'static str {
    "Hello, world!"
}

And now, let’s compare it to the code that you would write by not using Shuttle at all:

use axum::{routing::get, Router};

#[tokio::main]
async fn main() {
    let router = build_router();
    axum::Server::bind(&"0.0.0.0:8000".parse().unwrap())
        .serve(router.into_make_service())
        .await
        .unwrap();
}

fn build_router() -> Router {
    Router::new().route("/", get(hello_world))
}

async fn hello_world() -> &'static str {
    "Hello, world!"
}

As you can see, both examples are pretty similar.

The only part of code that is different is the entry point of your app. The rest is the same. So, you could potentially reuse the same code to deploy it in your own infrastructure if you ever want to get away of Shuttle, thus avoiding the vendor lock-in.

Adding a Postgres database

Adding a database is a piece of cake!

Forget about having to set up a user, dealing with connection strings and all that jazz. Shuttle will inject a sqlx pool that you can use right away from your main function to work with your database.

But in order for this to happen, you will have to add a couple of dependencies to your Cargo.toml first:

shuttle-shared-db = { version = "0.8.0", features = ["postgres"] }
sqlx = { version = "0.6.2", features = ["runtime-tokio-native-tls","postgres"] }

Finally, go to your previous code and add a parameter to your entry point function (note that it is annotated with a macro):

#[shuttle_service::main]
async fn axum(#[shuttle_shared_db::Postgres] pool: PgPool) -> shuttle_service::ShuttleAxum {
  // ...
}

That’s it! You can now start working with sqlx to deal with your new Postgres database.

For the moment, PostgreSQL and MongoDB are the only supported databases.

Initializing the database

Let’s say we want to create a simple table.

Create a file in the root of your project called db/schema.sql and add these lines to it:

CREATE TABLE IF NOT EXISTS test (
  id serial PRIMARY KEY,
  txt varchar(255) NOT NULL
);

Now, let’s add a few lines to the axum function:

#[shuttle_service::main]
async fn axum(#[shuttle_shared_db::Postgres] pool: PgPool) -> shuttle_service::ShuttleAxum {
    pool.execute(include_str!("../db/schema.sql"))
        .await
        .map_err(CustomError::new)?;
}

With that, every time our service starts, it will try to create the test table if it doesn’t already exist.

The Shuttle CLI

But, wait a moment!

You may be wondering: “_What if I want to connect to my database with another client (i.e. DBeaver) to inspect its contents or make a custom query?”.

Well, you can. I didn’t mention so far one of the most important Shuttle parts: its CLI.

Basically, in order to run your code locally and deploy it to the cloud, you’ll need to use a CLI tool called cargo-shuttle.

In order to install it, just run:

cargo install cargo-shuttle

It has lots of subcommands to help you set up a project from scratch and manage it.

Just by running cargo shuttle init, the CLI will guide you through a set of options that will let you choose, amongst other things, the name and location of your project, and the framework that you want to use.

Just to name a few of the available frameworks:

And no, I didn’t forget about the connection string 😉. Whenever you run or deploy your project, if cargo-shuttle detects you are using a database, it will let you know the connection string, so you can use it on your own.

Infrastructure from Code

This is the way they have chosen to describe how they provision all the resources for you (databases, caches, subdomains, static folder…).

As you already saw, they’re using annotations in your code to understand if you are going to need a database or not, and just depending on that, they will provision that for you.

The idea is that you don’t need to define infrastructure as code but that your code itself implicitly defines your infrastructure. Pretty cool and impressive, IMHO!

Many more things

There are many more things to discover.

Check out their amazing set of examples, which will get you up to speed in a blink of an eye.

On the other hand, I have prepared a small GitHub repository showcasing a simple Axum Rest API that uses a Postgres database.

This example is live, here. Feel free to use Postman to play with it. You have the request file available here.

Disclaimer

Note that I’m not in any way related to Shuttle and this is not a “marketing” post of something like that.

I was so amazed when I discovered their service that I immediately felt the need to share my excitement with the rest of the world 😉

In any case, bear in mind that they’re in public alpha so the service is not really that stable, and they may change things or even disrupt the service at any time.

Concluding thoughts

I think Shuttle is a great project and it’s definitely good news for the Rust community.

It not only allows us to deploy Rust backends easily for free , but also enables lots of web developers coming from other languages to approach the Rust web development ecosystem and realize that it’s not as intimidating as one could think in the first place.

Let’s leverage this great opportunity to get involved and be part of this interesting project!

Cheers!

Preview your VSCode extension in the marketplace

Roberto Huertas — Fri, 28 Oct 2022 22:00:00 +0000

Learn how to use the Marketplace Preview extension to see how your extension's manifest will look like in the Visual Studio Marketplace

So, are you building a Visual Studio Code extension?

If you're here, there's a good chance you are 😉. Then, eventually, at some point you will want to publish it to the Visual Studio Marketplace or even to the Open VSX Registry.

But first, just a little disclaimer about myself. I'm the author and main maintainer of the vscode-icons extension, which is about 6 years old at the moment of writing this and, back in the day, helped to bring icons to Visual Studio Code when there was no API available for that.

Aside from that, and after many years of non VSCode related work, I recently joined Datadog's IDE Integration team with the objective of building a great VSCode extension that can bring all the information that Datadog has to offer just where developers work most of the time: their code editor.

With all this, I just wanted to illustrate that I have also faced the moment of publishing to the Marketplace myself. And it's curious, because I clearly remember that the first time, I had to retouch the readme file more than once after publishing. And not only the readme file. The icon and the background color of the banner in the package.json file, too.

To be honest, I more or less suspected that modiying the icon, the background color or one of the many other properties that we usually set in the package.json file, would require to publish a new version of the extension. But I naively believed that the readme changes wouldn't require such a thing and they would be consumed directly from the repository.

Reducing the chances

The reality is that sometimes, when you're modifying your readme or the settings of your package.json file, you may not be sure of how that would look like in the Visual Studio Marketplace.

Yes, the readme is written in markdown, but the way it's rendered in GitHub, for instance, may differ from the way it's presented in the Visual Studio Marketplace, and there's a chance that when you see the result of your work in there, you may not really like it and then need to ship a new version "just" for that.

As we were planning to do a first release of the Datadog's VSCode extension soon and I had also some ideas for VSCode extensions that I wanted to explore, I thought I should find a tool to reduce the chances of having to publish again because of those unexpected aesthetic changes. And I found one! 🌟

Microsoft's Extension Manifest Editor

Enter Extension Manifest Editor, a VSCode extension built by Microsoft DevLabs that lets you preview the details page for your extension from within Visual Studio Code.

I was happy! Until I tried it...

Surprisingly, the extension didn't work. The last commit to that extension dates back from 2017 so it kind of makes sense.

Honestly, I'm not sure why Microsoft decided to discontinue the support for that extension. Especially, given that it has 11K downloads in the Marketplace, which is certainly not that much if you compare it with other extension's numbers, but anyway, it's a niche product, targeting only extension developers... so I would say that it's not such a bad number at all.

Let's build a new extension!

Probably you already know that feeling... You're seeking for a tool, and you want to find it because it will solve your problem... but... some part of you, deep inside, wishes not to find it at all, so you can have the opportunity to build it yourself and share it with the rest of the world.

And that's what happened to me! And that feeling ended up being transformed into this new Visual Studio Code extension called Marketplace Preview which I recently published.

It's quite simple but it does the job. And if you are building an extension yourself, I hope it can be helpful for you, too.

How it works

If you open a package.json file that happens to be a VSCode manifest, you will see this extra icon in the top right corner. Click over it to see how it would look in the Visual Studio Marketplace.

See how it works in the animated gif below:

If you give a try, let me know your thoughts and ideas. I'll be happy to accept contributions or suggestions in the extension's GitHub repository.

Open sourcing Licensebat

Roberto Huertas — Mon, 03 Jan 2022 23:00:00 +0000

Learn why I decided to open source Licensebat, the software that easily helps you manage the licenses of your project's dependencies.

What’s Licensebat?

The idea is pretty simple, it’s just a tool written in Rust that scans the dependencies of your project and generates a report of the different licenses that are used.

Initially, it was only a SaaS product running in the GitHub Marketplace with several pricing tiers. One of them being a free tier without any limitations for OSS projects and a few free requests per month for commercial projects.

After some months, though, I decided to open source Licensebat and release it as a FOSS project.

Why?

Although Licensebat as a SaaS product has access to the code hosted in GitHub it only accesses the code through the GitHub API. This allows us to keep the code private and only access the dependency manifest files, like package-lock.json, Cargo.lock or pubspec.yaml.

Anyway, some companies had some legit concerns about their code being potentially exposed to a third party service. It was clear to me that I’d need some other way for these customers to be able to get the service Licensebat was providing without exposing their code at all.

Licensebat CLI

The idea of having some sort of binary/container/executable that could be used by these companies in their CI/CD pipelines immediately came up. But then, anyway, if this binary was not open source, if they couldn’t compile it themselves, how would they be sure that it was completely safe to use? How would they know that it was not scanning the code under the hood?

That’s when I decided to open source not only the CLI but also all the crates that are used in order to give support for some programming languages, which at the time I’m writing this are:

The thought was that the community would eventually be able to use and evolve all these crates and support many more languages in the long term.

What about the SaaS?

You may be wondering what’s the value of the SaaS service in this context where any company could just download the CLI and use it for free. And you’re totally right.

To be honest, I’m not completely sure about this, but I think there will still be companies willing to use Licensebat as a GitHub integrated service.

I have plans to extend that service even further by optimizing the dependency resolution and the license scanning process, and providing more value by adding features such as a history of the licenses that have been scanned, and a way to share the results with other people.

Will it work? Who knows! 😁

In any case, I’m happy to be able to open source Licensebat and hope that it will be useful to other people. That’s what matters to me more.

Other conveniences

There are other cool things associated to the fact that this CLI is open source. The obvious one is that we can now leverage the fact that we can access the whole codebase without any concerns.

This means that, for some languages, we can find more convenient and optimized ways to access to the dependencies’ license information, speeding up the process.

Conclusion

If you want to give it a try or are just curious about the code, you can find it here, hosted in GitHub.

It would be great to have you as a contributor!

Cognito Authentication for your SvelteKit app

Roberto Huertas — Sat, 24 Jul 2021 22:00:00 +0000

Manage your users in your SvelteKit app with AWS Cognito.

Why Cognito?

To be honest, in my case, it’s just because we’re using AWS at Koa Health and it was the natural choice. I have many things to say about AWS Cognito, both good and bad, but this won’t be the article covering that 😉.

Well, hopefully, chances are that you may be already using AWS as your main cloud provider and this post can help you to integrate AWS Cognito in you SvelteKit app.

Disclaimer

I’m starting with Svelte so if you have any suggestions to improve what I’ll show here, you’re more than welcome to write a comment or contact me.

Alternatively, feel free to take a look at the GitHub repository hosting the final project and open an issue there if you prefer.

Setting up the SvelteKit app

We’re just going to use SvelteKit to build a simple app that will authenticate users, so let’s start with with some simple commands to create the project:

npm init svelte@next sveltekit-cognito-auth
# I'm choosing the Skeleton project,
# opting out of TypeScript for this tutorial
# and using ESlint and Prettier support
cd sveltekit-cogniton-auth
npm install

You can find here more details about how to get started with SvelteKit in case you want to learn more about it.

For now, let’s open the src/index.svelte file, delete its content and build this spartan UI:

<script>
    // we'll edit this later
    let user;
</script>

<svelte:head>
    <title>SvelteKit - Cognito</title>
</svelte:head>

{#if !user}
    <button>Log In with Cognito</button>
{:else}
    <button>Log Out</button>
{/if}

The idea here is to have a button that will trigger the Cognito authentication flow and another one that will log out the user. As simple as that!

For the moment, we’re going to skip all the logic to show one button or the other but we will come back to it later.

Let’s check that our app is working:

npm run dev -- --open

Hopefully, you should see a page with a button that says Log In with Cognito.

Setting up Cognito

Let’s create and configure our Cognito User Pool now.

First of all, let’s introduce the pool name: sveltekit-pool.

For this example, we’re going to use these simple settings.

Remember to tick the email checkbox in the Attributes section.

We’ll also relax the Password Policy to allow users to choose their own passwords without many hassles, so uncheck all the checkboxes related to the password strength.

Let’ press the Next Step button until we arrive to the App Clients section and let’s add one app client.

Leave it everything as it is and set the name that you want. For this example, I’m using sveltekit. Fairly original, isn’t it? 😉

Setting up the App Client

There are still a couple of things to do in our Cognito Pool. Ideally, we’d like to use the Cognito Hosted UI which will simplify a lot the Sign Up and Sign In flows.

For the moment, bear with me, fill the App Client settings with the following values and just click Save:

A couple of things to note here:

The callback url will be a backend endpoint we’ll be exposing in our app.
The Sign Out url is the final url to be redirected to once we’ve signed out from Cognito

Finally, for the Cognito Hosted UI to work, we need to set up a domain. Note that the domain you choose is subjected to availability.

Let’s move away from Cognito for the moment and let’s focus again on our Svelte app.

Leveraging SvelteKit Auth library

Luckily for us, RaviAnand Mohabir has created a super cool library called SvelteKit Auth, based on NextAuth.js, which will allow us to use some of the most popular identity providers, such as Google, Twitch, Facebook, Twitter and Reddit.

But not only that, it also provides a generic OAuth2 provider that we will leverage to authenticate our users with Cognito.

The library handles all the OAuth2 flow for you, including the redirects, the token exchange and the user profile retrieval.

Let’s add the dependency to our app before we continue:

npm i sk-auth --save

Creating our Cognito Provider

Let’s create a new file called src/lib/auth.js and paste the following code:

import { SvelteKitAuth, Providers } from 'sk-auth';

// this is the domain we set up in our Cognito Pool
const DOMAIN = 'sveltekit.auth.eu-west-1.amazoncognito.com';

// these are the configuration seetings for our OAUTH2 provider
const config = {
    accessTokenUrl: `https://${DOMAIN}/oauth2/token`,
    profileUrl: `https://${DOMAIN}/oauth2/userInfo`,
    authorizationUrl: `https://${DOMAIN}/oauth2/authorize`,
    redirect: 'https://robertohuertas.com',
    clientId: import.meta.env.VITE_CLIENT_ID,
    clientSecret: import.meta.env.VITE_CLIENT_SECRET,
    scope: ['openid', 'email'],
    id: 'cognito', // IMPORTANT: this is the id that we'll use to identify our provider
    contentType: 'application/x-www-form-urlencoded'
};

const oauthProvider = new Providers.OAuth2Provider(config);

// exposing our auth object
export const appAuth = new SvelteKitAuth({
    providers: [oauthProvider]
});

Probably, if it’s the first time you’ve seen something like import.meta.env.VITE_whatever, you’ll be wondering what the hell is that.

Basically, it’s the way to get some environment variables from an .env file in your root.

You can find more info about it in the SvelteKit FAQs.

So make sure to add the following to your .env file:

VITE_CLIENT_ID=<your_client_id>
VITE_CLIENT_SECRET=<your_client_secret>

Exposing the Auth object endpoints

Now, let’s create a new file called src/routes/api/auth/[...auth].js.

If that felt strange to you, you have to know that this kind of naming is a routing convention from SvelteKit.

You can use this piece of code to expose the auth endpoint:

// we're importing our auth object here
import { appAuth } from '$lib/auth';
// and exposing the get and post method handlers
export const { get, post } = appAuth;

Using the Auth object getSession method

The Auth object exposes a getSession method that we can use in our hooks.js file in order to return a session that it’s accessible on the client.

As we have used the Skeleton project, we don’t have any hooks.js file yet. Let’s create one in the src folder and add this code:

import { appAuth } from '$lib/auth';
export const { getSession } = appAuth;

With this, the claims of the access token will be available on the session store.

Adding some interaction in the UI

We’re going to add now some actions to the buttons in our UI, so let’s open our src/index.svelte file and edit it:

<script>
    import { signOut as authSignOut } from 'sk-auth/client';
    import { session } from '$app/stores';

    // getting the user from the session store
    $: user = $session.user;

    function signIn() {
        location.assign('/api/auth/signin/cognito?redirect=/');
    }

    function signOut() {
        authSignOut().then(session.set);
    }
</script>

<svelte:head>
    <title>SvelteKit - Cognito</title>
</svelte:head>

{#if !user}
    <button on:click="{signIn}">Log In with Cognito</button>
{:else}
    <h2>Welcome {user.email}!</h2>
    <p>Your username is {user.username} and your email has been verified: {user.email_verified}</p>
    <button on:click={signOut}>Log Out</button>
{/if}

Notice that we’ve added some actions to the buttons which are fairly self-explanatory.

The signIn action will redirect the user to our api/auth endpoint (the one we set up in the previous step) with signin and cognito as parameters (remember that we set the id of our provider to cognito).

Finally, the redirect querystring parameter will tell the endpoint to redirect the user to our / route once the user is authenticated.

Let’s run the app and click over the Log In with Cognito button. You should see the Cognito Hosted UI:

As we don’t have any account yet, let’s click over the Sign Up link and create a test user.

Use a real email address as Cognito will send you a verification email with a code you’ll need to enter in the UI.

Once you’ve confirmed your email, you’ll be redirected to the / route but, this time, you should see something like this:

If you check your Cognito pool, you should be able to see the user you just created.

Note that SvelteKit Auth will set an HttpOnly cookie in order to manage the session for you.

Conclusion

As you can see, setting up an OAuth2 provider is pretty easy with SvelteKit Auth. If you need further adjustments, check out the repository. If you take a look at the example app, you’ll see plenty of ways to adjust your flow.

As a final note, I would like to mention that our Log Out button is not really logging out our user from Cognito as we’re only removing the cookie from our browser. So if you try to log in again, you’ll probably be logged in without having to introduce any credentials.

As a way to mitigate this, you can clear the Cognito Hosted UI domain cookies or you just can extend the log out functionality by calling the logout endpoint from AWS Cognito. Give it a try!

--
Originally published at robertohuertas.com on Jul 25, 2021.

Deploy an IPv6 Azure VM while setting up a Nym node

Roberto Huertas — Sun, 18 Jul 2021 22:00:00 +0000

Learn how to deploy an Azure VM with IPv6 support while setting up a Nym node.

A little bit of context

I’m lately looking into cool crypto projects, mainly because most of them make an extensive use of Rust and that’s always and extra motivation for me 😁.

One that has caught my attention is Nym, which promises a huge improvement both in terms of privacy and security while we're using the internet. It has recently raised $6M so this not only shows that the project it's catching the attention of many diverse investors but also speaks of its capability to start growing and increasing its development pace.

Long story short, I was trying to set up a Nym node - as running a node in Milhon (their next version of the Testnet) will only be open to those who have run a node in Finney (their current version of the Testnet) - and thought of doing that in Azure.

I read the Nym docs and all seemed pretty straightforward. What could go wrong? It seems that for the Nym node to work ok, it has to have an IPv6 address and Azure was not going to let me walk away so easily…

TL;DR;

Setting up IPv6 in an Azure VM cannot be done by using the Portal. It’s also impossible to set it up in an already existing VM.

Here you are a link to the official Azure docs.

The steps to create an IPv6 VM

You could do all this via your command line but I will show you how I used the Azure Portal for everything except the VM creation.

These are the different steps:

Create a couple of public IP addresses
Create a Virtual Network
Create a Network Interface
Create a Network Security Group
Create the VM

Create a couple of public IP addresses

Let’s create a couple of public IP addresses, IPv4 and IPv6, respectively. We can create them both at the same time, which is cool as it will save us some time.

Let’s select the option in the Azure Portal and just click Create.

Select Both in the first option and just fill in the names of your IPs.

Then, just click Create and in a couple of seconds you’ll have your two new shiny IPs.

Create a Virtual Network

Next, we will create a Virtual Network which is the fundamental building block for your private network in Azure.

Similar to what we did in the previous section, we look for the virtual network option and click over the Create button:

Then, we just have to go through the different tabs, setting the name of the virtual network,

and then configuring the IP addresses in there. Be sure to select the Add IPv6 address space checkbox:

You’ll notice there’s a warning message stating that subnets should have at least one IPv6 address range. Let’s click over the default subnet name and a sidebar will open.

This time, tick the Add IPv6 address space checkbox, fill the IPv6 range, and click the Save button. You’ll notice that the previous warning is gone and you’ll be able to follow the process. Click Next and finally Createthe virtual network.

Create a Network Interface

Now, it’s time for us to create a Network Interface.

As you may probably guess by now, we’re going to look for the network interface option and click over the Create button:

You’ll find a form that you’ll have to fill.

Give a name to the network interface and tick the Private IP address (IPv6) checkbox before creating the resource.

Once you have created it, go the resource and select the IP configurations option in the left side menu. You’ll see a screen similar to the one below:

Let’s configure our IPs. Click over the Primary ip configuration, and make the appropriate selections. Once you select Associate in the Public IP address option you’ll be able to see a list with the IPs we created before. Select the IPv4 one.

Now, we will do the same for the Secondary ip configuration, the IPv6 one.

Finally, let’s select Properties in the left side menu and copy the Resource ID. We’ll need it later in order to create our VM.

Create a Network Security Group

You can also create a Network Security Group and set it up in the Network Interface so you can filter which ports are opened to the internet.

The process if fairly straightforward.

You just have to give it a name and create it.

Then, go to our Network interface (nym-nic) and select the Network Security Group option in the right side menu.

Create the VM

In order to create the VM we will use the Azure CLI.

Go to your command line and type:

az login

Once you are logged in, we’re going to create the VM by using this instruction.

az vm create --resource-group nym-node-burn --name nym --image UbuntuLTS --admin-username robertohuertasm --admin-password hZRAAJpjkwZ4KW --nics "/subscriptions/e3871cc7-3444-42ad-b4b8-5ee6bdbf6d54/resourceGroups/nym-node-burn/providers/Microsoft.Network/networkInterfaces/nym-nic" --location westeurope --size "Standard_B1s"

You can change the settings as you see fit but note that the --nics flag must be informed with the Resource ID we copied in the previous section.

And that’s it! Your VM should be up and running!

Finally, you just have to connect via ssh and run a Nym node 😊, but first, be sure you’ve enabled port 22 access in your Network Security Group:

Have fun!

--
Originally published at robertohuertas.com on Jul 19, 2021.

Aprende Rust en español

Roberto Huertas — Mon, 25 May 2020 22:00:00 +0000

Aprende Rust en español y a tu ritmo con estos vídeos de YouTube.

Motivación

Personalmente, siempre tengo la sensación de que hay pocos recursos técnicos disponibles en nuestro idioma y, aunque el inglés es algo que debemos controlar para poder avanzar en nuestra industria, no sé si por mi edad o qué, la mayoría de las veces me parece que mi cerebro gasta más ciclos de cpu de los necesarios para lograr comprender ciertos conceptos cuando leo o veo cosas que están en un idioma extranjero.

Así que, por solidaridad a todos aquellos que sufren de este sobrecoste a la hora de aprender, pensé que sería interesante empezar una serie de vídeos para introducir a Rust a todos aquellos interesados en aprenderlo. 😉

El plan

La idea es usar este canal de YouTube para ir publicando vídeos con una cadencia semanal.

Esta cadencia, lógicamente, podrá variar en función de mi logistica familiar pero mi intención es ir a buen ritmo, al menos al principio. Por el momento ya llevo 10 de unos 150 que tengo planificados.

Como podéis ver, el canal todavía no tiene ningún nombre especial porque YouTube se reserva ese derecho sólo para aquellos canales que tienen muchos subscriptores. ¡Así que ya podéis suscribiros y pasar el enlace a todos vuestros amigos! 😊

Una pequeña muestra

Estructura

Los vídeos se van a estructurar en dos grandes grupos:

Curso de Rust
Sesiones prácticas

Curso de Rust

Este curso intentará cubrir desde lo más básico a lo más complejo y, como he avanzado anteriormente, calculo que supondrá alrededor de 150 vídeos aproximadamente de unos 10 minutos de duración como máximo.

Los temas a tratar, más o menos, serían estos y cada uno de ellos tendrá una lista de reproducción asociada:

Introducción a Rust
Variables y tipos primitivos
Control de flujo
Tipos definidos por el usuario
Colecciones
Manejo de memoria
Packages, crates y módulos
Manejo de errores
Genéricos y Traits
Closures
Iteradores
Punteros inteligentes
Concurrencia
Errores personalizados
Tests
Programación asíncrona
Cargo
Extendiendo Cargo
Documentación
Macros
Rust en la web y WASM
Traits útiles
Trucos sobre tipos
Trucos sobre el manejo de memoria
Trucos sobre el compilador
Benchmarking, unsafe y CI/CD

Sesiones prácticas

Estas sesiones prácticas serán vídeos más largos en los que abordaré algún tipo de problemática concreta y daré una especie de taller para que lo podáis seguir en línea.

Entre los diferentes que tengo en mente:

Construir un servidor gRPC usando Tonic [échale un vistazo al repo]
Construir un servidor REST usando Actix-web
Construir un pequeño CLI

Declaración de intenciones

Hace ya unos años que empecé a programar con Rust y por algún motivo me atrapó este lenguaje. En seguida sentí la necesidad de contactar con más gente que compartiera la pasión por este lenguaje y en 2018 organicé el meetup de Rust de Barcelona y luego, al año siguiente, fui uno de los organizadores locales del Rustfest que se llevó a cabo en Barcelona, que ha sido el más grande organizado hasta la fecha.

Debido a las restricciones que nos ha impuesto el COVID-19, la actividad de nuestro meetup se ha visto reducida a cero y eso, unido a mi voluntad de llevar Rust a todos aquellos que están interesados en aprender, supongo que me ha llevado a la preparación de esta serie de vídeos.

No soy el mejor comunicador, ni el mejor youtuber pero lo voy a intentar. 😊

Espero que os guste la iniciativa y que le deis soporte, en la medida de lo posible.

¡Nos vemos pronto!

Roberto

--
Originally published at robertohuertas.com on May 26, 2020.

Sync your Figma assets with your dev environment

Roberto Huertas — Sat, 18 Apr 2020 22:00:00 +0000

Learn how to use Figma Asset Downloader CLI to work with Figma like a boss!

The problem

Recently, the Design team at Alpha started to use Figma in order to prepare the assets for the app I'm currently involved in, which is called Evermind and aims to help people to build the skills needed to manage stress and become more resilient.

Our Design team is really happy with the change from Sketch to Figma, and they really seem to be more productive and enjoy a lot the collaborative work that it promotes.

Suffice to say, that if they are happy, this has a direct impact in the product and, hence, also in the Development team. Although Figma has actually a way to export images directly from their tool it rapidly became a bottleneck for our Development team: it was a pain to be constantly exporting and reorganizing the assets in a convenient folder structure for our mobile app (2x, 3x and so on...).

The investigation

First, I decided to look for a tool that could be integrated in our development flow and automatically sync the assets from Figma. Some sort of CLI that would allow us to even integrate this into our CI pipelines if we needed it. The fact is that I couldn't find any kind of tool that suited our simple needs, so I decided to build my own.

As you may know if you've read some of my previous articles, I'm very fond of Rust so this was also an opportunity to write something with it and help to spread its usage 😉.

The solution

Luckily for me, Figma has a super nice API so it was really simple to build something that could export all the assets from any Figma Document I wanted with just a simple command call.

I just had to focus on building a nice CLI by leveraging the amazing StructOpt crate and some of the usual suspects like Tokio, Reqwest and Serde.

I stuck to the plan and eventually Figma Asset Downloader CLI came to life. It was simple, and it worked for us! 🎉

Read the docs if you want to learn more about it.

The improvements

After some days of using it, although we were happy with it, we found that the images generated from Figma were not being really optimized: jpg images were huge, and we felt that png optimization could be improved a little bit, too.

So I put some super cool crates into the mix, namely image for jpg and oxipng for png, and voilà, we got some cool image optimization capabilities into the CLI, just right away! 😮

The roadmap

The tool is open source and it's been published under the MIT license, so feel free to use it, adapt it and extend it.

Although the roadmap of this tool is mainly driven by the needs of my current team at Alpha Health, I'll be more than happy to hear other team's needs and perhaps try to implement these new features into it.

For the moment, as per one of my team colleagues suggestion, I'm considering to extend it by adding some sort of manifest that could help you to identify which files have changed their name, just to avoid having missing images in your app due to some misspelling in the Figma asset.

EDIT: This feature was added on April 30th, 2020

Of course, that's only an example.

Not only I'm hoping that this tool will be extended by the suggestions and efforts of other teams but I'm verifying that it's also increasing the interest of my teammates in Rust as a language, so, so far, it has been worth developing it.

Enjoy! 😊

--
Originally published at robertohuertas.com on April 19, 2020.

Wallpapers for the Rust of us

Roberto Huertas — Sat, 07 Dec 2019 23:00:00 +0000

Before Barcelona’s RustFest edition, I created some wallpapers for the occasion. It’s time to share them with the community.

Abstract

Download / Original source

Blue

Download / Original source

Broken

Download / Original source

Camouflage

Download / Original source

Circuits

Download / Original source

Circus

Download / Original source

Color

Download / Original source

Fuchsia

Download / Alt / Original source

Laser

Download / Original source

Metal

Download / Original source

Mondrian

Download / Alt / Original source

More color

Download / Original source

Old

Download / Original source

Old Paper

Download / Original source

Polygon

Download / Alt / Original source

Rainbow

Download / Original source

Red Fiber

Download / Original source

Rusty

Download / Original source

Sound

Download / Original source

Steam

Download / Original source

Sunset

Download / Original source

Wall

Download / Original source

Wood

Download / Original source

Attributions

These wallpapers were created by using some amazing wallpapers as the original background and my only contribution was to insert the Rust logo on them with more or less success, so all the credit goes to the original authors. You can access to the original source where I got these primary wallpapers below the images and besides the Download link.

Enjoy! 😊

--
Originally published at robertohuertas.com on December 8, 2019.

Rust once and share it with Android, iOS and Flutter

Roberto Huertas — Sat, 26 Oct 2019 22:00:00 +0000

What if I told you that you could use the same very performant code in Android, iOS or even in Flutter. In this article, we’ll see how to achieve this with Rust.

But, why would we want something like this?

Imagine that you have a mobile app that needs to process some audio in order to get some insights about the user but you don’t want the audio to be sent to the server to be processed. You want to preserve the privacy of the user. In this kind of scenario it would make sense to avoid having to write a library for Android and for iOS. That would save us from having to maintain two different codebases and would diminish the chance to get more bugs.

That’s nice, but how could we do something like this? Enter Rust. With Rust not only would you be able to share the same code among multiple platforms, but you could also take advantage of the boost of performance that you will get.

What are we going to do

We are going to write a simple shared Rust library and compile it to Android and iOS, and as a bonus, we will also write a Flutter plugin using the very same code.

As you can see, the scope of this article is quite broad so we’ll try to keep everything organized.

You can also read this post while taking a look at the associated GitHub repository.

Scaffolding our project

Let’s start by creating a folder called rust-for-android-ios-flutter and create four folders in it (android, ios, flutter & rust):

mkdir rust-for-android-ios-flutter
cd rust-for-android-ios-flutter
mkdir ios android rust flutter

Once we have it, just cd into the rust folder and create a new Rust library called rustylib:

cd rust
cargo init --name rustylib --lib

This library will have only one function that will get a string as its argument and will return a new string. Basically, just a Hello, World!. But just think of it as a function that could work as the entry point to a more complex process completely written in Rust.

Let’s install some targets

In order to compile our rustylib library to Android and iOS we will need to have some targets installed in our machine:

# Android targets
rustup target add aarch64-linux-android armv7-linux-androideabi i686-linux-android x86_64-linux-android

# iOS targets
rustup target add aarch64-apple-ios armv7-apple-ios armv7s-apple-ios x86_64-apple-ios i386-apple-ios

Tools for iOS

For iOS we have to be sure that we have Xcode installed in our computer and the Xcode build tools already set up.

# install the Xcode build tools.
xcode-select --install
# this cargo subcommand will help you create a universal library for use with iOS.
cargo install cargo-lipo
# this tool will let you automatically create the C/C++11 headers of the library.
cargo install cbindgen

As you can see, we have also installed cargo-lipo and cbindgen.

Tools for Android

For Android, we have to be sure that we have correctly set up the $ANDROID_HOME environment variable. In macOS this is tipically set to ~/Library/Android/sdk.

It is also recommended that you install Android Studio and the NDK. Once you have everything installed, ensure that the $NDK_HOME environment variable is properly set. In macOS this should be tipically set to ~/Library/Android/sdk/ndk-bundle.

Finally, we’re just going to install cargo-ndk, which handles finding the correct linkers and converting between the triples used in the Rust world to the triples used in the Android world:

cargo install cargo-ndk

Rust library configuration

The next step is to modify our Cargo.toml. Make sure it looks similar to this:

[package]
name = "rustylib"
version = "0.1.0"
authors = ["Roberto Huertas <roberto.huertas@outlook.com>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
name = "rustylib"
# this is needed to build for iOS and Android.
crate-type = ["staticlib", "cdylib"]

# this dependency is only needed for Android.
[target.'cfg(target_os = "android")'.dependencies]
jni = { version = "0.13.1", default-features = false }

iOS project

Now, let’s create an iOS project using Xcode.

In iOS you can use 2 different types of user interface. As we want to show how to use both of them, let’s create two different kind of projects.

Storyboard

We’re choosing Storyboard as our user interface and we’re naming the project rusty-ios-classic.

Save it in the previously created ios folder.

SwiftUI

Let’s create now a new iOS project. But this time we’re going to select SwiftUI as our user interface and name it rusty-ios.

Save it again in the ios folder.

Your treeview should look similar to this one:

Writing our first Rust code

Now, go to the Rust project, open the lib.rs file and make sure it looks exactly like this:

use std::ffi::{CStr, CString};
use std::os::raw::c_char;

#[no_mangle]
pub unsafe extern "C" fn hello(to: *const c_char) -> *mut c_char {
    let c_str = CStr::from_ptr(to);
    let recipient = match c_str.to_str() {
        Ok(s) => s,
        Err(_) => "you",
    };

    CString::new(format!("Hello from Rust: {}", recipient))
        .unwrap()
        .into_raw()
}

#[no_mangle]
pub unsafe extern "C" fn hello_release(s: *mut c_char) {
    if s.is_null() {
        return;
    }
    CString::from_raw(s);
}

The #[no_mangle] attribute is vital here to avoid the compiler from changing the name of the function. We want the name of the function to be exported as it is.

Note also that we’re using extern "C". This tells the compiler that this function will be called from outside Rust and ensures that it is compiled using C calling conventions.

You may be wondering why on Earth we need this hello_release function. The key here is to take a look at the hello function. Using CString and returning the raw representation keeps the string in memory and prevents it from being released at the end of the function. If the memory were to be released, the pointer provided back to the caller would now be pointing to empty memory or to something else entirely.

In order to avoid a memory leak, because we have now a string that sticks around after the function has finishing executing, we have to provide the hello_release function that takes a pointer to a C string and frees that memory. It’s very important not to forget calling this function from the iOS code if we don’t want to get into troubles. If you look closely at this function, you’ll notice that it leverages the way memory is managed in Rust by using the function’s scope in order to free the pointer.

This code will be the one that we’ll use for our iOS projects.

Compiling for iOS

Before we compile the library for iOS we’re going to generate a C header that will work as a bridge for our Swift code to be able to call our Rust code.

We’ll be leveraging cbindgen for this:

cd rust
cbindgen src/lib.rs -l c > rustylib.h

This should generate a file called rustylib.h containing the following code:

#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>

char *hello(const char *to);

void hello_release(char *s);

Note that cbindgen has automatically generated the C interface for us in a very convenient way.

Now, let’s proceed to compile our Rust library so it can be consumed in any iOS project:

# it's important to not forget the release flag.
cargo lipo --release

Check out your target/universal/release folder and look for a file called librustylib.a. That’s the binary we’re going to use in our iOS projects.

Using the iOS binary

First, we’re going to copy our librustylib.a and rustylib.h files into the ios folder:

# we're still in the `rust` folder so...
inc=../ios/include
libs=../ios/libs

mkdir ${inc}
mkdir ${libs}

cp rustylib.h ${inc}
cp target/universal/release/librustylib.a ${libs}

You should be seeing a treeview like this one, with an include and a libs file:

As you can imagine, having to manually do this every time you have to compile a new version of your Rust library would be very tedious. Fortunately, you can automate this process by using a simple bash script like this one.

Now, the following is something that you will need to do only once (twice if you have created two iOS projects as the article described).

Let’s open our rusty-ios-classic project in Xcode and let’s do the following:

Add the librustylib.a file in the General > Frameworks, Libraries and Embedded Content. Ensure that you see the name of the library there. If it doesn’t show, try it again. I’m not sure if it’s an Xcode bug but most of the times you’ll need to add it twice for it to work correctly.

After that, go to the Build Settings tab, search for search paths and add the header and library search paths. You can use relative paths or use the $(PROJECT_DIR) variable to avoid hardcoding your local path.

Finally, let’s add the Objective-C Bridging header. Search for bridging header in the Build Settings tab:

Repeat the same for our rusty-ios project in case you want to try both type of iOS projects.

In our rusty-ios project

if you’re using the project that uses SwiftUI as a user interface, then open the ContentView.swift file and make it look like this:

import SwiftUI

struct ContentView: View {
    let s = getName()
    var body: some View {
        Text(s)
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

func getName() -> String {
    let result = hello("Rob")
    let sr = String(cString: result!)
    // IMPORTANT: once we get the result we have to release the pointer.
    hello_release(UnsafeMutablePointer(mutating: result))
    return sr
}

Run the project in Xcode.

In this case, you should be able to see Hello from Rust: Rob in the emulator or device you’re using to test the app 🚀.

In our rusty-ios-classic project

In case you’re using a the project with a Storyboard user interface, open the ViewController.swift file and make it look like this:

import UIKit

class ViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view.
        let result = hello("Rob")
        let s_result = String(cString: result!)
        // IMPORTANT: once we get the result we have to release the pointer.
        hello_release(UnsafeMutablePointer(mutating: result))
        print(s_result)
    }
}

If everything is fine, you should be able to see Hello from Rust: Rob in the output pane. 😉

Android project

Let’s open Android Studio and let’s create our Android project: File > New...> New Project > Basic Activity.

Name it rusty-android and set the package name. We’ll choose Kotlin as our default language and minimum API 22.

You should end up with a treeview similar to this one:

JNI

If you remember when we discussed about how to create our iOS project, we needed to create a C header working as a bridge. In Android we will leverage the Java Native Interface or JNI for short and we will expose our functions through it. The way that JNI constructs the name of the function that will be called follows a specific convention: Java_<domain>_<qualified_classname>_<methodname>. In our case, that would be Java_com_robertohuertas_rusty_1android_MainActivity_hello (note that _1 represents underscores _ in the qualified class name).

As you can imagine, if we have to name our functions in such a specific way, this can pose a problem if we want to reuse this very same code in other Android apps. We have several alternatives, though. We can use some sort of proxy class that follows the same specific domain and class naming and include it in every project or we can create an Android Library and use it everywhere.

In our case, we’re going to create an Android Library.

Creating an Android Library

In Android Studio, File > New > New Module.... Then choose Android Library.

Your Android Studio project pane should look similar to the one below:

Adding a little bit more Rust

Ok, so let’s create our Android functions by leveraging the JNI naming conventions.

Let’s cd into our rust/src folder and create a new android.rs file:

cd rust/src
echo > android.rs

Once you have it, copy this code into it:

#![cfg(target_os = "android")]
#![allow(non_snake_case)]

use crate::hello;
use jni::objects::{JClass, JString};
use jni::sys::jstring;
use jni::JNIEnv;
use std::ffi::CString;

// NOTE: RustKt references the name rusty.kt, which will be the kotlin file exposing the functions below.
// Remember the JNI naming conventions.

#[no_mangle]
pub extern "system" fn Java_com_robertohuertas_rusty_1android_1lib_RustyKt_helloDirect(
  env: JNIEnv,
  _: JClass,
  input: JString,
) -> jstring {
  let input: String = env
    .get_string(input)
    .expect("Couldn't get Java string!")
    .into();
  let output = env
    .new_string(format!("Hello from Rust: {}", input))
    .expect("Couldn't create a Java string!");
  output.into_inner()
}

#[allow(clippy::similar_names)]
#[no_mangle]
pub extern "system" fn Java_com_robertohuertas_rusty_1android_1lib_RustyKt_hello(
  env: JNIEnv,
  _: JClass,
  input: JString,
) -> jstring {
  let java_str = env.get_string(input).expect("Couldn't get Java string!");
  // we call our generic func for iOS
  let java_str_ptr = java_str.as_ptr();
  let result = unsafe { hello(java_str_ptr) };
  // freeing memory from CString in ios function
  // if we call hello_release we won't have access to the result
  let result_ptr = unsafe { CString::from_raw(result) };
  let result_str = result_ptr.to_str().unwrap();
  let output = env
    .new_string(result_str)
    .expect("Couldn't create a Java string!");
  output.into_inner()
}

Wait, what’s going on here? 🤔

We’d better stop for a moment and explain a little bit the previous code.

First of all, on the top of the file we can see two different directives:

#![cfg(target_os = "android")]
#![allow(non_snake_case)]

The first one will enable this code only when we’re compiling for Android and the second one will allow us to name our functions however we want. Rust enforces snake_case but we need to opt out of this in order to comply with the JNI naming conventions.

Ok, but then, why have you created two different functions (helloDirect and hello) and not just one? 🤔

Well, the answer is that I wanted to show you two ways of handling the Android part and let you decide which one is more convenient for your kind of project.

The first function uses the jni crate without interacting with the lib.rs code (a.k.a. iOS code) and the second one uses the same code we have in the lib.rs file.

The difference is clear. The first function is way clearer and more succint than the second one. Plus, in the second one, we have to deal with the hello_release function and unsafe while in the first one we don’t.

So, what we should do? In my opinion, I would use the first one. This is a super simple example where we’re just building a string and returning it. Ideally, this logic should be also encapsulated in a pure Rust library that would be consumed by both the iOS and the Android code. These pieces of code should be only concerned about providing the glue to the iOS and Android land via C headers and JNI and that’s it. So, ideally, in our example, instead of duplicating the logic in the Java_com_robertohuertas_rusty_1android_1lib_RustyKt_helloDirect function we would call another library.

Anyway, for the sake of knowing that you have several options, I think it’s good to explore all the approaches. 😜

One more important thing. Note that we’re exporting our functions with system instead of C. This is just to stop cbindgen from generating signatures for these Android functions.

But wait, this won’t work! We haven’t exposed our android module.

Add this to the lib.rs file:

// add it below the use declarations.
#[cfg(target_os = "android")]
mod android;

The cfg attribute will prevent the android module we just created to be compiled in case we’re not targeting Android.

Compiling for Android

Let’s get ready to compile our code for Android.

Create a scripts folder inside the rust folder and add a file called android_build.sh with the following content:

#!/usr/bin/env bash
# set the version to use the library
min_ver=22
# verify before executing this that you have the proper targets installed
cargo ndk --target aarch64-linux-android --android-platform ${min_ver} -- build --release
cargo ndk --target armv7-linux-androideabi --android-platform ${min_ver} -- build --release
cargo ndk --target i686-linux-android --android-platform ${min_ver} -- build --release
cargo ndk --target x86_64-linux-android --android-platform ${min_ver} -- build --release

# moving libraries to the android project
jniLibs=../android/rusty-android/rusty-android-lib/src/main/jniLibs
libName=libdevicers.so

rm -rf ${jniLibs}

mkdir ${jniLibs}
mkdir ${jniLibs}/arm64-v8a
mkdir ${jniLibs}/armeabi-v7a
mkdir ${jniLibs}/x86
mkdir ${jniLibs}/x86_64

cp target/aarch64-linux-android/release/${libName} ${jniLibs}/arm64-v8a/${libName}
cp target/armv7-linux-androideabi/release/${libName} ${jniLibs}/armeabi-v7a/${libName}
cp target/i686-linux-android/release/${libName} ${jniLibs}/x86/${libName}
cp target/x86_64-linux-android/release/${libName} ${jniLibs}/x86_64/${libName}

Similarly to the previously suggested build script for iOS, this script will help us to compile and move the needed files to our previously created Android Library.

If you execute this bash script, once the compilation process ends you should be able to find a treeview similar to this one, with a newly created folder called jniLibs with several subfolders in it referencing several architectures:

Writing the Android Library

Finally, we’re going to write our Android Library code and consume it from our Android application.

Let’s create a new file under android/rusty-android-lib/src/main/java/com/robertohuertas/rusty_android_lib called rusty.kt. Note that the name must be the same that we used when defining our JNI functions in our Rust library.

Copy the following code in it:

package com.robertohuertas.rusty_android_lib

external fun hello(to: String): String
external fun helloDirect(to: String): String

fun loadRustyLib() {
    System.loadLibrary("rustylib")
}

Here, we just declared two signatures mirroring our two Rust functions (remember the name we gave them in our Rust code) and a function that will be called in order to dynamically load the library. Note that we’re not using the name of the library (librustylib.so) but the name we gave to the crate.

Compiling the Android Library

If you want to generate an .aar file ready to be consumed by any Android app just use the Gradle tab of your Android Studio and look for a task called assemble. Right-click on it and select Run. This will compile your library and you’ll be able to find it at android/rusty-android-lib/build/outputs/aar/rusty-android-lib-release.aar.

Consuming the Android Library

In this case, we don’t need to consume the library as an .aar file because we have it in the same project. In case you want to know how to consume it like that just take a look at the Android Documentation.

In our example, we just need to add the library as a dependency in our android/app/build.gradle:

dependencies {
    implementation project(':rusty-android-lib')
}

And then, in our Android Studio, choose File > Sync project with Gradle files.

Now, open the file content_main.xml located at android/app/src/main/res/layout and add an id to the TextView, so we can reference it later and programmatically change its value:

android:id="@+id/txt"

After this, we’re going to use our Android Library from our MainActivity.kt file located at android/app/src/main/java/com/robertohuertas/rust_android. Open it and write this:

package com.robertohuertas.rusty_android

import android.os.Bundle
import com.google.android.material.snackbar.Snackbar
import androidx.appcompat.app.AppCompatActivity
import android.view.Menu
import android.view.MenuItem
import android.widget.TextView

import kotlinx.android.synthetic.main.activity_main.*
import com.robertohuertas.rusty_android_lib.*

class MainActivity : AppCompatActivity() {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        setSupportActionBar(toolbar)

        loadRustyLib()

        findViewById<TextView>(R.id.txt).let {
            it?.text = hello("Rob")
        }

        var greeting2 = helloDirect("Rob Direct")

        fab.setOnClickListener { view ->
            Snackbar.make(view, greeting2, Snackbar.LENGTH_LONG)
                .setAction("Action", null).show()
        }
    }

    override fun onCreateOptionsMenu(menu: Menu): Boolean {
        // Inflate the menu; this adds items to the action bar if it is present.
        menuInflater.inflate(R.menu.menu_main, menu)
        return true
    }

    override fun onOptionsItemSelected(item: MenuItem): Boolean {
        // Handle action bar item clicks here. The action bar will
        // automatically handle clicks on the Home/Up button, so long
        // as you specify a parent activity in AndroidManifest.xml.
        return when (item.itemId) {
            R.id.action_settings -> true
            else -> super.onOptionsItemSelected(item)
        }
    }
}

We’re done! Run it in your emulator and you should see a text in the app saying Hello from Rust: Rob. Furthermore, if you click the button below, a snackbar will show Hello from Rust: Rob direct. As you can see, we’re using both functions, the one calling the iOS function and the one using only the jni-rs crate.

Flutter Project

And now… let’s go for the bonus points!

As we already have an Android Library and a working iOS project, making this work in a Flutter project shouldn’t be very difficult.

The basic idea is to create a Flutter plugin package so we can share our code in several Flutter projects.

So let’s start! 😀

# let's use the flutter folder
cd flutter
# create a plugin project, set its namespace and its name
flutter create --template=plugin --org com.robertohuertas rusty_flutter_lib
# now you'll have a folder called rusty_flutter_lib inside the flutter folder
# for convenience, we'll move everything to the parent directory (flutter)
# this last step is completely optional.
mv rusty_flutter_lib/{.,}* .
rm -rf rusty_flutter_lib

One of the cool things of the Flutter plugin packages is that the template comes with an example project so we can use it to test that our plugin works as expected. No need to create a new project to test our Flutter plugin.

We’re basically going to use our previous Android and iOS code and use it in our Flutter project. This should be very straightforward.

Importing the Android library

In order to use the Android Library that we built before we’re going to use Android Studio. Let’s open the flutter/android project.

Then, in order to import the Android Library, select File > New... > New Module.

Select import .JAR/.AAR Package:

And use the previously generated .AAR Package path:

By doing this, we should see a new folder called rusty-android-lib-release folder with our .aar package inside along with some other files:

Finally, open the build.gradle file in the flutter/android folder and add a new dependency:

dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
    // this is the line to add including the directory holding our .aar package:
    implementation fileTree(include: '*.aar', dir: 'rusty-android-lib-release')
}

Adding the Android platform code

Open flutter/android/src/main/kotlin/com/robertohuertas/rusty_flutter_lib/RustyFlutterLibPlugin.kt in your favorite IDE and replace the code in it with this one:

package com.robertohuertas.rusty_flutter_lib

// importing the Android library
import com.robertohuertas.rusty_android_lib.*

import io.flutter.plugin.common.MethodCall
import io.flutter.plugin.common.MethodChannel
import io.flutter.plugin.common.MethodChannel.MethodCallHandler
import io.flutter.plugin.common.MethodChannel.Result
import io.flutter.plugin.common.PluginRegistry.Registrar

class RustyFlutterLibPlugin: MethodCallHandler {
  companion object {
    @JvmStatic
    fun registerWith(registrar: Registrar) {
      val channel = MethodChannel(registrar.messenger(), "rusty_flutter_lib")
      channel.setMethodCallHandler(RustyFlutterLibPlugin())
      // dynamically loading the android library
      loadRustyLib()
    }
  }

  override fun onMethodCall(call: MethodCall, result: Result) {
    when {
      call.method == "getPlatformVersion" -> result.success("Android ${android.os.Build.VERSION.RELEASE}")
      call.method == "getHello" -> {
        val to = call.argument<String>("to")
          if (to == null) {
            result.success("No to parameter found")
          } else {
            // we're using the helloDirect function here
            // but you could also use the hello function, too.
            val res = helloDirect(to)
            result.success(res)
          }
      }
      else -> result.notImplemented()
    }
  }
}

Importing the iOS code

Importing the iOS code is fairly easy:

# copy the header to the Classes folder
cp ../rust/rustylib.h ios/Classes
# create a new libs folder
mkdir ios/libs
# copy the universal library into the libs folder
cp ../rust/target/universal/release/librustylib.a ios/libs/

Then open the rusty_flutter_lib.podspec file and add this line:

s.ios.vendored_library = 'libs/librustylib.a'

Adding the iOS plaform code

Open the flutter/ios/Classes/RustyFlutterLibPlugin.swift file and add the following code:

import Flutter
import UIKit

public class SwiftRustyFlutterLibPlugin: NSObject, FlutterPlugin {
  public static func register(with registrar: FlutterPluginRegistrar) {
    let channel = FlutterMethodChannel(name: "rusty_flutter_lib", binaryMessenger: registrar.messenger())
    let instance = SwiftRustyFlutterLibPlugin()
    registrar.addMethodCallDelegate(instance, channel: channel)
  }

  public func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) {
    if (call.method == "getPlatformVersion") {
      result("iOS " + UIDevice.current.systemVersion)
    } else if (call.method == "hello") {
      let res = hello("Rob")
      let sr = String(cString: res!)
      hello_release(UnsafeMutablePointer(mutating: res))
      result(sr)
    } else {
      result("No method found")
    }
  }
}

Connect the API and the platform code

Open the flutter/lib/rusty_flutter_lib.dart file and add a new static method in the RustyFlutterLib class:

static Future<String> hello({to: String}) async {
  final String greetings = await _channel.invokeMethod('hello', {'to': to});
  return greetings;
}

Testing the Flutter example app

Let’s open the flutter/example/lib/main.dart file and let’s consume our recently created Flutter package:

import 'package:flutter/material.dart';
import 'dart:async';

import 'package:flutter/services.dart';
import 'package:rusty_flutter_lib/rusty_flutter_lib.dart';

void main() => runApp(MyApp());

class MyApp extends StatefulWidget {
  @override
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  String _platformVersion = 'Unknown';
  String _greeting = '';

  @override
  void initState() {
    super.initState();
    initPlatformState();
  }

  // Platform messages are asynchronous, so we initialize in an async method.
  Future<void> initPlatformState() async {
    String platformVersion;
    String greeting;
    // Platform messages may fail, so we use a try/catch PlatformException.
    try {
      platformVersion = await RustyFlutterLib.platformVersion;
      greeting = await RustyFlutterLib.hello(to: 'Rob');
    } on PlatformException {
      platformVersion = 'Failed to get platform version.';
      greeting = 'Failed to get hello';
    }

    // If the widget was removed from the tree while the asynchronous platform
    // message was in flight, we want to discard the reply rather than calling
    // setState to update our non-existent appearance.
    if (!mounted) return;

    setState(() {
      _platformVersion = platformVersion;
      _greeting = greeting;
    });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Plugin example app'),
        ),
        body: Center(
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: <Widget>[
              Text(
                _greeting,
                style: TextStyle(fontSize: 20),
              ),
              Text('Running on: $_platformVersion\n'),
            ],
          ),
        ),
      ),
    );
  }
}

After that, go to the example/ios folder and execute this:

pod install

Then, open example/android/app/build.gradle and change the minSdkVersion from 16 to 22, as our library was built with this setting, too.

And finally, open the example/pubspec.yaml file and add the line below to avoid this issue when building the iOS version:

version: 1.0.0+1

Congratulations for arriving here! 👏

Let’s run the example app and see if it’s working.

In Android you should see something similar to the screenshot below:

Similarly, in iOS:

Thanks for finishing this long reading! I hope you found it useful! 😊

Bibliography

--
Originally published at robertohuertas.com on October 27, 2019.

Building an Android service that never stops running

Roberto Huertas — Sun, 30 Jun 2019 02:59:05 +0000

I've been struggling these days trying to find a way to run an endless service in Android. This is just a guide for all of you who pursue the same goal. Hope it helps! 😁

The problem

Due to Android battery optimizations introduced in Android 8.0 (API level 26), background services have now some important limitations. Essentially, they are killed once the app is in background for a while making them worthless for our purpose of running an always-running service.

According to Android recommendations, we should be using JobScheduler, which seems to work pretty well and will handle wakelocks for us, keeping the phone awake while the jobs are running.

Unfortunately, this won't work either. JobScheduler will run jobs at Android's discretion and, on top of that, once the phone enters in Doze Mode, the frequency of these jobs being run will constantly decrease. And even worst, if you ever want to access the network -say you need to send data to your server- you won't be able to. Check out the list of restrictions Doze Mode imposes.

JobScheduler works well if you don't mind about not having access to the network and you don't care about not controlling the periodicity either. In our case, we want our service to run at a very specific frequency and never be stopped, so we'll need something else.

It's all about Foreground Services

If you've been looking over the internet for a solution to this problem it's very likely that you've eventually arrived to this page from the Android's documentation.

There, we are introduced to the different types of services that Android provides. Take a look at the Foreground Service description:

A foreground service performs some operation that is noticeable to the user. For example, an audio app would use a foreground service to play an audio track. Foreground services must display a Notification. Foreground services continue running even when the user isn't interacting with the app.

It seems to be precisely what we're looking for... an it is, indeed! 😉

Show me the code

Creating a foreground service is really a straight-forward process so I will visit and explain all the steps needed to build a foreground service that never stops.

As usual, I've created a repository with all the code in case you want to take a look at it and skip the rest of the post.

Adding some dependencies

I'm using Kotlin for this example, so we will be leveraging coroutines and the Fuel library for HTTP requests.

In order to add these dependencies we must add them to our build.gradle file:

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation"org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
    implementation 'com.android.support:appcompat-v7:28.0.0'
    implementation 'com.android.support.constraint:constraint-layout:1.1.3'
    implementation 'com.jaredrummler:android-device-names:1.1.8'

    implementation 'com.github.kittinunf.fuel:fuel:2.1.0'
    implementation 'com.github.kittinunf.fuel:fuel-android:2.1.0'
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.0-M1'

    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
}

Writing our service

Foreground Services need a notification to be shown so the user is aware that the app is still running. That makes sense if you think about it.

Note that we will have to override some of the Service callback methods that handle key aspects of the service lifecycle.

It's also very important that we use a partial wakelock so our service never gets affected by Doze Mode. Bear in mind that this will have an impact in the battery life of our phone so we must evaluate whether our use case can be handled by any of the other alternatives Android offers in order to run processes in the background.

There are some utility function calls (log, setServiceState) and some custom enums (ServiceState.STARTED) in the code, but don't worry too much. If you want to see where they come from, just take a look at the example repository.

class EndlessService : Service() {

    private var wakeLock: PowerManager.WakeLock? = null
    private var isServiceStarted = false

    override fun onBind(intent: Intent): IBinder? {
        log("Some component want to bind with the service")
        // We don't provide binding, so return null
        return null
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        log("onStartCommand executed with startId: $startId")
        if (intent != null) {
            val action = intent.action
            log("using an intent with action $action")
            when (action) {
                Actions.START.name -> startService()
                Actions.STOP.name -> stopService()
                else -> log("This should never happen. No action in the received intent")
            }
        } else {
            log(
                "with a null intent. It has been probably restarted by the system."
            )
        }
        // by returning this we make sure the service is restarted if the system kills the service
        return START_STICKY
    }

    override fun onCreate() {
        super.onCreate()
        log("The service has been created".toUpperCase())
        var notification = createNotification()
        startForeground(1, notification)
    }

    override fun onDestroy() {
        super.onDestroy()
        log("The service has been destroyed".toUpperCase())
        Toast.makeText(this, "Service destroyed", Toast.LENGTH_SHORT).show()
    }

    private fun startService() {
        if (isServiceStarted) return
        log("Starting the foreground service task")
        Toast.makeText(this, "Service starting its task", Toast.LENGTH_SHORT).show()
        isServiceStarted = true
        setServiceState(this, ServiceState.STARTED)

        // we need this lock so our service gets not affected by Doze Mode
        wakeLock =
            (getSystemService(Context.POWER_SERVICE) as PowerManager).run {
                newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "EndlessService::lock").apply {
                    acquire()
                }
            }

        // we're starting a loop in a coroutine
        GlobalScope.launch(Dispatchers.IO) {
            while (isServiceStarted) {
                launch(Dispatchers.IO) {
                    pingFakeServer()
                }
                delay(1 * 60 * 1000)
            }
            log("End of the loop for the service")
        }
    }

    private fun stopService() {
        log("Stopping the foreground service")
        Toast.makeText(this, "Service stopping", Toast.LENGTH_SHORT).show()
        try {
            wakeLock?.let {
                if (it.isHeld) {
                    it.release()
                }
            }
            stopForeground(true)
            stopSelf()
        } catch (e: Exception) {
            log("Service stopped without being started: ${e.message}")
        }
        isServiceStarted = false
        setServiceState(this, ServiceState.STOPPED)
    }

    private fun pingFakeServer() {
        val df = SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.mmmZ")
        val gmtTime = df.format(Date())

        val deviceId = Settings.Secure.getString(applicationContext.contentResolver, Settings.Secure.ANDROID_ID)

        val json =
            """
                {
                    "deviceId": "$deviceId",
                    "createdAt": "$gmtTime"
                }
            """
        try {
            Fuel.post("https://jsonplaceholder.typicode.com/posts")
                .jsonBody(json)
                .response { _, _, result ->
                    val (bytes, error) = result
                    if (bytes != null) {
                        log("[response bytes] ${String(bytes)}")
                    } else {
                        log("[response error] ${error?.message}")
                    }
                }
        } catch (e: Exception) {
            log("Error making the request: ${e.message}")
        }
    }

    private fun createNotification(): Notification {
        val notificationChannelId = "ENDLESS SERVICE CHANNEL"

        // depending on the Android API that we're dealing with we will have
        // to use a specific method to create the notification
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            val notificationManager = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager;
            val channel = NotificationChannel(
                notificationChannelId,
                "Endless Service notifications channel",
                NotificationManager.IMPORTANCE_HIGH
            ).let {
                it.description = "Endless Service channel"
                it.enableLights(true)
                it.lightColor = Color.RED
                it.enableVibration(true)
                it.vibrationPattern = longArrayOf(100, 200, 300, 400, 500, 400, 300, 200, 400)
                it
            }
            notificationManager.createNotificationChannel(channel)
        }

        val pendingIntent: PendingIntent = Intent(this, MainActivity::class.java).let { notificationIntent ->
            PendingIntent.getActivity(this, 0, notificationIntent, 0)
        }

        val builder: Notification.Builder = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) Notification.Builder(
            this,
            notificationChannelId
        ) else Notification.Builder(this)

        return builder
            .setContentTitle("Endless Service")
            .setContentText("This is your favorite endless service working")
            .setContentIntent(pendingIntent)
            .setSmallIcon(R.mipmap.ic_launcher)
            .setTicker("Ticker text")
            .setPriority(Notification.PRIORITY_HIGH) // for under android 26 compatibility
            .build()
    }
}

Time to deal with the Android Manifest

We will need some extra permissions for FOREGROUND_SERVICE, INTERNET and WAKE_LOCK. Be sure you don't forget to include them because it won't work otherwise.

Once we put them in place we will need to declare our service.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" 
          package="com.robertohuertas.endless">

    <uses-permission android:name="android.permission.FOREGROUND_SERVICE"></uses-permission>
    <uses-permission android:name="android.permission.INTERNET"></uses-permission>
    <uses-permission android:name="android.permission.WAKE_LOCK" />

    <application
            android:allowBackup="true"
            android:icon="@mipmap/ic_launcher"
            android:label="@string/app_name"
            android:roundIcon="@mipmap/ic_launcher_round"
            android:supportsRtl="true"
            android:theme="@style/AppTheme">

        <service
                android:name=".EndlessService"
                android:enabled="true"
                android:exported="false">
        </service>

        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
    </application>
</manifest>

Can you tell me how I start the service?

Yes, you're right. You see, depending on the Android version we must start the service with a particular method.

If the Android version is below API 26 we must use startService. In any other case, startForegroundService is what we must use instead.

Here you can see our MainActivity, just a screen with two buttons to start and stop the service. That's all you need to start our endless service.

Remember that you can check out the complete code in this GitHub repository.

class MainActivity : AppCompatActivity() {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        title = "Endless Service"

        findViewById<Button>(R.id.btnStartService).let {
            it.setOnClickListener {
                log("START THE FOREGROUND SERVICE ON DEMAND")
                actionOnService(Actions.START)
            }
        }

        findViewById<Button>(R.id.btnStopService).let {
            it.setOnClickListener {
                log("STOP THE FOREGROUND SERVICE ON DEMAND")
                actionOnService(Actions.STOP)
            }
        }
    }

    private fun actionOnService(action: Actions) {
        if (getServiceState(this) == ServiceState.STOPPED && action == Actions.STOP) return
        Intent(this, EndlessService::class.java).also {
            it.action = action.name
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
                log("Starting the service in >=26 Mode")
                startForegroundService(it)
                return
            }
            log("Starting the service in < 26 Mode")
            startService(it)
        }
    }
}

Bonus: Start the service on Android boot

Ok, we now have our endless service making network requests every minute as we wanted but then the user restarts the phone... and our service doesn't start again... 😞

Don't worry, we can find a solution for this, too. We will create a BroadCastReceiver called StartReceiver.

class StartReceiver : BroadcastReceiver() {

    override fun onReceive(context: Context, intent: Intent) {
        if (intent.action == Intent.ACTION_BOOT_COMPLETED && getServiceState(context) == ServiceState.STARTED) {
            Intent(context, EndlessService::class.java).also {
                it.action = Actions.START.name
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
                    log("Starting the service in >=26 Mode from a BroadcastReceiver")
                    context.startForegroundService(it)
                    return
                }
                log("Starting the service in < 26 Mode from a BroadcastReceiver")
                context.startService(it)
            }
        }
    }
}

Then, we'll modify again our Android Manifest and add a new permission (RECEIVE_BOOT_COMPLETED) and our new BroadCastReceiver.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          package="com.robertohuertas.endless">
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE"></uses-permission>
    <uses-permission android:name="android.permission.INTERNET"></uses-permission>
    <uses-permission android:name="android.permission.WAKE_LOCK" />
    <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>

    <application
            android:allowBackup="true"
            android:icon="@mipmap/ic_launcher"
            android:label="@string/app_name"
            android:roundIcon="@mipmap/ic_launcher_round"
            android:supportsRtl="true"
            android:theme="@style/AppTheme">

        <service
                android:name=".EndlessService"
                android:enabled="true"
                android:exported="false">
        </service>

        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>

        <receiver android:enabled="true" android:name=".StartReceiver">
            <intent-filter>
                <action android:name="android.intent.action.BOOT_COMPLETED"/>
            </intent-filter>
        </receiver>

    </application>
</manifest>

Take into account that the service won't be rebooted unless it was already running. That's how we programmed it, it's not that it has to be like that.

Anyway, if you want to test this, just spin up one emulator with Google Services in it and be sure to be running adb in root mode.

adb root
# If you get an error then you're not running the proper emulator.
# Be sure to stop the service
# and force a system restart:
adb shell stop
adb shell start
# wait for the service to be restarted!

Bonus 2: Restart the service when the task is removed

Michal Materowski wrote to me with this case and its solution, so kudos for him!

Theoretically, according to Android documentation, returning RETURN_STICKY from the service's onStartCommand method should be enough for Android to keep the foreground service running.

Michal was testing all this with a Xiaomi Note 5 with Android Pie and every time he swiped an app from recent apps, it worked flawlessly. Unfortunately, whenever he pressed the Clear all recent apps button (MIUI specific), the service was stopped and the notification was gone. The Clear all recent apps button was probably doing some sort of battery life optimization by killing all processes and their associated services.

He found out that Android documentation states that onTaskRemoved "is called if the service is currently running and the user has removed a task that comes from the service's application.", so the plan was to leverage this to restart the service. Bear in mind, though, that onTaskRemoved is not called if the application is killed in any other way (e.g. stopped from the phone settings).

Add these lines to your service:

override fun onTaskRemoved(rootIntent: Intent) {
    val restartServiceIntent = Intent(applicationContext, EndlessService::class.java).also {
        it.setPackage(packageName)
    };
    val restartServicePendingIntent: PendingIntent = PendingIntent.getService(this, 1, restartServiceIntent, PendingIntent.FLAG_ONE_SHOT);
    applicationContext.getSystemService(Context.ALARM_SERVICE);
    val alarmService: AlarmManager = applicationContext.getSystemService(Context.ALARM_SERVICE) as AlarmManager;
    alarmService.set(AlarmManager.ELAPSED_REALTIME, SystemClock.elapsedRealtime() + 1000, restartServicePendingIntent);
}

You can check out the original Michal Materowski's PR with the whole code.

IMPORTANT: Michal had to manually set Autostart permission, otherwise the service was not started on boot.

According to Michal, some people mentioned that setting stopWithTask flag might help, but it didn't make a difference for him:

<service android:name=".EndlessService"" android:stopWithTask="false" />

Lots of kudos to Michal Materowski for his help on this case.

Enjoy! ❤️

--
Originally published at robertohuertas.com on June 29, 2019.