DEV Community: Sylvain Corsini

Why I choose to use a mono-repo for a large Gradle project

Sylvain Corsini — Thu, 18 Jun 2020 22:50:32 +0000

First of all, mono-repo doesn’t mean monolithic, even if monolithic means mono-repo ;)
And this post didn’t apply for Gradle project only.

I'm the only one developer in my company and working on a very large project. I have to quickly publish new features, fixes and other things.. When I'm vacant or when I'm busy on a feature, the project is directly impacted. I have to rapidly manage the whole project, be sure everything is robust because I can't waste time in trivialities.

I go through the multi-repo pattern. I split my project into different repositories. I quickly got dozens of repos and more as the project grows.

Multi-repo is not the best solution when you're the only one developer and you don't have the workload to manage everything. I will explain you why with my short-story.

The issue with multi-repo

Managing dependencies inside all my different repository became hard

One day I tried using a Kotlin feature I already used in another repository and it didn't work... What's missing in my project to use that simple feature introducing in the 1.3.60 version... Oh, I understand.

I saw that:

plugins {
    id "org.jetbrains.kotlin.jvm" version "1.3.61"
}

And in another repository:

plugins {
      id "org.jetbrains.kotlin.jvm" version "1.3.52"
}

Ok... I forgot to update my Kotlin version in the repository... Oups.

After investigations, a lot of dependencies through all repositories were out-dated and not only dependencies but also plugins. Updating the dependencies in all repository was very boring and difficult.

One day, a new GitHub feature appears : Dependabot. Look here. I will talk about it in another post.

I did add it in every repository to help me managing the dependencies version. Every week, I had to do the exact same thing through all my repositories... Bump dependencies version and be sure it wont break anything.

So time-consuming.. I'm alone and I waste time with dependencies... Frustrating.

Sharing code between repository is painful

The bad story began when I wanted to use something from a repo to another... I start copy-pasting the needed code through the repositories. Thinking, I will manage that later. We all know code duplication is a very bad idea.

I did a breaking change in that shared part, a very breaking one, and... A little repository lost in the big sea of repositories did not received its update. You know what ? Everything did blow. I quickly fix that of course but... What if my project was over-used like Facebook ?

This issue finished me. I said stop. Stop copy-pasting code, it's hell. I start publishing all my repo in a private maven repository. I quickly go through the free-tier but I will certainly found a solution for that.

Managing packages is very time-consuming. Really ! When I did publish a repo somewhere, I have to update it everywhere. And.... The same pattern did appear. I forgot to change the version in one repository and bugs appears again.

Publishing private repository was not the solution. (even with Dependabot since it didn't have read access to my private repository).

Multi-repositories = multiples projects in the IDE and hard to fix issues

When I have to add a new feature, I sometimes have to modify multiple services, since one change may have impact in others.

When I got a bug, it was sometimes difficult to know each part is really affected and which one create the bug. It may arrive that an issue comes from multiple services or we have to fix it in different services. Issue-tracking is hard through dozen of repository.

Managing all the different projects under my IntelliJ IDEA became difficult. I often had multiple window opened.

Mono-repo, my answer

You understand I did waste a lot of time with a lot of trivial things... Instead of being focused on features, I was focused on managing repositories and dependencies.

I dive into a mono-repo project. It was quite difficult at the first steps. I did lost some days and then.. I manage to have a mono-repo with all my repositories without losing commits. With one simple git command:

git subtree add --prefix=projects/service-A git://github.com/my-company/service-A.git master

By doing this for every repository, I finally got a directory (projects) with all my services.

The philosophy here is: one build.gradle to rule them all.

All dependencies and plugins are managed by the root build.gradle and every micro-services are splited into two project x-service and x-shared. And I can easily include the x-shared project from any other projects inside my mono-repo. I don't have to deal with maven packages and any other time-consuming thing.

I so have something like:

 - build.gradle
 - settings.gradle
 - projects
        - first-service
        - first-shared
        - second-service
        - third-service
        - thrid-shared
        - ...

The root build.gradle is managing everything for every sub-projects thanks to the allproject and subprojects component.

Since I got a lot of sub-projects, instead of included them all, I did a easy script in my root settings.gradle file:

file("projects").eachDir { p ->
    include p.name
    project(":${p.name}").projectDir = p
}

It includes every sub-projects from the projects directory and rename them to be :first-service instead of :projects:first-service which is awkward.

Merging everything in one repository is also so good for my IDE. Everything fall in my hand quickly when clicking on classes through the whole services.

A final note

Mono-repo comes with other complications. But I manage to deal with all of them and now, mono-repo help me a lot in my daily job. I can finally focus on my code.

Of course, mono-repo is not a solution for everyone. It's a case-by-case. Just : think about it, it's not a absurd, unstable or unviable solution as some peoples says.

I will explain how I manage dependencies, plugins, CI/CD and everything else with a Gradle mono-repo in others posts. Don't forgot to follow me to see what's coming next.

I will be glad to hear about your experiences in the comments !

Improve the performances of your Phoenix app with Rust: in both back and front

Sylvain Corsini — Wed, 10 Jun 2020 22:52:02 +0000

I know, Phoenix and Elixir are quite performant and quite fast. But sometimes, it may be necessary to improve a little bit the performances of your app. Rust is the solution in both your back-end and front-end code.

Here I will show you how to do that with very basic examples.

First let's install all required tools for Rust:

# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install rust-wasm
curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh 
# Install cargo generate
cargo install cargo-generate

I assume you have elixir (with the mix phx_new archive) and node installed.

Add Rust in our back-end code

In Erlang, we have the NIF functions which are callable functions implemented in C/C++ or even Rust directly from our Erlang/Elixir code. Thanks to rustler, making those functions became very easy !

We will start by making a new Phoenix project.

mix phx.new rust_phoenix --live --no-ecto
# if asked, install all deps
cd rust_phoenix
mix phx.server

You shall see the default live page which search in the HexDocs.
We will change that page to make a simple calculator, change the lib/rust_phoenix_web/live/page_live.html.leex file :

<form phx-change="change" phx-submit="compute">
    <input type="number" name="a" value="<%= @a %>" list="results" autocomplete="off"/>
    <input type="number" name="b" value="<%= @b %>" list="results" autocomplete="off"/>
    <h2>The result is: <%= @result %></h2>
    <button type="submit">Compute</button>
</form>

And then let's change the back-end code of our live page in lib/rust_phoenix_web/live/page_live.ex :

defmodule RustPhoenixWeb.PageLive do
  use RustPhoenixWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    result = 0
    {:ok, assign(socket, a: 0, b: 0, result: result)}
  end

  @impl true
  def handle_event(
        "compute",
        _,
        %{assigns: %{a: a, b: b}} = socket
      ) do
    result = a + b
    {:noreply, assign(socket, result: result)}
  end

  @impl true
  def handle_event(
        "change",
        %{"a" => a, "b" => b},
        socket
      ) do
    {a, _} = Integer.parse(a)
    {b, _} = Integer.parse(b)
    {:noreply, assign(socket, a: a, b: b)}
  end
end

Ok, easy. We just have two variables and we will compute the result when clicking on the "Compute" button.

It could be better but we will let it like that. When looking on our browser we see the both input and when typing 1 and 2 and then clicking on the compute button : we got 3. Awesome !

Let's now add our Rust project to make our computation in native code.
Add the rustler deps, in our mix.exs file add:

defp deps do
  [
    # your deps
    {:rustler, "~> 0.21.1"}
  ]
end

We have to register the rustler compiler, in our mix.exs add:

def project do
  [
    # ...
    # I did just add :rustler here
    compilers: [:phoenix, :gettext, :rustler] ++ Mix.compilers(),
  ]
end

And we will generate our awesome NIF project, when asked give the module name, I decided to give RustPhoenix.RustHelloWorld.

mix rustler.new

And now in native/rustphoenix_rusthelloworld we got our project. Take a look at native/rustphoenix_rusthelloworld/src/lib.rs file, here what it looks like:

#[macro_use]
extern crate rustler;

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler_atoms! {
        atom ok;
    }
}

rustler::rustler_export_nifs! {
    "Elixir.RustPhoenix.RustHelloWorld",
    [
        ("add", 2, add)
    ],
    None
}

fn add<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let num1: i64 = args[0].decode()?;
    let num2: i64 = args[1].decode()?;

    Ok((atoms::ok(), num1 + num2).encode(env))
}

(Add the #[macro_use] extern crate rustler; at the beginning of the file if it is missing.)

Let's add the glue to use our module in our Elixir project.
First we have to register our crate, in our mix.exs file add:

def project do
  [
    # ...
    # I did just add :rustler here
    rustler_crates: [
      rustphoenix_rusthelloworld: [
        mode: (if Mix.env() == :prod, do: :release, else: :debug)
      ]
    ],
  ]
end

Then we will create our glue module, create the file lib/rust_phoenix/rust_hello_world.ex and write :

defmodule RustPhoenix.RustHelloWorld do
  use Rustler, otp_app: :rust_phoenix, crate: :rustphoenix_rusthelloworld

  def add(_arg1, _arg2), do: :erlang.nif_error(:nif_not_loaded)
end

We did have exported ("add", 2, add) from our lib.rs file, so we have to declare the add/2 function in our module. All functions will be override by Rustler when the NIF functions will be correctly loaded.

Then let's change our live page to use our native code, edit lib/rust_phoenix_web/live/page_live.ex :

defmodule RustPhoenixWeb.PageLive do
  use RustPhoenixWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    {:ok, result} = RustPhoenix.RustHelloWorld.add(0, 0)
    {:ok, assign(socket, a: 0, b: 0, result: result)}
  end

  @impl true
  def handle_event(
        "compute",
        _,
        %{assigns: %{a: a, b: b}} = socket
      ) do
    {:ok, result} = RustPhoenix.RustHelloWorld.add(a, b)
    {:noreply, assign(socket, result: result)}
  end

  @impl true
  def handle_event(
        "change",
        %{"a" => a, "b" => b},
        socket
      ) do
    {a, _} = Integer.parse(a)
    {b, _} = Integer.parse(b)
    {:noreply, assign(socket, a: a, b: b)}
  end
end

Instead of doing a + b, we now do RustPhoenix.RustHelloWorld.add(a, b).

Now, test, and everything should work !

Add Rust in our front-end code

Let's go to our assets folder and create a native folder. Then generate our project and when asked type helloworld.

cd assets
mkdir native
cargo generate --git https://github.com/rustwasm/wasm-pack-template

In assets/native/helloworld/src/lib.rs we have our Rust code containing :

mod utils;

use wasm_bindgen::prelude::*;

#[cfg(feature = "wee_alloc")]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;

#[wasm_bindgen]
extern {
    fn alert(s: &str);
}

#[wasm_bindgen]
pub fn greet() {
    alert("Hello, World!");
}

It simply define a greet() function calling the alert javascript function.
Now let's compile our project:

wasm-pack build

We should now have a assets/native/helloworld/pkg directory created with all our glues and generated wasm file.
Now we can add our package in our asset webpack config, so edit assets/package.json :

"dependencies": {
  // ...
  "helloworld": "file:./native/helloworld/pkg"
},

And run:

npm install

We now have our Rust WASM code in our assets, ready to be used. The loading of WebAssembly code must be asynchronous and to achieve that, it's simple : let's modify our assets/js/app.js to add our new code:

import ...
import("helloworld")
    .catch(console.error)
    .then(module => window.helloworld = module);

Now we can call our code, edit lib/rust_phoenix_web/live/page_live.html.leex and add:

<button onClick="helloworld.greet()">Say Hello to Rust</button>

We now have to register the application/wasm MIME type required for our WebAssembly loader since Phoenix don't know it. Let's add it in our config file config/config.exs :

config :mime, :types, %{"application/wasm" => ["wasm"]}

And to rebuild the MIME deps, run:

mix deps.clean mime --build

Then when clicking on our button, it should pop-out our alert. Awesome !

Conclusion

Here the performances are not quite amaizing, but we can have so much more of course.

Rust is a very good language with high performance and high reliability. Used in both back and client-side code assure a solid foundation. However, we should not abuse of all of that good things, keep using Javascript and Elixir and only use Rust to optimize some heavy part of your code which are too slow.

Improvements

We can easily tell Webpack to watch for changes done in the native projects and recompile them. Take a look here.

I actually don't know how to watch, re-compile and live-reload the Rust NIF functions. Maybe tricking with nodemon (here) could be a good idea.

Docs and links

The repository with all the code : github.com/scorsi/rust_phoenix.
The Rust language main page on rust-lang.org.
The Rust WASM documentation on rustwasm.github.io, you should specially look here.
The Rustler documentation on hexdocs.pm.
Erlang NIF documentation on erlang.org.
The typical Discord use-case on blog.discord.com.
A good post on Rust NIF on theguild.nl.
WebAssembly use-case examples on webassembly.org.

I did create my first open source project for Elixir: Umbra, the next ExActor

Sylvain Corsini — Sat, 06 Jun 2020 12:57:41 +0000

I was looking for projects I can create for the Elixir community

When looking on the first used Hex packages, I did saw ExActor in the firsts pages. This project is not maintained since 2017 and I was really surprised to see "10K download yesterday"... It is a really good project which solves a minor issue with GenServer : there is a lot of boilerplat and code duplications... This is quite frustrating. ExActor solves that issues by adding macros to dynamically generate code at compile time.

What the issues with ExActor, you may ask ?

It is not extensible, it generates a lot of warnings, it did not generate optimized code and it didn't support last GenServer features like continue since the package is outdated.

Here we don't have any problem:

defmodule MyGenServer do
  use Exactor.GenServer

  defcast set_state(new_state),
    do: new_state(new_state)
end

defmodule Generated_MyGenServer do
  use GenServer

  def set_state(pid, new_state),
    do: GenServer.cast(pid, {:new_state, new_state})
  def handle_cast({:new_state, new_state}, _state),
    do: {:noreply, new_state}
end

But what's with this:

defmodule MyGenServer do
  use Exactor.GenServer

  defcast do_complicated_thing([head | _nothing], %{name: "name", id: id}, 42 = _b, %MyStruct{this: this} = c),
    do: complicated_stuff(id, head, c) 
end

defmodule Generated_MyGenServer do
  use GenServer

  def do_complicated_thing(pid, [head | _nothing], %{name: "name", id: id}, 42 = _b, %MyStruct{this: this} = c),
    do: GenServer.cast(pid, {:do_complicated_thing, [head | _nothing], %{name: "name", id: id}, _b, c)

  def handle_cast({:do_complicated_thing, [head | _nothing], %{name: "name", id: id}, 42 = _b, %MyStruct{this: this} = c}, _state),
    do: complicated_stuff(id, head, c)
end

And here it's a compiler warning party in the client-side function. Kind of errors here:

The variable this is defined but not used, declare it like _this.
The variable _b is used but named with a _, declare it like b.
The variable _nothing is used but named with a _, declare it like nothing.
...

But in fact the code the user wrote is absolutely right (for the server-side function at least) !

So Umbra did born

I so created my own package : Umbra. Which aims to resolve every issues ExActor does. It's still in development but I manage to have a working version which does resolves those ExActor issues. Unfortunately, my package is not complete at the actual time.

I read the whole ExActor code, I try to understand it... and from what I remember, the code is not really comprehensive not because it is not documented but because it goes all over the place.

I tried to create my package in a certain manner that it could be easily read and easily understand. The only complicated part of my package is when it navigate through the Elixir AST to alter the code to resolve shadow/unshadow warnings and to optimize it.

In the last example:

defmodule MyGenServer do
  use Umbra.GenServer

  # This is not the same function declaration as ExActor
  defcast {:do_complicated_thing, [head | _nothing], %{name: "name", id: id}, 42 = _b, %MyStruct{this: this} = c},
    do: complicated_stuff(id, head, c) 
end

defmodule Generated_MyGenServer do
  use GenServer

  def do_complicated_thing(pid, [_head | _nothing] = umbra_var_1, %{name: "name", id: _id} = umbra_var_2, 42 = b, %MyStruct{this: _this} = c),
    do: GenServer.cast(pid, {:do_complicated_thing, umbra_var_1, umbra_var_2, b, c)

  # The same handle_cast than ExActor
end

What's to notice:

Umbra created umbra_var_* to optimize calls,
Umbra did shadow unused variables,
Umbra did unshadow used variables.

Amazing.

For those who are asking how did I manage to do this, check the ArgumentsGenerator Module and the FunctionGenerator Module. I will create a post on how to understand and navigate through the Elixir AST code. This is pretty easy to do. And both module could be easily extracted to its own hex package.

Actually codes are poorly documented, sorry. I focused on features and tests before docs.

The next step of my projects is to add a lot of documentation for users and for contributors. Also guides on how to use it, how to create extensions, ...

Note: The project did have some known issues with guards, actually, variables used in when clause are potentially shadowed in the declaration which create a compile-time errors. This is why the 0.1.0 version of Umbra (on hex) did not use the last features like shadowing/unshadowing variables and variables optimizations.

Links

You can check out my package on hex or on GitHub directly.