DEV Community: Steve Pryde

A better home for your automated test results

Steve Pryde — Mon, 23 Feb 2026 20:30:00 +0000

In my previous job, I was tasked with creating the test automation for a SaaS startup, from scratch. It was a 10-person team working in a share-office. Over the next four years it would double in size each year and then got acquired by a fortune-100 company. The test automation story went from zero to a full test automation platform, with a test results database integrated with JIRA, automated test runners that auto-scale, and an interface where any developer could run their tests on demand, on any branch. This was more than a job. It was a passion project and I'm grateful for that experience.

I am proud of what I achieved in those four years, initially solo and later with a small test automation team that was eventually folded into the dev team as demand for tests increased. We were running 30,000 tests per month, and yet the automation infrastructure was still just getting started.

Two features in particular taught me what developers actually want from test tooling.

The first was the screenshot comparison tool. The MVP was written in two days and yet it became one of our most valuable features. A single endpoint for uploading a screenshot, triggered in tests by a single function call. Each screenshot was given a code in the test. The first upload with that code became the reference image. Subsequent uploads would be compared to the reference and given a score, as well as a new image generated showing the diff in two different shades. Later I built an entire UI around it including pan-and-zoom so you could compare the two images, approve/reject, or leave comments for that image code.

The second was the JIRA integration. Knowing when a test failed for the same reason that had already been captured in a ticket was invaluable. And if there wasn't a ticket, you could click a button and it would open the "create ticket" flow in JIRA with the description already pre-populated with the error message, some useful context, and a link to the original test result. Subsequent failures of the same type would search for the error message and then show that ticket against the test result. You could also mark it as a "known failure", so you would know not to waste time diagnosing it again. If the same test then failed for any other reason, you'd know it was a new failure mode.

What both of these had in common was a simple idea: test results are data, and they deserve better than a CI log.

Automated Future

Today, I'm taking what I learned from those four years and building my own platform. It's called Automated Future, and the core idea is straightforward: CI is for builds. AF is for tests.

Most projects start out with CI running your tests. That's fine for a short while but it doesn't scale. Your results end up buried in logs that expire, terminal output that's already gone, or artifacts scattered across half a dozen services. There's no single place to see your test health over time. No trends. No search. No history.

Automated Future gives your test results a dedicated home.

What it does today

AF is a test results platform with a dashboard, a CLI, and a public API, accessible via API Keys. Invite your team and set up projects for each of your apps. It works with any test framework that produces JUnit XML output, which covers Jest, Vitest, pytest, JUnit, RSpec, Go test, NUnit, xUnit, and more. It integrates into any CI pipeline — GitHub Actions, GitLab CI, Jenkins, whatever you use.

The fastest way to get results into AF is with the CLI. Install it, then wrap your existing test command:

af run -- npm test

That single command creates a test run, executes your tests, auto-discovers the JUnit XML files, parses and uploads the results, and exits with your test command's exit code so your pipeline still fails when it should. You don't change how your tests work. You just prefix the command.

Once results are flowing in, the dashboard gives you a real view of your test health:

Pass/fail charts and trend graphs over time, so you can see at a glance whether things are getting better or worse
A searchable, sortable history of every test run and result across all your projects
A custom query language for filtering your data — JIRA users will find the syntax familiar (status = Failed AND duration > 5000, started_at > -7d)
Artifact storage with in-browser previews — upload screenshots, videos, logs, or JSON to any test result and view them directly in the dashboard with syntax highlighting, image zoom, and video playback

Where it's going

The features I described earlier — the screenshot comparison, the JIRA integration, the known failure tracking — those are all on the roadmap. But rather than building them in an opinionated way, I want to build them with input from developers who are actually using the platform. The problems I solved at my previous company aren't unique. Every team doing test automation hits the same pain points. I'd rather build solutions shaped by real feedback than assumptions.

Try it out

Automated Future is live and free to get started with. You can be up and running in about five minutes:

Sign up at dashboard.automatedfuture.co
Create a project and grab your API key
Install the CLI in your pipeline
Wrap your test command with af run
Open the dashboard and see your results

The documentation has step-by-step guides for every major CI system and test framework.

If you're interested in helping to shape where this goes, I'd love to hear from you. Try it out, tell me what's missing, tell me what matters to you. Let's build a better home for test results together.

Create a desktop app in Rust using Tauri and Yew

Steve Pryde — Sun, 16 Jan 2022 05:31:38 +0000

I recently created a complete desktop app in Rust using Tauri and Yew. A few people expressed interest in how I did this, so here is a tutorial to show how to get started.

If you're into Rust and want to build desktop apps, Tauri is a great choice. It's still very early days for web front-ends in Rust but Yew is already quite usable as you will see.

So let's get started.

First let's create a directory for our new project. We'll create a monorepo containing separate directories for the front-end and the back-end.



mkdir tauri-yew-demo
cd tauri-yew-demo
git init

If you like, you can create a new github repository for it and add the git origin following the instructions here.

All of the code in this tutorial is available on my github if you want to just download it and follow along:
https://github.com/stevepryde/tauri-yew-demo

Front-end

Next we'll create the front-end directory. We do this first because later the tauri setup will ask where this directory is.



cargo new --bin frontend
cd frontend
mkdir public

(The public directory is where our CSS will go)

Since this will be the Yew part of the project, let's follow the setup instructions here: https://yew.rs/docs/getting-started/introduction



rustup target add wasm32-unknown-unknown
cargo install trunk
cargo install wasm-bindgen-cli

Next create index.html in the base of the frontend directory:

frontend/index.html



<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Tauri Yew Demo App</title>

    <link data-trunk rel="css" href="/public/main.css"/>
  </head>
  <body></body>
</html>

We'll also create main.css in the public/ directory we created earlier:

frontend/public/main.css



body {
    margin: 20px;
    background: #2d2d2d;
    color: white;
}

.heading {
    text-align: center;
}

You can now build your frontend app and view it in the browser! Let's test it out:



trunk build
trunk serve

Now open your browser and go to http::/localhost:8080. You should see a blank page with a grey background.

You may be wondering why we are viewing this in a browser when we are supposed to be creating a desktop app. We will get to that soon. Tauri essentially bundles your web app inside a desktop app using the OS-provided browser rendering engine to display your front-end app. So the process of creating your front-end part is very similar to creating a regular web front-end, except that instead of making HTTP requests to an external web server, we will make function calls to the tauri back-end via some Javascript glue code (there's very little Javascript required, I promise!).

Adding Yew

We haven't yet added yew to the project, so let's do that now.

To install rust packages to your project, I highly recommend the cargo-edit tool, which you can install via cargo install cargo-edit.

With cargo-edit installed you can do:



cargo add yew

If you prefer to add dependencies manually, just add the following lines to the dependencies section of Cargo.toml:



yew = "0.19.3"

Now open src/main.rs and replace its contents with the following:

frontend/src/main.rs



use yew::prelude::*;

fn main() {
    yew::start_app::<App>();
}

#[function_component(App)]
pub fn app() -> Html {
    html! {
        <div>
            <h2 class={"heading"}>{"Hello, World!"}</h2>
        </div>
    }
}

Now you should be able to re-run trunk serve and see the updated page in your browser. This is a real Rust front-end, compiled to WASM, running in your browser. Yes, it is that easy!

By the way, if you ever wish to make a regular Yew web front-end, you can follow the exact same process and just continue on with this front-end as a standalone project.

Adding Tauri

For this step you'll need to be back in the base tauri-yew-demo directory. Hit Ctrl+C if you're still running trunk serve.

If you're still in the frontend directory, go up one directory:



cd ..

Tauri has its own CLI that we'll use to manage the app.

There are various installation options but since we're using Rust, we'll install it via cargo.

The Tauri CLI installation instructions can be found here: https://tauri.studio/docs/getting-started/beginning-tutorial#alternatively-install-tauri-cli-as-a-cargo-subcommand



cargo install tauri-cli --locked --version ^1.0.0-rc

EDIT: Many readers have had trouble installing the Tauri CLI due to various bugs in previous versions or compatibility issues with newer Rust versions. I recommend checking the official docs and following the instructions there, especially as the Tauri CLI version gets updated in the future.

Then run cargo tauri init which will ask you some questions about the app:



$ cargo tauri init
What is your app name?: tauri-yew-demo
What should the window title be?: Tauri Yew Demo
Where are your web assets (HTML/CSS/JS) located, relative to the "<current dir>/src-tauri/tauri.conf.json" file that will be created?: ../frontend/dist
What is the url of your dev server?: http://localhost:8080

This will create a src-tauri directory, containing all of the back-end code for your app.

You can run cargo tauri info to check all of the installation details.

Before we can run the app, we need to tell tauri how to actually start the front-end server.

Edit the tauri.conf.json file that is in the src-tauri directory. This file contains all of the tauri config that tells tauri how to build your app. You will want to bookmark https://tauri.studio/en/docs/api/config/ because it contains a lot of very useful info about all of the options in this file. For now we will just update the build settings.

Replace the build section with the following:

src-tauri/tauri.conf.json



"build": {
    "distDir": "../frontend/dist",
    "devPath": "http://localhost:8080",
    "beforeDevCommand": "cd frontend && trunk serve",
    "beforeBuildCommand": "cd frontend && trunk build",
    "withGlobalTauri": true
  },

This means we won't need to manually run trunk serve in another tab while developing. We can develop in the front-end and back-end and both will hot-reload automatically when changes are made.

Ok, we're now ready to start the tauri dev server that you'll use while developing your app:



cargo tauri dev

And there's your desktop app, running with tauri!

Adding tauri commands

Tauri itself is written in Rust and your back-end application will be a Rust application that provides "commands" through tauri in much the same way that a Rust web-server provides routes. You can even have managed state, for things like sqlite databases!

Open src/main.rs in the src-tauri project and add the following below the main() function:

src-tauri/src/main.rs



#[tauri::command]
fn hello(name: &str) -> Result<String, String> {
  // This is a very simplistic example but it shows how to return a Result
  // and use it in the front-end.
  if name.contains(' ') {
    Err("Name should not contain spaces".to_string())
  } else {
    Ok(format!("Hello, {}", name))
  }
}

You will also need to modify the main() function slightly to tell tauri about your new command:

src-tauri/src/main.rs



fn main() {
  tauri::Builder::default()
    .invoke_handler(tauri::generate_handler![hello])
    .run(tauri::generate_context!())
    .expect("error while running tauri application");
}

That's all for the back-end.

For more details on what you can do with tauri commands, including async commands, error handling and accessing managed state, I highly recommend reading through all of the sections in the tauri docs here: https://tauri.studio/en/docs/guides/command

Accessing tauri commands from Rust in the front-end

In order to access tauri commands on the front-end in Rust/WASM we need to add some glue code in Javascript.

There is a tauri npm package but since we want to keep any Javascript usage to a minimum and prefer to use cargo and trunk to manage our front-end, we will use a different approach.

Tauri also exports its functions via the window object in Javascript.

In our front-end project, create a new file called glue.js inside the public/ directory.

frontend/public/glue.js



const invoke = window.__TAURI__.invoke

export async function invokeHello(name) {
    return await invoke("hello", {name: name});
}

That is all the Javascript we will add. I told you it would be minimal!

Notice that the Javascript function is async. That is because the tauri invoke() command returns a Javascript promise. And through WASM, Javascript promises can be converted into Rust futures. Just let that awesomeness sink in for a while :)

How do we call that Javascript function from Rust? We add more glue code, this time in Rust. Add the following in your frontend's main.rs:

frontend/src/main.rs



#[wasm_bindgen(module = "/public/glue.js")]
extern "C" {
    #[wasm_bindgen(js_name = invokeHello, catch)]
    pub async fn hello(name: String) -> Result<JsValue, JsValue>;
}

Here we tell wasm_bindgen about our javascript code in /public/glue.js. We give it the javascript function name, and the catch parameter tells wasm_bindgen that we want to add a catch() handler to the javascript promise and turn it into a Result in Rust.

This new code won't work yet, because we haven't added wasm-bindgen to our front-end dependencies. Let's do that now. We'll also add wasm-bindgen-futures which we'll use to call this later, web-sys which provides access to the browser window object, and js-sys which gives us the JsValue types we referenced above.



cargo add wasm-bindgen
cargo add wasm-bindgen-futures
cargo add web-sys
cargo add js-sys

This will add the following dependencies to Cargo.toml in the front-end:



wasm-bindgen = "0.2.78"
wasm-bindgen-futures = "0.4.28"
web-sys = "0.3.55"
js-sys = "0.3.55"

We also need to add some imports to the top of main.rs.

frontend/src/main.rs



use wasm_bindgen::prelude::*;
use wasm_bindgen_futures::spawn_local;
use web_sys::window;

So now we've added the glue code in Javascript and the Rust code that calls that method. What we need to do is wire that up to the front-end code and actually use it.

Calling Tauri commands in Yew code

In version 0.19 Yew added function components, which operate much like React hooks if you're familiar with the React framework in Javascript. If you're not familiar with that, I will attempt to give a brief explanation of the concepts used in this tutorial, but a fuller explanation of hooks is out of scope. The yew docs themselves go into some more detail.

Replace the function component in your front-end main.rs with the following:

frontend/src/main.rs



#[function_component(App)]
pub fn app() -> Html {
    let welcome = use_state_eq(|| "".to_string());
    let name = use_state_eq(|| "World".to_string());

    // Execute tauri command via effects.
    // The effect will run every time `name` changes.
    {
        let welcome = welcome.clone();
        use_effect_with_deps(
            move |name| {
                update_welcome_message(welcome, name.clone());
                || ()
            },
            (*name).clone(),
        );
    }

    let message = (*welcome).clone();

    html! {
        <div>
            <h2 class={"heading"}>{message}</h2>
        </div>
    }
}

fn update_welcome_message(welcome: UseStateHandle<String>, name: String) {
    spawn_local(async move {
        // This will call our glue code all the way through to the tauri
        // back-end command and return the `Result<String, String>` as
        // `Result<JsValue, JsValue>`.
        match hello(name).await {
            Ok(message) => {
                welcome.set(message.as_string().unwrap());
            }
            Err(e) => {
                let window = window().unwrap();
                window
                    .alert_with_message(&format!("Error: {:?}", e))
                    .unwrap();
            }
        }
    });
}

And this should compile and run.

Navigate up one directory (to the tauri-yew-demo directory), and run:



cargo tauri dev

You should see the application window showing Hello, World!.

Ok, that was a lot of code. Let's unpack it.

First we add some state to the component through the use_state_eq hook.



let welcome = use_state_eq(|| "".to_string());
let name = use_state_eq(|| "World".to_string());

Component state has an initial value, and a method for updating that value. Normally, updating a state value will cause the component to be re-rendered with the new value.

The use_state_eq hook gives us a state handle that will only cause the component to be re-rendered if the value has changed. It uses Rust's PartialEq trait for this.

You can dereference a UseStateHandle to get the underlying state value.



    {
        let welcome = welcome.clone();
        use_effect_with_deps(
            move |name| {
                update_welcome_message(welcome, name.clone());
                || ()
            },
            (*name).clone(),
        );
    }

We need to clone the welcome handle so that we can move it into the closure.

The use_effect_with_deps() hook is a variant of the use_effect hook where this one only runs the hook when some dependency changes.

It takes two arguments. The first is the closure we want to run when the dependency value changes, and the second is the dependency value itself.

In this example the dependency is specified as (*name).clone(), which means we are dereferencing the handle to get the underlying String and then cloning the String because we cannot move it.

That value is then passed as the argument to the closure.

The use_effect*() hook is generally used to do some longer operation or some async operation where the component will update with the result of that operation. This is why we pass in the welcome handle, because this allows the closure to set the value of that handle, which will cause the component to re-render, but only if the value has changed. Calls to tauri commands will generally go inside a use_effect or use_effect_with_deps hook.

The return value of the closure is always another closure that will run when the component is unloaded from the virtual DOM. You can use this closure to perform any cleanup relating to this effect hook. In this case there is nothing to clean up so we return an empty closure.

Let's look at the update_welcome_message() function.



fn update_welcome_message(welcome: UseStateHandle<String>, name: String) {
    spawn_local(async move {
        match hello(name).await {
            Ok(message) => {
                welcome.set(message.as_string().unwrap())
            }
            Err(e) => {
                ...
            }
        }
    });
}

The spawn_local() function is from wasm_bindgen_futures and it allows us to run async functions in the current thread. In this case we need it in order to call our hello() function, because it is async.

Next we call the hello() function, which returns Result<JsValue, JsValue>. We know the JsValue will be a string since the original tauri command we implemented returned Result<String, String>, so we can just unwrap() here.

Finally if the result was Ok(..) then we set the new value of the welcome state variable. Otherwise we display an alert.

Alerts are not a great way to handle errors like this, but as a simple example it shows how the Result type is passed through to the front-end. In a proper application you would probably want to show a toast popup and optionally log to the browser console (in debug builds).
See the wasm-logger crate for an easy way to send logs to the browser console using the log crate macros.

Wrapping up

Hopefully this tutorial has given you a taste for how to create a desktop app almost entirely in Rust, using Tauri for the back-end and Yew for the front-end.

You can find all of the code for this tutorial on my github:
https://github.com/stevepryde/tauri-yew-demo

Feel free to use it and modify it as a base for your own projects.

I highly recommend reading through the documentation for both Tauri and Yew.

Tauri: https://tauri.studio/en/docs/get-started/intro
Yew: https://yew.rs/docs/getting-started/introduction

If there's anything I missed or anything that is still unclear from this tutorial, please let me know in the comments and I'll aim to update it.

Thanks for reading!

Using Selenium with Rust

Steve Pryde — Sun, 29 Nov 2020 01:57:21 +0000

Selenium is a tool for automating web browsers, and is often used for testing websites, especially for end-to-end testing. Rust is commonly known as a Systems Programming language and might not be your first choice when it comes to browser testing but it can actually work pretty well.

First, let's introduce Rust.

For those who may not be familiar with the Rust language, let me give you a small taste of what its strengths are. This won't be a comprehensive introduction, but I hope to give you a sense of why this language exists and what it can offer you.

Performance
Rust code often performs very well due to it being a statically compiled language and also due to it encouraging patterns that naturally lead to good optimizations.
Safety
Rust prevents entire classes of common bugs and mistakes, which means your code is more likely to work correctly. Safety-by-default is a very common theme in Rust.
Rich type system
Rust has a very expressive type system which allows you to more accurately model your data objects and the relationships between them. It borrows many of the best features from other languages including its familiar syntax.
Awesome and welcoming community
Rust has a fantastic community behind it, willing to help you learn and succeed with the language. Click here for more info

Required tools for selenium

To use selenium, we first need to download the selenium server from here.

Selenium server is written in Java so you will also need Java installed.

We also need a webdriver to interact with your browser of choice. We will use Chrome, so you will need chromedriver which you can download from here.

Check your version of Chrome (open Chrome and select menu > Help > About Google Chrome) and then download the version of chromedriver that corresponds to your version of Chrome.

Unzip the downloaded file and place the chromedriver executable somewhere in your PATH. You can put it in the same directory as the selenium jar file if you want.

Now start selenium like this:

java -jar selenium-server-standalone-3.141.59.jar

This should start selenium server on port 4444.

Let's see the code

In order to interact with a web browser from Rust we will use the thirtyfour library (or "crate" in Rust lingo).

It is called "thirtyfour" because 34 is the atomic number for the Selenium chemical element.

Full Disclosure: I wrote the thirtyfour crate, with contributions from several other volunteers as well.

use thirtyfour::prelude::*;
use tokio;

#[tokio::main]
async fn main() -> color_eyre::Result<()> {
    color_eyre::install()?;

    let mut caps = DesiredCapabilities::chrome();
    caps.add_chrome_arg("--enable-automation")?;
    let driver = WebDriver::new("http://localhost:4444/wd/hub", &caps).await?;

    driver.get("https://wikipedia.org").await?;

    Ok(())
}

Some brief notes about other crates we are using:

We are using tokio for async/await. If you want to learn more about async in Rust, click here.
If you prefer not to use async, there is a sync version of the crate called thirtyfour_sync.
We are also using the awesome color_eyre crate for error reporting. It makes any error messages much clearer.

Ok, so firstly we set up a DesiredCapabilities struct. In this case we want to use the Chrome presets. There are also presets for other browsers (to use other browsers you need the webdriver for that browser in your PATH).

We want to enable the automation flags for Chrome, via this line:

caps.add_chrome_arg("--enable-automation")?;

And then we start the web browser and return a WebDriver struct for us to interact with it.

If you run the above code you should see a Chrome browser open and automatically navigate to https://wikipedia.org. Note that selenium will automatically wait for the page to load. This can all happen fairly quickly so if you need it to wait at the end you can add this line (requires the time feature of tokio):

tokio::time::sleep(Duration::from_secs(5)).await;

(In previous versions of tokio, sleep was named delay_for).

Aside: The ? symbol in Rust

You may be wondering what the ? symbol is for at the end of each line. This is an error-handling feature of Rust. You can see that the main() function returns a Result. This means it can either return Ok(value) or Err(error_value). The ? is a shorthand way of handling the error without having to type out all of the boilerplate code.

It is the equivalent of doing something like this:

let result = WebDriver::new("http://localhost:4444/wd/hub", &caps).await;
let driver = match result {
    Ok(value) => value,
    Err(error) => return Err(error)
};

But this would get very tedious so instead Rust provides the ? operator so we can just do this instead:

let driver = WebDriver::new("http://localhost:4444/wd/hub", &caps).await?;

Great! Now let's make it do something

First we want to find a particular element. To do this we use an element selector. We can search by Id, Name, Class name, Tag, and we can use CSS or XPath selectors as well.

To keep things simple, let's search by Id.

let elem_search = driver.find_element(By::Id("searchInput")).await?;
elem_search.send_keys("selenium").await?;

This will find the element with the id searchInput and then type the word selenium into it.

Note that we can also search for an element that is nested below another element. Another way to locate the searchInput element is as follows:

let elem_form = driver.find_element(By::Id("search-form")).await?;
let elem_search = elem_form.find_element(By::Id("searchInput")).await?;

This will first find the element with id search-form and then it will look for an element with id searchInput that is nested below the first element. This can be useful for performing more complex element searches, however you should also check out CSS and XPath selectors which can often do the same thing in one single query.

If you want to get more information about this element, you can get CSS properties like this:

println!("Font size: {}", elem_search.get_css_property("font-size").await?);

This will display the font size of the search input element.

Ok, now let's find a button and click it.

let elem_button = elem_form.find_element(By::Css("button[type='submit']")).await?;
elem_button.click().await?;

Here you can see the CSS selector being used to locate the button element.

If preferred you can shorten the code by simply chaining these calls together, like this:

elem_form.find_element(By::Css("button[type='submit']")).await?.click().await?;

At this point we could "sleep" for a few seconds to wait for the page to load, but using "sleep" in selenium tests is often very unreliable and can result in flaky tests.

Instead, it is better to locate some element on the page that indicates the page has finished loading. So let's wait for the page heading.

driver.find_element(By::ClassName("firstHeading")).await?;

By default, thirtyfour sets a 30 second implicit wait for all element searches. For more advanced polling, see thirtyfour_query, but the implicit wait will work fine for this particular example.

If for some reason the element was not found within 30 seconds, the above call would return an error (WebDriverError::NoSuchElement). That error would be propagated up by the ? operator and would return the error from the main() function, exiting the program.

Now we can display the page title, like this:

println!("Title = {}", driver.title().await?);

And this should display "Selenium - Wikipedia" if everything has worked correctly.

So hopefully this has given you a taste of how you can automate a web browser in Rust. This is also useful for testing your web applications, especially if those applications are also written in Rust.

A little "Rusty" feature

There is one more thing I want to point out, and it's something perhaps unique to Rust.

The thirtyfour crate uses a feature of Rust called lifetimes to guarantee at compile-time that no references to the selenium session can be used after the browser has been instructed to close. It does this by sharing a reference to the session between the WebDriver and all WebElement structs related to it. When you call WebDriver::quit() (or the WebDriver struct goes out of scope) this will consume the struct and destroy the session, and this means Rust knows that all of the other references to it are now invalid and cannot be used.

The following code will not compile:

let driver = WebDriver::new("http://localhost:4444/wd/hub", &caps).await?;
driver.get("https://wikipedia.org").await?;

let elem_search = driver.find_element(By::Id("searchInput")).await?;

// The session is "moved" here, preventing further use of either the driver or elements.
driver.quit().await?; 

// This attempts to "borrow" the session after it has been moved, which Rust will not allow.
elem_search.send_keys("this won't work").await?;

The Rust compiler tells us that the call to driver.quit() here is not allowed because the driver is also "borrowed" later in the call to elem_search.send_keys().

While other selenium libraries may be able to perform such checks at runtime, Rust is able to do the check at compile-time, saving you time in debugging and making your tests a little bit more reliable from the start.

Thank you for reading. Hopefully I have shown you that interacting with a web browser in Rust can be quite easy, especially if you are already familiar with Selenium :)

For more documentation on the thirtyfour crate, click here

Why Learn Rust?

Steve Pryde — Fri, 27 Nov 2020 12:35:59 +0000

In this article I want to share my thoughts on what Rust is really good for, why it exists, and what it can offer you that other languages might not.

I find that when I learn anything new, I first need to have a reason for learning it. There needs to be a purpose. What can it offer me? The same is true for programming languages. Learning anything can bring challenges and some setbacks, and overcoming those is going to be a lot easier if we can answer that simple question... "Why?"

Bugs

I hate bugs. I hate using software that has bugs. I hate it when programs crash, or go slow, or use all of my computer's memory, or leave my data vulnerable to security flaws.

In 2020 we are still using and writing software that has the same bugs, the same mistakes, the same security issues, that were present 20 or even 30 years ago.

The human factor

Most modern programming languages focus on making coding easier, and many have succeeded. But unfortunately they often make coding bugs easier too. We need a change.

Most programming languages still leave it up to the programmer to "get everything right" and it is up to us to remember to put in all of the right safety checks. But programmers are lazy (myself included). We take shortcuts and tell ourselves we'll fix it later. We never fix it later. And so we still have software that crashes.

We tell ourselves that bugs are the programmer's fault. We just need to write better code. Follow better practices. Use better tooling. Become better programmers. Become more experienced.

But is that enough? Even experienced programmers can (and often do) make the same mistakes. We're all human.

Today we have coding guidelines, formatters, linters, static analysis, profilers, benchmarks, unit tests, entire test frameworks, CI/CD, best practices, architects, analysts, experienced software engineers, and open source. And these are all fantastic things that really do help. But they're still not enough.

We still make the same mistakes. Null pointer access, double-free, use-after-free, data races, undefined variables, type mismatch, conversion errors, unhandled exceptions, uncaught errors, and a whole lot more!

Enter Rust

This is why Rust exists. It won't prevent all bugs. No language will do that. But it can prevent entire classes of bugs and that is something we haven't really seen in this industry before in a relatively mainstream language. No null pointers. No data races. Sum types (for more accurate modeling). Exhaustive pattern matching. And a strong focus on correctness and safety by default.

I believe the whole point of Rust, its reason for being, is to allow programmers like you and me to create software that does exactly what we want, with less bugs, and runs faster. By making many types of bugs impossible, I think Rust really does deliver on that goal. Yes it has some new concepts to learn that may be different from other languages, but in return it enables you to create software that is more reliable, with better error handling, and awesome performance. It is not uncommon to spend a little more time fixing errors reported by the compiler, and then much less time debugging.

Last but not least, learning Rust will teach you a lot about common programming mistakes and how to avoid them, even when you're programming in other languages. It will change the way you code everywhere, for the better.

Rust String vs &str

Steve Pryde — Sat, 10 Oct 2020 13:22:45 +0000

"Why does Rust have different types of Strings? Why not just one String type?"

I have seen these questions posed in different ways online and it is true that Rust's String and &str types can be confusing for people new to Rust, especially people who are familiar with dynamic programming languages. So let's talk about the two string types and see if we can clear up some of the confusion.

It will make sense eventually and Rust really does need both types. They are quite different and have different use cases.

TL;DR

It's all about "who owns the memory".

String

String contains a string in memory and owns the memory for it.
Use String for returning strings created within a function or (usually) when storing strings in a struct or enum.
If you have a String you can pass a reference to it to convert it to &str.

&str

&str is just a reference to another string (slice) but does not own the memory for it.
Prefer &str in function arguments to accept string slices and make it clear the function will not mutate the string.
If you have a &str and want a new String you can clone it either by to_owned() or to_string() (they are effectively the same - use whichever makes your code clearer to read and consistent). These will copy the memory and make a new String.

It's about memory and ownership

Firstly, we need to talk a little about how Rust manages memory. I'll try to keep this brief but it will be very important later when we discuss the difference between String and &str.

As you may be aware, Rust does not have a garbage collector. That means, the compiler needs to explicitly allocate and deallocate memory at specific points in the code, and it needs a mechanism for determining when something is no longer in use so that its memory can be deallocated safely.

This mechanism in Rust is called "ownership" (and "borrowing"). Consider the following code:

fn main() {
    let owned_string = get_string();
    print_string(&owned_string);
}

fn get_string() -> String {
    let s = String::from("Hello, World!");
    // In Rust we return s by omitting the semicolon
    s
}

fn print_string(my_string: &str) {
    println!("{}", my_string);
}

Here in get_string() we allocate memory for a String on the heap (to learn more about what the heap is, see here) and assign it to the variable s. At this point we can say that s "owns" that memory. The Rust compiler will now track the scope of s in order to know when to deallocate it. This would occur when s goes out of scope, unless s is instead "moved" (which you can think of like a transfer of ownership). In the above case, s is indeed moved, into the variable owned_string at the top.

When get_string() returns, the memory isn't copied and it doesn't go anywhere. Rather, the ownership of that memory is now transferred to the caller (in this case the variable owned_string).

The Rust compiler would now track this new owner owned_string in order to know when to deallocate. In this case it would be deallocated at the end of the main() function.

A String will only ever have 1 owner. That owner can allow others to "borrow" it temporarily (see the line print_string(&owned_string);) , but the memory will only be deallocated when the owner goes out of scope (unless it is moved, in which case it will be deallocated when the new owner goes out of scope, and so on).

Borrowing and references

So now let's look at the other function:

fn print_string(my_string: &str) {
    println!("{}", my_string);
}

The & symbol means this is a reference to a str (a string slice). You will almost never use str without & so it's easier to just think of them as &str, or "a reference to a string slice".

Consider the String we had earlier with its owned, allocated memory. What if we wanted to pass a reference to that string (or part of it), but without allowing the recipient to modify it. In Rust we call that an "immutable borrow", and that is what the & means. It is like a "reference" in other languages, but Rust goes a step further and lets you declare whether or not its contents can be mutated (changed).

In the above example we could have used &String in the function signature instead of &str and that would have worked ok because we passed in the entire string anyway. However, that wouldn't allow other callers to pass in a substring (at least without copying its memory into another String). This is where a string slice &str is much more useful.

&str is effectively a pointer directly to the string's memory, with a fixed length. You will notice in the original code the line print_string(&owned_string);. We passed a reference to a String into the function, but the function wanted &str. This works because the String type can automatically convert into &str - this is just returning a pointer to the very same memory, with a known length. &str is also an immutable reference so you cannot modify the memory it points to.

In fact, Rust will ensure that the &str is safe to use (even across threads!) by guaranteeing that nothing can modify the memory it points to (even the owner), for as long as the & reference variable is in scope. Rust will also guarantee that the owned String (that the &str points to) cannot go out of scope while the reference remains in scope (otherwise the reference would be a dangling pointer). These are very powerful features of Rust's borrow checker that prevent all kinds of nasty bugs you might encounter with other languages where you create a string over here then pass a reference to it over there but it accidentally modifies the original.

The difference between String and &str is...

String holds a string in memory and owns the memory for it.

&str is just a reference to another string but it doesn't own the memory for it.

When to use either one

In general, use String in structs and enums where you want the struct or enum to own the contents. Also use String when returning a string from a function where that string was allocated within the function (Rust will not let you return a reference in that case because the memory would be deallocated at the end of the function and the reference returned would be a dangling pointer)

Prefer &str for function arguments unless you explicitly want to move a String into the function and give up ownership of it (this tends to be rare).

One last time...

Use String when you need to own the memory, for example you created a string in a function and need to return it.

Use &str when you want an immutable reference to memory that is owned by another String variable (or a string literal in your code).

Intro to Rust Modules

Steve Pryde — Wed, 07 Oct 2020 13:13:23 +0000

The Rust module system can be very confusing and difficult to understand when you are just starting out learning Rust. Something as simple as just putting a function in another file and importing it into your main.rs can send you on a frustrating search and leave you with even more questions.

In this article I will share some info that I hope will be helpful in understanding how to make better use of the module system and perhaps even start to embrace the functionality it gives you.

Why not just import by file path?

I'm sure this is the approach many of us tried first and we were disappointed when it didn't work.

As an example, let's imagine we have some file called cat.rs and it contains the function meow():

fn meow() {
    println!("Meow, Meow!");
}

And then we try to call meow() from main.rs like so:

fn main() {
    meow();
}

But Rust says no:

error[E0425]: cannot find function `meow` in this scope
 --> src/main.rs:2:5
  |
2 |     meow();
  |     ^^^^ not found in this scope

Ok. We forgot to make meow() public using the pub keyword, so we will fix that now and try again. Nope, same error.

What we might try next is something like this:

use cat;

fn main() {
    cat::meow();
}

Now we get this error:

error[E0432]: unresolved import `cat`
 --> src/main.rs:1:5
  |
1 | use cat;
  |     ^^^ no `cat` external crate

The mistake in the above code is that it assumes there is a 1 to 1 mapping between module name and our directory structure. However, Rust does not work like that, and I think this is the key point that helps in understanding how Rust modules work.

Rust's module paths are not tied directly to the filesystem. Rather we need to tell Rust a little more about the module structure before we can use our modules in our code. That is, think of the module structure as its own thing independent of the file system layout. First we need to tell Rust where each file belongs in the module hierarchy, and then we can import our module using the "module path" rather than the filesystem path.

This can seem cumbersome at first, but you will come to understand the reasons why it works this way.

To demonstrate why Rust doesn't just use file paths as the module path, let's look at an example with multiple modules in one file, like this main.rs:

mod cat {
    pub fn meow() {
        crate::dog::woof();
    }
}

mod dog {
    pub fn woof() {
        println!("Woof, Woof!");
    }
}

fn main() {
    cat::meow();
}

Obviously using separate modules for this is overkill but it demonstrates the way modules work.

Note that we need to use the pub keyword to allow each function to be visible to other modules (and you can also make things private to a module by omitting the pub keyword).

This will compile. And when we run it we get:

Woof, Woof!

So we can see that mod cat created a kind of module path with the function meow() nested below it, even though it is in the same file. And the module itself is nested below a global namespace called crate.

In this way we can declare different module namespaces in a much more flexible way than if we could only use file paths. For a simple example like this you may think it is unnecessarily complex but when you are writing a library and you want to provide a nice import structure for your API without needing to match that exact structure in your filesystem, this is a really nice feature. If you have ever used a library that let you do: use somelib::prelude::*; that is one of the benefits.

What about different files?

Ok, so hopefully multiple modules in 1 file makes sense. What about importing functions from another file?

Let's return to our first example, with meow() declared in another file, named cat.rs.

The change we need to make within main.rs is to first tell Rust about the module cat.

The way we do this is by adding the following to our main.rs:

mod cat;

That tells Rust to look for either a cat.rs in the same directory OR alternatively a directory named cat with a mod.rs file within it (but you cannot have both a cat.rs file and a cat directory). We will get to directories shortly.

In our case we have cat.rs in the same directory, so now in our main.rs we can call the function within that module like so:

mod cat;

fn main() {
    cat::meow();
}

And that works!

What about directories?

Previously I mentioned that mod cat; could point to a directory named cat containing a file called mod.rs.

Let's create this directory for our example, and then move cat.rs into that directory. Actually we will also rename cat.rs to kitten.rs to disambiguate (it's in a "child" module hehe).

So now we have a main.rs and a directory named cat containing two files, mod.rs and kitten.rs.

We could just put our meow() function in mod.rs and that would work just fine. However let's show how to import from another file within that directory.

In mod.rs we now need to tell Rust about kitten.rs, like this:

pub mod kitten;

(Note the pub keyword which makes the kitten module visible to the parent module as well)

But now our main() would need to call meow() like this:

mod cat;

fn main() {
    cat::kitten::meow();
}

What if instead we wanted to use the original import path in main(), while keeping our new file structure? Can we do that?

Yes we can! This is where Rust's module system hopefully starts to make more sense.

Try this in mod.rs:

mod kitten;
pub use kitten::meow;

And now we can once again call cat::meow() from within our main() function.

What this does is within mod.rs we tell Rust about another module, called kitten (which here points to kitten.rs in the same directory as mod.rs), which then becomes visible to the current module scope. But without the pub keyword, the nested module is invisible to the parent. Rather than make the entire module visible, we can import the meow() function from that module via use and we can make only this import public (visible to the parent module) with pub use.

We could even change the name of the function when re-exporting it, like this, in mod.rs:

mod kitten;
pub use kitten::meow as woof;

and now our main() would need to look like this:

fn main() {
    cat::woof();
}

I hope you are starting to see the flexibility that the Rust module system gives you over your entire namespace hierarchy. You can organise your files in a way that is convenient for you, and you can export your structs and modules in the way that makes more sense for the caller.

Sometimes these will be the same, and sometimes they won't. Rust lets you choose.

Declaring the module hierarchy in one place

Putting module declarations in mod.rs files and so on is ok, but I personally think it can start to lead to too many module declarations scattered throughout the codebase and it can become difficult to follow.

In my own code I prefer to declare the entire module structure in main.rs (or lib.rs for library crates), which avoids the need for any mod.rs files while offering all of the same functionality.

For our above example, we can have our directory named cat and within it just the file kitten.rs (no mod.rs this time), and now our main.rs can look like this:

mod cat {
    mod kitten;
    pub use kitten::meow;
}

fn main() {
    cat::meow();
}

When I discovered this, that was the moment when I realised how powerful the Rust module system really is. This lets me describe my module structure all in one place, and then my whole crate can just refer to this structure to import things.

For example, let's add a new directory below cat called activities and containing a file called play.rs.
Here is play.rs:

pub struct CatToy {
    name: String
}

impl CatToy {
    pub fn new(name: &str) -> Self {
        Self { name: name.to_string() }
    }

    pub fn fetch(&self) {
        println!("Fetch the {}", self.name);
    }
}

We can describe this layout in main.rs like so:

mod cat {
    // We still have our imports from before.
    mod kitten;
    pub use kitten::meow;

    // And now we describe the additional structure.
    pub mod activities {
        pub mod play;
    }
}

fn main() {
    let ball = cat::activities::play::CatToy::new("ball");
    ball.fetch();
}

This is ok, but our import path is a bit long. We want to keep our files organised but we don't want the callers of these functions to have to worry about our directory structure. Or perhaps we want to modify our directory structure without modifying the import paths (so users of your crate don't need to update their code too).

So we can shorten the import path like this:

mod cat {
    mod kitten;
    pub use kitten::meow;

    mod activities {
        pub mod play;
    }

    pub use activities::play::*;
}

fn main() {
    let ball = cat::CatToy::new("ball");
    ball.fetch();
}

So you can see this greatly simplifies the use of the code for the caller.

One last thing (well, several)

If you're importing (with use) something into another module outside of main.rs, you would need to add crate:: to the start of the module path.

For example, if we wanted to use CatToy from another file:

use crate::cat::CatToy;

pub fn fetch_ball() {
    let ball = CatToy::new("ball");
    ball.fetch();
}

We didn't need to do that in the above examples because the main() function was already at the base level crate scope and all of the other modules were children of that scope.

Finally, if you just want to import and use a module somewhere without needing to re-export it to the parent module, you don't need pub use. Just use crate::module::path; will work fine. Only use pub when you need to make something visible to the parent scope.

Using Types To Prevent Unit Conversion Errors

Steve Pryde — Mon, 09 Mar 2020 03:10:27 +0000

What if we could use static types to prevent passing a value in Centimetres to a function that expected Inches? Or what if we could prevent passing a Person id to a function that expected a Book id?

In this article I will show you a technique I use all the time that can accomplish just that. I did not come up with this technique. I just find it incredibly useful.

I will give examples in Rust (and a few in TypeScript) but you can adapt this technique to other statically typed languages as well.

Wrapping Ids

Suppose you have a function like this:

fn give_book_to_person(person_id: i32, book_id: i32) {
    let person = get_person(person_id);
    person.books.push(book_id); 
}

What would happen if you accidentally called it with the ids in the wrong order? Like this:

give_book_to_person(book_id, person_id);

This could result in a runtime error (could not find book? could not find person?), or worse it could give the wrong book to the wrong person! We used static types, but both ids are the same type so the compiler won't complain. This kind of problem can be difficult to debug and cause headaches for you later.

So how do we avoid it?

We wrap the ids in their own types. This allows the type system to work for us, and can practically eliminate this class of bugs entirely.

Let's see how it works.

struct PersonId(i32);
struct BookId(i32);

fn give_book_to_person(person_id: PersonId, book_id: BookId) {
    let person = get_person(person_id);
    person.books.push(book_id);
}

Now if we call this with the ids the wrong way around, we will get a compile-time error. Congratulations, you have just saved yourself a headache and a lot of time!

But wait, how do we access the wrapped value?

In Rust, you can do it like this:

let id: i32 = person_id.0;

But it is much better to keep the internal value private and provide an accessor method instead.

struct PersonId(i32);

impl PersonId {
    pub fn id(&self) -> i32 {
        self.0
    }
}

fn main() {
    let person_id = PersonId(1);
    let id = person_id.id();
    assert_eq!(id, 1);
}

Another pattern is to use the dereference operator to access the internal value.

use std::ops::Deref;

struct PersonId(i32);

impl Deref for PersonId {
    type Target = i32;

    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

fn main() {
    let person_id = PersonId(1);
    let id: i32 = *person_id;
    assert_eq!(id, 1);
}

In other languages you might use an object and access the member variable by name. For example, in TypeScript you might do this:

class PersonId {
    constructor(private readonly id: number) {}
    get getId(): number { return this.id }
}

let personId = new PersonId(1);

// We can access the actual id like this:
let myId = personId.getId();

Typically the internal value should only be accessed when outputting the value, or when calling an external function from another library. By keeping all access to the internal value within the same file, we reduce the potential for mistakes.

Equality comparison

Sometimes you will want to compare an id against another to see if they match. Remember we want to limit any access to the internal value to as few places as possible. The best way to do this is to do the equality check via operator overloading, or via an instance method on the type itself.

In Rust we can simply derive the Eq (i.e. equality) trait.

#[derive(Eq, PartialEq)]
struct PersonId(i32);

fn main() {
    let person_a = PersonId(1);
    let person_b = PersonId(1);
    if person_a == person_b {
        println!("It's a match!");
    }
}

If your language supports operator overloading, you can use a similar approach. However, a language like TypeScript does not support operator overloading, so we would instead do something like this:

class PersonId {
    constructor(public readonly id: number) {}

    equals(other: PersonId): Boolean {
        return this.id === other.id;
    }
}

let personA = new PersonId(1);
let personB = new PersonId(1);
if (personA.equals(personB)) {
    console.log("It's a match!");
}

Feel free to add any other comparison methods you find useful. By only adding these methods to the class / object itself, we limit the places that access the internal value, and thus significantly reduce the possibility of making a mistake.

Everywhere else, we use the PersonId object, and the compiler will be able to check that we always supplied an instance of the correct type.

Unit Conversion

By now you should be able to see other uses for this technique. My other favourite is in unit conversion.

Suppose we want to convert between Centimetres and Inches. We might provide a function like this:

fn convert_inches_to_cm(inches: f32) -> f32 {
    inches * 2.54
}

fn get_rect(width: f32, height: f32) -> (f32, f32) {
    // How big is the rect going to be?

    let width_cm = convert_inches_to_cm(width);
    let height_cm = convert_inches_to_cm(height);
    (width_cm, height_cm)
}

Do you see the problem?

We (or someone else working on the same code) might call get_rect() with values that are already in cm, and get unexpected results.

How do we avoid this? We wrap each unit in its own type!

struct Centimetres(f32);

impl From<Inches> for Centimetres {
    fn from(value: Inches) -> Self {
        Centimetres(value.0 * 2.54)
    }
}

struct Inches(f32);

impl From<Centimetres> for Inches {
    fn from(value: Centimetres) -> Self {
        Inches(value.0 / 2.54)
    }
}

// Might as well make the Rect a type too.
struct RectInCM {
    width: Centimetres,
    height: Centimetres
}

fn get_rect_cm(width: Inches, height: Inches) -> RectInCM {
    RectInCM {
        width: width.into(),
        height: height.into(),
    }
}

Great! Now it is very unlikely you will get the units wrong again. More headaches and debugging time avoided!

But we can do even better. Rust supports generics, which means we can write the get_rect_cm() function in a way that supports either units.

fn get_rect_cm<T>(width: T, height: T) -> RectInCM 
where
    T: Into<Centimetres>
{
    RectInCM {
        width: width.into(),
        height: height.into(),
    }
}

Generics are beyond the scope of this article, but basically what this is saying is that the width and height parameters must both be the same type, and that type can be anything that implements the trait Into<Centimetres>.

Now you can call get_rect_cm() and pass in either Centimetres or Inches and it will always produce the correct RectInCM struct.

Pretty neat, hey?

I hope you find this technique useful.

In defence of (some) fragmentation

Steve Pryde — Sun, 08 Mar 2020 03:16:09 +0000

Any time a new project is announced, especially in a relatively young language such as Rust, there will be the inevitable responses regarding "fragmentation", "wasted effort", "duplication of work" and so on.

I believe these responses are usually born out of a genuine desire to build a better, more mature ecosystem, and the fear that competing projects will mean slower progress towards that goal. There may be cases where these fears are justified, but in general I disagree with them.

In this article I aim to show why I think these responses are usually counter-productive.

The benefits of pooling resources

Firstly, let's look at some of the positives that come from pooling resources.

Maturity

Anyone familiar with Open Source knows that lots of eyes on the code can lead to more bug reports, more contributions towards fixing those bugs, and thus a more stable and mature project.

There is also the ecosystem that can build up around a project in the form of plug-ins, supporting libraries, interoperability with other projects, and perhaps more importantly, better tutorials, books, and online courses. The guarantee that a project is going to be around for the long term should not be understated, and this allows developers and companies to invest resources in building products based on the stable and mature codebase. These are all very good things.

Productivity

When a project becomes stable and mature, it is likely that it will meet the needs of a greater percentage of downstream developers, companies, and so on. That means they can use the project to build products that benefit their customers, and they can spend their time and effort building their own products rather than debugging the upstream project or trying to figure out how it works. The increased educational resources mentioned above also lead to greater productivity across the board. Now companies can mention the project in their hiring criteria and they can likely find developers who already know how to work with the upstream software.

The cost of pooling resources

The above benefits are great, but there are also some downsides.

Are larger teams always more productive?

Let's assess the idea that if everyone worked on a single project, that project would move faster, and conversely if many people worked on different projects, progress would be slower.

Anyone who has managed a large team of people knows that progress in a software project does not scale linearly with the number of people contributing to it. There is real complexity that emerges once a team grows beyond certain milestones (the actual numbers are fuzzy). The more developers you have, the greater the potential there is for bugs, internal fragmentation (feature A does not interoperate well with feature B), and instability. Larger projects need more maintainers willing to oversee the project design and hold regular team meetings to determine the technical direction of the project.

I am not saying that large teams never work. Clearly they do. Projects such as the Linux kernel show that large software projects can work incredibly well. Open Source is built on the incredible ability of humans to collaborate and achieve amazing things. But this collaboration isn't easy, and it does come with several costs. More contributors will bring technical and practical challenges that must be overcome, and this will generally mean the project must move much slower in order to guarantee consistency and harmony between the contributions and the overall vision and direction of the project.

Do larger projects really move faster?

Once a project matures it will also see a growing user base, with many other projects that now depend on it. Stability and backwards-compatibility then become far more important goals than adding new experimental features. Every change requires more and more testing to ensure it does not break some other part of the software, and does not break any projects that rely on it.

The result of all of this is, contrary to the original assumption, projects with a larger codebase may actually move much slower. The slowdown comes from some combination of having a greater emphasis on API stability and backwards-compatibility, and also the technical and practical complexity that comes from managing a larger team of people.

In general (though there surely are exceptions), the larger the project, the less it is able to innovate rapidly. Backwards-compatibility and API stability are benefits that often come at the cost of reduced innovation.

An alternative view

If larger projects tend to move slower and innovate less, should we abandon all large projects in favour of smaller ones? Or should we cap the number of contributors?

No, of course not! As mentioned at the top of this article, there are clear benefits to having larger projects that are more robust, mature, and stable. This article is not suggesting we stop supporting these projects. We have to stop thinking in terms of "there can only be one". It's not either-or. We can do both.

Rather, what I am proposing is that we continue to support these larger projects where they provide benefits to us, but I think we should also allow space in the ecosystem for more upstarts and offshoots. Look at them as budding experiments that may or may not provide valuable benefits down the track.

Where does innovation come from?

Choose any large software project and take a look back in history to when it was just starting out. How did it become the project it is today? Did it do so by people only contributing their efforts onto other projects that existed before? Or was there instead some point where an individual or small team started a new project and tried to build something better. They may have seen a niche that needed to be filled, or a better way to do something. It is very often the entrepreneurial drive of such individuals that sees the creation of new paradigms, new frameworks, and new ways of making software.

Almost every piece of software we use today was once a competitor to some larger, more mature product. The new projects could innovate much faster because they did not have the baggage of legacy software, legacy features and a long history of dependencies and unfortunate design decisions. There will always be early-adopters willing to try a new piece of software and put up with the bugs and instability in exchange for some benefit not available elsewhere. As more people join the project, the trend towards maturity and away from innovation begins, but the original innovation remains. Innovation can still happen of course - but with every stable API release, the pace of innovation and experimentation must inevitably slow down.

Does fragmentation really slow the pace of software?

If we were building physical products, then it follows logically that if a person spends their time and energy on one product, that time and energy is not being spent on other products. However, software is not like that, especially when it comes to Open Source.

If I make a change to a piece of software, or implement a useful feature, the only thing preventing others from copying said feature would be Copyright/Trademark laws and the like. In the world of Open Source this restriction is often reduced or eliminated (depending on the licenses used). We can and should share code between projects, and benefit from each other's work, giving appropriate attribution generously as we go. Definitely ask for permission from the author if you are unsure. Software patents complicate this a lot but that is beyond the scope of this article.

If a feature is implemented in a competing project and becomes well-received by users, other projects should be encouraged to implement it also, either by re-using the code or the design or otherwise tweaking the overall idea to fit. In this way, fragmentation isn't zero-sum in terms of features and effort. Everyone benefits from the ideas of others, and we end up with the best of all possible features.

Fragmentation = Experimentation, and that is a Good Thing^TM

When we view competing projects as little experiments designed to explore some corner or niche in design space, the picture changes. No longer are we looking at "wasted effort", but rather we are looking at parallel development that might lead to exciting new innovations down the track. Some of those innovations may even be incorporated back into the other projects once they have proven their effectiveness and success.

True, most of these experiments will go nowhere and may fizzle out early, but we can learn from those outcomes too. It is this repeated process of experiment, implementation, evaluation, and reincorporation of what we learned, that drives software innovation right across the industry. It has always been thus, and we are all the beneficiaries of it.

The downsides of fragmentation

There are some notable downsides to fragmentation, however, and I would like to mention a few of them so that we can learn how best to avoid them.

Confusion for newcomers

Probably the main downside of fragmentation is the confusing landscape presented to newcomers. Anyone who has used Linux in the 90's or early 2000's knows of the confusing process of figuring out which distro they should try. Each distro had its own pros and cons, and each had its own band of proponents advocating one and disparaging others.

This is clearly counter-productive and can put new developers off entirely. The state of npm packages in Javascript is another example where it sometimes is not clear which of the 900 competing frameworks one should use. We use the number of weekly downloads as a proxy for determining which projects are reliable or stable. Over time some clear winners do emerge, such as React, Angular, and Vue, and their popularity will compound their success to the point where they become the mature projects we mentioned at the start of this article.

But how do we get there without the confusion?

Unfortunately I don't have any clear solutions for this, mostly because it's difficult to predict which project will become successful in order to know which to promote ahead of time. This kind of thing will usually naturally emerge over time.

My suggestion is for the core team of any programming language to at least provide a single page outlining the various options, and attempt to provide pros and cons for each (as unbiased as possible) and continually update the page to report the latest state of the ecosystem, including popularity within the community. The Rust community has such central pages, for example https://www.arewewebyet.org/ and these are extremely helpful. By reporting this feedback regularly in one central location, the stand-out projects will become known more quickly, and this will accelerate the community out of the confusing landscape and towards a more mature ecosystem containing a few large projects and many experiments from which they can draw inspiration.

Disagreements, insults, and factions

The tendency to view fragmentation as unwelcome competition or even as a threat can unfortunately bring out the worst human behaviours in all of us. We compete for "customers" for our projects as if our very livelihoods depended on it (perhaps in some cases it does).

But if we have learned anything from past experience, competition almost always leads to better outcomes for the consumer. Perhaps this sounds a little like free-market capitalism, and yes there are some parallels, though not intentionally. I think any system where individuals or groups compete in an evolutionary playing field will show similar traits. It is notable that the cases where such markets do not serve the consumer are precisely where there is a lack of competition, not too much of it. This would suggest we should look more favourably on those projects willing to experiment and try something different.

Too much reinventing the wheel

There probably is a point beyond which we should stop trying to reinvent the wheel or improve on past inventions, but the problem is that we don't know where on that scale we are. Perhaps if no new web framework delivers any benefits in 10 years that might suggest we should go back to incrementally improving existing ones, but I don't think we are there yet. The space of possible designs is potentially infinite, and we often cannot assess the effectiveness (or lack thereof) of a design until we try it out.

There is a practical aspect as well. When it comes to the modern world, we all need to earn money and get paid. In this market, there is a significant cost to working on failed experiments and an increased benefit to being more cautious and sticking to tried-and-tested designs. The space of innovation is still open, but the innovations often need to deliver benefit in the short term. The upshot of this is that we should probably not be too concerned with people reinventing the wheel. The projects that should succeed, probably will. The rest will cease due to lack of funding, or interest, or both.

So perhaps we should be hesitant to reinvent the wheel, but keep an open mind when it comes to reinventing the web framework, for example. We don't need to jump on board every new project, but we can wish their contributors all the best just the same.

Embrace fragmentation, but not too much

In general, fragmentation = experimentation. We should celebrate it, and encourage people to explore their ideas and see what they can build.

At the same time, we can (and should) still put our efforts and resources into building a stable ecosystem via the more popular community projects. This is not a zero-sum game. We can borrow ideas (and sometimes code if licenses permit) from other projects and all benefit as a result. Communities will form naturally around projects that deliver good results.

There is an optimal balance between the two which probably somewhat favours mature ecosystems and stable APIs while ensuring that experimentation can and does still freely occur.

At the end of the day, having a few mature projects that can incorporate new features that were designed and perfected via little experiments and other offshoot projects, is the best of both worlds. It's not fragmentation vs pooling resources. It's a big world full of amazing developers. We can support both approaches. And as consumers of the resulting software, we all benefit.

DEV Community: Steve Pryde

A better home for your automated test results

Automated Future

What it does today

Where it's going

Try it out

Create a desktop app in Rust using Tauri and Yew

Front-end

Adding Yew

Adding Tauri

Adding tauri commands

Accessing tauri commands from Rust in the front-end

Calling Tauri commands in Yew code

Wrapping up

Using Selenium with Rust

First, let's introduce Rust.

Required tools for selenium

Let's see the code

Aside: The ? symbol in Rust

Great! Now let's make it do something

A little "Rusty" feature

Why Learn Rust?

Bugs

The human factor

Enter Rust

Rust String vs &str

TL;DR

String

&str

It's about memory and ownership

Borrowing and references

The difference between String and &str is...

When to use either one

One last time...

Intro to Rust Modules

Why not just import by file path?

What about different files?

What about directories?

Declaring the module hierarchy in one place

One last thing (well, several)

Further reading

Using Types To Prevent Unit Conversion Errors

Wrapping Ids

Equality comparison

Unit Conversion

In defence of (some) fragmentation

The benefits of pooling resources

Maturity

Productivity

The cost of pooling resources

Are larger teams always more productive?

Do larger projects really move faster?

An alternative view

Where does innovation come from?

Does fragmentation really slow the pace of software?

Fragmentation = Experimentation, and that is a Good ThingTM

The downsides of fragmentation

Confusion for newcomers

Disagreements, insults, and factions

Too much reinventing the wheel

Embrace fragmentation, but not too much

Fragmentation = Experimentation, and that is a Good Thing^TM