DEV Community: johnnylarner

Thirty Days Of Agents: Motivation

johnnylarner — Mon, 23 Mar 2026 21:10:07 +0000

Thirty days of agents: motivation

Anyone who loves the beach must at some point in their life visit Rio de Janeiro. I have no doubt that Ipanema beach has something you want. Be it sun, surfable waves or refrigerated beverages. And let's not forget about the phat view. As the sun sets behind the Vidigal favela, you can take a moment to reflect on the day that has now passed. Your mind slips into a dream-like state and you feel grateful and warm until suddenly you are ripped out of lucidity by the fresh yet sterile glare of the beach floodlights.

This transition of light - from the sun's natural rays to artificial, mechanical light - turned out to be a fitting metaphor for the theme of this blog series. After all, we software engineers are witnessing in real time how artisinal hand-written code is being replaced by probabilistic streams of tokens.

When I found myself jogging under these bright lights in Rio a few weeks back I was listening to a podcast about Openclaw, the agentic development framework that has taken the tech world by storm. It was quite motivating to hear how the project's author Peter Steinberger had rediscovered his passion for coding whilst developing Openclaw as I too had been struggling to see the beauty in work I was doing. The last time I found myself in this situation was back in 2023 not long after the release of ChatGPT (wow - those years flew by!). Back then I was working as a data engineer in my first proper coding job since teaching myself to code during the covid pandemic. The antidote to my growing disaffection for my first language python was the discovery of my second language - rust. Having attended Pycon that year in Berlin I noticed that rust was becoming the language of choice for anyone who wanted to rewrite python libraries to make them faster and/or functionality better. One prime example was the evolution of the polars (rs for rust, you dummy) library which was starting to challenge the data wrangling hegemony that pandas had over python devs and data scientists. Back then it felt as if the future was rust and it was rewarding to find a way to ride that wave.

Agents and frameworks to manage them represent a fundamental rewrite of that future I had imagined. One the one hand, they represent a "democratization" (to borrow the vernacular of tech leaders in the 2010s) of access to software. Even bros in Berghain will tell you how they developed an app with Claude. For experienced developers the offer huge productivity gains and bootstrapping companies has never been easier. But importantly they also pose a risk to the industry of software at large:

Large tech companies will be able to maintain monopolies with fewer engineers
Engineers who aren't willing to embrace significant change will get left behind
Private equity firms with big bets on pre-agentic enterprise software fear loss of market share (white collar jobs replaced by agents)

And so in many ways it is a bittersweet moment for me, accepting that I must and therefore ought to want to change. I believe passion, interest and knowledge are not virtues born out of nothing. You have to work hard to cultivate them and that's why I want to share this journey with the wilder world. Our first stop is going to be getting Openclaw up and running. Watch this space.

Dummy Software Post 1

johnnylarner — Mon, 09 Mar 2026 19:35:36 +0000

This is a dummy software post for manual UI testing.

Dummy Software Post 3

johnnylarner — Mon, 09 Mar 2026 19:35:35 +0000

This is a third dummy software post for manual UI and publishing workflow testing.

Dummy CI Publish Test

johnnylarner — Mon, 09 Mar 2026 19:35:06 +0000

This is a dummy article used to validate the publishing pipeline.

If this post appears on both GitHub Pages and dev.to after merge, the workflow is working.

Dummy CI Publish Test 2

johnnylarner — Mon, 09 Mar 2026 19:35:05 +0000

This is a second dummy article for validating publishing automation.

How much do you nitpick other people's code?

johnnylarner — Thu, 28 Sep 2023 19:24:16 +0000

I recently watched a video posted by Prime Reacts based on this blog by Dan Lew on why nitpicking colleagues' code is bad. For those not familiar with the term, Dan Lew defines it as flagging small, insignificant problems in a code review that are not wrong but suboptimal. He also argues that fixing the "problems" will make you, the grouchy reviewer, feel smug but won't benefit your code base or your team's opinion of you.

Watching this video made me feel a petty criminal listening to a convict showing genuine remorse for a crime that I too had committed. Fortunately for me, I haven't been coding long enough to piss off hundreds of colleagues yet. But to those of you whose code I nitpicked - I apologise.

Upon reflection I realised that my previous career paths before software engineering had baked into me a culture of nitpicking. First as an editor, my task was to locate and remove language inconsistencies as well as create a coherent tone across an article. Then, as a compliance analyst I was tasked with nitpicking colleagues' work for typos, document dates and other trivialities that make you question the point of life.

But from here on out I'm a changed man. For the reasons clearly laid out in Dan's blog, I am turning a new leaf and accepting that code perfection does not make code better!

Now it's your turn

The reason you probably opened this blog was because you sought an answer to the question in the title. I've collected 4 code snippets and for each of them provided one or multiple issues. For relative consistency, there are no possible other issues (though please leave a comment if you want). Select things you would seriously considering leaving as feedback in a code review. At the end the blog I'll leave my two scores: my old self versus my new self. See how you fare.

Quiz

Question 1

def add_features(trip_df: DataFrame, zone_df: DataFrame) -> DataFrame:
    """Returns a pandas DataFrame containing
    all the features derived from NYC
    Taxi dataset to answer all business questions.
    """
    # Rename columns
    trip_df = rename_columns_as_lowercase(trip_df)
    zone_df = rename_columns_as_lowercase(zone_df)
    trip_df = update_payment_type_as_string_values(trip_df)

    # Add features
    trip_df = add_borough_and_zone(trip_df, zone_df, "pulocationid")
    trip_df = add_borough_and_zone(trip_df, zone_df, "dolocationid")
    return trip_df

Possible issues

Duplicated function calls can be refactored into loops
Comments are redundant or misleading
Function name is not descriptive

Question 2

def build_payload(
    train_type: str, 
    train_age: int, 
    train_length: float, 
    train_is_diesel: bool
) -> dict[str, str]:

    input_parameters = [
        train_type,
        train_age,
        train_length,
        train_is_diesel,
    ]

    payload_names = [
        "train_type",
        "train_age",
        "train_length",
        "train_is_diesel",
    ]
    payload = {}

    # Set payload defaults
    current_time = datetime.datetime.now()
    source = "pipeline"

    for i, p_name in enumerate(payload_names):
        if p_name == "train_is_diesel":
            if train_is_diesel:
                payload["engine_type"] = "diesel"

        else:
            payload[p_name] = input_parameters[i]

    return payload

Possible issues

Declare default values as global values for transparency
Zipping is a more intuitive way to iterate over the lists, so replace enumerate with zip
Refactoring the boolean function argument would make the payload mapping static
Current time makes the function indeterministic, it should be parameterised

Question 3

def write_df(df: DataFrame) -> None:
    # Select columns
    small_df = df.select("name", "age", "dob", "gender")

    # Cache df
    df.cache()

    df.write.csv("full_data.csv", df)
    small_df.write.csv("subset_data.csv", df)

Possible issues

This function should do something generic but lack of parameterisation makes it not possible to reuse
Small df isn't an expressive name
Comments don't give us useful information

Question 4

class DataRowExtractor:
    def __init__(self):
        pass

    def extract_rows(self, data) -> dict[str, Any]:
        extracted_rows = []
        for i, row in data.iterrows()
            extracted_data.append(row)
        return extracted_rows

def main():
    csv_path = os.path.join("/working_dir", "data", "table_data.csv")
    write_path = os.path.join("/working_dir", "data", "row_data.json")

    data = pd.read_csv(csv_path)
    data_extractor = DataRowExtractor()

    row_data = data_extractor.extract_rows(data)
    with open(write_path) as f:
        json_data = f"{row_data}"
        f.write(row_data)

Possible issues

OS Path module is verbose and should be replaced with the new pathlib Path API
The RowExtractor doesn't add any real abstraction and overcomplicates the code
Pandas has a built in method to create records from a dataframe
Data is not a descriptive name

Have I really changed?

As promised, I'd give you my scores:

question	old_me	new_me
1	1	1
2	3	1
3	2	1
4	4	2
Total	10	5

Conclusion

Now obviously I rigged the results to make my improvement look and feel real. Jokes aside, time (and colleagues) will tell whether I really improve at not nitpicking!

One instructive aspect I tried to get across in the examples is that - perhaps in a production environment - the bugs in question 2 and 4 (good work if you found them) may have gone unseen by a reviewer focussed on scrutinising irrelevant banalities.

This is particularly relevant for question two: if we focus on reviewing the design of the function as a whole, we see that by replacing a boolean argument with a string, we can simply return a dictionary or maybe even just drop the function all together. Simple design is less bug prone, and this case we'd avoid the bug in our nested if statement.

Now we can see concretely why nitpicking is dangerous. But what if (like me) - despite having only picked out critical issues in a code review - you still found a bunch of nits want to pick ( I'm thinking here about code simplification through better library knowledge i.e. with extract_rows) ? I spoke to a family friend with 25> years experience coding about this and he suggested offering your nits in a non-committal to your colleague (I found nits, if you want to discuss them, we can). This makes the feedback loop something your colleague controls and could turn nits into valuable insights.

30 Days of Rust - Day 30

johnnylarner — Wed, 05 Jul 2023 06:00:00 +0000

This is it, the final blog. It's long and arduous, so take some time to read it. Don't worry, I'll still be posting blogs, but probably not on the features of the Rust language - I think we all need a break from that for now.

Yesterday's questions answered

No questions to answer

Today's open questions

*No open questions

Associated types vs generics

We've already looked at how generics are a way to improve reusability of functions or traits in Rust. Associated types offer a different approach to reusability. There may be instances where a certain trait need only be implemented once for a given type. This is when an associate type definition on a trait may be more useful. Consider this example from the Rust book:

use std::ops::Add;

#[derive(Debug, Copy, Clone, PartialEq)]
struct Point {
    x: i32,
    y: i32,
}

impl Add for Point {
    type Output = Point;

    fn add(self, other: Point) -> Point {
        Point {
            x: self.x + other.x,
            y: self.y + other.y,
        }
    }
}

fn main() {
    assert_eq!(
        Point { x: 1, y: 0 } + Point { x: 2, y: 3 },
        Point { x: 3, y: 3 }
    );
}

Note how in the implementation of Add we declare the type Output. What's we're actually doing is defining the associate type of the Add trait. Without this, our code won't compile. The associate type serves as a contract in this function. As per the trait definition, our associated type serves as the return type of our add function. Here's the trait declaration:

trait Add<Rhs=Self> {
    type Output;

    fn add(self, rhs: Rhs) -> Self::Output;
}

What's interesting about the Add trait is that it also has default associate type. This is represented by the Rhs alias input argument.

Default types in this context provide flexibility to library authors: often most users of a library will use a trait in one way. This typical usage pattern can be expressed through the default type of a trait. By making this a parameter, however, you can give power users the chance to use the library in a way you may not have foreseen.

Fully qualified syntax for duplicated method and associated functions

Rust, like Python, provides no limitations on defining multiple methods of the same name on an object. That's probably because no programming language intended on its users to do that (except when overloading methods, of course).

In Rust, however, there is actually a relatively common situation where a struct may have multiple methods or associated functions of the same name. When implementing a trait from an external library, we may be forced to use a method name that we have already defined.

Syntax for method calls

Remember back to when we talked about multithreading, we briefly talked about the Send (and Sync) trait. Imagine you want code that implements not only Send from the standard library, but also from a third party library that instantiates threads in a different way. Suddenly you have two versions of the send method.

When calling these methods, we have to use fully qualified syntax. This consists of the trait and the method name:

let thread_struct = MyCustomThread::new();

StdLibrarySendTrait::send(&thread_struct);
ExternalLibrarySendTrait::send(&thread_struct);

Syntax for associated functions

As a struct method contains a reference to the struct as its first argument, when we call the method using fully qualified syntax, Rust knows which implementation block is being called as it knows the type of self:


impl ExternalLibrarySendTrait for MyCustomThread {
    fn send(&self) {
        // Do something
    }
}

impl ExternalLibrarySendTrait for SomeOtherType {
    fn send(&self) {
        // Do something
    }
}

ExternalLibrarySendTrait::send(&thread_struct);

This is different for associate functions which don't take self:

trait Animal {
    fn baby_name() -> String;
}

struct Dog;
struct Cat;

impl Dog {
    fn baby_name() -> String {
        String::from("Spot")
    }
}

impl Animal for Dog {
    fn baby_name() -> String {
        String::from("puppy")
    }
}

impl Animal for Cat {
    fn baby_name() -> String {
        String::from("kitten")
    }
}


Dog::baby_name() // No problems
Animal::baby_name() // Cat or Dog?

You can see here how the fully qualified syntax for methods isn't enough here (also, the code won't compile). That's actually because we didn't use fully fully qualified syntax. Before we could skip the type aliasing because it's implicit in the &self argument. Here is how we use fully qualified syntax which will get the above code to compile:

<Dog as Animal>::baby_name() // A puppy
<Cat as Animal>::baby_name() // A kitten

We could use this syntax for methods, too, but it's overly verbose.

Supertraits: declaring trait dependencies

When defining a trait we can declare an optional supertrait which must be implemented if that trait is to work as expected. Using a struct that does not implement the supertrait will cause you code to not compile:

trait PrettyPrint: fmt::Display {
    // Do something
}

This code declares a trait that can only be implemented on structs that implement Display.

Supertraits can help reduce code duplication at the cost of increasing code coupling and complexity.

Type wizardry

Until now we've barely touched on the type keyword. This keyword is used to declare type aliases:

type Kilometer = i32;

What's important to understand is that Kilometer is still an i32:

let a: Kilometer = 1;
let b = 1;

asserteq!(a + b, 2);

Often we see the newtype pattern in Rust. This is popular because of Rust's trait implementation rule:

A trait can only be implemented for a type if the trait or the type is local to your code.

We can implement the above code using the new type pattern:

struct Kilometer(i32);

let a = Kilometer(1);
let b = 1;

asserteq!(a + b, 2);

This code won't compile, because the Kilometer type doesn't implement the Add trait. i32 does implement the trait, which is is proof that the type alias really is an alias around i32.

When to alias and newtype?

Aliasing makes sense when:

You have a long type (nested Result or Options, for example) that can be shortened
A composite type where one part is always the same but the other should be treated as a generic
Giving a meaningful name to an existing type (like i32) can make your code easier to read.

Newtyping makes sense when:

You need to implement external traits
You want to use a public API to control access to internal code

The type you never knew about

Functions that return ! return the so-called never type. The never type is a Rust alias for the empty type. The point of a never type is that it expresses a function that never returns.

Functions that do this yield a result that can be coerced into any other type. This is useful when using match statements, as these must always return a single type. Keywords and macros such as continue, panic! and loop return the never type.

Sized types

A short note on sized types. Generic functions by default can only take sized types. A type is sized when its size is known at compile time. Sized types by default implement the Sized trait.

If you want to use a dynamically sized type with generics you need to use the ?Trait notation:

fn generic<T: ?Sized>(t: &T) {}

Note that we also expect to receive a reference to the dynamically sized type as these are also stored on the heap behind a pointer.

Passing and return functions

Functions can easily passed to functions using the fn type. This is different from the trait restraints used to indicate that a closure should be passed as an argument. As something of type fn implements all three Fn traits, you can always pass functions where closures are accepted.

If you want to return a closure from a function, you have to wrap it in a Box as a closure's size is determined at runtime.

Last but not least: Macros

The most advanced Rust feature is the macro. These are functions that use Rust code to generate more Rust code. This pattern is generally known as metaprogramming.

Rust has different kinds of macros. This have different implementation detail, and each type is suited to a different use case.

`macro_rules!` macros

The most common type of macro in Rust is the declarative macro. This is created by using the macro_rules! macro. In a declarative macro, you can match Rust source code to patterns and generate different kind of code based on which kind of pattern is matched. This makes it possible for these kind of macros to take an undefined number of arguments.

Procedural macros

The remaining three types of macros are procedural. These macros must take Rust source code as an input and return it.

The first of these is the derive macro. To create a derive macro, you need to derive from the proc_macro_derive macro of the standard library. Derive macros only work
on structs and enums.

Derive macros have a number of restrictions around packaging. When developing such macros, developers are encouraged to include them as a dependency of the main library being developed. Then the macro can simply be reexported as part of the main library for users to import.

Attribute macros are similar to derive macros, only that they are more flexible as they can be used for functions and other data types as well. They derive from the proc_macro_attribute macro.

Finally, you have function-like macros. Though procedural in nature, these macros can be used inline like a macro_rules! macro and can take an unspecified number of arguments. They derive from proc_macro.

30 Days of Rust - Day 29

johnnylarner — Tue, 04 Jul 2023 18:47:02 +0000

Good evening folks,
We're nearly there. Two posts left. I've got a final kick of energy to push through. Today we'll be zooming over some of Rust's object oriented, pattern matching and unsafe features.

Yesterday's questions answered

No questions to answer

Today's open questions

No open questions

Is Rust object-oriented?

There are many ways to define whether a programming language is object-oriented or not. When it comes to Rust, a useful way to think about this is to ask yourself: what are OO languages designed to achieve and how do they do this?

Encapsulating logic

Objects allow you to separate and encapsulate different parts of your code. Rust's features definitely support encapsulation. Structs and impl blocks allow programmers to write distinct code that is separated from other bits of code ✅

Objects as data storage

Often object-oriented languages express human concepts through grouping data in objects. In Rust enums and structs are also a great way to group conceptually similar data in one place. ✅

Reusing and types through inheritance

The most obvious missing aspect so far for new Rust developers is the absence of inheritance. In many programming languages, objects expressed as subclasses are able to inherit from parent classes. Inheritance can:

Reduce code duplication by defining common code at the parent level
Increase the breadth of composability as subclasses share their parent's type.

We know there are no classes in Rust, and that structs cannot inherit from each other. But this doesn't mean Rust doesn't have mechanisms to tackle the issues noted above.

trait objects provide a way of declaring default behaviour. If a trait implements a method and a struct implements that trait , the struct will take on the default implementation.

What's more traits also grant programmers access to typed interfaces across different structs. If a function or struct field has a trait object as its type definition, it will accept any struct that implements that trait. trait objects incur a small runtime penalty. As we don't know at compile time what all the types could be (imagine you're writing a library that would allow users to make their own components), trait objects are stored on the heap. This means we need to access them via a pointer at runtime to find the method we want to call. This is known as dynamic dispatch.

Match, match, match

When learning Rust, match statements are one of the first novel features you come across. What's particularly cool about match is that the compiler will often force you to exhaustively match all possible outcomes. This catches bugs before they even have a chance to compile.

match arms require so-called refutable patterns, patterns that can match or not match. Irrefutable patterns are most commonly seen in variable assignment:

let x = 5;

Nested matches

If you're working with nested enums or structs, you can use match patterns beyond the root level of the data structure:

enum MarriageStatus {
    Single,
    MarriedTo(String),
    Divorced,
}

struct TaxProfile {
    name: String,
    age: i32,
    marriage_status: MarriageStatus,
}

fn main() {
    let profile = TaxProfile {
    name: String::from("Max Mustermann"),
    age: 30,
    marriage_status: MarriageStatus::MarriedTo(String::from("Jane Doe")),
    };

match profile {
    TaxProfile {
        marriage_status: MarriageStatus::MarriedTo(spouse),
        ..
    } => {
        println!("Congrats, you and {spouse} get a tax break.");
    }
    _ => println!("Only married people are worthy of tax breaks!"),
    }
}

Here we want to print a different message based on whether a given tax profile gets a tax break or not. This code demonstrates several more advanced features of the match system:

struct matching requires you to temporarily instantiate a struct and compare the relevant fields
Irrelevant fields can be ignored using the range operator ... This is useful when we have more than one other field we want to ignore.
We can also match based on the inner field's value. This can also be used in the result of the match arm.

Match guards and bindings

In the above example the first arm of the match statement matched to an enum. If we wanted to only match if your spouse has a specific name, we'd also have to introduce a match guard:

match profile {
    TaxProfile {
        marriage_status: MarriageStatus::MarriedTo(spouse),
        ..
    } if spouse == "Angela Merkel" => {
        println!("Congrats, you and Angie get a tax break.");
    }
    _ => println!("Only married people are worthy of tax breaks!"),
    }
}

In this case, only the spouse of Angela Merkel would get a tax break. What if we wanted to give tax breaks to old people too? We could specify a range for the match arm and use a binding to extract the result. We can print this to the screen to give the user a personalised response:

match profile {
    TaxProfile {
        marriage_status: MarriageStatus::MarriedTo(spouse),
        ..
    } if spouse == "Angela Merkel" => {
        println!("Congrats, you and Angie get a tax break.");
    }
    TaxProfile {
        age: boomer_age @ 60..=79,
        ..
    } => println!("Even at age {boomer_age} you matter to society, have a tax break."),
    _ => println!("Sorry, no tax break for you."),
}

What is unsafe code?

Until now in these blog posts we've only talked about safe Rust code. Safe Rust code is effectively any code that meets the Rust compiler's ownership, borrowing and type rules. We call the code safe because we know that this code will not suffer from invalid pointers, null values, unintended side effects or security issues posed by incorrect pointer addresses.

Unsafe code is the inverse: we no longer get any guarantees from Rust that our program upon submission to the compiler won't have any of these issues at runtime. The trade off is that we no longer have to adhere to all of the compiler's rule. One example might be having more than one mutable reference in scope at any given time:

fn split_as_mut(values: &mut [i32], mid: usize) -> (&mut [i32], &mut [i32]) {
    let len = values.len();
    let ptr = values.as_mut_ptr();

    assert!(mid <= len);

    unsafe {
        (
            slice::from_raw_parts_mut(ptr, mid),
            slice::from_raw_parts_mut(ptr.add(mid), len - mid),
        )
    }
}

You can see we use the unsafe keyword to declare a block in which we can execute unsafe code. What makes this code unsafe is calling the associated function from_raw_parts_mut as this function is declared as unsafe. To make a function unsafe you can prepend the function declaration with the unsafe keyword.

Beyond the function being unsafe, we can tell from our function's return type that both calls from_raw_parts_mut a mutable reference for our array into scope. This is not permitted by the Rust compiler in a safe context. But was the code is unsafe, we don't have any issues when submitting our code.

The split_as_mut function is considered as an acceptable example of using unsafe Rust as:

The unsafe part of the code is isolated and wrapped in a safe API. This makes calling the function safe regardless of where you call it.
We manually assert that the pointers we create will point to a valid location in memory for the data structure we're trying to access
Both pointers access unique subsets of the array.

Multilingual Rust

We can also declare Rust APIs for other language using the extern keyword. Private extern functions call functions from other languages, while public functions can be called in other languages:

extern "C" {
    fn abs(input: i32) -> i32;
}

#[no_mangle]
pub extern "C" fn call_from_c() {
    println!("Just called a Rust function from C!");
}

30 Days of Rust - Day 28

johnnylarner — Sat, 24 Jun 2023 06:32:50 +0000

Hi folks, I've been really struggling to keep up the blog recently. This was in part to a change of scenery in Berlin. But als I've learned that writing continuously on the same subject for so long doesn't get kind of boring. That said, I'm gonna push through today and summarize the key points about threading in Rust.

Yesterday's questions answered

No questions to answer

Today's open questions

No open questions

Threading 101

A multi-threaded program can run code in a concurrent or parallelised way. This can increase the performance of your program at the cost of increasing its complexity. In older programming languages, threading also made debugging code more difficult: threading environments are hard to reproduce (as thread creation is usually managed by the OS) and understanding what a specific thread is doing can be impossible to tell without an advanced debugger.

Rust aims to solve some of the complexities via compiler rules that will make thread unsafe code fail at the compile step. This is achieved through the use of ownership guards and atomic reference counters which we'll cover later.

The API

We can create a thread really easily in Rust:


use std::thread;
use std::time::Duration;

fn main() { 
    thread::spawn(|| {
        for i in 1..10 { 
            println!("hi number {} from the spawned thread!", i); 
            thread::sleep(Duration::from_millis(1)); 
        } 
    });
}

By default the main thread will not wait for subthreads to exit before continuing. This is why we need to use a join handle to force the main thread to wait. As this returns a Result, we also unwrap it. Note that where you do this in your code will impact the runtime behaviour:

use std::thread;
use std::time::Duration;

fn main() { 
    let handle = thread::spawn(|| {
        for i in 1..10 { 
            println!("hi number {} from the spawned thread!", i); 
            thread::sleep(Duration::from_millis(1)); 
        } 
    });
    handle.join().unwrap();
}

What do threads own?

In the above example, we can see that threads take a closure as their first argument. If we think back, we'll remember that closures are special in Rust: they capture variables declared in their scope. Capturing actually means borrowing in ownership terms:

use std::thread;
use std::time::Duration;

fn main() {
    let my_vec = vec![1, 2, 3];
    let handle = thread::spawn(|| {
        for i in 1..10 { 
            println!("my vec: {:?}", my_vec); 
        } 
    });
    handle.join().unwrap();
}

The compiler will reject this code as we don't know how long a thread's lifetime will be. This means that a thread could outlive the duration of my_vec, leading to an invalid reference being used in a thread.

We can move the vector into the thread. However, we won't be able to reference the value anywhere after our instantiation of the thread, as it will no longer be in scope.

Chatty threads

Rust implements the concept of channels as a way to pass messages between threads. This is considered more robust that manipulating shared state. A channel consists of at least one producer and one receiver:

let (tx, rx) = mpsc::channel();

When sending messages between threads it's important to remember two things:

Sending stuff also moves ownership of the data to the receiving thread
You can force a thread to wait for a message with the recv. You can use try_recv to check if a message is available or not.

The first point is a great source of robustness for your Rust programs. Channels are a really safe way to pass around values as you know the compiler won't let you accidentally borrow data.

Sharing state

So far our examples have only covered situations where we create one thread. If we want to create multiple threads, we also need to use more advanced tools to manage ownership of shared resources among them. To be able to move ownership of data into multiple threads, we need to use combination of the Arc and Mutex type.

Mutually exclusive

The Mutex type is a was to guarantee singular access to a resource via a lock. When you aren't working with threads, Mutex can be used to access a value that it wraps in a different scope where it isn't owned and manipulate that value.

If we use a Mutex with threading, things get a bit more complicated. As we have to move values into a thread, the Mutex would end up having multiple owners. In a previous blog we looked at the Rc (reference counter) type. This isn't thread safe. Rc are cheaper than the thread safe Arc (atomic reference counter) type which will use here.

By wrapping our Mutex in an Arc, we instruct Rust to track all the owners of the Mutex. We can call the Arc's clone method with a reference of our Mutex to create multiple references to it. Deref coercion then helps us write clean code to dereference the reference and manipulate the underlying value.

Send and Sync

If you want data to be transferrable between threads, they have to implement the Send and Sync traits. These are the overheads I mentioned before. Implementing your own types with Send and Sync results in unsafe Rust. This is a topic I'll touch on a later blog.

30 Days of Rust - Day 27

johnnylarner — Tue, 13 Jun 2023 18:48:53 +0000

Good afternoon everyone, I'm entering the final blog sprint. I'm finishing my 30 days of learning this week. We'll be getting into some more advanced concepts over the next few days. The final sprint starts with a concept not totally unique to Rust: smart pointers.

Yesterday's questions answered

No questions to answer

Today's open questions

No open questions

What is the point (of reading my blog?)

We already know that pointers provide an efficient way of tracking large data structures during a program's runtime. Rather than copying the data each time we do some operation on it, we can simply pass around a pointer that points to the memory location of the result. References are the most common types of pointer and are represented by the & symbol.

Smart pointers in Rust are more powerful than a mere reference. Smart pointers can:

Help us efficiently allocate and track data on the heap
Count the numbers of references so that we can have multiple ownership (i.e. multithreaded code)
Enable us to run borrow rules at runtime instead of compile time

Put your data in a Box<T>

Most programs interact with data of an unknown quantity. The Box API allows us to allocate that data directly to program's heap while maintaining a pointer on the stack. There are certain types of data structures where this is required in Rust.

Any type that is of a recursive nature needs to be in a box. This includes binary trees and linked lists. The reason for this is that a type which declares one of its fields as being of its own type will set off a recursive chain. Consider a binary tree:

We could set a type for this structure if we knew that the root node would always have 2 children, and each of its children would have a left leaf node and no right leaf:

struct RootNode {
    left: ChildNode,
    right: ChildNode,
}

struct ChildNode {
    left: LeafNode,
}

struct LeafNode {}

But as soon as that structure changes, its type would also have to change. When working with recursive structures, we know that the size of each instance is very, very likely to vary (otherwise we'd be using some other data structure). Box is a way to tell this to the Rust compiler:

struct Node {
    left: Option<Box<Node>>,
    right: Option<Box<Node>>,
}

Here we are saying to the compiler: we will have a node which may have another node which may have another node and so on to infinity 🚀

Getting inside your Box

As a Box is a reference to data, you need to dereference a box if you want to access your data:

fn main() { 
    let x = 5; 
    let y = Box::new(x);
    assert_eq!(5, x); 
    assert_eq!(5, *y); }

Under the hood, the deference operator is calling deref method of the Deref trait of the Box object and then dereferencing the result. The verbosity of dereferencing is hidden from us as programmers.

Rust is also able to coercively dereference smart pointers. This means you can get references to the underlying value simply by calling the & operator. When you call this on a smart pointer, it will be implicitly dereferenced to yield the underlying value. As this behaviour is defined before compilation, Rust can determine the value of final result, which would be the last value to not implement the Deref trait.

Cleaning up your Box

Smart pointers also implement the Drop trait. This tells Rust what to do with a given resource when it goes out of scope. For Boxes, this will result in the pointer and the data on the heap being deallocated.

As with the Deref trait, Rust is actually calling the drop method behind the scenes. The Rust compiler won't let you call this method in your code. This could lead to two attempts being made to clear memory, this is known as a double free error. Instead you can use: std::mem::drop.

Sharing is caring

Above we talked about a case where want to express to the compiler that we don't know how many instances type may be nested in a structure: we didn't know the number of instances.

Another case for smart pointers is when we don't know how many owners a piece of data will have at compile time. For example, if we're trying to express a graph structure where each node is a person that has edges which represent connections to that node. We want obviously want to hold on to that node until we know each edge has been removed. If a node represents an expensive EC2 instance and an edge an active inbound connection, we only want autoshutdown to kick in when no more connections are active. Enter Rc<T>...

The reference counter smart pointer (Rc<T>) is a way of expressing this situation in Rust. Data wrapped in a Rc will only ever go out of scope once all reference owners themselves go out of scope.

Who cares what the compiler thinks!

Throughout this series, we've seen how the Rust compiler pushes us towards writing safer programs. One example of this is the borrowing rules:

There can be multiple immutable references in scope at one time if there are no mutable references in scope

There can be only one mutable reference in scope at any time

But sometimes we may want to break these rules. This is what the RefCell smart pointer does. Well strictly speaking, it doesn't break the rules, it defers validating the rules until runtime (instead of compile time). RefCell wraps data that might otherwise be immutable such that you can then mutate it at runtime if desired. A RefCell has the borrow and borrow_mut methods that return references to the data it is wrapping.

You may rightly ask: why not just use a mutable reference if you want to mutate data. Most of the time you probably should. But there are cases, particularly when mocking stuff for unit tests, where you may want to use mutability to validate the behaviour of your library. The Rust Book explains the case really well and I'll defer to that as this blog post is already getting pretty long...

Separating the weak from the strong

Some data structures may want to contain references to each other. For example our binary tree we look at earlier. We may want a root node to have access to all its children. Each parent and grandparent node should also access their children. Thus each node except the root will have an unknown number of owners at compile time. This means each node should be wrapped in a reference counter type.

But that means we aren't able to modify the hierarchy of our tree as reference counters return immutable references. We can wrap the reference counter in a RefCell if we want to adjust the hierarchy at run time:

use std::cell::RefCell; 
use std::rc::Rc; 

struct Node { 
    value: i32, 
    children: RefCell<Vec<Rc<Node>>>, 
}

A child node may also want to have a reference to its parent. And as parents can have multiple children, we'd want an reference counter wrapper type:

struct Node { 
    value: i32, 
    parent: Rc<Node>,
    children: RefCell<Vec<Rc<Node>>>, 
}

But now we have an issue: the child and parent now have circular references to each other. Why is that an issue? Reference counter types are only dropped when they have 0 strong references. If a Rc<Node> has 2 children with this setup, it will have 2 references. If those children are dropped or go out of scope, the Rc<Node> will have 0 references, meaning it will be dropped. This is probably not what we want to happen.

The Weak smart pointer is designed to solve this: when you want to have multiple references that should not influence the ownership of a given value:

struct Node { 
    value: i32, 
    parent: Weak<Node>,
    children: RefCell<Vec<Rc<Node>>>, 
}

Now when a node's children are dropped, the compiler will not forcibly drop the parent node 🙌

30 Days of Rust - Day 26

johnnylarner — Sun, 11 Jun 2023 20:03:03 +0000

Good evening everyone, as promised here is my third and final blog this week. I had a lovely trip over the weekend to Brandenburg on my bicycle. The pain in my legs was worth it: we took glorious forest routes and camped at a lovely lake. Back to reality today though with a deeper look into packaging in Rust with cargo.

Yesterday's questions answered

No questions to answer

Today's open questions

No open questions

We <3 cargo

The keen reader may have noticed that we already looked at some features of cargo in Day 1's blog . It has also featured in days 5, 7, 10, 13, 14, 19 and 23. It's packed with functionality: the package manager helps with testing, project builds and publishing, documentation and many other things.

Powerful publishing with autodocs and markdown

We already know that cargo can generate the documentation for any rust project in a locally stored project. This is because HTML is generated from documentation comments when a project is built using cargo.

documentation comments begin with ///. This is different to the // of a standard comment. What makes these comments powerful is that they support markdown notation. This means headings and code snippets can be designated for a given function or module which is then rendered into pretty HTML. Where Rust goes one step further is with the documentation testing. Any code snippets you write in your documentation that use an assert macro are tested when you run cargo test.

Metamarkdown

There is a third type of comment in Rust: //!. This is known as a style of doc comment. There can be used to edit the display HTML outside of the explicit API documentation. For example, if you want to give your crate a meaningful introduction, you'd write //! followed by some text. This would be rendered at a different position to the usual API code which is nested in with the functions it describes.

User friendly project structure

Often when you write code, the technical implementation can vary from the conceptual structure of the thing your are implementing. This can make your package harder to use: your public APIs are at locations that don't match that position they occupy in the conceptual domain. Also, this can lead to long import statements if the public interface to your code is deeply nested in some structure.

To solve this problem, you can use public re-exports. This is where you use the use keyword followed by the name of the module you want to make public. This also impacts how developers see your code in the crates.io documentation.

Another feature of cargo are so-called workspaces. These offer a way to group multiple projects around a single set of dependencies. You can use the [workspace] section of your Cargo.toml file to group subdirectories as subprojects. Each will have its own Cargo.toml, but the lock file will be managed at the common parent level.

The aim here is to ensure that projects you create that are mutually dependent on each other always contain the same, non-breaking dependencies. Nonetheless you have to explicitly add subprojects to another subproject's Cargo.toml for it to be marked as a dependency.

Things to remember about your crates

There are a couple of rules you need to remember when publishing your projects to crates.io:

You can't delete old versions. The best you can do is cargo yank them, but this will just prevent any external project from using that specific version.
You must specify a licence or a licence file, you can have multiple licences per project.
Your projects name must be unique on crates.io.

30 Days of Rust - Day 25

johnnylarner — Wed, 07 Jun 2023 18:14:46 +0000

What is up world, I promised you I'd be back again this week. Today's blog flog will shed some light on Rust's self-proclaimed functional aspects. I'm not really dedicated enough blog time to this topic, but I'm quite time-pressed this evening 🫣

Yesterday's questions answered

No questions to answer

Today's open questions

No open questions

5 days left until closure

Anonymous functions in Rust are known as closures. This feature of the language is particularly powerful due to the following fact:

Closures capture variables in the scope at which they are declared

We know that variable scope is fundamental point of thought when designing Rust code. Closures are the mechanism to get around some of the restrictions this poses. One obvious example of where this makes sense is creating a thread. You can make a variable available in that thread's scope.

Fn Traits

Closures (as well as normal functions) can implement so-called Fn Traits:

FnOnce -> Called once, moves captured data out when returning
FnMut -> Called more than once, may mutate captured values but won't move them
Fn -> Called more than once, no mutations, no captured values

These traits allow the Rust compiler to spot issues early. For example, if multiple threads call a closure that mutates a resource that all threads need access too. Similarly, if you move an item in a closure while its original owner is still active in scope, you'd encounter some nasty bugs (if the compiler let you).

Iterators are sexy

Turning collections into iterators opens up a bunch of useful operation to use as a developer. The map, filter and sum methods provide a readable, high-level way of writing code. Consider these two code snippets:

pub fn search<'a>(query: &str, contents: &'a str) -> Vec<&'a str> {
    contents
        .lines()
        .filter(|line| line.contains(query))
        .collect()
}


pub fn search<'a>(query: &str, contents: &'a str) -> Vec<&'a str> {
    let mut results = Vec::new();

    for line in contents.lines() {
        if line.contains(query) {
            results.push(line)
        }
    } 

    results
}

Not only is the first example shorter, the code is more expressive of the procedural nature of the task. What's more, we don't have to worry about state management (of the vector) which could become challenging in a multi-threaded system.

DEV Community: johnnylarner

Thirty Days Of Agents: Motivation

Thirty days of agents: motivation

Dummy Software Post 1

Dummy Software Post 3

Dummy CI Publish Test

Dummy CI Publish Test 2

How much do you nitpick other people's code?

Now it's your turn

Quiz

Question 1

Possible issues

Question 2

Possible issues

Question 3

Possible issues

Question 4

Possible issues

Have I really changed?

Conclusion

30 Days of Rust - Day 30

Yesterday's questions answered

Today's open questions

Associated types vs generics

Fully qualified syntax for duplicated method and associated functions

Syntax for method calls

Syntax for associated functions

Supertraits: declaring trait dependencies

Type wizardry

When to alias and newtype?

The type you never knew about

Sized types

Passing and return functions

Last but not least: Macros

macro_rules! macros

Procedural macros

30 Days of Rust - Day 29

Yesterday's questions answered

Today's open questions

Is Rust object-oriented?

Encapsulating logic

Objects as data storage

Reusing and types through inheritance

Match, match, match

Nested matches

Match guards and bindings

What is unsafe code?

Multilingual Rust

30 Days of Rust - Day 28

Yesterday's questions answered

Today's open questions

Threading 101

The API

What do threads own?

Chatty threads

Sharing state

Mutually exclusive

Send and Sync

30 Days of Rust - Day 27

Yesterday's questions answered

Today's open questions

What is the point (of reading my blog?)

Put your data in a Box<T>

Getting inside your Box

Cleaning up your Box

Sharing is caring

Who cares what the compiler thinks!

Separating the weak from the strong

30 Days of Rust - Day 26

Yesterday's questions answered

Today's open questions

We <3 cargo

Powerful publishing with autodocs and markdown

Metamarkdown

User friendly project structure

Things to remember about your crates

30 Days of Rust - Day 25

Yesterday's questions answered

Today's open questions

5 days left until closure

`macro_rules!` macros