DEV Community: Mete Can Eris

50 Ways to Handle Your Iterator in Rust

Mete Can Eris — Wed, 02 Mar 2022 16:07:23 +0000

The answer is easy if you
Take it logically
I'd like to help you in your struggle
To be free
There must be fifty ways
To leave your lover

That was Paul Simon on lovers. This is me on Rust iterators (yeah, living the life). Maybe not 50, but there are a lot of ways to handle your iterator in Rust. And they are equally easy if you take them logically.

Iterator, IntoInterator, and FromIterator traits

There are three main traits that we should look into to grasp iterators in Rust.

The Iterator trait requires the implementation of a single method called next(), which, when called, returns an Option<Item>.

  pub trait Iterator {
      type Item;

      fn next(&mut self) -> Option<Self::Item>;
  }

Calling next() will return Some(Item) as long as there are elements, and once they’ve all been exhausted, it will return None to indicate that iteration is finished.

In the code below, there is an implementation of the Iterator trait for a certain Counter type (the example is from the Introduction to Programming Using Rust book).

  struct Counter {
      max: i32,
      // `count` tracks the state of this iterator
      count: i32,
  }

  impl Counter {
      fn new(max: i32) -> Counter {
          Counter { count: -1, max: max }
      }
  }

  impl Iterator for Counter {
      type Item = i32;

      fn next(&mut self) -> Option<Self::Item> {
          self.count += 1;
          if self.count < self.max {
              Some(self.count)
          } else {
              None
          }
      }
  }

  for i in Counter::new(10) {
      println!("{}", i);
  }

The IntoIterator trait requires the implementation of an into_iter() method. This method defines how any data type can be converted into an interator.

  pub trait IntoIterator {
      type Item;
      type IntoIter: Iterator;
      fn into_iter(self) -> Self::IntoIter;
  }

Finally, the FromIterator trait, obviously, takes an iterator and returns a collection.

  pub trait FromIterator<A> {
      fn from_iter<T>(iter: T) -> Self
      where
          T: IntoIterator<Item = A>;
  }

The synergy between these traits allows an important flexibility when getting a custom type to be iterable. A developer can, of course, implement the Iterator trait for the type and be done with it. But they can also cheat a bit, in a sense, by implementing the IntoIterator and FromIterator traits to produce an existing iterable type, such as a Vec and offload the iteration responsibility to this intermediate data type.

Let's look into how the above example for the Counter type could have been implemented this way by using Range from the standard library (again, the example is from the Introduction to Programming Using Rust book).

  struct Counter {
      max: i32,
      // No need to track the state, 
      // because this isn't an iterator.
  }

  impl Counter {
      fn new(max: i32) -> Counter {
          Counter { max: max }
      }
  }

  impl IntoIterator for Counter {
      type Item = i32;
      type IntoIter = std::ops::Range<Self::Item>;

      fn into_iter(self) -> Self::IntoIter {
          std::ops::Range{ start: 0, end: self.max }
      }
  }

  for i in Counter::new(10) {
      println!("{}", i);
  }

into_iter, iter and iter_mut methods

We have already seen that the IntoIterator trait has only one required method, into_iter() which converts the thing implementing IntoIterator into, well, an iterator.

Additionally, collection types generally offer methods that provide iterators over references, called iter() and iter_mut(). All these do is borrow either an immutable or mutable reference to the collection and turn it into an iterator.

In other words, if you're confused when to use which, remember that the king is into_iter(), and iter() and iter_mut() are just wrapper methods to borrow and into_iter in one go.

  // assuming lovers is Vec<Lover>

  // Iter<Vec<Lover>>
  // because: impl<'a, T> IntoIterator for Vec<T>
  lovers.into_iter()  

  // Iter<&Vec<Lover>>
  // because: impl<'a, T> IntoIterator for &'a Vec<T>
  lovers.iter()       

  // Iter<&mut Vec<Lover>>
  // because: impl<'a, T> IntoIterator for &'a mut Vec<T>
  lovers.iter_mut()

for loops

Rust's for loop is just syntactic sugar for iterators.

So this:

  // assuming lovers is Vec<Lover>
  for lover in lovers {
      println!("{}", lover);
  }

is actually this:

  // assuming lovers is Vec<Lover>
  {
      let struggle = match IntoIterator::into_iter(lovers) {
          mut iter => loop {
              let next;

              match iter.next() {
                  // that's what she said
                  Some(lover) => next = lover,

                  // said no one ever 
                  None => break,                 
              };

              let lover = next;

              let () = { println!("{}", lover); };
          },
      };

      struggle
  }

As we have seen, into_iter() takes self, using a for loop to iterate over a collection consumes that collection. Yet, it's possible to loop with for without consuming. Remember, for a Vec, the IntoIterator trait is implemented multiple times.

  impl<T> IntoIterator for Vec<T>
  impl<'a, T> IntoIterator for &'a Vec<T>
  impl<'a, T> IntoIterator for &'a mut Vec<T>

Basically, if we loop over a vector directly, we'll consume it and get owned vector items within the loop. But if we loop over an immutable or mutable (& and &mut) reference to a vector, we'll get an immutable or mutable reference to vector items within the loop.

  // assuming lovers is Vec<Lover>


  // consuming lovers, lover is an owned Lover
  // ^_^ (see what I did there)
  for lover in lovers {}          

  // lover is &Lover
  // (because you can't own a lover)
  for lover in &lovers {}         

  // lover is &mut Lover
  // (but people don't change)
  for lover in &mut lovers {}

Lazy Iterators

As a final note, in Rust, iterators are lazy, meaning they have no effect until you call methods that consume the iterator.

Methods for consuming an iterator may be defined on the Iterator trait and are called consuming adapters because they fulfill whatever their purpose is by consuming the iterator via calling next() on it. Methods such as reduce(), find(), sum(), or collect() are consuming adapters.

On the other hand, certain methods called iterator adapters may also be defined on the Iterator trait. These take an iterator and return another. These methods are called between the creation of an iterator and its consumption. The primary example is, of course, map(). As a result of laziness, map() will do nothing unless fed into a consuming adapter.

Listing Modules under a Namespace in Elixir

Mete Can Eris — Mon, 21 Feb 2022 15:58:40 +0000

In Elixir, it is possible to get a list of compiled modules under an application using Erlang's application module.

Let's first see how we can obtain a list of all compiled modules, then we'll discuss the why and what.

How to get a list of compiled modules

I'll try to dig in and reference relevant tools by building upon this Stack Overflow answer by Aleksei Matiushkin.

Reading from Erlang's Reference Manual, the application module interacts with application controller, a process started at every Erlang runtime system. This module contains functions for controlling applications (for example, starting and stopping applications), and functions to access information about applications (for example, configuration parameters).

In our code, we can communicate with the application module via the lowercase atom :application, just like the rest of Erlang’s extensive standard library. All Erlang modules are represented by lowercase atoms such as :os and :timer.

:application.get_key/2 will return a list of all compiled modules if given :modules atom as the second argument.

{:ok, modules} = :application.get_key(:app_name, :modules)
[
 App,
 App.ModuleA, App.ModuleA.ModuleX,
 App.ModuleB, App.ModuleB.ModuleY
]

We can then:

filter certain modules by binary pattern matching or by using split/1 and concat/1 functions under Module
call their functions (using Kernel's apply/3)
do whatever that serves our purpose by using the capabilities of the Stream and Enum modules.

{:ok, modules} = :application.get_key(:app_name, :modules)

modules
|> Stream.map(&Module.split/1)      # split
|> Stream.filter(fn module ->       # filter
  case module do
    ["App", "Namespace", "Base"] -> false
    ["App", "Namespace", _] -> true
     _ -> false
  end
end)
|> Stream.map(&Module.concat/1)     # concat
|> Stream.map(&{&1, apply(&1, :some_module_fn, [])})
|> Enum.map(fn output ->
  # ... do whatever
end)

Why do this?

Well, besides the fact that we can, I'll give a few examples from my own projects.

In Rubik (a work-in-progress visual scripting language implementation for a thesis project), I represented the visual computation nodes as Elixir modules conforming to a specific Protocol, such as Nodes.Arithmetic.Add, Nodes.Logic.And, or Nodes.List.Count.

Rubik is an open-source library, so developers integrating the library into their applications can code new nodes (specific to their use cases) that their users then can use in the visual scripting interface.

The visual scripting interface provides a dropdown menu that a user can select a node from. This dropdown is essentially populated by a list of available modules under the Nodes namespace via a similar pipeline.

So to add a new node, all a developer has to do is to code a module under the Nodes namespace and compile, and the node appears in the dropdown.

In Watchtower, I try to categorize smart contracts based on their interface via available ABIs (Application Binary Interface). Again, I represented smart contract types as Elixir modules living under a Contracts namespace.

Each module has a module attribute @schema that contains a subset interface that I use to match an ABI to a contract type.

  @schema [
    {"name", [], [:address]},
    {"symbol", [], [:string]},
    {"token0", [], [:address]},
    {"token1", [], [:address]},
  ]

Each module also implements (better yet adopts via a Behaviour) a match/1 function that compares the given ABI to the subset under @schema. Basically, the pipeline becomes a meta factory that returns the correct module given an ABI.

Of course, it's always possible to implement a factory module that would do the same thing, but this way, a single module attribute gets the whole job done.