Posted on Dec 11, 2019

Day24:Multi Thread - 100DayOfRust

#rust

In rust it is easy to spawn a thread, for example:

use std::thread;

fn main() {
    let t1 = thread::spawn(|| {
        println!("{}", "hello from t1");
    });
    // wait t1 to finish
    t1.join();
}

From my last article Call Shell Command, say if we want to copy a file from source path to target path, we could do something like this:

use std::thread;
use std::process::Command;

fn main() {
    let t1 = thread::spawn(|| {
        let status = Command::new("cp")
                             .arg(&source_file_path)
                             .arg(&target_file_path)
                             .status()
                             .unwrap();
        println!("Status: {}", status);
    });
    t1.join();
}

And if we want to copy multiple files to target path in parallel, we can spawn multi threads.

However, say if we have 2000 files to copy, spawn 2000 threads simultaneously may not be a good idea, because unlike Go's lightweight go-routine, Rust's thread is the real thread in OS level. To create a huge amount of threads, it may consume too much resource. Plus, if in you thread you need to open file, in those 2000 threads means it may have 2000 files need to be open simultaneously, but there is a limit for opened file amount for one program.

Use ulimit -n to see it in your terminal, MacOS X will return 256, and Linux will return 1024. Although they are default value, we can change it to a bigger number, open too many files is not a good solution.

The real solution is, we can use a thread pool: in this pool we have a limited number of threads, and feed those threads with tasks.

How big the pool size is? Well, usually we will set it to be the number of how many tasks can be run parallel, and in Rust we have a package num_cpus can help us to get this number.

In Cargo.toml:

[dependencies]
num_cpus = "1.11.1"
threadpool = "1.7.1"

In main.rs:

use threadpool::ThreadPool;
use num_cpus;

fn main() {
    // here we create a pool with size num_cpus::get()
    let pool = ThreadPool::new(num_cpus::get());

    for task in tasks.iter() {
        pool.execute(|| {
            // do some task here
        });
    }
    // wait for all tasks to be finished
    pool.join();
}