DEV Community: Praveen Chaudhary

Build Your Own curl - Rust

Praveen Chaudhary — Sat, 23 Mar 2024 17:26:08 +0000

We will build curl from scratch by accepting the coding challenge posted on Coding Challenges FYI.

Before moving ahead, you must know how TCP client-server connections work.

You can read more about GeeksforGeeks.

On Server Side: -

Socket:- Socket object to expose our endpoints.
setsockopt:- This function sets the extra options for the sockets if needed.
Bind:- It binds the socket with the IP and port.
Listen:- Socket is now in listening state, and listen at the specified port for the incoming connection request. Here it queues incoming client requests to connect. The second argument specifies the maximum number of requests it can queue.
Accept:- In this phase, the server calls the accept function, which initiates the 3-way handshaking. The client sends a SYN packet to the server, the server responds with the SYN-ACK packet, and blocks(waits) the connection until it finally gets the ACK packet.
Send/Recv:- Once ACK is received from the client, communication can proceed to and fro.

On Client Side: -

Socket initialization:- In this setup, the socket is defined with all the configurations needed to connect.
Connect:- In this phase, the client calls the connect function, which sends the SYN packet to the server with the intent to connect.
Send/Recv: Once the connection is established, the client can send and receive the data.

We are doing to achieve in a few steps: -

Getting the CLI arguments
Creation of socket connection
Sending the request
Parsing the response

1. Getting the CLI arguments.

We will be using the library for Clap - A simple-to-use, efficient, and full-featured library for parsing command line arguments and subcommands.

The clap library provides two different ways to build parse objects. First is the Builder pattern(a creational design pattern to create complex things by the step-by-step process) and second Derive pattern in which the library automatically generates code based on the macros.

We are using the Builder pattern for our CLI tool.

But you can implement Derive pattern at doc.rs/clap/_derive

'cli.rs'

use clap::{Arg, ArgMatches, Command};

pub fn get_arguments()-> ArgMatches{
    Command::new("Ccurl - custom curl")
        .about("It helps to make HTTP methods")
        .version("1.0")
        .author("Praveen Chaudhary <chaudharypraveen98@gmail.com>")
        .arg(Arg::new("url").index(1).required(true))
        .arg(
            Arg::new("x-method")
                .help("Http method which you want to use")
                .long("x-method")
                .short('X'),
        )
        .arg(
            Arg::new("data")
                .help("Payload you want to send with the request")
                .long("data")
                .short('d'),
        )
        .arg(
            Arg::new("headers")
                .help("Request header")
                .long("header")
                .short('H')
                .action(ArgAction::Append),
        )
        .arg(
            Arg::new("verbose")
                .help("verbose mode")
                .long("verbose")
                .short('v')
                .action(clap::ArgAction::SetTrue),
        )
        .get_matches()
}

Firstly, we have to define the basic info like about, author, and version.

We have defined all the arguments needed for our curl. We have made one positional required argument url.

Argument matching or parsing

Clap makes it easier to match arguments.

For verbose, we have used action method .action(clap::ArgAction::SetTrue) because it will not contain any subsequent value.
For headers, similarly, we have used action method .action(ArgAction::Append), Append will append new values to the previous value if any value has already encountered.
For others, we have simply used get_one method to get the value.

let verbose_enabled = matches.contains_id("verbose") && matches.get_flag("verbose");
    let url = matches.get_one::<String>("url").unwrap();
    let data = matches.get_one::<String>("data");
    let method = matches.get_one::<String>("x-method");
    let headers: Vec<&str> = matches
        .get_many::<String>("headers")
        .unwrap_or_default()
        .map(|s| s.as_str())
        .collect();

2. Drafting request with input information [ HTTP 1.1 - RFC9110.]

We will be using the RFC9110 for HTTP 1.1 client.

we will start with an empty string, and append all the information needed for the Request according to RFC.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    let default_method = String::from("GET");
    let method = method.unwrap_or(&default_method);
    let mut res = String::new();
    res += &format!("{} /{} {}\r\n", method, path, protocol);
    res += &format!("Host: {}\r\n", host);
    res += "Accept: */*\r\n";
    res += "Connection: close\r\n";
    ....
    ....
    ....
    res += "\r\n";
    res
}

For PUT and POST, we need to add headers and data.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    ....
    ....

    if method == "POST" || method == "PUT" {
        if headers.len() > 0 {
            for head in headers {
                res += head;
            }
            res += "\r\n"
        } else {
            res += "Content-Type: application/json\r\n";
        }
        if let Some(data_str) = data {
            let data_bytes = data_str.as_bytes();
            res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
            res += data_str;
            res += "\r\n";
        }
    }

    ....
    res
}

According to RFC, for POST or PUT, we need to provide Content-Length and Content-Type header.

So now we have a complete request string. Let's move to the socket connection, sending this request string to the server.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    let default_method = String::from("GET");
    let method = method.unwrap_or(&default_method);
    let mut res = String::new();
    res += &format!("{} /{} {}\r\n", method, path, protocol);
    res += &format!("Host: {}\r\n", host);
    res += "Accept: */*\r\n";
    res += "Connection: close\r\n";

    if method == "POST" || method == "PUT" {
        if headers.len() > 0 {
            for head in headers {
                res += head;
            }
            res += "\r\n"
        } else {
            res += "Content-Type: application/json\r\n";
        }
        if let Some(data_str) = data {
            let data_bytes = data_str.as_bytes();
            res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
            res += data_str;
            res += "\r\n";
        }
    }

    res += "\r\n";
    res
}

3. Creation of socket connection

We will be using the standard rust network library for socket connection with the host server.

fn main() {
    ....
    ....

    let tcp_socket = TcpStream::connect(socket_addr);

    match tcp_socket {
        Ok(mut stream) => {
            ....
            ....
        }
        Err(e) => {
            eprintln!("Failed to establish connection: {}", e);
        }
    }
    ....
    ....
}

Once we are successfully connected, we can listen and send your request to the server.

4. Sending the request

First we have to check if verbose mode is enabled, and then we print out the request.
We have used the write_all to ensure our that our whole buffer is added to the stream.
Create a new empty buffer, and provide this buffer to the stream, to read the response data from the host.
Converts that bytes string to UTF-8 string using the from_utf8_lossy.
Print the response header and body.

fn main() {
    ....
    ....

    match tcp_socket {
        Ok(mut stream) => {
            if verbose_enabled {
                let lines = buffer_str.lines();
                for line in lines {
                    println!("> {}", line)
                }
            }
            stream
                .write_all(buffer_str.as_bytes())
                .expect("Failed to write data to stream");

            //initializing the buffer reads data from the stream and stores it in the buffer.
            let mut buffer = [0; 1024];
            stream
                .read(&mut buffer)
                .expect("Failed to read from response from host!");

            // converts buffer data into a UTF-8 encoded string (lossy ensures invalid data can be truncated).
            let response = String::from_utf8_lossy(&buffer[..]);

            // dividing the response headers and body
            let (response_header, response_data) = parse_resp(&response);
            if verbose_enabled {
                let lines = response_header.split("\r\n");
                for line in lines {
                    println!("< {}", line)
                }
            }
            println!("{}", response_data);
        }
        Err(e) => {
            eprintln!("Failed to establish connection: {}", e);
        }
    }
    ....
    ....
}

5. Time for Testing

cli - cargo run -- http://eu.httpbin.org:80/get
response -

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "eu.httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-65fec214-25771a3e732101c433ce67a7"
  }, 
  "origin": "49.36.177.79", 
  "url": "http://eu.httpbin.org/get"
}

Similarly, you can test others.

Hurray!! We have been able to make our curl.

Source Code

If this helps you, Do star and contribute.
Ccurl

Happy Hacking
Rustaceans!

Web Assembly With Rust

Praveen Chaudhary — Sun, 22 Jan 2023 12:45:52 +0000

It is a universal binary code that is designed for the web.

Wasm uses a low virtual machine with linear memory. It executes safely inside the sandbox.

Image Source :→ Tapadoo

When should we use WASM?

Whenever we need to do the heavy computation, we should use wasm as they are efficient and fast.

But why Rust?

You can choose any language that you wish to, but Rust provides low-level control over the memory and is free from non-deterministic garbage collection.

Let's create our Fibonacci wasm package

1. Creating/ Initializing our first lib

Make sure rust is installed. You can install it from here if not installed.

$  cargo new fibonacci-wasm --lib

2. Writing our Fibonacci function in Rust

It's pretty easy to write a function.

pub fn fibonacci(n: i64) -> i64 {
    if n == 0 {
        return 0;
    } else if n == 1 {
        return 1;
    } else {
        let mut a = 0;
        let mut b = 1;
        for _i in 2..n {
            let c = a + b;
            a = b;
            b = c;
        }
        return b;
    }
}

Apart from this, you can do whatever you can do in the browser like consoling, alerting etc.

You can use the web_sys which provides the web browser APIs in the Rust code.
web_sys is a part of wasm-bindgen which extends the functionality that you can do with the Rust code.
For JS bindings you can use the js_sys which provides binding to JavaScript’s standard, built-in objects, including their methods and properties

You can use the web_sys and js_sys to do whatever you thought of doing in the browser.

3. Let's create JS binding with our Rust code

It is a difficult part but we have crate/library to do this.

We will be using it to Facilitate high-level interactions between Wasm modules and JavaScript.

Binding our rust code with js is simple with the help of the wasm_bindgen macro.

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn fibonacci(n: i64) -> i64 {
    ....
}

Most of the things are done.

4. Let's compile and generate wasm package with wasm-pack

wasm-pack -> one-stop solution for working with rust-generated wasm code with the javascript.

YOu can use the wasm-pack to easily publish to npm registry or use it inside in node_modules folder.

You can install wasm-pack here.

$ wasm-pack build --target web

Now you can find a new folder created with the name pkg.

You can just import WebAssembly modules the same way you would import JavaScript modules.

5. Testing our package in the browser

Create a basic Html page and in the body, section adds the script tag with type module.
```
<script type="module">
```

Importing the wasm package.

    <script type="module">
    import init, { fibonacci } from "./pkg/fibonacci_wasm.js";
    ....
    </script>

Initializing our wasm code

<script type="module">
  import init, { fibonacci } from "./pkg/fibonacci_wasm.js";

  await init();
  ....
</script>

Running our Fibonacci function

<script type="module">
  import init, { fibonacci } from "./pkg/fibonacci_wasm.js";

  await init();
  console.log(fibonacci(10));
</script>

Results

Resources

iJS 2021: WebAssembly, Rust, and TypeScript – a Match Made in Heaven

Feel free to ask queries and make pull requests for changes and suggestions in GitHub.

Source Code

Github

Happy Hacking

Rustaceans!

Server-Sent Events in Rust

Praveen Chaudhary — Sun, 01 Jan 2023 08:36:21 +0000

What is a Server-Sent Events?

It is a standard that enables the servers to send a stream of events to the subscribed clients.

Why do we need SSE?

SSE helps to get real-time data from the server. Web sockets can do the same but they are bidirectional and the client can also send the data to the server.

They can be overkill for the applications which need to send updates to the clients only. So we need SSE for that purpose.

Some of the Uses Cases

Twitter feeds
Score Updates
Stocks rates

How does SSE works?

When a client requests, it forms an EventSource Object with the url of the server endpoint over the regular HTTP request.

The server responds back with an event stream and keeps the connection until it feels that it has no events to send (or stable) or the client explicitly close the connection.

How to implement SSE with actix-web?

Let's follow the easy steps:-

Libraries required:-

actix-web:- A powerful, pragmatic, and extremely fast web framework for Rust
actix-web-lab:- Experimental extractors, middleware, and other extras for possible inclusion in Actix Web.

1. Setting Up Actix-web

Initializing the Rust Project

Start a new project with the following cargo new <file-name>.

Implementing basic actix server

Let's clone the sample code from actix official docs.

Add dependency in cargo.toml

....
[dependencies]
actix-web = "4"
#sse
actix-web-lab = "0.18.5"
parking_lot = "0.12.1"
futures-util = { version = "0.3.25", default-features = false, features = ["std"] }
....

Let's write code for actix server.

use actix_web::{web, App, HttpServer};

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .route("/hello", web::get().to(|| async { "Hello World!" }))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

Implementing SSE using actix-web-lab

Broadcaster

Initialise the client struct
```
#[derive(Debug, Clone, Default)]
struct BroadcasterInner {
    clients: Vec<sse::Sender>,
}
```
- The Debug trait is used for printing or debugging
- The Clone trait let you clone the BroadcasterInner/ clients
- Default trait lets you assign default value on initialisation
Initialise the Broadcaster struct

    pub struct Broadcaster {
        inner: Mutex<BroadcasterInner>,
    }

We have used the Mutex to mutate the value safely across the threads.

Creating methods to implement ping, create a new instance, remove stale client, create new client, and broadcast message.

Create a new broadcaster instance

impl Broadcaster {
    ....
    pub fn create() -> Arc<Self> {
        let this = Arc::new(Broadcaster {
            inner: Mutex::new(BroadcasterInner::default()),
        });
        Broadcaster::spawn_ping(Arc::clone(&this));
        this
    }
    ....
}

This method will create a broadcaster instance which will ping the clients.

ping client

fn spawn_ping(this: Arc<Self>) {
    actix_web::rt::spawn(async move {
        let mut interval = interval(Duration::from_secs(10));

        loop {
            interval.tick().await;
            this.remove_stale_clients().await;
        }
    });
}

This ping method is used to know the active clients and remove the stable ones. You can modify the logic as per your requirement.

Remove stale clients

async fn remove_stale_clients(&self) {
    let clients = self.inner.lock().clients.clone();
    println!("active client {:?}",clients);

    let mut ok_clients = Vec::new();

    println!("okay active client {:?}",ok_clients);

    for client in clients {
        if client
            .send(sse::Event::Comment("ping".into()))
            .await
            .is_ok()
        {
            ok_clients.push(client.clone());
        }
    }

    self.inner.lock().clients = ok_clients;
}

This method will revalidate the client list and removes stale one.

Create a new client
```
pub async fn new_client(&self) -> Sse<ChannelStream> {
    println!("starting creation");
    let (tx, rx) = sse::channel(10);

    tx.send(sse::Data::new("connected")).await.unwrap();
    println!("creating new clients success {:?}",tx);
    self.inner.lock().clients.push(tx);
    rx
}
```
sse::channel will create sender and receiver pair. The channel will take a buffer parameter which is the count of unsend messages stored without waiting.

The first part -> sender can be cloned and shared across threads.

The second part -> receiver is the regular event stream.

Broadcast new message

pub async fn broadcast(&self, msg: &str) {
    let clients = self.inner.lock().clients.clone();

    let send_futures = clients
        .iter()
        .map(|client| client.send(sse::Data::new(msg)));

    // try to send to all clients, ignoring failures
    // disconnected clients will get swept up by `remove_stale_clients`
    let _ = future::join_all(send_futures).await;
}

It will broadcast messages to the active clients.

Complete Broadcaster

use std::{sync::Arc, time::Duration};
use actix_web::rt::time::interval;
use actix_web_lab::sse::{self, ChannelStream, Sse};
use futures_util::future;
use parking_lot::Mutex;

pub struct Broadcaster {
    inner: Mutex<BroadcasterInner>,
}

#[derive(Debug, Clone, Default)]
struct BroadcasterInner {
    clients: Vec<sse::Sender>,
}

impl Broadcaster {
    /// Constructs new broadcaster and spawns ping loop.
    pub fn create() -> Arc<Self> {
        let this = Arc::new(Broadcaster {
            inner: Mutex::new(BroadcasterInner::default()),
        });
        Broadcaster::spawn_ping(Arc::clone(&this));
        // println!("created success");

        this
    }

    /// Pings clients every 10 seconds to see if they are alive and remove them from the broadcast list if not.
    fn spawn_ping(this: Arc<Self>) {
        actix_web::rt::spawn(async move {
            let mut interval = interval(Duration::from_secs(10));

            loop {
                interval.tick().await;
                this.remove_stale_clients().await;
            }
        });
    }

    /// Removes all non-responsive clients from broadcast list.
    async fn remove_stale_clients(&self) {
        let clients = self.inner.lock().clients.clone();
        println!("active client {:?}",clients);

        let mut ok_clients = Vec::new();

        println!("okay active client {:?}",ok_clients);

        for client in clients {
            if client
                .send(sse::Event::Comment("ping".into()))
                .await
                .is_ok()
            {
                ok_clients.push(client.clone());
            }
        }

        self.inner.lock().clients = ok_clients;
    }

    /// Registers client with broadcaster, returning an SSE response body.
    pub async fn new_client(&self) -> Sse<ChannelStream> {
        println!("starting creation");
        let (tx, rx) = sse::channel(10);

        tx.send(sse::Data::new("connected")).await.unwrap();
        println!("creating new clients success {:?}",tx);
        self.inner.lock().clients.push(tx);
        rx
    }

    /// Broadcasts `msg` to all clients.
    pub async fn broadcast(&self, msg: &str) {
        let clients = self.inner.lock().clients.clone();

        let send_futures = clients
            .iter()
            .map(|client| client.send(sse::Data::new(msg)));

        // try to send to all clients, ignoring failures
        // disconnected clients will get swept up by `remove_stale_clients`
        let _ = future::join_all(send_futures).await;
    }
}

Creating a route for the sse stream and broadcast message

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let broadcaster = Broadcaster::create();

    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(AppState {
                broadcaster: Arc::clone(&broadcaster)
            }))
            // This route is used to listen events/ sse events
            .route("/events{_:/?}", web::get().to(sse_client))
            // This route will create notification
            .route("/events/{msg}", web::get().to(broadcast_msg))
    })
    .bind(format!("{}:{}","127.0.0.1", "8000"))?
    .run()
    .await
}

Let's implement the sse_client and broadcast_msg

sse_client

pub async fn sse_client(state: web::Data<AppState>) -> impl Responder {
    state.broadcaster.new_client().await
}

broadcast_msg

pub async fn broadcast_msg(
    state: web::Data<AppState>,
    Path((msg,)): Path<(String,)>,
) -> impl Responder {
    state.broadcaster.broadcast(&msg).await;
    HttpResponse::Ok().body("msg sent")
}

Complete main.rs

use actix_web::HttpResponse;
use actix_web::Responder;
use actix_web::{web, App, HttpServer};
mod broadcast;
use self::broadcast::Broadcaster;
use std::{io, sync::Arc};
use actix_web_lab::extract::Path;

pub struct  AppState{
    broadcaster:Arc<Broadcaster>
}

// SSE
pub async fn sse_client(state: web::Data<AppState>) -> impl Responder {
    state.broadcaster.new_client().await
}

pub async fn broadcast_msg(
    state: web::Data<AppState>,
    Path((msg,)): Path<(String,)>,
) -> impl Responder {
    state.broadcaster.broadcast(&msg).await;
    HttpResponse::Ok().body("msg sent")
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let broadcaster = Broadcaster::create();

    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(AppState {
                broadcaster: Arc::clone(&broadcaster)
            }))
            // This route is used to listen to events/ sse events
            .route("/events{_:/?}", web::get().to(sse_client))
            // This route will create a notification
            .route("/events/{msg}", web::get().to(broadcast_msg))
    })
    .bind(format!("{}:{}","127.0.0.1", "8000"))?
    .run()
    .await
}

Testing the SSE
- Open the /events to subscribe to the event stream
- Then hit the /events/ to send a message to the clients.

Hurray! we have finally implemented the Server-Sent Events.

Feel free to ask queries and make pull request for changes and suggestion in GitHub.

Source Code GitHub

Happy Hacking
Rustaceans!

Rust Generics

Praveen Chaudhary — Sun, 09 Oct 2022 11:44:57 +0000

It is a way of writing generalised algorithms in such a way that can be used for the types specified later. It can be initialised for a specified type with help of parameters.

What we use generics:-

Abstract Types: Instead of writing explicit type like i32 or float we can use the generics for different types by writing different code
Adds Flexibility: We can use different data types with the same piece of code.
Reduces Code Duplication: We can have the same piece of code or both for any two types.
No Runtime Cost: Unlike Object Traits, generics don't have runtime costs. The compiler generates the different types in the background during compile time.

Let's get started

Usually the generics type parameter is represented by <T>. But you are free to use any.

Structs

1. Single placeholder type

Defining the struct with the placeholder/generic type.

struct Point<T> {
    x: T,
    y: T,
}

Let's try to use this for i32

pub fn main() {
    let i32_point = Point { x: 45, y: 90 };
    println!("x is {} and y is {}", i32_point.x, i32_point.y);
}

If you hover on the i32_point then you can find the type of variable is Point i32 i32. But it is not limited. You can even use the float for both x and y but they must be the same.

So how can we use the different types for the two fields x and y?

It can be done using the two placeholder

2. Two placeholder

struct definition

struct TwoPoint<T, U> {
    x: T,
    y: U,
}

let's test

pub fn main() {
    let explicit_point = TwoPoint { x: 9, y: 90.3 };
    println!(
        "explicit_point x is {} and y is {}",
        explicit_point.x, explicit_point.y
    );
}

Similarly we can two different data types for the x and y, even vectors.

Enums

1. Generics with Enum

creating a result enum that will store the result whether pass, fail, or awaiting.

#[derive(Debug)]
enum Result<T> {
    Passed(T),
    Failed(T),
    Pending,
}

Used the Debug trait because the enum is not primitive because all the primitives include the debug capabilities/trait.

Testing

pub fn main() {
  let result = Result::Passed(43);
  match result {
      Result::Passed(a) => println!("Student Passed with {}", a),
      Result::Failed(b) => println!("Student Failed with {}", b),
      Result::Pending => println!("Student result is pending"),
  }
}

You can use different generics types for Passed and Failed as shown in the above case.

2. Using struct and enums together with generics

we are using the Result enum from the above example

struct Student<T> {
    name: String,
    roll_no: T,
    result: Result<T>,
}

Let's use it

pub fn main() {
  let student = Student {
      name: "Praveen".to_string(),
      roll_no: 78,
      result: Result::Passed(98),
  };

  match student.result {
      Result::Passed(a) => println!("Student Passed with {}", a),
      Result::Failed(b) => println!("Student Failed with {}", b),
      Result::Pending => println!("Student result is pending"),
  }
}

Functions with Generics

1. Function with single generic type parameter and single trait

function definition

fn custom_add<T: std::ops::Add<Output = T>>(a: T, b: T) -> T {
    a + b
}

Here Output=T ensures that after the + or - result must be the same. It can be different if you wish too like the addition of two atoms can result in the molecule(T+T=U)

let a = 4;
let b = 5;
println!("sum of {} and {} is {}", a, b, custom_add(a, b));

2. Function with single generic type parameter and multiple traits

fn custom_add_sub<T: std::ops::Add<Output = T> + std::ops::Sub<Output = T> + std::fmt::Debug>(
    a: T,
    b: T,
) -> T {
    // Debug trait is needed for this print statement
    println!("{:?} {:?}", a, b);
    a + b
}

Using the type

pub fn main() {
  println!("sub of {} and {} is {}", a, b, custom_add_sub(a, b));
}

I know it's easy for you now

3.Function with multiple generic type parameters and multiple traits and where keyword

// using the where to define a generic type
fn custom_add_sub_mul<T, E>(a: T, b: T, c: E) -> T
where
    T: std::ops::Mul<Output = T> + std::ops::Add<Output = T> + std::ops::Sub<Output = T>,
    E: std::fmt::Debug,
{
    println!("{:?}", c);
    a * b
}

where keyword makes our code clearer. It doesn't have any impact on how the code run.

pub fn main() {
  println!("mul of {} and {} is {}", a, b, custom_add_sub_mul(a, b, b));
}

4. Functions with custom trait

Instead of using the predefined trait, we can use the custom trait as we like to.

trait SomeCustomTrait {
    fn bla_bla(&self, a: &str, b: &str) -> String;
}
#[derive(Debug)]
struct TestStruct {
    something: i32,
}

impl SomeCustomTrait for TestStruct {
    fn bla_bla(&self, a: &str, b: &str) -> String {
        self.something.to_string() + "-" + a + "-" + b
    }
}

fn do_this<T>(some_var: &T) -> String
where
    T: SomeCustomTrait + std::fmt::Debug,
{
    println!("{:?}", some_var);
    some_var.bla_bla("first", "second")
}

In the function, we have used our SomeCustomTrait along with the Debug. We can even add more.

pub fn main() {
  let test = TestStruct { something: 1000 };
  let result = do_this(&test);
}

Impl with Generics

struct ImplStruct<T, U> {
    field1: T,
    field2: U,
}

impl<T, U> ImplStruct<T, U>
where
    T: std::fmt::Debug,
    U: std::fmt::Debug,
{
    fn log_something(&self) {
        println!("{:?} {:?}", self.field1, self.field2);
    }
}

You can notice here the generics type is defined at the impl block instead of a struct but you can define it at both blocks but Impl block is a must.

pub fn main() {
  let impl_test = ImplStruct{
      field1:5.6,
      field2:vec![1,2,3],
  };
  impl_test.log_something();

So, we have covered the basics of Rust Generics but you can do a lot more with Generics.

Resources

rust-lang by example
Doug Milford Source code RustyRustLessons Generics

Happy Hacking
Rustaceans!

React Native Push Notification - Firebase

Praveen Chaudhary — Sun, 18 Sep 2022 16:32:40 +0000

Notifications play an important role in targeting the right users at the right time. It helps in the following ways:-

Increasing App Engagement
Targeting the right user
Enhancing the conversion rate
Engaging user retention for a period of time

Libraries used: -

Firebase Cloud Messaging: It is a cross-platform messaging solution that lets you reliably send messages at no cost.
React Native MMKV (Optional): A library that allows you to easily use MMKV inside your React Native applications. We will need this if we want to store FCM token locally in a fast manner.

What we are going to do?

Let's divide the workflow in easy steps:-

Setting up React Native project.
Installing Firebase libraries needed for FCM Token generation and notification.
Integrating React Native MMKV for fast storage. (Optional)
Generating FCM token and storage
Requesting user permission for the notification.
Setting up notification Listener
Testing

1. Setting up React Native project.

React CLI will generate the template for you to get started quickly.

Initialize Project

npx react-native init SimpleReactNativeNotification

If you are new to React Native, then follow the React Native Official Docs to setup up the environment.

2. Installing Firebase libraries needed for FCM Token generation and notification.

Firebase libraries provide various method to support the firestore, real-time database firebase, Cloud messaging, and so on.

Installing the React Native Firebase library

# Using npm
npm install --save @react-native-firebase/app

# Using Yarn
yarn add @react-native-firebase/app

Additional steps for setup in Android and IOS

Android Visit Firebase console and start a new project.

Add a new Android application and enter your project details.

Download the google-services.json file and place it inside your project at the following location: /android/app/google-services.json.

For more info

I don't have a mac device💻. So you can follow here 😟

Adding Firebase messaging libraries along with above parent library

# Install the messaging module

yarn add @react-native-firebase/messaging

or 

npm install --save @react-native-firebase/messaging

# If you're developing your app using iOS, run this command
cd ios/ && pod install

3. Integrating React Native MMKV for fast storage. (Optional)

We don't need to generate the FCM token every time, as it is the same most of the time.

There are some exceptions, but we can handle them and reduce the number of calls to generate the token every time.

Validity of FCM Token

The app deletes Instance ID
The app is restored on a new device
The user uninstalls/reinstalls the app
The user clears app data.

Installing

yarn add react-native-mmkv
or 
npm install --save react-native-mmkv

cd ios && pod install

Creating instance to use throughout the app

Create a new file 'storage.js'

storage.js

import { MMKV } from 'react-native-mmkv'

export const storage = new MMKV()

Hurray! Our mmkv storage is ready to use

4. Generate FCM token and store it

Let's write a function to generate the FCM token.

Importing the messaging library which we installed earlier

import messaging from '@react-native-firebase/messaging';

1. Without storage

creating a function getFcmToken - './notifications.js'

const getFcmToken = async () => {
  try {
    const newFcmToken = await messaging().getToken();
    console.log(newFcmToken);
    return newFcmToken;
  } catch (error) {
    console.error(error);
    return null;
  }
};

Let's make changes in App.js

import {getFcmToken} from './notifications';

const App = () => {
  ....
  const [generatedToken, setGeneratedToken] = useState();

  useEffect(() => {
      const fetchToken = async () => {
        const token = await getFcmToken();
        if (token) {
          setGeneratedToken(token);
        }
      };
      void fetchToken();
    }, []);
  ....
}

2. With Storage

creating a function getFcmTokenFromLocalStorage - './notifications.js'

const getFcmTokenFromLocalStorage = async () => {
  const fcmtoken = localStorage.getString('fcmtoken');
  if (!fcmtoken) {
    try {
      const newFcmToken = await messaging().getToken();
      localStorage.set('fcmtoken', newFcmToken);
    } catch (error) {
      console.error(error);
    }
  } else {
    console.log('token found', fcmtoken);
  }
};

Let's make changes in App.js

import {getFcmToken} from './notifications';

const App = () => {
  ....
  const fcmToken = localStorage.getString('fcmtoken');

  useEffect(() => {
      const fetchTokenByLocal = async () => {
        await getFcmTokenFromLocalStorage();
      };
      void fetchTokenByLocal();
    }, []);
  ....
}

Use Hooks to see the tokens

useEffect(() => {
  console.log('storage', fcmToken, 'newly generated', generatedToken);
}, [fcmToken, generatedToken]);

5. Requesting user permission for the notification.

iOS prevent notification unless explicitly approved by the user.

On Android, you do not need to request user permission. This method can still be called on Android devices; however, and will always resolve successfully.

In Android, we don't need any special permission for notification, but this function can still be called and will resolve successfully

'./notifications.js'

const requestUserPermission = async () => {
  const authStatus = await messaging().requestPermission();
  const enabled =
    authStatus === messaging.AuthorizationStatus.AUTHORIZED ||
    authStatus === messaging.AuthorizationStatus.PROVISIONAL;

  if (enabled) {
    console.log('Authorization status:', authStatus);
  }
};

Setting up notification Listener

We can listen for real-time FCM messages on any device.

State	Description
Foreground	When the application is open and in view.
Background	When the application is open, however, in the background (minimized). This typically occurs when the user has pressed the "home" button on the device or has switched to another app via the app switcher.
Quit	When the device is locked or the application is not active or running. The user can quit an app by "swiping it away" via the app switcher UI on the device.

You can read more about here.

Let's Handle these three cases: -

'./notifications.js'

const notificationListener = () => {

  messaging().onNotificationOpenedApp(remoteMessage => {
    console.log(
      'Notification caused app to open from background state:',
      remoteMessage.notification,
    );
  });

  // Quiet and Background State -> Check whether an initial notification is available
  messaging()
    .getInitialNotification()
    .then(remoteMessage => {
      if (remoteMessage) {
        console.log(
          'Notification caused app to open from quit state:',
          remoteMessage.notification,
        );
      }
    })
    .catch(error => console.log('failed', error));

  // Foreground State
  messaging().onMessage(async remoteMessage => {
    console.log('foreground', remoteMessage);
  });
};

For handling the Background notification, we need to add a handler at the './index.js'

import messaging from '@react-native-firebase/messaging';

// Register background handler
messaging().setBackgroundMessageHandler(async remoteMessage => {
  console.log('Message handled in the background!', remoteMessage);
});

7. Testing

First, grab the token from the terminal generated by console.log.

Visit TestFcm.com

We need the server keys to test. We can get this from the firebase console.

Open the project and get the server key by following the below steps:-

If your cloud messaging API is not enabled then

Result

Foreground Notification

Background and Quiet Notification

Source Code
Simple-React-Native-Push-Notification

Resources: -

For any question, feel free to ask your doubts and open pull request for any changes or suggestion.

Thank you,
Happy Hacking

Rust - Interior mutability - Cell

Praveen Chaudhary — Thu, 18 Aug 2022 16:20:00 +0000

Interior mutability provides the ability to mutate the value even in case of immutable reference.

There is no legal way to convert the &T(exclusive reference) to the &mut T(mutable reference) and this is called undefined behavior.
But we have UnsafeCell which helps to cope with the immutability constraint of &T, and provides a shared reference &UnsafeCell which points to the value to be mutated called Interior mutability.

UnsafeCell
It is the building block of interior mutability. It is used to eliminate some Rust optimizations that assume exclusive mutable access (or aliasing) inside the cell.

Cell

It abstraction over the UnsafeCell (which is unsafe in nature), So it is unsafe, but taking some measures when we ensure the safety.

It is not suitable for vec, String, or anything that stores data in heap memory as it is expensive to use Copy trait.

Why to use Cell?

Suppose we have a cyclic data structure, we are using the mutable reference for accessing the value. Once we reach the end, we start from the first element, then we have two mutable references to a single value. Suppose now we used the mem::swap then bad thing will happen. It is undefined behaviour to ever have multiple mutable references to a single value concurrently, no matter whether you're in a single-threaded or multi-threaded context.

Why to use UnsafeCell with Cell?

If we try to mutate the value directly through the shared pointer without unsafeCell then it will always result in undefined behaviour because the compiler may try to optimize based on the current assumption that the pointer isn't shared, which would be invalid if it was shared.

It can be done using the UnsafeCell if you know it was truly not shared (even on the same thread), because that prevents some of those optimizations such is it safe to cache when we have &selfand when you have &mut self it makes sure that no two &mut T never points to same T.

How to mutate in Rust

Immutability can be possible by &T reference known as aliasing
Mutability can be only possible by having &mut T reference. This type of reference is exclusive in nature.

So what does shareable mutable reference means?

It means we have shared references(i.e &T type) but with the extra power to mutate in a controlled manner.

How do shared references allow mutation?

Interior mutability types which uses the shared references let you do because they maintain additional, special-case invariants that the compiler cannot check, that ensure that no conflicts arise even if you modify through a shared reference. There are lots of ways to do so with the help of CPU atomics, Mutexes, Cell, RefCell, etc.

How to ensure that we are using it in a controlled manner?

1. Not using Sync trait

Cell must not implement the Sync trait as it enables the usage of sharing references across threads which can result in adverse conditions as they can try to overwrite the value at the same time which leads to corrupted results.

Base code to understand the explanation

use std::cell::UnsafeCell;

pub struct Cell<T> {
    value: UnsafeCell<T>,
}

impl<T> Cell<T> {
    pub fn new(value: T) -> Self {
        Cell {
            value: UnsafeCell::new(value),
        }
    }

    pub fn set(&self, value: T) {
        unsafe { *self.value.get() = value };
    }

    pub fn get(&self) -> T
    where
        T: Copy,
    {
        unsafe { *self.value.get() }
    }
}

Let's implement some changes in the code to understand the working.

Implementing Sync Method for testing

unsafe impl<T> Sync for Cell<T>{}

Writing Test Case

#[cfg(test)]
mod test {

  use super::Cell;

  #[test]
  fn bad2(){
    use std::sync::Arc;
    let x = Arc::new(Cell::new(0));
    let x1 = Arc::clone(&x);
    let j1 = std::thread::spawn(move || {
      for _ in 0..1000000{
        let x = x1.get();
        x1.set(x+1);
      }
    });
    let x2 = Arc::clone(&x);
    let j2 = std::thread::spawn(move || {
      for _ in 0..1000000{
        let x = x2.get();
        x2.set(x+1);
      }
    });
    j1.join().unwrap();
    j2.join().unwrap();
    assert_eq!(x.get(),2000000)
  }
}

Arc is used to share the reference between the threads.

In the above code, we are spawning two threads and they are simultaneously mutating the value of x in each iteration 0 to 1000000.

Let's run test

running 1 test
test cell::test::bad2 ... FAILED

failures:

---- cell::test::bad2 stdout ----
thread 'cell::test::bad2' panicked at 'assertion failed: `(left == right)`
  left: `1170776`,
 right: `2000000`', src/cell.rs:96:5

Why does the assertion fail?

Instead, of getting 2000000, we get only 1170776. Because one or another thread read the old value of x and incremented and set it to a new value obtained.

2. The get method must implement the Copy trait which will give the cloned value, not the exclusive reference to it. If we don't use the Copy trait, then what happens?

For example:-

Let's try to understand by code

Returning the pointer to the value inside the Cell

pub fn get(&self)->&T{
  unsafe {&*self.value.get()}
}

Let's write some test

#[test]
fn bad1() {
  let x = Cell::new(true);
  let y = x.get();
  x.set(false);
  eprintln!("{}",y);
}

We have something in Cell and let it assign to a variable x then store the reference to y by making some change in get method. Then we try to change the value by set method.

Now, if we try to access the y. It should fail because once we set a new value, then the previous memory must be released.

⇒ If the test doesn't fail, it might be because the system might not free the memory instantly.

Conclusion:

Always use Cell when you have an immutable struct with numerous fields, and you want to change only 1-2 two fields.
It can be used for setting a flag in a single thread to know the status of something.

Special shoutout to Jon Gjengset
. It inspires me to write simple cron scheduler.

Reference taken:

Feel free to ask queries. You can connect to me on [LinkedIn].(https://www.linkedin.com/in/chaudharypraveen98/)

Happy Hacking
Rustaceans!

Scheduling Async Tasks made easy with Actix Cron.

Praveen Chaudhary — Sun, 24 Jul 2022 09:52:35 +0000

Cron is simply a scheduling daemon that executes the task at a specified interval. We usually use cron for backing up the data, report generation, synchronizing setting in background etc.

What will we learn : -

Setting Up Actix-web.
Adding duration for periodic tasks.
Adding cron parser for scheduling flexibility.
Performing DB operations using cronjobs.

Libraries required :-

actix-web :- A powerful, pragmatic, and extremely fast web framework for Rust
reqwest :- higher level HTTP client library
actix-rt :- Tokio-based single-threaded async runtime for the Actix ecosystem
cron :- A cron expression parser and schedule explorer.
chrono :- Date and time library for Rust

1. Setting Up Actix-web

Initializing the Rust Project

Start a new project with following cargo new <file-name>.

Implementing basic actix server

Let's clone the sample code from actix official docs.

Add dependency in cargo.toml

....
[dependencies]
actix-web = "4"
....

Let's write code for actix server.

use actix_web::{web, App, HttpServer};

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .route("/hello", web::get().to(|| async { "Hello World!" }))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

2. Adding duration for periodic tasks.

use std::{collections::HashMap, time::Duration};
use reqwest;
use actix_rt::time;

// async function to get data from url using reqwest library

async fn get_ips() -> HashMap<String, String> {
    let resp = reqwest::get("https://httpbin.org/ip")
        .await.unwrap()
        .json::<HashMap<String, String>>()
        .await.unwrap();
    resp
}

async fn main() -> std::io::Result<()> {

  actix_rt::spawn(async {
      let mut interval = time::interval(Duration::from_secs(20));
      loop {
          interval.tick().await;
          let result = get_ips().await;
          println!("20 sec {:?}",result);
      }
  });
  ....
}

Confused??

No, need to get confused. Let's try to understand step by step code.

actix_rt ()
It is a tokio-based single threaded async runtime for actix system.

Single threaded means only one command get executed at a time.

actix_rt::spawn : Spawns a future on the current thread as a new task. If not immediately awaited, the task can be cancelled using JoinHandle::abort (it detaches the current thread when dropped).

actix_rt::spawn(async move {}) : move will let you capture closure's environment variable.

Closures are simple anonymous function
that can be store in variable and doesn't require you to annotate the types.

tokio time::interval : Creates new Interval that yields with interval of duration. The first tick completes immediately.

interval.tick() : Completes when the next instant in the interval has been reached.

Duration::from_secs(20) : Creates a new Duration with the input whole number in seconds.

3. Adding cron parser for scheduling flexibility.

use std::{collections::HashMap, time::Duration, str::FromStr};
use chrono::{Local, FixedOffset};
use cron::Schedule;
use reqwest;
use actix_rt::{self, time};

use actix_web::{web, App, HttpServer};

async fn get_ips() -> HashMap<String, String> {
    let resp = reqwest::get("https://httpbin.org/ip")
        .await.unwrap()
        .json::<HashMap<String, String>>()
        .await.unwrap();
    resp
}

#[actix_web::main] // or #[tokio::main]
async fn main() -> std::io::Result<()> {
    actix_rt::spawn(async move {
        let expression = "1/50   *   *     *       *  *  *";
        let schedule = Schedule::from_str(expression).unwrap();
        let offset = Some(FixedOffset::east(0)).unwrap();

        loop {
            let mut upcoming = schedule.upcoming(offset).take(1);
            actix_rt::time::sleep(Duration::from_millis(500)).await;
            let local = &Local::now();

            if let Some(datetime) = upcoming.next() {
                if datetime.timestamp() <= local.timestamp() {

                    let result = get_ips().await;
                    println!("{:?}",result);
                }
            }
        }
    });
    HttpServer::new(|| {
        App::new()
            .route("/hello", web::get().to(|| async { "Hello World!" }))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

Let's try to understand what new happening here

cron expression : do follow the cron pattern. If you want to learn more about it, here.

//                  sec  min   hour   day of month   month   day of week   year
  let expression = "0   30   9,12,15     1,15       May-Aug  Mon,Wed,Fri  2018/2";

Schedule::from_str : IT will parse the value from string.

FixedOffset : The fixed offset is used to create the date instance. Positive secs mean the Northern Hemisphere and negative secs means the Western Hemisphere.

schedule.upcoming(offset).take(1) : upcoming will take the offset instance and returns an iterator of the DateTime object which matches the schedule.

take : it returns the first item in iterator.

thread::sleep : it is necessary to sleep before we check conditions because checking condition at every millisecond is not ideal as our use case is in seconds or greater than that. If you want minute milliseconds control over cron you can try lower time too.

Local::now() : It will return the current date time depending on local timezone.

if let Some(datetime) = upcoming.next() {if datetime.timestamp() <= local.timestamp() {} } : If we find a datetime object in iterator and local date time(current) is >= the iterator one then it will run the function.

4. Performing DB operations using cronjobs.

It's pretty similar, just like above two. I will try to give you the glimpse and the example code to understand by your self.

We will use the clousure(move) over the function to get the DB pool and logger.

Please find the code here, actix-question-bank-stackoverflow.

Special shoutout to cronjob. It inspires me to write simple cron scheduler.

Feel free to ask queries and make pull request for changes and suggestion in GitHub.

Github - Source Code

Happy Hacking
Rustaceans!

Form Validation in Rust (Actix-Web)

Praveen Chaudhary — Mon, 27 Jun 2022 16:16:36 +0000

Form validation is the important part of Software Development Lifecycle Process. It based on the data type define while creating a form. It basically ensures few things : -

Prevent form abuse from malicious users.
Reduce spams.
Ensure form users enter only legitimate data.

Tools Used : -

Actix-Web : A powerful, pragmatic, and extremely fast web framework for Rust
Validator : Macros 1.1 custom derive to simplify struct validation inspired by marshmallow and Django validators
lazy_static : A macro for declaring lazily evaluated statics in Rust.
regex : An implementation of regular expressions for Rust.

Dependencies

'./Cargo.toml'

[dependencies]
actix-web = "4"
validator = { version = "0.12", features = ["derive"] }
serde = { version = "1.0.104", features = ["derive"] }
lazy_static = "1.4.0"
regex = "1.5.6"

Let's get Rusty

Initializing the Rust Project

Start a new project with following cargo new <file-name>.

Implementing basic actix server

Let's clone the sample code from actix official docs.

use actix_web::{get, web, App, HttpServer, Responder};

#[get("/hello/{name}")]
async fn greet(name: web::Path<String>) -> impl Responder {
    format!("Hello {name}!")
}

#[actix_web::main] // or #[tokio::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .route("/hello", web::get().to(|| async { "Hello World!" }))
            .service(greet)
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

Creating Struct to hold data

In this project, we are creating a student model/struct.

pub struct Student {
    pub name: String,
    pub email: String,
    pub age: u32,
    pub username: String,
    pub password: String,
}

Implementing Serde Serialize and Deserailize traits

It will convert the json data to struct and vice versa.

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
pub struct Student {
    pub name: String,
    ....
}

You can read more here

Creating Route and Handler

It will be a post route which accepts json form data, validate it and return response depending on whether the validation is successful or failed.

Route

'./src/main.rs

async fn main() -> std::io::Result<()> {
    ....
    HttpServer::new(|| {
        App::new()
            .route("/", web::post().to(create_student))
    })
    ....
}

'./src/main.rs

Handler

pub async fn create_student(json: web::Json<Student>) -> impl Responder {
    let is_valid = json.validate();
    match is_valid {
        Ok(_) => HttpResponse::Ok().json("success"),
        Err(err) => HttpResponse::BadRequest().json(err),
    }
}

Here the line let is_valid = json.validate(); implements the validator. Now lets define the define validation rules.

Creating validation rules

Lets derive the Validator trait from validator.rs.

Add the validator trait along with serde Serialize and Deserialize traits.

use validator::Validate;

#[derive(Serialize, Deserialize, Validate)]
pub struct Student {
    ....
}

But what are the requirements for Student Struct.

Name : It must be greater than 3 characters.
Email : It must be valid Gmail address.
Age : It must be between 18 to 22 years.
Password : It must be 8 character with min one lower case alphabet,min one upper case alphabet, min one number and min one special character. Doesn't have white spaces.
Username : It must be alphanumeric and 6 characters long.

Implementing conditions

Name :
```
#[validate(length(min = 3,message = "Name must be greater than 3 chars"))]
pub name: String
```
length : It ensures that the length of data must be greater than or less than defined length.

message : It is that client side message which shows the error.
Email :
```
#[validate(
    email,
    contains(pattern = "gmail", message = "Email must be valid gmail address")
)]
pub email: String,
}
```
email : It ensures that the email contains which requirement like @ symbol,domain name and extension like .com and user-id.

contains and pattern : It ensures that the particular string must contain particular sub word(pattern) and here is gmail.

Age :

#[validate(range(min = 18, max = 22, message = "Age must be between 18 to 22"))]
pub age: u32,

range : It enforces that the size should fall in a particular range.

Username :

lazy_static! {
    static ref RE_USER_NAME: Regex = Regex::new(r"^[a-zA-Z0-9]{6,}$").unwrap();
}
pub struct Student {
    ....
    #[validate(
        regex(
            path = "RE_USER_NAME",
            message = "Username must number and alphabets only and must be 6 characters long"
        )
    )]
    pub username: String,
    ....
}

lazy_static : Lazy evaluated statics in Rust. It is executed at the runtime includes anything requiring heap allocations, like vectors or hash maps etc

regex : It requires a path/pattern to match. To learn more about the regex pattern used , you can refer to this article.

Password :

lazy_static! {
    static ref RE_SPECIAL_CHAR: Regex = Regex::new("^.*?[@$!%*?&].*$").unwrap();
}

fn validate_password(password: &str) -> Result<(), ValidationError> {
    let mut has_whitespace = false;
    let mut has_upper = false;
    let mut has_lower = false;
    let mut has_digit = false;

    for c in password.chars() {
        has_whitespace |= c.is_whitespace();
        has_lower |= c.is_lowercase();
        has_upper |= c.is_uppercase();
        has_digit |= c.is_digit(10);
    }
    if !has_whitespace && has_upper && has_lower && has_digit && password.len() >= 8 {
        Ok(())
    } else {
        return Err(ValidationError::new("Password Validation Failed"));
    }
}
pub struct Student {
....
#[validate(
    custom(
        function = "validate_password",
        message = "Must Contain At Least One Upper Case, Lower Case and Number. Dont use spaces."
    ),
    regex(
        path = "RE_SPECIAL_CHAR",
        message = "Must Contain At Least One Special Character"
    )
)]
pub password: String,
....
}

custom : It requires a function which implements validation and we can even add our custom message by message argument.

Complete Code

extern crate lazy_static;
extern crate regex;
extern crate serde;

use actix_web::{web, App, HttpResponse, HttpServer, Responder};
use lazy_static::lazy_static;
use regex::Regex;
use serde::{Deserialize, Serialize};
use validator::{Validate, ValidationError};

lazy_static! {
    static ref RE_USER_NAME: Regex = Regex::new(r"^[a-zA-Z0-9]{6,}$").unwrap();
    static ref RE_SPECIAL_CHAR: Regex = Regex::new("^.*?[@$!%*?&].*$").unwrap();
}

#[derive(Serialize, Deserialize, Validate)]
pub struct Student {
    #[validate(length(min = 3, message = "Name must be greater than 3 chars"))]
    pub name: String,
    #[validate(
        email,
        contains(pattern = "gmail", message = "Email must be valid gmail address")
    )]
    pub email: String,
    #[validate(range(min = 18, max = 22, message = "Age must be between 18 to 22"))]
    pub age: u32,
    #[validate(
        regex(
            path = "RE_USER_NAME",
            message = "Username must number and alphabets only and must be 6 characters long"
        )
    )]
    pub username: String,
    #[validate(
        custom(
            function = "validate_password",
            message = "Must Contain At Least One Upper Case, Lower Case and Number. Dont use spaces."
        ),
        regex(
            path = "RE_SPECIAL_CHAR",
            message = "Must Contain At Least One Special Character"
        )
    )]
    pub password: String,
}
fn validate_password(password: &str) -> Result<(), ValidationError> {
    let mut has_whitespace = false;
    let mut has_upper = false;
    let mut has_lower = false;
    let mut has_digit = false;

    for c in password.chars() {
        has_whitespace |= c.is_whitespace();
        has_lower |= c.is_lowercase();
        has_upper |= c.is_uppercase();
        has_digit |= c.is_digit(10);
    }
    if !has_whitespace && has_upper && has_lower && has_digit && password.len() >= 8 {
        Ok(())
    } else {
        return Err(ValidationError::new("Password Validation Failed"));
    }
}
pub async fn create_student(json: web::Json<Student>) -> impl Responder {
    let is_valid = json.validate();
    match is_valid {
        Ok(_) => HttpResponse::Ok().json("success"),
        Err(err) => HttpResponse::BadRequest().json(err),
    }
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    println!("Starting the server at http://127.0.0.1:8080/");
    HttpServer::new(|| App::new().route("/", web::post().to(create_student)))
        .bind(("127.0.0.1", 8080))?
        .run()
        .await
}

Validating

Let's spin our actix server by cargo run.

Password Validation

Payload

{
    "name":"prave",
    "email":"praveen@gmail.com",
    "age":19,
    "username":"praveee",
    "password":"123@4aA"
}

Output

Name Validation

Payload

{
    "name":"pr",
    "email":"praveen@gmail.com",
    "age":19,
    "username":"praveee",
    "password":"123d@4aA"
}

Output

Email Validation

Payload

{
    "name":"praveen",
    "email":"praveen@outlook.com",
    "age":19,
    "username":"praveee",
    "password":"123d@4aA"
}

Output

Age Validation

Payload

{
    "age": [
        {
            "code": "range",
            "message": "Age must be between 18 to 22",
            "params": {
                "min": 18.0,
                "value": 4,
                "max": 22.0
            }
        }
    ]
}

Output

Username Validation

Payload

{
    "name":"praveen",
    "email":"praveen@gmail.com",
    "age":19,
    "username":"prsdfdgfd@e",
    "password":"123d@4aA"
}

Output

Feel free to ask queries and make pull request for changes and suggestion in GitHub.

Source Code
Github Repo Link

Happy Hacking
Rustaceans!

Error Handling in Rust (Actix Web)

Praveen Chaudhary — Tue, 14 Jun 2022 08:38:51 +0000

Errors are part of software development. There are basically two types of errors in Rust i.e recoverable and unrecoverable.

Recoverable errors have type Result<T, E> and Unrecoverable errors have panic! macro that stops execution.

Errors serve two main purposes:

Control flow (i.e. determine what to do next);
Reporting (e.g. investigating, after the fact, what went wrong).

We can also distinguish errors based on their location:

Internal (i.e. a function calling another function within our application);
At the edge (i.e. an API request that we failed to fulfill).

	Internal	At the edge
Control Flow	Types, methods, fields	Status codes
Reporting	Logs/traces	Response body

You can refer more here

The first place where we are likely to get an error is pool.get() . When we are unable to get the pool instance whatever will be the reason like wrong credentials, database instance not running, etc.

Let's try how to handle it.

Using Unwrap to create panic.

Creating Panic is useful during the prototyping phase when we are more focused on logic implementation.

'/src/api_handlers.rs'

pub async fn get_tags(state: web::Data<AppState>) ->impl Responder  {
    // Just not handle the error and let the system to Panic (unrecoverable error)
    let client = state.pool
        .get()
        .await.unwrap();
    let result = db::get_tags_unwrap(&client).await;

    match result {
        Ok(tags) => HttpResponse::Ok().json(tags),
        Err(_) => HttpResponse::InternalServerError().into(),
    }
}

Calling unwrap will create panic and the execution stops. So, we have discovered that handling errors like this are not ideal in the production server.

Let's move to another way

Using the Result

Handling Single Error

The caller of the execution must be aware of whether the program was completed successfully or failed. For this use case, we can use simple ResultSignal enum

'./src/error.rs'

pub enum ResultSignal<Success> {
    Ok(Success),
    Err
}

It will return Ok status on success and an Error on failure. It is helpful now that our user is aware that something mishappen has occurred. It is suitable for a single kind of error but our system consists of different services and they can fail in different ways.

what is the reason? where has failure occurred?
To answer this, we need to handle various errors according to our needs like database errors, filter errors, etc

Handling Multiple Errors

Let's create an enum for different types of errors. For the sake of simplicity, we just considering two cases only Db Error and Not Found Error

'./src/error.rs'

pub enum AppErrorType {
    DbError,
    NotFoundError,
}

Then we need to implement the Debug and Display trait. It is used to print errors using println command.

#[derive(Debug)]
pub enum AppErrorType {
    DbError,
    NotFoundError,
}

impl fmt::Display for AppErrorType {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{:?}", self)
    }
}

#[derive(Debug)] : This macro will enable the Debug trait for the enum AppErrorType.
fn fmt : Display is for user-facing output.
write! and writeln! are two macros that are used to emit the format string to a specified stream. This is used to prevent intermediate allocations of format strings and instead directly write the output.

Let's implement the From trait to convert from one type to another like PoolError to AppErrorType.

impl From<PoolError> for AppErrorType {
    fn from(_error: PoolError) -> AppErrorType {
        AppErrorType::DbError
    }
}
impl From<Error> for AppErrorType {
    fn from(_error: Error) -> AppErrorType {
        AppErrorType::DbError
    }
}

Since, we will be using Result return type in api handlers then we need overwrite the ResponseError Trait.

Actix provide two methods error_response and status_code to handle errors response.

impl ResponseError for AppErrorType {
    fn error_response(&self) -> HttpResponse {
        HttpResponse::build(self.status_code()).finish()
    }
}

Then, we need to change the return type from impl Responder or HttpResponse to Result.

We have used the ? trait already implemented above in error.rs instead to unwrap.

'./src/api_handlers.rs'

pub async fn get_tags(state: web::Data<AppState>) ->Result<HttpResponse,AppErrorType> {
    let client: Client = state.pool.get().await?;
    let result = db::get_tags(&client).await;
    result.map(|tags| HttpResponse::Ok().json(tags))
}

Instead of using unwrap in client.prepare("select * from tag limit 10;").await.unwrap(), we can now use the ? as we have implemented the From trait and update the return type too Result<Vec<Tag>, AppErrorType>

'./src/db.rs'

pub async fn get_tags(client: &Client) -> Result<Vec<Tag>, AppErrorType> {
    let statement = client.prepare("select * from tag limit 10;").await?;
    let tags = client
        .query(&statement, &[])
        .await
        .expect("Error getting tags")
        .iter()
        .map(|row| Tag::from_row_ref(row).unwrap())
        .collect::<Vec<Tag>>();

    Ok(tags)
}

Let's spin our server and hit the endpoint.

The request returned a 500 status code. Instead of just panicking we are getting a status code that will help to debug.

But is only status code really helpful??

Not, not at all, A good error contains the cause of the error, the error status code, and a message for the client user which is human readable

Let's try to improve our error handling

Let's implement the AppError struct which contains our three fields' cause, error type, and message.

'./src/error.rs'

pub struct AppError {
    pub cause: Option<String>,
    pub message: Option<String>,
    pub error_type: AppErrorType,
}

Just like AppErrorType, Let's implement Debug and Display trait

#[derive(Debug)]
pub struct AppError {
    pub cause: Option<String>,
    pub message: Option<String>,
    pub error_type: AppErrorType,
}

impl fmt::Display for AppError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{:?}", self)
    }
}

Once we are done with display and debug trait, let's define ResponseError for AppError

impl ResponseError for AppError {

    fn status_code(&self) -> StatusCode {
        match self.error_type {
            AppErrorType::DbError => (StatusCode::INTERNAL_SERVER_ERROR),
            AppErrorType::NotFoundError => (StatusCode::NOT_FOUND),
        }
    }

    fn error_response(&self) -> HttpResponse {
        HttpResponse::build(self.status_code()).json(AppErrorResponse {
            error: self.message(),
        })
    }
}

In the above code, we have used status_code to match different errors and provide a status code according to it.

As soon the ResponseError is defined, We use From trait for error type conversion

impl From<PoolError> for AppError {
    fn from(error: PoolError) -> AppError {
        AppError {
            message: None,
            cause: Some(error.to_string()),
            error_type: AppErrorType::DbError,
        }
    }
}
impl From<Error> for AppError {
    fn from(error: Error) -> AppError {
        AppError {
            message: None,
            cause: Some(error.to_string()),
            error_type: AppErrorType::DbError,
        }
    }
}

Let's implement a default message for the error types.

impl AppError {
    // we are handling the none. the function name should match the field name
    fn message(&self) -> String {
        match &*self {
            // Error message is found then clone otherwise default message
            AppError {
                cause: _,
                message: Some(message),
                error_type: _,
            } => message.clone(),
            AppError {
                cause: _,
                message: None,
                error_type: AppErrorType::NotFoundError,
            } => "The requested item was not found".to_string(),
            _ => "An unexpected error has occurred".to_string(),
        }
    }
}

We are done with all the necessary changes in error.rs file. Let's start with API handlers and DB handlers

'./src/api_handlers.rs'

pub async fn get_tags(state: web::Data<AppState>) -> Result<HttpResponse, AppError> {
    let client: Client = state.pool.get().await?;
    let result = db::get_tags(&client).await;
    result.map(|tags| HttpResponse::Ok().json(tags))
}

Make sure to enable the logger to get a vivid description. You can use the actix default logger or the slog logger. You can read more about slog here.

async fn configure_pool(pool: Pool, log: Logger) -> Result<Client, AppError> {
    pool.get().await.map_err(|err| {
        let sublog = log.new(o!("cause"=>err.to_string()));
        crit!(sublog, "Error creating client");
        AppError::from(err)
    })
}


pub async fn get_tags(state: web::Data<AppState>) -> Result<HttpResponse, AppError> {
    let sublog = state.log.new(o!("handler" => "get_tags"));
    let client: Client = configure_pool(state.pool.clone(), sublog.clone()).await?;
    let result = db::get_tags(&client).await;
    result.map(|tags| HttpResponse::Ok().json(tags))
}

Let's change the Db handler file.

'./src/db.rs'

pub async fn get_tags(client: &Client) -> Result<Vec<Tag>, AppError> {
    let statement = client.prepare("select * from tag limit 10;").await?;
    let tags = client
        .query(&statement, &[])
        .await
        .expect("Error getting tags")
        .iter()
        .map(|row| Tag::from_row_ref(row).unwrap())
        .collect::<Vec<Tag>>();

    Ok(tags)
}

Let's start our server and hit the API endpoint.

Client side error

Server side error

Wow!! We have now logs and error messages for the client user.

For Awesome logs, we have used the slog logger. If you want to follow my blog then checkout here chaudharypraveen98 - Adding Slog Logger to Actix-web.

Source Code

GitHub Source Code

References

Added Comments for your quick revision and understanding.

Feel free to ask any questions or provide suggestions. I am too learning. So will be glad to get your feedback.

Happy Hacking
Rustaceans!

Adding Slog Logger to Actix Web

Praveen Chaudhary — Thu, 02 Jun 2022 16:42:45 +0000

We will learn how to use slog logger for logging in Actix web.
Actix web is a powerful, pragmatic, and extremely fast web framework for Rust and Slog is an ecosystem of reusable components for structured, extensible, composable logging for Rust. We will be using two crates of slog : slog-async and slog-term with the core Slog Core Package.

Why Slog over default log crate?

extensible
composable
flexible
structured and both human and machine-readable
contextual

Crates used

Actix Web : powerful web framework.
Slog Core Crate : core package to the gateway of logging modules.
Slog Term : Unix terminal drain and formatter for slog-rs.
Slog Term : Asynchronous drain for slog-rs.

Crated Version and Code

Simply paste the code in the cargo.toml file

slog = "2.7.0"
slog-term = "2.9.0"
slog-async = "2.7.0"

Default template for Actix web

It is a default hello world program of Actix web

use actix_web::{ web, App, HttpServer};

async fn index() -> &'static str {
    "Hello world!"
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    println!("Starting the server at 127.0.0.1:8080");

    HttpServer::new(|| {
        App::new()
            .service(web::resource("/index.html").to(|| async { "Hello world!" }))
            .service(web::resource("/").to(index))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

Configure Logger

use slog;
use slog::{Logger,o,Drain,info};
use slog_term;
use slog_async;

fn configure_log()->Logger{
    let decorator = slog_term::TermDecorator::new().build();
    let console_drain = slog_term::FullFormat::new(decorator).build().fuse();

    // It is used for Synchronization
    let console_drain = slog_async::Async::new(console_drain).build().fuse();

    // Root logger
    slog::Logger::root(console_drain,o!("v"=>env!("CARGO_PKG_VERSION")))
}

Let's break the configuration function and understand what is happening behind the scene.

TermDecorator Decorator : IT is used for formatting terminal output implemented using term crate. This decorator will add nice formatting to the logs it’s outputting. Note It does not deal with serialization so is !Sync. Run in a separate thread with slog_async::Async. We will be using the slog async with it. We can other decorator like CompactFormat, PlainRecordDecorator etc according to need.
FullFormat : It is a Drain that will take Decorator as an argument, used for terminal output. Decorator is for formatting and Drain is for outputting.
Synchronization via Async Slog : They are three ways slog to do synchronization out of which PlainSyncDecorator and slog_async are the efficient one depending on the need. Other than the two, the last Synchronization via Mutex is not efficient. You can read more here. We are using the synchronization with slog_async.
Logger::root :Logger is used to execute logging statements. It takes two arguments
1. drain - destination where to forward logging Records for processing.
2. context - list of key-value pairs associated with it.
o! : Macro for building group of key-value pairs used as a content in Logger.

fuse() : It is used for panicking if something went wrong. It is necessary to call fuse as the root logger must take a Drain which is error free.

Passing log instance to the handlers

Add the following line of code in main function

let log = configure_log();

It will configure the logger and ready to use now.

Passing a log instance

HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(log.clone()))
            .service(web::resource("/index.html").to(|| async { "Hello world!" }))
            .service(web::resource("/").to(index))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await

web::Data::new(log.clone()) : It is an application data stored with App::app_data() method available through the HttpRequest::app_data method at runtime.

Accessing the Log instance in function handlers

async fn index(log: web::Data<Logger>) -> &'static str {
    info!(log,
        "Inside Hello World"
    );
    "Hello world!"
}

info! : It is a macro used for the building Info Level Record Or Context(key-value pair) used by Logger to output. They are a bunch of macros you can be used for different level records

log: web::Data : -
Essentials helper functions and types for application registration.

Request Extractors

Data: Application data item
ReqData: Request-local data item
Path: URL path parameters / dynamic segments
Query: URL query parameters
Header: Typed header
Json: JSON payload
Form: URL-encoded payload
Bytes: Raw payload

We are using the Data method to access the application data initialised in server instance in main function.

Complete Code

use actix_web::{web, App, HttpServer};
// IT is used as a logging middleware. We can even use the default logger with actix.
use slog;
use slog::{Logger,o,Drain,info};
use slog_term;
use slog_async;

fn configure_log()->Logger{
    // Formatting the output https://docs.rs/slog-term/2.9.0/slog_term/index.html#
    let decorator = slog_term::TermDecorator::new().build();

    // Drain for outputting https://docs.rs/slog-term/2.9.0/slog_term/index.html#structs
    // fuse is used for panicking if something went wrong. It is necessary to call fuse as the root logger must take a Drain which is error free.
    let console_drain = slog_term::FullFormat::new(decorator).build().fuse();

    // It is used for Synchronization https://docs.rs/slog-term/2.9.0/slog_term/index.html#structs
    let console_drain = slog_async::Async::new(console_drain).build().fuse();
    slog::Logger::root(console_drain,o!("v"=>env!("CARGO_PKG_VERSION")))
}
async fn index(log: web::Data<Logger>) -> &'static str {
    info!(log,
        "Inside Hello World"
    );
    "Hello world!"
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let log = configure_log();

    info!(log,
        "Starting the server at http://127.0.0.1:8080/"
    );

    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(log.clone()))
            .service(web::resource("/index.html").to(|| async { "Hello world!" }))
            .service(web::resource("/").to(index))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

Logging Output

Source Code

GitHub Source Code

Added Comments for your quick revision and understanding.

Feel free to ask any questions or provide suggestions. I am too learning. So will be glad to get your feedback. Happy Hacking! Rustaceans!

Stackoverflow scraping with rust

Praveen Chaudhary — Sat, 26 Jun 2021 07:12:00 +0000

It will extract the title, question link, answers count, view count, and votes from StackOverflow depending on the tag parameter and count. This scraper is inspired by Kadekillary Scarper with updated libraries and some more features added.

Libraries Used

Reqwest >An ergonomic, batteries-included HTTP Client for Rust.
Select >A Rust library to extract useful data from HTML documents, suitable for web scraping
Clap >A simple-to-use, efficient, and full-featured library for parsing command line arguments and subcommands.

Features

Simple and Fast
Async get request
Cli mode

How to run

Build the executable by cargo build
Run by ./target/debug/stackoverflow-scraping-with-rust -t <tag> -c <count> < tag > is the topic from which you want to scrape < count > is the number of posts/threads to be scraped. Note the maximum limit is 16 Like this ./target/debug/stackoverflow-scraping-with-rust -t java -c 1

What We are going to do?

Getting argument from the command line using Clap library
Make a request using the Reqwest library
Scraping using the Selectrs library

Libraries Used

Reqwest >An ergonomic, batteries-included HTTP Client for Rust.
Select >A Rust library to extract useful data from HTML documents, suitable for web scraping
Clap >A simple-to-use, efficient, and full-featured library for parsing command line arguments and subcommands.

Installing Libraries
Simple add the following libraries in Cargo.toml

[dependencies]
reqwest = { version = "0.10", features = ["json"] }
tokio = { version = "0.2", features = ["full"] }
select = "0.6.0-alpha.1"
clap = "2.33.3"
rand = "0.8.4"

Before moving ahead, We must be aware of CSS selectors

What are selectors/locators?

A CSS Selector is a combination of an element selector and a value that identifies the web element within a web page.
The choice of locator depends largely on your Application Under Test

Id
An element’s id in XPATH is defined using: “[@id='example']” and in CSS using: “#” - ID's must be unique within the DOM.
Examples:

XPath: //div[@id='example']
CSS: #example

Element Type
The previous example showed //div in the XPath. That is the element type, which could be input for a text box or button, img for an image, or "a" for a link.

Xpath: //input or
Css: =input

Direct Child
HTML pages are structured like XML, with children nested inside of parents. If you can locate, for example, the first link within a div, you can construct a string to reach it. A direct child in XPATH is defined by the use of a “/“, while on CSS, it’s defined using “>”.
Examples:

XPath: //div/a
CSS: div > a

Child or Sub-Child
Writing nested divs can get tiring - and result in code that is brittle. Sometimes you expect the code to change, or want to skip layers. If an element could be inside another or one of its children, it’s defined in XPATH using “//” and in CSS just by whitespace.
Examples:

XPath: //div//a
CSS: div a

Class
For classes, things are pretty similar in XPATH: “[@class='example']” while in CSS it’s just “.”
Examples:

XPath: //div[@class='example']
CSS: .example

Step 1 -> Getting argument from the command line using Clap library

We are using the Clap library to get the argument from the command line.
There are three cases: -

Only tag is provided => We will get only the posts from the input tag with default no of post i.e 16
Only count is supplies => We will get only input number of post from any random topic using the Rand library
Both count and tag => We will get input number of post of input tag

First, we initialize the command line app names StackOverflow Scraper. Then mention all the three cases with their short and long name.

fn main() {
    let matches = App::new("StackOverflow Scraper")
        .version("1.0")
        .author("Praveen Chaudhary &lt;chaudharypraveen98@gmail.com&gt;")
        .about("It will scrape questions from StackOverflow depending on the tag.")
        .arg(
            Arg::with_name("tag")
                .short("t")
                .long("tag")
                .takes_value(true)
                .help("takes the tag and scrape according to this"),
        )
        .arg(
            Arg::with_name("count")
                .short("c")
                .long("count")
                .takes_value(true)
                .help("gives n count of posts"),
        )
        .get_matches();
        ....
        ....

Once we have mentioned all the cases. Now we need to extract the argument value using the match which helps us to find a particular pattern

    fn main() {
        .....
        .....

        if matches.is_present("tag") && matches.is_present("count") {
            let url = format!(
                "https://stackoverflow.com/questions/tagged/{}?tab=Votes",
                matches.value_of("tag").unwrap()
            );
            let count: i32 = matches.value_of("count").unwrap().parse::<i32>().unwrap();
            stackoverflow_post(&url, count as usize);
        } else if matches.is_present("tag") {
            let url = format!(
                "https://stackoverflow.com/questions/tagged/{}?tab=Votes",
                matches.value_of("tag").unwrap()
            );
            stackoverflow_post(&url, 16);
        } else if matches.is_present("count") {
            let url = get_random_url();
            let count: i32 = matches.value_of("count").unwrap().parse::<i32>().unwrap();
            stackoverflow_post(&url, count as usize);
        } else {        
            let url = get_random_url();        
            stackoverflow_post(&url, 16);
        }
    }

In the above code, we have used the stackoverflow_post function. We will learn about this in Step 3

Step 2 -> Make a request using the Reqwest library

We will use the reqwest library to make a get request to the StackOverflow website customized with the input tag

#[tokio::main]
async fn hacker_news(url: &str, count: usize) -> Result<(), reqwest::Error> {
    let resp = reqwest::get(url).await?;
    ....

Step 3 -> Scraping using the Selectrs library

We will use the CSS selectors to get the question post from the StackOverflow

#[tokio::main]
async fn hacker_news(url: &str, count: usize) -> Result<(), reqwest::Error> {
    ..... 
    ..... 

    let document = Document::from(&*resp.text().await?);

    for node in document.select(Class("s-post-summary")).take(count) {
        let question = node
            .select(Class("s-post-summary--content-excerpt"))
            .next()
            .unwrap()
            .text();
        let title_element = node
            .select(Class("s-post-summary--content-title").child(Name("a")))
            .next()
            .unwrap();
        let title = title_element.text();
        let question_link = title_element.attr("href").unwrap();
        let stats = node
            .select(Class("s-post-summary--stats-item-number"))
            .map(|stat| stat.text())
            .collect::<Vec<_>>();
        let votes = &stats[0];
        let answer = &stats[1];
        let views = &stats[2];
        let tags = node
            .select(Class("post-tag"))
            .map(|tag| tag.text())
            .collect::<Vec<_>>();
        println!("Question       => {}", question);
        println!(
            "Question-link  => https://stackoverflow.com{}",
            question_link
        );
        println!("Question-title => {}", title);
        println!("Votes          => {}", votes);
        println!("Views          => {}", views);
        println!("Tags           => {}", tags.join(" ,"));
        println!("Answers        => {}", answer);
        println!("-------------------------------------------------------------\n");
    }
    Ok(())
}

Complete Code

extern crate clap;
extern crate reqwest;
extern crate select;
extern crate tokio;

use clap::{App, Arg};
use rand::seq::SliceRandom;
use select::document::Document;
use select::predicate::{Attr, Class, Name, Or, Predicate};

fn main() {
    let matches = App::new("StackOverflow Scraper")
        .version("1.0")
        .author("Praveen Chaudhary &lt;chaudharypraveen98@gmail.com&gt;")
        .about("It will scrape questions from stackoverflow depending on the tag.")
        .arg(
            Arg::with_name("tag")
                .short("t")
                .long("tag")
                .takes_value(true)
                .help("takes the tag and scrape according to this"),
        )
        .arg(
            Arg::with_name("count")
                .short("c")
                .long("count")
                .takes_value(true)
                .help("gives n count of posts"),
        )
        .get_matches();

    if matches.is_present("tag") && matches.is_present("count") {
        let url = format!(
            "https://stackoverflow.com/questions/tagged/{}?tab=Votes",
            matches.value_of("tag").unwrap()
        );
        let count: i32 = matches.value_of("count").unwrap().parse::<i32>().unwrap();
        hacker_news(&url, count as usize);
    } else if matches.is_present("tag") {
        let url = format!(
            "https://stackoverflow.com/questions/tagged/{}?tab=Votes",
            matches.value_of("tag").unwrap()
        );
        hacker_news(&url, 16);
    } else if matches.is_present("count") {
        let url = get_random_url();
        let count: i32 = matches.value_of("count").unwrap().parse::<i32>().unwrap();
        hacker_news(&url, count as usize);
    } else {        
        let url = get_random_url();        
        hacker_news(&url, 16);
    }
}

#[tokio::main]
async fn hacker_news(url: &str, count: usize) -> Result&lt;(), reqwest::Error&gt; {
    let resp = reqwest::get(url).await?;
    // println!("body = {:?}", resp.text().await?);
    // assert!(resp.status().is_success());
    let document = Document::from(&*resp.text().await?);

    for node in document.select(Class("s-post-summary")).take(count) {
        let question = node
            .select(Class("s-post-summary--content-excerpt"))
            .next()
            .unwrap()
            .text();
        let title_element = node
            .select(Class("s-post-summary--content-title").child(Name("a")))
            .next()
            .unwrap();
        let title = title_element.text();
        let question_link = title_element.attr("href").unwrap();
        let stats = node
            .select(Class("s-post-summary--stats-item-number"))
            .map(|stat| stat.text())
            .collect::<Vec<_>>();
        let votes = &stats[0];
        let answer = &stats[1];
        let views = &stats[2];
        let tags = node
            .select(Class("post-tag"))
            .map(|tag| tag.text())
            .collect::<Vec<_>>();
        println!("Question       => {}", question);
        println!(
            "Question-link  => https://stackoverflow.com{}",
            question_link
        );
        println!("Question-title => {}", title);
        println!("Votes          => {}", votes);
        println!("Views          => {}", views);
        println!("Tags           => {}", tags.join(" ,"));
        println!("Answers        => {}", answer);
        println!("-------------------------------------------------------------\n");
    }
    Ok(())
}

// Getting random tag
fn get_random_url() -> String {
    let default_tags = vec!["python", "rust", "c#", "android", "html", "javascript"];
    let random_tag = default_tags.choose(&mut rand::thread_rng()).unwrap();
    let url = format!(
        "https://stackoverflow.com/questions/tagged/{}?tab=Votes",
        random_tag
    );
    url.to_string()
}

How to run our scraper?

Build the executable by cargo build
Run by ./target/debug/stackoverflow-scraping-with-rust -t <tag> -c <count> < tag > is the topic from which you want to scrape < count > is the number of posts/threads to be scraped. Note the maximum limit is 16. Like this ./target/debug/stackoverflow-scraping-with-rust -t java -c 1

Deployment

You can deploy on Heroku with the help of Circle CI
You can read more about on CircleCI Blog

Web Preview / Output

Google Drive

Github Link = > https://github.com/chaudharypraveen98/stackoverflow-scraping-with-rust

Farsite

Praveen Chaudhary — Mon, 31 May 2021 13:37:40 +0000

Farsite is a blockchain powered sandbox massively multiplayer online (MMO) real-time strategy (RTS) game. It has the elements of adventure in a science-fiction setting based on a post-apocalyptic story.

The gaming universe of Farsite is formed by Constellations containing stars, each with unique set of planets. "Homestead" is the first Constellation where all players start the game. By using some Credits, players can travel between Stars on spaceships following interstellar paths in faster than light modes, performing some social media tasks can increase your scores to get a powerful starter ship. If this alone has made you excited or has sparked your interest to the game, you may use this link to register, https://farsite.club/u/warliezdiaz.

Technology Used

Farsite is built using a mix of distributed ledger technology. The game also features Collateralized Digital Assets (cNFT) which is a proposed standard for ERC-20 collateral staking on non-fungible tokens (NFTs) Most of the in-game operations and trading will take place on Layer 2. Farsite will be open source and open to anyone who wants to play and contribute to the development of the game. It will be available to anyone with a web address and can be played by anyone with an internet connection.

Environment

Within the Farsite universe, Constellations play a central role. Only 4 to 5 Planets will be explored on the start of the game. Exploring a Planet will open its Sectors for acquisition and reveal resources spots and deposits. Sectors are a part of a Planet representing a construction site for a Base. These contain valuable resources that can be extracted using special buildings on the surface.

The ownership of a Sector can be traded or transferred to another player with all the contained Resources and constructed Bases. The construction of Stations can only be done by a player or a Corporation and requires a lot of resources and credits. Without a Station in a Star, players will need to fly to another Star to perform these transactions and require credits. All planets in the 'Homestead' Constellation will be available for colonization. There are 8 to 10 known Planets, unknown Planets and dead Planets.