DEV Community: Dave

How to call C code from Rust

Dave — Thu, 28 Oct 2021 14:05:47 +0000

Calling C code from Rust falls under FFI (Foreign Function Interface) and is unsafe by nature, as it's crossing Rust boundary and all the checks that comes with it.

The syntax itself is pretty simple, so create a project and:

// src/main.rs
extern "C" {
    fn print_num(num: i32);
}

fn main() {
    println!("[rust] start");
    unsafe { print_num(50) }
}

We are defining an external function which uses C ABI convention - which has got to do with the way stack and registers are arranged, when making a call to another function.

And for our C library:

// src/num.c
#include <stdio.h>

void print_num(int num) {
    printf("[  c ] num is %d", num);
}

But if we cargo run here, we'd get main.rs:8: undefined reference to print_num, meaning compiler is not finding our C function so we need to link the library:

$ # in root of project 
$ cc -c src/num.c
$ cc -shared num.o -o libnum.so

First cc would generate lib.o but rustc needs a shared obejct, which is generated from that and starts its name with lib. Now we can link it to our Rust code (omit the lib at start):

$ rustc -l num -L . src/main.rs

Which looks for (lib)num.so in current . path. This should compile and give you the main binary, which can't be executed:

$ ./main
./main: error while loading shared libraries: libnum.so: cannot open shared object file: No such file or directory

$ ldd main
    …
    libnum.so => not found

Because you've used a dynamic library, and ldd can't find it, you can't run your binary. You can either install libnum.so in appropriate path or use a static library:

$ ar rcs libnum.a num.o
$ rustc -l static=num -L. src/main.rs  
$ ./main
[rust] start
[  c ] num is 50

Perfect! Let's simplify our workflow, create build.rs in root of project:

fn main() {
    println!("cargo:rustc-link-search=.");
    println!("cargo:rustc-link-lib=static=num");
}

And now we can use cargo run and cargo build the usual way. We can also declare the name of library where it's being used so instead of println!("cargo:rustc-link-lib=static=num"); in build.rs:

// src/main.rs
#[link(name = "num", kind = "static")]
extern "C" {
    fn print_num(num: i32);
}

which links libnum.a and if we remove the kind part defaults to dynamic versin libnum.so.

Finally if you're not interested in handling the compilation of library files yourself, there's cc crate which takes care of that for you:

// Cargo.toml
…
[build-dependencies]
cc = "1.0"

And your build.rs:

// build.rs
fn main() {
    cc::Build::new()
        .file("src/num.c")
        .compile("anything");
}

And that's all!

Adding our own custom statement to Rust language

Dave — Fri, 30 Jul 2021 14:16:50 +0000

I wanted to learn more about Rust internals and decided why not add my own custom statement and see how it goes.

Without knowing much about either the code base or compilers in general, I was able to add my statement. Buckle up, this gonna be interesting!

Goal

Goal is to add the unless cond { block } statement, which only executes the block when cond is not met:

fn num_is_odd(n: u32) -> bool {
    return n % 2 == 1
}

fn main() {
    for num in 1..10 {
        unless num_is_odd(num) {
            print!("{} ", num);
        }
    }
}

// expected output:
// 2 4 6 8 ⏎

Setup

You need to clone the rust-lang/rust repository and setup everything it needs:

$ git clone https://github.com/rust-lang/rust.git
$ cd rust
$ ./x.py build -i library/std # this builds the compiler

This should take about 30 minutes, so get comfortable!

Start investigating

Once I had everything setup, meaning I could compile my own compiler, I started looking for keyword Stmt in the project.

The interesting file containing Stmt is at compiler/rustc_ast/src/ast.rs which is the module responsible for building the abstract syntax tree (AST).

After parsing the source code files, Rust creates the AST because it's easier to work with at further steps.

The file has an interesting enum containing expressions:

  pub enum ExprKind {
      .
      .
      /// A binary operation (e.g., `a + b`, `a * b`).
      Binary(BinOp, P<Expr>, P<Expr>),
      /// A unary operation (e.g., `!x`, `*x`).
      Unary(UnOp, P<Expr>),
      .
      .
      /// An `if` block, with an optional `else` block.
      ///
      /// `if expr { block } else { expr }`
      If(P<Expr>, P<Block>, Option<P<Expr>>),
      .
      .

Creating `Unless`

Our unless has the same logic as an if statement so we're going to model after that. Right under If we're going to place:

      /// An `unless` block, with an optional `else` block.
      ///
      /// `unless expr { block } else { expr }`
      Unless(P<Expr>, P<Block>, Option<P<Expr>>),

We can do without the optional expression but we chose to keep it. Save the file and try to compile, don't worry it will error out and won't take that much time.

$ ./x.py build -i library/std

error[E0004]: non-exhaustive patterns: `Unless(_, _, _)` not covered
    --> compiler/rustc_ast/src/ast.rs:1192:15
     |
1192 |           match self.kind {
     |                 ^^^^^^^^^ pattern `Unless(_, _, _)` not covered
...
1265 | / pub enum ExprKind {
1266 | |     /// A `box x` expression.
1267 | |     Box(P<Expr>),
1268 | |     /// An array (`[a, b, c, d]`)
...    |
1313 | |     Unless(P<Expr>, P<Block>),
     | |     ------ not covered
...    |
1411 | |     Err,
1412 | | }
     | |_- `ast::ExprKind` defined here
     |
     = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
     = note: the matched value is of type `ast::ExprKind`

error[E0004]: non-exhaustive patterns: `&mut Unless(_, _, _)` not covered
    --> compiler/rustc_ast/src/mut_visit.rs:1204:11
     |
1204 |       match kind {
     |             ^^^^ pattern `&mut Unless(_, _, _)` not covered
     |
    ::: compiler/rustc_ast/src/ast.rs:1265:1
     |
1265 | / pub enum ExprKind {
1266 | |     /// A `box x` expression.
1267 | |     Box(P<Expr>),
1268 | |     /// An array (`[a, b, c, d]`)
...    |
1313 | |     Unless(P<Expr>, P<Block>),
     | |     ------ not covered
...    |
1411 | |     Err,
1412 | | }
     | |_- `ast::ExprKind` defined here
     |
     = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
     = note: the matched value is of type `&mut ast::ExprKind`

error[E0004]: non-exhaustive patterns: `Unless(_, _, _)` not covered
    --> compiler/rustc_ast/src/visit.rs:738:11
     |
738  |       match expression.kind {
     |             ^^^^^^^^^^^^^^^ pattern `Unless(_, _, _)` not covered
     |
    ::: compiler/rustc_ast/src/ast.rs:1265:1
     |
1265 | / pub enum ExprKind {
1266 | |     /// A `box x` expression.
1267 | |     Box(P<Expr>),
1268 | |     /// An array (`[a, b, c, d]`)
...    |
1313 | |     Unless(P<Expr>, P<Block>),
     | |     ------ not covered
...    |
1411 | |     Err,
1412 | | }
     | |_- `ast::ExprKind` defined here
     |
     = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
     = note: the matched value is of type `ast::ExprKind`

error: aborting due to 3 previous errors

Since we added a new Enum, we need to fill in our type in places matching against that, those 3 files mentioned in errors:

In compiler/rustc_ast/src/ast.rs:

    pub fn precedence(&self) -> ExprPrecedence {
          match self.kind {
              .
              .
              ExprKind::If(..) => ExprPrecedence::If,
              ExprKind::Unless(..) => ExprPrecedence::If,
              .
              .

In compiler/rustc_ast/src/visit.rs:

pub fn walk_expr<'a, V: Visitor<'a>>(visitor: &mut V, expression: &'a Expr) {
    walk_list!(visitor, visit_attribute, expression.attrs.iter());

    match expression.kind {
        .
        .
        ExprKind::If(ref head_expression, ref if_block, ref optional_else) => {
            visitor.visit_expr(head_expression);
            visitor.visit_block(if_block);
            walk_list!(visitor, visit_expr, optional_else);
        }
        ExprKind::Unless(ref head_expression, ref unless_block, ref optional_unless) => {
            visitor.visit_expr(head_expression);
            visitor.visit_block(unless_block);
            walk_list!(visitor, visit_expr, optional_unless);
        }
        .
        .

In compiler/rustc_ast/src/mut_visit.rs:

pub fn noop_visit_expr<T: MutVisitor>(
    Expr { kind, id, span, attrs, tokens }: &mut Expr,
    vis: &mut T,
) {
    match kind {
        .
        .
        ExprKind::If(cond, tr, fl) => {
            vis.visit_expr(cond);
            vis.visit_block(tr);
            visit_opt(fl, |fl| vis.visit_expr(fl));
        }
        ExprKind::Unless(cond, tr, fl) => {
            vis.visit_expr(cond);
            vis.visit_block(tr);
            visit_opt(fl, |fl| vis.visit_expr(fl));
        }
        .
        .

Initially I'd negated the head_expression and cond, but I'd get:

error[E0600]: cannot apply unary operator `!` to type `&mut P<ast::Expr>`
    --> compiler/rustc_ast/src/mut_visit.rs:1250:28
     |
1250 |             vis.visit_expr(!cond);
     |                            ^^^^^ cannot apply unary operator `!`
     |
     = note: an implementation of `std::ops::Not` might be missing for `&mut P<ast::Expr>`

error[E0600]: cannot apply unary operator `!` to type `&P<ast::Expr>`
   --> compiler/rustc_ast/src/visit.rs:792:32
    |
792 |             visitor.visit_expr(!head_expression);
    |                                ^^^^^^^^^^^^^^^^ cannot apply unary operator `!`
    |
    = note: an implementation of `std::ops::Not` might be missing for `&P<ast::Expr>`

Which means, we have to use std::ops::Not to achieve this. For now, we proceed without it. All good, let's build the compiler now:

$ ./x.py build -i library/std
.
.
error[E0004]: non-exhaustive patterns: `Unless(_, _, _)` not covered
    --> compiler/rustc_ast_pretty/src/pprust/state.rs:1871:15
     |
1871 |         match expr.kind {
     |               ^^^^^^^^^ pattern `Unless(_, _, _)` not covered
     |
    ::: /.../rust-lang/rust/compiler/rustc_ast/src/ast.rs:1314:5
     |
1314 |     Unless(P<Expr>, P<Block>, Option<P<Expr>>),
     |     ------ not covered
     |
     = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
     = note: the matched value is of type `ExprKind`

error: aborting due to previous error

We missed that, so in rustc_ast_pretty/src/pprust/state.rs:

    fn print_expr_outer_attr_style(&mut self, expr: &ast::Expr, is_inline: bool) {
        .
        .
        match expr.kind {
            .
            .
            ast::ExprKind::If(ref test, ref blk, ref elseopt) => {
                self.print_if(test, blk, elseopt.as_deref())
            }
            ast::ExprKind::Unless(ref test, ref blk, ref elseopt) => {
                self.print_if(test, blk, elseopt.as_deref())
            }
            .
            .

Try again:

$ ./x.py build -i library/std
.
.
error[E0425]: cannot find value `Unless` in module `kw`
    --> compiler/rustc_parse/src/parser/expr.rs:1119:40
     |
1119 |         } else if self.eat_keyword(kw::Unless) {
     |                                        ^^^^^^ not found in `kw`
     |
help: consider importing one of these items
     |
1    | use crate::parser::ExprKind::Unless;
     |
1    | use rustc_ast::ExprKind::Unless;
     |

   Compiling rustc_middle v0.0.0 (/.../rust-lang/rust/compiler/rustc_middle)
error[E0004]: non-exhaustive patterns: `Unless(_, _, _)` not covered
    --> compiler/rustc_ast_lowering/src/expr.rs:29:30
     |
29   |             let kind = match e.kind {
     |                              ^^^^^^ pattern `Unless(_, _, _)` not covered
     |
    ::: /.../rust-lang/rust/compiler/rustc_ast/src/ast.rs:1314:5
     |
1314 |     Unless(P<Expr>, P<Block>, Option<P<Expr>>),
     |     ------ not covered
     |
     = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
     = note: the matched value is of type `rustc_ast::ExprKind`

error: aborting due to previous error

kw::Unless doesn't exist, if we follow kw definition, we reach rustc_span/src/symbol.rs and there we can add our keyword:

        If:                 "if",
        Unless:             "unless",
        Impl:               "impl",

In rustc_parse/src/parser/expr.rs:

    fn parse_bottom_expr(&mut self) -> PResult<'a, P<Expr>> {
        .
        .
        } else if self.eat_keyword(kw::If) {
            self.parse_if_expr(attrs)
        } else if self.eat_keyword(kw::Unless) {
            self.parse_if_expr(attrs)
        }
        .
        .

and in compiler/rustc_ast_lowering/src/expr.rs#L115:

                ExprKind::Unless(ref cond, ref then, ref else_opt) => match cond.kind {
                    ExprKind::Let(ref pat, ref scrutinee) => {
                        self.lower_expr_if_let(e.span, pat, scrutinee, then, else_opt.as_deref())
                    }
                    ExprKind::Paren(ref paren) => match paren.peel_parens().kind {
                        ExprKind::Let(ref pat, ref scrutinee) => {
                            // A user has written `if (let Some(x) = foo) {`, we want to avoid
                            // confusing them with mentions of nightly features.
                            // If this logic is changed, you will also likely need to touch
                            // `unused::UnusedParens::check_expr`.
                            self.if_let_expr_with_parens(cond, &paren.peel_parens());
                            self.lower_expr_if_let(
                                e.span,
                                pat,
                                scrutinee,
                                then,
                                else_opt.as_deref(),
                            )
                        }
                        _ => self.lower_expr_if(cond, then, else_opt.as_deref()),
                    },
                    _ => self.lower_expr_if(cond, then, else_opt.as_deref()),
                },

When do I stop copying?

So far, we only copied whatever If had for our Unless type. We have everything we want now to compile the compiler, but wait!

Before you start another compile and spend a lot of time, you can quickly check the status with:

$ ./x.py check

This is a faster command that is only going to check whether your code can compile. Don't worry if it errors at src/tools/clippy/clippy_utils/src/sugg.rs, you're good to go if you reach that far!

Implementing `Unless` logic

Right now, we have unless as a proxy for if but that's not what we want. We want our block to be run only when the condition is not met. We already tried with ! above but it told us to use std::ops::Not.

I tried that too, but that's not exactly the type we're looking for. Thank you, compiler!

Instead we need to modify our parser, because by the time code reaches ast module, expressions are set and we only have Visit logic.

So in rustc_parse/src/parser/expr.rs change:

} else if self.eat_keyword(kw::Unless) {
            self.parse_unless_expr(attrs)
}

and then:

    /// Parses an `unless` expression (`unless` token already eaten).
    fn parse_unless_expr(&mut self, attrs: AttrVec) -> PResult<'a, P<Expr>> {
        let lo = self.prev_token.span;
        let cond = self.parse_cond_expr()?;

        // Verify that the parsed `if` condition makes sense as a condition. If it is a block, then
        // verify that the last statement is either an implicit return (no `;`) or an explicit
        // return. This won't catch blocks with an explicit `return`, but that would be caught by
        // the dead code lint.
        let thn = if self.eat_keyword(kw::Else) || !cond.returns() {
            self.error_missing_if_cond(lo, cond.span)
        } else {
            let attrs = self.parse_outer_attributes()?.take_for_recovery(); // For recovery.
            let not_block = self.token != token::OpenDelim(token::Brace);
            let block = self.parse_block().map_err(|mut err| {
                if not_block {
                    err.span_label(lo, "this `if` expression has a condition, but no block");
                    if let ExprKind::Binary(_, _, ref right) = cond.kind {
                        if let ExprKind::Block(_, _) = right.kind {
                            err.help("maybe you forgot the right operand of the condition?");
                        }
                    }
                }
                err
            })?;
            self.error_on_if_block_attrs(lo, false, block.span, &attrs);
            block
        };
        let els = if self.eat_keyword(kw::Else) { Some(self.parse_else_expr()?) } else { None };
        let neg_cond = self.mk_expr(
            lo.to(self.prev_token.span),
            self.mk_unary(UnOp::Not, cond),
            AttrVec::new()
        );
        Ok(self.mk_expr(lo.to(neg_cond.span), ExprKind::If(neg_cond, thn, els), attrs))
    }

Our parser pretty much looks like If, except we negate the condition using Unary type, store that in neg_cond and then create a normal If block using our new condition.

Not only is this not a bad practice, but actually encouraged as it narrows down the building blocks in later stages and improvements on one of them would trickle. In our case a new compiler optimization for If would translate to a better Unless.

Compiling our goal

$ export RUST_SRC_PATH=/home/path-to-source/rust-lang/rust
$ export RUSTC_DEV=$RUST_SRC_PATH/build/x86_64-unknown-linux-gnu/stage1/bin/rustc
$ $RUSTC_DEV -vV # notice the dollar sign
rustc 1.56.0-dev # notice -dev here which tells us it's compiled from source
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.56.0-dev
LLVM version: 12.0.1

$ cat unless.rs
fn num_is_odd(n: u32) -> bool {
    return n % 2 == 1
}

fn main() {
    for num in 1..10 {
        unless num_is_odd(num) {
            print!("{} ", num);
        }
    }
}

$ $RUSTC_DEV unless.rs
$ ./unless
2 4 6 8

Github

You can see the commit here and if you want to try out the whole thing, checkout add-unless-statement branch.

II. Implementing ICMP in Rust

Dave — Sun, 18 Jul 2021 15:00:44 +0000

Make sure to checkout previous part where we talk about IP and ICMP layouts if you haven't already: I. Implementing ICMP in Rust

3. (Cont'd) Implementation in Rust

First thing we need to do is to create a TUN/TAP device that allows us to receive/send raw frames directly. It's a kernel-feature which enables us to have software-defined interfaces.

fn main() {
    let mut nic = tun_tap::Iface::without_packet_info("tun0", tun_tap::Mode::Tun).unwrap();
    let mut buf = [0u8; 1500];
    loop {
        let nbytes = nic.recv(&mut buf[..]).unwrap();

        match etherparse::Ipv4HeaderSlice::from_slice(&buf[..nbytes]) {
            Ok(iph) => {
                let src = iph.source_addr();
                let dst = iph.destination_addr();
                let proto = iph.protocol();

                if proto != 1 {
                    continue;
                }

                let data_buf = &buf[iph.slice().len()..nbytes];

                if let Some(mut c) = Connection::start(
                    iph,
                    data_buf,
                ).unwrap() {
                    println!("connection c started!");
                    c.respond(&mut nic).unwrap();
                    println!("responded to {} packet from {} ", proto, src);
                }
            }
            Err(e) => {
                eprintln!("ignoring weird packet {:?}", e);
            }
        }
    }
}

In our main, we create our device and a buf and keep reading from it in a loop. Whenever we find an IPv4 header, etherparse already takes care of the parsing for us and gives us the iph object.

For example, if you've read the previous installment you should know the proto == 1 for ICMP echo and echo reply packets, thus we skip everything else:

                if proto != 1 {
                    continue;
                }

Finally we read the actual data, which is everything except header bytes and create a connection:

                let data_buf = &buf[iph.slice().len()..nbytes];

                if let Some(mut c) = Connection::start(
                    iph,
                    data_buf,
                ).unwrap() {
                    println!("connection c started!");
                    c.respond(&mut nic).unwrap();
                    println!("responded to {} packet from {} ", proto, src);
                }

Since we're not dealing with a stateful protocol like TCP, we can get away with not using Connection but for sake of conforming to John's TCP implementation, I use one as well.

Still you need Connection to distinguish multiple streams and detect out of order sequence numbers.

pub struct Connection {
    ip: etherparse::Ipv4Header,
    icmp_id: u16,
    seq_no: u16,
}

impl Connection {
    pub fn start(iph: etherparse::Ipv4HeaderSlice, data: &[u8]) -> std::io::Result<Option<Self>> {
        let mut c = Connection {
            ip: etherparse::Ipv4Header::new(
                0,
                64,
                etherparse::IpTrafficClass::Icmp,
                [
                    iph.destination()[0],
                    iph.destination()[1],
                    iph.destination()[2],
                    iph.destination()[3],
                ],
                [
                    iph.source()[0],
                    iph.source()[1],
                    iph.source()[2],
                    iph.source()[3],
                ],
            ),
            icmp_id: u16::from_be_bytes(data[4..6].try_into().unwrap()),
            seq_no: u16::from_be_bytes(data[6..8].try_into().unwrap()),
        };

        Ok(Some(c))
    }

    pub fn respond(&mut self, nic: &mut tun_tap::Iface,) -> std::io::Result<usize> {
        let mut buf = [0u8; 1500];

        self.ip.set_payload_len(84-20 as usize);

        use std::io::Write;
        let mut unwritten = &mut buf[..];
        self.ip.write(&mut unwritten);

        let mut icmp_reply = [0u8; 64];
        // type
        icmp_reply[0] = ICMP_ECHO_REPLY;
        // code, always 0
        icmp_reply[1] = 0; 

        // checksum = 2 & 3, empty for now
        icmp_reply[2] = 0x00; 
        icmp_reply[3] = 0x00;

        // id = 4 & 5
        icmp_reply[4] = ((self.icmp_id >> 8) & 0xff) as u8; 
        icmp_reply[5] = (self.icmp_id & 0xff) as u8;

        // seq_no = 6 & 7
        icmp_reply[6] = ((self.seq_no >> 8) & 0xff) as u8; 
        icmp_reply[7] = (self.seq_no & 0xff) as u8;

        unwritten.write(&icmp_reply);

        let unwritten = unwritten.len();
        nic.send(&buf[..buf.len() - unwritten])?;

        Ok(0)
    }
}

We create our Connection type and implement two methods: start and respond. All start does is create an IP header with the source and destination swapped.

Then in respond, we start writing the actual ICMP packet. We start by setting IP's size to 84, which if you remember from last part was the Total Length of our IP packet. 20 bytes is the IP header so we deduce that and prepare a 64 bytes buffer to hold our ICMP.

All we do here is change the ICMP type to 0 which is echo reply, leave checksum as 0, and copy in the identifier and sequence number.

Remember ICMP layout from previous part?

08          Type             0th byte       = Echo message
00          Code             1st byte                    
492b        Checksum         2-3rd byte
5514        Identifier       4-5th byte     = id 21780 (in raw capture)
0001        Sequence Number  6-7th byte     = seq 1

4. Testing the implementation

We're ready to run our program.

Once it creates the device, we can ping it with any IP address associated with it and get:

# ping 192.168.0.3

PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=1626615676072 ms
wrong data byte #16 should be 0x10 but was 0x0
#16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#48 0 0 0 0 0 0 0 0
64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=1626615677081 ms
wrong data byte #16 should be 0x10 but was 0x0
#16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#48 0 0 0 0 0 0 0 0
64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=1626615678105 ms
wrong data byte #16 should be 0x10 but was 0x0
#16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#48 0 0 0 0 0 0 0 0

Turns out it's expecting our byte #16 to be 0x10, our #17 to be 0x11 and so on. So we add:

        for i in 16..64-8 {
            icmp_reply[i+8] = (0x10 + (i - 16)) as u8;
        }

to our respond method and try ping again:

# ping 192.168.0.3

PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=1626615769165 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=1626615770169 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=1626615771193 ms
64 bytes from 192.168.0.3: icmp_seq=4 ttl=64 time=1626615772217 ms
64 bytes from 192.168.0.3: icmp_seq=5 ttl=64 time=1626615773241 ms

We get a working ping back and forth, only all the times are all wrong.

After a lot of head scratches, and double and triple checking the wiki page of IP, ICMP, etc, I finally found the solution on RFC-792 page:

The data received in the echo message must be returned in the echo reply message.

So instead of blindly adding 0x10, 0x11 to my packet, I copy the original values:

icmp_reply[8..64].clone_from_slice(&self.data[8..64]);

and try once again:

# ping 192.168.0.3

PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=0.162 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=0.214 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=0.174 ms
64 bytes from 192.168.0.3: icmp_seq=4 ttl=64 time=0.226 ms

Everything seems to be working, except I don't have the correct checksum. Or any checksum for that mattter.

It's not showing up in ping but if you use Wireshark:

Following RFCs and Wikis only made me more confused when trying to write the checksum function. I had better luck writing one in Python and translating it to Rust after getting it right:

fn calculate_checksum(data: &mut [u8]) {
    let mut f = 0;
    let mut chk: u32 = 0;
    while f + 2 <= data.len() {
        chk += u16::from_le_bytes(data[f..f+2].try_into().unwrap()) as u32;

        f += 2;
    }

    while chk > 0xffff {
        chk = (chk & 0xffff) + (chk >> 2*8);
    }

    let mut chk = chk as u16;

    chk = !chk & 0xffff;

    // endianness
    //chk = chk >> 8 | ((chk & 0xff) << 8);

    data[3] = (chk >> 8) as u8;
    data[2] = (chk & 0xff) as u8;

}

And everything seems to be working, finally.

5. Github

The code is available on my Github xphoniex/icmp-rust.

I. Implementing ICMP in Rust

Dave — Sun, 18 Jul 2021 14:38:01 +0000

This post is inspired by Jon Gjengset series titled "Implementing TCP in Rust".

This should be an interesting read for anyone who wants to get a better understanding of networking protocols down to bits and doesn't mind diving into RFCs.

So what is ICMP anyway? It's the protocol that makes ping and traceroute possible. If you're a gamer or ever joined an online call, then you know how important those ping/latency numbers are.

Unlike TCP/UDP, which are Layer 4 protocols in OSI Model, ICMP actually belongs to Layer 3 which is the same level as IP, itself the main protocol of the Internet.

1. IP Packet

First, I'll show you what an IPv4 (RFC-791) header looks like, remember all your connections on the internet start with one of these:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options                    |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Example Internet Datagram Header

To capture a raw IP frame, ping localhost in another terminal and listen with:

$ tcpdump -i lo -p icmp -X
15:23:33.885666 IP localhost > localhost: ICMP echo request, id 21780, seq 1, length 64
    0x0000:  4500 0054 9210 4000 4001 aa96 7f00 0001  E..T..@.@.......
    0x0010:  7f00 0001 0800 492b 5514 0001 2d08 f460  ......I+U...-..`
    0x0020:  0000 0000 6c83 0d00 0000 0000 1011 1213  ....l...........
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567
15:23:33.885695 IP localhost > localhost: ICMP echo reply, id 21780, seq 1, length 64
    0x0000:  4500 0054 9211 0000 4001 ea95 7f00 0001  E..T....@.......
    0x0010:  7f00 0001 0000 512b 5514 0001 2d08 f460  ......Q+U...-..`
    0x0020:  0000 0000 6c83 0d00 0000 0000 1011 1213  ....l...........
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567

Let's start with the first IP and decode the header — the first 20 bytes of 4500 0054 9210 4000 4001 aa96 7f00 0001 7f00 0001 ...:

▼
4           Version                 = IPv4 version
5           IHL                     = 20 bytes
00          ToS
0054        Total Length            = 84 bytes
9210        Identification
4           Flags
000         Fragment Offset
40          Time to Live            = 64 TTL
01          Protocol                = ICMP
aa96        Header Checksum
7f00 0001   Source IP Address       = 127.0.0.1
7f00 0001   Destination IP Address  = 127.0.0.1

2. ICMP Packet

From RFC-792 we can see the layout of Echo or Echo reply message:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Type      |     Code      |          Checksum             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Identifier          |        Sequence Number        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Data ...
+-+-+-+-+-

And by replacing data bytes, the remaining bytes after the header, we get our ICMP header:

08          Type                    = Echo message
00          Code                     
492b        Checksum
5514        Identifier              = id 21780 (in raw capture)
0001        Sequence Number         = seq 1

If you compare this ICMP packet with the one from second IP frame, two fields have changed:

Type which has become 00 or Echo reply message
Checksum, naturally because content has changed

3. Implementation in Rust

Now that we know what an ICMP packet looks like, we are ready to implement it in Rust.

Unlike TCP/UDP, we have no concept of ports here and only use IPs to find our path. (Now you understand why ping has no port?).

Furthermore, astute reader may have noticed that the only thing that changes in the ICMP echo reply is the Type which goes from 08 to 00.

We've got all we need. Check out: II. Implementing ICMP in Rust

How to create a Kubernetes cluster on Alpine Linux

Dave — Tue, 19 May 2020 16:16:26 +0000

this post will help you understand kubeadm, kubelet flags, and nuances of alpine

Creating a production-ready K8s cluster is almost a breeze nowadays on most cloud platforms so I was curious to see how hard it'd be to create a cluster from scratch on my own set of VMs... turns out not very hard.

To accomplish this, you can either do it the hard way or use some automation. You're presented with two options:

kubespray which uses Ansible under the hood
kubeadm which is the official way to do it, part of k/k, and supported by amazing k8s team of VMWare

As kubeadm's binary already comes with kubernetes package on Alpine, I decided to go with that option.

I already had KVM installed on my machine and had an Alpine Linux 3.9 VM ready to go, so you need to pause here and provision your VMs if you haven't already before you proceed.

Once you have your VM, you need to add community and testing repositories so you can get the needed binaries for Kubernetes and Docker packages:

# echo "@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing/" >> /etc/apk/repositories
# echo "@community http://dl-cdn.alpinelinux.org/alpine/edge/community/" >> /etc/apk/repositories

then install required packages with:

# apk add kubernetes@testing
# apk add docker@community
# apk add cni-plugins@testing

at this point when I tried to start my docker service, I'd get an error:

# service docker start
supervise-daemon: --pidfile must be specified
failed to start Docker Daemon
ERROR: docker failed to start

This is apparently a bug on part of supervise-daemon and I created a merge request for this issue to alpine/aports but apparently this issue has been solved in newer versions of Alpine. In case you still run into this, you need to edit your /etc/init.d/docker file, add pidfile="/run/docker/docker.pid" and inside start_pre block add mkdir -p /run/docker.

Now you can duplicate your VM in KVM, and name the new one worker-1:

# hostname worker-1
# echo "worker-1" > /etc/hostname

make sure to do the same steps for master node but with the name master-1.

You're ready to create your control-plane on master node, run:

# kubeadm init --apiserver-advertise-address=[ Master Node's IP Here ] --kubernetes-version=1.17.5

Kubeadm runs in phases, and it was crashing when reaching:

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

Running another terminal (using SSH) and restarting the kubelet service fixed this issue. Turns out, Kubeadm starts the kubelet service first and then writes the config files it needs to start properly. On other OSes such as Ubuntu, Systemd -the OS init system- takes care of restarting the crashing service until the config files are there and kubelet can be run.

Alpine on the other hand, uses OpenRC as its init system which doesn't restart on crash loops. For that Gentoo community has introduced supervise-daemon which is experimental at the moment. To make this possible on Alpine, we fixed this issue directly on kubeadm with this PR.

Once kubeadm runs it course, it gives you two notes, one is the location of your kube config file. This is the file that kubectl uses to authenticate to API server on every call. You need to copy this file on any machines that needs to interact with cluster using kubectl.

Another one is a join statement like below, which is how you'll add your worker nodes to the cluster. First add your CNI on master node and then join from worker node:

# on master node
master-1 # kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# on worker node
worker-1 # kubeadm join 192.168.122.139:6443 --token hcexp0.qiaxub64z17up9rn --discovery-token-ca-cert-hash sha256:05653259a076769faa952024249faa9c9457b4abf265914ba58f002f08834006

Note:
Your join command should succeed now but when I initially tried this command, my kubelet service would again fail to start because config files were missing and, surprisingly, restarting kubelet service didn't help this time. (Shocking, I know!)

After some investigation I realized another mismatch between Systemd and OpenRC, --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf was missing from /etc/conf.d/kubelet and adding it fixed this but didn't specify CNI and my pods would get Docker IPs. You guessed it right, another kubelet argument missing. (See the full changes that were necessary here)

At this point you can deploy your workloads, and if you come from Ubuntu world one subtle difference is that you need to make sure you apps are compatible with musl as opposed to glibc. For example if you're deploying Go static binaries, make sure you're compiling with CGO_ENABLED=0 to create a statically-linked binary or if you're deploying node apps, make sure your npm install is being run inside an Alpine container.

That's it! Feel free to reach out to me if you need help with your k8s clusters.