64-bit RISC-V with Apache NuttX Real-Time Operating System

Lup Yuen Lee — Sat, 24 Jun 2023 00:05:17 +0000

Apache NuttX is a Real-Time Operating System (RTOS) that runs on many kinds of devices, from 8-bit to 64-bit.

(Think Linux, but a lot smaller and simpler)

In this article we'll...

Boot NuttX RTOS on a 64-bit RISC-V device
Explore the Boot Code that starts NuttX on RISC-V
And learn a little RISC-V Assembly!

But we need RISC-V Hardware?

No worries! We'll run NuttX on the QEMU Emulator for 64-bit RISC-V.

(Which will work on Linux, macOS and Windows machines)

Building Apache NuttX RTOS in 4 minutes

Boot NuttX on 64-bit RISC-V QEMU

We begin by booting NuttX RTOS on RISC-V QEMU Emulator (64-bit)...

Download and install QEMU Emulator.

For macOS we may use brew...
```
brew install qemu
```
Download nuttx from the NuttX Release...

nuttx: NuttX Image for 64-bit RISC-V QEMU

If we prefer to build NuttX ourselves: Follow these steps

Start the QEMU RISC-V Emulator (64-bit) with NuttX RTOS...

qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -smp 8 \
  -bios none \
  -kernel nuttx \
  -nographic

NuttX is now running in the QEMU Emulator! (Pic below)

uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
nx_start_application: Starting init thread

NuttShell (NSH) NuttX-12.1.0-RC0
nsh> nx_start: CPU0: Beginning Idle Loop
nsh>

(See the Complete Log)

Enter "help" to see the available commands...

nsh> help
help usage:  help [-v] [<cmd>]

    .         break     dd        exit      ls        ps        source    umount
    [         cat       df        false     mkdir     pwd       test      unset
    ?         cd        dmesg     free      mkrd      rm        time      uptime
    alias     cp        echo      help      mount     rmdir     true      usleep
    unalias   cmp       env       hexdump   mv        set       truncate  xd
    basename  dirname   exec      kill      printf    sleep     uname

Builtin Apps:
    nsh     ostest  sh

NuttX works like a tiny version of Linux, so the commands will look familiar...

nsh> uname -a
NuttX 12.1.0-RC0 275db39 Jun 16 2023 20:22:08 risc-v rv-virt

nsh> ls /dev
/dev:
console
null
ttyS0
zero

nsh> ps
  PID GROUP PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK           STACK   USED  FILLED COMMAND
    0     0   0 FIFO     Kthread N-- Ready              0000000000000000 002000 001224  61.2%  Idle Task
    1     1 100 RR       Task    --- Running            0000000000000000 002992 002024  67.6%  nsh_main

(See the Complete Log)

Let's talk about QEMU...

Apache NuttX RTOS on RISC-V QEMU

QEMU Emulator for RISC-V

Earlier we ran this command. What does it mean?

qemu-system-riscv64 \
  -kernel nuttx \
  -cpu rv64 \
  -smp 8 \
  -M virt,aclint=on \
  -semihosting \
  -bios none \
  -nographic

The above command starts the QEMU Emulator for RISC-V (64-bit) with...

Kernel Image: nuttx
CPU: 64-bit RISC-V
Symmetric Multiprocessing: 8 CPU Cores
Machine: Generic Virtual Platform (virt)
Handle Interrupts with Advanced Core Local Interruptor (ACLINT)

(Instead of the older SiFive Core Local Interruptor CLINT)

Enable Semihosting Debugging without BIOS

Which RISC-V Instructions are supported by QEMU?

QEMU's RISC-V Generic Virtual Platform (virt) supports RV64GC, which is equivalent to RV64IMAFDCZicsr_Zifencei (phew)...


RV64I	64-bit Base Integer Instruction Set
M	Integer Multiplication and Division
A	Atomic Instructions
F	Single-Precision Floating-Point
D	Double-Precision Floating-Point
C	Compressed Instructions
Zicsr	Control and Status Register (CSR) Instructions
Zifencei	Instruction-Fetch Fence

(Source)

We'll meet these instructions shortly.

QEMU Starts NuttX

What happens when NuttX RTOS boots on QEMU?

Let's find out by tracing the RISC-V Boot Code in NuttX!

Earlier we ran this command to generate the RISC-V Disassembly for the NuttX Kernel...

riscv64-unknown-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

This produces nuttx.S, the disassembled NuttX Kernel for RISC-V.

nuttx.S begins with this RISC-V code...

0000000080000000 <__start>:
nuttx/arch/risc-v/src/chip/qemu_rv_head.S:46
__start:
  /* Load mhartid (cpuid) */
  csrr a0, mhartid
    80000000:   f1402573  csrr  a0, mhartid

This says...

NuttX Boot Code is at qemu_rv_head.S
NuttX Kernel begins execution at address 0x8000 0000

(Why? What if NuttX is started by the U-Boot Bootloader?)

Now we head into the NuttX Boot Code...

RISC-V Boot Code for Apache NuttX RTOS

RISC-V Boot Code in NuttX

What's inside the NuttX Boot Code?

The RISC-V Assembly code in qemu_rv_head.S will...

Let's decipher the RISC-V Instructions in our Boot Code...

Get CPU ID

This is how we fetch the CPU ID in RISC-V Assembly: qemu_rv_head.S

/* Load mhartid (cpuid) */
csrr  a0, mhartid

Let's break it down...

csrr is the RISC-V Instruction that reads the Control and Status Register

(Which contains the CPU ID)

a0 is the RISC-V Register that will be loaded with the CPU ID.

According to the RISC-V EABI (Embedded Application Binary Interface), a0 is actually an alias for the Official RISC-V Register x10.

("a" refers to "Function Call Argument")

mhartid says that we'll read from the Hart ID Register, containing the ID of the Hardware Thread ("Hart") that's running our code.

(Equivalent to CPU ID)

So the above code will load the CPU ID into Register x10.

(We'll call it a0 for convenience)

Disable Interrupts

To disable interrupts in RISC-V, we do this: qemu_rv_head.S

/* Disable all interrupts (i.e. timer, external) in mie */
csrw  mie, zero

Which means...

csrw will write to the Control and Status Register

(Which controls interrupts and other CPU settings)

mie says that we'll write to the Machine Interrupt Enable Register

(0 to Disable Interrupts, 1 to Enable)

zero says that we'll read from Register x0...

Which always reads as 0!

Thus the above instruction will set the Machine Interrupt Enable Register to 0, which will disable interrupts.

(Yeah RISC-V has a funny concept of "0")

Wait for Interrupt

Now check out this curious combination of instructions: qemu_rv_head.S

/* Wait forever */
csrw  mie, zero
wfi

From the previous section, we know that "csrw mie, zero" will disable interrupts.

But wfi will Wait for Interrupt...

Which will never happen because we disabled interrupts!

Thus the above code will get stuck there, waiting forever. (Intentionally)

(wfi is probably the only instruction common to RISC-V and Arm CPUs)

Load Interrupt Vector

RISC-V handles interrupts by looking up the Interrupt Vector Table.

This is how we load the Address of the Vector Table into the CPU Settings: qemu_rv_head.S

/* Load address of Interrupt Vector Table */
la    t0, __trap_vec
csrw  mtvec, t0

la loads the Address of the Vector Table into Register t0

(Which is aliased to Register x5)

(trap_vec is defined here)

csrw writes t0 into the Control and Status Register at...
mtvec, the Machine Trap-Vector Base-Address Register

Which will load the Address of our Interrupt Vector Table into the CPU Settings.

(la is actually a Pseudo-Instruction that expands to auipc and addi)

(auipc loads an Address Offset from the Program Counter)

(addi adds an Immediate Value to a Register)

32-bit vs 64-bit RISC-V

Adapting 32-bit code for 64-bit sounds hard... But it's easy peasy for RISC-V!

Our Boot Code uses an Assembler Macro to figure out if we're running 32-bit or 64-bit RISC-V: qemu_rv_head.S

#ifdef CONFIG_ARCH_RV32
  /* Do this for 32-bit RISC-V */
  slli t1, a0, 2

#else
  /* Do this for 64-bit RISC-V */
  slli t1, a0, 3
#endif

Which means that the exact same Boot Code will run on 32-bit AND 64-bit RISC-V!

(slli sounds "silly", but it's Logical Shift Left)

(CONFIG_ARCH_RV32 is derived from our NuttX Build Configuration)

Other Instructions

What about the other RISC-V Instructions in our Boot Code?

Let's skim through the rest...

bnez branches to Label 1f if Register a0 is Non-Zero

  bnez  a0, 1f

(Source)

j jumps to Label 2f

(We'll explain Labels in a while)

  j  2f

(Source)

li loads the Value 1 into Register t1

(li is a Pseudo-Instruction that expands to addi)

(addi adds an Immediate Value to a Register)

  li  t1, 1

(Source)

blt branches to Label 3f if Register a0 is less than Register t1

(And grabs a sandwich)

  blt  a0, t1, 3f

(Source)

add sets Register t0 to the value of Register t0 + Register t1

(t1 is aliased to Register x15)

  add  t0, t0, t1

(Source)

REGLOAD is an Assembly Macro that expands to ld

ld loads Register t0 into the Stack Pointer Register

(Which is aliased to Register x2)

  REGLOAD  sp, 0(t0)

(Source)

jal (Jump And Link) will jump to the address qemu_rv_start and store the Return Address in Register x1

(Works like a Function Call)

  jal  x1, qemu_rv_start

(Source)

ret returns from a Function Call.

(ret is a Pseudo-Instruction that expands to jalr)

(jalr "Jump And Link Register" will jump to the Return Address stored in Register x1)

ret

(Source)

(See the list of all RISC-V Instructions)

(And RISC-V Pseudo-Instructions)

Why are the RISC-V Labels named "1f", "2f", "3f"?

"1f" refers to the Local Label "1" with a Forward Reference.

(Instead of a Backward Reference)

Let's jump to qemu_rv_start...

RISC-V Start Code for NuttX RTOS

Jump to Start

Our Boot Code jumps to qemu_rv_start...

What happens next?

qemu_rv_start is the very first C Function that NuttX runs when it boots on QEMU.

The function will...

Configure the Floating-Point Unit
Clear the BSS Memory
Initialise the Serial Port
Initialise the Memory Management Unit

(For Kernel Mode only)
Call nx_start

What happens in nx_start?

nx_start will initialise a whole bunch of NuttX things...

"After Primary Routine: nx_start"

Which will start the NuttX Shell that we've seen earlier.

And that's how NuttX RTOS boots on QEMU Emulator for RISC-V!

Why are we doing all this?

We're about to port NuttX to the StarFive JH7110 RISC-V SoC and Pine64 Star64 Single-Board Computer.

The analysis we've done today will be super helpful as we write the Boot Code for these RISC-V devices.

Stay tuned for updates in the next article!

What's Next

I hope this article has been an educational exploration of Apache NuttX RTOS on 64-bit RISC-V...

We booted NuttX RTOS on an emulated 64-bit RISC-V device
We peeked at the Boot Code that starts NuttX on RISC-V
And hopefully we learnt a little RISC-V Assembly!

As we've seen, NuttX is a tiny operating system that's perfect for experimenting with RISC-V gadgets. We'll do this and much more in the upcoming articles!

(We welcome your contribution to Apache NuttX RTOS)

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn't have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here...

lupyuen.github.io/src/riscv.md

Appendix: Build Apache NuttX RTOS for 64-bit RISC-V QEMU

The easiest way to run Apache NuttX RTOS on 64-bit RISC-V is to download the NuttX Image and boot it on QEMU Emulator...

"Boot NuttX on 64-bit RISC-V QEMU"

But if we're keen to build NuttX ourselves, here are the steps...

Install the Build Prerequisites, skip the RISC-V Toolchain...

"Install Prerequisites"
Download the RISC-V Toolchain for riscv64-unknown-elf...

"Download Toolchain for 64-bit RISC-V"

Download and configure NuttX...

mkdir nuttx
cd nuttx
git clone https://github.com/apache/nuttx nuttx
git clone https://github.com/apache/nuttx-apps apps

cd nuttx
tools/configure.sh rv-virt:nsh64
make menuconfig

In menuconfig, browse to "Build Setup > Debug Options"

Select the following options...

Enable Debug Features
Enable Error Output
Enable Warnings Output
Enable Informational Debug Output
Enable Debug Assertions
Scheduler Debug Features
Scheduler Error Output
Scheduler Warnings Output
Scheduler Informational Output

Save and exit menuconfig.

Build the NuttX Project and dump the RISC-V Disassembly...

make V=1 -j7

riscv64-unknown-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

(See the Build Log)

(See the Build Outputs)

If the build fails with...
```
sed: 1: "/CONFIG_BASE_DEFCONFIG/ ...": bad flag in substitute command: '}'
```
Please run "make menuconfig > Build Setup > Debug Options" and uncheck "Enable Debug Features". Save, exit menuconfig and rebuild NuttX with make.

This produces the NuttX Image nuttx that we may boot on QEMU RISC-V Emulator...

"Boot NuttX on 64-bit RISC-V QEMU"

Let's look at the GCC Command that compiles NuttX for 64-bit RISC-V QEMU...

Appendix: Compile Apache NuttX RTOS for 64-bit RISC-V QEMU

From the previous section, we see that the NuttX Build compiles the source files with these GCC Options...

riscv64-unknown-elf-gcc \
  -c \
  -fno-common \
  -Wall \
  -Wstrict-prototypes \
  -Wshadow \
  -Wundef \
  -Os \
  -fno-strict-aliasing \
  -fomit-frame-pointer \
  -ffunction-sections \
  -fdata-sections \
  -g \
  -march=rv64imac \
  -mabi=lp64 \
  -mcmodel=medany \
  -isystem nuttx/include \
  -D__NuttX__ \
  -DNDEBUG \
  -D__KERNEL__  \
  -pipe \
  -I nuttx/arch/risc-v/src/chip \
  -I nuttx/arch/risc-v/src/common \
  -I nuttx/sched \
  chip/qemu_rv_start.c \
  -o  qemu_rv_start.o

(See the Build Log)

The RISC-V Options are...

march=rv64imac: This generates Integer-Only 64-bit RISC-V code, no Floating-Point.

Which is surprising because RISC-V QEMU actually supports Floating-Point.

We'll fix this as we port NuttX to the StarFive JH7110 RISC-V SoC and Pine64 Star64 SBC.

mabi=lp64: This Application Binary Interface says that Long Pointers are 64-bit. No Floating-Point Arguments will be passed in Registers.

We might fix this for JH7110 SoC and Star64 SBC.

(More about this)

mcmodel=medany: Sounds like a burger (or fast-food AI model) but it actually generates code for the Medium-Any Code Model. (Instead of Medium-Low)

(More about this)

Appendix: Download Toolchain for 64-bit RISC-V

Follow these steps to download the 64-bit RISC-V Toolchain for building Apache NuttX RTOS on Linux, macOS or Windows...

Download the riscv64-unknown-elf RISC-V Toolchain for Linux, macOS or Windows...

-   [__Ubuntu Linux__](https://static.dev.sifive.com/dev-tools/freedom-tools/v2020.12/riscv64-unknown-elf-toolchain-10.2.0-2020.12.8-x86_64-linux-ubuntu14.tar.gz)

-   [__CentOS Linux__](https://static.dev.sifive.com/dev-tools/freedom-tools/v2020.12/riscv64-unknown-elf-toolchain-10.2.0-2020.12.8-x86_64-linux-centos6.tar.gz)

-   [__macOS__](https://static.dev.sifive.com/dev-tools/freedom-tools/v2020.12/riscv64-unknown-elf-toolchain-10.2.0-2020.12.8-x86_64-apple-darwin.tar.gz)

-   [__Windows MinGW__](https://static.dev.sifive.com/dev-tools/freedom-tools/v2020.12/riscv64-unknown-elf-toolchain-10.2.0-2020.12.8-x86_64-w64-mingw32.zip)

Extract the Downloaded Toolchain
Add the Extracted Toolchain to the PATH Environment Variable...
```
riscv64-unknown-elf-toolchain-.../bin
```
Check the RISC-V Toolchain...
```
riscv64-unknown-elf-gcc -v
```

DEV Community: Lup Yuen Lee

64-bit RISC-V with Apache NuttX Real-Time Operating System

Boot NuttX on 64-bit RISC-V QEMU

QEMU Emulator for RISC-V

QEMU Starts NuttX

RISC-V Boot Code in NuttX

Get CPU ID

Disable Interrupts

Wait for Interrupt

Load Interrupt Vector

32-bit vs 64-bit RISC-V

Other Instructions

Jump to Start

What's Next

Appendix: Build Apache NuttX RTOS for 64-bit RISC-V QEMU

Appendix: Compile Apache NuttX RTOS for 64-bit RISC-V QEMU

Appendix: Download Toolchain for 64-bit RISC-V