I normally write articles which highlight Arm technology and how to use it. As an AWS Community Builder I write about using AWS services based on Graviton processors. This article is a little different, it describes an application which is built on top of AWS Graviton processors. I think many readers are going to want to check it out, especially if you are trying to purchase a Raspberry Pi 4 and can't find one.
For many years engineers have used modeling techniques to enable software development before silicon is available. Commercial software such as Arm Fast Models and open source software such as QEMU have been used for early software development and testing. Hypervisors on Arm play an important role in allowing multiple users to share physical hardware. Hypervisors also play a key role in automotive software to consolidate multiple operating systems and isolate critical software.
Today, Arm announced an update to Arm Virtual Hardware which includes several new virtual devices including a virtual Raspberry Pi 4. Raspberry Pi products have made a significant impact in many areas of computing. This year the Raspberry Pi was hailed as the most successful computer ever created in the UK. More than 40 million boards have shipped during the 10 year history. Low cost boards running Linux made it easy to port software to the Arm architecture.
The virtual Raspberry Pi provided by Arm Virtual Hardware uses a combination of native software execution and extensions to model the architecture and peripherals of a board such as the Raspberry Pi 4. Virtual boards look and feel very much like the physical boards being modeled. Virtual boards run in the cloud and users connect to them using a browser, command line, or an API for test automation. AWS has pioneered general purpose cloud computing and it has become a regular part of software development. Until now, access to special purpose computing hardware in the cloud has not been widely available, and most developers rely on physical boards in a lab to do many software development and testing tasks.
My goal today is to introduce the virtual Raspberry Pi 4 and compare it to the physical Raspberry Pi 4 on my desk.
Background
For many years embedded system developers have used a combination of physical boards (on their desk or in a lab) and virtual hardware models. Historically, the highest performance virtual models have translated x64 instructions to Arm instructions. Each solution has pros and cons related to execution performance, availability, scalability, and model accuracy. Virtual models based on instruction translation have trouble keeping up with the performance needs of software developers. This is especially true for recent Cortex-A and Neoverse processors. Creation of complete boards using Fast Model technology has also been a challenge for users. Cloud based virtual boards combine the benefits of custom hardware modeling with the performance of physical Arm processors. This has resulted in virtual boards which appear very close to the physical board, run faster, and are easier to use.
There are many common questions about the differences between a virtual board and the physical board or device.
- What is different about them?
- What kind of software can be run?
- Can I measure performance on the virtual board?
Allan Turing devised the Turing test in 1950 to determine if a questioner can tell the difference between human and computer responses.
Today, let’s try to figure out if we can tell the difference between a Raspberry Pi 4 on my desk and a virtual Raspberry Pi 4 running in the cloud.
Raspberry Pi 4 hardware specs
First, let’s review the Raspberry Pi 4 specs.
- SoC: Broadcom BCM2711B0 quad-core A72 (ARMv8-A) 64-bit @ 1.5GHz
- GPU: Broadcom VideoCore VI
- Networking: 2.4 GHz and 5 GHz 802.11b/g/n/ac wireless LAN
- RAM: 1GB, 2GB, or 4GB LPDDR4 SDRAM
- Bluetooth: Bluetooth 5.0, Bluetooth Low Energy (BLE)
- GPIO: 40-pin GPIO header, populated
- Storage: microSD
- Ports: 2 × micro-HDMI 2.0, 3.5 mm analogue audio-video jack, 2 × USB 2.0, 2 × USB 3.0, Gigabit Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)
In many ways the Raspberry Pi 4 looks like a regular Arm based Linux computer. Recent performance increases, dual-monitor support, and the Raspberry Pi 400 kit make it a suitable desktop computer for many use cases. In other ways, the Raspberry Pi looks more like a board for creating embedded systems. It has GPIO pins for connecting to extra hardware and has inspired people around the world to create thousands of unique projects including weather stations, security cameras, photo frames, home automation, and many more. There are maker magazines filled with creative projects which have been constructed from Raspberry Pi products.
Let’s start investigating the differences between the real Raspberry Pi 4 and the virtual Raspberry Pi 4. I have the physical Raspberry Pi 4 on my desk and ready to go.
If you are eager to try this out, feel free to scroll straight to the bottom and register for free access to a virtual Raspberry Pi 4.
Hardware Identification
The first place to start is identifying the hardware. Both systems are running the 64-bit Raspberry Pi OS released January 2022. I have set the hostname of the virtual board to be “pimodel” and the default hostname of “raspberry” for the physical board just to make it clear which one is shown in the screenshots.
The first command to try is uname. The output is identical on both machines. One of them is shown here:
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.15.30-v8+ #1536 SMP PREEMPT Mon Mar 28 13:53:14 BST 2022 aarch64 GNU/Linux
Next, try out lscpu. The physical board shows the following:
pi@raspberrypi:~ $ lscpu
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: ARM
Model: 3
Model name: Cortex-A72
Stepping: r0p3
CPU max MHz: 1800.0000
CPU min MHz: 600.0000
BogoMIPS: 108.00
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Vulnerable
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm crc32 cpuid
The virtual board output is below. The CPU model name is Neoverse-N1 with a max frequency of 2 GHz. There are also numerous extra flags printed which are not shown by the real Raspberry Pi. One of the flags is atomics, which I covered in my previous article on Large System Extensions.
pi@pimodel:~ $ lscpu
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: ARM
Model: 1
Model name: Neoverse-N1
Stepping: r3p1
CPU max MHz: 2000.0000
CPU min MHz: 100.0000
BogoMIPS: 243.75
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdh
p cpuid asimdrdm lrcpc dcpop asimddp ssbs
The next check is lshw to see more info about other hardware. To install it:
$ sudo apt-get install lshw
The output is long and not all shown here. The key differences I spotted are the lack of a serial number and board revision on the model.
The real Raspberry Pi has:
pi@raspberrypi:~ $ lshw
WARNING: you should run this program as super-user.
raspberrypi
description: Computer
product: Raspberry Pi 4 Model B Rev 1.4
serial: 100000004d20671f
width: 64 bits
capabilities: smp cp15_barrier setend swp tagged_addr_disabled
The virtual Raspberry Pi has no serial number:
pi@pimodel:~ $ lshw
WARNING: you should run this program as super-user.
pimodel
description: Computer
product: Raspberry Pi 4 Model B
width: 64 bits
capabilities: smp cp15_barrier setend swp tagged_addr_disabled
I also hit the lack of serial number when installing remote.it. The install script tried to get the serial number and printed an error message. This can be confirmed as shown below.
pi@raspberrypi:~ $ more /sys/firmware/devicetree/base/serial-number
100000004d20671f
The virtual Raspberry Pi doesn’t have it.
pi@pimodel:~ $ more /sys/firmware/devicetree/base/serial-number
more: cannot open /sys/firmware/devicetree/base/serial-number: No such file or directory
Another difference in lshw is the wireless interface is missing for the virtual board. This is not significant as the wired network works fine and unless you were a software developer working on the wireless stack it wouldn’t matter.
*-network:1
description: Wireless interface
physical id: 2
logical name: wlan0
serial: dc:a6:32:b1:b6:18
capabilities: ethernet physical wireless
configuration: broadcast=yes driver=brcmfmac driverversion=7.45.241 firmware=01-703fd60 ip=192.168.68.96 multicast=yes wireless=IEEE 802.11
The networking interface has a slight difference in the Ethernet speed. The real Raspberry Pi reports 1 Gbit/s Ethernet and the virtual Raspberry Pi reports 10 Gbit/s Ethernet.
The initial survey shows the virtual board looks very close to a real Raspberry Pi. The virtual model of the Pi shows a 4 Gb RAM configuration while my physical Pi is 8 Gb RAM.
The CPU model name is really the only clear sign that this is not a real Raspberry Pi 4.
Performance
Let’s try some benchmarks to investigate the performance differences between the actual Raspberry Pi and the virtual Raspberry Pi.
Kernel Compile
One of the benefits of the Raspberry Pi compared to other Linux boards used in embedded projects is the ease of building the Linux kernel. The good news is the Linux kernel is very easy to build natively on the Raspberry Pi. The bad news is that it takes a very long time. The instructions even have a warning (with long in bold).
“this step can take a long time depending on the Raspberry Pi model in use”
I followed the Linux kernel information to build a kernel on the real Raspberry Pi and the virtual Raspberry Pi to see how long it would take. As a reference, I did the exact same thing on an x64 virtual machine in AWS using a cross compiler.
| System | Kernel compile time |
|---|---|
| Physical Raspberry Pi 4 (8 Gb RAM) | 81 min 17 sec |
| Virtual Raspberry Pi 4 (4 Gb RAM) | 20 min 6 sec |
| AWS EC2 c5.xlarge (4 vCPU, 8 Gb RAM) | 28 min 27 sec |
Kernel building is significantly faster on the virtual Pi and no cross compiling is required. Even cross compiling on an x64 machine was not faster than the virtual Raspberry Pi.
TensorFlow for machine learning
To see if TensorFlow can run on both the real Raspberry Pi 4 and the virtual Raspberry Pi 4 install it using:
$ pip install tensorflow-aarch64
I ran a TensorFlow quickstart example to compare the systems.
| System | Time to complete |
|---|---|
| Physical Raspberry Pi 4 | 1 min 9 sec |
| Virtual Raspberry Pi 4 | 22 sec |
Both systems produced the same results, and the virtual Pi is significantly faster.
Docker
Install Docker using the universal Linux instructions and check it works by running the hello-world image:
$ curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh
$ sudo usermod -aG docker $USER
$ newgrp docker
$ docker run hello-world
I tried out some Docker projects and they build and run as expected. Some simple ones are in my GitHub account in a project called hello-arm. I also successfully ran a few official images from Docker Hub such as Ubuntu. None of the projects detect anything different about the virtual Raspberry Pi 4 and performance is better than the physical Raspberry Pi in all cases. It’s also possible to run 32-bit Arm containers on the virtual Raspberry Pi with no problems.
Here are two common questions about the virtual Raspberry Pi.
Can I create a program that works on the virtual Raspberry Pi but not on a physical Raspberry Pi?
Yes, some of the software that is compiled for Arm Neoverse-N1 machines such as AWS Graviton2 will work on the virtual Raspberry Pi 4, but not on the actual Raspberry Pi.
One of the ideas for a benchmark was to check machine learning performance using the instructions of an article I published for using TensorFlow and PyTorch from Anaconda on Graviton2. Anaconda is clearly targeted for Graviton2 users, but I gave it a try anyway.
The installation of PyTorch worked perfectly well on the virtual Raspberry Pi 4 and I could run the MNIST example as outlined in the article. The physical Raspberry Pi 4 didn’t complete the installation.
As happens with software which uses architecture features newer than the Cortex-A72 the software stops with “Illegal instruction” and this is what happened to me on the physical Raspberry Pi 4.
The same thing will likely happen on any software compiled with instructions not included in the Cortex-A72 but present in the Neoverse-N1 processor. This may include dot product instructions or low-cost atomic instructions (also known as Large System Extensions, LSE).
Can I create a program that works on the physical Raspberry Pi but not on the virtual Raspberry Pi?
Another good question.
Let’s try Vulkan support on the Raspberry Pi 4. I followed the instructions for building some demo Vulkan apps. Both systems build the Vulkan software and demo applications correctly. The demos work fine on the physical Raspberry Pi 4. The traditional “gears” application and others on the referenced instructions work just like the article. On the virtual Raspberry Pi 4 the gears application fails with a Vulkan error.
Bluetooth
The physical Raspberry Pi 4 supports Bluetooth connections. Use bluetoothctl and scan for devices. I can connect my Android phone to the Raspberry Pi and create various applications to send and receive data between the Pi and the phone using Bluetooth.
The virtual Raspberry Pi 4 doesn’t have Bluetooth. There is no sign of it in the kernel boot log and bluetoothctl just hangs.
There are a few other points to be aware of when using the virtual Raspberry Pi 4.
Linux Perf
A common scenario is for a software developer to use perf to analyze performance. Install and enable it using the commands below. Make sure to become root for the last two commands.
$ sudo apt-get install linux-perf
$ sudo su -
# echo -1 > /proc/sys/kernel/perf_event_paranoid
# echo 0 > /proc/sys/kernel/kptr_restrict
I found that perf works fine on both the physical and virtual Raspberry Pi 4, but there are additional perf events on the virtual Raspberry Pi 4 because of the different underlying hardware.
To list the available perf events:
$ perf list
An example event which is included in the virtual Raspberry Pi 4 is stalled-cycles-backend. This is not available in the physical Raspberry Pi 4.
pi@pimodel:~ $ perf stat -e stalled-cycles-backend tar cfz test.tgz ./linux/
Performance counter stats for 'tar cfz test.tgz ./linux/':
27,701,570,350 stalled-cycles-backend # 0.00% backend cycles idle
44.461783841 seconds time elapsed
43.283230000 seconds user
2.121127000 seconds sys
If I run a tar command to zip the kernel source tree it shows a value for this event on the virtual Raspberry Pi 4.
The physical Raspberry Pi doesn’t have this event.
pi@raspberrypi:~ $ perf stat -e stalled-cycles-backend tar cfz test.tgz ./linux/
Performance counter stats for 'tar cfz test.tgz ./linux/':
<not supported> stalled-cycles-backend
143.177396310 seconds time elapsed
129.623757000 seconds user
8.292624000 seconds sys
KVM (kernel-based virtual machine)
Because the virtual Raspberry Pi 4 is running under the control of a hypervisor it cannot use KVM.
The physical Raspberry Pi does have KVM support and it can be seen in the kernel boot log and the device node.
pi@raspberrypi:~ $ dmesg | grep kvm
[ 0.297625] kvm [1]: IPA Size Limit: 44 bits
[ 0.298810] kvm [1]: vgic interrupt IRQ9
[ 0.299076] kvm [1]: Hyp mode initialized successfully
The virtual Raspberry Pi 4 does not have KVM. Probably doesn’t matter much for application development and this is the point of the virtual Raspberry Pi, it is an alternative to using qemu + kvm as a type 1 hypervisor.
pi@pimodel:~ $ dmesg | grep kvm
[ 0.779186] kvm [1]: HYP mode not available
Summary
Arm Virtual Hardware is a new technology which provides access to a virtual Raspberry Pi 4 as well as other boards used in embedded and IoT applications. The virtual Raspberry Pi 4 is very close to the physical Raspberry Pi 4, and runs much faster. Differences are related to the underlying processor details of the Neoverse-N1 versus the Cortex-A72. Some hardware is not available on the virtual Raspberry Pi. In some cases, additional modeling can be done, but in most cases the missing hardware is not needed.
To access the Arm Virtual Hardware Beta, including the virtual Raspberry Pi 4, click the Register button at https://avh.arm.com/
Top comments (3)
@jasonrandrews is AWS the sole way to access ARM virtualized hardware? I'm on the process of using Raspberry Pi 4 to emulate a cluster of smart devices. I went through the QEMU rabbit hole only to know that it is still not possible to do it. Then I came across your post.
Is there any way to access virtual GPIOs? How are they presented back to you? How can you simulate I/O on those?
Hello Patrick,
Yes, you can simulate the GPIO using custom vMMIO. You register a memory range in the hardware address map and when software running on the Pi accesses this range your code outside of the virtual Pi will be called. There is a simple example you can review at: intercom.help/arm-avh/en/articles/...