DEV Community

Cover image for ZeroTier Without Native. Managed ZtSharp and native VL1-only ZtLite for ESP32
Smersh
Smersh

Posted on

ZeroTier Without Native. Managed ZtSharp and native VL1-only ZtLite for ESP32

#esp32 #zerotier #csharp #cpp

Hello everyone.

I temporarily put the ESPB interpreter project on hold because interest in it turned out to be rather limited. So I approached the larger problem from another direction, and this time everything revolves around ZeroTier.

I had been interested in ZeroTier for quite a while. At one point I worked with libzt — sockets over the ZeroTier network. It is very convenient for applications whose endpoints are not necessarily on the same local network and may be located behind NAT.

That raised an interesting question:

What if we could use ZeroTier without depending on native libraries at all, and what if an ESP32 only needed the P2P part of ZeroTier rather than the entire virtual Ethernet stack?

This eventually resulted in three related projects:

Project Purpose
ZtSharp Fully managed ZeroTier implementation in C#/.NET with its own TCP/IP stack over VL2
ZtLiteESP Minimal VL1-only client for ESP32 for direct peer-to-peer messaging
ZtGenStorage Generates ZeroTier identity data during provisioning instead of doing it on the MCU

ZeroTier: Two Layers

Conceptually, ZeroTier can be divided into two layers.

VL1 (Virtual Layer 1) is the cryptographically addressed P2P layer. It provides identity, peer discovery, cryptography, path management, NAT traversal, and the exchange of control and application messages.

VL2 (Virtual Layer 2) provides virtual Ethernet on top of VL1: virtual networks, Ethernet frames, network configuration, and IP connectivity.

A simplified view looks like this:

ZeroTier
├── VL1
│    ├── identity
│    ├── discovery & crypto
│    ├── paths & NAT traversal
│    └── USER_MESSAGE
│
└── VL2
     ├── virtual Ethernet & frames
     ├── ARP / NDP
     └── IP connectivity
Enter fullscreen mode Exit fullscreen mode

For a conventional ZeroTier node, both layers are useful.
But not every application actually needs VL2.

ZtSharp: ZeroTier Entirely in C

Originally, I wanted to create a managed equivalent of libzt without native binaries and without having to maintain P/Invoke wrappers for every operating system.

That is where ZtSharp started.

ZtSharp is a fully managed implementation of a ZeroTier node in C#/.NET. The protocol core is based on the ZeroTier One node/ implementation, while the surrounding networking layer, APIs, storage, and managed infrastructure are implemented in C#.

The project currently includes:

  • VL1/VL2 protocol logic;
  • identity and peer management;
  • peer discovery and path establishment;
  • cryptographic functionality;
  • relay and multipath/bonding support;
  • USER_MESSAGE;
  • Planet/Moon support;
  • an embedded network controller;
  • pluggable state storage;
  • logging through Microsoft.Extensions.Logging.

It also includes a managed TCP/IP stack operating directly on the ZeroTier virtual Ethernet layer instead of using the operating system's native sockets.

The stack includes:

  • ARP;
  • NDP;
  • ICMP/ICMPv6;
  • IPv4;
  • IPv6;
  • UDP;
  • TCP;
  • a socket API.

The overall architecture looks like this:

.NET application
       │
       ▼
    ZtSocket
       │
       ▼
 managed TCP/IP
       │
       ▼
 virtual Ethernet
       │
       ▼
  ZeroTier node
    VL1 + VL2
Enter fullscreen mode Exit fullscreen mode

The goal was to make ZeroTier usable from managed .NET applications without requiring libzt, P/Invoke, or platform-specific native bindings.

Compatibility with the original libzt has also been tested. A managed ZtSharp node was able to communicate with a native libzt node over VL2.

USER_MESSAGE: Communication Without a Virtual Network

While working with the ZeroTier protocol, I became particularly interested in USER_MESSAGE.

This is a VL1 application-level message addressed directly to a specific peer.

At the API level it contains:

  • a destination address;
  • a 64-bit typeId;
  • an arbitrary payload.

The typeId can be used as an application-level message type. For example, it can distinguish telemetry from commands or separate different versions of an application protocol.

Conceptually:

Application
     │
     ▼
USER_MESSAGE
(typeId + payload)
     │
     ▼
ZeroTier VL1
(identity / path / crypto)
     │
     ▼
Peer
Enter fullscreen mode Exit fullscreen mode

No virtual network membership is required.

You do not need:

  • to join a VL2 virtual network;
  • a ZeroTier IP address;
  • TAP/TUN;
  • ARP/NDP;
  • IP routing.

There is, however, an important limitation.

USER_MESSAGE is not a reliable message queue with guaranteed delivery. Delivery confirmation, sequence numbers, retransmission, duplicate detection, and application-level ACK semantics must be implemented by the application if they are required.

That is exactly how I use it in the ESP32 test: the application sends messages such as:

hello from esp32:0
hello from esp32:1
hello from esp32:2
Enter fullscreen mode Exit fullscreen mode

and receives corresponding responses from the remote peer.

Those responses are application-level acknowledgements, not a built-in reliability mechanism provided by USER_MESSAGE.

For small telemetry and control messages, however, this is often enough.

Why VL1-only on ESP32?

For a server, the following architecture makes perfect sense:

VL1
 ↓
VL2
 ↓
Virtual Ethernet
 ↓
ARP / IPv4 / IPv6
 ↓
UDP / TCP
Enter fullscreen mode Exit fullscreen mode

But consider a much simpler use case:

The device sends its status to a server and occasionally receives a command.

For such a device, bringing the complete VL2 networking stack to the MCU may be unnecessary.

This is where ZtLiteESP comes in.

It keeps only what is actually required:

ESP32
  │
  ├── Identity
  ├── Discovery
  ├── WHOIS
  ├── Path
  ├── NAT traversal
  ├── Crypto
  └── USER_MESSAGE
          │
          ▼
      Application
Enter fullscreen mode Exit fullscreen mode

And intentionally does not include:

VL2
TAP
ARP/NDP
ZeroTier IP
IP routing
network rules
Enter fullscreen mode Exit fullscreen mode

The idea is not to create a new ZeroTier protocol.

Internally, ZtLite uses the platform-independent node/ core from ZeroTier One. The ESP32-specific layer provides the integration with ESP-IDF, FreeRTOS, and the UDP transport.

The transport itself is separated from the protocol core, which also makes it possible to debug the same profile outside the physical device:

                 ZtLite core
                      │
           ┌──────────┼──────────┐
           ▼          ▼          ▼
         ESP32      QEMU       Win32
         (lwIP)   (OpenEth)   (Winsock)
Enter fullscreen mode Exit fullscreen mode

Identity and Planet: Don't Generate Them on the Chip

Generating a ZeroTier identity involves a memory-hard proof-of-work based on SHA-512 and Salsa20.

On a PC, this operation is relatively inexpensive.

On a microcontroller, doing it during the first boot is much less attractive.

This is why ZtGenStorage was created.

It generates the ZeroTier identity in the required format:

identity.public
identity.secret
Enter fullscreen mode Exit fullscreen mode

and also prepares the required planet data.

The idea is simple:

Build / provisioning host
          │
          ├── generate identity
          ├── prepare planet
          │
          ▼
      ESP32 firmware
Enter fullscreen mode Exit fullscreen mode

The MCU does not need to generate an identity during boot. The required data can be generated ahead of time and embedded into the firmware image.

There is one critical rule:

One ZeroTier identity must belong to exactly one device.

If multiple devices are flashed with the same identity.secret, they will collide on the network.

For production provisioning, each device therefore needs its own generated identity.

Testing on a Real ESP32-C6

After debugging the implementation in QEMU, I tested ZtLite on a real ESP32-C6.

The test used:

  • ESP-IDF 6.0.2;
  • Wi-Fi;
  • UDP transport on port 9995.

After connecting to Wi-Fi, the ESP32 received its IP address, started the ZeroTier node, and reached the ONLINE state.

The device then started looking for the remote peer.

The peer identity was obtained through WHOIS, after which ZeroTier established the required path.

The sequence looked approximately like this:

ONLINE
   ↓
WHOIS → peer identity
   ↓
PATH up
   ↓
USER_MESSAGE
Enter fullscreen mode Exit fullscreen mode

Once the connection was established, the ESP32 sent several USER_MESSAGE packets and received replies from ZtSharp:

hello from esp32:0
hello from esp32:1
hello from esp32:2
hello from esp32:3
hello from esp32:4
hello from esp32:5
Enter fullscreen mode Exit fullscreen mode

The overall topology was:

ESP32-C6
   │
   ▼
Wi-Fi / IP
   │
   ▼
ZeroTier VL1
   │
   ├── WHOIS
   ├── discovery
   └── path
   │
   ▼
USER_MESSAGE
   │
   ▼
ZtSharp
Enter fullscreen mode Exit fullscreen mode

The ESP32 did not join a virtual ZeroTier network and did not receive a ZeroTier IP address through VL2.

For this communication model, VL1 and USER_MESSAGE were enough.

The bootstrap process still uses the standard ZeroTier Planet infrastructure for discovery. The ESP32 does not need to be configured as a member of a traditional ZeroTier virtual network.

Memory Usage

Memory consumption is one of the important questions when bringing a protocol stack to an ESP32.

During the test, the free heap changed approximately as follows:

  • after Wi-Fi initialization: ~303 KiB free;
  • after transport initialization: ~222 KiB;
  • after creating the ZeroTier node: ~192 KiB;
  • after reaching ONLINE: ~183 KiB;
  • minimum during the run: 180,664 bytes;
  • after completing the message exchanges: 228,816 bytes free.

Most of the overall memory usage still comes from the ESP-IDF infrastructure itself, especially the Wi-Fi driver and lwIP.

The sizeprobe utility reports the sizes of individual C++ objects using sizeof:

Node          = 4208 bytes
World         =  192 bytes
Identity      =   80 bytes
Salsa20       =   64 bytes
InetAddress   =   28 bytes
Peer          = 1344 bytes
Path          =  376 bytes
Enter fullscreen mode Exit fullscreen mode

For the runtime structures used in the test:

Trace           =    48 bytes
Switch          = 12592 bytes
Topology        =  1824 bytes
SelfAwareness   =    32 bytes
--------------------------------
total           = 14512 bytes
Enter fullscreen mode Exit fullscreen mode

These are sizeof values and therefore do not represent the complete runtime heap footprint. Dynamic allocations, buffers, container storage, and fragmentation also contribute to the real memory usage.

The largest structure is Switch, which is not particularly surprising because it is a central component of VL1 traffic processing.

At the end of the test, the main FreeRTOS task had 5,680 bytes free out of 28,672 bytes according to its stack high-water mark. The ZeroTier UDP task had almost 10 KB of free stack.

Firmware Size

The final firmware image for the test configuration was:

1,177,048 bytes — approximately 1.12 MiB.

A simplified breakdown:

Flash Code      1,063,670 B
.text             777,450 B
.rodata           247,096 B
.eh_frame          38,584 B
Enter fullscreen mode Exit fullscreen mode

DIRAM:

Used       140,830 B
Total      452,112 B
Usage       31.15 %
Free       311,282 B
Enter fullscreen mode Exit fullscreen mode

LP SRAM:

Used             24 B
Total        16,384 B
Usage          0.15 %
Enter fullscreen mode Exit fullscreen mode

These numbers describe the entire firmware, not the ZeroTier VL1 implementation alone.

The image contains:

  • ESP-IDF;
  • Wi-Fi;
  • lwIP;
  • FreeRTOS;
  • diagnostic code;
  • the ESP32-specific ZtLite integration.

So the firmware size should not be interpreted as the size of the ZeroTier core itself.

Still, combined with the runtime measurements, the result demonstrates an important point:

A VL1-only ZeroTier endpoint can realistically run on an ESP32-C6 while leaving a useful amount of RAM for the application.

What Has Actually Been Achieved?

The goal was not to create yet another incomplete "ZeroTier port for ESP32".

Instead, the roles of the components are deliberately separated.

ZtSharp uses ZeroTier as the foundation for a complete managed network:

VL1 + VL2 + managed TCP/IP stack + socket API
Enter fullscreen mode Exit fullscreen mode

ZtLiteESP uses only the parts that an embedded device actually needs:

identity
discovery
path establishment
cryptography
USER_MESSAGE
Enter fullscreen mode Exit fullscreen mode

And ZtGenStorage moves expensive identity/bootstrap preparation outside the MCU.

The main conclusion is:

ZeroTier can be used not only as a virtual Ethernet network, but also as a secure P2P transport layer.

For a server, a complete managed TCP/IP stack can be built on top of it.

For a small microcontroller, it can be enough to keep VL1 and exchange small messages directly between peers.

The same protocol foundation can therefore serve both endpoints:

                 ZeroTier VL1
                      │
          ┌───────────┴───────────┐
          │                       │
       ZtSharp                 ZtLiteESP
          │                       │
     VL1 + VL2                 VL1 only
          │                       │
   Managed TCP/IP            USER_MESSAGE
          │                       │
        .NET                   ESP32-C6
Enter fullscreen mode Exit fullscreen mode

For applications that only need telemetry, commands, remote control, or small peer-to-peer messages, this can be significantly simpler than deploying a complete VPN/IP stack on the MCU.

Projects

The source code and tools are available on GitHub:

ZtSharp

Fully managed ZeroTier implementation for C#/.NET with VL1/VL2, a managed TCP/IP stack, and socket APIs.

https://github.com/smersh1307n2/ZtSharp

ZtLiteESP

Minimal VL1-only ZeroTier client for ESP32 with direct peer-to-peer USER_MESSAGE communication.

https://github.com/smersh1307n2/ZtLiteESP

ZtGenStorage

Tool for preparing ZeroTier identity and planet data before flashing the device.

https://github.com/smersh1307n2/ZtGenStorage

Final Thoughts

The interesting part of this experiment is not simply getting ZeroTier to run on an ESP32.

The more interesting part is realizing that ZeroTier's VL1 can be useful as an independent secure P2P transport, without necessarily bringing the complete virtual Ethernet and IP stack to every embedded device.

For embedded systems that only need a secure path for telemetry and commands, this architecture can be much lighter than deploying a complete virtual network stack.

Top comments (0)