The Story of How I Built a VPN protocol: Part 1

Smile — Fri, 01 May 2026 22:21:13 +0000

🚨🚨🚨 Disclaimer 🚨🚨🚨

This article and the VPN itself are written for educational purposes only.

How It All Started

I recently switched to Arch. Everything started off well: I installed all the utilities I needed, and then I decided to install the VPN I used to use. And then a problem appeared — it doesn't work on Arch (even as an AppImage).

My provider also supported Shadowsocks, but instead of using it, I decided to write my own VPN. For more practice.

VPN Protocol

My VPN protocol is designed for maximum stealth. In my opinion, one of the most important things here is encryption from the very first packet. In my protocol, this is implemented just like in Shadowsocks — with a pre-shared key.

Encryption algorithm: ChaCha20-Poly1305.

It's also worth mentioning that the protocol works over TCP. A random amount of junk bytes is added to each packet for length obfuscation.

Packet Structure

Each packet has a 5-byte header that is masked as encrypted data using XOR with the first 5 bytes of the key.

First 2 bytes — total packet length. Needed to determine where the packet ends (since TCP can segment packets).
Third byte — flags byte. Currently only 2 flags are used:
- Bit 1 — indicates that this packet is fake and should not be processed (not yet implemented).
- Bit 2 — flag for performing ECDH (Elliptic Curve Diffie‑Hellman).
Last 2 bytes — ciphertext length, used to separate junk bytes from the ciphertext.

Then comes:

12 bytes — randomly generated nonce;
ciphertext;
AEAD (authentication tag);
junk bytes.

Handshake and Key Exchange

1. First packet from the client

The client sends its 16-byte username to the server (encrypted, of course).

2. Server response

If the server finds a user with that username, it:

sends the client a randomly generated 32-byte salt;
starts computing the keys:
- sending key (server → client)
- receiving key (server ← client)

3. Key computation on the server

The server stores the user's password in plaintext.

Receiving key (for decrypting from the client) = hash(password + first 16 bytes of salt).
Sending key (for encrypting to the client) = hash(password + last 16 bytes of salt).

4. Client actions

The client receives the salt, decrypts it, and does the same thing, but the key roles are inverted:

what is the sending key for the server becomes the receiving key for the client, and vice versa.

5. ECDH and connection finalization

After the client has generated the keys, it generates an ephemeral key pair based on the Curve25519 elliptic curve (this pair is needed for ECDH). It then sends a connection confirmation (first byte = 0xFF) along with the public ephemeral key, setting the ECDH flag.

The server receives the packet, deobfuscates it, and gets the confirmation and the client's ephemeral key. Then it:

assigns an IP address to the client from a local private network;
generates its own ephemeral key pair;
sends the client its assigned IP address and the server's public key;
performs the ECDH round.

After sending, the server updates its keys by hashing the old keys with the secret obtained from ECDH.

6. Client finalization

After receiving the packet with the IP address and the server's public ephemeral key, the client:

creates a local tunnel;
sets its IP address (received from the server);
performs the ECDH round;
updates its keys.

Main Work Loop

After the connection is established and keys are generated, the main work loop begins.

Client Side

3 goroutines run on the client side:

First goroutine (reading from the tunnel and preparing packets)

Reads packets from the tunnel.
Generates an 8-byte salt to update the sending key (by hashing the old sending key with the salt).
Adds this 8-byte salt to the beginning of the plaintext (the salt is followed by the packet read from the tunnel).
Encrypts everything.
Adds random junk bytes for obfuscation.
Stores the prepared packet in a buffer.

Second goroutine (sending packets)

Responsible for sending already prepared packets.
Packets are sent in batches of 1 to 5 packets (the protocol is of course segmented at OSI layers 3 and 4, but I can't influence that).

Third goroutine (receiving packets from the server)

Responsible for receiving packets from the server.
Performs deobfuscation and decryption.
Writes the decrypted data to the tunnel.

Server Side

The server has 3 main goroutines, plus additional goroutines for receiving packets from clients.

First goroutine (handshake handling)

Handles incoming handshake requests from clients. If the handshake is successful, a new goroutine is created to process packets sent by that client.

Second goroutine (reading from the tunnel)

Reads packets from the tunnel and sends them to clients.

Third goroutine (cleaning inactive connections)

Cleans up inactive connections.

Key Updates

Salt in every packet

Every packet (whether from client or server) contains a salt. It is used to update the keys:

The server, when sending a packet, includes a salt. After sending, it updates its sending key by hashing the old key with that salt.
The client, when receiving and decrypting a packet, also updates a key — but not the sending key, the receiving key.
When the client sends a packet, the same happens, but the roles are reversed.

Periodic ECDH updates

Every 4 minutes or after sending 2³² packets (whichever comes first), keys are updated using ECDH on elliptic curves. The keys are transmitted along with data packets.

And that, in fact, is the entire protocol. During implementation, I thought about writing it in Go or Rust. I chose Go for its simplicity.

Implementation Process

To be honest, the protocol architecture was mostly developed while writing the code. It has quite a few problems — both in terms of protocol design and implementation.

Example problems

Constant username packet length

The encrypted username packet has a constant length of 44 bytes (12 bytes nonce, 16 bytes ciphertext, and a 16-byte AEAD tag). Knowing this and that the user is using this protocol, you can calculate the 4th and 5th bytes of the key.
Repository duplication

I foolishly created two separate repositories — one for the client and one for the server. As a result, the branches containing common modules just duplicate each other.
Git flow

I tried to follow git flow, but failed here too.
Vulnerabilities

I also have a feeling that there are more vulnerabilities in the code than working logic.
No graceful shutdown

There is no proper negotiated client-server disconnect — just a connection break.

Although considering this is my first project, I think it didn't turn out too badly. If anyone wants to check out this mess, here are the links:

Client: https://github.com/SmileUwUI/smileTun-client
Server: https://github.com/SmileUwUI/smileTun-server

Currently, the implementation works. And I'm writing this article through my own VPN protocol.

Future Plans

Merge both repositories into one.
Add fake packet sending.
Add TLS mimicry.
And much more.

If anyone has any questions or recommendations — leave them in the comments. For now, I bid you farewell. Good luck to everyone!

DEV Community: Smile