ICP Vetkey Quickstart Guide

Artur Shirokov — Thu, 03 Apr 2025 19:21:01 +0000

Vetkeys Demystified: Encrypting Data in Your ICP App

If you're building an app on the Internet Computer (ICP) that requires
encrypting user data, you've likely heard about vetkeys. They offer a powerful,
decentralized way to manage encryption keys without relying on centralized key
servers. While the docs are alright they are by no means perfect and most of the information here I had to personally collect and experiment to get (first of many shoutouts to kristofer and his eth demo app).

I'll be straight with you, his guide is for developers looking to implement vetkey encryption in their ICP apps, if you are non-technical and hoping to get and overview of what vetkeys are and their use-cases this is not going to be covered in this guide, maybe I'll make one in the future, stay tuned. I'll walk through the core concepts and provide practical code
examples based on my experience building Canister Cloud, a full-stack
application using Rust for the backend and Svelte for the frontend. We'll focus
on the fundamental flow: encrypting data for storage and decrypting it upon
retrieval.

Most of the setup and implementation details will be useful regardless of your
specific stack, though you might need to adapt data types and logic flows
slightly depending on what you want to build. My goal is to ease the journey for a dev to implement vetkeys in their app by highlighting the key steps to implementing it, the challenges I ran into and explaining which parts are important and which are not.

What You'll Need

An ICP full-stack project (dfx).
A backend canister (Rust examples provided).
A frontend framework (Svelte examples provided).
The ic-vetkd-utils frontend package.
Basic familiarity with ICP development concepts (canisters, principals, Candid).

Core Concepts (Simplified)

Vetkeys utilize Identity-Based Encryption (IBE). Instead of generating a
traditional public/private key pair for each piece of data, IBE allows
deriving unique keys based on:

A Master Public Key: Provided by the vetkey system on ICP.
A Derivation ID: A unique identifier, often the user's Principal, which ties the derived key to a specific identity, but you can use whatever you want for it based on the business logic of your app.

The Basic Flow:

Encryption (Frontend):
1. Get the vetkd system's master public key from your backend.
2. Use the user's Principal (as bytes) as the derivation ID.
3. Use the ic-vetkd-utils library to encrypt the data using the master public key and the derivation ID.
4. Store the resulting encrypted blob.
Decryption (Frontend):
1. To decrypt data encrypted with a specific derivation ID (e.g., the user's Principal), you need the corresponding derived private key.
2. You can't get this key directly from the vetkey canister as that's not safe and your private key will be exposed publicly. Instead, you generate a temporary transport key pair (public/private) in the frontend.
3. You send the transport public key and the derivation ID (user's Principal) to your backend.
4. The backend asks the vetkd system API to derive the private key for that derivation ID and encrypt it using the provided transport public key.
5. The backend sends this encrypted derived key back to the frontend.
6. The frontend uses its transport private key to decrypt the derived key.
7. Finally, the frontend uses this decrypted derived key to decrypt the actual data blob.

This might sound complex, but it ensures that only the user associated with the
derivation ID can ultimately obtain the necessary key to decrypt their data,
facilitated securely by the backend and the vetkd system.

Backend Setup

1. Configure `dfx.json`

You need to tell dfx about the vetkd system canister. Add the following to
your canisters object in dfx.json:

"vetkd_system_api": {
  "candid": "./vetkeys/chainkey_testing_canister.did", // Adjust path if needed
  "type": "custom",
  "wasm": "./vetkeys/chainkey_testing_canister.wasm", // Adjust path if needed
  "declarations": {
    "output": "src/declarations/vetkd_system_api" // Or your preferred declarations path
  }
},

(Make sure you have the .did and .wasm files for the vetkd system API,
often provided in ICP examples or starter projects, you can download them from our repo).

2. Define Vetkey Methods in `service.did`

Expose the necessary vetkey methods in your backend canister's Candid interface
(.did file):

type VetkdEncryptedKeyResponse = variant {
  Ok : blob;
  Err : text;
};

type VetkdPublicKeyResponse = variant {
  Ok : blob;
  Err : text;
};

service : {
  // ... other methods
  vetkd_public_key : () -> (VetkdPublicKeyResponse);
  vetkd_encrypted_key : (blob, opt nat64) -> (VetkdEncryptedKeyResponse); // blob is transport_pubkey, opt nat64 is file_id
  // ... other methods
}

3. Implement Backend Rust Functions

You need Rust functions in your backend canister to interact with the vetkd
system API.

a) Getting the Master Public Key:

This function fetches the main vetkd public key used for encryption.

// backend/src/vetkd/controller/vetkd_public_key.rs
use crate::declarations::vetkd_system_api::{
    vetkd_system_api, VetkdCurve, VetkdPublicKeyArgs, VetkdPublicKeyArgsKeyId,
};
use ic_cdk::update;

#[update]
async fn vetkd_public_key() -> Result<Vec<u8>, String> {
    let args = VetkdPublicKeyArgs {
        key_id: VetkdPublicKeyArgsKeyId {
            // IMPORTANT: Use a production key ID in a real application!
            name: "insecure_test_key_1".to_string(),
            curve: VetkdCurve::Bls12381G2,
        },
        derivation_path: vec![], // Not typically needed for basic IBE
        canister_id: None,       // Use the canister's own ID
    };

    // Call the system API
    let (result,) = vetkd_system_api.vetkd_public_key(args).await.map_err(|e| format!("vetkd_public_key failed: {:?}", e))?;

    Ok(result.public_key.to_vec())
}

Note: This example uses "insecure_test_key_1". For production, you'll need
to use a proper key ID like "key_1" or "test_key_1" depending on the network
and desired security level.

b) Getting the Derived Encrypted Key:

This function handles the frontend's request for the derived key needed for
decryption. It takes the frontend's transport public key and derives the key
based on the file owner's principal (looked up via file_id).

// backend/src/vetkd/controller/vetkd_encrypted_key.rs
use crate::declarations::vetkd_system_api::{
    vetkd_system_api, VetkdCurve, VetkdDeriveEncryptedKeyArgs, VetkdDeriveEncryptedKeyArgsKeyId,
};
use crate::with_state; // Assuming you have a way to access shared state
use ic_cdk::update;
use serde_bytes::ByteBuf;

#[update]
async fn vetkd_encrypted_key(
    encryption_public_key: Vec<u8>, // Frontend's transport public key
    file_id: Option<u64>,           // ID to find the owner principal
) -> Result<Vec<u8>, String> {

    // Determine the derivation ID (owner's principal)
    let derivation_id_bytes = if let Some(id) = file_id {
        // Look up the file's owner principal from your state/metadata
        with_state(|state| {
            state
                .file_data // Assuming file_data map exists in your state
                .get(&id)
                .map(|file| file.metadata.requester_principal.as_slice().to_vec()) // Adjust field access as needed
                .ok_or_else(|| format!("File not found for ID: {}", id))
        })?
    } else {
        // Fallback or error if file_id is required for decryption context
        // For simplicity, let's assume file_id is always provided for decryption
         return Err("File ID is required to determine the correct derivation ID".to_string());
        // Alternatively, use caller if appropriate for your logic:
        // ic_cdk::api::caller().as_slice().to_vec()
    };

    let args = VetkdDeriveEncryptedKeyArgs {
        key_id: VetkdDeriveEncryptedKeyArgsKeyId {
            // IMPORTANT: Use the same key ID as in vetkd_public_key!
            name: "insecure_test_key_1".to_string(),
            curve: VetkdCurve::Bls12381G2,
        },
        derivation_path: vec![],
        derivation_id: ByteBuf::from(derivation_id_bytes), // Owner's principal
        encryption_public_key: ByteBuf::from(encryption_public_key), // Frontend's transport key
    };

    // Call the system API
    let (result,) = vetkd_system_api
        .vetkd_derive_encrypted_key(args)
        .await
        .map_err(|e| format!("vetkd_derive_encrypted_key failed: {:?}", e))?;

    Ok(result.encrypted_key.to_vec())
}

Remember to expose these Rust functions in your backend/src/lib.rs or
main.rs using #[ic_cdk::update] macros so they are callable from the
frontend.

Frontend Setup

1. Install `ic-vetkd-utils`

Add the package to your package.json. You might be using a .tgz file (you can download one from our repo) if it's not yet published or if you're using a specific version:

// package.json (dependencies)
"ic-vetkd-utils": "file:./ic-vetkd-utils-0.1.0.tgz", // Or the appropriate version/source

Run npm install or pnpm install.

2. Create a Vetkey Service

It's good practice to encapsulate the vetkey logic in a dedicated service class.

// frontend/src/frontend/src/lib/vetkeys/vetkdCrypto.ts
import * as vetkd from "ic-vetkd-utils";
import type { ActorType } from "$lib/shared/actor"; // Your backend actor type

export class VetkdCryptoService {
    constructor(private actor: ActorType) {}

    // Encryption method (details below)
    async encrypt(
        data: ArrayBuffer,
        userPrincipalBytes: Uint8Array,
    ): Promise<Uint8Array> {
        // ... implementation ...
    }

    // Decryption method (details below)
    async decrypt(
        encryptedData: Uint8Array,
        userPrincipalBytes: Uint8Array, // Current user's principal
        fileId: bigint, // Needed to get the correct derived key from backend
    ): Promise<Uint8Array> {
        // ... implementation ...
    }
}

Implementation: Encrypting & Uploading

Here's the step-by-step flow for encrypting a file before uploading:

Step 1: Get Vetkd Public Key

Your frontend needs the master public key from the backend. This is usually done
once or periodically (whenever you need to encrypt something).

// Inside an async function where you have access to your actor
const publicKeyResponse = await this.actor.vetkd_public_key();
if (!publicKeyResponse || "Err" in publicKeyResponse) {
    throw new Error(
        "Error getting public key: " +
            ("Err" in publicKeyResponse
                ? publicKeyResponse.Err
                : "empty response"),
    );
}
const vetkdPublicKey = publicKeyResponse.Ok as Uint8Array;

Step 2: Prepare Data and Derivation ID

Get the file content as an ArrayBuffer and the current user's principal as
Uint8Array.

const fileBytes: ArrayBuffer = await file.arrayBuffer(); // file is a File object
const userPrincipalBytes: Uint8Array = auth.authClient.getIdentity()
    .getPrincipal().toUint8Array(); // Assuming 'auth' service

Step 3: Encrypt using ic-vetkd-utils

Use the VetkdCryptoService.encrypt method, which wraps the ic-vetkd-utils
library call:

// frontend/src/frontend/src/lib/vetkeys/vetkdCrypto.ts (encrypt method)
async encrypt(
  data: ArrayBuffer,
  userPrincipalBytes: Uint8Array, // This is our Derivation ID
): Promise<Uint8Array> {
  try {
    // 1. Get vetkd public key (as shown in Step 1 above)
    const publicKeyResponse = await this.actor.vetkd_public_key();
    // ... (handle errors) ...
    const vetkdPublicKey = publicKeyResponse.Ok as Uint8Array;

    // 2. Generate a random seed (required by the library)
    const seed = window.crypto.getRandomValues(new Uint8Array(32));

    // 3. Ensure data is Uint8Array
    const encodedMessage = new Uint8Array(data);

    // 4. Encrypt using IBE
    const encryptedData = vetkd.IBECiphertext.encrypt(
      vetkdPublicKey,
      userPrincipalBytes, // Derivation ID
      encodedMessage,
      seed,
    );

    // 5. Serialize the result for storage/transport
    return encryptedData.serialize();
  } catch (error) {
    console.error("Encryption error:", error);
    throw error;
  }
}

Step 4: Upload Encrypted Blob

Your upload service takes the encryptedData (the Uint8Array returned by
encrypt) and uploads it, likely in chunks or however else you want to handle your data.

// frontend/src/frontend/src/lib/services/upload.ts (simplified relevant parts)
import { VetkdCryptoService } from "../vetkeys/vetkdCrypto";
// ... other imports ...
export const CHUNK_SIZE = 2_000_000;

export class UploadService {
  // ... constructor, auth state ...
  private vetkdCryptoService: VetkdCryptoService;

  async uploadFile({ file, ... }: { file: File, ... }) {
    // ... get userPrincipalBytes ...
    const fileBytes = await file.arrayBuffer();

    // Encrypt the data
    const encryptedData = await this.vetkdCryptoService.encrypt(
      fileBytes,
      userPrincipalBytes,
    );

    // Chunk and upload the encryptedData
    const numChunks = Math.ceil(encryptedData.length / CHUNK_SIZE);
    const firstChunk = encryptedData.subarray(0, CHUNK_SIZE);

    // Call backend to start upload (atomic or first chunk)
    const fileId = await this.auth.actor.upload_file_atomic({ // Or upload_file
      content: firstChunk,
      name: file.name,
      file_type: file.type,
      num_chunks: BigInt(numChunks),
    });

    // Upload remaining chunks in parallel
    await this.uploadChunks(encryptedData, fileId, onChunkUploaded);

    // ... handle completion/errors ...
  }

  private async uploadChunks(content: Uint8Array, fileId: bigint, ...) {
     // ... logic to upload chunks from index 1 onwards using upload_file_continue ...
  }
}

Implementation: Downloading & Decrypting

Now, the reverse process:

Step 1: Download Encrypted Blob

Fetch all the encrypted chunks from the backend and merge them into a single
Uint8Array.

// frontend/src/frontend/src/lib/services/decrypt.ts (simplified relevant parts)
export class DecryptService {
  // ... constructor, auth state ...

  async decryptFile({ fileId }: { fileId: bigint }): Promise<...> {
    // ... get file metadata ...

    // Download chunk 0
    let downloadResponse = await this.auth.actor.download_file(fileId, 0n);
    // ... handle errors ...
    let mergedEncryptedData = downloadResponse.found_file.contents;
    const totalChunks = Number(downloadResponse.found_file.num_chunks);

    // Download remaining chunks (1 to n-1)
    for (let i = 1; i < totalChunks; i++) {
      const chunkResponse = await this.auth.actor.download_file(fileId, BigInt(i));
      // ... handle errors ...
      const chunkData = chunkResponse.found_file.contents;

      // Merge chunks
      const temp = new Uint8Array(mergedEncryptedData.length + chunkData.length);
      temp.set(mergedEncryptedData, 0);
      temp.set(chunkData, mergedEncryptedData.length);
      mergedEncryptedData = temp;
    }

    // Now 'mergedEncryptedData' holds the full encrypted file content
    // ... proceed to decryption ...
  }
}

Step 2: Generate Ephemeral Transport Keys

Create a temporary key pair in the frontend.

// Inside VetkdCryptoService.decrypt or where decryption is initiated
const seed = window.crypto.getRandomValues(new Uint8Array(32));
const transportSecretKey = new vetkd.TransportSecretKey(seed);
const transportPublicKeyBytes = transportSecretKey.public_key();

Step 3: Request Derived Key from Backend

Call the backend vetkd_encrypted_key endpoint, passing the transport public
key and the fileId (so the backend knows which owner's principal to use for
derivation).

// Inside VetkdCryptoService.decrypt
const privateKeyResponse = await this.actor.vetkd_encrypted_key(
    transportPublicKeyBytes,
    [fileId], // Pass fileId as Option<nat64>
);
// ... handle errors ...
const encryptedDerivedKey = privateKeyResponse.Ok as Uint8Array;

Step 4: Decrypt Derived Key

Use the transport private key to decrypt the response from the backend. You
also need the master vetkd public key and the original derivation ID (the
user's principal in this case) used for encryption.

// Inside VetkdCryptoService.decrypt (simplified for non-shared files)

// Get vetkd public key (if not already available)
const publicKeyResponse = await this.actor.vetkd_public_key();
// ... handle errors ...
const vetkdPublicKey = publicKeyResponse.Ok as Uint8Array;

// Get current user's principal (used as derivation ID during encryption)
const userPrincipalBytes = this.auth.authClient.getIdentity().getPrincipal()
    .toUint8Array();

// Decrypt the key received from the backend
const decryptedDerivedKey = transportSecretKey.decrypt(
    encryptedDerivedKey,
    vetkdPublicKey,
    userPrincipalBytes, // The derivation ID used to encrypt
);

(Note: The actual decrypt code in your repo handles shared files by
potentially using the owner's principal here. For this guide, we assume the user
is decrypting their own file).

Step 5: Decrypt Data

Finally, use the decryptedDerivedKey to decrypt the actual file content.

// Inside VetkdCryptoService.decrypt
const ibeCiphertext = vetkd.IBECiphertext.deserialize(mergedEncryptedData);
const decryptedDataBytes = ibeCiphertext.decrypt(decryptedDerivedKey);

// decryptedDataBytes is the original file content as Uint8Array
return decryptedDataBytes;

The DecryptService would call this decrypt method after downloading the
chunks.

// frontend/src/frontend/src/lib/services/decrypt.ts (calling decrypt)
// ... after merging chunks into mergedEncryptedData ...

try {
    const userPrincipalBytes = this.auth.authClient.getIdentity().getPrincipal()
        .toUint8Array();

    // Decrypt using the service
    const decryptedData = await this.vetkdCryptoService.decrypt(
        mergedEncryptedData as Uint8Array,
        userPrincipalBytes,
        fileId,
    );

    // Return the decrypted content
    return {
        // ... name, dataType, etc. ...
        contents: decryptedData.buffer as ArrayBuffer,
    };
} catch (err) {
    console.error("Decryption failed:", err);
    // Handle decryption error
}

Key Considerations & Tips

Experimental: Vetkeys are still somewhat experimental. APIs and libraries might change. In the month it took me to write this guide there were already several updates and helpful packages published.
Test Keys: Remember to switch from test key IDs (like "insecure_test_key_1") to appropriate production key IDs ("key_1") when deploying.
Derivation ID: The choice of derivation ID is critical. Using the user's principal is common for encrypting user-specific data. Ensure you consistently use the correct ID for both encryption and decryption key derivation requests.
Error Handling: Robust error handling is essential, especially around backend calls and the decryption process, as failures can leave data inaccessible.

Conclusion

Vetkeys provide a native ICP solution for decentralized encryption key
management. While the flow involves a few steps, particularly for decryption, it
eliminates the need for external key servers and ties data access directly to
user identity on the Internet Computer. By setting up the backend endpoints and
leveraging the ic-vetkd-utils library on the frontend, you can implement
robust end-to-end encryption for your application's data.

If you have any questions, anything you didn't understand or just want help with your project please reach out to me and I'll try to help whatever way I can (check the My Contacts section below)

Check out the Canister Cloud for the full codebase context.

My contacts

Telegram: Nth_typonomy
Email: artur.wiseman@proton.me
Discord: ahegao_burger

Coming in the Future

Implementing file sharing guide with vetkeys (handling decryption for recipients).
Minimal MVP app demonstrating vetkey functionality.
Non-technical guide about possibilities and use-cases for vetkey encryption

I hope this guide helps you integrate vetkey encryption into your own ICP
projects!

Choosing the right tools for the job

Artur Shirokov — Sat, 18 May 2024 21:15:46 +0000

It's crucial for a new developer to choose the right tools for the job and not the right job for the tools. This expression is very common, but it rings true in our profession. It’s particularly relevant to new and beginner developers who are in the early stages of their careers and deciding what to learn and where to focus their efforts. This moment is pivotal in a developer's professional journey, as choosing the right tools can set the foundation for future success.

So, how do you actually go about choosing the right tools? Well, there are a couple of ways to do it, and I'll speak from my personal experience because I believe that is the most efficient way and probably the one that will resonate with most people since it's based on real experience.

Initially, my approach was flawed. I’m someone who is easily influenced by others, especially those more experienced than me. My first decision was based exactly on what others recommended. That choice was SvelteKit. I don’t regret it—SvelteKit offers a fantastic developer experience. However, as developers, we aren't the end-users of our own services. The true end-users are the clients or the companies we work for, or in the case of freelancers like me, our clients.

Acknowledging that we are not the primary consumers of our services, we need to think about who is. Whether it's a company or a client, we can look at statistics on the most demanded tools in the industry. Tools like React, Next.js, and Python for the backend are highly sought after. In contrast, Svelte and SvelteKit, while excellent technologies, don't have the same level of demand.

It's crucial to take great care and thought when deciding which tools to invest your time and resources in learning and mastering. You can either approach this smartly, using statistics and a good understanding of the industry, or you can do what I did—listen to others and end up having to learn multiple tools to reach the same level of proficiency you could have achieved with one. The tool I ultimately settled on is Next.js. It shares many similarities with SvelteKit but differs in some aspects, notably in the job market where Next.js has far more opportunities.

Dear colleagues, I hope you learned something from this post. Have a great day, and thank you for reading.

DEV Community: Artur Shirokov