DEV Community

Cover image for Storage solutions with Midnight - handling off-chain data with privacy
Levai!
Levai!

Posted on

Storage solutions with Midnight - handling off-chain data with privacy

#zk #midnightnetwork #zeroknowledge #privacy

Blockchain storage is a premium resource. While storing data directly on the ledger ensures immutability, it is often prohibitive for large files and can compromise user privacy if sensitive information is leaked.

On the Midnight network, storing large files directly on-chain is impractical and expensive. This is where off-chain storage comes in. You store your data off-chain, then store the proof of that data on-chain.

In this guide, you will learn how to integrate off-chain storage solutions like IPFS (InterPlanetary File System) or Arweave with Midnight smart contracts. You will build a document management system that encrypts files before upload, stores them off-chain, and uses Midnight's shielded state to manage access and verify data integrity.

By the end of this guide, you will:

  • Understand the strategic trade-off between on-chain and off-chain storage and when to use both.
  • Learn how to use content hashes and cryptographic commitments to verify data integrity.
  • Encrypt sensitive data using AES-256-GCM before uploading to IPFS or Arweave.
  • Implement access control using derived viewing keys and hybrid encryption.
  • Build and deploy a complete document management system on Midnight.

Prerequisites

Before starting this tutorial, ensure you have the following:

On-chain vs. off-chain storage

As with building anything, before you begin writing code, you must first decide where your data lives. There are two options for storing data: on-chain and off-chain. These are not interchangeable as they serve fundamentally different purposes.

On-chain storage refers to the data storage system on the ledger itself. The ledger provides an immutable and privacy-preserving state; it is not designed to hold high-volume data like PDFs, high-resolution images, or massive data records.

Off-chain storage, on the other hand, refers to any system that exists outside the ledger. Instead of overloading the ledger with large files and incurring massive costs, you store the files in an external system (like IPFS) and store a small pointer or "commitment" on-chain. This pointer serves to prove that the off-chain data exists and has not been tampered with.

To determine what belongs on-chain, follow this simple rule:
"If the data is essential for the smart contract to function, then it belongs on-chain."

Such data includes:

  • Ownership state (Who owns the document?)
  • Cryptographic commitments (Hashes of the content)
  • Metadata required for zero-knowledge (ZK) proofs

Everything else belongs off-chain. This includes raw files like PDFs or images, large bodies of text, and sensitive records that only require verification or retrieval rather than on-chain computation.

Criteria On-Chain Off-Chain
Use Case Access rules, ownership proofs, commitments Raw files, large data, retrievable records
Capacity Limited (kilobytes) Unlimited (gigabytes+)
Privacy Selective disclosure Encrypted before upload for privacy
Persistence Permanent, immutable Depends on provider
Latency Consensus required Immediate retrieval

Building the document manager

Now that you understand what belongs on-chain and what belongs off-chain, it is time to build. You will create a system where file metadata is shielded on Midnight, while the files themselves are encrypted and stored off-chain.

A repository has been prepared with the complete code for this project. This guide walks through the implementation of off-chain storage as demonstrated in the repository.

Step 1: Project setup

First, clone the repository and install dependencies:

git clone https://github.com/Mackenzie-OO7/midnight-doc-manager.git
cd midnight-doc-manager
npm install
Enter fullscreen mode Exit fullscreen mode

Configure the environment

Configure the connection to the storage provider (Pinata) and the Midnight network.

# Copy the example configuration file
cp .env.example .env
Enter fullscreen mode Exit fullscreen mode
Setting up Pinata (IPFS):
  1. Log in to your Pinata Dashboard (https://app.pinata.cloud).
  2. In the sidebar on your left under the "Developer" section, select API Keys and click New Key.
  3. Input a name for your new API key and enable the admin checkbox. Then click Create.
  4. Copy the JWT (the long string) and replace the placeholder value for PINATA_JWT in your .env file.
  5. Navigate to “Gateways” in your Pinata Dashboard. Copy your dedicated gateway URL (it looks like ‘https://your-name.mypinata.cloud’) and set it as PINATA_GATEWAY in your .env file.

Step 2: Understanding the smart contract

Head over to contracts/document-manager.compact. This file defines the rules of the system. It specifies what data is stored, and who can change it.

The contract handles three critical things:

Storing document metadata

The DocumentRecord struct links the off-chain file to its on-chain proof.

    struct DocumentRecord {
        contentHash: Bytes<32>,       // SHA-256 of the ORIGINAL file
        storageCid: Opaque<"string">, // IPFS Address (encrypted content)
        ownerCommitment: Bytes<32>,   // Proof of ownership (hidden)
        fileType: Opaque<"string">,   // MIME type (e.g., "application/pdf")
        isActive: Boolean             // For "soft deletes"
    }
Enter fullscreen mode Exit fullscreen mode
Proving integrity

The verifyDocument circuit checks if a provided file hash matches the one stored on the ledger. This allows anyone to verify a file's authenticity without trusting the storage provider.

    export circuit verifyDocument(documentId: Bytes<32>, providedHash: Bytes<32>): Boolean {
        const doc = documents.lookup(disclose(documentId));
        return doc.contentHash == disclose(providedHash) && doc.isActive;
    }
Enter fullscreen mode Exit fullscreen mode
Managing access control

The AccessGrant struct allows secure sharing. It stores an encrypted key that only the intended recipient can unlock.

    struct AccessGrant {
        encryptedKey: Opaque<"string">,
        nonce: Opaque<"string">,
        senderPublicKey: Opaque<"string">
    }
Enter fullscreen mode Exit fullscreen mode

The contract relies on the following components to manage state and privacy:

  • Ledger: The documents map stores records, and accessGrants stores permissions.
  • Witness Function: witness secretKey(): Bytes<32>. It allows your local wallet to supply a secret key to the contract for zero-knowledge proof generation, without ever revealing that key to the network.
  • Circuits: The contract defines the following actions:
    • registerDocument: Creates a new immutable record linked to the owner's private key.
    • updateDocument: Allows the owner to update the file hash or storage location.
    • deactivateDocument: Enables the owner to soft-delete a document, marking it as inactive.
    • verifyDocument: Publicly proves that a provided file matches the on-chain record without revealing the owner.
    • grantAccess: Securely stores an encrypted key for a specific recipient.
    • revokeAccess: Removes a shared key, effectively revoking access for that user.
    • hasAccess: Checks if an access grant exists for a given recipient.
    • getAccessGrant: Retrieves the encrypted key for an authorised recipient.
    • getDocument: Retrieves the full document record by ID.

Compile the smart contract

Next, compile the contract:

    npm run compile
Enter fullscreen mode Exit fullscreen mode

This generates the necessary TypeScript bindings and stores them in a new sub-directory contracts/managed/document-manager/.

Step 3: The application layer

Now that you have the smart contract, you need to build the application layer to interact with it.

From what is defined in the smart contract, there are three distinct features to implement:

  1. Encryption: First, secure the data itself. Create utilities to encrypt files client-side using AES-256-GCM.
  2. Key management: Second, handle the keys. Implement key derivation to generate specific keys for each user and document.
  3. Storage: Finally, store the data. Create providers for IPFS and Arweave to hold the encrypted data blobs.

Let's begin with encryption.

Encryption utility

If you simply uploaded files to IPFS, they would be public, and anyone with the CID could access them, including the storage provider. To create a private document manager, you must encrypt your data before it ever leaves the user's device.

The AES-256-GCM standard is widely considered the gold standard for symmetric encryption. It provides confidentiality, making the data impossible to read without the key, and integrity via an authentication tag. If even a single bit of the encrypted file is altered on IPFS, decryption fails instantly.

Here is the core logic from src/utils/encryption.ts:

    export function encryptFile(fileData: Buffer, key: Buffer): EncryptedData {
        // 1. Validate the key length
        if (key.length !== 32) throw new Error("Encryption key must be 32 bytes");

        // 2. Generate a unique IV (Initialisation Vector)
        const iv = randomBytes(12);

        // 3. Create the cipher and encrypt the data
        const cipher = createCipheriv("aes-256-gcm", key, iv);
        const ciphertext = Buffer.concat([cipher.update(fileData), cipher.final()]);

        // 4. Return ciphertext, IV, and Auth Tag (Integrity Proof)
        return { ciphertext, iv, authTag: cipher.getAuthTag() };
    }
Enter fullscreen mode Exit fullscreen mode

Uploading to IPFS packs these three components: iv, authTag, and ciphertext into a single binary blob. This ensures the downloader has everything needed to verify and decrypt the file, except the key.

Key management in Midnight

If I encrypt a file with a key on my machine, how do I let others read it without managing thousands of loose keys? And how do I prove ownership without exposing my wallet history?

The solution is a two-part key strategy. First, derive a deterministic key from your wallet seed for ownership proofs on-chain. Then generate an independent encryption key pair for sharing files securely.

A. The contract witness key

This key is derived using a SHA-256 hash of your wallet seed and a unique prefix. It serves as a private input (witness) for zero-knowledge proofs, allowing you to prove you own a document without revealing your identity. Below is the derivation from src/api/contract.ts:

    // src/api/contract.ts — inside the connect() method
    this.secretKeyBytes = new Uint8Array(
        createHash('sha256')
            .update('midnight-doc-manager:owner-key:')
            .update(seed)
            .digest(),
    );
Enter fullscreen mode Exit fullscreen mode

The witness function then supplies this key to the contract at proof generation time:

    // src/api/witnesses.ts
    export function createWitnesses(secretKey: Uint8Array) {
        if (secretKey.length !== 32) throw new Error("Secret key must be 32 bytes");
        return {
            secretKey(context: WitnessContext<any, DocumentManagerPrivateState>):
                [DocumentManagerPrivateState, Uint8Array] {
                return [context.privateState, secretKey];
            },
        };
    }
Enter fullscreen mode Exit fullscreen mode

To prove ownership without revealing the key itself, the application computes an ownership commitment using Midnight's persistentHash and not plain SHA-256, so the result matches the on-chain check:

    export function computeOwnerCommitment(secretKey: Uint8Array): Uint8Array {
        if (secretKey.length !== 32) throw new Error("Secret key must be 32 bytes");
        return persistentHash(BYTES32_DESCRIPTOR, secretKey);
    }
Enter fullscreen mode Exit fullscreen mode
B. The encryption identity key (viewing key)

This is your identity for file sharing. Unlike the witness key, it is not derived from your wallet seed. It is an independent X25519 keypair generated randomly via nacl.box.keyPair() and stored locally in .midnight-doc-keys.json. It has nothing to do with your mnemonic or wallet address. Your public key can be distributed so that others can encrypt files for you, while your private key remains secret and is used only to unwrap and decrypt keys shared with you. If you lose it, you cannot decrypt documents you have uploaded or that have been shared with you.

Why derive separate keys?

Ideally, you never want to use your raw wallet seed directly in an app. If that seed is compromised, your entire wallet is at risk. By deriving specific keys for specific tasks, you ensure that even if the app's keys are compromised, your funds and other DApps remain secure

wallet seed to file key flow

The sharing mechanism (key wrapping)

Now, how do we share files? You don't actually share the file itself. The file is already on IPFS and anyone can download the encrypted blob. What you need to share is the key to decrypt it.

Key Wrapping securely sends this key.

Consider this scenario:

  1. Alice (Sender) has the document, which is encrypted with a random AES Key; let's call it the File Key.
  2. Alice wants to share the File Key with Bob. She fetches Bob's Encryption Public Key.
  3. Alice uses her Encryption Private Key and Bob's Encryption Public Key to generate a secure shared secret.
  4. Alice uses this shared secret to encrypt the File Key. This encrypted key is called the Wrapped Key.
  5. Alice stores this Wrapped Key on the Midnight contract.

The result:
Bob can download the Wrapped Key from the contract. He uses his Encryption Private Key to "unwrap" it, revealing the File Key, which he then uses to decrypt the actual file from IPFS.

The storage layer

So far, you have implemented secure encryption to ensure files cannot be viewed publicly, and you have created viewing keys to grant read access to authorised parties. However, something important is missing: where do you actually store these files?

When it comes to decentralised storage, two solutions lead the way: IPFS and Arweave.

IPFS

IPFS allows you to store files using content addressing. Instead of a location (like myserver.com/file.pdf), the address is a hash of the content (the CID). If the file changes, the CID changes. This makes IPFS perfect for verification: store the CID on-chain. If the data on IPFS is tampered with, the CID changes immediately, alerting you to the discrepancy.

Arweave

Arweave offers permanent storage. You pay once, and the data is stored forever on the "blockweave." This is ideal for critical documents like legal contracts or deeds that must never be deleted.

In src/storage.ts, implement a flexible StorageProvider interface that supports both IPFS and Arweave so you can switch between them seamlessly:

    export interface StorageProvider {
        upload(data: Buffer, metadata?: Record<string, string>): Promise<string>;
        download(storageId: string): Promise<Buffer>;
        getGatewayUrl(storageId: string): string;
        delete?(storageId: string): Promise<void>;
    }
Enter fullscreen mode Exit fullscreen mode

Step 4: Deployment

Finally, let's put it all together. A CLI has been included in the repository to help you deploy and test the application.

You can either deploy locally via docker, or to the public preprod testnet.

Note that the mnemonic phrase you use for one network cannot be used on the other. Each network is separate and requires its own funded wallet.

Option A: The local network

The local network runs entirely in Docker and includes the node, indexer, and proof server. You will need two terminals open.

In terminal 1, start the local network:

    git clone https://github.com/midnightntwrk/midnight-local-dev
    cd midnight-local-dev
    npm install
    npm start
Enter fullscreen mode Exit fullscreen mode

Once the network is up and running, you will see the genesis master wallet logged. This is the network's internal funding source, not your wallet. You will then see the funding menu:

  [1] Fund accounts from the config file (NIGHT + DUST registration)
  [2] Fund accounts by public key (NIGHT transfer only)
  [3] Display wallets
  [4] Exit
Enter fullscreen mode Exit fullscreen mode

Select [1]. When prompted for a path, enter ./accounts.json. The repo ships with an accounts.json containing a pre-configured development wallet. You can use the mnemonic already there or replace it with your own. Note the mnemonic down as you will need it in Terminal 2.

Leave Terminal 1 running once funding is complete. Navigate to Terminal 2, where you will deploy the contract:

First, confirm the wallet was funded:


    npm run check-balance “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

You should see a non-zero NIGHT and DUST balance. Then deploy the contract:

    npm run deploy “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

On success, the contract address is printed and saved to deployment.json.

Option B: Preprod testnet

Preprod is Midnight's public testnet. You will need two terminals open. Ensure that you are in the midnight-doc-manager directory on both terminals.

In terminal 1, start the proof server:

    npm run proof-server
Enter fullscreen mode Exit fullscreen mode

In terminal 2, deploy the contract:

First, set .env to target preprod:

    MIDNIGHT_NETWORK=preprod
Enter fullscreen mode Exit fullscreen mode

Next, create a new Lace wallet and select 'Preprod' in Settings → Network. Lace will show you a 24-word mnemonic when you create the wallet. Save it, as this is what you will pass to all CLI commands.

Get your unshielded address:

    npm run check-balance “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

Get tNIGHT from the Midnight faucet by pasting your unshielded address. Wait for the transaction to confirm, then run the check-balance command again to verify the funds arrived and that DUST is non-zero before continuing.

Deploy the contract:

    npm run preprod “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

Deployment takes 60-120 seconds on preprod while waiting for block confirmations. On success, the contract address is saved to deployment.json.

Using the document manager

Once the contract is deployed, you can upload, retrieve, share, and verify documents. All commands use the same mnemonic you used to deploy.

1. Generate an encryption keypair

Create the keys you will use to share documents:

    npm run cli -- keys generate
Enter fullscreen mode Exit fullscreen mode

This saves a keypair to .midnight-doc-keys.json in the current directory and prints your public key. If you lose this file, you will not be able to decrypt documents you have uploaded.

2. Upload a document

With the smart contract deployed, you can now upload your first document. Create a test file and upload it:

    echo “Midnight is great.” > demo.txt
    npm run cli -- upload demo.txt “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

This command encrypts the file with AES-GCM, uploads the encrypted blob to IPFS, and records the content hash and ownership proof on the Midnight smart contract.

When the upload completes, the CLI prints a document ID. Save this value as you will need it for all subsequent commands.

3. Verify integrity 

    npm run cli -- verify demo.txt <DOCUMENT_ID> “<YOUR_MNEMONIC_PHRASE>”
Enter fullscreen mode Exit fullscreen mode

The CLI computes the SHA-256 hash of the local file and asks the smart contract to verify that the hash matches the immutable record on the ledger. A match proves the file is completely identical to what was uploaded.

For the full CLI reference, including download, file sharing, and access revocation, see the README.

Conclusion

In this guide, you learned how to:

  • Distinguish between on-chain and off-chain storage and decide where your data belongs.
  • Encrypt files client-side using AES-256-GCM before uploading to decentralised storage.
  • Derive deterministic keys from a wallet seed for ownership proofs and secure sharing.
  • Use key wrapping (hybrid encryption) to share encrypted files with other users.
  • Build storage providers for IPFS and Arweave.
  • Deploy a Midnight smart contract that manages document metadata and access control.

Next, try implementing off-chain storage in your own Midnight projects. Experiment with the provided code examples and adapt the patterns for your specific use cases. Feel free to share your implementations and challenges in the Midnight forum.

Related resources:

Top comments (0)