DEV Community: WeaveDB

WAO - Wizard AO SDK

KYOHEI ITO — Fri, 09 May 2025 09:14:11 +0000

original: wao

WAO SDK streamlines Arweave/AO development with elegant syntax enhancements and seamless message piping for enjoyable coding experiences. GraphQL operations are also made super easy.

Additionally, it includes a drop-in replacement for aoconnect, allowing the testing of lua scripts 1000x faster than the mainnet by emulating AO units in memory. It's even 100x faster than testing with arlocal and ao-localnet.

Quick Start
- Installation
- Drop-in aoconnect Replacement for Tests
- Setting up a Project
- Writing Tests
- Using WAO SDK
- Cherry-Picking Outputs
- Determining Message Success
- Async Message Tracking with receive()
- Logging
- Fork Wasm Memory
- WeaveDrive
- Local Persistent Server
- HyperBEAM
API Reference
- AO
- Process
- Function Piping
- AR
- GQL
- ArMem
- HB

Quick Start

WAO is still actively being developed; please use it at your discretion.

Installation

yarn add wao

Drop-in `aoconnect` Replacement for Tests

By replacing aoconnect with WAO connect, everything runs in memory with zero latency and your tests are executed 1000x faster. The APIs are identical. So, there's no need to change anything else in your code.

//import { spawn, message, dryrun, assign, result } from "@permaweb/aoconnect"
import { connect, acc } from "wao/test";
const { spawn, message, dryrun, assign, result } = connect();

Setting up a Project

It's super easy to set up a test AO project manually.

mkdir wao-test && cd wao-test
yarn init && yarn add wao

Add test and test-only commands to your package.json.

{
  "scripts": {
    "test": "node --experimental-wasm-memory64",
    "test-only": "node --experimental-wasm-memory64 --test-only"
  }
}

Create test directory and test.js file.

mkdir test && touch test/test.js

Writing Tests

Write a simple test in test.js.

import assert from "assert";
import { describe, it } from "node:test";
import { connect, acc } from "wao/test";
const { spawn, message, dryrun } = connect();
const signer = acc[0].signer;
const src_data = `
Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = "Hello, World!" })
end)
`;
describe("WAO", function () {
  it("should spawn a process and send messages", async () => {
    const pid = await spawn({
      signer,
      module: "Do_Uc2Sju_ffp6Ev0AnLVdPtot15rvMjP-a9VVaA5fM",
      scheduler: "_GQ33BkPtZrqxA84vM8Zk-N2aO0toNNu_C-l-rawrBA",
    });

    // on mainnet, you need to wait here till the process becomes available.
    // WAO automatically handles it. No need with in-memory tests.
    // await wait({ pid })

    await message({
      process: pid,
      tags: [{ name: "Action", value: "Eval" }],
      data: src_data,
      signer,
    });
    const res = await dryrun({
      process: pid,
      tags: [{ name: "Action", value: "Hello" }],
      signer,
    });
    assert.equal(res.Messages[0].Data, "Hello, World!");
  });
});

Note that generating random Arweave wallets for every test takes time and slows down your test executions, so Wao connect provides pre-generated accounts for your tests, which saves hours if you are to run your tests thousands of times.

acc[0] = { jwk, addr, signer }

Run the test.

yarn test test/test.js

Using WAO SDK

WAO comes with elegant syntactic sugar and makes writing AO projects an absolute joy.

The same test can be written as follows.

import assert from "assert";
import { describe, it } from "node:test";
import { AO, acc } from "wao/test";

const src_data = `
Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = "Hello, World!" })
end)
`;
describe("WAO", function () {
  it("should spawn a process and send messages", async () => {
    const ao = await new AO().init(acc[0]);
    const { p } = await ao.deploy({ src_data });
    assert.equal(await p.d("Hello", false), "Hello, World!");
  });
});

The AO class is not only for in-memory tests, but also for production code. You just need to import from a different path.

import { AR, AO, GQL } from "wao";

Cherry-Picking Outputs

You often need to pick a specific piece of data from returned results with multiple spawned messages. You need to go through all the returned messages and further go through tags and data to find it. That's too much code to write. AO comes with get parameter to simplify it.

Consider the following Lua handlers.

local json = require('json')

Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = json.encode({ Name = "Bob" })})
end)

Handlers.add("Hello2", "Hello2", function (msg)
  msg.reply({ Data = "Hello, World!", Name = "Bob", Age = "30" })
end)

Handlers.add("Hello3", "Hello3", function (msg)
  msg.reply({ Profile = json.encode({ Name = "Bob", Age = "30" })})
end)

// by default it extracts JSON decoded Data
const out = await p.d("Hello");
assert.deepEqual(out, { Name: "Bob" });

// equivalent
const out2 = await p.d("Hello", { get: true });
assert.deepEqual(out2, { Name: "Bob" });

// get string Data
const out3 = await p.d("Hello2", { get: false });
assert.equal(out3, "Hello, World!");

// get a tag
const out4 = await p.d("Hello2", { get: "Age" });
assert.equal(out4, "30");

// get multiple tags
const out5 = await p.d("Hello2", {
  get: { obj: { firstname: "Name", age: "Age" } },
});
assert.deepEqual(out5, { firstname: "Bob", age: "30" });

// shortcut if keys don't include name, data, from, json
const out6 = await p.d("Hello2", { get: { firstname: "Name", age: "Age" } });
assert.deepEqual(out6, { firstname: "Bob", age: "30" });

// await p.d("Hello2", { get: { name: "Name", age: "Age" } }) doesn't work

// handle tag as json
const out7 = await p.d("Hello3", {
  get: { prof: { name: "Profile", json: true } },
});
assert.deepEqual(out7, { prof: { Name: "Bob", Age: "30" } });

Determining Message Success

To determine if your message is successful, you often need to track down a chain of asynchronous messages and examine resulted tags and data. This is actually a fairy complex operation and too much code to write. Luckily for you, AO comes with check parameter to extremely simplify it. check tracks down messages and lazy-evaluates if your check conditions are met.

// check if Data exists
await p.m("Hello2", { check: true })

// check if Data is a certain value
await p.m("Hello2", { check: "Hello, World! })

// check if a tag exists
await p.m("Hello2", { check: "Name" })

// check if tags are certain values
await p.m("Hello2", { check: { Name: "Bob", Age: "30" } })

// it throws an Error if the conditions are not met
try{
  await p.m("Hello2", { check: { Name: "Bob", Age: "20" } })
}catch(e){
  console.log("something went wrong!")
}

// check if Name is Bob and Age exists, then get Age
const age = await p.m("Hello2", { check: { Name: "Bob", Age: true }, get : "Age" })
assert.equal(age, "30", "Bob is not 30 yo!")

Async Message Tracking with receive()

AOS2 introduced a handy function receive() to send a message to another process and receive a reply in the same handler.

Handlers.add("Hello3", "Hello3", function (msg)
  msg.reply({ Data = "How old are you?" })
  local age = Send({
    Target = msg.To, Action = "Get-Age", Name = msg.Who
  }).receive().Data
  msg.reply({ Data = "got your age!", Name = msg.Who, Age = age })
end)

Since the second reply will be a part of another message triggerd by the Target process reply, you cannot get the final reply simply with the arconnect result function. You need to keep pinging the process results or track down the chain of messages to examine what went wrong. The AO get and check automatically handle this complex operation in a lazy short-circuit manner in the background for you. A proper timeout (ms) should be specified.

const age = await p.m(
  "Hello3",
  { Who: "Bob", To: DB_PROCESS_ID }, // second argument can be tags
  { get: "Age", check: "got your age!", timeout: 5000 }
);
assert.equal(age, "30");

There are so many more powerful tricks you can utilize to make complex AO development easier.

Read on to the API reference section to find out!

Logging

WAO hot-patches the core AOS module code so ao.log automatically is forwarded to JS console.log and whatever you log will be directly displayed in your terminal. Lua tables will be auto-converted to JSON objects. It doesn't affect your production code, it only hot-paches the module during testing. This makes complex debugging so easy.

Handlers.add("Hello4", "Hello4", function (msg)
  ao.log("Hello, Wordl!") -- will be displayed in the terminal
  ao.log({ Hello = "World!" }) -- will be auto-converted to JSON

  -- passing multiple values
  ao.log("Hi", 3, true, [ 1, 2, 3 ], { Hello = "World!" })
end)

You can get logs even when an error occurs in the handler, which is extremely handy to identify the error causes.

Fork Wasm Memory

You can fork wasm memory to a new process. This could come in handy to create checkpoints for tests.

It only works with in-memory testing.

const src_counter = `
local count = 0
Handlers.add("Add", "Add", function (msg)
  count = count + tonumber(msg.Plus)
end)
Handlers.add("Get", "Get", function (msg)
  msg.reply({ Data = tostring(count) })
end)
`;
const ao = await new AO().init(acc[0]);
const { p, pid } = await ao.deploy({ boot: true, src_data: src_counter });
await p.m("Add", { Plus: 3 });
assert.equal(await p.d("Get"), "3");

const ao2 = await new AO().init(acc[0]);
// pass the exisiting wasm memory to a new process
const { p: p2 } = await ao2.spwn({ memory: ao.mem.env[pid].memory });
assert.equal(await p2.d("Get"), "3");
await p2.m("Add", { Plus: 2 });
assert.equal(await p2.d("Get"), "5");

You can also get mainnet process memory from the CU endpoint (GET /state/{pid}) and fork it for tests.

WeaveDrive

The WeaveDrive extension is fully emulated with WAO. You can use attest and avail functions from AO.

import { blueprint, AO, acc } from "wao/test";
const attestor = acc[0];
const handler = `
apm.install('@rakis/WeaveDrive')
Drive = require('@rakis/WeaveDrive')
Handlers.add("Get", "Get", function (msg)
  msg.reply({ Data = Drive.getData(msg.id) })
end)`;

describe("WeaveDrive", () => {
  it("should load Arweave tx data", async () => {
    const ao = await new AO().init(attestor);

    const { p } = await ao.deploy({
      tags: { Extension: "WeaveDrive", Attestor: attestor.addr },
      loads: [await blueprint("apm"), handler],
    });

    const { id } = await ao.ar.post({ data: "Hello" });
    await ao.attest({ id });

    assert.equal(await p.d("Get", { id }), "Hello");
  });
});

Local Persistent Server

You can run a local WAO server with persistent storage, which enables connections with outside components such as frontend apps.

npx wao

port : Arweave port, the ports of AO units are based on this port (default to 4000)
- AR: localhost:4000
- MU: localhost:4002
- SU: localhost:4003
- CU: localhost:4004
db : a directory to store data (default to .cache)
reset : to reset the database

npx wao --port 5000 --db .custom_cache_dir --reset

In this case, the ports will be, AR => 5000, MU => 5002, SU => 5003, CU => 5004.

You can use WAO SDK or AOConnect to connect with the WAO units, but the following tags will be automatically set with WAO SDK.

AOS2.0.1 Module: Do_Uc2Sju_ffp6Ev0AnLVdPtot15rvMjP-a9VVaA5fM
Scheduler: _GQ33BkPtZrqxA84vM8Zk-N2aO0toNNu_C-l-rawrBA
Authority: eNaLJLsMiWCSWvQKNbk_YT-9ydeWl9lrWwXxLVp9kcg

import { describe, it } from "node:test";
import assert from "assert";
import { AO } from "wao";

const src_data = `
Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = "Hello, World!" })
end)`;

describe("WAO Server", () => {
  it("should connect with WAO SDK", async () => {
    const ao = await new AO(4000).init(YOUR_JWK);
    const { p } = await ao.deploy({ src_data });
    assert.equal(await p.d("Hello"), "Hello, World!");
  });
});

With AOConnect,

import { describe, it } from "node:test";
import assert from "assert";
import { connect, createDataItemSigner } from "@permaweb/aoconnect";
const { spawn, message, dryrun, assign, result } = connect({
  MU_URL: `http://localhost:4002`,
  CU_URL: `http://localhost:4003`,
  GATEWAY_URL: `http://localhost:4000`,
});

const src_data = `
Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = "Hello, World!" })
end)`;

describe("WAO Server", () => {
  it("should connect with WAO SDK", async () => {
    const pid = await spawn({
      module: "Do_Uc2Sju_ffp6Ev0AnLVdPtot15rvMjP-a9VVaA5fM",
      scheduler: "_GQ33BkPtZrqxA84vM8Zk-N2aO0toNNu_C-l-rawrBA",
      tags: [
        {
          name: "Authority",
          value: "eNaLJLsMiWCSWvQKNbk_YT-9ydeWl9lrWwXxLVp9kcg",
        },
      ],
      signer: createDataItemSigner(YOUR_JWK),
    });

    // wait till the process becomes available

    const mid = await message({
      process: pid,
      tags: [{ name: "Action", value: "Eval" }],
      data: src_data,
      signer: createDataItemSigner(acc[0].jwk),
    });

    console.log(await result({ process: pid, message: mid }));

    const res = await dryrun({
      process: pid,
      data: "",
      tags: [{ name: "Action", value: "Hello" }],
      anchor: "1234",
    });

    assert.equal(res.Messages[0].Data, "Hello, World!");
  });
});

Connecting with the AOS terminal,

aos \
  --gateway-url http://localhost:4000 \
  --cu-url http://localhost:4004 \
  --mu-url http://localhost:4002 \
  --tag-name Authority \
  --tag-value eNaLJLsMiWCSWvQKNbk_YT-9ydeWl9lrWwXxLVp9kcg

HyperBEAM

HyperBEAM integration is highly experimental at this stage.

Clone WAO and install the dependencies.

git clone https://github.com/weavedb/wao.git && cd wao && yarn

Run a WAO server with in-memory mode (the persistent-mode with HB is not well-tested yet).

yarn server --memory --hb http://localhost:10001

The CU should be running at http://localhost:4004.

Make sure you followed the Hyperbeam setup guide and successfully compiled the latest version.

But don't run rebar3 shell just yet.

Instead, start a Hyperbeam node with the following command from the WAO root directory.

yarn hb PATH_TO_HB_DIR

The HyperBEAM should be running at http://localhost:10001.

You can test HyperBEAM requests with the WAO server as a local CU.

import assert from "assert";
import { after, describe, it } from "node:test";
import { acc } from "wao/test";
import { HB } from "wao";

const data = `
local count = 0
Handlers.add("Inc", "Inc", function (msg)
  count = count + 1
  msg.reply({ Data = "Count: "..tostring(count) })
end)

Handlers.add("Get", "Get", function (msg)
  msg.reply({ Data = "Count: "..tostring(count) })
end)
`;
describe("Hyperbeam", function () {
  it("should interact with a hyperbeam node", async () => {
    const hb = await new HB({ url: "http://localhost:10001" }).init(acc[0].jwk);
    const metrics = await hb.metrics();
    const info = await hb.info();
    const process = await hb.process();
    const slot = await hb.schedule({ process, data });
    const r = await hb.compute({ process, slot });
    const slot2 = await hb.schedule({ process, action: "Inc" });
    const r2 = await hb.compute({ process, slot: slot2 });
    assert.equal(r2.Messages[0].Data, "Count: 1");
    const r3 = await hb.dryrun({ process, action: "Get" });
    assert.equal(r3.Messages[0].Data, "Count: 1");
  });
});

API Reference

AO

Instantiate
deploy
msg
dry
res
ress
asgn
load
eval
spwn
aoconnect Functions
postModule
postScheduler
attest
avail
wait
var

Instantiate

You can initialize AO in the same way as AR.

import { AO } from "wao";
const ao = await new AO().init(jwk || arweaveWallet);

If you need to pass AR settings, use ar. ao.ar will be automatically available.

const ao = await new AO({ ar: { port: 4000 } }).init(jwk || arweaveWallet);
const addr = ao.ar.addr;
await ao.ar.post({ data, tags });

AO Core Functions

deploy

Spawn a process, get a Lua source, and eval the script. src is an Arweave txid of the Lua script.

const { err, res, pid, p } = await ao.deploy({ data, tags, src, fills });

You can directly pass the Lua script with src_data instead of src.

const { err, res, pid, p } = await ao.deploy({ data, tags, src_data, fills });

boot will use On-Boot tag to initialize the process instead of Eval action. You can set either true to use src_data, or set a txid of an existing script. In case of true, data should be undefined so the src_data can fill it with spawn.

const { err, res, pid, p } = await ao.deploy({
  boot: true,
  tags,
  src_data,
  fills,
});

fills replace the Lua source script from src.

local replace_me = '<REPLACE_ME>'
local replace_me_again = '<REPLACE_ME_AGAIN>'
local replace_me_with_hello_again = '<REPLACE_ME>'

const fills = { REPLACE_ME: "hello", REPLACE_ME_AGAIN: "world" };

This will end up in the following lua script.

local replace_me = 'hello'
local replace_me_again = 'world'
local replace_me_with_hello_again = 'hello'

In case you have multiple scripts, use loads and pass src and fills respectively.

await ao.deploy({
  tags,
  loads: [
    { src, fills },
    { src: src2, fills: fills2 },
  ],
});

You can also pass an array of string data to loads.

const num = `num = 0`;
const inc = `Handlers.add("Inc", "Inc", function () num = num + 1 end)`;
const get = `Handlers.add("Get", "Get", function (m) m.reply({ Data = num }) end)`;

const { p } = await ao.deploy({ tags, loads: [num, inc, get] });
await p.m("Inc");
assert.equal(await p.d("Get"), 1);

msg

Send a message.

const { err, mid, res, out } = await ao.msg({
  pid,
  data,
  act,
  tags,
  check,
  get,
  mode,
  limit,
});

check determins if the message call is successful by checking through Tags in Messages in res.

When using from either in check or get, mode needs to be set gql. mode defaults to aoconnect which uses the aoconnect.results function to track down results, which cannot tell where results come from. gql mode doesn't sometimes catch all results if used with AO/Arweave mainnet since there are lags due to the block finality time. limit specifies how many transactions or results to fetch for the check.

const check = { Status: "Success" }; // succeeds if Status tag is "Success"
const check2 = { Status: true }; // succeeds if Status tag exists

Assigning either a string or boolean value checks Data field instead of Tags.

const check3 = "Success"; // succeeds if Data field is "Success"
const check4 = true; // succeeds if Data field exists
const check5 = /ok/; // succeeds if Data field is string containing "ok"
const check6 = (n) => +n > 10; // succeeds if Data field is bigger than 10
const check7 = { json: { a: 3 } }; // succeeds if Data field is JSON and a is 3
const check8 = { json: { a: 3, b: 4 }, eq: true }; // deep equal JSON
const check9 = { data: true, tags: { Status: true, Balance: (n) => +n > 10 } };
const check10 = { data: true, from: PID }; // specify message sender process

Use an array to check multiple conditions.

const check11 = ["Success", { Age: "30" }]; // Data is Success and Age tag is 30
const check12 = [
  { data: "Success", from: PID },
  { Age: "30", from: PID2 },
];

get will return specified data via out.

const get = "ID"; // returns the value of "ID" tag
const get2 = { name: "Profile", json: true }; // "Profile" tag with JSON.parse()
const get3 = { data: true, json: true }; // returns Data field with JSON.parse()
const get4 = true; // => { data: true, json: true }
const get5 = false; // => { data: true, json: false }
const get6 = { obj: { age: "Age", who: "Name" } }; // => { age: 30, who: "Bob" }
const get7 = { age: "Age", who: "Name" }; // same as get6
const get8 = { name: "Profile", json: true, from: PID }; // specify sender process
const get9 = { age: { name: "Age", from: PID }, who: "Name" }; // another example
const get10 = {
  data: true,
  json: true,
  match: (val, index, res) => val.Age < 10,
};

check and get lazy-evaluate tags and data by tracking down async messages. As soon as the conditions are met, they won't track further messages. With receive() added with AOS 2.0, you can only get spawned messages up to the receive() function from result. But WAO automatically tracks down further messages and determines if check conditions are met beyond the receive() function.

For example, consider the following handler.

Handlers.add("Hello", "Hello", function (msg)
  msg.reply({ Data = "Hello, World!" })
  local name = Send({ Target = msg.to, Action = "Reply" }).receive().Data
  msg.reply({ Data = "Hello, " .. name .. "!" })
end)

you can only get the first Hello, World!, but not the second "Hello, " .. name .. "!" from the aoconnect result function.

dry

Dryrun a message without writing to Arweave.

const { err, res, out } = await ao.dry({ pid, data, action, tags, check, get });

res

res does the same thing as msg but for an existing result with mid.

const { err, res, out } = await ao.res({ pid, mid, check, get });

ress

ress gets multiple results from a process.
next() will be returned for pagenation if there are more messages.

const { err, out: msgs, res, next } = await ao.ress({ pid, limit, asc, from });

if (next) {
  const { out: msgs2 } = await next();
}

pid : process ID
limit : how many to get
asc : messages are sorted descendingly by default, set asc=true to reverse
from : cursor to get from

asgn

Assign an existing message to a process.

const { err, mid, res, out } = await ao.asgn({ pid, mid, check, get });

load

Get a Lua source script from Arweave and eval it on a process.

const { err, res, mid } = await ao.load({ src, fills, pid });

eval

Eval a Lua script on a process.

const { err, res, mid } = await ao.eval({ pid, data });

spwn

Spawn a process. module and scheduler are auto-set if omitted.

const { err, res, pid } = await ao.spwn({ module, scheduler, tags, data });

aoconnect Functions

The original aoconnect functions message | spawn | result | assign | dryrun are also available.

createDataItemSigner is available as toSigner.

const signer = ao.toSigner(jwk);
const process = await ao.spawn({ module, scheduler, signer, tags, data });
const message = await ao.message({ process, signer, tags, data });
const result = await ao.result({ process, message });

Advanced Functions

postModule

data should be wasm binary. overwrite to replace the default module set to the AO instance.

const { err, id: module } = await ao.postModule({ data, jwk, tags, overwrite });

postScheduler

This will post Scheduler-Location with the jwk address as the returning scheduler.

const { err, scheduler } = await ao.postScheduler({
  url,
  jwk,
  tags,
  overwrite,
});

attest

Attest Arweave transactions for WeaveDrive.

const { err, res, id } = await ao.attest({ id, tags, jwk });

avail

Make Arweave transactions available for WeaveDrive.

const { err, res, id } = await ao.avail({ ids, tags, jwk });

wait

wait until the process becomes available after spwn. This is mostly used internally with deploy.

const { err } = await ao.wait({ pid });

var

var reads a Lua variable from the current state with dryrun.

const { pid } = await ao.deploy({
  src_data: `Table = { String = "Hello", Array = { "str", 3, true } }`,
});

const table = await ao.var({ pid, data: "Table" });

It strips off pretty tags from the output and auto-converts Lua tables to JSON, you can disable it with json and pretty.

const table = await ao.var({ pid, data: "Table", json: false, pretty: true });

Process

You can go for even more concise syntax with Process class.

Instantiate
msg
m
dry
d
res
r
v

Instantiate

const p = ao.p(pid);

const { p, pid } = await ao.deploy({ data, tags, src, fills });

msg

The first argument is Action, the second argument is Tags, and the third argument is the rest of the options.

const { mid, res, out, err } = await p.msg(
  "Action",
  { Tag1: "value1", Tag2: "value2" },
  { get: true, check: { TagA: "valueA" }, jwk }
);

The default third argument is { get: true } to return the JSON decoded Data.

const { mid, out } = await p.msg("Action", { Tag1: "value1", Tag2: "value2" });

The third parameter defaults to get if it's not an object.

const { mid, out } = await p.msg("Action", { Tag1: "value1" }, "TagA");

is equivalent to

const { mid, out } = await p.msg("Action", { Tag1: "value1" }, "TagA");

You can omit the second argument if there is no tag to pass to.

const { mid, out } = await p.msg("Action", { check: "success!" }}

m

You can only get out with m. This is the most extreme form.

const out = await p.m("Action", { Tag1: "value1", Tag2: "value2" });

This is a quite common pattern during testing. Doing the same with aoconnect requires an enormous amount of code, especially if it involves async/await receive().

const { p } = await ao.deploy({ tags, src_data, fills });
const out = await p.m("Action", { Tag1: "value1", Tag2: "value2" }); // get Data
assert.equal(out, EXPECTED_JSON);

dry

const { mid, out } = await p.dry("Action", { Tag1: "value1", Tag2: "value2" });

d

const out = await p.d("Action", { Tag1: "value1", Tag2: "value2" });

res

const { err, res, out } = await p.res({ mid, check, get });

r

const out = await p.r({ mid, check, get });

v

v is a shortcut for var to get a Lua variable with dryrun.

const { p } = await ao.deploy({
  src_data: `Table = { String = "Hello", Array = { "str", 3, true } }`,
});
const table = await p.v("Table"); // { String: "Hello", Array: [ "str", 3, true ] }

To disable the auto JSON conversion and enable pretty print, use the 2nd and the 3rd arguments.

const table = await p.v("Table", false, true);

Function Piping

pipe
bind
then
err
cb

pipe

Most functions return in the format of { err, res, out, pid, mid, id }, and these function can be chained with pipe, which makes executing multiple messages a breeze.

For example, the following is how deploy uses pipe internally. The execution will be immediately aborted if any of the functions in fns produces an error.

let fns = [
  {
    fn: "spwn",
    args: { module, scheduler, tags, data },
    then: { "args.pid": "pid" },
  },
  { fn: "wait", then: { "args.pid": "pid" } },
  { fn: "load", args: { src, fills }, then: { "args.pid": "pid" } },
];
const { err, res, out, pid } = await this.pipe({ jwk, fns });

bind

If the function comes from other instances rather than AO, use bind.

const fns = [{ fn: "post", bind: this.ar, args: { data, tags } }];

then

You can pass values between functions with then. For instance, passing the result from the previous functions to the next function's arguments is a common operation.

const fns = [
  {
    fn: "post",
    bind: ao.ar,
    args: { data, tags },
    then: ({ id, args, out }) => {
      args.tags.TxId = id; // adding TxId tag to `msg` args
      out.txid = id; // `out` will be returned at last with `pipe`
    },
  },
  { fn: "msg", args: { pid, tags } },
];
const {
  out: { txid },
} = await ao.pipe({ fns, jwk });

If then returns a value, pipe will immediately return with that single value. You can also use err to abort pipe with an error.

const fns = [
  {
    fn: "msg",
    args: { pid, tags },
    then: ({ inp }) => {
      if (inp.done) return inp.val;
    },
  },
  {
    fn: "msg",
    args: { pid, tags },
    err: ({ inp }) => {
      if (!inp.done) return "something went wrong";
    },
  },
];
const val = await ao.pipe({ jwk, fns });

then has many useful parameters.

res : res from the previous result
args : args for the next function
out : the final out result from the pipe sequence
inp : out from the previous result
_ : if values are assigned to the _ fields, pipe returns them as top-level fields in the end
pid : pid will be passed if any previous functions return pid ( e.g. deploy )
mid : mid will be passed if any previous functions return mid ( e.g. msg )
id : id will be passed if any previous functions return id ( e.g. post )

then can be a simplified hashmap object.

let fns = [
  {
    fn: "msg",
    args: { tags },
    then: { "args.mid": "mid", "out.key": "inp.a", "_.val": "inp.b" },
  },
  { fn: "some_func", args: {} }, // args.mid will be set from the previous `then`
];
const {
  out: { key },
  val,
} = await ao.pipe({ jwk, fns });

err

err has the same signature as then. If err returns a value, pipe will throw an Error with that value.

const fns = [
  {
    fn: "msg",
    args: { pid, tags },
    err: ({ inp }) => {
      if (!inp.done) return "something went wrong!";
    },
  },
];
const val = await ao.pipe({ jwk, fns });

cb

cb can report the current progress of pipe after every function execution.

await ao.pipe({
  jwk,
  fns,
  cb: ({ i, fns, inp }) => {
    console.log(`${i} / ${fns.length} functions executed`);
  },
});

AR

AR handles operations on the base Arweave Storage layer as well as wallet connections.

Instantiate
Set or Generate Wallet
toAddr
mine
balance | toAR | toWinston
transfer
checkWallet
post
tx
data
bundle

Instantiate

import { AR } from "wao";
const ar = new AR();

host, port, and protocol can be set to access a specific gateway rather than https://arweave.net.

const ar = new AR({ host: "localhost", port: 4000, protocol: "http" });

In the case of local gateways, you can only set port and the rest will be automatically figured out.

const ar = new AR({ port: 4000 });

AO class auto-instantiates AR internally.

import { AO } from "wao";
const ao = new AO();
const ar = ao.ar;

Set or Generate Wallet

You can initialize AR with a wallet JWK or ArConnect.

const ar = await new AR().init(jwk || arweaveWallet);

Or you can generate a new wallet. In case of ArLocal, you can mint AR at the same time.

const { jwk, addr, pub, balance } = await ar.gen("100"); // mint 100 AR

Once a wallet is set in one of these 3 ways, you cannot use the instance with another wallet unless you re-initialize it with another wallet. This is to prevent executing transactions with the wrong wallet when the browser connected active address has been changed unknowingly.

You can go on without calling init or gen, in this case, AR generates a random wallet when needed, and also using different wallets will be allowed. This is useful, if you are only calling dryrun with AO, since AO requires a signature for dryrun too, but you don't want to bother the user by triggering the browser extension wallet for read only calls.

Once a wallet is set, ar.jwk and ar.addr will be available.

Token Related Methods

toAddr

Convert a jwk to the corresponding address.

const addr = await ar.toAddr(jwk);

mine

Mine pending blocks (only for arlocal).

await ar.mine();

balance | toAR | toWinston

Get the current balance of the specified address in AR. addr will be ar.addr if omitted.

const balance_AR = await ar.balance(); // get own balance
const balance_Winston = ar.toWinston(balance_AR);
const balance_AR2 = ar.toAR(balance_Winston);
const balance_AR3 = await ar.balance(addr); // specify wallet address

transfer

Transfer AR token. amount is in AR, not in winston for simplicity.

const { id } = await ar.transfer(amount, to);

You can set a jwk to the 3rd parameter as a sender. Otherwise, the sender is ar.jwk.

const { id } = await ar.transfer(amount, to, jwk);

For most write functions, jwk can be specified as the last parameter or a field like { data, tags, jwk }.

checkWallet

checkWallet is mostly used internally, but it returns this.jwk if a wallet has been assigned with init, or else it generates a random wallet to use. The following pattern is used in many places. With this pattern, if a wallet is set with init and the jwk the user is passing is different, checkWallet produces an error to prevent the wrong wallet. If no wallet has been set with init or gen and the jwk is not passed, it generates and returns a random wallet.

some_class_method({ jwk }){
  let err = null
  ;({ err, jwk } = await ar.checkWallet({ jwk }))
  if(!err){
    // do something with the jwk
  }
}

Storage Related Methods

post

Post a data to Arweave.

const { err, id } = await ar.post({ data, tags });

tags are not an Array but a hash map Object for brevity.

const tags = { "Content-Type": "text/markdown", Type: "blog-post" };

If you must use the same name for multiple tags, the value can be an Array.

const tags = { Name: ["name-tag-1", "name-tag-2"] };

tx

Get a transaction.

const tx = await ar.tx(txid);

data

Get a data.

const data = await ar.data(txid, true); // true if string

bundle

Bundle ANS-104 dataitems.

const { err, id } = await ar.bundle(dataitems);

dataitems are [ [ data, tags ], [ data, tags ], [ data, tags ] ].

const { err, id } = await ar.bundle([
  ["this is text", { "Content-Type": "text/plain" }],
  ["# this is markdown", { "Content-Type": "text/markdown" }],
  [png_image, { "Content-Type": "image/png" }],
]);

GQL

GQL simplifies the Arwave GraphQL operations to query blocks and transactions.

Instantiate
Txs
Blocks

Instantiate

You can instantiate the GQL class with an endpoint url.

import { GQL } from "wao";
const gql = new GQL({ url: "https://arweave.net/graphql" }); // the default url

AR class auto-instantiates GQL internally.

import { AO } from "wao";
const ao = new AO();
const gql = ao.ar.gql;

import { AR } from "wao";
const ar = new AR();
const gql = ar.gql;

Txs

Get latest transactions.

const txs = await gql.txs();

asc

Get transactions in ascending order.

const txs = await gql.txs({ asc: true });

first

Get the firxt X transactions.

const txs = await gql.txs({ first: 3 });

after

Get transactions after a specific one to paginate. Pass a cursor.

const txs = await gql.txs({ first: 3 });
const txs2 = await gql.txs({ first: 3, after: txs[2].cursor });

Easier pagination with next.

const { next, data: txs0_2 } = await gql.txs({ first: 3, next: true });
const { next: next2, data: txs3_5 } = await next();
const { next: next3, data: txs6_8 } = await next2();

res.next will be null if there's no more transactions to paginate.

block

Get transactions within a block height range.

const txs = await gql.txs({ block: { min: 0, max: 10 } });

const txs = await gql.txs({ block: [0, 10] });

You can also specify only min or max.

by Transaction IDs

Get transactions by transaction ids.

const txs = await gql.txs({ id: TXID });

const txs = await gql.txs({ ids: [TXID1, TXID2, TXID3] });

by Recipients

Get transactions by recipients.

const txs = await gql.txs({ recipient: ADDR });

const txs = await gql.txs({ recipients: [ADDR1, ADDR2, ADDR3] });

by Owners

Get transactions by owners.

const txs = await gql.txs({ owner: ADDR });

const txs = await gql.txs({ owners: [ADDR1, ADDR2, ADDR3] });

by Tags

Get transactions that match tags.

const txs = await gql.txs({ tags: { Name: "Bob", Age: "30" } });

fields

Choose fields to be returned.

const txs = await gql.txs({ fields: ["id", "recipient"] });

For nested objects,

const txs = await gql.txs({ fields: ["id", { owner: ["address", "key"] }] });

You can use a hashmap to specify fields too.

const txs = await gql.txs({
  fields: { id: true, { owner: { address: true, key: true } } }
})

If you assign false, the other fields will be returned.

const txs = await gql.txs({
  fields: { id: true, { block: { previous: false } } }
})

For example, the above will exclude previous from block and return id, timestamp and height.

The entire available fields for transactions as in a graphql query are as follows.

const tx_fields = `{
  id 
  anchor 
  signature 
  recipient 
  owner { address key } 
  fee { winston ar } 
  quantity { winston ar } 
  data { size type } 
  tags { name value } 
  block { id timestamp height previous } 
  parent { id }
  bundledIn { id }
}`;

Blocks

Get latest blocks.

const blocks = await gql.blocks();

asc

Get blocks in ascending order.

const blocks = await gql.blocks({ asc: true });

first

Get the firxt X blocks.

const blocks = await gql.blocks({ first: 3 });

after

Get blocks after a specific one to paginate. Pass a cursor.

const blocks = await gql.blocks({ first: 3 });
const blocks2 = await gql.blocks({ first: 3, after: blocks[2].cursor });

by Block IDs

Get blocks by block ids.

const blocks = await gql.blocks({ id: BLCID });

const blocks = await gql.blocks({ ids: [BLKID1, BLKID2, BLKID3] });

height

Get blocks within a block height range.

const blocks = await gql.blocks({ height: { min: 0, max: 10 } });

const blocks = await gql.blocks({ height: [0, 10] });

Easier pagination with next.

const { next, data: blocks0_2 } = await gql.blocks({ first: 3, next: true });
const { next: next2, data: blocks3_5 } = await next();
const { next: next3, data: blocks6_8 } = await next2();

res.next will be null if there's no more blocks to paginate.

fields

const blocks = await gql.blocks({
  fields: ["id", "timestamp", "height", "previous"],
});

The entire available fields for blocks as in a graphql query are as follows.

const block_fields = `{ id timestamp height previous }`;

ArMem

ArMem stands for Arweave in memory and is a class to emulate an Arweave node and AO units in memory, which is internally used in the WAO testing framework. You can instantiate ArMem and control multiple emulators by passing it between other classes.

Instantiate

Instantiate

When you instantiate WAO connect or AO from wao/test, it automatically and internally instantiates ArMem.

import { connect } from "wao/test";
const { spawn, message, dryrun, assign, result, mem } = connect(); // aoconnect APIs

import { AO } from "wao/test";
const ao = new AO(); // ao.mem

You can instantiate ArMem and pass it to other classes.

import { ArMem, AO, AR, connect } = "wao/test"
const mem = new ArMem()
const { spawn, message, dryrun, assign, result } = connect(mem)
const ao = new AO({ mem })
const ar = new AR({ mem })

If you don't pass the same ArMem instance, the two AO instances will have different environments.

import { AO } = "wao/test"
const ao = new AO() // ao.mem
const ao2 = new AO() // ao2.mem

ao.mem and ao2.mem are not connected. They are on different networks.

import { AO } = "wao/test"
const ao = new AO()
const ao2 = new AO({ mem: ao.mem })

This will connect the two.

HB

HB handles interactions with a Hyperbeam node.

Instantiate
metrics
info
messages
process
schedule
compute
dryrun
get
post
request

Instantiate

import { HB } = "wao"
const hb = await new HB({ url: "http://localhost:10000" }).init(jwk)

metrics

Get node metrics.

const metrics = await hb.metrics();

info

Get node info.

const info = await hb.info();
const operator = info.address;

messages

Get messages on a process. You can get next messages by message.next.

const msgs = await hb.messages({ target, from, to })
for(const item of msgs.edges){
  const { cursor: slot, node: { assignment, message } } = item
}
if(msgs.next) const msgs2 = await msg.next()

target : a process id
from | to : the slot numbers. Messages are marked by integer on HB, not by txid. 0 is the Type=Process message that spawed the process. to is inclusive. Both are optional.

process

Equivalent to spawning a process.

const pid = await hb.process({ tags, data });

schedule

Equivalent to sending a message.

const slot = await hb.schedule({ tags, data, process: pid, action });

compute

Equivalent to getting a result.

const res = await hb.compute({ process: pid, slot });
const { Messages, Spawns, Assignments, Output } = res;

dryrun

const res = await hb.dryrun({ tags, data, process: pid, action });
const { Messages, Spawns, Assignments, Output } = res;

get

Sending a signed http request with GET method to HyperBEAM.

const res = await hb.get({ device, path });

post

Sending a signed http request with POST method to HyperBEAM.

const res = await hb.post({ device, path, tags });

request

Sending a signed http request to HyperBEAM.

const res = await hb.request({ device, path, tags, method });

ArNext - Next.js framework that supports Arweave and Vercel simultaneously

KYOHEI ITO — Fri, 09 May 2025 09:11:49 +0000

original: ArNext

ArNext is a NextJS-based framework that lets you deploy the same codebase both on Vercel and Arweave.

This allows permaapps to have all kinds of cloud-powered performance optimizations, which were not possible before.

Limitations of PermaApps

Permaapps are SPAs (single-page applications) deployed on Arweave, which require complete SSG (static site generation) to build the app before deployment. Permaapps traditionally allow only client-side hash routing within the SPA due to the lack of server-side routing.

An example permaapp.

https://arweave.net/pnZ34u3byeagp93CCKZtnedhBMPStBMNZO1ieD7a2eg

There are a number of limitations permaapps introduce.

Loading of dynamic content is very slow due to the lack of SSR (server-side rendering).
Only the root URL is accessible with client-side hash routing. Regular browser routing without the hash will take you to inaccessible pages when reloading.
Social media cards don't work for dynamically generated pages. You can have only one card for the root page because dynamically generated meta tags with Javascript are not read by social apps such as X (ex-Twitter).
No server-side optimizations such as SSR, ISR, and edge CDN are possible.

With these limitations, permaapps cannot practically onboard millions of users.

What ArNext Does?

An ArNext app is fundamentally a NextJS app, but it also allows the building of an identical permaapp from the same codebase through a series of hacks. It's a multi-page SPA (!?) with SSR on Vercel and client-side routing on Arweave.

You can deploy the identical apps on Vercel and Arweave.
The permaapps deployed on Arweave function as a censorship-resistant permanent backup.
The app deployed on Vercel can have all kinds of cloud-powered optimizations.
The app works as a statically generated multi-page site, but once a page is loaded, it works as a SPA with client-side routing.

Server-Side Performance Enhancements

Now compare the performance of the permaapp deployed on Vercel below with that of the one deployed on Arweave above.

https://arnext-ao.vercel.app

The one on Vercel is extremely fast with ISR (incremental static regeneration).

OpenGraph Tags for Dynamic Pages

Also, the open graph meta tags for dynamically generated pages are enabled with SSR.

Social media previews of arnext-ao.vercel.app/post/sbVC...

No More Hash Routing

Hash routing is no longer required on Arweave-deployed permaapps.

https://uz3hpyxn3pe6nifh3xbarjtntxtwcbgd2k2bgdle5vrhqpw23hua.arweave.net/pnZ34u3byeagp93CCKZtnedhBMPStBMNZO1ieD7a2eg/post/sbVCVIVPwMdeYxXgUqe9-rCgr9-JEInUco7EJ_yJ5tI

This was enabled by utilizing fallback of the Arweave Manifests v0.2.0.

ArNext generates optimized manifests with fallback with a modified version of arkb.

Dynamically Generating Relative Paths for Assets

The biggest challenge to building a static permaapp from the NextJS codebase is the static file linking. Arweave gateways deploy permaapps with a subdirectory URL, and you cannot know the subdirectory name before building the app because it's the hash of the built files. NextJS allows basePath to build the app for subdirectory deployment, but you cannot use that if you don't know the hash before the build. And basePath only allows absolute paths. So we cannot use that.

You need to dynamically calculate and insert relative paths for assets after the app is loaded on a URL, but NextJS doesn't allow such things. In fact, no major web frameworks allow such file linking, for that matter.

There are 3 hacks combined to solve this issue.

Dynamically calculate the relative path and insert tags in the HTML head (_document.js)
Manually rewrite asset tags after building the app code, delete unnecessary html files, and modify the webpack generated js file to produce correct paths during runtime (arweave.mjs)
Generate a proper manifest.json with modified arkb before deployment (deploy.js)

How to Build ArNext Apps

Create ArNext App

npx create-arnext-app myapp
cd myapp && yarn
yarn dev

Deploy on Vercel

vercel --prod

Build for Arweave

yarn arweave

Now, the static app to deploy on Arweave is running on localhost:3000.

Deploy on Arweave

Using arkb

yarn deploy -w WALLET_PATH

Using Turbo

Deploying using Turbo requires a wallet funded with Turbo Credits

yarn deploy:turbo -w WALLET_PATH

A JSON object will be printed to the console with upload information for each file, the arweave manifest generated, and the upload response from uploading the manifest. The manifest id for your deployment can be found at manifestResponse.id.

ArNext Utils

There are a couple of minor changes to make in your NextJS app code to make it work the same on both Vercel and Arweave.

Currently, ArNext only works with the NextJS page router and react-router-dom for Arweave.

Link / useParams / useRouter

Replace Link, useParams, and useRouter with the ones from arnext. It will align next/router and react-router-dom.

import { Link, useParams, useRouter } from "arnext"

export default function Post() {
  const router = useRouter() // router.push(pathname)
  const { id } = useParams()
  ...
  return <Link href={`/post/${id}`}>/post/{id}</Link>
}

getStaticProps / ssr

Wrap getStaticProps with ssr.

import { ssr } from "arnext"

export const getStaticProps = ssr(async ({}) => {
  ...
  return { props }
})

Migration Guide

Currently, ArNext only works with the page router without src directory and JS.

If you have an existing NextJS app, follow the steps below. Using create-arnext-app and manually copying the source code might be less hassle.

Remove unsupported features

ArNext doesn't support certain NextJS features at the moment. If your app is using any of the following features, find alternative ways to implement them.

Typescript
App Router
src directory
next/font/local
next/image
next/router only supports push

They will be supported in future releases.

Install necessary dependencies.

yarn add arnext
yarn add arnext-arkb cheerio cross-env starknet @ardrive/turbo-sdk --dev

Wrap config in next.config.mjs.

/** @type {import('next').NextConfig} */
import arnext from "arnext/config";
const nextConfig = { reactStrictMode: true };
export default arnext(nextConfig);

Replace Head in _document.js.

import { Html, Main, NextScript } from "next/document";
import { Head } from "arnext";

export default function Document() {
  return (
    <Html lang="en">
      <Head />
      <body>
        <Main />
        <NextScript />
      </body>
    </Html>
  );
}

Replace Component with ArNext in _app.js.

import { ArNext } from "arnext";
export default function App(props) {
  return <ArNext {...props} />;
}

Wrap getStaticProps with ssr

import { ssr } from "arnext"

export const getStaticProps = ssr(async ({}) => {
  ...
  return { props }
})

Replace Link, useParams, useRouter with the ones from arnext.

import { Link, useParams, useRouter } from "arnext"

export default function Post() {
  const router = useRouter() // router.push(pathname)
  const { id } = useParams()
  ...
  return <Link href={`/post/${id}`}>/post/{id}</Link>
}

Copy arweave.mjs file to the app root directory.
Add script commands to package.json.

{
  ...
  "scripts": {
    "arweave": "npm run build:arweave && npx serve -s out",
    "deploy": "node node_modules/arnext-arkb deploy out",
    "deploy:turbo": "turbo upload-folder --folder-path out",
    "build:arweave": "cross-env NEXT_PUBLIC_DEPLOY_TARGET='arweave' next build && node arweave.mjs",
    ...
  },
  ...
}

Build for Arweave

yarn arweave

The static version of the NextJS app to be deployed on Arweave should be running at localhost:3000.

Deploy on Arweave

If everything is working, you can deploy the app either using arkb or turbo.

yarn deploy -w KEYFILE

yarn deploy:turbo -w KEYFILE

zkJSON Litepaper v1.0

KYOHEI ITO — Fri, 09 May 2025 09:07:29 +0000

original: zkJSON Litepaper v1.0

Zero Knowledge Provable JSON

zkJSON makes any arbitrary JSON data provable with zero knowledge proof, and makes them verifiable both offchain and onchain (blockchain).

EVM blockchains like Ethereum will get a hyper scalable NoSQL database extension whereby off-chain JSON data are directly queriable from within Solidity smart contracts.

Why

Most offchain data on the web are represented in JSON format, and blockchains have been failing to connect with them efficiently for some critical reasons.

Blockchains are not scalable to the web level
There is no decentralized general-purpose database alternative to cloud databases
The current decentralized database solutions are too domain-specific
The current oracle / indexer solutions are limited to a great extent

As a result, data on web2 (offchain) and web3 (onchain) are divided and web3 is missing a great wide variety of use cases with offchain data. What if we could verify any offchain JSON data in onchain smart contracts, and also build a general-purpose database with web2-like performance and scalability? zkJSON and zkDB will allow direct connections from smart contract to offchain database. And we will further make it practical and sustainable with modular blockchain rollups (Ethereum security + Arweave permanency and scalability) and a decentralized physical infrastructure network (DePIN) using Cosmos IBC.

This entire tech stack will enable novel use cases to web3 such as decentralized oracles and indexers, as well as provide a decentralized database alternative to web2 with the performance and scalability of cloud databases. We could, for instance, build a fully decentralized Twitter without any centralized components. Connecting securely with offchain data with privacy is also the way to bring in enterprise use cases to web3 in combination with DID and Verifiable Credentials (VC). We are working on it too with PolygonID and ICP VETKeys.

We envision the web where offchain data are seamlessly connected with blockchains. Our ultimate goal is to liberate the web2 data silos and redirect the huge monopolistic web2 revenue models such as ad networks and future AI-based networks to web3. Any offchain data without zkJSON are not legit, since they are not verifiable onchain.

Onchain verifiability is what scales the decentralized web. Onchain is the new online, and zkJSON expands what's online/onchain (verifiable).

How

There are 4 steps to build a complete solution.

make any JSON provable with zk circuits - zkJSON
build a database structure with merkle trees and zkJSON - zkDB
commit db states to an EVM blockchain - zkRollup
make it queriable with Solidity - zkQuery

And 3 bonus steps to make it practical and sustainable (using Arweave & Cosmos IBC).

make zkDB feature-rich to bear any web2/web3 usages - WeaveDB
make WeaveDB performant, scalable, and secure with Arweave+EVM hybrid rollup - WeaveDB Rollup
make the rollups sustainable with DePIN - WeaveChain

This repo contains only the first 4 steps. You can find the rest here.

zkJSON

The key to making JSON verifiable with zkp is to invent a deterministic encoding that is friendly to zk circuits. zk circuits can only handle arithmetic operations with natural numbers, so we need to convert any JSON to a series of natural numbers back and forth, then pack everything into as few uint as possible to efficiently save space. The default storage block in Solidity is uint256 and Circom uses a modulo just below the 256 bit range. So optimizing for uint makes sense. Just to clarify, you cannot simply convert JSON to a binary format or any existing encoding formats, because it has to specifically make sense to the circuit logic and Solidity.

Encoding

zk circuits can neither handle objects nor dynamically nested arrays. So we first need to flatten all the paths into a simple array.

{
  "a": 1,
  "c": false,
  "b": { "e": null, "d": "four" },
  "f": 3.14,
  "ghi": [ 5, 6, 7 ],
}

becomes

[
  ["a", 1],
  ["c", false][("b.e", null)],
  ["b.d", "four"],
  ["f", 3.14],
  ["ghi", [5, 6, 7]],
];

Each path will be converted to an unicode number.

[
  [[[97]], 1],
  [[[99]], false][([[98], [101]], null)],
  [[[98], [100]], "four"],
  [[[102]], 3.14],
  [[[103, 104, 105]], [5, 6, 7]],
];

To make it deterministic, items must be lexicographically sorted by the paths.

[
  [[[97]], 1],
  [[[98], [100]], "four"],
  [[[98], [101]], null],
  [[[99]], false][([[102]], 3.14)],
  [[[103, 104, 105]], [5, 6, 7]],
];

Here's a tricky part, if the value is an array, we need to create a path for each element, but we need to tell the difference between ghi.0 and ghi[0] with just numbers. ghi.0 is a path to an object, ghi[0] is a path to an array element. Also there is a case where the key is empty like { "" : "empty" }. Another case to note is that just a primitive value without the top level element being an object is also a valid JSON, such as null, true, [ 1, 2, 3], 1. You can express the paths with empty string , or something like a..b for { "a" : { "" : { "b" : 1 } } }.

To address all these edge cases, we prefix each array key with the number of characters that follow, or 0 if the key is empty (followed by 1) or an array index (followed by another0).

[
  [[[1, 97]], 1],
  [
    [
      [1, 98],
      [1, 100],
    ],
    "four",
  ],
  [
    [
      [1, 98],
      [1, 101],
    ],
    null,
  ],
  [[[1, 99]], false][([[1, 102]], 3.14)],
  [
    [
      [3, 103, 104, 105],
      [0, 0, 0],
    ],
    5,
  ],
  [[[3, 103, 104, 105]], [0, 0, 1], 6],
  [[[3, 103, 104, 105]], [0, 0, 2], 7],
];

Now we flatten the paths but also prefix them with how many nested keys each path contains.

[
  [ 1, 1, 97 ], 1 ],
  [ 2, 1, 98 , 1, 100 ], "four" ],
  [ 2,  1, 98, 1, 101 ], null ],
  [ 1, 1, 99 ], false ]
  [ 1, 1, 102 ], 3.14 ],
  [ 2, 3, 103, 104, 105, 0, 0, 0 ], 5 ],
  [ 2, 3, 103, 104, 105, 0, 0, 1 ], 6 ],
  [ 2, 3, 103, 104, 105, 0, 0, 2 ], 7 ]
]

If the top level is a non-object value such as 1 and null, the flattened path is always [ 0 ].

Let's numerify the values in a similar fashion. There are only 6 valid data types in JSON ( null / boolean / number / string / array / object ), and since the paths are flattened, we need to handle only 4 primitive types. We assign a type number to each.

null (0)
boolean (1)
number (2)
string (3)
array | object (4)

The first digit will always be the type number.

null (0)

null is always [ 0 ] as there's nothing else to tell.

boolean (1)

There are only 2 cases. true is [ 1, 1 ] and false is [ 1, 0 ].

number (2)

number is a bit tricky as we need to differentiate integers and floats, and also positive numbers and negative ones. Remember that circuits can only handle natural numbers. A number contains 4 elements.

1st element - type 2
2nd - sign, 0 for negative, 1 for positive
3rd - how many digits after ., 0 in case of an integer
4th - actual number without .

for instance,

1 : [ 2, 1, 0, 1 ]
-1 : [ 2, 0, 0, 1 ]
3.14 : [ 2, 1, 2, 314 ]

string (3)

The first digit is the type 3 and the second digit tells how many characters, then each character is converted to a unicode number (e.g. abc = [ 3, 3, 97, 98, 99 ]).

array | object (4)

In the case of an array and object, it prefixes 4 and recursively encodes all the nested values. The final array includes internal paths too.

[ 1, 2 ] : [ 4, 1, 0, 0, 0, 2, 1, 0, 1, 1, 0, 0, 1, 2, 1, 0, 2 ]

Note that the path to 1 is 1, 0, 0, 0 and the path to 2 is 1, 0, 0, 1, and they are included.

Now let's convert the values in our original JSON example.

[
  [
    [1, 1, 97],
    [2, 1, 0, 1],
  ],
  [
    [2, 1, 98, 1, 100],
    [3, 4, 102, 111, 117, 114],
  ],
  [[2, 1, 98, 1, 101], [0]],
  [
    [1, 1, 99],
    [1, 0],
  ],
  [
    [1, 1, 102],
    [2, 1, 2, 314],
  ],
  [
    [2, 3, 103, 104, 105, 0, 0, 0],
    [2, 1, 0, 5],
  ],
  [
    [2, 3, 103, 104, 105, 0, 0, 1],
    [2, 1, 0, 6],
  ],
  [
    [2, 3, 103, 104, 105, 0, 0, 2],
    [2, 1, 0, 7],
  ],
];

Now we are to flatten the entire nested arrays, but each number must be prefixed by the number of digits that contains, otherwise, there's no way to tell where to partition the series of digits. And here's another tricky part, if the number contains more than 9 digits, you cannot prefix it with 10, 11, 12 ... because when all the numbers are concatenated later, 10 doesn't mean that 10 digits follow, but it means 1 digit follows and it's 0. So we allow max 8 digits in each partition and 9 means there will be another partition(s) following the current one.

123 : [ 3, 123 ]
12345678 : [ 8, 12345678 ]
1234567890 : [ 9, 12345678, 2, 90 ]

By the way, digits are in fact stored as strings, so a leading 0 won't disappear.

1234567809 : [ "9", "12345678", "2", "09" ]

This is the prefixed version.

[
  [
    [1, 1, 1, 1, 2, 97],
    [1, 2, 1, 1, 1, 0, 1, 1],
  ],
  [
    [1, 2, 1, 1, 2, 98, 1, 1, 3, 100],
    [1, 3, 1, 4, 3, 102, 3, 111, 3, 117, 3, 114],
  ],
  [
    [1, 2, 1, 1, 3, 98, 1, 1, 3, 101],
    [1, 0],
  ],
  [
    [1, 1, 1, 1, 2, 99],
    [1, 1, 1, 0],
  ],
  [
    [1, 1, 1, 1, 3, 102],
    [1, 2, 1, 1, 1, 2, 3, 314],
  ],
  [
    [1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 0],
    [1, 2, 1, 1, 1, 0, 1, 5],
  ],
  [
    [1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 1],
    [1, 2, 1, 1, 1, 0, 1, 6],
  ],
  [
    [1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 2],
    [1, 2, 1, 1, 1, 0, 1, 7],
  ],
];

Then this is the final form all flattened.

[
  1, 1, 1, 1, 2, 97, 1, 2, 1, 1, 1, 0, 1, 1, 1, 2, 1, 1, 2, 98, 1, 1, 3, 100, 1,
  3, 1, 4, 3, 102, 3, 111, 3, 117, 3, 114, 1, 2, 1, 1, 3, 98, 1, 1, 3, 101, 1,
  0, 1, 1, 1, 1, 2, 99, 1, 1, 1, 0, 1, 1, 1, 1, 3, 102, 1, 2, 1, 1, 1, 2, 3,
  314, 1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 0, 1, 2, 1, 1, 1, 0,
  1, 5, 1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 1, 1, 2, 1, 1, 1, 0,
  1, 6, 1, 2, 1, 3, 3, 103, 3, 104, 3, 105, 1, 0, 1, 0, 1, 2, 1, 2, 1, 1, 1, 0,
  1, 7,
];

It's 144 integers, or 182 digits. The original JSON was 66 character long when JSON.stringified, so it's not too bad considering integer vs character (let's say one ascii char takes up 3 digits and one unicode char takes up 7 digits). And zk circuits and Solidity cannot handle just stringified JSONs anyway. But it gets better.

When passed to a circuit, all digits will be concatenated into one integer. Circom by default uses a modulo with

21888242871839275222246405745257275088548364400416034343698204186575808495617 (77 digits)

which means up to 76 digits are safe and a 77-digit number could overflow, which is also within the range of uint / uint256 in Solidity.

So to convert the encoded array to a circuit signal, it becomes

[
  1111297121110111211298113100131431023111311731141211298113101101111299111011,
  1131021211123314121331033104310510101012111015121331033104310510101112111016,
  121331033104310510101212111017,
];

If you observe carefully, there's room for more compression. Most digits are a single digit with a prefix of 1, so we can remove the prefixes and join the succession of single digits, and we can use 0 and the number of single digits in the succession. For instance 121110111211 becomes 06210121, and we save 4 digits.

We will prefix each integer with 1, since now 0 could come at the beginning and it disappears without the prefix. So

032123314121331033104310509000210523310331043105090012106233103310431051010

will be prefixed with 1 and become

1032123314121331033104310509000210523310331043105090012106233103310431051010

otherwise the first 0 will disapper when being evaluated as a number.

[
  1111129706210121298113100131431023111311731141211298113101030112990410113102,
  1032123314121331033104310509000210523310331043105090012106233103310431051010,
  10522107,
];

Now it's much shorter than before. What's surprising here is that the entire JSON is compressed into just 3 integers in the end (well, almost 2 integers). It's just uint[3] in Solidity. This indeed is extreme efficiency! The zkJSON circuit by default allows up to 256 integers (256 * 76 safe digits), which can contain a huge JSON data size, and Solidity handles it efficiently with a dynamic array uint[], which is optimized with Yul assembly language. What's even better is that the only bits passed to Solidity is the tiny bits of the value at the queried path, and not the entire JSON bits. So if you are querying the value at the path a, 1111297(path: "a") and 1042101(value: 1) are the only digits passed to Solidity as public signals of zkp.

Now we can build a circuit to handle these digits and prove the value of a selected path without revealing the entire JSON. It's easy to explain the encoding, but harder to write the actual encoder/decoder and a circuit to properly process this encoding. But fortunately, we already did write them!

You can use zkjson node package to encode and decode JSON.

yarn add zkjson

const { encode, decode, toSignal, fromSignal } = require("zkjson");

const json = { a: 1 };
const encoded = encode(json); // [ 1, 1, 97, 2, 1, 0,  1 ]
const signal = toSignal(encoded); // [ '11111297042101' ]
const encoded2 = fromSignal(signal); // [ 1, 1, 97, 2, 1, 0, 1 ]
const decoded = decode(encoded2); // { a : 1 }

const { encodePath, decodePath, encodeVal, decodeVal } = require("zkjson");

const path = "a";
const encodedPath = encodePath(path); // [ 1, 1, 97 ]
const decodedPath = decodePath(encodedPath); // "a"

const val = 1;
const encodedVal = encodeVal(val); // [ 2, 1, 0, 1 ]
const decodedVal = decodeVal(encodedVal); // 1

zkDB

Once we get zkJSON, we can build a database structure with zkJSON as base building blocks.

A document-based NoSQL database would have collections, and each collection in turn would have a bunch of documents, which are JSONs.

Collection

We can use a sparse merkle tree (SMT) to represent all the document data in a collection with a root hash. SMT is perfect because curcuits cannot handle dynamic tree sizes and SMT can represent a large number of documents efficiently, and any data membership or non-membership can be proven efficiently with a zk proof without the actual merkle proof. This is what enables efficient direct queries to offchain databases from within EVM smart contracts.

Each leaf node will be the poseidon hash of zkJSON encoding of the data. To hash 256 * 76 digits, 16 poseidon hashes are hashed together into another poseidon hash. This allows a fairly large JSON size to be proven.

And each leaf node has an index number, so we need to somehow convert the document IDs to numbers without collisions. How many leaf nodes a SMT has depends on the pre-defined depth of the tree. For example, a 32-level SMT can have 2 ** 32 = 4294967296 leaf nodes. The level must be pre-defined at the circuit compile time, so we need to find the right conversion and balance.

Due to this constraint, we only allow 64 characters to keep things compact and efficient, although there can be different optimized setups for your specific use cases.

A-Z (0 - 25)
a-z (26 - 51)
0-9 (52 - 61)
- (62)
_ (63)

Now 2 digits can represent one character with collision free, which means we can have only up to 4 characters in document IDs with a 32-level SMT. The last allowed digit will always have the possibility of overflowing, so we prefix the converted numbers with 1 to differentiate A from AA (they are both 0 without the prefix 1).

A = 100
AA = 10000
ABC = 1000102
abcd = 126272829

We can of course increase the level to have more characters, but the more levels, the more computation with the circuit, so we need to find the right balance. For instance, to allow 10 characters we need 67 levels of SMT.

zk_WeaveDB = 151366322302647300301

You can use zkjson to convert the string to an SMT index.

const { toIndex, fromIndexs } = require("zkjson");

const index = toIndex("zkJSON"); // 1513609181413
const str = fromIndex(index); // "zkJSON"

Practically a 100-level SMT allows 15 character IDs and 1,267,650,600,228,229,401,496,703,205,376 documents in a collection. It should be sufficient for most applications if the IDs are designed wisely.

One way to have a longer ID length with the same depth is to restrict the allowed characters to less than 31 since 31 * 31 = 961. In this case 3 digits can represent 2 characters instead of 4 digits representing 2 characters. But we won't cover it here.

Collection Circuit

Database

For the database, we can take the exact same approach with the collections. We can use an SMT to represent multiple collection states in a DB with one root hash, and each leaf node will be the merkle root of a collection, which in turn represents the entire documents in the collection. We could give each collection an ID with the same ID-to-index conversion as the documents, however, collection IDs are not as essential as document IDs since document IDs are usually a part of access control rules, but collection IDs are not. We can use an incremental count for collection IDs and no well-structured DB has so many collections as documents. Let's say 2 ** 8 = 256, so an 8 level SMT can give us 256 collections and it should be more than enough for most applications. If you need alphanumeric IDs for collections, you could map them with numeric indexes offchain (e.g. 0 = FirstCollection, 1 = AnotherCollection, 2 = YetAnotherCollection...). Note that this is different from the deterministic toIndex / fromIndex conversion. In this way we can use a smaller tree and keep the circuit small.

Now we can write a circuit to prove a collection root hash, then we can write another circuit to prove a database root hash, which represents multiple collections within the database. This circuit can also prove any value in any JSON document in any collection in a database without revealing the entire JSON data. zkJSON enables this.

DB Circuit

zkRollup

How do we make zkDB secure and queriable from other blockchains? We can write a circuit to prove the merkle tree hash transitions and deploy a Solidity contract to verify those proofs onchain. Fortunately, Circom auto-generates a Solidity verifier for us, so we can use that function in our verifier contract. We need to keep track of the current database root merkle hash as a Solidity contract state.

interface IZKRollup {
  address public committer;
  uint public root;
  function commit (uint[] memory zkp) external returns (uint);
}

zkQuery

Finally, we can deploy the previous zkDB query circuit verifier as a Solidity contract too, and make it possible to securely query any paths with the right proof. When querying, the Solidity contract must check the DB root hash to verify the queried value against the current database state.

interface IZKQuery {
  function qNull (uint[] memory path, uint[] memory zkp) external pure returns (bool);
  function qBool (uint[] memory path, uint[] memory zkp) external pure returns (bool);
  function qInt (uint[] memory path, uint[] memory zkp) external pure returns (int);
  function qFloat (uint[] memory path, uint[] memory zkp) external pure returns (uint[3] memory);
  function qString (uint[] memory path, uint[] memory zkp) external pure returns (string memory);
  function qRaw (uint[] memory path, uint[] memory zkp) external pure returns (uint[] memory);
  function qCond (uint[] memory path, uint[] memory cond, uint[] memory zkp) external pure returns (bool);
}

path[0] is a collection index, and path[1] is a doc index, then the rest of the path follows.

qNill returns true only if the value is null and otherwise throws an error. And qFloat returns the array of encoded numbers without the type prefix ( e.g. [ 1, 2, 314 ] ) since Solidity cannot handle float numbers.

qRaw returns the raw encoded value for non-primitive data types (array and object), and you can further query the raw value with the getX functions. Pass the raw value returned from qRaw with the path to query, instead of zkp proof.

interface IZKQuery {
  function getNull (uint[] memory path, uint[] memory raw) external pure returns (bool);
  function getBool (uint[] memory path, uint[] memory raw) external pure returns (bool);
  function getInt (uint[] memory path, uint[]  memory raw) external pure returns (int);
  function getFloat (uint[] memory path, uint[] memory raw) external pure returns (uint[3] memory);
  function getString (uint[] memory path, uint[] memory raw) external pure returns (string memory);
}

Conditional Operators

qCond queries a field with a conditional operator and returns true if the condition is met.

with boolean, number, string : $gt $gte $lt $lte
with any types : $eq $ne $in $nin
with array : $contains $contains_any $contains_all $contains_none

const { encodeQuery } = require("zkjson");

const json = { num: 5, arr: [1, 2, 3] };

// for num field
const num_gt = encodeQuery(["$gt", 4]);
const num_gte = encodeQuery(["$gte", 5]);
const num_lt = encodeQuery(["$lt", 6]);
const num_lte = encodeQuery(["$lte", 5]);
const num_eq = encodeQuery(["$eq", 5]);
const num_ne = encodeQuery(["$ne", 7]);
const num_in = encodeQuery(["$in", [4, 5, 6]]);
const num_nin = encodeQuery(["$nin", [1, 2, 3]]);

// for arr field
const arr_contains = encodeQuery(["$contains", 3]);
const arr_contains_any = encodeQuery(["$contains_any", [3, 4, 5]]);
const arr_contains_all = encodeQuery(["$contains_all", [2, 3]]);
const arr_contains_none = encodeQuery(["$contains_none", [4, 5, 6]]);

Other Structures

You could also write a function to get an array of numbers or a specific data structure, but it's up to your applications what data types to extract, so we will leave it up to you.

Going Further

With the first 4 components zkJSON / zkDB / zkRollup / zkQuery, it's now technically possible to build a fully verifiable zkp-based DB connecting blockchains and offchain data. But this doesn't mean it's practical in real use cases. We will briefly introduce 3 additional steps of WeaveDB / WeaveDB Rollup / WeaveChain to implement zkDB in our real world.

WeaveDB

WeaveDB is a general-purpose NoSQL database as a smart contract. It utilizes SCP (Storage-based Consensus Paradigm) enabled by Arweave, and the entire database including indexes is a SmartWeave contract. It has a powerful DSL called FPJSON to operate on JSON objects, which enables highly advanced features a decentralized database would require.

Crypto Wallet Authentication
ZKP verifiability
Data Schema
Access Control Rules
Auto-executed Crons
Triggers to chain queries
Onchain indexers
Relayers to connect with offchain data / other blockchains

WeaveDB queries are almost compatible with Firestore from Google but way more powerful thanks to FPJSON. In the future, we will write a circuit to prove all the FPJSON operations so zkDB will be even more secure with powerful data manipulations.

Each data block of WeaveDB will be a zkJSON document, so we can query WeaveDB data directly from Ethereum smart contracts as well as from Arweave smart contracts (SmartWeave).

WeaveDB Rollup

SmartWeave (Arweave smart contract) provides scalability and cost-effectiveness with lazy offchain computation. This is the only way to hyperscale the decentralized web. But when it comes to databases, a blockchain sequencer is a bottleneck to performance and latency because of how the sequencer processes transactions in sequence and how a DB must maintain ACID properties with hyper-low latency. So WeaveDB has developed a L3 rollup to the L2 sequencer (Warp) to the L1 Arweave permanent storage. In this way, we can have a centralized node for parallel query executions with high performance and low latency of web2 cloud databases, but still keep full decentralization with L1 verifiability and L2 composability.

Although Arweave already guarantees permanent data verifiability and full decentralization, WeaveDB rollups can optionally inherit the Ethereum (or any EVM) security and interoperability with EVM smart contracts via zkp by turning on the zkRollup feature.

WeaeDB Rollup Docs (WIP)

WeaveDB Rollup will roll out in 3 phases.

Rollup Development Kit (RDK)
Rollup as a Service (RaaS)
Decentralized Physical Infrastructure Network (DePIN)

WeaveChain

WeaveChain will be a CosmosSDK based DePIN blockchain and a marketplace to match database developers / dapps with rollup operators. It's basically a Filecoin for database. zkDB/WeaveDB is to WeaveChain as IPFS is to Filecoin. We will introduce 2 unique components to connect with real-world data and web2.

WeavePoint Protocol - Offchain DB points backed by ERC20 via zkJSON
WeaveWallet - Biometric wallet with PolygonID (DID / VC / ZKP) utilizing Passkeys and ICP VETKeys

WeaveChain will also be a PoS network to manage rollup nodes and IBC compatible to communicate between different chains.

Resources

Tutorials

API Reference

Demos

ARJSON

KYOHEI ITO — Fri, 09 May 2025 09:02:21 +0000

original: ARJSON

ARJSON leverages bit-level optimizations to encode JSON at lightning speed while compressing data more efficiently than other self-contained JSON encoding/compression algorithms, such as MessagePack and CBOR.

How ARJSON Outperforms Every Other Encoding Algorithm

MessagePack generates the smallest encoded data sizes among existing self-contained JSON serialization formats. However, ARJSON outperforms MessagePack in 90% of cases with random data, with the remaining 10% yielding nearly identical sizes. More importantly, for structured data with repetitive patterns and duplicate values, ARJSON far surpasses MessagePack, achieving up to around 95% better compression rates (20x smaller in size).

How could beating the current status quo even be possible at all, let alone by a wide margin?

Bit-level optimization over traditional byte-level encoding MessagePack and other serialization algorithms are optimized at the byte level, often leaving unused bits scattered throughout the encoded data. Since 1 byte consists of 8 bits, data is typically packed into bytes in a way that aligns with standard memory structures. In contrast, ARJSON leverages bit-level optimizations with variable-length data units, ensuring that every bit is utilized efficiently, leaving no space wasted.
Columnar structure reorganization for enhanced compression As ARJSON scans JSON data, it dynamically reorganizes it into a columnar structure. Modern databases achieve high performance by storing values of the same field together, which naturally results in sequences of similar data. This structural alignment makes data size more uniform and predictable, increasing redundancy and minimizing differences between neighboring values, ultimately leading to superior compression efficiency.
Delta packing for efficient storage of sequential and repetitive data ARJSON leverages its columnar structure to encode differences between consecutive values and efficiently pack repeating sequences. Since meaningful data naturally forms patterns—whether timestamps, counters, or structured records—delta packing takes full advantage of these repetitions, drastically reducing storage and improving compression efficiency.
Simple, Deterministic, and Metadata-Efficient Despite its advanced compression capabilities, ARJSON is based on simple deterministic principles. With just a single scan, it dynamically applies bit-level optimizations, columnar reorganization, and delta packing without requiring multiple passes or complex heuristics. The deterministic order eliminates the need for unnecessary metadata, further reducing storage overhead and ensuring a compact and streamlined encoding process.
Direct Data Extraction and ZK Compatibility ARJSON’s structured encoding allows data to be accessed directly without full decoding, making it highly efficient for on-the-fly processing. This is especially important for zero-knowledge circuits (ZK circuits), where only arithmetic operations are allowed, eliminating the need for costly parsing. Additionally, ARJSON’s compact and predictable format enables EVM assembly-level optimizations, reducing gas costs and improving execution efficiency in blockchain environments.

Bit Level Optimization

LEB128 is a byte-level optimized integer encoding scheme that represents base-128 numbers using 7-bit chunks per byte. The most significant bit (MSB) serves as a continuation flag, indicating whether additional bytes are needed to fully encode the number. This allows efficient storage of variable-length integers while minimizing wasted space.

For instance,

3 : 00000011
120 : 01111000
150 : 10010110 00000001 = 22 (00010110) + 128 * 10 ** 1 (00000001)

However, small numbers in LEB128 waste bits. 3 requires only 2 bits (11) but consumes 8 bits in LEB128.

ARJSON eliminates wasted bits by introducing a 2-bit flag to specify the number encoding scheme, followed by the minimal bit representation required for the number:

00 : 2 bit integer
01 : 3 bit integer
10 : 4 bit integer
11 : LEB128

For instance, 3 is stored as 0011 (flag: 00, integer: 11), using only 4 bits — a 50% reduction compared to LEB128.

If you store 1,000 numbers, ARJSON saves 4 bits per number, resulting in 5,000 bits (500 bytes) of storage savings—a significant improvement in compression efficiency.

Additionally, ARJSON adapts its integer encoding based on context, using three optimized schemes:

Short
- 00 : 2 bit integer
- 01 : 3 bit integer
- 10 : 4 bit integer
- 11 : LEB128
Uint
- 00 : 3 bit integer
- 01 : 4 bit integer
- 10 : 6 bit integer
- 11 : LEB128
LEB128
- no flags requried

This adaptive approach ensures minimal storage overhead while maintaining fast encoding and decoding speeds.

For integers with a known maximum length, ARJSON eliminates the need for additional flags. For example, since ARJSON defines only 8 value types, each type can be represented within 3 bits. In such cases, the 2-bit flag is omitted, and a 3-bit integer encoding is used instead.

This saves 2 bits per number, leading to a total 5-bit reduction compared to LEB128’s minimum 8-bit representation.

Additionally, ARJSON performs a single scan over the data, restructuring it into a columnar format in a deterministic order. This approach eliminates the need for explicit markers or metadata, as bit positions inherently define what each number represents.

Columnar Restructuring

ARJSON converts JSON data into integer representations and organizes them into column-based chunks during scanning. The encoded data is divided into 13 groups of similar integers, ensuring that related values are packed together. The packing order is crucial, as earlier groups act as metadata for subsequent groups.

1. First Bit

The very first bit classifies whether the data is a primitive type including an empty array ([]) and an object ({}), or a no-empty array or an object. Since primitive types require no additional structural metadata, they follow a separate optimized encoding path, often requiring just 1 byte.

0 : a non-empty structured object
1 : a primitive value or an empty array or an empty object

2. Value Count

If the first bit is 0, the following bits represent the number (short) of values in the JSON.

3. Value Flags

ARJSON builds a structure map, linking values to keys and keys to their parent objects, while also marking objects and arrays. For instance, with { a: 1, b: 2, c: [3, 4] }, it first discovers an object {} then goes through a => 1 => b => 2 => c => [] => 3 => 4, which generates a keymap of [ -1, 0, 0, 0, 3 ] and a value map of [ 1, 2, 4, 4 ]. The keymap corresponds to [ "{}", "a", "b", "c", "[]" ] and the value map correcponds to [ 1, 2, 4, 4 ] respectively. The valuemap is converted to deltas if the difference is less than 4, which usually gives us small integers saving bits. So [ 1, 2, 4, 4 ] becomes [ 1, 1, 2, 0 ]. If a delta decreases, an offset of 4 is added to indicate a negative shift.

[ 1, 2, 4, 4 ] => [ 1, 1, 2, 0 ]
[ 1, 2, 4, 2 ] => [ 1, 1, 2, -2 ] => [ 1, 1, 2, 6 ]
[ 1, 2, 4, 10 ] => [ 1, 1, 2, 10 ]

Value flags are 1 bit flags to specify if the corresponding item is a delta (1) or non-delta (0).

[ 1, 1, 2, 0 ] => 1111 (all delta)
[ 1, 1, 2, 6 ] => 1111 (all delta)
[ 1, 1, 2, 10 ] => 1110 (delta except for 10)

For delta-encoded values, ARJSON uses only 3 bits per number (range 0-8), saving multiple bits compared to non-delta values.

4. Value Links

Value links represent the value map ([ 1, 1, 2, 0 ]), with each integer’s bit-length deterministically determined by combining the value map with the key map. For example, given the key map [ -1(0), 0(1), 0(2), 0(3), 3(4) ] and the value map [ 1(v), 2(v), 4(v), 4(v) ], preserving the scanning order results in [ -1(0), 0(1), 1(v), 0(2), 2(v), 0(3), 3(4), 4(v), 4(v) ]. Since key indexes are always incremental, 1(v) can only refer to previous indexes, ensuring it is safe to encode using only 2 bits. However, since 0 is reserved for bit increment, ARJSON increments non-delta values by 1 before encoding, so [ 1, 2, 4, 4 ] becomes [ 2, 3, 5, 5 ], and 2(v) actually consumes 3 bits while remaining fully deterministic. Delta values always consume only 3 bits.

0 with the current expected number of bits is used to indicate increments in the number of bits required for each link. For example, in the value map [ 1, 2, 2, 7, 15 ] (40 bits), it becomes [ 1, 0, 0, 7, 15 ] with delta conversion and the required bit lengths are [ 3(delta), 3(delta), 3(delta), 3, 4 ]. To encode this efficiently, ARJSON inserts 0 before each increment except for deltas as we always know a delta requires 3 bits, resulting in 22 bits, saving 18 bits.

001(delta) 000(delta) 000(delta) 0(inc) 00(inc) 111(7) 000(inc) 1111(15)

This ensures that ARJSON can always determine the minimum number of bits needed to read the next value link without additional metadata.

Delta Packing

If the same delta pattern occurs more than three times consecutively, ARJSON applies delta packing to eliminate redundant storage. For example, with the value map [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ], delta conversion results in [ 1, 1, 1, 1, 1, 1, 1, 1, 1 ], requiring 27 bits (3 bits per value). Instead of storing repetitive deltas, ARJSON compresses the sequence by indicating delta packing with 0 (3 bits), specifying the length 9 (4 bits), and encoding the delta value 1 (3 bits), resulting in just 10 bits (000 1001 001). This saves 17 bits compared to the naive approach.

5.Key Flags

Key flags are 1-bit indicators that specify whether a key uses delta encoding, just like value flags.

6. Key Links

Key links represent the keymap ([ -1, 0, 0, 0, 3 ]), but since the first element is always -1, it is removed. Additionally, 0 is reserved for delta packing, so [ -1, 0, 0, 0, 3 ] is transformed into [ 1, 1, 1, 4 ], with delta conversion and packing further optimizing it to [ 1, 0, 0, 3 ].

7. Key Types

Key types are 2-bit indicators that define the type of each key in the keymap:

00 (0) : array
01 (1) : object
10 (2) : base64url string key
11 (3) : string key

For example, given the key map ["{}", "a", "b", "c", "[]"], the key types would be [1, 3, 3, 3, 0]. base64url keys save at least 2 bits per character since they are converted to numbers less than 64 and encoded in just 6 bits each, whereas regular string keys are stored using LEB128. This optimization applies only to keys consisting solely of base64url characters (alphanumeric, -, and _).

For keys classified as base64url or regular strings (types 2 and 3), key types are followed by the string length in a compressed format. For example, the key abc is stored as 10 00 11 (10 for base64url, 00 indicating a 2-bit short encoding, and 11 representing a length of 3).

Key types are followed by a key length (short) + 1 (since 0 is reserved for duplicate references) if the key type is string (10 or 11). For instance, if the key is abc, the key type is 10 00 11.

Duplicate string keys and types are replaced with a 0 flag followed by a deterministic index, significantly reducing storage overhead. For instance, in [ { "name": "Bob" }, { "name": "Alice" } ], the second occurrence of name is not stored as base64url again but instead referenced with 10 000, pointing to the first occurrence, saving 22 bits (name vs 0, 24 bits vs 2 bits). This method extends to cross-referencing values as well. In [ { "name": "Bob" }, { "Bob": "name" } ], both name and Bob are replaced with deterministic indexes 0 and 1, respectively, further reducing redundancy.

8. Keys

Keys are either base64url strings or regular strings. As explained earlier, duplicate keys are replaced with deterministic indexes, significantly reducing redundancy and saving bits.

9. Data Types

ARJSON defines 7 data types, each represented using only 3 bits:

001 (1) : null
010 (2) : base64url string
011 (3) : bool
100 (4) : positive integer
101 (5) : negative integer
110 (6) : float
111 (7) : string

The value 0 is reserved for type packing. If the same type appears more than three times consecutively, it is compressed using a repeat flag (0), a short-length encoding, and the actual type. For example, [ 3, 3, 3, 3, 3 ] is compressed into [ 0, 5, 3 ], which is stored as 000 101 011, saving 6 bits.

10. Boolean Values

Boolean values are stored in 1 bit:

0 : false
1 : true

11. Number Values

Positive and negative integers are stored using a 2-bit flag in a custom number system:

00 (0) : delta
01 (1) : 4 bits
10 (2) : 6 bits
11 (3) : LEB128

If a value is delta-encoded (0 flag), and the next bit is 7, it indicates delta packing, followed by a short length and a 3-bit delta value. For example, [ 1, 2, 3, 4, 5 ] (30 bits) is first transformed into [ 1, 1, 1, 1, 1 ] via delta conversion, then further compressed to 00 111 001 using delta packing, saving 22 bits.

Floats are handled differently, using a sign, precision, and integer representation. The first integer combines the sign and precision, adding 4 to the precision when the number is negative. If the precision is greater than 2, the first integer stores 0 for positive numbers and 4 for negative numbers, while the second integer holds the actual precision. The last integer is always the numerical value itself.

For example, -3.14 is a negative number with precision 2, which makes the first integer 6 (2 + 4) and the second integer 314 in LEB128 format. So -3.14 results in 01 0110 11 10011010 00000010 which is 24 bits (3 bytes). In contrast, MessagePack consumes 9 bytes to store -3.14.

12. String Values

String values follow the same compression rules as keys but are stored in value positions rather than key positions. For example, in { "name": "Bob" }, name is encoded as a key, while Bob is stored separately as a string value.

13. Padding

To ensure proper byte alignment, the encoded bitstream is zero-padded to the nearest multiple of 8 bits. If the encoded data is 29 bits, ARJSON adds three padding bits to extend it to 32 bits (4 bytes), maintaining efficient byte-level storage.

Special Optimization for Primitive Values and Empty Objects

When the first bit is 1, ARJSON applies special encoding rules using the next 7 bits, which differ from those used for structured objects. These optimizations allow primitive values and empty objects to be stored with minimal overhead, ensuring maximum compression efficiency.

0 : other value types
- 00000 (0) : null (00000000)
- 00001 (1) : true (00000001)
- 00010 (2) : false (00000010)
- 00011 (3) : "" (00000011)
- 00100 (4) : [] (00000100)
- 00101 (5) : {} (00000101)
- 00110 (6) : negative integer followed by uint (-1 => 00000110 001)
- 00111 (7) : positive float followed by uint (precision) and uint (integer) (3.14 = > 00000111 00010111 01110100 00000010)
- 01000 (8) : negative float followed by uint (precision) and uint (integer) (-3.14 = > 00001000 00010111 01110100 00000010)
- 01001 - 111100 (9-60) : alphabetical single character (charmap + 9)
- 111101 (61) : non-alphabetical single character : followed by LEB128 (charcode)
- 111110 (62) : alphabetical multi characters : followed by short (charmap)
- 111111 (63) : non-alphabetic multi characters : followed by LEB128 (charcode)
1 : positive integer : followed by either
- 6 bit interger if smaller than or equal to 63 (62 => 01111110)
- or 111111 and LEB128 if bigger than 63 (70 => 63 + 7 => 01111111 00000111)

const charmap = { 
  A: 0, B: 1, C: 2, D: 3, E: 4, F: 5, G: 6, H: 7, I: 8,
  J: 9, K: 10, L: 11, M: 12, N: 13, O: 14, P: 15, Q: 16, R: 17,
  S: 18, T: 19, U: 20, V: 21, W: 22, X: 23, Y: 24, Z: 25,
  a: 26, b: 27, c: 28, d: 29, e: 30, f: 31, g: 32, h: 33, i: 34,
  j: 35, k: 36, l: 37, m: 38, n: 39, o: 40, p: 41, q: 42, r: 43,
  s: 44, t: 45, u: 46, v: 47, w: 48, x: 49, y: 50, z: 51
}

With these special rules, positive integers less than 64 and single alphhabetical characters, as well as null, true, false "", [], {}, are all encoded in just 1 byte, maximizing efficiency while minimizing storage overhead.

Benchmarking

Benchmarks against MessagePack, CBOR and BSON

WeaveDB Litepaper v0.1

KYOHEI ITO — Fri, 09 May 2025 08:55:02 +0000

original: WeaveDB Litepaper v0.1

Decentralized NoSQL Database as a Smart Contract

Blockchains are revolutionizing fintech with censorship-resistant and unstoppable networks with decentralization. But so far, their primary use cases are token transfers and their scalability is not even enough for mass financial adoption. However, the internet and the web are composed of data and web3 is an antithesis to centralized data silos beyond financial sectors. Blockchains, as of today, can never scale to the web level.

Many have attempted to build a decentralized social network but failed to build fully decentralized ones. This is primarily due to the fact that a practical alternative to web2 cloud databases has not been possible by now. There have been many decentralized database projects, but all of them are domain-specific, such as blockchain indexers, p2p graph DBs, and zkp-based privacy DBs, and not a general-purpose database to replace web2 databases. As a result, most web3 social networks call themselves sufficiently decentralized and use centralized servers and databases for data indexing and complex service logic and APIs.

This is obviously a huge issue for decentralized applications because if you lose the centralized database, the services relying on the APIs for data extraction will cease their operations. And as long as a centralized entity is managing the database layer, it's bound to stop sooner or later. The database layer is what costs the most when running a scalable service and the dev team cannot keep paying for the infrastructure forever. There have been many such examples where the provider arbitrarily changes the API pricing or stops APIs altogether and closes down the service itself.

You could build a fully decentralized application without a centralized database, but then all the logic has to run on smart contracts, and it would be super feature-limited without effective data indexing.

WeaveDB aims to solve these issues by building a fully decentralized database with the performance and scalability of web2 cloud databases.

4 Bottlenecks

A general-purpose fully decentralized database has not been realized so far due to 4 big bottlenecks:

Cost

Blockchain computation and storage are prohibitively expensive, including L2 rollups. Ethereum requires $10M to store 1GB of data.
Performance

Blockchain finality is slow, ranging from 500ms to 10 mins. But a practical DB requires less than 1ms finality with 10,000 - 100,000 TPS.
Scalability

Storage size is bottlenecked by the prohibitive cost. As of 2024, the total Ethereum storage size is just around 1TB. More than 300M TB of data is created every day on the web and it's accelerating.
Access Control

Unlike web2 databases with admins, decentralized databases are permissionless, and anyone can write anything from anywhere by default. Precise access control is required, but if we should go beyond simple data ownership, it requires highly flexible logic with complex implementations, and no database projects have figured it out.

So, building a practical decentralized database as a smart contract seems impossible, but in this paper, we would like to present a solution with the Arweave permanent storage.

Why Smart Contract Database?

Before getting into the solution, I'd like to put an emphasis on the smart contract database. You could build a decentralized database using existing technology and decentralize the network of nodes running the database. However, this approach limits the usability of the database to a great extent since the core components of web3 databases and web2 databases should be totally different. Instead, we built a brand new database from scratch, all as smart contract components, which unlocks the key features within the context of decentralized networks.

Verifiability of Computations and States All the access logic and computation for data manipulations should be recorded on-chain and verifiable by anyone. Only smart contracts and their storage-based states can enable such public verifiability.
Composability with Other Smart Contracts Smart contracts are the building blocks of decentralized applications and are composable with other smart contracts. WeaveDB can be a part of other smart contract protocols, such as DeFi, SocialFi, NFT, DAO, blockchain games, etc.
Interoperability with Other Blockchains Once it's a smart contract protocol and managed by a decentralized network, it will open up interoperability with other blockchains. We implement 3 types of interoperability, explained in detail in the later sections.
- Zero Knowledge Proof (ZKP)
- Inter-Blockchain Communication (IBC)
- Threshold Signature Scheme (TSS)

All these features will be unlocked with the blockchain standard protocols only if the database itself is a smart contract on a blockchain.

We define smart contracts as an immutable script verifiable by anyone on a blockchain with decentralized networks. In this sense, Solidity on EVM is not the only smart contract implementation; smart contracts can be written in any language on any VM.

How Do We Solve This Problem

Arweave Storage-based Consensus Paradigm (Cost)

Arweave is a blockchain-like protocol (Blockweave) specializing in permanent storage. You need to pay a small one-time fee when uploading, and your data will be stored virtually forever (at least 200 years in the most pessimistic estimation). It costs around $5 per GB, which is 400 thousand times cheaper than Ethereum, and it even gets cheaper as the storage hardware cost decreases every year.

There is a novel way to build a smart contract layer on permanent storage, and this methodology is called Storage-based Consensus Paradigm (SCP). SCP is an idea to use the storage consensus with off-chain computation to drastically lower the cost of the smart contract layer. In essence, all the data required for smart contract evaluations are stored permanently on Arweave, and state evaluations can be lazy-executed on the client side and off-chain, which costs nothing. So it's a combination of cheap permanent storage and free off-chain computations that enable the smart contract layer. It's called SmartWeave.

Warp SmartWeave Sequencer (Performance)

Arweave does not adopt absolute finality, so the safe block time is something around 20 minutes. This, of course, is too slow for a smart contract execution layer, so Warp took it further with an efficient sequencer to immediately order and finalize transactions without any evaluation. It bundles up smart contract transactions and uploads to Arweave via bundle protocols such as Irys and ArIO. This decreases time to finality (TTF) to just 10ms.

As mentioned earlier, the Warp sequencer doesn't evaluate the contract states and it's a lazy evaluation on the client side, which enables even faster performance and lower latency than web2 cloud databases. This, however, opens up a different set of issues that led us to the WeaveDB Rollup development, addressed at the end of this paper.

KV Storage Adaptors (Scalability)

The entire database is implemented as a smart contract, but storing all the data as a smart contract state is not scalable and portable at all. Every time the database needs to evaluate something, it needs to load the entire database to memory, which is impossible if you have, for instance, 100 million datasets.

Warp SmartWeave, fortunately, has implemented a protocol to enable underlying KV storage with only interface (get/put/del). This works as a KVS adaptor, and how you store the data via the KV interface is up to the node implementation. A node can use any existing database technology to deterministically store data via the interface, and how much the database can scale will be up to the KV implementation of the node, and not up to the smart contract constraints. This is more future-proof since the hardware and database technology will improve in the future. In this way, we make the database indefinitely scalable. This is equivalent to Ethereum client diversity.

WeaveDB only stores minimum database configurations as the contract state. The actual data and indexes are stored in the underlying KV storage set up by the node in the way the node adaptor script defines.

Besides this, the Arweave network itself is supposed to be indefinitely scalable. The upper bound is only constrained by the storage cost. So, if a database needs to store a huge amount of data, in theory, someone just has to pay for it. Then, how to handle the data via the smart contract KVS interface is up to the node KVS.

FPJSON (Access Control)

Simple data ownership logic can be implemented with the simple signature verification flow, but that's not how web2 databases work. Web2 database is indeed a combination of application logic on the server/client side and data access rules set to the database. Since you cannot control where users access the database from, to fully cover such web2 use cases, a web3 database should contain highly advanced application logic on the database side as well as advanced access control rules. In other words, data ownership should be programmable within the database logic. How do we achieve all these with an immutable smart contract?

The answer is FPJSON, a domain-specific language (DSL) we developed to enable functional programming in the form of JSON arrays. FPJSON comes with more than 250 composable functions and you can express any level of advanced logic as a JSON array.

["add", 1, 2][("difference", [1, 2, 3], [3, 4, 5])] = // = 3
  [1, 2][(["map", ["inc"]], [1, 2, 3])][ // = [4, 5, 6]
    (["compose", ["map", ["inc"]], ["difference"]], [1, 2, 3], [3, 4, 5])
  ]; // = [2, 3]

FPJSON plays a pivotal role in a smart contract database because any logic can be stored as a contract state and reused without deploying a new contract to extend database functionalities. Without FPJSON, one would have to deploy a new smart contract every time to update access control rules to the database.

Here's an example of how complex access control rules are defined with FPJSON.

[
  // define a query type to follow a user ($signer=$from follows $to)
  "set:follow",
  [
    // split the document id (ids are in "from:to" format)
    ["split()", [":", "$id", ["=$from", "=$to"]]],

    // check if the signer is the from (signer is verified with signature)
    ["=$isFromSigner", ["equals", "$from", "$signer"]],

    // modify the uploaded data to add who follows whom with the query timestamp
    ["mod()", { from: "$from", to: "$to", date: "$ms" }],

    // accept the query only if $isFromSigner is true
    ["allowif()", "$isFromSigner"],
  ],
];

WeaveDB comes with a very powerful set of unique features required for a decentralized database, and they are all defined with FPJSON. You don't have to upgrade your database by a new contract deployment, but you can simply update a contract state in JSON format to upgrade your database with new functionalities. WeaveDB itself is more powerful than any smart contract, such as EVM's Solidity.

await setRules(fpjson_snippet, COLLECTION_NAME);

This is all it takes to upgrade your database logic, which is as powerful as deploying a new smart contract.

The 3 big takeaways about FPJSON.

Any level of complex logic is constructible by composing 250+ functions in JSON format.
No need to deploy a new smart contract to upgrade the database logic since FPJSON is just reusable JSON data stored as smart contract states
Everything is JSON in WeaveDB, and FPJSON enables powerful features unique to decentralized databases

FPJSON is what enables a general-purpose decentralized database with programmamble data ownership and also allows complete replacements of web2 cloud databases.

WeaveDB Queries

WeaveDB queries are mostly compatible with the Firestore queries, but with a much simpler syntax. Everything within WeaveDB is expressed as JSON including queries due to storability as a smart contract state.

With the Firestore SDK, you would make a query like

const ppl = (
  await firestore
    .collection("people")
    .orderBy("age", "desc")
    .where("age", ">", 20)
    .limit(5)
).map((doc) => docv.data());

The WeaveDB SDK has syntactic sugar and simplifies the queries.

const ppl = await weavedb.get("people", ["age", "desc"], ["age", ">", 20], 5);

It's not just for simplicity but for compatibility with the JSON format. The WeaveDB queries are represented as a JSON array in FPJSON snippets and reused as a part of access control rules, crons and triggers. With the Firestore query syntax, we cannot do this.

const query = ["get", "people", ["age", "desc"], ["age", ">", 20], 5]]

Available Query Types.

await get("people", "Bob");
await add({ name: "Mike", age: 15 }, "people"); // docID will be auto-generated
await set({ name: "Bob", age: 20 }, "people", "Bob");
await update({ age: 30 }, "people", "Bob");
await upsert({ name: "Alice", age: 40 }, "people", "Alice");
await delete ("people", "Alice");

To get multiple docs, you can use conditional queries.

// let's add some people
await add({ name: "Bob", age: 15, drinks: ["milk", "juice"] }, "ppl")
await add({ name: "Alice", age: 20, drinks: ["cola", "mocha"] }, "ppl")
await add({ name: "Mike", age: 30, drinks: ["water", "beer"] }, "ppl")

// then get them with a condition
await get("ppl", ["age", "==" 20]) // Alice
await get("ppl", ["age", "!=" 20]) // Bob, Mike
await get("ppl", ["age", ">" 20]) // Mike
await get("ppl", ["age", ">=" 20]) // Alice, Mike
await get("ppl", ["age", "<" 20]) // Bob
await get("ppl", ["age", "<=" 20]) // Bob, Alice
await get("ppl", ["age", "in", [ 20, 21, 22 ]]) // Alice
await get("ppl", ["age", "not-in", [ 20, 21, 22 ]]) // Bob, Mike
await get("ppl", ["drinks", "array-contains", "beer"]]) // Mike
await get("ppl", ["drinks", "array-contains-any", ["milk", "cola"]]]) // Bob, Alice

You can also define custom queries described earlier. If the data fields are auto-filled with mod() in the access rules, you can send an empty data {}.

await query("set:follow", {}, "follows", "Bob_addr:Alice_addr");
// { from: "Bob_addr", to: "Alice_addr", date: TIMESTAMP } will be saved

batch query bundles multiple queries from the same signer, and bundle query bundles single queries and batch queries from different signers. The WeaveDB rollup uses bundle queries to commit transactions to the upper layer.

await db.batch([
  ["set", { name: "Bob" }, "ppl", "Bob"],
  ["upsert", { name: "Alice" }, "ppl", "Alice"],
  ["delete", "John"],
]);

const batch = [
  ["set", { name: "Bob" }, "ppl", "Bob"],
  ["upsert", { name: "Alice" }, "ppl", "Alice"],
];
const batch_query = await db.sign("batch", batch, { evm: wallet });
const single_query1 = await db.sign("add", { name: "Mike" }, "ppl", {
  evm: wallet2,
});
const single_query2 = await db.sign("add", { name: "Beth" }, "ppl", {
  evm: wallet3,
});

await db.bundle([batch_query, single_query1, single_query2]);

WeaveDB Indexers

WeaveDB uses a unique B+ tree optimized for the Warp KVS adaptor interface. All indexes are on-chain permanently stored on Arweave, which means you will never lose access to the indexes unlike off-chain centralized databases. Technically speaking, the logic to build indexes is stored on-chain, so if you evaluate the smart contract, the contract will automatically rebuild the indexes. Indexes themselves are stored on the underlying KV storage with the node.

Single-field indexes are automatically added, but the contract owner needs to add multi-field indexes to make conditional queries using them avairswiiable.

await db.addIndex("ppl", [
  ["age", "desc"],
  ["height", "asc"],
]);

This call will add a B+ tree to index ppl by sorting them by age in descending order first, then by height in ascending order second. Then queries like

await get("ppl", ["age", "desc"], ["height", "asc"], ["age", ">", 20]);

will be available on ppl collection.

This design, however, makes the contract owner the single authority to add indexes to the database. So unless the specific indexes required for your app are added by the owner, the database will be useless for your app.

Modular Database

To address this issue, we took a modular database approach. In this design, indexers are detached from the data source, and anyone can add arbitrary indexers to the core data contract permissionlessly for their own applications. The architecture consists of 4 layers.

Data Sources

The core database contract to handle configurations and data storing.
Indexers

Indexers to handle efficient data extractions. Optimized B+ trees for NoSQL database.
Query Parsers

Evaluate API queries and articulate which indexers to use to extract the requested data.
Public APIs

The public interfacing contract to define APIs to access the database.

This modular architecture introduces a new paradigm to decentralized databases. The same data sources are shared with multiple applications with permissionless indexers, and we can even build different database types (RDB/NoSQL/GraphDB) on top of the same data sources. The public APIs are permanently available, and dapps relying on the APIs can function forever without ever losing access to the data.

One can also build off-chain indexers using the permanent data sources depending on your use cases.

WeaveDB Features

The data ownership and access control of decentralized databases are completely different from web2 databases. So, we need a very different set of features to build decentralized versions of web2 applications. To articulate what features are required, we have built a fully decentralized Twitter on top of WeaveDB and found out just decentralizing web2 database would never work, and implemented all the missing database features. WeaveDB is the first decentralized database capable of building a fully decentralized Twitter alternative on top.

I will list the unique set of features, and why decentralized databases need them.

Crypto Wallet Authentication

Decentralized applications need a way to authenticate users permissionlessly, and public key cryptography adopted by many blockchains is the best way to do so. WeaveDB integrated multiple types of public key algorithms, which are compatible with many blockchain networks.

secp256k1 (Bitcoin / Ethereum / EVM)
Ed25519 (Dfinity / Cosmos)
RSA (Arweave)
BabyJubJub (PolygonID / Intmax)

WeaveDB can also authenticate other web3 identity protocols via a relayer mechanism.

Lens Profile NFT (Lens Protocol)
NextID (Mask Network)
WebAuthn (Biometric Authentication)

The authentication component is extensible. Schnorr and BLS (Boneh-Lynn-Shacham) are also planned to be integrated.

Data Schema

Decentralized database typically requires precise data schema, even with document-based NoSQL databases, because anyone can update a malformed data set and break the applications expecting consistent data structures. We believe flexible data schema has an advantage over SQL table schema, but freestyle data schema (or no schema) has zero benefit in the first place because most applications should expect a certain fixed data model.

WeaveDB adopts JSONLogic to define the data schema for each collection.

{
  follows: {
    type: "object",
    required: ["from", "to", "date"],
    properties: {
      from: { type: "string" },
      to: { type: "string" },
      date: { type: "number" },
    },
  },
}

Access Control Rules

If you want just 1 to 1 data ownership, all you need to do is to verify a signature of the data uploader, and most p2p database projects are some variants of it. But this is not how web2 databases work at all. Web3 is about data ownership, so assigning an owner to each data is a good start, but the ownership should be programmabble and most data should have shared ownership with predefined logic. This is where FPJSON comes in.

With FPJSON you can set up any level of complex access control rules for each collection, and you can also modify the uploaded data within the rules to add/remove data fields, and mutate data before committing to the database. You can also set up any number of custom queries with separate rules.

Example custom query rules to follow a user explained in the previous section.

[
  "set:follow",
  [
    ["split()", [":", "$id", ["=$from", "=$to"]]],
    ["=$isFromSigner", ["equals", "$from", "$signer"]],
    ["mod()", { from: "$from", to: "$to", date: "$ms" }],
    ["allowif()", "$isFromSigner"],
  ],
];

Auto-Executed Crons

By utilizing the lazy-execution nature of SmartWeave, WeaveDB enables periodically executed queries without producing transactions. Contract states are only lazy-evaluated when some data are requested on the client side. Once a cronjob is defined, WeaveDB can calculate the execution schedule and retrospectively apply the execution before returning the latest data. In this way, you don't need a keeper node to make periodical transactions like Ethereum smart contracts.

Auto-Executed crons are useful for managemental tasks such as deleting transient/expired data and calculating trending posts based on the number of likes in a certain period.

Query Triggers

Triggers are one of the most essential features when it comes to building complex applications. A trigger can trigger one query from another based on conditions defined with FPJSON. For instance, let's think about how we should update data collections when someone likes a post. A good database structure would be to have 2 collections, one for posts and the other for likes. Each post doc should have a like count, but post should only be updatable by the author. So, the liker doesn't have write access to the post he/she is liking. In this case, a query to add a like should trigger an internal query to increment the like count in the post. In this way, the liker can update the like count of the posts they don't have access to update.

This was a simple example, but this kind of query chaining is everywhere if you are building a web2 application. Firestore can handle this with the internal system with serverless functions, but decentralized databases don't come with such an integrated system. WeaveDB triggers enable that function within smart contract logic. You can never build a fully decentralized Twitter without the internal query chaining to overload data access and ownership.

Relayers

Smart contracts need an oracle to bring data from outside sources. WeaveDB has a built-in relayer mechanism to verify the relayer's signature and add/modify values with predefined logic. For example, you can set up a relayer to check the ownership of some NFTs on Ethereum, and the relayer signs for the validity and adds the owner's address to the uploaded data. WeaaveDB verifies the relayer's signature and merges the extra data with the uploaded data before committing to the database.

You can combine the WeaveDB relayer with the Lit protocol's PKP to build verifiable relayers.

ZKP verifiers

WeaveDB also comes with an adaptor interface to connect with zkp verifiers. You can write a zk circuit and make WeaveDB verify the proofs to validate some data without revealing the data itself. This opens up many novel use cases connecting with off-chain data. For instance, you could verify only a specific piece of information in your verifiable credential (VC) without revealing the actual data.

zkDB / zkRollup

In a separate litepaper, we wrote about zero knowledge provable JSON (zkJSON) which makes any JSON data verifiable directly from Ethereum smart contracts. WeaveDB is a NoSQL database, and each document is indeed zkJSON, which forms a zkDB as a whole. So whatever the data you store on WeaveDB will be directly queriable on other blockchain smart contracts if you turn on the zkRollup feature. We will talk about zkRollup in yet another litepaper.

WeaveDB smart contract stores merkle trees for zkDB and anyone can verify the latest merkle hash with a simple call. The Warp SmartWeave has plug-in extensions (equivalent to EVM precompiles), and WeaveDB comes with a precompiled zk prover, so anyone can generate proofs for any data on the client-side hain and make a query from Ethereum or other EVM blockchains. This is kind of equivalent to an on-chain zk prover.

Issues with This Approach

Decentralized database as a SmartWeave contract enables the performance and scalability of web2 databases with super low-cost thanks to the Arweave permanent storage. This design, however, brings up a new set of issues.

Client-Side Scalability

Since computations are lazy-executed on the client side, now clients have the burden to evaluate the current state of the contract and to do so, they need to cache the entire transaction history of the contract by downloading transactions from an Arweave gateway. This takes minutes if there are millions of transactions. And each client has to do this. This will never scale on the client side. One way to solve this is to have an intermediary evaluation node clients can access to instantly get the latest contract states. But this approach again doesn't solve the second problem described in the next section.

Data Consistency with Low Latency

SmartWeave can provide performance of cloud databases, or even better, this is enabled by 2 factors.

Computations are executed locally on the client side
The Warp sequencer doesn't evaluate anything

Due to these 2 optimizations, data consistency is actually not guaranteed because your client is not sure if it's in sync with other clients, and the sequencer doesn't evaluate the contract state and ensure the state validity, so you don't know what your data will turn out to be until you get finality from the sequencer and evaluate the current state with your new finalized cache. If you really wanted to make sure of the data consistency like any database should, you would need to execute queries in sequence and one at a time, which would take 3 to 5 seconds to evaluate the current state before committing and after getting finality.

If your application can tolerate either high latency or data inconsistency, you will be just fine with bare SmartWeave contracts.

But to replace web2 use cases of high-performance applications with data consistency and low latency, we will need further optimizations, which will be explained in the WeaveDB Rollup litepaper.

DEV Community: WeaveDB

WAO - Wizard AO SDK

Quick Start

Installation

Drop-in aoconnect Replacement for Tests

Setting up a Project

Writing Tests

Using WAO SDK

Cherry-Picking Outputs

Determining Message Success

Async Message Tracking with receive()

Logging

Fork Wasm Memory

WeaveDrive

Local Persistent Server

HyperBEAM

API Reference

AO

Instantiate

AO Core Functions

deploy

msg

dry

res

ress

asgn

load

eval

spwn

aoconnect Functions

Advanced Functions

postModule

postScheduler

attest

avail

wait

var

Process

Instantiate

msg

m

dry

d

res

r

v

Function Piping

pipe

bind

then

err

cb

AR

Instantiate

Set or Generate Wallet

Token Related Methods

toAddr

mine

balance | toAR | toWinston

transfer

checkWallet

Storage Related Methods

post

tx

data

bundle

GQL

Instantiate

Txs

asc

first

after

next

block

by Transaction IDs

by Recipients

by Owners

by Tags

fields

Blocks

Drop-in `aoconnect` Replacement for Tests