DEV Community: Phat

Implement WitnessCalc in native apps Pt.2

Phat — Thu, 21 Nov 2024 20:06:08 +0000

Previously we have implemented witness calculation in android native

Now, let’s do the same for iOS

Open our root ios folder in xCode and after indexing we will see:

Our project will be under the Pods/Development Pods:

scroll down and search for our module:

open RnWtnscalcs.mm file which is the analogue of our RnWtnscalcsModule.kt file in android project

As we haven’t implemented our generated Spec file yet, the xCode will offer you to fix that:

To go to our spec file, you could just cmd+click on RnWtnscalcs implementation

and cmd+click on NativeRnWtnscalcsSpec

Now let’s get back to our RnWtnscalcs.mm file and implement plus function

as it is objective-c, we could just import our RnWtnscalcs.h file and be able to call functions straight from RnWtnscalcs.cpp file

and mock generateAuthWtns for a while

now, in android project, from the android studio, let’s remove “#include ” from rn-wtnscalcs.cpp if your file contains it, and move it to cpp-adapter.cpp if it not exists there

this include related only to android project and shouldn't be in common cpp file

And rebuild our project for ios:

npx expo prebuild --platform ios --clean && npx expo run:ios

after start running this command, ur xcode will offer you to re-save project, so wait till the generation ends, and click-re-save

And then “Read from disk”

Finally we got our functions works:

Okay, after we get familiar with iOS part of implementing methods, let’s integrate witnesscalc

Let’s see, how it will looks like at the end:

You could see familiar things here, e.g. WtnsUtils, or “dat” in datAssets.xcassets, but let’s get it step by step:

first things first - the library. And that is the most confusing part

We need a libgmp - the helper library, that our witnesscalc_auth depends on

move to out witnesscalc repo and open ./build_gmp.sh file

we will se build variants here, and if you testing on real device - choose iOS e.g.

./build_gmp.sh ios

for simulators would be:

./build_gmp.sh ios_simulator

Why?

Because xCode wouldn’t let you build the app for Simulator if the library was compiled for iOS and vise-versa

Next, we need to build the witnesscalc_auth itself, for iOS, as we did for android

And here we have 2 options, both can run on device and simulator, but it depends on “which one" exactly, for example “ios” option is running only on devices and simulators on arm64 architecture, such as iPhone 15, 14, …etc, (and simulators too)
same thing for x86_64

In this guide we will test on iPhone 15 Pro simulator, so:

make ios

After compile - we will see freshly created build_witnesscalc_ios folder, and new subfolders under ./depends/gmp directory

build_witnesscalc_ios folder is where our “target” library will be

and the ./depends/gmp folder is where our helper library is, let’s start with it first:

you need a libgmp.a file:

chose your variant and copy to our module/wtnscalcs/cpp folder

Now, open the ./build_witnesscalc_ios folder in xCode

you should see something like this:

we need to chose witnesscalc_authStatic variant:

and build for our iPhone 15 Pro simulator

Unfortunately we couldn’t chose “Any iOS simulator device” cuz it contains x86_64, and the libs we have compiled are not support it.

but for production you could build apropriate variants in witnesscalc repo and chose any iOS device(arm64).

So, let's run build for witnesscalc_authStatic and also “fr”

It’s also the library, that witnesscalc_authStatic depends on

build it, and then open products folder, our builded libraries would be there:

copy them to our ./modules/wtnscalcs/cpp folder next to libgmp.a file:

Then, create bridge.h file:

#ifndef bridge_h
#define bridge_h

#include "witnesscalc_auth.h"

#endif /* bridge_h */

it will let our iOS project access this libraries

Now, open root project ios folder in xCode and add new File to our module ios folder

select Asset Catalog under Resources section

open that file and press “add”(plus symbol at the bottom-left corner), chose Data Set

name it “authDat” as it shown above and drag-n-drop our auth.dat file inside, the same auth.dat file, which we use in android app, just copy it.

Now we need to configure our podspec file, so our changes and linking will not vanish out after each pod-install or app rebuild

s.resource_bundles = {
  "RnWtnscalcsDatAssets" => ["ios/datAssets.xcassets"]
}

s.source_files = "ios/**/*.{h,m,mm,swift}", "cpp/**/*.{hpp,cpp,c,h}"

s.vendored_libraries = "cpp/*.a"

s.preserve_paths = ['cpp/*.a']
s.libraries = "gmp"

# Add GMP headers
s.public_header_files = 'cpp/*.h'
s.xcconfig = { 'HEADER_SEARCH_PATHS' => '"$(PODS_ROOT)/Headers/Public/rn-wtnscalcs/cpp"' }

s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'OTHER_LINKING_FLAGS' => '-lc++' }

These configurations will provide correct linking to our libraries and datAssets.

After that, get back to our react-native “frontend" app, and run:

npx pod-install

wait till-the end and “re-save” our xCode project as we did before.

Don’t be afraid if the file structure in your module will change, it’s okay, for example mine was like that after pod-install:

So, our preparations done and we could move to the code:

Create WtnsUtils m and h files under ios folder as it shown above

define our future WtnsUtils class and import witnesscalc_auth.h file

#ifndef WtnsUtils_h
#define WtnsUtils_h

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>
#import "../cpp/witnesscalc_auth.h"

//NS_ASSUME_NONNULL_BEGIN

@interface WtnsUtils : NSObject

+ (NSData *)calcWtnsAuth:(NSData *)privateInputsJson error:(NSError **)error;

@end

//NS_ASSUME_NONNULL_END

#endif /* WtnsUtils_h */

import it in WtnsUtils.m file and create our WtnsUtils class

#import <Foundation/Foundation.h>
#import "WtnsUtils.h"

@implementation WtnsUtils

static const NSUInteger ERROR_SIZE = 256;
static const unsigned long WITNESS_SIZE = 100 * 1024 * 1024;

+ (void)initialize {
    if (self == [WtnsUtils self]) {
        NSLog(@"WtnsUtils initialized");
    }
}

+ (NSData *)calcWtnsAuth:(NSData *)privateInputsJson error:(NSError **)error {
    NSBundle *bundle = [NSBundle bundleWithIdentifier:@"org.cocoapods.RnWtnscalcsDatAssets"];

    // Check if bundle is loaded correctly
    if (!bundle) {
        NSLog(@"Bundle not found");
        if (error) {
            *error = [NSError errorWithDomain:@"WtnsUtilsErrorDomain" code:1 userInfo:@{NSLocalizedDescriptionKey: @"Resource bundle not found"}];
        }
        return nil;
    }

    // Attempt to load the data asset
    NSDataAsset *authDat = [[NSDataAsset alloc] initWithName:@"authDat" bundle:bundle];
    if (!authDat) {
        NSLog(@"authDat file not found in bundle");
        if (error) {
            *error = [NSError errorWithDomain:@"WtnsUtilsErrorDomain" code:1 userInfo:@{NSLocalizedDescriptionKey: @"authDat file not found in bundle"}];
        }
        return nil;
    }

    return [self _calcWtnsAuth:authDat.data privateInputsJson:privateInputsJson error:error];
}

+ (NSData *)_calcWtnsAuth:(NSData *)descriptionFileData privateInputsJson:(NSData *)privateInputsJson error:(NSError **)error {
    unsigned long *wtnsSize = (unsigned long *)malloc(sizeof(unsigned long));
    *wtnsSize = WITNESS_SIZE;
    char *wtnsBuffer = (char *)malloc(WITNESS_SIZE);
    char *errorBuffer = (char *)malloc(ERROR_SIZE);

    int result = witnesscalc_auth(
      [descriptionFileData bytes], (unsigned long)[descriptionFileData length],
      [privateInputsJson bytes], (unsigned long)[privateInputsJson length],
      wtnsBuffer, wtnsSize,
      errorBuffer, ERROR_SIZE);

    if (result != WITNESSCALC_OK) {
        [self handleWitnessError:result errorBuffer:errorBuffer wtnsSize:wtnsSize error:error];
        free(wtnsSize);
        free(wtnsBuffer);
        free(errorBuffer);
        return nil;
    }

    NSData *resultData = [NSData dataWithBytes:wtnsBuffer length:(NSUInteger)*wtnsSize];
    free(wtnsSize);
    free(wtnsBuffer);
    free(errorBuffer);
    return resultData;
}

+ (void)handleWitnessError:(int32_t)result errorBuffer:(char *)errorBuffer wtnsSize:(unsigned long *)wtnsSize error:(NSError **)error {
    if (result == WITNESSCALC_ERROR) {
        NSString *errorMessage = [[NSString alloc] initWithBytes:errorBuffer length:ERROR_SIZE encoding:NSUTF8StringEncoding];
        if (errorMessage) {
            errorMessage = [errorMessage stringByReplacingOccurrencesOfString:@"\0" withString:@""];
            if (error) {
                *error = [NSError errorWithDomain:@"com.example.WtnsUtils" code:result userInfo:@{NSLocalizedDescriptionKey: errorMessage}];
            }
        }
    } else if (result == WITNESSCALC_ERROR_SHORT_BUFFER) {
        NSString *errorMessage = [NSString stringWithFormat:@"Buffer too short, should be at least: %lu", *wtnsSize];
        if (error) {
            *error = [NSError errorWithDomain:@"com.example.WtnsUtils" code:result userInfo:@{NSLocalizedDescriptionKey: errorMessage}];
        }
    }
}

@end

as you see, we could access witnesscalc_auth method straight from .a library, cuz obj-c can interact with c++, so we don’t need any bindings and CMakeFiles like we did for android

And here we will get access to our bundle, that we defined in podspec file

s.resource_bundles = {
  "RnWtnscalcsDatAssets" => ["ios/datAssets.xcassets"]
}

and search for authDat asset, where our auth.dat file is.

note: that ur bundle identifier could be named differently

if you unsure, you could log all your bundle identifier and find the right one:

NSArray *bundles = [NSBundle allBundles];
for (NSBundle *bundle in bundles) {
    NSLog(@"Bundle Identifier: %@", [bundle bundleIdentifier]);
}

Now, import WtnsUtils.h file in RnWtnscalcs.mm

and implement our generateAuthWtns method:

- (void)generateAuthWtns:(NSString *)jsonInputsBase64 resolve:(RCTPromiseResolveBlock)resolve reject:(RCTPromiseRejectBlock)reject {
    NSError *error = nil;

    // Convert base64 encoded string to NSData
    NSData *jsonData = [[NSData alloc] initWithBase64EncodedString:jsonInputsBase64 options:0];

    if (!jsonData) {
        reject(@"error", @"Failed to decode base64 string", nil);
        return;
    }

    NSData *result = [WtnsUtils calcWtnsAuth:jsonData error:&error];

    if (error) {
        NSLog(@"Error: %@", error.localizedDescription);
        reject(@"error", error.localizedDescription, error);
    } else {
        // Encode result to base64 and resolve
        NSString *resultBase64 = [result base64EncodedStringWithOptions:0];

        resolve(resultBase64);
    }
}

Now, rebuild the app, and run

npx expo prebuild --clean && npx pod-install && npx expo run:ios

🎉🎉🎉

Part 3 (.AAR & .xcframework) | Wrap everything up to simplify development and delivery process

Implement WitnessCalc in native apps Pt.1

Phat — Thu, 21 Nov 2024 20:03:45 +0000

In this ~~short~~ guide we will roughly implement generated witnesscalc .cpp files from circom in react-native app.

I'll do it in expo project.

Circom files will be borrowed from passport-zk-circuits

And finally, we will use 0xPolygonID/witnesscalc to compile executable files for different native platforms.

Prerequisities:
By following this guide, let's assume you are already familiar with react-native

Android

Getting started

install Android Studio
open the settings and install ndk v.23.1.7779620

configure path in your .bashrc or .zshrc config file

clone https://github.com/rarimo/passport-zk-circuits, open project, and install all deps.
follow readme to compile circuits
we will test on auth.circom
after compile we should get something similar to:
auth.cpp and auth.dat is the files we will use
clone https://github.com/0xPolygonID/witnesscalc
follow readme "preparation" to install necessary deps

Now let's take a look at src folder:

As you can see, for every [name].cpp and [name].dat files, there is also attached witnesscalc_[name].cpp and witnesscalc_[name].h files. And we need to do the same.

So, let's do it step-by-step:

First things first: wrap content inside our auth.cpp file,

#include ... 
#include ... 

namespace CIRCUIT_NAME {

// millions of code lines here

} // namespace

remove #include and replace all "assert(" with "check(", that's will return exception onError in runtime, instead of crash

Open the root ./CMakeLists.txt file and scroll down to install(TARGETS...) and add the name of our files, e.g. "auth"

create witnesscalc_auth.cpp and witnesscalc_auth.h files under src folder

You could check all the same files, their content is exact same, except the name, so we have done like so.

now add these files to ./CMakeLists.txt

And finally, open ./src/CMakeLists.txt and add this code at the end of file

as before, you should notice, that this code is the same as code above in this file, so we just copied them and changed name

And these steps should be done for every .cpp & .dat files you will add.

in this case you can use this bash script

Also you could comment all unnecessary files in compile list, cuz it’s very time consuming process

in ./src/CMakeLists.txt file

in root ./CMakeLists.txt file

except fr, it’s necessary

and also except src/witnesscalc.h, it’s necessary too

Now you can run these commands to compile your executable files

./build_gmp.sh android // only once
make android // every time u want to compile new wtns calcs

The result:

libwitnesscalc_auth.so - is what we need for our android app, along with witnesscalc_auth.h file

Now we can jump to our react-native app.

note, that in this guide i'm using new architecture, so you could follow this guide to turn it on.

We will implement our generated file in native-module

And use https://github.com/callstack/react-native-builder-bob for that.

so let's start

npx create-react-native-library@latest wtnscalcs

After that, run yarn install, rebuild app, then run android, and we will be able to use this module, cuz there is already been example methods generated by “builder-bob”

npx expo prebuild --platform android --clean && npx expo run:android

the usage is simple, as usual npm-package

import { multiply } from 'rn-wtnscalcs'

console.log(multiply(2, 3))

Now, let’s dive deep into our module

this is what we see after scaffolding our module, and to modify the code, it’s would be better to open via android-studio

we ARE NOT going to open this android folder, but should open a root android folder, this one:

and after opening it in android-studio and waiting for indexing we will see:

here is our module, and by modifying it, our code in ./modules/[module]/android folder will be changed too

so what we see now:

The entry for us would be build.gradle file of our module:

and the cmake structure which points to our CMakeLists.txt, which is register our future cpp file with logic

next, the rn-wtnscalcs.cpp file, where common logic are placed

and then .h file for it

The bridging file, which bridging cpp code to java/kotlin code

Now, let’s back to our "frontend perspective" e.g IDE or editor where your react-native project are, and take a look in to spec file in module, which responsible to register our module with native code and define top-level interface

By defining methods in this "Spec" interface

and after run

npx expo prebuild --platform android --clean && npx expo run:android

we will generate abstract class in our android project

And after that we could implement this class in main module .kt file

And by exposing method from index.tsx we will be able to call method from module as a regular npm-pkg

Okay, let's add new "plus" method just for practice:

import { multiply, plus } from 'rn-wtnscalcs'

console.log(multiply(2, 3))
console.log(plus(2, 3))

Nice, now let's jump straight in to hard mode:

let's define what we need:
1) first of all, at the very top level we have "inputs" which witness calculator will accept in the future
2) we need to send these inputs to our native module, pass it to .so file
3) execute and return bytes, it should be the same as circom compiles .wtns file, but we will keep file content in ram, instead of file

generate spec file

npx expo prebuild --platform android --clean && npx expo run:android

Now, let's write this implementation code, and leave it empty for a while

Now, the most important part - we will add our .so library in to project

Add new raw directory in our res folder

and paste there our auth.dat file

add our witnesscalc_auth.h file to cpp folder

get back to "Frontend perspective", create lib folder under modules/android directory and paste there a .so file

and finally register our library in CMakeLists.txt file

cmake_minimum_required(VERSION 3.4.1)
project(RnWtnscalcs)

set (CMAKE_VERBOSE_MAKEFILE ON)
set (CMAKE_CXX_STANDARD 14)

add_library(rn-wtnscalcs            SHARED
            ../cpp/rn-wtnscalcs.cpp
            cpp-adapter.cpp
)

# Specifies a path to native header files.
include_directories(
            ../cpp
)

link_directories(lib)

add_library(auth SHARED IMPORTED)
set_target_properties(auth PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/lib/libwitnesscalc_auth.so)

target_link_libraries(rn-wtnscalcs
        android
        auth)

what we have done:

We linked directories where we've placed our .so file
Registered our .so file as library
And finally ‘target_link_libraries’ will compile it for us in one executable file, which we can call in runtime

After these steps we will be able to execute .so file in runtime

let's write the code to call it:

object WtnsUtil {
  external fun auth(
    circuitBuffer: ByteArray?,
    circuitSize: Long,
    jsonBuffer: ByteArray?,
    jsonSize: Long,
    wtnsBuffer: ByteArray?,
    wtnsSize: LongArray?,
    errorMsg: ByteArray?,
    errorMsgMaxSize: Long
  ): Int

  init {
    System.loadLibrary("rn-wtnscalcs")
  }
}

define our object with external method of our auth witness calculator

and init will load our .so library, which we have defined in CMakeLists.txt

Now, we could open options for our auth method, cuz there is no binded code for external function

these steps will create binded function in our rn-wtnscalcs.cpp file, but let’s move it to cpp-adapter.cpp

that’s the place where all external function will be called from, and now let’s just add this code inside as shown above:

    const char *circuitBuffer = reinterpret_cast<const char *>(env->GetByteArrayElements(
            circuit_buffer, nullptr));
    const char *jsonBuffer = reinterpret_cast<const char *>(env->GetByteArrayElements(json_buffer,
                                                                                      nullptr));
    char *wtnsBuffer = reinterpret_cast<char *>(env->GetByteArrayElements(wtns_buffer, nullptr));
    char *errorMsg = reinterpret_cast<char *>(env->GetByteArrayElements(error_msg, nullptr));

    unsigned long wtnsSize = env->GetLongArrayElements(wtns_size, nullptr)[0];


    int result = witnesscalc_auth(
            circuitBuffer, static_cast<unsigned long>(circuit_size),
            jsonBuffer, static_cast<unsigned long>(json_size),
            wtnsBuffer, &wtnsSize,
            errorMsg, static_cast<unsigned long>(error_msg_max_size));

    // Set the result and release the resources
    env->SetLongArrayRegion(wtns_size, 0, 1, reinterpret_cast<jlong *>(&wtnsSize));

    env->ReleaseByteArrayElements(circuit_buffer,
                                  reinterpret_cast<jbyte *>(const_cast<char *>(circuitBuffer)), 0);
    env->ReleaseByteArrayElements(json_buffer,
                                  reinterpret_cast<jbyte *>(const_cast<char *>(jsonBuffer)), 0);
    env->ReleaseByteArrayElements(wtns_buffer, reinterpret_cast<jbyte *>(wtnsBuffer), 0);
    env->ReleaseByteArrayElements(error_msg, reinterpret_cast<jbyte *>(errorMsg), 0);

    return result;

Now, lets create a class next to WtnsUtil object, which will accept our external function as a parameter, allocate memory and execute calculation, almost it should get access to our dat file, so we need to pass AssetManager and context:

class WtnsCalculator(val context: Context, val assetManager: AssetManager) {
  fun calculateWtns(
    datFile: Int,
    inputs: ByteArray,
    wtnsCalcFunction: (
      circuitBuffer: ByteArray,
      circuitSize: Long,
      jsonBuffer: ByteArray,
      jsonSize: Long,
      wtnsBuffer: ByteArray,
      wtnsSize: LongArray,
      errorMsg: ByteArray,
      errorMsgMaxSize: Long
    ) -> Int
  ): ByteArray {
    val DFile = openRawResourceAsByteArray(datFile)

    val msg = ByteArray(256)

    val witnessLen = LongArray(1)
    witnessLen[0] = 100 * 1024 * 1024

    val byteArr = ByteArray(100 * 1024 * 1024)

    val res = wtnsCalcFunction(
      DFile,
      DFile.size.toLong(),
      inputs,
      inputs.size.toLong(),
      byteArr,
      witnessLen,
      msg,
      256
    )

    if (res == 2) {
      throw Exception("Not enough memory for wtns calculation")
    }

    if (res == 1) {
      throw Exception("Error during wtns calculation ${msg.decodeToString()}")
    }

    val witnessData = byteArr.copyOfRange(0, witnessLen[0].toInt())

    return witnessData
  }

  private fun openRawResourceAsByteArray(resourceName: Int): ByteArray {
    val inputStream = context.resources.openRawResource(resourceName)
    val byteArrayOutputStream = ByteArrayOutputStream()

    try {
      val buffer = ByteArray(1024)
      var length: Int

      while (inputStream.read(buffer).also { length = it } != -1) {
        byteArrayOutputStream.write(buffer, 0, length)
      }

      return byteArrayOutputStream.toByteArray()
    } finally {
      // Close the streams in a finally block to ensure they are closed even if an exception occurs
      byteArrayOutputStream.close()
      inputStream.close()
    }
  }
}

and use it in implemented function in our module.kt file, which we have leaved empty:

@RequiresApi(Build.VERSION_CODES.O)
  override fun generateAuthWtns(jsonInputsBase64: String, promise: Promise) {
    val wtnsCalc = WtnsCalculator(
      reactApplicationContext,
      reactApplicationContext.assets
    )

    try {
      val res = wtnsCalc.calculateWtns(
        R.raw.auth,
        Base64.getDecoder().decode(jsonInputsBase64),
        WtnsUtil::auth
      ).let {
        // base64
        String(Base64.getEncoder().encode(it))
      }

      promise.resolve(res)
    } catch (e: Exception) {
      promise.reject(e)
    }
  }

now, rebuild the project and let's test it:

npx expo prebuild --platform android --clean && npx expo run:android

import { Buffer } from 'buffer'
import { generateAuthWtns } from 'rn-wtnscalcs'

const runAuthCalc = async () => {
    try {
      const res = await generateAuthWtns(Buffer.from(JSON.stringify(authInputs)).toString('base64'))
      console.log(Buffer.from(res, 'base64'))
    } catch (error) {
      ErrorHandler.processWithoutFeedback(error)
    }
  }