DEV Community: Richard Leddy

Running process from a web page

Richard Leddy — Mon, 05 Sep 2022 23:26:21 +0000

I put together a little process manager for running communicating programs.

The point was to put up something simple and quickly. That was not too hard to accomplish by using Svelte for the web page, node.js, etc. for my server. And, I added a class to message-relay-services.

You can install it and use it yourself:

npm install -g copious-transitions-manager

You can get an example config file by using get-npm-assets

get-npm-assets copious-tm

Then you can run your program by calling:

copious-tm manager.conf

Once it is running, you can surf to the web page:

https://localhost:8989/

From there, you can control the processes you see in the list.

The reader is encouraged to dig around in the code, propose issues, do pull requests, etc. Indeed, the code includes Svelte, node.js, polka.js, websockets, and message-relay-services examples.

copious-transitions-manager README.md follows:

A simple web app for managing procs and sharing data between them.

The runtime provides those two functions:

A web page server with a web page for starting, stopping, updating, etc. processes.
A simple messaging system that allows the processes to share messages

The processes are managed as child processes of the copious-transitions-manager process. The child processes may send messages to the copious-transitions-manager process to store object or to fetch objects. These are simple applications of JavaScript data structures.

So far, this tool is good for managing a basic configuration for those trying out processes which may also communicate with each other. The runtime is compatible with processes that use communication interfaces provided by message-relay-services.

Install

npm install -g copious-transitions-manager

Basic Ops

Install a process from npm
Remove a process
Start a process
Stop a process
List processes
Set top level config parameters....

Run

copious-tm manager.conf

The configuration file is a JSON object that describes how to call programs from the command line.

You can get an example manager.conf file into you working directory by using get-npm-assets

get-npm-assets copious-tm

Here is what it looks like:

{
    "password" : "your password",
    "web_page_port" : 8989,
    "wss_app_port" : 8990,
    "all_procs" : {

        "test1.sh" : {
            "name" : "test1.sh",
            "run_on_start" : false,
            "attempt_reconnect" : false,
            "runner" : "bash",
            "args" : ["test/test1.sh","Print this message"]
        },
        "test2.js" : {
            "name" : "test2.js",
            "run_on_start" : false,
            "attempt_reconnect" : true,  
            "runner" : "node",
            "args": [
                "test/test2.js",
                "test/test2.conf"
            ]
        },
        "test3.js" : {
            "name" : "test3.js",
            "run_on_start" : false,
            "attempt_reconnect" : false,  
            "runner" : "node",
            "args": [
                "test/test3.js",
                "test/test3.conf"
            ]
        }

    }
}

The top level fields are as follows:

password :: you have to type this in on the web page to make anything work
web_page_port :: this is the html server port. Surf to your computer's IP and specifiy this port, e.g.: http://<my machine's ip>:8989
wss_app_port :: this is for websockets.
all_procs :: this is a map of process names to process descriptions. The web page will know about these at startup.

Have a look at the different process descriptions. You will see that they refer to programs and their descriptions.

Let's have a closer look at the fields for describing the processes.

name :: a name that identifies the process
run_on_start :: whether or not to start a process as soon as the manager comes up or wait for the admin to start it from the web page
attempt_reconnect :: if the process fails, should the manager try to restart it?
runner :: a program name that uses the parameters. This will be what runs at the outermost part of the process. It may run a script mentioned in the arguments, or it may be the name of a binary executable.
args parameters that are fed to the runner

For example, "test3.js" has a "node" runner node.js. The first parameter of the argments list is a javascript file in the test directory. So, "test3.js" will be available to run from the test page.

Once the manager is running, you can surf to the web page:

 http://<my machine's ip>:8989

On the web page you can select the process, find the "op" section and click on the "run" button.

The Web Page

You can see what the web page looks like here: web page

Note: nothing works on this page. The computer is not available for public operation. For viewing purposes only...

On the front page you can see buttons:

overview
stdout
ops

Clicking on these shows different panels.

The first panel, overview, shows a list box (processes will be seen there if you try it). Selecting a process makes it available for use in the ops panel. It can be removed as well using "remove" on the front panel.

On the overview, you can add a processes to the configuration file. You can run a command line using he exec button. You can install npm modules or uninstall them. The buttons lead to dialogs to help you enter field values.

On the stdout panel, you can view the terminal output of a process.

On the ops panel, you can run, stop, restart, update or remove a process (remove is the same as the front panel). Or, you can get access to the configuration file, config.

The confg button will look at the arguments array and ask if that is the config file you want to edit. A rudimentary editor will be opened in a panel with the config file contents. The editor expects that you will specify the configuration in JSON. The configuration will only update with parseable JSON. The JSON will be seen on a box on the left.

TO SEE THE WHOLE PAGE, you need to try out examples.

A Look at Example Apps

Here are two test program that can be managed by the copious-transitions-manager.

In the following, test2.js is the process that gets values from the communication channel, while test3.js writes values to the communication channel. You can see that test2.js calls messenger.get_on_path, while test3.js calls messenger.set_on_path.

test2.js

const {IPCChildClient} = require('message-relay-services')
const {load_json_file} = require('../lib/utils')
const {XXHash32} = require('xxhash32-node-cmake')

console.log("TEST 3 STARTS")

let hasher = new XXHash32(9347597)

let conf = load_json_file(process.argv[2])

if ( conf === undefined ) process.exit(0)

let messenger = new IPCChildClient(conf)


console.log("THIS IS TEST 2")

let any_old_key = [
    "this is a key for this is a test",
    "another key 94r9w487r you should know tests when you see them",
    "more keys are as much fun as testing lots of junk"
]

setInterval(async () => {
    let the_time =  new Date()
    let time_report = the_time.toLocaleString()
    //
    let rand_pick = Math.trunc(Math.random()*(any_old_key.length - 1))
    let txt = any_old_key[rand_pick]
    let hash = hasher.hash(txt)
    //
    let status = await messenger.get_on_path({
        "table" : "key_value",
        "hash" : hash
    },"test2")
    //
    console.log(status)
    //
    console.log("test2: One more tick: "  + time_report)

},2000)

test3.js


//
const {IPCChildClient} = require('message-relay-services')
const {load_json_file} = require('../lib/utils')
const {XXHash32} = require('xxhash32-node-cmake')


console.log("TEST 3 STARTS")


let hasher = new XXHash32(9347597)

let conf = load_json_file(process.argv[2])

if ( conf === undefined ) process.exit(0)

let messenger = new IPCChildClient(conf)

let any_old_key = [
    "this is a key for this is a test",
    "another key 94r9w487r key for testing tests",
    "more keys are as much fun as testing lots of tests"
]

let things_to_send = [
    "this is a test",
    "you should know tests when you see them - right?",
    "have fun testing lots of test with more tests"
]

setInterval(async () => {
    //
    let the_time =  new Date()
    let time_report = the_time.toLocaleString()
    //
    let rand_pick = Math.trunc(Math.random()*(any_old_key.length - 1))
    let txt = any_old_key[rand_pick]
    let hash = hasher.hash(txt)
    //
    let status = await messenger.set_on_path({
        "table" : "key_value",
        "hash" : hash,
        "v" : things_to_send[rand_pick]
    },"test2")
    //
    console.log(status)
    //
    console.log("test3: One more tick: "  + time_report)
},2000)

check it out

You can run these from the web page and see the messages they print in the stdout panel.

Later, hypothetically, you could use a different class from message-relay-services and a more sophisticated server to run the same programs. All you have to do is change the class in the require line:

const {IPCChildClient} = require('message-relay-services')

For example:

const {MessageRelayer} = require('message-relay-services')

Givent the endpoint server the programs attach to provides an appropiate hash table functionality, the programs would not have to change more.

Issues?

issues on github

dependencies

The web page have been developed in Svelte. Find the project in the manager-app diretory.

Thanks from copious.world

get-npm-assets for an easier installation

Richard Leddy — Fri, 02 Sep 2022 00:36:48 +0000

It's easy enough to do a post install in your npm package. But, after a while, package after package, the shell scripting gets tedious.

I have some packages that install executable node.js scripts in the npm global bin directory. Then, I had to go fishing around for a template for the config file. I just wanted a command that would get the stuff for me. And, I needed the command to run for a particular script in the bin directory.

Maybe it's out there. But, it seemed easy enough to make and pass on. So, today, I made get-npm-assets.

You can read the Read.me to see how to make use of it.

Just to give you a highlight, I have already put it to use for one of my other packages copious-endpoints. For that, I made a asset-map.json file. This file maps the bin scripts released by copious-endpoints to the files that are needed to run each program.

Here is the file contents:

{
    "copious-contacts" : "contact-service.conf",
    "copious-user" : "relay-service.conf"
}

This is file relates to the bin field in package.json. Here is the bin field from the module's package.json file:

  "bin" : {
    "copious-user" : "./bin/endpoint-user.js",
    "copious-contacts" : "./bin/endpoint-contacts.js",
    "add-mini-links" : "./bin/add-mini-links.js"
  },

So, to use one of the programs released the user will need to call something like this:

$copious-user relay-services.conf

But, where is that file? What should it look like? If I knew once, what if it changed?

But, I could call get-npm-assets as follows:

$get-npm-assets copious-user

Now, as a user, I don't have to look into the module directory to get an example. And, this version of the command will get the relay-services.conf instead of contact-service.conf.

You an also set up an assets directory and have the whole directory copied into a local assets directory that get-npm-assets will make for you.

Enjoy. Please leave issues in the github repository.

Svelte Nested Tabs

Richard Leddy — Tue, 07 Sep 2021 05:43:15 +0000

Svelte is Pretty Cool

Maybe something worth talking about.

What this is About

svelte-nested-tabs

Experience

I got some stuff up in a hurry. And, that may have been to do with Svelte.

I decided to make a blogging platform, one with music and videos among other things. I decided that I might have several entry pages, depending on the kind of media that is being presented. So, I needed to make grids I could just drop stuff into.

So, I made a Svelte component that I am calling The grid of things. grid-of-things. I figured that if you want an Internet of Things, you might want to see them, sometimes in a grid.

One cool thing about Svelte is that you can have a slot. A slot is called out in HTML within a component that you might use in your web page. It goes something like this:

... Bunches of HTML
<!- suddenly you are here -->
<slot>
Oh! My gosh! Someone forget to put a value in!
</slot>

So, if that is part of a component, says "Surprises.svelte". Then when the application uses it, it can put in a value from the application. For example:

<script>
import * as everything from "Surprises.svelte"

// do JavaScript stuff.

</script>
<!-- now for the HMTL part of the app -->
<div>
<Surprises {...parameters_in_an_object} >
I did not forget to put values in!
</Surprises>
<span class="stylish">So there!</span>
</div
<style> 
  .stylish { font-style: cursive }
</style>

It's like the Svelte components have attitude or something.

So, it's all great fun.

I was piking down the Svelte road, until one day...

I Needed Nested Tabs

Strange! A little hard to come by. Well, maybe its in some ultra large library that finishes loading on your page for that date you have with Rip Van Winkle. But, how about something simple?

Was making another little web app for sending message back and forth in a public P2P file system. It's doesn't have much purpose except replacing email with end to end encrypted contact mail that never touches a mail server. But, other than that it's just a page that needs nested tabs.

You can find that software in the copious-world repository called, interplanetary-contact.

I am going to make some changes to the code there. Maybe I will call it "intergalactic-contact" later on. I am going to start using UCWIDs and derived keys. But, that is for another conversation. Right now, I am worried about nested tabs.

Tabs and Panels

OK. You probably have a clue as to what I am talking about. But, just in case:

A tab is a likely a button that you will see in a row at the top of a display area.
A panel is like a box ('div' in HTML-ese) that contains information to be displayed when one of the tab buttons is clicked.

Here is a general idea:

<div class="tab-row">
<button on:click={activate_me}>tab 1</button>
<button on:click={activate_me}>tab 2</button>
<button on:click={activate_me}>tab 3</button>
</div>
<div id="tab1-panel"  class="panel"></div>
<div id="tab2-panel"  class="panel"></div>
<div id="tab3-panel"  class="panel"></div>

So, the panels are visible one at a time, depending on the button that got clicked.

I Looked Around

I looked around and found some pretty good components that will make great tabs if you just use one layer. I used one, and just decided that all the page could be under separate tabs. If I recall (just months ago) I was hoping to keep the message lists under one tab with tabs for kinds of messages nested below it. I made some compromises.

I started wondering if maybe Svelte was just too tricky to work with. I started looking around again. I found nested tabs that look wonderful and are in vanilla JS or use jQuery. OK. But, I was hoping to keep things organized with in one scheme, not many.

I tried slots. So, in my grid-of-things component, there is a single slot that is replicated many times. In fact, it is the same object allowing in different data. But, all the data has the same structure. So, each presentation is similar, but it might have a different picture, SVG, or a different link to some media.

At first I thought named slots might work. But, really name slots are just like slots without names. They allow a datum to be parsed into them for a one view. Again the structure is expected to be the same. I tried using a formula for the name, but I got a message in the compiler saying "slot names cannot be dynamic". So, that is actually blocked. I was imaging an application loop going through a list of tabs somehow generating a list of matched slots in the components.

The Nested Tabs Need Different Components in Each Panel

Something more like this:

<div class="tab-row">
<button on:click={activate_me}>tab 1</button>
<button on:click={activate_me}>tab 2</button>
<button on:click={activate_me}>tab 3</button>
</div>
<div id="tab1-panel"  class="panel">{stuff_from_ace[1]}</div>
<div id="tab2-panel"  class="panel">{stuff_from_acme[0]}</div>
<div id="tab3-panel"  class="panel">{stuff_from_the_bay[4]}</div>

Like that. One for each provider - say. Or, maybe they are all the same type. But, what you want to do is generate the panels in a for loop. The same for the buttons. Here is the Svelte way.

<div class="tab-row">
{#each tab_list as tab (tab.id)}
<button on:click={activate_me}>tab.name</button>
{/each}
</div>
{#each tab_list as tab (tab.id)}
<div id={tab.id) >
{tab.content}
</div>
{/each}

So, right there where it says tab.content, you want something like a slot. But, you can't have a slot. At least you can't have a different slot at each point in the loop.

There is a Way

Svelte does have a trick up its sleeve, it's called svelte:component.

What you do with this is use it where you might put a slot. And, you tell it what component to put there. For example:

<svelte:component this={component} {...parameters} />

Here, you bind "this" to the component to say what it's type is. And, you can pass parameters in a generic way, using the ellipsis. So, as long as you can pass a component to the tabs component, where the passed component is one imported into the Svelte app calling the nested tab component, you can put anything into the tab's panel. The parameters object would be passed as well. So, you get you data straight in the application and send it down.

For my Tabs component, I put in a loop to get my components lined up.

{#each tab_list as tab (tab.id) }
{#if active === tab.id }
<div class="tab-panel" style={(tab.style && tab.style.panel) ? tab.style.panel : (style && style.panel ? style.panel : "") }>
    {#if (tab_panels[tab.id] !== undefined) && tab_panels[tab.id] }
    <svelte:component this={tab_panels[tab.id].component} {...(tab_panels[tab.id].parameters)} />
    {:else}
        {@html tab.content}
    {/if}
</div>
{/if}
{/each}

So, you see, I passed in a list of tabs, tab|_list and a map of tab|_panels. The tab|_panels have fields pointing to the component class, and field point to parameters to parameters that go into the component.

It Needs to be Said : the Component Parameter may be a Tabs Component

So, there you have it.

The application can now decide what it wants inside a Tabs component.

<script>
import Tabs from "svelte-nested-tabs"

</script>
<!-- now for the HMTL part of the app -->
<main>
    <h1>Hello {name}!</h1>
    <div class="tab-container" >
        <Tabs {...tab_def} />
    </div>  
</main>

The innocuous variable, tab_def has a longish definition. One you might want to grab from a DB or something. Here is a version of one that does one nested tab.

let tab_list_def = {
        "tab_list" : [
            { 
                "id" : "l1_1",
                "name" : "tutorial",
                "content" : "not much"
            },
            { 
                "id" : "l1_2",
                "name" : "example",
                "content" : "won't see this"
            },
            { 
                "id" : "l1_3",
                "name" : "colors",
                "content" : "superfluous..."
            }
        ],
        "tab_panels" : {
            "l1_1" : false,
            "l1_2" : {
                "component" : Tabs,
                "parameters" : {
                    "tab_list" : [
                                    { 
                                        "id" : "l2_1",
                                        "name" : "jumping jack",
                                        "content" : "bouncy"
                                    },
                                    { 
                                        "id" : "l2_2",
                                        "name" : "nick nack",
                                        "content" : "likes momentos"
                                    }
                                ],
                    "tab_panels" : {
                        "l2_1" : false,
                        "l2_2" : false
                    },
                }
            }
        }
    }

It looks longer than it is. But, you can load parts and put them together, and you don't have it in your code. Notice that tab_list_def.tab_panels.l1_2.component is a Tabs component, the same from the import statement.

Conclusion

You can customize it, too. There is room for improvement there if you want to help out.

You can find out more by checking out the repository: svelte-nested-tabs

And, you can get a working example from my brand new (not yet rewritten readme) example repository. svelte-examples. The example uses the styling capabilities.

Not bad for an afternoon! Gotta go!.

Hope you enjoy!

Thanks. Bye.

Elliptic Curve TLSv1.3 for Node.js

Richard Leddy — Fri, 20 Aug 2021 01:06:22 +0000

So... This is short.

I spent a lot of time (I mean an awful amount of time) yesterday assuring myself that the few short steps in the node.js docs for TLS are as easily done with elliptic curves. After reading up on a lot of things and running my freshly secure application, I am finally satisfied that it's OK to just use the openssl ecparam and ec parameters. And, it's OK to slip them into an X509 format at the end to get files for node.js TLS connections.

Why is that something that you wouldn't just do without thinking it about? Well...

Problem #1

I overthink things as a habit.

Problem #2

I've been playing this development game for too many years. So, now I am suspicious of all documentation, all code snippets, and anyone saying he's got a quick fix.

Helpful Articles

The articles are helpful for explaining lots of things. Except, many are now quite a few years old. My! How time flies!

So, you start opening many articles returned by your favorites search engine, only to find a repetition of the first one some twenty articles back with the same code snippet from the node.js documentation.

In the end, the best thing to read was the openssl documentation. See openssl elliptics

There's lots of great articles with pretty pictures that explain elliptic curve cryptography. Some of it is beautiful. I will have to write up some review some time. Later...

Of course, if you want to go deeper into what the TLS standard is, you can always immerse yourself in the IETF documents IETF on TLSv1.3

Why this Bother?

Well... Let's say you have working processes in back-end computers. They don't all have to be HTTP/S. They can just pass messages around with their own format. For instance, I am using my own little JSON message relay stack. Find it here: message-relay-services

So, you want connections only; no other overhead.

And, yes, you want secure connections between computers that aren't even facing the outside world. Maybe within a specific cluster you can forego some security. But, these messages go between loosely connected (tiny) servers fairly close to the front of the operation. Better safe than sorry.

As a result, my implementation case even has specific client keys being configured into the servers. I have endpoint servers (those that finally do something to a table on disk or similar). They know who their client is. Of course, other than one admin desktop app, the client is most likely a middle message exchange, which serves many clients itself.

So... What are those Commands?

The ellipses once again!

Here is the two step key generation using openssl:

$ openssl ecparam -name secp384r1 -genkey -out keys/ec_key.pem

$ openssl req -new -x509 -key keys/ec_key.pem -sha256 -nodes -out keys/ec_crt.crt -days 365

And, yes, I copied them from the docs.

The first generates the secret key. Notice that I put it in a key directory. The directory is about keeping down clutter. Put it where you need it.

Also, notice that I chose a particular curve, secp384r1. This is for 384 bits of key. There are many other curves. Check the node.js docs on how to find out what they are.

The next command generates the public key and puts it into an X509 file. The file is the cert. The req command with the X509 format guides you through putting in required fields. The fields are not magical, just where you are and who you are and a way to contact you.

That's it!

Just make sure that you do this for the server and the client.

Using the Keys in node.js

Now, this where the examples don't change. Below is code if you don't want to follow links like this one node.js tls doc.

The code is from message-relay-services. Notice that I used a configuration object. The file reading happens at initialization. It is not safe code per se. Later, I may move the file reading to an earlier point in initialization, so it can crash sooner.

You might ask, "Does this basically copy the docs?" Yes, it does, almost to the letter. Read the docs.

But, the point is:

You don't have to change old TLS code to use the EC curves. Just prepare the files differently.

This exercise has been carried out on node version v16.6.2

Server:

let base = process.cwd()
const options = {
    key: fs.readFileSync(`${base}/${this.tls_conf.server_key}`),
    cert: fs.readFileSync(`${base}/${this.tls_conf.server_cert}`),
    requestCert: true,  // using client certificate authentication
    ca: [ fs.readFileSync(`${base}/${this.tls_conf.client_cert}`) ] //client uses a self-signed certificate
};
if ( this.extended_tls_options !== false ) {
    options = Object.assign({},options,this.extended_tls_options)
}
this.connection = tls.createServer(options,((sock) => { this.onClientConnected_func(sock) }));

Client:

let base = process.cwd()
const tls_options = {
    // Necessary only if the server requires client certificate authentication.
    key: fs.readFileSync(`${base}/${this.tls_conf.client_key}`),
    cert: fs.readFileSync(`${base}/${this.tls_conf.client_cert}`),
    // Necessary only if the server uses a self-signed certificate.
    ca: [ fs.readFileSync(`${base}/${this.tls_conf.server_cert}`) ],
    // Necessary only if the server's cert isn't for "localhost".
    checkServerIdentity: () => { return null; },
};
if ( this.extended_tls_options !== false ) {
    tls_options = Object.assign({},tls_options,this.extended_tls_options)
}
this.socket = tls.connect(this.port, this.address, tls_options, () => {
    if ( this.socket.authorized ) {
        this._connection_handler()
    } else {
        this.socket.end()
    }
    this.writer = this.socket
});

EXIT

If this saved you time, then maybe life has purpose.

Crypto-wraps

Richard Leddy — Tue, 10 Aug 2021 02:21:54 +0000

So, I put this npm package out there: crypto-wraps

For this package, which is on GitHub crypto-wraps-GitHub

npm test

I tested it in FireFox and node.js

As with many wrapper libraries, some of the methods pick which encryption standard they use. So, while crypto.subtle gives you just about every standard crypto routine you can think of, this library just uses some of them.

More on the crypto.subtle library can be found at MDN: Crypto/subtle.

What is there to say? Well, calling the crypto.subtle methods has its ins and outs. The code gets a little long, and you are always worrying about the types of the data buffers going in and out of the methods. Then, since the crypto.subtle methods always use ArrayBuffers or more specifically Uint8Arrays, your program has to put the buffers into text (hex or base64url) to send them anywhere.

Actually, I ran into some trouble storing buffers straight up in the IndexedDB. But, they stored well as blobURls which are usually a base64 encoding of the blob along with some mime type information. But, yes, the first time I stuck a blob straight into the IndexedDB to have it disappear, I was a little surprised and started converting the buffers to base text.

So, here is a code example from the node.js test:

let cwraps = require('../lib/crypto-wraps')
let hashit = require('../lib/crypto-hash');
...

    let again_text = "Finally the last test again until the next last test."
    let again_cipher_text = await cwraps.encipher_message(again_text,aes_key,nonce)
    let wrapped_key = await cwraps.key_wrapper(aes_key,key_pack.pk_str)
    let again_decipher_text =  await cwraps.decipher_message(again_cipher_text,wrapped_key,key_pack.priv_key,nonce)

Now, I saved myself a lot lines. Here is the code for just the key_wrapper:

async function key_wrapper(key_to_wrap,pub_wrapper_key) {
    try {
        let wrapper_jwk = JSON.parse(pub_wrapper_key)
        let wrapper = await g_crypto.importKey(
                "jwk",
                wrapper_jwk,
                {   //these are the wrapping key's algorithm options
                    name: "RSA-OAEP",
                    modulusLength: 4096, //can be 1024, 2048, or 4096
                    publicExponent: new Uint8Array([0x01, 0x00, 0x01]),
                    hash: { name: "SHA-256" },
                },
                true,
                ["wrapKey"]
        );

        let wrapped_key = await g_crypto.wrapKey(
                                            "jwk",
                                            key_to_wrap,
                                            wrapper,
                                            {   //these are the wrapping key's algorithm options
                                                name: "RSA-OAEP"
                                            }
                                        );
        let type8 = new Uint8Array(wrapped_key)
        let tranportable = to_base64_from_uint8array(type8)
        return tranportable
    } catch(e) {
        console.log(e)
    }
    return false
}

So, you see how things can grow. Here the program is parsing JSON to get an "jwk" object carrying the public wrapper key. This is the key you would give someone to wrap something to send to you, that only you can unwrap with your private key. The code imports the wrapper key from the "jwk" object. The key to wrap is passed in as an object of type Cryptokey. The wrapper does not know too much about it. But, it is an AES key that I got out of the key gen method.

The methods are using this global variable: g_crypto. The reason for that is that node.js exposed the crypto.subtle methods from one global object, while the browser uses another:

node.js way:

let g_crypto = crypto.webcrypto.subtle

browser way:

window.g_crypto = window.crypto ? window.crypto.subtle : null

It turns out that node.js and the browser are very much in agreement with the exposition of the crypto.subtle methods. The modules, crypto-wraps.js, for the two versions are almost identical. For base64url encoding, they both import to methods from the hash module:

to_base64_from_uint8array
from_base64_to_uint8array

These methods do differ between node.js and the browser. But, at the next level up, they are used exactly the same.

For the browser version, I borrowed base64 encoding from a Gist by Egor Nepomnyaschih. base64. That proved to be very helpful. The node version makes use of easy Buffer class conversions.

That's all for now folks.

Have fun with it. Leave notes in the issues. Thanks.

IPFS CIDS v.s. CWIDS

Richard Leddy — Tue, 03 Aug 2021 07:59:39 +0000

So, I just wrote a program, call it more a prototype, for something like email, but using IPFS. I call it interplanetary-contact for the moment. You can find a copy of it at copious-world. I was hoping for email without a server, but for this you still have to invoke the node.js server included there. And, interface is a Svelte application.

I got it up and working. But, I still have much more to do with it. One things I was doing was using the Mutable File System, MFS which is part of IPFS. And, I was allowing the server to act as an agent for IPFS. Not the worst thing you can do, but not a true player in P2P. Also, the server is being relied upon to make CIDs or fetch them from IPFS.

So, I need to make CIDs in a browser. This led me to the anatomy of CID. And, as I dug deeper, I wanted more from a CID and less code in the browser, too. So, I came up with a CWID. It's really a minor difference. And, I will explain. But, I will also fill you in on a CID just in case you are not the CID aficionado.

CIDs

A CID is a content identifier. It refers to a unique set of bytes, that is one byte array per CID. It also has information, a prefix, preceding the true identifier telling the programs and programmers the format in which the identifier is being presented. So, the if identifier contains a SHA256 hash of the content, the hash may be presented in a base, such as Hex = base16, or base64url, or btc58, or binary, or other. Another hashing algorithm may be used, and the length will be part of the encoded data. At the very front of the string is single character indicating the base. All the rest of the characters are in the base.

So, for a CID not all formats can simply be tacked together by concatenating parts of the CID. Hex can be done that way. But, base64 requires a little more finesse. Before putting the base flag at the front of the string, all the descriptors must be placed into a binary buffer, and the whole buffer gets presented in the base.

So, for a Hex string using SHA256, prior to putting an 'f' in front of the lower case hex string, the buffer will be arranged as follows:

version number: 01 - this is the CIDv1
the hash storage structure: 'raw' = 0x55
the algorithm: 'sha2-256' = 0x12
length of the hash buffer: 0x20 = 256bits/8 = 32bytes

The final string will look something like the following:

f01551220a9uf9ac9af9ua0c8a9f890a8f0af0...

So, if you have a system that is solid in one kind of CID, you can store f01551220 as a constant. But, that works if you are using Hex or another very predictable base.

If you get good CIDs, you can try them out using the IPFS CID Inspector.

My CID problem

So, I was thinking why use all that IPFS code to make a CID, and what if I want to use base64? So, I started toying around with CIDs and I came to a point where the idea of concatenating strings and have the SHA256 hash clearly exposed would be nice. But, CIDs are not so amazingly clear with base64.

So, here is a SHA256 hash of "MUCH TO DO ABOUT IDENTITY" in base64url:

KSrJ8BfYBJFRaHsFbClKvhwVFeapgccN2QCxShd54GY=

Here is the IPFS CID for just this text:

uAVUSICkqyfAX2ASRUWh7BWwpSr4cFRXmqYHHDdkAsUoXeeBm

So, where did the SHA hash go?

Well, it is in there, but the length of the SHA256 hash, 32 bytes, which you might have seen before in the hex prefix, has started the base64 encoding, changing the rest of the string.

In fact, the base64 encoding of the prefix, which might never change, is uAVUSIA==. You can see the uAVUSI part. But, the size of hash is not going to change either.

So, what if I want to get get the hash from the CID. There are times one wants to do that. For instance, you might be verifying a hash in a Merkel tree. So, this would mean that for all hashes, you have to run the CID parser and pull out the bytes.

But, you could just make a string that keeps the hash exposed. You want the hash, and neither its function nor its accessibility doesn't change from how you pack the string. But, that is a functional point about the hash doing its job as an indicator. You can make the string harder to access in the sense that the computer works harder to pull it out. But, that's all that will be happening. So, with millions and millions of hashes, the energy bill goes up.

CWID

Does CWID rhyme with quid?

Well, I put a program in this repository: CWID

A CWID is not much different than a CID. I just made it so that the prefix is separate from the hash. I put in a Sheffer stroke '|' to indicate the separation.

So, here is my CWID for "MUCH TO DO ABOUT IDENTITY":

uAVUSIA==|KSrJ8BfYBJFRaHsFbClKvhwVFeapgccN2QCxShd54GY=

Maybe the padding could be taken out and put back in for checking it out. Maybe something other than a Sheffer stroke should be used (it gets URI encoded by some browsers). Or, maybe that is not too big a issue.

So, if you are making lots of CWIDs, just make a constant out of the prefix and have it waiting in a buffer to write the hash after it. That works well in C++. The JavaScript might not gain too much from that.

I was wondering if the prefix really had to be in the same base as the hash. I guess if you are on a really small machine that you might only have room for one encoding function. Would that just be a special prefix?

Well, now that I have this trick, I can revisit Interplanetary-contact and switch to making CWIDs in the browser. Maybe I will have to rename it to Intergalactic-contact.

You can play with the code.

npm test

This should launch your default browser and start up http-server to serve the page. You may have to install http-server.

npm i http-server@0.8.5

The CWID repository is not an npm package. But, you can clone it off of GitHub. Please feel free to raise issues.

Also, note that this is very new (did it today), and there is very little support for all the different formats. That is, there is just base64url and Hex.

Following up

Hope that is entertaining for now. For later, I am planning to discuss identity and how to have email without having email. With the contact system, no email addresses are used. Rather, your description of yourself is turned into a content identifier. But, that's all for now...