DEV Community: Diogen Vagrant

Metastocle - a decentralized data storage

Diogen Vagrant — Sat, 13 Jun 2020 12:10:14 +0000

What if you need to store a variety of data decentralized? Objects, arrays, dates, numbers, strings, yes anything. Is it necessary to develop a powerful DBMS for this? Indeed, often we just need to store and receive data in a distributed, openly, but as simple as possible and without any special claims.

In this article, I would like to reveal a little bit about the metastocle library, which can be used to solve the above problem easily, but with some limitations.

A bit of background

About a year ago, there was a desire and need to create a music storage. See this article for more details. From the very beginning, it was clear that you need to write everything so that you can do the same with other entities in the future: books, videos, etc. it was decided to divide everything into layers that can be used independently.

Metastocle is one of the layers that allows you to store and retrieve many types of data (but not files), as opposed to the storacle layer, which implements working with files.

When we save files, we need to write the hashes somewhere so that we can access them later. This is exactly why we need metastocle. It's where we keep everything we need: the names of the songs, links to files etc.

As a result, all this was brought to a certain universal form, and the system consists of three main entities:

Collections - an entity for defining the data structure, various options, and so on.
Documents - data itself, as objects.
Actions(Instructions) - a set of rules for processing the required data: filtering, sorting, limiting, and so on.

Let's look at a couple of examples:

Server:

const Node = require('metastocle').Node;

(async () => {  
  try {
    const node = new Node({
      port: 4000,
      hostname: 'localhost'
    });
    // Creating a collection
    await node.addCollection('test', { limit: 10000, pk: 'id' });
    await node.init();
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

Client:

  const Client = require('metastocle').Client;

(async () => {  
  try {
    const client = new Client({
      address: 'localhost:4000'
    });
    await client.init();

    // Adding a document
    const doc = await client.addDocument('test', { text: 'hi' });

    // Updating this document
    await client.updateDocuments('test', { text: 'bye' }, {
      filter: { id: doc.id }
    });

    // Adding another document
    await client.addDocument('test', { id: 2, text: 'new' });

    // Getting the second document
    const results = await client.getDocuments('test', {
      filter: { id: 2 }
    });

    // Getting it differently
    const doc2 = await client.getDocumentById('test', 2)); 

    // Adding more documents
    for(let i = 10; i <= 20; i++) {
      await client.addDocument('test', { id: i, x: i });
    }

    // Getting the documents that meet all the conditions
    const results2 = await client.getDocuments('test', {
      filter: { id: { $gt: 15 } },
      sort: [['x', 'desc']],
      limit: 2,
      offset: 1,
      fields: ['id']
    });

    // Deleting documents with id > 15
    await client.deleteDocuments('test', {
      filter: { id: { $gt: 15 } }
    });
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

Customers can not create a collection. The network sets the structure itself, and users only work with documents. Collections can also be described declaratively, via node options:

const node = new Node({
  port: 4000,
  hostname: 'localhost',
  collections: {
    test: { limit: 10000, pk: 'id' }
  }
});

Main collection parameters:

pk - a primary key field. You can omit this if it is not required. If this field is specified, a uuid hash is created by default. But you can pass any integer or string.
limit - maximum number of documents per node
queue - queue mode: if enabled, when the limit is reached, certain documents are deleted to record new ones
limitationOrder - if the limit and queue are enabled, then you can specify sorting rules to determine which documents to delete. By default, those that have not been used for a long time are deleted.
schema - document field structure
defaults - default values for document fields
hooks - document field hooks
preferredDuplicates - you can specify the preferred number of duplicate documents in the network

The structure of the collection fields (schema) can be described as:

{ 
  type: 'object',
  props: {
    count: 'number',
    title: 'string',
    description: { type: 'string' },
    priority: {
      type: 'number',
      value: val => val >= -1 && val <= 1
    },
    goods: {
      type: 'array',
      items: {
        type: 'object',
        props: {
          title: 'string',
          isAble: 'boolean’
        }
      }
    }
  }
}

All rules can be found in the function utils.validateSchema() in https://github.com/ortexx/spreadable/blob/master/src/utils.js

Default values and hooks can be like that:

{ 
  defaults: {
    date: Date.now
    priority: 0
    'nested.prop': (key, doc) => Date.now() - doc.date
  },
  hooks: {
    priority: (val, key, doc, prevDoc) => prevDoc? prevDoc.priority + 1: val
  }
}

Main features of the library:

Working on the CRUD principle
Storing all Javascript data types that can be serialized, including nested ones.
Data can be added to storage through any node.
Data can be duplicated for greater reliability.
Queries may contain nested filters

Isomorphism

The client is written in javascript and is isomorphic, it can be used directly from your browser.
You can upload a file https://github.com/ortexx/metastocle/blob/master/dist/metastocle.client.js as a script and get access to window.ClientMetastocle or import via the build system etc

Client Api

async Client.prototype.addDocument() - adding a document to the collection
async Client.prototype.getDocuments() - getting documents from the collection according some instructions
async Client.prototype.getDocumentsСount() - getting the number of documents in the collection
async Client.prototype.getDocumentByPk() - getting a document from a collection using the primary key
async Client.prototype.updateDocuments() - updating documents in the collection according to some instructions
async Client.prototype.deleteDocuments() - deleting documents from the collection according to some instructions

Basic actions (instructions)

.filter - data filtering, example:

{ 
  a: { $lt: 1 },
  $and: [
    { x: 1 },
    { y: { $gt: 2 } },
    { 
      $or: [
        { z: 1 },
        { "b.c": 2 }
      ] 
    }
  ]
}

.sort - data sorting, example:

{ sort: [['x', 'asc'], ['y.z', 'desc']] }

.limit - amount of data

.offset - starting position for data selection

.fields - the required fields

All instructions and possible values are described in more detail in the readme.

Using the command line

The library can be used via the command line. To do this you need to install it globally: npm i -g metastocle --unsafe-perm=true --allow-root. After that, you can run the necessary actions from the project directory.

For example, metastocle -a getDocumentByPk -o test -p 1 -c ./config.js, to get a document with the primary key 1 from the collection "test". All actions can be found in https://github.com/ortexx/metastocle/blob/master/bin/actions.js

Limitations

All data is first stored in memory, and later written to a file, at certain intervals, and when exiting the process. Therefore, first, you need to have enough RAM, and second, keep in mind that you will not be able to run multiple processes to work with the same database.
Sharding at the level of the entire network has not been implemented very effectively yet. Priority is given to duplication, because the size of the network is unstable: nodes can be disconnected, connected, and so on at any time. So if you want to get a large amount of data from the network, keep in mind that all this will be collected via the HTTP protocol, without much optimization.

I came to the choice of the stack and these restrictions deliberately, because there was no goal and possibility to create a full-fledged DBMS.

Although the library is still a bit crude in terms of optimizing data requests, but if you follow certain rules, everything is fine:

You need to narrow the selection of data as much as possible, and try to organize everything so that you get documents by keys, or by some other fields, but filtered to the optimal size.
If you still need to pull a lot of data, you will have to limit each server, based on their optimal size, to transfer them over the network. For example, if 10,000 documents in a collection weigh 100 KB in compressed form, then by limiting the collection at each node to this value, we will get everything at an acceptable speed.

For any questions, please contact:

Storacle - a decentralized file storage

Diogen Vagrant — Sat, 09 May 2020 11:05:30 +0000

Before the start, I'd like to leave a link to the previous article, to clarify what exactly we are talking about.

In this article, I want to introduce the layer that is responsible for storing files and how it can be used by anyone. Storace is an independent library. You can organize storage of any files.

In my previous article I was too hard on ipfs, but it is due to the context of my task.

In fact, I think that project is really cool. I just prefer the ability to create different networks for different tasks. This allows you to better organize the structure and reduce the load on each node and the network as a whole. If necessary, you can even split the network into pieces within a single project based on certain criteria, reducing the overall load.

So, storacle uses the spreadable mechanism to organize the network. Main features:

Files can be added to the storage through any node.
Files are saved as a whole, not in blocks.
Each file has its own unique content hash for further work with it.
Files can be duplicated for greater reliability
The number of files on a single node is limited only by a file system (there is an exception, which will be discussed later)
The number of files in the network is limited by the capabilities of spreadable by the number of allowed nodes in the network, which in the second version can allow you to work with an infinite number of nodes (more on this in another article)

A simple example of how it works from the program:

The server:

const  Node  =  require('storacle').Node;

(async () => {
  try {
    const  node  =  new  Node({
      port:  4000,
      hostname:  'localhost'
    });
    await node.init();
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

The client:

const  Client  =  require('storacle').Client;

(async () => {
  try {
    const  client  =  new  Client({
      address:  'localhost:4000'
    });
    await client.init();
    const  hash  =  await client.storeFile('./my-file');
    const  link  =  await client.getFileLink(hash);    
    await client.removeFile(hash);

  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

Inside peek

There's nothing supernatural under the hood. Information about the number of files, their total size and other points are stored in the in-memory database and updated when deleting and adding files, so there is no need to frequently access the file system. An exception is the inclusion of the garbage collector when file circulation is needed, rather than limiting their number. In this case, you have to go through the storage from time to time. And working with a large number of files (let's say more than one million files) can lead to significant loads. It is better to store less files and run more nodes. If the "cleaner" is disabled, there is no such problem.

The file storage consists of 256 folders and 2 levels of nesting. Files are stored in second-level folders. So, if we have 1 million files in each folder, there are about 62500 pieces (1000000 / sqrt (256)).

Folder names are formed from the file hash to provide you quick access if necessary.

This structure was chosen based on a large number of different storage requirements: support for weak file systems, where it is not desirable to have many files in a single folder, fast crawl of the folders if necessary, and so on.

Caching

When files are added or received, links to files are written to the cache. This often means that you don't need to search the entire network for a file. This speeds up getting links and reduces the load on the network. Caching also occurs via http headers.

Isomorphism

The client is written in javascript and is isomorphic, it can be used directly from your browser.

You can upload a file https://github.com/ortexx/storacle/blob/master/dist/storacle.client.js as a script and get access to window.ClientStoracle or import via the build system, etc

Deferred links

An interesting feature is also the "deferred link". This is a link to the file that can be obtained synchronously, here and now, and the file will be pulled up when it is found in the storage. This is very convenient, for example, when you need to show some images on the site. Just put a deferred link in the src and that's it. You can come up with a lot of cases.

Api of the client

async Client.prototype.storeFile() - file storing
async Client.prototype.getFileLink() - getting a direct link to a file
async Client.prototype.getFileLinks() - getting a list of direct links to a file from all nodes where it exists
async Client.prototype.getFileToBuffer() - getting a file as a buffer
async Client.prototype.getFileToPath() - getting a file to the file system
async Client.prototype.getFileToBlob() - getting a file as a blob (for the browser version)
async Client.prototype.removeFile() - file deletion
Client.prototype.createRequestedFileLink() - creating a deferred link

Export files to another server

To transfer files to another node, you can:

Just copy the entire storage folder along with the settings. (this may not work in the future).
Copy only the file folder. But in this case, you have to run the node.normalizeFilesInfo() function once to recalculate all the data and put it in the database.
Use the node.exportFiles() function, which starts copying files.

The main node settings

When running the storage node, you can specify all the necessary settings. Only the most basic ones are listed below:

storage.dataSize - size of the file folder
storage.tempSize - size of the temporary folder
storage.autoCleanSize - minimum size of storage that you want to keep. If you specify this parameter, the most underused files will be deleted as soon as there is not enough space.
file.maxSize - maximum file size
file.minSize - minimum file size
file.preferredDuplicates - preferred number of duplicate files in the network
file.mimeWhitelist - acceptable file types
file.mimeBlacklist - unacceptable file types
file.extWhitelist - acceptable file extensions
file.extBlacklist - unacceptable file extensions
file.linkCache - link caching settings

Almost all parameters related to sizes can be set in both absolute and relative values.

Using the command line

The library can be used via the command line. You need to install it globally: npm i -g storacle. After that, you can run the necessary actions from the project directory where the node is located.

For example, storacle -a storeFile -f ./file.txt -c ./config.js to add a file. All actions can be found in https://github.com/ortexx/storacle/blob/master/bin/actions.js

Why would you want to use that

If you want to create a decentralized project where you are going to store and work with files using convenient methods. For example, the music project, that is described in the link at the beginning of the article, uses storacle.
If you are working on any other project where you need to store files distributed. You can easily build your own closed network, flexibly configure nodes and add new ones when you need it.
If you just need to store the files of your site somewhere and you have to write everything yourself. Perhaps this library is better than others, in your case.
If you have a project in which you work with files, but want to perform all manipulations from the browser. You can avoid writing server-side code.

My contacts:

Museria - a decentralized music storage

Diogen Vagrant — Tue, 05 May 2020 07:00:21 +0000

One day I decided to write an application for selecting music. I wanted to organize a stream of music to select something and listen to it at any time. Nothing complicated, right? The architecture was successfully designed, the preparations were completed. But a "little problem" arose. Where to get the song files? At that moment I realized that the whole world of music was plunged into commerce. It's not bad when people want to earn money on their work, but at what cost? When in the end no one but large companies can create a full-fledged music application without resorting to all sorcery.

Then two tasks arose before me:

Organizing the free distribution of music, using methods that are convenient for most people, including programmable ones.
Offer earning alternatives to music creators.

Global decentralized music storage.

Initially, I tried to find existing solutions and create everything based on this. After some time searching, ipfs was the first one I liked. I started implementing my idea, but after a while I found several critical problems in this solution:

Ipfs is the repository for everything. There are texts, images, music, videos, anything. By and large, this is such a big planetary "garbage". When you start your node, you immediately get a huge load.
Some kind of unfinished "garbage collection" mechanism. I don't know how it is now, but at that moment, if you specified in the config that you want to limit the storage to ten gigabytes of data, it didn't mean anything. The repository was growing, ignoring many configuration options. In the end, you had to have a huge supply of hard drive, while ipfs figured out how to reset the unnecessary.
At the time of using the library (I don't know about now), the client did not have timeouts. You send a request to get a file, and if it is not there, then you just hang. Of course, people came up with all kinds of workarounds that partly solved the problem, but it was crutches. Such things should be out of the box.

There were still many minor problems that I don't remember now, but the impression was unambiguous: this cannot be used for my project. I continued to search for a solution, studied various options, but did not find anything suitable.

In the end, I decided that I should try writing a decentralized storage myself. Let it not pretend to be interplanetary, but it will solve my specific task.

So they appeared: spreadable, storacle, metastocle, museria, museria-global.

spreadable - the main, lowest layer that allows you to combine nodes into a network. It contains an algorithm that I partially implemented based on the calculation of about 10,000 servers. The full version of the algorithm is much more difficult to implement and would require several additional months (maybe more).

It uses http/https.
You can create a separate network for a specific task, which will significantly reduce the load on each individual project than if they were all on the same network.
The timeout mechanism and other important details were thought out from the very beginning for all methods both in the client and node. You can flexibly manage parameters from your app.
The library is written with nodejs. Performance problems are compensated by the decentralized nature. The load can be "smeared" by increasing the number of nodes. Instead, there are many advantages: a huge community, simplicity and convenience, an isomorphic client, and so on.

storacle - a layer inherited from spreadable that allows you to store files on the network. Each file has its own content hash, which can be used to get it later. Files are not divided into blocks.

metastocle - a layer inherited from spreadable that allows you to store data on the network, but not files. The interface is similar to a nosql database. For example, you can add a file to storage, get its hash, and write it to metastocle by linking it to something.

museria - inherited from storace and metastocle. This layer is directly responsible for storing music. The storage only works with mp3 files and id3 tags.

As the "key" to the song, its full name is used in the form of Artist (TPE1) - Title (TIT2). For example:

Brimstone - The Burden
Hi-rez - Lost My Way (feat. Emilio Rojas, Dani Devinci)

You can learn as much as possible about how song titles are formed here:

https://github.com/ortexx/museria/blob/master/src/utils.js

Look at utils.beautifySongTitle().

A key match is the percentage specified in the node settings. For example, a value of 0.85 means that if the key comparison function (song names) found a similarity of more than 85%, then this is the same song.
Algorithm for determining similarity in the same place, in the function utils.getSongSimilarity().

An example of working with the storage you can find in the readme.

All of the above layers are self-sufficient and can be used separately as lower layers for other projects. For example, now there is an idea to make a layer for storing books.

museria-global - a configured git repository to run your own node in the global music storage network. Clone it and then: npm i && npm start. That's it. You can configure in more detail, run in docker, etc. Detailed information is available on github.

When the repository is updated, you should update your node as well. If the major or minor version number changes, then this action is required, otherwise your node will be ignored by the network.

You can work with songs manually or programmatically. Each node starts the server for different tasks. In particular, when you visit the default endpoint, you will get an interface for working with music. For example, you can go to the root node (the link may not be up to date later, you can get input nodes in telegram, or view updates on github).

There you can search and upload songs. Songs can be uploaded in two modes: normal and moderated. The second mode means that the work is done by a person, not a program. And if you check this box when adding it, you will need to solve the captcha. Songs can be added with the priorities of -1, 0 or 1. Priority 1 can only be set in moderated mode. Priorities are needed to let the storage more effectively decide what to do when you try to replace an existing song with a new one. The higher the priority, the more likely you will overwrite an existing file. This helps fight spam and increases the quality of downloaded songs.

How files are technically added, in a nutshell:

The client gets the address of the free node, which will become the coordinator for a while.
The function of adding a song (by a person or code) is triggered, and a request is made to the coordinator's endpoint to add a file.
The coordinator calculates how many duplicates to make (configurable param).
Search for the most suitable nodes to save.
The file goes directly to these nodes.

How files are received technically:

The client gets the address of the free node, which will become the coordinator for a while.
The function of getting a song (by a person or code) is triggered, and a request is made to the coordinator's endpoint to get a file.
The coordinator checks for the link in the cache. If there is one and it is working, it is immediately returned to the client, otherwise the nodes are polled for availability.
The file is received by the link, if there is one.

Alternatives for music creators

I have always been interested in the question, how can we objectively estimate the cost of creative works? Why does anybody sell their music album for $10, $20 or $100. Where is the algorithm? When we are talking about a physical product, or even many types of services, we have at least the prime cost.

Okay, let's take $10. Is it effective? Let's say I listened to an album somewhere or a song from there and decided to thank you. But according to my feelings and material possibilities, $3 is my top. What should I do? I probably just won't do anything, like most people. By setting a fixed price for creative work, you only limit yourself, do not allow a large number of people to send you less money, which in total can be more impressive than those who will buy at the price you set. It seems to me that creativity is exactly the sphere where donations should rule in the first place. To do this, we need:

Teach people to thank in this way. The creators themselves must clearly show that they would like to receive donations, add links to different payment methods everywhere, and so on.
More mechanisms are needed to simplify and strengthen these processes.

For example, someone can create a global website where you can donate through the special author's links.

Something like that:

http://someartistsdonationsite.com/{category}/{artist}?{external-info}

If to narrow it down to musicians,then:

http://someartistsdonationsite.com/music/eminem?song=blabla

Artists need to verify their nickname and link themselves to it.

In the museria client, we add a function for generating such a link, and all projects using the storage can place buttons for donations with these links next to the songs on their sites/apps. Users can quickly and easily make a donation. Naturally, this approach can be used in any project and creative category, not just through the storage and music.

Why you need the music storage, and how you can participate in it.

If you are working on a project related to music, or are planning to create one, then this is what it was all about. You can use museria to store and receive songs, increasing the flow of songs on the network. If, at the same time, you have the ability to run and hold at least one own node, it will be the best contribution to the development of the network.
Perhaps you are ready to take on some other role: help with the code, or fill in and moderate the database, distribute information about the project to your friends, and so on.
Maybe you like the idea and are ready to help financially, so that it all lives and develops. The more nodes, the more songs.
Or you may just need to find and download a song at some point. You can do this very simply, for example, through the telegram bot.

The project is now at the very beginning stage. The test network is running, nodes can frequently restart, require updates, and so on. if there are no critical problems during the estimation period, this same network is transformed into the main one.

To view information about the node from the outside: the number of songs, free space, and so on, follow the link below

http://node-address/status or http://node-address/status?pretty

My contacts:

Telegram group

Well, the site that gave birth to this idea.