One day I decided to write an application for selecting music. I wanted to organize a stream of music to select something and listen to it at any time. Nothing complicated, right? The architecture was successfully designed, the preparations were completed. But a "little problem" arose. Where to get the song files? At that moment I realized that the whole world of music was plunged into commerce. It's not bad when people want to earn money on their work, but at what cost? When in the end no one but large companies can create a full-fledged music application without resorting to all sorcery.
Then two tasks arose before me:
- Organizing the free distribution of music, using methods that are convenient for most people, including programmable ones.
- Offer earning alternatives to music creators.
Global decentralized music storage.
Initially, I tried to find existing solutions and create everything based on this. After some time searching, ipfs was the first one I liked. I started implementing my idea, but after a while I found several critical problems in this solution:
- Ipfs is the repository for everything. There are texts, images, music, videos, anything. By and large, this is such a big planetary "garbage". When you start your node, you immediately get a huge load.
- Some kind of unfinished "garbage collection" mechanism. I don't know how it is now, but at that moment, if you specified in the config that you want to limit the storage to ten gigabytes of data, it didn't mean anything. The repository was growing, ignoring many configuration options. In the end, you had to have a huge supply of hard drive, while ipfs figured out how to reset the unnecessary.
- At the time of using the library (I don't know about now), the client did not have timeouts. You send a request to get a file, and if it is not there, then you just hang. Of course, people came up with all kinds of workarounds that partly solved the problem, but it was crutches. Such things should be out of the box.
There were still many minor problems that I don't remember now, but the impression was unambiguous: this cannot be used for my project. I continued to search for a solution, studied various options, but did not find anything suitable.
In the end, I decided that I should try writing a decentralized storage myself. Let it not pretend to be interplanetary, but it will solve my specific task.
So they appeared: spreadable, storacle, metastocle, museria, museria-global.
spreadable - the main, lowest layer that allows you to combine nodes into a network. It contains an algorithm that I partially implemented based on the calculation of about 10,000 servers. The full version of the algorithm is much more difficult to implement and would require several additional months (maybe more).
- It uses http/https.
- You can create a separate network for a specific task, which will significantly reduce the load on each individual project than if they were all on the same network.
- The timeout mechanism and other important details were thought out from the very beginning for all methods both in the client and node. You can flexibly manage parameters from your app.
- The library is written with nodejs. Performance problems are compensated by the decentralized nature. The load can be "smeared" by increasing the number of nodes. Instead, there are many advantages: a huge community, simplicity and convenience, an isomorphic client, and so on.
storacle - a layer inherited from spreadable that allows you to store files on the network. Each file has its own content hash, which can be used to get it later. Files are not divided into blocks.
metastocle - a layer inherited from spreadable that allows you to store data on the network, but not files. The interface is similar to a nosql database. For example, you can add a file to storage, get its hash, and write it to metastocle by linking it to something.
museria - inherited from storace and metastocle. This layer is directly responsible for storing music. The storage only works with mp3 files and id3 tags.
As the "key" to the song, its full name is used in the form of Artist (TPE1) - Title (TIT2). For example:
- Brimstone - The Burden
- Hi-rez - Lost My Way (feat. Emilio Rojas, Dani Devinci)
You can learn as much as possible about how song titles are formed here:
https://github.com/ortexx/museria/blob/master/src/utils.js
Look at utils.beautifySongTitle().
A key match is the percentage specified in the node settings. For example, a value of 0.85 means that if the key comparison function (song names) found a similarity of more than 85%, then this is the same song.
Algorithm for determining similarity in the same place, in the function utils.getSongSimilarity().
An example of working with the storage you can find in the readme.
All of the above layers are self-sufficient and can be used separately as lower layers for other projects. For example, now there is an idea to make a layer for storing books.
museria-global - a configured git repository to run your own node in the global music storage network. Clone it and then: npm i && npm start. That's it. You can configure in more detail, run in docker, etc. Detailed information is available on github.
When the repository is updated, you should update your node as well. If the major or minor version number changes, then this action is required, otherwise your node will be ignored by the network.
You can work with songs manually or programmatically. Each node starts the server for different tasks. In particular, when you visit the default endpoint, you will get an interface for working with music. For example, you can go to the root node (the link may not be up to date later, you can get input nodes in telegram, or view updates on github).
There you can search and upload songs. Songs can be uploaded in two modes: normal and moderated. The second mode means that the work is done by a person, not a program. And if you check this box when adding it, you will need to solve the captcha. Songs can be added with the priorities of -1, 0 or 1. Priority 1 can only be set in moderated mode. Priorities are needed to let the storage more effectively decide what to do when you try to replace an existing song with a new one. The higher the priority, the more likely you will overwrite an existing file. This helps fight spam and increases the quality of downloaded songs.
How files are technically added, in a nutshell:
- The client gets the address of the free node, which will become the coordinator for a while.
- The function of adding a song (by a person or code) is triggered, and a request is made to the coordinator's endpoint to add a file.
- The coordinator calculates how many duplicates to make (configurable param).
- Search for the most suitable nodes to save.
- The file goes directly to these nodes.
How files are received technically:
- The client gets the address of the free node, which will become the coordinator for a while.
- The function of getting a song (by a person or code) is triggered, and a request is made to the coordinator's endpoint to get a file.
- The coordinator checks for the link in the cache. If there is one and it is working, it is immediately returned to the client, otherwise the nodes are polled for availability.
- The file is received by the link, if there is one.
Alternatives for music creators
I have always been interested in the question, how can we objectively estimate the cost of creative works? Why does anybody sell their music album for $10, $20 or $100. Where is the algorithm? When we are talking about a physical product, or even many types of services, we have at least the prime cost.
Okay, let's take $10. Is it effective? Let's say I listened to an album somewhere or a song from there and decided to thank you. But according to my feelings and material possibilities, $3 is my top. What should I do? I probably just won't do anything, like most people. By setting a fixed price for creative work, you only limit yourself, do not allow a large number of people to send you less money, which in total can be more impressive than those who will buy at the price you set. It seems to me that creativity is exactly the sphere where donations should rule in the first place. To do this, we need:
- Teach people to thank in this way. The creators themselves must clearly show that they would like to receive donations, add links to different payment methods everywhere, and so on.
- More mechanisms are needed to simplify and strengthen these processes.
For example, someone can create a global website where you can donate through the special author's links.
Something like that:
http://someartistsdonationsite.com/{category}/{artist}?{external-info}
If to narrow it down to musicians,then:
http://someartistsdonationsite.com/music/eminem?song=blabla
Artists need to verify their nickname and link themselves to it.
In the museria client, we add a function for generating such a link, and all projects using the storage can place buttons for donations with these links next to the songs on their sites/apps. Users can quickly and easily make a donation. Naturally, this approach can be used in any project and creative category, not just through the storage and music.
Why you need the music storage, and how you can participate in it.
- If you are working on a project related to music, or are planning to create one, then this is what it was all about. You can use museria to store and receive songs, increasing the flow of songs on the network. If, at the same time, you have the ability to run and hold at least one own node, it will be the best contribution to the development of the network.
- Perhaps you are ready to take on some other role: help with the code, or fill in and moderate the database, distribute information about the project to your friends, and so on.
- Maybe you like the idea and are ready to help financially, so that it all lives and develops. The more nodes, the more songs.
- Or you may just need to find and download a song at some point. You can do this very simply, for example, through the telegram bot.
The project is now at the very beginning stage. The test network is running, nodes can frequently restart, require updates, and so on. if there are no critical problems during the estimation period, this same network is transformed into the main one.
To view information about the node from the outside: the number of songs, free space, and so on, follow the link below
http://node-address/status or http://node-address/status?pretty
My contacts:
Top comments (0)