Decentralized music storage
One day I decided to write an app to select music for me and listen to it at home/outdoors/exercise, etc. And to make it all work in flow, with minimal involvement from me. I came up with the architecture, sketched a prototype and ended up with one ″little problem″.
And it was unclear where to get the song files themselves from. By that time vkontakte had already closed the api, at major music portals, too, all is silent, even the songs are given in pieces, so as not to parcel. There were only some separate one-day sites with tons of advertising and garbage, some dubious grabbers and other "dirty" variants. All in all, not a single really good solution. You can, of course, buy a subscription to some Yandex music or something like that. But again, there's no open public api anywhere and you don't have access to the music programmatically. A few of the big companies have essentially restricted the rest of us from accessing music. Why did this happen in the first place? Digging deeper, it's clear that the main problem is copyright. The current solution in the form of subscriptions suits many commercial copyright free music site and creators and those very companies. That said, non-commercial and not-for-profit music are also on the common list. You either pay for everything or you don't listen to anything at all.
So I started thinking about what to do with all this. How do you organize the free distribution of music? What would I do if I was making music myself and wanted to make money from it? Would I like it if my songs started to be pirated? What alternative solution is there anyway?
In the end, there are two main problems that need to be solved:
• Organizing the free distribution of music in ways most people are comfortable with, including software.
• Offering alternatives to music creators to make money
Global decentralized music storage
Initially, I tried to find already existing solutions and create everything based on that. After some time of searching, the first one that caught my eye was ipfs. I started to implement my idea, but after some time I discovered some critical problems with this solution:
ipfs is storage for anything and everything. It has images and music and videos and everything. It's basically a big planetary "dump". So when you start up your node, you immediately get a huge load. The machine just squirms in pain.
Some kind of unfinished "garbage" assembly mechanism. I don't know about it now, but at that time, if you wrote in the configuration that you wanted to limit the storage to ten gigabytes of data, it meant nothing. The storage was growing, ignoring a lot of the configuration parameters. As a result, you had to have a huge supply of hard drive while ipfs figured out how to dump the unneeded stuff.
At the time of using the library (I don't know how it is now), the client didn't have timeouts implemented. You send a request for a file, and if it's not there, you just hang. Sure, people came up with all sorts of workarounds that partially solved the problem, but those were crutches. Things like that should be out of the box.
There were many more minor problems, the impression I got at the time was unequivocal: it couldn't be used for the project. I continued to look for storage, exploring different options, but did not find anything suitable.
In the end, I decided that it was worth trying to write a decentralized repository myself. Suppose it will not claim to be interplanetary, but will solve the problem.
This is how spreadable, storacle, metastocle, museria, museria-global came about.
Spreadable is the basic, lowest layer, which allows you to combine nodes into a network. It contains an algorithm, which I have so far partially implemented for about 10,000 servers. The full version of the algorithm is much harder to implement and would require a few more months (maybe more).
I am not going to describe it in detail in this article, I would rather write a separate one sometime. Here I will just note some of the features:
It works via http/https.
You can create a separate network for the specific task, which significantly reduce the load on each individual project than if they were all in one network.
The mechanism with timeouts and other little things was originally thought out. And it works for all methods both in the client and in the node. You can flexibly manage the parameters from your application.
The library is written in nodejs. Problems with stack performance are compensated by the decentralized nature. The load can be "smeared out" by increasing the number of nodes. In return, there are many pluses: huge community, simplicity and usability, isomorphic client, no external dependencies, etc.
The node is a layer inherited from spreadable, which allows you to store files on the network. Each file has its own hash on the contents, by which it can be retrieved later. The files are not divided into blocks, but stored as a whole.
Metastocle is a layer inherited from spreadable which allows to store data but not files on the network. The interface is similar to nosql databases. You can, for example, add a file to storacle, get its hash and write it to metastocle with a link to something.
museria - inherited from storacle and metastocle. This layer is directly responsible for storing music. It works only with mp3 files and id3 tags.
museria-global is an already configured git repository to start your own node in the global music network. Clone, npm i && npm start and that′s basically it. You can configure it in more detail, run it in docker, etc. Detailed information is on the github.
When the repository is updated, you need to update your node as well. If the major or minor version number changes, this action is mandatory, otherwise the old nodes will be ignored by the network.
Songs can be handled manually and programmatically. Each node runs the server for different tasks. Including when you visit the default endpoint, you will get an interface to work with music. For example, you can go to the root node (the link may not be up to date later, you can also get the input nodes on telegram, or look for updates on githab).
This way you can search and upload songs to the repository. Uploading songs can take place in two modes: normal and moderated. The second mode means that the work is done by a person, not a program. And if you put this checkbox when adding, you will need to solve the captcha. You can add songs with a priority of -1, 0 or 1. Priority 1 can only be put in moderated mode. Priorities are needed so that the repository can more efficiently decide what to do when you try to replace an existing song with a new one. The higher the priority, the more likely you are to overwrite an existing file. This helps fight spam and increases the quality of the downloaded songs.
If you start adding songs to the repository, try to attach images (cover) as well, although this field is not required. In 99% of cases, the first pictures googled by song title are album covers.