This is a follow up to the introductory article of Quarks:
https://dev.to/lucpattyn/quarks-a-new-approach-with-a-new-mindset-to-programming-10lk
The introductory article discussed the philosophy behind Quarks but the scaling aspect of it wasn't covered fully because the idea behind it evolved. Hence the new article which sheds more light on the scaling side of things.
Terms
To understand scaling, a few terms must be understood first:
1) {B} - Quarks Bubble
2) {M} - Master Quarks
3) {L} - Logger Quarks
4) {C} - Quarks Core
5) {CL} - Lower Core or Left Core
6) {CR} - Higher Core or Right Core
7) {CLx, CRx} - Left/Right Cores according to levels
so a level-1, level-2 left core would be represented
as CL1, CL2 and so on..
and a level-1, level-2 right core would be represented
as CR1, CR2 and so on..
*Lower and Higher is decided based on key comparison in cores.
*All {CR} has keys greater than collection of keys in {CL}
*Master Quarks {M} stands at level-0
Now, on to the illustration before the explanation :
Let's approach this from top down:
The Quarks Bubble {B} can be thought of as part of the expandable layer (Bubble layer).
This layer take hits from clients, caches data and grow horizontally based on cpu and memory usage. So the bubbles are equivalent to Docker instances.
Data Write
If a client (can be mobile app, laptop or an external server) invokes a write data operation, the command first go through a bubble that saves the data, send request respond to client and at the same time publish through a message broker to send it to the lower layer (Cloud layer).
{B} has a cache limit, so it can save limited number of data for quick retrieval.
At the lower layer, the Master Quarks {M} and the logger Quarks {L} both is listening for the data.
*Note a) The role of {M} is to decide where should this data be saved for later retrieval.
At this point the cores become important.
Quarks Core {C} is the basic level element that saves data and they are discussed in details in the introductory article.
{C} can be configured to have a data limit (for example 1 million elements).
The data is sharded/distributed based on Keys. The key comparison is nothing but basic string compare of the keys.
If {C} hits the limit, then new entries are sent to the lower level cores.
{C} has the maxKey attribute which denotes the highest key saved so far (based on it's string value).
If the new key is greater than maxKey it goes to next level {CR} otherwise next level {CL}.
{M} too works like {C} at level-0 and passes on data to next level when data limit is reached.
Data Read
If a client (can be mobile app, laptop or an external server) invokes a read operation, the command first go through a bubble that checks whether that data is available. If available, then cached data is sent back to client.
Otherwise, the read request is published through the message broker and sent to the lower layer (Cloud layer).
At Cloud layer, {M} is listening for read requests.
When a request comes, {M} identifies which core or cores {CLx, CRx} has the data, communicates to the core through broker for the data, and once all data is gathered, send it to the Bubble layer (via message broker) which finally responds back to client with the requested data.
Backup and Restore
At Cloud layer, Logger Quarks {L} is listening for write requests.
(*There can be multiple loggers).
Whenever a new write operation comes it carries out two responsibilities:
1) Logs down the data in multiple storage
2) Publishes the write operation for other subscribers to save data in their own system and run analytics.
If {M} goes down it can be quickly restored by replicating a copy that is stored by {L}. A health checker with the aid of a script is able to do it.
If any of the other cores go down ,
the health checker notifies {M} to fetch data from {L} and restore it quickly.
{M} has the knowledge of saved data as already mentioned (refer to ^note a).
The beauty of the architecture is - it is easy to deploy, because {M}, {L}, {C}, {B} etc are nothing but instances of Quarks {Q} configured by a few parameters in config file or through command line parameters.
Getting {Q} and it's variants running is a breeze.
Till next time,
{Mukit}
Top comments (1)
This has been replaced by a better concept in another article:
dev.to/lucpattyn/quarks-replicatio...