DEV Community: Simer Plaha

Decentralised systems - SwayDB

Simer Plaha — Thu, 12 Nov 2020 17:13:55 +0000

TimeOrder in SwayDB can be used by decentralised systems for managing un-ordered writes and/or replication requests.

Before going into details I will briefly explain the two types of Ordering in SwayDB.

KeyOrder - Similar to any sorted collection type this simply orders data by key. Simple stuff!
TimeOrder - Orders the sequence in which updates occur on a single key.

How is Time used?

Time is always stored with each write (even for range updates).

In the following example we've inserted key1 with value 1 at time 0 and then updated the value by incrementing it by 5 first and then multiplying it by 10 which sets the final value to (1 + 5) * 10 = 60.

map.put(key = "key1", value = 1, time = 0)  
map.update(key = "key1", value = oldValue => oldValue + 5, time = 1)  
map.update(key = "key1", value = oldValue => oldValue * 10, time = 2)

If we were to submit the same updates in reverse order the final value would still be the same (60) since compaction applies these updates based on time and not the actual request order.

map.update(key = "key1", value = oldValue => oldValue * 10, time = 2)
map.update(key = "key1", value = oldValue => oldValue + 5, time = 1)

Without time, the above updates would've resulted in different final value of 1 * 10 + 5 = 15 and not our intended 60.

How can it be used by decentralised systems?

I'm going to consider the use-case that writes can occur on any machine, at any time, for any key which may or may not be replicated to other machines.

SwayDB's job is to ensure that writes by a decentralised system should eventually result in the same final value regardless of when a write or replication request was received by the machine. The only requirement is that each machine's SwayDB instance is initialised with the same TimeOrder function (documented below).

Time is just a serialisable type so it can be any class with any number of fields. For our example we will use the following CustomTime class with two fields

case class CustomTime(machineId: Int,   
                      machineLocalTime: Long)

machineId - unique ID assigned to each machine on boot that does not change even if the machine reboots.
machineLocalTime - assigned to each write (put, update, remove, expire etc) initialised by each machine that creates the write request.

Example write behaviour

Suppose machine0 receives the following requests (in any order) that are replicated to machine1 (in any order).

map.put(key = "key1", value = 1, time = CustomTime(0, 0))  
map.update(key = "key1", value = oldValue => oldValue + 5, time = CustomTime(0, 1))  
map.update(key = "key1", value = oldValue => oldValue * 10, time = CustomTime(0, 2))

And machine1 receives the following requests (in any order) that are also replicated to machine0 (in any order).

map.update(key = "key1", value = oldValue => oldValue - 10, time = CustomTime(1, 1))  
map.update(key = "key1", value = oldValue => oldValue * 20, time = CustomTime(1, 2))

Expectation - We want the final value of key1 to be (1 + 5 * 10 - 10) * 20 = 820 on both machines regardless of the order in which the request were received.

Custom TimeOrder

We can achieve the above expectation with just the following TimeOrder function which orders writes by machineId first and then machineLocalTime.

val customTimeOrder =  
  new TimeOrder[CustomTime] {  
    override def compare(left: CustomTime, right: CustomTime): Int =  
      if (left.machineId == right.machineId)  
        left.machineLocalTime compareTo right.machineLocalTime  
      else  
        left.machineId compareTo right.machineId  
  }

API Status

Currently TimeOrder API is private to SwayDB's core and is enabled when functions are turned on. It is used by all Levels to execute non-blocking reads and it decouples reads from compaction i.e. if compaction is running on a Segment it does not block or lock reads.

TimeOrder's public API is pending test-cases.

MultiMap - Tables in SwayDB

Simer Plaha — Thu, 30 Jul 2020 04:50:13 +0000

MultiMap is an extension of Map data type which allows storing multiple Maps within a single SwayDB instance. It mimics SQL's Table which can be used to seggregate data.

MultiMap is just a thin layer that uses custom types and KeyOrdering to manage data segregation.

The following code snippets demo few core functionality of MultiMap like nesting, expiration, streaming, transactions, replace & remove.

Initialise `MultiMap`

import swaydb._ //import SwayDB's API
import swaydb.serializers.Default._ //import default serialisers

//initialise an in-memory multiMap  
val map = memory.MultiMap_EAP[String, Long, String, Nothing, Bag.Less]()

//or initialise an persistent multiMap 
val pMap = persistent.MultiMap_EAP[String, Long, String, Nothing, Bag.Less]("myDir")

String - The type of Map ID. In SQL this would be Table Name.
Long - The type of Key for key-values stored in this Map. In SQL this would be primary key.
String - The type of Value for key-values stored in the Map. In SQL this would be the Table row.
Nothing - The function type. Using Nothing disables functions. Instead of a query language, SwayDB allows complex data modifications by submitting pure JVM functions. In SQL these would be CREATE, INSERT & UPDATE queries. You can read more about this here.
Bag.Less - The effect type for all API calls. We can use Cats, Monix, ZIO, Try, Future or any other custom effect type. Using Bag.Less disables effect management and simply returns the result as output.

Creating Tables

The following creates two tables under the root map.

val usersTable = map.schema.init(mapKey = "Users Table")  
val productsTable = map.schema.init(mapKey = "Products Table")

//this create another child table under Users Table.
val userTablesChild = usersTable.schema.init(mapKey = "Users Table's child")

//read all tables keys of userTable
val keys: Stream[String] = usersTable.schema.keys
//read all child tables of userTable
val tables = usersTable.schema.stream

Schema tree

Creating tables with default expiration

Similar to above we can also create Tables/Maps that expire.

import scala.concurrent.duration._  

map.schema.init(mapKey = "Users Table", expireAfter = 10.days)  
map.schema.init(mapKey = "Products Table", expireAt = 10.seconds.fromNow)

Removing and replacing tables

Replace creates a new table clearing all existing data including all existing child tables.

//replace
map.schema.replace(mapKey = "Users Table", expireAfter = 10.days)

//or remove to delete the map/table
map.schema.remove(mapKey = "Users Table")

Transactions

Modifications to multiple Tables can be performed as a single batch transaction. The following snippet creates and commits two transaction entries for two different tables.

//prepare statement for userTable and productTable  
val transaction =  
  usersTable.toTransaction(Prepare.Put(key = 1L, value = "user1")) ++  
    productsTable.toTransaction(Prepare.Put(key = 1L, value = "product1"))  

//commit to root map  
map.commit(transaction)  

usersTable.get(1L) shouldBe Some("user1")  
productsTable.get(1L) shouldBe Some("product1")

See transaction for a list of all Prepare statements.

Code

See MultiMap_Example.scala on Github which uses custom types with circe for serialisation.

Please consider staring the SwayDB project if you find it useful.

EAP

MultiMap is currently in preview version (EAP). See performance improvement task #235.

GitHub repos

Check out related GitHub repos

SwayDB on GitHub.
Scala examples repo.
Java examples repo.
Kotlin examples repo.
Documentation.

Functions as Queries @ SwayDB

Simer Plaha — Sun, 29 Mar 2020 16:43:52 +0000

Conditional updates & deletes are performed by submitting any Java, Scala, Kotlin or any native JVM code/function. There is no query language.

Being just JVM functions/code we can

Be more expressive in writing our query logic as we are not restricted by rules of a query language.
Use external Java libraries within our queries like Joda-Time for time management, Jackson or Circe for JSON parsing etc.
Write test-cases using familiar testing libraries like JUnit, kotlin.test, ScalaTest etc.
Type-safe - catch errors during compile time.
Fast - No runtime parsing & compiling performance cost. See these benchmarks that show 482,000+ writes per second. Function updates follow the same write path as any basic write and do not incur any extra cost.

Example

A simple example which creates a Map<String, Double> that stores productName as key and price as value. You can use custom classes (Product.java for values) here but for simplicity String and Double is used.

Update logic: If a product has less than 2 days left to expiration, give a generous 50% discount!

//Our function that implements the update logic. 
//You can also use PureFunction.OnValue or PureFunction.OnKey.
PureFunction.OnKeyValue<String, Double, Return.Map<Double>> discount =  
  (String product, Double price, Optional<Deadline> deadline) -> {
    //if there are less than 2 days to expiry then apply discount.  
    if (deadline.isPresent() && deadline.get().timeLeft().minusDays(2).isNegative()) {  
      double discountedPrice = price * 0.50; //50% discount.
      if (discountedPrice <= 10) { //If the price is below $10
        //return Return.expire(Duration.ZERO); //expire it
        return Return.remove(); //or remove the product
      } else {  
        return Return.update(discountedPrice); //else update with the discounted price
      }  
    } else {
      return Return.nothing(); //else do nothing.
    }
  };  

//create our map with functions enabled.    
Map<String, Double, PureFunction<String, Double, Return.Map<Double>>> products =  
  MapConfig.functionsOn(stringSerializer(), doubleSerializer())  
    .registerFunction(discount) //register the discount function
    .get();  

//insert two products that expire after a day. 
products.put("MacBook Pro", 2799.00, Duration.ofDays(1));  
products.put("Tesla", 69275.0, Duration.ofDays(1));  

//apply the discount function.
products.applyFunction("MacBook Pro", "Tesla", discount);

We can print the content of the Map with the following

products.stream().forEach(System.out::println);

Before the discount is applied will print

KeyVal(MacBook Pro, 2799.0)
KeyVal(Tesla, 69275.0)

After the discount is applied prints

KeyVal(MacBook Pro, 1399.5)
KeyVal(Tesla, 34637.5)

See the above example on GitHub - DiscountApp.java.

What is function ID?

Each PureFunction type also implement a String id value which gets stored in the database. Currently the id is defaulted to this.getClass().getName(); which can be overridden by your function.

Note & TODOs

The function itself is not serialised. Only the function's id gets serialised and stored in the database. TODO - Serialisable functions.
.mightContainFunction can be used to check if a function is in use. TODO - Automatic function removal.
TODO - stronger types when registering and applying function.

Useful links

SwayDB on GitHub.
Java examples repo.
Kotlin examples repo.
Scala examples repo.
Documentation.

Caching & IOStrategy @ SwayDB

Simer Plaha — Fri, 27 Mar 2020 04:30:41 +0000

We all know that Memory IO is 50-200 times faster than Disk IO!

Caching plays a decent role in boosting read and compaction performance but some machines & smaller devices (eg: mobile phones) might have not have enough memory for cache so a configurable cache allowing disabled, partial or full cache is required.

Note - A Segment(.seg) file in SwayDB is simply a byte array that stores other bytes arrays like keys, values, indexes etc (Array<Array<Byte>>). All these bytes can be cached based on any condition which is configurable.

Configuring IO and Cache

When accessing any file with a custom format we generally

Open the file (OpenResource)
Read the file header or info (ReadDataOverview) to understand the files content eg: format etc.
Finally read the content of the file (Compressed or Uncompressed data).

The following sample ioStrategy function does exactly that where we get an IOAction that describes what IO is being performed by SwayDB and in our function we define how we want to perform IO (IOStrategy) for that action and also configure caching for the read data/bytes.

.ioStrategy(  
  (IOAction ioAction) -> {  
    if (ioAction.isOpenResource()) {  
      //Here we are just opening the file so do synchronised IO because
      //blocking when opening a file might be cheaper than thread 
      //context switching. Also set cacheOnAccess to true so that other 
      //concurrent threads accessing the same file channel do not 
      //open multiple channels to the same file.
      return new IOStrategy.SynchronisedIO(true);  
  } else if (ioAction.isReadDataOverview()) {  
      //Data overview is always small and less than 128 bytes and can be 
      //read sychronously to avoid switching threads. Also cache
      //this data (cacheOnAccess) for the benifit of other threads and to save IO.
      return new IOStrategy.SynchronisedIO(true);  
  } else {  
      //Here we are reading actual content of the file which can be compressed
      //or uncompressed. 
      IOAction.DataAction action = (IOAction.DataAction) ioAction;  
      if (action.isCompressed()) {
        //If the data is compressed we do not want multiple threads to concurrently
        //decompress it so perform either Async or Sync IO for decompression
        //and then cache the compressed data. You can also read the compressed 
        //and decompressed size with the following code
        //IOAction.ReadCompressedData dataAction = (IOAction.ReadCompressedData) action;  
        //dataAction.compressedSize();  
        //dataAction.decompressedSize();
        return new IOStrategy.AsyncIO(true);  
      } else {  
        //Else the data is not compressed so we allow concurrent access to it. 
        //Here cacheOnAccess can also be set to true but that could allow multiple
        //threads to concurrently cache the same data. If cacheOnAccess is required 
        //then use Asyc or Sync IO instead.
        return new IOStrategy.ConcurrentIO(false);  
      }  
    }  
  }  
)

You will find the above ioStrategy property in all data-blocks that form a Segment - SortedKeyIndex, RandomKeyIndex, BinarySearchIndex, MightContainIndex & ValuesConfig.

A Segment itself is also a data-block and it's ioStrategy can also be configured via SegmentConfig.

Cache control/limit with MemoryCache

Caching should be controlled so that it does not lead to memory overflow!

You can enable or disable caching for any or all of the following

Bytes within a Segment (ByteCacheOnly).
Parsed key-values (KeyValueCacheOnly).
Or all the above (MemoryCache.All).

By default ByteCacheOnly is used because KeyValueCacheOnly uses an in-memory SkipList and inserts to a large SkipList are expensive which is not useful for general use-case. But KeyValueCacheOnly can be useful for applications that perform multiple reads to the same data and if that data rarely changes.

An Actor configuration is also required here which manages the cache in the background. You can configure the Actor to be a Basic, Timer or TimerLoop.

The following demoes how to configured all caches.

//Byte cache only
.setMemoryCache(  
  MemoryCache  
    .byteCacheOnlyBuilder()
    .minIOSeekSize(4096)
    .skipBlockCacheSeekSize(StorageUnits.mb(4))  
    .cacheCapacity(StorageUnits.gb(2))  
    .actorConfig(new ActorConfig.Basic((ExecutionContext) DefaultConfigs.sweeperEC()))  
)
//or key-value cache only
.setMemoryCache(  
  MemoryCache  
    .keyValueCacheOnlyBuilder()  
    .cacheCapacity(StorageUnits.gb(3))  
    .maxCachedKeyValueCountPerSegment(Optional.of(100))  
    .actorConfig(new Some(new ActorConfig.Basic((ExecutionContext) DefaultConfigs.sweeperEC())))  
)
//or enable both the above.
.setMemoryCache(  
  MemoryCache  
    .allBuilder()  
    .minIOSeekSize(4096)  
    .skipBlockCacheSeekSize(StorageUnits.mb(4))  
    .cacheCapacity(StorageUnits.gb(1))  
    .maxCachedKeyValueCountPerSegment(Optional.of(100))  
    .sweepCachedKeyValues(true)  
    .actorConfig(new ActorConfig.Basic((ExecutionContext) DefaultConfigs.sweeperEC()))  
)

`minIOSeekSize`

The blockSize which set the minimum number of bytes to read for each IO. For example in the above configuration if you ask for 6000 bytes then 4096 * 2 bytes will be read.

The value to set depends on your machines block size. On Mac this can be read with the following command:

diskutil info / | grep "Block Size"

which returns

Device Block Size: 4096 Bytes
Allocation Block Size: 4096 Bytes

`skipBlockCacheSeekSize`

This skips the BlockCache and perform direct IO if the data size is greater than this value.

`cacheCapacity`

Sets the total memory capacity. On overflow the oldest data in the cache is dropped by the Actor.

`maxCachedKeyValueCountPerSegment`

If set, each Segment is initialised with a dedicated LimitSkipList. This cache is managed by the Actor or by the Segment itself if it gets deleted or when the max limit is reached.

`sweepCachedKeyValues`

Enables clearing cached key-values via the Actor. If false, key-values are kept in-memory indefinitely unless the Segment gets deleted. This configuration can be used for smaller databases (eg: application configs) that read the same data more often.

Memory-mapping (MMAP)

MMAP can also be optionally enabled for all files.

Map<Integer, String, Void> map =  
  MapConfig  
    .functionsOff(Paths.get("myMap"), intSerializer(), stringSerializer())  
    .setMmapAppendix(true) //enable MMAP for appendix files
    .setMmapMaps(true)  //enable MMAP for LevelZero write-ahead log files
    .setSegmentConfig(  //configuring MMAP for Segment files
      SegmentConfig  
        .builder()  
         ...
         //either disable memory-mapping Segments
        .mmap(MMAP.disabled()) 
         //or enable for writes and reads.
        .mmap(MMAP.writeAndRead())  
         //or enable for reads only. 
        .mmap(MMAP.readOnly())  
        ...
     )  
    .get();  

map.put(1, "one");  
map.get(1); //Optional[one]

Summary

You are in full control of Caching & IO and can configure it to suit your application needs. If your IOStrategy configurations uses only AsyncIO and ConcurrentIO then you can truely build reactive applications which are non-blocking end-to-end other than the file system IO performed by java.nio.* classes. Support for Libio to provide aysnc file system IO can be implemented as a feature if requested.

Useful links

SwayDB on GitHub.
Java examples repo.
Kotlin examples repo.
Scala examples repo.
Documentation.

Compression @ SwayDB

Simer Plaha — Wed, 25 Mar 2020 12:13:44 +0000

Enabling full compression can reduce data size by 89.3%!

Compression is important for performance because more data can be stored & read into memory within a block, it reduces IOps and the cost of hosting data on servers/cloud. But over-compression can get expensive due to time required for decompression which could effect read performance but it can also be useful for applications that need cold storage or just high compression.

A configurable compression strategy is required to support unique storage requirements allowing us to define how much & how often we want to compress data. For example: if your data file (Segment) is of size 100MB with 100K key-values you can express your compression requirements like

Compress the full 100MB data file.
Compress data every 4MB.
Compress data every 1000 key-values.
Compress all data but reset compression every 10^th key-value
Do not compress all data but compress every 10^th key-value.
Compress binary-search-indexes & hash-indexes but not other data-blocks.

Note: data-block is referred to a logical set of bytes (Array<Byte>) that are stored within a Segment like indexes, keys, values etc. A Segment itself is a data-block that stores other data-blocks within itself (Array<Array<Byte>>).

Compression strategies

You can combine, enable or disable any or all of the following compression strategies

Internal-compression includes prefix compression and duplicate value elimination.
External-compression uses LZ4 and/or Snappy which can applied selectively to parts of a file or to the entire file.

Prefix compression

Prefix compression stores all keys in a compressed group format in their sorted order. Reading a single key from that group requires decompressing all keys that exist before the searched key within the group, so for read performance it is useful to leave some keys uncompressed. You can also prefix compress all keys if you just want high compression.

The following Map is configured to compress 4 keys into a group and starts a new group every 5^th key. The input boolean parameter named keysOnly is set to true which applies prefix-compression to keys only, if false, it applies prefix-compression to keys and all metadata that gets written with that key which would result in higher compression.

Map<Integer, String, Void> map =  
  MapConfig  
    .functionsOff(Paths.get("myMap"), intSerializer(), stringSerializer())  
    .setSortedKeyIndex(  
      SortedKeyIndex  
        .builder()  
        .prefixCompression(new PrefixCompression.Enable(true, PrefixCompression.resetCompressionAt(5)))  
        ...
    )  
    .get();  

map.put(1, "one");  
map.get(1); //Optional[one]

In the following configuration prefix compression is applied to every 5^th key.

.prefixCompression(new PrefixCompression.Enable(false, PrefixCompression.compressAt(5)))

Prefix compression can also be disabled which optionally allows optimising sorted-index for direct binary-search without creating a dedicated binary-search byte array. You can read more about normaliseIndexForBinarySearch here.

.prefixCompression(new PrefixCompression.Disable(false))

Duplicate value elimination

Time-series or events data like weather, electricity, solar etc often contains duplicate values. Such duplicate values can be detected and eliminated with the following configuration.

ValuesConfig  
  .builder()  
  .compressDuplicateValues(true)  
  .compressDuplicateRangeValues(true)

Duplicate value elimination is very cost effective because it does not create or leave decompression markers on compressed data, instead all decompression information for that key is embedded within an already existing 1-2 bytes space.

Range values created by the range APIs like remove-range, update-range & expire-range are most likely to have duplicate values and can be eliminated/compressed with the compressDuplicateRangeValues(true) config.

External compression

Every data-block written into Segment file is compressible! A Segment file is nothing special but just another data-block that stores other data-blocks within itself.

You will find a compression property that configures external compression in all data-blocks that form a Segment - SortedKeyIndex, RandomKeyIndex, BinarySearchIndex, MightContainIndex, ValuesConfig & SegmentConfig.

All LZ4 instances and Snappy are both supported.

The following snippet demos how to apply compression to a SortedKeyIndex/Linear-search-index. It tries to compress with LZ4 first at minimum 20.0% compression savings, if the compression was lower than 20.0% then Snappy is tried with the same.

SortedKeyIndex
  .builder()
  .compressions((UncompressedBlockInfo info) ->
    Arrays.asList(
      //try running LZ4 with minimum 20.0% compression  
      Compression.lz4Pair(
        new Pair(LZ4Instance.fastestJavaInstance(), new LZ4Compressor.Fast(20.0)),
        new Pair(LZ4Instance.fastestJavaInstance(), LZ4Decompressor.fastDecompressor())
      ),
      //if not try Snappy  
      new Compression.Snappy(20.0)
    )
    ...
  )

UncompressedBlockInfo provides the data size (info.uncompressedSize()) of the data-block being compressed which can optionally be used to determine if it should be compressed or not. For example: if the data size is already too small then you can disable compression by returning Collections.emptyList().

.compression(
  (UncompressedBlockInfo blockInfo) -> {
    if (blockInfo.uncompressedSize() < StorageUnits.mb(1)) {
      return Collections.emptyList();
    } else { //else do compression
      return {your compression};
    }
  }

How to apply compression at a file Level? Similar to above you can apply file level compression with SegmentConfig.

.setSegmentConfig(  
  SegmentConfig  
    .builder()  
    ...  
    .compression((UncompressedBlockInfo info) ->
       {your compression config here}
    )  
)

How to limit compression by size & key-value count?

The property minSegmentSize sets the compressible size of Segment and if the above compression property is defined for a data-block then that data gets compressed every minSegmentSize data block.

The property maxKeyValuesPerSegment also controls the compressible limit of a Segment which, along with minSegmentSize enables checks to limit the maximum count/number of key-values to store within a compressible Segment.

.setSegmentConfig(  
  SegmentConfig  
    .builder()  
    .minSegmentSize(StorageUnits.mb(4))
    .maxKeyValuesPerSegment(100000)
    ...
)

Summary

SwayDB's compression is highly configurable and can be tuned for unique storage requirements with different tradeoffs.

GitHub repos

SwayDB on GitHub.
Java examples repo.
Kotlin examples repo.
Scala examples repo.
Documentation.

Hash-Index @ SwayDB

Simer Plaha — Sun, 22 Mar 2020 06:01:48 +0000

Hash-index @ SwayDB

Hash-indexes improves random read performance by 55%!

Hash-indexes like HashSet or HashMap are useful for serving random read requests. They increase performance for randomly accessed data, reduce total number of IOps and save CPU time. But they can also be costly for applications that rarely do random reads like

Time-series or event-source data might only require fast sequential reads.
Cold storage or archived data might not require random reads at all.

A configurable hash-index is required for SwayDB so various data storage requirements could be handled. The following were the requirements:

Optionally enable or disable hash-index.
Allow creation of perfect or nearly perfect hash-indexes.
Create partially indexed hash-indexes with fallback to alternative indexes like binary-search & linear-search.
Copy keys directly into the hash-index.
Controlled concurrency.
Compression.

Example configuration

The following creates a persistent Map where hash-index is disabled.

Map<Integer, String, Void> map =  
  MapConfig  
    .functionsOff(Paths.get("myMap"), intSerializer(), stringSerializer())  
    .setRandomKeyIndex(RandomKeyIndex.disable())  
    .get();

map.put(1, "one");  
map.get(1); //Optional[one]

The following enables hash-index (RandomKeyIndex) with custom configuration.

Map<Integer, String, Void> map =  
  MapConfig  
    .functionsOff(Paths.get("myMap"), intSerializer(), stringSerializer())  
    .setRandomKeyIndex(  
      RandomKeyIndex  
        .builder()  
        .maxProbe(10) //re-hash 10 times to resolve hash collision
        .minimumNumberOfKeys(20) //minimum keys required
        .minimumNumberOfHits(10) //minimum indexed keys required 
        //use reference format. IndexFormat.copyKeys() can also be used here
        //to copy keys directly into the hash-index
        .indexFormat(IndexFormat.reference())
        .allocateSpace(  
          (RandomKeyIndex.RequiredSpace optimalSpace) ->
            //allocates 3 times more storage than the 
            //default optimal space for higher hit rate. 
            optimalSpace.requiredSpace() * 3   
         )  
         //allow async IO for all IO operations
        .ioStrategy((IOAction ioAction) -> new IOStrategy.AsyncIO(true))  
         //Use LZ4 and fallback to Snappy.
        .compression((UncompressedBlockInfo info) ->  
          Arrays.asList(  
             //try running LZ4 with minimum 20.0% compression    
            Compression.lz4Pair(  
              new Pair(LZ4Instance.fastestInstance(), new LZ4Compressor.Fast(20.0)),  
              new Pair(LZ4Instance.fastestInstance(), LZ4Decompressor.fastDecompressor())  
            ),  
            //if LZ4 fails try Snappy with 20.0% compression.    
            new Compression.Snappy(20.0)  
         )  
       )  
    )  
    .get();

map.put(1, "one");  
map.get(1); //Optional[one]

For a detail documentation of the configurations refer to documentation website.

Useful links

SwayDB on GitHub.
Java examples repo.
Kotlin examples repo.
Scala examples repo.
Documentation website.

Intro to SwayDB

Simer Plaha — Sat, 08 Feb 2020 13:57:21 +0000

SwayDB is an embeddable persistent and in-memory key-value store/database.

Here I will try to demonstrate on how to create a Map in Scala and play with some basic APIs to do simple CRUD operations.

Creating a Map

You can ignore Nothing and Bag.Less for now which will explained in future posts.

import swaydb._ //swaydb's APIs
import swaydb.serializers.Default._ //default serialisers

val map = memory.Map[Int, String, Nothing, Bag.Less]().get //create a memory map

Similarly a persistent Map can be created by replacing memory.Map with persistent.Map and providing a folder name.

val map = persistent.Map[Int, String, Nothing, Bag.Less](dir = "target/myMap").get

Basic Put & Get


map.put(key = 1, value = "one")
map.get(key = 1) //returns Some("one")

Expiring & Updating data

//put and expire after 1 hour
map.put(key = 2, value = "two", expireAfter = 1.hour) 
//updates value but leave the expiration unchanged.
map.update(key = 2, value = "two updated")
//returns expiration deadline.
map.expiration(key = 2)

Remove/Deleting data

Batch removes both keys.

map.remove(keys = Seq(1, 2))
//return true - Map is now empty.
map.isEmpty

Run on Scastie.

Done!

DEV Community: Simer Plaha

Decentralised systems - SwayDB

How is Time used?

How can it be used by decentralised systems?

Example write behaviour

Custom TimeOrder

API Status

MultiMap - Tables in SwayDB

Initialise MultiMap

Creating Tables

Schema tree

Creating tables with default expiration

Removing and replacing tables

Transactions

Code

EAP

GitHub repos

Functions as Queries @ SwayDB

Example

What is function ID?

Note & TODOs

Useful links

Caching & IOStrategy @ SwayDB

Configuring IO and Cache

Cache control/limit with MemoryCache

minIOSeekSize

skipBlockCacheSeekSize

cacheCapacity

maxCachedKeyValueCountPerSegment

sweepCachedKeyValues

Memory-mapping (MMAP)

Summary

Useful links

Compression @ SwayDB

Compression strategies

Prefix compression

Duplicate value elimination

External compression

How to limit compression by size & key-value count?

Summary

GitHub repos

Hash-Index @ SwayDB

Hash-index @ SwayDB

Example configuration

Useful links

Intro to SwayDB

Creating a Map

Basic Put & Get

Expiring & Updating data

Remove/Deleting data

Initialise `MultiMap`

`minIOSeekSize`

`skipBlockCacheSeekSize`

`cacheCapacity`

`maxCachedKeyValueCountPerSegment`

`sweepCachedKeyValues`