loading...
Cover image for Just in! A New Persistent NoSQL Database (18 KiB only!)

Just in! A New Persistent NoSQL Database (18 KiB only!)

trinly01 profile image Trinmar Boado ・5 min read

Welcome to Trin.DB!

A fast RESTful persistent or in memory NoSQL database (18 KiB only!)

Github Repo: https://github.com/trinly01/TrinDB

Installation

npm install trin.db

or

yarn add trin.db

Usage

const express = require('express')
const app = express()
const port = process.env.PORT || 3000
const trinDB = require('trin.db')

app.use(express.json())               // required for RESTful APIs

app.listen(port, async () => {
  app.trinDB = {
    todos: await trinDB({             // Todos Service
      filename: 'trinDb/todos.db',    // get records from a file
      inMemoryOnly: false,            // Optional
      restful                         // Optional
    })
  }
})

// Other Options

const restful = {                     // Optional
  app,                                // express app
  url: '/todos',                      // API end-point
  hooks                               // Optional
}

const hooks = ({app, service}) => ({  // Hooks Example
  before: {
    all: [
      (req, res, next) => {
        console.log('before all hook')
        next()
      }
    ],
    get: [],
    find: [],
    create: [],
    patch: [],
    remove: []
  }
  after: {
    all: [
      (result) => {
        console.log(result)
        return result
      }
    ],
  }
})

await trinDB(<object>)

Returns a trinDB Service

Object Prop Type Description Default Mode
filename <string> Path to file. Required if in persistent mode n/a persistent
inMemoryOnly <boolean> ( Optional ) If true, database will be in non-persistent mode false in-memory
restful <object> ( Optional ) { app, url, hooks } n/a persistent

service.create(<object>)

Returns the created <object>

/* example existing records (service.data)
  {
    asd: { _id: 'asd', text: 'Hello', read: true, nested: { prop: 'xander' } },
    zxc: { _id: 'zxc', text: 'World', read: false, nested: { prop: 'ford' } }
  }
*/

const result = service.create({
  text: 'Trinmar Pogi'
})

console.log(result)
// { _id: 'qwe', text: 'Trinmar Pogi' }

console.log(service.data)
/* service.data with the newly created object
  {
    asd: { _id: 'asd', text: 'Hello', read: true, nested: { prop: 'xander' } },
    zxc: { _id: 'zxc', text: 'World', read: false, nested: { prop: 'ford' } },
    qwe: { _id: 'qwe', text: 'Trinmar Pogi' }
  }
*/

RESTful API

curl --location --request POST 'http://localhost:3000/todos' \
--header 'Content-Type: application/json' \
--data-raw '{
  "text": "Trinmar Pogi"
}'

service.find(<object>)

Returns found data <object>

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 }
  }
*/

// Equality
result = service.find({
  query: {
    lastName: 'Pogi' // equality
  },
  limit: 10, // default 10
  skip: 0 // default 0
})
console.log(result)
/*
  {
    total: 1,
    limit: 10,
    skip: 0,
    data: {
      asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 }
    }
  }
*/

RESTful API

curl --location --request GET 'http://localhost:3000/todos?lastName=Pogi&$limit=10&$skip=0'

Complex Query (conditional >, >==, <, <==, &&, || )

// Map data or select specific props
result = service.find({
  query (obj) {
    return ob.age < 20
  },
  map (obj) {
    return {
      fullName: obj.firstName + ' '+  obj.lastName
    }
  }
})
console.log(result)
/*
  {
    total: 2,
    limit: 10,
    skip: 0,
    data: {
      zxc: { _id: 'zxc', firstName: 'Trinly Zion Boado' },
      qwe: { _id: 'qwe', firstName: 'Lovely Boado' }
    }
  }
*/

service.search(keywords)

fuzzy search finds data based on the keywords (<String>) and returns it sorted by _score

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Boado' },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado' },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado' }
  }
*/

result = service.search('ly oad')

console.log(result)
/*
  {
    total: 3,
    data: {
      qwe: { _score: 2, _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 },
      zxc: { _score: 2, _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
      asd: { _score: 1, _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    }
  }
*/

RESTful API

curl --location --request GET 'http://localhost:3000/todos?$search=ly%20oad'

service.patch(_id, <object>)

Returns the created <object>

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Boado' nested: { counter: 123 } }

const result = service.patch('q12m3k', {
  lastName: 'Pogi',
  children: ['Trinly Zion'],
  'nested.counter': 456
})

console.log(result)
// { _id: 'q12m3k', lastName: 'Pogi' children: ['Trinly Zion'], 'nested.counter': 456 }

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "lastName": "Pogi",
    "children": ["Trinly Zion"],
    "nested.counter": 456
}'

service.remove(_id)

Returns the removed <object>

service.remove('q12m3k')

console.log(service.data['q12m3k'])
// undefined

RESTful API

curl --location --request DELETE 'http://localhost:3000/todos/:_id'

service.removeProps(_id, <object>)

Returns the removed <object> props

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

service.removeProps('q12m3k', {
  lastName: true,
  'nested.prop': true
  firstName: false
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "removeProps"
    "lastName": true,
    "nested.prop": true,
    "firstName": false
}'

service.inc(_id, <object>)

Increments specific props and returns the <object>

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

service.inc('q12m3k', {
  'nested.prop': 5
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 461 }, children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "inc"
    "nested.prop": 5
}'

service.splice(_id, <object>)

removes element by index and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.splice('q12m3k', {
  'children': 1
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "splice"
    "children": 1
}'

service.push(_id, <object>)

adds one or more elements to the end of an array and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.push('q12m3k', {
  'children': 'Lovely Boado'
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado', 'Lovely Boado'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "push"
    "children": "Lovely Boado'"
}'

service.unshift(_id, <object>)

adds one or more elements to the beginning of an array and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.unshift('q12m3k', {
  'children': 'Lovely Boado'
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Lovely Boado', 'Trinly Zion', 'Trinmar Boado'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "unshift"
    "children": "Lovely Boado'"
}'

service.sort(data,<object>)

Sorts the data based on the <object> and returns the sorted data

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 }
  }
*/

// Descending (-1)
result = service.sort({
  data: service.data, // (Optional) if not defined, service.data will be used
  params: {
    age: -1
  }
})

console.log(result)
/*
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 }
  }
*/

service.copmact(filename, <object>)

writes the compact data to a file
| param | Type | Description | Default |
|--|--|--|--|
| filename | <string> | (Optional) Path to file | current |
| | <object> | ( Optional ) a TrinDB object | service.data |

service.copmact('test.db', service.data)

Github Repo: https://github.com/trinly01/TrinDB

Join and support our Community
Web and Mobile Developers PH
[ Facebook Page | Group ]

Posted on by:

trinly01 profile

Trinmar Boado

@trinly01

An ICT Advocate, Trainer and Consultant. He is giving talks and facilitating Hands-on Trainings to schools and other IT

Discussion

markdown guide
 

This isn't very good:

module.exports = (path, data, fs) => {
  fs.writeFileSync(path, '')
  for (const key in data) {
    const line = JSON.stringify({
      action: 'create',
      data: data[key]
    })
    fs.appendFileSync(path, `${ line }\r\n`)
  }
}
  1. Open the file properly, consider if locks are needed.
  2. Make it non-blocking.
  3. Why break it into lines?
  4. Use binary serialization such as msgpack.
  5. Consider compression.
 

npmjs.com/package/msgpack

node-msgpack is currently slower than the built-in JSON.stringify() and JSON.parse() methods. In recent versions of node.js, the JSON functions have been heavily optimized. node-msgpack is still more compact, and we are currently working performance improvements. Testing shows that, over 500k iterations, msgpack.pack() is about 5x slower than JSON.stringify(), and msgpack.unpack() is about 3.5x slower than JSON.parse().

 

I'm aware of this though it's a bit more complex. Non-native msgpack implementations will find it hard to compete against native implementations.

I did a lot of research on binary serialization. Specifically writing several of my own, comparing to igbinary and msgpack for speed, size when uncompressed and size when compressed.

Two things were difficult:

  1. Making any difference in size, except for string deduplication (igbinary does quite well with this even post compression). Note that most improvements made to the binary to make it smaller really had much the same effect as pre-compressing so rarely made any difference after compression.
  2. Anything, and I mean getting anything much faster in JS be it in JS itself or C++ was very difficult to get the same performance gain you get compared to something like PHP C extensions. A lot of the overhead in JS is baked right into the objects used under the hood in C++ if I remember not just the interpretation.

msgpack being slower is a problem though I'm not sure why it should be so at least on the backend. I only ever considered that a problem on the frontend. It should have all the same limitations and advantages as the native JSON code does. Unless, which I can't be sure about as it was a while ago, there might have also been some barriers between the engine and extensions.

In theory a backend implementation of msgpack, at least minimal, would be little more than a copy and paste of the one for JSON (though if it uses something like YACC it's overkill). It should basically be the same but with less so faster.

It doesn't help that JS isn't entirely binary friendly (another area which it sorely hurt compared to PHP, which I mention a lot because a lot of infrastructure is a mix of the two and they need to talk to each other).

JSON does in some circumstances have size benefits but they're not often brought out.

It's a shame that msgpack isn't as fast at least frontend due to no native (tried that webasm stuff / cutting edge JS though, typed arrays, etc)?

Regardless, it's not always the main thing. It's still potentially much smaller which might be important if bandwidth or the amount written / read is a concern.

I haven't checked recently, but msgpack may still not support dictionaries as an extra option. You can however do this yourself. In the serialized stream you might look for if there are repeat strings and then store those only once in an array then the string type points to those instead.

That has an interesting benefit as you can reduce memory use as well as a side effect reducing copies of the same string. Though I don't know if JS engines are able to do that themselves in any situation.

A similar thing might be done with object arrays. IE if they're all the same, instead of [{a: 123, b: false}, {a: 321, b: true}] have ['a', 'b', 123, false, 321, false] though that kind of things you might want to better do yourself over the serialization. You can do the same for a string dictionary like igbinary has which is surprisingly effective even post compression but might have complications or limitations with streaming and memory usage patterns.

This is a similar concept to gson and interned strings. I've been using those successful and for proven gain for some time now.

There's probably a missing pre/post pack library out there for these cases. They can all take more or less CPU but it's basically the same trade off the same as with compression, spend more time but get better results.

I found it very annoying with very large things that you can't get the best of both worlds with msgpack. IE, you notice both the CPU time and net time.

It wouldn't surprise me if one of the API's somewhere sneaking into modern JS has some binary serialisation that might be native.

I'm seeing C++ for msgpack so it should at least in theory be possible to get it up to JSON speed on the backend. It's sometimes more important on the backend because that's where the contention is.

Thanks for the detailed explanation ❤️

My goal is to be pure JS implementation with no binary dependencies.

I think streams are optimized to minimize CPU and memory overheads

There's always a lot to learn, I still learn every day.

No bins is good for supporting front and back as well as being hardware agnostic.

You should take a look at fs.open, and see if there's a flock, etc.

You might decide you don't need to implement flock but if you make that decision it's crucial to understand concurrency issues.

To test this might be easier than you think. See if there's a sleepSync. Then after writing each line sleep.

Then run a basic program filling the in memory store then saving it twice at the same time.

You'll see lines from both. Ok it doesn't matter you mgiht think they're the same and it's just the same op twice but initialise each database so that they have the same keys but different values.

If you really push it, just run the program thousands of times at the same time with no delays, then you can get something like the file will have one line that suddenly stops and then the line from another program continues meaning the lines wont even be parsable in JSON.

When people are starting out the first thought is "how can I make this work". The next step, the revolution, is thinking "how can I break this" :D.

 

Thanks for the suggestion. <3
I already converted it from appendFileSync to fileWriteStream

 

These are concepts you might find it useful to study and learn as well as experiment with.

Non-blocking is a crucial concept in JS that gives it much of it's performance (async). While this write operation is happening everything is blocked and nothing can happen.

By also writing to the file like that, it's opening it everytime, writing the line then closing it. You can instead open it, write all the lines, then close it.

fs.writeFileSync(path, JSON.stringify(data)) is enough in this case or fs.writeFileSync(path, JSON.stringify(['create', data])).

You can probably also get away with just \n for lines in many cases. Line endings only matter these days in niche situations and \n works well as a standard.

You should check options to see if writeFileSync supports locking but otherwise you may need to look up the lock functions. If it's not concurrency safe and is non-blocking it might need to say that in the documentation otherwise using it can mess up someone's application either due to data corruption or routine freezes.

It might suit your app, IE, you only run it once at a time per path and either can tolerate delays or are working with small amounts of data but for others it would not be even half way stable.

Once you learn it and apply it once it's like riding a bicycle. It's much more enjoyable though if you write some minimal benchmarks and tests. It's then nice to see it doing better so you get a reward for it.

 

Would love if you showed some benchmarks instead of copy-pasting the whole API here.

 

Hi Atta,
Nice suggestion!
You may contribute to this project by adding a benchmark.
Will love to merge it ❤️

 

No it's your job to provide benchmarks. :(

To be honest, I get your point, but I don't like this wording when it comes to open source projects :)

Projects are projects. You can't expect other people to provide benchmarks for your project. You are the one advertising it, so you need to provide benchmarks. So yes, it is your job. Open source is not an excuse.

 

Looks very interesting! How is the performance when you have 100.000 - 1.000.000 records?

 

Hi Roelof, Contributors are welcome.
Need help to test it using real world scenarios.

 

Currently I am interested in JSON schema validation, perhaps with Ajv.

 

All suggestions are being considered. Some are already implemented. Thank you to all of insights and inputs ❤️