DEV Community: tato

MongoDB Performance 3.4 vs 4.4 vs 5.0 vs 6.0

tato — Mon, 26 Sep 2022 13:41:01 +0000

We've been using MongoDB v3.4.24 for quite a long time. And ever since MongoDB v6 got released, we have wanted to benchmark all these new versions to see if they fit our needs. We did such a benchmark years ago against MongoDB v4, but it turned out that MongoDB v4 was worse than MongoDB v3 in terms of performance. This time, we'll do a more detailed benchmark on MongoDB 3.4.24, 4.4.16, 5.0.12, and 6.0.1.

We ran the benchmark with https://github.com/mongodb/mongo-perf.

To better observe system resource usage and minimize the noises and randomness of the results, we've separated the testing server and client into 2 VMs with dedicated CPUs:

Server	DigitalOcean CPU-Optimized 4 CPUs / 8 GB, Ubuntu 20.04
Client	DigitalOcean CPU-Optimized 2 CPUs / 4 GB, Ubuntu 20.04 + MongoDB Shell (v5.0.12) + mongo-perf

Server benchmark

Here is the python script for running the benchmark (change 6.0.1 to the version that is being tested, change 10.1.0.2 to the internal IP of the testing server, and change -t 1 to -t 4 to run with four threads):

#!/usr/bin/bash
tests=("simple_insert" "simple_multi_update" "simple_query" "simple_remove" "simple_text" "simple_update" "simple_update_mms" "simple_geo")
for i in {1..10}; do
  echo "==== Round $i ===="
  for t in "${tests[@]}"; do
    python3 -u benchrun.py -f testcases/$t.js --host 10.1.0.2 -t 1 --trialTime 1 --out 6.0.1-$t-$i.json
  done
done

We ran the tests ten times for each server, cut out the lowest and the highest value, and averaged the rest. We didn't run through all tests provided by the repo, but that's enough for our use case.

Here are the results for one thread -t 1 (higher is better, the winner for each test is highlighted in red):

The results for the four threads -t 4 look similar on winner distributions.

We are also attaching the system resource usage of the server with -t 4 if someone is interested.

It is disappointing that MongoDB v3.4.24 leads in almost all write operation tests (insert, update and delete) and most aggregation tests. v4.4.16 wins in the rest of the aggregation tests. v5.0.12 is the biggest loser. And finally, 6.0.1 wins for all queries.

There's no reason for us to migrate to v4.4.16 and v5.0.12. As for v6.0.1, although it received an impressive query performance bump, probably related to the Slot-Based Query Execution Engine, its write operation performance was only around 60% of v3.4.24, which was the worst among all four versions. We won't risk trading writing performance with querying.

Driver benchmark

We've done further tests on MongoDB ruby drivers (mongo gem, versions v2.13.0 and v2.18.1) by inserting a certain number of documents and measuring the duration. Here's the script for testing:

gem 'mongo', '=2.13.0'
require 'mongo'
require 'digest'
require 'benchmark'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test')

puts Benchmark.measure {
  100_000.times do |index|
    client[:users].insert_one(name: Digest::MD5.hexdigest(index.to_s))
  end
}

Results: (mongo v2.18.1 doesn't support MongoDB v3.4.24)

We can see that mongo v2.13.0 is slightly better than v2.18.1 on insert time, which is frustrating. Also, inserting performance degrades from v3.4.24 to v6.0.1.

Out-of-memory test on `$near`

Our final test was an OOM-Killer situation that occurred in our v3.4.24 setup. It failed once we concurrently fed 5k $near queries to the server.

Here is the script used for testing:

gem 'mongo', '=2.13.0'
require 'mongo'
require 'timeout'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test', max_pool_size: 5000)

col = client[:locations]

5000.times.map do
  Thread.new do
    col.find({
      :gps => {
        "$near" => {
          "$geometry": { type: "Point",  coordinates: [ rand(-90.000000000...90.000000000), rand(-180.000000000...180.000000000) ] },
        }
      }
    }).to_a
  end
end.each(&:join)

The location collection was seeded with 10k entries.

Unfortunately, all four versions (v3.4.24, v4.4.16, v5.0.12, v6.0.1) still got OOM-killed within 30 seconds after running the script.

There's an issue still unresolved, claiming that $near is using unbounded memory.

We've also tried to stop the script right before the server crashes. Hoping that it would stop at a high memory consumption. We wanted to see if the extra memory would be released just like with GC. But sadly, it didn't happen (we waited for 3 hours).

Conclusion

To sum up, MongoDB had not put its focus on performance optimization, at least for write operations. Writing performance became even worse on more recent versions. The only thing encouraging is that MongoDB v6 has introduced a new query engine that boosts query performance.

It is also frustrating to find out that newer versions of mongo ruby driver have poorer writing performance than older versions.

The memory leak issue regarding the $near query has not been resolved in any of the tested versions.

These results ultimately killed our enthusiasm for upgrading MongoDB at this time. We will continue to monitor and test the performance of newer versions.

Creating collabrative javascript spreadsheet made easy

tato — Tue, 26 Apr 2022 09:43:58 +0000

In the last post, we have introduced how to create a javascript spreadsheet using FortuneSheet. This time, we will show you how to enable it's collabration feature.

Backend

Before we get into the topic, the first thing to be considered is backend storage, as the sheet data have to be saved so that others can see the lastest sheet state when they enter the page.

Websocket is our main transport of exchanging live data among clients. Here we choose Express as backend server, and MongoDB as database. The main role of express server is to manage websocket connection, serve initial data and process incremental op messages.

In this demo we simply use all documents of a collection for the sheets of our demo workbook.

async function getData() {
  const db = client.db(dbName);
  return await db.collection(collectionName).find().toArray();
}

We will skip the code of creating a simple express server, and focus on the core code. Don't worry, you can find the complete code at the end of this post.

Then, add some code for processing websocket messages, and manage connections for broadcasting.

const connections = {};

const broadcastToOthers = (selfId, data) => {
  Object.values(connections).forEach((ws) => {
    if (ws.id !== selfId) {
      ws.send(data);
    }
  });
};

const wss = new SocketServer({ server, path: "/ws" });

wss.on("connection", (ws) => {
  ws.id = uuid.v4();
  connections[ws.id] = ws;

  ws.on("message", async (data) => {
    const msg = JSON.parse(data.toString());
    if (msg.req === "getData") {
      ws.send(
        JSON.stringify({
          req: msg.req,
          data: await getData(),
        })
      );
    } else if (msg.req === "op") {
      await applyOp(client.db(dbName).collection(collectionName), msg.data);
      broadcastToOthers(ws.id, data.toString());
    }
  });

  ws.on("close", () => {
    delete connections[ws.id];
  });
});

Here, the applyOp function is the core of collabration. It reads the Ops sent by our frontend library, and perform data mutations to the database.

Processing Ops

The Ops are generated by the patch of Immer.js, for example, here is an op when user sets the cell font to be bold on cell A2.

[
    {
        "op": "replace",
        "index": "0",
        "path": ["data", 1, 0, "bl"],
        "value": 1
    }
]

We have to convert this op into MongoDB update query. If we convert it directly, the result will look like:

db.updateOne(
  { index: "0" },
  { $set: { "data.1.0.bl": 1 } }
);

However, considering storage size, we stores cell data sparsely in the database, that is, instead of storing the entire 2-dimension cell array, we store a 1-dimension array of cells that contain values. Thus, a cell in the database is in the form of

{
   r: number, // row index
   c: number, // column index
   v: any, // cell value
}

and the above update query becomes:

db.updateOne(
  { index: "0" },
  { $set: { "celldata.$[e].v.bl": 1 } },
  { arrayFilters: [{ "e.r": 1, "e.c": 0 }] }
);

Updates to others fields of the sheet are similar.

And that's all of our backend server. For the complete code, see https://github.com/ruilisi/fortune-sheet/tree/master/backend-demo

Frontend

Now let's focus on the frontend part, which is quite simple.

Step 1, create a websocket connection.

const wsRef = useRef<WebSocket>();

useEffect(() => {
  const socket = new WebSocket("ws://localhost:8081/ws");
  wsRef.current = socket;

  socket.onopen = () => {
    socket.send(JSON.stringify({ req: "getData" }));
  };
}, []);

Step 2, send ops from local change, receive ops from others and apply them into the workbook.

// A ref of Workbook
const workbookRef = useRef<WorkbookInstance>(null);

// In useEffect
socket.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.req === "getData") {
    setData(msg.data);
  } else if (msg.req === "op") {
    workbookRef.current?.applyOp(msg.data);
  }
};

// Workbook declaration
<Workbook
  ref={workbookRef}
  onOp={(op) => socket.send(JSON.stringify({ req: "op", data: op }))} />

For the complete code, refer to https://github.com/ruilisi/fortune-sheet/blob/master/stories/Collabration.stories.tsx

Thanks for reading

The repo is hosted on Github, completely open source. Give us a star if you find it useful 😄. Feedbacks are much appreciated!

Create Excel-like javascript spreadsheet in less than 10 lines of code

tato — Sat, 16 Apr 2022 03:02:02 +0000

Spreadsheets are one of the most important format that stores and calculates data, and are also key components of products that focus on online document.

In this blog, we will see how to create a rich-featured javascript spreadsheet in just 10 minutes.

This is what it will look like when you finish:

Setup

Let's start by creating an empty project with create-react-app

If you already have a react project, you can skip this step.



npx create-react-app my-spreadsheet

or if you are using yarn



yarn create react-app my-spreadsheet

Then, go into my-spreadsheet folder, install our spreadsheet library



# using npm
npm install @fortune-sheet/react
# or using yarn
yarn add @fortune-sheet/react

Using the library

Replace src/App.js with the following content



import { Workbook } from "@fortune-sheet/react";
import "@fortune-sheet/react/dist/index.css"


function App() {
  return <Workbook data={[{ name: "Sheet1" }]} />
}

export default App;

Add the following lines into src/index.css



html, body, #root {
  height: 100%;
}

Great, all things are set up!

Now, start the project by running



# using npm
npm start
# or using yarn
yarn start

and there you go!

You can play around it and try the features.

In the next blog we will show you how to persist sheet data and make it collabrative, which will end up like this:

The spreadsheet library is completely opensource, for more information, check out

https://github.com/ruilisi/fortune-sheet

It is under active development, feedbacks are appreciated!