TOSS: 1 Secret to Achieve Eventual Consistency of Edges in Nebula Graph

Nebula Graph has just released v.2.6. In this version, TOSS is certainly one of the important features. Here a detailed explanation about TOSS will be given.

Let’s start from a GO statement

As we all know, there are two types of edges, directed and undirected edges. When traversing directed edges, you can traverse forward or reversely. Nebula Graph also supports this semantics. For example:

go from "101" over known reversely yield known.kdate, id($$);

The above statement starts from Vertex 101 to find all the corresponding neighbors reversely. However, when you insert an edge into Nebula Graph, the command will be like:

insert edge known(degree) VALUES "100" -> "101":(299792458);

Seemingly, the above statement only specifies the outgoing edge. This is because Nebula Graph will specify the incoming edge in the background when you insert an edge.

How to insert an edge into Nebula Graph

Take the INSERT statement above as an example, the background execution process contains the following:

Nebula Console sends the INSERT request to the Nebula Graph server.

After the Nebula Graph server receives the request, it adds an incoming edge for each outgoing edge and sends AddEdgeRequest to their hosts respectively.

After the host (Nebula Storage server) receives AddEdgeRequest, it inserts the edge locally (via the Raft protocol) and returns the result to the Nebula Graph server.

The Nebula Graph server then returns the results from both hosts to the Nebula Console for querying purpose.

The flow diagram is as follows:

If you are familiar with network/distributed programming, you may see the problem now. The graph service uses RPC to call both storage services. When the INSERT operation is executed enough times, one RPC succeeds while the other fails due to timeout. In other words, an INSERT operation may succeed on the outgoing edge while fail on the incoming-edge.

If now a user requires consistent property settings for both outgoing edge and incoming edge, the client has to retry the query infinitely. It is inappropriate for Nebula Graph, s a database product, to rely on the client for data atomicity.

A requirement thus comes into being, that is, to ensure the atomicity of outgoing edge and incoming edge. This means that the outgoing edge and incoming edge should be updated either successfully or they should fail at the same time. And TOSS (Transaction On Storage Side) is the feature to ensure the eventual consistency of edges upon INSERT, UPDATE, and UPSERT operations.

How to use TOSS

With the release of Nebula Graph v2.6.0, the TOSS function has also been launched. The feature is set to Disabled by default due to performance and stability considerations. You can find the enable_experimental_feature option in the Nebula Graph server configuration file and set it to True. Then you need to restart the graphd service for the feature to take effect. The command is as follows:

--enable_experimental_feature=true

Then the operations CREATE SPACE / CREATE EDGE / INSERT / UPDATE will achieve eventual consistency of edges in Nebula Graph. (Just execute the operations as before)

Note: The TOSS feature will be only applied to incremental data after it is enabled.