DEV Community: lisahui

How to use one Dashboard to see your cluster heath

lisahui — Tue, 01 Mar 2022 07:23:28 +0000

Distributed database systems like Nebula Graph perform well in storing data but they make DevOps difficult and complicated. Building and managing clusters is pain and time-consuming. Not to mention it is not easy to back up and upgrade the system, especially in the production environment.

Introducing Nebula Dashboard, a visualization tool that helps you manage your Nebula Graph clusters in an intuitive web user interface. Nebula Dashboard can help DevOps engineers and database administrators (DBAs) reduce the daily cost of managing a Nebula Graph cluster and ensure the stability of their systems.

If you want a free trial of Nebula Dashboard enterprise edition, click here: https://dashboard.nebula-graph.io/

The structure

The following figure gives the six main features of Nebula Dashboard. They are cluster management, monitoring, alarming, single cluster configuration, access control, and system setting.

The one dashboard to rule them all

First, let's dive into the main features of Nebula Dashboard: multi-cluster orchestration & lifecycle management, monitoring and alerting, and access control.

Multi-cluster orchestration & lifecycle management

The lifecycle for a Nebula Graph cluster refers to the journey from the creation of a cluster to management to eventual recycling.

Nebula Dashboard can manage the full lifecycle of your Nebula Graph cluster in a visualized way.

When you have not created a Nebula Graph cluster, Nebula Dashboard enables you to start the creation of a new cluster. If you are already working with Nebula Graph clusters, you can import those clusters into Nebula Dashboard in batch and monitor or manage them there. We will go through the entire process of creating and managing clusters in this chapter.

Create a cluster from Nebula Dashboard

There are five steps to create a cluster, as shown in the figure above:

Input a unique cluster name
Choose the Nebula Graph version that you want to create a cluster. Please note that Nebula Dashboard only supports Nebula Graph v2.0 and above. By default, Nebula Dashboard provides three built-in installation package: v2.6.1, v2.5.1, and v2.0.1. It will also support Nebula Graph v3.0.0 once it is released in mid February 2022.
Add nodes to the cluster. You may need to authorize the step from SSH.
Select the services to be deployed on different nodes. For simplicity, you can also use the "Auto Add Service” feature to evenly deploy services to different nodes.
Confirm everything is OK and click “Create Cluster.” And voilà, you have your Nebula Graph cluster up and running.

Importing a cluster into Nebula Dashboard

If you have a Nebula Graph cluster running, you can import it into Nebula Dashboard to manage it in a graphic user interface.

Cluster operations and maintenance

Once you have completed the above two steps, you can use Nebula Dashboard to manage your clusters. Cluster management can be tedious. It involves repeated actions such as management of clusters, nodes and services. Nebula Dashboard can free DevOps engineers from this unbearable boredom by simplifying the whole process of cluster management.

Here are what you can do with Nebula Dashboard to manage your cluster:

Node and service management
Nebula Dashboard’s Node management and Service management modules will give you an intuitive overview of all the information about running nodes in the cluster and services in each node. You can do actions such as adding empty nodes and starting or stopping services in a node.

Be informed. Be alerted.

Monitoring the health of a cluster is the first priority forNebula Dashboard. Nebula Dashboard provides cluster overview, node metrics monitoring, service metrics monitoring, and alarm notification capabilities to help DevOps engineers stay informed of their Nebula Graph cluster health.

Cluster overview

In the overview section, DevOps engineers can quickly understand the overall situation of the current cluster, including the distribution of services in the node, operational status, and customize indicators of their most concerned metrics to stay alarmed.

Node monitoring

Node monitoring is used to present all the information of a particular Nebula Graph node, including its CPU usage, memory, load, and disk. You can also set a baseline for a specific metric so that you can be alerted when the metric exceeds the baseline.

Service monitoring

Service monitoring mainly involves the metrics of three services of Nebula Graph clusters. The three services are graphd, metad and storaged. Currently, Nebula Dashboard can present dozens of cluster metrics and monitor them based on their aggregations and averages. These metrics include the number of slow queries and errors in the graphd service, vertex and edge latencies of storaged, and heartbeat latencies of the metad service.

Alert

Nebula Dashboard's alert module is a service for alerting the monitoring indicators of Nebula Graph clusters. You can view alert information, set alert rules and alert recipients in the module.

You can set up alert baselines of indicators that you care about, and the frequency and duration of the alert triggering, as well as the notification message template, so that the system will automatically send out notifications to alert you when any of the indicators triggers the corresponding rules.

You can view history alert messages, in-site alert messages, set up email and webhook notifications, and set alert rules in the alert module. Here is how the module works:

Create custom alert rules or activate alert rules.
An alert message will be sent when the system monitors any abnormal metric.
A pop-up in-site message of the alert is shown on the top right of Nebula Dashboard. The system will send out an alert email if you have set an email recipient.
You can then perform troubleshooting based on the alert message.

Fine-grained access control

As a multi-cluster management tool, Nebula Dashboard has enabled a fine-grained access control system to ensure security. Nebula Dashboard has designed two roles, and they are admins and users. While admins are authorized to perform overall configuration of the Dashboard, users are only entitled to operate within the scope that is assigned to them.

LDAP login

Nebula Dashboard supports login with Lightweight Directory Access Protocol (LDAP). Once the LDAP information is provided in the deployment phase, users can log in with their enterprise account. Admins can invite users by sending them an email containing a verification link.

Feature roadmap

Nebula Dashboard is designed to simplify the daily tasks of DevOps engineers and database administrators (DBAs). It will support the following features in the forthcoming versions.

Back up and upgrade a cluster with one click. This feature will only be compatible with Nebula Graph v3.0 and above.
Visualized management of Storage Zone
TV dashboard that allows you to display key metrics in a customized way in a display in your office
Slow query management that allows DBAs to quickly identify slow queries.

Wanna try?

Check out the Nebula Dashboard playground to see how Nebula Dashboard works in practice.. You can also click this link to request a 15-day free trial to test Nebula Dashboard enterprise edition out in your own environment.

If you encounter any problems in the process of using Nebula Graph, please refer to Nebula Graph Database Manual to troubleshoot the problem. It records in detail the knowledge points and specific usage of the graph database and the graph database Nebula Graph.

Join our Slack channel if you want to discuss with the rest of the Nebula Graph community!

What I learned working on Nebula Graph, an open source and distributed graph database

lisahui — Thu, 24 Feb 2022 10:27:05 +0000

This article is based on a talk given by Dr. Min Wu, a senior expert at vesoft. Dr. Wu talked about the status quo of the global graph database market, the design and features of Nebula Graph as a distributed graph database, as well as Nebula Graph’s open-source community.

The Global Graph Database Market

Let’s start with some numbers. Markets and Markets anticipates the graph database market will reach $2.4 billion by 2023 from $821.8 million in 2018.

Graph database is still in a rising trend and it is one of Gartner’s top 10 data and analytics trends in 2021. This is because graph databases can be used in many more areas than traditional databases, including computing, processing, deep learning, and machine models.

Graph databases gained the most in popularity in the past 10 years, according to data compiled by DB-Engines. The data is based on social media mentions, Stack Overflow questions, and search trends.

Advantages of Graph Databases

One of the most significant advantages of graph databases is that they are intuitive. If you want to express the character relationships of Game of Thrones, you can use both traditional tabular databases and graph databases. But as it is shown below, using graph data is much more intuitive, though they both express the same data model.

(Tabular data)

(Graph data)

In another example, we can compare how to search in SQL databases and graph databases. For example, here is how to find out how many posts and comments were created in a given time frame and rank the results in SQL databases and graph databases.

SQL:

--PostgreSQL 

WITH RECURSIVE post_all (psa_threadid

                      , psa_thread_creatorid, psa_messageid

                      , psa_creationdate, psa_messagetype

                       ) AS (

    SELECT m_messageid AS psa_threadid

         , m_creatorid AS psa_thread_creatorid

         , m_messageid AS psa_messageid

         , m_creationdate, 'Post'

      FROM message

     WHERE 1=1 AND m_c_replyof IS NULL -- post, not comment

       AND m_creationdate BETWEEN :startDate AND :endDate

  UNION ALL

    SELECT psa.psa_threadid AS psa_threadid

         , psa.psa_thread_creatorid AS psa_thread_creatorid

         , m_messageid, m_creationdate, 'Comment'

      FROM message p, post_all psa

     WHERE 1=1 AND p.m_c_replyof = psa.psa_messageid

     AND m_creationdate BETWEEN :startDate AND :endDate

)

SELECT p.p_personid AS "person.id"

     , p.p_firstname AS "person.firstName"

     , p.p_lastname AS "person.lastName"

     , count(DISTINCT psa.psa_threadid) AS threadCount

END) AS messageCount

     , count(DISTINCT psa.psa_messageid) AS messageCount

  FROM person p left join post_all psa on (

       1=1   AND p.p_personid = psa.psa_thread_creatorid

   AND psa_creationdate BETWEEN :startDate AND :endDate

   )

 GROUP BY p.p_personid, p.p_firstname, p.p_lastname

 ORDER BY messageCount DESC, p.p_personid

 LIMIT 100;
Here is how to realize the same query using the Cypher graph query language:

--Cypher
MATCH (person:Person)<-[:HAS_CREATOR]-(post:Post)<-[:REPLY_OF*0..]-(reply:Message)
WHERE  post.creationDate >= $startDate   AND  post.creationDate <= $endDate
  AND reply.creationDate >= $startDate   AND reply.creationDate <= $endDate
person. RETURN
id,   person.firstName,   person.lastName,   count(DISTINCT post) AS threadCount,
  count(DISTINCT reply) AS messageCount
ORDER BY
  messageCount DESC,  person.id ASC
LIMIT 100

In addition, the graph ecosystem is diversified. The following is the graph technology landscape in 2020, and we can expect more graph related technologies coming along in 2021.

Nebula Graph

Now, let’s take Nebula Graph, a distributed graph database, as an example to talk about the evolution of graph technology. I will also share the challenges the team had faced when developing Nebula Graph and how we had solved them.

When we started designing the blueprint of Nebula Graph in late 2018, the team had set four goals for the database. They are scalability, production-ready, OLTP(Online Transaction Processing), and open source. The four goals are still influencing the roadmapping of Nebula Graph until today.

Scalability

Scalability is the No.1 design principle of Nebula Graph. This is because we believe the data that businesses will process in the future must be massive and there will be no way that single machines can handle them. That’s why we designed Nebula Graph in a way that it is capable of handling graph data with trillions of vertices and edges.

Production-ready

Nebula Graph is also designed to be production-ready on the first day, including the design of its query language, visualization, programmability, and DevOps.

OLTP

OLTP (online transactional processing) enables the real-time execution of large numbers of database transactions by large numbers of users, typically over the internet. One of the priorities of Nebula Graph’s design is OLTP. This makes Nebula Graph an online, high-concurrency, and low-latency graph database.

Open source

Nebula Graph is also devoted to building an open-source community and integrating with the big data world, supporting graph computing and training frameworks like Tencent Plato and Spark GraphX.

The Nebula Core

The above figure shows the ecosystem built around Nebula Graph. The section with red background is the Nebula Graph Core, which consists of three parts called Meta, Graph, and Storage.

Nebula Graph’s query language is our in-house nGQL, which is also compatible with openCypher. We have also developed clients in languages including Java, C++, Python, and Go. Then on the top we have a number of SDKs that can work with frameworks like Spark, Flink, GraphX, Tencent Plato.

Let’s dive into the Nebula core, which, as I mentioned above, consists of the meta, graph, and storage services. The meta service deals with metadata, while the storage service stores the data, and the graph service is in charge of querying. The three modules run on their independent processes, ensuring the separation of compute and storage.

The meta service manages the schema. Nebula Graph is not a schema-free database, and it requires the properties of vertices and edges to be pre-configured. The meta service also manages storage spaces, long-duration tasks, and data cleaning.

Nebula Graph can handle data with trillions of vertices and edges. This means the system must segment the data in storage and handling. Nebula Graph uses the segmentation of edges and stores vertices in partitions. Each partition may have a few replicas and run on different machines. The query engine is stateless, meaning that all query data should either be retrieved from the meta service or the storage service and there is no communication between query services.

The above is about Nebula Graph’s separation of compute and storage. Now let’s talk about data characteristics. We have mentioned that Nebula Graph is not a schema-free database and that all the data stored is pre-defined by Data Definition Languages (DDLs). We call types of vertices Tag and types of edges EdgeType. Vertices are defined using a 2-tuple consisting of the vid and the tag. Edges are defined using a 4-tuple consisting of the endpoints, EdgeType, and rank.

Nebula Graph supports primitive data types like boolean, int, and double as well as composite types like list, set, or graph data types like path and subgraph. If there is long string data stored in the database, it is usually indexed by Elasticsearch.

Here is some additional information about the storage engine. For the query engine graphd, the external interface exposed by the storage engine is a distributed graph service, but it can also be used as a distributed key-value (KV) service if necessary. In the storage engine, partitions adopt the Raft consensus protocol. Nebula Graph stores vertices and edges in separated partitions. The following figure is about how KV is implemented using the storage partitioning.

In Nebula Graph, each edge is stored as two pieces of data. As mentioned above, the storage layer relies on VID and guarantees strong consistency using the Raft protocol.

Nebula Graph uses ElasticSearch for full-text index. From Nebula Graph v2.x, our R&D team has optimized the write performance of Nebula’s indexing capability. Since v2.5.0, Nebula Graph has started to support the combination of data expiration TTL and indexing. And from v2.6.0, Nebula Graph started to support the TOSS (Transaction on storage side) function to achieve the eventual consistency of edges. That is to say, edges are either successfully written or failed at the same time when they are inserted or modified.

Nebula Graph uses its in-house nGQL as its query language. In June 2021, the International Organization for Standardization has drafted the standard for the syntax and semantics of GQL and there is a consensus between major graph database vendors.

From Nebula Graph v2.0, nGQL started to be compatible with openCypher, which was an open-source version of Neo4j’s Cypher query language. Now, nGQL supports the Doctrine Query Language (DQL) of openCypher and has developed its own vanilla nGQL syntax style in Data Manipulation Language (DML) and Data Definition Language (DDL).

We also mentioned that Nebula Graph is born to be production-ready. So it supports a wide range of operation features like data isolation, user permission and authentication, and replica configuration. Also, Nebula Graph supports clustering. Nebula Operator, which was released in April 2021, started to support Kubernetes.

As for the performance of Nebula Graph, most performance tests are carried out by users in the community, such as engineers from tech companies like Meituan, WeChat, 360 DigiTech, and WeBank, etc. The figures below are performance reports compiled by users.

As for the performance of Nebula Graph, most performance tests are carried out by users in the community, such as Meituan, WeChat, 360 DigiTech, WeBank. The figures below are performance reports compiled by users.

We have mentioned that one of the priorities of Nebula Graph’s design is OLTP. But that doesn’t mean it neglects analytical processing (AP). Nebula Graph has integrated AP frameworks like Apache Spark’s GraphX and Plato developed by Tencent.

Generally speaking, Nebula Graph performance has improved significantly when the deep traversal is conducted.

Nebula Graph meets users’ AP (Access Point) requirements of OLTP (Online Transactional Processing ) by docking with Spark’s Graph X and supports Plato, the graph computing engine of Tencent’s WeChat team. Plato docking is actually the data connection between the two engines, which needs to change the internal data format of Nebula Graph to be that of Plato and then Partition to map them one by one.

The Nebula Graph Community

Nebula Graph became open source in May 2019. Its v1.0 GA was released in June 2020, even though some companies had applied Nebula Graph in production before that. Nebula Graph first entered DB-Engines’ graph database management system ranking two years ago and now it ranks 15th on the list.

NOTE: The screenshot is the DB-Engine ranking in Apr. 2021 when this talk was given.

Nebula Graph is also one of the top open source players in China. This following report, which was published by the X-Lab of East China Normal University, ranks the companies according to the community popularity of their open source products. Vesoft Inc., the maker of Nebula Graph, ranks the eighth, before TikTok parent Bytedance, and just one position after Huawei.

Here are some of my thoughts about open source graph databases. Open source is very common in the graph database industry because it is a relatively new area and only got traction in recent years. This is why Nebula Graph chose to be open source from day one. Open source software can also attract more developers to use and gain valuable feedback from adopters.

Nebula Graph v3.0.0 Release Note

lisahui — Thu, 24 Feb 2022 09:58:52 +0000

Nebula Graph v3.0.0 is here! The new version introduced a series of new features including enhanced backup and restore, better support for the openCypher query language, and more fine-grained user management.

Features

Support backup and restore. https://github.com/vesoft-inc/nebula/pull/3469 https://github.com/vesoft-inc/nebula-agent/pull/1 https://github.com/vesoft-inc/nebula-br/pull/22
Support openCypher queries with multiple MATCH statements. https://github.com/vesoft-inc/nebula/pull/3519 https://github.com/vesoft-inc/nebula/pull/3318
Support Standalone Nebula Graph. https://github.com/vesoft-inc/nebula/pull/3310
Support key value separation for the storage engine. https://github.com/vesoft-inc/nebula/pull/3281
Support topN push down for LOOKUP. https://github.com/vesoft-inc/nebula/pull/3499
Support vertex without tag. https://github.com/vesoft-inc/nebula/pull/3316 https://github.com/vesoft-inc/nebula/pull/3335 https://github.com/vesoft-inc/nebula/pull/3328 https://github.com/vesoft-inc/nebula/pull/3286
Support parameterized queries. https://github.com/vesoft-inc/nebula/pull/3379
Support queries without specifying VIDs but a LIMIT clause must be used to restrict the number of results. https://github.com/vesoft-inc/nebula/pull/3320 https://github.com/vesoft-inc/nebula/pull/3329 https://github.com/vesoft-inc/nebula/pull/3262
Support duration. https://github.com/vesoft-inc/nebula/pull/3338
Support most UTF-8 encoded characters of 1 to 4 bytes in Schema. https://github.com/vesoft-inc/nebula/pull/3380 https://github.com/vesoft-inc/nebula/pull/3440
Support DESCRIBE USER. https://github.com/vesoft-inc/nebula/pull/3300

Optimizations

Refactor cluster management. https://github.com/vesoft-inc/nebula/pull/3343
Add log monitor to check free bytes of log disks, change log level when space is almost full. https://github.com/vesoft-inc/nebula/pull/3576
Support any string for tag names in apostrophe. https://github.com/vesoft-inc/nebula/pull/3424
Support that the storage service sends partition disk paths to the meta. https://github.com/vesoft-inc/nebula/pull/3369 https://github.com/vesoft-inc/nebula/pull/3416
Add constraints on invalid password attempts. https://github.com/vesoft-inc/nebula/pull/3573 https://github.com/vesoft-inc/nebula/pull/3629
Support DELETE in TOSS. https://github.com/vesoft-inc/nebula/pull/3374
Support to use logrotate. https://github.com/vesoft-inc/nebula/pull/3541
Support more metrics. https://github.com/vesoft-inc/nebula/pull/3446 https://github.com/vesoft-inc/nebula/pull/3605 https://github.com/vesoft-inc/nebula/pull/3590
Enhancement datetime parser. https://github.com/vesoft-inc/nebula/pull/3179
Remove read lock in meta service to reduce the side effect of read-write locks. https://github.com/vesoft-inc/nebula/pull/3256
Refactor storage indexes to solve the coupling problem between services. https://github.com/vesoft-inc/nebula/pull/3196
Support specifying the floating point accuracy of the round() function. https://github.com/vesoft-inc/nebula/pull/3178
Support https for ES client. https://github.com/vesoft-inc/nebula/pull/3150
Move version info outside of heartbeat. https://github.com/vesoft-inc/nebula/pull/3378
Support empty list, set, map. https://github.com/vesoft-inc/nebula/pull/3302
Support specifying s2 region coverage parameters when creating a geo index. https://github.com/vesoft-inc/nebula/pull/3396
Add version info for SHOW HOSTS. https://github.com/vesoft-inc/nebula/pull/3702

Bugfix

Fix the bug that memory isn’t released when a default value is used and no value is specified in nGQL. https://github.com/vesoft-inc/nebula/pull/3666
Fix the bug that the function coalesce() cannot be used. https://github.com/vesoft-inc/nebula/pull/3653
Fix the bug that using multiple INSERT on an indexed tag will lead to incorrect LOOKUP results. https://github.com/vesoft-inc/nebula/pull/3627
Fix the crash when the expression exceeds the depth. https://github.com/vesoft-inc/nebula/pull/3606
Disable the aggregate function in YIELD clause and WHERE clauses of nGQL. https://github.com/vesoft-inc/nebula/pull/3597
Fix the crash when using the aggregate function in UNWILD and WHERE clauses. https://github.com/vesoft-inc/nebula/pull/3397 https://github.com/vesoft-inc/nebula/pull/3355
Fix the bug that the tag index is rebuilt with an old schema version value. https://github.com/vesoft-inc/nebula/pull/3332
Fix the bug that the query results will still contain the expired edges if we use GO...REVERSELY. https://github.com/vesoft-inc/nebula/pull/3536
Fix the memory estimation error info in CentOS 6.0. https://github.com/vesoft-inc/nebula/pull/3534
Fix the crash when the LOOKUP statement contains a filter that consists of a logical And expression and an IN expression with only one element. https://github.com/vesoft-inc/nebula/pull/3525
Fix the bug that metad progress is suspended under high load. https://github.com/vesoft-inc/nebula/pull/3482
Fix the unwinding subgraph crash. https://github.com/vesoft-inc/nebula/pull/3506
Fix the DROP SPACE crash when rebuilding an index. https://github.com/vesoft-inc/nebula/pull/3406
Fix the bug of reading memory stats under cgroup v2. https://github.com/vesoft-inc/nebula/pull/3419
Fix the bug that DROP TAG INDEX deletes the edge index with same name unexpectedly, and vice versa for the deletion of the tag index. https://github.com/vesoft-inc/nebula/pull/3413
Fix the bug that edges are not shown after a graph space is cloned. https://github.com/vesoft-inc/nebula/pull/3351
Fix the index existence check problem. https://github.com/vesoft-inc/nebula/pull/3315
Fix a bug that running the ALTER statement to query the type property may lead to a null pointer obtained by the graph service. https://github.com/vesoft-inc/nebula/pull/3325
Optimize the raft to make the system more stable. https://github.com/vesoft-inc/nebula/pull/3172 https://github.com/vesoft-inc/nebula/pull/3435 https://github.com/vesoft-inc/nebula/pull/3358 https://github.com/vesoft-inc/nebula/pull/3322 https://github.com/vesoft-inc/nebula/pull/3031
Cancel memory check when the ratio is greater than 1.0. https://github.com/vesoft-inc/nebula/pull/3289
Fix building with ninja error. https://github.com/vesoft-inc/nebula/pull/3195
Fix the bug that creating tag and edge with same name at the same time may be both succeed. https://github.com/vesoft-inc/nebula/pull/3735
Fix the bug that failed to create full-text index for the same tag or edge internal id in different SPACE. https://github.com/vesoft-inc/nebula/pull/3747
Fix variable inconsistency in YIELD clause and GO statement. https://github.com/vesoft-inc/nebula/pull/3430
Fix the crash when schema version is greater than 256. https://github.com/vesoft-inc/nebula/pull/3893

Incompatibility

Nebula Graph 3.0.0 does not support most ecosystem tools of v2.x, please upgrade the ecosystem tools.

The storage services added in the configuration files cannot be read or written directly. The configuration files only register the storage services into the meta services. You must run the ADD HOSTS command to read and write data on storage servers. https://github.com/vesoft-inc/nebula/pull/3343
Disable ZONE and GROUP. https://github.com/vesoft-inc/nebula/pull/3776 https://github.com/vesoft-inc/nebula/pull/3825 https://github.com/vesoft-inc/nebula/pull/3330
Disable BALANCE DATA. https://github.com/vesoft-inc/nebula/pull/3756
Modify the default session timeout from 0 to 28800 seconds, limit the value between 1 and 604800 seconds. https://github.com/vesoft-inc/nebula/pull/3357 https://github.com/vesoft-inc/nebula/pull/3807
Add SHOW LOCAL SESSIONS and SHOW LOCAL QUERIES commands, and deprecate SHOW ALL QUERIES. https://github.com/vesoft-inc/nebula/pull/3488
A tag is not required for a vertex. DELETE VERTEX only deletes the vertices, and does not delete the related outgoing and incoming edges of the vertices. At this time, there will be dangling edges by default. https://github.com/vesoft-inc/nebula/pull/3316 https://github.com/vesoft-inc/nebula/pull/3335 https://github.com/vesoft-inc/nebula/pull/3328 https://github.com/vesoft-inc/nebula/pull/3286
Disable the YIELD clause to return custom variables. https://github.com/vesoft-inc/nebula/pull/3271
The YIELD clause is required in the FETCH, FIND PATH, LOOKUP, GET SUBGRAPH and GO statements. https://github.com/vesoft-inc/nebula/pull/3056 https://github.com/vesoft-inc/nebula/pull/3139 https://github.com/vesoft-inc/nebula/pull/2957
Add non-reserved keywords: s2_max_level, s2_max_cells. https://github.com/vesoft-inc/nebula/pull/3396
It is required to specify a tag to query properties of a vertex in a MATCH statement. For example, from return v.name to return v.player.name. https://github.com/vesoft-inc/nebula/pull/3255

Join our Slack channel if you want to discuss with the rest of the Nebula Graph community!

A method to compute the Betweenness Centrality against Nebula Graph

lisahui — Mon, 20 Dec 2021 03:48:36 +0000

Betweenness Centrality (BC for short) reflects the significance of vertices in the entire network. This article will introduce how to compute Betweenness Centrality against Nebula Graph.

Relevant Concepts

Centrality represents how central a vertex is in the entire network graph, including Degree Centrality, Closeness Centrality, and Betweenness Centrality, etc. Degree Centrality reflects the popularity of a vertex by counting the number of its incoming and outgoing edges, while Closeness Centrality computes the sum of the length of the shortest paths between a vertex and all other vertices in the graph. Thus, the more central a vertex is, the closer it is to all other vertices.

Betweenness Centrality counts the number of times a vertex appears on the shortest path between any two other vertices, so as to represent the significance of this vertex to the network connectivity.

The Betweenness Centrality of a vertex is the proportion of the number of paths passing through this vertex in all the shortest paths to the total number of shortest paths.

Computing the Betweenness Centrality of a vertex in a graph can be conducted in a weighted graph or in an unweighted graph. For unweighted graphs, Breadth-First Search (BFS for short) is used to find the shortest path, while for weighted graphs, Dijkstra’s algorithm is used.

The following algorithms are all targeted at undirected graphs.

Applicable Scenarios

Betweenness Centrality reflects the significance of vertices in the entire network by measuring how a vertex bridges all other vertices in a graph or network. As we can see, Vertex C in the following figure acts as an important bridging vertex.

Betweenness Centrality can be used to identify

a. The intermediary entities in anti-fraud scenarios in the field of financial risk control.

b. Specific disease control genes in the medical field to improve drug targets.

Betweenness Centrality Algorithm
The Betweenness Centrality of a vertex can be computed as follows:

C_B = \sum_{s{\not=} v {\not=} t \in V} \frac{\sigma_{st}(v)}{\sigma_{st}}

（Formula 1）

In this formula,

\sigma_{st}(v) is the number of shortest paths from Vertex s to Vertex t.

\sigma_{st} is the number of shortest paths that pass through Vertex v.

Vertex s and Vertex t are a pair of vertices belonging to the vertex set.

To make it more convenient, the betweenness of each pair of vertices can be computed as:
\delta_{st}(v) = \frac{\sigma_{st}(v)}{\sigma_{st}}

（Formula 2）

So Formula 1 can be replaced by Formula 2, which gives rise to Formula 3 as follows:

C_B(v) = \sum_{s{\not=} v {\not=} t \in V} \delta_{st}(v)

（Formula 3）

Solution Procedure

To get the Betweenness Centrality of Vertex v, namely to get $$\frac{, we need to know whether Vertex v lies on the path from Vertex s to Vertex t.

(1) To know whether Vertex v lies on the shortest path from Vertex s to Vertex t, use the following formula d_G represents the shortest path from Vertex s to Vertex t):

If Vertex v lies on the shortest path from Vertex s to Vertex t, then d_G(s,t) = d_G(s,v) + d_G(v,t)
is satisfied.

（Formula 4）

d_G(s,v) and d_G(v,t) are mutually independent. According to the rules of combination, the total number of shortest paths from Vertex s to Vertex t is the product of the number of shortest paths from Vertex s to Vertex t and the number of shortest paths from Vertex s to Vertex t.

Based on the above two situations, Formula 5 can be inferred:

\sigma_{st}(v) = \begin{cases} \sigma_{sv} \times \sigma_{vt} &\text{if } d(s,v) + d(v,t) = d(s,t) \ 0 &\text{if } other \end{cases}

（Formula 5）

（2）According to the above formula, we can get the conclusion that the number of shortest paths from Vertex s to Vertex t that pass through Vertex w is the result of \sigma_{st}(w) = \sigma_{sw} \times \sigma_{wt}

. In the graph, Vertex v is the preceding vertex of Vertex w. Therefore, the formula to count the number of shortest paths from Vertex s to Vertex t passing through Vertex v to Vertex w is:

\sigma_{st}(v,{v,w}) = \sigma_{sv} \times \sigma_{wt}

（Formula 6）

Here are two situations, t = w
and t \not= w

a. If t = w
, then \sigma_{wt}
will not exist and we can get

\delta(v,{v,w}) = \frac{\sigma_{sv}}{\sigma_{sw}}

（Formula 7）

b. If t \not= w
, then we can get
\delta(v,{v,w}) = \frac{\sigma_{sw}(v)}{\sigma_{sw}} \times \frac{\sigma_{st}(w)}{\sigma_{st}}

（Formula 8）

(3) So considering the above two situations, we can get

\delta_s(v) = \sum_{w:v \in P_s(w)}(\frac{\sigma_{sw}(v)}{\sigma_{sw}} + \sum_{t \not= w \in V}\frac{\sigma_{sw}(v)}{\sigma_{sw}} \times \frac{\sigma_{st}(w)}{\sigma_{st}}) \ = \sum_{w:v \in P_s(w)}\frac{\sigma_{sw}(v)}{\sigma_{sw}}(1 + \sum_{t \not= w \in V} \frac{\sigma_{st}(w)}{\sigma_{st}}) \ = \sum_{w:v \in P_s(w)}\frac{\sigma_{sw}(v)}{\sigma_{sw}} (1 + \delta_s(w))

（Formula 9）

In w:v \in P_s(w)
, Vertex v is the predecessor of Vertex w in the path from Vertex s to Vertex w.

(4) According to the above formula that gets the result of , the algorithm workflow of Betweenness Centrality in unweighted graphs can be given as follows:

For unweighted graphs, follow the above process.

To compute the Betweenness Centrality in weighted graphs requires Dijkstra’s algorithm, that is, to change the code in the first while loop.

The Betweenness Centrality against Nebula Graph has achieved the algorithms for both weighted and unweighted graphs. For the code, see https://github.com/vesoft-inc/nebula-algorithm/blob/master/nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/lib/BetweennessCentralityAlgo.scala.

Computation Examples

Firstly, read the graph data in Nebula Graph to specify the edge data for data reading.

Secondly, make a topological graph based on the edge data of Nebula Graph and perform centrality computation.

The graph data read in Nebula Graph can be illustrated in the following unweighted graph:

Compute the BC of Vertex 1:

The vertex pair with the shortest path passing through Vertex 1	The total number of shortest paths between the vertex pair	The number of the shortest path passing through Vertex 1
2-4	3 （2-3-4, 2-5-4, 2-1-4）	1
The BC of Vertex 1:	1/3

Compute the BC of Vertex 2:

The vertex pair with the shortest path passing through Vertex 2	The total number of shortest paths between the vertex pair	The number of the shortest path passing through Vertex 2
1-3	2 （1-2-3, 1-4-3）	1
3-5	2（3-2-5, 3-4-5）	1
The BC of Vertex 2:	1

Compute the BC of Vertex 3:

The vertex pair with the shortest path passing through Vertex 3	The total number of shortest paths between the vertex pair	The number of the shortest path passing through Vertex 3
2-4	3 （2-3-4, 2-5-4, 2-1-4）	1
The BC of Vertex 3:	1/3
Compute the BC of Vertex 4:	1

The vertex pair with the shortest path passing through Vertex 4	The total number of shortest paths between the vertex pair	The number of the shortest path passing through Vertex 4
1-3	2 （1-4-3, 1-2-3）	1
3-5	2（3-4-5, 3-2-5)	1
The BC of Vertex 4:	1

Compute the BC of Vertex 5:

The vertex pair with the shortest path passing through Vertex 5	The total number of shortest paths between the vertex pairs	The number of the shortest path passing through Vertex 5
2-4	3 （2-3-4, 2-5-4, 2-1-4）	1
The BC of Vertex 5:	1/3

Therefore, the BC of each vertex is: Vertex 1: 1/3 Vertex 2: 1 Vertex 3: 1/3 Vertex 4: 1 Vertex 5: 1/3

Result Examples

Data: Read the edge data in the Nebula Graph test, and take srcId, dstId, and rank as the triplet of edges in the topological graph (Source Vertex, Destination Vertex, and Rank).

(root@nebula) [test]> match (v:node) -[e:relation] -> () return e
+------------------------------------+
| e |
+------------------------------------+
| [:relation "3"->"4" @1 {col: "f"}] |
+------------------------------------+
| [:relation "2"->"3" @2 {col: "d"}] |
+------------------------------------+
| [:relation "2"->"5" @4 {col: "e"}] |
+------------------------------------+
| [:relation "4"->"5" @2 {col: "g"}] |
+------------------------------------+
| [:relation "1"->"5" @1 {col: "a"}] |
+------------------------------------+
| [:relation "1"->"2" @3 {col: "b"}] |
+------------------------------------+
| [:relation "1"->"4" @5 {col: "c"}] |
+------------------------------------+
Read the edge data in Nebula Graph, set the graph as unweighted, and compute the Betweenness Centrality of each vertex. The results are as follows:

vid: 4 BC: 1.0
vid: 1 BC: 0.3333333333333333
vid: 3 BC: 0.3333333333333333
vid: 5 BC: 0.3333333333333333
vid: 2 BC: 1.0
Read the edge data of Nebula Graph, set the graph as weighted, and compute the Betweenness Centrality of each vertex. The results are as follows:

vid: 4 BC: 2.0
vid: 1 BC: 0.5
vid: 3 BC: 1.0
vid: 5 BC: 2.0
vid: 2 BC: 0.0

References

Paper: A Faster Algorithm for Betweenness Centrality
The source code of Python’s NetworkX realizing Betweenness Centrality: https://github.com/networkx/networkx/blob/master/networkx/algorithms/centrality

Perform a Load Testing against Nebula Graph with K6

lisahui — Thu, 02 Dec 2021 04:08:19 +0000

Why Load Testing Matters in Nebula Graph?

The load testing for the database needs to be conducted usually so that the impact on the system can be monitored in different scenarios, such as query language rule optimization, storage engine parameter adjustment, etc.

The operating system in this article is the x86 CentOS 7.8.

The hosts where nebula is deployed are configured with 4C 16G memory, SSD disk, and 10G network.

Tools Needed for the Load Testing

nebula-ansible deploys Nebula Graph services.
nebula-importer imports data into Nebula Graph clusters.
k6-plugin is a K6 extension that is used to perform load testing against the Nebula Graph cluster. The extension integrates with the nebula-go client to send requests during the testing.
nebula-bench generates the LDBC dataset and then imports it into Nebula Graph.
ldbc_snb_datagen_hadoop is a LDBC data generator.

Load Testing Process Overview

The load testing conducted in this article uses the LDBC dataset generated by ldbc_snb_datagen. The testing process is as follows.

To deploy the topology, use one host as the load testing runner, and use three hosts to form a Nebula Graph cluster.

To make monitoring easier, the load testing runner also deploys:

Prometheus
Influxdb
Grafana
node-exporter
The hosts where Nebula Graph is installed also deploy:

node-exporter
process-exporter

Load Testing Steps

Use nebula-ansible to deploy Nebula Graph

Set up SSH login without passwords a. Log in 192.168.8.60, 192.168.8.61, 192.168.8.62, and 192.168.8.63 respectively. Create a vesoft user and join in sudoer with NOPASSWD. b. Log in 192.168.8.60 to set up SSH.

ssh-keygen

ssh-copy-id vesoft@192.168.8.61
ssh-copy-id vesoft@192.168.8.62
ssh-copy-id vesoft@192.168.8.63

Download nebula-ansible, install Ansible, and modify the Ansible configuration.

sudo yum install ansible -y
git clone https://github.com/vesoft-inc/nebula-ansible
cd nebula-ansible/

The following is an example of inventory.ini.

[all:vars]
# GA or nightly
install_source_type = GA
nebula_version = 2.0.1
os_version = el7
arc = x86_64
pkg = rpm

packages_dir = {{ playbook_dir }}/packages
deploy_dir = /home/vesoft/nebula
data_dir = {{ deploy_dir }}/data

# ssh user
ansible_ssh_user = vesoft

force_download = False

[metad]
192.168.8.[61:63]

[graphd]
192.168.8.[61:63]

[storaged]
192.168.8.[61:63]

Install and deploy Nebula Graph.

ansible-playbook install.yml
ansible-playbook start.yml

Monitor hosts

Using docker-compose to deploy a monitoring system is convenient. Docker and Docker-Compose need to be installed on the hosts first.

git clone https://github.com/vesoft-inc/nebula-bench.git

cd nebula-bench
cp -r third/promethues ~/.
cp -r third/exporter ~/.



cd ~/exporter/ && docker-compose up -d

cd ~/promethues
# Modify the exporter address of monitoring nodes
# vi prometheus.yml
docker-compose up -d

# Copy exporter to 192.168.8.61, 192.168.8.62, and 192.168.8.63, and then start docker-compose

Configure the Grafana data source and dashboard. For details, see https://github.com/vesoft-inc/nebula-bench/tree/master/third.

Generate the LDBC dataset

cd nebula-bench

sudo yum install -y git \
                    make \
                    file \
                    libev \
                    libev-devel \
                    gcc \
                    wget \
                    python3 \
                    python3-devel \
                    java-1.8.0-openjdk \
                    maven

pip3 install --user -r requirements.txt

# Using `snb.interactive.1` parameter in ldbc_snb_datagen_hadoop, for more infor https://github.com/ldbc/ldbc_snb_datagen_hadoop/wiki/Configuration

python3 run.py data

# Date generated by mv

mv target/data/test_data/ ./sf1

Import data

cd nebula-bench
# Modify .evn
cp env .env
vi .env

The following is the example of .env

DATA_FOLDER=sf1
NEBULA_SPACE=sf1
NEBULA_USER=root
NEBULA_PASSWORD=nebula
NEBULA_ADDRESS=192.168.8.61:9669,192.168.8.62:9669,192.168.8.63:9669
#NEBULA_MAX_CONNECTION=100
INFLUXDB_URL=http://192.168.8.60:8086/k6

# Compile nebula-importer and K6
./scripts/setup.sh

# Import data
python3 run.py nebula importer

During the import process, you can focus on the following network bandwidth and disk IO writing.

Execute the load testing

python3 run.py stress run

According to the code source in the file scenarios, the js file will be automatically rendered and K6 will be used to test all scenarios.

After the execution is over, the js file and the result will be saved in the output folder.

Among them, latency is the latency time returned by the server, and responseTime is the time from initiating execute to response by the client. The measurement unit is μs.

[vesoft@qa-60 nebula-bench]$ more output/result_Go1Step.json
{
    "metrics": {
        "data_sent": {
            "count": 0,
            "rate": 0
        },
        "checks": {
            "passes": 1667632,
            "fails": 0,
            "value": 1
        },
        "data_received": {
            "count": 0,
            "rate": 0
        },
        "iteration_duration": {
            "min": 0.610039,
            "avg": 3.589942336582023,
            "med": 2.9560145,
            "max": 1004.232905,
            "p(90)": 6.351617299999998,
            "p(95)": 7.997563949999995,
            "p(99)": 12.121579809999997
        },
        "latency": {
            "min": 308,
            "avg": 2266.528722763775,
            "med": 1867,
            "p(90)": 3980,
            "p(95)": 5060,
            "p(99)": 7999
        },
        "responseTime": {
            "max": 94030,
            "p(90)": 6177,
            "p(95)": 7778,
            "p(99)": 11616,
            "min": 502,
            "avg": 3437.376111156418,
            "med": 2831
        },
        "iterations": {
            "count": 1667632,
            "rate": 27331.94978169588
        },
        "vus": {
            "max": 100,
            "value": 100,
            "min": 0

[vesoft@qa-60 nebula-bench]$ head -300 output/output_Go1Step.csv | grep -v USE
timestamp,nGQL,latency,responseTime,isSucceed,rows,errorMsg
1628147822,GO 1 STEP FROM 4398046516514 OVER KNOWS,1217,1536,true,1,
1628147822,GO 1 STEP FROM 2199023262994 OVER KNOWS,1388,1829,true,94,
1628147822,GO 1 STEP FROM 1129 OVER KNOWS,1488,2875,true,14,
1628147822,GO 1 STEP FROM 6597069771578 OVER KNOWS,1139,1647,true,30,
1628147822,GO 1 STEP FROM 2199023261211 OVER KNOWS,1399,2096,true,6,
1628147822,GO 1 STEP FROM 2199023256684 OVER KNOWS,1377,2202,true,4,
1628147822,GO 1 STEP FROM 4398046515995 OVER KNOWS,1487,2017,true,39,
1628147822,GO 1 STEP FROM 10995116278700 OVER KNOWS,837,1381,true,3,
1628147822,GO 1 STEP FROM 933 OVER KNOWS,1130,3422,true,5,
1628147822,GO 1 STEP FROM 6597069771971 OVER KNOWS,1022,2292,true,60,
1628147822,GO 1 STEP FROM 10995116279952 OVER KNOWS,1221,1758,true,3,
1628147822,GO 1 STEP FROM 8796093031179 OVER KNOWS,1252,1811,true,13,
1628147822,GO 1 STEP FROM 10995116279792 OVER KNOWS,1115,1858,true,6,
1628147822,GO 1 STEP FROM 6597069777326 OVER KNOWS,1223,2016,true,4,
1628147822,GO 1 STEP FROM 8796093028089 OVER KNOWS,1361,2054,true,13,
1628147822,GO 1 STEP FROM 6597069777454 OVER KNOWS,1219,2116,true,2,
1628147822,GO 1 STEP FROM 13194139536109 OVER KNOWS,1027,1604,true,2,
1628147822,GO 1 STEP FROM 10027 OVER KNOWS,2212,3016,true,83,
1628147822,GO 1 STEP FROM 13194139544176 OVER KNOWS,855,1478,true,29,
1628147822,GO 1 STEP FROM 10995116280047 OVER KNOWS,1874,2211,true,12,
1628147822,GO 1 STEP FROM 15393162797860 OVER KNOWS,714,1684,true,5,
1628147822,GO 1 STEP FROM 6597069770517 OVER KNOWS,2295,3056,true,7,
1628147822,GO 1 STEP FROM 17592186050570 OVER KNOWS,768,1630,true,26,
1628147822,GO 1 STEP FROM 8853 OVER KNOWS,2773,3509,true,14,
1628147822,GO 1 STEP FROM 19791209307908 OVER KNOWS,1022,1556,true,6,
1628147822,GO 1 STEP FROM 13194139544258 OVER KNOWS,1542,2309,true,91,
1628147822,GO 1 STEP FROM 10995116285325 OVER KNOWS,1901,2556,true,0,
1628147822,GO 1 STEP FROM 6597069774931 OVER KNOWS,2040,3291,true,152,
1628147822,GO 1 STEP FROM 8796093025056 OVER KNOWS,2007,2728,true,29,
1628147822,GO 1 STEP FROM 21990232560726 OVER KNOWS,1639,2364,true,9,
1628147822,GO 1 STEP FROM 8796093030318 OVER KNOWS,2145,2851,true,6,
1628147822,GO 1 STEP FROM 21990232556027 OVER KNOWS,1784,2554,true,5,
1628147822,GO 1 STEP FROM 15393162796879 OVER KNOWS,2621,3184,true,71,
1628147822,GO 1 STEP FROM 17592186051113 OVER KNOWS,2052,2990,true,5,

It is also possible to execute the load testing in one scenario and continuously adjust the configuration parameters for comparison.

Concurrent reading

#Run Go2Step with 50 virtual users and 300 seconds of duration
python3 run.py stress run -scenario go.Go2Step -vu 50 -d 300

INFO[0302] 2021/08/06 03:55:27 [INFO] finish init the pool

     ✓ IsSucceed

     █ setup

     █ teardown

     checks...............: 100.00% ✓ 1559930     ✗ 0
     data_received........: 0 B     0 B/s
     data_sent............: 0 B     0 B/s
     iteration_duration...: min=687.47µs avg=9.6ms       med=8.04ms max=1.03s  p(90)=18.41ms p(95)=22.58ms p(99)=31.87ms
     iterations...........: 1559930 5181.432199/s
     latency..............: min=398      avg=6847.850345 med=5736   max=222542 p(90)=13046   p(95)=16217   p(99)=23448
     responseTime.........: min=603      avg=9460.857877 med=7904   max=226992 p(90)=18262   p(95)=22429   p(99)=31726.71
     vus..................: 50      min=0         max=50
     vus_max..............: 50      min=50        max=50

Every metric can be monitored at the same time.

checks is to verify whether the request is executed successfully. If the execution fails, the failed message will be saved in the CSV file.

awk -F ',' '{print $NF}' output/output_Go2Step.csv|sort |uniq -c

# Execute Go2Step with 200 virtual users and 300 seconds of duration
python3 run.py stress run -scenario go.Go2Step -vu 200 -d 300

INFO[0302] 2021/08/06 04:02:34 [INFO] finish init the pool

     ✓ IsSucceed

     █ setup

     █ teardown

     checks...............: 100.00% ✓ 1866850    ✗ 0
     data_received........: 0 B     0 B/s
     data_sent............: 0 B     0 B/s
     iteration_duration...: min=724.77µs avg=32.12ms      med=25.56ms max=1.03s  p(90)=63.07ms p(95)=84.52ms  p(99)=123.92ms
     iterations...........: 1866850 6200.23481/s
     latency..............: min=395      avg=25280.893558 med=20411   max=312781 p(90)=48673   p(95)=64758    p(99)=97993.53
     responseTime.........: min=627      avg=31970.234329 med=25400   max=340299 p(90)=62907   p(95)=84361.55 p(99)=123750
     vus..................: 200     min=0        max=200
     vus_max..............: 200     min=200      max=200

K6 metrics to be monitored with Grafana
![Concurrent reading](https://user-images.githubusercontent.com/90186547/143537954-780fade2-ae2a-4882-a33e-3df47ad68402.png）

Concurrent writing

#Execute insert with 200 virtual users and 300 seconds of duration. By default, batchSize is 100.

python3 run.py stress run -scenario go.Go2Step -vu 200 -d 300

The js file can be modified manually to adjust batchSize

sed -i 's/batchSize = 100/batchSize = 300/g' output/InsertPersonScenario.js

# Run K6 manually

scripts/k6 run output/InsertPersonScenario.js -u 400 -d 30s --summary-trend-stats "min,avg,med,max,p(90),p(95),p(99)" --summary-export output/result_InsertPersonScenario.json --out influxdb=http://192.168.8.60:8086/k6

If the batchSize is 300 with 400 virtual users, an error will be returned.

INFO[0032] 2021/08/06 04:03:49 [INFO] finish init the pool

     ✗ IsSucceed
      ↳  96% — ✓ 31257 / ✗ 1103

     █ setup

     █ teardown

     checks...............: 96.59% ✓ 31257       ✗ 1103
     data_received........: 0 B    0 B/s
     data_sent............: 0 B    0 B/s
     iteration_duration...: min=12.56ms avg=360.11ms      med=319.12ms max=2.07s   p(90)=590.31ms p(95)=696.69ms p(99)=958.32ms
     iterations...........: 32360  1028.339207/s
     latency..............: min=4642    avg=206931.543016 med=206162   max=915671  p(90)=320397.4 p(95)=355798.7 p(99)=459521.39
     responseTime.........: min=6272    avg=250383.122188 med=239297.5 max=1497159 p(90)=384190.5 p(95)=443439.6 p(99)=631460.92
     vus..................: 400    min=0         max=400
     vus_max..............: 400    min=400       max=400

awk -F ',' '{print $NF}' output/output_InsertPersonScenario.csv|sort |uniq -c

 31660
   1103  error: E_CONSENSUS_ERROR(-16)."
      1 errorMsg

If E_CONSENSUS_ERROR occurs, it should be that the appendlog buffer of raft is overflown when the concurrency is large, which can be solved by adjusting relevant parameters.

Summary

The load testing uses the LDBC dataset standard to ensure data uniform. Even when bigger data volume, say one billion vertices, is generated, the graph schema is the same.
K6 is more convenient than Jmeter for the load testing. For more details, please refer https://k6.io/docs/.
You can easily find the bottleneck of the system resources by simulating various scenarios or adjust parameters in Nebula Graph with the mentioned tools.

TOSS: 1 Secret to Achieve Eventual Consistency of Edges in Nebula Graph

lisahui — Thu, 11 Nov 2021 07:35:27 +0000

Nebula Graph has just released v.2.6. In this version, TOSS is certainly one of the important features. Here a detailed explanation about TOSS will be given.

Let’s start from a GO statement

As we all know, there are two types of edges, directed and undirected edges. When traversing directed edges, you can traverse forward or reversely. Nebula Graph also supports this semantics. For example:

go from "101" over known reversely yield known.kdate, id($$);

The above statement starts from Vertex 101 to find all the corresponding neighbors reversely. However, when you insert an edge into Nebula Graph, the command will be like:

insert edge known(degree) VALUES "100" -> "101":(299792458);

Seemingly, the above statement only specifies the outgoing edge. This is because Nebula Graph will specify the incoming edge in the background when you insert an edge.

How to insert an edge into Nebula Graph

Take the INSERT statement above as an example, the background execution process contains the following:

Nebula Console sends the INSERT request to the Nebula Graph server.

After the Nebula Graph server receives the request, it adds an incoming edge for each outgoing edge and sends AddEdgeRequest to their hosts respectively.

After the host (Nebula Storage server) receives AddEdgeRequest, it inserts the edge locally (via the Raft protocol) and returns the result to the Nebula Graph server.

The Nebula Graph server then returns the results from both hosts to the Nebula Console for querying purpose.

The flow diagram is as follows:

If you are familiar with network/distributed programming, you may see the problem now. The graph service uses RPC to call both storage services. When the INSERT operation is executed enough times, one RPC succeeds while the other fails due to timeout. In other words, an INSERT operation may succeed on the outgoing edge while fail on the incoming-edge.

If now a user requires consistent property settings for both outgoing edge and incoming edge, the client has to retry the query infinitely. It is inappropriate for Nebula Graph, s a database product, to rely on the client for data atomicity.

A requirement thus comes into being, that is, to ensure the atomicity of outgoing edge and incoming edge. This means that the outgoing edge and incoming edge should be updated either successfully or they should fail at the same time. And TOSS (Transaction On Storage Side) is the feature to ensure the eventual consistency of edges upon INSERT, UPDATE, and UPSERT operations.

How to use TOSS

With the release of Nebula Graph v2.6.0, the TOSS function has also been launched. The feature is set to Disabled by default due to performance and stability considerations. You can find the enable_experimental_feature option in the Nebula Graph server configuration file and set it to True. Then you need to restart the graphd service for the feature to take effect. The command is as follows:

--enable_experimental_feature=true

Then the operations CREATE SPACE / CREATE EDGE / INSERT / UPDATE will achieve eventual consistency of edges in Nebula Graph. (Just execute the operations as before)

Note: The TOSS feature will be only applied to incremental data after it is enabled.

1 Article tells how Nebula Clients work with fbthrift

lisahui — Wed, 03 Nov 2021 03:44:41 +0000

Overview

Nebula Clients provide users with APIs in multiple programming languages to interact with Nebula Graph and repackages the data structure returned by the server for better use.

Currently, Nebula Clients support C++, Java, Python, Golang, and Rust.

Framework for service communication

Nebula Clients use fbthrift https://github.com/facebook/fbthrift as the RPC framework for service communication between servers and clients to implement cross-language interaction.

At a high level, fbthrift is:

A code generator: fbthrift has a code generator that generates data structures that can be serialized using Thrift in different languages.
A serialization framework: fbthrift has a set of protocols to serialize the generated structures created from the code generator.
An RPC framework: fbthrift has a framework to send messages between clients and servers and to call application-defined functions when receiving messages in different languages.

Examples

Take the Golang client as an example to show the application of fbthrift in Nebula Graph.

The definition of the Vertex structure in servers:

struct Vertex {
    Value vid;
    std::vector<Tag> tags;

    Vertex() = default;
};

Define some data structures in src/interface/common.thrift:

struct Tag {
        1: binary name,
        // List of <prop_name, prop_value>
        2: map<binary, Value> (cpp.template = "std::unordered_map") props,
} (cpp.type = "nebula::Tag")

struct Vertex {
        1: Value     vid,
        2: list<Tag> tags,
} (cpp.type = "nebula::Vertex")

In the above example, we define a Vertex structure. (cpp.type = "nebula::Vertex") indicates this structure corresponds to the nebula::Vertex of the server.

fbthrift will automatically generate the data structure in Golang:

// Attributes:
//  - Vid
//  - Tags
type Vertex struct {
    Vid *Value `thrift:"vid,1" db:"vid" json:"vid"`
    Tags []*Tag `thrift:"tags,2" db:"tags" json:"tags"`
}

func NewVertex() *Vertex {
    return &Vertex{}
}

...

func (p *Vertex) Read(iprot thrift.Protocol) error { // Deserialization
    ...
}

func (p *Vertex) Write(oprot thrift.Protocol) error { // Serialization 
    ...
}

In MATCH (v:Person) WHERE id(v) == "ABC" RETURN v, the client requests a vertex (nebula::Vertex) from the server. The server will serialize it after finding it. After the server finds this vertex, it will be serialized and sent to the client through the transport of the RPC communication framework. When the client receives this data, it will be deserialized to generate the corresponding data structure (type Vertex struct) defined in the client.

Clients

In this section, we will take nebula-go as an example to introduce different modules of the client and their main interfaces.

Configs provides the whole configuration options.

type PoolConfig struct {
    // Set the timeout threshold. The default value 0 means it does not time out. Unit: ms
    TimeOut time.Duration
    // The maximum idle time of each connection. When the idle time exceeds this threshold, the connection will be disconnected and deleted. The default value 0 means permanently idle and the connection will not be disconnected
    IdleTime time.Duration
    // max_connection_pool_size: Set the maximum number of connections in the connection pool. The default value is 10
    MaxConnPoolSize int
    // The minimum number of idle connections. The default value is 0
    MinConnPoolSize int
}

Session provides an interface for users to call directly.

// Manage the specific information of Session
type Session struct {
    // Use for identity verification or message retry when executing commands
    sessionID  int64
    // Currently held connections
    connection *connection
    // Currently used connection pools
    connPool   *ConnectionPool
    // Log tools
    log        Logger
    // Use to save the time zone used by the current session
    timezoneInfo
}

The definition of interfaces is as follows:

// Execute nGQL. The return data type is ResultSet. This interface is non-thread-safe
    func (session *Session) Execute(stmt string) (*ResultSet, error) {...}
    // Re-acquire a connection from the connection pool for the current Session
    func (session *Session) reConnect() error {...}
    // Signout, release the Session ID, and return the connection to the pool
    func (session *Session) Release() {

ConnectionPool manages all connections. The main interfaces are as follows:

// Create a new connection pool and complete the initialization with the entered service address
func NewConnectionPool(addresses []HostAddress, conf PoolConfig, log Logger) (*ConnectionPool, error) {...}
// Validate and get the Session example
func (pool *ConnectionPool) GetSession(username, password string) (*Session, error) {...}

Connection packages the network of thrift and provides the following interfaces:

// Establish a connection with the specified ip and port
func (cn *connection) open(hostAddress HostAddress, timeout time.Duration) error {...}
// Authenticate the username and password
func (cn *connection) authenticate(username, password string) (*graph.AuthResponse, error) {
// Execute query
func (cn *connection) execute(sessionID int64, stmt string) (*graph.ExecutionResponse, error) {...}
// Generate a temp sessionID 0 and send the query "YIELD 1" to test if the connection is usable.
func (cn *connection) ping() bool {...}
// Release sessionId to the graphd process.
func (cn *connection) signOut(sessionID int64) error {...}
// Disconnect.
func (cn *connection) close() {...}

LoadBalance is used in the connection pool.
Policy: Polling

Interaction of modules

Connection pool

Initialize:

When using it, the user needs to create and initialize a connection pool. During initialization, the connection pool will establish a connection at the address of the Nebula service specified by the user. If multiple Graph services are deployed in a cluster deployment method, the connection pool will use a polling policy to balance the load and establish a nearly equal number of connections for each address.
Manage connections:
Two queues are maintained in the connection pool, idle connection queue and active Connection Queue. The connection pool will periodically detect expired idle connections and close them. These two queues will use read-write lock to ensure the correctness of multi-thread execution when adding or deleting elements.
When Session requests a connection to the connection pool, it will check whether there are usable connections in the idle connection queue. If there are any usable connections, they will be directly returned to the Session for users to use. If there are no usable connections and the current total number of connections does not exceed the maximum number of connections defined in the configuration, a new connection is created to the Session. If it reaches the maximum number of connections, an error is returned.
Generally, the connection pool needs to be closed only when you close the program. All connections in the pool will be disconnected when the program is closed.

Session

Session is generated through the connection pool. The user needs to provide the password for authentication. After the authentication succeeds, the user will get a Session example and communicate with the server through the connection in the Session. The most commonly used interface is execute(). If an error occurs during execution, the client will check the error type. If it is a network error, it will automatically reconnect and try to execute the statement again.
Note that a Session does not support being used by multiple threads at the same time. The correct way is that multiple sessions are applied by multiple threads, and one session is used by each thread.
When the Session is released, the connection held by it will be put back into the idle connection queue of the connection pool so that it can be reused by other sessions later.

Connection

Each connection example is equivalent and can be held by any Session. The purpose of this design is to allow these connections to be reused by different Sessions, reducing repeatedly enabling and disabling Transport.
The connection will send the client’s request to the server and return the result to the Session.

Example

// Initialize connection pool
pool, err := nebula.NewConnectionPool(hostList, testPoolConfig, log)
if err != nil {
    log.Fatal(fmt.Sprintf("Fail to initialize the connection pool, host: %s, port: %d, %s", address, port, err.Error()))
}
// Close all connections in the pool when program exits
defer pool.Close()

// Create session
session, err := pool.GetSession(username, password)
if err != nil {
    log.Fatal(fmt.Sprintf("Fail to create a new session from connection pool, username: %s, password: %s, %s",
        username, password, err.Error()))
}
// Release session and return connection back to connection pool when program exits
defer session.Release()

// Excute a query
resultSet, err := session.Execute(query)
if err != nil {
    fmt.Print(err.Error())
}

Returned data structure

The client packages the returned query results by part of the complex servers and adds an interface for convenience use.

nebula::Value will be packaged as ValueWrapper in the client and converted to other structures through interfaces. (i.g. node = ValueWrapper.asNode())

Analysis of data structure

For MATCH p= (v:player{name:"Tim Duncan"})-[]->(v2) RETURN p, the returned result is:

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| p                                                                                                                                                                                                                         |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| <("Tim Duncan" :bachelor{name: "Tim Duncan", speciality: "psychology"} :player{age: 42, name: "Tim Duncan"})<-[:teammate@0 {end_year: 2016, start_year: 2002}]-("Manu Ginobili" :player{age: 41, name: "Manu Ginobili"})> |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Got 1 rows (time spent 11550/12009 us)

We can see that the returned result contains one row, and its type is a path. At this time, you can execute as follows to get the properties of the destination vertex of the path (v2).

// Excute a query
resultSet, _ := session.Execute("MATCH p= (v:player{name:"\"Tim Duncan"\"})-[]->(v2) RETURN p")

// Get the first row of the result. The index of the first row is 0
record, err := resultSet.GetRowValuesByIndex(0)
if err != nil {
    t.Fatalf(err.Error())
}

// Take the value of the cell in the first column from the first row
// At this time, the type of valInCol0 is ValueWrapper
valInCol0, err := record.GetValueByIndex(0)

// Convert ValueWrapper into PathWrapper objects.
pathWrap, err = valInCol0.AsPath()

// Get the destination vertex through pathWrap.GetEndNode()
node, err = pathWrap.GetEndNode()

// Get all properties through node.Properties()
// The type of props is map[string]*ValueWrapper
props, err = node.Properties()

Address of clients

The GitHub addresses of clients are as follows:

https://github.com/vesoft-inc/nebula-cpp
https://github.com/vesoft-inc/nebula-java
https://github.com/vesoft-inc/nebula-python
https://github.com/vesoft-inc/nebula-go
https://github.com/vesoft-inc/nebula-rust

Nebula Explorer: A Tool to Visualize Graph Data Easily

lisahui — Thu, 28 Oct 2021 06:11:10 +0000

Nebula Explorer is a visualization tool of the Nebula Graph ecosystem. With it, you can easily access Nebula Graph, and then query and retrieve graph data via GUI without having to learn nGQL. It can display graph data as a graph on a canvas for you to visually analyze data.

Querying Graph Data

Nebula Explorer provides various methods for querying graph data. For example, you can specify a VID list or tags (with their indexes) to query vertices, or specify one or more VIDs to query a subgraph. When the queried vertices are retrieved, you can select one or more for further exploration. For example, you can query for the shared neighbors and the related paths of each pair of vertices. Through progressive queries, you can improve and enrich the data on the canvas for further graph analysis.

Displaying Graph Data

When the graph data is retrieved, you can manipulate it on the canvas flexibly, including drag-and-drop, zoom-in or zoom-out, click-and-drag, and marking vertices with colors or icons. Nebula Explorer enables you to have fun with the graph data model by providing various features.

Roadmap

Besides querying and exploring data, Nebula Explorer will support graph computing in the future, which will enable the Nebula Graph users to use appropriate methods to query and analyze graph data in various business scenarios.

Currently, Nebula Explorer is only available for Enterprise users. If you are interested, please send an email with “Require Dashboard” as the subject to inquiry@vesoft.com.

1 way tells how an execution plan is generated

lisahui — Fri, 22 Oct 2021 08:16:00 +0000

In the last article, we mentioned that Validator will convert an AST generated by Parser to an execution plan. In this article, we will explain how an execution plan is generated.

Overview

Planner is an execution plan generator. It generates an execution plan based on the semantically valid AST that was validated by Validator, and then passes the plan to Optimizer to generate an optimized execution plan. Finally, Executor will execute the optimized plan. An execution plan is composed of a series of nodes (PlanNode).

Structure of Source Files

Here is the structure of source files for Planner.

src/planner
├── CMakeLists.txt
├── match/
├── ngql/
├── plan/
├── Planner.cpp
├── Planner.h
├── PlannersRegister.cpp
├── PlannersRegister.h
├── SequentialPlanner.cpp
├── SequentialPlanner.h
└── test

The Planner.h file defines the data structure of SubPlan and the interfaces of Planner.

struct SubPlan {
    // root and tail of a subplan.
    PlanNode*   root{nullptr};
    PlanNode*   tail{nullptr};
};

PlannersRegister is responsible for registering available planners. So far, SequentialPlanner, PathPlanner, LookupPlanner, GoPlanner, and MatchPlanner have been registered for Nebula Graph.

The corresponding sentence of SequentialPlanner is SequentialSentences, which is a combined sentence composed of multiple sentences separated with semicolons. Each sentence can be a GO, LOOKUP, or MATCH statement. Therefore, SequentialPlanner generates multiple execution plans by calling other sentence planners and then calling Validator::appendPlan to connect the plans end to end.

The match/ directory defines the planners and connection strategies of SubPlans of some statements and clauses compatible with openCypher, such as MATCH, UNWIND, WITH, RETURN, WHERE, ORDER BY, SKIP, and LIMIT. SegmentsConnector uses an appropriate strategy, such as AddInput, addDependency, or innerJoinSegments, to connect the SubPlans end to end to generate a complete execution plan.

src/planner/match
├── AddDependencyStrategy.cpp
├── AddDependencyStrategy.h
├── AddInputStrategy.cpp
├── AddInputStrategy.h
├── CartesianProductStrategy.cpp
├── CartesianProductStrategy.h
├── CypherClausePlanner.h
├── EdgeIndexSeek.h
├── Expand.cpp
├── Expand.h
├── InnerJoinStrategy.cpp
├── InnerJoinStrategy.h
├── LabelIndexSeek.cpp
├── LabelIndexSeek.h
├── LeftOuterJoinStrategy.h
├── MatchClausePlanner.cpp
├── MatchClausePlanner.h
├── MatchPlanner.cpp
├── MatchPlanner.h
├── MatchSolver.cpp
├── MatchSolver.h
├── OrderByClausePlanner.cpp
├── OrderByClausePlanner.h
├── PaginationPlanner.cpp
├── PaginationPlanner.h
├── PropIndexSeek.cpp
├── PropIndexSeek.h
├── ReturnClausePlanner.cpp
├── ReturnClausePlanner.h
├── SegmentsConnector.cpp
├── SegmentsConnector.h
├── SegmentsConnectStrategy.h
├── StartVidFinder.cpp
├── StartVidFinder.h
├── UnionStrategy.h
├── UnwindClausePlanner.cpp
├── UnwindClausePlanner.h
├── VertexIdSeek.cpp
├── VertexIdSeek.h
├── WhereClausePlanner.cpp
├── WhereClausePlanner.h
├── WithClausePlanner.cpp
├── WithClausePlanner.h
├── YieldClausePlanner.cpp
└── YieldClausePlanner.h

The ngql/ directory defines the planners of nGQL statements such as GO, LOOKUP, and FIND PATH.

src/planner/ngql
├── GoPlanner.cpp
├── GoPlanner.h
├── LookupPlanner.cpp
├── LookupPlanner.h
├── PathPlanner.cpp
└── PathPlanner.h

The plan/ directory defines seven categories, with a total of more than 100 plan nodes.

src/planner/plan
├── Admin.cpp
├── Admin.h
├── Algo.cpp
├── Algo.h
├── ExecutionPlan.cpp
├── ExecutionPlan.h
├── Logic.cpp
├── Logic.h
├── Maintain.cpp
├── Maintain.h
├── Mutate.cpp
├── Mutate.h
├── PlanNode.cpp
├── PlanNode.h
├── Query.cpp
├── Query.h
└── Scan.h

Here is an introduction to the purpose of plan nodes:

Admin: For the nodes related to database administration.
Algo: For the nodes related to the algorithms of paths, subgraphs, and so on.
Logic: For the nodes related to logic controlling, such as loop and binary selection.
Maintain: For the nodes related to schema.
Mutate: For the nodes related to DML.
Query: For the nodes related to query computation.
Scan: For the nodes related to indexing and scanning.
In the Executor phase, each PlanNode generates an executor, and each executor is responsible for a specific functionality.

For example, here is the source code of the GetNeighbors node.

static GetNeighbors* make(QueryContext* qctx,
                              PlanNode* input,
                              GraphSpaceID space,
                              Expression* src,
                              std::vector<EdgeType> edgeTypes,
                              Direction edgeDirection,
                              std::unique_ptr<std::vector<VertexProp>>&& vertexProps,
                              std::unique_ptr<std::vector<EdgeProp>>&& edgeProps,
                              std::unique_ptr<std::vector<StatProp>>&& statProps,
                              std::unique_ptr<std::vector<Expr>>&& exprs,
                              bool dedup = false,
                              bool random = false,
                              std::vector<storage::cpp2::OrderBy> orderBy = {},
                              int64_t limit = -1,
                              std::string filter = "")

GetNeighbors is the semantic encapsulation of the KV of an edge in the storage layer. Based on the source vertex of the given edge type, it will find the destination vertex of an edge. During the finding edge course, GetNeighbors can retrieve the properties of the edge (edgeProps). Additionally, the outgoing edge is stored with its source vertex in one partition (shard), so the properties of the source vertex (vertexProps) can be retrieved easily.

Here is the source code of the Aggregate node.

static Aggregate* make(QueryContext* qctx,
                               PlanNode* input, 
                               std::vector<Expression*>&& groupKeys = {},
                               std::vector<Expression*>&& groupItems = {})

The Aggregate node is for aggregate computing. It groups the table according to groupKeys, and does aggregate calculation on groupItems.

Here is the source code of the Loop node.

static Loop* make(QueryContext* qctx,
                      PlanNode* input,
                      PlanNode* body = nullptr,
                      Expression* condition = nullptr);

The Loop node is for looping. It keeps on executing the PlanNode segement between the body and the next Start node until the value of condition is changed to false.

Here is the source code of the InnerJoin node.

static InnerJoin* make(QueryContext* qctx,
                           PlanNode* input,
                           std::pair<std::string, int64_t> leftVar,
                           std::pair<std::string, int64_t> rightVar,
                           std::vector<Expression*> hashKeys = {},
                           std::vector<Expression*> probeKeys = {})

The InnerJoin node aims to perform inner join between two tables (Table or DataSet). leftVar and rightVar refer to the two tables respectively.

Entry Functions

The entry function of Planner is Validator∷toPlan().

Status Validator::toPlan() {
    auto* astCtx = getAstContext();
    if (astCtx != nullptr) {
        astCtx->space = space_;
    }
    auto subPlanStatus = Planner::toPlan(astCtx);
    NG_RETURN_IF_ERROR(subPlanStatus);
    auto subPlan = std::move(subPlanStatus).value();
    root_ = subPlan.root;
    tail_ = subPlan.tail;
    VLOG(1) << "root: " << root_->kind() << " tail: " << tail_->kind();
    return Status::OK();
}

Steps

Calling getAstContext() Firstly, getAstContext() is called to obtain the validated (by Validator) and rewritten AST contexts. The data structure of these contexts are defined in src/context/.

src/context/ast
├── AstContext.h
├── CypherAstContext.h
└── QueryAstContext.h
struct AstContext {
    QueryContext*   qctx; // The context of each query request
    Sentence*       sentence; // The AST of each query statement
    SpaceInfo       space; // The current graph space
};

CypherAstContext defines the AST contexts of the openCypher compatible statements. QueryAstContext defines the AST contexts of the nGQL statements.

2.Calling Planner::toPlan(astCtx)
Secondly, Planner∷toPlan(astCtx) is called. Based on the AST contexts, it will find the registered planners for the query statement in PlannerMap, and then the corresponding execution plan is generated.

Each plan is composed of a series of PlanNodes. There are two major relationships between PlanNodes, execution dependency and data dependency.

Execution dependency: From the perspective of execution order, an execution plan is a directed acyclic graph, and the dependencies between nodes are determined when the plan is generated. In the execution phase, the executor generates an operator for each node, and starts scheduling from the root node. If the root node is found dependent on another node, a recursive calling is executed for the node that the root node depends on. The process repeats until it finds a node that is not dependent on any other nodes. And then, the node is executed. After the execution is done, the executor will continue to execute the nodes that depend on it until the root node is reached.
Data dependency: The data dependency between nodes is like the execution dependency, that is, the output of the previous execution is the input of the next execution. Let’s take the InnerJoin node as an example. The inputs of InnerJoin may be the outputs of some nodes that are not adjacent to it.

(In the preceding figure, the solid lines represent the execution dependencies and the dashed lines represent the data dependencies.)

An Example

In this section, I will take MatchPlanner as an example to show how an execution plan is generated.

Here is the example statement.

MATCH (v:player)-[:like*2..4]-(v2:player)\
WITH v, v2.age AS age ORDER BY age WHERE age > 18\
RETURN id(v), age

After validated by MatchValidator and rewritten, this statement will be output as a tree composed of contexts.

Each context corresponds to a clause or a subclause.

enum class CypherClauseKind : uint8_t {
    kMatch,
    kUnwind,
    kWith,
    kWhere,
    kReturn,
    kOrderBy,
    kPagination,
    kYield,
};

struct CypherClauseContextBase : AstContext {
    explicit CypherClauseContextBase(CypherClauseKind k) : kind(k) {}
    virtual ~CypherClauseContextBase() = default;

    const CypherClauseKind  kind;
};

struct MatchClauseContext final : CypherClauseContextBase {
    MatchClauseContext() : CypherClauseContextBase(CypherClauseKind::kMatch) {}

    std::vector<NodeInfo>                       nodeInfos; // The vertices involved in the pattern
    std::vector<EdgeInfo>                       edgeInfos; // The edges involved in the pattern
    PathBuildExpression*                        pathBuild{nullptr}; // Constructing the expression of Path
    std::unique_ptr<WhereClauseContext>         where; // filter SubClause
    std::unordered_map<std::string, AliasType>* aliasesUsed{nullptr}; // The specified alias
    std::unordered_map<std::string, AliasType>  aliasesGenerated; // The generated alias
};
...

And then, these steps are followed:

1.Finding Planner for the Statement
This is a MATCH statement, so MatchPlanner is found from the PlannerMap.

2.Generating a Plan
MatchPlanner::transform is called to generate an execution plan.

StatusOr<SubPlan> MatchPlanner::transform(AstContext* astCtx) {
    if (astCtx->sentence->kind() != Sentence::Kind::kMatch) {
        return Status::Error("Only MATCH is accepted for match planner.");
    }
    auto* matchCtx = static_cast<MatchAstContext*>(astCtx);

    std::vector<SubPlan> subplans;
    for (auto& clauseCtx : matchCtx->clauses) {
        switch (clauseCtx->kind) {
            case CypherClauseKind::kMatch: {
                auto subplan = std::make_unique<MatchClausePlanner>()->transform(clauseCtx.get());
                NG_RETURN_IF_ERROR(subplan);
                subplans.emplace_back(std::move(subplan).value());
                break;
            }
            case CypherClauseKind::kUnwind: {
                auto subplan = std::make_unique<UnwindClausePlanner>()->transform(clauseCtx.get());
                NG_RETURN_IF_ERROR(subplan);
                auto& unwind = subplan.value().root;
                std::vector<std::string> inputCols;
                if (!subplans.empty()) {
                    auto input = subplans.back().root;
                    auto cols = input->colNames();
                    for (auto col : cols) {
                        inputCols.emplace_back(col);
                    }
                }
                inputCols.emplace_back(unwind->colNames().front());
                unwind->setColNames(inputCols);
                subplans.emplace_back(std::move(subplan).value());
                break;
            }
            case CypherClauseKind::kWith: {
                auto subplan = std::make_unique<WithClausePlanner>()->transform(clauseCtx.get());
                NG_RETURN_IF_ERROR(subplan);
                subplans.emplace_back(std::move(subplan).value());
                break;
            }
            case CypherClauseKind::kReturn: {
                auto subplan = std::make_unique<ReturnClausePlanner>()->transform(clauseCtx.get());
                NG_RETURN_IF_ERROR(subplan);
                subplans.emplace_back(std::move(subplan).value());
                break;
            }
            default: { return Status::Error("Unsupported clause."); }
        }
    }

    auto finalPlan = connectSegments(astCtx, subplans, matchCtx->clauses);
    NG_RETURN_IF_ERROR(finalPlan);
    return std::move(finalPlan).value();
}

A MATCH statement may be composed of multiple MATCH, UNWIND, WITH, and RETURNclauses. Therefore, with MatchPlanner::transform, the corresponding ClausePlanners are called directly to generate the corresponding SubPlans, and then the SubPlans are connected end to end by SegmentsConnector according to the appropriate connection strategies.

In the example statement, the first clause is a MATCH clause: MATCH (v:player)-[:like*2..4]-(v2:player), so MatchClausePlanner::transform is called.

StatusOr<SubPlan> MatchClausePlanner::transform(CypherClauseContextBase* clauseCtx) {
    if (clauseCtx->kind != CypherClauseKind::kMatch) {
        return Status::Error("Not a valid context for MatchClausePlanner.");
    }

    auto* matchClauseCtx = static_cast<MatchClauseContext*>(clauseCtx);
    auto& nodeInfos = matchClauseCtx->nodeInfos;
    auto& edgeInfos = matchClauseCtx->edgeInfos;
    SubPlan matchClausePlan;
    size_t startIndex = 0;
    bool startFromEdge = false;

    NG_RETURN_IF_ERROR(findStarts(matchClauseCtx, startFromEdge, startIndex, matchClausePlan));
    NG_RETURN_IF_ERROR(
        expand(nodeInfos, edgeInfos, matchClauseCtx, startFromEdge, startIndex, matchClausePlan));
    NG_RETURN_IF_ERROR(projectColumnsBySymbols(matchClauseCtx, startIndex, matchClausePlan));
    NG_RETURN_IF_ERROR(appendFilterPlan(matchClauseCtx, matchClausePlan));
    return matchClausePlan;
}

The MatchClausePlanner::transform method performs these steps:

Finding the starting vertex of the expansion.
Currently, three strategies are available for finding the starting vertex. They are registered in startVidFinders.

// MATCH(n) WHERE id(n) = value RETURN n
startVidFinders.emplace_back(&VertexIdSeek::make);

// MATCH(n:Tag{prop:value}) RETURN n
// MATCH(n:Tag) WHERE n.prop = value RETURN n
startVidFinders.emplace_back(&PropIndexSeek::make);

// seek by tag or edge(index)
// MATCH(n: tag) RETURN n
// MATCH(s)-[:edge]->(e) RETURN e
startVidFinders.emplace_back(&LabelIndexSeek::make);

Of these three strategies, VertexIdSeek is the best, which can locate the specific VID of the starting vertex. PropIndexSeek is the second, which is converted to an IndexScan that filters vertices by the property. LabelIndexSeek will be converted to an IndexScan.

For each strategy of finding the starting vertex, the findStarts function will traverse all the nodes in the MATCH pattern until it finds a node that can be used as the node of the starting vertex, and generates corresponding PlanNodes for finding the starting vertex.

For this example statement, LabelIndexScan is used and the starting vertex is v. Finally, an IndexScan node is generated and the indexes on the player tag are used.

According to the starting vertex and the MATCH pattern, an expansion across multiple steps is executed.
For the example statement, the MATCH pattern is (v:player)-[:like*1..2]-(v2:player). It means v is the starting vertex, and an expansion across one or two steps along the like edge is executed, and the end vertex is of the player tag.

Here is how the expansion is executed.

Status Expand::doExpand(const NodeInfo& node, const EdgeInfo& edge, SubPlan* plan) {
    NG_RETURN_IF_ERROR(expandSteps(node, edge, plan));
    NG_RETURN_IF_ERROR(filterDatasetByPathLength(edge, plan->root, plan));
    return Status::OK();
}

An expansion across multiple steps will generate a Loop node. The body of the Loop node is expandStep , which means a one-step expansion is executed from the given starting vertex and such an expansion generates a GetNeighbors node. The end vertex of each expansion is the starting vertex of the next expansion. It keeps looping until the maximum number of steps specified in the pattern is reached.

To do the Step M expansion, the end vertex of the M-1 steps long path is used as the starting vertex of the expansion. By expanding one step more, the expansion result is constructed as a 1-step long path consisting of the source vertex of an edge and the edge itself. And then InnerJoin is performed to the 1-step long path and the previous M-1 steps long path to obtain a set of paths of M steps long.

This set of paths are filtered to remove the paths with duplicate edges, which are not allowed for path expansion in openCypher. Finally, the end vertex is used as the starting vertex of the next expansion. Such expansions continue until the specified maximum number of steps is reached.

After Loop, a UnionAllVersionVar node is generated. It combines the paths varying from 1-step to M-steps long that are generated from the execution of each loop of the body. The filterDatasetByPathLength() function will generate a Filter node to filter out all the paths that are shorter than the minimum number of steps specified in the MATCH pattern.

After the expansion, the path looks like (v)-like-()-e-(v)-?, where the properties of the end vertex is still missing. At this point, generating a GetVertices node is needed. When the end vertex is obtained, an InnerJoin is performed to it and the M-steps long path, and then we will have a set of paths that meet the requirements of the MATCH pattern.

More information about the expansion across multiple steps of MATCH will be introduced in a new article “Variable Length Pattern Match”.

// Build Start node from first step
SubPlan loopBodyPlan;
PlanNode* startNode = StartNode::make(matchCtx_->qctx);
startNode->setOutputVar(firstStep->outputVar());
startNode->setColNames(firstStep->colNames());
loopBodyPlan.tail = startNode;
loopBodyPlan.root = startNode;

// Construct loop body
NG_RETURN_IF_ERROR(expandStep(edge,
                              startNode,                // dep
                              startNode->outputVar(),   // inputVar
                              nullptr,
                              &loopBodyPlan));

NG_RETURN_IF_ERROR(collectData(startNode,           // left join node
                               loopBodyPlan.root,   // right join node
                               &firstStep,          // passThrough
                               &subplan));
// Union node
auto body = subplan.root;

// Loop condition
auto condition = buildExpandCondition(body->outputVar(), startIndex, maxHop);

// Create loop
auto* loop = Loop::make(matchCtx_->qctx, firstStep, body, condition);

// Unionize the results of each expansion which are stored in the firstStep node
auto uResNode = UnionAllVersionVar::make(matchCtx_->qctx, loop);
uResNode->setInputVar(firstStep->outputVar());
uResNode->setColNames({kPathStr});

subplan.root = uResNode;
plan->root = subplan.root;

A table is output and its column names are determined.
All named symbols specified in the MATCH pattern are used as the column names to generate a table for the subsequent clauses, which will generate a Project node.

The second clause in the example statement is WITH. It calls WithClause::transform to generate SubPlans.

WITH v, v2.age AS age ORDER BY age WHERE age > 18

This WITH clause yields a table with two columns named v and v2.age. These columns are sorted by age, and then the table is used as a filter.

The YIELD part will generate a Project node. The ORDER BY part will generate a Sort node. And the WHERE part will generate a Filter node.

The third clause is RETURN. It will generate a Project node.

RETURN id(v), age

The complete execution plan of the example statement is shown as follows.

This is the end of this article.

1 simple way to implement variable-Length Pattern Matching

lisahui — Fri, 22 Oct 2021 06:06:50 +0000

At the very heart of openCypher, the MATCH clause allows you to specify simple query patterns to retrieve the relationships from a graph database. A variable-length pattern is commonly used to describe paths and it is Nebula Graph’s first try to get nGQL compatible with openCypher in the MATCH clause.

As can be seen from the previous articles of this series, an execution plan is composed of physical operators. Each operator is responsible for executing unique computational logics. To implement the MATCH clause, the operators such as GetNeighbors, GetVertices, Join, Project, Filter, and Loop, which have been introduced in the previous articles, are needed. Unlike the tree structure in a relational database, the execution process expressed by an execution plan in Nebula Graph is a cyclic graph. How to transform a variable-length pattern into a physical plan in Nebula Graph is the focus of the Planner. In this article, we will introduce how variable-length pattern matching is implemented in Nebula Graph.

Problem Analysis

Fixed-Length Pattern

In a MATCH clause, a fixed-length pattern is commonly used to search for a relationship. If a fixed-length pattern is considered a special case of the variable-length pattern, that is, a pattern describing a path of a specified length, the implementations of both can be unified. Here are the examples.

// Fixed-length pattern MATCH (v)-[e]-(v2)
// Variable-length pattern MATCH (v)-[e*1..1]-(v2)

The preceding examples differ from each other in the type of the e variable. In the fixed-length pattern, e represents an edge, while in the variable-length one, e represents a list of edges of length 1.

Connecting Variable-Length Patterns

According to the syntax of openCypher, a MATCH clause allows you to specify a combination of various patterns for describing complicated paths. For example, two variable-length patterns can be connected as follows.

MATCH (v)-[e*1..3]-(v2)-[ee*2..4]-(v3)

The pattern combination in the preceding example is extendable, which means by connecting variable-length and fixed-length patterns in different ways, various complicated paths can be queried. Therefore, we must find a pattern to generate an execution plan to iterate the whole process recursively. The following conditions must be considered:

The following variable-length path depends on the preceding one.
The variables in the following pattern depend on the preceding pattern.
Before the next traversal step, the starting vertex must be de-duplicated.
From the following example, you can see that as long as an execution plan can be generated for the part of ()-[:like*m..n]-, combinations and iterations may be applied to generate plans for the subsequent parts.

()-[:like*m..n]- ()-[:like*k..l]- ()
 \____________/   \____________/   \_/
    Pattern1         Pattern2       Pattern3

Execution Plan

In this section, we will introduce how the ()-[:like*m..n]- part in the preceding example is transformed into a physical execution plan in Nebula Graph. This pattern describes a graph of a minimum of m hops and a maximum of n hops. In Nebula Graph, a one-step traversal is completed by the GetNeighbors operator. To implement a multi-step traversal, each traversal step must call the GetNeighbors operator again on the basis of the previous step, and when the traversal of all the steps are completed, all the retrieved vertices and edges are connected end to end to form a single path. What users need is the paths of m to n relationships. However, in the execution process, paths of length 1 to length n are queried and are stored for output or for the next traversal, but only the paths of length m to n are retrieved.

One-Step Traversal

Let’s see what the one-step traversal looks like. In Nebula Graph, the source vertex is stored together with its outgoing edges, so retrieving them does not need to access data across partitions. However, the destination vertex and its incoming edges are stored in different partitions, so GetVertices is necessary for retrieving the properties of the vertex. In addition, to avoid replicated scanning of Storage, the source vertices must be de-duplicated before the traversal. The execution plan of a one-step traversal is shown as follows.

Multi-Step Traversal

The process of a multi-step traversal is the repetition of one-step traversal. However, we can see that the GetNeighbors operator can retrieve the properties of an edge’s source vertex, so the GetVertices operator can be omitted in the previous step. Here is an execution plan of a two-step traversal.

Storing Paths

The paths retrieved in each traversal step may be needed at the end of the traversal, so all the paths must be stored. The paths for a two-step traversal are connected by the Join operator. In the result of the example ()-[e:like*m..n]-, e represents a list of data (edges), so Union is needed to merge the results of each traversal step. The execution plan will be evolved further as follows.

Connecting Variable-Length Patterns

After the implementations of the preceding process, a physical plan will be generated for the ()-[e:like*m..n]- pattern. If multiple similar patterns are connected together, such a process is iterated. However, before the iteration, the results of the previous process must be filtered to get the paths of length m to length n. The retrieved dataset of the previous process involves the paths of length 1 to length n, so filtering them by path length is needed. When the variable-length patterns are connected together, the execution plan becomes as follows.

After the step-by-step decomposition of the patterns, the expected execution plan for the MATCH clause is finally generated. As you can see, it takes a lot of effort to transform a complicated pattern into the underlying interfaces for a traversal. Of course, the execution plan can be optimized, such as the multi-step traversal can be encapsulated by using the Loop operator and the sub-plan of a one-step traversal can be reused, which will not be detailed in this article. If you are interested, please refer to the source code of Nebula Graph.

Summary

This article demonstrated the process of generating an execution plan for a MATCH clause with a variable-length pattern. While reading the article, you may have this question: Why such a basic and simple path query will generate such a complicated execution plan in Nebula Graph? It’s not like Neo4j, where only a few operators are needed to complete the same job. In Nebula Graph, complicated directed acyclic graphs (DAG) are generated.

The answer is that in Nebula Graph, the operators are closer to the underlying interfaces and there is a lack of semantic abstractions for higher-level graph operations. The operator granularity is too fine, so too many details need to be considered to implement the optimization of the upper layer. We will further study the execution operators to gradually improve the functionality and the performance of the MATCH clause.

2 Modules in Nebula Graph: Scheduler & Executor

lisahui — Mon, 11 Oct 2021 07:26:16 +0000

You may have learned the optimizer of Nebula Graph’s query engine in the last article. In this article, we will introduce how the Scheduler and the Executor, the last two modules of the query engine, are implemented.

Overview

In the execution phase, the execution engine uses the Scheduler to transform a physical execution plan, generated by the Planner, into a series of Executors to drive their execution. Each PlanNode in a physical execution plan has a corresponding Executor.

Structure of Source Files

The source code of the Scheduler is under the src/scheduler directory.

src/scheduler
├── AsyncMsgNotifyBasedScheduler.cpp
├── AsyncMsgNotifyBasedScheduler.h
├── CMakeLists.txt
├── Scheduler.cpp
└── Scheduler.h

The Scheduler abstract class defines the common interfaces of the schedulers, which can inherit the features from the class to implement various types of schedulers. The AsyncMsgNotifyBasedScheduler scheduler has been implemented. By using the asynchronous message communication and breadth-first search algorithm, it can be prevented from stack overflow errors. The source code of the Executor is under the src/executor directory.

src/executor
├── admin
├── algo
├── CMakeLists.txt
├── ExecutionError.h
├── Executor.cpp
├── Executor.h
├── logic
├── maintain
├── mutate
├── query
├── StorageAccessExecutor.cpp
├── StorageAccessExecutor.h
└── test

Process

First, the Scheduler starts the traversal of the entire execution plan from its root node by using the breadth-first search algorithm and builds their notification mechanism according to the dependencies between nodes. During the execution phase, each node will be scheduled to be executed after being notified that all the nodes it depends on have been executed successfully. For a node, once executed, it will notify its dependent nodes until the entire plan is executed successfully.

void AsyncMsgNotifyBasedScheduler::runExecutor(
    std::vector<folly::Future<Status>>&& futures,
    Executor* exe,
    folly::Executor* runner,
    std::vector<folly::Promise<Status>>&& promises) const {
    folly::collect(futures).via(runner).thenTry(
        [exe, pros = std::move(promises), this](auto&& t) mutable {
            if (t.hasException()) {
                return notifyError(pros, Status::Error(t.exception().what()));
            }
            auto status = std::move(t).value();
            auto depStatus = checkStatus(std::move(status));
            if (!depStatus.ok()) {
                return notifyError(pros, depStatus);
            }
            // Execute in current thread.
            std::move(execute(exe)).thenTry(
                [pros = std::move(pros), this](auto&& exeTry) mutable {
                    if (exeTry.hasException()) {
                        return notifyError(pros, Status::Error(exeTry.exception().what()));
                    }
                    auto exeStatus = std::move(exeTry).value();
                    if (!exeStatus.ok()) {
                        return notifyError(pros, exeStatus);
                    }
                    return notifyOK(pros);
                });
        });
}

Each Executor goes through four phases: “create”, “open”, “execute”, and then “close”.

create

In the “create” phase, an appropriate Executor will be generated according to the node type.

open

In the “open” phase, before the execution starts, the Executor is initialized, the slow queries are terminated, and the memory watermark is checked. When using Nebula Graph, you can use kill to terminate a query, so the status of the current execution plan must be checked before the execution of each Executor. If the plan is in the killed status, the execution will be terminated. Before the execution of each query Executor, it is necessary to check whether the amount of free memory has fallen below the watermark. If the watermark is reached, the execution will be terminated, which may avoid OOM.

Status Executor::open() {
    if (qctx_->isKilled()) {
        VLOG(1) << "Execution is being killed. session: " << qctx()->rctx()->session()->id()
            << "ep: " << qctx()->plan()->id()
            << "query: " << qctx()->rctx()->query();
        return Status::Error("Execution had been killed");
    }
    auto status = MemInfo::make();
    NG_RETURN_IF_ERROR(status);
    auto mem = std::move(status).value();
    if (node_->isQueryNode() && mem->hitsHighWatermark(FLAGS_system_memory_high_watermark_ratio)) {
        return Status::Error(
            "Used memory(%ldKB) hits the high watermark(%lf) of total system memory(%ldKB).",
            mem->usedInKB(),
            FLAGS_system_memory_high_watermark_ratio,
            mem->totalInKB());
    }
    numRows_ = 0;
    execTime_ = 0;
    totalDuration_.reset();
    return Status::OK();
}

execute

The input and output of a query Executor are in the form of tables (DataSet). The execution of an Executor is based on the iterator model, which means that for each calculation, the next() method of the iterator of the input table is called to retrieve a row of data and then the calculation is performed. Such a process is repeated until the traversal of the entire input table is done. The results of the calculations are constructed into a new table and output to the next Executor as its input.

folly::Future<Status> ProjectExecutor::execute() {
    SCOPED_TIMER(&execTime_);
    auto* project = asNode<Project>(node());
    auto columns = project->columns()->columns();
    auto iter = ectx_->getResult(project->inputVar()).iter();
    DCHECK(!!iter);
    QueryExpressionContext ctx(ectx_);

    VLOG(1) << "input: " << project->inputVar();
    DataSet ds;
    ds.colNames = project->colNames();
    ds.rows.reserve(iter->size());
    for (; iter->valid(); iter->next()) {
        Row row;
        for (auto& col : columns) {
            Value val = col->expr()->eval(ctx(iter.get()));
            row.values.emplace_back(std::move(val));
        }
        ds.rows.emplace_back(std::move(row));
    }
    VLOG(1) << node()->outputVar() << ":" << ds;
    return finish(ResultBuilder().value(Value(std::move(ds))).finish());
}

If the input table of the current Executor cannot be used by the other Executors as their input, the memory occupied by the table will be dropped in the execution phase to reduce memory usage.

void Executor::drop() {
    for (const auto &inputVar : node()->inputVars()) {
        if (inputVar != nullptr) {
            // Make sure use the variable happened-before decrement count
            if (inputVar->userCount.fetch_sub(1, std::memory_order_release) == 1) {
                // Make sure drop happened-after count decrement
                CHECK_EQ(inputVar->userCount.load(std::memory_order_acquire), 0);
                ectx_->dropResult(inputVar->name);
                VLOG(1) << "Drop variable " << node()->outputVar();
            }
        }
    }
}

close

After the execution of an Executor is done, some collected execution information, such as execution time and the number of rows in the output table, is added to the profiling statistics. You can run a PROFILE statement and then view the statistics in the returned result.

Execution Plan (optimize time 141 us)

-----+------------------+--------------+-----------------------------------------------------+--------------------------------------
| id | name             | dependencies | profiling data                                      | operator info                       |
-----+------------------+--------------+-----------------------------------------------------+--------------------------------------
|  2 | Project          | 3            | ver: 0, rows: 56, execTime: 147us, totalTime: 160us | outputVar: [                        |
|    |                  |              |                                                     |   {                                 |
|    |                  |              |                                                     |     "colNames": [                   |
|    |                  |              |                                                     |       "VertexID",                   |
|    |                  |              |                                                     |       "player.age"                  |
|    |                  |              |                                                     |     ],                              |
|    |                  |              |                                                     |     "name": "__Project_2",          |
|    |                  |              |                                                     |     "type": "DATASET"               |
|    |                  |              |                                                     |   }                                 |
|    |                  |              |                                                     | ]                                   |
|    |                  |              |                                                     | inputVar: __TagIndexFullScan_1      |
|    |                  |              |                                                     | columns: [                          |
|    |                  |              |                                                     |   "$-.VertexID AS VertexID",        |
|    |                  |              |                                                     |   "player.age"                      |
|    |                  |              |                                                     | ]                                   |
----------+------------------+--------------+-----------------------------------------------------+--------------------------------------
|  3 | TagIndexFullScan | 0            | ver: 0, rows: 56, execTime: 0us, totalTime: 6863us  | outputVar: [                        |
|    |                  |              |                                                     |   {                                 |
|    |                  |              |                                                     |     "colNames": [                   |
|    |                  |              |                                                     |       "VertexID",                   |
|    |                  |              |                                                     |       "player.age"                  |
|    |                  |              |                                                     |     ],                              |
|    |                  |              |                                                     |     "name": "__TagIndexFullScan_1", |
|    |                  |              |                                                     |     "type": "DATASET"               |
|    |                  |              |                                                     |   }                                 |
|    |                  |              |                                                     | ]                                   |
|    |                  |              |                                                     | inputVar:                           |
|    |                  |              |                                                     | space: 318                          |
|    |                  |              |                                                     | dedup: false                        |
|    |                  |              |                                                     | limit: 9223372036854775807          |
|    |                  |              |                                                     | filter:                             |
|    |                  |              |                                                     | orderBy: []                         |
|    |                  |              |                                                     | schemaId: 319                       |
|    |                  |              |                                                     | isEdge: false                       |
|    |                  |              |                                                     | returnCols: [                       |
|    |                  |              |                                                     |   "_vid",                           |
|    |                  |              |                                                     |   "age"                             |
|    |                  |              |                                                     | ]                                   |
|    |                  |              |                                                     | indexCtx: [                         |
|    |                  |              |                                                     |   {                                 |
|    |                  |              |                                                     |     "columnHints": [],              |
|    |                  |              |                                                     |     "index_id": 325,                |
|    |                  |              |                                                     |     "filter": ""                    |
|    |                  |              |                                                     |   }                                 |
|    |                  |              |                                                     | ]                                   |
----------+------------------+--------------+-----------------------------------------------------+--------------------------------------
|  0 | Start            |              | ver: 0, rows: 0, execTime: 1us, totalTime: 19us     | outputVar: [                        |
|    |                  |              |                                                     |   {                                 |
|    |                  |              |                                                     |     "colNames": [],                 |
|    |                  |              |                                                     |     "type": "DATASET",              |
|    |                  |              |                                                     |     "name": "__Start_0"             |
|    |                  |              |                                                     |   }                                 |
|    |                  |              |                                                     | ]                                   |
----------+------------------+--------------+-----------------------------------------------------+--------------------------------------

So far, the explanation of the query engine source code has been completed. Next time we will explain the implementation of some features of Nebula Graph.

Nebula Operator Kind, oneliner installer for Nebula K8s Operator Playground

lisahui — Thu, 30 Sep 2021 08:54:38 +0000

Nebula-Kind, an one-liner command to try K8s Operator based Nebula Graph Cluster on your machine, with the help of KIND (K8s in Docker)

Nebula-Operator-Kind

As a Cloud Native Distributed Database, Nebula Graph comes with an open-source K8s Operator to enable boostrap and maintain Nebula Graph Cluster from a K8s CRD.

Normally it takes you some time to setup all the dependencies and control plane resources of the Nebula Operator. If you are as lazy as I am, this Nebula-Operator-Kind is made for you to quick start and play with Nebula Graph in KIND.

Nebula-Operator-Kind is the one-liner for setup everything for you including:

Docker
K8s(KIND)
PV Provider
Nebula-Operator
Nebula-Console
nodePort for accessing the Cluster
Kubectl for playing with KIND and Nebula Operator

How To Use

Install Nebula-Operator-Kind:

curl -sL nebula-kind.siwei.io/install.sh | bash

You will see this after it’s done

You can connect to the cluster via ~/.nebula-kind/bin/console as below:

~/.nebula-kind/bin/console -u user -p password --address=127.0.0.1 --port=30000

It’s in GitHub with more information you may be intrested in ;-), please try and feedback there~ https://github.com/wey-gu/nebula-operator-kind

Install on KubeSphere all-in-on cluster：

curl -sL nebula-kind.siwei.io/install-ks-1.sh | bash

Install on existing K8s cluster:

curl -sL nebula-kind.siwei.io/install-on-k8s.sh | bash