DEV Community: Bassem Naguib

Gremlin DSL Implementation in C#

Bassem Naguib — Tue, 23 Mar 2021 08:47:36 +0000

Introduction

Developers can extend the Gremlin traversal language by defining higher-level, domain-specific steps that are made from the lower-level, standard graph traversal steps. This can be used to define steps that are specific to your application domain. Or to define more general-purpose steps to make traversing graphs easier.

These extensions are called Gremlin Domain-Secific Languages (DSLs). They are comparable to SQL stored procedures in that they encapsulate data access logic. But the big difference is that SQL stored procedures are defined on the database server-side. While Gremlin DSLs are defined on the client-side and the server does not know anything about them.

In this article, I will explain how to implement Gremlin.Net DSLs in C#. I will show how to define a ProjectBy() function that can be used instead of the standard Gremlin syntax Project().By().

Sample Graph Data

The following code populates the graph with some sample data: a "post" vertex that is related by "hasComment" edges to three "comment" vertices.

static void PopulateGraph(GraphTraversalSource g)
{
    g.AddV("post")
        .Property("slug", "post-1")
        .Property("author", "Author 1")
        .Property("publishedOn", new DateTime(2021, 3, 19, 17, 7, 52))
        .Property("lastUpdate", new DateTime(2021, 4, 2, 8, 45, 3))
        .Property("title", "Post 1")
        .Property("body", "Body 1")
        .As("p1")

    .AddV("comment")
        .Property("userName", "User 1")
        .Property("postedOn", new DateTime(2021, 4, 15, 20, 59, 50))
        .Property("title", "Comment 1 Title")
        .Property("body", "Comment 1 body")
        .Property("wasEdited", false)
        .As("c1")
    .AddE("hasComment").From("p1").To("c1")

    .AddV("comment")
        .Property("userName", "User 2")
        .Property("postedOn", new DateTime(2021, 4, 18, 16, 48, 44))
        .Property("title", "Comment 2 Title")
        .Property("body", "Comment 2 body")
        .Property("wasEdited", true)
        .As("c2")
    .AddE("hasComment").From("p1").To("c2")

    .AddV("comment")
        .Property("userName", "User 3")
        .Property("postedOn", new DateTime(2021, 4, 22, 4, 12, 21))
        .Property("title", "Comment 3 Title")
        .Property("body", "Comment 3 body")
        .Property("wasEdited", false)
        .As("c3")
    .AddE("hasComment").From("p1").To("c3")

    .Iterate();
}

The following is a visual representation of the graph data.

The Standard Gremlin Project Syntax

In standard Gremlin, you add a "Project" step by calling Project() and passing the project keys as arguments. Then you modulate this Project step by calling By() multiple times to specify the value sources for each of the keys. This works well when you are projecting by a small number of keys/values. But I do not like this syntax when the lists of keys and values are long and the key is too far from the corresponding value source.

The following code uses the standard Project syntax to get the post and the related comments information.

static IDictionary<string, object> GetPostUsingStandardProject(GraphTraversalSource g)
{
    return g.V().Has("post", "slug", "post-1")
        .Project<object>("id", "slug", "author", "publishedOn", "lastUpdate", "title", "body", "comments")
        .By(T.Id)
        .By("slug")
        .By("author")
        .By("publishedOn")
        .By("lastUpdate")
        .By("title")
        .By("body")
        .By(
            __.Out("hasComment")
            .Project<object>("id", "userName", "postedOn", "title", "body", "wasEdited")
            .By(T.Id)
            .By("userName")
            .By("postedOn")
            .By("title")
            .By("body")
            .By("wasEdited")
            .Fold()
        ).Next();
}

Defining an Alternative "ProjectBy()" Function

Let's create an alternative projection function that takes an IDictionary<string, ITraversal> so that the keys are always near the value sources no matter the number of projection keys/values.

This ProjectBy() function is an example general-purpose DSL. You add Gremlin.Net DSLs as extension methods to the GraphTraversal<S, E> class or the GraphTraversalSource class. Or by adding static methods that build anonymous traversals.

Here are the GraphTraversal<S, E> extension methods.

public static class GraphTraversalExtensions
{
    public static GraphTraversal<S, IDictionary<string, E2>> ProjectBy<S, E, E2>(
        this GraphTraversal<S, E> tIn, IDictionary<string, ITraversal> projectionMap)
    {
        if (tIn is null)
            throw new ArgumentNullException(nameof(tIn));
        if (projectionMap is null)
            throw new ArgumentNullException(nameof(projectionMap));
        if (!projectionMap.Any())
            throw new ArgumentException("The projection map should contain at least one item.", nameof(projectionMap));

        GraphTraversal<S, IDictionary<string, E2>> tOut = tIn.Project<E2>(
            projectionMap.First().Key, projectionMap.Skip(1).Select(kv => kv.Key).ToArray());

        foreach (KeyValuePair<string, ITraversal> keyValue in projectionMap)
        {
            if (keyValue.Value is null)
                tOut = tOut.By(keyValue.Key);
            else
                tOut = tOut.By(keyValue.Value);
        }

        return tOut;
    }

    public static GraphTraversal<S, IDictionary<string, object>> ProjectBy<S, E>(
        this GraphTraversal<S, E> tIn, IDictionary<string, ITraversal> projectionMap)
    {
        return ProjectBy<S, E, object>(tIn, projectionMap);
    }
}

Our ProjectBy() function takes an IDictionary<string, ITraversal>. The dictionary keys are the projection result keys (column headers). And the dictionary values are anonymous traversals that describe how to get the projection values.

Our function calls the standard Gremlin Project() function which takes the projection keys as arguments. Then our function calls By() for each of the projection items. If no traversal is provided for the projection item, then it is assumed that the key is also the projected property name.

The second ProjectBy() overload saves the caller from having to specify the type of the projected values. Usually the type Object has to be used because the projected values are of different types.

Now let's create static ProjectBy() methods that build anonymous traversals with a "Project" step.

public static class __Ext
{
    public static GraphTraversal<object, IDictionary<string, E2>> ProjectBy<E2>(
        IDictionary<string, ITraversal> projectionMap)
    {
        return new GraphTraversal<object, object>().ProjectBy<object, object, E2>(projectionMap);
    }

    public static GraphTraversal<object, IDictionary<string, object>> ProjectBy(
        IDictionary<string, ITraversal> projectionMap)
    {
        return ProjectBy<object>(projectionMap);
    }

    public static GraphTraversal<object, object> Values(params string[] propertyKeys)
    {
        return __.Values<object>(propertyKeys);
    }
}

Our static ProjectBy() method creates an empty traversal then calls the extension ProjectBy() method that we defined earlier. So we did not have to repeat the implementation logic.

We created a ProjectBy() overload that saves the caller from having to specify the type of the projected values. And we created a Values() function that does not require the user to provide a type parameter. Specifying these types is often useless when the result of the projection has to be an IDictionary<string, object> anyway.

As I mentioned before, a common way of extending Gremlin.Net is by adding extension methods to the GraphTraversalSource class. But this will not be demonstrated in this article because it does not make any sense to add a ProjectBy() method to the GraphTraversalSource.

Using the "ProjectBy()" Function

Here is how to use the "ProjectBy()" Function.

static IDictionary<string, object> GetPostUsingProjectBy(GraphTraversalSource g)
{
    return g.V().Has("post", "slug", "post-1")
        .ProjectBy(new Dictionary<string, ITraversal>
        {
            { "id", __.Id() },
            { "slug", null },
            { "writer", __Ext.Values("author") },
            { "publishedOn", null },
            { "updateDate", __Ext.Values("lastUpdate") },
            { "title", null },
            { "body", null },
            {
                "comments",
                __.Out("hasComment")
                .ProjectBy(new Dictionary<string, ITraversal>
                {
                    { "id", __.Id() },
                    { "userName", null },
                    { "postedOn", null },
                    { "subject", __Ext.Values("title") },
                    { "body", null },
                    { "wasEdited", null }
                }).Fold()
            }
        }).Next();
}

Now the keys are right next to the value sources which makes the traversal more readable and maintainable in my opinion. Also we do not have to specify the value source when we want to get the value of a property which name is the same as the projection key.

Here is a link to the "Gremlin DSL Implementation in C#" project repo on GitHub https://github.com/JoinTheGraph/GremlinDslImplementationInCSharp

Hosting Multiple Graphs on JanusGraph

Bassem Naguib — Fri, 05 Mar 2021 23:13:17 +0000

Introduction

In this article, I will explain how to configure JanusGraph to host multiple graphs on the same JanusGraph instance. I will start by explaining how you can add graphs from the JanusGraph configuration files. Then I will explain how to configure JanusGraph to use the ConfiguredGraphFactory so you can add and remove graphs dynamically without having to edit the configuration files or restart the JanusGraph server.

View the Default Graph Configuration

Out of the box, JanusGraph is configured to host only one graph. And to use the in-memory storage backend to save this graph data. Looking at this default graph configuration will help us determine what we need to do to add more graphs.

Go to the JanusGraph root folder and open the file "conf/gremlin-server/gremlin-server.yaml".

cd /opt/janusgraph-0.5.3/
vim conf/gremlin-server/gremlin-server.yaml

You will see that this configuration file has a "graphs" list containing one "graph" item.

graphs: {
  graph: conf/janusgraph-inmemory.properties
}

The key "graph" is the identifier that can be used by remote clients to access this graph. And the value is a path to a properties/configuration file that specifies things like the storage backend used to save this graph data and the mixed index backend.

Next, open the file "scripts/empty-sample.groovy".

vim scripts/empty-sample.groovy

The last line in this file adds the identifier "g" and its value (the graph traversal source) to the globals map.

globals << [g : graph.traversal()]

So we should be able to use the identifiers "graph" and "g" from a remote client to access the graph and the graph traversal source respectively. Let's try this!

Start the JanusGraph server.

bin/gremlin-server.sh

And from a different terminal window start the Gremlin Console.

bin/gremlin.sh

Then connect the Gremlin Console to the Gremlin server.

gremlin> :remote connect tinkerpop.server conf/remote.yaml
==>Configured localhost/127.0.0.1:8182
gremlin> :remote console
==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182] - type ':remote console' to return to local mode

And try evaluating the identifiers "graph" and "g".

gremlin> graph
==>standardjanusgraph[inmemory:[127.0.0.1]]
gremlin> g
==>graphtraversalsource[standardjanusgraph[inmemory:[127.0.0.1]], standard]

They point to the graph and the graph traversal source just as expected.

Let's add a second graph and instruct JanusGraph to save the graph data in HBase.

Start HBase

Navigate to the HBase root folder and start the HBase server.

cd opt/hbase-2.1.5/
bin/start-hbase.sh

And start the HBase shell to see the tables that JanusGraph creates in HBase.

bin/hbase shell

You can run the list command to see the tables that you have initially. I do not have any tables as shown in the shell output below.

hbase(main):001:0> list
TABLE
0 row(s)
Took 0.3452 seconds
=> []

Add a Graph From the Configuration

We need to create a properties file that specifies the properties of the second graph we want to create.

Navigate to the JanusGraph root folder, copy the file "conf/janusgraph-hbase.properties", then open the copy for editing.

cd /opt/janusgraph-0.5.3/
cp conf/janusgraph-hbase.properties conf/graph2-hbase.properties
vim conf/graph2-hbase.properties

I will add only one line to this properties file to specify the name of the HBase table that will be used to store the graph data.

storage.hbase.table = graph2

Now open the "conf/gremlin-server/gremlin-server.yaml" file for editing.

vim conf/gremlin-server/gremlin-server.yaml

And add a second item under "graphs". The key/identifier will be "graph2" and the value will be the path to the properties file we just created.

graphs: {
  graph: conf/janusgraph-inmemory.properties,
+ graph2: conf/graph2-hbase.properties
}

Then open the file "scripts/empty-sample.groovy" for editing.

vim scripts/empty-sample.groovy

And add a new graph traversal source to the globals map.

  globals << [g : graph.traversal()]
+ globals << [g2 : graph2.traversal()]

To apply the new configuration, shutdown the JanusGraph server by pressing CTRL + c if it is already running, then start the server again.

bin/gremlin-server.sh

JanusGraph should create the HBase table for the new graph on startup. Use the HBase shell to see if this table was created successfully.

hbase(main):002:0> list
TABLE
graph2
1 row(s)
Took 0.2073 seconds
=> ["graph2"]

Now go to the Gremlin Console. Since we restarted the server, we will need to close the connection from the console then reconnect.

gremlin> :remote close
==>Removed - Gremlin Server - [localhost/127.0.0.1:8182]
gremlin> :remote connect tinkerpop.server conf/remote.yaml
==>Configured localhost/127.0.0.1:8182
gremlin> :remote console
==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182] - type ':remote console' to return to local mode

And let's test the identifiers "graph2" and "g2" to see if they are recognized.

gremlin> graph2
==>standardjanusgraph[hbase:[127.0.0.1]]
gremlin> g2
==>graphtraversalsource[standardjanusgraph[hbase:[127.0.0.1]], standard]

They are indeed recognized. So we successfully added a graph from the configuration. Which is great! But note that we had to edit the configuration files and restart the server. Depending on your use case, this may or may not be acceptable. In the next sections, I will show how to add and remove graphs dynamically without service interruptions.

Configure JanusGraph to Use the "ConfiguredGraphFactory"

The "ConfiguredGraphFactory" provides methods for dynamically managing the graphs hosted on the server. The default JanusGraph configuration does not enable the ConfiguredGraphFactory. So we need to make some changes to the configuration before we can use it.

The ConfiguredGraphFactory needs to store information about the graphs it is tracking. It uses a graph called the "Configuration Management Graph" to store this information.

We need a properties file for this configuration management graph. JanusGraph comes with two properties files that can be used for this purpose: "janusgraph-cassandra-configurationgraph.properties" and "janusgraph-cql-configurationgraph.properties". But I want to store the configuration graph in HBase. So I will create a new file "janusgraph-hbase-configurationgraph.properties" by copying one of the two files that came with JanusGraph. Then I will open the new file for editing.

cp conf/janusgraph-cql-configurationgraph.properties conf/janusgraph-hbase-configurationgraph.properties
vim conf/janusgraph-hbase-configurationgraph.properties

There is only one line that I need to change in the new properties file to set the storage backend to "hbase" instead of "cql".

- storage.backend=cql
+ storage.backend=hbase

And there are three changes that need to be made in the YAML configuration file. So let's open it for editing.

vim conf/gremlin-server/gremlin-server.yaml

The three changes are:

Add a graph named "ConfigurationManagementGraph" under "graphs".
Add a "graphManager" property.
Change the value of the "channelizer" property.

- channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
+ channelizer: org.janusgraph.channelizers.JanusGraphWebSocketChannelizer
+ graphManager: org.janusgraph.graphdb.management.JanusGraphManager
  graphs: {
    graph: conf/janusgraph-inmemory.properties,
    graph2: conf/graph2-hbase.properties,
+   ConfigurationManagementGraph: conf/janusgraph-hbase-configurationgraph.properties
  }

To apply the new configuration, shutdown the JanusGraph server by pressing CTRL + c if it is already running, then start it again.

bin/gremlin-server.sh

JanusGraph should create the "ConfigurationManagementGraph" on startup. Let's make sure this happened by entering the list command in the HBase shell.

hbase(main):003:0> list
TABLE
ConfigurationManagementGraph
graph2
2 row(s)
Took 0.0848 seconds
=> ["ConfigurationManagementGraph", "graph2"]

Create a Template and Graphs Dynamically From the Client-Side

Switch to the Gremlin Console. We will need to close the old connection and reconnect because the server was restarted.

gremlin> :remote close
==>Removed - Gremlin Server - [localhost/127.0.0.1:8182]
gremlin> :remote connect tinkerpop.server conf/remote.yaml session
==>Configured localhost/127.0.0.1:8182-[be2df6ef-df68-4061-9ee0-c172e40c1d92]
gremlin> :remote console
==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182]-[be2df6ef-df68-4061-9ee0-c172e40c1d92] - type ':remote console' to return to local mode

Note than this time I connected to the server using a "sessioned connection". This is because it is much easier to build the template configuration map by sending multiple commands to the server.

The following code creates a template configuration. Then creates two graphs: "dynamic1" and "dynamic2" based on this template.

templateConfMap = new HashMap();
templateConfMap.put("storage.backend", "hbase");
templateConfMap.put("storage.hostname", "localhost");
ConfiguredGraphFactory.createTemplateConfiguration(new MapConfiguration(templateConfMap));

ConfiguredGraphFactory.create("dynamic1")
ConfiguredGraphFactory.create("dynamic2")

Creating these graphs adds the global identifiers "dynamic1" and "dynamic2" for the graph objects. And adds the global identifiers "dynamic1_traversal" and "dynamic2_traversal" for the graph traversal source objects. These identifiers enable clients to access the new graphs.

We cannot use these identifiers right now because we are using a sessioned connection that was established before the identifiers were created. So we need to disconnect from the server and reconnect to bind the new identifiers.

:remote close
:remote connect tinkerpop.server conf/remote.yaml
:remote console

Now try evaluating the new identifiers to make sure they are defined.

gremlin> dynamic1
==>standardjanusgraph[hbase:[localhost]]
gremlin> dynamic1_traversal
==>graphtraversalsource[standardjanusgraph[hbase:[localhost]], standard]
gremlin> dynamic2
==>standardjanusgraph[hbase:[localhost]]
gremlin> dynamic2_traversal
==>graphtraversalsource[standardjanusgraph[hbase:[localhost]], standard]

These identifiers will NOT be forgotten when the JanusGraph server is restarted.

Now list the tables from the HBase shell to see the tables that JanusGraph created for the new graphs.

hbase(main):004:0> list
TABLE
ConfigurationManagementGraph
dynamic1
dynamic2
graph2
4 row(s)
Took 0.0779 seconds
=> ["ConfigurationManagementGraph", "dynamic1", "dynamic2", "graph2"]

Other Functions of the ConfiguredGraphFactory

The ConfiguredGraphFactory can be used to get the list of names of the graphs that it is tracking.

gremlin> ConfiguredGraphFactory.getGraphNames()
==>dynamic1
==>dynamic2

It can also be used to drop graphs.

gremlin> ConfiguredGraphFactory.drop('dynamic2')
==>null
gremlin> ConfiguredGraphFactory.getGraphNames()
==>dynamic1

Using Gremlin.Net From an ASP.NET Web Application

Bassem Naguib — Tue, 23 Feb 2021 09:23:37 +0000

Introduction

In this article, I will explain how to use Gremlin.Net to connect and send traversals to a Gremlin server from an ASP.NET Web application.

If you would like to follow along, you will need the "Gremlin Server" and the "Gremlin Console" which can be downloaded from the Apache Tinkerpop Website. And you will need the .NET SDK which can be downloaded from here.

Run the Gremlin Server and Add Sample Data

Navigate to the Gremlin Server root folder then run ./bin/gremlin-server.sh if you are on Linux or Mac. Or run .\bin\gremlin-server.bat if you are on Windows. This should start the Gremlin Server and tell you which port it is listening to. The port is "8182" by default.

Then navigate to the Gremlin Console root folder and run ./bin/gremlin.sh if you are on Linux or Mac. Or run .\bin\gremlin.bat if you are on Windows. This should start the Gremlin Console.

Now enter the following commands in the Gremlin Console.

:remote connect tinkerpop.server conf/remote.yaml
:remote console

The first command connects the Gremlin Console to the Gremlin Server. And the second command instructs the Gremlin Console to send all the following commands to the server so you do not have to precede every command with :>.

Now let's create a couple of vertices so we can pull them and display them in the ASP.NET Web application that we will create later.

g.addV('person').property('firstName', 'First').property('lastName', 'Person')
g.addV('person').property('firstName', 'Second').property('lastName', 'Person')

Create the ASP.NET Web Application

Navigate to the folder where you wish to create the ASP.NET Web application then run the following command

dotnet new webapp

Then add the Gremlin.Net NuGet package by running the following command.

dotnet add package Gremlin.Net

Register the Gremlin Client Objects

Then open "Startup.cs" in the web application project root folder. We need to write some code in the ConfigureServices() method to add the Gremlin client objects (or their factory methods) to the container.

The Gremlin client objects we need are GremlinClient and GraphTraversalSource. The GremlinClient can be used to send scripts to the Gremlin Server. Which is something you should avoid as much as possible. But sometimes it is your only option for tasks like defining the graph schema and creating indices. The GraphTraversalSource wraps the GremlinClient and provides a better interface for building Gremlin traversals and sending them to the server. So the GraphTraversalSource is what you will use most of the time.

The big question is: "What should be the lifetime of these objects?". The ASP.NET service container gives you three lifetime options for registering types:

Transient: A new object is created each time an instance is requested from the container.
Scoped: A new object is created only once per HTTP request and is reused as many times as needed during this same request processing.
Singleton: Only one object is created per application lifetime. And this one object is used by all the web application threads potentially running in parallel to process concurrent HTTP requests.

GremlinClient is thread safe. So it can be registered as a singleton. And you will probably want to do that because every instance of GremlinClient contains a connection pool. And every time you create an instance of GremlinClient it opens as many connections to the database server as needed to fill the pool. So creating an instance of GremlinClient is a very expensive operation that you do not normally want to do more than once in the application lifetime. The GremlinClient instance is bound to a particular database server, so the only reason to create multiple instances is if you want to connect to multiple database servers.

GraphTraversalSource is thread safe too. So it can be registered as a singleton. But creating an instance of GraphTraversalSource is very cheap when you already have the GremlinClient required to create it. So it will not hurt if you register the GraphTraversalSource as a transient or a scoped service. The GraphTraversalSource is bound to a particular graph on the database server. So if your application needs to access multiple graphs, you will need to create multiple GraphTraversalSource instances.

Update the ConfigureServices() function in "Startup.cs" to register the GremlinClient and the GraphTraversalSource as singletons.

public void ConfigureServices(IServiceCollection services)
{
    services.AddSingleton<GremlinClient>(
        (serviceProvider) =>
        {
            var gremlinServer = new GremlinServer(
                hostname: "localhost",
                port: 8182,
                enableSsl: false,
                username: null,
                password: null
            );

            var connectionPoolSettings = new ConnectionPoolSettings
            {
                MaxInProcessPerConnection = 32,
                PoolSize = 4,
                ReconnectionAttempts = 4,
                ReconnectionBaseDelay = TimeSpan.FromSeconds(1)
            };

            return new GremlinClient(
                gremlinServer: gremlinServer,
                connectionPoolSettings: connectionPoolSettings
            );
        }
    );

    services.AddSingleton<GraphTraversalSource>(
        (serviceProvider) =>
        {
            GremlinClient gremlinClient = serviceProvider.GetService<GremlinClient>();
            var driverRemoteConnection = new DriverRemoteConnection(gremlinClient, "g");
            return AnonymousTraversalSource.Traversal().WithRemote(driverRemoteConnection);
        }
    );

    services.AddRazorPages();
}

The string "g" given to the DriverRemoteConnection is the identifier of the server-side GraphTraversalSource that the local GraphTraversalSource should be bound to. If the server has multiple graphs, each of them will be exposed to remote clients by a different GraphTraversalSource identifier.

Execute a Gremlin Traversal

Open "Pages/Index.cshtml.cs" under the project root folder. And change the private property and the constructor to inject an instance of GraphTraversalSource instead of the ILogger.

private readonly GraphTraversalSource _g;

public IndexModel(GraphTraversalSource g)
{
    _g = g;
}

Then add a public property on the IndexModel to hold the people data that we will fetch from the database.

public IList<IDictionary<string, object>> People { get; set; }

Unfortunately the Gremlin traversals do not return strongly-typed models. Here the list items will represent people. The dictionary will hold the property names and values of the person. Of course there are ways to map this properties dictionary to a strongly-typed object. But I will not do this here because I want to focus on the basics in this article.

Next, write the Gremlin traversal in the OnGet() method to fetch the data from the Gremlin server and assign it to the People property.

public void OnGet()
{
    People = _g.V()
        .HasLabel("person")
        .Project<object>("Id", "FirstName", "LastName")
            .By(T.Id)
            .By("firstName")
            .By("lastName")
        .ToList();
}

There is a small enhancement that I would like to make to the code above. It is always better to execute IO operations asynchronously so we do not block the thread. And Gremlin.Net does provide a Promise() function to execute the traversals asynchronously. So let's make the OnGet() method async. And replace .ToList() by .Promise(traversal => traversal.ToList()).

public async Task OnGetAsync()
{
    People = await _g.V()
        .HasLabel("person")
        .Project<object>("Id", "FirstName", "LastName")
            .By(T.Id)
            .By("firstName")
            .By("lastName")
        .Promise(traversal => traversal.ToList());
}

Display the Traversal Results In the View

Open "Pages/Index.cshtml" under the project root folder and write the following Razor code to display the people data fetched from the graph database.

@if (!Model.People.Any())
{
    <p>There are no people in the database.</p>
}
else
{
    <table border="1" cellpadding="5">
        <tr>
            <th>ID</th>
            <th>First Name</th>
            <th>Last Name</th>
        </tr>

        @foreach (Dictionary<string, object> person in Model.People)
        {
            <tr>
                <td>@person["Id"]</td>
                <td>@person["FirstName"]</td>
                <td>@person["LastName"]</td>
            </tr>
        }
    </table>
}

Run the Web Application

Navigate to the web application root folder and run the following command.

dotnet run

Then, from your favorite web browser, navigate to https://localhost:5001/
You may need to replace the port number by whatever port your web server is listening on.

If everything is working as expected, you should see two "person" records displayed on the home page. These two records are coming from the two vertices that we added earlier from the Gremlin Console.

The code snippets in this article are from the "GremlinWeb" project published on Github https://github.com/JoinTheGraph/GremlinWeb

Installing Apache HBase for JanusGraph

Bassem Naguib — Sun, 31 Jan 2021 08:38:12 +0000

Introduction

This is the fourth article in a series of articles about installing and configuring JanusGraph and its storage backends on a Linux Ubuntu server.

In this article, I will explain how to install and run the Apache HBase database. And how to configure JanusGraph to use it as its storage backend.

Download and Extract HBase

At the time of writing this article, JanusGraph's latest release page on GitHub says that JanusGraph is compatible with HBase 2.1.5. So I will get this version's download link from the Apache HBase archive, and I will use the wget shell command to download it to my "/opt" directory. Then I will use the tar command to extract the contents of the downloaded archive.

cd /opt
wget https://archive.apache.org/dist/hbase/2.1.5/hbase-2.1.5-bin.tar.gz
tar -xf hbase-2.1.5-bin.tar.gz

Create a Linux User for Running HBase

I think it is a good idea to create a Linux user for running the HBase server. This user can be given only the permissions needed by HBase and not more. This dedicated user will also make it easier to find the HBase server process if we need to monitor it or kill it.

adduser hbase

You will be prompted to enter the password for this new user.

Now let's make this newly created user the owner of the HBase root folder and all its contents.

chown -R hbase:hbase hbase-2.1.5

Find the Java Home Folder

We must set the JAVA_HOME environment variable before starting HBase. So we need to find the path to the Java Home directory. This path will be different depending on the Linux distribution and the installed JRE/JDK package.

I used the which command to find the location of the java command. Then I used the ll command to get the details of this file. It turned out to be a symbolic link pointing to another symbolic link pointing to the actual "java" executable.

$ which java
/usr/bin/java
$ ll /usr/bin/java
lrwxrwxrwx 1 root root 22 Dec 15 17:22 /usr/bin/java -> /etc/alternatives/java*
$ ll /etc/alternatives/java
lrwxrwxrwx 1 root root 46 Dec 15 17:22 /etc/alternatives/java -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*
$ ll /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-rwxr-xr-x 1 root root 14632 Nov  8 19:38 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*

So the path to the java executable is "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java". And the Java Home folder is one level above the "bin" folder containing the "java" executable. So the Java Home folder is "/usr/lib/jvm/java-8-openjdk-amd64/jre/".

Set the JAVA_HOME Environment Variable

We can set the JAVA_HOME environment variable by entering the following command.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/

But I don't want to have to enter this command manually every time before running the HBase server. HBase gives you a better way to set the environment variables for the server. All you need to do is edit the file "conf/hbase-env.sh" under the HBase root folder to specify the environment variables that need to be set before starting HBase.

vim /opt/hbase-2.1.5/conf/hbase-env.sh

This file already has a line for setting the JAVA_HOME environment variable. You just need to uncomment this line and change its value to point to the Java Home folder on your machine.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/

Run the HBase Server

Switch to the "hbase" user and run the "start-hbase.sh" shell script.

su hbase
/opt/hbase-2.1.5/bin/start-hbase.sh

An HBase instance should now be running in the background. You can use the shell script "stop-hbase.sh" to stop it when you want.

Configure JanusGraph to Use HBase

Open the file "/opt/janusgraph-0.5.2/conf/gremlin-server/gremlin-server.yaml" for editing. And change the value of "graphs > graph" as shown below.

graphs: {
  graph: conf/janusgraph-hbase.properties
}

This will instruct JanusGraph to use HBase as its storage backend. You can look at the file "/opt/janusgraph-0.5.2/conf/janusgraph-hbase.properties" to see the configuration details.

Start the JanusGraph Server

Switch to the "janus" user that we created in a previous article in this series. Then execute "gremlin-server.sh".

su janus
/opt/janusgraph-0.5.2/bin/gremlin-server.sh

The screenshot above shows the JanusGraph server output. The most important piece of information is the port number that JanusGraph is listening to.

Test JanusGraph From the Gremlin Console

Open a new terminal window. Switch to the "janus" user. Then run the Gremlin Console.

su janus
/opt/janusgraph-0.5.2/bin/gremlin.sh

Connect the Gremlin Console to the JanusGraph server.

:remote connect tinkerpop.server conf/remote.yaml

Then enter

:remote console

to send all the following commands to the server without having to precede them with :>

Let's check the string representation of the graph object to make sure that JanusGraph is configured to use HBase.

gremlin> graph
==>standardjanusgraph[hbase:[127.0.0.1]]

Create two vertices and an edge to connect them.

gremlin> g.addV('person').property('name', 'p1')
==>v[4208]
gremlin> g.addV('person').property('name', 'p2')
==>v[4192]
gremlin> g.addE('knows').from(g.V(4208)).to(g.V(4192))
==>e[1la-38w-2dx-38g][4208-knows->4192]

Since these commands were executed successfully, this means that JanusGraph is talking to HBase without problems.

View the Graph Data Saved In HBase

Open another terminal window. Switch to the "hbase" user. And run the "HBase shell".

su hbase
/opt/hbase-2.1.5/bin/hbase shell

Now let's use the list command to list all the tables in HBase.

hbase(main):001:0> list
TABLE
janusgraph
1 row(s)
Took 0.3954 seconds
=> ["janusgraph"]

You see that JanusGraph created a table "janusgraph". Let's use the describe command to get the description of the table.

hbase(main):002:0> describe 'janusgraph'
Table janusgraph is ENABLED
janusgraph
COLUMN FAMILIES DESCRIPTION
{NAME => 'e', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'f', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'g', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'h', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'i', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'l', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => '604800 SECONDS (7 DAYS)', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 'm', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 's', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
{NAME => 't', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS =>
'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'GZ', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
9 row(s)
Took 0.1346 seconds

Now let's use the scan command to get the data in this table.

hbase(main):003:0> scan 'janusgraph'
ROW                                      COLUMN+CELL
 \x00\x00\x00\x00\x00\x00\x00\x03        column=i:\xFF\xFF\xFF\xFF\xFF\xFE\xC7\x7F\x00\x00\x01wP\x91\xF5\xA17f000101604-Bassem-Laptop1, timestamp=1611964478
                                         881, value=
 \x00\x00\x00\x00\x00\x00\x00\x04        column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x9B\x00\x00\x01wP\x91\xF4M7f000101604-Bassem-Laptop1, timestamp=1611964478541
                                         , value=
 \x00\x00\x00\x00\x00\x00\x00\x04        column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xCD\x00\x00\x01wP\x91\xF2\xF67f000101604-Bassem-Laptop1, timestamp=1611964478
                                         199, value=
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x02, timestamp=1611964480067, value=\x00\x01\x08\x80
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x10\xC0, timestamp=1611964480067, value=\xA0vl\x1Eperso\xEE\x04\x80
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x10\xC2\x80\x18\x00, timestamp=1611964480067, value=\x8F\x00\x01\x8E\x00\x8F\x80
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x10\xC2\x80\x1C\x00, timestamp=1611964480067, value=\x8F\x00\x01\x8E\x00\x90\x80
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x10\xC4, timestamp=1611964480067, value=\x00\x82\x0C\x80
 \x00\x00\x00\x00\x00\x00\x02\x0D        column=e:\x10\xC8, timestamp=1611964480067, value=\x00\x80\x00\x01wP\x91\xF6\xFE\x14\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x02, timestamp=1611964480067, value=\x00\x01$\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC0, timestamp=1611964480067, value=\xA0rt\x1Enam\xE5 \x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x804\x00, timestamp=1611964480067, value=\x8F\x00\x01\x8E\x00\x80\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x808\x00, timestamp=1611964480067, value=\x99\x81\x01\x8E\x00\x81\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x80<\x00, timestamp=1611964480067, value=\xAD\x80\x01\x8E\x00\x82\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x80@\x00, timestamp=1611964480067, value=\x99\x81\x01\x8E\x00\x83\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x80D\x00, timestamp=1611964480067, value=\xAE\x83\x01\x8E\x00\x84\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x80H\x00, timestamp=1611964480067, value=\xBC\x92\x01\x8E\x00\x85\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC2\x80L\x00, timestamp=1611964480067, value=\xB3\x82\x01\x8E\x00\x87\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC4, timestamp=1611964480067, value=\x00\x81(\x80
 \x00\x00\x00\x00\x00\x00\x08\x05        column=e:\x10\xC8, timestamp=1611964480067, value=\x00\x80\x00\x01wP\x91\xF9\xB90\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x02, timestamp=1611964537095, value=\x00\x01T\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC0, timestamp=1611964537095, value=\xA0rt\x1Eknow\xF3P\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80d\x00, timestamp=1611964537095, value=\x8F\x00\x01\x8E\x00\x80\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80h\x00, timestamp=1611964537095, value=\x99\x81\x01\x8E\x00\x81\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80l\x00, timestamp=1611964537095, value=\xAD\x80\x01\x8E\x00\x82\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80p\x00, timestamp=1611964537095, value=\x99\x81\x01\x8E\x00\x83\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80t\x00, timestamp=1611964537095, value=\xAE\x80\x01\x8E\x00\x84\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80x\x00, timestamp=1611964537095, value=\xB0\x82\x01\x8E\x00\x86\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC2\x80|\x00, timestamp=1611964537095, value=\xB3\x82\x01\x8E\x00\x87\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC4, timestamp=1611964537095, value=\x00\x80X\x80
 \x00\x00\x00\x00\x00\x00\x0C\x15        column=e:\x10\xC8, timestamp=1611964537095, value=\x00\x80\x00\x01wP\x92\xD8\x95`\x80
 3\x1B\x96|\x10\xA5\xA0rt\x1Enam\xE5     column=g:\x00, timestamp=1611964480067, value=\x10\x85
 CWc\x02\x11\xA5\x81                     column=g:\x00\x10\x85(\x80, timestamp=1611964480067, value=\x10\x85
 `\x00\x00\x00\x00\x00\x00\x00           column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xD8\xEF\x00\x00\x01wP\x92\x1A-7f000101604-Bassem-Laptop1, timestamp=1611964488237
                                         , value=
 `\x00\x00\x00\x00\x00\x00\x03           column=i:\xFF\xFF\xFF\xFF\xFF\xFE\xC7\x7F\x00\x00\x01wP\x92\x1B\x817f000101604-Bassem-Laptop1, timestamp=1611964488
                                         577, value=
 `\x00\x00\x00\x00\x00\x00\x80           column=e:\x02, timestamp=1611964488937, value=\x00\x01\x04\x8C
 `\x00\x00\x00\x00\x00\x00\x80           column=e:$, timestamp=1611964488937, value=\x04\x8D\x08\x8C\xFF
 `\x00\x00\x00\x00\x00\x00\x80           column=e:P\xC0, timestamp=1611964488937, value=\xA0p\xB2\x0C\x8C
 `\x00\x00\x00\x00\x00\x00\x80           column=e:p\xE1\x80 p\x80\x10\x0E, timestamp=1611964536987, value=
 configuration                           column=s:cache.db-cache, timestamp=1611963652207001, value=\x8F\x01
 configuration                           column=s:cache.db-cache-clean-wait, timestamp=1611963652282001, value=\x8C\xA8
 configuration                           column=s:cache.db-cache-size, timestamp=1611963652274001, value=\x94?\xE0\x00\x00\x00\x00\x00\x00
 configuration                           column=s:cache.db-cache-time, timestamp=1611963652277001, value=\x8D\x80\x00\x00\x00\x00\x02\xBF
 configuration                           column=s:graph.janusgraph-version, timestamp=1611963652291001, value=\x92\xA00.5.\xB2
 configuration                           column=s:graph.storage-version, timestamp=1611963652298001, value=\x92\xA0\xB2
 configuration                           column=s:graph.timestamps, timestamp=1611963652302001, value=\xB6\x82
 configuration                           column=s:hidden.frozen, timestamp=1611963652313001, value=\x8F\x01
 configuration                           column=s:system-registration.7f000101604-Bassem-Laptop1.startup-time, timestamp=1611963670839, value=\xC1\x80\x00\x
                                         00\x00`\x14\x9D\x16\x06 \x10\x7F\x80
 c\x83\x94r\x11\xA5\x80                  column=g:\x00\x18\x95X\x80, timestamp=1611964537095, value=\x18\x95
 p\x00\x00\x00\x00\x00\x00\x00           column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xD8\xEF\x00\x00\x01wP\x91\xF7\x067f000101604-Bassem-Laptop1, timestamp=1611964479
                                         239, value=
 p\x00\x00\x00\x00\x00\x00\x03           column=i:\xFF\xFF\xFF\xFF\xFF\xFE\xC7\x7F\x00\x00\x01wP\x91\xF8Z7f000101604-Bassem-Laptop1, timestamp=1611964479579
                                         , value=
 p\x00\x00\x00\x00\x00\x00\x80           column=e:\x02, timestamp=1611964479943, value=\x00\x01\x04\x8E
 p\x00\x00\x00\x00\x00\x00\x80           column=e:$, timestamp=1611964479943, value=\x04\x8D\x08\x8E\xFF
 p\x00\x00\x00\x00\x00\x00\x80           column=e:P\xC0, timestamp=1611964479943, value=\xA0p\xB1\x0C\x8E
 p\x00\x00\x00\x00\x00\x00\x80           column=e:p\xE0\x80 `\x80\x10\x0E, timestamp=1611964536987, value=
 \xAB\xBA\x8B\x5C\x10\xA5\xA0rt\x1Eknow\ column=g:\x00, timestamp=1611964537095, value=\x18\x95
 xF3
 \xFA<[T\x11\xA5\x82                     column=g:\x00\x04\x8D\x0C\x80, timestamp=1611964480067, value=\x04\x8D
 \xFA\x8F\xE4\xF9\x10\xA5\xA0vl\x1Eperso column=g:\x00, timestamp=1611964480067, value=\x04\x8D
 \xEE
18 row(s)
Took 0.1781 seconds

You see that it took 18 rows to store these two vertices, their properties and the edge connecting them. The data in the table is not human readable, but at least we know where it is.

Installing Apache Cassandra for JanusGraph

Bassem Naguib — Mon, 11 Jan 2021 06:53:09 +0000

Introduction

This is the third article in a series of articles about installing and configuring JanusGraph and its storage backends on a Linux Ubuntu server.

In this article, I will explain how to install and run the Apache Cassandra database. And how to configure JanusGraph to use it as its storage backend.

Download and Extract Cassandra

At the time of writing this article, JanusGraph's latest release page on GitHub says that JanusGraph is compatible with Cassandra 3.11.0. So I will get this version's download link from the Apache Cassandra archive, and I will use the wget shell command to download it to my /opt directory. Then I will use the tar command to extract the contents of the downloaded archive.

cd /opt
wget http://archive.apache.org/dist/cassandra/3.11.0/apache-cassandra-3.11.0-bin.tar.gz
tar -xf apache-cassandra-3.11.0-bin.tar.gz

Create a Linux User for Running Cassandra

I think it is a good idea to create a Linux user for running the Cassandra server. This user can be given only the permissions needed by Cassandra and not more. This dedicated user will also make it easier to find the Cassandra server process if we need to monitor it or kill it.

adduser cassandra

You will be prompted to enter the password for this new user.

Now let's make this newly created user the owner of the Cassandra root folder and all its contents.

chown -R cassandra:cassandra apache-cassandra-3.11.0

Run the Cassandra Server

Switch to the "cassandra" user and run the "cassandra" shell script.

su cassandra
/opt/apache-cassandra-3.11.0/bin/cassandra -f

The -f flag runs the Cassandra server in the foreground so we can see the output messages and easily shutdown the server when we want by pressing CTRL + c

The screenshot above shows the Cassandra server output. The most important piece of information is the port number that Cassandra is listening to.

Configure JanusGraph to Use Cassandra

Open a new terminal window. Then open the file /opt/janusgraph-0.5.2/conf/gremlin-server/gremlin-server.yaml for editing. And change the value of graphs > graph as shown below.

graphs: {
  graph: conf/janusgraph-cql.properties
}

Note: Use janusgraph-cql.properties instead of janusgraph-cassandra.properties. The former instructs JanusGraph to use Cassandra's recommended CQL protocol. While the latter instructs JanusGraph to use Cassandra's deprecated Thrift protocol.

Start the JanusGraph Server

Switch to the "janus" user that we created in a previous article in this series. Then execute "gremlin-server.sh".

su janus
/opt/janusgraph-0.5.2/bin/gremlin-server.sh

The screenshot above shows the JanusGraph server output. The most important piece of information is the port number that JanusGraph is listening to.

Test JanusGraph From the Gremlin Console

Open a new terminal window. Switch to the "janus" user. Then run the Gremlin Console.

su janus
/opt/janusgraph-0.5.2/bin/gremlin.sh

Connect the Gremlin Console to the JanusGraph server.

:remote connect tinkerpop.server conf/remote.yaml

Then enter

:remote console

to send all the following commands to the server without having to precede them with :>

Let's check the string representation of the graph object to make sure that JanusGraph is configured to use Cassandra/CQL.

gremlin> graph
==>standardjanusgraph[cql:[127.0.0.1]]

Create two vertices and an edge to connect them.

gremlin> g.addV('person').property('name', 'p1')
==>v[4160]
gremlin> g.addV('person').property('name', 'p2')
==>v[4144]
gremlin> g.addE('knows').from(g.V(4160)).to(g.V(4144))
==>e[1l4-37k-2dx-374][4160-knows->4144]

Since these commands were executed successfully, this means that JanusGraph is talking to Cassandra without problems.

View the Graph Data Saved In Cassandra

Open yet another terminal window. Switch to the "cassandra" user. And run "cqlsh" which is a Cassandra client. Note that you will need to install Python if it is not already installed.

su cassandra
/opt/apache-cassandra-3.11.0/bin/cqlsh

Get the list of keyspaces.

cqlsh> DESCRIBE KEYSPACES;

system_schema  system      system_distributed
system_auth    janusgraph  system_traces

You see that JanusGraph created a keyspace "janusgraph". Let's use this keyspace and get the list of tables under it.

cqlsh> USE janusgraph;
cqlsh:janusgraph> DESCRIBE TABLES;

edgestore_lock_  graphindex_lock_         janusgraph_ids
txlog            systemlog                graphindex
edgestore        system_properties_lock_  system_properties

The most important table is "edgestore". This is where all the graph data (vertices, edges and properties) are stored. So let's view this table data.

cqlsh:janusgraph> SELECT * FROM edgestore;

 key                | column1            | value
--------------------+--------------------+----------------------------
 0x0000000000000c15 |               0x02 |                 0x00015480
 0x0000000000000c15 |             0x10c0 |   0xa072741e6b6e6f77ee5080
 0x0000000000000c15 |       0x10c2806400 |           0x8f00018e008080
 0x0000000000000c15 |       0x10c2806800 |           0x9981018e008180
 0x0000000000000c15 |       0x10c2806c00 |           0xad80018e008280
 0x0000000000000c15 |       0x10c2807000 |           0x9981018e008380
 0x0000000000000c15 |       0x10c2807400 |           0xae80018e008480
 0x0000000000000c15 |       0x10c2807800 |           0xb082018e008680
 0x0000000000000c15 |       0x10c2807c00 |           0xb382018e008780
 0x0000000000000c15 |             0x10c4 |                 0x00805880
 0x0000000000000c15 |             0x10c8 |   0x008005b873e162aaf86080
 0x4000000000000080 |               0x02 |                 0x00010488
 0x4000000000000080 |               0x24 |               0x048d0888ff
 0x4000000000000080 |             0x50c0 |               0xa070b10c88
 0x4000000000000080 | 0x70e0802030801008 |                         0x
 0x0000000000000805 |               0x02 |                 0x00012480
 0x0000000000000805 |             0x10c0 |     0xa072741e6e616de52080
 0x0000000000000805 |       0x10c2803400 |           0x8f00018e008080
 0x0000000000000805 |       0x10c2803800 |           0x9981018e008180
 0x0000000000000805 |       0x10c2803c00 |           0xad80018e008280
 0x0000000000000805 |       0x10c2804000 |           0x9981018e008380
 0x0000000000000805 |       0x10c2804400 |           0xae83018e008480
 0x0000000000000805 |       0x10c2804800 |           0xbc92018e008580
 0x0000000000000805 |       0x10c2804c00 |           0xb382018e008780
 0x0000000000000805 |             0x10c4 |                 0x00812880
 0x0000000000000805 |             0x10c8 |   0x008005b873dcbb75c03080
 0x3000000000000080 |               0x02 |                 0x00010486
 0x3000000000000080 |               0x24 |               0x048d0886ff
 0x3000000000000080 |             0x50c0 |               0xa070b20c86
 0x3000000000000080 | 0x70e1802040801008 |                         0x
 0x000000000000020d |               0x02 |                 0x00010880
 0x000000000000020d |             0x10c0 | 0xa0766c1e706572736fee0480
 0x000000000000020d |       0x10c2801800 |           0x8f00018e008f80
 0x000000000000020d |       0x10c2801c00 |           0x8f00018e009080
 0x000000000000020d |             0x10c4 |                 0x00820c80
 0x000000000000020d |             0x10c8 |   0x008005b873dcb0e2b81480

(36 rows)

The data is not human readable. But at least we know where it is.

Configuring JanusGraph to Use Oracle Berkeley DB

Bassem Naguib — Sat, 26 Dec 2020 22:56:45 +0000

Introduction

This is the second article in a series of articles about installing and configuring JanusGraph and its storage backends on a Linux Ubuntu server.

In this article, I will explain how to configure JanusGraph to use "Oracle Berkeley DB Java Edition" as its storage backend.

About Oracle Berkeley DB

Oracle Berkeley DB is an embedded key-value database library. The words "embedded" and "library" mean that Berkeley DB will be loaded and executed in the JanusGraph process as opposed to running in a separate process. So Berkeley DB is a database management library, not a stand-alone database server like Cassandra or HBase.

The Berkeley DB JAR file is included in the JanusGraph package that we downloaded in the previous article. The path to the JAR file is "lib/je-18.3.12.jar". So there is no need to download or install Berkeley DB separately.

Configure JanusGraph to Use Berkeley DB

Open the file "conf/gremlin-server/gremlin-server.yaml" for editing (I suggest you backup the original file before making any changes). Then change the "graph" configuration value from

graphs: {
  graph: conf/janusgraph-inmemory.properties
}

graphs: {
  graph: conf/janusgraph-berkeleyje-lucene.properties
}

Now the yaml file is pointing to another configuration file "conf/janusgraph-berkeleyje-lucene.properties" which tells JanusGraph to use "Oracle Berkeley DB Java Edition" as the storage backend. And "Apache Lucene" as the mixed index backend. In the next section, I will make a couple of small tweaks to this "properties" file.

Change the Data and Index Storage Directories

Open the file "conf/janusgraph-berkeleyje-lucene.properties". There is a line in this file that specifies the directory where Berkeley DB should place the database data files.

storage.directory=../db/berkeley

All paths are relative to the JanusGraph root folder. So the default configuration will place the database data files outside the JanusGraph root folder. This will not work for me because I like to run JanusGraph with a special user that is not allowed to write outside the JanusGraph root folder. So I will replace the .. in the path by . to place the database folder and files under the JanusGraph folder.

storage.directory=./db/berkeley

I also need to change the path to the mixed index data directory. So I will change the configuration line

index.search.directory=../db/searchindex

index.search.directory=./db/searchindex

Now the JanusGraph server process will not need to write anything outside the JanusGraph root folder.

Run and Test the JanusGraph Server

Execute the shell script "bin/gremlin-server.sh" to start the JanusGraph server. Then open another terminal window and execute the shell script "bin/gremlin.sh" to start the Gremlin console.

Type this command in the Gremlin console to connect it to the server.

:remote connect tinkerpop.server conf/remote.yaml

Then type this command to send all the following commands to the server without having to precede them with :>

:remote console

The server should create a graph and a g object based on the configuration. Let's make sure these variables are defined.

gremlin> graph
==>standardjanusgraph[berkeleyje:./db/berkeley]
gremlin> g
==>graphtraversalsource[standardjanusgraph[berkeleyje:./db/berkeley], standard]

The string representation of the graph object indicates that it is using Berkeley DB and the storage directory that we specified in the configuration. Perfect!

Let's make sure that the graph is initially empty.

gremlin> g.V().count()
==>0

Then add a couple of vertices

gremlin> g.addV('person').property('name', 'p1')
==>v[4136]
gremlin> g.addV('person').property('name', 'p2')
==>v[4152]

Ensure That the Graph Data Is Persisted

Switch to the terminal that was used to start the JanusGraph server and press CTRL + c to shutdown the server. Then run "bin/gremlin-server.sh" to start it again.

Go to the Gremlin Console terminal. Close the connection to the server, then reconnect.

:remote close
:remote connect tinkerpop.server conf/remote.yaml
:remote console

Now let's see if the two vertices we added are still remembered after the server restart.

gremlin> g.V().count()
==>2

The result shows that the vertices were remembered. This is the expected result because we configured JanusGraph to use a storage backend that persists the graph data.

You can explore the file system to see the folders that were created under the JanusGraph root folder to store the graph data and the mixed index data. The folder "db/berkeley" contains a few files. This is where the graph data (vertices, edges, properties, ...) is stored. These files are managed by Berkeley DB. The other folder "db/searchindex" is empty because we did not create any mixed indexes. But if we do, Lucene will write the mixed index data under this folder.

Critique of Berkeley DB

I am currently planning for an application that will use a graph database. And I will probably use JanusGraph unless I find a better alternative. I am still trying to decide on the storage backend that I will use with JanusGraph. My database/graph will be small enough to fit in one machine, so I do not think I will need horizontal scalability. So is Berkeley DB a good storage backend for my use case? I still do not know. But from my research so far, I found some pros and cons.

The Pros

Berkeley DB runs in the same process as JanusGraph. Other storage backends run as servers and listen to a port. So using Berkeley DB will save the overhead of communication between JanusGraph and the storage backend server.
Open source with a very permissive license (Apache 2.0). Note that I am talking about "Oracle Berkeley DB Java Edition". The other members of the Oracle Berkeley DB family have a much more restrictive license (AGPL 3). But I only care about the Java Edition because it is the only one that can be used with JanusGraph.

The Cons

Not future-proof at all. It seems like dead software. Even Oracle does not care very much about it. You can download Berkeley DB's source code from Oracle's website. But I could not find any public repository. There is a discussion board for the product, but it is very inactive. The latest version downloadable from Oracle's website is 7.5.11. I have no idea when this version was released. Probably a long time ago. The version of the Berkeley DB library that comes with JanusGraph is 18.3.12. The JanusGraph developers got this JAR file from Maven Central.

This all seems messy to me. So I do not think I will use Berkeley DB in production unless it turns out to be much faster than Cassandra and HBase in my use case.

Installing JanusGraph and Testing it With the InMemory Storage Backend

Bassem Naguib — Thu, 17 Dec 2020 02:44:19 +0000

Introduction

In this article series, I will explain how to install JanusGraph on a Linux Ubuntu server. And how to install three of its officially supported storage backends (Apache Cassandra, Apache HBase, and Oracle Berkeley DB Java Edition). I will also show how to configure JanusGraph to specify the storage backend we want to use.

In this first article, I will explain how to install JanusGraph itself and how to test it with the InMemory storage backend.

Install Java 8 JRE

JanusGraph is a Java application. So we need to have the Java JRE or JDK installed on the server to be able to run JanusGraph. At the time of writing this article, JanusGraph latest release page on Github tells us that it was tested with Java 1.8 (which is the same as Java 8). So it is better to install this particular version on the server.

I do not remember the exact package names for Java 8. So I will start by running the apt search command.

apt search openjdk

The following image shows some of the output of this command.

The "JDK" (Java Development Kit) is what you install when you need to compile and run Java applications. But in our case, we will download a pre-compiled version of JanusGraph and we will just need to run it. So the "JRE" (Java Runtime Environment) will be enough. Now the choice is between the "JRE" and the "JRE headless" packages. Normally, on servers, you install the headless version which is a minimal version of Java that should be enough for executing non-GUI applications. But for some reason, JanusGraph gave me errors when I tried to run it on the headless version of the JRE. So I ended up having to install the full JRE.

Most of the following shell commands will require Superuser privileges. So switch to the Superuser then enter the command to install the JRE.

sudo su
apt install openjdk-8-jre

Download and Extract JanusGraph

I place prepackaged application bundles under "/opt". So you can go to this directory or any other directory you like. Then use the "wget" command to download the JanusGraph ZIP file from Github. Then the "unzip" command to extract the files from the ZIP archive.

cd /opt
wget https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-0.5.2.zip
unzip janusgraph-0.5.2.zip

You can get the download link from JanusGraph's latest release page on Github. And you may need to install the "unzip" package if you do not already have it.

Create a Linux User for Running JanusGraph

I think it is a good idea to create a Linux user for running the JanusGraph server. This user can be given only the permissions needed by JanusGraph and not more. This dedicated user will also make it easier to find the JanusGraph server process if we need to monitor it or kill it.

adduser janus

You will be prompted to enter the password for this new user.

Now let's make this newly created user the owner of the JanusGraph directory and all its contents.

chown -R janus:janus janusgraph-0.5.2

Run the JanusGraph Gremlin Server

Switch to the "janus" user and run the "gremlin-server.sh" shell script

su janus
/opt/janusgraph-0.5.2/bin/gremlin-server.sh

This will start the Gremlin Server and it will listen to port 8182. The following image shows the output of the Gremlin Server.

Test the Server From the Gremlin Console

Keep the first terminal window/tab open to see the server output logs. And open another terminal. Switch to the "janus" user. And run the JanusGraph Gremlin Console.

su janus
/opt/janusgraph-0.5.2/bin/gremlin.sh

To connect the Gremlin Console to the Gremlin Server running on the same machine, enter the following command in the Gremlin Console.

:remote connect tinkerpop.server conf/remote.yaml

I suggest you look at the configuration file "remote.yaml" under "/opt/janusgraph-0.5.2/conf/". The most important lines in this file tell the Gremlin Console the hostname "localhost" and the port number "8182" that the server is running on.

To make the Gremlin Console send a command to the server, you need to put :> in front of the command. For example

:> 1 + 1

If you want to make the Gremlin Console send all the commands to the server without having to remember to add :>, then you can use the following command

:remote console

You do not need to create the "Graph" and the "Graph traversal source" objects. The server already has the variables graph and g defined for you.

gremlin> graph
==>standardjanusgraph[inmemory:[127.0.0.1]]
gremlin> g
==>graphtraversalsource[standardjanusgraph[inmemory:[127.0.0.1]], standard]

The graph will be initially empty

gremlin> g.V().count()
==>0

So let's create a couple of vertices

g.addV('person').property('name', 'p1')
g.addV('person').property('name', 'p2')

And make sure they were saved

gremlin> g.V().count()
==>2

Bummer! The Data Is Not Persisted

The default server configuration file "janusgraph-0.5.2/conf/gremlin-server/gremlin-server.yaml" makes JanusGraph use the "InMemory Storage Backend". So the data is not written to any persistent storage. And when you shutdown the server, all the graph data will be lost. Let's test this!

Open the terminal window that was used to run the Gremlin Server and press CTRL + C on your keyboard to shutdown the server. Then enter the following command to start the server again.

/opt/janusgraph-0.5.2/bin/gremlin-server.sh

Then switch to the Gremlin Console terminal. Because we restarted the server, we will need to disconnect the Gremlin Console from the server then reconnect it. Then enter the :remote console command to send all subsequent commands to the server.

:remote close
:remote connect tinkerpop.server conf/remote.yaml
:remote console

Now let's see the vertices count after the server restart

gremlin> g.V().count()
==>0

This confirms that all the data was lost when we restarted the server.

In the next articles, I will show how to make JanusGraph use storage backends that actually persist the graph data.