DEV Community: g-despot

Analyzing Real-Time Movie Reviews with Redpanda and Memgraph

g-despot — Sun, 05 Dec 2021 13:58:04 +0000

In recent years, it has become apparent that almost no production system is complete without real-time data. This can also be observed through the rise of streaming platforms such as Apache Kafka, Apache Pulsar, Redpanda, and RabbitMQ.

This tutorial focuses on processing real-time movie ratings that are streamed through Redpanda, a Kafka-compatible event streaming platform. The data can be used to generate movie recommendations with the help of Memgraph and the Cypher query language.

Prerequisites

To follow this tutorial, you will need:

Docker and Docker Compose (included in Docker Desktop for Windows and macOS)
Memgraph Lab - an application that can visualize graphs and execute Cypher queries in Memgraph.
A clone of the data-streams repository. This project contains the data stream, a Redpanda setup and Memgraph.

Data model

In this example, we will use the reduced MovieLens dataset streamed via Redpanda.
Each JSON message will be structured like the one below:

{
  "userId": "112",
  "movie": {
    "movieId": "4993",
    "title": "Lord of the Rings: The Fellowship of the Ring, The (2001)",
    "genres": ["Adventure", "Fantasy"]
  },
  "rating": "5",
  "timestamp": "1442535783"
}

So how are we going to store this data as a graph?
There are three different types of nodes: Movie, User, and Genre.

Each movie can be connected with an OF_GENRE edge to a different genre. A user can
rate movies, and these ratings will be modeled with the edge RATED. This edge contains the properties rating, which can range from 1.0 to 5.0, and timestamp.

Each Movie has the properties id and title while each User has the property id. A Genre only contains the property name.

1. Start the Redpanda stream

We created a Redpanda topic which you can connect to for the purpose of this tutorial. Clone the data-streams repository:

git clone https://github.com/memgraph/data-streams.git

Run the following command to start the Redpanda stream:

python start.py --platforms redpanda --dataset movielens

After the container starts, you should see messages being consumed in the console.

2. Start Memgraph

Usually, you would start Memgraph independently using Docker but this time we are going to use the data-streams project. Given that we need to access the data stream running in a separate Docker container, we need to run Memgraph on the same network.

1. Position yourself in the data-streams directory you cloned earlier.

2. Build the Memgraph image with:

docker-compose build memgraph-mage

3. Start the container:

docker-compose up memgraph-mage

Memgraph should be up and running. You can make sure by opening Memgraph Lab and connecting to the empty database.

3. Create the transformation module

Before we can connect to a data stream, we need to tell Memgraph how to transform the incoming messages, so they can be consumed correctly. This will be done through a simple Python transformation module:

import mgp
import json


@mgp.transformation
def rating(messages: mgp.Messages
             ) -> mgp.Record(query=str, parameters=mgp.Nullable[mgp.Map]):
    result_queries = []

    for i in range(messages.total_messages()):
        message = messages.message_at(i)
        movie_dict = json.loads(message.payload().decode('utf8'))
        result_queries.append(
            mgp.Record(
                query=("MERGE (u:User {id: $userId}) "
                       "MERGE (m:Movie {id: $movieId, title: $title}) "
                       "WITH u, m "
                       "UNWIND $genres as genre "
                       "MERGE (m)-[:OF_GENRE]->(:Genre {name: genre}) "
                       "CREATE (u)-[:RATED {rating: ToFloat($rating), timestamp: $timestamp}]->(m)"),
                parameters={
                    "userId": movie_dict["userId"],
                    "movieId": movie_dict["movie"]["movieId"],
                    "title": movie_dict["movie"]["title"],
                    "genres": movie_dict["movie"]["genres"],
                    "rating": movie_dict["rating"],
                    "timestamp": movie_dict["timestamp"]}))

    return result_queries

Each time we receive a JSON message, we need to execute a Cypher query that will map it to a graph object:

MERGE (u:User {id: $userId}) 
MERGE (m:Movie {id: $movieId, title: $title}) 
WITH u, m 
UNWIND $genres as genre 
MERGE (m)-[:OF_GENRE]->(:Genre {name: genre}) 
CREATE (u)-[:RATED {rating: ToFloat($rating), timestamp: $timestamp}]->(m)

This Cypher query creates a User and Movie if they are missing from the database. Movies are also connected to the genres they belong to. In the end, an edge of the type RATED is created between the user and the movie, indicating a rating.

Now that we have created the transformation module, another question arises. How to load a transformation module into Memgraph?

1. First, find the id of the container (CONTAINER_ID) where Memgraph is running:

docker ps

Note the id of the memgraph-mage container.

2. Now, you can copy the movielens.py transformation module to the Memgraph container with the following command:

docker cp movielens.py CONTAINER_ID:/usr/lib/memgraph/query_modules/movielens.py

3. Load the module with the following Cypher query:

CALL mg.load("movielens");

If you don't receive an error, the module was loaded successfully.

4. Connect to the Redpanda stream from Memgraph

1. Open Memgraph Lab and select the Query tab from the left sidebar.
2. Execute the following query in order to create the stream:

CREATE KAFKA STREAM movielens_stream 
TOPICS ratings
TRANSFORM movielens.rating 
BOOTSTRAP_SERVERS "redpanda:29092";

3. Now, that we have created the stream, it needs to be started in order to consume messages:

START STREAM movielens_stream;

4. It's time to check if the stream was created and started correctly:

SHOW STREAMS;

That's it! You just connected to a real-time data source with Memgraph and can start exploring the dataset. If you open the Overview tab in Memgraph Lab, you should see that a number of nodes and edges has already been created.

Just to be sure, open the tab Graph Schema and click on the generate button to see if the graph follows the Data model we defined at the beginning of the article.

5. Analyze the streaming data

For data analysis, we will use Cypher, the most popular query language when it comes to graph databases. It provides an intuitive way to work with property graphs. Even if you are not familiar with it, the following queries shouldn't be too hard to understand if you have some knowledge of SQL.

1. Let's return 10 movies from the database:

MATCH (movie:Movie)
RETURN movie.title
LIMIT 10;

2. Find movies that are of genre Adventure and Fantasy:

MATCH (movie:Movie)-[:OF_GENRE]->(:Genre {name:"Fantasy"})
MATCH (movie)-[:OF_GENRE]->(:Genre {name:"Adventure"})
RETURN movie.title
ORDER BY movie.title
LIMIT 10;

3. Calculate the average rating score for the movie Matrix:

MATCH (:User)-[r:RATED]->(m:Movie)
WHERE m.title = "Matrix, The (1999)"
RETURN avg(r.rating)

4. It's time for a more serious query. Let's find a recommendation for a specific user, for example, with the id 6:

MATCH (u:User {id: "6"})-[r:RATED]-(p:Movie)
      -[other_r:RATED]-(other:User)
WITH other.id AS other_id,
     avg(r.rating-other_r.rating) AS similarity,
     count(*) AS similar_user_count
ORDER BY similarity
LIMIT 10
WITH collect(other_id) AS similar_user_set
MATCH (some_movie: Movie)-[fellow_rate:RATED]-(fellow_user:User)
WHERE fellow_user.id IN similar_user_set
WITH some_movie, avg(fellow_rate.rating) AS prediction_score
RETURN some_movie.title AS Title, prediction_score
ORDER BY prediction_score DESC;

And that's it, you have generated recommendations based on the similarity of ratings between each user. If you want to find out more about this query, definitely check out our tutorial where we go more into detail.

Conclusion

Analyzing real-time data from streaming platforms has never been easier. This also applies to graph analytics, as we have demonstrated through the use of Redpanda and Memgraph. By applying network analytics and graph databases on streaming data, we can uncover hidden insights almost instantaneously while not sacrificing performance.

If you have any questions or comments, check out the Memgraph Discord server or leave a post on the forum.

The Cypher Query Language - Best Practices

g-despot — Wed, 20 Oct 2021 06:31:59 +0000

Cypher, like any other programming or query language, has a defined set of rules for writing readable and well-designed constructs. By following this guide, you will learn how to format and organize Cypher queries so that naming conventions and formatting are consistent and understandable to everyone.

Don't worry if you are still new to graph databases or Cypher. Most example queries are very easy to understand just from context and don't require any advanced knowledge.

Data model styling

Defining your data model is one of the most important steps when it comes to styling. Because these constructs will be used throughout your whole project, it would be very wise to think about the naming convention and formatting rules before committing to specific ones.
Sticking to shared best practices is by far the best choice because it enables other users to easily read and understand your data model and queries. In this section, you'll find recommendations on how to name nodes and relationships, their properties, variables... and so on.

Nodes

When it comes to nodes, the most important factor is styling labels. Node labels should be defined in CamelCase, which means that the first letter of each word begins with a capital letter. Because Cypher is case-sensitive, it is important to uphold this style throughout every query you write.

(:Country)
(:City)
(:CapitalCity)

Relationships

Relationship types are styled upper-case and use the underscore character _ to separate multiple words.

[:LIVES_IN]
(:BORDERS_WITH)

Properties, variables, parameters, aliases, and functions

Property keys, variables, parameters, aliases, and functions should have a camelCase style where the first letter of the word is lower-case, and the first letter of each following word is a capital letter. All of these constructs are case-sensitive, so capitalization must match either what is in the database (properties), what is already defined in the query (variables, parameters, aliases), or Cypher definitions (functions).

dateOfBirth // Property key
largestCountry // Variable
size() // Function
countryOne // Alias

Clauses

Clauses should be defined with capital letters, even if they consist of two or more words. Each new clause should be placed at the beginning of a new line to ensure complete readability. While clauses are not case sensitive, we strongly discourage you from using any other style to avoid confusion.

MATCH (c:Country)
WHERE c.name = 'UK'
RETURN c;

WITH "2021-01-01" as currentDate
MATCH (p:Person)
WHERE p.birthdate > currentDate
RETURN p.name;

Keywords

Aside from clauses, there is a number of keywords that should be in upper case even though they are not case sensitive. These include: DISTINCT, IN, STARTS WITH, CONTAINS, NOT, AND, OR and AS.

MATCH (c:Country)
WHERE c.name CONTAINS 'United' AND c.population > 9000000
RETURN c AS Country;

Indentations and line breaks

Sometimes it's helpful to separate new clauses with an indent. Even though they are in a new line, subqueries should be indented to ensure readability.
If there are multiple subqueries, they can be further grouped by using curly brackets.

//Indent 2 spaces on lines with ON CREATE or ON MATCH subqueries
MATCH (p:Person {name: 'Helga'})
MERGE (c:Country {name: 'UK'})
MERGE (p)-[l:LIVES_IN]->(c)
  ON CREATE SET l.movedIn = date({year: 2020})
  ON MATCH SET l.modified = date()
RETURN p, l, c;

//Indent 2 spaces with braces for subqueries
MATCH (p:Person)
WHERE EXISTS {
  MATCH (p)-->(c:Country)
  WHERE c.name = 'UK'
}
RETURN p;

An exception to this rule would be a one-line subquery where you don't need to use a new line or an indent.

MATCH (p:Person)
WHERE EXISTS { MATCH (p)-->(c:Country {name: 'UK'}) }
RETURN p

Metacharacters

Quotes

When it comes to quotes, a simple rule is to use whichever provides the fewest escaped characters in the string. If escaped characters are not needed or their number is the same for both single and double quotes, then single quotes should be favored.

// Bad syntax
RETURN 'Memgraph\'s mission is: ', "A very famous quote is: \"Astra inclinant, sed non obligant.\""

// Recommended syntax
RETURN "Memgraph's mission is: ", 'A very famous quote is: "Astra inclinant, sed non obligant."'

Semicolons

In most cases, a semicolon at the end of a Cypher query is unnecessary. The exception is when you have a script or a block with multiple separate Cypher queries that should be executed independently.

MATCH (c:Country {name: 'UK'})
RETURN c;

MATCH (c:Country {name: 'Germany'})
RETURN c;

Null and Boolean Values

Boolean literals and null values should always be lower case.

// Bad syntax
MATCH (c:Country)
WHERE c.island = NULL
  SET islandCountry = True
RETURN c

// Recommended syntax
MATCH (c:Country)
WHERE c.island = null
  SET islandCountry = true
RETURN c

Pattern styling

When you have a pattern that is too long for one line, the recommended practice is to break after an arrow, not before it.

// Bad syntax
MATCH (:Country)-->(:Person)-->(:Person)
      <--(c:Country)
RETURN c.name

// Recommended syntax
MATCH (:Country)-->(:Person)-->(:Person)<--
      (c:Country)
RETURN c.name

If you don't plan on using a variable in the query, it's better to use anonymous nodes and relationships instead.

// Bad syntax
MATCH (c:Country {name: 'UK'})<-[l:LIVES_IN]-(p:Person)
RETURN p.name

// Recommended syntax
MATCH (:Country {name: 'UK'})<-[:LIVES_IN]-(p:Person)
RETURN p.name

Nodes with assigned variables should come before anonymous nodes and relationships if possible. The same goes for nodes that are starting points or the central focus of the query.

// Bad syntax
MATCH (:Person)-[:LIVES_IN]->(c:Country {name: 'UK'})
RETURN c

// Recommended syntax
MATCH (c:Country {name: 'UK'})<-[:LIVES_IN]-(:Person)
RETURN c

Patterns should be ordered so that left-to-right relationships (arrows) come at the beginning of the query.

// Bad syntax
MATCH (c:Country)<-[:BORDERS_WITH]-(:Country)<-[:LIVES_IN]-(:Person)
RETURN c

// Recommended syntax
MATCH (:Person)-[:LIVES_IN]->(:Country)-[:BORDERS_WITH]->(c:Country)
RETURN c

Spacing

Spacing has a big impact on the readability of queries. Some of the recommended practices when it comes to spacing are:

There should be one space between property predicates and label/type predicates.

// Bad syntax
MATCH (:Country   {name: 'UK'})<-[:LIVES_IN{since: 2010}]-(p:Person)
RETURN p.name

// Recommended syntax
MATCH (:Country {name: 'UK'})<-[:LIVES_IN {since: 2010}]-(p:Person)
RETURN p.name

There should be no spaces in label predicates.

// Bad syntax
MATCH (c: Country:  City)
RETURN c.name

// Recommended syntax
MATCH (c:Country:City)
RETURN c.name

There should be no spaces in patterns.

// Bad syntax
MATCH (c:Country) --> (:City) 
RETURN c.name

// Recommended syntax
MATCH (c:Country)-->(:City) 
RETURN c.name

There should be one space on both sides of an operator.

// Bad syntax
MATCH (c:Country)
WHERE population>100000
RETURN c.name

// Recommended syntax
MATCH (c:Country)
WHERE population > 100000
RETURN c.name

There should be one space between elements in a list (after each comma).

// Bad syntax
WITH ['UK','US','Germany'] as list
MATCH (c:Country)
WHERE c.name IN list
RETURN c.name

// Recommended syntax
WITH ['UK', 'US', 'Germany'] as list
MATCH (c:Country)
WHERE c.name IN list
RETURN c.name

Function call parentheses should only have one space after each comma.

// Bad syntax
RETURN split( 'A', 'B' , 'C' )

// Recommended syntax
RETURN split('A', 'B', 'C')

Map literals should only have one space after each comma and one space separating a colon and a value.

// Bad syntax
WITH { name :'UK' ,population  :  70000000 } AS country
RETURN country

// Recommended syntax
WITH {name: 'UK', population: 70000000} AS country
RETURN country

Conclusion

Having a well-defined style for writing queries is a technical must-have. It makes it easier and faster to understand queries that other people have written while also enabling everyone to contribute in a meaningful way without having to worry about the format. Whenever you are not sure about a naming convention or the structure of your query, just take a quick look at this guide.

If you have trouble writing a specific query or would just like to learn about the concepts that are available in Cypher, go through our Cypher Cheat Sheet.

Hacktoberfest 2021 - Join the Festivities!

g-despot — Sat, 16 Oct 2021 12:31:09 +0000

The annual Hacktoberfest is
underway! We may be a bit late in joining the festivities because of our
Memgraph 2.0 launch, but
we are very much committed to furthering the open-source spirit and contributing
to such noteworthy efforts.

For those who are not familiar with the concept, the basic idea is:

Register on the Hacktoberfest website.
Open 4 pull requests during October 2021 in repositories with the tag hacktoberfest.
If successful and the PRs are merged, you can choose between planting a tree or receiving some swag!

What's more, you can also fix any bugs or provide better implementations
for certain features! Feel free to inform the community by filling an issue
and start hacking! If you're new to contributing code, you can always ask for
help on our Discord server.

Memgraph has repositories with different
programming languages and technology stacks, so find what works for you and
start coding there:

MAGE - an open-source graph algorithm library with algorithms written in C/C++/Python/Rust.
Memgraph - the Memgraph core repository for C/C++ wizards.
GQLAlchemy - a library developed to assist in writing and running queries on Memgraph written in Python.

If you want to contribute to any other of our repositories, just follow the same
principle and open an issue. We will handle the rest.

What can you do to participate?

Read the Contribution Guidelines in the selected repo.
Open an issue (for big ideas you may want the community to know) or comment on the Hacktoberfest 2021 issue for your new implementations, fixes, or functionalities.
Start hacking!
Open a pull request for your contribution.
Wait until your pull requests get reviewed or merged! In the meantime, you can review someone else's pull requests too.

Last remarks

Join our Discord server to get started. It's definitely easier with the help of the Memgraph engineering team or other community members.
As part of the event, we would like to invite everyone to adhere to Hacktoberfest's values and our Code of Conduct.
Please observe Hacktoberfest's rules, terms, and FAQ. If a pull request is marked invalid or spam it would not be counted, and the person may be disqualified from the event.
Hacktoberfest 2021 is an event presented by DigitalOcean, Intel, appwrite, and deepsource.

Understanding Community Detection Algorithms with Python NetworkX

g-despot — Tue, 06 Apr 2021 08:26:08 +0000

Introduction

While humans are very good at detecting distinct or repetitive patterns among a few components, the nature of large interconnected networks makes it practically impossible to perform such basic tasks manually. Groups of densely connected nodes are easy to spot visually, but more sophisticated methods are needed to perform these tasks programmatically. Community detection algorithms are used to find such groups of densely connected components in various networks.

M. Girvan and M. E. J. Newman have proposed one of the most widely adopted community detection algorithms, the Girvan-Newman algorithm. According to them, groups of nodes in a network are tightly connected within communities and loosely connected between communities.

In this article, you will learn the basic principles behind community detection algorithms, their specific implementations, and how you can run them using Python and NetworkX.

Community Detection Example Applications

Because networks are an integral part of many real-world problems, community detection algorithms have found their way into various fields, ranging from social network analysis to public health initiatives.

A known use case is the detection of terrorist groups in social networks by tracking their activities and interactions. Similarly, groups of malicious bots can be detected on online social platforms.
Community detection can be used to study the dynamics of certain groups that are susceptible to epidemic diseases. Other types of diseases can be studied similarly to discover common links among patients.
One of the most recent use cases, community evolution prediction, involves the prediction of changes in network structures.

Community Detection Techniques

There are two main types of community detection techniques, agglomerative and divisive.

Agglomerative methods generally start with a network that contains only nodes of the original graph. The edges are added one-by-one to the graph, while stronger edges are prioritized over weaker ones. The strength of an edge, or weight, is calculated differently depending on the specific algorithm implementation.

On the other hand, divisive methods rely on the process of removing edges from the original graph iteratively. Stronger edges are removed before weaker ones. At every step, the edge-weight calculation is repeated, since the weight of the remaining edges changes after an edge is removed. After a certain number of steps, we get clusters of densely connected nodes, a.k.a. communities.

Community Detection in NetworkX

Girvan-Newman algorithm: The Girvan-Newman algorithm detects communities by progressively removing edges from the original network.
Fluid Communities algorithm: This algorithm is based on the simple idea of fluids interacting in an environment, expanding and pushing each other.
Label Propagation algorithm: Label propagation is a semi-supervised machine learning algorithm that assigns labels to previously unlabeled data points.
Clique Percolation algorithm: The algorithm finds k-clique communities in a graph using the percolation method.
Kernighan-Lin algorithm: This algorithm partitions a network into two sets by iteratively swapping pairs of nodes to reduce the edge cut between the two sets.

Girvan-Newman Algorithm

For the detection and analysis of community structures, the Girvan-Newman algorithm relies on the iterative elimination of edges that have the highest number of shortest paths between nodes passing through them. By removing edges from the graph one-by-one, the network breaks down into smaller pieces, so-called communities. The algorithm was introduced by Michelle Girvan and Mark Newman.

How does it work?

The idea is to find which edges in a network occur most frequently between other pairs of nodes by finding edge betweenness centrailities. The edges joining communities are then expected to have a high edge betweenness. The underlying community structure of the network will be much more fine-grained once the edges with the highest betweenness are eliminated which means that communities will be much easier to spot.

The Girvan-Newman algorithm can be divided into four main steps:

For every edge in a graph, calculate the edge betweenness centrality.
Remove the edge with the highest betweenness centrality.
Calculate the betweenness centrality for every remaining edge.
Repeat steps 2–4 until there are no more edges left.

In this example, you can see how a typical graph looks like when edges are assigned weights based on the number of shortest paths passing through them. To keep things simple, we only calculated the number of undirected shortest paths that pass through an edge. The edge between nodes A and B has a strength/weight of 1 because we don’t count A->B and B->A as two different paths.

The Girvan-Newman algorithm would remove the edge between nodes C and D because it is the one with the highest strength. As you can see intuitively, this means that the edge is located between communities.
After removing an edge, the betweenness centrality has to be recalculated for every remaining edge. In this example, we have come to the point where every edge has the same betweenness centrality.

Betweenness centrality

Betweenness centrality measures the extent to which a node or edge lies on paths between nodes. Nodes and edges with high betweenness may have considerable influence within a network under their control over information passing between others.

The calculation of betweenness centrality is not standardized and there are many ways to solve it. It is defined as the number of shortest paths in the graph that passes through the node or edge divided by the total number of shortest paths.

In this example, you can see an undirected graph colored based on the betweenness centrality of each node from lowest (red) to highest (blue).

Pseudocode

In each iteration, calculate the betweenness centrality for all edges in the graph. Remove the edge with the highest centrality. Repeat until there are no more edges left.

REPEAT
    LET n BE number of edges in the graph
    FOR i=0 to n-1
        LET B[i] BE betweenness centrality of edge i
        IF B[i] > max_B THEN
            max_B = B[i]
            max_B_edge = i
        ENDIF
    ENDFOR
    REMOVE edge i FROM graph
UNTIL number of edges in graph is 0

Community Detection Algorithms in NetworkX

girvan_newman(G, most_valuable_edge=None)

Method input

The first input parameter of the method, G, is a NetworkX graph.
The second parameter, most_valuable_edge, is a function that takes a graph as input and returns the edge that should be removed from the graph in each iteration. If no function is specified, the edge with the highest betweenness centrality will be chosen in each iteration.

Method output

The output of the method is an iterator over tuples of sets of nodes in G. Each set of nodes represents a community and each tuple is a sequence of communities at a particular level (iteration) of the algorithm.

Example

import matplotlib.pyplot as plt
import networkx as nx
from networkx.algorithms.community.centrality import girvan_newman

G = nx.karate_club_graph()
communities = girvan_newman(G)

node_groups = []
for com in next(communities):
  node_groups.append(list(com))

print(node_groups)

color_map = []
for node in G:
    if node in node_groups[0]:
        color_map.append('blue')
    else: 
        color_map.append('green')  
nx.draw(G, node_color=color_map, with_labels=True)
plt.show()

The output is:

[[0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21], 
[2, 8, 9, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]]

The network has been divided into two distinct communities:

Community Detection in Memgraph

While the NetworkX package may be enough on its own to learn about graph theory and algorithms, in production, we often require a permanent storage solution. Using Memgraph, an in-memory graph database, as the storage solution provides additional benefits and functionalities to NetworkX.
In this section, we will focus on how to implement a custom Cypher procedure that uses query modules in Memgraph to perform community detection.

Creating a Custom Query Module

Memgraph supports extending the query language with user-written procedures. These procedures are grouped into modules (Query Modules), which can then be loaded on startup or later on. We are going to create such a procedure to work with the NetworkX package.

To get started, we need to create and mount a volume to access the query_modules directory. This directory contains all of the built-in query modules and it's where you save new custom query modules. Create an empty directory ~modules on your host machine and execute the following command:

docker volume create --driver local --opt type=none  --opt device=$(pwd)/modules --opt o=bind modules

Now, you can start Memgraph and mount the created volume:

docker run -it --rm -v modules:/usr/lib/memgraph/query_modules -p 7687:7687 memgraph

Everything from the directory /usr/lib/memgraph/query_modules will be visible/editable in your mounted modules volume and vice versa. If you take a look at the /query_modules directory, you will find built-in query modules that come prepackaged with Memgraph. To learn more about them, visit the How to Use Query Modules Provided by Memgraph? guide.

Create a file called community.py in the modules directory and copy the code below into it.

import mgp
import networkx as nx
from networkx.algorithms import community
from mgp_networkx import MemgraphMultiDiGraph


@mgp.read_proc
def detect(
    ctx: mgp.ProcCtx
    ) -> mgp.Record(communities=mgp.List[mgp.List[mgp.Vertex]]):

    networkxGraph = nx.DiGraph(MemgraphMultiDiGraph(ctx=ctx))
    communities_generator = community.girvan_newman(networkxGraph)

    return mgp.Record(communities=[
        list(s) for s in next(communities_generator)])

If you are using a Linux machine and don't have the right permissions to create a new file in your modules directory, run the following command:

sudo chown -R <user-name> modules

We just created a new query module with the procedure detect() that utilizes the Girvan–Newman method for finding communities in a graph. Before we can call it, the newly created query module has to be loaded. Execute the following command in Memgraph Lab:

CALL mg.load_all()

We need actual data in our database instance to try out the detect() procedure. In Memgraph Lab, navigate to the Datasets tab in the left sidebar and load the Capital cities and borders dataset.
Now, you can call the procedure with:

CALL community.detect() 
YIELD communities
UNWIND communities as community
RETURN community

The result is 52 communities with varying numbers of components. Some of the cities in the dataset have been grouped by continent, while some are isolated because of their remote island locations.

Conclusion

Community detection is a powerful tool for graph analysis. From terrorist detection to healthcare initiatives, these algorithms have found their way into many real-world use cases.

The Python NetworkX package offers powerful functionalities when it comes to analyzing graph networks and running complex algorithms like community detection. However, if you're looking to operationalize your graph algorithms and are looking for functionalities such as incremental updates, data persistency, and better performance, you will need to consider using a graph database in conjunction with NetworkX. If you want to find out more about using NetworkX algorithms in Memgraph, read our Reference Guide.

How to Develop a Credit Card Fraud Detection Application using Memgraph, Flask, and D3.js

g-despot — Fri, 12 Mar 2021 08:22:48 +0000

Introduction

There is a large and ever-growing number of use cases for graph databases and many of them are centered around one important functionality: relationship traversals. While in traditional relational databases the concept of foreign keys seems like a simple and efficient idea, the truth is that they result in very complex joins and self-joins when the dataset becomes too inter-related.

Graph databases offer powerful data modeling and analysis capabilities for many real-world problems such as social networks, business relationships, dependencies, shipping, logistics… and they have been adopted by many of the world's leading tech companies.

The use case you'll be working on is Fraud Detection in large transaction networks. Usually, such networks contain millions of relationships between POS devices, logged transactions, and credit cards which makes it a perfect target for graph database algorithms.

In this tutorial, you will learn how to build a simple Python web application from scratch. You will get a basic understanding of the technologies that are used, and see how easy it is to integrate a graph database in your development process.

You can also find all of the code here if you don't want to work on it as you go through the tutorial. If at any point in this tutorial you have a question or something is not working for you, feel free to post on StackOverflow with the tag memgraphdb.

Prerequisites

Since you will be building a complete web application there is a number of tools that you will need to install before getting started:

Flask: a very powerful web framework that provides you with tools, libraries and technologies used in web development. A Flask application can be as small as a single web page or as complex as a management interface.
Docker and Compose: an open platform for developing, shipping, and running applications. It enables you to separate your application from your infrastructure (host machine). If you are installing Docker on Windows, Compose will be already included. For Linux and macOS visit this site.
Memgraph DB: a native fully distributed in-memory graph database built to handle real-time use-cases at enterprise scale. Follow the Docker Installation instructions. While it's completely optional, I encourage you to also install Memgraph Lab so you can execute Cypher queries on the database directly and see visualized results.

Understanding the Payment Fraud Detection Scenario

First, let's define all the roles in this scenario:

Card - a credit card used for payment.
POS - a point of sale device that uses a card to execute transactions.
Transaction - a stored instance of buying something.

Your application will simulate how a POS device gets compromised, then a card in contact with that POS device gets compromised as well and in the end, a fraudulent transaction is reported.
Based on these reported transactions, Memgraph is used to search for the root-cause (a.k.a. the compromised POS) of the reported fraudulent transactions and all the cards that have fallen victim to it as shown below.

Because this is a demo application you will create a set number of random cards, POS devices, and transactions. Some of these POS devices will be marked as compromised. If you find a compromised POS device, while searching for frauds in the network, then you'll mark the card as compromised as well. If the card is compromised, there is a 0.1% chance the transaction is fraudulent and detected (regardless of the POS device). You can then visualize all the transactions and cards connected to that POS device and resolve them as not fraudulent if need be.

Defining the Graph Schema

After we defined the scenario, it's time to create the graph schema!

A graph schema is a "dictionary" that defines the types of entities, vertices, and edges, in the graph and how those types of entities are related to one another.

Ok, so you know that there are three main entities in your model: Card, POS and Transaction.

The next step is to determine how these entities are represented in the graph and how they are connected. If you are not familiar with graph databases, a good rule of thumb is to use a relational analogy to get you started. All of these entities would be separate tables in a relational database and therefore they could be separate types of nodes in a graph. And so it is!

Each type of node has a different label: Card, Pos and Transaction.
All of them have the property id so you can identify them. The nodes Card and Pos also have the boolean property compromised to indicate if fraudulent activity has taken place. The node Transaction has a similar boolean property with the name fraudReported.

But how are these nodes connected? The nodes labeled Card and Transaction are connected via a relationship of type :USING. Internalize the meaning by reading it out loud: a transaction is executed USING a card. In the same fashion, a transaction is executed AT a POS device so the relationship between Transaction and POS is of type :AT.

Using the Cypher query language notation, the data structure when there are no frauds looks like this:

(:Card {compromised:false})<-[:USING]-(:Transaction)-[:AT]->(:Pos {compromised: false})

The data structure when frauds occur:

(:Card {compromised:true})<-[:USING]-(:Transaction)-[:AT]->(:Pos {compromised: true})
(:Card {compromised:true})<-[:USING]-(:Transaction {fraudReported:true})-[:AT]->(:Pos)

Building the Web Application Backbone

This is presumably the easy part. You need to create a simple Python web application using Flask to be your server. Let's start by creating a root directory for your project and naming it card_fraud. There you need to create a requirements.txt file containing the necessary PIP installs. For now, only one line is needed:

Flask==1.1.2

You can install the specified package by running:

pip3 install -r requirements.txt

Add a new file to your root directory with the name card_fraud.py and the following code:

from flask import Flask

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
   return "Hello World"

You are probably rolling your eyes while reading this, but don't mock the Hello World example! Let's compile and run your server to see if everything works as expected. Open a terminal, position yourself in the root directory and execute the following two commands:

export FLASK_APP=card_fraud.py
export FLASK_ENV=development

This way, you have defined the entry point of your app and set the environment to development. This will enable development features like automatic code reloading. Don't forget to change this to production when you're ready to deploy your app. To run the server, execute:

flask run --host 0.0.0.0

You should see a message similar to the following, indicating that your server is up and running:

* Serving Flask app "card_fraud"
* Running on http://0.0.0.0:5000/

Dockerizing the Application

In the root directory of the project create two files, Dockerfile and docker-compose.yml. At the beginning of the Dockerfile, you specify the parent image and instruct the container to install CMake, mgclient, and pymgclient. CMake and mgclient are necessary to install pymgclient, the Python driver for Memgraph DB.

You don’t have to focus too much on this part, just copy the code to your Dockerfile:

FROM python:3.8

# Install CMake
RUN apt-get update && \
  apt-get --yes install cmake

# Install mgclient
RUN apt-get install -y git cmake make gcc g++ libssl-dev && \
  git clone https://github.com/memgraph/mgclient.git /mgclient && \
  cd mgclient && \
  git checkout dd5dcaaed5d7c8b275fbfd5d2ecbfc5006fa5826 && \
  mkdir build && \
  cd build && \
  cmake .. && \
  make && \
  make install

# Install pymgclient
RUN git clone https://github.com/memgraph/pymgclient /pymgclient && \
  cd pymgclient && \
  python3 setup.py build && \
  python3 setup.py install

# Install packages
COPY requirements.txt ./
RUN pip3 install -r requirements.txt

COPY card_fraud.py /app/card_fraud.py
WORKDIR /app

ENV FLASK_ENV=development
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

ENTRYPOINT ["python3", "card_fraud.py"]

If you are not familiar with Docker, do yourself a favor and take look at this: Getting started with Docker.

Next, you need to create a docker-compose.yml file. Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. For this project, you'll need two services. One is the web application (sng_demo) and the other a database instance (memgraph).

If you followed the instructions on how to setup Memgraph DB with Docker correctly you only need to add the following code to your docker-compose.yml file to run the container:

version: "3"
services:
  memgraph:
    image: "memgraph"
    ports:
      - "7687:7687"
  card_fraud:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MG_HOST: memgraph
      MG_PORT: 7687
    depends_on:
      - memgraph

When it comes to the ports key, there is an important distinction between the HOST_PORT and the CONTAINER_PORT. The first number in the key is the HOST_PORT and it can be used to connect from your host machine to the service (for example with Memgraph Lab). The second number specifies the CONTAINER_PORT which is used for service-to-service communication. More specifically, your service card_fraud can use this port to access the service memgraph and connect to the database.

The depends_on key is used to start services in dependency order because
need the database to start before the web application.

The build key allows us to tell Compose where to find the build instructions as well as the files and/or folders used during the build process. By using the volumes key, you bypass the need to constantly restart your image to load new changes to it from the host machine.

Congratulations, you now have a dockerized app! This approach is great for development because it enables you to run your project on completely different operating systems and environments without having to worry about compatibility issues.

To make sure we are on the same page, your project structure should look like this:

card_fraud
├── card_fraud.py
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

Let’s start your app to make sure you don’t have any errors. In the project root directory execute:

docker-compose build

The first build will take some time because Docker has to download and install a lot of dependencies. After it finishes run:

docker-compose up

The URL of your web application is http://0.0.0.0:5000/. You should see the message Hello World which means that the app is up and running correctly.

Defining the Bussines Logic

At this point, you have a basic web server and a database instance. It's time to add some useful functionalities to your app. To communicate with the database, your app needs some kind of OGM - Object Graph Mapping system. You can just reuse this one: custom OGM. Add the database directory with all of its contents to the root directory of your project.

Also, delete the contents of card_fraud.py because you are starting from scratch.

Let's fetch the environment variables you defined in the docker-compose.yml file by adding the following code in card_fraud.py:

import os

MG_HOST = os.getenv('MG_HOST', '127.0.0.1')
MG_PORT = int(os.getenv('MG_PORT', '7687'))
MG_USERNAME = os.getenv('MG_USERNAME', '')
MG_PASSWORD = os.getenv('MG_PASSWORD', '')
MG_ENCRYPTED = os.getenv('MG_ENCRYPT', 'false').lower() == 'true'

No web application is complete without logging, so let's at least add the bare minimum:

import logging
import time

log = logging.getLogger(__name__)

def init_log():
    logging.basicConfig(level=logging.INFO)
    log.info("Logging enabled")
    logging.getLogger("werkzeug").setLevel(logging.WARNING)

init_log()

It would also be convenient to add an input argument parser so you can run the app with different configurations without hardcoding them. Add the following import and function:

from argparse import ArgumentParser

def parse_args():
    '''
    Parse command-line arguments.
    '''
    parser = ArgumentParser(description=__doc__)
    parser.add_argument("--app-host", default="0.0.0.0",
                        help="Allowed host addresses.")
    parser.add_argument("--app-port", default=5000, type=int,
                        help="App port.")
    parser.add_argument("--template-folder", default="public/template",
                        help="The folder with flask templates.")
    parser.add_argument("--static-folder", default="public",
                        help="The folder with flask static files.")
    parser.add_argument("--debug", default=True, action="store_true",
                        help="Run web server in debug mode")
    parser.add_argument('--clean-on-start', action='store_true',
                        help='Should the DB be emptied on script start')
    print(__doc__)
    return parser.parse_args()


    args = parse_args()

Now, you can connect to your database and create an instance of a Flask server by adding the following code:

from flask import Flask, Response, request, render_template
from database import Memgraph

db = Memgraph(host=MG_HOST, port=MG_PORT, username=MG_USERNAME,
              password=MG_PASSWORD, encrypted=MG_ENCRYPTED)
app = Flask(__name__,
            template_folder=args.template_folder,
            static_folder=args.static_folder,
            static_url_path='')

Finally, you come to the business logic and all the interesting functions. Get ready because there are many things you need to implement. If you'd rather just copy them and read their descriptions later, that's fine too. You can find the complete card_fraud.py script here and can continue the tutorial on this section.

Clearing the Database

You need to start with an empty database so let's implement a function to drop all the existing data from it:

def clear_db():
    """Clear the database."""

    db.execute_query("MATCH (n) DETACH DELETE n")
    log.info("Database cleared")

Adding Initial Cards and POS Devices

There is a fixed number of initial cards and POS devices that need to be added to the database at the beginning.

def init_data(card_count, pos_count):
    """Populate the database with initial Card and POS device entries."""

    log.info("Initializing {} cards and {} POS devices".format(
        card_count, pos_count))
    start_time = time.time()

    db.execute_query("UNWIND range(0, {} - 1) AS id "
                     "CREATE (:Card {{id: id, compromised: false}})".format(
                         card_count))
    db.execute_query("UNWIND range(0, {} - 1) AS id "
                     "CREATE (:Pos {{id: id, compromised: false}})".format(
                         pos_count))

    log.info("Initialized data in %.2f sec", time.time() - start_time)

Adding a Single Compromised POS Device

You need the option of changing the property compromised of a POS device to true given that all of them are initialized as false at the beginning.

def compromise_pos(pos_id):
    """Mark a POS device as compromised."""

    db.execute_query(
        "MATCH (p:Pos {{id: {}}}) SET p.compromised = true".format(pos_id))
    log.info("Point of sale %d is compromised", pos_id)

Adding Multiple Random Compromised POS Devices

You can also compromise a set number of randomly selected POS devices at once.

from random import sample

def compromise_pos_devices(pos_count, fraud_count):
    """Compromise a number of random POS devices."""

    log.info("Compromising {} out of {} POS devices".format(
        fraud_count, pos_count))
    start_time = time.time()

    compromised_devices = sample(range(pos_count), fraud_count)
    for pos_id in compromised_devices:
        compromise_pos(pos_id)

    log.info("Compromisation took %.2f sec", time.time() - start_time)

Adding Credit Card Transactions

This is where the main analysis for fraud detection happens. If the POS device is compromised, then the card in the transaction gets compromised too. If the card is compromised, there is a 0.1% chance the transaction is fraudulent and detected (regardless of the POS device).

from random import randint

def pump_transactions(card_count, pos_count, tx_count, report_pct):
    """Create transactions. If the POS device is compromised, 
    then the card in the transaction gets compromised too. 
    If the card is compromised, there is a 0.1% chance the 
    The transaction is fraudulent and detected (regardless of 
    the POS device)."""

    log.info("Creating {} transactions".format(tx_count))
    start_time = time.time()

    query = ("MATCH (c:Card {{id: {}}}), (p:Pos {{id: {}}}) "
             "CREATE (t:Transaction "
             "{{id: {}, fraudReported: c.compromised AND (rand() < %f)}}) "
             "CREATE (c)<-[:Using]-(t)-[:At]->(p) "
             "SET c.compromised = p.compromised" % report_pct)

    def rint(max): return randint(0, max - 1)
    for i in range(tx_count):
        db.execute_query(query.format(rint(card_count),
                                      rint(pos_count),
                                      i))

    duration = time.time() - start_time
    log.info("Created %d transactions in %.2f seconds", tx_count, duration)

Resolving Transactions and Cards on a POS Device

You also need to have the functionality to resolve suspected fraud cases. This means marking all the connected components of a POS device as not compromised if they are cards and not fraudulent if they are transactions. This function is triggered by a POST request to the URL /resolve-pos. The request body contains the variable pos which specifies the id of the POS device.

import json

@app.route('/resolve-pos', methods=['POST'])
def resolve_pos():
    """Resolve a POS device and card as not compromised."""

    data = request.get_json(silent=True)
    start_time = time.time()

    db.execute_query("MATCH (p:Pos {{id: {}}}) "
                     "SET p.compromised = false "
                     "WITH p MATCH (p)--(t:Transaction)--(c:Card) "
                     "SET t.fraudReported = false, c.compromised = false".format(data['pos']))

    duration = time.time() - start_time
    log.info("Compromised Point of sale %s has been resolved in %.2f sec",
             data['pos'], duration)

    response = {"duration": duration}
    return Response(
        json.dumps(response),
        status=201,
        mimetype='application/json')

Fetching all Compromised POS Devices

This function searches the database for all POS devices that have more than one fraudulent transaction connected to them. It's is triggered by a GET request to the URL /get-compromised-pos.

@app.route('/get-compromised-pos', methods=['GET'])
def get_compromised_pos():
    """Get compromised POS devices."""

    log.info("Getting compromised Point Of Service IDs")
    start_time = time.time()

    data = db.execute_and_fetch("MATCH (t:Transaction {fraudReported: true})-[:Using]->(:Card)"
                                "<-[:Using]-(:Transaction)-[:At]->(p:Pos) "
                                "WITH p.id as pos, count(t) as connected_frauds "
                                "WHERE connected_frauds > 1 "
                                "RETURN pos, connected_frauds ORDER BY connected_frauds DESC")
    data = list(data)

    log.info("Found %d POS with more then one fraud in %.2f sec",
             len(data), time.time() - start_time)

    return json.dumps(data)

Fetching all Fraudulent Transaction

With a very simple query, you can return all the transactions that are marked as fraudulent. The function is triggered by a GET request to the URL /get-fraudulent-transactions.

@app.route('/get-fraudulent-transactions', methods=['GET'])
def get_fraudulent_transactions():
    """Get fraudulent transactions."""

    log.info("Getting fraudulent transactions")
    start_time = time.time()

    data = db.execute_and_fetch(
        "MATCH (t:Transaction {fraudReported: true}) RETURN t.id as id")
    data = list(data)

    duration = time.time() - start_time
    log.info("Found %d fraudulent transactions in %.2f",
             len(data), duration)

    response = {"duration": duration, "fraudulent_txs": data}
    return Response(
        json.dumps(response),
        status=201,
        mimetype='application/json')

Generating Demo Data

Your app will have an option to generate a specified number of cards, POS devices, and transactions, so you need a function that will be responsible for creating them and marking a number of them as compromised/fraudulent. It's triggered by a POST request to the URL /generate-data. The request body contains the variables:

pos: specifies the number of the POS device.
frauds: specifies the number of compromised POS devices.
cards: specifies the number of the cards.
transactions: specifies the number of the transactions.
reports: specifies the number of reported transactions.

@app.route('/generate-data', methods=['POST'])
def generate_data():
    """Initialize the database."""

    data = request.get_json(silent=True)

    if data['pos'] < data['frauds']:
        return Response(
            json.dumps(
                {'error': "There can't be more frauds than devices"}),
            status=418,
            mimetype='application/json')

    start_time = time.time()

    clear_db()
    init_data(data['cards'], data['pos'])
    compromise_pos_devices(data['pos'], data['frauds'])
    pump_transactions(data['cards'], data['pos'],
                      data['transactions'], data['reports'])

    duration = time.time() - start_time

    response = {"duration": duration}
    return Response(
        json.dumps(response),
        status=201,
        mimetype='application/json')

Fetching POS device Connected Components

This function finds all the connected components of a compromised POS device and returns them to the client. It's triggered by a POST request to the URL /pos-graph.

@app.route('/pos-graph', methods=['POST'])
def host():
    log.info("Client fetching POS connected components")

    request_data = request.get_json(silent=True)

    data = db.execute_and_fetch("MATCH (p1:Pos)<-[:At]-(t1:Transaction {{fraudReported: true}})-[:Using] "
                                "->(c:Card)<-[:Using]-(t2:Transaction)-[:At]->(p2:Pos {{id: {}}})"
                                "RETURN p1, t1, c, t2, p2".format(request_data['pos']))
    data = list(data)

    output = []
    for item in data:
        p1 = item['p1'].properties
        t1 = item['t1'].properties
        c = item['c'].properties
        t2 = item['t2'].properties
        p2 = item['p2'].properties
        print(p2)
        output.append({'p1': p1, 't1': t1, 'c': c, 't2': t2, 'p2': p2})

    return Response(
        json.dumps(output),
        status=200,
        mimetype='application/json')

Rendering Views

These functions will return the requested view. More on them in the Client-side Logic section. They are triggered by GET requests to the URLs / and /graph.

@app.route('/', methods=['GET'])
def index():
    return render_template('index.html')


@app.route('/graph', methods=['GET'])
def graph():
    return render_template('graph.html',
                           pos=request.args.get('pos'),
                           frauds=request.args.get('frauds'))

Creating the Main Function

The function main() has three jobs:

Clear the database if so specified in the input arguments.
Create indexes for the nodes Card, Pos and Transaction. You can learn more about indexing here.
Start the Flask server with the specified arguments.

def main():

    if args.clean_on_start:
        clear_db()

    db.execute_query("CREATE INDEX ON :Card(id)")
    db.execute_query("CREATE INDEX ON :Pos(id)")
    db.execute_query("CREATE INDEX ON :Transaction(fraudReported)")

    app.run(host=args.app_host, port=args.app_port, debug=args.debug)


if __name__ == "__main__":
    main()

Adding the Client-Side Logic

Now, that your server is ready, let's create the client-side logic for your web application.
I'm sure that you're not here for a front-end tutorial and therefore I leave it up to you to experiment and get to know the individual components. Just copy this public directory with all of its contents to the root directory of your project and add the following code to the Dockerfile under the line RUN pip3 install -r requirements.txt:

COPY public /app/public

Just to get you started, here is a basic summary of the main components in the public directory:

img: this directory contains images and animations.
js: this directory contains the JavaScript scripts.
- graph.js: this script handles the graph.html page. It fetches all the connected components of a POS device, renders them in the form of a graph, and can resolve a POS device and all of its connected components as not fraudulent/compromised.
- index.js: this script handles the index.html page. It initializes all of the necessary components, tells the server to generate the initial data, and fetches the fraudulent transactions.
- render.js: this script handles the graph rendering on the graph.html page using the D3.js library.
libs: this directory contains all the locally stored libraries your application uses. For the purpose of this tutorial we only included the memgraph-design library to style your pages.
template: this directory contains the HTML pages.
- graph.html: this is the page that renders a graph of a compromised POS device with all of its connected components.
- index.html: this is the main page of the application. In it, you can generate new demo data and retrieve the compromised POS devices.

Starting the App

It's time to test your app. First, you need to build the Docker image by executing:

docker-compose build

Now, you can run the app with the following command:

docker-compose up

The app is available on the address 0.0.0.0:5000.

Hopefully, you see a screen similar to the image below and are smiling because you just finished your graph-powered credit card fraud detection web application!

Conclusion

Relational database-management systems model data as a set of predetermined structures. Complex joins and self-joins are necessary when the dataset becomes too inter-related. Modern datasets require technically complex queries which are often very inefficient in real-time scenarios.

A graph database is the perfect solution for such complex and large networks. From the underlying storage capabilities to the built-in graph algorithms, every aspect of a graph database is fine-tuned to deliver the best experience and performance when dealing with such problems.

In this, you built a graph-powered credit card fraud detection application from scratch using Memgraph, Flask, and D3.js. You got a good overview of the end-to-end development process using a graph database, and hopefully some ideas for your own projects. We can't wait to see what other graph applications you come up with!

As mentioned at the beginning of this tutorial, feel free to ask us any questions about this tutorial or Memgraph in general on StackOverflow with the tag memgraphdb or on our official forum.

Happy coding!

How to Visualize a Social Network in Python with a Graph Database: Flask + Docker + D3.js

g-despot — Tue, 12 Jan 2021 15:55:23 +0000

When you think about a web application, a graph database doesn’t usually spring to mind. Instead, most people just take the familiar route of using an SQL database to store information. While this is perfectly acceptable for most use cases there are sometimes those that would see tremendous benefits by using a graph database.
In this tutorial, I will show you how to make a basic web application using Flask that stores all of its information in a graph database. To be more precise we are using Memgraph DB, an in-memory database that can easily handle a lot of information and perform read/write instructions quite quickly.

Our use case is a Social Network Graph (in the code referred to as SNG for convenience) representing users and the connections between them. Usually, such a graph would contain millions of relationships and the algorithms that are performed on them don’t do well with data being stored in relational databases.

In this tutorial, I will show you step by step how to build a simple Python web application from the bottom up so you get a basic understanding of the technologies that are used. You can also find all of the code here if you don't want to work on it as you go through the tutorial. If at any point in this tutorial you have a question or something is not working for you, feel free to post on StackOverflow with the tag memgraphdb.

Grandjean, Martin (2014). La connaissance est un réseau. CC BY-SA 3.0

Prerequisites

Because we are building a complete web application there is a number of tools that you will need to install before we begin:

Poetry: a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
Flask: a very powerful web framework that provides you with tools, libraries, and technologies used in web development. A Flask application can be as small as a single web page or as complex as a management interface.
Docker and Compose: an open platform for developing, shipping, and running applications. It enables us to separate our application from our infrastructure (host machine). If you are installing Docker on Windows, Compose will be already included. For Linux and macOS visit this site.
Memgraph DB: a native fully distributed in-memory graph database built to handle real-time use-cases at enterprise scale. Follow the Docker Installation instructions on the Quick Start page. While it's completely optional, I encourage you to also install Memgraph Lab so you can execute openCypher queries on the database directly and see visualized results.

Creating the Project Structure and Handling Dependencies

Sometimes standard packaging systems and dependency management in Python can be confusing for beginners so we decided to use Poetry.

To start building our project structure choose a working directory and run:

poetry new sng-demo

Now you should have a directory with the following content:

sng-demo
├── pyproject.toml
├── README.rst
├── sng_demo
│  └── __init__.py
└── tests
   ├── __init__.py
   └── test_poetry_demo.py

In this tutorial, we won’t use the testing functionalities so go on ahead and delete the directory tests as well as the file README.rst.

Now we need to add the dependencies for our project. Given that we are going to run the app inside a Docker container we don't need the dependencies installed locally, only inside the container. Copy the files project.toml and poetry.lock and place them in the root directory of the project. The only other thing we need to do about dependency management is to tell Docker how to run Poetry on startup so it can install/update all the necessary dependencies inside the container.

Dockerizing an Application

In the root directory of the project create two files, Dockerfile and docker-compose.yml. At the beginning of the Dockerfile, we specify the Python version and instruct the container to install CMake, poetry, mgclient and pymgclient. Poetry is necessary to manage our dependencies inside the container while CMake and mgclient are required for pymgclient, the Python driver for Memgraph DB.

You don’t have to focus too much on this part just copy the code to your Dockerfile:

FROM python:3.7

#Install CMake
RUN apt-get update && \
  apt-get --yes install cmake

#Install poetry
RUN pip install -U pip \
  && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
ENV PATH="${PATH}:/root/.poetry/bin"

#Install mgclient
RUN apt-get install -y git cmake make gcc g++ libssl-dev && \
  git clone https://github.com/memgraph/mgclient.git /mgclient && \
  cd mgclient && \
  git checkout 5ae69ea4774e9b525a2be0c9fc25fb83490f13bb && \
  mkdir build && \
  cd build && \
  cmake .. && \
  make && \
  make install

#Install pymgclient
RUN git clone https://github.com/memgraph/pymgclient /pymgclient && \
  cd pymgclient && \
  python3 setup.py build && \
  python3 setup.py install

Next, we define the working directory with:

WORKDIR /app
COPY poetry.lock pyproject.toml /app/

The second command will enable us to cache the project requirements and only reinstall them when pyproject.toml or poetry.lock are changed.

RUN poetry config virtualenvs.create false && \
  poetry install --no-interaction --no-ansi

We don’t need to create a virtual environment because our application is already isolated by being in a Docker container. To disable it virtualenvs.create needs to be set to false.

The second line in the command ensures that Poetry asks us no interactive questions while installing/updating dependencies and it makes the output more log friendly.

COPY . /app
EXPOSE 5000
ENTRYPOINT [ "poetry", "run" ]

This is where we essentially create all the directories and files inside of our container. The EXPOSE command informs Docker that the container listens on the specified network port at runtime.

Next, we need to create a docker-compose.yml file. Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. For our project, we need two services. One is the web application (sng_demo) and the other a database instance (memgraph).

If you followed the instructions on how to setup Memgraph DB with Docker correctly you only need to add the following code to your docker-compose.yml file to run the container:

version: '3'
services:
  memgraph:
    image: "memgraph"
    ports:
      - "7687:7687"
  sng_demo:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MG_HOST: memgraph
      MG_PORT: 7687
    depends_on:
      - memgraph

When it comes to the ports key, there is an important distinction between the HOST_PORT and the CONTAINER_PORT. The first number in the key is the HOST_PORT and it can be used to connect from your host machine to the service (for example with Memgraph Lab). The second number specifies the CONTAINER_PORT which is used for service-to-service communication. More precisely, our service sng_db can use this port to access the service memgraph and connect to the database.

The environment key contains MG_HOST and MG_PORT which represent environment variables in the service’s container. They store the memgraph service address and port which are needed to establish a database connection.
The depends_on key is used to start services in dependency order because we need the database to start before the web application.

The build key allows us to tell Compose where to find the build instructions as well as the files and/or folders used during the build process. By using the volumes key, we bypass the need to constantly restart our image to load new changes to it from the host machine.

Finally, we have a dockerized project that utilizes Poetry! This approach is great for development because it enables us to run our project on completely different operating systems and environments without having to worry about compatibility issues.

Web Development with Flask

Flask is very simple to use so why not create a Hello World! page to try out our Docker+Poetry setup.

In the project root directory create a file called app.py with the following code:

from flask import Flask

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
   return "Hello World"

First, we imported the Flask class and then created an instance of it. The route() decorator tells Flask what URL should trigger our function.
Now, we need to tell Docker how to run our app. This can be done by creating a simple script in the project root directory. Let’s call it start.sh:

#!/bin/bash
export FLASK_APP=app.py
export FLASK_ENV=development
flask run --host 0.0.0.0

Setting FLASK_ENV to development will enable the debug mode. This makes Flask use an interactive debugger and reloader.

Setting FLASK_APP to app.py specifies how to start the application.

We need to tell Docker when and how to run this script so put the following code in your Dockerfile after the line EXPOSE 5000 :

ADD start.sh /
RUN chmod +x /start.sh

The command chmod +x makes the script executable by setting the right permission.

To execute the script, add the following command after the line ENTRYPOINT [ "poetry", "run" ]:

CMD ["/start.sh"]

That’s it! Our first web page is ready so let’s start our app to make sure we don’t have any errors.

In the project root directory execute:

docker-compose build

The first build will take some time because Docker has to download and install a lot of dependencies.

After it finishes run:

docker-compose up

The URL of our web application is http://localhost:5000/. When you open it there should be a message Hello World! which means that the app is up and running.

Now it’s time to create a more complex web page that will contain our Social Network Graph. In the project root directory create a folder called templates and in it a file with the name base.html. This will be our base HTML template for other pages. Copy the code:

<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
    <link rel="stylesheet" href="/static/css/style.css">

    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
    <script src="https://d3js.org/d3.v4.min.js" charset="utf-8"></script>

    <title>Social Network Graph Demo</title>
</head>

<body>
    {% block content %} {% endblock %}
</body>

</html>

We also need to create an HTML file for our actual landing site that utilizes this base file and an accompanying JavaScript file. Create the HTML file in the same location with the name index.html and copy the following code into it:

{% extends 'base.html' %} {% block content %}
<div class="container">
  Hello World!
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

In the project root directory create a folder called static with one subfolder called js and another called css. The js folder will contain all of the needed local JavaScript files while the css folder will contain all the CSS stylesheets. In the js folder create a file called index.js and in the css folder one called style.css. Just leave them empty for now.

If you want to find out more about web development with Flask I suggest you try out this tutorial.

Your current project structure should like this:

sng-demo
├── sng_demo
│  └── __init__.py
├── templates
│  └── index.html
├── static
│  ├── css
│  │  └── style.css
│  └── js
│     ├── base.html
│     └── index.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

The Data Model and Database Connection

In the app directory sng-demo create a folder called database. This folder will contain all of the modules that we need to communicate with the database. You can find them here and just copy their contents. They are closely related to the database driver and if you wish to examine them a bit more I suggest you look up the driver documentation here.
In the app directory sng-demo create the module db_operations.py. This is where all the custom database related commands will be located.

The sng_demo directory should look like this:

sng_demo
├── __init__.py
├── db_operations.py
└── database
   ├── __init__.py
   ├── memgraph.py
   ├── connection.py
   └── models.py

We will use a very simple data model that can be easily upgraded later on.

There is only one node with the label User and each User has two properties, a numerical id and a string name. Nodes are connected with edges of the type FRIENDS:

There are several methods to populate our database (more on that here) but we will be doing it manually by executing openCypher queries so you can get a better understanding of how to communicate with the database. You will find all the necessary queries to populate the database in the files data_big.txt and data_small.txt. The former just has a larger dataset than the latter.

In the project root directory create a folder called resources and place the files in it. Now you can add an import method to your web application.

In the module db_operations.py add the following import and method:

import json

def clear(db):
  command = "MATCH (node) DETACH DELETE node"
  db.execute_query(command)

def populate_database(db, path):
  file = open(path)
  lines = file.readlines()
  file.close()
  for line in lines:
      if len(line.strip()) != 0 and line[0] != '/':
          db.execute_query(line)

The method clear() deletes any data that might have been left in the database before populating it.

The method populate_database() reads all of the openCypher queries in the specified file and executes them.

In the module app.py change the imports and method index() to:

from flask import Flask, render_template, request, jsonify, make_response
from sng_demo.database import Memgraph
from sng_demo import db_operations

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
    db = Memgraph()
    db_operations.clear(db)
    db_operations.populate_database(db, "resources/data_small.txt")
    return render_template('index.html')

Now every time we refresh our index page the database is cleared and repopulated with new data. While this is not suitable for the production stage, it is highly useful during development because it will enable us to make changes in the data without having to restart the whole application or working directly on the database.

If you want to examine the graph before proceeding, I suggest you open Memgraph Lab and run the query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;.

The result should be:

We also need a method in our app to fetch all the relevant data from the database when a client requests it.

Let’s call it get_graph() and place it in the db_operations.py module:

def get_graph(db):
   command = "MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;"
   relationships = db.execute_and_fetch(command)

   link_objects = []
   node_objects = []
   added_nodes = []
   for relationship in relationships:
       e = relationship['e']
       data = {"source": e.nodes[0], "target": e.nodes[1]}
       link_objects.append(data)

       n1 = relationship['n1']
       if not (n1.id in added_nodes):
           data = {"id": n1.id, "name": n1.properties['name']}
           node_objects.append(data)
           added_nodes.append(n1.id)

       n2 = relationship['n2']
       if not (n2.id in added_nodes):
           data = {"id": n2.id, "name": n2.properties['name']}
           node_objects.append(data)
           added_nodes.append(n2.id)
   data = {"links": link_objects, "nodes": node_objects}

   return json.dumps(data)

First, we need to execute the openCypher query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e; and return its results from the database. These results will contain all the edges in the graph as well as all the nodes that are connected to those edges. Nodes that don't have connections will not be returned and that's ok for now.

The results (the object relationships) are in the form of a generator which we can iterate over and access its contents by using the node/edge names specified in our initial query (n1,n2 and e).

We also need to check if a node has already been appended to the node_objects list because multiple edges can contain (point to or from) the same node. All of the objects are stored in key-value pairs suitable for later JSON conversion.

The final result is a JSON object containing:

links: all the relationships that are in the graph as pairs of source and target id properties,
nodes: all the nodes from the graph that form relationships with other nodes.

In your app.py module add the following method:

@app.route("/get-graph", methods=["POST"])
def get_graph():
   db = Memgraph()
   response = make_response(
       jsonify(db_operations.get_graph(db)), 200)
   return response

This method is responsible for responding to POST requests from the client. It returns the graph data that we fetched from the server in the previous method.

Now let's do something with this data! Copy the contents for your index.js file from here
and the style.css file from here.

We also need to add the actual SVG graphic to our page so change the index.html file to:

{% extends 'base.html' %} {% block content %}
<div class="container">
    <svg class="border rounded mt-3" width="960" height="600" style="background-color:white"></svg>
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

I won't go into much detail about how to use D3.js so if you want to find out more I encourage you to visit their website.

In short, we fetch all the nodes and edges from the database and add them to an SVG element. The visual representation of the graph is made by simulating how physical forces act on particles (charge and gravity). You can drag and drop the nodes, hover over them to see the value of their name property, zoom in and out of the graph and move the SVG graphic.

Additional Functionalities

Go ahead and copy the file query.js to the directory static/js and query.html to the directory templates. You can find the updated base.html file here. Copy the necessary methods from the db_operations.py module and app.py module.

After you made the changes, just open http://localhost:5000/query/ and see the results.

This page will make your life easier if you want to debug the data being fetched from the server. It returns all the nodes or edges and shows them in a JSON highlighted format.

Your current project structure should like this:

sng-demo
├── resources
│  ├── data_big.py
│  └── data_small.txt
├── sng_demo
│  ├── __init__.py
│  ├── db_operations.py
│  └── database
│     ├── __init__.py
│     ├── memgraph.py
│     ├── connection.py
│     └── models.py
├── templates
│  ├── base.html
│  ├── index.html
│  └── query.html
├── static
│   ├── css
│   │  └── style.css
│   └── js
│      ├── index.js
│      └── query.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

Conclusion

Even though graph databases have been around for a long time, they are still not considered a mainstream tool in software development. Relational database-management systems model data as a set of predetermined structures. Complex joins and self-joins are necessary when the dataset becomes too inter-related. Modern datasets require technically complex queries which are often very inefficient in real-time scenarios.

Graph databases offer powerful data modeling and analysis capabilities for many real-world problems such as social networks, business relationships, dependencies, shipping, logistics... and they have been adopted by many of the world's leading tech companies. With this tutorial, I hope to shed some light on how easy it is to integrate a graph database in your development process and I encourage you to try it out yourself.

As I said at the beginning, feel free to ask us any questions about this tutorial or Memgraph in general on StackOverflow with the tag memgraphdb or on our official forum. Good luck with your coding!