DEV Community: Memgraph

MCP for Agents: The Security Gap Most Teams Miss

Sabika Tasneem — Mon, 16 Feb 2026 12:31:41 +0000

MCP is exciting because it turns an LLM into something that can execute actions through tool calls. One protocol, many tools. Your agent can pull data, update tickets, call APIs, and trigger workflows. That is exactly why teams are rushing to ship MCP based agents.

That speed comes with a tradeoff. Once an LLM can touch live systems, mistakes stop being “bad answers” and start becoming real actions. The point of this post is not to criticize MCP. It is to help you ship agents that stay useful without unintentionally expanding your blast radius.

Let’s dive in!

What MCP Gives You (And What it Does Not)

MCP standardizes how tools and context are exposed to a model, which is great for developer velocity. What it does not do is decide what is safe or appropriate in your environment. You still own boundaries and behavior.

In production, the gaps show up fast:

Which tool should be used for this request
What data is allowed for this user or team
Which actions should be blocked or require approval
How you can audit tool use after something goes wrong

If you want the spec level overview, start with Anthropic’s MCP introduction.

Building Agents with MCP: 3 Problems You Will Hit First

The first failure is rarely a headline breach. It usually looks like a normal product bug, except now the bug can trigger emails, update records, or touch production data. For instance:

The Agent Does the “Helpful” Thing You Did Not Ask For

A user asks, “Can you check which customers are impacted?” The agent decides that notifying customers is helpful and drafts a mass email. Nothing was hacked. The model was just optimizing for task completion, and you gave it a tool that made the wrong idea easy.
A Demo Tool Becomes a Production Hazard

Most teams start with a broad tool set because it makes the demo work. Later, the agent gets a slightly different question and reaches for the most powerful tool available. If that tool can write, delete, or trigger workflows, you now have an outsized failure scope. That is the blast radius.
The Agent Guesses and Guesses Wrong

If your agent can query a database, it will try. If it does not have the right context about what is allowed and what the data means, it will guess. Sometimes the guess is harmless. Sometimes it pulls data it should not have pulled, or it produces results that look right but are based on the wrong assumptions.

Why Prompt Rules Are Not Enforcement

The common response is to add more instructions: “Read-only,” “confirm before sending,” “never delete.” Those rules help, but they do not enforce anything.

There is a simple reason. Prompts influence the model’s behavior. They do not change the system’s capabilities. If a write tool is exposed, the model can still call it, even if you told it not to. If a broad SQL tool is exposed, the model can still retrieve more data than you intended, even if you asked it to be careful.

This is why prompt-only safety tends to decay over time. As you add tools, edge cases, and new workflows, the instruction layer becomes a long list of exceptions. The agent still has the same tool surface, but now it is operating under a growing set of text rules that are easy to miss, conflict, or misapply.

The fix is capability control. Reduce what the agent can do, scope what it can see, and require explicit approvals for actions that have a real blast radius.

The Practical Fix: Shrink the Tool Surface at Runtime

Do not rely on the model to always choose correctly. Make wrong choices harder. The simplest way to do that is to reduce what the agent can do by default, then expand capabilities only when you have a clear reason.

Start with these guardrails:

Expose fewer tools by default
Only expose tools that match the current task
Separate read tools from write tools
Require approvals for irreversible actions
Log tool calls so you can trace what happened

This is least-privilege design applied to agent tool access, enforced at runtime.

Where GraphRAG Fits in an MCP Tooling Stack

Most RAG stacks start with vectors. Vectors are great at finding semantically similar text, but they are not built to represent relationships like who owns which data, which rule is current, or which tool is allowed for this workflow.

Graphs are good at that because they model relationships directly. When you add a graph-based context layer, you can give the model a smaller, cleaner slice of context tied to the user and the task.

For example, you can make use of label-based access controls that determine which node labels and relationship edge types a given user or workflow can touch. That reduces overload and lowers the chance your agent reaches for the wrong tool.

A Checklist You Can Actually Use

If you are shipping MCP-powered agents, do not treat guardrails as a final polish step. Treat them as part of the build. The fastest way to end up in trouble is to bolt safety on after you have already exposed a wide tool surface to an LLM.

Start with a simple baseline and improve it as you learn. The point is not to predict every edge case up front. The point is to make tool behavior observable, reversible where possible, and scoped to what the agent should be doing right now.

If you are shipping MCP-powered agents, start here:

List your tools and label them read or write
Turn off anything irreversible by default
Add a human approval step for high impact actions
Keep tool descriptions short and specific
Log every tool call with who requested it and what tool ran
Review misfires weekly and treat them as product bugs

This checklist is not about paranoia. It is about making MCP workflows predictable enough to ship. If your plan is “we will fix it in the prompt,” you are in for some trouble.

What Memgraph adds to an MCP agent stack

At some point in production, most enterprise teams realize they need a real context layer. Memgraph is an in-memory graph database used as a real-time context engine, which makes it a good fit when your agent needs fast traversal, connected context, and governance that changes as your systems change.

In practice, you can use Memgraph to store and query the relationships your agent depends on, then apply GraphRAG patterns to retrieve a connected context slice instead of stuffing everything into a prompt.

This is also where Memgraph’s Atomic GraphRAG comes in. Instead of stitching together multiple retrieval steps in your application code, Atomic GraphRAG aims to generate context in a single query so it is simpler, faster, and easier to review and tweak.

For you, that means fewer moving parts, clearer failure modes, and a smaller surface area for accidental tool misuse.

If you are exploring MCP specifically, Memgraph provides an MCP Server to expose graph context to agents, and an MCP Client inside Memgraph Lab to compose workflows across MCP servers.

Wrapping Up

MCP is a doorway to useful agents. It also makes mistakes expensive. If you want to ship responsibly, focus on runtime guardrails: shrink the tool surface, keep context clean, and log everything.

If you want to explore a graph-based context layer for MCP, start here. And remember, tool access is part of your attack surface, so review it alongside your production code.

Innovation Graph Analytics Powered by Embeddings and LLM’s

André Vermeij — Wed, 27 Nov 2024 10:52:44 +0000

Guest Author: André Vermeij, Founder of Kenedict Innovation Analytics & Developer of Kenelyze

Intro & Recap: Innovation Graphs

The first post in our series on Innovation Graphs introduced the usage of graphs in the analysis of innovation and its output, such as patents, scientific publications and research grants.

Innovation graphs focus on mapping the connections between technologies, organisations and people and can provide new insights into the actual underpinnings of innovative activity within topics or organisations of interest. They can be constructed based on all kinds of metadata and often focus on visually mapping three complementary perspectives:

Graphs of documents to gain deeper insight into technology/topic clusters.
Graphs of organisations and institutions to focus on sector-wide collaboration patterns.
Graphs of people/experts to get a better understanding of team-level collaboration and key players in a field of expertise.

In this second post on Innovation Graphs, we’ll focus on the creation and LLM-powered analysis of the first type of graph mentioned above—graphs detailing clusters of technologies and topics within a specific sector of interest. Specifically, we’ll dive into how we can use text embeddings to construct document similarity graphs, and how we can automatically analyse the content and label the graph’s clusters using locally running Large Language Models.

Text Embeddings & Graph Creation

Mapping clusters of technology and the connections between them is a key part of most innovation analytics projects. A common way to create the related document similarity graphs is to collect the unstructured text related to documents in a dataset (for example, abstracts for scientific publications or summaries of R&D project reports), convert the text into vectors/embeddings, and then calculate pairwise similarities to get similarity scores for each pair of documents to construct the final graph.

The Classical Way: TF-IDF

Converting unstructured text into ready-to-analyse vectors can be done in various ways. A classical way to approach this is to use a variant of Term Frequency-Inverse Document Frequency (TF-IDF). Here, all unstructured text is initially pre-processed using common techniques in Natural Language Processing (tokenization, lemmatization, stop-word removal, etc.), after which each token in a document is assigned a TF-IDF score. This score is based on how often the token appears in the document itself (TF) and on the inverse of how often it appears across all documents in the dataset (IDF). For each document, a vector with a length equalling the total number of unique tokens across all documents is then created, holding the TF-IDF scores for all tokens in the document and zeroes for any tokens that do not occur in the document.

Although this is a pretty intuitive way of converting text into vectors, it comes with several challenges. The main drawback is that semantic similarity is mostly overlooked in this approach, since the scores are simply based on term counts within and across documents. Also, the vectors resulting from TF-IDF are generally very sparse and can easily consist of thousands of elements per vector, depending on the size of the overall text corpus.

The Modern Way: Embedding Models

The rise of Large Language Models and Generative Artificial Intelligence has also resulted in the availability of a wide variety of embedding models and APIs to convert unstructured text into fixed-length vectors. For example, Nomic, Mixedbread, Jina, and OpenAI all offer APIs to get embeddings based on input of unstructured text of your choice. Some key use cases for these embedding models are query and document embedding for Retrieval Augmented Generation, but they also serve as an excellent basis for the large-scale embedding of datasets to create document similarity graphs.

The main benefits of these embedding models are that they also consider semantic similarity between concepts and are usually of a fixed, dense size (often 768 or 1024 elements, often called dimensionality). A challenge is that users need to carefully pick the parameters when using these models since these can significantly impact the overall outcome when converting the vectors into document similarity graphs.

Graph Creation Based on Embeddings

We can construct a document similarity graph based on all pairwise similarities between the document vectors as soon as embeddings are generated for all documents in our dataset. The nodes in the graph are simply the original documents from our dataset, with weighted links drawn between nodes when they have a certain degree of similarity. A commonly used similarity metric is cosine similarity, with scores ranging from 0 to 1, where 1 denotes identical texts/vectors. Links between nodes can be determined by setting a threshold similarity value.

The exact value used here can have a significant impact on the readability of the graph: setting the threshold too low will often lead to a hairball/spaghetti bowl visualization (too many links between nodes), while setting it too high will show many disparate clusters with no connections between them. When constructing a graph, it is therefore important to give this some thought and also relate it to the actual size of the text fragments you are dealing with – shorter strings (titles) usually go well with higher threshold values, while longer strings (abstracts, summaries) usually combine well with lower threshold values.

Community Detection for Technology Cluster Identification

As soon as the nodes and links in the graph have been constructed based on the embedding similarities and the threshold set, we can start analysing the graph of documents to uncover clusters of related content. In practice, this is a very important step to make the graph more readable and understandable.In innovation analytics, gaining insight into which technology clusters are present in a dataset and how they connect and evolve is often key to a project’s success.

An excellent way to uncover these clusters is by using the Leiden community detection algorithm, now available in Memgraph. Based on the structure of the graph, this algorithm detects densely connected subsets of nodes and iteratively assigns them to the same communities. In the end, when colouring nodes based on the communities they are assigned, we have an excellent basis to start labelling and annotating the graph to make sense of its contents.

LLM-Powered Innovation Cluster Labelling

In the analysis of technology and topic graphs, providing clear labelling and annotation of the resulting graph visualizations is key to gaining insights by stakeholders in an innovation analytics project. Annotated visuals are often used to provide initial high-level overviews of a graph’s contents in presentations, and often serve as a basis for further deep dives into specific clusters of interest.

A classical approach to initial cluster labelling is to treat each cluster's contents as a separate corpus of documents and then run a version of TF-IDF to extract the top-5 highest-scoring tokens or phrases for each cluster. The resulting labels often provide a decent first indication of a cluster’s contents, but they do require subsequent manual analysis and improvement to improve their readability.

An exciting alternative way to label clusters is to use a Large Language Model to summarize cluster contents. In our case, we utilize locally running models such as Llama 3.1 in Ollama or LM Studio based on the following high-level process:

For each detected community, we first gather relevant unstructured text from the attributes of the nodes in the cluster. In most cases, we have found that sending over collections of document titles per cluster works very well for cluster labelling. This collection of texts is then added to a prompt that specifies exactly how the LLM should respond in its summarization: based on the texts provided, return a short summary/label consisting of a maximum of 5 words with an indication of the high-level topic. Many LLM’s are prone to adding a lot of introductory (“Absolutely! Here is a summary of…”) and concluding text to answers, so the prompt also specifies that it should never do this and purely focus on returning the labels.

As soon as the LLM finishes providing the labels for all communities, we replace the nodes’ initial community attribute with the newly created label. Of course, these labels do require manual checks to see whether they make sense and sometimes require slight adjustments because they are too high-level. The quality of the labels is also dependent on the LLM itself: we’ve found that larger models such as Llama3.1 (8B parameters) generally provide better labels than smaller models such as Llama 3.2 (3B parameters).

Cluster Summarization Using LLM’s

Another valuable way to use LLM’s in document similarity graph analysis is to further enhance users’ understanding of clusters by providing point-and-click larger summaries of what the documents in a cluster are about. The approach here is similar to the LLM-based labelling described above, with the prompt sent to the model focusing on providing an overall summary consisting of 3 to 5 phrases instead.

Practically speaking, users of a graph visualization select nodes of their interest using a free-form selection tool and point out which unstructured text attribute should be used for the analysis, after which the LLM returns a summary based on the collection of texts sent to it. The summary is then printed in a window right on top of the visual, as in the example below:

Up Next: Step-By-Step Real-Life Examples and Visuals

This post provided an overview of how text embeddings can be used to construct document similarity graphs for innovation analysis, and how Large Language Models can aid in the labeling and summarization of the resulting graphs. Our next post in this series will show examples of this in practice using Kenelyze, based on a real-life dataset of the innovation output of a major high-tech company. It will also discuss the importance of local LLM’s when working with sensitive data, and highlight some technical considerations when picking and configuring a local LLM.

Innovation as a Graph: Improved Insight into Technology Clusters, Collaboration and Knowledge Networks

André Vermeij — Wed, 18 Sep 2024 12:42:47 +0000

Guest Author: André Vermeij, Founder of Kenedict Innovation Analytics & Developer of Kenelyze

Organisations focused on innovation come in many forms, including corporations with large Research & Development (R&D) departments, universities, research institutions active in advancing science, and startups working on the potentially next big thing. Innovation-related data has become increasingly important for each of these organisations to inform decision-making and stay ahead of market developments. For example, an R&D-intensive corporation could use data to benchmark its own technology portfolio with its direct competitors, while a startup might be analysing data to assess previous activity and potential market entry in a sector of interest.

Traditional Innovation Analysis

The traditional way to look at innovation-related data is to report on output within a topic or organisation of interest based on counts and sums of variables of interest. When analysing its competition, a business may for example gather information on a competitor’s recent output and report on the number of documents in each technology domain, produce a list of the companies the competitor has worked with, or generate an overview of the most active inventors or researchers in a field of interest. Although all these analyses can be valuable in their own right, they’re missing out on a key aspect of an innovation ecosystem: the connections between technologies, organisations, and people.

A Graph of Innovation

Viewing innovation and its output as a graph of interconnected data points allows us to get a much deeper understanding of the technology and knowledge structures in a context of interest. Using the metadata in a wide array of innovation-related data sources, which will be discussed more in the following section, it is possible to create graphs of connected documents, organisations and people and gain new insights into the actual underpinnings of innovative activity.

For example, innovation graphs allow us to answer questions, such as:

Which clusters of activity can we distinguish within a topic or organisation of interest, and how has this evolved?
What do the organisational collaboration networks in an area of interest look like, and who are the key players in network connectivity?
How are teams of individual experts in a specific field composed, and who are the leading experts in a given topic?

Open Data Sources for Innovation Analytics

Until just a few years ago, quality innovation data was quite hard to come by without a subscription to an expensive database hosting patent information or scientific publications. Luckily, in recent years, there has been a move towards more openly available data, which can serve as an excellent basis for setting up a wide variety of innovation graphs.

Here’s a quick overview of common data sources:

Patents: organisations apply for patents to protect their inventions against commercialisation by third parties. Patent applications and grants are published online by national patent offices around the world, with databases gathering data from all jurisdictions and providing a wide array of metadata. A great open data source is the European Patent Office's Open Patent Services (OPS) API, or the EPO's search platform Espacenet.
Scientific publications: journal publications, conference proceedings, book chapters and various other types of scientific output are gathered in databases which bring together output from many sources. Paid databases such as Scopus are still used often by large organisations – great open alternatives include OpenAlex and Semantic Scholar.
Subsidies & funding programmes: governmental subsidies to stimulate innovation and R&D in specific areas are often structured in openly available data sources. A good example is the European Union’s CORDIS data for the Horizon Europe programme. Many national enterprise agencies also publish their granted subsidies and projects online.
Internal data: the above data sources are often augmented with internal, unpublished data (e.g., internal project reports, unfiled patent applications, scientific output in the review stage) to get a view on very recent activity within an organisation. This is especially valuable when creating knowledge graphs within organisations or carrying out an innovation portfolio analysis for a specific client.

In a typical Innovation Analytics project, combining data from multiple of the above data sources is often key to gaining the best insights. For example, organisations applying for patents often also have scientific output related to the same theme and may also apply for governmental funding. To get a picture of innovative activity that is as complete as possible, it is therefore important to look at activity from multiple data sources and graph perspectives.

Graphs of Documents: Insight into Technology and Knowledge Clusters

The analysis and visualisation of innovation graphs often starts with looking at the relationships between documents based on a shared characteristic.

Depending on the goals of the analysis, there are various ways to link documents together:

Text similarity: unstructured text data in the form of document titles, abstracts and summaries can be used to connect documents when there is a high similarity between their contents. This relies on vectorisation of the text of interest and subsequent calculation of pairwise cosine similarities, where a link is then drawn between documents based on a minimum similarity score.
Knowledge flows / shared authors: another way to generate clusters of connected documents is to link them when the same people have worked on them. The authorship data on documents can be used to accomplish this. The key assumption here is that documents are part of the same “knowledge cluster” when persons with specific expertise have (co-) written them.
Citations: numerous citations to other documents can be found in both scientific publications and patent applications. We can use these citations to create various types of graphs:
- Shared references: connect documents when they cite the same sources, often with a minimum number of shared citations set as the weight for the links.
- Shared citing documents: connect documents when they have been cited by the same other documents, again often with a minimum weight set.
- Direct citations: creation of citation graphs where links are drawn between documents when they cite each other.
Technology classifications: patent documents are categorised using classification codes designating the technology areas which they fall into. These can be used to connect documents when they share one or multiple codes, essentially creating clusters of documents based on technological overlap.

The following graph is an example of a text similarity approach, where scientific publications in the area of autonomous vehicles are connected when they share significant textual content. Colors depict clusters of activity based on the outcomes of a community detection algorithm, and nodes are sized based on the number of times they were cited by other papers:

Figure 1: Graph of scientific publications linked based on text similarity approach

Graphs of Organisations: Insight into Collaboration Ecosystems

Another graph perspective, which is very common in innovation analysis, focuses on mapping the connections between organisations (businesses, universities, research institutions, public bodies, hospitals, etc.).

Many of the data sources above hold extensive metadata on the organisations responsible for the documents—scientific authors are affiliated with their employers, patents are applied for by the parties seeking protection of their invention and governmental subsidies are often received by consortia of collaborating organisations.

It is common to attach weights to the links based on the number of collaborations between two organisations. Using these weights, it is then possible to filter the graph to focus only on the strongest / most frequently occurring collaborations.

The graph below shows an example of collaboration in radiotherapy innovation, where colors are based on the type of organisation (e.g. blue = universities, green = hospital and medical centers) and node sizes based on their betweenness centrality scores:

Figure 2: Collaboration in radiotherapy

Graphs of People: Insight into Expertise and Knowledge Networks

This is a graph perspective that often follows after mapping organizational collaboration networks, focusing on the actual person-to-person collaborations taking place to produce the analysed output.

Using the author/inventor metadata on documents, we draw links between people when they have co-authored a document. Similar to the organisational networks, we can also attach weights to the links, which correspond to the number of documents which have been worked on jointly by two authors. This perspective can provide a deep understanding of the actual team structures and knowledge networks within and outside of organisations.

Here’s an example of the (relatively large!) network of inventors who have worked on Apple patents. Nodes are sized based on their betweenness centralities, and colors are based on clusters detected by a community detection algorithm:

Figure 3: Apple's inventor network

Graph Metrics & Innovation Insights

The above examples show various ways to convert innovation data into actionable graph visualisations. In the actual analysis and interpretation of these graphs, it is important to make good use of the many metrics available in graph analytics. These metrics can help us understand which clusters are present in a network, and can aid in determining the importance of nodes based on centrality measures.

The following metrics are valuable for analysing the overall graph structure in innovation analysis:

Component analysis: determining the components (interconnected subsets of nodes) in the graph to be able to see how far the graph is interconnected (how many nodes can reach each other directly or indirectly) and to determine the impact of the largest connected components versus smaller components.
K-Cores: to determine highly connected subsets of nodes in graphs, k-Cores can be used to highlight subgraphs in which all nodes have at least a degree of k. This can be used to focus on so-called cliques of nodes quickly and is especially valuable when analysing collaboration and knowledge networks.
Community detection: using an algorithm such as the Leiden community detection algorithm to determine which clusters we can distinguish within the components. These clusters then serve as the basis for graph annotation, where clusters are labeled based on their actual contents (see the labels in the autonomous driving graph above).

On the individual node level, degree and betweenness centrality measures can be used to determine the importance of nodes in innovation graphs:

Degree Centrality: determining simple connection counts per node to quickly see which actors are most important in terms of the number of other nodes they are connected to. Since most innovation graphs are weighted (links have weights associated with them), weighted degree centrality is also used regularly.
Betweenness Centrality: this is a frequently and often used metric to determine who holds key positions in a graph in terms of hub positions – which organizations/people are the “key connectors” between clusters/teams? It is calculated by determining how often each node appears on the shortest paths between all other nodes in the network.

Up Next: Use Cases

Now that you have an initial idea of the main ideas behind innovation graphs, we will showcase practical use cases, real-world client examples and common challenges in innovation graph analysis in the next blog post. Stay tuned!

In-memory vs. disk-based databases: Why do you need a larger than memory architecture?

Memgraph — Tue, 05 Sep 2023 14:41:54 +0000

Memgraph is an in-memory graph database that recently added support for working with data that cannot fit into memory. This allows users with smaller budgets to still load large graphs to Memgraph without paying for (more) expensive RAM. However, expanding the main-memory graph database to support disk storage is, by all means, a complex engineering endeavor. Let’s break this process down into pieces.

On-disk databases

Disk-based databases have been, for a long time, a de facto standard in the database development world. Their huge advantage lies in their ability to store a vast amount of data relatively cheaply on disk. However, the development can be very complex due to the interaction with low-level OS primitives. Fetching data from disk is something that everyone strives to avoid since it takes approximately 10x more time than using it from main memory. Neo4j is an example of a graph, an on-disk database that uses disk as its main storage media while trying to cache as much data as possible to main memory so it could be reused afterward.

In-memory databases

In-memory databases avoid the fundamental cost of accessing data from disk by simply storing all its data in the main memory. Such architecture also significantly simplifies the development of the storage part of the database since there is no need for a buffer pool. However, the biggest issue with in-memory databases is when the data cannot fit into the random access memory since the only possible way out is to transfer the data to a larger and, consequently, more expensive machine.

In-memory database users rely on the fact that durability is still secured through durability mechanisms like transaction logging and snapshots so that data loss does not occur.

Larger-than-memory architecture

Main memory computation

Larger-than-memory architecture describes a database architecture when the majority of computations are still within the main memory, but the database offers the ability to store a vast amount of data on disk, too, without having the computational complexity of interacting with buffer pools.

Identify hot & cold data

The larger-than-memory architecture utilizes the fact that there are always hot and cold parts of the database in terms of accessing it. The goal is then to find cold data stored and move it to the disk so that transactions still have fast access to hot data. Cold data identification can be done either by directly tracking transactions’ access patterns (online) or by offline computation in which a background thread analyzes data.

The second very important feature of the larger-than-memory architecture is the process of evicting cold data. This can be done in two ways:

DB tracks the memory usage and starts evicting data as soon as it reaches a predefined threshold.
Eviction can be done only when new data is needed.

Transaction management

Different systems also behave differently regarding transaction management. If the transaction needs data that is currently stored on the disk, it can:

Abort the transaction, fetch data stored on the disk, and restart the transaction.
Stall the transaction by synchronously fetching data from the disk.

Transaction must fit into memory

The question is, what happens when the transaction data cannot fit into random access memory? In Memgraph, we decided to start with an approach that all transaction data must fit into memory. This means that some analytical queries cannot be executed on a large dataset, but this is the tradeoff we were willing to accept in the first iteration.

Benefits of larger-than-memory databases

Memgraph uses RocksDB as a key-value store for extending the capabilities of the in-memory database. Not to go into too many details about RocksDB, but let’s just briefly mention that it is based on a data structure called Log-Structured Merge-Tree (LSMT) (instead of B-Trees, typically the default option in databases), which are saved on disk and because of the design come with a much smaller write amplification than B-Trees.

The in-memory version of Memgraph uses Delta storage to support multi-version concurrency control (MVCC). However, for larger-than-memory storage, we decided to use the Optimistic Concurrency Control Protocol (OCC) since we assumed conflicts would rarely happen, and we could make use of RocksDB’s transactions without dealing with the custom layer of complexity like in the case of Delta storage.

We’ve implemented OCC in a way that every transaction has its own private workspace, so potential conflicts are detected at the commit time. One of our primary requirements before starting to add disk-based data storage was not to ruin the performance of the main memory-based storage. Although we all knew there was no such thing as zero-cost abstraction, we managed to stay within 10% of the original version. We decided to use snapshot isolation as an appropriate concurrency isolation level since we believed it could be the default option for the large majority of Memgraph users.

Disadvantages of larger-than-memory databases

As always, not everything is sunshine and flowers, especially when introducing such a significant feature to an existing database, so there are still improvements to be made. First, the requirement that a single transaction must fit into memory makes it impossible to use large analytical queries.

It also makes our LOAD CSV command for importing CSV files practically unusable since the command is executed as a single transaction. Although RocksDB is really good, fits really well into our codebase, and has proved to be very efficient in its caching mechanisms, maintaining an external library is always hard.

In retrospect

Albeit the significant engineering endeavor, the larger-than-memory architecture is a super valuable asset to Memgraph users since it allows them to store large amounts of data cheaply on disk without sacrificing the performance of in-memory computation. We are actively working on resolving issues introduced with the new storage mode, so feel free to ask, open an issue, or pull a request. We will be more than happy to help. Until next time 🫡

Exciting News: LangChain Now Supports Memgraph!

Memgraph — Fri, 25 Aug 2023 07:06:32 +0000

We're thrilled to announce a powerful integration between LangChain and Memgraph, bringing you an unparalleled natural language interface to your Memgraph database. Say goodbye to complex queries and welcome a seamless and intuitive way to interact with your data.

Memgraph QA chain tutorial

If you've ever wanted to effortlessly query your Memgraph database using natural language, this tutorial is for you. This step-by-step guide will walk you through the process, ensuring you have all the tools you need to get started.

Prerequisites

Before you dive in, make sure you have Docker and Python 3.x installed on your system.

Get started

Launch a Memgraph Instance: With a few simple commands, you can have your Memgraph instance up and running using Docker. Just follow our script to set it up.

Install dependencies: We've got you covered with the required packages. Use pip to install langchain, openai, neo4j, and gqlalchemy. Don't forget the --user flag to ensure smooth permissions.

Code playtime: Whether you prefer working within this notebook or want to use a separate Python file, the tutorial offers code snippets to guide you through the process.

What's inside

Explore the rich features and functionalities that LangChain and Memgraph offer together:

API reference: We provide an overview of the key components you'll be working with, such as ChatOpenAI, GraphCypherQAChain, and MemgraphGraph.

Populating the database: Learn how to populate your Memgraph database effortlessly using the Cypher query language. We guide you through the process of seeding data that serves as the foundation for your work.

Refresh graph schema: Familiarize yourself with refreshing the graph schema, a crucial step in setting up the Memgraph-LangChain graph for Cypher queries.

Querying the database: Discover how to interact with the OpenAI API and configure your API key. We'll show you how to utilize the GraphCypherQAChain to ask questions and receive informative responses.

Chain modifiers: Customize your chain's behavior with modifiers like return_direct, return_intermediate_steps, and top_k. Tailor the experience to your preferences.

Advanced querying: Delve into advanced querying techniques and uncover tips for refining your prompts to improve query accuracy.

Ready to take your data interaction to the next level? Join us in exploring the seamless synergy between LangChain and Memgraph. No more wrangling with queries – just natural language and meaningful insights. Simplify complexity, elevate your insights, and share your projects in our community.

What is a Graph Database?

Ani Ghazaryan — Wed, 23 Aug 2023 15:22:01 +0000

While relational databases have been the go-to choice for data storage, they fall short when it comes to handling complex relationships and traversing interconnected data, which puts graph databases in a special spotlight. A graph database is a specialized database system designed to store, manage, and query highly connected data using graph theory principles. As data volumes continue to explode, companies need efficient and scalable solutions to handle the complexities of their data.

Specialized database systems like graph databases offer a more natural and efficient way to model, query, and store data, leading to improved performance and better data insights.

Understanding graphs

In simple terms, at the core of graph databases lies the concept of a graph. In mathematics and computer science, a graph is a collection of nodes (also known as vertices) connected by edges. Nodes represent entities or objects, while edges depict the relationships or connections between them. This straightforward yet effective structure forms the foundation of graph databases.

Components of graphs

It's your time to shine! Let's reiterate on what we've learned so far. Graphs consist of two fundamental components: nodes and edges.

Nodes represent entities or objects and can have various attributes associated with them.
Edges, on the other hand, depict the relationships or connections between nodes and can also carry properties.

Together, nodes and edges create a rich network of connected data.

Graph theory basics

Another common term you may hear here and there, alluding to graphs or graph databases, is graph theory, which is a branch of mathematics, that provides the theoretical underpinning for understanding and analyzing graphs. It defines vertices as the fundamental building blocks of a graph and edges as the connections between vertices. Relationships in a graph can be represented by directed or undirected edges, capturing the nature and direction of the connections.

Relational databases vs. graph databases

Opinions split when it comes to choosing a database, however, the debate around relational vs. graph databases is still hot. Relational databases have long been the dominant database model, organizing data into structured tables with predefined schemas. They excel in handling structured data and transactions but face challenges when dealing with complex relationships and traversing connected data. This is largely due to their rigid tabular structure.

Joining multiple tables and navigating through numerous relationships can lead to performance bottlenecks and complex query formulations. This limits their effectiveness in scenarios where relationships play a crucial role.

Graph databases excel in modeling and querying relationships. They store connections explicitly, allowing for efficient traversals between nodes and enabling complex relationship queries with ease. And of course, graph databases provide flexibility, scalability, and performance advantages over a relational database when it comes to handling interconnected data.

Characteristics of graph databases

So far, you've been introduced to a few qualities that are typical to graphs, so let's put the learnings into structure and build off of what you've grasped.

Schema-less nature

Unlike relational databases, graph databases are schema-less, meaning they do not require a predefined structure or schema for data. This flexibility allows for the dynamic addition of new node types, properties, and relationships, making graph databases highly adaptable to evolving data models.

Native graph processing

Graph databases are purpose-built for processing graph data. They employ optimized algorithms and data structures to efficiently traverse and manipulate the graph structure, resulting in faster query response times and improved performance compared to non-native graph databases.

Graph traversal and pattern matching

One of the key strengths of graph databases is their ability to traverse and explore relationships between nodes. Graph traversal algorithms can efficiently navigate the graph to discover patterns, uncover hidden connections, and retrieve data based on specific criteria. This capability is particularly valuable in applications such as recommendation engines, fraud detection, and knowledge graphs, which we will explore in the sections to come.

Use cases of graph databases

Unlike a traditional relational database that relies on tabular data, a graph database utilizes a flexible and intuitive data model, allowing for the representation of intricate relationships between entities. With its ability to efficiently capture and traverse vast networks of data, a graph database has emerged as an advanced tool for diverse domains, including:

Social networks and recommendation engines

Graph databases have revolutionized social networking platforms and recommendation engines. They enable personalized recommendations, friend suggestions, and social network analysis by leveraging the rich network of connections between users, interests, and entities.

Fraud detection and network analysis

Graphs also excel in fraud detection and network analysis. By representing complex networks of relationships, they can identify suspicious patterns, detect fraudulent activities, and uncover hidden connections that might indicate illicit behavior, making them an invaluable tool for cybersecurity.

Knowledge graphs and semantic networks

Last but not least, graph databases serve as a foundation for building knowledge graphs and semantic networks. By representing data as nodes and relationships, they capture the semantics and context of information, enabling sophisticated knowledge discovery, semantic search, and data integration across disparate sources.

Sneak peek into graph algorithms

Surely enough, graph algorithms play a crucial role in leveraging the power of graph databases and unlocking valuable insights from connected data. In this section, we provide a sneak peek into some fundamental graph algorithms that form the backbone of graph database operations.

Breadth-First Search (BFS): Breadth-First Search is a fundamental algorithm used to explore and traverse a graph in a breadth-first manner. Starting from a given source node, BFS systematically explores all the neighboring nodes before moving deeper into the graph. This algorithm is commonly used to find the shortest path between two nodes, identify connected components, and perform level-based analysis.

Depth-First Search (DFS): Depth-First Search is another crucial graph algorithm that explores a graph with a depth-first principle. DFS starts from a given source node and traverses as far as possible along each branch before backtracking. The algorithm is useful for identifying cycles in a graph, performing topological sorting, and searching for specific nodes or patterns.

PageRank algorithm: Developed by Google's founders, PageRank is a graph algorithm used to measure the importance or relevance of nodes in a graph, particularly in web graphs. PageRank assigns each node a numerical value based on the number and quality of incoming links, and plays a vital role in search engine ranking, recommendation systems, and social network analysis.

These are just a few examples of the numerous graph algorithms available, however, graph databases employ a wide range of algorithms to perform tasks such as community detection, centrality analysis, graph clustering, and more.

Sum up

In this article, we explored the world of graph databases and their significance in modern data management. We defined graph databases and highlighted their importance in handling complex relationships and interconnected data. If you're curious and want to learn more about the fascinating world of graphs, make sure to check out our blog and give us a shout in our community.

Security Analysis with JupiterOne’s Starbase and Memgraph

Matea Pesic — Tue, 22 Aug 2023 16:36:38 +0000

Starbase is an open-source graph-based security analysis tool that unifies all of JupiterOne’s integrations into one. It collects assets and relationships from services and systems, including cloud infrastructure, SaaS applications, security controls, and more, into an intuitive graph visualization. With over 115 open-source graph integrations, Starbase collaborates with your existing toolkit enabling easy and insightful cyber security analysis.

In this article, we’ll dig into Starbase, guiding you through the setup of two example integrations and enabling Starbase to work with Memgraph for easy ingestion and visualization of your graph data.

Prerequisites

Installed Yarn package manager.
Installed Node.js.
A running Memgraph instance—visit Memgraph’s docs for instructions on how to install and connect to Memgraph.

Setting up Starbase

To kick-start your Starbase setup, first, you need to clone the JupiterOne/Starbase repo into your local directory and ensure you have Yarn and Node.js installed.
Once you’ve successfully cloned the repository and installed the prerequisites, place yourself in the terminal in the directory where you cloned the repo and run the yarn command. The command installs all of the necessary project dependencies.
The next step is setting up configurations for your integration of choice. You can find a list of all integrations on JupiterOne’s GitHub repo. Moving forward, we are going to explore two options for possible integration, Zoom, and GitHub.

Setting up integrations

In order to set up an integration, you need to register an account in the system the integration targets for ingestion and obtain the necessary API credentials. Starbase leverages credentials from external services to authenticate and collect data. When Starbase is started, it reads configuration data from a single configuration file named config.yaml at the root of the project.

Zoom integration

In order to configure the Zoom integration, we need to create a Zoom app to retrieve the needed credentials:

Go to the Zoom App Marketplace and sign into your Zoom account.
In the top right corner, go to the Develop dropdown menu and select Build App.
Choose to create an OAuth type of app.
Take note of your Account ID, Client ID, and Client secret which we’ll need for the configuration file later on.
In the Scopes section, add group:read:admin, role:read:admin, user:read:admin, and account:read:admin.

After you’ve successfully created your Zoom App, open up the starbase repo in your editor of choice and create your config.yaml file. This is an example of a config.yaml file for Zoom integration:

integrations:
  -
    name: graph-zoom
    instanceId: testInstanceId
    directory: ./.integrations/graph-zoom
    gitRemoteUrl: <https://github.com/JupiterOne/graph-zoom.git>
    config:
      ACCOUNT_ID: <ACCOUNT_ID>
      CLIENT_ID: <CLIENT_ID>
      CLIENT_SECRET: <CLIENT_SECRET>
      SCOPES: 'read:admin role:read:admin user:read:admin account:read:admin'

GitHub integration

In order to configure GitHub integration, we need to create a GitHub app to retrieve the needed credentials:

Go to the GitHub Apps and select to create a new GitHub App
Name your app, and enter a homepage URL (in this case, you can use the JupiterOne’s Starbase repo URL), uncheck the webhook and adjust the repository permissions. The following permissions need to be set to read-only: -Repository Permissions: Actions, Environments, Issues, Pull Requests and Secrets -Organization Permissions: Administration, Members, Secrets. The rest of the permissions are No access by default.

Read-only access for secrets repo doesn’t give read-only access to actual secret content, it only gives read-only info to the existence of the metadata about the secrets.

Select Any account and create your GitHub App.

After you’ve successfully created your GitHub App, open up the cloned Starbase repository in your editor of choice and create your config.yaml file. Generate your private key and retrieve other needed credentials from the GitHub App you previously created. Below is an example of a config.yaml file for a GitHub integration:

integrations:
    -
     name: graph-github
     instanceId: testInstanceId
     directory: ./.integrations/graph-github
     gitRemoteUrl: <https://github.com/JupiterOne/graph-github.git>
     config:
        GITHUB_APP_ID = <GITHUB_APP_ID>
GITHUB_APP_LOCAL_PRIVATE_KEY_PATH={YOURPATH}/{YOURFILENAME}.private-key.pem
        INSTALLATION_ID=<INSTALLATION_ID>
        GITHUB_API_BASE_URL=https://api.github.com

Use Starbase with Memgraph

After you’ve successfully created your config.yaml file, the last step is to adjust your queries to work with Memgraph. In order to do that, run the following steps:

First, you need to place yourself in the terminal in the folder you cloned your Starbase repo and run yarn starbase setup command to clone or update all integrations listed in the config.yaml file, as well as install all dependencies for each integration.
Run your Memgraph instance. Follow the instructions from Memgraph’s docs on how to connect to Memgraph, or if you are using Docker, simply run the following command:
docker run -it -p 3000:3000 -p 7444:7444 -p 7687:7687 memgraph/memgraph-platform
By modifying just a single line of code, you are ready to use Starbase with Memgraph. Inside the neo4jGraphStore.js file, locate the addEntities() function. To enable compatibility with Memgraph, simply update the following line:

await this.runCypherCommand(`CREATE INDEX index_${entity._type} IF NOT EXISTS FOR (n:${entity._type}) ON (n._key, n._integrationInstanceID);`);

With:

await this.runCypherCommand(`CREATE INDEX index_${entity._type} IF NOT EXISTS FOR (n:${entity._type}) ON (n._key, n._integrationInstanceID);`);
await this.runCypherCommand

You are all set to utilize Starbase with Memgraph. The instance is actively listening to port 7687, as defined in the code.

The final step is to run the yarn starbase run command. Afterward, launch your browser and go to localhost:3000 to access Memgraph Lab or open your desktop version to explore and visualize your graph data.

Explore your dataset

Below, we’ve provided a few query examples that demonstrate how you can dig into your dataset and extract valuable insights. The following examples assume the use of GitHub integration.

With the following query, you are retrieving the information of all of the extracted GitHub users from a certain organization:

MATCH (n:github_user) RETURN n LIMIT 3;

If you also want to determine which code owners of organization repositories grant access to outside contributors, execute the following query:

MATCH (account:github_account) - [e:OWNS] -> (repo:github_repo) -> [f:ALLOWS] -> (user:github_user {role: ‘OUTSIDE’}) RETURN account, repo, user, e, f;

Takeaways

Starbase is a powerful tool that simplifies security analysis by unifying integrations into a user-friendly graph view, enhancing cybersecurity insights. Incorporating Memgraph for data ingestion adds another dimension by enhancing its capabilities and visualizing your data. If you are curious about graphs and would like to learn more, make sure to check out our blog and join our community on Discord.

Memgraph vs. TigerGraph

Vlasta Pavicic — Fri, 18 Aug 2023 06:43:44 +0000

In today's data-driven world, the necessity to process and interpret complex relationships within massive datasets is making organizations continually search for the go-to graph database, leaving the traditional relational database options behind. After the initial DB-Engines consultations, two names commonly arise in conversations: TigerGraph and Memgraph.

Background on both solutions

Founded in 2012 by Dr. Yu Xu, TigerGraph's core objective is to provide a scalable and efficient graph database platform that enables organizations to leverage the power of interconnected data, supporting applications ranging from fraud detection to AI and machine learning.

Memgraph is an in-memory, open-source graph database with roots in the UK and Croatia. Founded by Marko Budiselic and Dominik Tomicevic in 2016, and backed by American investors, Memgraph prioritizes high performance and developer accessibility. With a robust community edition, the platform offers a blend of ease of use and practical functionality, all presented through clear and uncomplicated licensing, making it the backbone of many cybersecurity solutions.

Memgraph vs. TigerGraph differences

Although both TigerGraph and Memgraph have been developed in C++ and aim to provide performant solutions for real-time data analytics, there exist some important differences between the two platforms that set them apart. Let’s check what those are.

Query language

The choice of query language plays a significant role in the overall user experience.

GSQL, TigerGraph's proprietary query language, does offer an expressive, Turing-complete language tailored for graph pattern-matching and analytics functions but might present a steeper learning curve for those new to graphs. It has been specifically designed for TigerGraph, and the skillset may not be easily transferred from or to other graph database platforms.

In contrast, Cypher query language is an open-source, declarative language known for its user-friendly syntax. Cypher's human-readable style has propelled it into a standard for querying graph databases. It has been developed by Neo4j but is utilized by various systems, including Memgraph. Due to its simplicity, and broad community support, it is a preferred choice for many developers who know their applications will need minimum changes if they require a switch to another database vendor.

Data storage

TigerGraph and Memgraph offer distinctive approaches to handling data in their graph databases, each reflecting a unique strategy to balance performance, scalability, and flexibility.

TigerGraph employs a hybrid memory-disk approach, leveraging RAM for storing frequently accessed data and disk storage for large graphs that may exceed available memory. This hybrid model allows TigerGraph to achieve real-time analytics, where active datasets are immediately available, while also scaling to handle massive datasets without being constrained by RAM.

In contrast, Memgraph's architecture has been built natively for in-memory data analysis and storage, focusing on lightning-fast data processing. Being ACID compliant, it ensures consistency and reliability in its core design. However, Memgraph also offers flexibility. An analytical storage mode that bypasses ACID compliance is available, accelerating analytics and data import operations when absolute consistency is not required. Additionally, an on-disk storage option allows users to weigh performance against budget constraints, thus achieving a balance tailored to specific needs.

While TigerGraph's hybrid approach offers a comprehensive solution for both speed and scalability, Memgraph's focus on in-memory processing with adaptable options reflects a commitment to performance with versatility to suit various requirements. The distinction between these two models shows the innovation in graph database technology, catering to diverse needs in data management, analysis, and storage.

Data isolation level

TigerGraph employs a read-committed data isolation level, meaning that a transaction can access data which is committed before and during this transaction’s execution.
For example, two same READ queries inside one transaction can return different results because between them another transaction was committed.

On the other hand, Memgraph uses snapshot isolation by default, where each query operates on a consistent snapshot of the data at the query's start time, with the option to change the isolation level, but snapshot isolation offers an advantage as it provides a more consistent view of the data, reducing the chance of reading partial or uncommitted changes. This ensures more accurate query results and a smoother transaction experience, making snapshot isolation generally considered a more reliable approach in many scenarios.

Pricing models and support

TigerGraph has a free version that allows users to work with up to 50GB of data, making it suitable for small projects or initial exploration. Memgraph offers something different with its Community Edition, which is not only free but also open-source and packed with features.

For example, both TigerGraph and Memgraph offer high availability features to ensure that data is consistently accessible and resistant to failures, but Memgraph's replication is available even in the Community Edition of the product. This means that the Community Edition is not "crippleware" but a fully functional version that allows users to "kick the tires" on the product and properly test it to ensure it meets requirements before deploying it in a production environment.

By offering this, and a plethora of other features in the Community Edition, Memgraph not only shows a commitment to performance and reliability but also to accessibility, empowering users to explore and validate the capabilities of the software without barriers.

Due to the lack of complicated layers of management that some larger companies might have, in Memgraph you can talk directly to engineers if you have questions or need help. It's a more hands-on, direct way of working that puts you closer to the people who built the product, and it can make working with Memgraph a more pleasant and efficient experience.

Overview of features

Takeaways on both graph databases

Memgraph and TigerGraph both offer graph database solutions, but Memgraph's native in-memory design sets it apart. Built for speed without losing stability or ACID compliances, Memgraph provides efficient real-time querying and analytics. Although TigerGraph claims to be "The World’s Fastest Graph Analytics Platform for the Enterprise", clients have reported increased performance after switching to Memgraph. If speed, reliability, direct interaction, and support from engineers are key priorities, Memgraph may be the more appealing choice.

Check the performance of Memgraph on your own dataset using Benchgraph, a graph database performance benchmark, and feel free to contact us about making the switch.

What is a Key-Value Database?

Kruno Golubic — Thu, 03 Aug 2023 09:17:43 +0000

In the landscape of data management solutions, key-value databases stand apart, offering a unique blend of simplicity and performance tailored to the big data era. By abstracting data into simple key-value pairs and values, this type of database offers an intuitive and flexible data model that can seamlessly scale to accommodate large datasets. Before delving into the complexities of key-value databases, their advantages, use cases, and potential drawbacks, let's first establish a foundational understanding of these databases and how they represent a distinctive paradigm in data storage and retrieval.

Understanding a key-value store

In the ever-expanding world of databases, key-value databases have emerged as a vital element. They are designed for storing, retrieving, and managing associative arrays, a data structure commonly known as a dictionary or hash table. The dictionary contains a collection of records, each with different fields containing data. These records are stored and retrieved using a unique key. This unique approach to data management differs from traditional relational databases and makes a key-value database an optimal solution for use cases where simplicity, speed, and scalability are paramount. Key-value databases are the simplest form of NoSQL databases. Each item in the database is stored as an attribute name, or key, together with its value.

Defining key-value stores

A key-value database, or key-value store, uses a simple key-value method to store data. Each key-value pair represents a specific piece of data. The 'key' serves as a unique identifier that is used to find the data within the database. Key-value databases treat the data as a single opaque collection, which may have different fields for every record. This offers considerable flexibility and more closely aligns with modern concepts like object-oriented programming.

The primary features of a key-value database include simplicity, high performance, and the ability to handle large volumes of data efficiently. Because placeholders or input parameters do not represent optional values, key-value databases often use far less memory to store the same data, which can lead to significant performance gains in certain workloads.

Working with a key-value database

Key-value databases are known for their ease of use, making them a good starting point for beginners in the database realm. Thanks to their straightforward data structure, they provide a simple, user-friendly interface. The process involves storing unique keys associated with their corresponding data values and retrieving the associated value using its unique key.

Key-value databases excel in use cases that involve frequently accessed data. For instance, in e-commerce platforms, a key-value database could be used to store product details. The unique identifier or 'key' could be the product ID, with all the related product data stored as the associated 'value'. Another scenario is user logs, where key-value stores prove to be efficient. Popular key-value database examples include Redis, DynamoDB, Riak, and RocksDB have active community support and extensive documentation, making it easier for developers to get started and troubleshoot issues.

Benefits and advantages of a key-value database

Developers and organizations choose key-value databases for several reasons.

Speed

One of the main reasons is their speed. Key-value databases can handle large volumes of reads and writes with minimal latency. The simplicity of the data structures used in a key-value store makes it easy for developers to quickly store and retrieve data. They offer impressive speed and performance, handling large volumes of data and providing rapid, random data access.

Horizontal scaling

Scalability is a critical factor for modern applications dealing with ever-growing data volumes and increasing user demands. Key-value stores offer an advantage in this area, as they can easily scale horizontally. Horizontal scaling involves adding more servers to the existing network to distribute the data and workload across multiple nodes. As a result, key-value databases can maintain their performance levels even as data grows, ensuring that applications can handle a high number of concurrent users and massive data storage needs. This ability to scale out effectively makes them a preferred choice for applications that require elasticity and flexibility.

Ease of use and development

Key-value databases provide a simple and straightforward interface for data storage and retrieval. Developers can easily interact with the database using basic operations like "get" and "put" based on the associated keys. The straightforward API and data model reduce the complexity of the application code, making it easier to develop and maintain. Additionally, the ease of integration with various programming languages and frameworks allows developers to quickly incorporate key-value stores into their applications without a steep learning curve.

Flexible data models

Key-value databases are schema-less, meaning that they do not enforce a fixed data structure or data types. This flexibility allows developers to store and manage various types of data within the same database without strict predefined schemas. This is particularly advantageous for applications dealing with diverse and evolving data formats. Whether it's storing user profiles, session data, configurations, or complex objects, key-value databases can handle different data formats efficiently, eliminating the need for complex and costly data transformations.

Potential drawbacks of a key-value database

While key-value databases are powerful, they aren't always the ideal choice.

Complex queries

They lack the ability to perform complex queries or handle sophisticated relationships between data, which relational databases excel at. If your use case involves complex queries or requires understanding the relationships between different data entities, then a relational database or a graph database might be a better fit. Also, they may not be suitable for applications that require multi-record transactions or complex data manipulation.

Lack of data relationships

Key-value databases do not inherently support relationships between data items. While data can be organized and retrieved efficiently based on keys, establishing complex relationships between different pieces of data requires additional application logic. This can lead to potential data inconsistencies and increased complexity in managing relationships outside the database layer. For applications heavily reliant on data relationships, such as social networks or recommendation systems, graph databases may provide more natural and performant solutions, as they are specifically designed to handle and traverse complex data relationships efficiently.

Handling complex data types

Another challenge with key-value databases is the handling of complex data types. While they can handle structured, semi-structured, and unstructured data, they lack the advanced capabilities of a document database in managing complex objects.

Data size and indexing

As the data size grows significantly, key-value databases may face challenges with indexing and maintaining performance. While they excel at handling fast key-based lookups, large datasets can impact the efficiency of these operations. Proper index design and tuning are crucial to ensure optimal performance. Additionally, some key-value databases may not offer sophisticated indexing options, limiting their capabilities to efficiently handle specific types of queries or data access patterns.

Overcoming limitations of a key-value database

Despite these limitations, there are strategies and tools that can help overcome them. For example, secondary indexing and query languages like Redis's RediSearch and Amazon DynamoDB's Query Language can help perform more complex queries. Hybrid models, combining a key-value database with other database types like document databases, can handle complex queries or manage complex relationships between data entities. Additional technologies that can be incorporated into key-value databases to enhance functionality include secondary indexing to create efficient index structures to accelerate application responses.

Takeaways

Key-value databases offer a compelling mix of simplicity, speed, and scalability that can be highly beneficial in certain situations. As with any technology, it's important to understand its strengths and weaknesses to make the most of it. The key-value database world is constantly evolving, and it's an exciting area to explore for any developer or organization looking to optimize their data storage and retrieval needs.

They are a fantastic choice for use cases that involve handling large volumes of frequently accessed data or where horizontal scalability is required. However, it's crucial to understand the capabilities and limitations of key-value databases before choosing them as a solution. Overall, with ongoing evolution and innovation in the database world, the future looks promising for the key-value database scene.

When to Use a NoSQL Database

Katarina Supe — Fri, 21 Jul 2023 13:44:05 +0000

How we manage and process data changes rapidly over the years, and staying on track with the newest technology trends is important. Traditional relational database management systems, such as MySQL, Oracle, and SQL Server, have always been the first choice to maintain data integrity and for fast querying, but with the rise of big data, NoSQL (Not Only SQL) databases emerged and became more popular due to their speed, flexibility, and scalability. Now, the question that pops up naturally is when is the right time to use a NoSQL database?

This blog post will explore NoSQL databases and how they differ from traditional databases. Besides, you will learn when and why to utilize NoSQL databases. Whether you’re a developer, data engineer, or decision maker, this blog post will provide insights into modern databases that will come in handy to boost your projects.

Understanding NoSQL databases

NoSQL databases are non-relational databases with flexible schema designed for high performance at a massive scale. Unlike traditional relational databases, which use tables and predefined schemas, NoSQL databases use a variety of data models. There are 4 main types of NoSQL databases - document, graph, key-value, and column-oriented databases. NoSQL databases generally are well-suited for unstructured data, large-scale applications, and agile development processes. The most popular examples of NoSQL databases are MongoDB (document), Memgraph (graph), Redis (key-value store) and Apache HBase (column-oriented).

These types of NoSQL databases store data in their own unique ways offering pros and cons, which are presented below:

Advantages

Flexibility: If you’ve used an SQL database before, you are probably familiar with its strict schema. That can be a hassle with data rapidly changing. NoSQL databases, with their dynamic and flexible schemas, have a huge advantage in handling unstructured and semi-structured data, making an excellent choice for diverse data types.
Scalability: Data volumes can grow exponentially, and scalability becomes crucial, especially with the rise of big data. Many NoSQL databases offer horizontal scalability, allowing you to add more servers to increase the capacity. On the other hand, SQL databases usually scale only vertically.
Performance: Many NoSQL databases are optimized to deliver high performance, even when dealing with large data volumes or data streams.

Disadvantages

Maturity: SQL (standing for Structured Query Language) databases have been popular in the database world for a long time, and because of that, they offer robustness, tools and community that NoSQL databases have not managed to match yet. That does not mean that NoSQL will not overtake over the years, just like it happens often with modern technologies.
Complexity: The learning curve with NoSQL databases is steeper than with SQL databases, especially because there is no standard query language. This follows from the maturity and the fact that NoSQL databases are more complex to design, implement and manage. But, the results you gain by storing your data in a proper database might be worth it.
ACID compliance: Opposite to NoSQL databases, many SQL databases are usually ACID-compliant. However, as an exception, Memgraph is a NoSQL graph database that is ACID compliant. If you’re unsure what that means for your application, you are invited to check out our recent blog on ACID transactions and isolation levels.

5 reasons to choose a NoSQL database

Based on the above-mentioned advantages and disadvantages of NoSQL databases, there are various scenarios where NoSQL databases are the optimal solution for your project. Let’s take a look at 5 scenarios where NoSQL databases are great:

1. Handling large volumes of data at scale seamlessly

NoSQL databases can handle large amounts of data by spreading it across multiple servers. Hence, the NoSQL database is a perfect solution if you have large amounts of growing data. You don’t have to worry that you don’t have enough resources to scale vertically; instead, you should design your database system properly to get the most insights from your data.

2. Working easily with unstructured or semi-structured data

Predefined schemas can be good sometimes, but if you’ve always found that a hassle for your unstructured or semi-structured data which is often changing, NoSQL databases are the correct approach. They are built with flexible data schema, which speeds up the development process and lowers the efforts on database management. Problems with data types like XML, JSON and others have become issues of the past with NoSQL databases.

3. Enabling rapid development

Again, because of the NoSQL flexibility, your data model can rapidly change, meaning you can update your application on the fly without schema updates. Nowadays, development is fast, and iterations are quick, so removing the task of updating schemas constantly saves you valuable time that can be spent on even faster development.

4. High read/write speed

If you’re working on applications that require real-time data processing, you might be stuck with traditional databases that don’t offer the required speed. NoSQL databases are optimized for high-speed read and write operations and are ideal for chat, IoT, gaming, fraud detection, and similar applications.

5. Managing complex relationships

Complex data relationships, such as parent-child or many-to-many relationships, arise in relational databases when data from different tables is related or somehow interconnected. Querying it requires hopping from one table to another and joining it with slow and resource-intensive join operations. A graph database is a type of NoSQL database that handles highly-connected data especially well and is a good choice for social networks, recommendation engines, fraud detection, or any application where you need to traverse through interconnected data points.

Concluding thoughts

If you still don’t know whether you need an NoSQL or a relational database management system, we prepared another article on that topic that may speed up your decision process. Keep in mind that it doesn’t always have to be a fight, SQL vs NoSQL, you can surely consider a hybrid approach and get the best out of both SQL and NoSQL databases. But if you did find that your use case fits the description above, then NoSQL is a database for you! We are always open to discussions on this and other topics in the database scene, so join the conversation on our Discord server to be a part of the community.

How to Build a Graph Web Application With Python, Flask, Docker & Memgraph

Memgraph — Thu, 20 Jul 2023 11:24:36 +0000

The goal is straightforward (or at least it seems simple enough). Let's build a
web application in Python that can visualize a graph and run some cool graph
algorithms out of the box. Maybe it's not your flavor, but I prefer the
Flask web framework for such occasions, so bear with me through this
tutorial.

Now, I am going to show you an example of how to accomplish this. You can also
take a look at the finished
app
on GitHub if you want to see the complete code.

The general outline of the tutorial is:

Create a Flask server
Dockerize your application
Import the data into Memgraph
Query the database

Graph visualizations will be covered in part two of the tutorial so stay
tuned! Spoiler alert, we are going to use D3.js to draw our graph.

Prerequisites

For this tutorial, you will need to install:

Docker
Docker Compose (which is included with Docker on Windows and macOS)

With Docker, we don't need to worry about installing Python, Flask, Memgraph...
essentially anything. Everything will be installed automatically and run
smoothly inside Docker containers!

Disclaimer: Docker fanboy alert

1. Create a Flask server

I included comments in the code to make it more understandable, but if at any
point you feel like something is unclear, join our Discord
Server and share your thoughts. First, create the
file app.py with the following contents:

import json
import logging
import os
from argparse import ArgumentParser
from flask import Flask, Response, render_template
from gqlalchemy import Memgraph

log = logging.getLogger(__name__)

def init_log():
    logging.basicConfig(level=logging.DEBUG)
    log.info("Logging enabled")
    # Set the log level for werkzeug to WARNING because it will print out too much info otherwise
    logging.getLogger("werkzeug").setLevel(logging.WARNING)

Other than the imports, the first few lines focus on setting up the logging. No
web application is complete without logging, so we will add the bare minimum and
disable the pesky werkzeug logger, which sometimes prints too much info.

Now, let's create an argument parser. This will enable you to easily change the
behavior of the app on startup using arguments.

# Parse the input arguments for the app
def parse_args():
    """
    Parse command line arguments.
    """
    parser = ArgumentParser(description=__doc__)
    parser.add_argument("--host", default="0.0.0.0", help="Host address.")
    parser.add_argument("--port", default=5000, type=int, help="App port.")
    parser.add_argument("--template-folder", default="public/template", help="Flask templates.")
    parser.add_argument("--static-folder", default="public", help="Flask static files.")
    parser.add_argument("--path-to-input-file", default="graph.cypherl", help="Graph input file.")
    parser.add_argument("--debug", default=True, action="store_true", help="Web server in debug mode.")
    print(__doc__)
    return parser.parse_args()

args = parse_args()

It’s time to create your server instance:

# Create the Flask server instance
app = Flask(
    __name__,
    template_folder=args.template_folder,
    static_folder=args.static_folder,
    static_url_path="",
)

You can finally create the view functions that will be invoked from the browser
via HTTP requests. In layman's terms, the homepage is called by:

# Retrieve the home page for the app
@app.route("/", methods=["GET"])
def index():
    return render_template("index.html")

The only thing that’s left is to implement and call the main() function:

# Entrypoint for the app that will be executed first
def main():
    # Code that should only be run once
    if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
        init_log()
    app.run(host=args.host,
            port=args.port,
            debug=args.debug)

if __name__ == "__main__":
    main()

The somewhat strange statement os.environ.get("WERKZEUG_RUN_MAIN") == "true"

will make sure that this code is only executed once. Confused? A problem arises
when working with Flask in development mode because each code change triggers a
reload of the server, which in turn could result in parts of your code executing
multiple times (for example, like the main function).

So, if you need to execute something only once in Flask at the beginning like
loading data, this is the perfect place for it.

The next step is to create the following files, which we will work on in the
next tutorial:

index.html in public/template
index.js in public/js
style.css in public/css

One more file is needed and this one will specify all the Python dependencies
that need to be installed. Create requirements.txt with the following
contents:

gqlalchemy==1.0.6
Flask==2.0.2

Your current project structure should look like this:

app
├── public
│  ├── css
│  │   └── style.css
│  ├── js
│  │   └── index.js
│  └── templates
│      └── index.html
├── app.py
└── requirements.txt

2. Dockerize your application

This is much simpler than you might think. Most often, you will need a
Dockerfile in which you will specify how your Docker image should be created.
Let's take a look at our Dockerfile :

FROM python:3.9

# Install CMake
RUN apt-get update && \
  apt-get --yes install cmake && \
  rm -rf /var/lib/apt/lists/*

# Install Python packages
COPY requirements.txt ./
RUN pip3 install -r requirements.txt

# Copy the source code
COPY public /app/public
COPY app.py /app/app.py
WORKDIR /app

# Set the environment variables
ENV FLASK_ENV=development
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

# Start the web application
ENTRYPOINT ["python3", "app.py"]

The first line indicates that we are basing our image on a Linux image that has
Python 3.9 preinstalled. The next step is to install CMake (which is needed for
the Memgraph Python driver) with RUN and the standard Linux installation
command apt-get ... .

We copy the requirements.txt file and install the Python packages with
pip. The source code also needs to be copied to the image in order for us to
start the web application. The ENTRYPOINT command is responsible for starting
the desired process inside the container.

But we are not finished with Docker yet. We need to create a
docker-compose.yml file that will tell Docker which containers to start.

version: "3"
services:
  server:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MEMGRAPH_HOST: memgraph
      MEMGRAPH_PORT: "7687"
    depends_on:
      - memgraph

  memgraph:
    image: "memgraph/memgraph"
    ports:
      - "7687:7687"

There are two services/containers in our app:

Server: Uses the Dockerfile to build a Docker image and run it.
Memgraph: This is our database. Docker will automatically download the image and start it.

Because we are supplying environment variables, let's load them in app.py
right after the imports:

MEMGRAPH_HOST = os.getenv("MEMGRAPH_HOST", "memgraph")
MEMGRAPH_PORT = int(os.getenv("MEMGRAPH_PORT", "7687"))

Your current project structure should look like this:

app
├── public
│  ├── css
│  │   └── style.css
│  ├── js
│  │   └── index.js
│  └── templates
│      └── index.html
├── app.py
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

Now, we can even start our app with the following commands:

docker-compose build
docker-compose up

3. Import the data into Memgraph

This task will be done inside the main() function because it only needs to be
executed once:

memgraph = None

def main():
    if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
        init_log()
        global memgraph
        memgraph = Memgraph(MEMGRAPH_HOST,
                            MEMGRAPH_PORT)
        load_data(args.path_to_input_file)
    app.run(host=args.host,
            port=args.port,
            debug=args.debug)

How do we import the data into Memgraph? I prepared a file with the Cypher
queries that need to be executed in order to populate the database. You just
need to download the file in your root directory and add the following
load_data() function:

def load_data(path_to_input_file):
    """Load data into the database."""
    try:
        memgraph.drop_database()
        with open(path_to_input_file, "r") as file:
            for line in file:
                memgraph.execute(line)
    except Exception as e:
        log.info(f"Data loading error: {e}")

First, we clear everything in the database, and then we go over each line in the
file graph.cypherl and execute them. And that's it. Once we start the web
application, Memgraph will import the dataset.

4. Query the database

We will create a function that will execute a Cypher query and return the
results. It returns the whole graph, but we will limit ourselves to 100 nodes:

def get_graph():
    results = memgraph.execute_and_fetch(
        f"""MATCH (n)-[]-(m)
                RETURN n as from, m AS to
                LIMIT 100;"""
    )
    return list(results)

The view function get_data() which fetches all the nodes and relationships
from the database, filters out the most important information, and returns it in
JSON format for visualization. To can the network load at a minimum, you will
send a list with every node id (no other information about the nodes) and a list
that specifies how they are connected to each other.

@app.route("/get-graph", methods=["GET"])
def get_data():
    """Load everything from the database."""
    try:
        results = get_graph()

        # Sets for quickly checking if we have already added the node or edge
        # We don't want to send duplicates to the frontend
        nodes_set = set()
        links_set = set()
        for result in results:
            source_id = result["from"].properties['name']
            target_id = result["to"].properties['name']

            nodes_set.add(source_id)
            nodes_set.add(target_id)

            if ((source_id, target_id) not in links_set and
                    (target_id, source_id,) not in links_set):
                links_set.add((source_id, target_id))

        nodes = [
            {"id": node_id}
            for node_id in nodes_set
        ]
        links = [{"source": n_id, "target": m_id} for (n_id, m_id) in links_set]

        response = {"nodes": nodes, "links": links}
        return Response(json.dumps(response), status=200, mimetype="application/json")
    except Exception as e:
        log.info(f"Data loading error: {e}")
        return ("", 500)

What’s next?

As you can see, it’s very easy to connect to Memgraph and run graph algorithms,
even from a web application. While this part of the tutorial focused on the
backend, in the next one, we will talk about graph visualizations and the D3.js
framework.

How to Visualize a Social Network in Python with a Graph Database: Flask + Docker + D3.js

Memgraph — Wed, 19 Jul 2023 13:15:58 +0000

When you think about a web application, a graph database doesn’t usually spring to mind. Instead, most people just take the familiar route of using an SQL database to store information. While this is perfectly acceptable for most use cases there are sometimes those that would see tremendous benefits by using a graph database.
In this tutorial, I will show you how to make a basic web application using Flask that stores all of its information in a graph database. To be more precise we are using Memgraph DB, an in-memory database that can easily handle a lot of information and perform read/write instructions quite quickly.

Our use case is a Social Network Graph (in the code referred to as SNG for convenience) representing users and the connections between them. Usually, such a graph would contain millions of relationships and the algorithms that are performed on them don’t do well with data being stored in relational databases.

In this tutorial, I will show you step by step how to build a simple Python web application from the bottom up so you get a basic understanding of the technologies that are used. You can also find all of the code here if you don't want to work on it as you go through the tutorial. If at any point in this tutorial you have a question or something is not working for you, feel free to post on StackOverflow with the tag memgraphdb.

Prerequisites

Because we are building a complete web application there is a number of tools that you will need to install before we begin:

Poetry: a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
Flask: a very powerful web framework that provides you with tools, libraries and technologies used in web development. A Flask application can be as small as a single web page or as complex as a management interface.
Docker and Compose: an open platform for developing, shipping, and running applications. It enables us to separate our application from our infrastructure (host machine). If you are installing Docker on Windows, Compose will be already included. For Linux and macOS visit this site.
Memgraph DB: a native fully distributed in-memory graph database built to handle real-time use-cases at enterprise scale. Follow the Docker Installation instructions on the Quick Start page. While it's completely optional, I encourage you to also install Memgraph Lab so you can execute openCypher queries on the database directly and see visualized results.

Creating the Project Structure and Handling Dependencies

Sometimes standard packaging systems and dependency management in Python can be confusing for beginners so we decided to use Poetry.

To start building our project structure choose a working directory and run:

poetry new sng-demo

Now you should have a directory with the following content:

sng-demo
├── pyproject.toml
├── README.rst
├── sng_demo
│  └── __init__.py
└── tests
   ├── __init__.py
   └── test_poetry_demo.py

In this tutorial, we won’t use the testing functionalities so go on ahead and delete the directory tests as well as the file README.rst.

Now we need to add the dependencies for our project. Given that we are going to run the app inside a Docker container we don't need the dependencies installed locally, only inside the container. Copy the files project.toml and poetry.lock and place them in the root directory of the project. The only other thing we need to do about dependency management is to tell Docker how to run Poetry on startup so it can install/update all the necessary dependencies inside the container.

Dockerizing an Application

In the root directory of the project create two files, Dockerfile and docker-compose.yml. At the beginning of the Dockerfile, we specify the Python version and instruct the container to install CMake, poetry, mgclient and pymgclient. Poetry is necessary to manage our dependencies inside the container while CMake and mgclient are required for pymgclient, the Python driver for Memgraph DB.

You don’t have to focus too much on this part just copy the code to your Dockerfile:

FROM python:3.7

#Install CMake
RUN apt-get update && \
  apt-get --yes install cmake

#Install poetry
RUN pip install -U pip \
  && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
ENV PATH="${PATH}:/root/.poetry/bin"

#Install mgclient
RUN apt-get install -y git cmake make gcc g++ libssl-dev && \
  git clone https://github.com/memgraph/mgclient.git /mgclient && \
  cd mgclient && \
  git checkout 5ae69ea4774e9b525a2be0c9fc25fb83490f13bb && \
  mkdir build && \
  cd build && \
  cmake .. && \
  make && \
  make install

#Install pymgclient
RUN git clone https://github.com/memgraph/pymgclient /pymgclient && \
  cd pymgclient && \
  python3 setup.py build && \
  python3 setup.py install

Next, we define the working directory with:

WORKDIR /app
COPY poetry.lock pyproject.toml /app/

The second command will enable us to cache the project requirements and only reinstall them when pyproject.toml or poetry.lock are changed.

RUN poetry config virtualenvs.create false && \
  poetry install --no-interaction --no-ansi

We don’t need to create a virtual environment because our application is already isolated by being in a Docker container. To disable it virtualenvs.create needs to be set to false.

The second line in the command ensures that Poetry asks us no interactive questions while installing/updating dependencies and it makes the output more log friendly.

COPY . /app
EXPOSE 5000
ENTRYPOINT [ "poetry", "run" ]

This is where we essentially create all the directories and files inside of our container. The EXPOSE command informs Docker that the container listens on the specified network port at runtime.

Next, we need to create a docker-compose.yml file. Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. For our project, we need two services. One is the web application (sng_demo) and the other a database instance (memgraph).

If you followed the instructions on how to setup Memgraph DB with Docker correctly you only need to add the following code to your docker-compose.yml file to run the container:

version: '3'
services:
  memgraph:
    image: "memgraph"
    ports:
      - "7687:7687"
  sng_demo:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MG_HOST: memgraph
      MG_PORT: 7687
    depends_on:
      - memgraph

When it comes to the ports key, there is an important distinction between the HOST_PORT and the CONTAINER_PORT. The first number in the key is the HOST_PORT and it can be used to connect from your host machine to the service (for example with Memgraph Lab). The second number specifies the CONTAINER_PORT which is used for service-to-service communication. More precisely, our service sng_db can use this port to access the service memgraph and connect to the database.

The environment key contains MG_HOST and MG_PORT which represent environment variables in the service’s container. They store the memgraph service address and port which are needed to establish a database connection.
The depends_on key is used to start services in dependency order because we need the database to start before the web application.

The build key allows us to tell Compose where to find the build instructions as well as the files and/or folders used during the build process. By using the volumes key, we bypass the need to constantly restart our image to load new changes to it from the host machine.

Finally, we have a dockerized project that utilizes Poetry! This approach is great for development because it enables us to run our project on completely different operating systems and environments without having to worry about compatibility issues.

Web Development with Flask

Flask is very simple to use so why not create a Hello World! page to try out our Docker+Poetry setup.

In the project root directory create a file called app.py with the following code:

from flask import Flask

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
   return "Hello World"

First, we imported the Flask class and then created an instance of it. The route() decorator tells Flask what URL should trigger our function.
Now, we need to tell Docker how to run our app. This can be done by creating a simple script in the project root directory. Let’s call it start.sh:

#!/bin/bash
export FLASK_APP=app.py
export FLASK_ENV=development
flask run --host 0.0.0.0

Setting FLASK_ENV to development will enable the debug mode. This makes Flask use an interactive debugger and reloader.

Setting FLASK_APP to app.py specifies how to start the application.

We need to tell Docker when and how to run this script so put the following code in your Dockerfile after the line EXPOSE 5000 :

ADD start.sh /
RUN chmod +x /start.sh

The command chmod +x makes the script executable by setting the right permission.

To execute the script, add the following command after the line ENTRYPOINT [ "poetry", "run" ]:

CMD ["/start.sh"]

That’s it! Our first web page is ready so let’s start our app to make sure we don’t have any errors.

In the project root directory execute:

docker-compose build

The first build will take some time because Docker has to download and install a lot of dependencies.

After it finishes run:

docker-compose up

The URL of our web application is http://localhost:5000/. When you open it there should be a message Hello World! which means that the app is up and running.

Now it’s time to create a more complex web page that will contain our Social Network Graph. In the project root directory create a folder called templates and in it a file with the name base.html. This will be our base HTML template for other pages. Copy the code:

<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
    <link rel="stylesheet" href="/static/css/style.css">

    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
    <script src="https://d3js.org/d3.v4.min.js" charset="utf-8"></script>

    <title>Social Network Graph Demo</title>
</head>

<body>
    {% block content %} {% endblock %}
</body>

</html>

We also need to create an HTML file for our actual landing site that utilizes this base file and an accompanying JavaScript file. Create the HTML file in the same location with the name index.html and copy the following code into it:

{% extends 'base.html' %} {% block content %}
<div class="container">
  Hello World!
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

In the project root directory create a folder called static with one subfolder called js and another called css. The js folder will contain all of the needed local JavaScript files while the css folder will contain all the CSS stylesheets. In the js folder create a file called index.js and in the css folder one called style.css. Just leave them empty for now.

If you want to find out more about web development with Flask I suggest you try out this tutorial.

Your current project structure should like this:

sng-demo
├── sng_demo
│  └── __init__.py
├── templates
│  ├── base.html
│  └── index.html
├── static
│  ├── css
│  │  └── style.css
│  └── js
│     └── index.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

The Data Model and Database Connection

In the app directory sng-demo create a folder called database. This folder will contain all of the modules that we need to communicate with the database. You can find them here and just copy their contents. They are closely related to the database driver and if you wish to examine them a bit more I suggest you look up the driver documentation here.
In the app directory sng-demo create the module db_operations.py. This is where all the custom database related commands will be located.

The sng_demo directory should look like this:

sng_demo
├── __init__.py
├── db_operations.py
└── database
   ├── __init__.py
   ├── memgraph.py
   ├── connection.py
   └── models.py

We will use a very simple data model that can be easily upgraded later on.

There is only one node with the label User and each User has two properties, a numerical id and a string name. Nodes are connected with edges of the type FRIENDS:

There are several methods to populate our database (more on that here) but we will be doing it manually by executing openCypher queries so you can get a better understanding of how to communicate with the database. You will find all the necessary queries to populate the database in the files data_big.txt and data_small.txt. The former just has a larger dataset than the latter.

In the project root directory create a folder called resources and place the files in it. Now you can add an import method to your web application.

In the module db_operations.py add the following import and method:

import json

def clear(db):
  command = "MATCH (node) DETACH DELETE node"
  db.execute_query(command)

def populate_database(db, path):
  file = open(path)
  lines = file.readlines()
  file.close()
  for line in lines:
      if len(line.strip()) != 0 and line[0] != '/':
          db.execute_query(line)

The method clear() deletes any data that might have been left in the database before populating it.

The method populate_database() reads all of the openCypher queries in the specified file and executes them.

In the module app.py change the imports and method index() to:

from flask import Flask, render_template, request, jsonify, make_response
from sng_demo.database import Memgraph
from sng_demo import db_operations

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
    db = Memgraph()
    db_operations.clear(db)
    db_operations.populate_database(db, "resources/data_small.txt")
    return render_template('index.html')

Now every time we refresh our index page the database is cleared and repopulated with new data. While this is not suitable for the production stage, it is highly useful during development because it will enable us to make changes in the data without having to restart the whole application or working directly on the database.

If you want to examine the graph before proceeding, I suggest you open Memgraph Lab and run the query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;.

The result should be:

We also need a method in our app to fetch all the relevant data from the database when a client requests it.

Let’s call it get_graph() and place it in the db_operations.py module:

def get_graph(db):
   command = "MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;"
   relationships = db.execute_and_fetch(command)

   link_objects = []
   node_objects = []
   added_nodes = []
   for relationship in relationships:
       e = relationship['e']
       data = {"source": e.nodes[0], "target": e.nodes[1]}
       link_objects.append(data)

       n1 = relationship['n1']
       if not (n1.id in added_nodes):
           data = {"id": n1.id, "name": n1.properties['name']}
           node_objects.append(data)
           added_nodes.append(n1.id)

       n2 = relationship['n2']
       if not (n2.id in added_nodes):
           data = {"id": n2.id, "name": n2.properties['name']}
           node_objects.append(data)
           added_nodes.append(n2.id)
   data = {"links": link_objects, "nodes": node_objects}

   return json.dumps(data)

First, we need to execute the openCypher query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e; and return its results from the database. These results will contain all the edges in the graph as well as all the nodes that are connected to those edges. Nodes that don't have connections will not be returned and that's ok for now.

The results (the object relationships) are in the form of a generator which we can iterate over and access its contents by using the node/edge names specified in our initial query (n1,n2 and e).

We also need to check if a node has already been appended to the node_objects list because multiple edges can contain (point to or from) the same node. All of the objects are stored in key-value pairs suitable for later JSON conversion.

The final result is a JSON object containing:

links: all the relationships that are in the graph as pairs of source and target id properties,
nodes: all the nodes from the graph that form relationships with other nodes.

In your app.py module add the following method:

@app.route("/get-graph", methods=["POST"])
def get_graph():
   db = Memgraph()
   response = make_response(
       jsonify(db_operations.get_graph(db)), 200)
   return response

This method is responsible for responding to POST requests from the client. It returns the graph data that we fetched from the server in the previous method.

Now let's do something with this data! Copy the contents for your index.js file from here
and the style.css file from here.

We also need to add the actual SVG graphic to our page so change the index.html file to:

{% extends 'base.html' %} {% block content %}
<div class="container">
    <svg class="border rounded mt-3" width="960" height="600" style="background-color:white"></svg>
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

I won't go into much detail about how to use D3.js so if you want to find out more I encourage you to visit their website.

In short, we fetch all the nodes and edges from the database and add them to an SVG element. The visual representation of the graph is made by simulating how physical forces act on particles (charge and gravity). You can drag and drop the nodes, hover over them to see the value of their name property, zoom in and out of the graph and move the SVG graphic.

Additional Functionalities

Go ahead and copy the file query.js to the directory static/js and query.html to the directory templates. You can find the updated base.html file here. Copy the necessary methods from the db_operations.py module and app.py module.

After you made the changes, just open http://localhost:5000/query/ and see the results.

This page will make your life easier if you want to debug the data being fetched from the server. It returns all the nodes or edges and shows them in a JSON highlighted format.

Your current project structure should like this:

sng-demo
├── resources
│  ├── data_big.py
│  └── data_small.txt
├── sng_demo
│  ├── __init__.py
│  ├── db_operations.py
│  └── database
│     ├── __init__.py
│     ├── memgraph.py
│     ├── connection.py
│     └── models.py
├── templates
│  ├── base.html
│  ├── index.html
│  └── query.html
├── static
│   ├── css
│   │  └── style.css
│   └── js
│      ├── index.js
│      └── query.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

Conclusion

Even though graph databases have been around for a long time, they are still not considered a mainstream tool in software development. Relational database-management systems model data as a set of predetermined structures. Complex joins and self-joins are necessary when the dataset becomes too inter-related. Modern datasets require technically complex queries which are often very inefficient in real-time scenarios.

Graph databases offer powerful data modelling and analysis capabilities for many real-world problems such as social networks, business relationships, dependencies, shipping, logistics... and they have been adopted by many of the worlds leading tech companies. With this tutorial, I hope to shed some light on how easy it is to integrate a graph database in your development process and I encourage you to try it out yourself.

As I said at the beginning, feel free to ask us any questions about this tutorial or Memgraph in general on StackOverflow with the tag memgraphdb or on our Discord server. Good luck with your coding!