DEV Community

aymane
aymane

Posted on

OBDA and Virtual Knowledge Graphs: A Practical Introduction

#ai #ontology #knowledgegraph

When working with enterprise data, one problem appears again and again: the data is spread across different systems, but applications need to work with it as if it belonged to one coherent model.

A company might have:

  • PostgreSQL databases
  • Oracle databases
  • data warehouses
  • legacy systems
  • APIs
  • CSV files
  • different databases owned by different teams

The schemas are usually different, even when the systems describe related things.

For example, one system might have:

customer_id
Enter fullscreen mode Exit fullscreen mode

another:

client_number
Enter fullscreen mode Exit fullscreen mode

and another:

account_id
Enter fullscreen mode Exit fullscreen mode

All three might represent the same business concept: a customer.

One approach is to copy everything into a new database or knowledge graph.

Another approach is to leave the data where it is and put a semantic layer on top of it.

This is where Ontology-Based Data Access (OBDA) and Virtual Knowledge Graphs (VKGs) come in.

What is OBDA?

Ontology-Based Data Access is an approach where users access data through an ontology instead of directly interacting with the schemas of the underlying databases.

The ontology provides the conceptual model.

For example:

Customer
Order
Product

Customer ── places ──> Order
Order ── contains ──> Product
Enter fullscreen mode Exit fullscreen mode

The database doesn't have to use these names.

It could look like:

crm_customer
------------
cust_no
cust_nm
country_code
Enter fullscreen mode Exit fullscreen mode

and:

sales_order
-----------
order_no
cust_no
total
Enter fullscreen mode Exit fullscreen mode

The ontology can still expose these tables as:

Customer
Order
Enter fullscreen mode Exit fullscreen mode

The connection between the physical schema and the ontology is defined through mappings.

The important idea is:

Ontology
   ↓
Mappings
   ↓
Data sources
Enter fullscreen mode Exit fullscreen mode

The ontology describes what the data means.

The mappings describe where that data comes from.

The actual data remains in the original systems.

The Three Main Components

An OBDA system can be understood through three main components:

  1. Ontology
  2. Mappings
  3. Data sources

1. Ontology

The ontology describes the concepts and relationships that users work with.

For example:

Customer
Order
Product
Enter fullscreen mode Exit fullscreen mode

with relationships:

Customer ── places ──> Order
Order ── contains ──> Product
Enter fullscreen mode Exit fullscreen mode

It can also contain additional semantic information.

For example:

PremiumCustomer subclassOf Customer
Enter fullscreen mode Exit fullscreen mode

or:

Order hasCustomer Customer
Enter fullscreen mode Exit fullscreen mode

The ontology represents the domain from the point of view of the application or user.

It does not need to mirror the database schema.

2. Mappings

Mappings connect the ontology to the physical data.

Suppose the database contains:

SELECT id, name, country
FROM customers;
Enter fullscreen mode Exit fullscreen mode

A mapping can define that the rows returned by this query represent instances of:

Customer
Enter fullscreen mode Exit fullscreen mode

and that:

customers.name
Enter fullscreen mode Exit fullscreen mode

represents:

Customer.name
Enter fullscreen mode Exit fullscreen mode

while:

customers.country
Enter fullscreen mode Exit fullscreen mode

represents:

Customer.country
Enter fullscreen mode Exit fullscreen mode

The database might therefore look like:

customers
---------
id
name
country
Enter fullscreen mode Exit fullscreen mode

while the semantic model looks like:

Customer
 ├── id
 ├── name
 └── country
Enter fullscreen mode Exit fullscreen mode

The mapping connects the two.

This separation is important because the physical schema can change without necessarily changing the conceptual model.

3. Data Sources

The data sources are the systems containing the actual data.

For example:

PostgreSQL
Oracle
MySQL
SQL Server
Data warehouse
Legacy database
API
Enter fullscreen mode Exit fullscreen mode

An OBDA system can sit above these sources and provide a semantic interface.

The important point is that the data does not have to be copied into the semantic layer.

What is a Virtual Knowledge Graph?

A knowledge graph represents information as entities and relationships.

For example:

Customer123 ── places ──> Order456

Order456 ── contains ──> Product789
Enter fullscreen mode Exit fullscreen mode

With RDF, this can be represented as triples:

Customer123 → places → Order456
Order456    → contains → Product789
Enter fullscreen mode Exit fullscreen mode

A Virtual Knowledge Graph provides this graph representation without requiring the underlying data to be physically stored as RDF.

The database remains the source of truth.

Conceptually:

Relational database
        ↓
     Mappings
        ↓
Virtual RDF graph
Enter fullscreen mode Exit fullscreen mode

The graph is virtual because it is derived from the source data when needed.

Materialized Knowledge Graph vs Virtual Knowledge Graph

There are two different approaches.

Materialized Knowledge Graph

The data is extracted and loaded into a graph database.

Database
   │
   │ ETL
   ▼
Knowledge Graph
Enter fullscreen mode Exit fullscreen mode

The graph contains its own copy of the data.

This can be useful when graph queries need to be served independently of the original database.

But it introduces another copy of the data.

You now have to think about:

Source database
       ↓
ETL
       ↓
Knowledge graph
Enter fullscreen mode Exit fullscreen mode

and keeping the two synchronized.

Virtual Knowledge Graph

With a VKG:

Database
   │
   ▼
Mappings
   │
   ▼
Virtual graph
Enter fullscreen mode Exit fullscreen mode

The graph is a semantic view over the existing data.

There is no requirement to first copy every row into a graph database.

If the database changes, queries against the virtual graph can see the new data because the data is still being obtained from the source.

This is one of the main reasons VKGs are useful when organizations already have large relational systems.

Querying a Virtual Knowledge Graph

A VKG is normally queried using SPARQL, the standard query language for RDF.

For example:

SELECT ?customer
WHERE {
    ?customer :hasCountry "Morocco" .
}
Enter fullscreen mode Exit fullscreen mode

The user is querying the semantic model.

The database does not necessarily contain a column called:

hasCountry
Enter fullscreen mode Exit fullscreen mode

The mapping might tell the system that:

:hasCountry
Enter fullscreen mode Exit fullscreen mode

corresponds to:

customers.country
Enter fullscreen mode Exit fullscreen mode

The VKG system can then generate a database query similar to:

SELECT id
FROM customers
WHERE country = 'Morocco';
Enter fullscreen mode Exit fullscreen mode

The user works with the ontology.

The database executes SQL.

The system connects the two.

Query Rewriting

Query rewriting is one of the central mechanisms behind OBDA.

Consider this query:

SELECT ?customer
WHERE {
    ?customer :hasCountry "Morocco" .
}
Enter fullscreen mode Exit fullscreen mode

The query is expressed using the ontology.

The OBDA system has to determine how that semantic query can be answered using the underlying database.

A simplified process is:

SPARQL query
     ↓
Ontology-level rewriting
     ↓
Mapping expansion
     ↓
SQL query
     ↓
Database
Enter fullscreen mode Exit fullscreen mode

For example:

:hasCountry
Enter fullscreen mode Exit fullscreen mode

could be mapped to:

customers.country
Enter fullscreen mode Exit fullscreen mode

and:

:Customer
Enter fullscreen mode Exit fullscreen mode

could be mapped to:

customers
Enter fullscreen mode Exit fullscreen mode

The resulting SQL can then be executed directly by the database.

The VKG Query Pipeline

A simplified VKG query pipeline looks like this:


SPARQL query
      ↓
Query rewriting
      ↓
Rewritten query
      ↓
Mapping unfolding
      ↓
SQL query
      ↓
Database
      ↓
SQL result
      ↓
SPARQL result
Enter fullscreen mode Exit fullscreen mode

There are two important transformations here.

Rewriting

The ontology is used to expand the query according to the semantic relationships defined in the ontology.

Unfolding

The mappings are used to translate the ontology-level query into a query over the actual data sources.

The result is a database query that can be executed by the underlying system.

Why Use an Ontology?

A database schema tells you how data is stored.

An ontology is concerned with what the data means.

Consider two databases:

Database A

client_id
client_name
Enter fullscreen mode Exit fullscreen mode

and:

Database B

customer_number
customer_name
Enter fullscreen mode Exit fullscreen mode

From the database perspective, these are different schemas.

From the business perspective, both might represent:

Customer
Enter fullscreen mode Exit fullscreen mode

An ontology can provide that common conceptual layer.

             Customer
             /      \
            /        \
       Database A   Database B
Enter fullscreen mode Exit fullscreen mode

The applications don't need to know that one system calls it client_id and another calls it customer_number.

OBDA vs Data Warehouse

OBDA and a data warehouse solve different problems.

A data warehouse physically stores integrated data.

Source A ─┐
Source B ─┼──> ETL ──> Data Warehouse
Source C ─┘
Enter fullscreen mode Exit fullscreen mode

OBDA instead creates a semantic access layer:

                 Ontology
                    │
                 Mappings
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
       Source A  Source B  Source C
Enter fullscreen mode Exit fullscreen mode

This doesn't mean one approach is better than the other.

They can also be used together.

For example:

                    Semantic Layer
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
         PostgreSQL    Warehouse     Data Lake
Enter fullscreen mode Exit fullscreen mode

The ontology provides a common vocabulary while the underlying systems remain responsible for storing and processing the data.

OBDA vs Knowledge Graph

The terms OBDA and knowledge graph are related, but they are not the same thing.

A knowledge graph is a way of representing information as entities and relationships.

OBDA is an architecture for accessing data through an ontology.

A knowledge graph can be materialized:

Database
   ↓
ETL
   ↓
Graph database
Enter fullscreen mode Exit fullscreen mode

or virtual:

Database
   ↓
Mappings
   ↓
Virtual Knowledge Graph
Enter fullscreen mode Exit fullscreen mode

So a VKG can be viewed as a knowledge graph that is exposed virtually rather than requiring the entire graph to be materialized first.

OBDA Architecture

Putting the pieces together:


                 Application
                      │
                      │ SPARQL
                      ▼
                 ┌───────────┐
                 │ Ontology  │
                 └─────┬─────┘
                       │
                    Mappings
                       │
              ┌────────┼────────┐
              ▼        ▼        ▼
           Database  Database  Database
Enter fullscreen mode Exit fullscreen mode

A real OBDA implementation adds more components around this basic model.

For example:

SPARQL
   ↓
Query Rewriter
   ↓
Mapping / Query Unfolder
   ↓
SQL Generator
   ↓
Database
Enter fullscreen mode Exit fullscreen mode

The exact architecture depends on the implementation.

Where the Complexity Actually Goes

At first glance, the architecture looks simple:

Ontology
   +
Mappings
   +
Database
Enter fullscreen mode Exit fullscreen mode

The difficult part starts when the semantic query becomes complicated.

For example:

Customer
    ↓
Order
    ↓
Product
    ↓
Supplier
Enter fullscreen mode Exit fullscreen mode

might correspond to several joins in the underlying relational schema.

The OBDA engine needs to transform the semantic query into an efficient database query.

That means the system has to deal with things such as:

  • query rewriting
  • SQL generation
  • join optimization
  • database constraints
  • query planning
  • source capabilities
  • different SQL dialects
  • large intermediate results

This is why OBDA is closely related to database query processing.

The ontology gives the system the semantic model, but the database still has to execute the final query efficiently.

A Simple Mental Model

A useful way to think about the whole architecture is:

Ontology
"What does this data mean?"

        ↓

Mappings
"Where does this information come from?"

        ↓

VKG
"What does the integrated data look like semantically?"

        ↓

Query rewriting
"How can this semantic query be answered?"

        ↓

SQL
"How do I ask the underlying database?"

        ↓

Database
"Execute it."
Enter fullscreen mode Exit fullscreen mode

That separation is the core idea behind OBDA and Virtual Knowledge Graphs.

Conclusion

OBDA provides a way to put a semantic layer over existing data sources.

The basic model is:

Ontology
    +
Mappings
    +
Data Sources
Enter fullscreen mode Exit fullscreen mode

A Virtual Knowledge Graph builds on this idea by exposing the underlying data as a virtual RDF graph.

The data can remain in relational databases while users query it through concepts such as:

Customer
Order
Product
Supplier
Enter fullscreen mode Exit fullscreen mode

The system then handles the translation:

SPARQL
   ↓
Ontology reasoning / rewriting
   ↓
Mapping unfolding
   ↓
SQL
   ↓
Database
Enter fullscreen mode Exit fullscreen mode

The main advantage is the separation between how data is stored and how data is understood.

The database can continue using tables and columns designed for storage and operations, while the semantic layer provides a model designed around the domain.

That separation is what makes OBDA and Virtual Knowledge Graphs useful when working with heterogeneous data sources without immediately moving everything into a new physical knowledge graph.

Top comments (0)