DEV Community: Pieter Heyvaert

Learn how to create a website on top of decentralized data

Pieter Heyvaert — Wed, 02 Sep 2020 11:44:41 +0000

Walder offers an easy way to set up a website or Web API on top of decentralized knowledge graphs. Knowledge graphs incorporate data together with the meaning of that data. This makes it possible to combine data from multiple knowledge graphs, even if different, independent parties maintain or host them. Knowledge graphs can be hosted via Solid PODs, SPARQL endpoints, Triple Pattern Fragments interfaces, RDF files, and so on. Using content negotiation, Walder makes the data in these knowledge graphs available to clients via HTML, RDF, and JSON-LD. Users define in a configuration file which data Walder uses and how it processes this data. This tutorial introduces you to Walder.

Table of contents

Before we start the tutorial
Example
How to start a configuration file
How to add knowledge graphs
How to add paths
How to add views
How to run Walder
How to process query results
How to add parameters to paths
Bonus: content negotiation
Wrapping up
More information

1. Before we start the tutorial

1.1. Learning objective

At the end of the tutorial you will be able to set up a website on top of decentralized knowledge graphs by using Walder.

1.2. Prerequisites

We assume that you are familiar with

knowledge graphs and more specific the Resource Description Framework (RDF)
bash
Node.js (v12 or higher) and npm

1.3. How to use the tutorial

There are two ways to complete this tutorial: you read the explanations and either

read the examples, or
try out the examples yourself using your computer.

You can find the files of this tutorial in this repository.

2. Example

For this tutorial we want to create a website that displays our ratings of the tv shows we watched. Our website is getting the required data from two knowledge graphs containing the following data:

our tv show ratings
details about these tv shows, such as title, release year and so on.

We will combine the data in both knowledge graphs to create these two Web pages:

list of tv shows with their ratings at the path /ratings
list of tv shows with "x"-star ratings at the path /rating/{x}, where "x" is a value between 1 and 5.

3. How to start a configuration file

A configuration file for Walder is a valid OpenAPI 3.0 file with Walder-specific extensions. These extensions start with x-walder- in the configuration file.

We do the following steps

Create a config file called config.yaml
Add the following content to the file:

openapi: 3.0.2
info:  
  title: 'Ratings website'
  version: 1.0.0

Let's have a look at this.

openapi states the OpenAPI version we are using.
info contains the title and version of our website.

4. How to add knowledge graphs

You add knowledge graphs, also called data sources, to Walder by listing them under the section x-walder-datasources. For the two aforementioned knowledge graphs this results in

x-walder-datasources:
  - https://pieterheyvaert.com/data/example/walder/ratings.ttl # Turtle file
  - https://data.betweenourworlds.org/latest # Triple Pattern Fragments server

Add this section to the root of config.yaml.

Although both data sources are hosted using different technologies, you do not need to specify these technologies
as Walder determines the correct way to query a data source
on the fly. If interested, you can open both data sources in your browser and explore them.

5. How to add paths

You add paths under the section paths, which is defined in the OpenAPI specification. The following is an entry for the path ratings:

paths:
  /ratings:
    get:
      summary: Returns a list of all tv shows and their ratings.
      responses:
        200:
          description: List of all tv shows and their ratings.
          x-walder-input-text/html: ratings.handlebars
      x-walder-query:
        graphql-query: >
          {
            id @single
            title @single
            review @single {
              rating @single {
                value @single
              }
            }
          }
        json-ld-context: >
          {
            "schema": "http://schema.org/",
            "review": "schema:review",
            "rating": "schema:reviewRating",
            "value": "schema:ratingValue",
            "title": "schema:name"
          }

Add this section to the root of config.yaml.

Let's have a closer look at this entry.

/ratings is the path we want and is the section where we add the details about that path.
get says that for this path we want to support an HTTP GET and it is the section where we add the details about the GET.
summary contains a summary of the request.
responses is a section that contains all supported responses based on their HTTP status code.
200 is a section that contains all the information to handle a response with status code 200.
description contains a description of the response.
x-walder-input-text/html points to a template-engine file that renders an HTML file, which is send back to the client as response. Walder supports different template engines, such as Pug and Handlebars. In our example, we have a Handlebars file called ratings.handlebars (which we will create later).
x-walder-query is a section that contains information about query that is executed over the knowledge graphs. Walder gives the result of this query to the template engine, which uses this result during the rendering of the HTML file.
graphql-query contains the GraphQL-LD query that is executed over the knowledge graphs.
json-ld-context contains the JSON-LD context that is needed for the GraphQL-LD query.

Let's have a closer look at the query and the context. The GraphQL-LD query is

{
  id @single
  title @single
  review @single {
    rating @single {
      value @single
    }
  }
}

This query returns the URLs, titles, and reviews of the shows. The variable id contains the URL. The variable title is a string with the title. The variable review is an object with a rating, which is also an object having the variable value with the actual rating value. We use @single to get strings and objects instead of arrays of strings and objects, which is the default.

The query in itself is not enough,because these variables are not defined across all data sources. To tackle this, we use this JSON-LD context:

{
  "schema": "http://schema.org/",
  "review": "schema:review",
  "rating": "schema:reviewRating",
  "value": "schema:ratingValue",
  "title": "schema:name"
}

The context defines the prefix for http://schema.org/ and the URLs for all the variables. For example, review has as URL http://schema.org/review.

6. How to add views

In our path entry, we have x-walder-input-text/html: ratings.handlebars. Walder will look for the file ratings.handlebars in the folder views relative to the location of the config file. We do the following steps:

Create a folder views.
Add a file ratings.handlebars with the following content:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
</head>
<body>
<h1>Ratings</h1>
<div>
    <ul>
       {{#each data}}
            <li>{{this.title}}: {{this.review.rating.value}}/5 </li>
        {{/each}}
    </ul>
</div>
</body>
</html>

It is important to note here that the results of the query are accessed via the variable data, as done on the line 10 with {{#each data}} in the above code.

You can find the files we created up until this point here.

7. How to run Walder

We now have all the files in place to use Walder for the first time. If you do not have Walder installed, you can do so via

npm i -g walder

You run Walder via

walder -c config.yaml

You will see this in the output

info 2020-08-26T13:37:03.152Z : Starting static server...
info 2020-08-26T13:37:03.153Z : Static server running.
info 2020-08-26T13:37:03.154Z : Parsing route /ratings
info 2020-08-26T13:37:03.154Z :     - Parsing method get
info 2020-08-26T13:37:03.209Z : Starting server...
info 2020-08-26T13:37:03.212Z : Started listening on port: 3000.

The output is configurable via the -l, --log option. See walder -h for more information and other options.

Next, open a browser and navigate to http://localhost:3000/ratings. The result is a list of tv shows together with their ratings:

8. How to process query results

When checking the results on http://localhost:3000/ratings
we see that there are duplicate results: certain tv shows are shown multiple times. This is due to the fact that tv shows have different titles, possibly in different languages.

Walder offers the functionality to process the results of a query. You do this via the x-walder-postprocessing section, which has a section for every function that needs to be executed on the results.

In our example this is

x-walder-postprocessing:
  keepSingleENTitle:
    source: utils.js

This means that we want to use the function keepSingleENTitle which can be found in the file utils.js. Walder will look for this file in the folder pipe-modules relative to the location of the config file. We do the following steps:

Create a folder pipe-modules.
Add a file utils.js with the following content:

module.exports = {
  keepSingleENTitle: (queryResults) => {
    const originalData = queryResults.data;

    // Keep only English titles.
    const english = /^[A-Za-z0-9 .]*$/;
    const showsWithEnglishTitles = originalData.filter(show => english.test(show.title) );

    // Filter duplicate shows.
    const ids = []
    const shows = []

    showsWithEnglishTitles.forEach(show => {
      // Only add show when the id hasn't been encountered yet.
      if (ids.indexOf(show.id) === -1) {
        ids.push(show.id);
        shows.push(show);
      }
    });

    queryResults.data = shows;
    return queryResults;
  }
};

The resulting config file is

openapi: 3.0.2
info:
  title: 'Ratings website'
  version: 1.0.0
x-walder-datasources:
  - https://data.betweenourworlds.org/latest
  - https://pieterheyvaert.com/data/example/walder/ratings.ttl
paths:
  /ratings:
    get:
      summary: Returns a list of all tv shows and their ratings.
      responses:
        200:
          description: List of all tv shows and their ratings.
          x-walder-input-text/html: ratings.handlebars
      x-walder-query:
        graphql-query: >
          {
            id @single
            title @single
            review @single {
              rating @single {
                value @single
              }
            }
          }
        json-ld-context: >
          {
            "schema": "http://schema.org/",
            "review": "schema:review",
            "rating": "schema:reviewRating",
            "value": "schema:ratingValue",
            "title": "schema:name"
          }
      x-walder-postprocessing:
        keepSingleENTitle:
          source: utils.js

If you run Walder again and browse to http://localhost:3000/ratings we see every show only once with an English title:

You can find the files we created up until this point here.

9. How to add parameters to paths

The second Web page of our website lists all tv shows with "x" star ratings at the path /rating/{x}, where "x" is a value between 1 and 5. The corresponding path entry is

/rating/{value}:
get:
  summary: Returns a list of all tv shows with a given rating.
  parameters:
    - in: path
      name: value
      required: true
      schema:
        type: integer
      description: Rating of the tv shows.
  responses:
    200:
      description: List of all tv shows with a given rating.
      x-walder-input-text/html: selected-rating.handlebars
  x-walder-query:
    graphql-query: >
      {
        id @single
        title @single
        review @single {
          rating(value: $value)
        }
      }
    json-ld-context: >
      {
        "schema": "http://schema.org/",
        "review": "schema:review",
        "rating": "schema:reviewRating",
        "value": "schema:ratingValue",
        "title": "schema:name"
      }
  x-walder-postprocessing:
    keepSingleENTitle:
      source: utils.js

The differences with the first path entry are

the addition of a parameters section which contains all information about the supported parameters, and
the use of a parameter value in the GraphQL-LD query.

The parameters section contains an entry for every parameter. This path entry only has one parameter:

in: path
name: value
required: true
schema:
  type: integer
description: Rating of the tv shows.

in defines where the parameter can be found. In this case that is the path.
name defines the name of the parameter. In this case the name is value.
required defines whether a parameter is required or not. In this case the parameter is required.
schema defines the schema of the parameter. In this case we only make us of the type key, which states the data type of the parameter. The type is integer, because a rating is an integer between 1 and 5.
description contains the description of the parameter.

To use the value of the parameter in our query, we use $value. That is $ and the name of the parameter. Our query is

{
  id @single
  title @single
  review @single {
    rating(value: $value)
  }
}

rating(value: $value) means that we only want the ratings that have values that match $value.

The view for this path is in the file selected-rating.handlebars with this content:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
</head>
<body>
<h1>TV shows</h1>
<div>
    <ul>
        {{#each data}}
            <li>{{this.title}}</li>
        {{/each}}
    </ul>
</div>
</body>
</html>

The only difference with the previous view is that in this one the ratings are not shown, because they are the same for all tv shows on the Web page and determined by the parameter in the path.

Do the following steps to update our files:

Add the aforementioned path in config.yaml.
Add the file selected-rating.handlebars to the folder views.

If you run Walder again and browse to http://localhost:3000/rating/2 we only see one show with a two-star rating:

You can find all files here.

10. Bonus: content negotiation

Besides offering HTML pages, Walder can also offer RDF and JSON-LD. This is supported through the use of content negotiation. You use the Accept header to achieve this. Let's have a look at the following cURL commands.

curl -H 'accept: text/turtle' http://localhost:3000/rating/2

This command returns all shows with a two-star rating in the Turtle serialization:

<https://betweenourworlds.org/anime/bakuman> <http://schema.org/name> "Bakuman.";
    <http://schema.org/review> _:b1_b0.
_:b1_b0 <http://schema.org/reviewRating> "bc_0_n3-71".

curl -H 'accept: application/ld+json' http://localhost:3000/rating/2

This command returns the same data, but uses JSON-LD instead:

{
  "@context": {
    "schema":"http://schema.org/",
    "review":"schema:review",
    "rating":"schema:reviewRating",
    "value":"schema:ratingValue",
    "title":"schema:name"
  },
  "@graph":[
    {
      "title":"Bakuman.",
      "review":{
        "rating":["bc_0_n3-71"]
      },
      "@id":"https://betweenourworlds.org/anime/bakuman"
    }
  ]
}

curl -H 'accept: text/html' http://localhost:3000/rating/2

This command returns the HTML version of the data, which is what is shown in the browser.

11. Wrapping up

Congratulations! You have created your first website using Walder that

offers two Web pages
is based on two different, decentralized knowledge graphs
supports content negotiation to return the data in different formats and serializations.

Nice work! We hope you now feel like you have a decent grasp on how Walder works.

12. More information

You can find more information in the following:

Generate RDF from an XML file

Pieter Heyvaert — Wed, 22 Apr 2020 06:57:26 +0000

Table of Contents

Before we start the tutorial
Example
What rules are needed
How to start a document with RML rules
What data to use
How to generate subjects
How to generate predicates and objects
Complete Turtle document with RML rules
Wrapping up
More information

1 Before we start the tutorial

1.1 What you learn

At the end of the tutorial you will be able to generate RDF from an XML file using RML rules.

What you need

We assume that you understand

RDF
XML
vocabularies and ontologies, such as classes, properties, and datatypes

How you use the tutorial

There are two ways to complete this tutorial: you read the explanations and either

Read the examples.
Try out the examples yourself by writing and executing RML rules on your computer.

For the second option you need a tool that executes RML rules. Suggestions are the RMLMapper and the RMLStreamer.

2 Example

Consider the following XML file called "characters.xml":

<?xml version="1.0" encoding="UTF-8"?>

<characters>
  <character id="0">
    <firstname>Ash</firstname>
    <lastname>Ketchum</lastname>
    <hair>black</hair>
  </character>
  <character id="1">
    <firstname>Misty</firstname>
    <hair>orange</hair>
  </character>
</characters>

It contains the information about two different characters. The id, first name, last name, and hair color are included. The latter two are optional. We want to annotate every character and generate the corresponding RDF triples.

For example, consider the character described by the first XML element:

<character id="0">
  <firstname>Ash</firstname>
  <lastname>Ketchum</lastname>
  <hair>black</hair>
</character>

We want to generate the corresponding RDF triples for this element:

@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .
@prefix character: <http://example.org/character/> .

character:0 a schema:Person;
  schema:givenName "Ash";
  schema:lastName "Ketchum";
  dbo:hairColor "black".

In the following sections we explain

what rules you need to generate these triples, and
how you write them using RML.

3 What rules are needed

Two sets of rules are needed:

rules that describe the XML file
rules that define how the RDF terms are generated from the XML file, and how these terms are used to generate triples.

In our example we need rules that define that:

The IRI representing a character is generated by concatenating http://example.org/character/ with the character's id.
This IRI is used as subject of the triples.
A character is annotated with the class schema:Person.
The first name is annotated with the property schema:givenName.
The last name is annotated with the property schema:lastName.
The hair color is annotated with the property dbo:hairColor.

4 How to start a document with RML rules

We write the RML rules in a Turtle document. RML rules are RDF themselves.

We add the following prefixes:

Prefix	Description
`rml`	RML ontology
`rr`	The R2RML ontology, which is extended by RML
`ql`	The Query Language vocabulary, which is used together with RML
`rdf`	The RDF Concepts Vocabulary
empty	The prefix used for our RML rules
`schema`	The schema.org vocabulary
`dbo`	The DBpedia ontology

The last two are added because they are used for the classes and properties.

The prefixes are added in Turtle like this:

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

5 What data to use

In our example the data of the characters is stored in an XML file. We add the following RML rules that define what XML file is used and how we iterate over the elements in it:

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source "characters.xml";
    rml:referenceFormulation ql:XPath;
    rml:iterator "/characters/character"
  ].

The different rules work as follows:

:TriplesMap a rr:TriplesMap; defines the Triples Map that groups all rules for the characters.
The blank node of :TriplesMap rml:logicalSource [ ... ] contains all rules about the XML file. The class of the blank node is implicitly of the class rml:LogicalSource.
[rml:source "characters.xml"] says that we access the XML file characters.xml.
[rml:referenceFormulation ql:XPath] says that we use XPath the access the data in the XML file.
[rml:iterator "/characters/character"] says that we iterate over all elements that match the XPath expression /characters/character.

6 How to generate subjects

We add the following rules that define how the subject IRI of a character is generated:

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{@id}"
].

The different rules work as follows:

:TriplesMap rr:subjectMap [ ... ] contains all the rules about the subject of a triple. The class of the blank node is implicitly of the class rr:SubjectMap.
[rr:template "http://example.org/character/{@id}"] says that the IRI of the subject is generated by concatenating http://example.org/character/ with the attribute id of the character element.

7 How to generate predicates and objects

7.1 How to annotate with a class

In our example we need to annotate every character with the class schema:Person. We add the following RML rules:

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:constant schema:Person
].

The different rules work as follows:

:TriplesMap rr:predicateObjectMap [ ... ] contains all the rules about a specific predicate of a triple. The class of the blank node is implicitly of the class rr:PredicateObjectMap.
[rr:predicate rdf:type] says that we use the predicate rdf:type.
[rr:objectMap [ ... ]] contains all the rules about the object of a triple. The class of the blank node is implicitly of the class rr:ObjectMap.
[rr:constant schema:Person] says that the object of the triple is schema:Person for every character.

Putting all rules we have so far together results in

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source "characters.xml";
    rml:referenceFormulation ql:XPath;
    rml:iterator "/characters/character"
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{@id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
    rr:constant schema:Person
  ]
].

You can download the Turtle file here. If we execute these rules, the following triples are generated:

@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person .
<http://example.org/character/1> a schema:Person .

Two triples are generated: one for each character. There is a unique subject IRI for each character and each character is annotated with the class schema:Person.

7.2 How to annotate with a property

In our example we need to annotate the values in the tags firstname
with the property schema:givenName. We add the following rules:

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

The rules are different from when annotating with a class: rml:reference is used instead of rr:constant because the object is not the same for every character. More specific, [rml:reference "firstname"] says that the data in the tag firstname is used for the object.

Putting all rules we have so far together results in

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source "characters.xml";
    rml:referenceFormulation ql:XPath;
    rml:iterator "/characters/character"
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{@id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
   rr:constant schema:Person
 ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

You can download the Turtle file here. If we execute these rules, the following triples are generated:

@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person;
  schema:givenName "Ash" .

<http://example.org/character/1> a schema:Person;
  schema:givenName "Misty" .

Two triples are added: one for the first name of each character.

We add the following rules to annotate the last name and the hair color in the same way as the first name:

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:lastName;
  rr:objectMap [
    rml:reference "lastname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate dbo:hairColor;
  rr:objectMap [
    rml:reference "hair"
  ]
].

8 Complete Turtle document with RML rules

The complete Turtle document with RML rules is

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source "characters.xml";
    rml:referenceFormulation ql:XPath;
    rml:iterator "/characters/character"
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{@id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
   rr:constant schema:Person
 ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:lastName;
  rr:objectMap [
    rml:reference "lastname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate dbo:hairColor;
  rr:objectMap [
    rml:reference "hair"
  ]
].

You can download the Turtle file here. If we execute these rules, the final triples are generated:

@prefix dbo: <http://dbpedia.org/ontology/> .
@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person;
  dbo:hairColor "black";
  schema:givenName "Ash";
  schema:lastName "Ketchum" .

<http://example.org/character/1> a schema:Person;
  dbo:hairColor "orange";
  schema:givenName "Misty" .

9 Wrapping up

Congratulations! You have created your own RML rules that generate RDF from data in an XML file. Nice work! We hope you now feel like you have a decent grasp on how RML rules work.

10 More information

You can find more information about RML in its specification. There is also a human readable text-based representation available for RML rules called YARRRML. It is a subset of YAML, a widely used data serialization language designed to be human-friendly.

If you have any questions or remarks, don't hesitate to contact me via email or via Twitter.

Generate RDF from TSV file

Pieter Heyvaert — Tue, 07 Apr 2020 13:55:54 +0000

Table of Contents

Before we start the tutorial
Example
What rules are needed
How to start a document with RML rules
What data to use
How to generate subjects
How to generate predicates and objects
Complete Turtle document with RML rules
Wrapping up
More information

1 Before we start the tutorial

1.1 What you learn

At the end of the tutorial you will be able to generate RDF from a TSV file using RML rules.

What you need

We assume that you understand

RDF
TSV
vocabularies and ontologies, such as classes, properties, and datatypes

How you use the tutorial

There are two ways to complete this tutorial: you read the explanations and either

Read the examples.
Try out the examples yourself by writing and executing RML rules on your computer.

For the second option you need a tool that executes RML rules.
Suggestions are the RMLMapper and the RMLStreamer.

2 Example

Consider the following TSV file called "characters.tsv":

id firstname lastname hair
0 Ash Ketchum black
1 Misty  orange

For example, consider the character described by the first row:

0 Ash Ketchum black

We want to generate the corresponding RDF triples for this row:

@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .
@prefix character: <http://example.org/character/> .

character:0 a schema:Person;
  schema:givenName "Ash";
  schema:lastName "Ketchum";
  dbo:hairColor "black".

In the following sections we explain

what rules you need to generate these triples, and
how you write them using RML.

3 What rules are needed

Two sets of rules are needed:

rules that describe the TSV file
rules that define how the RDF terms are generated from the TSV file, and how these terms are used to generate triples.

In our example we need rules that define that:

The IRI representing a character is generated by concatenating http://example.org/character/ with the character's id.
This IRI is used as subject of the triples.
A character is annotated with the class schema:Person.
The first name is annotated with the property schema:givenName.
The last name is annotated with the property schema:lastName.
The hair color is annotated with the property dbo:hairColor.

4 How to start a document with RML rules

We write the RML rules in a Turtle document. RML rules are RDF themselves.

We add the following prefixes:

Prefix	Description
`rml`	RML ontology
`rr`	The R2RML ontology, which is extended by RML
`ql`	The Query Language vocabulary, which is used together with RML
`csvw`	The CSVW Vocabulary, which is used to describe the TSV file
`rdf`	The RDF Concepts Vocabulary
empty	The prefix used for our RML rules
`schema`	The schema.org vocabulary
`dbo`	The DBpedia ontology

{.table .table-responsive}

The last two are added because they are used for the classes and properties.

The prefixes are added in Turtle like this:

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix csvw: <http://www.w3.org/ns/csvw#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

5 What data to use

In our example the data of the characters is stored in a TSV file. We add the following RML rules that define what TSV file is used:

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source [
      a csvw:Table;
      csvw:url "characters.tsv";
      csvw:dialect [
        a csvw:Dialect;
        csvw:delimiter "\t"
      ]
    ];
    rml:referenceFormulation ql:CSV
  ].

The different rules work as follows:

:TriplesMap a rr:TriplesMap; defines the Triples Map that groups all rules for the characters.
The blank node of :TriplesMap rml:logicalSource [ ... ] contains all rules about the TSV file and how to access the data in the file. The class of the blank node is implicitly of the class rml:LogicalSource.
The blank node in [rml:source [ ... ] contains all rules about the TSV file.
[csvw:url "characters.tsv"] says that we access the TSV file characters.tsv.
[csvw:dialect [csvw:delimiter "\t"]] says that the delimiter used in this file is a tab (\t).
[rml:referenceFormulation ql:CSV] says that we use the column names to access the data in the TSV file.

Note that we access a TSV file as if it is a CSV file because we consider it a CSV file with a different delimiter.

6 How to generate subjects

We add the following rules that define how the subject IRI of a character is generated:

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{id}"
].

The different rules work as follows:

:TriplesMap rr:subjectMap [ ... ] contains all the rules about the subject of a triple. The class of the blank node is implicitly of the class rr:SubjectMap.
[rr:template "http://example.org/character/{id}"] says that the IRI of the subject is generated by concatenating http://example.org/character/ with the value in the column id.

7 How to generate predicates and objects

7.1 How to annotate with a class

In our example we need to annotate every character with the class schema:Person. We add the following RML rules:

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:constant schema:Person
].

The different rules work as follows:

:TriplesMap rr:predicateObjectMap [ ... ] contains all the rules about a specific predicate of a triple. The class of the blank node is implicitly of the class rr:PredicateObjectMap.
[rr:predicate rdf:type] says that we use the predicate rdf:type.
[rr:objectMap [ ... ]] contains all the rules about the object of a triple. The class of the blank node is implicitly of the class rr:ObjectMap.
[rr:constant schema:Person] says that the object of the triple is schema:Person for every character.

Putting all rules we have so far together results in

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix csvw: <http://www.w3.org/ns/csvw#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source [
      a csvw:Table;
      csvw:url "characters.tsv";
      csvw:dialect [ 
        a csvw:Dialect;
        csvw:delimiter "\t"
      ]
    ];
    rml:referenceFormulation ql:CSV
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
    rr:constant schema:Person
  ]
].

You can download the Turtle file here. If we execute these rules, the following triples are generated:

@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person .
<http://example.org/character/1> a schema:Person .

Two triples are generated: one for each character. There is a unique subject IRI for each character and each character is annotated with the class schema:Person.

7.2 How to annotate with a property

In our example we need to annotate the values in the column firstname with the property schema:givenName. We add the following rules:

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

Putting all rules we have so far together results in

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix csvw: <http://www.w3.org/ns/csvw#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source [
      a csvw:Table;
      csvw:url "characters.tsv";
      csvw:dialect [ 
        a csvw:Dialect;
        csvw:delimiter "\t"
      ]
    ];
    rml:referenceFormulation ql:CSV
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
   rr:constant schema:Person
 ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

You can download the Turtle file here. If we execute these rules, the following triples are generated:

@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person;
  schema:givenName "Ash" .

<http://example.org/character/1> a schema:Person;
  schema:givenName "Misty" .

Two triples are added: one for the first name of each character.

We add the following rules to annotate the last name and the hair color in the same way as the first name:

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:lastName;
  rr:objectMap [
    rml:reference "lastname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate dbo:hairColor;
  rr:objectMap [
    rml:reference "hair"
  ]
].

8 Complete Turtle document with RML rules

The complete Turtle document with RML rules is

@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix csvw: <http://www.w3.org/ns/csvw#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <http://example.org/rules/> .
@prefix schema: <http://schema.org/> .
@prefix dbo: <http://dbpedia.org/ontology/> .

:TriplesMap a rr:TriplesMap;
  rml:logicalSource [
    rml:source [
      a csvw:Table;
      csvw:url "characters.tsv";
      csvw:dialect [ 
        a csvw:Dialect;
        csvw:delimiter "\t"
      ]
    ];
    rml:referenceFormulation ql:CSV
  ].

:TriplesMap rr:subjectMap [
  rr:template "http://example.org/character/{id}"
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate rdf:type;
  rr:objectMap [
   rr:constant schema:Person
 ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:givenName;
  rr:objectMap [
    rml:reference "firstname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate schema:lastName;
  rr:objectMap [
    rml:reference "lastname"
  ]
].

:TriplesMap rr:predicateObjectMap [
  rr:predicate dbo:hairColor;
  rr:objectMap [
    rml:reference "hair"
  ]
].

You can download the Turtle file here. If we execute these rules, the final triples are generated:

@prefix dbo: <http://dbpedia.org/ontology/> .
@prefix schema: <http://schema.org/> .

<http://example.org/character/0> a schema:Person;
  dbo:hairColor "black";
  schema:givenName "Ash";
  schema:lastName "Ketchum" .

<http://example.org/character/1> a schema:Person;
  dbo:hairColor "orange";
  schema:givenName "Misty";
  schema:lastName "" .

9 Wrapping up

Congratulations! You have created your own RML rules that generate RDF from data in a TSV file. Nice work! We hope you now feel like you have a decent grasp on how RML rules work.

10 More information

If you have any questions or remarks, don't hesitate to contact me via email or via Twitter.

What is a knowledge graph - Pokémon edition

Pieter Heyvaert — Thu, 16 Jan 2020 10:54:33 +0000

Originally posted on my website.

The word "knowledge graph" is used more and more, but what exactly is a knowledge graph? In this blog post I will explain what it is through an example, based on the approach of the Resource Description Framework. And, as title suggests, the example will have as theme: Pokémon! Let's jump right in it!

But before we can really get started you need to pick your starter, trainer! Who will it be?

Charmander: a bipedal, reptilian fire-type Pokémon with a primarily orange body and blue eyes.
Bulbasaur: a small, quadruped grass/poison-type Pokémon that has blue-green skin with darker patches.
Squirtle: a small water-type Pokémon that resembles a light blue turtle.

Which one do you want?
I choose Charmander!
I choose Bulbasaur!
I choose Squirtle!

Charmander

Great! You picked the fire-type Pokémon. Here are some facts about Charmander:

Charmander is a fire-type Pokémon introduced in Generation I.
It evolves into Charmeleon starting at level 16, which evolves into Charizard starting at level 36. source

Let's see how we can represent the information in this text as a knowledge graph. At the centre of our graph is Charmander, as you can see below.

Graphs, including knowledge graphs, are formed of nodes and edges. Our graph contains one node: Charmander. Now let's have a look at the first sentence:

Charmander is a fire-type Pokémon introduced in Generation I.

This sentence can be split up in two bits of information:

Charmander is a fire-type Pokémon.
Charmander is introduced in Generation I.

We represent a fire-type Pokémon by the word "FIRE" in an orange rectangle:

Now our graph contains two nodes: Charmander and the fire-type Pokémon. What is the relationship between these two? "Charmander is a fire-type Pokémon" means that there is a relationship between our two nodes. We add that information to our graph by drawing an edge that connects the two nodes.

We are not done yet though. We know that there is a relationship, but what kind of relationship is it? "Charmander is a fire-type Pokémon" means that Charmander belongs to the group of fire-type Pokémon, or that Charmander is of the type fire. Thus, we add "type" to the relationship.

Note that the direction of the arrow is from Charmander to fire and not the other way around. If that were the case then our sentence would have been "A fire-type Pokémon is Charmander", or something like that. What doesn't make sense.

To add the second bit of information from the first sentence to the graph, we add a "I" to the graph that represents Generation I. Now there are three nodes in our graph.

Next, we draw an edge between Charmander and I to show that there is a relationship between the nodes. Finally, we also add what kind of relationship it is. Here we add "generation" to the relationship.

The second sentence is

It evolves into Charmeleon starting at level 16, which evolves into Charizard starting at level 36.

This sentence can be split up in two bits of information:

Charmander evolves into Charmeleon, starting at level 16.
Charmeleon evolves into Charizard, starting at level 36.

We add a new node for Charmeleon and add an "evolves" relationship between Charmander and Charmeleon.

Now, we want to add the fact that the evolution can only happen at level 16 or higher. Again, we add a node. This time for "16", but where do we add the relationship? Do we add it between Charmander and 16? Do we add it between Charmeleon and 16? This is possible, but what happens if a Pokémon can evolve into two different Pokémon? How do you know to which evolution the level belongs? Actually, we want to say that that specific evolve relationship only happens starting at level 16. Thus, we want to add "16" to the relationship between Charmander and Charmeleon. But that is not possible: extra information cannot be added to a relationship. But don't despair trainer! If we alter our current graph we can still add that information where we want it.

We introduce a dedicated node for a single evolution. In our case the evolution from Charmander to Charmeleon. But instead of a relationship between Charmander and Charmeleon directly, we now have a relationship between Charmander and the evolution, and a relationship between the evolution and Charmeleon, as shown below.

Now we can add as much information to that specific evolution as we want. Charmeleon is already one of them, and the level, via the level relationship, is the second one:

We can do the same for the fact that Charmeleon evolves into Charizard starting at level 36:

Nice job! We already have a pretty decent knowledge graph. But we are not done yet: we still need to add semantics to the graph. We do this by explicitly defining what each relationship means, so that there is no doubt when others try to understand the data in our knowledge graph. For the relationships we used we can define the following:

type: Pokémon belong to one or more types. When there is a type relationship between a subject and object, then this means that the subject is of the type represented by the object. The subject is always a Pokémon, such as Charmander, and the object is always a type, such as Fire.
generation: Pokémon are introduced in a specific generation. When there is a generation relationship between a subject and object, then this means that the subject is introduced in the generation represented by the object. The subject is always a Pokémon, such as Charmander, and the object is a always a generation, such as I.
evolves: Some Pokémon can evolve. This relationship is used between a Pokémon, the subject, and a representation of a specific evolution, the object. The subject can be for example Charmander, and the object is then an evolution that is linked to Charmeleon. Note that the object of the evolves relationship is not a Pokémon.
into: When a Pokémon evolves, they evolve into another Pokémon. This relationship says into which Pokémon another Pokémon evolves. The subject of this relationship is an evolution, and the object is the Pokémon into which the original Pokémon evolves. The subject is for example an evolution for Charmander, and the object is Charmeleon.
level: When a Pokémon evolves is often determined by the level they should have. The subject of this relationship is the representation of an evolution, and the object is the level that is required before an evolution can take place. The subject is for example an evolution for Charmander, and then the object is the 16.

By incorporating this information in the graph, we now have a real knowledge graph: we have the data together with the meaning of the data. Note that in practice the meaning of the data is not described in plain text, but through the use of ontologies (which are knowledge graphs themselves).

Great work trainer! This is the end of our example with Charmander. I hope you now have a better understanding of what knowledge graphs actually are. If you are eager to see more ~~Pokémon~~ knowledge graphs, do not hesitate to check the examples with Bulbasaur and Squirtle.

If you have any questions or remarks, don’t hesitate to contact me via email or via Twitter.

Bulbasaur

Great! You picked the grass/poison-type Pokémon. Here are some facts about Bulbasaur:

Bulbasaur is a grass/poison-type Pokémon introduced in Generation I.
It learns Leech Seed at level 7 and Vine Whip at level 13. source

Let's see how we can represent the information in this text as a knowledge graph. At the centre of our graph is Bulbasaur, as you can see below.

Graphs, including knowledge graphs, are formed of nodes and edges. Our graph contains one node: Bulbasaur. Now let's have a look at the first sentence:

Bulbasaur is a grass/poison-type Pokémon introduced in Generation I.

This sentence can be split up in three bits of information:

Bulbasaur is a grass-type Pokémon.
Bulbasaur is a poison-type Pokémon.
Bulbasaur is introduced in Generation I.

We represent a grass-type Pokémon by the word "GRASS" in a green rectangle:

Now our graph contains two nodes: Bulbasaur and the grass-type Pokémon. What is the relationship between these two? "Bulbasaur is a grass-type Pokémon" means that there is a relationship between our two nodes. We add that information to our graph by drawing an edge that connects the two nodes.

We are not done yet though. We know that there is a relationship, but what kind of relationship is it? "Bulbasaur is a grass-type Pokémon" means that Bulbasaur belongs to the group of grass-type Pokémon, or that Bulbasaur is of the type grass. Thus, we add "type" to the relationship.

Note that the direction of the arrow is from Bulbasaur to grass and not the other way around. If that were the case then our sentence would have been "A grass-type Pokémon is Bulbasaur", or something like that. What doesn't make sense.

We do the same for the fact that Bulbasaur is a poison-type Pokémon:

To add the third bit of information from the first sentence to the graph, we add a "I" to the graph that represents Generation I. Now there are four nodes in our graph.

Next, we draw an edge between Bulbasaur and I to show that there is a relationship between the nodes. Finally, we also add what kind of relationship it is. Here we add "generation" to the relationship.

The second sentence is

It learns Leech Seed at level 7 and Vine Whip at level 13.

This sentence can be split up in two bits of information:

Bulbasaur learns Leech Seed at level 7.
Bulbasaur learns Wine Whip at level 13.

We add a new node for the move "Leech Seed" and add a learns relationship between Bulbasaur and Leech Seed.

Now, we want to add the fact that Leech Seed is learned at level 7. Again, we add a node. This time for "7", but where do we add the relationship? Do we add it between Bulbasaur and 7? Do we add it between Leech Seed and 7?
This is possible, but what happens if Leech Seed can be learned by multiple Pokémon at different levels? How do you know to which Pokémon the level belongs? Actually, we want to say that that specific learns relationship only happens at level 7 (for Bulbasaur). Thus, we want to add "7" to the relationship between Bulbasaur and Leech Seed. But that is not possible: extra information cannot be added to a relationship. But don't despair trainer! If we alter our current graph we can still add that information where we want it.

We introduce a dedicated node for the learning of a single move. In our case the learning of Leech Seed by Bulbasaur. But instead of a relationship between Bulbasaur and Leech Seed directly, we now have a relationship between Bulbasaur and the learning of a move, and a relationship between the learning of a move and Leech Seed, as shown below.

Now we can add as much information to that specific learning of a move as we want. Leech Seed is already one of them, and the level, via the level relationship, is the second one:

We can do the same for the fact that Bulbasaur learns Vine Whip at level 13:

type: Pokémon belong to one or more types. When there is a type relationship between a subject and object, then this means that the subject is of the type represented by the object. The subject is always a Pokémon, such as Bulbasaur, and the object is always a type, such as Grass.
generation: Pokémon are introduced in a specific generation. When there is a generation relationship between a subject and object, then this means that the subject is introduced in the generation represented by the object. The subject is always a Pokémon, such as Bulbasaur, and the object is a always a generation, such as I.
learns: Pokémon can learn new moves. This relationship is used between a Pokémon, the subject, and the representation of learning a specific move, the object. The subject can be for example Bulbasaur, and the object the learning of a move that is linked to Leech Seed. Note that the object of the learns relationship is not a move.
which move:
This relationship says which moves a Pokémon learns. The subject of this relationship is the learning of a specific move, and the object is that move which the Pokémon learns. The subject is for example the learning of a specific move by Bulbasaur, and the object is Leech Seed.
level:
When a Pokémon learns a specific move is often determined by the level they should have. The subject of this relationship is the representation of learning a specific move, and the object is the level that is required before that move can be learned. The subject is for example the learning of a specific move by Bulbasaur, and then the object is the 7.

By incorporating this information in the graph, we now have a real knowledge graph: we have the data together with the meaning of the data.
Note that in practice the meaning of the data is not described in plain text, but through the use of ontologies (which are knowledge graphs themselves).

Great work trainer! This is the end of our example with Bulbasaur.
I hope you now have a better understanding of what knowledge graphs actually are. If you are eager to see more ~~Pokémon~~ knowledge graphs,
do not hesitate to check the examples with Charmander and Squirtle.

If you have any questions or remarks, don’t hesitate to contact me via email or via Twitter.

Squirtle

Great! You picked the water-type Pokémon. Here are some facts about Squirtle:

Squirtle is a water-type Pokémon introduced in Generation I.
It is a starter in Pokémon Blue and Red and
can be received in Pokémon Yellow from Officer Jenny after receiving the Thunder Badge. source

Let's see how we can represent the information in this text as a knowledge graph. At the centre of our graph is Squirtle, as you can see below.

Graphs, including knowledge graphs, are formed of nodes and edges. Our graph contains one node: Squirtle. Now let's have a look at the first sentence:

Squirtle is a water-type Pokémon introduced in Generation I.

This sentence can be split up in two bits of information:

Squirtle is a water-type Pokémon.
Squirtle is introduced in Generation I.

We represent a water-type Pokémon by the word "WATER" in a blue rectangle:

Now our graph contains two nodes: Squirtle and the water-type Pokémon. What is the relationship between these two? "Squirtle is a water-type Pokémon" means that there is a relationship between our two nodes. We add that information to our graph by drawing an edge that connects the two nodes.

We are not done yet though. We know that there is a relationship,
but what kind of relationship is it? "Squirtle is a water-type Pokémon" means that Squirtle belongs to the group of water-type Pokémon, or that Squirtle is of the type water. Thus, we add "type" to the relationship.

Note that the direction of the arrow is from Squirtle to water and
not the other way around. If that were the case then our sentence would have been "A water-type Pokémon is Squirtle", or something like that. What doesn't make sense.

To add the second bit of information from the first sentence to the graph, we add a "I" to the graph that represents Generation I. Now there are three nodes in our graph.

Next, we draw an edge between Squirtle and I to show that there is a relationship between the nodes. Finally, we also add what kind of relationship it is. Here we add "generation" to the relationship.

The second sentence is

It is a starter in Pokémon Blue and Red and
can be received in Pokémon Yellow from Officer Jenny after receiving the Thunder Badge.

This sentence can be split up in three bits of information:

Squirtle is a starter in Pokémon Blue.
Squirtle is a starter in Pokémon Red.
Squirtle can be received in Pokémon Yellow from Officer Jenny after receiving the Thunder Badge.

For the first bit of information, we add a new node for the game "Pokémon Blue" and add a starter relationship between Squirtle and Pokémon Blue.

We do the same for the second bit: a node for Pokémon Red and a starter relationship between Squirtle and that node.

Next, we add a new node for Pokémon Yellow too and a receive relationship between Squirtle and that node for the third bit for information.

Now, we want to add the fact that Squirtle is received from Officer Jenny in Pokémon Yellow. Again, we add a node for Officer Jenny, but where do we add the relationship? Do we add it between Squirtle and Officer Jenny?
Do we add it between Pokémon Yellow and Officer Jenny? This is possible, but what happens if a Squirtle can be received from different people in different games? How do you know from whom in which game? Actually, we want to say that that specific receive relationship only happens in a specific game from a specific person (for Squirtle). Thus, we want to add Officer Jenny to the relationship between Squirtle and Pokémon Yellow. But that is not possible: extra information cannot be added to a relationship.
But don't despair trainer! If we alter our current graph we can still add that information where we want it.

We introduce a dedicated node for receiving something called "give away".
In our case the receiving of Squirtle from Officer Jenny in Pokémon Yellow. But instead of a relationship between Squirtle and Pokémon Yellow directly, we now have a relationship between Squirtle and the give away, and a relationship between the give away and Pokémon Yellow, as shown below.

Now we can add as much information to that give away as we want. Pokémon Yellow is already one of them, and Officer Jenny, via the from relationship, is the second one:

Finally, we can also add the fact that the Squirtle can only received if you have the Thunder Badge:

type: Pokémon belong to one or more types. When there is a type relationship between a subject and object, then this means that the subject is of the type represented by the object. The subject is always a Pokémon, such as Squirtle, and the object is always a type, such as Grass.
generation: Pokémon are introduced in a specific generation. When there is a generation relationship between a subject and object, then this means that the subject is introduced in the generation represented by the object. The subject is always a Pokémon, such as Squirtle, and the object is always a generation, such as I.
starter: Certain Pokémon can be picked as a starter in the games. When there is a starter relationship between a subject and object, then this means that the subject can be picked as a starter in the object. The subject is always a Pokémon, such as Squirtle, and the object is always a game, such as Pokémon Blue.
receive: Certain Pokémon can be received from characters in the games. When there is a receive relationship between a subject and object, then this means that the subject can be received according to the details described by the object. The subject is always a Pokémon, such as Squirtle, and the object is a representation of giving away a Pokémon.
game: Pokémon can be received in specific games. This relationship is used between representation of giving away a Pokémon, the subject, and a game, the object. The subject can be for example the representation of giving away Squirtle, and the object Pokémon Yellow. Note that the subject of the game relationship is not a Pokémon.
from: Pokémon can be received from specific characters in the games.
This relationship is used between representation of giving away a Pokémon, the subject, and a character from which the Pokémon is received, the object. The subject can be for example the representation of giving away Squirtle, and the object Officer Jenny.
requires: Sometimes requirements have to fulfilled before Pokémon can be received in games. This relationship is used between representation of giving away a Pokémon, the subject, and a requirement that needs to be fulfilled, the object. The subject can be for example the representation of giving away Squirtle, and the object the Thunder Badge.

Great work trainer! This is the end of our example with Squirtle. I hope you now have a better understanding of what knowledge graphs actually are. If you are eager to see more ~~Pokémon~~ knowledge graphs, do not hesitate to check the examples with Charmander and Bulbasaur.

If you have any questions or remarks, don’t hesitate to contact me via email or via Twitter.

Pokémon and Pokémon character names are trademarks of Nintendo.

Caching JavaScript data file results when using Eleventy

Pieter Heyvaert — Fri, 27 Sep 2019 19:14:56 +0000

Eleventy by Zach Leatherman has become my default static site generator. It is simple, uses JavaScript, and is easy to extend. It allows me to include custom code to access additional data sources,
such as RDF datasets.

Querying data can take up some time, for example, when using an external Web API. During deployment of a website this is not a big deal, as this probably doesn't happen every minute. But when you are developing then it might become an issue: you don't want to wait for query results every time you make a change that doesn't affect the results, such as updating a CSS property, which only affects how the results are visualized. Ideally, you want to reuse these results without querying the data over and over again. I explain in this blog post how that can be done by introducing a cache.

The cache has the following features:

The cache is only used when the website is locally served (eleventy --serve).
The cached data is written to and read from the filesystem.

This is done by using the following two files:

serve.sh: a Bash script that runs Eleventy.
cache.js: a JavaScript file that defines the cache method.

An example Eleventy website using these two files is available on Github.

Serve.sh

#!/usr/bin/env bash

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT

function ctrl_c() {
  rm -rf _data/_cache
  exit 0
}

# Remove old folders
rm -rf _data/_cache # Should already be removed, but just in case
rm -rf _site

# Create needed folders
mkdir _data/_cache

ELEVENTY_SERVE=true npx eleventy --serve --port 8080

This Bash script creates the folder for the cached data and serves the website locally. First, we remove the cache folder and the files generated by Eleventy, which might still be there from before. Strictly speaking removing the latter is not necessary, but I have noticed that removed files are not removed from _site, which might result in unexpected behaviour. Second, we create the cache folder again, which of course is now empty. Finally, we set the environment variable ELEVENTY_SERVE to true and start Eleventy: we serve the website locally on port 8080. The environment variable is used by cache.js to check if the website is being served, because currently this information can't be extracted from Eleventy directly. Note that I have only tested this on macOS 10.12.6 and 10.14.6, and Ubuntu 16.04.6. Changes might be required for other OSs.

Cache.js

const path = require('path');
const fs = require('fs-extra');

/**
 * This method returns a cached version if available, else it will get the data via the provided function.
 * @param getData The function that needs to be called when no cached version is available.
 * @param cacheFilename The filename of the file that contains the cached version.
 * @returns the data either from the cache or from the geData function.
 */
module.exports = async function(getData, cacheFilename) {
  // Check if the environment variable is set.
  const isServing = process.env.ELEVENTY_SERVE === 'true';
  const cacheFilePath = path.resolve(__dirname, '_data/_cache/' + cacheFilename);
  let dataInCache = null;

  // Check if the website is being served and that a cached version is available.
  if (isServing && await fs.pathExists(cacheFilePath)) {
    // Read file from cache.
    dataInCache = await fs.readJSON(cacheFilePath);
    console.log('Using from cache: ' + cacheFilename);
  }

  // If no cached version is available, we execute the function.
  if (!dataInCache) {
    const result = await getData();

    // If the website is being served, then we write the data to the cache.
    if (isServing) {
      // Write data to cache.
      fs.writeJSON(cacheFilePath, result, err => {
        if (err) {console.error(err)}
      });
    }

    dataInCache = result;
  }

  return dataInCache;
};

The method defined by the JavaScript file above takes two parameters: getData and cacheFilename. The former is the expensive function that you don't want to repeat over and over again. The latter is the filename of the file with the cached version. The file will be put in the folder _data/_cache relative to the location of cache.js. The environment variable used in serve.sh is checked here to see if the website is being served. Note that the script requires the package fs-extra, which adds extra methods to fs and is not available by default.

Putting it all together

To get it all running, we put both files in our Eleventy project root folder. Do not forget to make the script executable and run serve.sh.

When executing the aforementioned example, we see that the first time to build the website it takes 10.14 seconds (see screencast below). No cached version of the query results is available at this point and thus the Web API has to be queried. But the second time, when we update the template, it only takes 0.03 seconds. This is because the cached version of the query results is used instead of querying the Web API again.

Screencast: When the Web API is queried it takes 10.14 seconds. When the cached version of the query results is used it takes 0.03 seconds.

How a chess app interacts with Solid

Pieter Heyvaert — Thu, 04 Apr 2019 10:32:15 +0000

Last year I started playing around with the Solid platform, introduced by Tim Berners-Lee. In order to get more familiar with the different concepts and technologies used, I created a proof-of-concept app: a browser chess game.

The main concept of the Solid platform is the use of Solid PODs, which provide personal online data storage. The idea of Solid is that a user has one or more PODs to store their data and different apps are able to interact with these PODs, but these apps can only read and write the data to which the user has granted them access. So, instead of having to store all data of all users on a single server per app, users now store and control (!) their own data, leading to a decentralized approach. The focus of this blog post is on the interaction between the app and the Solid PODs.

First, we list the different features of the chess app. Second, we elaborate on the three high-level components of the app. Third, we list the different steps that are taken when certain actions are done by a player. Finally, we discuss how the data generated by this app can be used by other apps.

Features

The app provides the following features:

authenticate a player
start a new chess game and invite your opponent
join a chess game you are invited to
do moves in a chess game
continue an existing chess game

High-level components

The app consists of three high-level components: the GUI, the chess game engine, and the interaction with the POD. The GUI uses the chessboard.js library, which offers the chessboard and the interaction of the players with the board, such as the moving of the pieces. It does not provide a chess engine, i.e., it will not check whether the moves are valid or not and keep track of which player's turn it is. To accomplish this we make use of the chess.js library. Although it provides the required information to play a chess game, the data is not presented as Linked Data, which is preferred by the Solid platform. Linked Data is a method of publishing structured data so that it can be interlinked and become more useful through semantic queries. Therefore, I created a "wrapper" library around chess.js called semantic-chess that outputs RDF, a technology used to materialize Linked Data, which can be used during the interaction of the app with the POD. The focus of this blog post is on this interaction.

Actions

Authentication

When the app is launched, one of the first actions a player does is logging in. For this we use the solid-auth-client library. It keeps track of the session and provides a fetch method that is used whenever we want to interact with the POD. Whenever a player is logged in the fetch method is able to store and read data from PODs to which the logged-in player has access.

Figure 1: pop-up window that appears when clicking the "Log in" button in the app.

In the GUI there is a classic log-in button. When clicked, a pop-up appears that allows selecting your identify provider to which you want to authenticate (see Figure 1). You can provide your own HTML file to render the pop-up or you can use the default one provided by the library. After successfully logging in, the pop-up is closed and the focus is back on the app.

The solid-auth-client library has tracked the fact that a player has logged in and now a session object is available. This session object contains, for example, the Web ID of the logged in player. A Web ID is an HTTP URI that denotes an agent on an HTTP-based network. In line with the Linked Data principles, when a Web ID is de-referenced, it resolves to a profile document that describes its referent, i.e., the player. Furthermore, the fetch method provided by the library will now act on behalf of the player.

Start new game

When a player starts a new game he needs to select an opponent, the app needs to store the game data on the player's POD, and invite the opponent.

Select opponent

Figure 2: sequence diagram of the steps taken when a player selects an opponent.

With this action, the player selects an opponent for a new game. In Figure 2, we describe the corresponding steps between the player, the player's app, the player's POD, and the PODs of his friends. The steps are:

1) The app does a GET to the player's Web ID. If the Web ID of the player is

https://player.solid.community/profile/card#me

then the corresponding curl command is

curl https://player.solid.community/profile/card#me

2) The player's POD returns RDF describing the player. An example of such RDF is

@prefix : <https://player.solid.community/profile/card#>.
@prefix inbox: <https://player.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix c0: <https://opponent.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows c0:me;
  foaf:name "Player".

3) The app queries the RDF to determine the player's friends, via the predicate foaf:knows. An example of SPARQL query using this predicate is

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX player: <https://player.solid.community/profile/card#>

SELECT ?friend 
WHERE {
  player:me foaf:knows ?friend.
}

For every friend the following three steps are taken.

4) The app does a GET to the friend's Web ID. If the Web ID of the player is

https://opponent.solid.community/profile/card#me

then the corresponding curl command is

curl https://opponent.solid.community/profile/card#me

5) The friend's POD returns RDF describing the friend. This RDF is similar to the RDF of step 2. An example of such RDF is

@prefix : <https://opponent.solid.community/profile/card#>.
@prefix inbox: <https://opponent.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix player: <https://player.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows player:me;
  foaf:name "Opponent".

6) The app queries the RDF to determine the friend's name, via the predicate foaf:name. An example of SPARQL query using this predicate for an friend with the Web ID

https://opponent.solid.community/profile/card#me

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX opponent: <https://opponent.solid.community/profile/card#>

SELECT ?name 
WHERE {
  opponent:me foaf:name ?name.
}

7) The app shows a list of friends to the player.
8) The player selects the desired friend as his opponent.

Remarks

SPARQL is used to query the RDF to find friends and their names, because it is the default query language for RDF. However, alternatives such as LDFlex and GraphQL-LD are also available.
The FOAF ontology, with prefix foaf, is one of several ontologies, such as Schema.org and vCard, that can be used to describe the basic information of a person. At the time of writing, when creating a default profile with Solid mostly FOAF is used. However, bear in mind that other Solid PODs might use (a combination of) other ontologies.
We only store the Web IDs of a player's friends on the player's POD, because additional information about a friend can be acquired by downloading data from the friend's POD.

Store game data on POD

Figure 3: sequence diagram of the steps taken when storing game data on the POD.

With this action, the game data of a new game is stored on the player's POD. In Figure 3, we describe the corresponding steps between the player, the player's app, and the player's POD. The steps are:

1) The player starts the game.
2) The player's app generates RDF that describing this specific instance of a chess game. An example of such RDF is

 @prefix : <https://player.solid.community/public/chess.ttl#>.
 @prefix chess: <http://purl.org/NET/rdfchess/ontology/>.
 @prefix stor: <http://example.org/storage/>.
 @prefix schema: <http://schema.org/>.
 @prefix player: <https://player.solid.community/profile/card#>.
 @prefix opponent: <https://opponent.solid.community/profile/card#>.

 :game
   a chess:ChessGame;
   stor:storeIn :;
   chess:providesAgentRole :jo8deywv, :jo8deyww;
   chess:starts :jo8deywv;
   schema:name "Test game".

 :jo8deywv a chess:WhitePlayerRole;
   chess:performedBy player:me.

 :jo8deyww a chess:BlackPlayerRole;
   chess:performedBy opponent:me.

It includes

the unique URL of the game: uniquely identifies this specific chess game on the Web
the name of the game: displayed in the app
the Web IDs of the two players: uniquely identifies the players of this specific chess game on the Web
the color of each player

3) The player's app does a PATCH with a SPARQL UPDATE query to the player's POD with the RDF. A PATCH is used because the file where we want to store the RDF, chosen by the player, might already contain data, which we do not want to overwrite. An example of such a PATCH is

PREFIX : <https://player.solid.community/public/chess.ttl#>
PREFIX chess: <http://purl.org/NET/rdfchess/ontology/>
PREFIX stor: <http://example.org/storage/>
PREFIX schema: <http://schema.org/>
PREFIX player: <https://player.solid.community/profile/card#>
PREFIX opponent: <https://opponent.solid.community/profile/card#>

INSERT DATA {
  :game a chess:ChessGame;
    stor:storeIn <>;
    chess:providesAgentRole :jo8deywv, :jo8deyww;
    chess:starts :jo8deywv;
    schema:name "Test game".

    :jo8deywv a chess:WhitePlayerRole; 
      chess:performedBy player:me.

    :jo8deyww a chess:BlackPlayerRole; 
      chess:performedBy opponent:me.
}

Invite opponent

Figure 4: sequence diagram of the steps taken when an opponent is invited.

With this action, an invitation is sent to the opponent to join the newly created game. In Figure 4, we describe the corresponding steps between
the player's app and the opponent's POD. The steps are:

1) The player's app generates RDF describing the invitation. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix schema: <http://schema.org/>.
@prefix player: <https://player.solid.community/profile/card#>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

:invitation a schema:InviteAction>;
  schema:event :game;
  schema:agent player:me;
  schema:recipient opponent:me.

2) The player's app does PATCH with a SPARQL UPDATE query to store the invitation on the player's POD. An example of such a PATCH is

 PREFIX schema: http://schema.org/>
 PREFIX : <https://player.solid.community/public/chess.ttl#>
 PREFIX player: <https://player.solid.community/profile/card#>
 PREFIX opponent: <https://opponent.solid.community/profile/card#>

 INSERT DATA {
   :invitation a schema:InviteAction;
     schema:event :game;
     schema:agent player:me;
     schema:recipient opponent:me.
 }

3) The player's app generates RDF for a notification of the invitation. An RDF example of such an notification is

@prefix schema: <http://schema.org/> .
@prefix : <https://player.solid.community/public/chess.ttl#>.

:invitation a schema:InviteAction .

4) The player's app does a GET to the Web ID of the opponent. If the Web ID of the player is

https://player.solid.community/profile/card#me

then the corresponding curl command is

curl https://player.solid.community/profile/card#me

5) The opponent's POD returns RDF describing the opponent. An example of such RDF is

@prefix : <https://opponent.solid.community/profile/card#>.
@prefix inbox: <https://opponent.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix c0: <https://player.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows c0:me;
  foaf:name "Opponent".

6) The player's app queries the RDF to determine the opponent's inbox via the predicate ldp:inbox. An example of a SPARQL query using this predicate is

PREFIX ldp: <http://www.w3.org/ns/ldp#>

SELECT ?inbox
 HERE {
  <https://opponent.solid.community/profile/card#> ldp:inbox ?inbox.
}

7) The player's app does a POST to the inbox with the notification of the invitation. When we do a POST to an inbox a new file, containing the notification, is created in the inbox.

Join game

Figure 5: sequence diagram of the steps taken when an opponent joins a game.

With this action, the opponent joins the game for which he received an invitation. In Figure 5, we describe the corresponding steps between the player, the opponent, the player's app, the player's POD, the opponent's app, and the opponent's POD. The steps are:

1) The opponent's app does a GET to the Web ID of the the logged-in user, which in this case is the opponent.
2) The opponent's POD returns RDF describing the opponent. An example of such RDF is

@prefix : <https://opponent.solid.community/profile/card#>.
@prefix inbox: <https://opponent.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix player: <https://player.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows player:me;
  foaf:name "Opponent".

3) The opponent's app queries the RDF to determine the opponent's inbox via the predicate ldp:inbox. An example of a SPARQL query using this predicate is

PREFIX ldp: <http://www.w3.org/ns/ldp#>
PREFIX opponent: <https://opponent.solid.community/profile/card#>

SELECT ?inbox
WHERE {
  opponent:me ldp:inbox ?inbox.
}

4) The opponent's app does a GET to the inbox, which contains links that identify the different notifications.
5) The opponent's POD returns RDF describing the inbox. An example of such RDF is

@prefix inbox: <https://opponent.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix terms: <http://purl.org/dc/terms/>.
@prefix XML: <http://www.w3.org/2001/XMLSchema#>.
@prefix st: <http://www.w3.org/ns/posix/stat#>.

inbox:
  a ldp:BasicContainer, ldp:Container;
  terms:modified "2019-01-10T15:00:26Z"^^XML:dateTime;
  ldp:contains inbox:765a0510-14e8-11e9-a29e-5d8e3e616ac9, n0:;
  st:mtime 1547132426.207;
  st:size 4096.

inbox:765a0510-14e8-11e9-a29e-5d8e3e616ac9
  a ldp:Resource;
  terms:modified "2019-01-10T15:00:26Z"^^XML:dateTime;
  st:mtime 1547132426.207;
  st:size 1369.

6) The opponent's app queries the RDF to determine the notifications, i.e., their links, via the class ldp:Resource. An example of a SPARQL query using this predicate is

PREFIX ldp: <http://www.w3.org/ns/ldp#>

SELECT ?notification
WHERE {
  ?notification a ldp:Resource .
}

The opponent's app iterates over all notifications in the inbox.
7) The opponent's app does a GET to the link of the notification.
8) The opponent's POD returns RDF describing the notification.
An example of such RDF is

@prefix schema: <http://schema.org/> .
@prefix : <https://player.solid.community/public/chess.ttl#>.

:invitation a schema:InviteAction .

9) The app queries the RDF to determine if the notification contains an invitation, via the class schema:InviteAction. An example of a corresponding SPARQL query is

PREFIX schema: <http://schema.org/>

SELECT ?invitation
WHERE {
  ?invitation a schema:InviteAction .
}

10)If the notification contains an invitation, the opponent's app does a GET to the link of the invitation.
11) The player's POD returns RDF describing the invitation. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix schema: <http://schema.org/>.
@prefix player: <https://player.solid.community/profile/card#>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

:invitation a schema:InviteAction;
  schema:event :game;
  schema:agent player:me;
  schema:recipient opponent:me.

12) The opponent's app queries the RDF to determine the link of the game and the Web ID of the opponent. A corresponding SPARQL query is

PREFIX : <https://player.solid.community/public/chess.ttl#>
PREFIX schema: <http://schema.org/>   

SELECT ?game ?opponent
  WHERE {
   :invitation schema:event ?game;
     schema:agent ?opponent.
  }

13) The opponent's app shows the invitation to the opponent.
14) The opponent accepts the invitation.
15) The opponent's app generates RDF describing the response. An example of such RDF is

@prefix response: <https://opponent.solid.community/public/chess.ttl#response>.
@prefix invitation: <https://player.solid.community/public/chess.ttl#invitation>.
@prefix schema: <http://schema.org/>.
@prefix player: <https://player.solid.community/profile/card#>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

response: a schema:RsvpAction;
  schema:rsvpResponse schema:RsvpResponseYes;
  schema:agent opponent:me;
  schema:recipient player:me.

invitation: schema:result response:.

16) The opponent's app does PATCH with a SPARQL UPDATE query to store the response on the opponent's POD. An example of such a PATCH is

PREFIX response: <https://opponent.solid.community/public/chess.ttl#response>
PREFIX invitation: <https://player.solid.community/public/chess.ttl#invitation>
PREFIX schema: <http://schema.org/>
PREFIX player: <https://player.solid.community/profile/card#>
PREFIX opponent: <https://opponent.solid.community/profile/card#>

INSERT DATA {
  response: a schema:RsvpAction;
    schema:rsvpResponse schema:RsvpResponseYes;
    schema:agent opponent:me;
    schema:recipient player:me.

  invitation: schema:result response:.
}

17) The player's app generates RDF for a notification of the response. An RDF example of such an notification is

@prefix response: <https://opponent.solid.community/public/chess.ttl#response>.
@prefix schema: <http://schema.org/>.

response: a schema:RsvpAction.

18) The opponent's app does a PATCH to the opponent's POD to store the relevant game data. This is mostly data to know later that the opponent is participating in the game. Details about the game are not stored on the opponent's POD, because they are stored on the player's POD and are retrievable by doing a GET to the link of the game.
19) The opponent's app does a GET to the Web ID of the player. This Web ID is available through the invitation.
20) The player's POD returns RDF describing the player. An example of such RDF is

@prefix : <https://player.solid.community/profile/card#>.
@prefix inbox: <https://player.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows opponent:me;
  foaf:name "Player".

21) The app queries the RDF to determine the player's inbox via the predicate ldp:inbox.
22) The app does a POST to the player's inbox with notification of the response.
23) The player's app does a GET to the notification. For clarity we skip the iteration over the different notifications, because this is the same as earlier.
24) The player's POD returns RDF describing the notification. An example of such RDF is

@prefix response: <https://opponent.solid.community/public/chess.ttl#response>.
@prefix schema: <http://schema.org/>.

response: a schema:RsvpAction.

25) The player's app queries the RDF to determine if the notification contains a response, via the class schema:RsvpAction.
26) If the notification contains a response, the app does a GET to the link of the response.
27) The opponent's POD returns RDF describing the response. An example of such RDF is

@prefix response: <https://opponent.solid.community/public/chess.ttl#response>.
@prefix invitation: <https://player.solid.community/public/chess.ttl#invitation>.
@prefix schema: <http://schema.org/>.
@prefix player: <https://player.solid.community/profile/card#>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

response: a schema:RsvpAction;
  schema:rsvpResponse schema:RsvpResponseYes;
  schema:agent opponent:me;
  schema:recipient player:me.

invitation: schema:result response:.

28) The player's app queries the response to determine the corresponding invitation, from which the corresponding game can be determined. An example of a corresponding SPARQL query is

PREFIX schema: <http://schema.org/>

SELECT ?invitation 
WHERE {
  response: schema:result ?invitation. 
}

29) The player's app shows the response to the player.

Do move

Figure 6: sequence diagram of the steps taken when a player does a move.

With this action, the player does a new move, which is shown to the opponent.
In Figure 6, we describe the corresponding steps between the player, the player's app, the player's POD, the opponent's app, and the opponent's POD. The steps are:

1) The player does a move.
2) The player's app generates RDF that describes this move. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix schema: <http://schema.org/>.
@prefix chess: <http://purl.org/NET/rdfchess/ontology/>.
@prefix opponent: <https://opponent.solid.community/public/chess.ttl#>.

:game chess:hasHalfMove :move2.

:move2 a chess:HalfMove;
  schema:subEvent :game;
  chess:hasSANRecord "e3"^^xsd:string;
  chess:resultingPosition "rnbqkbnr/pppp1ppp/4p3/8/8/4P3/PPPP1PPP/RNBQKBNR w KQkq -".

opponent:move1 chess:nextHalfMove :move2.

where

https://player.solid.community/public/chess.ttl#move2

is the link of the new move and

https://opponent.solid.community/public/chess.ttl#move1

is the link of the previous move.
3) The player's app does a PATCH to the player's POD to store the move.
4) The player's app generates a notification for the move. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix opponent: <https://opponent.solid.community/public/chess.ttl#>.

opponent:move1 chess:nextHalfMove :move2.

5) The player's app does a POST to the opponent's POD with this notification.
6) The opponent's app does a GET to the link of the notification.
7) The opponent's POD returns RDF describing the move. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix opponent: <https://opponent.solid.community/public/chess.ttl#>.

opponent:move1 chess:nextHalfMove :move2.

8) The opponent's app queries the RDF to determine if the notification contains a move.
9) The opponent's app does a GET to the link of the move.
10) The player's POD returns RDF describing the move. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix schema: <http://schema.org/>.
@prefix chess: <http://purl.org/NET/rdfchess/ontology/>.

:move2 a chess:HalfMove;
  schema:subEvent :game;
  chess:hasSANRecord "e3"^^xsd:string;
  chess:resultingPosition "rnbqkbnr/pppp1ppp/4p3/8/8/4P3/PPPP1PPP/RNBQKBNR w KQkq -".

11) The opponent's app queries the RDF to determine the details of the move. A corresponding SPARQL query is

PREFIX : <https://player.solid.community/public/chess.ttl#>
PREFIX schema: <http://schema.org/>
PREFIX chess: <http://purl.org/NET/rdfchess/ontology/>

SELECT ?game ?san
WHERE {
  :move2 schema:subEvent ?game;
  chess:hasSANRecord ?san.
}

12) The opponent's app shows the new move to the opponent.
13) The opponent's app stores the link of the new move on the opponent's POD. This is done to reconstruct the chess game based on the sequence of individual moves.

Continue game

Figure 7: sequence diagram of the steps taken when a player continues a game.

With this action, the player continues a game that he started before. Chess does not have to be played real time, i.e., it is possible that a player does a move in the morning, but that the opponent does the next move only in the evening. Therefore, players should be able to continue a game at any point in time. In Figure 7, we describe the corresponding steps between the player, the player's app, the player's POD, and the opponent's POD. The steps are:

1) The player's app does a GET to the player's Web ID.
2) The player's POD returns RDF describing the player. An example of such RDF is

@prefix : <https://player.solid.community/profile/card#>.
@prefix inbox: <https://player.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix c0: <https://opponent.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows c0:me;
  foaf:name "Player".

3) The player's app queries the RDF to determine the chess games in which the player participates. The app iterates over all games.
4) The player's app does a GET to the link of each game.
5) The player's POD returns RDF describing the game. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix chess: <http://purl.org/NET/rdfchess/ontology/>.
@prefix stor: <http://example.org/storage/>.
@prefix schema: <http://schema.org/>.
@prefix player: <https://player.solid.community/profile/card#>.
@prefix opponent: <https://opponent.solid.community/profile/card#>.

:game a chess:ChessGame;
  stor:storeIn :;
  chess:providesAgentRole :jo8deywv, :jo8deyww;
  chess:starts :jo8deywv;
  schema:name "Test game".

:jo8deywv a chess:WhitePlayerRole;
  chess:performedBy player:me.

:jo8deyww a chess:BlackPlayerRole; 
  chess:performedBy opponent:me.

6) The player's app queries the RDF to determine the name of the game and the opponent's Web ID. An example of a corresponding SPARQL query is

PREFIX : <https://player.solid.community/public/chess.ttl#>
PREFIX chess: <http://purl.org/NET/rdfchess/ontology/>
PREFIX schema: <http://schema.org>
PREFIX player: <https://player.solid.community/profile/card#>

SELECT ?name ?opponent
WHERE {
  :game schema:name ?name;
    chess:providesAgentRole ?role.

  ?role chess:performedBy ?opponent.

  MINUS {?role chess:performedBy player:me}
}

7) The player's app does a GET to the opponent's Web ID.
8) The opponent's POD returns RDF describing the opponent. An example of such RDF is

@prefix : <https://opponent.solid.community/profile/card#>.
@prefix inbox: <https://opponent.solid.community/inbox/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix foaf: <http://xmlns.com/foaf/0.1/>.
@prefix player: <https://player.solid.community/profile/card#>.

:me ldp:inbox inbox:;
  foaf:knows player:me;
  foaf:name "Opponent".

9) The player's app queries the RDF to determine the name of the player via the predicate foaf:name. An example of SPARQL query using this predicate for an friend with the Web ID

https://opponent.solid.community/profile/card#me

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX opponent: <https://opponent.solid.community/profile/card#>

SELECT ?name 
WHERE {
  opponent:me foaf:name ?name.
}

10) The player selects the game he wants to continue.
11) The player's app iterates over all moves of the selected game.
12) If the move is from the player then a GET to the link of the move by the player's app goes to the player's POD.
13) The player's POD returns RDF describing the move. An example of such RDF is

@prefix : <https://player.solid.community/public/chess.ttl#>.
@prefix chess: <http://purl.org/NET/rdfchess/ontology/>.
@prefix schema: <http://schema.org>.

:move2 a chess:HalfMove;
  schema:subEvent :game;
  chess:hasSANRecord "e3"^^xsd:string;
  chess:resultingPosition "rnbqkbnr/pppp1ppp/4p3/8/8/4P3/PPPP1PPP/RNBQKBNR w KQkq -".

14) If the move is from the opponent then a GET to the link of the move by the player's app goes to the opponent's POD.
15) The opponent's POD returns RDF describing the move. An example of such RDF is

@prefix : <https://opponent.solid.community/public/chess.ttl#>.
@prefix player: <https://opponent.solid.community/public/chess.ttl#>.
@prefix chess: <http://purl.org/NET/rdfchess/ontology/>.
@prefix schema: <http://schema.org>.

:move1 a chess:HalfMove;
  schema:subEvent :game;
  chess:hasSANRecord "e6"^^xsd:string;
  chess:resultingPosition "rnbqkbnr/pppp1ppp/4p3/8/8/8/PPPPPPPP/RNBQKBNR w KQkq -";
  chess:nextHalfMove player:move2.

16) The player's app adds the move to the instance of the game.
17) The player's app shows the game to the player.

Beyond this app

The data generated by this app is not tied to this specific app, because the data is materialized as RDF and follows the Linked Data principles. Furthermore, it is stored in the PODs of the players, which the players themselves control. As a result, the data can be used by other apps, completely independent of the app described in this blog post. Examples of such other apps are:

other chess apps which can continue an existing game
leaderboard apps that list the best players
artificial intelligence apps that analyze the moves of a player with the goal to provide suggestions to the player as how to improve their skills

If you have any questions or remarks, don’t hesitate to contact me via email or via Twitter.

Add content negotiation to website when using NGINX

Pieter Heyvaert — Fri, 01 Mar 2019 06:51:18 +0000

co-authored by Julián Rojas

Content negotiation (conneg) is an HTTP mechanism that makes it possible to serve different representations of a resource at the same URI. This allows agents to specify which representation of a resource they prefer.For example, on the one hand, when a browser opens http://example.com/house it will ask for an HTML document from the server.On the other hand, another application that works with XML documents, can request the same resource at http://example.com/house, but the application will ask for the resource's XML representation. In this blog post we will explain how you can achieve conneg for your website when using NGINX.

We want to achieve the following:

Use NGINX as reverse proxy.
Serve a static website, which has different documents for the same resource.
Enable conneg for that website.

First, we have a look at the tools we use. Second, we discuss the setup and configuration. Finally, we show two live examples and summarize this blog post.

Tools

NGINX does not support conneg out of the box. There are tutorials and blog posts available that explain how to achieve some kind of conneg, but they are heavily tied to a specific use case. Thus, they require significant changes to make it work for your use case and are not easily reusable. If NGINX doesn't support conneg, then why not just pick something else. Well, although NGINX lacks this feature, it does work great for the other things such as high performance, caching, acting as a reverse, and so on.

So what do we use to enable conneg? We use a fork of http-server.
http-sever is "a simple, zero-configuration command-line http server" build using Node.js. We created a fork that supports conneg. In more details: the server looks for the correct file depending on the MIME Types in the Accept header. MIME Types are linked to their corresponding extensions. For example, when you do a request to http://localhost:8080/test with Accept header text/turtle, the server looks for the file /test.ttl. When you do a request to http://localhost:8080/test with Accept header application/n-triples, the server looks for the file /test.nt.
Accept headers with multiple types, optionally weighted with a quality value, are also supported. For example, when you do a request to http://localhost:8080/test with Accept header text/turtle;q=0.5, application/n-triples, the server looks for the file /test.nt.

Setup

You can find an overview of the different components of our setup in the figure above. We have three services running: NGINX, the http-server, and a service X. The latter service can be anything, and there is no limit to the number of services. All requests are received by NGINX and the ones for our website, for which we want conneg, are redirected to the http-server. NGINX is unaware of any conneg, as all requests are just forwarded to the http-server. Service X is added to show that other services behind NGINX are also unaware of the conneg, as all conneg logic is handled in the http-server. Of course, service X can also support conneg, but that does not influence the (conneg of the) http-server.

Configuration

We need to configure both NGINX and the http-server in order for both to work together. The relevant part of the NGINX config looks as follows:

location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://127.0.0.1:4000/;
        proxy_redirect off;
}

The most important line is proxy_pass http://127.0.0.1:4000/; where it says that requests are forwarded to the http-server, which is running on the same server as NGINX and on port 4000. Of course, you can run it on another server and port.

When you enable caching via NGINX together with conneg, it is important to add the Accept header to the proxy_cache_key, for example, via proxy_cache_key "$request_uri $http_accept";. If you don't do this, then the Accept header is not considered for the cache, because the default value for proxy_cache_key is $scheme$proxy_host$request_uri. Note that $http_accept is missing. As a result, once a cached version of a resource is available, that cached version is returned regardless of the value of the Accept header in new requests.

The http-server is running on port 4000, with conneg and the automatic addition of trailing slashes enabled, in silent mode. This is achieved via http-server -P 4000 --conneg --trailing -s where:

-P 4000: use port 4000,
--conneg: enable conneg,
--trailing: enabled automatic addition of trailing slashes, and
-s: enable silent mode.

For more details on the available parameters, we refer to the README.

Examples

The setup described above is already available in the wild: the descriptions of Pieter Heyvaert and the Open Velopark Ontology are available in different representations through conneg.

Describing a person via HTML and RDF

A person's website main goal is to provide information about that person to others. When browsing to that website, an HTML document is downloaded from the server and displayed. However, applications might prefer another representation that is easier for them to process. By offering conneg for a persons's website, applications can decide which representation to request. For example, browsers request an HTML representation to display to users, while Linked Data applications request an RDF representation to easily process the data.

Basic information about Pieter has been made available in both HTML and RDF (serialized in Turtle and RDF/XML) through conneg, using the setup described above.

When you do a GET to https://pieterheyvaert.com/#me without an Accept header, an HTML document is returned that provides information about Pieter, such as topics of interest and ongoing work. Try it out yourself by browsing to https://pieterheyvaert.com/#me or via

curl https://pieterheyvaert.com/#me

When you do a GET with the Accept header text/turtle, RDF in Turtle format that describes Pieter is returned. Try it out yourself via

curl -H 'accept: text/turtle' https://pieterheyvaert.com/#me

And finally, when you do a GET with the Accept header application/rdf+xml, RDF in XML format is returned. Try it out yourself via

curl -H 'accept: application/rdf+xml' https://pieterheyvaert.com/#me

Explaining an ontology via HTML and RDF

Ontologies provide structured descriptions of domain models. They define the different concepts and elements of a certain domain and describe how they are related to each other. Take for example the Open Velopark Ontology. This ontology provides a description of concepts and properties related to bicycle parkings. This description can be used, for example, by people wanting to know what different types of bike parkings are contemplated in the domain model or by machines to automatically discover and reason over data about bike parkings.

The information contained in the ontology needs to be delivered in the right format for the right audience. For humans using a browser, a HTML representation would be the most appropriate format while for machines, a machine-readable format (e.g., RDF, JSON, XML and so on) would probably be the best fit. This is where conneg comes into play. By exposing different representations of the same resource through its URI, we are saying that anyone and anything can request the information contained in that resource using the format they need (if available of course).

The Velopark Ontology has been made available in both HTML and RDF (serialized in Turtle) through conneg, using the setup described above.

When you do a GET to http://velopark.ilabt.imec.be/openvelopark/vocabulary without an Accept header, an HTML document is returned that provides information about the ontology. Try it out yourself by browsing to http://velopark.ilabt.imec.be/openvelopark/vocabulary or via

curl http://velopark.ilabt.imec.be/openvelopark/vocabulary

When you do a GET with the Accept header text/turtle, RDF in Turtle format that describes the ontology is returned. Try it out yourself via

curl -H 'accept: text/turtle' http://velopark.ilabt.imec.be/openvelopark/vocabulary

Summary

In this blog post we show that we can add conneg to a website by using NGINX together with the http-server. Both NGINX and the http-server can be easily configured to support this. Different documents corresponding with the different representations for each resource are served by the http-server depending on the Accept header of the request. NGINX and other services running behind it are unaware of conneg.

If you have any questions or remarks, don’t hesitate to contact me via email or via Twitter.

Originally published at pieterheyvaert.com on February 25, 2019.