Here's our journey through data !

How we manage data at Lenra

Data management is a very important part for every application, even for Lenra. Today, I'm going to introduce you to our data management needs and how we solve them. But first, I will introduce you to the type of data we need to manage at Lenra.

www.lenra.io is is made to accelerate the creation of your applications and simplify their hosting while preserving the personal data of your users.
Lenra is built around the values of digital sobriety and data protection, meaning users of its tools anchor their practice in a more responsible and ethical digital world.

Malware protection is a key point for our platform and has been respected through a process of improving the security of your data.

The platform allows GDPR compliance and also the control by the user of his personal data.

Data ?

A Lenra application is made of 2 components: widgets and listeners. The widgets are the graphical components, they are in JSON format and represent a description of the graphical interface. The listeners are the functions called in reaction to the user's actions on the graphical interface, such as pressing a button. All these components are described in a file called "manifest". It lists all the components that can be used within the application. In this part we will see the components as they were before the new data system was implemented. Schematically, here is how an application works

Widget

Widgets are therefore the graphical components of an application, they describe the graphical interface in JSON format. A bit like HTML, the developer will be able to embed "lenra-components", a library made by Lenra. The "lenra-components" library is a set of flutter widgets reworked to fit the needs of the company.

(data, props) => {
 return {
   "type": "text",
   "value": "Hello World!"
 }
}

As shown in this example we have a widget that is a function taking two parameters as input and returning some JSON corresponding to the interface. The two parameters are :

data : A JSON object representing the user data

props : A JSON object allowing the developer to give properties to the widget

This widget in JSON format will be validated and transformed into a Flutter widget using the JSON Schema. Once transformed, it will be displayed on the application. The widgets also have the possibility to be built according to the data, indeed the data of the application are passed to the widget so that it can use them in order to have a modular interface according to the data, for example:

(data, props) => {
 return {
   "type": "text",
   "value": data.text
 }
}

Listeners

The listeners are functions reacting to the user's actions and making modifications on the data.

(data, props, event) => {
  return {
    text: "world"
  }
}

As shown in the example, a listener is a function taking 3 parameters:

data : A JSON object containing the user's data.

props : A JSON object that allows to give properties to the listener.

event : A String that allows to specify to the listener the event that triggered its call.

What we need

The system we have just seen is the system that was in place on Lenra. It was quite functional and quite easy to use but it could very quickly present some limits. Indeed on a small application like a "hello world" this system is quite sufficient, but on a larger application and storing different types of data, having only one data object can quickly become restrictive especially for data querying. A new system has therefore been designed.
We had some specifications to respect for the new data system:

We wanted to allow all developers to manage their data schema as they wish, a bit like Firebase does, we wanted a system that would abstract the entire data system behind the Lenra interface and allow developers to create their data easily through HTTP requests.

We also wanted a system that is GDPR compliant by design, meaning that if the developer implements the obligatory listeners, he will have a GDPR compliant application by default.

What we do

Following these needs, we have therefore imagined a data system that is close to an object-oriented database such as SQL for example. So we have two types of objects: Datastore and Data. A datastore can correspond to a SQL table and a data to an entry of this table. We thus have a Datastore containing data in JSON format which is a mix between a SQL system and a data format similar to NoSQL. In order to link data between them, we have added a DataReference object which allows to make a many_to_many relation between two Data.

Database schema

Here is the database schema

Here we find the tables of the three objects mentioned above, datastores, datas and data_references, as well as the applications and environments tables mentioned in the previous point. The last table user_datas allows us to identify to which user a data belongs, we will come back more precisely on how it works in the part which is reserved to it.

API

Creation of an API that handles create/add/delete on our objects

scope "/app", LenraWeb do
    pipe_through([:api, :ensure_auth_app])
    post("/datastores", DatastoreController, :create)
    delete("/datastores/:_datastore", DatastoreController, :delete)
    get("/datastores/user/data/@me", DataController, :get_me)
    get("/datastores/:_datastore/data/:_id", DataController, :get)
    get("/datastores/:_datastore/data", DataController, :get_all)
    post("/datastores/:_datastore/data", DataController, :create)
    delete("/datastores/:_datastore/data/:_id", DataController, :delete)
    put("/datastores/:_datastore/data/:_id", DataController, :update)
    post("/query", DataController, :query)
  end

If you want to know more about how our controllers work, you can find it at: https://github.com/lenra-io/application-runner/tree/b4dac9754ae9a1925b5ab9a921ed5129bad7a624/lib/controllers

Query system

After implementing the new data system, we had to think of a way to query the data. The application data must be accessible from two different locations.
First, we need to be able to retrieve and perform actions from the listeners. Indeed, listeners do not receive an object containing all the data of a user for an environment but must be able to request this data with an API call.
Second, the data must be able to be queried from the widgets. As a reminder, widgets are written in JSON format, so we had to think about a JSON querying system. The listeners use this same format and will send a POST request on the /app/query route.

query format

VALUE: (STRING, NUMBER, BOOLEAN, OBJECT, ARRAY),
QUERY: { ...MATCH_BODY: { ( ...PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION }, | ...PROPERTY_CHECK)+ }, },
FIND_FUNCTION: { $find: { ...MATCH_BODY, _datastore: STRING } },
MATCH_BODY: { ( ...PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION }, | ...PROPERTY_CHECK)+ },
PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION },
BOOLEAN_MATCHING_FUNCTION: ( MATCH_MATCHING_FUNCTION | EQ_MATCHING_FUNCTION | AND_MATCHING_FUNCTION | OR_MATCHING_FUNCTION | LT_MATCHING_FUNCTION | GT_MATCHING_FUNCTION | NOT_MATCHING_FUNCTION ),
BOOLEAN_MATCHING_FUNCTION_LIST: [ MATCH_BODY+ ]
EQ_MATCHING_FUNCTION: ( { $eq: VALUE } | VALUE ),
MATCH_MATCHING_FUNCTION:  { $match: MATCH_BODY },
AND_MATCHING_FUNCTION: { $and: BOOLEAN_MATCHING_FUNCTION_LIST },
OR_MATCHING_FUNCTION: { $or: BOOLEAN_MATCHING_FUNCTION_LIST },
NOT_MATCHING_FUNCTION: { $not: MATCH_BODY },
GT_MATCHING_FUNCTION: { $gt: NUMBER },
LT_MATCHING_FUNCTION: { $lt: NUMBER },

query parser

You can find our complete parser here: https://github.com/lenra-io/query-parser/blob/2d3883e450826b35cfa09c1c80c24fc453dc1b71/lib/ast/parser.ex

Once we have defined the format of the requests as seen before, we had to implement a parser that will easily transform this JSON into an Ecto request. To do so, we proceeded by steps, the first one being to transform the JSON into a format that can be easily processed by Elixir. To do this, we chose to make an AST tree, the AST tree corresponding to the requests follows this pattern:

the pattern matching system provided by Elixir allows us to easily identify the patterns present in the JSON object in order to build step by step the ast tree with the corresponding object.
Our parser is composed of two main functions parse_expr and parse_fun, here are the declinations of each function:

In the case where we have a list as input, it is actually a $and:

defp parse_expr(clauses, ctx) when is_map(clauses) do
    parse_expr({"$and", Map.to_list(clauses)}, ctx)
end

A key that starts with $ is a function:

defp parse_expr({"$" <> _ = k, val}, ctx) do
  parse_fun({k, val}, ctx)
end

A simple k => v clause:

defp parse_expr({k, v}, ctx) do
  ctx = Map.merge(ctx, %{left: from_k(k, ctx)})
  parse_expr(v, ctx)
end

If there is a left in context, and is not a function, this is a simplified $eq function:

defp parse_expr(value, %{left: _} = ctx) do
  parse_expr({"$eq", value}, ctx)
end

Final parse_expr create Value object:

defp parse_expr(clauses, ctx) when is_list(clauses) do
  %ArrayValue{values: Enum.map(clauses, &parse_expr(&1, ctx))}
end
defp parse_expr(value, _ctx) when is_bitstring(value) do
  %StringValue{value: value}
end

defp parse_expr(value, _ctx) when is_number(value) do
    %NumberValue{value: value}
end

Parse function and with only one clause

defp parse_fun({"$and", [clause]}, ctx) do
    parse_expr(clause, ctx)
end

Parse function and with many clauses

defp parse_fun({"$and", clauses}, ctx) when is_list(clauses) do
    %And{clauses: Enum.map(clauses, &parse_expr(&1, ctx))}
end

Parse function eq

defp parse_fun({"$eq", val}, %{left: _} = ctx) do
    {left, ctx} = Map.pop(ctx, :left)
    %Eq{left: left, right: parse_expr(val, ctx)}
end

Using the same system of pattern matching with parse the AST tree into Ecto query that can be executed by our server

The limit

This system is functional and allows the developer to manage the data as he wants, only this system forces us to maintain a query system while Mongo already has a complete system.
So we can ask ourselves why keep this system and not just switch to Mongo, at the moment, the only answer is that we can't keep the data_reference but the developer can make the reference he wants by putting the data id in another data.
So we decided to rework the Postgres data management system to replace it with a system using Mongo.