DEV Community: Ivan S Glazunov

Associative links

Ivan S Glazunov — Sun, 12 Sep 2021 08:52:42 +0000

What is the ideal "quantum of information" or "minimal unit of meaning" and why to ask such a question? There is a lot of pain associated with this, often not even realized. We offer a solution, but first you need to figure out what it is for. We recommend you to read the first article in a series.

Retrospective

Relational tables, document oriented databases, jsonb stores ... We often use the model in the table/collection as the minimum unit of meaning. The model consists of columns/fields with a different structure, and reflects some real object or abstract concept defined by the author.

This approach creates problems:

Lots of address spaces. (If you do not see it as a problem, then soon where will be an article exclusively on this topic, and you will see why this problem is real).
Many control points of the data structure that are expensive to change.
Compatibility only at the level defined by the author.
Failure to fully describe something new without creating another table/collection.
The inability to separate objects from the relationships between them.

All of these problems are addressed in the Deep.Case associative model of data.

If you are interested in optimization or search by ranges in a graph, there will be a link here to that article.

Let's dig into it

Deep.Case operates with the Link concept as a minimal unit of meaning / quantum of information.

Links table

Links (a network of links) is very similar to a graph, however, the graph has a division into nodes and edges referring only to nodes. Links do not divide these address spaces. This allows links to refer to links and carry any meaning that the author of the data models can put in them.

Link structure

The link structure consists of the required unique id and type_id used as the type of this link. The optional from_id and to_id fields can only be specified together.

Types for creating links are easily created by the user himself within the project model, or supplied with imported model packages.

The specified from_id and to_id fields refer to other links. Any relationship can be described with such links.

{
  links(where: { id: { _eq: 8 } }) {
    id
    type_id
    from_id
    to_id
  }
}

{
  "data": {
    "links": [
      {
        "id": 8,
        "type_id": 1,
        "from_id": 7,
        "to_id": 6
      }
    ]
  }
}

Link as a node

You can leave from_id and to_id fields empty in links. Such links play the role of a node, a point, an object of relations.

{
  links(where: { id: { _eq: 28 } }) {
    id
    type_id
    from_id
    to_id
  }
}

{
  "data": {
    "links": [
      {
        "id": 28,
        "type_id": 6,
        "from_id": 0,
        "to_id": 0
      }
    ]
  }
}

Link as a link

If from_id and to_id are specified, the link plays the role of a relationship between nodes or links. It can answer the question of how they are related using type_id. The environment of a node with links can answer many questions about its meaning, relationships, purpose, value and state.

{
    links(where: { id: { _in: [6,7,8] } }) {
    id
    type_id
    from_id
    to_id
  }
}

{
  "data": {
    "links": [
      {
        "id": 6,
        "type_id": 1,
        "from_id": 0,
        "to_id": 0
      },
      {
        "id": 7,
        "type_id": 1,
        "from_id": 0,
        "to_id": 0
      },
      {
        "id": 8,
        "type_id": 1,
        "from_id": 7,
        "to_id": 6
      }
    ]
  }
}

Unified GraphQL API

This API allows to do a lot of things, such as traversal to all children/parents in certain subtrees using index. But there will be separate articles about this.

Obviously, the structure of the link itself is available there, and also from a link you can go to other links both by its links (from_id and to_id) and by backwards references from other links (outgoing and incoming). For example, you can go to all links that refer to this link by from_id (these links go from it) using relationship out, or by to_id (these links go to it) using relationship in.

{
    links(where: { id: { _eq: 8 } }) {
        id
        type { id }
        from {
            id
            type_id
            from_id
            to_id
            out {
                id
                type_id
                from_id
                to_id
            }
        }
        to {
            id
            type_id
            from_id
            to_id
            in {
                id
                type_id
                from_id
                to_id
            }
        }
    }
}

{
  "data": {
    "links": [
      {
        "id": 8,
        "type": {
          "id": 1
        },
        "from": {
          "id": 7,
          "type_id": 1,
          "from_id": 0,
          "to_id": 0,
          "out": [
            {
              "id": 8,
              "type_id": 1,
              "from_id": 7,
              "to_id": 6
            }
          ]
        },
        "to": {
          "id": 6,
          "type_id": 1,
          "from_id": 0,
          "to_id": 0,
          "in": [
            {
              "id": 8,
              "type_id": 1,
              "from_id": 7,
              "to_id": 6
            },
            {
              "id": 13,
              "type_id": 1,
              "from_id": 6,
              "to_id": 6
            },
            {
              "id": 24,
              "type_id": 22,
              "from_id": 23,
              "to_id": 6
            }
          ]
        }
      }
    ]
  }
}

SQL-like queries

We can receive not only the data on the structures described above. We can use the sql-like where predicate to perform a variety of complex filtering at each level.

{
  links(where: {id: { _eq: 6 } }) {
    id
    type { id }
    from { id }
    to { id }
    out(where: { from_id: { _eq: 6} }) {
      id
      type_id
      from_id
      to_id
    }
    in(where: { type_id: { _eq: 1} }, limit: 1) {
      id
      type_id
      from_id
      to_id
    }
  }
}

{
  "data": {
    "links": [
      {
        "id": 6,
        "type": {
          "id": 1
        },
        "from": null,
        "to": null,
        "out": [
          {
            "id": 13,
            "type_id": 1,
            "from_id": 6,
            "to_id": 6
          }
        ],
        "in": [
          {
            "id": 8,
            "type_id": 1,
            "from_id": 7,
            "to_id": 6
          }
        ]
      }
    ]
  }
}

Subscriptions

And all of this already with support for subscriptions. You can simply describe to which structure you want to subscribe - and you will receive updates on the results of your query in real-time.

subscription {
  links(where: { from_id: { _eq: 7 }, to_id: { _eq: 6 } }) {
    id
    type_id
  }
}

Support associative technologies

We create a development environment (boxed CE / EE version and SaaS version), which allows you to use data storage for solving your business problems with the ability to adapt to any business changes using an associative approach. We will create a distributed network of server clusters in a single associative space, including so as not to think about regional legislation when creating a project. We create a culture of publishing reusable data models with their behavior.
Join our community on Discord. Sign up for early access on our Waitlist or support us on Patreon.
On our site you can find links to drafts of future articles in development, links to code sources, plans for project and product management and invest presentations.

Refactoring factor

Ivan S Glazunov — Mon, 30 Aug 2021 21:06:24 +0000

The article was translated by Deep Foundation’s co-founder Konstantin Dyachenko.

Code is dependent on data and models, and therefore on the abstractions used in them, so refactoring is inevitable today. Why? Usually, refactoring means changes in your code needed in order to use data in a new way. We'll talk about the most common and least favorite type of refactoring that causes the snowball effect. It occurs when data models, table structures and business logic are changed.

The Deep's philosophy describes everything using the concept of a Link. Any object is a Link, any relationship is a Link. The relationship always has from and to fields specified. In an independent object, from and to fields are not specified. This also distinguishes Deep's philosophy from graph databases, where edge cannot serve as an object of relationships.

What can be changed depending on data models?

Fields renaming.
Changing of one to one/many relationships
Adding abstractions
Deprecating abstractions

You can always try to patch up backward compatibility issues by adding new adapters and layers. But this fight against symptoms will only postpone the consequences of the real problem. With every change in business logic, the onion effect will only grow. And there will be more abstractions that are intertwined with each other.

Many programmers will argue - this is a matter of clean code culture. We disagree. The culture of clean code is only about its implementation. The problem is not how we write the code, no matter how much we want to believe it. The problem is that it is, in principle, dependent on abstractions of business logic. Programmers are trying to fix the consequences of this problem, and it does not reduce the complexity of the code, but only multiplies it. At the same time, programmers surround themselves with imaginary feelings of comfort and control over the situation.

Deep.Case aims to defeat this enemy. Data models can evolve without refactoring at all. How? In order to explain, we'll have to dig to the root causes of problems.

Renaming of fields

There are many types of code architecture. You can use GraphQL and schema generators, or you can map APIs and data abstractions through the ORM/ODM yourself, forwarding table rules to your code. You can have functional API and REST API from server. But, probably, in any case, the way of working with these APIs will be defined either at the API level or at the level of column names in tables. Thus, we compensate for the induced changes by paying developers to update the database, generators, resolvers and APIs. The problem here is the separation of the implementation of the functionality and its integration into the business logic. This one layer of abstraction is multiplied by the intersection of the business logic rules and the number of ways columns are connected, ultimately making the price of each wave of refactoring highly dependent on the age of the project. It was not possible to accurately calculate the dependency factor, but this price always turns out to be a multiple times greater than the cost of only the field modification and behavior changes in the business scenario.

Here, you must also always take user permissions and any other business rules into account. This increases the cost of such waves of refactoring in multiples of the number of business rules.
Deep.Case allows you to completely forget about this problem!

Changing one to one/many relationships

No matter how we indulge ourselves in the illusion of being able to predict everything, this is obviously not the case. Even the most ideal model will change. What we considered a single connection will become multiple. What we considered to be multiple will begin to be referenced from several places. Models conceived for use in some places become necessary to be applied in many others. Every such change today requires developer involvement, again and again.

With Deep.Case data is divided into:

links used as nodes (objects|points) answering the WHAT question, such as: payment, volume, need, moment, and other
links used as relationships (links) answering the HOW IS RELATED question, such as: X wants Y, W watched Z, T owns R, T answers P
custom tables supplied by data models, accessible from a node of corresponding type. The purpose of such tables is storage of custom non-referenced data that is more convenient to describe by a separate table.

Links describing relationships are not initially limited by the system as one or many. Rules for creating links of each type can be described at the model level, and subsequently easily changed.

This means that for transition from one to many, it is enough to allow the creation of multiple links of a certain type.

It is recommended that all handlers and client interfaces assume the iterability of each subset. It is very cheap to take this into account in initial development. Even if we think that there is only one instance of this type of link, we must assume that there can be many of them.

Adding new levels of abstraction

For example, when you need to create a new type of link and new entities. With Deep.Case you no longer need to go through a full cycle of changing data structures, ORM/ODM models, or take into account how the code reflects the required abstractions, and its inflexibility slows down changes in business logic. You no longer need a backend developer for this kind of change. Just create a new type of links and describe under what circumstances this link can be created and changed in the surrounding space. It will immediately become available through a single GraphQL/REST API without damaging the old abstractions and links, and therefore the dependent code.

Deprecation of old levels of abstraction

Since there is no longer a self-written API resolver layer, the deprecation of old abstractions is reduced to a recommendation not to use certain links in the new code. But since some subset of interfaces and handlers can be tied to their existence, their existence can simply be maintained at the database level. Even if the package that supplies the data model wants to switch to a different type of links, nothing prevents, while creating new links, to maintain mirror structures, that are based on these links.

Support associative technologies

Join our community on Discord. Sign up for early access on our Waitlist or support us on Patreon. https://deep.foundation/

On our site you can find links to drafts of future articles in development, links to code sources, plans for project and product management and invest presentations.