DEV Community: Erwan Carriou

Creating an ODM with JavaScript

Erwan Carriou — Sun, 11 Feb 2018 12:45:43 +0000

I think the first time I heard about ORM was 10 years ago. I discovered this technique on a NetBeans tutorial that explained how to generate Java classes from a MySQL database. I made some tests and it worked pretty well. I really liked the concepts but not the execution. I was quite frustrated with the generation step because I had always to regenerate classes for every structure update of the database. This problem was in fact related to the language used for the mapping that needs to be compiled. I said to myself that it could be simpler to use a dynamic language that could generate these classes at runtime. That’s why I started at that time to create my own ORM with JavaScript. It worked quite well but I get stuck with a big limitation: MySQL. The relational structure of the tables did not match with JavaScript native objects. So the mapping was not as easy as I wanted.

But things changed a few years later when NoSQL databases became more and more popular. We could use pure JSON objects as documents and we could manage NoSQL data as native JavaScript objects.

I will show you in this post how it is easy to create now an ODM (Object-Document Mapping) with JavaScript.

My first ODM

Let’s start by choosing a NoSQL database. We will use my favorite one, I call it the universal database: {}.

const db = {};

It is light, can work on a server or on a browser. Everything I like!

Now that we have the database, let’s stop a minute to think about object creation in JavaScript. Generally we use many parameters to create an object, like this:

const luke = new Jedi('Luke', 'Skywalker');

But we could also pass one object as parameter:

const luke = new Jedi({
  firstName: 'Luke',
  lastName: 'Skywalker'
});

Did you notice that this parameter looks like a document? That is the main idea of ODM: use a document as a parameter of the class constructor.

Now we have that in mind, let’s create the class that will manage the mapping between documents and class instances:

class ODM {
  constructor(document) {
    // get class name
    const name = this.constructor.name;

    // add unique id
    if (!document._id) document._id = Math.random().toString();

    // create document
    if (!db[name]) db[name] = {};
    db[name][document._id] = document;

    // define accessors
    const configuration = {};
    Object.keys(document).forEach((prop) => {
      configuration[prop] = {
        get() {
          return db[name][document._id][prop];
        },
        set(value) {
          db[name][document._id][prop] = value;
        }
      };
    });

    // set accessors
    Object.defineProperties(this, configuration);
  }
}

In this class we made several things:

we get the name of the collection in the database: in our case the class name,
we generate a unique id for the document,
we add the document in the database and
we create getter and setter for every property of the instance that will manage the related document in the database.

Now let’s do some tests with it:

// create a Jedi class
class Jedi extends ODM { };

// create an instance with a document
const luke = new Jedi({
  _id: 'luke',
  firstName: 'Luke',
  lastName: 'Skywaker'
}); 

// update the instance
luke.lastName = 'Skywalker';

// check that the value has been changed in the database
db.Jedi.luke.lastName;
// => 'Skywalker'

We have now a complete synchronization between documents and instances. And we did that with only 30 lines of code!

Documents exportation

Let go further. And if want to export documents? It is very easy to do that:

db.export = (name) => {
  return JSON.stringify(db[name]);
};

In our case we suppose that all the documents are JSON valid so that we can export them with a native JavaScript API.

Now let’s make some tests with it:

// create the Jedi class
class Jedi extends ODM { };

// create an object with a document
const luke = new Jedi({
  _id: 'luke',
  firstName: 'Luke',
  lastName: 'Skywaker'
}); 

db.export('Jedi');
// => '{\"luke\":{\"firstName\":\"Luke\",\"lastName\":\"Skywaker\",\"_id\":\"luke\"}}'

In this example we export all the documents created for a specific class. That means that we can now serialize all the objects into a string. Pretty cool, does it?

Documents importation

Now we will do something a little more complicated with the importation of documents. When we import documents on a specific collection, we want to create their related objects:

// create classes list
const classes = {}; 

db.import = (name, documents) => {
  db[name] = JSON.parse(documents);

  // create instances
  Object.keys(db[name]).forEach((id) => {
    new classes[name](db[name][id]);
  });
};

Now let’s update a bit the main class for that purpose:

// create instances list
const instances = {}; 

class ODM {

  constructor(document) {
    // get class name
    const name = this.constructor.name;

    // add unique id
    if (!document._id) document._id = Math.random().toString();

    // create document
    if (!db[name]) db[name] = {};
    db[name][document._id] = document;

    // define accessors
    const configuration = {};
    Object.keys(document).forEach((prop) => {
      configuration[prop] = {
        get() {
          return db[name][document._id][prop];
        },
        set(value) {
          db[name][document._id][prop] = value;
        }
      };
    });

    // set accessors
    Object.defineProperties(this, configuration);

    // add it to the list of instances
    instances[document._id] = this;
  }
}

The difference with the previous class is that we add now the created instance in the list instances.

Let ‘s test it:

// create Jedi class
classes.Jedi = class Jedi extends ODM {};

// import Jedi documents
db.import('Jedi', '{\"luke\":{\"firstName\":\"Luke\",\"lastName\":\"Skywalker\",\"_id\":\"luke\"}}');

// access the created instance
instances.luke.firstName;
// => 'Luke'

We can now deserialize datas into objects. Moreover we can also know the exact number of created objects at any moments, it is the number of objects in my instances list.

Managing data relationships

And what about relations? In NoSQL world, we can simulate relations by using the id of a document as a value of a property to create a link. If we follow this pattern, managing relations becomes very simple:

class ODM {

  constructor(document) {
    // get class name
    const name = this.constructor.name;

    // add unique id
    if (!document._id) document._id = Math.random().toString();

    // create document
    if (!db[name]) db[name] = {};
    db[name][document._id] = document;

    // define accessors
    const configuration = {};
    Object.keys(document).forEach((prop) => {
      configuration[prop] = {
        get() {
          const value = db[name][document._id][prop];
          // return an instance or a value
          return value.indexOf('@') !== -1 ? instances[value.replace('@','')] : value;
        },
        set(value) {
          if (classes[value.constructor.name]) {
            // store the id of the instance
            db[name][document._id][prop] = value._id;
          } else {
            db[name][document._id][prop] = value;
          }
        }
      };
    });

    // set accessors
    Object.defineProperties(this, configuration);

    // add it to the list of instances
    instances[document._id] = this;
  }
}

To distinguish a value from a link, we add this new rule: if a value begins with @, it means that it represents the id of a document.

Let’s create now a link between objects:

const vador = new classes.Jedi({
  _id: 'vador',
  'firstName': 'Dark',
  'lastName': 'Vador'
)};

const luke = new classes.Jedi({
  _id: 'luke',
  'firstName': 'Luke',
  'lastName': 'Skywalker',
  'father': '@vador'
)};

luke.father.lastName;
// => 'Vador'

Now, let’s do this link at API level:

const vador = new classes.Jedi({
  _id: 'vador',
  'firstName': 'Dark',
  'lastName': 'Vador'
});

const luke = new classes.Jedi({
  _id: 'luke',
  'firstName': 'Luke',
  'lastName': 'Skywalker'
});

// set father link  
luke.father = vador;

db.export('Jedi');  
// => '{\"vador\":{\"_id\":\"vador\",\"firstName\":\"Dark\",\"lastName\":\"Vador\"},
// \"luke\":{\"_id\":\"luke\",\"firstName\":\"Luke\",\"lastName\":\"Skywalker\",\"father\":\"@vador\"}}'

As you see, we can create one-to-one relationship very easily with ODM.

Conclusion

ODM is a technique that you have to use more often in your code, it is not complicated and very powerful. Because of the heavy coupling between documents and objets, you know at every moments what are the components of your application, how many they are and what data they manage.

If you think harder, you realize that in fact ODM is a way to manage the store of your application. Every kind of components has its own store (i.e. collection in the database) and can be managed like in Redux. But here you are at the functional level (you manage functional objects), not at the technical level (where you manage data).

I made some CodePen examples so that you can start to play right now with ODM:

If you want to go deeper, you can have a look at System Runtime, a JavaScript library that I have created that applies all the pattern I talked about in this post.

Credits: cover image by Sebastien Gabriel.

Designing Models in JavaScript

Erwan Carriou — Sun, 28 Jan 2018 12:57:49 +0000

My grand-mother was a brodeuse, french name for embroiderer. She made beautiful traditional clothes for my family that we keep preciously. Recently when cleaning her home, we found a treasure, the patterns she used for creating her clothes. She used to draw her patterns on big, large papers. We could clearly understand the kind of clothes she wanted to do by looking at the patterns.

I could stop myself making an analogy with our work. We avoid too many times this fundamental step of design when developing. We start coding before designing the model of our application just because we think the code is the model. And it is not. The code is just the fabric of which the application is made. By reading the code, we only could guess what model, patterns were used to create the application. Moreover, because we are human, because our way of thinking differ from one developer to another developer, we need also to understand the mental space organization that the developer had when she/he was coding. And it is not that easy.

Separation of concerns

That is why no matter the frameworks you use and the code you write, the most important is the model that you define to create your application. And this model do not have to be in the code. It has to be somewhere else, normalized in a human readable format. But what format? We could use UML, it is quite common now and there are many design tools that use it. But the problem is that it is not a format, it is a language. It is quite complex and not so human friendly when you need to understand a diagram. I know that UML diagrams can be saved in XMI format but it is for machine not human.

Create the schema of the model

Let’s find the format we need with an example. Lets say we want to create a model of a Jedi. A Jedi is defined by its first name and its last name, she/he has got a mother and a father, and can have children. Let’s say we put all these informations related to this model into a JSON file, we could have something like that:

{
  "_name": "Jedi",
  "firstName": "property",
  "lastName": "property",
  "mother": "link",
  "father": "link",
  "children": "collection"
}

When reading this JSON, we easily understand that:

a Jedi has two properties, firstName and lastName,
a Jedi is linked to a mother and a father and
a Jedi has a collection of children.

This JSON is in fact the schema of our model.

Generate the model and extend it

It would be great to be able to define a model like that, right? Now we have this schema, let’s go further. And what if we could generate UML class diagrams from the model to get something like this:

In order to do that, we are missing some basic informations, like the type of the properties. How to do this with our file? By simply generating a complete model with default values from the schema and then edit it:

{
  "_name": "Jedi",
  "firstName": {
    "type": "any",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "lastName": {
    "type": "any",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "mother": {
    "type": "Component",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "father": {
    "type": "Component",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "children": {
    "type": ["Component"],
    "readOnly": false,
    "mandatory": false,
    "default": []
  }
}

We have here all informations defined in the schema that we can edit in a human readable format. For example, for firstName we can set its type (that has for default value any which means that it can have any type) to string.

From this JSON it will be easy to generate a UML class diagram. We have a complete model to build the application.

Use the model at runtime

Now what about the code? In Model Driven Architecture approach, we generate code from the model, generally from UML diagrams. It is great, but not perfect because the model we have defined has higher risk to be desynchronized from the code. So how to avoid this? By simply using the model definition of the application at runtime. The application has to read the schema and the model that we defined to create the classes, methods and components of the model.

Yes, seems pretty cool, but how do I do to implement the methods if it is generated at runtime? To do that you need to think in systems. A method is just an action that the system does when reacting to an event.

In our case, let’s say we need to have in Jedi a method to get her/his full name. So we will edit the schema to have something like this:

{
  "_name": "Jedi",
  "firstName": "property",
  "lastName": "property",
  "mother": "link",
  "father": "link",
  "children": "collection",
  "fullName": "method"
}

We have added fullName method in the schema. So we will have in the generated model:

{
  "_name": "Jedi",
  "firstName": {
    "type": "any",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "lastName": {
    "type": "any",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "mother": {
    "type": "Component",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "father": {
    "type": "Component",
    "readOnly": false,
    "mandatory": false,
    "default": ""
  },
  "children": {
    "type": ["Component"],
    "readOnly": false,
    "mandatory": false,
    "default": []
  },
  "fullName": {
    "params": [
      {
        "name": "param",
        "type": "any",
        "mandatory": false,
        "default": null
      }
    ],
    "result": "any"
  }
}

By default fullName takes one optional parameter and can return a value of any type. Now let’s say we want to implement the method. How to do that? By simply adding a behavior when the fullName event is send:

// require the Jedi class
const Jedi = runtime.require('Jedi');
// add a behavior
Jedi.on('fullName', () => this.firstName() + ' ' + this.lastName());

And when this event is send? When fullName method is called in the code. In fact this method has been generated from the model, and when we invoke it, it just send a synchronous fullName event. The existing behavior related to that event will be then executed.

Now if we change the model of the method, it will never override its behavior.

Conclusion

Here are the different steps to follow when designing a model:

Define your model at high level in a human readable format.
A model will be generated from that schema.
Edit the generated model to specify the types, default values, … of your model.
Use this model to create at runtime classes, methods and components of your model.
Architecture your application with an event-driven approach to add behaviors to the methods.
Use your model to generate UML class diagrams.

If you want to see a concrete use of this approach, you can have a look at two of my projects:

System Runtime, a JavaScript library to run and manage systems and
System Designer, a web IDE to design systems

that use deeply this methodology. For that purpose I have created the MSON format (for Metamodel JavaScript Object Notation) to define models in a human readable format. But it is not the only possible implementation of this methodology, find the one that matches your needs.

Credits: cover image by Christian Kaindl.

Designing Systems

Erwan Carriou — Sun, 21 Jan 2018 13:06:20 +0000

In my previous post I explained that, as a developer, you do not create applications but in fact systems, living entities that react to their environment (browsers, servers, …). On this post I will describe a process that you can follow in order to create systems. This process has been inspired by the amazing work of Donella H. Meadows, the B-Method that I learned at the University and the Domain Driven Design.

This process is composed of 5 steps:

Understand the problem
Design your system
Agree on the design
Refine your design
Generate your system

1. Understand the problem

Always remember that you are human

In order to create systems you need first to realize that you perceive things and understand the world as a human beeing. It means that when we think to a specific problem, we resolve it with our own mental representation of the world. We are never objective when we think. We are limited by our culture, life experience, mother tongue, … We can miss useful informations because we can not see them from our perspective. As a result the more you are and diverse you are to solve a problem, the better solution you get. So create the most diverse team you can and invite them to a meeting to start designing the system.

Be inclusive

First of all listen without interruption people that are asking for a feature/solving a problem/… . Then ask questions about the context. Get the maximum informations you can and make all the team talk. It is very important. Everybody needs to feel heard during the meeting in order to be able to share without difficulty their own understanding of the context.

Speak the same language

The main issue in project is always communication, so be sure that everyone speak the same language, the user language, and not a technical one. For that purpose note on a paper all the words you use to define the context and write their definitions. So that everyone will know what you are talking about.

2. Design your system

Take a paper and a pen

Because if you can not design it, you can not explain it to other.

Create the border

Draw a big circle that represents the border of the system. Then write all the words taken from the list you made.

Write inside the circle the concepts that you think are part of the system. Write outside the circle the concepts that you think are not part of the system.

The border depends on your comprehension of the system and could change from one perspective to another. For example: when designing a web app, you can include the server in the system… or not. It depends on where you put this border.

Find what are the events that provoke a response from the system

Then look at the concepts that you put outside the system. Think how these concepts can interact with your system.

Create arrows that go into the system for each message that could be send to the system. And name the event responsible for this message.

Create arrows that go out the system for each message that could be send from the system. And name the event responsible for this message.

Design the model

Now look at all the words that you put inside the system. Create a square around each word. Then draw a line between each square that you think are related together. Do not add information on the link, just draw a line. The idea is to have an global overview of the concepts and their relations.

Find the core components

Name the components that you need to have to start the system and draw little circles that represents them. We will call them the core components.

3. Agree on the design

Now we have a complete overview of the system. We know:

what is the border of the system,
what are the events that provoke a response from the system,
what is the model of the system and
what are the core components of the system.

Be sure that everyone agree on the design and understand how the system is composed.

4. Refine your Design

Describe the model

Now that everyone agree on the design, you can describe more precisely the model. To do that, you need to ask:

What are the properties of the model?
What are the behaviors of the model?
What are the type of links. Are they collection, inheritance, …?
What are the events send by the model?

I encourage you to use UML to define the model. But keep it simple like we did before. Always use design that everybody can understand.

Find the initial states of the core components

Find what are the initial values of the core components of the system. The question you have to answer is simple: at which states the core components needs to be in order to start the system?

I encourage you to describe these values in a JSON object. It is human readable format that you can easily update.

Define the messages

Define what are the types of the messages send to the system and returned by the system.

Use also UML to define the structure of the messages.

5. Generate your system

Now you have define the model and find the initial states of the core components, generate the system from the model. Do not start to code, otherwise your code will be always desynchronized with your model. Then implement the behavior of your system from the generated code.

There are many tools that can do the code generation, find the one that fills your needs. If you focus on JavaScript, you can try System Designer, a web IDE that I have created in order to design and create systems. It is open source and free.

Conclusion

Designing a system is a complex task that needs the work of everyone. Technical skills are not necessary to design a system, but human skills are mandatory.

It was a quick overview of the process I use when I create systems. I did not go into specific details, I only described the main steps of this process so that you can adapt it to your work.

In my next post I will go more deeper on the model designing process and explain how to have a model synchronized with your running system.

Credits: cover image by Sergey Zolkin.

Thinking in Systems with JavaScript

Erwan Carriou — Sun, 07 Jan 2018 17:08:26 +0000

Everytime I have to fix a bug, I follow the same workflow: when someone at the QA team found a bug, she/he send me a bug report with the steps to reproduce (STR) the issue. And if I do not understand the STR, I generally come to see her/him so that she/he can show me the bug on her/his computer.

This workflow seems quite common in many companies and I often say to myself that it must have a better way to get the context of an application before a bug happened.

Dynamic Bundling

But what could be this better way? Ideally it could be great to make a snapshot of the application before the bug occured so that we could fix the issue from that state.

For example let’s say that we have an application in the state we want and that we can save that application into a format that can be then loaded to restore that state. For sure the resolution of the fix would be faster and easier.

In fact, the new worflow could be something like that:

In this example we have a todo app (the basic TodoMVC app) in a specific context (with one task). We export the context of the application in a bundle (a stringified JSON object), open a new blank page and then install that bundle. Then we see our application running on the new page with the correct context. So we can start using the application from that context.

It means that, as a developper, I will have only to load a JSON that someone at the QA team send me to get the context of the application and fix that bug. Much simpler, does it?

How does it work?

In the video, we can export the state of this application and restore it at runtime because:

the application was designed as a system and
the application objects (components, methods, models) are stored at runtime in a tiny NoSQL Database.

Your application is a system

When we create an application, we create in fact a system. A system is defined by a model, is composed by components and reacts to events with behaviors. As you see, these concepts are quite common with the applications we create every day.

So what differ a system from an application ? With systems we focus first on the design before focusing on the code. How to do that ?

First design the model of your system,
then find all the components that you will need to start your system and
then create these components and implement its behaviors (with methods).

You need to make a complete disctinction between the design and the runtime of your system. Design must always be declarative and execution imperative. How to do that? Use UML to define your model and integrate it into your development workflow.

Everything is a document

Once we have the system ready to be executed, we need to store it in a NoSQL Database. It is possible because everything you have created can be managed as a document. Let’s say we want to store an object in a database, we need to serialize it in JSON, but if we only store its state this process will be easier. And it is what is done in the video. Models and behaviors are also serialized so that the entire system is stored in a database.

And what about runtime? What if we update an object in the current application? Because all objects states are stored in a database, we have a complete ODM (Object-Document Mapper). It means that an update to an object of the system will automatically update its state on the database.

So now exporting the current state of the system is like making a dump of the database. And restoring the state of the system is like importing the dump into the database. Quite simple, isn’t it?

Want to learn more?

I will develop the concepts and patterns I talked about in a forthcoming post but if you want to create some systems right now you can:

install System Runtime, a JavaScript library to create and manage systems and
read the book Thinking in Systems from Donella H. Meadows. A great introduction to the world of systems.

Credits: cover image by Ant Rozetsky.