DEV Community: Lenra

Join us now on Lenra, a unique place to create a cross-platform app that scales automatically to follow your users growth.

Alienor — Tue, 06 Jun 2023 10:06:00 +0000

Hey, you! Yeah, you...
Have you ever wanted to make a difference in the world, but you don't have the money or the resources to do it?
Well, I'm here to tell you that you can still make a difference without breaking the bank. How, you ask? By contributing to our open source project, of course!

Contributing to Lenra, an app creation platform, isn't just about making a difference. It's also about learning new skills. Whether you're a developer, a designer, or just someone who's interested in tech, you can learn a lot from collaborating with others on our project.

Lenra offers a range of benefits that can help developers simplify the app development and hosting process, while ensuring the privacy and security of user data.

Here are some insights :

Faster app development: Lenra provides an all-in-one platform with optimized dev tools, which can help developers accelerate the development process and reduce time-to-market[1].
Simplified app hosting: Lenra offers a range of hosting services that can simplify app deployment and management, including automatic scaling and reduced carbon footprint[2].
Enhanced data privacy: Lenra is GDPR-compliant and offers a range of security features, including malware protection and a Single Sign-On (SSO) system, which can help ensure the privacy and security of user data[3].
Ethical development: Lenra is a Big Tech-free platform with sober code, hosted within the European Union, which can help developers support ethical development practices[4].
Open-source and transparent: Lenra is an open-source platform that allows anyone to take ownership of the platform, and offers transparency in its development and operations[5].

Contribute now and take your part of the new world ;)

Sources:

Why YAML is better than JSON (read before screaming)

Alienor — Fri, 17 Mar 2023 09:20:10 +0000

You have probably seen or used the YAML format in configuration files.
YAML (a recursive acronym for “YAML Ain’t Markup Language”) is a human-friendly data serialization language for all programming languages, like the JSON format.

The YAML files are mostly written using the Python-style indentation to indicate nesting such as the following example of a Dofigen file:

# An anchor
from: &image "docker.io/bitnami/node:18"
# A string field
workdir: /app
# An object list
builders:
  # An object
  - name: module-loader
    from: *image
    workdir: /tmp/module
    adds:
      - package.json
      - package-lock.json
    script:
      - npm i --production --cache /tmp/cache
    caches:
      - /tmp/cache
  - name: builder
    from: *image
    workdir: /tmp/app
    adds:
      - .
    script:
      - npm i --cache /tmp/cache
      - npm run build
    caches:
      - /tmp/cache
artifacts:
  - builder: module-loader
    source: /tmp/module/
    destination: "."
  - builder: builder
    source: /tmp/app/dist/
    destination: dist/
  - builder: builder
    source: /tmp/app/resources/
    destination: resources/
# A string list
cmd:
  - npm
  - start
# An integer list
ports:
  - 3000
ignores:
- "**"
- "!/*.json"
- "!/src/"
- "!/resources/"

You may say:

OK, the YAML is great and very readable but how could it be better than JSON ?
They are just two different formats.

That's true, but I will show you how YAML is at least as good as JSON.

YAML is at least as good as JSON

The YAML format permits many ways to define the data.
We will see here the different ways:

strings
arrays/list
objects/strutures

strings

The basic string description is just writing the string:

myString: My super string

In order to disambiguate other data types like numbers or booleans, we can define the string by surrounding the value with simple quotes(') or double quotes("):

myString: "My super string"
alsoString: '10'
stringNotBoolean: "true"

For multiline strings we also can use two specific descriptions:

myString: >
  My super 
  multiline string

  Second line
otherString: |
  The other super multiline string
  Second line

Look at this website to read more about it.

arrays/lists

The main array description in YAML is the next one:

myArray:
- My string value
- 42

But YAML also permits the use of another syntax, by surrounding the array elements with brackets ([ and ]), very useful for short or empty arrays:

myShortArray: [first, second]
emptyArray: []

objects/structures

The main object description in YAML is the following one:

name: My string
age: 24
subobject: 
  nestedField: The value

That would result to the next JSON object:

{
  "name": "My string",
  "age": 24,
  "subobject": {
    "nestedField": "The value"
  }
}

But the YAML format also permits the use of braces ({ and }) to surround the object fields and they can be separated by a comma+space instead of a new line.
It also permits the use of simple quotes(') or double quotes(") arround the field name.
Here is the same object with this syntax:

{
  'name': My string,
  "age": 24,
  subobject: { nestedField: The value }
}

Merged YAML secondary format

Do you see where I'm going with this ?

Here is another way to describe the initial YAML example, but using the syntaxes seen previously:

{
  # An anchor
  "from": &image "docker.io/bitnami/node:18",
  # A string field
  "workdir": "/app",
  # An object list
  "builders": [
    # An object
    {
      "name": "module-loader",
      "from": *image,
      "workdir": "/tmp/module",
      "adds": [
        "package.json",
        "package-lock.json"
      ],
      "script": [ "npm i --production --cache /tmp/cache" ],
      "caches": [ "/tmp/cache" ]
    },
    {
      "name": "builder",
      "from": *image,
      "workdir": "/tmp/app",
      "adds": [ "." ],
      "script": [
        "npm i --cache /tmp/cache",
        "npm run build"
      ],
      "caches": [ "/tmp/cache" ]
    }
  ],
  "artifacts": [
    {
      "builder": "module-loader",
      "source": "/tmp/module/",
      "destination": "."
    },
    {
      "builder": "builder",
      "source": "/tmp/app/dist/",
      "destination": "dist/"
    },
    {
      "builder": "builder",
      "source": "/tmp/app/resources/",
      "destination": "resources/"
    }
  ],
  # A string list
  "cmd": [ "npm", "start" ],
  # An integer list
  "ports": [ 3000 ],
  "ignores": [
    "**",
    "!/*.json",
    "!/src/",
    "!/resources/"
  ]
}

Does it look familiar ?
Yes, the YAML format is fully compatible with JSON data.

But there are still some elements that you can't find in JSON format.

YAML additional features

The YAML format also has many interesting additional features, let's take a look at some of those I use in my projects.

Comments

One of the features that I miss the most in JSON files is the comments.
How many times I tried to comment a dependency in a package.json file or a configuration...

In YAML, you can comment a line by just adding a hashtag (#) before the line content.
So simple !

# my comment

Anchors

YAML format also lets you avoid repeating data in your configuration files thanks to the anchors.
With this feature you can define an anchor (&) by setting its name before a value (of any type).
You can then use an alias (*) as value of another field (later in the same YAML file).

The anchors let you change a value used many times at a single point like from Docker image in the initial example:

# An anchor
from: &image "docker.io/bitnami/node:18"
...
builders:
  # An object
  - name: module-loader
    from: *image
...

This is even more useful for objects and arrays.

YAML anchors also let you extend and override an object for a new value by entering <<: before the alias.
Here is an example with a builder:

&base
from: "docker.io/bitnami/node:18"
workdir: /app
builders:
  - <<: *base
    name: module-loader # extension to add the name
    workdir: /tmp/module # override the workdir
    adds:
      - package.json
      - package-lock.json
    script:
      - npm i --production --cache /tmp/cache
    caches:
      - /tmp/cache
    builders: # override the build to null to avoid circular references

This feature is also very useful but could lead to less readable files if not done wisely.

Conclusion

We have seen previously that the YAML format is fully compatible with the JSON one and that it has many additionnal features, but I wrote this article focusing on the human-friendly part of those languages.
This is not the only important aspect for a format.
The permissivity of the YAML format makes it more readable and easy to use (at least to me ^^), but it also can make it less efficient to process in a program...

To look for all the additionnal features see the full specification (at the current date).

Sources:

Are you ready to go green ?

Alienor — Thu, 16 Mar 2023 10:28:26 +0000

Do you consider the environmental impact of the applications you are developing?

The world becomes increasingly concerned about climate change, and most businesses are looking for ways to reduce their carbon footprint and promote sustainability.
However, I don't feel like there's massive changes in the tech game in order to switch habits and search for a more virtuous circle.
The energy consumption and carbon emissions of applications are significant, and developers have an important role to play in creating low carbon apps.

But do you really care ? Are you open to using alternative technologies or tools to reduce the carbon footprint of your applications?

Where are you in the tree ? (imagine the tree is "green technology")

Wherever you see yourself in that tree, I would love to know why... What's on your mind ?

Do your data smell gas ?!

Alienor — Tue, 06 Dec 2022 10:11:19 +0000

Digital sovereignty called into question

With the implementation of the GDPR in 2018, the European Union has equipped itself with strong legislation to protect the privacy of its citizens’ data. This advance has brought order to the abusive practices that led to the mass surveillance scandals by U.S. intelligence agencies.

Europe’s energy dependence on the scale

Europe’s hard-won digital sovereignty appears to be challenged by a shift in the balance of power. Indeed, the Russian-Ukrainian conflict has reshuffled the cards in the international distribution of energy resources. This new context has made it necessary for the EU to seek new sources of supply. The United States needed no less to seize this opportunity and renegotiate the issue of the sovereignty of European personal data.
Thus, the announcement of an energy alliance between Europe and the United States on March 25, 2022 was followed seven months later (on September 7) by another announcement on the regulatory framework for data transfer on both sides of the Atlantic. This is a resumption of the discussions since the European Court of Justice invalidated the Privacy Shield (July 16, 2020). It is difficult not to make the link between these two events to explain the loss of European ground on this highly sensitive issue.

A look back at a long-standing legal tug-of-war

The Safe Harbor agreement on data transfer between the EU and the United States, which dates back to the 2000s, was annulled in 2015 by the EU Court of Justice. The lack of data protection for European citizens was at issue in this decision.
A new agreement, the Privacy Shield, was drawn up in July 2016 and invalidated 4 years later for the same reasons. The implementation of the famous RGPD in 2018 appears to be a protective bulwark against the shameless exploitation of European data by American digital giants.

The persistence of American pressure

The Americans react quickly to this strong stance by the EU. Shortly afterwards, they put in place the Cloud Act (Clarifying Lawful Overseas Use of Data Act) to circumvent Article 5 of the GDPR. With this act, the American authorities assume the right to obtain data stored on American servers, regardless of their origin (thus including abroad).
The exploitation of personal data is the core business of Big Tech, which is not ready to give it up. However, they find themselves stuck with conflicting legislation in this showdown between the two political entities. And despite the dominant position of the American digital behemoths, the European pressure to impose heavy financial penalties or ban the use of data is still hovering.
Faced with the inability to reach an agreement on this issue and break the legal deadlock, Meta waved the threat of removing Facebook and Instagram from the European continent.

In the end,

The United States were able to deftly exploit the situation by asserting the best interests of its national security. The coincidence with a U.S. liquid gas delivery agreement leaves no one fooled by the maneuver. However, the decree still has to be submitted to the European Commission for approval, which is not likely to happen before the spring of 2023. The suspense remains.

Do's & don't's of writing Elixir

Alienor — Tue, 15 Nov 2022 08:18:33 +0000

In this article, we are going to explore do's and don't's of writing Elixir code. We will briefly learn the key challenges of switching from other languages to Elixir. For simplicity, we will write all the examples and code snippets using Python & Elixir and learn some common mistakes that programmers often do when switching to Elixir from a different programming language.

What is Elixir?

Elixir is a functional progamming language that runs on the powerful Erlang VM. Elixir takes full advantage of the VM and provides the perfect environment to the developer to build fault tolerant,low-latency, distributed systems.

According to stackoverflow's 2022 Developer Survey, it is the Second most loved Programming language of the Year.

The Elixir Community is growing as more and more people & companies adopting elixir to their code base.

The Key Challenges of Switching to Elixir

Unlike other functional programming languages such as Erlang, Clojure, etc. Elixir has a more beginner friendly syntax. This also opens the door to bring the baggage of other programming languages to Elixir.

For example, Python programmers with no previous functional programming experience might attempt to apply coding patterns of Object Oriented programming that are not suitable for Functional Programming. Even though, they might get away with writing codes that are more "Pythonic" than "Elixir". It will hinder their growth & productivity in the long term.

The lack of functional programming knowledge will also prevent them from taking advantage of various functional programming paradigms. It will also challenge their core concepts in at least three different layers:

The Functional way of thinking (i.e. immutability/streams/transformation/pipeline)
OTP way of thinking (i.e. concurrency, fault-tolerence, distributed system)
Patterns and Pipes.

So they will have to unlearn what they already know and think with a fresh perspective. This initial step is relatively hard but once, they master the concepts they will become productive in no time.

Now let's take a look at a few examples:

Mutable vs Immutable

In a Python environment, Variables or objects are mutable. Consider the following code snippet:

my_list = [1, 2, 3, 4]
do_something(my_list)
print(my_list)

Here, we can't reliably say, what will be the output of the print statement without looking at the source code of function do_something. In Python, Lists are passed as reference, so it is possible that the do_something function can alter the original my_list e.g. add something to the list or remove a thing or two. If we think in concurrency or parallel programming this could lead to unintended bugs. Two processes accessing the same object can get different values based on execution order or moment.

In Elixir, this is not an issue cause, variables are always immutable:

my_list = [1, 2, 3]
do_something(my_list)
IO.inspect(my_list)

Here the output of IO.inspect() is guaranteed to get the list [1, 2, 3]. In other words, the do_something() function can not modify the my_list variable. Even if we spawn hundreds or thousands of processes and access my_list all of those processes will get the same value of my_list. This is also true for complex data such as Nested List of Lists, Maps, or Database Records, etc.

Mastering the concepts of immutability is essential to write Good Elixir Code.

Nested Function Calls vs Piping

Nested function calls are common in most programming languages. When we want to pass the return value of a function as an agrument to another function the nesting occurs. it is more like a series of transformation to data. Elixir also allows nested function calls to transform data. Consider the below example:

decorate(bake(add_flavor(mix(ingredients))))

You will have to take some time to read the statement carefully and figure out which function is getting called in which sequence, what are their effects. And, how the data is getting transformed in each steps.

Fortunately, Elixir solves this problem elegantly by introducing a new operator called "Pipe" (|>). It takes the result of the expression to its left/top and inserts it as the first parameter of the function invocation to its right/bottom. Using the pipe operator we can rewrite the above example:

ingredients
|> mix()
|> add_flavor()
|> bake()
|> decorate()

Here the steps & sequence of function calls are clear. We are passing the ingredients to mix(). The result of mix() is getting piped to add_flavor(). Then we bake() the result and finally we decorate()

The Bottomline is, do not write Elixir codes with nested function calls use piping instead. It will make your code clear, concise and much more readable.

Functions & Pattern Matching

Pattern Matching is one of the most powerful feature of Elixir Programming Language. Mastering pattern matching techniques will boost productivty and improve code quality. The key is to know when and where we can apply pattern matching. Let's explore the below python function:


def flip(coin):
    if coin == "head":
        return "tail"
    elif coin == "tail":
        return "head"
    else:
        return "invalid"

This function simply flips a coin. If the coin is head it returns tail and vice versa. It also returns invalid for any other arguments. Now, if we want we can write it in a similar way in elixir:

def flip(coin) do
    if coin == "head" do
        "tail"
    else
        if coin == "tail" do
            "head"
        else
            "invalid"
        end
    end
end

This sure works, but it doesn't look good. First of all, it has a nested if...else block. Whenever you notice your code has nested blocks, take a few moment and think it thoroughtly. There must be a better way of doing it. In the above case, we do have a better way of achieving the same functionality. We can refactor this code with 3 simple lines of code:

def flip("head"), do: "tail"
def flip("tail"), do: "head"
def filp(_), do: "invalid"

Looks neat, isn't it? We are using pattern matching to match the argument directly from the function parameter. When the function flip will be called, Elixir will take each method signature and match it with the input argument. As soon as the match occurs, it will execute the function and stop. So for example, if we call the flip function with the parameter "tail". Elixir will first try to match the argument with the first function once, it fail it will move to the next one. Since, the second function takes "tail" as argument, it will get executed. We will get an output "head".

It will also work for complex Data Structure such as Dictionary or Maps. For example, see the below function that takes a dictionary person as argument and prints the full name.

def get_fullname(person):
    firstname = person.get("firstname") 
    lastname = person.get("lastname")
    return f"{firstname} {lastname}"

The dict person has two key value pairs, firstname and lastname. We first retrieve the values from the dict and store them in two variables and later we print them by concatenating them using f-strings.

In Elixir, we can do the pattern matching in the function signature like below:


def get_fullname(%{firstname: first_name, lastname: last_name}) do
    "#{first_name} #{last_name}"
end

just like dict Elixir has a data structure named Map which we are using for this example. The firstname and lastname are atoms, they are the key of the Map. The first_name and last_name variable will get the value of the Map using pattern matching. And, in the function body we just return the concatenated string.

The above examples are pretty basic but, they give us the glimpse of how pattern matching works. And, how it can help us to improve Elixir code.

Conclusion

Before calling it a day, let's quickly recap what we've explored so far. We learned the key challenges of switching from different programming language to Elixir. We also explored some common mistakes and how to avoid them by taking advantage of Elixir features and best practices.

Contribute to Lenra, a SaaS to create your crossplatform app, using any language

Alienor — Tue, 11 Oct 2022 14:15:27 +0000

Hello !

Check our website for more info www.lenra.io about our project :)

V / php / swift / java / C# / elixir / python .... We need contributors for our templates ! Feel free to help !

https://github.com/lenra-io/template-v

https://github.com/lenra-io/template-php

https://github.com/lenra-io/template-swift

https://github.com/lenra-io/template-java-springboot

https://github.com/lenra-io/template-csharp

https://github.com/lenra-io/template-elixir

https://github.com/lenra-io/template-python

Or if you're a NodeJS fan, you can help us with the migration of our data here :

https://github.com/lenra-io/Lenra/issues/143

Happy Hacking everyone !!

Not so green Meta

Alienor — Tue, 11 Oct 2022 14:12:36 +0000

Virtual worlds, a not so green Meta

We hear a lot about the metaverse, but the concept is still unclear to the general public. The company Meta, ex-Facebook, has launched a campaign “Impact will be real” to educate on the subject. The commercial shows a surgeon training on a virtual mannequin and a history student projected in ancient Greece.

The promise of the metaverse

With 3D, the metaverse facilitates the transmission of school and university knowledge, as well as the training of professionals. It allows creation of experiences without putting learners in dangerous situations, thanks to the projection in artificial environments.
Advocates of the metaverse also point the positive impact on the carbon footprint, due to the elimination of unnecessary travel. Users can work without commuting or attend a conference on the other side of the world without traveling.
For many, the metaverse is already shaping up the future of the Web. That alone is enough to justify the huge investments made by the Meta company, which wants to maintain a dominant position. Many other large companies are hoping in.
However, can we say that the metaverse is really virtuous? To what extent is it compatible with the current issues of responsible digital technology?

The real environmental impact of the metaverse

Today, the digital sector is responsible for 4% of global gas emissions. Servers, terminals, the use of rare minerals and the waste produced represent the other side of the coin.
As for the metaverse, it requires much more computing power than today (we’re talking about an order of magnitude 1,000 times more). This means more and more data centers to transmit these very high definition images in real time.
We must also take into account the renewal of equipment such as the purchase of virtual reality helmets and goggles, and more powerful computers, transferrring even more data on servers. The manufacture of these terminals requires the extraction, by a very toxic process, of rare minerals, mainly from China.

What is at stake in Green IT ?

The metaverse can be questioned with regard to the criteria of responsible digital technology, which concern accessibility, security, attention grabbing and digital sobriety.
In terms of accessibility, it is obvious that the situation will worsen. Indeed, the metaverse is likely to further widen the inequality gap in terms of equipment power and connection speed.
Security will probably also be put aside, under the pretext of offering the user an ever more fluid experience.
As far as capturing attention through design processes is concerned, it is safe to say that the metaverse will only intensify the problem. Its immersive power already makes the user captive. The quest for time, money and user data pushes companies to deploy more and more ethically objectionable processes. The metaverse will increase the effect of dark patterns, elements designed to induce behavior contrary to the user’s interest.
Finally, as far as digital sobriety is concerned, the aspects already mentioned concerning the cost of resources show that the gamble is not won.

In summary, the metaverse seems to go against the grain of the Digital Responsibility movement. However, considering the urgency of the environmental situation, the idea of imposing sobriety values on the metaverse cannot be without hope.

Hacktoberfest : contribute to Lenra, the app creation game-changing platform !!

Alienor — Mon, 03 Oct 2022 09:41:30 +0000

Lenra is an open source framework to create your app using any language, and deploy it without any Ops scale, built on ethical values.

Check our website www.lenra.io for more info !

We are thrilled to share our project during HacktoberFest, and hope you'll find a way to contribute to any part of the platform you want.

Data lover ? With the migration of data management, from postgres to mongoDB, we have some changes to make in the application to be compatible here :

https://github.com/lenra-io/Lenra/issues/143

Rust fan ? Create a dev interactive command in lenra_cli. The dev command builds the app, starts it and displays the app logs when you run it. It's here :

https://github.com/lenra-io/Lenra/issues/144

Go, Python, Ruby, Elixir, V ... As we create a platform usable in any langage, you can participate in developing any template here :

https://github.com/lenra-io/Lenra/issues/142

Hope you'll have fun !!

Here's our journey through data !

Alienor — Mon, 26 Sep 2022 12:28:37 +0000

How we manage data at Lenra

Data management is a very important part for every application, even for Lenra. Today, I'm going to introduce you to our data management needs and how we solve them. But first, I will introduce you to the type of data we need to manage at Lenra.

www.lenra.io is is made to accelerate the creation of your applications and simplify their hosting while preserving the personal data of your users.
Lenra is built around the values of digital sobriety and data protection, meaning users of its tools anchor their practice in a more responsible and ethical digital world.

Malware protection is a key point for our platform and has been respected through a process of improving the security of your data.

The platform allows GDPR compliance and also the control by the user of his personal data.

Data ?

A Lenra application is made of 2 components: widgets and listeners. The widgets are the graphical components, they are in JSON format and represent a description of the graphical interface. The listeners are the functions called in reaction to the user's actions on the graphical interface, such as pressing a button. All these components are described in a file called "manifest". It lists all the components that can be used within the application. In this part we will see the components as they were before the new data system was implemented. Schematically, here is how an application works

Widget

Widgets are therefore the graphical components of an application, they describe the graphical interface in JSON format. A bit like HTML, the developer will be able to embed "lenra-components", a library made by Lenra. The "lenra-components" library is a set of flutter widgets reworked to fit the needs of the company.

(data, props) => {
 return {
   "type": "text",
   "value": "Hello World!"
 }
}

As shown in this example we have a widget that is a function taking two parameters as input and returning some JSON corresponding to the interface. The two parameters are :

data : A JSON object representing the user data

props : A JSON object allowing the developer to give properties to the widget

This widget in JSON format will be validated and transformed into a Flutter widget using the JSON Schema. Once transformed, it will be displayed on the application. The widgets also have the possibility to be built according to the data, indeed the data of the application are passed to the widget so that it can use them in order to have a modular interface according to the data, for example:

(data, props) => {
 return {
   "type": "text",
   "value": data.text
 }
}

Listeners

The listeners are functions reacting to the user's actions and making modifications on the data.

(data, props, event) => {
  return {
    text: "world"
  }
}

As shown in the example, a listener is a function taking 3 parameters:

data : A JSON object containing the user's data.

props : A JSON object that allows to give properties to the listener.

event : A String that allows to specify to the listener the event that triggered its call.

What we need

The system we have just seen is the system that was in place on Lenra. It was quite functional and quite easy to use but it could very quickly present some limits. Indeed on a small application like a "hello world" this system is quite sufficient, but on a larger application and storing different types of data, having only one data object can quickly become restrictive especially for data querying. A new system has therefore been designed.
We had some specifications to respect for the new data system:

We wanted to allow all developers to manage their data schema as they wish, a bit like Firebase does, we wanted a system that would abstract the entire data system behind the Lenra interface and allow developers to create their data easily through HTTP requests.

We also wanted a system that is GDPR compliant by design, meaning that if the developer implements the obligatory listeners, he will have a GDPR compliant application by default.

What we do

Following these needs, we have therefore imagined a data system that is close to an object-oriented database such as SQL for example. So we have two types of objects: Datastore and Data. A datastore can correspond to a SQL table and a data to an entry of this table. We thus have a Datastore containing data in JSON format which is a mix between a SQL system and a data format similar to NoSQL. In order to link data between them, we have added a DataReference object which allows to make a many_to_many relation between two Data.

Database schema

Here is the database schema

Here we find the tables of the three objects mentioned above, datastores, datas and data_references, as well as the applications and environments tables mentioned in the previous point. The last table user_datas allows us to identify to which user a data belongs, we will come back more precisely on how it works in the part which is reserved to it.

API

Creation of an API that handles create/add/delete on our objects

scope "/app", LenraWeb do
    pipe_through([:api, :ensure_auth_app])
    post("/datastores", DatastoreController, :create)
    delete("/datastores/:_datastore", DatastoreController, :delete)
    get("/datastores/user/data/@me", DataController, :get_me)
    get("/datastores/:_datastore/data/:_id", DataController, :get)
    get("/datastores/:_datastore/data", DataController, :get_all)
    post("/datastores/:_datastore/data", DataController, :create)
    delete("/datastores/:_datastore/data/:_id", DataController, :delete)
    put("/datastores/:_datastore/data/:_id", DataController, :update)
    post("/query", DataController, :query)
  end

If you want to know more about how our controllers work, you can find it at: https://github.com/lenra-io/application-runner/tree/b4dac9754ae9a1925b5ab9a921ed5129bad7a624/lib/controllers

Query system

After implementing the new data system, we had to think of a way to query the data. The application data must be accessible from two different locations.
First, we need to be able to retrieve and perform actions from the listeners. Indeed, listeners do not receive an object containing all the data of a user for an environment but must be able to request this data with an API call.
Second, the data must be able to be queried from the widgets. As a reminder, widgets are written in JSON format, so we had to think about a JSON querying system. The listeners use this same format and will send a POST request on the /app/query route.

query format

VALUE: (STRING, NUMBER, BOOLEAN, OBJECT, ARRAY),
QUERY: { ...MATCH_BODY: { ( ...PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION }, | ...PROPERTY_CHECK)+ }, },
FIND_FUNCTION: { $find: { ...MATCH_BODY, _datastore: STRING } },
MATCH_BODY: { ( ...PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION }, | ...PROPERTY_CHECK)+ },
PROPERTY_CHECK: { STRING: BOOLEAN_MATCHING_FUNCTION },
BOOLEAN_MATCHING_FUNCTION: ( MATCH_MATCHING_FUNCTION | EQ_MATCHING_FUNCTION | AND_MATCHING_FUNCTION | OR_MATCHING_FUNCTION | LT_MATCHING_FUNCTION | GT_MATCHING_FUNCTION | NOT_MATCHING_FUNCTION ),
BOOLEAN_MATCHING_FUNCTION_LIST: [ MATCH_BODY+ ]
EQ_MATCHING_FUNCTION: ( { $eq: VALUE } | VALUE ),
MATCH_MATCHING_FUNCTION:  { $match: MATCH_BODY },
AND_MATCHING_FUNCTION: { $and: BOOLEAN_MATCHING_FUNCTION_LIST },
OR_MATCHING_FUNCTION: { $or: BOOLEAN_MATCHING_FUNCTION_LIST },
NOT_MATCHING_FUNCTION: { $not: MATCH_BODY },
GT_MATCHING_FUNCTION: { $gt: NUMBER },
LT_MATCHING_FUNCTION: { $lt: NUMBER },

query parser

You can find our complete parser here: https://github.com/lenra-io/query-parser/blob/2d3883e450826b35cfa09c1c80c24fc453dc1b71/lib/ast/parser.ex

Once we have defined the format of the requests as seen before, we had to implement a parser that will easily transform this JSON into an Ecto request. To do so, we proceeded by steps, the first one being to transform the JSON into a format that can be easily processed by Elixir. To do this, we chose to make an AST tree, the AST tree corresponding to the requests follows this pattern:

the pattern matching system provided by Elixir allows us to easily identify the patterns present in the JSON object in order to build step by step the ast tree with the corresponding object.
Our parser is composed of two main functions parse_expr and parse_fun, here are the declinations of each function:

In the case where we have a list as input, it is actually a $and:

defp parse_expr(clauses, ctx) when is_map(clauses) do
    parse_expr({"$and", Map.to_list(clauses)}, ctx)
end

A key that starts with $ is a function:

defp parse_expr({"$" <> _ = k, val}, ctx) do
  parse_fun({k, val}, ctx)
end

A simple k => v clause:

defp parse_expr({k, v}, ctx) do
  ctx = Map.merge(ctx, %{left: from_k(k, ctx)})
  parse_expr(v, ctx)
end

If there is a left in context, and is not a function, this is a simplified $eq function:

defp parse_expr(value, %{left: _} = ctx) do
  parse_expr({"$eq", value}, ctx)
end

Final parse_expr create Value object:

defp parse_expr(clauses, ctx) when is_list(clauses) do
  %ArrayValue{values: Enum.map(clauses, &parse_expr(&1, ctx))}
end
defp parse_expr(value, _ctx) when is_bitstring(value) do
  %StringValue{value: value}
end

defp parse_expr(value, _ctx) when is_number(value) do
    %NumberValue{value: value}
end

Parse function and with only one clause

defp parse_fun({"$and", [clause]}, ctx) do
    parse_expr(clause, ctx)
end

Parse function and with many clauses

defp parse_fun({"$and", clauses}, ctx) when is_list(clauses) do
    %And{clauses: Enum.map(clauses, &parse_expr(&1, ctx))}
end

Parse function eq

defp parse_fun({"$eq", val}, %{left: _} = ctx) do
    {left, ctx} = Map.pop(ctx, :left)
    %Eq{left: left, right: parse_expr(val, ctx)}
end

Using the same system of pattern matching with parse the AST tree into Ecto query that can be executed by our server

The limit

This system is functional and allows the developer to manage the data as he wants, only this system forces us to maintain a query system while Mongo already has a complete system.
So we can ask ourselves why keep this system and not just switch to Mongo, at the moment, the only answer is that we can't keep the data_reference but the developer can make the reference he wants by putting the data id in another data.
So we decided to rework the Postgres data management system to replace it with a system using Mongo.

Traefik middleware - Forward authentication

Alienor — Mon, 19 Sep 2022 13:38:28 +0000

In this article we will explain how to use Traefik middlewares and routers to manage authentication to many applications on Kubernetes.

Custom authentication

Custom auth delegates management to an external server.
To manage custom authentication we will use the ForwardAuth Middleware.

In this exemple, we will create a NodeJS Express server that persist authentication with a cookie containing a JWT token.

For every request passing through the middleware it will check the JWT token.

If it is not present or expired, it will send a login form to the client with a 401 status code.

We will use our previous secret basic-auth-users-secret to manage the accounts.

This is one kind of implementation, but you can imagine many authentication workflows.

We will not explain how to dockerize the NodeJS server, but you can find many tutorials.

Implement the server

The following environment variables need to be defined:

COOKIE: the name of the cookie that will persists the JWT
SECRET: The JWT secret to manage JWT
USERS_PATH: the path to the users file
VALIDITY: the authentication validity duration in seconds

In this example, the form values are sent in a HTTP header because the body of the request is not forwarded the server.

const express = require('express');
const router = express.Router();
const jwt = require('jsonwebtoken');
const atob = require('atob');
const htpasswd = require('htpasswd-js');
const cookie = process.env.COOKIE;
const key = process.env.SECRET;
const usersPath = process.env.USERS_PATH;
const validity = parseInt(process.env.VALIDITY);//in seconds

router.all('/', function (req, res, next) {
  let forwarded = {
    method: req.header("X-Forwarded-Method"),
    protocol: req.header("X-Forwarded-Proto"),
    host: req.header("X-Forwarded-Host"),
    uri: req.header("X-Forwarded-Uri"),
    ip: req.header("X-Forwarded-For"),
  };

  let url = `${forwarded.protocol}://${forwarded.host}${forwarded.uri}`;
  let roles = req.query.roles;

  try {
    if (cookie in req.cookies) {
      // Check JWT token
      let decoded = jwt.verify(req.cookies[cookie], key);
      res.sendStatus(200);
      return;
    }
    else if (forwarded.method.toUpperCase() == "POST") {
      // Check login/password
      let form = req.header("Form-Content");
      form = atob(form);
      let [login, password] = form.split(":");
      let loggedIn = htpasswd.authenticate({
        username: login,
        password,
        file: usersPath
      });
      if (!loggedIn) throw new Error("Not logged in");
      console.log(`Logged in ${login}. Redirect to ${url}`);
      // Create JWT token
      let val = (req.query.validity || validity) * 1000;
      let expire = val + Date.now();
      let token = jwt.sign({ user: login, exp: Math.floor(expire / 1000) }, key);
      res.cookie(cookie, token, { 
        secure: true,
        httpOnly: true,
        expire: new Date(expire)
      });
      res.status(302);
      res.header("Location", url);
      // redirect to initial url
      res.render('redirect', { title: 'Redirect', url });
      return;
    }
  }
  catch (er) { 
    console.error(er);
    res.clearCookie(cookie);
  }
  res.status(401);
  res.render('index', { title: 'Login', url , js: `function sendForm(e) {
    var form = e.currentTarget;
    var xhr = new XMLHttpRequest();
    var formData = new FormData(form);
    xhr.addEventListener('load', function(event) {
      window.location.reload();
    });
    xhr.addEventListener('error', function(event) {
      alert('Oups! Something went wrong.');
    });
    xhr.open('POST', '');
    xhr.setRequestHeader('Form-Content', btoa(form.login.value+':'+form.password.value));
    xhr.send(formData);

    e.preventDefault();
    e.stopPropagation();
    return false;
  }`});
});

module.exports = router;

Here is the index view:

extends layout

block content
  h1= "Login"
  form(method="post", onsubmit="return sendForm(event);")
    input(name="login")
    input(name="password", type="password")
    input(name="redirect", type="hidden", value= url)
    button= "Login"
  script= js

And here is the redirect view:

extends layout

block content
  h1= "Redirection en cours"
  script= `window.location = "${url}";`

Deploy the authentication middleware

First we will define a Secret containing the needed environment variables:

apiVersion: v1
kind: Secret
metadata:
  name: custom-auth-secret
stringData:
  # The cookie name
  COOKIE: my-cookie-name
  # The JWT 
  SECRET: The jwt secret key
  # The JWT and cookie validity in seconds
  VALIDITY: "1800"

Let's define the server Deployment and Service:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: custom-auth
  labels:
    k8s-app: custom-auth
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: custom-auth
  template:
    metadata:
      name: custom-auth
      labels:
        k8s-app: custom-auth
    spec:
      containers:
        - name: custom-auth
          image: custom-auth-server:latest
          env:
            - name: USERS_PATH
              value: /auth/users
          envFrom:
            - secretRef:
                name: custom-auth-secret
          volumeMounts:
            - name: users
              mountPath: "/auth"
              readOnly: true
      volumes:
      - name: users
        secret:
          secretName: basic-auth-users-secret
---
kind: Service
apiVersion: v1
metadata:
  name: custom-auth
  labels:
    k8s-app: custom-auth
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    k8s-app: custom-auth

Now we define the Traefik ForwardAuth Middleware in the same namespace of the server:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: custom-auth
spec:
  forwardAuth:
    address: http://auth.traefik:8080

It can now be used in Traefik routers:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: test-custom-auth
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`test-custom-auth.lenra.io`)
      middlewares:
        - name: test-custom-auth
          # Define the middleware namespace if you use is it another one
          # namespace: my-namespace
      services:
        - kind: Service
          name: my-service
          port: 8080

Go further

We can imagine many things to improve this authentication system:

pass the allowed users to the custom authentication server by defining it in the middleware address field
define the authentication cookie at the main domain in order to login only once
manage authentication with a third-part OAuth provider, like Github

Traefik middleware - Basic authentication

Alienor — Wed, 07 Sep 2022 13:38:50 +0000

In this article we will explain how to use Traefik middlewares and routers to manage authentication to many applications on Kubernetes.

Traefik authentication middleware
- Basic Authentication
- Deploy the authentication middleware
- Go further

Basic Authentication

Basic auth is a good way to restrict access to users you choose.

To manage Basic Authentication with Traefik, we will use the BasicAuth Middleware.

First, you need to define a secret with the Basic Authentication users at the format name:hashed-password with password hashed using MD5, SHA1, or BCrypt.

You can hash passwords using the following command: 'htpasswd -nb user password'.

apiVersion: v1
kind: Secret
metadata:
  name: basic-auth-users-secret
stringData:
  users: |>
    user1:$apr1$rKM7TQZQ$FUFmIklAwWBy80pHFcZlM0
    user2:$apr1$elAEjLXh$l5NtPMsZ5YmZhjKPJ5u4r/

and the middleware instance:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-basic-auth
spec:
  basicAuth:
    secret: basic-auth-users-secret

You can now use the middleware in Traefik routers:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: test-basic-auth
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`test-basic-auth.lenra.io`)
      middlewares:
        - name: test-basic-auth
      services:
        - kind: Service
          name: my-service
          port: 8080

Stay tuned, we will see soon how to set up the forward authentication middleware !

Introducing Dofigen !

Alienor — Tue, 16 Aug 2022 16:52:00 +0000

Glad to share with you our very first rust project, Dofigen !
It's a simple tool that generates dockerfiles from YAML (or JSON) structure.
Available on Github here

And it's opensource !! Enjoy !