DEV Community: Vitaliy Stoliarov

Rete.js 2 stable is now available!

Vitaliy Stoliarov — Wed, 12 Jul 2023 22:44:40 +0000

After months of active development, it's finally here! ⌛
Upgrade now and enjoy the stability without any breaking changes

Check out the intro and examples on the website https://retejs.org

Code generation: transform JavaScript into visual programming language and back

Vitaliy Stoliarov — Tue, 27 Jun 2023 21:32:09 +0000

I would like to introduce to you Rete Studio, a general-purpose code generation tool that seamlessly bridges the gap between textual and visual programming languages

The implementation of this solution proved to be more challenging than expected. The primary focus was finding the scalable method to convert the Abstract Syntax Tree (AST) of the programming language (JavaScript was used as the foundation) into a user-friendly node editor. Later on, the task became even more complex, requiring the conversion back into code

retejs / rete-studio

A general-purpose code generation tool powered by Rete.js

Rete Studio

Rete.js application

Introduction

Rete Studio is a general-purpose code generation tool powered by Rete.js. Its primary goal is to seamlessly bridge the gap between textual and visual programming languages. With Rete Studio, you can transform a textual programming language into a visual representation, which can then be transformed back into textual language.

Key features

Playground: input your code and check its graph representation
Lab: debugging tool for code transformation into a graph and back
Editor: open your local project to edit code visually as a graph

Roadmap

Currently, JavaScript is the only supported language. Considering the complexity of the transformation process, our current priority is building a robust application. Once we have fine-tuned our methodologies and algorithms (including different programming languages), we will introduce them as a separate plugin for Rete.js.

If you believe in the potential of this project to revolutionize the way code…

View on GitHub

Visually programmed Chatbot powered by Rete.js 2

Vitaliy Stoliarov — Wed, 03 May 2023 21:22:41 +0000

This is a simple demonstration of a chatbot whose behavior is programmed using a visual editor.

In this implementation, all elements are customized, including animated connections. The chat window is presented as a separate node that can be moved only using special handles at the bottom and top. A dataflow and control flow-based engine is used to process the graph. There is also validation of connections by checking socket compatibility.

retejs / rete

JavaScript framework for visual programming and creating node editor #StandWithUkraine

Rete.js

JavaScript framework for visual programming

#StandWithUkraine 💛💙

#RussiaInvadedUkraine on 24 of February 2022, at 5.00 AM the armed forces of the Russian Federation attacked Ukraine. Please, Stand with Ukraine, stay tuned for updates on Ukraine’s official sources and channels in English and support Ukraine in its fight for freedom and democracy in Europe.

Help to defend Ukraine — donate to Ukraine’s main charity fund

Help to defend Ukraine — donate to the fund of the National Bank of Ukraine

Introduction

Rete is a modular framework for visual programming. Rete allows you to create node-based editor directly in the browser. You can define nodes and workers that allow users to create instructions for processing data in your editor without a single line of code.

Next version

The beta version of Rete.js 2 is available in the next branch

Documentation

Check the docs and learn about the components and capabilities.

…

View on GitHub

Sankey diagram, built using Rete.js 2

Vitaliy Stoliarov — Fri, 28 Apr 2023 13:43:15 +0000

This example showcases the versatility of the framework, which can be easily customized to accommodate a diverse range of graph or flow diagram types

retejs / rete

JavaScript framework for visual programming and creating node editor #StandWithUkraine

Rete.js

JavaScript framework for visual programming

#StandWithUkraine 💛💙

Help to defend Ukraine — donate to Ukraine’s main charity fund

Help to defend Ukraine — donate to the fund of the National Bank of Ukraine

Introduction

Next version

The beta version of Rete.js 2 is available in the next branch

Documentation

Check the docs and learn about the components and capabilities.

…

View on GitHub

Rete.js 2: visual programming for React.js, Angular and Vue.js

Vitaliy Stoliarov — Mon, 24 Apr 2023 18:56:40 +0000

In this article I will introduce you to Rete.js, a framework for creating node editors with customization and built-in features for processing. From its inception to its current state (version 2 beta), I will provide a brief history of the framework, highlighting its evolution and growth.

While this article won’t delve into detailed guides on how to use Rete.js, you can easily find them on the official website at retejs.org. Instead, you can leave your feedback in the comments section below or ask questions on our Discord server.

History

My interest in visual programming began back in 2017 when I came up with the idea of creating an tool for data processing through a graphical interface. At that time, I was already familiar with Blender’s node editor and Unreal Engine’s Blueprint, but I couldn’t find any ready-made solutions in JS that met my needs.

So, I decided to develop my own solution using d3.js. I started with a node editor graphical interface that was similar in capabilities to the aforementioned solutions. Then I focused on processing these schemes. My goal was to create a library that allowed me to create different types of nodes without worrying about anything other than the input data the node should receive and the output it should provide.

The d3-node-editor package was created to fulfill my needs for an interface that could create nodes of various types and offer data processing functionality. The demo is still available on Codepen. Soon, after it received a few stars on the repository, I noticed that it was gaining interest from others as well. This motivated me to shift my focus from my original project to the library, and I began to consider how it could be improved.

One of the first issues I tackled was how to make it customizable and flexible. It was important that new features could be added without disrupting the main codebase. This was especially crucial because breaking changes could cause issues for users who relied on the library in their projects. To solve this issue, I decided to implement an event-based architecture.

During the implementation of the new architecture, I decided to give it a shorter name that was available on NPM. That’s how the framework was born, complete with a package called rete. In addition, the plugin system was implemented that allows to split functionality into different packages and install them on demand. Over time, I also added the ability to render nodes using a variety of frameworks, including Angular, Vue.js, and React.js.

As the main functionality was implemented, the framework gained popularity with and without my involvement (thanks to those who mentioned it in articles or shared links). I dedicated time to fixing bugs and adding features, but this was mainly driven by motivation and external factors.

As a result, we have now reached a stage where the framework has a second major version in beta.

Disadvantages of v1

After several years since the first release I can summarize its disadvantages. Am I saying that the first version should not be used? Partly yes, that’s why I’d like to discuss it here. Is it worth to use the v1 while the v2 is still in beta? Absolutely, because the first version has been tested extensively by the community.

Time and skills

One of the unfortunate truths about open-source projects: libraries or frameworks are typically provided “as is”, especially if they are developed purely out of enthusiasm. The quality of these solutions depends directly on the time and skills invested in them. As a result, there is a possibility that your issue may remain unanswered, or that a bug may not be fixed for years. Therefore, it’s crucial to contribute to a project that you’re passionate about in right way.

Design

During the early stages, like the design phase, some unforeseen factors can have a significant impact on subsequent stages.

Let’s review the critical points:

TypeScript: it wasn’t used from the beginning, since version 1.0.0. Support for static typing could be improved, but significant improvements may require breaking changes.
Component abstraction: it enables the creation of nodes of varying types, simplifying the development process but also restricting developers. Although the import/export feature is built-in, this approach can cause confusion in distinguishing between nodes and components.
Event-based architecture and plugin system: they have undoubtedly provided significant assistance in terms of flexibility. However, it can be quite challenging to debug. All events were concentrated in the core. Plugins can create their own events, leading to an overwhelming number of events that aren’t isolated. Also the connected plugins are just a list, where the order of connection is crucial.
Engine: graph processing was previously uncharted territory. Initially, I had planned to only work with dataflow, but due to implementation limitations, recursion support was not possible. Eventually, the task plugin was developed, which partially solved the issue of creating control flow solutions on top of the existing engine.

Documentation

In the first versions I used GitHub Wiki for documentation and later switched to Read the Docs. However, I was still struggling with some limitations of Markdown, so I decided to create a website called rete.js.org.

Despite the site provided some documentation and examples, it didn’t cover everything as I discovered from questions on GitHub.

Package building

I’ve discovered a litany of pitfalls. Firstly, relying on the Rollup and hoping for seamless integration was naive. I have repeatedly encountered uninformative errors when attempting to use the plugins. Secondly, managing multiple packages while working with HMR is a significant challenge. Especially when using npm link, which can cause problems with dependency linking that require several hours of debugging. As a result, resolving these issues can take up a lot of time.

What has been done?

The development of the second version of the framework and its accompanying materials has taken many times more efforts than the first version. The codebase was completely rewritten from scratch, with a focus on full TypeScript support and flexibility in customization. The only thing that remained unchanged is the recognizable visual design.

Overall, my goal wasn’t to implement a lot of features. Flexibility and extensibility have a higher priority than a multitude of features that can be easily enabled with a flag like enableThisCoolFeature. Of course, it's great to have many features, but alongside this we would also end up with dozens of options, each requiring 5-10 more options to customize these features.

Architecture

The new version of the framework features an architecture that is TypeScript-first and more scalable. It includes two key components: an alternative to event-based architecture and a cascading plugin system. In short, plugins can be connected not only to the editor instance but also to other plugins, resembling a cascade. This enables data, also known as signals, to be transmitted from the parent plugin to all child plugins, where they can be transformed or prevented.

For further details, please refer to the Plugin system page.

Presets

The key innovation in this version is the implementation of presets. These are a set of ready-made features that can be used by default or replaced with an alternative preset. Instead of using flags such as enableThisCoolFeature, presets can be added without the need to modify the plugin's source code.

For more information, please refer to the Presets section.

Engine

The Engine is now implemented in a separate package called rete-engine. It is designed to process schemes using dataflow and control flow approaches, as well as their combinations. This implementation is more flexible than the one in the previous version and is easier to understand from a theoretical perspective regarding the different problems these approaches solve.

Learn more on the Engine article.

Rete CLI

This tool has existed since the first version, but significant improvements have been made in the new version, including the resolution of some issues. This is particularly useful when a project is divided into different packages and repositories. The tool now supports TypeScript by default, along with linting and test runner. The issue with polyfill integration has been resolved, whereby polyfills were excluded from bundles in order to reduce their size, causing errors in certain environments where they were not automatically included, such as regeneratorRuntime is not defined. This issue has now been resolved using @babel/runtime.

For more details, please refer to the Rete CLI documentation.

Rete Kit

The idea of creating this tool did not come up immediately, but it proved to be extremely useful for both testing purposes and for familiarizing other developers with the project on their preferred stack, such as Angular, React.js or Vue.js.

This tool does several things: it creates a plugin base (which was done by the previous version of Rete CLI), creates an application using Rete.js, and performs bulk build during development. You can read more about this in the Rete Kit article. Here I just want to highlight the creation of an application for different stacks with the necessary features, which proved to be an extremely useful functionality. Using only one command rete-kit app --next you can get a working application with an editor.

Check out details on Rete Kit

Rete QA

Finally, another tool has been added to the framework in order to maintain quality without spending too much time. This tool is presented as a separate package called rete-qa and is responsible for performing regression UI tests on different stacks and browsers. The first aspect is achieved by Rete Kit, while the second is accomplished through Playwright under the hood, which runs all tests for each stack on Chromium, WebKit, and Firefox.

More details can be found in the Quality Assurance article.

Documentation and examples

This time, a lot of effort was put into developing documentation, examples, and the website itself. The website was migrated to Nuxt 3 with the use of accompanying modules, which unfortunately had an impact on the release time.

Many articles were added, which provide an overview of the project ecosystem and various formalities, as well as a FAQ page and guides for different parts of the functionality, along with links to examples.

Currently, there are 32 examples, most of which are hosted on Codesandbox.

Additionally, Algolia’s DocSearch has been integrated for improved search experience.

Conclusion

The story of Rete.js began several years ago when customizable JavaScript solutions were hard to come by. Currently, Rete.js continues to evolve and has significantly improved, although it is still in beta. It is more flexible and extensible than its previous version. With a more careful design and the creation of testing tools, as well as a focus on supporting TypeScript, Rete.js has become an even more reliable and flexible tool for creating node editors.