In this article I will introduce you to Rete.js, a framework for creating node editors with customization and built-in features for processing. From its inception to its current state (version 2 beta), I will provide a brief history of the framework, highlighting its evolution and growth.
While this article won’t delve into detailed guides on how to use Rete.js, you can easily find them on the official website at retejs.org. Instead, you can leave your feedback in the comments section below or ask questions on our Discord server.
History
My interest in visual programming began back in 2017 when I came up with the idea of creating an tool for data processing through a graphical interface. At that time, I was already familiar with Blender’s node editor and Unreal Engine’s Blueprint, but I couldn’t find any ready-made solutions in JS that met my needs.
So, I decided to develop my own solution using d3.js. I started with a node editor graphical interface that was similar in capabilities to the aforementioned solutions. Then I focused on processing these schemes. My goal was to create a library that allowed me to create different types of nodes without worrying about anything other than the input data the node should receive and the output it should provide.
The d3-node-editor package was created to fulfill my needs for an interface that could create nodes of various types and offer data processing functionality. The demo is still available on Codepen. Soon, after it received a few stars on the repository, I noticed that it was gaining interest from others as well. This motivated me to shift my focus from my original project to the library, and I began to consider how it could be improved.
One of the first issues I tackled was how to make it customizable and flexible. It was important that new features could be added without disrupting the main codebase. This was especially crucial because breaking changes could cause issues for users who relied on the library in their projects. To solve this issue, I decided to implement an event-based architecture.
During the implementation of the new architecture, I decided to give it a shorter name that was available on NPM. That’s how the framework was born, complete with a package called rete. In addition, the plugin system was implemented that allows to split functionality into different packages and install them on demand. Over time, I also added the ability to render nodes using a variety of frameworks, including Angular, Vue.js, and React.js.
As the main functionality was implemented, the framework gained popularity with and without my involvement (thanks to those who mentioned it in articles or shared links). I dedicated time to fixing bugs and adding features, but this was mainly driven by motivation and external factors.
As a result, we have now reached a stage where the framework has a second major version in beta.
Disadvantages of v1
After several years since the first release I can summarize its disadvantages. Am I saying that the first version should not be used? Partly yes, that’s why I’d like to discuss it here. Is it worth to use the v1 while the v2 is still in beta? Absolutely, because the first version has been tested extensively by the community.
Time and skills
One of the unfortunate truths about open-source projects: libraries or frameworks are typically provided “as is”, especially if they are developed purely out of enthusiasm. The quality of these solutions depends directly on the time and skills invested in them. As a result, there is a possibility that your issue may remain unanswered, or that a bug may not be fixed for years. Therefore, it’s crucial to contribute to a project that you’re passionate about in right way.
Design
During the early stages, like the design phase, some unforeseen factors can have a significant impact on subsequent stages.
Let’s review the critical points:
- TypeScript: it wasn’t used from the beginning, since version 1.0.0. Support for static typing could be improved, but significant improvements may require breaking changes.
- Component abstraction: it enables the creation of nodes of varying types, simplifying the development process but also restricting developers. Although the import/export feature is built-in, this approach can cause confusion in distinguishing between nodes and components.
- Event-based architecture and plugin system: they have undoubtedly provided significant assistance in terms of flexibility. However, it can be quite challenging to debug. All events were concentrated in the core. Plugins can create their own events, leading to an overwhelming number of events that aren’t isolated. Also the connected plugins are just a list, where the order of connection is crucial.
- Engine: graph processing was previously uncharted territory. Initially, I had planned to only work with dataflow, but due to implementation limitations, recursion support was not possible. Eventually, the task plugin was developed, which partially solved the issue of creating control flow solutions on top of the existing engine.
Documentation
In the first versions I used GitHub Wiki for documentation and later switched to Read the Docs. However, I was still struggling with some limitations of Markdown, so I decided to create a website called rete.js.org.
Despite the site provided some documentation and examples, it didn’t cover everything as I discovered from questions on GitHub.
Package building
I’ve discovered a litany of pitfalls. Firstly, relying on the Rollup and hoping for seamless integration was naive. I have repeatedly encountered uninformative errors when attempting to use the plugins. Secondly, managing multiple packages while working with HMR is a significant challenge. Especially when using npm link, which can cause problems with dependency linking that require several hours of debugging. As a result, resolving these issues can take up a lot of time.
What has been done?
The development of the second version of the framework and its accompanying materials has taken many times more efforts than the first version. The codebase was completely rewritten from scratch, with a focus on full TypeScript support and flexibility in customization. The only thing that remained unchanged is the recognizable visual design.
Overall, my goal wasn’t to implement a lot of features. Flexibility and extensibility have a higher priority than a multitude of features that can be easily enabled with a flag like enableThisCoolFeature. Of course, it's great to have many features, but alongside this we would also end up with dozens of options, each requiring 5-10 more options to customize these features.
Architecture
The new version of the framework features an architecture that is TypeScript-first and more scalable. It includes two key components: an alternative to event-based architecture and a cascading plugin system. In short, plugins can be connected not only to the editor instance but also to other plugins, resembling a cascade. This enables data, also known as signals, to be transmitted from the parent plugin to all child plugins, where they can be transformed or prevented.
For further details, please refer to the Plugin system page.
Presets
The key innovation in this version is the implementation of presets. These are a set of ready-made features that can be used by default or replaced with an alternative preset. Instead of using flags such as enableThisCoolFeature, presets can be added without the need to modify the plugin's source code.
For more information, please refer to the Presets section.
Engine
The Engine is now implemented in a separate package called rete-engine. It is designed to process schemes using dataflow and control flow approaches, as well as their combinations. This implementation is more flexible than the one in the previous version and is easier to understand from a theoretical perspective regarding the different problems these approaches solve.
Learn more on the Engine article.
Rete CLI
This tool has existed since the first version, but significant improvements have been made in the new version, including the resolution of some issues. This is particularly useful when a project is divided into different packages and repositories. The tool now supports TypeScript by default, along with linting and test runner. The issue with polyfill integration has been resolved, whereby polyfills were excluded from bundles in order to reduce their size, causing errors in certain environments where they were not automatically included, such as regeneratorRuntime is not defined. This issue has now been resolved using @babel/runtime.
For more details, please refer to the Rete CLI documentation.
Rete Kit
The idea of creating this tool did not come up immediately, but it proved to be extremely useful for both testing purposes and for familiarizing other developers with the project on their preferred stack, such as Angular, React.js or Vue.js.
This tool does several things: it creates a plugin base (which was done by the previous version of Rete CLI), creates an application using Rete.js, and performs bulk build during development. You can read more about this in the Rete Kit article. Here I just want to highlight the creation of an application for different stacks with the necessary features, which proved to be an extremely useful functionality. Using only one command rete-kit app --next you can get a working application with an editor.
Check out details on Rete Kit
Rete QA
Finally, another tool has been added to the framework in order to maintain quality without spending too much time. This tool is presented as a separate package called rete-qa and is responsible for performing regression UI tests on different stacks and browsers. The first aspect is achieved by Rete Kit, while the second is accomplished through Playwright under the hood, which runs all tests for each stack on Chromium, WebKit, and Firefox.
More details can be found in the Quality Assurance article.
Documentation and examples
This time, a lot of effort was put into developing documentation, examples, and the website itself. The website was migrated to Nuxt 3 with the use of accompanying modules, which unfortunately had an impact on the release time.
Many articles were added, which provide an overview of the project ecosystem and various formalities, as well as a FAQ page and guides for different parts of the functionality, along with links to examples.
Currently, there are 32 examples, most of which are hosted on Codesandbox.
Additionally, Algolia’s DocSearch has been integrated for improved search experience.
Conclusion
The story of Rete.js began several years ago when customizable JavaScript solutions were hard to come by. Currently, Rete.js continues to evolve and has significantly improved, although it is still in beta. It is more flexible and extensible than its previous version. With a more careful design and the creation of testing tools, as well as a focus on supporting TypeScript, Rete.js has become an even more reliable and flexible tool for creating node editors.
Top comments (0)