Low-code drag-and-drop tool for building RESTful APIs with in minutes.

Romel Sikdar — Thu, 04 Jul 2024 17:48:49 +0000

Hello, everyone!

I am Romel Sikdar, a computer application student from India, who recently completed my master's degree at Narula Institute of Technology. When I was in eighth grade, Facebook was incredibly popular. Inspired by this, I dreamt of creating something simpler and better than Facebook 😂. It was an ambitious dream for a 13-14-year-old. At the time, I was new to coding, but I managed to implement major functions like authentication, user validation, and live messaging. Through this process, I discovered my passion for designing and developing backend systems rather than frontend.

As you might have guessed, I was still in school then. Due to my studies and final board exams for Secondary and Higher Secondary, I couldn't dedicate full time to my project—what I now call a childish dream 😂. Unfortunately, my hard drive failed, and I lost all the code I had written.

However, my journey didn't end there. During college, I completed several projects on web development and IoT. Given my interest in backend development, I often focused on creating REST APIs and backend logic. As a somewhat lazy person, I frequently wondered if there was an easier way to build and integrate backend logic and REST APIs without writing numerous lines of code.

After searching the web, I couldn't find anything that met my requirements. So, I decided to create a solution that would fulfill my needs and help others design and integrate backend logic and REST APIs without worrying about writing multiple lines of code.

I would like to introduce you to my project, which I have been working on since August 13, 2023, called EcoFlowJS.

⎆ Overview

EcoFlowJS is a powerful and user-friendly framework for creating, developing, and managing RESTful APIs within minutes. It's a flow-based, low-code, drag-and-drop visual programming system that requires minimal coding.

It features a simple interface for managing database connections and manipulating database records. This connection enables the creation and management of database records via RESTful API calls.

The application also provides a robust interface for managing users based on roles and permissions.

Additionally, EcoFlowJS includes an intuitive interface for creating, updating, and removing environment variables during runtime.

Configuring EcoFlowJS is straightforward, with an easy-to-use interface for setting up basic configurations, directory structures, API routing, CORS settings, and more.

EcoFlowJS allows for the creation and installation of custom module packages, enabling you to build and customize your RESTful APIs further. Documentation for creating and installing custom modules can be found here.

Note: A complete documentation can be found here

🚀 Features

🧩 Visual API Builder: Easily create backend logic and RESTful APIs by dragging and dropping nodes, connecting them based on the desired logic.
🗄️ Multiple Database Connections: Support for various databases simultaneously. Currently supported databases are MySQL, PostgreSQL, SQLite, and MongoDB. Support for other databases is possible via installation of external packages.
📊 Database Management: Easily monitor and manipulate database records using the provided database editor.
🔑 User Management: Role and permission-based user system.
🌐 Environment Variables: Update environment variables during runtime without restarting the application.
⚙️ Flexible Configuration: Manage all configurations, such as API Router, CORS, and Directories, from the admin panel without accessing configuration files.
- API Router configuration
- CORS configuration
- Directories configuration

Restart is required after setting configurations for the application to work properly.

📦 Package Management: Install and remove packages as needed.
🛠️ Custom Modules: Create and install custom modules for extended functionality.

📸 Snapshots

API Builder

Database Management

Environment Variables

Configuration

✨ Inspiration

During a college project in the field of IoT, I came across a simple and powerful solution for wiring together hardware devices called NODE-RED developed originally by IBM. The project was very simple as it only involved controlling electrical appliances and sensing room temperature using a temperature sensor. The whole hardware system was connected to the network using the MQTT Protocol, and using Node-Red, it was just a few minutes of work to connect all sensors and respond accordingly.

Below is the screenshot of the workflow of the project described above:

After some days, my sister, who was in class 2 then, came to me and showed me the first program she wrote. It was not a code-based program but a visual program using software called Scratch 3.0. It is similar to NODE-RED but with a different approach, focusing more on programming than wiring together hardware devices. It contains all the node blocks needed to build a simple program without any coding knowledge and is very user-friendly for children new to computer programming.

Below is a sample of the Scratch software:

Seeing these two programs, I thought of building something similar but with a different approach—a flow-based, low-code visual programming tool with all the minimum requirements needed for building backend logic and RESTful APIs. This would solve the problem of writing multiple lines of code. I also aimed to follow the approach of NODE-RED, allowing the installation of multiple external node packages so users can build their own packages and use them as needed for extended functionality.

📍 How EcoFlowJS Helps in backend

EcoFlowJS is particularly useful in backend due to its visual programming interface, which simplifies the process of integrating different backend logics and protocols, such as HTTP. By providing a flow-based environment similar to Node-Red, it allows for rapid prototyping and development, enabling even those with minimal programming knowledge to build complex backend systems.

For instance, you can create a workflow that reads database records, processes it, and sends back to the user, all within the visual interface. This can significantly reduce development time and complexity, making backend project implementation more efficient and accessible.

A detailed documentation can be found here

🚧 Challenges Faced

No project is without its challenges. Some key challenges I faced included:

Dividing the application sections based on roles and permissions: This was a critical challenge. Without proper role-based access control, all users would have full permissions, making it difficult to restrict actions. I overcame this by assigning each application section a unique access key, ensuring that only users with the appropriate key could perform certain actions.
Simultaneous implementation of multiple databases: Handling multiple database connections concurrently was challenging. To provide the ability to use multiple database connections for building backend RESTful APIs, I assigned a unique connection name to each database and accessed them using these names throughout the application.

Live updation of environment variables: The challenge was to update environment variables every time a user made changes. I solved this by creating a custom function that gets called whenever environment variables change. Below is an example of the function used:

import _ from "lodash";
import path from "path";
import fse from "fs-extra";
import dotenv from "dotenv";
import { homedir } from "os";

const loadEnvironments = () => {
  const { server, config } = ecoFlow;
  const envDir = _.isEmpty(config.get("envDir"))
    ? process.env.configDir ||
      homedir().replace(/\\/g, "/") + "/.ecoflow/environment"
    : fse.existsSync(config.get("envDir"))
    ? fse.lstatSync(config.get("envDir")).isDirectory()
      ? config.get("envDir")
      : process.env.configDir ||
        homedir().replace(/\\/g, "/") + "/.ecoflow/environment"
    : fse.ensureDirSync(config.get("envDir"));

  const ecosystemEnv = path.join(envDir!, "/ecoflow.environments.env");
  const userEnv = path.join(envDir!, "/user.environments.env");
  fse.ensureFileSync(ecosystemEnv);
  fse.ensureFileSync(userEnv);

  dotenv.config({ path: ecosystemEnv });
  dotenv.config({ path: userEnv });
};

export default loadEnvironments;

Flexible configuration of the application: Ensuring the configuration was simple and flexible was crucial. Users should be able to configure the application from the interface without accessing configuration files manually. This was achieved by updating the configuration file from the application itself, with a restart prompt to load the new configuration.
Building and installation of external packages: This challenge was divided into four major parts:
- Building of Nodes: Nodes were built with a JSON object containing their descriptions, allowing users to create custom nodes easily.
- Building of Packages: Packages were built with a section containing an object with the key as the name and value as the controller file in the package.json. A function returned a JSON object describing the packages and all its nodes.
- Publishing of Packages: Packages were published to the official npm registry for version management.
- Installation of Packages: Packages were verified to ensure they were valid for EcoFlowJS by checking for specific keywords.

Designing the Flow Editor: The flow editor is the heart of the project. Initially, I considered building it from scratch but found React Flow, which provided almost everything needed. Some custom implementations included:

on Connecting a Node :

  const onConnect = useCallback(
    (connections: Edge | Connection) => {
      if (connections.source === connections.target) return false;

      /**
       * Filters the edges array to find the target node IDs that match the given source node.
      * @param {Array} edges - The array of edges to filter.
      * @param {string} connections.source - The source node to match.
      * @returns An array of target node IDs that match the given source node.
      */
      const targetNodeIDs = edges
        .filter((edge) => edge.source === connections.source)
        .map((e) => e.target);

      /**
       * Checks if there are nodes with specific IDs and types in the given array of nodes.
      * @param {Array} nodes - The array of nodes to filter through.
      * @param {Array} targetNodeIDs - The array of target node IDs to check for.
      * @param {string} connections.target - The target connection ID to check against.
      * @returns {boolean} Returns false if the conditions are met, otherwise true.
      */
      if (
        nodes.filter(
          (node) =>
            targetNodeIDs.includes(node.id) && node.type === "Response"
        ).length > 0 &&
        nodes.filter(
          (node) => node.id === connections.target && node.type === "Response"
        ).length > 0
      )
        return false;

      /**
       * Updates the edges in the graph based on the existing connections and nodes.
      * @param {function} existingConnections - The existing connections in the graph.
      * @returns None
      */
      setEdges((existingConnections) => {
        const sourceNode = nodes.filter(
          (node) => node.id === connections.source
        )[0];
        const targetNode = nodes.filter(
          (node) => node.id === connections.target
        )[0];
        let updatedEdge = addEdge(
          {
            ...connections,
            animated:
              false || sourceNode.data.disabled || targetNode.data.disabled,
            data: { forcedDisabled: false },
          },
          existingConnections
        );

        /**
         * Finds the target node based on the connection target ID and filters the updated edges
        * to get the target IDs based on the source connection.
        * @param {Array} nodes - The array of nodes to search for the target node.
        * @param {string} connections.target - The ID of the target connection.
        * @param {Array} updatedEdge - The array of updated edges to filter.
        * @param {string} connections.source - The ID of the source connection.
        * @returns {Array} An array of target nodes and an array of target IDs.
        */
        const targetNodes = nodes.find(
          (node) => node.id === connections.target
        );
        const targetIds = updatedEdge
          .filter((e) => e.source === connections.source)
          .map((e) => e.target);

        /**
         * Checks if the targetNodes type is "Response" or if any of the nodes with targetIds
        * includes the id and has a type of "Response".
        * @param {object} targetNodes - The target nodes object to check.
        * @param {array} nodes - The array of nodes to filter through.
        * @param {array} targetIds - The array of target ids to check against.
        * @returns {boolean} True if the condition is met, false otherwise.
        */
        if (
          (targetNodes && targetNodes.type === "Response") ||
          nodes.filter(
            (node) => targetIds.includes(node.id) && node.type === "Response"
          ).length > 0
        )
          /**
           * Filters the edges in the updatedEdge array based on the source property matching the connections.source value.
          * For each matching edge, if the corresponding node is of type "Middleware", updates the edge with new properties.
          * @param {Array} updatedEdge - The array of edges to filter and update.
          * @param {Object} connections - The connections object containing the source property to match.
          * @param {Array} nodes - The array of nodes to search for the target node.
          * @returns None
          */
          updatedEdge
            .filter((edge) => edge.source === connections.source)
            .forEach((edge) => {
              const node = nodes.find((node) => node.id === edge.target);
              if (node && node.type === "Middleware")
                updatedEdge = updateEdge(
                  { ...edge, animated: true, data: { forcedDisabled: true } },
                  {
                    source: edge.source,
                    target: edge.target,
                    sourceHandle: edge.sourceHandle!,
                    targetHandle: edge.target!,
                  },
                  updatedEdge
                );
            });

        /**
         * Filters out duplicate edges from the given array of edges based on their 'id' property.
        * @param {Array} updatedEdge - The array of edges to filter.
        * @returns {Array} An array of unique edges with no duplicates based on their 'id' property.
        */
        return updatedEdge.filter(
          (edge, index, edges) =>
            edges.indexOf(edges.filter((e) => e.id === edge.id)[0]) === index
        );
      });
    },
    [nodes, edges]
  );

on Deleting a node:

  const onNodesDelete = useCallback(
    (nodeLists: Node[]) => {
      /**
       * Extracts the IDs of nodes from an array of node lists.
      * @param {Array} nodeLists - An array of node lists.
      * @returns {Array} An array of IDs of the nodes extracted from the node lists.
      */
      const deletedNodeIDs = nodeLists.map((node) => node.id);

      /**
       * Updates the node configurations by filtering out the configurations of deleted nodes.
      * @param {NodeConfiguration[]} nodeConfigurations - The array of node configurations to update.
      * @returns None
      */
      setNodeConfigurations((nodeConfigurations) =>
        nodeConfigurations.filter(
          (nodeConfiguration) =>
            !deletedNodeIDs.includes(nodeConfiguration.nodeID)
        )
      );
    },
    [nodes, nodeConfigurations]
  );

on Drag Over:

  const onDragOver = useCallback((event: DragEvent<HTMLDivElement>) => {
    event.preventDefault();
    event.dataTransfer.dropEffect = "move";
  }, []);

on Drop:

  const onDrop = useCallback(
    (event: DragEvent<HTMLDivElement>) => {
      event.preventDefault();

      /**
       * Parses the JSON data from the event data transfer and extracts the moduleID, type, label,
      * configured, and nodeDescription properties.
      * @param {string} event.dataTransfer.getData("application/ecoflow/nodes") - The JSON data to parse.
      * @returns An object containing moduleID, type, label, configured, and nodeDescription properties.
      */
      const { moduleID, type, label, configured, nodeDescription } =
        JSON.parse(event.dataTransfer.getData("application/ecoflow/nodes"));

      /**
       * Checks if the type is undefined or falsy, and returns early if it is.
      * @param {any} type - The type to check for undefined or falsy value.
      * @returns None
      */
      if (typeof type === "undefined" || !type) return;

      /**
       * Converts screen coordinates to flow coordinates using the reactFlowInstance.
      * @param {object} event - The event object containing clientX and clientY properties.
      * @returns The position object with x and y coordinates in flow space.
      */
      const position = reactFlowInstance.screenToFlowPosition({
        x: event.clientX,
        y: event.clientY,
      });

      /**
       * Generates a unique node ID.
      * @returns {string} A unique node ID.
      */
      const nodeID = generateNodeID();

      /**
       * Creates a new node with the specified properties.
      * @param {string} nodeID - The ID of the node.
      * @param {string} type - The type of the node.
      * @param {Position} position - The position of the node.
      * @param {string} moduleID - The ID of the module.
      * @param {string} label - The label of the node.
      * @param {boolean} configured - Indicates if the node is configured.
      * @param {string} nodeDescription - The description of the node.
      * @param {NodeAppearanceConfigurations} defaultNodeAppearance - The default appearance of the node.
      * @returns A new node with the specified properties.
      */
      const newNode: Node<FlowsNodeDataTypes & { nodeDescription?: string }> =
        {
          id: nodeID,
          type,
          position,
          data: {
            moduleID,
            label,
            configured,
            disabled: false,
            description: "",
            appearance: defaultNodeAppearance,
            nodeDescription: nodeDescription,
            openDrawer: (
              label: string,
              configured: boolean,
              disabled: boolean,
              description: string,
              appearance: NodeAppearanceConfigurations
            ) =>
              openConfigurationDrawer(
                nodeID,
                moduleID,
                label,
                configured,
                disabled,
                description,
                appearance
              ),
          },
        };

      /**
       * Concatenates a new node to the existing list of nodes using the setNodes function.
      * @param {Array} nds - The current list of nodes.
      * @param {any} newNode - The new node to be added to the list.
      * @returns None
      */
      setNodes((nds) => nds.concat(newNode));

      /**
       * Updates the node configurations by adding a new configuration object for a specific node ID.
      * @param {Function} setNodeConfigurations - The function to update the node configurations.
      * @param {string} nodeID - The ID of the node to add configuration for.
      * @returns None
      */
      setNodeConfigurations((configurations) =>
        configurations.concat([{ nodeID, configs: {} }])
      );
    },
    [reactFlowInstance]
  );

Configuration of API Router based the visual flow design: Ensuring conversion of graphical logic to routing structure was crucial. This was achieved by implementing a function which was developed by me to convert the graphical to routing structural logic containing all needed elements such as the methods, routes endpoints, parameters and controllers. Below is the code for the conversion logic function:

/**
 * Asynchronously generates route configurations based on the provided request stack and middleware stack.
* @param {RequestStack} requestStack - The stack of requests to generate routes for.
* @param {MiddlewareStack} middlewareStack - The stack of middleware to apply to the routes.
* @returns A promise that resolves to an array of tuples containing API method, request path, and Koa controller function.
*/
async function generateRoutesConfigs(
  requestStack: RequestStack,
  middlewareStack: MiddlewareStack
): Promise<[API_METHODS, string, (ctx: Context) => void][]> {
  let result: [API_METHODS, string, (ctx: Context) => void][] = [];
  this._isDuplicateRoutes = {};
  for await (const node of requestStack) {
    const { ecoModule } = ecoFlow;
    const { type, controller } = (await ecoModule.getNodes(
      node.data.moduleID._id
    ))!;
    const inputs = this._configurations.find(
      (configuration) => configuration.nodeID === node.id
    )?.configs;

    if (type !== "Request") continue;
    const [method, requestPath] = await this.buildRouterRequest(
      controller,
      inputs
    );

    const checkPath = `${method} ${requestPath}`;
    if (this._isDuplicateRoutes[checkPath]) {
      this._isDuplicateRoutes[checkPath].push(node.id);
      continue;
    }
    this._isDuplicateRoutes[checkPath] = [node.id];

    const koaController = await this.buildKoaController(
      middlewareStack.find((mStack) => mStack[0].id === node.id)?.[1]
    );
    result.push([method, requestPath, koaController]);
  }

  Object.keys(this._isDuplicateRoutes).forEach((key) => {
    if (this._isDuplicateRoutes[key].length === 1)
      delete this._isDuplicateRoutes[key];
  });

  if (Object.keys(this._isDuplicateRoutes).length > 0) {
    const routes = Object.keys(this._isDuplicateRoutes);
    const nodesID: string[] = [];
    routes.forEach((route) =>
      this._isDuplicateRoutes[route].forEach((nodeID) => nodesID.push(nodeID))
    );

    throw {
      msg: "Duplicate routes",
      routes,
      nodesID,
    };
  }

  return result;
}

Time Complexity of the API Router Responses: This was a significant challenge. Ensuring low time complexity is crucial for any application. I designed the system to have a time complexity close to O(1) for each response. This was achieved by implementing a custom controller calling wrapper that directly interacts with the actual controller of the application. Below is the code for the custom controller calling wrapper:

/**
 * Builds a Koa controller function with the given middlewares.
* @param {NodesStack} [middlewares=[]] - An array of middleware functions to be executed.
* @returns {Promise<void>} A Koa controller function that handles the middleware execution.
*/
async function buildKoaController(middlewares: NodesStack = []) {
  return async (ctx: Context) => {
    const { _ } = ecoFlow;
    const controllerResponse = Object.create({});
    const middlewareResponse = Object.create({});
    const concurrentMiddlewares: TPromise<Array<void>, void, void>[] =
      middlewares.map(
        (middleware) =>
          new TPromise<Array<void>, void, void>(async (resolve) => {
            const controllers = await buildUserControllers(
              middleware,
              this._configurations
            );
            let isNext = true;
            let lastControllerID: string | null = null;

            const ecoContext: EcoContext = {
              ...ctx,
              payload: { msg: (<any>ctx.request).body || {} },
              next() {
                isNext = true;
              },
            };

            for await (const controller of controllers) {
              const [id, type, datas, inputs, userControllers] = controller;
              if (_.has(controllerResponse, id)) continue;

              if (!isNext && type === "Middleware") {
                controllerResponse[id] = lastControllerID
                  ? controllerResponse[lastControllerID]
                  : (ecoContext.payload as never);
                lastControllerID = id;
                continue;
              }
              isNext = false;
              ecoContext.inputs = inputs;
              ecoContext.moduleDatas = datas;

              if (type === "Middleware")
                await middlewareController(
                  id,
                  ecoContext,
                  userControllers,
                  controllerResponse
                );

              if (type === "Response")
                await responseController(
                  ecoContext,
                  ctx,
                  userControllers,
                  middlewareResponse
                );

              if (type === "Debug")
                await debugController(
                  ecoContext,
                  lastControllerID
                    ? controllerResponse[lastControllerID]
                    : {},
                  userControllers
                );

              lastControllerID = id;
            }
            resolve();
          })
      );

    await Promise.all(concurrentMiddlewares);
    if (!_.isEmpty(middlewareResponse)) ctx.body = middlewareResponse;
  };
}

🛠️ Tech Stack

Commander: CLI interface tool for interacting with the application such as starting of the application, passing startup parameters, etc.
Koa: A NodeJs backend web framework for the development of the backend server, setting up the routes endpoint for the RESTful API, etc.
Passport: Authentication middleware used for user authentication to the application.
JWT: Secure transmission information between parties used by the application to validate the user authorization. This is also used to generate access-token and refresh-token for the user authorization.
Lodash: Utility library used in the application to makes working with arrays, numbers, objects, strings, etc. easier and time effective.
Knex: SQL query builder used in the application to manage sql databases queries in a easier way by letting knex handel all the hassle for us.
Mongoose: MongoDB object modeling in the application to manage mongodb queries and collections in a easier and hassle free way.
query-registry: npm registry API wrapper used in the application for querying, searching, installing, removal of packages and much more.
Socket.io: Real-time communication used by the application to provide live update of some feature and settings.
React: Frontend library for designing and building the frontend user interfaces.
React Flow: Interactive diagram builder used with in the application for designing and build of the backend and RESTful APIs logics.

🗺️ Future Roadmap

Implementation admin CLI commands
Enhancement of the default CLI commands
Integrate Socket.io as request and emitter nodes and much more
Implementation file manipulation operations
Implementation of updation of frontend projects and serve it within the application natively.
Add more official packages for providing extended functionality.
Create an official registry on top of npm registry to provide easy access to the packages available.
A drag-and-drop UI/UX generator for creating beautiful and responsive user interfaces.

📝 License

This project is MIT licensed.

🎭 Conclusion

Working on this project has been an incredible journey, filled with learning and growth. I am immensely proud of what we have achieved and look forward to the future developments. This is my first open source project I am taking part and the whole project was completely done by myself with the help from one of the junior, Soumya Paramanik, with the UI/UX design.

I am also thankful to my childhood teacher Subhojit Karmakar for guiding me throughout the process of build this project.

If you haven’t checked out EcoFlowJS yet, I invite you to give it a try and share your feedback. If you have ideas for new features, improvements, or bug fixes, feel free to open an issue or submit a pull request on GitHub. Thank you for joining me on this journey. Stay tuned for more updates and insights on my projects.

🔗 Useful Links

Project: EcoFlowJS

Repository: EcoFlowJS

Documentation:

Made with ❤️ by EcoFlowJS Team

DEV Community: Romel Sikdar