DEV Community: Abdul

How to build a Web IDE like codesandbox?

Abdul — Sun, 16 Jun 2024 08:24:06 +0000

Introduction

I first came across Web IDE’s during my college days through online competitive coding platforms like leetcode and hackerrank, and since then the concept of web based IDE’s or Editors has always intruded me and this being relatively very niche topic I could not find much resources on its workings. However by the end of my final year I came up with a prototype web IDE that lets you build react applications in the browser, the prototype had a file explorer, working terminal, code editor and a live preview window. Back then i was too embarrassed about my code quality and architecture that i did not post or open-source the project. Fast forward to today, I revisited the project again, laughed at my mistakes and ended up improving the IDE to my best knowledge and now i have decided to open-source it. If you find my content interesting, leave a 💖 and follow me for more.

Features:
Code editor (Monaco)
File explorer
Xterm.js terminal
LSP (Language server protocol) Support

Tech stack:
Node.js
Docker
React

Available Implementations
Before I explain my implementation of online code editor, I would like to mention few other available approaches/websites such as:

VS Code.dev: Core VS Code is open-source (other than extensions marketplace) and since it is built with web technologies like HTML, CSS and JS through electron framework, all most all of the editor front end is compatible to run inside a web browser, but for editor backend it makes use of remote servers or docker containers.

StackBlitz: Uses WASM and WebContainers to power their code playgrounds that run entirely inside a web browser.

My Build:

Since I wanted to build (or at least assemble) my own editor VS Code wasn’t the approach that i was interested in. The StackBlitz is cutting edge and a great piece of tech but there is limitation to amount of stuff that can be done through WASM.

So my approach was to take all the available open-source tools and stitched them together to run as a web editor.

FrontEnd: I used monaco and xterm.js as my editor and terminal respectively (VS Code also uses these). I wrote a small recursive file explorer component for viewing, creating and deleting files/folders. The entire editor is responsive thanks to split.js.

The frontend is build with react and deployed as a static site inside my S3 bucket.

The link to frontend repo

Backend: The backend is written in node.js and it runs inside a docker container. The backend is responsible for serving project files and writing content to the files and also spawning a terminal instance for the user (with non root privileges).

The link to backend repo

Enough talk show me code…!

For Editor there are many react monaco wrappers out there, but after trying out a few of them the best one I found is from typefox called monaco-language client. The great thing about this is, it seamlessly integrates with almost all vs-code syntax highlighting extensions and also it provides a way to integrate Language servers over web sockets (this is the coolest part, more on this in later section).

The coolest part of this repo is that it strips down and makes use of existing VS Code extensions for providing language specific syntax highlighting (check line no 1). Checkout Monarch if you want to build syntax highlighting by yourself.

Also here you can see the golang lsp configuration, but what is LSP ?

LSP stands for Language Server Protocol. It’s basically a special language that lets code editors and development environments (like Visual Studio Code) to talk to language servers.

These language servers are like mini programs that understand the ins and outs of a specific programming language. The editor can ask the server questions about your code, and the server can provide cool features like auto-completion, finding errors, auto importing packages, or jumping to where a function is defined.

Before Microsoft introduced the LSP protocol the language developers used to write language support plugins for each and every editor out there, but now thanks to this protocol the language server is implemented once and used everywhere.

When a language support extension is installed in VS Code, it basically gets the syntax highlighting and sets up the language server. The VS Code runs the installed language server as a process and communicates through Json-RPC calls. But for our web editor we cannot run these language servers in browser, so we run the language server inside the docker container and transfer all the JSON-RPC calls made by the editor to the language server via a web socket connection. You can think of this as a way of piping | the RPC calls over sockets.

{
    serverName: "GOPLS",
    pathName: "/gopls",
    serverPort: 80,
    runCommand: "gopls",
    runCommandArgs: ["serve"],
 }

Checkout the backend repo for complete code.

Let’s talk about the terminal

VS Code uses Xterm.js for its integrated terminal. Xterm.js is a terminal emulator with rich plugin/add-on ecosystem. Since our entire user workspace is inside a docker container and only the front end is being served through web, we need a way to attach our Xterm.js terminal emulator to a process that in turn interacts with pty on docker.

To achieve this I made use of node-pty, xterm.js attach add on. The node-pty spawns a child processes that can interact with the operating system like you would on a real terminal, but we need a way to transfer the commands and responses from web to backend running inside the docker container, for this I made use of attach add-on for xterm.js. This works similar to the LSP setup i.e, each keystroke in terminal (xterm.js) is transferred to backend via socket connection and then its fed into the node-pty process and response generated in turn is brought back.

Backend code for terminal:

This concludes the terminal section of the web IDE.

Additional features:

We have a file explorer and a terminal, since files or directories could also be modified via terminal we need a way to communicate those changes to frontend. For this I made use of Chokidar, this package lets us setup file system watcher (nodemon also makes use of chokidar) and through following callbacks we can communicate file system changes.

The above piece of code listens for new file additions and sends a socket event to frontend so that UI stays in sync even if the files are being created from the terminal or any other process.

Security and deployment:

Through a web IDE we are basically giving people on the internet access to free cloud compute and chances of malicious actors taking advantage of such system is high. Since the backend is in docker, we could easily limit the CPU and Memory usage of each container and instead of giving root access to the user we can create a low privileged user and spawn the terminal through this user.

Dockerfile:

Run the docker container with command

docker run -p 80:80 -d playground

This is just a fun little project, its far from being a perfect and secure code editor, if you find any issues please report them on github.

If you found this article helpful, leave a 💖 here or a ⭐ on github.

Follow me for interesting content

Socials:

https://www.linkedin.com/in/abdul-mohammad-a567b2183/

https://github.com/iam-abdul

Simplified OpenAPI Docs Using Serverless Framework for Node.js Developers

Abdul — Sun, 19 May 2024 18:58:10 +0000

Introduction:

OpenAPI formerly known as swagger docs is a specification. It's a standardised way to describe web APIs using a machine-readable format, typically YAML or JSON. OpenAPI docs could also be written in line as comments in javascript/typescript files. This format outlines the different endpoints, what kind of data they expect (request body), what kind of data they return (response), and other details that are crucial for developers and tools (eg: postman) to understand and interact with your API effectively.

Serverless framework lets you build and deploy applications without managing servers. You write code for specific functions that execute in response to events (like an API request). The cloud provider handles server provisioning, scaling, and maintenance, allowing you to focus on core functionality and benefit from pay-per-use pricing.

Setting up swagger/OpenAPI with serverless:

Install the following dependencies
npm i swagger-jsdoc with this dependency you will be able to extract swagger doc from comments in javascript files.

npm i swagger-ui-express this serves interactive swagger ui.

Following is a utility function that sets up the swagger configuration, the function accepts express app as the input

swagger docs extracts the api definitions from the .js files mentioned in the apis array.

Swagger needs the original URL to correctly generate the api documentation, but the api gateway (proxy) modifies the request URL and store a copy of original URL on the "x-forwarded-prefix" header and with this middleware the original URL gets assigned to req.originalUrl.

A sample swagger doc

Result

I had trouble setting up swagger docs with AWS and i hope this article helps you out.
Thank you for reading :)
Abd038

Socials:
https://www.linkedin.com/in/abdul-mohammad-a567b2183/
https://twitter.com/iam_abd038
https://github.com/iam-abdul

Vitest setup with serverless framework: Node.js Express.js

Abdul — Thu, 16 May 2024 04:26:25 +0000

While established testing frameworks like Jest and Mocha have long served the Node.js community, Vitest emerges as a compelling alternative, especially for projects that leverage TypeScript and modern tooling.

Vitest boasts a rich testing ecosystem out of the box, including features like test coverage and optional visual test explorers for enhanced development experience.

In this article i will walk over the setup for vitest with serverless framework for node and express applications.

The serverless.yml file remains almost unchanged, just add an exclude section under package to discard the test files when serverless build is getting created and deployed.

Vitest.config.json
Since the serverless framework encourages the use of config.stage.json files and environment files, i had slight trouble getting environment files to work with vite, so i came up with the above function that reads the .json as well as .env files and loads the environment variables.

The variable CONFIG_FILE is passed during the run time.

I hope you like this short article, thanks for reading :)

Abd038

Socials:

https://www.linkedin.com/in/abdul-mohammad-a567b2183/
https://twitter.com/iam_abd038
https://github.com/iam-abdul

I built an open-source schema visualisation tool for mongoose

Abdul — Wed, 15 May 2024 04:20:46 +0000

Working with Mongoose models can sometimes feel like keeping track of a complex web. Unlike many SQL databases, finding free and easy tools to visualize your Mongoose schemas can be tough. This can make it difficult to understand how your data is connected and collaborate with other developers.

To solve this problem, I built an open-source tool that visualizes your Mongoose schemas. Now you can easily see how your data fits together and keep your projects organised!

URL: http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/

Repo: https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool

NPM Package: https://www.npmjs.com/package/mongoose-parser

My Approach

While it might be tempting to directly query the database using the connection URI to extract model relationships, this approach has a few drawbacks:

Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials.
Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool.
To address these concerns, I opted to obtain the schema by parsing the JavaScript files. This approach has some key advantages:

Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe.
Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database.
Parsing the JavaScript files does require some additional logic, but the benefits in terms of security and efficiency make it a worthwhile trade-off. It’s important to note that this entire operation happens within your browser. Since it’s a static site, there are no API calls involved. You simply upload your JavaScript files, and the tool does its magic to visualize your Mongoose schema relationships, all without ever touching your database directly.

Parsing the Schema Files
Extracting model relationships from JavaScript files can be done using regular expressions. However, this approach has some limitations:

Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity.
Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run.
To address these limitations, I opted for a more robust approach: parsing the Abstract Syntax Tree of the JavaScript files.

Abstract syntax tree

What’s an AST?

An AST, or Abstract Syntax Tree, is a tree-like representation of the code’s structure. Imagine it like a family tree, but for your code! It breaks down the code into its core elements, like variables, functions, and expressions, making it easier to understand the relationships between different parts of the code.

Where are ASTs Used?

ASTs are a powerful tool used in various programming contexts beyond this schema visualisation tool. Here are a few examples:

Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyse and optimise the code before generating the final output.
Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it.

Benefits of Using an AST for Schema Visualisation
By converting the JavaScript files to ASTs, tool gains several advantages:

Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code.
Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles.
Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve.
While parsing with an AST requires a bit more initial setup compared to regular expressions, the benefits in terms of accuracy, flexibility, and maintainability make it a worthwhile investment for this tool. It allows for robust and reliable extraction of model relationships from your JavaScript schema files.

Since mongoose schemas are javascript files, I made use of acorn (https://www.npmjs.com/package/acorn) a javascript parser written in javascript 🙃 to build the AST.

Acron creates the AST for the given javascript files and from that i look for mongoose methods and extract the associated model definitions. I converted these set of functions into an
NPM package of its own (https://www.npmjs.com/package/mongoose-parser). This package is used by the tool for extracting the schemas.

Once the schema is extracted the frontend react app converts this schema into set of nodes and edges and creates the visual representation with the help of react-flow (https://www.npmjs.com/package/reactflow).

Usage

Just drag and drop the JavaScript files containing your Mongoose schema definitions here, and watch your schemas come to life! http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/

Thanks for reading :)

Socials

Linkedin: https://www.linkedin.com/in/abdul-mohammad-a567b2183/

Github: https://github.com/iam-abdul