DEV Community: Shin'ya Ueoka

Migrating node-fetch/form-data to Node.js native APIs

Shin'ya Ueoka — Sat, 27 Jul 2024 12:31:04 +0000

The release of Node v16.15.0 and v17.5.0 brought in the Fetch API and FormData. Before that, the node-fetch and form-data packages were used to use browser-like APIs in Node.js. To migrate node-fetch/form-data packages to Node.js native APIs, some changes can be made by removing import statements of 3rd-party packages, but there are important considerations to keep in mind when using Node.js features such as file system. This article outlines about migrating from node-fetch/form-data packages to Node.js native APIs.

Overview of migration

 import fs from 'node:fs';
-import FormData from 'form-data';
-import fetch from 'node-fetch';

-const file = fs.createReadStream('secret.txt');
+const file = await fs.openAsBlob('secret.txt', { type: 'text/plain' });
 const form = new FormData();
-form.append('file', file);
+form.append('file', file, 'secret.txt');

 const response = await fetch('https://example.com/upload', {
   method: 'POST',
   body: form,
 });

Details

Incompatibility of types

The form-data package and Node.js FormData are not compatible and cannot be passed directly with the Fetch API. If you pass form-data to the native Fetch API, the request body will be the string [object FormData].

import FormData from 'form-data';

const form = new FormData();
await fetch('https://example.com/upload', {
  method: 'POST',
  body: form,
});

// [object FormData]

The form-data package accepts fs.ReadStream in Node.js. The native API accepts a browser-compatible Blob. To create a Blob from a file, we can sue fs.openAsBlob(),added in Node.js v19.8.0. If you pass fs.ReadStream to the native API's FormData, the value will be [object Object].

import fs from 'node:fs';

const file = fs.createReadStream('secret.txt');
const form = new FormData();
form.append('file', file);

// Content-Disposition: form-data; name="file"
// 
// [object Object]

Incompatibility of internal behavior

The form-data package obtains the file name from fs.ReadStream and uses it as the filename field (internal implementation). It also determines the file type from the file name and sets the Content-Type header (internal implementation).

Blob does not have a file name, it needs to be specified manually. The native API's FormData defaults to application/octet-stream if the Content-Type is not specified. We can specify the Content-Type explicitly from the argument of fs.openAsBlob().

const file = await fs.openAsBlob('secret.txt');
const form = new FormData();
form.append('file', file);

// Content-Disposition: form-data; name="file"; filename="blob"
// Content-Type: application/octet-stream

const file = await fs.openAsBlob('secret.txt', { type: 'text/plain' });
const form = new FormData();
form.append('file', file, 'secret.txt');

// Content-Disposition: form-data; name="file"; filename="secret.txt"
// Content-Type: text/plain

Conclusion

Fetch API and FormData have become Stability: 2 (Stable) from Node.js v21.0.0. fs.openAsBlob() is still Stability: 1 (Experimental) in the latest version of Node.js (v22.5.1 as of at July 27, 2024). We need to take care when running in old runtimes or using experimental APIs which has potential feature changes.

Vimmatic - borwsing web pages with Vim-like key binds

Shin'ya Ueoka — Mon, 10 Apr 2023 14:02:14 +0000

Using a browser is a part of life these days. Most Vim users have wanted to navigate a browser by Vim-like key binds. Today, I will introduce a new browser extension, Vimmatic, which enables you to use your browser with Vim-like key binds.

Install

Vimmatic supports both Firefox and Google Chrome. You can get Firefox add-on or Chrome Extensions from the following:

Firefox	Chrome
Get an add-on	Get an extension (experimental support)

Note that the Firefox requires an permission to run add-on on certain page. To run the add-on on any pages, ensure that "Access your data for all websites" is enabled on permissions tab on add-on settings.

Quick start

Vimmatic provides Vim-like key binds. To scroll a page in the browser, press k, j, h and l keys. You can scroll to the top or the bottom of a page by gg and G.

To select a left and right of current tab, use K and J respectively. To close current tab, use d and to restore closed tabs, use u.

See also Keymaps on official document for more detailed of keymaps.

Follow mode

To open a link, press f to enter the follow mode to select a link. Then you can select links by alphabetic keys.

Using command line

Vimmatic supports command line to run commands that control tabs and opens a tab. To open command line, press :.

To open a tab with URL, use open command as the following:

:open https://github.com/ueokande/vimmatic

or search keywords with search engine (such as Google) like:

:open How to use Vim

You can see completed commands on Console commands.

Install your chrome extensions and firefox add-ons to the browser with playwright

Shin'ya Ueoka — Sun, 21 Aug 2022 13:36:44 +0000

Playwright is one of the most remarkable frameworks to achieve browser automation. It officially supports running Chromium, Firefox, and WebKit. How can we install chrome extensions or firefox add-ons to the browser on playwright? Playground introduces a practice for chromium on the document, but firefox is not.

Playwright-webextext stands to enable the installation of chrome extensions and firefox add-ons to the browser. It extends playground APIs and test fixtures that allow you to use chrome extensions or firefox add-ons from your local file.

Install

Playwright-webextext is distributed on npm. You can install it by npm:



$ npm install --save playwright-webextext

Or install with yarn:



$ yarn add playwright-webextext

Usage

withExtension()

This method extends playwright's core APIs. The method returns a custom BrowserType to load temporary extensions or add-ons from the local filesystem.



import { firefox } from "playwright";
import { withExtension } from "playwright-webextext";

(async () => {
  const browserTypeWithExtension = withExtension(
    // base browser type
    firefox,
    // path to a directory containing manifest.json
    "path/to/your/extensions",
  );
  // launch a browser
  const browser = await browserTypeWithExtension.launch();

  const page = await browser.newPage();
  await page.goto("https://example.com/");
})();

Note that using chrome extensions has two limitations: 1) the browser should run on headed mode, and 2) you should launch the browser with a persistent context.



const browser = await browserTypeWithExtension.launchPersistentContext("", {

  headless: false,

});

createFixture()

This method provides custom fixtures to run tests with a custom BrowserType. The method prepares a browser to install chrome extensions or firefox add-ons before tests start.



import { createFixture } from "playwright-webextext";

const { test, expect } = createFixture("path/to/your/extensions");

test("should launch browser with extensions", async ({ page }) => {

  await page.goto("https://example.com/");

// ...

});

How does it work

Chromium

Chromium supports installing extensions via command-line options. Playwright-webextext launches a chromium browser with command-line options with extension paths. See the following document for more details:

Chrome Extensions | Playwright

Firefox

Firefox provides a remote debugging server to control the browser via a remote debugging protocol. This protocol also enables installing a temporary add-on. Playwright-webextext installs add-ons by this protocol.

Building a monolithic application with Go and React

Shin'ya Ueoka — Sun, 14 Nov 2021 06:11:45 +0000

The ways and practices of building web services are more and more diversified today due to the business requirements, reliability, and scalability. The microservices or micro-frontend are well-known practices to divide a huge service and huge organization into self-organized teams so that they can maintain their services themself. That not only achieve to increase your development cycle but also makes the service more sustainable. Although these technics have a position with the future on scaling the service, complex architecture brings on complex development and deployment. Sometimes they are overmuch for the non-critical services for your business like the internal services in the company.

This article lets you build a monolith service using Go and React and introduces the boilerplate I published.

Monolithic Go and React Application Boilerplate

ueokande / go-react-boilerplate

Boilerplate for building a monolighic Go and React application

Monolithic Go and React Application Boilerplate

This repository provides a simple and monolithic service with a server written in Go and frontend with React. The service does not orient the modern services (like microservices or micro frontends), but you can see the simplicity and easiness of development and deployment. Sometimes monolithic services are helpful for a small-scale or non-critical business situation such as an internal system.

This project introduce the following languages and frameworks:

Server-side
- Golang
- gorilla/mux
Frontend
- TypeScript
- React
- react-router
- axios
- MUI

The application serves a tiny social blog as a sample. The users can publish the articles and add comments. Feel free to customize and build your applications based on this project.

Quickstart (for development)

The repository contains two projects; for the frontend and server-side. The frontend project is in frontend sub-project, you can install dependencies and run the debug server by yarn:

$ cd frontend
$

…

View on GitHub

The project introduce the following languages and frameworks:

Server-side
- Golang
- gorilla/mux
Frontend
- TypeScript
- React
- react-router
- axios
- MUI

The application serves a tiny social blog as a sample. The users can publish the articles and add comments. Feel free to customize and build your applications based on this project.

Quickstart (for development)

The repository contains two projects; for the frontend and server-side. The frontend project is in frontend sub-project, you can install dependencies and run the debug server by yarn:



$ cd frontend
$ yarn install && yarn start

Then run server-side service in debug mode by go run:



$ go run main.go  -debug

You can see the sample application on http://localhost:8000

Sample application: A minimal social blog

The application provides a minimal social blog as a sample. You can retrieve the articles and publish your article via your browser. You are also able to send your comment to the articles.

Any frontend page consists of a single page known as a single-page application; the server responds with a single HTML. The pages transit seamlessly. The frontend scripts fetch and send articles and comments via APIs asynchronously using axios. The server-side application provides the following REST APIs:

GET /api/health and GET /api/ready: The heath check endpoints to ensure the application lives. You can see more detail in Kubernetes docs.
GET /api/articles: Get all articles with summaries.
GET /api/articles/{article_id}: Get a article with full content.
POST /api/articles: Create a new article.
GET /api/articles/{article_id}/comments: Get comments of the article article_id.
POST /api/articles/{article_id}/comments: Create a new comment for the article_id article_id.

How to develop it

The server-side go service has a debug mode to serve the webpack development server on the same endpoint with APIs. This is useful to make the frontend scripts able to access APIs with no CORS headers. Any requests excluding the path starting with /api/ returns assets served from webpack

Server-side

The server-side program has two packages, repository and web. The repository packages contain interfaces and implementation to read and persist user's requests. Note that the current implementation server never keeps data. The server loses the article you published or the comments you sent when the restart. If you wish to persist them, try to implement repositories instead of mock storing to the in-memory.

The web packages provide the routes of the request from user's URLs and present responses for the request. They are known as the controller layer in MVC or DDD. If you want to add some APIs or add models, implement them to repository and web, respectively. If you need more complex logic or use cases, feel free to implement or add new packages.

Frontend

Any page consists of a single page as known as a single-page application. The react-router is a library to achieve client-side routing.

The frontend sub-project is initialized by react-scripts and keeps the directory structure and build processes. Every component is in frontend/src in flatten.

How to deploy it to the production

Manual deploy

You should not use a webpack development server by yarn start and go run in production. You need to build them before deployment. To generate frontend assets, run yarn build. This command generates minified HTML, JavaScript, and CSS.



$ (cd frontend && yarn install && yarn build)

To compile a server-side application to create an executable binary, use go build:



$ CGO_ENABLED=0 go build -o go-react-boilerplate -trimpath .

Then you can confirm to run it by the following:



./go-react-boilerplate -webroot ./frontend/build

The server-side also serves asset files from the directory specified by -webroot, so you do not need some 3rd-party HTTP server like Apache HTTP server or NGINX.

Building a Docker container

The repository contains Dockerfile. Using docker containers allows you to develop applications faster and make deployment easier. The Dockerfile uses multi-stage builds; it contains steps to build frontend and server-side phases. You can build a docker image and run the container from the image by docker build and docker run:



$ docker build -t go-react-builderplate .
$ docker run --rm -p 8000:8000 go-react-builderplate

How to evict specific pods on the Kubernetes cluster

Shin'ya Ueoka — Sun, 24 Oct 2021 08:31:28 +0000

Introduction

Kubernetes is a very popular and the most widely used container orchestration. It has a sophisticated and flexible scheduling system to deploy your application on production. The administrators would use a kubectl CLI to manage the cluster and deployed pods. The kubectl CLI provides enough features to manage the cluster, but the developer can customize CLI with a kubectl plugin. This article will present what pod eviction is and introduce a kubectl plugin to do that, kubectl-evict.

Pod Eviction

Kubernetes provides pod eviction, which stops pods and removes them from the node. The pod eviction is usually used when the node is shutting down, or the cluster performs a rolling restart of nodes to minimize the downtime of the service. The kubectl CLI has a sub-command kubectl drain to evict pods from the specified node and, the command makes the node as unschedulable:



$ kubectl drain your-node

kubectl-evict: evicts specific pods

The kubectl CLI does not provide a feature to evict specific pods. I sometimes need this feature to safely remove pods from the node or test a PodDisruptionBudget. The kubectl-evict is a kubectl plugin that allows evicting certain pods from the node.

ueokande / kubectl-evict

A kubectl plugin to evict pods

Although there are already plugins to evict pods such as dwradcliffe/kubectl-evict and kubectl-evict-pod, the above plugin has a high compatibility interface with kubectl logs or kubectl exec and provides the following features:

Allows evicting pods by a label selector
Evicts pods created by Deployment or DaemonSet
Supports --dry-run and --grace-period flags

You can get the plugin by the following:



$ go install github.com/ueokande/kubectl-evict@latest

Usage

To evict a specific pod:



$ kubectl evict nginx-abcd-1234

It also allows evicting pods by label selector. The following shows to evict pods which has a label app=nginx:



$ kubectl evict -l app=nginx

Evict pods created by Deployment:



$ kubectl evict deployment/nginx

You can of course evict pods from the node:



$ kubectl evict node/worker-1

Note that kubectl evict does NOT make the node as unschedulable (cordon), so use kubectl cordon or kubectl drain if you need to do this.