DEV Community: Penn

how to retain position of markdown element in remark.js

Penn — Mon, 04 Sep 2023 06:42:53 +0000

remark.js is great tool that transforms markdown with plugins.

I usually combine remark-parse, remark-rehype and rehype-react to transform markdown into react components. The configuration of the processor is like:

import { unified } from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeReact from 'rehype-react'

const processor = unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeReact, {
    components: {
      //custom components here
    },
  })

The problem

I recently found that the position of the markdown elements is not available in the react scope. For example, if we have a markdown file like:

# Headline

The expected position prop should be passed to the corresponding react component like:

function Headline(props: { position: { line: { start: number; end: number } } }) {}

This is an important feature for the markdown editor. When the user scrolls the preview view, the editor view should scroll to the corresponding position. If the position of the origin markdown element is available, then the editor knows very well which line to scroll to.

The investigation

This first thing I asked myself to do is checking whether remark already knows the position. Luck me, the answer is yes.

I learnt there by inspect the mdast(markdown-abstract-syntax-tree). Run the following test.js and it will generate the description of the tree in json format.

import { unified } from 'unified'
import remarkParse from 'remark-parse'

const processor = unified().use(remarkParse)

const markdown = `# Headline`

console.log(JSON.stringify(processor.parse(markdown), null, 2))

And you will get the result as:

{
  "type": "root",
  "children": [
    {
      "type": "heading",
      "depth": 1,
      "children": [
        {
          "type": "text",
          "value": "Headline",
          "position": {
            "start": {
              "line": 1,
              "column": 3,
              "offset": 2
            },
            "end": {
              "line": 1,
              "column": 11,
              "offset": 10
            }
          }
        }
      ],
      "position": {
        "start": {
          "line": 1,
          "column": 1,
          "offset": 0
        },
        "end": {
          "line": 1,
          "column": 11,
          "offset": 10
        }
      }
    }
  ],
  "position": {
    "start": {
      "line": 1,
      "column": 1,
      "offset": 0
    },
    "end": {
      "line": 1,
      "column": 11,
      "offset": 10
    }
  }
}

As you can see, the position of the headline is

{
  "start": {
    "line": 1,
    "column": 1,
    "offset": 0
  },
  "end": {
    "line": 1,
    "column": 11,
    "offset": 10
  }
}

So which plugin remove the position object?

The plugin remark-rehype is the one that remove the position object. It generates hast(html-abstract-syntax-tree) from each mdast using specify handler. The default handlers don't expose position property to the hast. Here is the source code of the headline handler.

export function heading(state, node) {
  /** @type {Element} */
  const result = {
    type: 'element',
    tagName: 'h' + node.depth,
    properties: {},
    children: state.all(node),
  }
  state.patch(node, result)
  return state.applyData(node, result)
}

PS: source code

The solution

The answer to the solution is custom all the handlers of remark-rehype. The following code is the custom handler for the headline.

/**
 * @typedef {import('hast').Element} Element
 * @typedef {import('mdast').Heading} Heading
 * @typedef {import('../state.js').State} State
 */
import gatherPosition from './gather-position.js'
/**
 * Turn an mdast `heading` node into hast.
 *
 * @param {State} state
 *   Info passed around.
 * @param {Heading} node
 *   mdast node.
 * @returns {Element}
 *   hast node.
 */
function gatherPosition(node) {
  return {
    [`data-startline`]: node.position.start.line,
    [`data-startcolumn`]: node.position.start.column,
    [`data-startoffset`]: node.position.start.offset,
    [`data-endline`]: node.position.end.line,
    [`data-endcolumn`]: node.position.end.column,
    [`data-endoffset`]: node.position.end.offset,
  }
}

export function heading(state, node) {
  /** @type {Element} */
  const result = {
    type: 'element',
    tagName: 'h' + node.depth,
    properties: { ...gatherPosition(node) },
    children: state.all(node),
  }
  state.patch(node, result)
  return state.applyData(node, result)
}

The custom handler is almost the same as the default one. The only difference is that it adds the position object to the properties of the hast node.

Thanks for reading.

NextJS Authentication over Nginx

Penn — Thu, 27 Apr 2023 07:23:24 +0000

NextJS doens’t provided internal authencation support such as private root, redirection for unauthenticated request.

This article introduce a approuch to use Nginx to help fill the gaps.

Steps

Add an API to authenticate incoming request

/next-app/pages/api/auth.tsx



import { NextApiRequest, NextApiResponse } from 'next'
import isValid from 'token/isValid'

export default async function handler(req: NextApiRequest, res: NextApiResponse) {

  const token = req.cookies['access_token']
  // isValid is an function that check if the token is valid or not
  if (await isValid(token)) {
        res.status(200).json({ message: 'Hello from Next App!' })
  } else {
    res.status(401).json({ message: 'unauthorized' })
  }
}

PS: Please use the real isValid function for your project. 🙂

Protect private routes, redirect to signin page if unauthencated

Let’s say the Next App is serving on port 3030. Nginx listens incoming request on 3000 and make redirection per configuration.

nginx.conf



server {
  listen 3000;
  listen  [::]:3000;
  server_name localhost;
  large_client_header_buffers 4 12k;
  gzip  on;
  gzip_types application/javascript application/json text/css;

  # Place this before the next one in order to take the high porior.
  location ~* ^/_next/data {
    auth_request     /auth;
    auth_request_set $auth_status $upstream_status;
    error_page 401 =401 /401;
    proxy_pass              http://localhost:3030;
  }

  location ~* \.(gif|jpg|jpeg|svg|png|ico|woff|woff2|js|css|json|yaml)$ {
    expires 1y;
    add_header Cache-Control "public";
    proxy_pass              http://localhost:3030;
  }

  location /_next/static {
    expires 1y;
    add_header Cache-Control "public";
    proxy_pass              http://localhost:3030;
  }

  # public routes
  location ~* ^/(sign_in|other-public-routes)/?$ {
    proxy_pass              http://localhost:3030;
  }

  location /  {
    auth_request     /auth;
    auth_request_set $auth_status $upstream_status;
    error_page 401  =3=2 $schema$host/sign_in?redirect=$request_uri;
    proxy_pass              http://localhost:3030;
  }

  location = /auth {
    internal;
    proxy_pass              http://localhost:3030/api/auth;
    proxy_pass_request_body off;
    proxy_set_header        Content-Length "";
    proxy_set_header        X-Original-URI $request_uri;
  }

}

Note: Make sure the location ~* ^/_next/data take a high prior by placing it before the static asset location location ~* \.(gif|jpg|jpeg|svg|png|ico|woff|woff2|js|css|json|yaml)$ .

The incoming request will be redirected to /auth (actually /api/auth on Next App). When the response code is 401, a [302](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/302) redirection to /signin will be returned to the client. Then the user has to finish the signin process before actually visiting the protected pages/apis.

The sequence diagram would be:

Tips: use the online test to verify matched location with request uri.

Alternative when `$schema` or `$host` is transformed

Sometimes the $schema (either HTTPS or HTTP) or $host (e.g. www.notion.com) is not the same with the one that the user types in the address bar in the browser since the request was proxies by another gateway so the original host was replaced. In this case, using server-side redirect could resolve the problem.

nginx.conf



location /  {
    auth_request     /auth;
    auth_request_set $auth_status $upstream_status;
    error_page 401  /signin?redirect=$request_uri;
    proxy_pass              http://localhost:3030;
  }

Since it’s server-side redirection, the redirect param has to be handled on the server side as well.

Data Fetching: getServerSideProps | Next.js

Summary

Add authencation API in NextJS app.
Perform redirection with Nginx

Nginx plays the key role in the sample. And you may get it that the authencation API could be replaced by any third part API. And the flexibility is provided by Nginx.

How to Disable SSL & Authencation of Elasticsearch

Penn — Wed, 21 Dec 2022 06:45:09 +0000

disable `SSL` and user authencation

For local elasticsearch, sometimes we’d like to visit the instance without SSL(so that we can visit elastic on HTTP not HTTPS) and authencation. This article shows up how to diable the SSL and the authencation.

ps. The instance in the sample is hosted as a docker image and its version is 8.5.3 (https://hub.docker.com/_/elasticsearch), and we are going to use vscode as the editor to view and update the configuration of elasticsearch. Read more on attach to a running container

The configuration yaml file is at /usr/share/elasticsearch/config/elasticsearch.yml. Let’s open it to see what’s in it.

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 21-12-2022 05:50:49
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["1c17b10563ba"]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Let’s change some lines in the file as below.

Disable SSL

Set xpack.security.http.ssl.enabled as false

xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

Disable user authencation

This is no option to disable the authencation directly, instead, we are going to leverage the anonymous-access which perfect match our need. Add the follow section at the end of the elasticsearch.yml.

xpack.security.authc:
  anonymous:
    username: anonymous_user 
    roles: superuser
    authz_exception: true

It does make sense to use roles other than superuser (I know it’s a big risk but anyway it’s up to you).

Then restart the elasticsearch server to apply the changes. If no errors reported, you will be able to visit the endpint on http and no authorization needed. Have fun!

References

simluate `&&` and `||` by using Nodejs

Penn — Sat, 23 Jul 2022 09:32:00 +0000

In the last article I expound how to run two Nodejs Modules in one without shell. The secret is child process offered by Nodejs iteself. Today I will cover other two different types with it that is to simulate && and || as shell.

Found a great explaination of && and || about shell.

&& strings commands together. Successive commands only execute if preceding ones succeed.
Similarly, || will allow the successive command to execute if the preceding fails.

The signal that shell depends on to determ if the commands succeed ends or not is the exit code.

The `exit code`

The exit code is the code that returned from your program. In the world of NodeJS, we us process.exit to exit and set a proper code for exit. e.g.

// exit without error
process.exit(0)

// exit with errors
process.exit(1)

In general, 0 used as a success code, and others for anything else. We usually don't need to set the exit code explictly as the above script, NodeJS handles properly when the program exits.

Exit code of child process

The child process will emits exit event with the code as the data after it ends. And from the main process we can use the following script to inspect the code and make the following actions such as execute another script or not.

childprocess.on('exit', (code) => {
  // inspect the code and operate as "&&" or "||"
})

So here we go

const fork = require('child_process').fork

// Equivalent to &&
const c1 = fork('./child1.js')
c1.on('exit', (code) => {
  if (code === 0) {
    const c2 = fork('./child2.js', {
      detached: true
    })
  }
})

and

const fork = require('child_process').fork
// Equivalent to ||
const c1 = fork('./child1.js')
c1.on('exit', (code) => {
  if (code !== 0) {
    fork('./child2.js', {
      detached: true
    })
  }
})

Thanks for Reading.

References

https://shapeshed.com/unix-exit-codes/

How to run two nodejs modules in distroless docker image

Penn — Tue, 19 Jul 2022 05:15:37 +0000

Recently I tried to replace the Alpine docker image with Distroless to get more benifits. But I met an error said no shell available.
The CMD of the dockerfile is like:

CMD node dosomething.js & node server.js

In this form(aka shell-form), it invokes a command shell to run the commands(node dosomething.js & node server.js).

If you use the shell form of the CMD, then the will execute in /bin/sh -c

Since Distroless doesn't have the builtin shell the error happens.

"Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

I started from the idea that to move the code of dosomething.js into server.js and finally found it's too complicated since the server.js was generated from the source code and it's difficult to insert something and wish it still works.

Then it came to the second idea that is let a runner to start the dosomething.js and server.js as child process. Its code was like:

runner.js

const { fork } = require('child_process')
const path = require('path')

fork(
  path.resolve('dosomething.js'),
  null,
  {
  detached: true
  }
)

fork(
  path.resolve('server.js'),
  null,
  {
    detached: true,
  }
)

Fork will start a child process from the module(the first argument, e.g. path.resolve('server.js')) defined as the first argument.
The second argument is the list of the arguments for server.js. When the server.js accepts arguments to control its behavior then we may add something here. I make it as null since no extra arguments needs for this sample.
The last argument is the options and the detached:true indicates that the child process started from the module(e.g. server.js) run independently of its parent process. It has different behavior on differnt platform but in short you can think it makes the child process continue runing no matter the main dies or not. Suit my requirements.

So finally the CMD in docker file as:

CMD ["runner"]

So above approuch fix the issue. But only when we needs both dosomthing.js and server.js runs it works. What if I want to server.js starts only when dosomthing.jsends with no error or with any error. It could be easily implemented using command shell with the operator && and ||. I will cover this in the next post.

Thanks for reading.

Ways to load env variables for your script

Penn — Fri, 18 Jun 2021 09:23:00 +0000

Several years ago when I decided to write an article, I got this inject-env-variables-with-no-tears. At that time, I know nothing about makefile and shell script(almost), the only idea in my brain is just Nodejs. Years later, I got new and better solutions.

Firstly, let's talk about what env variables act in software like a project written in Nodejs. Let's say it's an app that is bootstrapped by create-react-app. By default, this app will be served on the root path of a host, like medium.com. If it's supposed to be served on a sub-path like /apps/my-app, you have to tell the build script what the exact path it is. The way it receives the path value is through the env variable, actually as PUBLIC_URL.
The reason why it(react-scripts) prefers the env variable not an inline argument is env variable is the most unified. Different operations may have different ways in commands, data format, etc. But they all have same basic support for the env variable. That means the great cross-platform. That's it.

Now, let talk about how to inject env variables into the project.

Shell Script

For example, you are going to set an env variable for the app.

PUBLIC_URL=/apps/my-app

Let's create a file named build.sh

#!/bin/bash
export PUBLIC_URL=/apps/my-app
/node_modules/.bin/react-scripts build

If there are multiple stacks for deployment, just created more build-[stack-name].sh, tell the build manager(like Jenkins) that executes the correct shell script.

Makefile

Makefile works like shell script plus tasks.

PATH := ./node_modules/.bin:${PATH}
.PHONY: start build
start:
    react-scripts start
build:
    export PUBLIC_URL=/apps/my-app; \
    react-scripts build

The start and build are tasks (called target by Makefile) defined by the Makefile. In the build target, the first line is to set the PUBLIC_URL. Please be noticed it ends with a semicolon followed by a backslash. This is because each line in a target is executed in a standalone shell by default, which means the env variable set in the first line is not readable by the second line. The sign that a semicolon followed by a backslash asks Makefile that the next line executes in the current shell, which means they share the same context. In this way, the second line receives the env variable.
There is another way to declare that the lines in the target share the same shell, one-shell. It is not available in some old implementations, such as mine. So I still use the old mark. If you have a new Makefile, try yourself if one-shell works.

Why not Nodejs

Compare to Nodejs(usually dot-env or set env manully by process.env[name] = value), shell script works more straightforward. And the great part is, shell script works for all unix like OS, not only Nodejs. In this point of view, it's sort of cross-platform.

DEV Community: Penn

how to retain position of markdown element in remark.js

The problem

The investigation

The solution

NextJS Authentication over Nginx

Steps

Add an API to authenticate incoming request

Protect private routes, redirect to signin page if unauthencated

Alternative when $schema or $host is transformed

Summary

How to Disable SSL & Authencation of Elasticsearch

disable SSL and user authencation

Disable SSL

Disable user authencation

References

simluate `&&` and `||` by using Nodejs

The exit code

Exit code of child process

So here we go

References

How to run two nodejs modules in distroless docker image

Ways to load env variables for your script

Shell Script

Makefile

Why not Nodejs

Alternative when `$schema` or `$host` is transformed

disable `SSL` and user authencation

The `exit code`