DEV Community: onichandame

Blog made by Gatsby

onichandame — Wed, 27 May 2020 01:25:43 +0000

Background
Challenges
- Static Site Generation
- Internationalization
- Gatsby Plugin i18n
- Translate Myself
- Markdown and JSX
- GraphQL
- Heading Anchor
- Deployment
Implementation
- Gatsby Build System
- GraphQL
- Internationalization
- Page Level
- Field Level

Having read this blog, I am convinced to host my personal blog. The most convncing point is that self-hosting blogs give more ways to monetize. Every developer is keen on monetizing his/her skills, just like me.

So the strategy has been made. Only tactics left undecided. There are 2 ways to build a self-hosted blog in general: using a blog builder such as WordPress, or build from "scratch". The difference between these 2 options is not actually well defined. I personally consider a tool requiring not web development skills as a blog builder.

As a half-frontend-developer, I decided to use a proper develper's framework for the job: Gatsby.

Background

Gatsby is a multi-page website generator based on React.js . The stack can be visualized as follows:

It can be seen that to use Gatsby, the user is expected to know HTML, CSS, JS, TS and React. Thus it is a great opportunity to self-learn frontend skills.

Challenges

Before diving into this project, I had no knowledge of Gatsby as a framework. Thus the path I took to learn is also applicable for any other newbie developer who wish to learn Gatsby.

Static Site Generation

As a Next.js developer, I am used to use server-side code during runtime. However it is forbidden for Gatsby as a Static Site Generator(SSG). This descrepancy has lead me to a dead end for many times. So these are the differences between SSG and SSR(Server Side Rendering):

SSR	SSG
allow server side code at runtime	does not allow server side code at runtime
shipped with server executable	shipped as plain html/css/... files
dynamically generate pages	requires rebuild to change page content

Based on the principle of SSG, 3 baselines need to be followed:

separate codes for buildtime and code for runtime
write runtime code as if writing in browser's dev tool
make sure all data can be statically fetched at buildtime

Internationalization

Working in a non-English-speaking country, it is necessary to serve audience from both English background and Chinese background. Thus my blog needs to serve contents in both English and Chinese. This is one of the considerations I took when I decided to make my own blogger as no other blogger is popular in both China and the outside world.

Coming to the implementation part, there are several tools and examples available online that can guide me.

Gatsby Plugin i18n

The first approach I tried was using a gatsby plugin. This is the best option for a non-blog site. I have used a similar tool for Next.js so this is not too hard to catch up. However this tool does not suit the needs of a blog site as it only provides field level translation. A blog utilizing field level translation is extremely difficult to maintain as all translation sources are kept in JSON files. A blog stored in JSON? No...

Translate Myself

According to the official guide, it is possible to achieve field level translation and page level translation at the same time.

Based on the guide, I made a template that bundles gatsby with mdx, page level translation and field level translation.

Markdown and JSX

The most significant difference between a blog and a regular site is that blogs are served in virtually the same layout. For the ease of maintenance, it would be better to keep the source of the blogs in a stable file format that has not changed in the past 10 years. It would also be beneficial if the source and the output are very similar so that overhead is small, and it is visually clear during writing.

Combining the above considerations, markdown is the best option. Markdown standard is very stable. Although some improvements have been added in the past few years, all plugins keep a very good interoperability and good backward compatibility.

However, the original markdown is quite limited in terms of functionality. It's implicity constrains its potential, especially for the flexbility of UI. Hence an enhancement needs to be made.

JSX is a great tool for creating fancy UI. Using markdown as the backbone and occasionally decorated with JSX will provide a decent layout for blogs. Thus gatsby-plugin-mdx is chosen.

Having decided to adopt mdx, several issues came in the way.

GraphQL

mdx in Gatsby allows page query. However, I personally do not like to use page query. Moreover, it is not usable in non-page components, which significantly limits its usage.

Sadly, component level query is not available in mdx as the JSX components in mdx are not compiled to retrieve static data during build time. Therefore the codes like useStaticQuery is left to run in runtime, where the query will not succeed.

Heading Anchor

A very useful markdown enhancement is the anchor of the headings. This is one of the basic requirements for table of contents. However, gatsby-plugin-mdx is not shipped with this support.

According to this blog, this feature can be easily added.

Deployment

The last problem is to host the site. There are generally 3 methods to choose:

buy a managed hosting service
host on a self-managed server
find a free hosting service

The option 1 and 2 are for lazy users who do not want to spend time digging the internet. I happen to know a hosting service for free: Github Pages.

Github Pages can be easily set up following the official guide. Bear in mind that there are 2 types of Page: project and personal. The one used for the purpose of this blog is the personal one.

Implementation

Here I record the detailed implementation of the blog site I made. Basic knowledge of React.js is preassumed.

Gatsby Build System

The first thing to learn is the build system of gatsby.

The source code is written in JSX and the output is static website. There are several fundamental issues here, most have been addressed by Gatsby itself, such as client side routing. One thing that requires special care is the data retrieval.

People from the background of server side coding may intuitively write codes that are left to run on server during runtime. This is a big no when using Gatsby, or any other SSG tool. Gatsby only allows 2 runtimes: buildtime and browser runtime.

During buildtime, all data are retrieved to create static HTML. During browser runtime, only browser API is available.

The codes for buildtime reside in the following locations:

page query and static query(GraphQL queries are run once at buildtime)
gatsby-node.js, gatsby-config.js, gatsby-browser.js and gatsby-ssr.js under the project's root directory

All other codes are compiled to static HTML at buildtime.

For details of the gatsby-node.js etc. see the conceptual guid.

GraphQL

The usage of GraphQL in Gatsby is somewhat unusual. In traditional data retrieval modes, a client fires a query to a backend API and the backend respond with the required data. Naively I thought it happens during runtime. But Gatsby runs all GraphQL queries during buildtime. Then all the queries are stripped away from the output.

Therefore all the queries must be given all the required data and schema during buildtime. One of the main sources of data is gatsby-config.js and gatsby-node.js. check the guide for details.

Internationalization

To unify page level and field level internationalization, the following design is made.

Page Level

Only mdx files are accepted for page level translation, such as posts and resume page.

The example source structure of a post is:

All translations of the page are kept in one folder, which is named as the slug of the post. The file name represents the locales. All the information is extracted by codes in gatsby-node.js then is passed down to the corresponding pages.

As all the pages have the same layout, a template is passed to the gatsby-plugin-mdx that is applied to all post pages.

Field Level

Field level translations are kept in a similar structure, where the file name represents the locale. A custom hook is used to retrieve those translations.

Check my blog for the final result!

CentOS 8 Bluetooth Headset

onichandame — Thu, 21 May 2020 01:47:39 +0000

On a fresh-new CentOS 8 desktop, it is not trivial to get a Bluetooth headset connected for the first time.

Symptoms

When trying to connect to a paired headset in Bluetooth settings, it keeps failing but not error log can be seen. The logs can be retrieved by running command journalctl -xe and scroll up to find the read lines similar to a2dp-sink profile connect failed for 5C:C6:E9:B9:9D:DE: Protocol not available.

This error message tells that the bluetooth service successfully identifies the headset, but having trouble to hand over the connection to the audio service.

Fix

The most common audio service shipped with CentOS is pulseaudio. The quick fix is to restart the pulseaudio service.

pulseaudio -k
pulseaudio --start

Note that both commands are meant to be run as non-root user.

Credits

Thanks to this question

Rootless systemd and podman

onichandame — Mon, 18 May 2020 06:34:11 +0000

Systemd is a useful tool programmatically managing routine tasks. I encountered a specific situation with some constraints:

start task on boot as a non-root user
auto restart on failure

The solution to this issue is to create a systemd task as the non-root user.

Steps

Create the service

systemctl edit --user --full --force <service_name>

add the content

[Unit]
Description=<Service>

[Service]
Restart=always
ExecStart=<Servece_Start>
ExecStop=<Service_Stop>

[Install]
WantedBy=default.target

the last line registers the service to the dependency list for auto-start

enable the service

systemctl enable --user <service_name>

(optional) enable linger

 sudo loginctl enable-linger "$USER"

This is to allow the service continuing to run after the user is logged out.

Podman Service

When running podman container as a systemd service, the ExecStart command should be podman start -a <container_name> and the ExecStop command should be podman stop -t <seconds_to_wait> <container_name>. Details can be found at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/managing_containers/running_containers_as_systemd_services_with_podman

Credits

Thanks to this issue

ulimit on CentOS 8.1

onichandame — Wed, 13 May 2020 08:53:34 +0000

Problem

On a fresh-installed Centos 8.1(kernel 4.18.0-147.8.1.el8_1) machine, the max number of opened file descriptors is set to H: 2048, S: 1024.

By editing /etc/security/limits.conf the limits can be upped. But this only takes effect in non-gnome-terminals, such as ssh sessions.

Cause

The gnome terminals do not see /etc/security/limits.conf during login as they are started under systemd(I hate it).

Fix

According to this, /etc/systemd/user.conf sets the soft limit and /etc/systemd/system.conf sets the hard limit for the user under systemd. The variable for file descriptors is DefaultLimitNOFILE.

If you are using Node.js, one more step is required: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p according to this issue.

Collaboration using Blender 3D

onichandame — Sat, 09 May 2020 03:37:59 +0000

The Blender 3D is the best 3D modelling software on Linux. I use Linux for work, and I need to process the 3D models made by my colleagues. Hence I have to learn to use Blender.

Scenario

There are 2 constraints in my situation, which are quite common I guess:

models are made in SolidWorks 2018
colleagues do not have any knowledge of Linux, open-source 3D model formats or any other thing that a non-IT worker is not assumed to know.

Problems

In this section, I will record all the important issues solved during my work.

Multi-language

The working environment of my colleagues is in Chinese. So inevitably they export files with non-English characters.

Blender's default setting disables the multi-language support. In order to display non-English characters on the UI, one has to check the box next to the Translation tab in the Preferences>Interface dialogue.

File Format

The biggest challenge is to move the models from SolidWorks to Blender. Thankfully they both support a wide range of file formats for import/export. Hence the quickest solution is to find a format to which SolidWorks can export, and from which Blender can import.

The best open source format in my mind is GLTF. Unfortunately, SolidWorks only began to support GLTF since ver. 2019.

After a quick search, 2 formats were found to suit my needs:

WRL file exported from SolidWorks cannot be properly imported to Blender. Nothing on the canvas is shown after the import finishes. The scene collection does not show any change either.

So STL is the only choice left. The bad thing about it is that SolidWorks exports many different files, each responsible for one component. But this is not a major bug which prevents using it.

View Clipping

This problem is not collaboration related but general for any usage of Blender.

If you know the basics of the camera parameters in 3D modelling, viewing frustum should not be too alienated. Basically in a virtual world composed of 3D models, you are only presented with a finite volume. This constraint is to prevent infinity. The farthest plane orthogonal to your viewing vector is one of the constraints. Anything beyond that plane is not rendered. This parameter defaults to a quite small value. I had to change it as the model imported is very large. Press "N" key to open the following dialogue and change the Clip Start and End value.

EPICS Deployment

onichandame — Fri, 08 May 2020 11:20:17 +0000

As a newbie in the world of EPICS and hardware control system, every day many new challenges come and many are solved. This is a record of the experience so that when the same problem comes in the future, I don't need to get through it again.

System Architecture

The details of the architecture are not covered here. Broadly speaking, the proxy converts EPICS protocol to GraphQL or vice versa.

Challenges

This section records all major challenges I have met. Some are trivial but I was not smart enough to find a solution immediately.

Connection between Different Hosts

The connection between CA client and IOC server is the most anti-intuitive part for me. EPICS provides a convenient technology "UDP broadcasting", which helps establish connection when both parties have no knowledge of the IP address of each other.

However, This method only works as expected when both parties are in the same and the only subnet. e.g. when the CA client has 2 IP addresses in 2 different subnets, such as 10.0.0.1 and 192.168.1.1, it is not guaranteed to be able to find an IOC whose address is 192.168.1.100.

When no better architecture design is feasible, one has to provide the IP address of the IOC server to the CA client, so that the client knows where to look for the IOC. The way to tell is by setting an environmental variable EPICS_CA_ADDR_LIST when starting the CA client. For example:

export EPICS_CA_ADDR_LIST=192.168.1.100
# CA connect here

However, it is generally a better practice to minimize the necessary configurations. A better solution here is putting the CA client in the subnet where IOC servers are found and no other subnets. In my architecture, it means that the web services and IOCs, as well as the proxy, are in the same subnet. This is achievable when the number of IOCs is relatively small. When there is large number of IOCs, they must be divided into different subnets for better management. In that case, a workaround can be made by introducing another layer of proxy between web services and the EPICS proxy, so that the EPICS proxy can stay with 1 IP.

React-based 3D components

onichandame — Sat, 02 May 2020 08:29:15 +0000

This is a record of my personal experience on the 3D React component development. The content may be updated at any time because the project is still under development.

Background

I need to design and build a 3D based building information modelling(BIM) due to the nature of BIM. This BIM component needs to reside in an existing system based on Next.js, a UI framework based on React and specialized in SSR optimization.

Demand Analysis

End-User Experience

The end user is an untrained worker who is not specialzed in either IT or BIM. Therefore the functions are simple and understandable for every computer user.

An intrinsic mapping between the UI and the real building, down to the component level
An intrinsic interactive pattern based on ray casting
All texts and numbers on the UI must be clear and minimal
A camera reset button, a scene reload button and a virtual button on each component to enter the corresponding page of details

Technical Constraints

Must be integrated into a Next.js project
Must be able to load 3D models like GLTF and OBJ
Must be code-splitted to minimize the impact on the performance of the original website

Technology Analysis

Based on the demand analysis, I conducted research to find some existing technologies that I can make use of. Many good frameworks are found but they all boils down to 3 fundamental technologies.

React 360

This is a framework built by Facebook for VR development. It also supports 3D UI out of the box. By default it loads models from GLTF and OBJ formats. At the first glance it may be the best choice for the following reasons:

3D out of the box
interactive out of the box
supports open source models GLTF and OBJ

But it is based on React Native, which makes it difficult to integrate into the existing webpage. Truly it can be embedded in <iframe>. But this workaround brings more complexities into the deployment phase. Moreover, this framework aims at VR, which is not a 100% overlap with BIM. Not to say that this framework is not as popular as its competitors, which means that it is more defficult to get help from the community.

Babylon

This framework supports the integration with React officially, according to the official docs. However the docs also mentions that there may be performance overhead if used with React. The optimal choice is to use the pure JavaScript, which is what I would like to avoid. There are feasible ways such as React DOM or reconciler. But again the community is quite small. The existing solution has just 140 stars on GitHub at the time of writing this sentence. As an individual developer the size of community is at the top of my list.

Three-based Solution

Now the main course comes. Three.js is the most popular 3D framework in Web development. It has the largest most active community. However, it is not designed for React hence the React-Three community is not comparable to the entire Three community.

React-Three-Fiber

This is the most suitable framework in this condition.

Deep integration with React with no significant performance overhead
Able to load GLTF models using useLoader hook, GLTF loader from Three and React Suspense
Just a wrapper around three so info from three community is most likely helpful
Many helper packages from the same team that help ray casting, animation and more.

The only hack needed here is the dynamic import of Next.js with SSR disabled. It is not a traditional hack that needs a proper fix, but a non-intuitive but reasonable solution for the problem: 2D rendering is compatible with SSR as SSR produces plain HTML, but 3D requires runtime loading of the resources such as models. If models are wrapped in React components, these components must be dynamically loaded using next/dynamic.

Also, dynamic importing of Next.js provides code-splitting out of the box.

Implementation

Currently, only the feasibility analysis is done. The package has not been designed yet.

An EPICS Proxy

onichandame — Mon, 20 Apr 2020 07:45:43 +0000

Background

EPICS is a well-known framework for controlling a wide range of hardware. Just like other inventions from the particle physicists, it reaches beyond the field of the particle physics experiments.

Scenario

EPICS provides a comprehensive bucket of tools for almost all use cases. In the age of the internet, however, many unprecedented demands have emerged. One of which is to allow remote control over the internet.

Here comes a trivial problem that has never been addressed before: how to communicate as a web front with the EPICS IOCs?

Technical Consideration

As a never-before new system, one of the most fundamental engineering principles is to stick with the existing standards as much as possible. Based on this principle, 3 sub-principles must be followed.

use loosely-coupled microservices each provides a limited and well-defined functionality
communication protocols and data structure are kept simple and consistent across the smallest necessary scope.
use standardized protocols, deployment workflow, and libraries

Hence a bridge between IOCs of the 20th century and the web frameworks of the 21st century is necessary.

Design

Data Flow

The data flow is the foundation of any software system and should be drafted before the actual development.

Communication Protocol

According to the basic structure, the proxy needs to handle 2 types of protocols: one for the microservices and one for the IOCs. The former has many standards I can choose from. The latter only has two standards at the moment: CA and PV.

As PV is only available to EPICS 7+, for better compatibility of the proxy, CA should be supported first.

Considering that caget caput are compatible with the stateless protocols, but camonitor requires a stateful protocol like WebSocket, the protocol to the microservices must support both cases. Hence the most popular communication protocol HTTP-based REST is not an option. After a quick research, GraphQL from Facebook is found to satisfy all the demands.

Framework

Both CA and GraphQL are too complicated to make from scratch. Therefore some existing frameworks must be utilized. Based on the choice of the protocols, 2 frameworks are required:

a GraphQL server
a CA library

The most popular GraphQL server is the apollo-server, and it is the only server that supports real-time subscriptions out of the box.

The only CA library comes from EPICS-base, in the form of dynamic libraries and executable binaries.

Implementation

The development of the proxy was not smooth and simple, therefore I will divide this section into challenges.

Feasibility

The very first question is: is such proxy even possible?

This question can be broken down into 2:

is it possible to integrate CA library with GraphQL server?
is the proxy able to communicate with the IOCs on the other hosts in the network?

The first question is answered by this tool and this tool. Thanks to the great pioneers who made these tools!

The second question can be cleared by a simple test using the tools mentioned above.

Setup Apollo Server

GraphQL is way more complicated than it sounds if you are new to it. There are many useful resources out there to help newbies set up their first server. I found this and this to be very helpful.

The biggest challenge here is to understand the concept of the resolver. It is a standardized component of GraphQL so many different packages can work together without a problem. As I use TypeScript to code, type-graphql suits my needs best. Although a code-first GraphQL server is better, the Apollo server does not support the code-first approach.

Having implemented caget, caput and camonitor as Query, Mutation and Subscription respectively, the server part is done.

Connect to IOC

Here comes the most tricky part. As Apollo Server only runs with Node.js, using JS/TS is the only choice. However, node-epics is too old to support the latest Node.js. Hence I made my own fork. Sadly it depends on ref-napi which only works on node before 13. Therefore it requires node <13, which puts this proxy to the same restriction. Luckily we have containers that minimize the impact of such restriction.

After some head-scratching, this tool is published. The connection to the IOC is easily implemented and tested.

This issue took me a week before the first version is published whereas the other issues only took me a couple of days.

Deployment

As the proxy relies on a specific version of the Node, it is better shipped with the Node of the correct version. Hence docker is the best solution.

Personally I use Kubernetes to manage the containers. I recommend anyone who needs docker to consider to switch to Kubernetes as it is just awesome.

Links

click here for the proxy
click here for the JS binding of CA