DEV Community: mrgao

Private projects use github action to package docker images and deploy them to servers

mrgao — Thu, 01 Jun 2023 08:11:36 +0000

There are always some projects that don't want to be open source. For the convenience of deployment, they hope to automatically deploy to different servers through docker after the code is changed. This article briefly describes the whole process, I hope it will help you

1. First create a project

First create a private project on github

Here we mainly take a front-end project as an example (a project created using umi)

mkdir myapp && cd myapp
pnpm dlx create-umi@latest

2. Talk about the basic commands of Docker

1. Common commands related to mirroring

# To download the mirror, first look for it from the local, did not go to the mirror, and finally did not go to the hub, if the label is not written, the default is latest

docker pull [image_name]:[image_tag]

# show all of images
docker image ls
docker images

# delete image file, -f force to delete
docker rmi [image_name][:Tag]
docker rmi [image_name1][:Tag1] [image_name2][:Tag2]    # delete muti images
docker rmi $(docker ps -a -q)    # delete all of images

# query the image name, --no-trunc displays the complete image description, --filter=stars=30 lists the images whose star is not less than the specified value, --filter=is-automated=true lists the images of the automatic build type
docker search [keyword]

# download the image, if the tag is not written, the default is latest, or you can add it yourself, for example: 3.2.0
docker pull [image_name1][:Tag]

2. Common commands related to containers

# list the running containers of this machine, -a lists all containers of this machine including the terminated container, -q silent mode only displays the container number, -l displays the most recently created container

docker container ls     
docker ps

# create new container
docker run [option] [container_name]

#  start container
docker start [container_id]/[container_name]

# restart container
docker restart [container_id]/[container_name]

docker stop [container_id]  # stop container
docker stop $(docker ps -a -q) # stop all of containers

# remove container by id
docker rm [container_id]
docker rm `docker ps -aq`    

# set the container to start automatically when docker starts
$ docker container update --restart=always [container_name]

I want to talk about docker runit option, because it is the most commonly used:

--name specifies a name for the container;
-d The container enters the background after starting and returns the container ID, that is, starts the daemon container;
-P random port mapping;
-p 80:8080 maps local port 80 to port 8080 of the container;
-i runs the container in interactive mode, usually used together with -t;
-t reassigns a pseudo-input terminal for the container, the shell of the container will be mapped to the current shell, and then the commands entered in the local window will be passed into the container, usually used together with -i;
--rm automatically deletes container files after the container terminates;
--restart=always set the container to start automatically;
-v /xxx:/yyy mapping command, map the xxx directory of the local machine to the yyy directory in the container, that is to say, if the content in the xxx directory of the local machine is changed, the content in the yyy directory of the container will also change;

3. Write Dockerfile

umiSince the product will be placed distunder the directory after the build, we can write the following Dockerfile;

Dockerfile is used to declare the steps of image execution; students who are not familiar with the following commands can click to view this article (opens new window)

FROM nginx:latest # use nginx to base image
LABEL maintainer "your github email"
ADD ./dist /usr/share/nginx/html # put the product in the /usr/share/nginx/html directory of nginx
EXPOSE 80 #

4. Write github action

Since only one private image is hub.docker.iocurrently supported, we will not consider using it for the time being; it is recommended that you use githubthe built-in image source ghcr.io

Preparatory work

Go to https://github.com/settings/tokens to generate a token, note that this token needs permission to read and write packages, please see here for details (opens new window)
Prepare the account number and password of your server to be deployed (the current page can use the secret key to log in, and the password is used here for now)
Think about your image name (here we take web-image as an example)

In .github/workflowsthe directory create apublish-image.yml

name: Publish Image

on:
  push: # push to the main branch for automatic release
    branches: ["main"]
    paths-ignore: # ignore some unnecessary files
      - ".gitignore"
      - "README.md"
      - ".vscode/**"
  pull_request:
    branches: ["main"]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v1
        with:
          registry: ghcr.io # declare mirror source
          username: ${{ github.actor }} # current github username
          password: ${{ secrets.HUB_GITHUB_TOKEN }} #  you need to go to https://github.com/settings/tokens to generate a token named token, note that this token requires permissions such as reading and writing packages
      - uses: actions/setup-node@v2
        with:
          node-version: "16.14"
      - uses: actions/cache@v2
        id: cache
        with:
          path: node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
        # Runs a single command using the runners shell
      - name: pnpm install
        run: npm install -g pnpm
      - name: Install
        if: steps.cache.outputs.cache-hit != 'true'
        run:  pnpm i

      - name: Build
        run: npm run build
      - name: Show Dir
        run: ls

      - name: Build the  Docker image
        run:
          | # use the Dockerfile written in the previous step to build the image and publish it to the private warehouse; after the release is complete, you can go to https://github.com/your_github_name?tab=packages to view
          docker build . --file Dockerfile --tag ghcr.io/your_github_name/web-image:latest 
          docker push ghcr.io/your_github_name/web-image:latest
      - name: Update to Server
        uses: appleboy/ssh-action@v0.1.10
        with:
          host: ${{ secrets.SERVER_HOST }} # Server ip address; you need to go to the settings/secrets/actions of the warehouse to create
          username: ${{ secrets.SERVER_USER }} # server username
          password: ${{ secrets.SERVER_PWD }} # server password
          port: ${{ secrets.SERVER_PORT }} # server port
          script: | # Re-update the mirror and deploy
            docker stop web-image 
            docker rm web-image 
            docker login -u your_github_name -p your_github_token https://ghcr.io 
            docker pull ghcr.io/your_github_name/web-image:latest 
            docker run -dp 80:80  --restart=always --name web-image ghcr.io/your_github_name/web-image

        # The above uses port 80 to start, of course you can also change to the port you want

5. Private docker image

The private image we githubused ghcr; of course, you can also create a private image based on docker/register; you can run the following command on your own server

# 1、download images
docker pull registry:2

# 2、run container 
$ docker run  -d --restart=always  -v /opt/registry:/var/lib/registry -p 5000:5000 --name myregistry registry:2

Then build the image locally and upload the image:



# privatization build
$ docker build . --file Dockerfile --tag server_ip_address:5000:web-image:latest

# privatization push

$ docker push server_ip_address:5000:web-image:latest

# Note that an error was reported when uploading the image：http: server gave HTTP response to HTTPS client

# 【Execute on the client side of docker push, not the private server】Solution 1：
# 1、edit /etc/docker/daemon.json，write in the file: (if your image is actually on linux/centos)

{ "insecure-registries":["server_ip_address:5000"] }

# 【Execute on the client of docker push, not a private server】 or solution 2 (if your image is on linux/centos)

$ sudo vim /usr/lib/systemd/system/docker.service
# and change content

ExecStart=/usr/bin/dockerd  --containerd=/run/containerd/containerd.sock --insecure-registry server_ip_address:5000

# 2、restart server and docker 

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
# push once
$ docker push server_ip_address:5000:web-image:latest

If you are using windows/mac, you can directly add it in the docker settings

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "features": {
    "buildkit": true
  },
  "insecure-registries": ["server_ip_address:5000"]
}

React Native SDK upgrade issues and split jsbundle

mrgao — Fri, 13 Aug 2021 09:51:41 +0000

Background:

Our team has always integrated ReactNative (hereinafter referred to as RN) as a sub-module into the existing android/ios application; the original RN version used was 0.55 ; as the times change, RN has 0.65 the version; the upgrade span is larger ; Here I will give a brief summary of the problems encountered in the recent SDK upgrade.

Question 1: How to split RN bundle

Preface

In the previous version RN , metro does not currently support the use of processModuleFilter for module filtering; if you google for RN split bundle, you will find it difficult to have an article detailing how RN performs split bundle; this article will detail how to perform RN split bundle.

RN split bundle, in the new version of metro , in fact, most of us only need to pay attention to the two apis of metro:

createModuleIdFactory : Create a unique id for each module of RN;
processModuleFilter : Select which modules are needed for the current build

First, let's talk about how to give an Id name to a module. The name according to the id that comes with metro is self-increasing according to the number:

function createModuleIdFactory() {
  const fileToIdMap = new Map();
  let nextId = 0;
  return (path) => {
    let id = fileToIdMap.get(path);
    if (typeof id !== "number") {
      id = nextId++;
      fileToIdMap.set(path, id);
    }
    return id;
  };
}

According to this, the moduleId will be incremented from 0 in turn;

Let's talk about processModuleFilter , a simplest processModuleFilter as follows:

function processModuleFilter(module) {
  return true;
}

It means that all modules of RN are needed, and there is no need to filter some modules;

With the above foundation, let's start to consider how to split bundle the RN; I believe everyone is clear about the general situation. We divide the entire jsbundle into the common package and the bussiness package; the common package is generally built into the App; and the bussiness package It is issued dynamically. Following this line of thinking, let's start subcontracting;

common package split bundle scheme

As the name suggests, the package is a common resource for all RN pages. Generally, there are several requirements for common

module will not change frequently
module is universal
generally does not put all npm packages under node_modules in the base package

According to the above requirements, a project basis we will generally react , react-native , redux , react-redux and other changes infrequently general public npm package on package; So how do we divide public bag? There are generally two ways:

Scheme 1 [PASS]. to analyze service entry as an entry packet, in processModuleFilter (module.path) to manually remove the module through the module path past

const commonModules = ["react", "react-native", "redux", "react-redux"];
function processModuleFilter(type) {
  return (module) => {
    if (module.path.indexOf("__prelude__") !== -1) {
      return true;
    }
    for (const ele of commonModules) {
      if (module.path.indexOf(`node_modules/${ele}/`) !== -1) {
        return true;
      }
    }
    return false;
  };
}

If you follow this way, trust me, you will definitely give up. Because it has a huge disadvantage: needs to manually handle the dependencies of packages such as react/react-native ; that is to say, it’s not that you wrote 4 modules and packaged these 4 modules. It is possible that these 4 modules depend on others. Module, so when running the common package, the basic package will directly report an error.

This led to the second plan:

Create a public package entry in the root directory and import the modules you need; use this entry when packaging.

Note: provides an entry file for the public package, so the code after packaging will report error Module AppRegistry is not registered callable module (calling runApplication) ; you need to manually delete the last line of code ;

For detailed code, please see: react-native-dynamic-load

common-entry.js entry file

// some module that you want
import "react";
import "react-native";
require("react-native/Libraries/Core/checkNativeVersion");

can write createModuleIdFactory

function createCommonModuleIdFactory() {
  let nextId = 0;
  const fileToIdMap = new Map();

  return (path) => {
    if (!moduleIdByIndex) {
      const name = getModuleIdByName(base, path);
      const relPath = pathM.relative(base, path);
      if (!commonModules.includes(relPath)) {
        // record path
        commonModules.push(relPath);
        fs.writeFileSync(commonModulesFileName, JSON.stringify(commonModules));
      }
      return name;
    }
    let id = fileToIdMap.get(path);

    if (typeof id !== "number") {
      // Use numbers for module id, and record the path and id for subsequent subcontracting of business packages, and filter out public packages
      id = nextId + 1;
      nextId = nextId + 1;
      fileToIdMap.set(path, id);
      const relPath = pathM.relative(base, path);
      if (!commonModulesIndexMap[relPath]) {
        commonModulesIndexMap[relPath] = id;
        fs.writeFileSync(
          commonModulesIndexMapFileName,
          JSON.stringify(commonModulesIndexMap)
        );
      }
    }
    return id;
  };
}

write metro.common.config.js

const metroCfg = require("./compile/metro-base");
metroCfg.clearFileInfo();
module.exports = {
  serializer: {
    createModuleIdFactory: metroCfg.createCommonModuleIdFactory,
  },
  transformer: {
    getTransformOptions: async () => ({
      transform: {
        experimentalImportSupport: false,
        inlineRequires: true,
      },
    }),
  },
};

Run packaging command

react-native bundle --platform android --dev false --entry-file common-entry.js --bundle-output android/app/src/main/assets/common.android.bundle --assets-dest android/app/src/main/assets --config ./metro.base.config.js --reset-cache && node ./compile/split-common.js android/app/src/main/assets/common.android.bundle

be careful:

above does not use processModuleFilter , since for common-entry.js inlet, all modules are required;
There are two ways to generate moduleId in the above implementation: one is a number, the other is a path; the difference between the two is not big, but it is recommended to use a number. The reasons are as follows:

The number is smaller than the string, the smaller the bundle size;
Multiple modules may have the same name, and the use of strings may cause module conflicts in multiple modules; if you use numbers, you won’t, because the numbers are random;

Numbers are more secure, if the app is attacked, it is impossible to know exactly which module the code is

business package and split bundle plan

I talked about the subcontracting of the public package. When the public package is subcontracted, the module path and module id in the public package will be recorded; for example:

{
  "common-entry.js": 1,
  "node_modules/react/index.js": 2,
  "node_modules/react/cjs/react.production.min.js": 3,
  "node_modules/object-assign/index.js": 4,
  "node_modules/@babel/runtime/helpers/extends.js": 5,
  "node_modules/react-native/index.js": 6,
  "node_modules/react-native/Libraries/Components/AccessibilityInfo/AccessibilityInfo.android.js": 7,
  "node_modules/@babel/runtime/helpers/interopRequireDefault.js": 8,
  "node_modules/react-native/Libraries/EventEmitter/RCTDeviceEventEmitter.js": 9
  // ...
}

In this way, when subcontracting the business package, you can judge by the path whether the current module is already in the basic package, if it is in the public package, use the corresponding id directly; otherwise, use the logic of business package subcontracting;

write createModuleIdFactory

function createModuleIdFactory() {
  // Why use a random number? It is to avoid rn module conflicts in singleton mode due to the same moduleId
  let nextId = randomNum;
  const fileToIdMap = new Map();

  return (path) => {
    // Use name as id
    if (!moduleIdByIndex) {
      const name = getModuleIdByName(base, path);
      return name;
    }
    const relPath = pathM.relative(base, path);
    // Whether the current module is already in the basic package, if it is in the public package, use the corresponding id directly; otherwise, use the logic of business package split bundle
    if (commonModulesIndexMap[relPath]) {
      return commonModulesIndexMap[relPath];
    }

    let id = fileToIdMap.get(path);
    if (typeof id !== "number") {
      id = nextId + 1;
      nextId = nextId + 1;
      fileToIdMap.set(path, id);
    }
    return id;
  };
}

Write to filter the specified module

// processModuleFilter
function processModuleFilter(module) {
  const { path } = module;
  const relPath = pathM.relative(base, path);
  if (
    path.indexOf("**prelude**") !== -1 ||
    path.indexOf("/node_modules/react-native/Libraries/polyfills") !== -1 ||
    path.indexOf("source-map") !== -1 ||
    path.indexOf("/node_modules/metro-runtime/src/polyfills/require.js") !== -1
  ) {
    return false;
  }
  if (!moduleIdByIndex) {
    if (commonModules.includes(relPath)) {
      return false;
    }
  } else {
    // The modules in the public package are directly filtered out
    if (commonModulesIndexMap[relPath]) {
      return false;
    }
  }
  return true;
}

Run commands to package

react-native bundle --platform android --dev false --entry-file index.js --bundle-output android/app/src/main/assets/business.android.bundle --assets-dest android/app/src/main/assets --config ./metro.business.config.js --reset-cache

The packaged effect is as follows:

// bussiness.android.js
d(function(g,r,i,a,m,e,d){var t=r(d[0]),n=r(d[1])(r(d[2]));t.AppRegistry.registerComponent('ReactNativeDynamic',function(){return n.default})},832929992,[6,8,832929993]);
// ...
d(function(g,r,i,a,m,e,d){Object.defineProperty(e,"**esModule",
r(832929992);

General code for subcontracting

How RN performs dynamic subcontracting and dynamic loading, please see: https://github.com/MrGaoGang/react-native-dynamic-load

Question 2: Cookie expiration problem

background

To Android for example, the common will Cookie use android of CookieManager manage; but we did not use it for internal management; the 0.55 version of the initialization time when you can set up a RN CookieProxy :

        ReactInstanceManagerBuilder builder = ReactInstanceManager.builder()
                .setApplication(application)
                .setUseDeveloperSupport(DebugSwitch.RN_DEV)
                .setJavaScriptExecutorFactory(null)
                .setUIImplementationProvider(new UIImplementationProvider())
                .setNativeModuleCallExceptionHandler(new NowExceptionHandler())
                .setInitialLifecycleState(LifecycleState.BEFORE_CREATE);
                .setReactCookieProxy(new ReactCookieProxyImpl());

Among them, ReactCookieProxyImpl can be implemented by yourself, or you can control how the Cookie is written to RN;

But in the latest RN, okhttp is used for network request, and andrid's CookieManager used for management; the code is as follows:

// OkHttpClientProvider
OkHttpClient.Builder client = new OkHttpClient.Builder()
.connectTimeout(0, TimeUnit.MILLISECONDS)
.readTimeout(0, TimeUnit.MILLISECONDS)
.writeTimeout(0, TimeUnit.MILLISECONDS)
.cookieJar(new ReactCookieJarContainer());

// ReactCookieJarContainer
public class ReactCookieJarContainer implements CookieJarContainer {

@Nullable
private CookieJar cookieJar = null;

@Override
public void setCookieJar(CookieJar cookieJar) {
this.cookieJar = cookieJar;
}

@Override
public void removeCookieJar() {
this.cookieJar = null;
}

@Override
public void saveFromResponse(HttpUrl url, List<Cookie> cookies) {
if (cookieJar != null) {
cookieJar.saveFromResponse(url, cookies);
}
}

@Override
public List<Cookie> loadForRequest(HttpUrl url) {
if (cookieJar != null) {
List<Cookie> cookies = cookieJar.loadForRequest(url);
ArrayList<Cookie> validatedCookies = new ArrayList<>();
for (Cookie cookie : cookies) {
try {
Headers.Builder cookieChecker = new Headers.Builder();
cookieChecker.add(cookie.name(), cookie.value());
validatedCookies.add(cookie);
} catch (IllegalArgumentException ignored) {
}
}
return validatedCookies;
}
return Collections.emptyList();
}
}

Then there is no use android.CookieManagercase of how toReactNativeinjectionCookie it?

solution

One possible idea is that clients have their own CookieManager when synchronizing update android.CookieManager ; but this scheme is the need for client support students;
The client gets the cookie and passes it to RN, and RN uses jsb to pass the cookie to android/ios

We adopted the second option:

The first step, the client will cookie by props passed to the RN


Bundle bundle = new Bundle();
// get cookie in native
String cookie = WebUtil.getCookie("https://example.a.com");
bundle.putString("Cookie", cookie);
// start
rootView.startReactApplication(manager, jsComponentName, bundle);

The second step, RN gets the Cookie

// this.props is the rn root component props
document.cookie = this.props.Cookie;

The third step is to set cookies to the client

const { RNCookieManagerAndroid } = NativeModules;
if (Platform.OS === "android") {
  RNCookieManagerAndroid.setFromResponse(
    "https://example.a.com",
    `${document.cookie}`
  ).then((res) => {
    // `res` will be true or false depending on success.
    console.log("RN_NOW: 设置 CookieManager.setFromResponse =>", res);
  });
}

The premise of use is that the client already has a corresponding native module. For details, please see:

https://github.com/MrGaoGang/cookies

Among them, the version of the rn community is mainly modified. Android cookies cannot be set at one time. You need to set

    private void addCookies(String url, String cookieString, final Promise promise) {
        try {
            CookieManager cookieManager = getCookieManager();
            if (USES_LEGACY_STORE) {
                // cookieManager.setCookie(url, cookieString);
                String[] values = cookieString.split(";");
                for (String value : values) {
                    cookieManager.setCookie(url, value);
                }
                mCookieSyncManager.sync();
                promise.resolve(true);
            } else {
                // cookieManager.setCookie(url, cookieString, new ValueCallback<Boolean>() {
                //     @Override
                //     public void onReceiveValue(Boolean value) {
                //         promise.resolve(value);
                //     }
                // });
                String[] values = cookieString.split(";");
                for (String value : values) {
                    cookieManager.setCookie(url, value);
                }
                promise.resolve(true);

                cookieManager.flush();
            }
        } catch (Exception e) {
            promise.reject(e);
        }
    }

Question 3: Window isolation problem in singleton mode

Background In the RN singleton mode, if each page uses the window for global data management, the data needs to be isolated; the industry-wide method is to use the micro front end qiankun to proxy window This is indeed a good method, but it may be more responsible in RN; the method used by the author is:

Use babel to replace global variables, so that you can ensure that for different pages, setting and using window have different effects on the following; for example:

// business code
window.rnid = (clientInfo && clientInfo.rnid) || 0;
window.bundleRoot = (clientInfo && clientInfo.bundleRoot) || "";
window.clientInfo = clientInfo;
window.localStorage = localStorage = {
  getItem: () => {},
  setItem: () => {},
};
localStorage.getItem("test");

The code after escaping is:

import _window from "babel-plugin-js-global-variable-replace-babel7/lib/components/window.js";

_window.window.rnid = (clientInfo && clientInfo.rnid) || 0;
_window.window.bundleRoot = (clientInfo && clientInfo.bundleRoot) || "";
_window.window.clientInfo = clientInfo;
_window.window.localStorage = _window.localStorage = {
  getItem: () => {},
  setItem: () => {},
};

_window.localStorage.getItem("test");