DEV Community: Maverick Fung

Amazon Route 53 Transfer domain and trouble shooting

Maverick Fung — Wed, 01 Nov 2023 06:35:44 +0000

Hi Developers,I'm Maverick,I want share a case for you today,Let me show you

Background

I hava a customer,who have 2 aws account ,I will call that Account-A and Account-B,in Account-A have a domain(like example.com) register by Route 53,and the customer have new project need deploy to Account-B,so the customer want to transfer this domain(example.com) from Account-A to Account-B.Then I login to Account-A Route 53 console and want transfer domain(example.com) to Account-B.

Step by Step

Route 53 Domain Operation

First login to Route 53 console page
in left navigation bar of the page,we need choose Registered Domains
Jump in you want transfer domain details page
make sure your domain "Transfer lock" status is "OFF"
Select "Transfer to another AWS account" from the drop-down box in the upper right corner
paste your target account ID
Then you will get the transfer password
Login another AWS account,like Account-B,same operations step 1-2,and choose "Transfer in"---"Single domain"
In details page,just put request infomation

Trouble shooting

The domain is not transferable at this time

Introduce

Actually,I don't know how to deal this problem,and I trying to search by using google,and I found have some body also have same question,but no answer!

Same question link:https://repost.aws/questions/QUsUOoB4fHSG-WBb9QWTlMxA/why-is-transferring-domain-between-accounts-unavailable-at-this-time

Solution

Make sure your Domain status code is "ok" and no more code,In this case , customer domain just buy about 2 days,the Domain status code is "addPeriod ok",you have to wait about 5 days or more time

Use Boto3 create EventBridge-Rule tigger Lambda Function

Maverick Fung — Thu, 18 Aug 2022 03:07:00 +0000

Hi,Guys,This article I will teach you how to use python3 to create EventBridge Rule and tigger it to lambda function

If you don't have too much time,I will do this

somebody say:"Cut the crap and show me your code"

import os
import time
import boto3
from botocore.config import Config

lambda_client = boto3.client('lambda',config=Config(region_name=os.environ['AWS_REGION']))
event_client = boto3.client('events',config=Config(region_name=os.environ['AWS_REGION']))

def lambda_handler(event, context):
    rule_name = 'LeifengRule' # Define a var for rule_name
    cron_sec = 'cron(59 23 * * ? *)' # Define a var for cron
    lambda_fc_name = 'LeifengFC' # Define a var for lambda name
    lambda_fc_arn = 'arn:aws:lambda:us-east-1:xxxx:function:LeifengFC' # Here you need copy the lambda_fc_name function arn
    add_permission_role_arn = 'arn:aws:iam::xxxx:role/add_permission' # put create role ARN
    # use boto3 create a rule
    create_rule_resp = event_client.put_rule(
            Name=rule_name, # There put your rule name
            ScheduleExpression=cron_sec, # there put your cron
            State='ENABLED', # there set the rule state ENABLED or DISABLED
            EventBusName='default', # set eventbus ,I use default
            RoleArn=add_permission_role_arn
        )

    put_target_resp = event_client.put_targets(
            Rule=rule_name,
            Targets=[{
                'Id': lambda_fc_name,
                'Arn': lambda_fc_arn
            }]
        )

    # use if to determine the lambda_fc_arn weather '$' exists
    # if the '$' in lambda_fc_arn,just remove from $

    if '$' in lambda_fc_arn:
        lambda_fc_arn = lambda_fc_arn[:-8]
    add_lambda_permission = lambda_client.add_permission(
            FunctionName=lambda_fc_arn,
            StatementId=str(time.time())[-5:]+lambda_fc_name,
            Action='lambda:InvokeFunction',
            Principal='events.amazonaws.com',
            SourceArn=create_rule_resp['RuleArn']
        )

1.Create IAM Role and Policy

1.1 Create add_permission role

1.1.1 Open AWS IAM console

Click here:https://us-east-1.console.aws.amazon.com/iam/home

1.1.2 Create a role

1.1.2.1 use the json file

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

1.1.2.2 attach 2 policy to the role(lambda and event full access)

1.1.2.3 Remember the add_permission role ARN

1.2 Create lambda execute role

1.2.1 Open AWS IAM console

Click here:https://us-east-1.console.aws.amazon.com/iam/home

1.2.2 Create a role for lambda

1.2.3 set a name lambda_exec_role

1.2.4 check the role(not attach any policy)

1.3 Create a policy for lambda_exec_role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "events:DeleteRule",
                "events:PutTargets",
                "events:DescribeRule",
                "events:ListRuleNamesByTarget",
                "events:EnableRule",
                "events:PutRule",
                "events:ListRules",
                "events:RemoveTargets",
                "events:ListTargetsByRule",
                "events:DisableRule",
                "lambda:ListFunctions",
                "lambda:AddPermission",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

1.4 attach lambda_exec_role_policy to lambda_exec_role

2.Create a lambda function

2.1 create a lambda function set runtime as python3.9

2.2 Copy the code to lambda

please replace rule_name cron_sec lambda_fc_name and lambda_fc_arn value

this code just show you

import os
import time
import boto3
from botocore.config import Config

lambda_client = boto3.client('lambda',config=Config(region_name=os.environ['AWS_REGION']))
event_client = boto3.client('events',config=Config(region_name=os.environ['AWS_REGION']))

def lambda_handler(event, context):
    rule_name = 'LeifengRule' # Define a var for rule_name
    cron_sec = 'cron(59 23 * * ? *)' # Define a var for cron
    lambda_fc_name = 'LeifengFC' # Define a var for lambda name
    lambda_fc_arn = 'arn:aws:lambda:us-east-1:xxxx:function:LeifengFC' # Here you need copy the lambda_fc_name function arn
    add_permission_role_arn = 'arn:aws:iam::xxxx:role/add_permission' # put create role ARN
    # use boto3 create a rule
    create_rule_resp = event_client.put_rule(
            Name=rule_name, # There put your rule name
            ScheduleExpression=cron_sec, # there put your cron
            State='ENABLED', # there set the rule state ENABLED or DISABLED
            EventBusName='default', # set eventbus ,I use default
            RoleArn=add_permission_role_arn
        )

    put_target_resp = event_client.put_targets(
            Rule=rule_name,
            Targets=[{
                'Id': lambda_fc_name,
                'Arn': lambda_fc_arn
            }]
        )

    # use if to determine the lambda_fc_arn weather '$' exists
    # if the '$' in lambda_fc_arn,just remove from $

    if '$' in lambda_fc_arn:
        lambda_fc_arn = lambda_fc_arn[:-8]
    add_lambda_permission = lambda_client.add_permission(
            FunctionName=lambda_fc_arn,
            StatementId=str(time.time())[-5:]+lambda_fc_name,
            Action='lambda:InvokeFunction',
            Principal='events.amazonaws.com',
            SourceArn=create_rule_resp['RuleArn']
        )

If this article can help you, I will be very happy,Thank you ,have a nice day!

amazing! Millions of people are learning Docker!(Chinese)

Maverick Fung — Wed, 10 Aug 2022 16:29:54 +0000

1、环境简介

1.1、系统简介

1.1.1、Docker简介

:::info
💡Docker 是一个开源的应用容器引擎，让开发者可以打包他们的应用以及依赖包到一个可移植的容器中，然后发布到任何流行的Linux机器上，也可以实现虚拟化，容器是完全使用沙箱机制，相互之间不会有任何接口。
:::
一个完整的Docker有以下几个部分组成：

dockerClient客户端
Docker Daemon守护进程
Docker Image镜像
DockerContainer容器

Docker引擎Docker Engine是C/S架构，主要有以下部件组成：
服务器（Docker daemon）：后台运行的Docker daemon进程。Daemon进程用于管理Docker对象，包括镜像（images）、容器（containers）、网络（networks）、数据卷（data volumes）。
REST接口：同daemon交互的REST API接口。
客户端（Docker client）：命令行（CLI）交互客户端。客户端使用REST API接口同Docker daemon进行访问。Docker服务的架构图如图1-1所示。

图1-1 Docker服务的架构图

1.1.2、Docker平台组成

图1-2 Docker服务组成图
运行一个Docker服务，组成包括Docker daemon服务器、Docker Client客户端、Docker Image镜像、Docker Registry库、Docker Contrainer容器，如图1-2所示。
（1）Docker镜像：
是一个只读模板，用于创建Docker容器，由Dockerfile文本描述镜像的内容。镜像定义类似“面对对象的类”，从一个基础镜像（Base Image）开始。构建一个镜像实际就是安装、配置和运行的过程，Docker镜像基于UnionFS把以上过程进行分层（Layer）存储，这样更新镜像可以只更新变化的层。Docker的描述文件为Dockerfile，Dockerfile是一个文本文件，基本指令包括：
FROM：定义基础镜像。
MAINTAINER ：作者或维护者。
RUN：运行linux 命令。
ADD：增加文件或目录。
EVN：定义环境变量。
CMD：运行进程。
（2）Docker容器：
是一个镜像的运行实例。容器有镜像创建，运行过程例如：
运行ubuntu操作系统镜像，-i 前台交互模型，运行命令为/bin/bash

$ docker run -i -t ubuntu /bin/bash

拉（pull）镜像，Docker Engine 检查ubuntu 镜像是否存在，如果本地已经存在，使用该镜像创建容器，如果不存在，Docker Engine从镜像库拉镜像。
使用该镜像创建新容器。
分配文件系统，挂载一个读写层，在读写从加载镜像。
分配网络/网桥接口，创建一个网络接口，让容器和主机通信。
从可用的IP池选择IP地址，分配给容器。
执行命令/bin/bash。
捕获和提供执行结果。
（3）Docker 仓库：
Docker仓库是Docker镜像库。Docker Registry也是一个容器。Docker Hub是Docker公司提供的互联网公共镜像仓库。可以构建自己本地的镜像仓库，国内有些公司也构建了镜像仓库。包括阿里云、新浪等。Docker 集群服务：Docker集群服务运行承租的Docker节点一起工作。目前支持swarm模式。
一个 Docker Registry 节点中可以包含多个仓库（Repository）；每个仓库可以包含多个标签（Tag）；每个标签对应一个镜像。
一般而言，一个仓库包含的是同一个软件的不同版本的镜像，而标签则用于对应于软件的的不同版本。
可以通过 <仓库名>:<标签> 的格式来指定具体是哪个版本的镜像。如果不给出标签，将以 latest 作为默认标签。
以Ubuntu镜像为例，ubuntu 是仓库的名字，其内包含有不同的版本标签，如，14.04, 16.04。可以通过 ubuntu:14.04，或者 ubuntu:16.04 来具体指定所需哪个版本的镜像。如果忽略了标签，比如ubuntu，那将视为 ubuntu:latest。

docker pull ubuntu:16:04
docker pull ubuntu:14:04
docker pull ubuntu  #这里指代  ubuntu:latest

1.2、预装环境说明

1.2.1、软件版本说明

宿主机系统:CentOS 7.9
Docker版本: 1.13.1

1.2.2、基础环境配置

:::info
💡本次实验环境采用下面的基础环境1-4步骤进行基础设置，请勿遗漏
:::
1.关闭selinux

# vi /etc/sysconfig/selinux
SELINUX=disabled # 修改enforcing为disabled
# getenforce
Disabled

2.关闭防火墙

systemctl stop firewalld

3.删除iptables防火墙规则

iptables –F
iptables –X
iptables –Z
/usr/sbin/iptables-save

4.修改系统内核

打开内核转发功能。
vi /etc/sysctl.conf
编辑配置文件/etc/sysctl.conf，将以下内容添加,添加完后在命令模式下 :wq 保存退出：
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

#修改完成后使用命令生效。
sysctl –p

2、服务安装

2.1、安装Docker服务

2.1.1、CentOS安装Docker

yum install -y docker  #安装docker服务
systemctl restart docker #启动docker服务
systemctl enable docker #设置开机启动

2.1.2、Ubuntu安装Docker

sudo apt-get update
sudo apt-get install -y docker.io

3、镜像

:::info
在之前的介绍中，我们知道镜像是 Docker 的三大组件之一。 Docker 运行容器前需要本地存在对应的镜像，如果镜像不存在本地，Docker 会从镜像仓库下载（默认是 Docker Hub 公共注册服务器中的仓库）。本章将介绍更多关于镜像的内容，包括：从仓库获取镜像；管理本地主机上的镜像；介绍镜像实现的基本原理。

:::
本节将介绍更多关于镜像的内容，包括：

从仓库获取镜像；
管理本地主机上的镜像；
介绍镜像实现的基本原理。

3.1、设置镜像加速（阿里云）

点击此处设置阿里云镜像加速服务:https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors

修改为阿里云镜像加速

3.2、拉取镜像(pull)

可以使用 docker pull 命令来从仓库获取所需要的镜像。
这里使用nginx镜像作为演示

# 以nginx为例

docker pull nginx

3.3、列出镜像(images)

# 列出本地所有镜像
docker images

在列出信息中，可以看到几个字段信息
来自于哪个仓库(REPOSITORY)---> docker.io/nginx
比如 nginx 镜像的标记(TAG)，比如 latest
它的 ID 号（唯一）---> b692a91e4e15
创建时间 ---> 2 days ago
镜像大小 ---> 142 MB

3.4、创建镜像(commit/import/Dockerfile)

3.4.1、使用commit创建镜像

:::info
💡其中， -m 来指定提交的说明信息，跟我们使用的版本控制工具一样； -a 可以指定更新的用户信息；之后是用来创建镜像的容器的 ID；最后指定目标镜像的仓库名和 tag 信息。创建成功后会返回这个镜像的 ID 信息。
:::

# 运行一个名为nginx的容器 映射宿主机 8081到80端口
docker run -dit --name nginx -p 8081:80 nginx
# 查看所有容器
docker ps -a
# 使用exec 命令进入 nginx容器内
docker exec -it nginx bash

# 使用echo 替换 index.html页面显示内容
echo "this is v2 website" > /usr/share/nginx/html/index.html


# 此处的 ee9d263ea0f7 为之前创建的 nginx 容器 id
docker commit -m "this is v2 web" -a "UniStars" ee9d263ea0f7 nginx:v2

# 使用新构建的镜像运行容器
docker run -dit --name nginx2 nginx:v2

3.4.2、使用Dockerfile创建镜像

Dockerfile是一个包含用于组合映像的命令的文本文档。可以使用在命令行中调用任何命令。 Docker通过读取Dockerfile中的指令自动生成映像。

# 使用mkdir 命令在当前目录下创建一个dockerfile目录
mkdir dockerfile
# 使用vi命令 创建并编写 Dockerfile 文件
vi Dockerfile

# 添加如下内容
FROM centos:7
RUN yum -y install httpd
RUN echo "Build By Dockerfile" > /var/www/html/index.html
RUN echo "ServerName localhost:80" >> /etc/httpd/conf/httpd.conf
EXPOSE 80
CMD ["/usr/sbin/httpd","-D","FOREGROUND"]

# 运行docker build 命令来构建该镜像，这里的-t 后面跟的是这个镜像的TAG
 docker build -t httpd:v1 .
# 运行 docker images 查看构建成功的镜像
docker images

# 使用docker run 来运行刚刚构建的镜像
docker run -dit --name http -p 8088:80 httpd:v1

3.4.3、Dockerfile详解

Dockerfile 由一行行命令语句组成，并且支持以 # 开头的注释行。
Dockerfile 基本的语法如下

使用 # 来注释
MAINTAINER 镜像作者信息
FROM 指令告诉 Docker 使用哪个镜像作为基础
RUN 开头的指令会在创建中运行，比如安装一个软件包，在这里使用 yum 来安装了一些软件
COPY 从 Docker 宿主机复制文件至创建的新镜像文件
ADD 类似于 COPY 指令，ADD 支持 tar 文件和 URL 路径
WORKDIR 用于为 Dockerfile 中所有的 RUN、CMD、ENTRYPOINT、COPY、ADD 指定设定工作目录
VOLUME 数据卷，用于在 image 中创建挂载点目录，以挂载 Docker host 上的卷或者其他容器上的卷
EXPOSE 为容器打开指定的监听端口以实现与外部通信
ENV 用于为镜像定义所需的环境变量，可以被 Dockerfile 文件中其他命令调用(ENV、ADD、COPY、RUN、CMD)
CMD 启动容器指定默认要运行的程序或命令，默认"/bin/sh -c"运行
ENTRYPOINT 类型 CMD 指令的功能，用于为容器指定默认运行程序或命令
USER 指定当前用户

3.4.3.1、FROM

格式为 FROM 或 FROM :TAG。
第一条指令必须为 FROM 指令。并且，如果在同一个Dockerfile中创建多个镜像时，可以使用多个 FROM 指令（每个镜像一次）。

FROM centos:7

3.4.3.2、MAINTAINER

格式为 MAINTAINER ，指定维护者信息。

MAINTAINER FengLei

3.4.3.3、RUN

格式为:

RUN
RUN ["executable", "param1", "param2"]

RUN yum -y install httpd

前者将在 shell 终端中运行命令，即 /bin/sh -c ；后者则使用 exec 执行。指定使用其它终端可以通过第二种方式实现，例如 RUN ["/bin/bash", "-c", "echo Build_Dockerfile"] 。
每条 RUN 指令将在当前镜像基础上执行指定命令，并提交为新的镜像。当命令较长时可以使用 \ 来换行。

RUN yum clean all; \
    yum install -y httpd; \
    mkdir -p /var/www/html/;

3.4.3.4、COPY

格式：

COPY ...
COPY ["",... ""]

和 RUN 指令一样，也有两种格式，一种类似于命令行，一种类似于函数调用。
复制本地主机的 .（为 Dockerfile 所在目录的相对路径）到容器中的。
当使用本地目录为源目录时，推荐使用 COPY 。
COPY 指令将从构建上下文目录中 <源路径> 的文件/目录复制到新的一层的镜像内的 <目标路径> 位置。比如：

COPY index /usr/share/nginx/html

<源路径> 可以是多个，甚至可以是通配符，其通配符规则要满足 Go 的 filepath.Match 规则，如：
COPY /root/* /mnt/
COPY /root/.html /usr/share/nginx/html/
<目标路径>* 可以是容器内的绝对路径，也可以是相对于工作目录的相对路径（工作目录可以用 WORKDIR 指令来指定）。目标路径不需要事先创建，如果目录不存在会在复制文件前先行创建缺失目录。此外，还需要注意一点，使用 COPY 指令，源文件的各种元数据都会保留。比如读、写、执行权限、文件变更时间等。这个特性对于镜像定制很有用。特别是构建相关文件都在使用 Git 进行管理的时候。

3.4.3.5、ADD

格式为 ADD 。
该命令将复制指定的到容器中的。
其中可以是Dockerfile所在目录的一个相对路径；也可以是一个 URL；还可以是一个 tar 文件（自动解压为目录）。

FROM centos:7
ADD httpd-2.4.tar.gz /

3.4.3.6、WORKDIR

格式为 WORKDIR
为后续的 RUN 、 CMD 、 ENTRYPOINT 指令配置工作目录。
可以使用多个 WORKDIR 指令，后续命令如果参数是相对路径，则会基于之前命令指定的路径。例如

WORKDIR /mnt/
WORKDIR dir1
WORKDIR dir2

最终路径为 /mnt/dir1/dir2

3.4.3.7、USER

格式为 USER <用户名>
USER 指令和 WORKDIR 相似，都是改变环境状态并影响以后的层。 WORKDIR 是改变工作目录， USER 则是改变之后层的执行 RUN , CMD 以及 ENTRYPOINT 这类命令的身份。
注意: USER只是切换用户，该用户必须事先存在

RUN groupadd -r ug && useradd -r -g ug user1
USER user1

3.4.3.8、VOLUME

格式为 VOLUME ["/data"] 。
创建一个可以从本地主机或其他容器挂载的挂载点，一般用来存放数据库和需要保持的数据等。

3.4.3.9、EXPOSE

格式为 EXPOSE [...] 。
告诉 Docker 服务端容器暴露的端口号，供互联系统使用。
这只是一个声明，在运行时并不会因为这个声明应用就会开启这个端口的服务。在 Dockerfile 中写入这样的声明有两个好处，一个是帮助镜像使用者理解这个镜像服务的守护端口，以方便配置映射；另一个用处则是在运行时使用随机端口映射时，也就是 docker run -P 时，会自动随机映射 EXPOSE 的端口。

FROM centos:7
EXPOSE 80,22

3.4.3.10、ENV

格式有两种：

ENV
ENV =

这个指令就是设置环境变量，无论是后面的其它指令，如 RUN ，还是运行时的应用，都可以直接使用这里定义的环境变量。定义了环境变量，那么在后续的指令中，就可以使用这个环境变量。
这里的例子演示了定义MYSQL的用户和密码

ENV MYSQL_USER docker_user
ENV MYSQL_PASS docker_password

3.4.3.11、CMD

支持三种格式，分别是:

CMD ["executable","param1","param2"] 使用 exec 执行（推荐）；
CMD command param1 param2 在 /bin/sh 中执行，提供给需要交互的应用；
CMD ["param1","param2"] 提供给 ENTRYPOINT 的默认参数；

指定启动容器时执行的命令，每个 Dockerfile 只能有一条 CMD 命令。
如果指定了多条命令，只有最后一条会被执行。
如果用户启动容器时候指定了运行的命令，则会覆盖掉 CMD 指定的命令。
在运行时可以指定新的命令来替代镜像设置中的这个默认命令，比如， centos 镜像默认的 CMD 是 /bin/bash ，如果我们直接 docker run -it centos ,会直接进入 bash 。
也可以在运行时指定运行别的命令，如 docker run -it centos cat /etc/redhat-release。这就是用 cat /etc/redhat-release 命令替换了默认的 /bin/bash 命令

在指令格式上，一般推荐使用** exec **格式，这类格式在解析时会被解析为 JSON 数组，因此一定要使用双引号 " ，而不要使用单引号。
如果使用 shell 格式的话，实际的命令会被包装为 sh -c 的参数的形式进行执行。比如：
CMD echo $USERNAME在实际执行中，会将其变更为： CMD [ "sh", "-c", "echo $USERNAME" ]

CMD [ "sh", "-c", "echo $USERNAME" ]

这就是为什么我们可以使用环境变量的原因，因为这些环境变量会被 shell 进行解析处理。提到 CMD 就不得不提容器中应用在前台执行和后台执行的问题。这是初学者常出现的一个混淆。

3.4.3.12、ENTRYPOINT

ENTRYPOINT 成为入口点
ENTRYPOINT 的格式和 RUN 指令格式一样，分为 exec 格式和 shell 格式。
拥有两种格式：

ENTRYPOINT ["executable", "param1", "param2"]
ENTRYPOINT command param1 param2 （shell中执行）。

配置容器启动后执行的命令，并且不可被 docker run 提供的参数覆盖。每个 Dockerfile 中只能有一个 ENTRYPOINT ，当指定多个时，只有最后一个起效。
ENTRYPOINT 的目的和 CMD 一样，都是在指定容器启动程序及参数。 ENTRYPOINT 在运行时也可以替代，不过比 CMD 要略显繁琐，需要通过 docker run 的参数 --entrypoint 来指定。当指定了 ENTRYPOINT 后， CMD 的含义就发生了改变，不再是直接的运行其命令，而是将 CMD 的内容作为参数传给 ENTRYPOINT 指令，换句话说实际执行时，将变为：
"CMD"

3.4.4、Dockerfile演示

3.4.4.1、构建带有openssh-server功能的容器镜像

centos-7 镜像为基础镜像,安装openssh-server服务，暴露端口 22，容器开机运行 sshd

FROM centos:7

RUN yum install -y openssh-server sudo
RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config 

#RUN useradd admin
#RUN echo "admin:admin" | chpasswd
RUN echo "root:123456" | chpasswd
#RUN echo "admin  ALL=(ALL)    ALL" >> /etc/sudoers

RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

编写完成 Dockerfile 后可以使用 docker build 来生成镜像。
其中 -t 标记来添加 tag，指定新的镜像的用户信息。
“.” 是 Dockerfile 所在的路径（当前目录），也可以替换为一个具体的 Dockerfile 的路径

3.5、标记(tag)

:::info
以阿里云为例,稍后会在仓库小节进行详细描述
:::

docker tag 镜像id 远程/本地仓库/镜像名称:版本号

例如:
docker tag 37bb9c63c8b2 registry-vpc.cn-hangzhou.aliyuncs.com/acs/agent:0.7-dfb6816

37bb9c63c8b2 为镜像id
registry-vpc.cn-hangzhou.aliyuncs.com/acs/  为仓库地址
agent 为镜像名称
0.7-dfb6816 为版本号

3.6、上传镜像(push)

:::info
以阿里云为例,稍后会在仓库小节进行详细描述
:::

3.7、导出/导入(save/load)

如果要导出镜像到本地文件，可以使用 docker save 命令。
可以使用 docker load 从导出的本地文件中再导入到本地镜像库

# 列出镜像,找出要导出
docker images

# 导出镜像
docker save -o nginx_sv2.tar nginx:v2

# 查看
ls -l

# 查看准备导入的镜像包
ls -l

# 先删除已存在的 nginx:v2
docker rmi 2ee0ee0dc960

# 导入镜像,2种语法

docker load --input nginx_v2.tar
docker load < nginx_v2.tar

3.8、使用supermin5构建镜像

使用 supermin5 命令构建 centos7系统的 docker 镜像，镜像名称为 centos-7，镜像预装 yum、net-tools、initscripts 和 vi 命令

# 安装supermin
yum -y install supermin*

# supermin5 添加预装工具 yum net-tools initscripts vi
supermin5 -v --prepare bash coreutils  yum  net-tools initscripts vim-minial  -o supermin.d
# supermin5 构建
supermin5 -v --build --format chroot supermin.d -o appliance.d

echo 7 >  appliance.d/etc/yum/vars/releasever
# 镜像打包
tar --numeric-owner -cpf centos-7.tar -C appliance.d .

# 导入镜像
cat centos-7.tar | docker import - centos-7
# 运行
docker run -dit --name centos7 centos-7 /bin/bash
cat /etc/redhat-release

3.9、删除镜像(rmi)

docker rmi 镜像iD

例:
docker rmi b692a91e4e15
docker rmi b6
b692a91e4e15 代表nginx镜像的id 也可以只写 前两位

4、容器

:::info
容器是 Docker 又一核心概念。
简单的说，容器是独立运行的一个或一组应用，以及它们的运行态环境。对应的，虚拟机可以理解为模拟运行的一整套操作系统（提供了运行态环境和其他系统环境）和跑在上面的应用。
本节将具体介绍如何来管理一个容器，包括创建、启动和停止等

:::

4.1、查看容器

# 查看正在运行的容器
docker ps 

# 查看所有容器
docker ps -a

4.2、启动/停止容器

4.2.1、创建并启动新容器

常用附加参数:
-d 在后台运行容器并打印容器ID
-t 选项让Docker分配一个伪终端（pseudo-tty）并绑定到容器的标准输入上
-i 则让容器的标准输入保持打开。
-p 映射端口宿主机端口:容器端口
-P 随机映射端口
-v 绑定卷
-w 设置工作目录
--rm 退出容器自动删除
--restart 重启条件，若宿主机开机要求该容器也运行那么请设置 --restart=always
--name 设置容器名称

# 启动一个随宿主机启动,并映射宿主机80端口到容器80端口 的 nginx 容器
docker run -dit --name nginx -p 80:80 --restart=always nginx

4.2.2、启动已停止的容器

docker start 容器名称/容器id

4.2.3、停止正在运行的容器

docker stop 容器名称/容器id

4.3、删除容器

# 删除已停止的容器
docker rm 容器名称/容器id

# 删除正在运行的容器  -f 代表 强制
docker rm -f 容器名称/容器id

# 删除所有停止的容器
docker container prune

4.4、连接容器

在使用 -d 参数时，容器启动后会进入后台。某些时候需要进入容器进行操作，包括使用 docker attach 命令或 docker exec 命令，推荐使用 docker exec 命令

4.4.1、attach命令

使用 attach 命令有时候并不方便。当多个窗口同时 attach 到同一个容器的时候，所有窗口都会同步显示。当某个窗口因命令阻塞时,其他窗口也无法执行操作了。
当使用attach退出时，容器也会退出

docker attach centos7

4.4.2、exec命令

docker exec 后边可以跟多个参数，一般使用 -i -t参数。只用 -i 参数时，由于没有分配伪终端，界面没有我们熟悉的 Linux 命令提示符，但命令执行结果仍然可以返回。当 -i -t 参数一起使用时，则可以看到我们熟悉的 Linux 命令提示符。

docker exec -it centos7 bash

4.5、导入导出容器

4.5.1、导出容器

docker export 容器id > tar包名

例如
docker export 82be49d3c283 > nginx.tar

4.5.2、导入容器

# cat tar包 | docker import - 仓库/镜像名:tag
# 例：
cat nginx.tar | docker import - unistars/nginx:v1

5、仓库

仓库（Repository）是集中存放镜像的地方。
一个容易混淆的概念是注册服务器（Registry）。实际上注册服务器是管理仓库的具体服务器，每个服务器上可以有多个仓库，而每个仓库下面有多个镜像。从这方面来说，仓库可以被认为是一个具体的项目或目录。
例如对于仓库地址 registry.cn-hangzhou.aliyuncs.com/fl_test/docker_demo 来说

registry.cn-hangzhou.aliyuncs.com 是注册服务器地址
fl_test 是仓库名
docker_demo为镜像名称 ## 5.1、远程仓库 :::info 目前 Docker 官方维护了一个公共仓库 Docker Hub，其中已经包括了超过 15,000 的镜像。大部分需求，都可以通过在 Docker Hub 中直接下载镜像来实现。下面以阿里云容器镜像服务为大家演示 :::

5.1.1、创建阿里云镜像仓库

点击创建阿里云镜像仓库(杭州站)：https://cr.console.aliyun.com/cn-hangzhou/instance/dashboard

仓库信息---设置命名空间仓库名称仓库类型以及说明
设置代码源 ---- 本地仓库

5.1.2、登录远程仓库

可以通过执行 docker login 命令来输入用户名、密码和邮箱来完成注册和登录。注册成功后，本地用户目录的 .dockercfg 中将保存用户的认证信息。

5.2、本地私有仓库

有时候出于安全和网络情况考虑，云上仓库并不方便，那么，用户可以创建一个本地仓库供私人使用。本节介绍如何使用本地仓库。 docker-registry 是官方提供的工具，可以用于构建私有的镜像仓库。

5.2.1、安装并运行本地 registry 仓库

# 运行本地仓库容器
docker run -dit --name registry --restart=always -p 5000:5000 registry

# 修改docker配置文件
vi /etc/sysconfig/docker
# 添加
# 172.31.36.113 为宿主机局域网IP地址，请自行更改为自己的IP地址
ADD_REGISTRY='--add-registry 172.31.36.113:5000'
INSECURE_REGISTRY='--insecure-registry 172.31.36.113:5000'

vi /etc/docker/daemon.json
# 添加
# 注意上一行后面需增加 , 
"insecure-registries":["172.31.36.113:5000"]

5.2.2、查看仓库中镜像

curl 172.31.36.113:5000/v2/_catalog

6、数据管理

6.1、数据卷

数据卷是一个可供一个或多个容器使用的特殊目录，它绕过 UFS，可以提供很多有用的特性：

数据卷可以在容器之间共享和重用
对数据卷的修改会立马生效
对数据卷的更新，不会影响镜像
卷会一直存在，直到没有容器使用

数据卷的使用，类似于 Linux 下对目录或文件进行 mount。

6.1.1、创建数据卷

# 创建一个名为 new-volume 的数据卷
docker volume create new-volume
# 列出
docker volume ls

# 查看卷信息
docker volume inspect new-volume

6.1.2、挂载数据卷

# 启动一个挂载新卷的nginx容器
docker run -dit --name nginx -v new-volume:/mnt nginx

# 查看容器的详细信息 grep -a 这里显示具体前后行数
 docker inspect nginx|grep Mounts -a5

6.1.3、删除数据卷

数据卷是被设计用来持久化数据的，它的生命周期独立于容器，Docker 不会在容器被删除后自动删除数据卷，并且也不存在垃圾回收这样的机制来处理没有任何容器引用的数据卷。如果需要在删除容器的同时移除数据卷。可以在删除容器的时候使用 docker rm -v 这个命令。

docker volume rm new-volume

无主的数据卷可能会占据很多空间，要清理请使用以下命令

docker volume prune

6.2、挂载主机目录

6.2.1、挂载主机目录作为数据卷

也可以使用 -v 标记也可以指定挂载一个本地主机的目录到容器中去。

# 在root目录下创建web目录，并创建index.html文件
mkdir /root/web
echo "This is mount root dir" > /root/web/index.html

# 运行挂载主机目录 /root/web/ 到 容器/usr/share/nginx/html
docker run -dit --name nginxweb -p 8088:80 -v /root/web/:/usr/share/nginx/html/ nginx

# curl查看内容
curl localhost:8088

上面的命令加载主机的 /root/web 目录到容器的** /usr/share/nginx/html 目录。
这个功能在进行测试的时候十分方便，比如用户可以放置一些程序到本地目录中，来查看容器是否正常工作。本地目录的路径必须是绝对路径，如果目录不存在 Docker 会自动为你创建它。
注意：Dockerfile 中不支持*这种用法，这是因为 Dockerfile 是为了移植和分享用的。然而，不同操作系统的路径格式不一样，所以目前还不能支持。
Docker 挂载数据卷的默认权限是读写，用户也可以通过 :ro 指定为只读。

docker run -dit --name nginxweb -p 8088:80 -v /root/web/:/usr/share/nginx/html/:ro nginx

6.2.2、挂载主机文件作为数据卷

同样，使用 -v 来进行挂载，从主机挂载单个文件到容器

# 创建演示文件
echo "This is mount single file" > /root/index.html

# 创建挂载单个主机文件的容器
docker run -dit --name nginxweb -p 8088:80 -v /root/index.html:/usr/share/nginx/html/index.html nginx

6.3、容器间共享卷

在这个案例中，将使用三个nginx容器进行演示，分别为 nginxweb1、nginxweb2、nginxweb3


# 启动nginxweb1 创建共享卷
docker run -dit --name nginxweb1 -p 8081:80 -v /usr/share/nginx/html nginx
# 启动nginxweb2、nginxweb3 并指定共享卷为 nginxweb1
docker run -dit --name nginxweb2 --volumes-from nginxweb1 -p 8082:80 nginx
docker run -dit --name nginxweb3 --volumes-from nginxweb1 -p 8083:80 nginx
# curl测试
curl localhost:8081
curl localhost:8082
curl localhost:8083
# 进入nginxweb1 进行修改index.html文件，并测试
docker exec -it nginxweb1 bash
容器内操作
echo "This is nginxweb1 website" > /usr/share/nginx/html/index.html
# 退出容器测试 exit
curl localhost:8081
curl localhost:8082
curl localhost:8083

7、网络管理

7.1、映射网络

容器中可以运行一些网络应用，要让外部也可以访问这些应用，可以通过 -P 或 -p 参数来指定端口映射。
当使用 -P 标记时，Docker 会随机映射一个 49000~49900 的端口到内部容器开放的网络端口。
使用 docker ps 可以看到，本地主机的 49153 被映射到了容器的 80 端口。此时访问本机的 49153 端口即可访问容器内 web 应用提供的界面。


# 指定映射
docker run -dit --name nginxweb1 -p 8081:80 nginx

# 随机映射
docker run -dit --name nginxweb4 -P nginx

# 查看所有容器
docker ps -a

7.2、使用主机网络

可以使用 --network=host 参数来直接使用宿主机网络

docker run -dit --network=host --name nginx-host nginx

7.3、容器互联

7.3.1、新建网络

-d 参数指定 Docker 网络类型，有 bridge overlay 。其中 overlay 网络类型用于 Swarm mode，在本小节中你可以忽略它。

docker network create -d bridge new-net

7.3.2、容器互联

docker run -dit --name box1 --network new-net busybox
docker run -dit --name box2 --network new-net busybox

7.3.3、ping测试

# 进入box1容器
docker exec -it box1 sh

# ping测试
ping box2

7.4、自定义docker0网络

7.4.1、移除原有网络

7.4.1.1、安装btctl命令

yum install -y bridge-utils

7.4.1.2、查询网桥信息

brctl  show

7.4.1.3、关闭docker服务

systemctl stop docker

7.4.1.4、停止docker0网桥

ip link set dev docker0 down

7.4.1.5、删除docker0网桥

brctl delbr docker0

7.4.1.6、查询所有网桥信息

brctl show

7.4.2、自定义新网络

7.4.2.1、创建新网桥bridge0

brctl addbr bridge0

7.4.2.2、查询创建的bridge0

brctl show

7.4.2.3、设置新网桥的网络段地址

ip addr add 192.168.0.1/24 dev bridge0

7.4.2.4、启动bridge0网桥

ip link set dev bridge0 up

7.4.2.5、查询bridge0网桥信息

ifconfig bridge0

7.4.2.6、添加新网桥到配置文件

vi /etc/sysconfig/docker
# 添加 -b=bridge0 到 OPTIONS 中
OPTIONS='-b=bridge0'

7.4.2.7、加载配置文件，重启docker服务

systemctl daemon-reload
systemctl restart docker

7.4.3、使用新网络部署应用

7.4.3.1、创建一个nginx容器

docker run -dit --name nginx-net nginx

7.4.3.2、查看容器的状态

docker ps -a

7.4.3.3、查看容器的bridge

docker inspect -f {{.NetworkSettings.Networks.bridge}} nginx-net

8、Nginx负载均衡(轮询)

8.1、创建测试网页

# 创建web1文件夹 以及页面
mkdir /root/web1
echo "This is website 1 Page 11111111111111111" > /root/web1/index.html

# 创建web2文件夹 以及页面
mkdir /root/web2
echo "This is website 2 Page 22222222222222222" > /root/web2/index.html

8.2、创建测试容器 nginx1 nginx2

# 创建nginx1容器挂载 web1页面
docker run -dit --name nginx1 -v /root/web1/index.html:/usr/share/nginx/html/index.html -p 8091:80 nginx
# 创建nginx2容器挂载 web2页面
docker run -dit --name nginx2 -v /root/web2/index.html:/usr/share/nginx/html/index.html -p 8092:80 nginx

8.3、创建nginx.conf配置文件

:::info
配置文件中的 172.31.36.113 为宿主机的内网IP地址，请按实际情况修改
:::

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;




events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;


    include /etc/nginx/conf.d/*.conf;
# 定义Tomcat服务的负载均衡
upstream tomcat_alb {

    server 172.31.36.113:8091;

    server 172.31.36.113:8092;

}
server {
    listen       80;
    server_name  172.31.36.113;#宿主机的内网IP

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;
# 这个是调用Tomcat服务的负载均衡
    location / {
        proxy_pass http://tomcat_alb;
    }
}
}

8.4、创建nginx负载均衡容器

# -v 标记挂载本地的 /root/nginx.conf 文件
docker run -dit --name nginx-alb -v /root/nginx.conf:/etc/nginx/nginx.conf -p 80:80 nginx

8.5、负载测试

# 循环测试
for i in `seq 1 6`;do curl 172.31.36.113;done

Install Minikube on your Ubuntu 18.04(AWS)

Maverick Fung — Sat, 11 Jun 2022 13:28:35 +0000

Step1.Lanuch EC2 Instance

1.Choose Image(Ubuntu 18.04)

2.Choose Instance Type(T3.medium)

3.Add Storage(Least 10GiB)

4.Add Tags(Options)

5.Add SecurityGroup

6.Check and lanuch

7.View Your Instance

Step2.Login into your Instance

1.Open SecureCRT Connect Tool

2.Add .PEM key

3.Login use ubuntu User

Step3.Install Kubectl(Use ubuntu User)

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Step4.Install Docker(Use ubuntu User)

sudo apt-get update && sudo apt-get install docker.io -y
sudo systemctl start docker

Step5.Install and check Minikube

1.Install conntrack(Root User)

sudo -i
apt install conntrack

2.Install Minikube

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Step6.Start and check Minikube

minikube start --vm-driver=none
minikube status

AWS uses EC2 to reduce the training cost of deep racer. Practical operation of deepracer-for-cloud

Maverick Fung — Thu, 19 May 2022 08:20:37 +0000

AWS uses EC2 to reduce the training cost of deep racer. Practical operation of deepracer-for-cloud

The operation video here:https://youtu.be/ADbhcXwQzDY

The official AWS Deepracer-for-Cloud blog document：
The direct link：https://aws.amazon.com/cn/blogs/china/use-amazon-ec2-to-further-reduce-the-cost-of-deepracer-training/

comes first

** As the blog has written a specific program, here will be extracted from the script, and the corresponding problems to solve **

Deep Learning AMI (Ubuntu 18.04) Version 60.1 is used

Example preparation, I see the article mentioned using G and P system instances, as follows:

g4dn.2xlarge：Cost-effective training method, based on GPU acceleration, training speed slightly faster than DeepRacer Console training
p3.2xlarge：Training speed is much faster than DeepRacer Console training, fast iteration model, achieve training results

g4dn.2xlarge is used again in this practice
So I'm using the spot request instance instead of the on-demand, which reduces the cost by 70% or more,But also can appear some problems, it is US - area of the US-EAST-1 G and P series of serious shortage of instance, may be running for a while will be stopped, so we could start with the basic environment deployment good, direct Make AMI, so can quickly pull up instances when used in the back, do not need to be on the basis of environment too much

Note that you need to check your EC2 limit, by default there is no capacity for G and P series, you need to submit the case to raise limit, do not submit too many requests when submitting the limit, it may be rejected by you

For convenience, I made an EC2 startup template here, which is also a foundation for the subsequent training, so as not to have to manually configure each time

Article directory structure:

One、Create IAM roles for EC2
- 1.Access the IAM console
- 2.Select EC2 and click Next
- 3.Add permissions
- 4.Name, view, and create
- 5.View the created role
Two、Create a storage bucket
- 1.Access the S3 console
- 2.Create a storage bucket
- 3.View the created storage bucket
Three、Create a Lanuch template
- 1.Define the template name
- 2.Select a mirror
- 3.Set the instance type and key pair
- 4.Define subnets and security groups
- 5.Configure storage
- 6.Advanced details
- 7.View the summary and create a template
Four、Create an instance
- 1.Select start the instance from the template
- 2.View spot requests
Five、Connect instances and build the base environment
- Basic.Connect to your instance
- ①.SecureCRT Imports the key
- ②.SecureCRT connection example
- Next, the setup of the base environment
- Step-1.Pull code from Github
- Step-2.Install the base components required for DeepRacer local training
- Error scenario 1, solution
- Error scenario 2, solution
- Step-3.Reconnect the EC2 instance and execute the second-stage environment initialization code
- Step-4.Load the scripts required to train DeepRacer
- Step-5.Edit reward function, training information, vehicle information
- Step-6.Edit the environment file run.env
- ①.Add bucket information
- ②.Edit track information
- Step-7.Update the Python version
- Step-8.Update the configuration
- Step-9.Upload dr-upload-custom-files to S3 buckets
- Step-10.Start training
Six、Follow-up Operations (retraining)
- 1.If the instance terminates, pull the instance again and continue the last training
- ①.Modify the run.env file
- ②.Updates the configuration to take effect
- ③.If you have modified the custom_files file, run the following command to upload the custom_files file again
- ④.start training
Seven、Parameter and command definition
Eight、Problems encountered
- 1.NO PUBKEY
- 2.Python3.6 fails and the Python version is updated
- 3.The lock cannot be obtained
- 4.Sagemaker is not running

The following is the specific operation scheme

One、Create IAM role for EC2

As stated in the official AWS blog, we use EC2 to train models using the following three services

AmazonKinesisVideoStreams
CloudWatch
S3

So here is a demonstration of creating a new EC2 role and empowering it

1. Access the IAM console

IAM_console:https://us-east-1.console.aws.amazon.com/iamv2/home#/roles/create?step=selectEntities

2. Select EC2 and click Next

3. Add permissions

Search S3, cloudwatch and Amazon kinesisvideostreams in turn, as shown in the following figure
You need to select the end of fullaccess to represent full access

After adding all three permissions, click next

4. Name, view, and create

①.Setting the Role Name

②.View the additional rights of a role

③.Ready to create

Whether to add labels according to their own needs (optional)

5.View created roles

Two、Creating a Storage Bucket

1.Accessing the S3 Console

S3 Bucket control panel:https://s3.console.aws.amazon.com/s3/

2.Creating a Storage Bucket

Here, you only need to set the bucket name and area, and other settings can remain the default

Slide to the bottom and click Create bucket

3.View the created storage bucket

Three、Creating a Lanuch Template

EC2启动模板控制面板:https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchTemplates

1.Defining template names

2.First you need to select the mirror

Please select Deep Learning AMI(Ubuntu 18.04) image

3.Set the instance type and key pair

Although the official tip is not to include these two parameters in the template, this experiment is to save time for multiple deployments later, so it is defined in the template in advance

4.Define subnets and security groups

Since I need to use spot requests this time, I will not set the subnet. Later spot will make requests in any area of US-EAST-1

5.Configure storage

The configuration instructions given by the official blog are: the root volume must be at least higher than 150gib. Here, it is set to 160gib. We can also see that AMI has a storage volume with 225gb

6.Advanced details

Here I checked the request spot instance. In order to reduce the cost, you can check it according to the user's own situation
Then, in the Iam role, we need to select the role we created in advance for EC2

7.View summary and create template

Check the created template and click create if there is no error

Four、Create Instance

EC2 instance console:https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances

1.Select Lanuch instance from template

Here, select the template you created earlier

After confirmation, start the instance

View successfully created instances
If you check the spot request instance, and an error is reported when starting the instance, indicating that there is no spot quota, and you are ensuring that your account has sufficient limit, there may be no spot instances available for you in this period of time in this area. Please use the on-demand instance to create

2.View spot requests

If you checked the spot request instance in the instance template and started it successfully, you can see your instance in the spot request control panel

Five、Connect instances and build the underlying environment

Here is the installation procedure given in the official AWS blog
I'd like to show you how to connect EC2. I use the SecureCRT connection tool here

Basic. Connect your instance

Here only supports the use of PEM type key (shared with openssh), if you are For PPK type, please use putty to connect

①.SecureCRT import key

②.SecureCRT connection instance

Please note that since we are using the image of Ubuntu, the user name used here is Ubuntu

After connecting, if a prompt box pops up, please click accept and save

Next, enter the construction of the basic environment

The installation and training script of AWS deep racer for cloud is as follows

Step-1. Pull code

Enter the created EC2 instance and execute the following command to pull the code from GitHub:

git clone https://github.com/aws-deepracer-community/deepracer-for-cloud.git

Step-2. Install the basic components required for deep Racer local training

Execute the environment pre configuration code in the first stage, which will install the basic components required for deeperacer local training, and then restart the EC2 instance:
There are only 2 lines of code officially given in this step, so in the actual implementation process, some new problems are derived due to the change of version

cd deepracer-for-cloud && ./bin/prepare.sh
sudo reboot

Because the scenario that will make mistakes is considered here, additional operations are carried out here
The actual error reporting scenario 1 is posted here to prompt No_ PUBKEY

Error scenario 1, solution

### Add pubkey
wget -qO - http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub | sudo apt-key add -
### update
sudo apt-get update
### Re execute. Please make sure you are not in the deep for cloud directory. Please go back to the home directory of Ubuntu
cd deepracer-for-cloud && ./bin/prepare.sh
### reboot
sudo reboot

If the following progress bar appears, it indicates that the error has been fixed and the basic environment is being installed

Error scenario 2, solution

If the progress bar does not appear and the following errors appear, it may appear many times

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Please run the code solution and re execute the basic environment installation command

sudo rm /var/lib/dpkg/lock-frontend
sudo rm /var/lib/dpkg/lock
### Re execute. Please make sure you are not in the deep for cloud directory. Please go back to the home directory of Ubuntu
cd deepracer-for-cloud && ./bin/prepare.sh
### Reboot
sudo reboot

Step-3. Reconnect the EC2 instance and execute the environment initialization code in the second stage

### Please make sure you are in the home directory of Ubuntu at this time
cd deepracer-for-cloud/ && bin/init.sh -c aws -a gpu

At this time, the environment initialization needs to pull many container images. Please wait patiently for completion

The environment initialization is completed as follows

Step-4. Load scripts required for training deepracer

source bin/activate.sh

Step-5. Edit reward function, training information and car information

There is a custom in the deep for cloud directory_ The files directory contains three files:

reward_function.py #Reward function file
hyperparameters.json #Training information file
model_metadata.json #Car information file

Edit reward function in dealer-for-cloud/custom_files/reward_function.py file

    def reward_function(params):
    '''
    Example of penalize steering, which helps mitigate zig-zag behaviors
    '''

    # Read input parameters
    distance_from_center = params['distance_from_center']
    track_width = params['track_width']
    steering = abs(params['steering_angle']) # Only need the absolute steering angle

    # Calculate 3 marks that are farther and father away from the center line
    marker_1 = 0.1 * track_width
    marker_2 = 0.25 * track_width
    marker_3 = 0.5 * track_width

    # Give higher reward if the car is closer to center line and vice versa
    if distance_from_center <= marker_1:
        reward = 1
    elif distance_from_center <= marker_2:
        reward = 0.5
    elif distance_from_center <= marker_3:
        reward = 0.1
    else:
        reward = 1e-3  # likely crashed/ close to off track

    # Steering penality threshold, change the number based on your action space setting
    ABS_STEERING_THRESHOLD = 15

    # Penalize reward if the car is steering too much
    if steering > ABS_STEERING_THRESHOLD:
        reward *= 0.8

    return float(reward)

In deepracer-for-cloud/custom_files/hyperparameters.json file edit training information, such as:

    {
    "batch_size": 64,
    "beta_entropy": 0.01,
    "discount_factor": 0.995,
    "e_greedy_value": 0.05,
    "epsilon_steps": 10000,
    "exploration_type": "categorical",
    "loss_type": "huber",
    "lr": 0.0003,
    "num_episodes_between_training": 20,
    "num_epochs": 10,
    "stack_size": 1,
    "term_cond_avg_score": 350.0,
    "term_cond_max_episodes": 1000,
    "sac_alpha": 0.2
  }

Edit vehicle information in deepracer-for-cloud/custom_files/model_metadata.json file, including action space, sensor and neural network type, for example:

    {
    "action_space": [
        {
            "steering_angle": -30,
            "speed": 0.6
        },
        {
            "steering_angle": -15,
            "speed": 0.6
        },
        {
            "steering_angle": 0,
            "speed": 0.6
        },
        {
            "steering_angle": 15,
            "speed": 0.6
        },
        {
            "steering_angle": 30,
            "speed": 0.6
        }
    ],
    "sensor": ["FRONT_FACING_CAMERA"],
    "neural_network": "DEEP_CONVOLUTIONAL_NETWORK_SHALLOW",
    "training_algorithm": "clipped_ppo",
    "action_space_type": "discrete",
    "version": "3"
}

Step-6. Edit the environment file run env

①. Add bucket information

Edit deepracer-for-cloud/run.env file, add as follows:

DR_LOCAL_S3_BUCKET=<YOU_CREATED_BUCKET_NAME>
DR_UPLOAD_S3_BUCKET=<YOU_CREATED_BUCKET_NAME>

You can also use the command, please put <YOU_CREATED_BUCKET_NAME> replace with the name of the S3 bucket you created earlier

sed -i '1i\DR_LOCAL_S3_BUCKET=<YOU_CREATED_BUCKET_NAME>' run.env
sed -i '1i\DR_UPLOAD_S3_BUCKET=<YOU_CREATED_BUCKET_NAME>' run.env

The bucket name I created here is: deepracer-demo-bk, as a demonstration

②.Edit track information

I use re: invent 2018 track here, and its DR_WORLD_NAME is reinvent_ base
Please in run.env file found DR_WORLD_NAME modify its value

Step-7. Update Python version

Since the image comes with Python 3.6, it is no longer supported here, so you need to update the python version
Please execute the python 3 command

### install python 3.8
sudo apt-get -y install python3.8

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2

Step-8. Update configuration

### Execute the following code to make the configuration here effective
dr-update

Step-9. Upload dr-upload-custom-files to S3 bucket

dr-upload-custom-files

At this time, the custom files in the S3 bucket should contain the following files

Step-10. Start training

Execute the following command to start training

dr-start-training

Normal training is shown in the figure

At this point, your model has started training on EC2

Six、Follow-up Operations (retraining)

1.If the instance terminates, pull the instance again, and then the last training

Please do the following

①.Edit run.env file

Modify the following parameters

### This is the last training storage folder. Please check the folder name in S3 bucket
DR_LOCAL_S3_MODEL_PREFIX=<Directory of S3 storage bucket for this training>
### Determine whether the training or assessment should be based on the model created in the previous session
DR_LOCAL_S3_PRETRAINED=True
### Set the directory of this training
DR_LOCAL_S3_PRETRAINED_PREFIX=<Directory of S3 bucket stored in the last training>
### This training defaults to last from the checkpoint setting of the previous training
DR_LOCAL_S3_PRETRAINED_CHECKPOINT=best

This is before the modification

This is a modified version

Take a look at the files in the S3 bucket

②.Update to make this configuration effective

dr-update-env

③.If you modify custom_files, please execute the following command to upload custom_files again

dr-upload-custom-files

④.Start training

dr-start-training

If sagemaker is not running is prompted, execute dr-start-training -w

dr-start-training -w

Seven、run.evn configuration file parameter setting, dr command interpretation

Refer to for specific parameters
Deepracer-for-Cloud GitHub

Eight、Problems encountered

1.NO_PUBKEY

W: GPG error: http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
E: The repository 'http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Treatment scheme
Due to the use of Ubuntu 18.04
So use the following solution

wget -qO - http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub | sudo apt-key add -

sudo apt-get update

2.Python3. 6. Report an error and update the python version

sudo apt-get install python3.8
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2

Note:

The first parameter --install indicates that the service name is registered with update-alternatives.


The second parameter is to register the final address. After success, the command will be used as the soft chain of real commands at this fixed destination address. Later management is to manage this soft chain;

( --install link name path priority)

Where link is the public link directory of software with the same function in the system, such as /usr/bin/java (absolute directory is required); Name is the name of the command linker. For example, Java path is the directory where you want to use the new command and new software. Priority is the priority. When the command link already exists, it needs to be higher than the current value, because when alternative is in automatic mode, the system enables the link with high priority by default# Integer the priority set according to the version number (the changed priority needs to be greater than the current one)

The third parameter: service name, which will be used as the association basis for future management.

The fourth parameter is the absolute path of the managed command.

The fifth parameter is priority. The higher the number, the higher the priority.

3.Unable to obtain lock

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

resolvent:

$ sudo rm /var/lib/dpkg/lock-frontend
$ sudo rm /var/lib/dpkg/lock

4.If `sagemaker is not running` is prompted, execute `dr-start-training -w`

dr-start-training -w

I hope this basic tutorial can help you

Spark programming basics (Python version)

Maverick Fung — Tue, 29 Mar 2022 08:20:15 +0000

1.Master the installation and environment configuration of spark

2.Master the installation and environment configuration of spark

3.Master Python version management and third-party installation under Ubuntu

4.Master the synchronous connection between pychar and Ubuntu under Windows

5.Master the data read by spark from the file system

Reference website:

https://spark.apache.org/docs/1.1.1/quick-start.html

1.Write first: experimental environment

You Must be have Hadoop environment,You can read my other blog:Click Here


Operating system: Ubuntu 16 04；
Spark version: 2.4.6;
Hadoop version: 2.7.2.
Python version: 3.5.

Click Here Download:spark-2.4.6-bin-without-hadoop.tgz

2.Master the installation and environment configuration of spark

1. Unzip the spark package and move it

ubuntu@adserver:~$ tar zxf spark-2.4.6-bin-without-hadoop.tgz 
ubuntu@adserver:~$ ls -lh
total 536M
-rw-rw-r--  1 ubuntu ubuntu 203M Mar 28 09:59 hadoop-2.7.2.tar.gz
-rw-rw-r--  1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz
-rw-rw-r--  1 ubuntu ubuntu  458 Mar 28 13:31 jps
drwxr-xr-x 13 ubuntu ubuntu 4.0K May 30  2020 spark-2.4.6-bin-without-hadoop
-rw-rw-r--  1 ubuntu ubuntu 161M Mar 26 19:34 spark-2.4.6-bin-without-hadoop.tgz
ubuntu@adserver:~$ sudo mv spark-2.4.6-bin-without-hadoop /usr/local/
ubuntu@adserver:~$ cd /usr/local/
ubuntu@adserver:/usr/local$ sudo mv spark-2.4.6-bin-without-hadoop/ spark
ubuntu@adserver:/usr/local$ ls -lh spark/

2.Modify spark environment variable file

ubuntu@adserver:~$ cd /usr/local/spark/conf/
ubuntu@adserver:/usr/local/spark/conf$ pwd
/usr/local/spark/conf
ubuntu@adserver:/usr/local/spark/conf$ cp spark-env.sh.template spark-env.sh
ubuntu@adserver:/usr/local/spark/conf$ vi spark-env.sh

Edit spark env SH file (vi ./conf/spark-env.sh), add the following configuration information on the first line:

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop-2.7.2/bin/hadoop classpath)

With the above configuration information, spark can store data in Hadoop distributed file system HDFS or read data from HDFS. If the above information is not configured, spark can only read and write local data, but cannot read and write HDFS data. After configuration, it can be used directly without running the startup command like Hadoop. Verify whether spark is successfully installed by running the example provided by spark.
During execution, a lot of running information will be output, and the output result is not easy to find. It can be filtered through the grep command (2 > & 1 in the command can output all information to stdout, otherwise it will be output to the screen due to the nature of the output log):

ubuntu@adserver:/usr/local/spark$ ./bin/run-example SparkPi 2>&1 | grep "Pi is"

Modify the contents of /usr/local/spark/bin/pyspark file
Change line 45 Python to python3

Execute commandsudo update-alternatives --install /usr/bin/python python /usr/bin/python3 150

ubuntu@adserver:/usr/local/spark/bin$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 150

3.Master Python version management and third-party installation under Ubuntu

whereis python3  #  Confirm Python 3 directory
cd  /usr/lib/python3.5   # Switch directory 
sudo apt-get install python3-pip   #  install pip 
sudo pip3 install -i https://pypi.doubanio.com/simple matplotlib  #  install matplotlib

ubuntu@adserver:~$ whereis python3
python3: /usr/bin/python3 /usr/bin/python3.5 /usr/bin/python3.5m /usr/lib/python3 /usr/lib/python3.5 /etc/python3 /etc/python3.5 /usr/local/lib/python3.5 /usr/share/python3 /usr/share/man/man1/python3.1.gz
ubuntu@adserver:~$ cd /usr/lib/python3.5/
ubuntu@adserver:/usr/lib/python3.5$ sudo apt-get install python3-pip
ubuntu@adserver:/usr/lib/python3.5$ sudo pip3 install -i https://pypi.doubanio.com/simple matplotlib

4.Master the synchronous connection between pychar and Ubuntu under Windows

1) Open pychart and open file -- > settings -- >

Click the + sign, and then select SSH interpreter to set the server; Enter the IP address, user name and password of the virtual machine Ubuntu

5.Master the data read by spark from the file system

1）Read the local file "/home/Hadoop/test.txt" of Linux system in pyspark, and then count the number of lines of the file;

First create the test file

$ vi /home/hadoop/test.txt
this is line 1
this is line 2
this is line 3
this is line 4

ubuntu@adserver:/usr/local/spark/bin$ pwd
/usr/local/spark/bin
ubuntu@adserver:/usr/local/spark/bin$ ./pyspark 
Python 3.5.2 (default, Jan 26 2021, 13:30:48) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/03/28 15:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Python version 3.5.2 (default, Jan 26 2021 13:30:48)
SparkSession available as 'spark'.
>>> lines = sc.textFile("file:/home/hadoop/test.txt")
>>> lines.count()
4
>>>

2）Read the HDFS system file "/user/Hadoop/test.txt" (if the file does not exist, please create it first), and then count the number of lines of the file;

ubuntu@adserver:~$ cd /home/hadoop/
ubuntu@adserver:/home/hadoop$ hadoop fs -ls /
Found 1 items
drwxrwx---   - ubuntu supergroup          0 2022-03-28 17:15 /tmp
ubuntu@adserver:/home/hadoop$ hadoop fs -mkdir -p /user/hadoop
ubuntu@adserver:/home/hadoop$ hadoop fs -put test.txt /user/hadoop/
ubuntu@adserver:/home/hadoop$ hadoop fs -ls /user/hadoop/
Found 1 items
-rw-r--r--   3 ubuntu supergroup         60 2022-03-28 17:17 /user/hadoop/test.txt
ubuntu@adserver:/home/hadoop$ /usr/local/spark/bin/pyspark 
>>> lines = sc.textFile("/user/hadoop/test.txt")     
>>> lines.count()

3）Write an independent application, read the HDFS system file "/user/Hadoop/test.txt" (if the file does not exist, please create it first), and then count the number of lines of the file;

ubuntu@adserver:/home/hadoop$ sudo vi st-app.py

from pyspark import SparkContext

logFile = "/user/hadoop/test.txt"  # Should be some file on your system
sc = SparkContext("local", "Simple App")
logData = sc.textFile(logFile).cache()
print("The HDFS file lines : ",logData.count())

ubuntu@adserver:/home/hadoop$ /usr/local/spark/bin/spark-submit --master local[4] st-app.py 2>&1 | grep "The HDFS"

Hey,If you think this blog is useful, please like it and Follow Me. Thank you.Have a nice day!!!!!!

Build Hadoop 2.X Fully distributed Environment by Ubuntu 16.04

Maverick Fung — Tue, 29 Mar 2022 01:30:22 +0000

1.Experimental model and architecture description

2.Basic environment configuration

3.JDK Hadoop environment configuration

4.Hadoop configuration file modification

5.Start Hadoop cluster (all in AdServer operation)

1.Experimental model and architecture description

Written in the front, these pictures are from another blog of mine. Don't worry

My Chinese Blog:CSDN Address

This experiment uses three Ubuntu 16 04 instance

HostName	IPAddress	OS	Running Services	Role
adserver	192.168.200.10	ubuntu 16.04	NameNode、SecondaryNameNode、ResourceManager、JobHistoryServer	Master
monserver	192.168.200.20	ubuntu 16.04	DataNode、NodeManager	Slave
osdserver-1	192.168.200.30	ubuntu 16.04	DataNode、NodeManager	Slave

2.Basic environment configuration

Change Hostname and the network interface name

Ubuntu 16.04 static IP address only

Check the IP address first

Modify the network card configuration file to make it a static IP address. After modification, restart the instance to take effect
$ sudo vi /etc/network/interfaces
Modify host name, use here

$ sudo hostnamectl set-hostname YOUR_HOSTNAME

Modify hosts and configure FQDN domain name

Create Hadoop users and configure password free login. Each node needs to be configured

sudo useradd -d /home/hadoop -m hadoop 
sudo passwd hadoop 
echo "hadoop ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/hadoop 
sudo chmod 0440 /etc/sudoers.d/hadoop

To configure SSH password free login, you need to install openssh-server(sudo apt get install openssh-server) first

ubuntu@adserver:~$ ssh-keygen 
ubuntu@adserver:~$ ssh-copy-id adserver
ubuntu@adserver:~$ ssh-copy-id monserver
ubuntu@adserver:~$ ssh-copy-id osdserver-1

ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@adserver  
ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@monserver  
ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@osdserver-1

3.JDK Hadoop environment

1. Configure JDK environment. Jdk-8u77 is used here

Download JDK jdk-8u77-linux-x64.tar.gz
https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html

ubuntu@adserver:~$ ls -lh
total 173M
-rw-rw-r-- 1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz
ubuntu@adserver:~$ tar -zxf jdk-8u77-linux-x64.tar.gz 
ubuntu@adserver:~$ ls -lh
total 173M
drwxr-xr-x 8 ubuntu ubuntu 4.0K Mar 21  2016 jdk1.8.0_77
-rw-rw-r-- 1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz

ubuntu@adserver:~$ sudo mkdir /usr/lib/jdk      
ubuntu@adserver:~$ sudo mv jdk1.8.0_77/ /usr/lib/jdk/
ubuntu@adserver:~$ sudo ls /usr/lib/jdk/
jdk1.8.0_77

ubuntu@adserver:~$ sudo vi /etc/profile 
Add JDK environment
#JDK
export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77 
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH

2. Configure Hadoop environment

Download HadoopHadoop-2.7.2
Download Link：
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz

Unzip：

ubuntu@adserver:~$ tar -zxf hadoop-2.7.2.tar.gz 
ubuntu@adserver:~$ ls -lh

Move Hadoop to /usr/local/ directory

Add Hadoop environment variable

$ sudo vi /etc/profile

#HADOOP
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

$ vi ~/.bashrc
#HADOOP
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

$ source /etc/profile
$ source ~/.bashrc
$ hadoop version

4.Hadoop configuration file modification

Modify Hadoop configuration file

Modify the /hadoop-2.7.2/etc/hadoop directory hadoop-env.sh、yarn-env.sh、slaves、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml

1. Create the tmp folder and its subdirectories under the Hadoop directory

ubuntu@adserver:~$ sudo mkdir -p /usr/local/hadoop-2.7.2/tmp/dfs/data
ubuntu@adserver:~$ sudo mkdir -p /usr/local/hadoop-2.7.2/tmp/dfs/name

Modify profile:

First enter the corresponding folder

ubuntu@adserver:~$ cd /usr/local/hadoop-2.7.2/etc/hadoop/

2. Add Java_ Home to Hadoop environment profile

① Add Java home to Hadoop env sh

export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77

② Add Java home to yarn env SH, just add it directly in the first line

export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77

③ Add slave hostname to slave

ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi slaves 
monserver
osdserver-1

3. Modify the corresponding configuration file

④Modification core-site.xml

ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi core-site.xml

Add the following contents to <configuration></configuration>

<property>
                <name>fs.defaultFS</name>
                <value>hdfs://adserver:8020</value>
       </property>
       <property>
               <name>hadoop.tmp.dir</name>
               <value>file:/usr/local/hadoop-2.7.2/tmp</value>
               <description>Abase for other temporary   directories.</description>
       </property>

⑤Modify hdfs-site.xml file,

ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi hdfs-site.xml

Add the following contents to <configuration></configuration>

       <property>
                <name>dfs.namenode.secondary.http-address</name>
               <value>adserver:50090</value>
       </property>
     <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop-2.7.2/tmp/dfs/name</value>
       </property>
      <property>
              <name>dfs.datanode.data.dir</name>
              <value>file:/usr/local/hadoop-2.7.2/tmp/dfs/data</value>
       </property>
       <property>
               <name>dfs.replication</name>
               <value>3</value>
        </property>
        <property>
                 <name>dfs.webhdfs.enabled</name>
                  <value>true</value>
         </property>

⑥Modify mapred-site,xml, you need to copy the file as mapred-site.xml, and then

ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ cp mapred-site.xml.template mapred-site.xml
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi mapred-site.xml

Add the following contents to <configuration></configuration>

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>adserver:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>adserver:19888</value>
        </property>

⑦、Modify yarn-site.xml

ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi yarn-site.xml

Add the following contents to <configuration></configuration>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>adserver:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>adserver:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>adserver:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>adserver:8033</value>
  </property>
  <property>
     <name>yarn.resourcemanager.webapp.address</name>
     <value>adserver:8088</value>
  </property>

4.Use the SCP command to add /etc/hosts, /etc/profile, ~/.bashrc, JDK and Hadoop are distributed to two slave nodes respectively. Here is only a demonstration of copying to monserver

ubuntu@adserver:~$ scp /etc/hosts ubuntu@monserver:~/
ubuntu@adserver:~$ scp -r /etc/profile ubuntu@monserver:~/           
ubuntu@adserver:~$ scp -r .bashrc  ubuntu@monserver:~/                 
ubuntu@adserver:~$ scp -r /usr/local/hadoop-2.7.2/  ubuntu@monserver:~/
ubuntu@adserver:~$ scp -r /usr/lib/jdk/  ubuntu@monserver:~/

ubuntu@monserver:~$ sudo mv hosts /etc/hosts
ubuntu@monserver:~$ sudo mv hadoop-2.7.2/ /usr/local/
ubuntu@monserver:~$ sudo mv jdk/ /usr/lib/           
ubuntu@monserver:~$ sudo update-alternatives --install /usr/bin/java java /usr/lib/jdk/jdk1.8.0_77/bin/java 300 
ubuntu@monserver:~$ sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jdk/jdk1.8.0_77/bin/javac 300
ubuntu@monserver:~$ source /etc/profile
ubuntu@monserver:~$ source .bashrc     
ubuntu@monserver:~$ java -version
ubuntu@monserver:~$ hadoop version

Set the folder permissions of all nodes hadoop-2.7.2 to 0777：

ubuntu@adserver:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2
ubuntu@monserver:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2
ubuntu@osdserver-1:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2

5.Start Hadoop cluster (all in AdServer operation)

① initialization namenode

ubuntu@adserver:~$ hadoop namenode -format
Note: initialization is required for the first run, but not after.
If it runs successfully, it should return exiting with status 0 and prompt shutting down namenode at AdServer/xxx.xxx.xxx.XX (IP address of AdServer). The specific results are shown in the figure below：

②Start Hadoop daemons (namenode, datanode, ResourceManager, nodemanager, etc.)）

A、Start namenode, secondarynamenode and datanode first

Execute on the AdServer node

ubuntu@adserver:~$start-dfs.sh

At this time, the processes running on the master node are：NameNode、SecondaryNameNode

At this time, the processes running on the slave node are：DataNode

B、Start ResourceManager、NodeManager

$ start-yarn.sh

Yarn is separated from MapReduce and is responsible for resource management and task scheduling. Yarn runs on MapReduce and provides high availability and scalability

At this time, the processes running on the master node are：NameNode、SecondaryNameNode、ResourceManager

The processes running on the slave node are：DataNode、NodeManager

C、Start JobHistoryServer

$ mr-jobhistory-daemon.sh start historyserver

Note: a jobhistoryserver process will be added to the master node

After TMP and logP directories on each node are created several times, be sure to delete them again

View the operation status of three nodes

ubuntu@adserver:~$ jps
ubuntu@adserver:~$ ssh monserver "/usr/lib/jdk/jdk1.8.0_77/bin/jps"
ubuntu@adserver:~$ ssh osdserver-1 "/usr/lib/jdk/jdk1.8.0_77/bin/jps"

If this article is helpful to you, please like it and Follow Me. Thank you,Have a Nice Day!!!!!

How To SSH Your ECS Server(Running Fargate)

Maverick Fung — Wed, 12 Jan 2022 15:38:30 +0000

1.Install ACLI V2

2.Install and Run Docker On EC2

3.Use ECR create repository

4.Attach IAM Role To EC2

5.Build and Push Image to Repository

6.Create ECS Task Definition

7.Create Cluster

8.Create Service

9.Use SSH Tool connect your container

1.Install ACLI V2 On Ami-2

In here,I use Amazon Linux 2, t3.micro instance type.

First,We need give the instance right IAM Role

Install ACLI V2,I will put the code

rm -rf /bin/aws
rm -rf /bin/aws*
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
ln -s /usr/local/bin/aws /bin/aws

[root@ip-172-31-43-69 ~]# yum install docker && systemctl start docker

Use ECR create repository

Attach IAM Role To EC2

https://docs.amazonaws.cn/en_us/AmazonECR/latest/userguide/registry-permissions-create.html

Build and Push Image to Repository

1.Write Dockerfile

FROM centos:7

RUN yum install -y openssh-server sudo
RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config 
RUN useradd admin  
RUN echo "admin:123456" | chpasswd
RUN echo "admin   ALL=(ALL)       ALL" >> /etc/sudoers 

RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

2.Build Image

[root@ip-172-31-43-69 ~]# docker build -t sshd .

3.Tag and Push Image

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
docker tag sshd:latest aws_account_id.dkr.ecr.region.amazonaws.com/sshd:latest
docker push aws_account_id.dkr.ecr.region.amazonaws.com/sshd:latest

Create Fargate Task Definitions(I put my json code)

Notice:Replace the capital letter identification part

{
  "ipcMode": null,
  "executionRoleArn": "YOUR_EXEC_ROLE_ARN": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/sshd_task",
          "awslogs-region": "YOUR_REGION",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 22,
          "protocol": "tcp",
          "containerPort": 22
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": 2048,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "YOU_CREATE_IMG_URL",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "sshd_container"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": "USE_YOURSELF_TASK_ROLE",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "USE_YOURSELF_ARN",
  "family": "sshd_task",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 2,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

7.Create Cluster

8.Create Service

9.Use SSH Tool connect your container

Use Python+AWS Create Sentiment Analysis Application

Maverick Fung — Mon, 10 Jan 2022 14:02:31 +0000

Hi,Today I will share my experience to you,It's about Use Python And Amazon Comprehend ,and I also use Lambda,Api Gateway togather build the tool

Python Stage(I will put my code in this stage) #### Notice:I use Qtdesigner and PYUIC

# -*- coding: utf-8 -*-

# Form implementation generated from reading ui file '1.ui'
#
# Created by: PyQt5 UI code generator 5.15.6
#
# WARNING: Any manual changes made to this file will be lost when pyuic5 is
# run again.  Do not edit this file unless you know what you are doing.


import sys

import openpyxl
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtWidgets import QFileDialog, QWidget
from PyQt5.QtWidgets import QApplication
import requests
import json
import datetime
import ctypes
import images

#Change Task manager icon
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID("myappid")

filename = '1'#Waiting analysis file name
flag = 1 #analysis flag

class About_page(QWidget):
    def __init__(self):
        super(About_page, self).__init__()
        self.resize(400,300)
        self.setWindowTitle("About")
        icon = QtGui.QIcon()
        icon.addPixmap(QtGui.QPixmap(':/Image-2.ico'), QtGui.QIcon.Normal)
        self.setWindowIcon(icon)
        self.label = QtWidgets.QLabel(self)
        self.label.setGeometry(QtCore.QRect(30, 40, 161, 21))
        self.label.setObjectName("label")
        self.label_2 = QtWidgets.QLabel(self)
        self.label_2.setGeometry(QtCore.QRect(30, 75, 161, 21))
        self.label_2.setObjectName("label_2")
        self.label_3 = QtWidgets.QLabel(self)
        self.label_3.setGeometry(QtCore.QRect(30, 110, 91, 21))
        self.label_3.setObjectName("label_3")
        self.label_4 = QtWidgets.QLabel(self)
        self.label_4.setGeometry(QtCore.QRect(130, 110, 221, 21))
        self.label_4.setObjectName("label_4")

        self.retranslateUi(self)
        QtCore.QMetaObject.connectSlotsByName(self)

    def retranslateUi(self,About_page):
        _translate = QtCore.QCoreApplication.translate
        self.setWindowTitle(_translate("Form", "About"))
        self.label.setText(_translate("Form", "Copyright ownership:T-UniStar"))
        self.label_2.setText(_translate("Form", "User:Lei Feng"))
        self.label_3.setText(_translate("Form", "Powered by"))
        self.label_4.setText(_translate("Form",
                                        u'<a href="https://aws.amazon.com/cn/" style="color:#0000ff;"><b>Amazon Web Services, Inc.</b></a>'))
        self.label_4.setOpenExternalLinks(True)

class Ui_Form(object):
    def setupUi(self, Form):
        Form.setObjectName("Form")
        Form.resize(750, 548)
        Form.setFixedSize(QtCore.QSize(780, 548))
        Form.setMaximumSize(QtCore.QSize(750, 548))
        icon = QtGui.QIcon()
        icon.addPixmap(QtGui.QPixmap(':/Image-1.ico'), QtGui.QIcon.Normal)
        Form.setWindowIcon(icon)
        self.label = QtWidgets.QLabel(Form)
        self.label.setGeometry(QtCore.QRect(220, 10, 261, 61))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(26)
        self.label.setFont(font)
        self.label.setObjectName("label")
        self.label_2 = QtWidgets.QLabel(Form)
        self.label_2.setGeometry(QtCore.QRect(250, 70, 211, 21))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(12)
        self.label_2.setFont(font)
        self.label_2.setObjectName("label_2")
        self.pushButton = QtWidgets.QPushButton(Form)
        self.pushButton.setGeometry(QtCore.QRect(20, 140, 111, 28))
        self.pushButton.setObjectName("pushButton")
        self.pushButton.clicked.connect(self.RealTime_analysis)
        self.textEdit = QtWidgets.QTextEdit(Form)
        self.textEdit.setGeometry(QtCore.QRect(160, 140, 291, 31))
        self.textEdit.setObjectName("textEdit")
        self.label_3 = QtWidgets.QLabel(Form)
        self.label_3.setGeometry(QtCore.QRect(20, 100, 201, 31))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(20)
        self.label_3.setFont(font)
        self.label_3.setObjectName("label_3")
        self.label_4 = QtWidgets.QLabel(Form)
        self.label_4.setGeometry(QtCore.QRect(20, 200, 481, 31))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(18)
        self.label_4.setFont(font)
        self.label_4.setObjectName("label_4")
        self.pushButton_2 = QtWidgets.QPushButton(Form)
        self.pushButton_2.setGeometry(QtCore.QRect(40, 420, 111, 28))
        self.pushButton_2.setObjectName("pushButton_2")
        self.pushButton_2.clicked.connect(self.multWordsAy)
        self.label_5 = QtWidgets.QLabel(Form)
        self.label_5.setGeometry(QtCore.QRect(20, 250, 171, 31))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(10)
        font.setBold(True)
        font.setWeight(75)
        self.label_5.setFont(font)
        self.label_5.setObjectName("label_5")
        self.label_6 = QtWidgets.QLabel(Form)
        self.label_6.setGeometry(QtCore.QRect(20, 330, 171, 31))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(10)
        font.setBold(True)
        font.setWeight(75)
        self.label_6.setFont(font)
        self.label_6.setObjectName("label_6")
        self.label_7 = QtWidgets.QLabel(Form)
        self.label_7.setGeometry(QtCore.QRect(80, 280, 31, 51))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(20)
        font.setBold(False)
        font.setWeight(50)
        self.label_7.setFont(font)
        self.label_7.setObjectName("label_7")
        self.label_8 = QtWidgets.QLabel(Form)
        self.label_8.setGeometry(QtCore.QRect(80, 360, 31, 51))
        font = QtGui.QFont()
        font.setFamily("Arial")
        font.setPointSize(20)
        font.setBold(False)
        font.setWeight(50)
        self.label_8.setFont(font)
        self.label_8.setObjectName("label_8")
        self.label_9 = QtWidgets.QLabel(Form)
        self.label_9.setGeometry(QtCore.QRect(200, 250, 561, 31))
        self.label_9.setObjectName("label_9")
        self.textBrowser = QtWidgets.QTextBrowser(Form)
        self.textBrowser.setGeometry(QtCore.QRect(190, 410, 351, 131))
        self.textBrowser.setObjectName("textBrowser")
        self.lineEdit = QtWidgets.QLineEdit(Form)
        self.lineEdit.setGeometry(QtCore.QRect(180, 330, 301, 31))
        self.lineEdit.setObjectName("lineEdit")
        self.lineEdit.setEnabled(False)
        self.pushButton_3 = QtWidgets.QPushButton(Form)
        self.pushButton_3.setGeometry(QtCore.QRect(510, 330, 111, 31))
        self.pushButton_3.setObjectName("pushButton_3")
        self.pushButton_3.clicked.connect(self.openFile)
        self.label_10 = QtWidgets.QLabel(Form)
        self.label_10.setGeometry(QtCore.QRect(470, 110, 72, 21))
        self.label_10.setObjectName("label_10")
        self.label_11 = QtWidgets.QLabel(Form)
        self.label_11.setGeometry(QtCore.QRect(560, 110, 72, 21))
        self.label_11.setObjectName("label_11")
        self.label_12 = QtWidgets.QLabel(Form)
        self.label_12.setGeometry(QtCore.QRect(650, 110, 72, 21))
        self.label_12.setObjectName("label_12")
        self.textEdit_2 = QtWidgets.QTextEdit(Form)
        self.textEdit_2.setGeometry(QtCore.QRect(460, 140, 81, 31))
        self.textEdit_2.setObjectName("textEdit_2")
        self.textEdit_2.setEnabled(False)
        self.textEdit_3 = QtWidgets.QTextEdit(Form)
        self.textEdit_3.setGeometry(QtCore.QRect(550, 140, 81, 31))
        self.textEdit_3.setObjectName("textEdit_3")
        self.textEdit_3.setEnabled(False)
        self.textEdit_4 = QtWidgets.QTextEdit(Form)
        self.textEdit_4.setGeometry(QtCore.QRect(640, 140, 81, 31))
        self.textEdit_4.setObjectName("textEdit_4")
        self.textEdit_4.setEnabled(False)
        self.pushButton_4 = QtWidgets.QPushButton(Form)
        self.pushButton_4.setGeometry(QtCore.QRect(650, 500, 93, 28))
        self.pushButton_4.setObjectName("pushButton_4")
        self.pushButton_4.clicked.connect(self.open_About_page)

        self.retranslateUi(Form)
        QtCore.QMetaObject.connectSlotsByName(Form)

    def retranslateUi(self, Form):
        _translate = QtCore.QCoreApplication.translate
        Form.setWindowTitle(_translate("Form", "Sentiment Analysis System"))
        self.label.setText(_translate("Form", "Sentiment Analysis System"))
        self.label_2.setText(_translate("Form", "Please follow the stage work"))
        self.pushButton.setText(_translate("Form", "Start Single words Analysis"))
        self.textEdit.setHtml(_translate("Form", "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0//EN\" \"http://www.w3.org/TR/REC-html40/strict.dtd\">\n"
"<html><head><meta name=\"qrichtext\" content=\"1\" /><style type=\"text/css\">\n"
"p, li { white-space: pre-wrap; }\n"
"</style></head><body style=\" font-family:\'SimSun\'; font-size:9pt; font-weight:400; font-style:normal;\">\n"
"<p style=\" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;\"><span style=\" font-style:italic;\">Please Input Need Analysis words</span></p></body></html>"))
        self.label_3.setText(_translate("Form", "Single Words Analysis"))
        self.label_4.setText(_translate("Form", "More Words Analysis(Only support XLSX format)"))
        self.pushButton_2.setText(_translate("Form", "Start More Words Analysis"))
        self.label_5.setText(_translate("Form", "Stage 1、Format head"))
        self.label_6.setText(_translate("Form", "Stage 2、Choose File"))
        self.label_7.setText(_translate("Form", "↓"))
        self.label_8.setText(_translate("Form", "↓"))
        self.label_9.setText(_translate("Form", "The First head is：Text、Positive、Negative、Neutral、Emotion judgment、Language"))
        self.pushButton_3.setText(_translate("Form", "Choose File"))
        self.label_10.setText(_translate("Form", "Positive"))
        self.label_11.setText(_translate("Form", "Negative"))
        self.label_12.setText(_translate("Form", "Emotion Judgment"))
        self.pushButton_4.setText(_translate("Form", "About"))

    def RealTime_analysis(self):
        stat_words = self.textEdit.toPlainText()
        #print(stat_words)
        #print(data)
        url = "https://yuzcb6dtb3.execute-api.us-east-1.amazonaws.com/endpoint"
        payload = json.dumps({
            "Text": stat_words
        })
        headers = {
            'Content-Type':'application/json'
        }

        RTResponse = requests.request('POST',url,headers=headers,data=payload)
        RTRp_text = RTResponse.text

        RT_Positive = json.loads(RTRp_text)['body']['SentimentScore']['Positive']
        RT_Negative = json.loads(RTRp_text)['body']['SentimentScore']['Negative']
        RT_Neutral = json.loads(RTRp_text)['body']['SentimentScore']['Neutral']
        # print(float(str(json.loads(RTRp_text)['body']['SentimentScore']['Positive'])))
        # self.textEdit_2.setText(RT_Positive)
        # self.textEdit_3.setText(RT_Negative)
        if RT_Positive > RT_Negative and RT_Positive > RT_Neutral:
            self.textEdit_2.setText(str(float(RT_Positive) * 100)[:8])
            self.textEdit_3.setText(str(float(RT_Negative)*10)[:8])
            self.textEdit_4.setText('Postive')
        elif RT_Positive < RT_Negative and RT_Neutral < RT_Negative:
            self.textEdit_2.setText(str(float(RT_Positive)*10)[:8])
            self.textEdit_3.setText(str(float(RT_Negative)*100)[:8])
            self.textEdit_4.setText('Negative')
        else:
            self.textEdit_2.setText(str(float(RT_Positive) * 10)[:8])
            self.textEdit_3.setText(str(RT_Negative)[:8])

            self.textEdit_4.setText('Neutral')

        print(RTRp_text)


    def openFile(self):
        openFileName = QFileDialog.getOpenFileName(None,'Choose File', '', 'Excel files(*.xlsx)')
        print(openFileName)
        openF = openFileName[0].split('/')
        global filename;
        filename = openFileName[0];
        name = openF[len(openF) - 1];
        self.lineEdit.setText(str(name))


    def stopAnalysis(self):
        self.pushButton_2.setText("Start More Words Analysis")
        self.pushButton_2.clicked.disconnect(self.stopAnalysis)
        self.pushButton_2.clicked.connect(self.multWordsAy)
        self.textBrowser.append("Stopped by user：" + datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
        global flag;
        flag = -1;
        return

    def multWordsAy(self):
        #Check file exist
        global flag;
        flag = 1
        import os
        if (not os.path.exists(filename)):
            self.textBrowser.append('This file not exist！')
            return;
        self.pushButton_2.setText("Stop Analysis")
        self.textBrowser.append("Start Analysis：" + datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
        self.pushButton_2.clicked.disconnect(self.multWordsAy)
        self.pushButton_2.clicked.connect(self.stopAnalysis)

        def getMultResult(text):
            senment_words = text
            # print(stat_words)
            # print(data)
            url = "https://yuzcb6dtb3.execute-api.us-east-1.amazonaws.com/endpoint"
            mult_payload = json.dumps({
                "Text": senment_words
            })
            headers = {
                'Content-Type': 'application/json'
            }

            multResponse = requests.request('POST', url, headers=headers, data=mult_payload)
            mult_resp_text = multResponse.text
            mu_Positive = json.loads(mult_resp_text)['body']['SentimentScore']['Positive']
            mu_Negative = json.loads(mult_resp_text)['body']['SentimentScore']['Negative']
            mu_Neutral = json.loads(mult_resp_text)['body']['SentimentScore']['Neutral']
            LanguageCode = json.loads(mult_resp_text)['LanguageCode']
            return [mu_Positive,mu_Negative,mu_Neutral,LanguageCode]



        book = openpyxl.load_workbook(filename)
        sheet = book.get_sheet_by_name(book.get_sheet_names()[0])
        rows = sheet.max_row
        showText = "Read file successful，amount"+str(rows-1)+"records waiting analysis。。。\nCurrent Time："+datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        self.textBrowser.append(showText)
        if (sheet.cell(1, 1).value != 'Text' or sheet.cell(1, 2).value != 'Positive' or sheet.cell(1,3).value != 'Negative' or sheet.cell(1, 4).value != 'Negative' or sheet.cell(1, 5).value != 'Emotion judgment' or sheet.cell(1,6).value != 'Language'):
            self.textBrowser.append('The execl not stand format！')
            return;
        for i in range(2,rows+1):
            detectWords = sheet.cell(i,1).value
            detectResult = getMultResult(detectWords)
            self.textBrowser.append(f'Analysising {i-1}Record,Current Time:{datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}')
            print(f'Analysising{i-1}Record,Current Time:{datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}')
            sheet.cell(i,2,detectResult[0])
            sheet.cell(i,3,detectResult[1])
            sheet.cell(i,4,detectResult[2])
            sheet.cell(i,6,detectResult[3])
            if detectResult[0] > detectResult[1] and detectResult[0] > detectResult[2]:
                sheet.cell(i,5,"Positive")
            elif detectResult[1] > detectResult[0] and detectResult[1] > detectResult[2]:
                sheet.cell(i,5,"Negative")
            else:
                sheet.cell(i,5,"Neutral")
            QApplication.processEvents()
        book.save(filename)
        self.textBrowser.append('Analysis ended')

        self.textBrowser.append(str(rows - 1) + "Records analysis done！")
        self.pushButton_2.setText("Strat Analysis")
        self.pushButton_2.clicked.disconnect(self.stopAnalysis)
        self.pushButton_2.clicked.connect(self.multWordsAy)

    def open_About_page(self):
        self.about_form = About_page()
        self.about_form.show()




if __name__=="__main__":
    app=QtWidgets.QApplication(sys.argv)
    widget=QtWidgets.QMainWindow()
    # widget.setFixedSize(widget.width(), widget.height()-80) #ban max
    ui=Ui_Form()
    ui.setupUi(widget)
    widget.showMaximized()
    sys.exit(app.exec_())

Lambda And Api Gateway Stage #### Notice You need give right IAM

Here is the Lambda Code

import json
import boto3

comprehend = boto3.client(service_name='comprehend', region_name='ap')

def lambda_handler(event, context):
    # TODO implement
    LanguageCode = comprehend.detect_dominant_language(Text=event['Text'])
    comprehend_sentiment = comprehend.detect_sentiment(Text = event['Text'],LanguageCode=LanguageCode['Languages'][0]['LanguageCode'])
    print(LanguageCode)
    return {
        'statusCode': 200,
        'body': comprehend_sentiment,
        'LanguageCode':LanguageCode['Languages'][0]['LanguageCode']
    }

This is Apigateway(When you finished,You need deploy it and use postman to test)

The Final display

Use AWS EC2 create VPN(SSR)(Chinese and English description)

Maverick Fung — Wed, 25 Aug 2021 02:56:31 +0000

Hi,guys,Today I will show you how to use aws ec2 to create ShadowsocksR

First,You should launch a server by aws ec2

1.Launch a ec2 server

2.Choose what you want instance type

3.Open the instance security group ports,such as 22

(Notice:When the shadowsocksr is deployed later, the corresponding port will be opened)

4.Connect to you create ec2-instance

Second,Install ShadowsocksR Script

1.Login root user

[ec2-user@ip-172-31-95-56 ~]$ sudo -i

2.git and run ShadowsocksR script

wget -N --no-check-certificate https://raw.githubusercontent.com/ToyoDAdoubi/doubi/master/ssr.sh && chmod +x ssr.sh && bash ssr.sh

Congiure the ShadowSocksR config

1.First you will see this

You need type the number `1` to install ShadowSocksR

2.Choose the ShadowSocksR server port,and you need open the ec2 security group inbound port

(default: 2333)

3.Set the shadowsocksR server connect password

(Default: doub.io)

4.Choose the Encryption Mode

[Note] if auth is used_ chain_ A protocol, please select none as the encryption method, and confusion is optional (recommended)
[Note] the salsa20 / chachacha20 - * series encryption method requires additional installation (depends on libsodium, otherwise shadowsocksr cannot be started!)
(Default: 5. aes-128-ctr)

5.Choose Protocol plug-in

[Note] if auth is used_ chain_ A protocol, please select none as the encryption method, and confusion is optional (recommended)

(Default: 2. auth_sha1_v4)

6.Choose Confusion plug-in

[note] if you use shadowsocksr to accelerate the game, please select obfuscation compatible with the original or plain obfuscation, and then the client selects plain, otherwise the delay will be increased!
In addition, if you select tls1.2_ ticket_ Auth, then the client can choose tls1.2_ ticket_ Fastauth, so you can disguise without adding delay!
If you are building in popular areas such as Japan and the United States, you may have a lower chance of being blocked by the wall!

(Default: 1. plain)

7.Set Connect client numbers,Single thread speed,Single port total speed

Please enter the number of devices (auth) to be limited by the shadowsocksr account to be set_* Series protocols are incompatible (only the original version is valid)
[Note] device number limit: the number of clients that can be linked at the same time on each port (multi port mode, each port is calculated independently). It is recommended that there be at least 2.
(Default: No-limit)

Please enter the upper limit of single thread speed limit per port to be set (unit: KB / s)
[note] single thread speed limit: the upper limit of single thread speed limit on each port, and multithreading is invalid.
(Default: No-limit)

Please enter the upper limit of the total speed limit of each port to be set (unit: KB / s)
[note] total speed limit of ports: the upper limit of total speed limit of each port, and the overall speed limit of a single port.
(Default: No-limit)