DEV Community: Shingai Zivuku

From Disaster to Recovery: A Practical Case Study on Kubernetes etcd Backups

Shingai Zivuku — Sat, 14 Mar 2026 00:19:17 +0000

In Kubernetes (K8s) clusters, etcd functions as the "brain." It stores all state data for the entire cluster ranging from Pod configurations and service registrations to network policy definitions, making the cluster's stability entirely dependent on etcd. Any loss or corruption of etcd data can paralyze the cluster and severely impact business operations.

However, real world operations are fraught with unexpected events such as human error, hardware failure, and network anomalies, all of which threaten data integrity. Therefore, building a reliable backup mechanism, specifically periodic automated backups, is a critical link in ensuring K8S cluster stability and business continuity.

This article focuses on periodic etcd data backups within a K8S environment. Through a hands-on case study, we will demonstrate how to build an efficient and stable automated backup solution to help O&M personnel navigate data security challenges.

1. Backup Solution Approach

While using Ceph RBD or Object Storage for backend storage is often more effective, the operational logic remains similar. Therefore, this demonstration will utilize PV-NFS.

The core logic of this solution is to leverage the native Kubernetes CronJob controller for periodic task scheduling. The etcdctl utility performs the backup, while

vcluster.com

Persistent Volumes provide shared storage to persist backup files and certificates. Key components include:

Backup Tool: Utilizing the official etcdctl CLI to perform snapshot backups.
Scheduling Control: Defining the backup cycle (e.g., daily at 2:00 AM) via a K8S CronJob to trigger tasks automatically.
Certificate Management: Since etcd typically enables TLS encryption, the backup process must mount the CA certificate, client certificate, and private key to ensure etcdctl can authenticate with the cluster.
Storage Solution: Certificates and backup files are mounted via a PVC dynamically provisioned by the csi-nfs-storageclass, utilizing NFS shared storage for persistence.

2. Detailed Implementation Steps

Preparation: Creating Storage Paths on the NFS Server

NFS storage relies on server-side directory exports. You must first create dedicated directories for certificates and backups on the NFS server and configure the appropriate permissions.

Verify the NFS Shared Directory
In this environment, the NFS server's shared directory is /data/nfs-server (verified via the exportfs command), which is accessible by all K8S nodes.
Create Certificate and Backup Directories
Create a certificate directory (etcd-certs) and a backup directory (etcd-backup) within /data/nfs-server. Assign read/write permissions to ensure the K8S Pods can access them:

root@k8s-master:~# mkdir -p /data/nfs-server/etcd-certs
root@k8s-master:~# mkdir -p /data/nfs-server/etcd-backup
root@k8s-master:~# ll /data/nfs-server/
total 32
drwxr-xr-x 8 root root 4096 Jul 30 16:21 ./
drwxr-xr-x 3 root root 4096 Jul 30 14:54 ../
-rw-r--r-- 1 root root    0 Jul 30 14:55 1.txt
drwxr-xr-x 2 root root 4096 Jul 30 16:21 etcd-backup/
drwxr-xr-x 2 root root 4096 Jul 30 16:20 etcd-certs/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv1/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv2/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv3/
drwxr-xr-x 3 root root 4096 Jul 30 15:22 sc/

root@k8s-master:~# chmod 755 /data/nfs-server/etcd-certs
root@k8s-master:~# chmod 755 /data/nfs-server/etcd-backup

Ensure that the worker nodes can properly mount and use the NFS storage.

root@k8s-node1:~# install -d /data/nfs-server/
root@k8s-node1:~# mount -t nfs 10.0.0.6:/data/nfs-server /data/nfs-server/
root@k8s-node1:~# ll /data/nfs-server/
total 32
drwxr-xr-x 8 root root 4096 Jul 30 16:21 ./
drwxr-xr-x 3 root root 4096 Jul 30 17:02 ../
-rw-r--r-- 1 root root    0 Jul 30 14:55 1.txt
drwxr-xr-x 2 root root 4096 Jul 30 16:21 etcd-backup/
drwxr-xr-x 2 root root 4096 Jul 30 16:22 etcd-certs/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv1/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv2/
drwxrwxrwx 2 root root 4096 Jul 30 15:03 pv3/
drwxr-xr-x 5 root root 4096 Jul 30 16:26 sc/

Copy etcd Certificates to the NFS Directory

Copy the etcd certificates to the etcd-certs directory on the NFS server (the source certificate path remains /etc/kubernetes/pki/etcd/):

root@k8s-master:~# cp /etc/kubernetes/pki/etcd/{ca.crt,peer.crt,peer.key} /data/nfs-server/etcd-certs/
root@k8s-master:~# ll /data/nfs-server/etcd-certs/
total 20
drwxr-xr-x 2 root root 4096 Jul 30 16:22 ./
drwxr-xr-x 8 root root 4096 Jul 30 16:21 ../
-rw-r--r-- 1 root root 1094 Jul 30 16:22 ca.crt
-rw-r--r-- 1 root root 1204 Jul 30 16:22 peer.crt
-rw------- 1 root root 1675 Jul 30 16:22 peer.key

Writing the PVC Manifest: Creating Persistent Volumes Based on the NFS StorageClass

Create two PersistentVolumeClaims (PVCs): one for mounting certificates (Read-Only) and one for mounting the backup directory (Read-Write).

Create the Certificate PVC (etcd-certs-pvc)

Create a new file named static-pv-pvc-etcd-certs.yaml. This will be used to mount the NFS etcd-certs directory (certificates require ReadOnlyMany access).

apiVersion: v1
kind: PersistentVolume
metadata:
  name: static-pv-etcd-certs  # PV Name
spec:
  capacity:
    storage: 1Gi  # Capacity is for identification; does not restrict actual storage
  accessModes:
    - ReadOnlyMany  # Read-only, allowing access from multiple nodes
  persistentVolumeReclaimPolicy: Retain  # Retain data; do not delete files when PVC is deleted
  nfs:
    server: 10.0.0.6  # NFS Server IP (as specified in your environment)
    path: /data/nfs-server/etcd-certs  # Manually created directory for certificates
  storageClassName: ""  # No StorageClass specified to avoid dynamic provisioning

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-pvc-etcd-certs  # PVC Name
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  volumeName: static-pv-etcd-certs  # Manually bound to the PV defined above
  storageClassName: ""  # Must match the PV configuration

Create the Backup PVC (etcd-backup-pvc)

Create a new file named etcd-backup-pv-pvc.yaml. This will be used to mount the NFS etcd-backup directory (requires Read-Write access for storing backup files):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: static-pv-etcd-backup  # PV Name
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany  # Read-Write, allowing multi-node access
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 10.0.0.6  # NFS Server IP
    path: /data/nfs-server/etcd-backup  # Manually created backup directory
  storageClassName: ""

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-pvc-etcd-backup  # PVC Name
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: static-pv-etcd-backup  # Manually bound to the PV above
  storageClassName: ""

Create PVCs and Verify Status

Run the following commands to create the PVCs and confirm that their status is Bound:

# Create static PVs
kubectl apply -f static-pv-etcd-certs.yaml
kubectl apply -f static-pv-etcd-backup.yaml

# Create static PVCs (will automatically bind to the PVs with the same names)
kubectl apply -f static-pvc-etcd-certs.yaml
kubectl apply -f static-pvc-etcd-backup.yaml

# Verify status (Ensure STATUS is "Bound")
root@k8s-master:~# kubectl get pv,pvc

Key Points:

AccessModes:

Certificate Directory using ReadOnlyMany (ROX): Allows Pods on multiple nodes to mount the volume as read-only, preventing accidental modification of sensitive certificates.
Backup Directory using ReadWriteMany (RWX): Allows Pods on any node to perform read/write operations, which is essential since the CronJob Pod may be scheduled on different nodes across the cluster.

StorageClass:
Must be specified as the existing nfs-csi to ensure the PVC can dynamically bind to the NFS volumes.

Writing the Dockerfile

The Dockerfile only defines the etcdctl utility and the backup command environment within the container. It is independent of the storage type, so we can directly reuse the previous configuration:

root@k8s-master:~/bak-etcd# cat Dockerfile 
FROM alpine:latest 
LABEL matainer="NovaCaoFc" \
      role="bak" \
      project="etcd"
COPY etcdctl /usr/local/bin/ 
CMD ["/bin/sh","-c","etcdctl --endpoints=${ETCD_HOST}:${ETCD_PORT} --cacert=/certs/ca.crt --cert=/certs/peer.crt --key=/certs/peer.key snapshot save /backup/etcd-`date +%F-%T`.backup"]

Download the etcdctl binary from GitHub and save the executable.

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system

etcd

Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases.

etcd is a distributed reliable key-value store for the most critical data of a distributed system, with a focus on being:

Simple: well-defined, user-facing API (gRPC)
Secure: automatic TLS with optional client cert authentication
Fast: benchmarked 10,000 writes/sec
Reliable: properly distributed using Raft

etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

etcd is used in production by many companies, and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as Kubernetes, locksmith, vulcand, Doorman, and many others. Reliability is further ensured by rigorous robustness testing.

See etcdctl for a simple command line client.

Original image credited to xkcd.com/2347, alterations…

View on GitHub

root@k8s-master:~/bak-etcd# ll
total 16096
drwxr-xr-x  2 root root     4096 Jul 30 16:37 ./
drwx------ 13 root root     4096 Jul 30 16:37 ../
-rw-r--r--  1 root root      302 Jul 30 16:31 Dockerfile
-rwxr-xr-x  1 cao  cao  16466072 Jul 26 02:17 etcdctl*

As shown in the directory listing above, the preparation involves placing the etcdctl binary in the same build context as your Dockerfile. This ensures that when the image is built, the tool is available inside the container to execute the snapshot commands.

Building the Image

Click to view the full build log

root@k8s-master:~/bak-etcd# docker build -t etcd-bak:v1 .
[+] Building 0.1s (7/7) FINISHED                                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 353B                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load build context                                                                                                                       0.0s
 => => transferring context: 31B                                                                                                                        0.0s
 => [1/2] FROM docker.io/library/alpine:latest                                                                                                          0.0s
 => CACHED [2/2] COPY etcdctl /usr/local/bin/                                                                                                           0.0s
 => exporting to image                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                 0.0s
 => => writing image sha256:8a29a144172a91e01eb81d8e540fb785e9749058be1d6336871036e9fb781adb                                                            0.0s
 => => naming to docker.io/library/etcd-bak:v1          

root@k8s-master:~/bak-etcd# docker images |grep bak
etcd-bak                                                                                         v1                                          8a29a144172a   7 minutes ago   24.8MB

# Save the image as a tar archive
root@k8s-master:~# docker save -o etcd-bak.tar etcd-bak:v1 

# Distribute the image to other nodes via scp
root@k8s-master:~# scp etcd-bak.tar 10.0.0.7:/root/
etcd-bak.tar                                                                                100%   24MB  73.5MB/s   00:00    
root@k8s-master:~# scp etcd-bak.tar 10.0.0.8:/root/
etcd-bak.tar                                          

# Import the image on other nodes
root@k8s-node1:~# docker load -i etcd-bak.tar 
418dccb7d85a: Loading layer [==================================================>]  8.596MB/8.596MB
39e2b60cb098: Loading layer [==================================================>]  16.47MB/16.47MB
Loaded image: etcd-bak:v1

This process manually distributes the backup image across the cluster nodes. In a production environment, it is generally recommended to push the image to a Private Container Registry (like Harbor or Azure Container Registry) and configure an imagePullSecret to allow the nodes to pull the image automatically.

Writing the CronJob Manifest (Adapted for NFS Mounts)

Operational Steps:

Create a new file named cj-backup-etcd-nfs.yaml with the following:

root@k8s-master:~# cat cj-backup-etcd-nfs.yaml 
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-etcd
spec:
  schedule: "* * * * *" # Use "every minute" for initial testing
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
          - name: certs
            persistentVolumeClaim:
              claimName: static-pvc-etcd-certs  # Referencing the static certificate PVC
          - name: bak
            persistentVolumeClaim:
              claimName: static-pvc-etcd-backup  # Referencing the static backup PVC
          containers:
          - name: etcd-backup
            image: etcd-bak:v1
            imagePullPolicy: IfNotPresent
            volumeMounts:
            - name: certs
              mountPath: /certs
              readOnly: true
            - name: bak
              mountPath: /backup
            env:
            - name: ETCD_HOST
              value: "10.0.0.6"  # Your etcd node IP
            - name: ETCD_PORT
              value: "2379"
          restartPolicy: OnFailure

Deploying the CronJob:

root@k8s-master:~# kubectl apply -f cj-backup-etcd-nfs.yaml
cronjob.batch/backup-etcd created

Testing and Verification: Confirming NFS Backup Success

The primary goal of this stage is to verify that the backup files are being correctly generated and stored in the /etcd-backup directory on the NFS server.

Check CronJob and Pod Status:

root@k8s-master:~# kubectl get -f  cj-backup-etcd-nfs.yaml 
NAME          SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
backup-etcd   * * * * *   <none>     False     0        <none>          17s

root@k8s-master:~# kubectl get pods
NAME                         READY   STATUS             RESTARTS      AGE
backup-etcd-29231120-9s9dm   0/1     Completed          0             3m4s
backup-etcd-29231121-nfjl5   0/1     CrashLoopBackOff   1 (11s ago)   2m4s
backup-etcd-29231122-gl4df   0/1     Completed          1             64s
backup-etcd-29231123-tfw5f   0/1     Completed          0             4s
root@k8s-master:~# ll /data/nfs-server/etcd-backup/
total 50824
drwxr-xr-x 2 root root     4096 Jul 30 17:23 ./
drwxr-xr-x 8 root root     4096 Jul 30 16:21 ../
-rw------- 1 root root 13004832 Jul 30 17:22 etcd-2025-07-30-09:22:50.backup
-rw------- 1 root root 13004832 Jul 30 17:22 etcd-2025-07-30-09:22:53.backup
-rw------- 1 root root 13004832 Jul 30 17:23 etcd-2025-07-30-09:23:01.backup

3. Data Recovery Steps

Pre-recovery Preparation

Before initiating a restore, it is critical to ensure the integrity of your backup file.

Verify Backup Validity First, confirm that the backup file is complete and usable. Use the etcdctl utility to verify the snapshot status:

# Assume the backup file is in the NFS directory; first, copy it to the local node (e.g., the Master node)
cp /data/nfs-server/etcd-backup/etcd-2025-07-30-09:22:50.backup /tmp/etcd-backup.db

# Verify the backup file (ensure you are using an etcdctl version that matches your cluster version)
etcdctl --write-out=table snapshot status /tmp/etcd-backup.db

When restoring etcd data, you must stop all control plane components that depend on etcd. This prevents data write conflicts and ensures the restoration process has exclusive access to the data directory.

# Execute on the Master node (adjust based on your actual components)
systemctl stop kubelet

# Stop the control plane containers
docker stop $(docker ps -q --filter name=k8s_kube-apiserver*)
docker stop $(docker ps -q --filter name=k8s_kube-controller-manager*)
docker stop $(docker ps -q --filter name=k8s_kube-scheduler*)
docker stop $(docker ps -q --filter name=k8s_etcd*)

Perform Recovery

Multi-node etcd Cluster Recovery (Production Environment)

In a high-availability cluster (typically 3 nodes), the restoration must be performed on all nodes. Each node must rebuild its data directory from the same snapshot to ensure consistency across the cluster.

Back Up Existing Data on All etcd Nodes

Back up the original data on all etcd nodes:

mv /var/lib/etcd /var/lib/etcd.bak

Perform recovery on the first node:

etcdctl snapshot restore /tmp/etcd-backup.db \
  --data-dir=/var/lib/etcd \
  --name=etcd-1 \  # Node name (e.g., etcd-1, etcd-2, or etcd-3)
  --initial-cluster=etcd-1=https://10.0.0.6:2380,etcd-2=https://10.0.0.7:2380,etcd-3=https://10.0.0.8:2380 \
  --initial-cluster-token=etcd-cluster-token \
  --initial-advertise-peer-urls=https://10.0.0.6:2380  # Peer address of the current node

Execute the restoration on the other nodes (modify --name and --initial-advertise-peer-urls to match the information for each specific node):

etcdctl snapshot restore /tmp/etcd-backup.db \
  --data-dir=/var/lib/etcd \
  --name=etcd-2 \
  --initial-cluster=etcd-1=https://10.0.0.6:2380,etcd-2=https://10.0.0.7:2380,etcd-3=https://10.0.0.8:2380 \
  --initial-cluster-token=etcd-cluster-token \
  --initial-advertise-peer-urls=https://10.0.0.7:2380

Post-Recovery Verification and Service Startup

1. Fix Directory Permissions

After performing the restoration, you must ensure that the etcd data directory has the correct ownership and permissions. If the permissions are incorrect (e.g., owned by root while the service expects the etcd user), the etcd service will likely fail to start.

chown -R 1000:1000 /var/lib/etcd  # etcd defaults to running as user 1000

2. Start Control Plane Components

systemctl start kubelet
# Wait for the containers to restart automatically, or start them manually
docker start $(docker ps -aq --filter name=k8s_etcd*)
docker start $(docker ps -aq --filter name=k8s_kube-apiserver*)
docker start $(docker ps -aq --filter name=k8s_kube-controller-manager*)
docker start $(docker ps -aq --filter name=k8s_kube-scheduler*)

3. Verify Cluster Status

Confirm Successful Recovery:

# View node status
kubectl get nodes

# View Pod status across all namespaces
kubectl get pods --all-namespaces

# Check etcd cluster health status
etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  endpoint health

⚠️ Important Considerations

Restoration Overwrites Existing Data: The restoration process wipes the current etcd data. Always verify your backup file before proceeding.

Version Compatibility: The etcdctl version must strictly match the etcd cluster version (e.g., use v3.5.0 etcdctl for a v3.5.0 etcd cluster); otherwise, the restoration may fail.

Production Environment Recommendations:

Backup Before You Restore: Always take a fresh snapshot of the current (even if corrupted) etcd data (etcdctl snapshot save) before starting the recovery. This provides a rollback point in case of operational errors.

Plan for Downtime: The restoration process causes brief cluster unavailability. It is highly recommended to perform this during off-peak hours.

Post-Recovery Sync Check: After restoring a multi-node etcd cluster, verify that all nodes have successfully joined and are in sync using etcdctl member list.

By following these steps, you can leverage etcd snapshot backups to restore Kubernetes cluster data, ensuring rapid service recovery in the event of data anomalies.

Nginx Deep Dive: Architecture, Configuration, and Practical Examples

Shingai Zivuku — Thu, 19 Feb 2026 00:38:26 +0000

Introduction

Nginx (“Engine-X”) is a high-performance HTTP and reverse proxy server widely used in various scenarios such as web services, load balancing, API gateways, reverse proxies, and static resource servers. Due to its high performance, low resource consumption, and flexible configuration, Nginx has become the preferred choice for many internet companies, enterprises, and developers.

This article will begin with a basic introduction to Nginx, delve into its working principles, and use practical examples to help readers better understand how to configure and optimize Nginx.

Introduction to Nginx Basics

History and Background of Nginx

Nginx was originally developed by Russian programmer Igor Sysoev and released in 2004. It was initially designed to address the C10K problem (the problem of handling 10,000 simultaneous connections), thus exhibiting excellent performance in handling high-concurrency requests. Due to its outstanding performance and scalability, Nginx has become one of the world’s most popular web servers, particularly excelling in handling static resources and reverse proxying.

Nginx’s Core Functions

Nginx has the following core functionalities:

Reverse proxy: Nginx can act as a reverse proxy server, forwarding client requests to backend servers.
Load balancing: Nginx supports a variety of load balancing algorithms, such as Round Robin, IP Hash, and Least Connections.
Static file service: Nginx can efficiently serve static files such as HTML, CSS, JavaScript, and images.
HTTP caching: Nginx provides a caching mechanism that can cache the response content of HTTP requests, improving access performance.
SSL/TLS support: Nginx fully supports the SSL/TLS protocol and can provide support for HTTPS services.
Reverse proxy combined with load balancing: Nginx can forward requests to multiple backend servers, distributing traffic and providing high availability.
WebSocket support: Nginx supports the WebSocket protocol and can handle long-lived connections.

How Nginx Works

Nginx’s high performance and scalability stem from its unique design philosophy and working principles, as follows:

Event-Driven Model

Nginx employs an event-driven architecture. Unlike traditional multi-threaded or multi-process models, Nginx uses a single-process model and handles client connection requests through asynchronous event-driven mechanisms. Whenever a new request arrives, Nginx places it in an event queue, which is then scheduled by the main thread.

In this way, Nginx can handle a large number of concurrent connections in a single process, while avoiding the context switching and memory consumption problems caused by traditional multithreading.

Request Processing Flow

The Nginx request processing flow can be divided into the following stages:

Receiving requests: Nginx listens for client requests and adds them to the event queue.
Parsing requests: Driven by the event-driven model, Nginx parses the HTTP request headers (such as URL, method, hostname, etc.) of the client request.
Selecting the appropriate service: Based on the instructions in the configuration file, Nginx will select the appropriate backend server for request forwarding, or directly provide static resources.
Response generation: Nginx generates response data based on the response from the backend server or local static files and sends it back to the client.
Logging: Nginx logs requests for analysis and debugging by operations and development personnel.

Nginx Configuration File Structure

Nginx’s configuration file is usually located at /etc/nginx/nginx.conf, and its basic structure is as follows:

Global block: Sets global configurations, such as the number of worker processes, user permissions, log paths, etc.
HTTP block: Configures HTTP server settings such as caching, compression, and load balancing.
Server block: Defines virtual hosts and handles requests from different domains.
Location block: Used to match request URLs and configure the processing method for different URLs.

Advanced Features of Nginx

Load Balancing

Nginx provides various load-balancing algorithms to distribute client requests evenly across multiple backend servers. Common load balancing algorithms include:

Round Robin: The default load balancing method, which distributes requests evenly across all servers.
Least Connections: Distribute requests to the server with the fewest connections.
IP Hash: Distributes requests to specific backend servers based on the hash value of the client’s IP address.

Example configuration:

http {
    upstream backend {
        server backend1.example.com;
        server backend2.example.com;
        server backend3.example.com;
    }

    server {
        location / {
            proxy_pass http://backend;
        }
    }
}

Caching and Compression

Nginx supports caching of static files and HTTP responses, which can significantly improve access speed and reduce the burden on the backend. In addition, Nginx supports compression, which can reduce the amount of data transmitted over the network and improve page load speed.

Example of cache configuration:

http {
    proxy_cache_path /tmp/cache keys_zone=my_cache:10m;

    server {
        location / {
            proxy_cache my_cache;
            proxy_pass http://backend;
        }
    }
}

Compression configuration example:

http {
    gzip on;
    gzip_types text/plain application/javascript text/css;
    gzip_min_length 1000;

}

SSL/TLS Configuration

Nginx supports SSL/TLS, providing secure HTTPS services for websites. Configuring SSL/TLS can prevent man-in-the-middle attacks and protect user privacy.

server {
    listen 443 ssl;
    server_name www.example.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        root /var/www/html;
        index index.html;
    }
}

Nginx Practical Case Studies

Static File Server

Suppose we have a website that needs to serve static resources via Nginx. The configuration is as follows:

server {
    listen 80;
    server_name www.example.com;

    root /var/www/html;
    index index.html index.htm;

    location /images/ {
        root /var/www/assets;
    }
}

In this configuration, Nginx will serve the website homepage located at /var/www/html and forward requests under the /images/ path to resources in the /var/www/assets folder.

Reverse Proxy and Load Balancing

Assuming we have three backend application servers, we will use Nginx for load balancing:

http {

upstream backend {
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

server {
    listen 80;
    server_name www.example.com;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    }
}

}

URL Rewriting and Redirection

Sometimes we need to redirect old URLs to new URLs. For example, redirecting all http requests to new https requests:

server {

listen 80;
server_name www.example.com;

return 301 https://$host$request_uri;

}

Performance Optimization and Monitoring

Nginx performance optimization can be approached from the following aspects:

Adjust the number of worker processes: Configure them appropriately based on the number of CPU cores on the server worker_processes.
Use caching: Cache static resources and reverse proxy caching to reduce backend pressure.
Enable GZIP compression: Reduce the amount of data transmitted and improve page loading speed.

For monitoring, you can use Nginx’s stub_status module to monitor server status:

server {
    listen 80;
    server_name status.example.com;

    location /status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

Conclusion

Nginx, as a high-performance web server, is an indispensable component of modern web services due to its unique event-driven architecture, reverse proxy, load balancing, and caching capabilities. This article provides a foundation for further discussion.

Kubernetes Configuration Management: Mastering Helm Values Files for Multi-Environment Deployments

Shingai Zivuku — Wed, 18 Feb 2026 23:50:38 +0000

Introduction

Overview of Kubernetes

Kubernetes (K8s) is an open-source container orchestration platform that automates the deployment, scaling, and management of containers. In K8s, applications typically run as containers, organized into different resource objects such as Deployments, Services, ConfigMaps, and Secrets. To ensure stable operation of applications in different environments (development, testing, pre-production, and production), customized configurations are needed for each environment.

The Importance of Values Files

Values files are configuration files used when managing Kubernetes application deployments using Helm, typically in YAML format. Helm is the package manager for Kubernetes, packaging and managing Kubernetes resources as a whole, called a Chart. Values files play a crucial role in this, allowing us to modify and adjust application configurations in different deployment environments without altering the application code or Kubernetes resource definition files. This ensures the application runs as expected at different stages, while maintaining configuration flexibility and maintainability.

Differences in configuration requirements for different environments

Development environment: Primarily used by developers for code development and debugging. It typically involves frequent configuration updates, may use relatively small resources, and has relatively lower performance and reliability requirements. For example, a development environment can use a local database or a simple development environment database, and the log level can be set to verbose to allow developers to promptly identify and resolve issues in the code.
Test environment: Used for functional testing, integration testing, and performance testing, it needs to have a similar configuration to the production environment, but allows for a certain degree of flexibility. The database in the test environment may be dedicated to testing, and its data and performance requirements may vary depending on the testing purpose; for example, it may simulate the data volume or load of the production environment.
Pre-production environment: This is a rehearsal of the production environment, designed to simulate the production environment as closely as possible, including configuration, resource scale, and performance requirements, to ensure that potential problems are identified and resolved before the official launch.
Production environment: requires the highest stability, performance, and security, needs to use real production databases, usually has a large resource scale, and has strict requirements for service availability and performance.

Implementation Method

Creating a Helm Chart

First, you need to create a Helm Chart, which is a directory structure containing the application's Kubernetes resource template and default Values files. You can create a new Helm Chart using the following command:

helm create my-application

This command will create a my-application directory named <directory_name> containing multiple subdirectories and files. The templates directory contains template files for Kubernetes resources, while values.yaml is the default Values file.

Modify the default Values file

In the values.yaml configuration file, you can define various configuration parameters for the application, such as:

replicaCount: 1
image:
  repository: my-image
  tag: latest
service:
  type: ClusterIP
  port: 80
environment: development
database:
  url: jdbc:mysql://localhost:3306/devdb
  username: devuser
  password: devpass

This section defines the number of replicas, image information, service type, port, and database connection information. However, these configurations are generic, and we need to customize them for different environments.

Create customized Values files for different environments

Create a separate Values file for each environment and save it in the Chart directory. For example:

Development environment: values-dev.yaml

replicaCount: 1
image:
  repository: my-image-dev
  tag: latest-dev
service:
  type: ClusterIP
  port: 8080
environment: development
database:
  url: jdbc:mysql://dev-db-server:3306/devdb
  username: devuser
  password: devpass
  logging:
    level: debug

Test environment: values-test.yaml

replicaCount: 2
image:
  repository: my-image-test
  tag: latest-test
service:
  type: NodePort
  port: 8081
environment: testing
database:
  url: jdbc:mysql://test-db-server:3306/testdb
  username: testuser
  password: testpass
  logging:
    level: info

Pre-release environment: values-preprod.yaml

replicaCount: 3
image:
  repository: my-image-preprod
  tag: latest-preprod
service:
  type: LoadBalancer
  port: 80
environment: preproduction
database:
  url: jdbc:mysql://preprod-db-server:3306/preproddb
  username: preproduser
  password: preprodpass
  logging:
    level: warn

Production environment: values-prod.yaml

replicaCount: 5
image:
  repository: my-image-prod
  tag: latest-prod
service:
  type: LoadBalancer
  port: 80
environment: production
database:
  url: jdbc:mysql://prod-db-server:3306/proddb
  username: produser
  password: prodpass
  logging:
    level: error

Using template languages

In the templates Kubernetes resource template file within the directory, you can use Helm's template language to reference configurations from the Values file. For example, in the deployment.yaml template file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{.Release.Name }}-my-application
spec:
  replicas: {{.Values.replicaCount }}
  selector:
    matchLabels:
      app: {{.Release.Name }}-my-application
  template:
    metadata:
      labels:
        app: {{.Release.Name }}-my-application
    spec:
      containers:
        - name: my-application
          image: {{.Values.image.repository }}:{{.Values.image.tag }}
          ports:
            - containerPort: {{.Values.service.port }}
          env:
            - name: DATABASE_URL
              value: {{.Values.database.url }}
            - name: DATABASE_USER
              value: {{.Values.database.username }}
            - name: DATABASE_PASSWORD
              value: {{.Values.database.password }}
            - name: LOGGING_LEVEL
              value: {{.Values.database.logging.level }}

Here, we use {{.Values.xxx }} syntax to reference the configuration in the Values file and insert them into the appropriate locations in the Kubernetes resources.

Deploying the application

Use Helm commands to deploy using the appropriate Values file based on different environments:

Development environment:

helm install my-application-dev my-application --values my-application/values-dev.yaml

Test environment:

helm install my-application-test my-application --values my-application/values-test.yaml

Pre-release environment:

helm install my-application-preprod my-application --values my-application/values-preprod.yaml

Production environment:

helm install my-application-prod my-application --values my-application/values-prod.yaml

Implementation Case

Suppose we have a simple web application; here is a complete implementation example:

Creating a Helm Chart

helm create my-webapp

Modify the `values.yaml` document:

appName: my-webapp
replicaCount: 1
image:
  repository: my-webapp-image
  tag: latest
service:
  type: ClusterIP
  port: 80
database:
  url: jdbc:mysql://default-db:3306/defaultdb
  username: defaultuser
  password: defaultpass
logging:
  level: info

Creating Values files for different environments:

Development environment: values-dev.yaml

appName: my-webapp-dev
replicaCount: 1
image:
  repository: my-webapp-dev-image
  tag: latest-dev
service:
  type: ClusterIP
  port: 8080
database:
  url: jdbc:mysql://dev-db:3306/devdb
  username: devuser
  password: devpass
logging:
  level: debug

Test environment: values-test.yaml

appName: my-webapp-test
replicaCount: 2
image:
  repository: my-webapp-test-image
  tag: latest-test
service:
  type: NodePort
  port: 8081
database:
  url: jdbc:mysql://test-db:3306/testdb
  username: testuser
  password: testpass
logging:
  level: info

Pre-release environment: values-preprod.yaml

appName: my-webapp-preprod
replicaCount: 3
image:
  repository: my-webapp-preprod-image
  tag: latest-preprod
service:
  type: LoadBalancer
  port: 80
database:
  url: jdbc:mysql://preprod-db:3306/preproddb
  username: preproduser
  password: preprodpass
logging:
  level: warn

Production environment: values-prod.yaml

appName: my-webapp-prod
replicaCount: 5
image:
  repository: my-webapp-prod-image
  tag: latest-prod
service:
  type: LoadBalancer
  port: 80
database:
  url: jdbc:mysql://prod-db:3306/proddb
  username: produser
  password: prodpass
logging:
  level: error

Modify the template file

In templates/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{.Release.Name }}-{{.Values.appName }}
spec:
  replicas: {{.Values.replicaCount }}
  selector:
    matchLabels:
      app: {{.Release.Name }}-{{.Values.appName }}
  template:
    metadata:
      labels:
        app: {{.Release.Name }}-{{.Values.appName }}
    spec:
      containers:
        - name: {{.Values.appName }}
          image: {{.Values.image.repository }}:{{.Values.image.tag }}
          ports:
            - containerPort: {{.Values.service.port }}
          env:
            - name: DATABASE_URL
              value: {{.Values.database.url }}
            - name: DATABASE_USER
              value: {{.Values.database.username }}
            - name: DATABASE_PASSWORD
              value: {{.Values.database.password }}
            - name: LOGGING_LEVEL
              value: {{.Values.logging.level }}

Deploying the application

Use the following command to deploy the application to different environments:

Development environment:

helm install my-webapp-dev my-webapp --values my-webapp/values-dev.yaml

Test environment:

helm install my-webapp-test my-webapp --values my-webapp/values-test.yaml

Pre-release environment:

helm install my-webapp-preprod my-webapp --values my-webapp/values-preprod.yaml

Production environment:

helm install my-webapp-prod my-webapp --values my-webapp/values-prod.yaml

In this way, we can easily deploy applications in different environments and customize the configuration according to the characteristics of each environment. This approach ensures the flexibility and manageability of applications in different environments, while reducing problems caused by configuration errors.

Summary

Using Values files to customize application configurations across different Kubernetes environments is a powerful and flexible approach. Leveraging Helm's template capabilities, it allows developers and operations personnel to easily adjust application configurations at different deployment stages without modifying code or Kubernetes resource definition files. By using Values files appropriately, we can better manage application development, testing, pre-production, and production environments, ensuring smooth operation at each stage and meeting corresponding performance, security, and availability requirements. Simultaneously, this approach facilitates continuous integration and continuous deployment (CI/CD) processes, enabling teams to deliver and maintain software more efficiently.

Please remember that in practical applications, appropriate adjustments and optimizations are needed based on the specific requirements of the application and the characteristics of the Kubernetes cluster to ensure the accuracy and effectiveness of the configuration. At the same time, pay attention to the management of sensitive information (such as database passwords), avoid storing it in plaintext in configuration files, and consider using Kubernetes' Secret resources for storage and management.

Frontend Refresh Project - An Electronic Spider

Shingai Zivuku — Tue, 17 Sep 2024 18:03:41 +0000

Preface: After learning JavaScript, you can use JavaScript to achieve some interesting effects. This article explains how to use JavaScript purely to implement an electronic spider on a web page.

Before we start learning how to write a web spider, let's take a look at what this electronic spider looks like:

You can see that it will move with our mouse, so how do you achieve this effect? Let's start explaining.

HTML Code

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Dreaming</title>
    <!-- External JavaScript files -->
    <script src=".test.js"></script>
    <style>
        /* Remove default padding and margins from body */
        body {
            margin: 0px;
            padding: 0px;
            position: fixed;
            /* Set the background color of webpage to black */
            background: rgb(0, 0, 0);
        }
    </style>
</head>

<body>
    <!-- Create a canvas for drawing -->
    <canvas id="canvas"></canvas>
</body>

</html>

As you can see our HTML code is very simple, let's start working on it!

Before you start writing JavaScript code, create a plan:

Overall process

When the page loads, the canvas element and drawing context are initialized.
Define tentacle objects . Each tentacle consists of multiple segments.
Listen to mouse movement events and update the mouse position in real time.
The tentacles are drawn through an animation loop, and they change dynamically according to the position of the mouse, creating a smooth animation effect.

The general process is the above steps, but I believe that you may not understand the above process before you have completed the writing of this code, but it doesn't matter anyways, so let's start writing our web spider:

Preface: In order to help you better understand the logic of the code, I have added comments to each code. I hope that you can understand the code bit by bit with the help of the comments:

JavaScript Code

// Define requestAnimFrame function
window.requestAnimFrame = function () {
    // Check if the browser supports requestAnimFrame function
    return (
        window.requestAnimationFrame ||
        window.webkitRequestAnimationFrame ||
        window.mozRequestAnimationFrame ||
        window.oRequestAnimationFrame ||
        window.msRequestAnimationFrame ||
        // If all these options are unavailable, use setTimeout to call the callback function
        function (callback) {
            window.setTimeout(callback)
        }
    )
}

// Initialization function to get canvas element and return related information
function init(elemid) {
    // Get canvas element
    let canvas = document.getElementById(elemid)
    // Get 2d drawing context, note that 'd' is lowercase
    c = canvas.getContext('2d')
    // Set canvas width to window inner width and height to window inner height
    w = (canvas.width = window.innerWidth)
    h = (canvas.height = window.innerHeight)
    // Set fill style to semi-transparent black
    c.fillStyle = "rgba(30,30,30,1)"
    // Fill the entire canvas with the fill style
    c.fillRect(0, 0, w, h)
    // Return drawing context and canvas element
    return { c: c, canvas: canvas }
}

// Execute function when page is fully loaded
window.onload = function () {
    // Get drawing context and canvas element
    let c = init("canvas").c,
        canvas = init("canvas").canvas,
        // Set canvas width to window inner width and height to window inner height
        w = (canvas.width = window.innerWidth),
        h = (canvas.height = window.innerHeight),
        // Initialize mouse object
        mouse = { x: false, y: false },
        last_mouse = {}

    // Function to calculate distance between two points
    function dist(p1x, p1y, p2x, p2y) {
        return Math.sqrt(Math.pow(p2x - p1x, 2) + Math.pow(p2y - p1y, 2))
    }

    // Define segment class
    class segment {
        // Constructor to initialize segment object
        constructor(parent, l, a, first) {
            // If it's the first tentacle segment, position is the tentacle top position
            // Otherwise, position is the nextPos coordinates of the previous segment object
            this.first = first
            if (first) {
                this.pos = {
                    x: parent.x,
                    y: parent.y,
                }
            } else {
                this.pos = {
                    x: parent.nextPos.x,
                    y: parent.nextPos.y,
                }
            }
            // Set segment length and angle
            this.l = l
            this.ang = a
            // Calculate coordinates for the next segment
            this.nextPos = {
                x: this.pos.x + this.l * Math.cos(this.ang),
                y: this.pos.y + this.l * Math.sin(this.ang),
            }
        }
        // Method to update segment position
        update(t) {
            // Calculate angle between segment and target point
            this.ang = Math.atan2(t.y - this.pos.y, t.x - this.pos.x)
            // Update position coordinates based on target point and angle
            this.pos.x = t.x + this.l * Math.cos(this.ang - Math.PI)
            this.pos.y = t.y + this.l * Math.sin(this.ang - Math.PI)
            // Update nextPos coordinates based on new position coordinates
            this.nextPos.x = this.pos.x + this.l * Math.cos(this.ang)
            this.nextPos.y = this.pos.y + this.l * Math.sin(this.ang)
        }
        // Method to return segment to initial position
        fallback(t) {
            // Set position coordinates to target point coordinates
            this.pos.x = t.x
            this.pos.y = t.y
            this.nextPos.x = this.pos.x + this.l * Math.cos(this.ang)
            this.nextPos.y = this.pos.y + this.l * Math.sin(this.ang)
        }
        show() {
            c.lineTo(this.nextPos.x, this.nextPos.y)
        }
    }

    // Define tentacle class
    class tentacle {
        // Constructor to initialize tentacle object
        constructor(x, y, l, n, a) {
            // Set tentacle top position coordinates
            this.x = x
            this.y = y
            // Set tentacle length
            this.l = l
            // Set number of tentacle segments
            this.n = n
            // Initialize tentacle target point object
            this.t = {}
            // Set random movement parameter for tentacle
            this.rand = Math.random()
            // Create first segment of the tentacle
            this.segments = [new segment(this, this.l / this.n, 0, true)]
            // Create other segments
            for (let i = 1; i < this.n; i++) {
                this.segments.push(
                    new segment(this.segments[i - 1], this.l / this.n, 0, false)
                )
            }
        }
        // Method to move tentacle to target point
        move(last_target, target) {
            // Calculate angle between tentacle top and target point
            this.angle = Math.atan2(target.y - this.y, target.x - this.x)
            // Calculate tentacle distance parameter
            this.dt = dist(last_target.x, last_target.y, target.x, target.y)
            // Calculate tentacle target point coordinates
            this.t = {
                x: target.x - 0.8 * this.dt * Math.cos(this.angle),
                y: target.y - 0.8 * this.dt * Math.sin(this.angle)
            }
            // If target point is calculated, update position coordinates of last segment object
            // Otherwise, update position coordinates of last segment object to target point coordinates
            if (this.t.x) {
                this.segments[this.n - 1].update(this.t)
            } else {
                this.segments[this.n - 1].update(target)
            }
            // Iterate through all segment objects, update their position coordinates
            for (let i = this.n - 2; i >= 0; i--) {
                this.segments[i].update(this.segments[i + 1].pos)
            }
            if (
                dist(this.x, this.y, target.x, target.y) <=
                this.l + dist(last_target.x, last_target.y, target.x, target.y)
            ) {
                this.segments[0].fallback({ x: this.x, y: this.y })
                for (let i = 1; i < this.n; i++) {
                    this.segments[i].fallback(this.segments[i - 1].nextPos)
                }
            }
        }
        show(target) {
            // If distance between tentacle and target point is less than tentacle length, draw tentacle
            if (dist(this.x, this.y, target.x, target.y) <= this.l) {
                // Set global composite operation to "lighter"
                c.globalCompositeOperation = "lighter"
                // Begin new path
                c.beginPath()
                // Start drawing line from tentacle starting position
                c.moveTo(this.x, this.y)
                // Iterate through all segment objects and use their show method to draw lines
                for (let i = 0; i < this.n; i++) {
                    this.segments[i].show()
                }
                // Set line style
                c.strokeStyle = "hsl(" + (this.rand * 60 + 180) +
                    ",100%," + (this.rand * 60 + 25) + "%)"
                // Set line width
                c.lineWidth = this.rand * 2
                // Set line cap style
                c.lineCap = "round"
                // Set line join style
                c.lineJoin = "round"
                // Draw line
                c.stroke()
                // Set global composite operation to "source-over"
                c.globalCompositeOperation = "source-over"
            }
        }
        // Method to draw tentacle's circular head
        show2(target) {
            // Begin new path
            c.beginPath()
            // If distance between tentacle and target point is less than tentacle length, draw white circle
            // Otherwise draw cyan circle
            if (dist(this.x, this.y, target.x, target.y) <= this.l) {
                c.arc(this.x, this.y, 2 * this.rand + 1, 0, 2 * Math.PI)
                c.fillStyle = "white"
            } else {
                c.arc(this.x, this.y, this.rand * 2, 0, 2 * Math.PI)
                c.fillStyle = "darkcyan"
            }
            // Fill circle
            c.fill()
        }
    }
    // Initialize variables
    let maxl = 400, // Maximum tentacle length
        minl = 50, // Minimum tentacle length
        n = 30, // Number of tentacle segments
        numt = 600, // Number of tentacles
        tent = [], // Array of tentacles
        clicked = false, // Whether mouse is pressed
        target = { x: 0, y: 0 }, // Tentacle target point
        last_target = {}, // Previous tentacle target point
        t = 0, // Current time
        q = 10; // Step length for each tentacle movement

    // Create tentacle objects
    for (let i = 0; i < numt; i++) {
        tent.push(
            new tentacle(
                Math.random() * w, // Tentacle x-coordinate
                Math.random() * h, // Tentacle y-coordinate
                Math.random() * (maxl - minl) + minl, // Tentacle length
                n, // Number of tentacle segments
                Math.random() * 2 * Math.PI, // Tentacle angle
            )
        )
    }
    // Method to draw image
    function draw() {
        // If mouse moves, calculate deviation between tentacle target point and current point
        if (mouse.x) {
            target.errx = mouse.x - target.x
            target.erry = mouse.y - target.y
        } else {
            // Otherwise, calculate x-coordinate of tentacle target point
            target.errx =
                w / 2 +
                ((h / 2 - q) * Math.sqrt(2) * Math.cos(t)) /
                (Math.pow(Math.sin(t), 2) + 1) -
                target.x;
            target.erry =
                h / 2 +
                ((h / 2 - q) * Math.sqrt(2) * Math.cos(t) * Math.sin(t)) /
                (Math.pow(Math.sin(t), 2) + 1) -
                target.y;
        }

        // Update tentacle target point coordinates
        target.x += target.errx / 10
        target.y += target.erry / 10

        // Update time
        t += 0.01;

        // Draw tentacle target point
        c.beginPath();
        c.arc(
            target.x,
            target.y,
            dist(last_target.x, last_target.y, target.x, target.y) + 5,
            0,
            2 * Math.PI
        );
        c.fillStyle = "hsl(210,100%,80%)"
        c.fill();

        // Draw center points of all tentacles
        for (i = 0; i < numt; i++) {
            tent[i].move(last_target, target)
            tent[i].show2(target)
        }
        // Draw all tentacles
        for (i = 0; i < numt; i++) {
            tent[i].show(target)
        }
        // Update previous tentacle target point coordinates
        last_target.x = target.x
        last_target.y = target.y
    }
    // Function to loop animation drawing
    function loop() {
        // Use requestAnimFrame function to loop
        window.requestAnimFrame(loop)

        // Clear canvas
        c.clearRect(0, 0, w, h)

        // Draw animation
        draw()
    }

    // Listen for window resize event
    window.addEventListener("resize", function () {
        // Reset canvas size
        w = canvas.width = window.innerWidth
        h = canvas.height = window.innerHeight

        // Loop animation drawing function
        loop()
    })

    // Loop animation drawing function
    loop()
    // Use setInterval function to loop
    setInterval(loop, 1000 / 60)

    // Listen for mouse move event
    canvas.addEventListener("mousemove", function (e) {
        // Record previous mouse position
        last_mouse.x = mouse.x
        last_mouse.y = mouse.y

        // Update current mouse position
        mouse.x = e.pageX - this.offsetLeft
        mouse.y = e.pageY - this.offsetTop
    }, false)

    // Listen for mouse leave event
    canvas.addEventListener("mouseleave", function (e) {
        // Set mouse to false
        mouse.x = false
        mouse.y = false
    })
}

Here I roughly sort out the process of the above code:

Initialization phase

initFunction: When the page is loaded, the function init is called to get the canvas element and set its width and height to the size of the window. The obtained 2D drawing context is used for subsequent drawing.
window.onload: After the page is loaded, initialize canvas and set context to the initial state of the mouse.

Definition of tentacle objects

segmentClass: This is a segment of a tentacle. Each segment has a starting point (pos), length (l), angle (ang), and the position of the next segment is calculated by the angle ( nextPos).
tentacleClass: Represents a complete tentacle, consisting of several segment. The starting point of the tentacle is at the center of the screen, and each tentacle contains multiple segments.

The main methods of tentacle are:
move: Update the position of each segment according to the mouse position.
show: Draw the path of the tentacle.

Event monitoring

canvas.addEventListener('mousemove', ...): When the mouse moves, the mouse position is captured and stored in the mouse variable. Each mouse movement updates the coordinates of mouse and last_mouse for subsequent animations.

Animation loop

drawFunction : This is a recursive function used to create animation effects.

First, it fills the canvas with a semi-transparent background in each frame, causing previously drawn content to gradually disappear, creating a smear effect.
Then, iterate over all tentacles, calling their move and show methods, updating their positions and drawing each frame.
Finally, use requestAnimFrame(draw) to make continuous recursive draw calls to form an animation loop.

Tentacle Behavior

The movement of the tentacle move is implemented through the function. The last segment of the tentacle updates its position first, and then the other segments follow in sequence.
The drawing of the tentacles is done through the show function, which iterates through all segments and draws lines, and finally displays them on the screen.

In this way, you have completed the production of the electronic spider!!!

Finally, let's take a look at the final effect:

AI empowered micro-services: Future application development driven by Spring Boot and Machine Learning

Shingai Zivuku — Tue, 14 May 2024 01:45:36 +0000

Overview

The rapid growth of AI technology has led companies and developers to explore integrating AI into applications to enhance user experience, optimize processes, and innovate business models. Microservice architecture, favored for its flexibility, scalability, and maintainability, pairs well with AI. Combining AI and microservices allows for intelligent services and flexible model deployment. This article introduces building AI-powered microservice applications using Spring Boot and machine learning.

Spring Boot

Spring Boot simplifies the setup and development of Spring applications by providing non-business functions like configuration management, service discovery, and load balancing. It allows developers to focus on implementing business logic and supports the quick creation of standalone, production-grade Spring applications. Spring Boot easily integrates with databases, message queues, and caching systems.

Machine Learning

Machine learning is an important branch of the field of artificial intelligence. It studies how to use calculations to enable computers to learn and make predictions or decisions based on data. Machine learning models usually include steps such as data preprocessing, feature engineering, model training, evaluation and optimization. The trained model can be integrated into applications to provide intelligent recommendations, image recognition, speech recognition, natural language processing and other functions.

Combination of AI and Microservices

Combining AI with microservices can achieve rapid development and deployment of intelligent services. Specifically, machine learning models can be encapsulated into microservices and provided services through REST API or gRPC. This design allows the model to be deployed and upgraded independently of business logic, and also facilitates integration and collaboration with other microservices.

Practical Cases

Suppose we want to develop an e-commerce recommendation system microservice based on Spring Boot and machine learning. Here's a simple example to illustrate how to do this.

Environment preparation

First, we need to prepare the Spring Boot development environment and introduce relevant machine learning libraries, such as TensorFlow or PyTorch's Java bindings.

Create Spring Boot project

Use Spring Initializr to create a new Spring Boot project and add necessary dependencies, such as Spring Web, Spring Data JPA, etc.

Define model service interface

Define an interface in the Spring Boot project to describe the services provided by the machine learning model. For example:

public interface RecommendationService {
    List<Product> recommendProducts(User user);
}

Implement model services

Implement the above interface, load the trained machine learning model, and provide the function of recommending products. It is assumed here that we already have a trained TensorFlow model and convert it to TensorFlow Lite format for use in Java.

import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.Tensor;
// ... other necessary imports

@Service
public class TensorFlowRecommendationService implements RecommendationService {
    private Interpreter tflite;
    // ... initialize model, load model and other codes

    @Override
    public List<Product> recommendProducts(User user) {
        // ... Convert user features to model input format
        // ... Call the model for inference
        // ... Convert model output to product list and back
    }
}

Create REST controller

Create a Spring MVC controller to handle requests from clients and call the model service for recommendations.

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
// ... Other necessary imports

@RestController
@RequestMapping("/api/recommendations")
public class RecommendationController {
    private final RecommendationService recommendationService;

    public RecommendationController(RecommendationService recommendationService) {
        this.recommendationService = recommendationService;
    }

    @GetMapping
    public ResponseEntity<List<Product>> getRecommendations(@RequestParam String userId) {
        try {
            User user = getUserById(userId); // Suppose there is a method to get user information based on user ID
            List<Product> recommendedProducts = recommendationService.recommendProducts(user);
            return ResponseEntity.ok(recommendedProducts);
        } catch (Exception e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(null);
        }
    }

    private User getUserById(String userId) {
        // Implement the logic of obtaining user information. This is just an example. The specific implementation is based on business needs.
        return new User(userId, "UserName", /* Other user attributes */);
    }
}

Running and testing

Start the Spring Boot application and use tools such as Postman or curl to send GET requests /api/recommendations?userId=xxxto test the recommended functionality. Ensure that the machine learning model can load correctly and return reasonable recommendation results.

Conclusion

By combining Spring Boot with machine learning technology, we can quickly build powerful microservice applications and realize the development and deployment of intelligent services. This article shows how to integrate Spring Boot and machine learning models to build microservices through a simple e-commerce recommendation system example. In practical applications, more factors need to be considered, such as model performance optimization, service scalability, security, etc. In the future, with the continuous development of AI technology, we can expect more innovations and breakthroughs to appear in the field of microservices.

Ansible for Automated Ops & Maintenance: Deployment & Checklist

Shingai Zivuku — Tue, 19 Mar 2024 19:20:08 +0000

What is Automated Operations?

Automated Ops refers to the automation of a large number of repetitive tasks in daily operations. Manual tasks are converted into automated operations. Automation is the sublimation of IT operations work. IT operations automation is not simply a maintenance process, but also a process of management improvement. It is a higher level of operations and the development trend of the future.

Problems Solved By Operations Automation

Improves the overall work efficiency of projects;
Reduces human errors;
Facilitates information transmission, aggregates configuration information, and makes the information chain more complete;
Leaves traces of transactions, making it easy to track and trace;
Makes operations work easier and more flexible;
Improves the value of operations work and manages more resources and service objects.

Classification of Automated Operations Tools

System installation: PXE, MetalLB, Terraform
Application configuration: Puppet, Ansible, Saltstack
Command execution and control: Fabric, Ansible
Application release: git/svn (version management), Jenkins/Gitlab Runner (continuous integration)

Comparison of Automated Operations Tools

Here we only compare Puppet, Ansible and Saltstack

	Puppet	Saltstack	Ansible
Development Language	Ruby language	Python language	Python language
Client Support	Yes	Yes (salt-ssh has no client)	No
Secondary Development Support	No	Yes	Yes
Communication Encryption	Standard SSL encryption	AES encryption	OpenSSH
Platform Support	AIX, BSD, HP-UX, Linux, Mac OSX, Solaris, Windows	BSD, Linux, Mac OS X, Solaris, Windows	AIX, BSD, HP-UX, Linux, Mac OS X, Solaris
Web UI	Yes	Yes	Commercial
Configuration File Syntax	Ruby syntax format	YAML	YAML
Command Line Execution	No (configuration implementation)	Yes	Yes

Ansible

Ansible is an automation unified configuration management tool developed using Python. It is modular and relies on the ssh protocol for implementation. Automation is mainly reflected in the fact that Ansible integrates rich modules and functional components. A series of operations can be completed through one command, which can reduce repetitive work and maintenance costs, and improve work efficiency.

Advantages of Ansible

Simple installation and deployment (Ansible is installed on the master node, and no additional client needs to be installed)
Implemented based on the existing ssh protocol (1.3 and above)
Ansible has no daemon and does not need to be started
Logs are centralized on the master node, which is convenient for troubleshooting
Simple and easy to use, no programming foundation required
Powerful functions through modules

Ansible Architecture

Inventory: Ansible's Inventory defines the scope of hosts that Ansible needs to operate on.
Playbook: Ansible's configuration file, which defines multiple tasks in the playbook, is automatically executed by Ansible.
Core Modules: Connect to the host to implement operations. It relies on specific modules to do specific things.
Custom Modules: Write specific modules according to your own needs.
Connection Plugins: Used to connect to the host and connect to the managed end.
Plugins: Complete the function supplement of the module.

The most important point is: Ansible is modular, and all its operations depend on modules.

Ansible Modes

There are two modes in Ansible: ad-hoc mode and playbook mode:

Ad-hoc: In short, it is a "temporary command" that will not be saved;

Playbook: Translated into a script, the execution process is saved in the file.

Ansible Deployment



sudo yum install ansible

[root@localhost ~]# ansible --version

ansible 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Ansible Commonly Used Parameters



-m : Specifies the name of the module to be used. If not specified, the command module is used by default.
-a : Specifies the module parameters and the specific actions of the module. It specifies the specific command to be executed.
--syntax-check: Checks the syntax.

Uncommonly Used Parameters



--version: Displays the Ansible version information.
-v: Displays detailed information.
-i: Specifies the host inventory file path. The default path is /etc/ansible/hosts.
-k: Prompts you to enter the SSH password instead of using SSH-based key authentication.
-C: Simulates the execution test but does not actually execute it.
-T: Specifies the timeout for executing the command.
-f: Specifies the number of results to return at a time.

Ansible Help Commands



ansible-doc: Help command
ansible-doc -l: Lists all modules
ansible-doc module_name: Displays detailed information about the module
ansible-doc module_name -s: Displays the usage instructions for the module options

Ansible Host Inventory

The host asset inventory is used to define the authentication information of the managed hosts, such as the SSH login username, password, and key information.

How to Check Ansible Configuration File Path:



[root@localhost ~]# rpm -qc ansible
/etc/ansible/ansible.cfg
/etc/ansible/hosts

Host Inventory Location:

The default location is /etc/ansible/hosts.

Specify Host Inventory with -i:



ansible -i /path/to/inventory_file command

Specify Default Host Inventory File in Ansible Configuration File:

The configuration file path is /etc/ansible/ansible.cfg.

Example:



[defaults]
inventory = /path/to/inventory_file

In this example, Ansible will use /path/to/inventory_file as the default host inventory file.

Ansible.cfg Common Configuration Analysis



[defaults]
#inventory = /etc/ansible/hosts     #Specifies the host list configuration file.
#library = /usr/share/my_modules/   #Specifies the directory where the library files are stored.
#remote_tmp = ~/.ansible/tmp        #Specifies the directory where temporary py files are stored on the remote host.
#local_tmp = ~/.ansible/tmp         #Specifies the local temporary execution directory.
#forks = 5                          #Specifies the default number of concurrent processes.
#sudo_user = root                   #Specifies the default sudo user.
#ask_sudo_pass = True               #Specifies whether to ask for the sudo SSH password each time it is executed.
#ask_pass = True                    #Specifies whether to ask for the SSH password each time it is executed.
#remote_port = 22                   #Specifies the remote host port.
host_key_checking = False           #Skips checking the host fingerprint.
log_path = /var/log/ansible.log     #Specifies the Ansible log file.

[privilege_escalation]
#become = True                      #Specifies whether to enable privilege escalation.
#become_method = sudo               #Specifies the privilege escalation method.
#become_user = root                 #Specifies the user to become after privilege escalation.
#become_ask_pass = False            #Specifies whether to ask for the sudo password after privilege escalation.

If host_key_checking and log_path are not configured, it will cause an error.
If host_key_checking is not configured, it will cause an error when executing the Ansible command:



172.16.11.209 | FAILED | rc=-1 >>
Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your known_hosts file to manage this host.

Host Inventory Configuration Rules

(Configured in /etc/ansible/hosts)

Host:

You can use the hostname (domain name) or IP address.
Supports hostname wildcards and regular expressions.
Supports different variables for different hosts, including password, port number, user, etc.
Single Host Configuration



# Method 1: IP + Port + Username + User Password

[root@localhost ansible]# vim /etc/ansible/hosts 
[web01]
172.16.11.209 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass='123123'
[web02]
172.16.10.232 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass='123123'

# Test: ansible hostname -m module_name
[root@localhost ansible]# ansible web01 -m ping
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

[root@localhost ansible]# ansible web02 -m ping
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

If you encounter the following error:

Enable these two in the ansible.cfg configuration file:



host_key_checking = False       #Skip checking host fingerprint
log_path = /var/log/ansible.log #ansible log



# Method Two: IP + Username and Password
[root@localhost ansible]# vim /etc/ansible/hosts 
[web01]
172.16.11.209 ansible_ssh_pass='123123'
[web02]
172.16.10.232 ansible_ssh_pass='123123'

# Test: ansible hostname -m module_name
[root@localhost ansible]# ansible web01 -m ping
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
[root@localhost ansible]# ansible web02 -m ping
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

#When no user is defined, the current logged-in user is used by default.

Multi-host Configuration



# IP + Port + Password
[root@localhost ansible]# vim /etc/ansible/hosts 
[web]
172.16.11.209 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass='123123'
172.16.10.232 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass='123123'

# Test: ansible hostname -m module_name
[root@localhost ansible]# ansible web -m ping
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

Host group:

Nested [group name: children]
Define variables for the group [group name: vars]



# Host Group Variables + Host + Password
[root@localhost ansible]# vim /etc/ansible/hosts 
[web_group]
172.16.11.209
172.16.10.232

[web_group:vars]
ansible_ssh_pass='123123'


# Test: ansible hostname -m module_name
[root@localhost ansible]# ansible web_group -m ping
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}



# Define multiple groups, nested groups

# The webservers group includes two subgroups [apache, nginx]
[root@localhost ansible]# vim /etc/ansible/hosts 
# Define total ip
[web_group]
172.16.11.209
172.16.10.232
172.16.10.129

# Two Apache servers
[apache]
172.16.11.209
172.16.10.232

# One Nginx server
[nginx]
172.16.10.129 ansible_ssh_pass='csnginx123'

# Define password
[apache:vars]
ansible_ssh_pass='123123'

[web_group:children]
apache
nginx



# Test: ansible hostname -m specify module
[root@localhost ansible]# ansible web_group -m ping
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
172.16.10.129 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

[root@localhost ansible]# ansible apache -m ping
172.16.11.209 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
172.16.10.232 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

[root@localhost ansible]# ansible nginx -m ping
172.16.10.129 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

In addition to using a password to connect, you can also configure a secret key for password free login. At this time, there is no need to specify a password in the host configuration file.



1. Generate key pair
[root@localhost ansible]# ssh-keygen

2. Share public key
[root@localhost ansible]# ssh-copy-id 172.16.11.209
[root@localhost ansible]# ssh-copy-id 172.16.10.232

Detailed Explanation of JavaScript Prototype Chain

Shingai Zivuku — Sun, 29 Oct 2023 20:22:28 +0000

Every object in JavaScript has a prototype property, which references another object. The referenced object also has its own prototype, and so on, forming a prototype chain. The top of the prototype chain is Object.prototype, which is the root prototype of all objects.

When accessing a property of an object, JavaScript will search the prototype chain for a matching property, starting with the object itself and moving up the chain until it reaches the end of the chain. If no matching property is found, the property lookup fails.

How the JavaScript prototype chain works

The working principle of the JavaScript prototype chain is very simple. Here's a very simple example of how the prototype chain works:

// Define a human constructor
function Person(name, age) {
    this.name = name;
    this.age = age;
}

// Add a method for greeting
Person.prototype.greet = function() {
    console.log(`Hi, my name is ${this.name} and I'm ${this.age} years old.`);
};

// Create a human instance
const person = new Person('John', 30);

// Access the person's name attribute and output "John"
console.log(person.name);

// Access the greet method of person and output "Hi, my name is John and I'm 30"
person.greet()

In this example, we create a Person constructor and add a greet method to its prototype object. When we create a person instance using the new keyword, it inherits the greet method from the Person constructor's prototype object.

Because the person object itself does not have a greet method, JavaScript will look up the prototype chain upwards until it finds a matching method.

Multiple Ways to Create Objects

In JavaScript, we have many ways to create objects. Depending on how it is created, the prototype chain of a JavaScript object will be different.

Creating Objects Using Literals

The prototype of objects created using literals is Object.prototype, that is, all objects created by literals are instances of Object.

// Create an empty object 
const emptyObj = {};
// Create an object with properties and methods
const obj = {
    name: 'John',
    age: 30,
    greet() {
        console.log(`Hi, my name is ${this.name} and I'm ${this.age} years old.`);
    }
};

Creating Objects Using Constructor Methods

The prototype of an object created using a constructor is the prototype of the constructor function.

// Define a Person constructor
function Person(name, age) {
    this.name = name;
    this.age = age;
}
// Add a greet method
Person.prototype.greet = function () {
    console.log(`Hi, my name is ${this.name} and I'm ${this.age} years old.`);
};
// Create a Person instance
const person = new Person('John', 30);

In this example, we create a Person object using the constructor method. When we use the new keyword to create a person instance, it inherits the greet method from the Person constructor's prototype.

Class-based Object Creation

Introduced in ES6, the class keyword makes classes and inheritance in JavaScript more intuitive and straightforward. While still underpinned by the prototype chain mechanism, this syntactic sugar streamlines the object creation and inheritance process.

class Person {
    constructor(name, age) {
      this.name = name;
      this.age = age;
    }

    greet() {
      console.log(`Hi, my name is ${this.name} and I'm ${this.age} years old.`);
    }
  }

  const person = new Person('John', 30);

Class-based objects are identical to constructor-based objects. Their prototype is the object referenced by the Class constructor's prototype property.

Modifying Prototype and Inheritance

Modifying an object's prototype can change its inheritance relationship. When an object's prototype changes, so does its prototype chain, allowing objects to inherit properties and methods from their prototype chain. This enables code reuse and extension.

// Define an Animal constructor function
class Animal {
  constructor(legs) {
    this.legs = legs;
  }

  move() {
    console.log('Moving...');
  }
}

// Define a Bird constructor function
class Bird extends Animal {
  constructor(name, legs) {
    super(legs);
    this.name = name;
  }

  fly() {
    console.log('Flying...');
  }
}

// Create a new bird object
const bird = new Bird('Pigeon', 2);

// Print the bird's name and legs
console.log(bird.name); // Pigeon
console.log(bird.legs); // 2

// Call the move and fly methods
bird.move(); // Moving...
bird.fly(); // Flying...

Here we define an Animal constructor with a move method. We then define a Bird constructor that inherits the prototype object of the Animal constructor using the Object.create() method, thus enabling the Bird constructor to inherit the move method. Finally, we add a fly method to the Bird constructor and create a bird object.

Performance and Optimization of JavaScript Prototype Chain

The prototype chain in JavaScript can introduce a performance overhead, as accessing a property requires searching the chain step-by-step until the property is found or the end of the chain is reached. A deep prototype chain can therefore lead to performance degradation. To optimize performance, design object prototype chains carefully to avoid overly large and complex structures.

Advanced JavaScript Prototype Chain

Deep dive into the implementation details of the prototype chain, including [[Prototype]] attributes, __proto__ properties, and the Object.getPrototypeOf() method.
Explore the application of the prototype chain in functions and constructors, as well as its role in function prototype extension and method inheritance.
Examine the relationship between the prototype chain and the scope chain, and explain why global variables can be accessed in the prototype object.
Discuss the use cases of the Object.create() and Object.setPrototypeOf() methods in prototype chain operations.
Recommend excellent resources and articles, such as MDN documentation and the JavaScript authoritative guide, for further study.

Conclusion

By deeply understanding the concept, working principle, and applications of the JavaScript prototype chain, we can better understand the underpinnings of JavaScript object-oriented programming. The prototype chain is not only the foundation for inheritance and property lookup in JavaScript, but it also helps us to gain a deeper understanding of the design philosophy of the JavaScript language. By skillfully applying the prototype chain, we can write efficient and scalable front-end code.

Scopes, Scope Chains, and Closures Explained

Shingai Zivuku — Sun, 08 Oct 2023 12:31:42 +0000

In JavaScript, scope refers to the scope of a variable that is accessible in code. Understanding JavaScript scopes and scope chains is crucial to writing quality code.

In this article, I will introduce the concepts of lexical scope, scope chain, and closure in JavaScript in detail, and explore their application scenarios in actual development.

Lexical Scope

Lexical scope is the most common type of scope in JavaScript. It is determined during the code writing phase, not the code execution phase. In lexical scope, access to variables is determined by their location in the code.

Example

function outer() {
  var outerVariable = "Hello";

  function inner() {
    var innerVariable = "World";
    console.log(outerVariable + " " + innerVariable);
  }

  inner();
}

outer(); output: Hello World

In the above example, the inner function can access the variable outerVariable, which is defined in the outer function outer. This is because they are in the lexical scope. Lexical scope ensures that variables can be accessed correctly at the code writing stage.

Application Scenarios of Lexical Scope

Lexical scope has a wide range of application scenarios in JavaScript, including:

Variable access control: Lexical scope prevents variables from being accessed by functions outside of their scope, which avoids naming conflicts and variable pollution.

Encapsulation: Lexical scope allows functions to encapsulate their variables and prevent them from being modified by external functions. This helps to improve the modularity and maintainability of code.

Function nesting: Lexical scope allows internal functions to access the variables of external functions. This is a common programming pattern in JavaScript and is used to implement closures.

Scope Chain

Scope chain is a mechanism in JavaScript for finding variables. It consists of variable objects from the current scope and all parent scopes. When accessing a variable, the JavaScript engine will first search in the variable object of the current scope. If it cannot find it, it will search up the scope chain until it finds the variable or reaches the global scope.

Global Execution Context
   |
   +-- Function Execution Context 1
   |      |
   |      +-- Function Execution Context 2
   |             |
   |             +-- Function Execution Context 3
   |
   +-- Function Execution Context 4

Example

var globalVariable = "Global";

function outer() {
  var outerVariable = "Hello";

  function inner() {
    var innerVariable = "World";
    console.log(globalVariable + " " + outerVariable + " " + innerVariable);
  }

  inner();
}

outer(); // Output: Global Hello World

In the above example, the inner function can access the variable globalVariable, which is defined in the global scope, and the variable outerVariable, which is defined in the outer function outer. This is because the JavaScript engine searches for variables in the order of the scope chain.

Application Scenarios of Scope Chain

Scope chain has many application scenarios in JavaScript, including:

Variable lookup: The scope chain determines the order in which variables are searched. This allows JavaScript to correctly find and access variables. For example, if a function declares a variable with the same name as a global variable, the function's local variable will be used.

Closures: A closure is a function that can access the variables of the scope in which it was created, even after that scope has closed. This is possible because closures capture the scope chain of the function in which they were created. Closures are used in a variety of JavaScript features, such as callback functions, event handlers, and modules.

Modular development: The scope chain allows you to implement modular code organization by encapsulating variables and functions in private scopes. Variables and functions can only be accessed by the code within the scope in which they were declared. This helps improve the readability, maintainability, and modularity of code.

Closure

A closure is a function that has access to the variables of the scope in which it was created, even after that scope has closed. This is because closures capture the lexical environment of the function in which they were created. The lexical environment of a function is the set of variables that are accessible to the function.

JavaScript commonly uses closures to create private variables and implement modular development.

Example

function createCounter() {
  var count = 0;

  return function() {
    count++;
    console.log(count);
  };
}

var counter = createCounter();
counter(); // Output: 1
counter(); // Output: 2

In the above example, the function createCounter() returns an inner function that has access to the outer function's variable count. Even after the outer function has finished executing, the inner function can still access and modify the variable count. This is the characteristic of closures.

Application Scenarios of Closure

Closures have many application scenarios in JavaScript, including:

Private variables: Closures provide a mechanism to implement private variables, which are variables that can only be accessed by the functions in which they are declared. This helps to improve the encapsulation and security of code.

Modular development: Closures can be used to create modular code, which is code that is divided into self-contained units that can be reused and maintained independently. This makes code more readable, maintainable, and scalable.

Delayed execution: By using closures, the execution of functions can be delayed. This is useful for implementing asynchronous operations and event processing.

Conclusion

Scope, scope chain, and closures are essential concepts in JavaScript. They work together to provide a robust mechanism for variable access and code organization.

Lexical scope allows us to control the visibility and access rights of variables, implement modular code organization, and avoid naming conflicts and variable pollution.

The scope chain determines the order in which JavaScript searches for variables, ensuring that the correct variable is always found. It also provides the foundation for closures.

Closures allow us to create private variables, implement modular code organization, and delay the execution of functions.

A deep understanding of scope, scope chains, and closures is essential for writing maintainable and efficient JavaScript code.

Comparing Caddy to Nginx and Apache

Shingai Zivuku — Tue, 26 Sep 2023 12:43:55 +0000

As the complexity of modern Web applications increases, choosing a suitable Web server is not just a technical choice, but also a matter of performance, security, and maintenance costs. Caddy, Nginx, and Apache are all excellent choices, but what makes them different?

What is Caddy?

Caddy is an open-source HTTP/2 web server written in Go language. Its main advantages are automatic support for HTTPS and simple configuration.

your-domain.com {
    reverse_proxy /api/* 127.0.0.1:8080
}

Advanced Configuration of Caddy

From automatic HTTPS to static file serving, Caddy provides a wealth of features, such as:

header / {
    X-Custom-Header "Some value"
}

Other Configurations of Caddy

Caddy provides a very rich set of configuration options to meet various needs. The following are some commonly used configurations in Caddyfile:

Compression: Use the encode directive to enable content encoding for responses.

encode gzip zstd

Logging: log directives allow you to configure the output and format of logs.

log {
    output file /path/to/logfile.log
    format json
}

Request rewrite: rewrite Directive can change the properties of the request.

rewrite /api/* /new-api/*

Request and Response Headers: You can use the header directive to add, remove, or modify HTTP headers.

header / {
    X-Custom-Header "Some value"
    -Server
}

Rate Limiting: limitDirectives can rate limit requests.

limit rate 10r/s burst 5

Basic Authentication: Use basicauth to enable Basic HTTP authentication for a specific path.

basicauth /protected/* {
    user1 hashed_password1
    user2 hashed_password2
}

Cross-Origin Resource Sharing (CORS): CORS headers can be manually configured using third-party plugins or header directives.

Health checks: When using load balancing, health checks can be configured to determine which backends are healthy.

reverse_proxy / {
    to backend1:8080 backend2:8080
    health_path /healthcheck
    health_interval 30s
}

Static File Service: file_serverThe directive can make Caddy act as a static file server.

file_server

Custom Error Pages: Use handle_errorsto define how errors are handled.

handle_errors {
    rewrite * /error{http.error.status_code}.html
    file_server
}

Caddy is a scalable and flexible web server that can be deeply customized to meet your needs. If you need more features or configuration options, please consult Caddy's official documentation.

How to configure Caddy as a reverse proxy and load balancer for a Spring Boot project

To configure load balancing and reverse proxy for your Spring Boot project using Caddy, you can follow these steps:

Install Caddy

First, make sure you have Caddy installed. If it is not installed yet, please refer to the official documentation to install the appropriate version.

Create Caddyfile

Caddyfile is the configuration file of Caddy. Create a file named Caddyfile in the directory where you plan to run Caddy.

Configure Caddyfile

The following is an example configuration of a Caddyfile to provide load balancing and reverse proxy for two Spring Boot instances:

your-domain.com {
    reverse_proxy / {
        lb_policy round_robin
        to 127.0.0.1:8080 127.0.0.1:8081
    }
}

Here is an explanation of the configuration:

your-domain.com: Replace this with your domain name.
reverse_proxy /: Configure Caddy as a reverse proxy.
lb_policy round_robin: Set the load balancing policy to round-robin.
to 127.0.0.1:8080 127.0.0.1:8081: Specify the address and port of the backend Spring Boot instance. In this example, we have two instances running on 8080 and 8081 ports.

Start Caddy

caddy run

This will start Caddy, which will provide load balancing and reverse proxy for your Spring Boot project based on the configuration of the Caddyfile.

Precaution

Make sure that your Spring Boot project instance is indeed running on the configured port and is accessible from the Caddy server. If your Spring Boot project and Caddy are on different machines, you need to ensure that the port and IP address are configured correctly.

Additionally, if you are using Caddy's automatic HTTPS feature, make sure that the domain name correctly points to the IP address of the Caddy server and that the DNS settings are configured correctly.

Nginx and Apache: Tradition and Strength

Nginx is a high-performance HTTP and reverse proxy server. Its configuration method is declarative and very flexible.

location / {
    proxy_pass http://localhost:8080;
}

Apache

Apache is a long-established web server that supports rich modules and .htaccess configurations.

<VirtualHost *:80>
    ProxyPass /app/ http://localhost:8080/
</VirtualHost>

Caddy vs Nginx vs Apache

I have listed the main features and functions of these three servers through a comparison table, from development languages to cross-platform capabilities, to provide you with a comprehensive perspective.

The following is a comparison table of Caddy, Nginx, and Apache, listing some of their key features and functions:

Features/Functions	Caddy	Nginx	Apache
Development language	Go	C	C
License	Apache 2.0	2-Clause BSD-like	Apache 2.0
Automatic HTTPS	Yes (supported by default)	Requires additional modules and configuration	Requires additional modules and configuration
HTTP/2 & HTTP/3	Support	Support	Supported (additional configuration required)
Configuration method	Caddyfile	nginx.conf	.htaccess and httpd.conf
Reverse proxy	Native support	Native support	Requires mod_proxy module
Load balancing	Native support	Native support	Requires mod_proxy_balancer module
Module/plugin system	Support (dynamic loading)	Supported (usually compiled statically)	Support (dynamic loading)
Performance	High (especially in default configuration)	High	Moderate (but can be optimized)
Safety	Designed to be secure (default HTTPS)	Safe, but configuration needs to be taken care of	Safe, but need to pay attention to configuration and modules
Beginner friendly	High (automatic HTTPS, simple configuration)	Moderate (configuration is slightly complex)	Low (configuration and module management are more complex)
Cross-platform	Yes	Yes	Yes

This table is a simplified comparison, focusing primarily on general functionality and key features. Each server has unique features and advantages, and your choice depends on your specific needs and preferences. For example, while Caddy's automatic HTTPS and simple configuration are very popular with beginners, Nginx and Apache may have advantages in some complex configurations and large deployment scenarios.

Conclusion

Choosing a web server depends not only on technical requirements but also on long-term maintenance and learning costs. Caddy's simplicity and automatic HTTPS capabilities make it popular in modern web development. But Nginx and Apache also have their unique advantages and long-term stability.

Optimizing File Processing in React with Multipart Uploads and Downloads

Shingai Zivuku — Sun, 24 Sep 2023 17:54:53 +0000

Preface

Front-end developers now need to be able to process file streams efficiently and reliably. Whether uploading, downloading, reading, displaying, or performing other file-processing operations, binary data must be processed efficiently and reliably. With increasing file sizes and the limitations of network transmission, file uploading and downloading in slices has gradually become a necessary technology to improve performance and user experience.

File multi-part upload and download makes file transmission more reliable and efficient by splitting large files into multiple small fragments and taking advantage of resuming upload at breakpoints. In this process, front-end developers should be familiar with Blob objects and ArrayBuffer. This familiarity can help you process and manipulate binary data. Using the React framework can make it more convenient to manage and operate file objects and quickly implement multi-part upload and download functions for files.

This article will delve into how to use React to implement file uploading and downloading in slices and introduce related basic concepts and technologies. I will focus on how to handle the binary data of files efficiently and how to leverage file stream operations to optimize file processing tasks in front-end development. By studying this article, you can master a set of methods for efficiently processing file stream operations and provide better solutions for your front-end development work. Let's start exploring together!

Introduction

File transfer is a fundamental requirement for many front-end applications. However, traditional methods for uploading and downloading large files can lead to performance and user experience problems.

Fortunately, front-end technology provides several efficient solutions, including file stream operations and slice download and upload. This article will delve into these technologies and explain how to use them to optimize file transfer efficiency and improve user experience.

Front-end File Flow Operation

In front-end development, file stream operations allow you to process files through a data flow, performing operations such as reading, writing, and deleting files. This article will introduce several basic concepts and technologies of front-end file stream operations in detail.

Basic Concepts of Data Flow and File Processing

In front-end development, files can be processed as data streams. A data stream is a sequence of data that is transmitted from one source to another destination.

Blob objects and ArrayBuffer: Processing Binary Data

When processing files on the front end, you often need to process binary data. Blob (Binary Large Object) objects are a type of object in JavaScript that can store large amounts of binary data. Blob objects can be created through the constructor or generated through other APIs, such as the FormData object. ArrayBuffers are another type of object in JavaScript that can store binary data. They are typically used for lower-level operations, such as manipulating and processing binary data directly.

Look at the example below:

import React, { useState } from "react";

function FileInput() {
  const [fileContent, setFileContent] = useState("");

  // Read file content to ArrayBuffer
  function readFileToArrayBuffer(file) {
    return new Promise((resolve, reject) => {
      const reader = new FileReader();

      // Register callback function when file reading is complete
      reader.onload = function(event) {
        const arrayBuffer = event.target.result;
        resolve(arrayBuffer);
      };

      // Read file content to ArrayBuffer
      reader.readAsArrayBuffer(file);
    });
  }

  // Convert ArrayBuffer to hexadecimal string
  function arrayBufferToHexString(arrayBuffer) {
    const uint8Array = new Uint8Array(arrayBuffer);
    let hexString = "";
    for (let i = 0; i < uint8Array.length; i++) {
      const hex = uint8Array[i].toString(16).padStart(2, "0");
      hexString += hex;
    }
    return hexString;
  }

  // Handle file select event
  function handleFileChange(event) {
    const file = event.target.files[0]; // Get selected file

    if (file) {
      readFileToArrayBuffer(file)
        .then(arrayBuffer => {
          const hexString = arrayBufferToHexString(arrayBuffer);
          setFileContent(hexString);
        })
        .catch(error => {
          console.error("File read failed:", error);
        });
    } else {
      setFileContent("Please select a file");
    }
  }

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <div>
        <h4>File content:</h4>
        <pre>{fileContent}</pre>
      </div>
    </div>
  );
}

export default FileInput;

In the above code, I created a functional component called FileInput. This component contains a file selection box and a preformatted text element for displaying the file contents. When the user selects a file, the file content is read to an ArrayBuffer using FileReader. The ArrayBuffer is then converted to a hexadecimal string and the result is displayed on the page.

Use FileReader to Reading Files

FileReader is a front-end browser API that allows you to read file contents asynchronously and convert them into usable data formats, such as text or binary data. It provides methods such as readAsText() and readAsArrayBuffer(), which you can choose based on your needs.

Display the File Stream

Once you have successfully read the contents of the file, you can display the file stream on the front-end page. The specific display method depends on the file type. For example, you can display text files directly in a text box or area, picture files using an img tag, and audio and video files using audio or video tags. By displaying the file stream on the front-end page, can preview and view the file content online.

File Slice Download

File slice download is the topic of today's article. Let's first take a look at the main process.

graph LR
Start (A) --> Select file {B}
B -- User selects file --> Slice file into multiple slices [C]
C --> Upload slices {D}
D -- Upload complete --> Merge slices into a complete file [E]
E -- File merge complete --> Upload success (F)
D -- Upload interrupted --> Save upload progress [G]
G -- Upload resumed --> {D}
G -- Upload canceled --> Upload canceled (H)

Performance Issues With Traditional File Downloads

File slice downloading is a technology that improves file download efficiency by splitting large files into smaller fragments (slices) and downloading them concurrently. This speeds up the overall download speed, especially for users with unreliable or slow internet connections.

Large file downloads can be slow and inefficient using traditional methods. The server must send the entire file to the client, which can lead to the following:

Long wait times: Users may have to wait a long time to start using large files.
Network congestion: Other users may experience slow download speeds if the network bandwidth is used up by large file downloads.
Difficulty resuming downloads: If a network failure or user interruption occurs, the entire file must be downloaded again.

Use File Slicing to Improve Download Efficiency

File slicing downloads split files into small fragments, typically a few hundred KB to several MB in size. This allows clients to:

Start downloading quickly: Only the first slice needs to be downloaded, so clients can start using the file sooner.
Download concurrently: Multiple concurrent requests can be used to download slices, which fully utilizes bandwidth and increases overall download speeds.
Resume downloads: If a download is interrupted, only the unfinished slices need to be redownloaded, not the entire file.

Slice upload code example:

const [selectedFile, setSelectedFile] = useState(null);
const [progress, setProgress] = useState(0);

// Handle file selection event
function handleFileChange(event) {
  setSelectedFile(event.target.files[0]);
}

// Handle file upload event
function handleFileUpload() {
  if (selectedFile) {
    // Calculate the number of chunks and the size of each chunk
    const fileSize = selectedFile.size;
    const chunkSize = 1024 * 1024; // Set chunk size to 1MB
    const totalChunks = Math.ceil(fileSize / chunkSize);

    // Create a FormData object and add file information
    const formData = new FormData();
    formData.append('file', selectedFile);
    formData.append('totalChunks', totalChunks);

    // Loop through and upload chunks
    for (let chunkNumber = 0; chunkNumber < totalChunks; chunkNumber++) {
      const start = chunkNumber * chunkSize;
      const end = Math.min(start + chunkSize, fileSize);
      const chunk = selectedFile.slice(start, end);
      formData.append(`chunk-${chunkNumber}`, chunk, selectedFile.name);
    }

    // Make a file upload request
    axios.post('/upload', formData, {
      onUploadProgress: progressEvent => {
        const progress = Math.round((progressEvent.loaded / progressEvent.total) * 100);
        setProgress(progress);
      }
    })
      .then(response => {
        console.log('File upload successful:', response.data);
      })
      .catch(error => {
        console.error('File upload failed:', error);
      });
  }
}

Slice uploading and downloading is typically implemented using the file processing functions provided by the front-end library or framework. The back-end service implementation is responsible for handling the uploaded or downloaded chunks and assembling them into the complete file.

In the above example, we implemented slice uploading using the following steps:

The handleFileChange() function is called when the user selects a file. This function saves the selected file to the selectedFile state variable.
The handleFileUpload() function is called when the user clicks the upload button. This function calculates the number of slices and the size of each slice and then creates a FormData object to store the file information and slice data.
The FormData object is then used to make a file upload request to the back-end service.
The back-end service handles the uploaded chunks and assembles them into the complete file.

Solution to Implement Client-side Slice Downloading

The basic solution to implement client-side slice downloading is as follows:

The server side cuts the large file into multiple slices and generates a unique identifier for each slice.
The client sends a request to obtain the slice list and starts downloading the first slice.
During the download process, the client initiates concurrent requests to download other slices based on the slice list, and gradually splices and merges the downloaded data.
When all slices are downloaded, the client merges the downloaded data into a complete file.

Here is an example

function downloadable() {

  // Make a file download request
  fetch('/download', {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json',
    },
  })
    .then(response => response.json())
    .then(data => {
      const totalSize = data.totalSize;
      const totalChunks = data.totalChunks;

      // Initialize variables
      let downloadedChunks = 0;
      let chunks = [];

      // Download each chunk
      for (let chunkNumber = 0; chunkNumber < totalChunks; chunkNumber++) {
        fetch(`/download/${chunkNumber}`, {
          method: 'GET',
        })
          .then(response => response.blob())
          .then(chunk => {
            downloadedChunks++;
            chunks.push(chunk);

            // When all chunks are downloaded
            if (downloadedChunks === totalChunks) {
              // Merge chunks
              const mergedBlob = new Blob(chunks);

              // Create an object URL to generate a download link
              const downloadUrl = window.URL.createObjectURL(mergedBlob);

              // Create an <a> element and set attributes
              const link = document.createElement('a');
              link.href = downloadUrl;
              link.setAttribute('download', 'file.txt');

              // Simulate a click to download
              link.click();

              // Release resources
              window.URL.revokeObjectURL(downloadUrl);
            }
          });
      }
    })
    .catch(error => {
      console.error('File download failed:', error);
    });
}

Let's take a look at the code. First, use the Blob object to create a total object URL, which is used to generate a download connection. Then create a label and set the href attribute to the object URL just created. Continue to set the attribute of the label to download the file name, so that the file can be automatically downloaded when clicked.

Show Download Progress and Completion Status

To display the download progress and completion status on the client, you can implement the following functions:

Progress bar: The client can calculate the overall download progress by tracking the download progress of each slice. This progress can then be displayed in a progress bar to give the user a visual indication of how long the download will take.

Completion status: When all slices have been downloaded, the client can display a completion status, such as a completion icon or text. This lets the user know that the download is complete and that they can now access the file.

Here we can continue the slice upload code example to write:

// Handle file download event

function handleFileDownload() {
  axios.get('/download', {
    responseType: 'blob',
    onDownloadProgress: progressEvent => {
      const progress = Math.round((progressEvent.loaded / progressEvent.total) * 100);
      setProgress(progress);
    }
  })
    .then(response => {
      // Create a temporary URL object for download

      const url = window.URL.createObjectURL(new Blob([response.data]));
      const link = document.createElement('a');
      link.href = url;
      link.setAttribute('download', 'file.txt');
      document.body.appendChild(link);
      link.click();
      document.body.removeChild(link);
    })
    .catch(error => {
      console.error('File download failed:', error);
    });
}

When the user clicks the download button, the handleFileDownload() function is called to handle the file download event.

The handleFileDownload() function uses the axios library to initiate a file download request with the responseType set to blob to return binary data.

The function then listens for the onDownloadProgress event to track the download progress and update the display of the progress bar.

Once the download is complete, the function creates a temporary URL object for downloading and dynamically creates an <a> element to simulate a click and download the file.

Problems and Solutions For Large File Uploads

Large file uploads can be slow, inefficient, and unreliable, but some solutions can improve the performance and stability of the process.

Problems With Traditional File Upload Methods

Large files take a long time to upload and can easily cause timeouts.
They occupy server and network bandwidth resources, which can affect the access speed of other users.
If the upload is interrupted, the entire file needs to be re-uploaded, which is inefficient.
It is difficult to display and control the upload progress.

Advantages of Front-end File Slicing Upload

Splits large files into smaller chunks for faster and more reliable uploads.
Monitors and displays the upload progress to improve the user experience.
Makes full use of the browser's concurrent upload capabilities to reduce the load on the server.
Implements a breakpoint resumption function to avoid re-uploading uploaded chunks.

Method to Implement Front-end Slice Upload

Use the File API in JavaScript to get the file object, and use the Blob.prototype.slice() method to cut the file into multiple slices.

To upload a large file in slices, you can use the FormData object to send the slices to the server via AJAX or the Fetch API.
The backend server will receive the slices and save them temporarily. Once all the slices have been received, the server will merge them into the complete file.
The client can monitor the upload progress event and display the progress in a progress bar or prompt.

Here is an example:

const [file, setFile] = useState(null); // To store the file that I uploaded locally

const chunkSize = 1024 * 1024; // The size of each slice is 1MB

const upload = () => {
  if (!file) {
    alert("Please select a file to upload!");
    return;
  }

  const start = 0;
  const end = Math.min(chunkSize, file.size);

  while (start < file.size) {
    const chunk = file.slice(start, end);

    // Create a FormData object
    const formData = new FormData();
    formData.append("file", chunk);

    // Send the slice to the server
    fetch("upload-api", {
      method: "POST",
      body: formData
    })
      .then(response => response.json())
      .then(data => {
        console.log(data); // Handle the response result
      })
      .catch(error => {
        console.error(error); // Handle the error
      });

    start = end;
    end = Math.min(start + chunkSize, file.size);
  }
};

return (
  <div>
    <input type="file" onChange={handleFileChange} />
    <button onClick={upload}>Upload</button>
  </div>
);

The Upload function component uses React's useState hook to manage the selected file. It monitors changes in the file input box through onChange events and updates the file state in the handleFileChange() function.

When the "Upload" button is clicked, the upload() function is called. This function cuts the file into multiple equal-sized slices and uses FormData objects and the fetch() function to send the slice data to the server

How to Implement Breakpoint Resumption: Record and Restore Upload Status

On the front end, you can use localStorage or sessionStorage to store uploaded slice information, including uploaded slice index and slice size.
Before each upload, check whether the uploaded slice information exists in the local storage. If it exists, continue uploading from the breakpoint.
On the backend, a temporary folder or database can be used to record received tile information, including uploaded tile index and tile size.
Before the upload completes, save the upload status so that upload progress can be resumed if the upload is interrupted.

import React, { useState, useRef, useEffect } from "react";

function Upload() {
  const [file, setFile] = useState(null); // The file that I uploaded locally
  const [uploadedChunks, setUploadedChunks] = useState([]); // The list of chunks that have been uploaded
  const [uploading, setUploading] = useState(false); // Whether the upload is in progress
  const uploadRequestRef = useRef(null); // A reference to the current upload request

  const handleFileChange = (event) => {
    const selectedFile = event.target.files[0];
    setFile(selectedFile);
  };

  const uploadChunk = async (chunk) => {
    // Create a FormData object
    const formData = new FormData();
    formData.append("file", chunk);

    // Send the slice to the server
    return await fetch("your-upload-url", {
      method: "POST",
      body: formData
    })
      .then(response => response.json())
      .then(data => {
        console.log(data); // Handle the response result
        return data;
      });
  };

  const upload = async () => {
    if (!file) {
      alert("Please select a file to upload!");
      return;
    }

    const chunkSize = 1024 * 1024; // 1MB
    const totalChunks = Math.ceil(file.size / chunkSize);

    let start = 0;
    let end = Math.min(chunkSize, file.size);

    setUploading(true);

    for (let i = 0; i < totalChunks; i++) {
      const chunk = file.slice(start, end);
      const uploadedChunkIndex = uploadedChunks.indexOf(i);

      if (uploadedChunkIndex === -1) {
        try {
          const response = await uploadChunk(chunk);
          setUploadedChunks((prevChunks) => [...prevChunks, i]);

          // Save the list of uploaded chunks to local storage
          localStorage.setItem("uploadedChunks", JSON.stringify(uploadedChunks));
        } catch (error) {
          console.error(error); // Handle the error
        }
      }

      start = end;
      end = Math.min(start + chunkSize, file.size);
    }

    setUploading(false);

    // Upload is complete, clear the list of uploaded chunks from local storage
    localStorage.removeItem("uploadedChunks");
  };

  useEffect(() => {
    const storedUploadedChunks = localStorage.getItem("uploadedChunks");

    if (storedUploadedChunks) {
      setUploadedChunks(JSON.parse(storedUploadedChunks));
    }
  }, []);

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <button onClick={upload} disabled={uploading}>
        {uploading ? "Uploading..." : "Upload"}
      </button>
    </div>
  );
}

The Upload function component uses React's useState hook to create a uploadedChunks state to hold the uploaded slice index array, and the useRef hook to create an uploadRequestRef reference to store the current upload request.

When the user selects a file to upload, the handleFileChange() function updates the file state.

The uploadChunk() function sends a slice to the server and returns a Promise object to handle the response.

The upload() function resumes the download at a breakpoint by getting the total number of slices and setting the uploading state to true to disable the upload button. It then iterates through all the slices and checks if the slice index is already contained in the uploadedChunks array. If it is not, the function uploads the slice and adds the uploaded slice index to the uploadedChunks array. The function then uses localStorage to save the uploaded slice information. Finally, after the upload is complete, the function sets the uploading state to false and clears the locally stored slice information.

When uploading large files, it is important to consider the server's processing power and storage space, as well as security issues. It is also important to avoid uploading the same files concurrently to ensure the accuracy of resumed uploads. You can use unique file identifiers or user session identifiers to distinguish them.

Optimize User Experience: Application Scenarios of Slice Upload and Download

Slice upload and download can optimize user experience in the following application scenarios:

File Download and Upload in the Background Management System:

File download: Users may need to download large files, such as reports, log files, and database backups. Slicing files for download can improve download speed and stability, and allow users to interrupt downloads and resume where they left off.

File upload: Users may need to upload large files, such as data imports and file backups. Slice upload can improve upload efficiency, upload file slices in batches, and display the upload status progress.

Image/Video Upload and Preview:

Image Upload and Preview: Speed up image uploads and display progress in real-time with slice upload. After the upload is complete, provide a preview function so users can immediately view the uploaded images.

Video upload and preview: Slice upload ensures reliable and efficient upload of large video files, with real-time progress status display. After the upload is complete, slice download technology allows users to watch the video smoothly without waiting for the entire file to be downloaded.

File Operations on Cloud Storage and Cloud Disk Applications:

File upload: Cloud storage and cloud disk applications can use chunking to improve the upload speed and stability of large files. This allows users to upload large files more quickly and reliably, even if their internet connection is unstable.

File download: When users need to download large files from cloud storage or cloud disks, they can use chunking to speed up the download process. This is especially useful for users with slow internet connections.

File preview and online editing: By slicing files and previewing and editing them online, users can get a better user experience. For example, users can preview and edit large video files without having to wait for the entire file to download.

Conclusion

Multipart file upload and download is an effective solution for optimizing front-end file streaming operations. By slicing large files into multiple small fragments and taking advantage of the breakpoint resume feature, you can improve the reliability and efficiency of file transfers.

The React framework can easily manage and operate file objects, providing a more convenient programming method for multipart file upload and download. Familiarity with tools such as Blob objects and ArrayBuffer is also key to processing and manipulating binary data.

In this article, you have gained an in-depth understanding of the basic concepts and technologies of file stream operations, and how to use React to implement file upload and download in slices. You have also mastered how to handle binary data and leverage streaming and resumable technology to optimize file transfer performance and user experience.

The implementation of file uploading and downloading in slices provides an efficient file processing method for front-end development. It can also cope with the challenges of network limitations and large file transmission. Mastering this knowledge and technologies will give you a powerful toolbox for handling file stream operations in front-end development.

Secure PHP APIs with Signature Verification Using OpenSSL

Shingai Zivuku — Fri, 15 Sep 2023 08:21:37 +0000

Application Programming Interfaces (APIs) are essential in web development. To ensure the security of API requests, it is often necessary to verify the authenticity of these requests. This article will discuss how to design a set of API signature verification programs in PHP, and provide specific design steps and code samples.

Overview: Signature Verification Process With OpenSSL

The API signature verification process is a way to ensure that API requests have not been tampered with or come from an unauthorized source. It works by using a digital signature to bind the request data to the identity of the sender.

The signature verification process typically includes the following steps:

The client signs the request data by encrypting it with their private key. The signature is included in the request parameters.
The server extracts the signature from the request parameters.
The server reproduces the signature by encrypting the request data with the same private key and algorithm.
The server compares the original and reproduced signatures. If they match, the request is authenticated. Otherwise, verification fails.

Implementation Guide

This section provides a step-by-step guide to implementing API signature verification with OpenSSL.

Generate Key Pair

First, generate a pair of public and private keys for encryption and decryption. You can use the openssl extension to generate a key pair. The specific code is as follows:

// Configuration for key generation
$config = array(
  "private_key_bits" => 2048, // Key size: 2048 bits 
  "private_key_type" => OPENSSL_KEYTYPE_RSA, // RSA algorithm
);

// Generate new private and public keys 
$res = openssl_pkey_new($config);

// Extract the generated private key 
openssl_pkey_export($res, $private_key); 

// Extract the public key component
$public_key = openssl_pkey_get_details($res);
$public_key = $public_key["key"]; 

// Save keys to files
file_put_contents('private.key', $private_key);
file_put_contents('public.key', $public_key);

The above code will generate a private key and save it to private.key file, and save the public key to a public.key file.

Implementing Signature Functions

This section covers implementing the core signature functions.

Loading Keys

Before signature generation and verification, you need to load the private and public keys. You can create a Signature class and load the private and public keys in the constructor like this:

class Signature {
    private $private_key;
    private $public_key;

    public function __construct() {
        $this->private_key = openssl_pkey_get_private(file_get_contents('private.key'));
        $this->public_key = openssl_pkey_get_public(file_get_contents('public.key'));
    }

    // Other methods and code will be introduced later
}

Generate Signature

To generate a signature, you need a method that accepts the request parameters and returns the signature. You can use OpenSSL's openssl_sign() function:

public function generateSignature($params) {

  // Convert the request parameters array to a string.
  $data = http_build_query($params);

  // Sign the data with the private key.
  openssl_sign($data, $signature, $this->private_key);

  // Encode the signature result in Base64.
  $signature = base64_encode($signature);

  // Return the signature.
  return $signature;

}

Verify Signature

public function verifySignature($params, $clientSignature) {

  // Convert request params array to string
  $data = http_build_query($params);

  // Base64 decode client's signature
  $clientSignature = base64_decode($clientSignature);

  // Verify signature using public key
  $result = openssl_verify($data, $clientSignature, $this->public_key);

  // Return true if signatures match
  return $result === 1;

}

Use an SDK to call APIs

To make it easier for clients to call an API and perform signature verification, you can create a simple SDK. The SDK will provide encapsulated methods that you can use to easily call APIs and perform signature verification.

Here is the code for a simple SDK:

class APIClient {

  private $apiURL;
  private $signature;

  public function __construct($apiURL, $privateKeyPath, $publicKeyPath) {

    $this->apiURL = $apiURL;

    // Initialize signature validator
    $this->signature = new Signature($privateKeyPath, $publicKeyPath);

  }

  public function callAPI($params) {

    // Generate signature
    $signature = $this->signature->generateSignature($params);

    // Add signature to request parameters
    $params['signature'] = $signature;

    // Send API request
    $response = $this->sendRequest($params);

    // Verify signature
    $isValid = $this->signature->verifySignature($params, $response['signature']);

    // If verification succeeds, return API response data
    if ($isValid) {
      return $response['data'];
    } else {
      throw new Exception("Invalid signature");
    }

  }

  private function sendRequest($params) {

    // Use cURL library to send HTTP request
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $this->apiURL);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params)); 
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    $response = curl_exec($ch);
    curl_close($ch);

    // Parse API response data
    return json_decode($response, true);

  }

}

SDK usage example:


$apiURL = 'http://xyz.com/api.php';
$privateKeyPath = '/path/to/private.key'; 
$publicKeyPath = '/path/to/public.key';

// Create APIClient instance  
$client = new APIClient($apiURL, $privateKeyPath, $publicKeyPath);

// Prepare API request parameters
$params = array(
  'param1' => 'value1',
  'param2' => 'value2', 
);

try {

  // Call API and get response  
  $result = $client->callAPI($params);

  // Handle API response data
  // ...

} catch (Exception $e) {

  // Handle exception
  // ...

}

Conclusion

This article demonstrated implementing API signature verification in PHP by generating a key pair with OpenSSL, creating signature functions with openssl_sign and openssl_verify, and building an SDK to simplify usage. Applying these cryptographic techniques secures API communications by allowing clients to digitally sign requests which can be validated by the server, preventing tampering and ensuring integrity.

The signature binding provides non-repudiation and protects against unauthorized access. While further enhancements are possible, the methods covered provide a solid foundation for applying signatures to safeguard APIs.

JavaScript Mastery: Take Your Skills to the Next Level

Shingai Zivuku — Sun, 10 Sep 2023 18:59:39 +0000

In learning JavaScript, you must understand some basic knowledge, such as variables, functions, classes, loops, etc. These basics are the basis for our use of JavaScript . However, in daily business development, we need some more advanced skills to better solve problems.

By reading this article, you will learn advanced knowledge points and practical application skills of JS, such as advanced data structures and algorithms, functional programming, asynchronous programming and object-oriented programming. I will use code examples to help everyone better understand these knowledge points. At the same time, I will also provide some practical case demonstrations and usage techniques, so that you can better apply these technologies.

Advanced Data Structures and Algorithms

Map and Set Data Structures

The Map data structure is usually used to store key-value pairs, which can use arbitrary types as keys and values. Set data structure is used to store a collection of unique values.

// Create a Map object
const map = new Map();

// Set key-value pairs
map.set('name', 'Tom');
map.set('age', 20);

// Get key-value pairs
console.log(map.get('name')); // 'Tom'
console.log(map.get('age')); // 20

// Create a Set object
const set = new Set();

// Add elements
set.add(10);
set.add(20);
set.add(30);

// Remove elements
set.delete(20);

// Check if an element exists
console.log(set.has(10)); // true
console.log(set.has(20)); // false

Heap, Stack and Queue

Heap and stack are commonly used memory allocation methods. The stack is a last-in, first-out (LIFO) data structure, and the heap is a dynamically allocated memory structure. A queue is a first-in, first-out (FIFO) data structure that is commonly used in caching and concurrent programming.

// Simulate a heap using an array
const arr = [1, 2, 3, 4];
arr.push(5); // Push to the heap
console.log(arr.pop()); // Pop from the heap

// Simulate a stack using an array
const stack = [1, 2, 3, 4];
stack.push(5); // Push to the stack
console.log(stack.pop()); // Pop from the stack

// Simulate a queue using an array
const queue = [1, 2, 3, 4];
queue.push(5); // Push to the queue
console.log(queue.shift()); // Pop from the queue

Depth-first Search and Breadth-first Search

Depth-first search (DFS) and breadth-first search (BFS) are commonly used graph traversal algorithms. DFS is typically used to solve depth-first problems, while BFS is suitable for breadth-first problems.

// Depth-first traversal
function dfs(node) {
  if (node === null) return;
  console.log(node.value); // Print the value of the node
  dfs(node.left); // Recursively traverse the left subtree
  dfs(node.right); // Recursively traverse the right subtree
}

// Breadth-first traversal
function bfs(node) {
  const queue = [node]; // Create a queue and enqueue the root node
  while (queue.length) {
    const curr = queue.shift(); // Dequeue the first node from the queue
    console.log(curr.value); // Print the value of the node
    if (curr.left) queue.push(curr.left); // Enqueue the left child of the node
    if (curr.right) queue.push(curr.right); // Enqueue the right child of the node
  }
}

Commonly Used Algorithms

Commonly used algorithms include sorting, search, and lookup.

Sorting Algorithm: Quick Sort uses the idea of divide and conquer, sorting by dividing the array into smaller blocks.

function quickSort(arr) {
  if (arr.length < 2) {
    return arr;
  }

  let pivot = arr[0];
  let left = [];
  let right = [];

  for (let i = 1; i < arr.length; i++) {
    if (arr[i] < pivot) {
      left.push(arr[i]);
    } else {
      right.push(arr[i]);
    }
  }

  return [...quickSort(left), pivot, ...quickSort(right)];
}

// Search algorithm：
function binarySearch(arr, target) {
  let left = 0;
  let right = arr.length - 1;

  while (left <= right) {
    const mid = Math.floor((left + right) / 2);
    if (arr[mid] === target) {
      return mid;
    } else if (arr[mid] < target) {
      left = mid + 1;
    } else {
      right = mid - 1;
    }
  }

  return -1;
}

Functional Programming

Higher-order Functions and Currying

Higher-order functions and currying are common concepts in functional programming that allow us to create more abstract and flexible functions.

// Higher order function
function higherOrderFunction(func) {
  return function (num) {
    return func(num);
  };
}

function double(num) {
  return num * 2;
}

const doubleFunc = higherOrderFunction(double);
console.log(doubleFunc(10)); // 20

// Currying
function curry(func) {
  return function curried(...args) {
    if (args.length >= func.length) {
      return func.apply(this, args);
    } else {
      return function (...args2) {
        return curried.apply(this, [...args, ...args2]);
      };
    }
  };
}

function sum(a, b, c) {
  return a + b + c;
}

const curriedSum = curry(sum);
console.log(curriedSum(1)(2)(3)); // 6

Closures and Scope

Closures and scope are relatively common concepts in JavaScript . Closures allow us to maintain state within a function, and scope determines the visible range of variables.

// Closure
function closure() {
  let i = 0;
  return function () {
    return ++i;
  };
}

const func = closure();
console.log(func()); // 1
console.log(func()); // 2

// Scope
let a = 10;

function foo() {
  let a = 20;
  console.log(a); // 20
}

foo();
console.log(a); // 10

Common Patterns in Functional Programming

There are many common patterns in functional programming, such as map, filter, and reduce.

// map
const arr = [1, 2, 3];
const mapArr = arr.map((item) => item * 2);
console.log(mapArr); // [2, 4, 6]

// filter
const filterArr = arr.filter((item) => item > 1);
console.log(filterArr); // [2, 3]

// reduce
const reduceArr = arr.reduce((sum, curr) => sum + curr, 0);
console.log(reduceArr); // 6

Asynchronous Programming

Promise and async/await

Promise and async/await are common asynchronous programming methods, which allow us to better deal with problems in asynchronous programming.

// Promise
function promise() {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      resolve('done');
    }, 1000);
  });
}

promise().then((result) => console.log(result)); // 'done'

// async/await
async function asyncFunc() {
  const result = await promise();
  console.log(result);
}

asyncFunc(); // 'done'

Event loop and EventEmitter

// Event loop
console.log('start');
setTimeout(() => {
  console.log('setTimeout');
}, 0);
Promise.resolve().then(() => console.log('promise'));
console.log('end');

// EventEmitter
const { EventEmitter } = require('events');
const emitter = new EventEmitter();

emitter.on('doSomething', (arg1, arg2) => {
  console.log(`${arg1} ${arg2}`);
});

emitter.emit('doSomething', 'Hello', 'World'); // 'Hello World'

Web Worker

Web Workers allow us to move long-running tasks off the main thread to avoid blocking the UI.

// main thread
const worker = new Worker('worker.js');

worker.onmessage = (event) => {
  console.log(event.data);
};

worker.postMessage('start');

// worker.js
self.onmessage = (event) => {
  const result = longCalculation(event.data);
  self.postMessage(result);
};

Object Oriented Programming

Classes and inheritance

Classes and inheritance in JavaScript are similar to other object-oriented programming languages.

class Animal {
  constructor(name) {
    this.name = name;
  }

  speak() {
    console.log(`${this.name} makes a noise.`);
  }
}

class Cat extends Animal {
  constructor(name, breed) {
    super(name);
    this.breed = breed;
  }

  speak() {
    console.log(`${this.name} meows.`);
  }

  get description() {
    return `${this.name} is a ${this.breed} cat.`;
  }

  set nickname(nick) {
    this.name = nick;
  }
}

const cat = new Cat('Fluffy', 'Persian');
cat.speak(); // 'Fluffy meows.'
console.log(cat.description); // 'Fluffy is a Persian cat.'
cat.nickname = 'Fuffy';
console.log(cat.name); // 'Fuffy'

Encapsulation, Inheritance, Polymorphism

Encapsulation, inheritance, and polymorphism are important concepts in object-oriented programming.

// Encapsulation
class Person {
  constructor(name) {
    this._name = name;
  }

  get name() {
    return this._name.toUpperCase();
  }

  set name(newName) {
    this._name = newName;
  }
}

const person = new Person('John');
console.log(person.name); // 'JOHN'
person.name = 'Lisa';
console.log(person.name); // 'LISA'

// Inherit
class Shape {
  constructor(color) {
    this.color = color;
  }

  draw() {
    console.log('Drawing a shape...');
  }
}

class Circle extends Shape {
  constructor(color, radius) {
    super(color);
    this.radius = radius;
  }

  draw() {
    console.log(`Drawing a ${this.color} circle with radius ${this.radius}.`);
  }
}

const circle = new Circle('red', 10);
circle.draw(); // 'Drawing a red circle with radius 10.'

// Polymorphism
function drawShape(shape) {
  shape.draw();
}

drawShape(new Shape('blue')); // 'Drawing a shape...'
drawShape(new Circle('green', 20)); // 'Drawing a green circle with radius 20.'

Conclusion

In this article, I introduced some advanced knowledge points of JavaScript , such as advanced data structures and algorithms, functional programming, asynchronous programming, and object-oriented programming.

📖 Implement concurrent requests through Promise.all

function fetchData(urls) {
  const promises = urls.map((url) => fetch(url));
  return Promise.all(promises).then((responses) =>
    Promise.all(
      responses.map((response) => {
        if (!response.ok) throw new Error(response.statusText);
        return response.json();
      })
    )
  );
}

📖 Use async/await to implement asynchronous calls

async function getData(url) {
  const response = await fetch(url);
  if (!response.ok) throw new Error(response.statusText);
  const data = await response.json();
  return data;
}

📖 Using factory pattern in object-oriented programming

class Product {
  constructor(name, price) {
    this.name = name;
    this.price = price;
  }
}

class ProductFactory {
  createProduct(name, price) {
    return new Product(name, price);
  }
}

const productFactory = new ProductFactory();
const product = productFactory.createProduct('Apple', 1);
console.log(product.name); // 'Apple'
console.log(product.price); // 1

The above are some simple examples, but in practice, you need more complex and specific cases to solve real-world problems. I hope this article can provide you with some starting points, so that everyone can better understand the advanced usage of JavaScript and become a JavaScript master.

DEV Community: Shingai Zivuku

From Disaster to Recovery: A Practical Case Study on Kubernetes etcd Backups

1. Backup Solution Approach

2. Detailed Implementation Steps

Preparation: Creating Storage Paths on the NFS Server

Copy etcd Certificates to the NFS Directory

Writing the PVC Manifest: Creating Persistent Volumes Based on the NFS StorageClass

Key Points:

Writing the Dockerfile

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system

etcd

Building the Image

Writing the CronJob Manifest (Adapted for NFS Mounts)

Testing and Verification: Confirming NFS Backup Success

3. Data Recovery Steps

Pre-recovery Preparation

Perform Recovery

Back Up Existing Data on All etcd Nodes

Post-Recovery Verification and Service Startup

⚠️ Important Considerations

Nginx Deep Dive: Architecture, Configuration, and Practical Examples

Introduction

Introduction to Nginx Basics

History and Background of Nginx

Nginx’s Core Functions

How Nginx Works

Event-Driven Model

Request Processing Flow

Nginx Configuration File Structure

Advanced Features of Nginx

Load Balancing

Caching and Compression

SSL/TLS Configuration

Nginx Practical Case Studies

Static File Server

Reverse Proxy and Load Balancing

URL Rewriting and Redirection

Performance Optimization and Monitoring

Conclusion

Kubernetes Configuration Management: Mastering Helm Values Files for Multi-Environment Deployments

Introduction

Overview of Kubernetes

The Importance of Values Files

Differences in configuration requirements for different environments

Implementation Method

Creating a Helm Chart

Modify the default Values file

Create customized Values files for different environments

Using template languages

Deploying the application

Implementation Case

Creating a Helm Chart

Modify the values.yaml document:

Creating Values files for different environments:

Modify the template file

Deploying the application

Summary

Frontend Refresh Project - An Electronic Spider

HTML Code

Overall process

JavaScript Code

AI empowered micro-services: Future application development driven by Spring Boot and Machine Learning

Overview

Spring Boot

Machine Learning

Combination of AI and Microservices

Practical Cases

Environment preparation

Create Spring Boot project

Define model service interface

Implement model services

Create REST controller

Running and testing

Conclusion

Ansible for Automated Ops & Maintenance: Deployment & Checklist

What is Automated Operations?

Problems Solved By Operations Automation

Classification of Automated Operations Tools

Comparison of Automated Operations Tools

Modify the `values.yaml` document: