DEV Community: John Cheng

Introduction to a General IoT Platform Architecture

John Cheng — Thu, 08 Oct 2026 04:04:42 +0000

1. Overview

A typical Internet of Things (IoT) system consists of several major components, including end devices, IoT communication infrastructure, business services, data storage, and user-facing applications.

The primary goal of an IoT platform is to:

Allow users to interact with and manage connected devices through applications such as mobile apps and web applications, while enabling devices to send data, status updates, and events to the cloud and receive commands from cloud services.

A commonly used high-level architecture looks like this:

┌─────────────────────────────────────────────────────────────┐
│                     Application Layer                       │
│                                                             │
│             Mobile App / Web / Mini Program / API           │
└──────────────────────────┬──────────────────────────────────┘
                           │
                     HTTP / WebSocket
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                      Business Services                      │
│                                                             │
│       API Service / Business Service / User Service         │
│                                                             │
│  User Mgmt │ Device Mgmt │ Authorization │ Command Mgmt      │
└───────────────┬──────────────────────┬──────────────────────┘
                │                      │
                ▼                      ▼
        ┌───────────────┐      ┌────────────────┐
        │ Cache / Redis │      │ Database       │
        │               │      │                │
        │ Real-time     │      │ Business data  │
        │ state / cache │      │ / history      │
        └───────────────┘      └────────────────┘
                                       ▲
                                       │
                              ┌────────┴────────┐
                              │                 │
                              │                 │
┌─────────────────────────────┴─────────────────┴─────────────┐
│                    IoT Communication Layer                  │
│                                                             │
│                       MQTT Broker                            │
│                                                             │
│   Message Routing │ Connection Mgmt │ Topic │ QoS │ Session │
└─────────────────────────────┬───────────────────────────────┘
                              │
                            MQTT
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                         Device Layer                        │
│                                                             │
│       MCU / Gateway / Sensor / Controller / IoT Device      │
│                                                             │
└─────────────────────────────────────────────────────────────┘

This is not the only possible architecture, but it represents a common and practical design pattern for IoT systems.

2. Major Components of an IoT System

A typical IoT platform can be divided into several logical layers:

Application Layer
        ↓
Business Service Layer
        ↓
IoT Communication Layer
        ↓
Device Layer

Supporting the entire system:

Data Storage
Caching
Message Queues
Logging & Monitoring
Security

Different products may implement these components differently, but the layered architecture provides a useful and reusable design model.

3. Device Layer

The Device Layer connects the digital system to the physical world.

Devices may include:

Microcontroller-based devices
Sensors
Controllers
Gateways
Industrial equipment
Smart home devices
Vehicles
Embedded systems
Other connected devices

Devices are typically responsible for:

Collecting data
Executing commands
Storing essential local configuration
Establishing communication with the cloud
Reporting status and events
Providing basic offline functionality when network connectivity is unavailable

A device normally does not need to maintain a direct business connection with a mobile application. Instead, it communicates with the cloud through the IoT communication layer.

A typical communication path is:

Device
   │
   │ MQTT
   ▼
MQTT Broker

MQTT is widely used in IoT because of its lightweight communication model, publish/subscribe mechanism, and suitability for devices operating over constrained or unreliable networks.

4. IoT Communication Layer

The IoT Communication Layer is responsible for:

Providing reliable and efficient message exchange between devices and cloud services.

An MQTT Broker is a common component of this layer.

For example:

Device A ──┐
Device B ──┤
Device C ──┼──→ MQTT Broker
Device D ──┤
Device E ──┘

A device can publish messages:

Device
   │
   │ publish
   ▼
MQTT Broker

Cloud services can also publish commands to devices:

MQTT Broker
   │
   │ publish
   ▼
Device

The MQTT Broker primarily handles message transport and routing rather than complex business logic.

Typical broker responsibilities include:

MQTT Broker
    │
    ├── Message routing
    ├── Client connection management
    ├── Topic management
    ├── QoS handling
    └── Session management

Business logic such as:

User authorization
Device management
Business rules
Data processing
Command management

should normally be handled by the Business Service Layer.

5. Business Service Layer

The Business Service Layer is the core of the IoT platform.

It connects:

User Applications
        ↕
Business Services
        ↕
IoT Platform

Typical responsibilities include:

User management
Device management
User-device relationships
Authorization and access control
Command management
Business rules
Device state management
Data processing
APIs
WebSocket connections
Message processing

For example, when a user performs an operation in a mobile app:

APP
 │
 │ WebSocket / HTTP
 ▼
Service
 │
 │ MQTT
 ▼
MQTT Broker
 │
 ▼
Device

The Service validates the user's identity, permissions, and applicable business rules before sending the corresponding command to the device.

After the device executes the command, it can return the result through MQTT:

Device
   │
   │ MQTT
   ▼
MQTT Broker
   │
   ▼
Service
   │
   ├──→ Database
   │
   └──→ WebSocket
          │
          ▼
         APP

This creates a complete end-to-end communication cycle.

6. Communication Between Applications and Services

Mobile and web applications typically do not communicate directly with MQTT Brokers.

A common architecture is:

APP
 │
 ├── HTTP / HTTPS
 │
 └── WebSocket
       │
       ▼
    Service

HTTP/HTTPS is commonly used for operations such as:

Login
Querying data
Updating configuration
Retrieving historical data
Managing user information

WebSocket is particularly useful for:

Real-time status updates
Real-time events
Command execution results
Server-initiated notifications

Both mechanisms can coexist.

For example:

HTTP
 ↓
Query device list
Query historical data
Update configuration

WebSocket
 ↓
Real-time device status
Real-time events
Command results

7. Data Storage Layer

IoT systems can generate large amounts of data. Therefore, different types of data should be handled according to their characteristics and purpose.

Not every piece of data needs to be written directly to a database.

A common approach is:

                    Data
                     │
          ┌──────────┼──────────┐
          │          │          │
          ▼          ▼          ▼
        Cache     Database   Message Queue
          │          │          │
      Real-time   Persistent   Asynchronous
        state        data       processing

7.1 Cache

A caching system such as Redis can be used for:

Device online status
Latest device state
Sessions
Temporary data
Frequently accessed data

The key characteristic is:

A cache is optimized for fast access and should generally not be the only source of persistent data.

7.2 Database

A database is used for data that needs to be stored reliably over time, such as:

Users
Devices
Device configuration
Permissions
Historical records
Operation records
Event records
Command records

The primary role of the database is:

Reliable persistence and queryable storage of business data.

7.3 Message Queue

As the system grows, a message queue can be introduced:

MQTT
  │
  ▼
Service
  │
  ▼
Message Queue
  │
  ├──→ Data Processing
  ├──→ Statistics
  ├──→ Notifications
  ├──→ Logging
  └──→ Other Services

This prevents every MQTT message from having to be processed synchronously inside the message-receiving path.

8. Real-Time Communication vs. Persistent Storage

This is an important principle in IoT architecture.

Suppose a device reports:

temperature = 25.3

The system may need to handle this data in two different ways.

For the current device state:

Real-time state
      ↓
    Redis
      ↓
Current temperature = 25.3°C

For historical analysis:

Historical data
      ↓
   Database
      ↓
Timestamp
Device
Temperature

Therefore:

Real-time communication and persistent data storage are separate concerns.

The database should not simply be used as a relay for every real-time message.

A more appropriate design is:

                 ┌──→ Redis
                 │
MQTT → Service ──┼──→ Database
                 │
                 └──→ Message Queue

Different data paths can then be selected according to their purpose.

9. Commands, Events, and State

It is useful to distinguish three fundamental types of IoT data from the beginning of the system design.

Command

A Command represents:

An instruction requesting a device or service to perform an operation.

Examples:

SetConfig
Reboot
Start
Stop
UpdateFirmware

Typical direction:

APP
 ↓
Service
 ↓
MQTT
 ↓
Device

Event

An Event represents:

Something that happened in the system or on a device.

Examples:

DeviceStarted
SensorTriggered
ConfigurationChanged
ErrorOccurred

Typical direction:

Device
 ↓
MQTT
 ↓
Service
 ↓
APP / Database / Other Service

State

State represents:

The current condition or status of a device or system component.

Examples:

online
offline
running
stopped
temperature
battery
firmware_version

Current state is often stored in a cache for fast access and may also be persisted in a database when required by the business.

10. Example of an End-to-End Data Flow

Consider a user performing an operation on a device through a mobile application.

Step 1: The user initiates an operation

APP
 │
 │ WebSocket
 ▼
Service

Step 2: The Service processes the request

Service
 │
 ├── Authenticate user
 ├── Check authorization
 ├── Check device state
 └── Create Command

Step 3: The command is sent to the device

Service
 │
 │ MQTT
 ▼
MQTT Broker
 │
 ▼
Device

Step 4: The device executes the command

Device
 │
 ├── Execute command
 └── Generate Event

Step 5: The device reports the result

Device
 │
 │ MQTT
 ▼
MQTT Broker
 │
 ▼
Service

Step 6: The Service processes the result

Service
 │
 ├──→ Database
 │
 ├──→ Redis
 │
 └──→ WebSocket
          │
          ▼
         APP

The user can then see the operation result in the application.

11. Why Is This Architecture So Common?

One of the major advantages of this architecture is decoupling.

For example:

APP
 ↓
Service
 ↓
MQTT
 ↓
Device

Each layer has a clearly defined responsibility.

The application does not need to understand the device's low-level communication details.

The device does not need to understand the application's user interface or business logic.

The MQTT Broker does not need to understand complex business rules.

The Service Layer connects these components and implements the business logic between them.

This makes it possible to evolve individual components independently:

Application can change
        ↓
Service remains largely unchanged

Device implementation can change
        ↓
Service remains largely unchanged

MQTT infrastructure can change
        ↓
Business logic remains largely unchanged

This loose coupling makes the system easier to maintain and extend.

12. Architecture as the System Scales

As the number of devices and users increases, the Business Service Layer can be further divided into specialized services:

                         APP / Web
                             │
                       API Gateway
                             │
              ┌──────────────┼──────────────┐
              │              │              │
              ▼              ▼              ▼
         User Service   Device Service   Data Service
              │              │              │
              └──────────────┼──────────────┘
                             │
                       Message Queue
                             │
                             ▼
                       MQTT Service
                             │
                             ▼
                       MQTT Broker
                             │
                             ▼
                          Devices

Supporting infrastructure may include:

                 ┌── Redis
                 │
Services ────────┼── Database
                 │
                 └── Object Storage

This allows individual components to scale independently according to their workloads.

For example:

Device Service
    × 3

MQTT Consumers
    × 10

Data Service
    × 4

There is no need to scale the entire system as a single unit.

13. Security

Security should be treated as a fundamental part of the IoT architecture.

At a minimum, the system should consider:

Users
 │
 ├── Authentication
 ├── Authorization
 └── Session / Token

Devices
 │
 ├── Device ID
 ├── Device Credentials
 ├── TLS
 └── MQTT ACL

Communication
 │
 ├── HTTPS
 ├── WSS
 └── MQTT over TLS

Other important security considerations include:

Device identity and authentication
User authentication
Authorization
Data encryption
Firmware integrity
Key management
Audit logging
Protection against device impersonation
Protection against replay attacks

Security should not be added only after the system is completed. It should be considered from the beginning of the architecture design.

14. Logging, Monitoring, and Operations

Observability is another important part of an IoT platform.

A production system should ideally provide:

                 Monitoring
                     │
       ┌─────────────┼─────────────┐
       │             │             │
     Metrics        Logs         Traces
       │             │             │
       ▼             ▼             ▼
   CPU / Memory    Service       Request
   MQTT sessions   Logs          Tracing
   Device count    Errors        Message flow

This makes it possible to determine whether a problem is caused by:

Device
   ↓
Network
   ↓
MQTT Broker
   ↓
Service
   ↓
Database
   ↓
Application

rather than simply observing that “the application is not responding.”

15. A More Complete General IoT Architecture

Combining the concepts above, a general-purpose IoT platform can be represented as follows:

┌──────────────────────────────────────────────────────────────┐
│                       Application Layer                      │
│                                                              │
│              Mobile App / Web / API / Other Clients         │
└─────────────────────────────┬────────────────────────────────┘
                              │
                       HTTPS / WebSocket
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│                        API Gateway                            │
└─────────────────────────────┬────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│                     Business Services                        │
│                                                              │
│  User │ Device │ Command │ Data │ Notification │ Other       │
└─────────────┬──────────────────────────────┬─────────────────┘
              │                              │
              ▼                              ▼
        ┌───────────┐                  ┌──────────────┐
        │   Redis   │                  │   Database   │
        │   Cache   │                  │              │
        └───────────┘                  └──────────────┘
              ▲                              ▲
              │                              │
              └──────────────┬───────────────┘
                             │
                       Message Queue
                             │
                             ▼
┌──────────────────────────────────────────────────────────────┐
│                     IoT Platform Layer                       │
│                                                              │
│                       MQTT Broker                            │
│                                                              │
│    Connection │ Routing │ QoS │ Session │ Authentication      │
└─────────────────────────────┬────────────────────────────────┘
                              │
                             MQTT
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│                        Device Layer                          │
│                                                              │
│   Sensor │ MCU │ Gateway │ Controller │ Embedded Device      │
└──────────────────────────────────────────────────────────────┘


              ┌───────────────────────────────┐
              │     Security & Observability │
              │                               │
              │ Logs │ Metrics │ Tracing      │
              │ IAM │ TLS │ Audit │ Key Mgmt  │
              └───────────────────────────────┘

16. General Architecture Principles

When planning an IoT platform, the following principles provide a useful starting point.

Principle 1: Use Clear Layers

Define clear responsibilities for:

Application
Service
MQTT
Device
Database

Each layer should focus on its own responsibilities.

Principle 2: Keep Components Decoupled

The application should not depend directly on device implementation details, and devices should not depend on application-specific business logic.

Principle 3: Separate Real-Time Communication from Persistence

Do not treat the database as a general-purpose relay for all real-time messages.

Principle 4: Distinguish Commands, Events, and State

Clearly separating:

Command
Event
State

makes the system easier to understand and extend.

Principle 5: Prefer Stateless Services

Whenever possible, avoid storing critical state inside an individual Service instance.

Instead:

Service × N
    │
    ├── Redis
    ├── Database
    └── Message Queue

This makes horizontal scaling much easier.

Principle 6: Design for Intermittent Connectivity

IoT devices cannot assume that network connectivity will always be available.

Depending on the product requirements, devices should support appropriate mechanisms such as:

Local state
Local configuration
Offline operation
Automatic reconnection
Message retry

17. Summary

A general IoT platform can be understood as:

                Users
                  │
                  ▼
             APP / Web
                  │
                  ▼
               Service
                  │
        ┌─────────┴─────────┐
        │                   │
        ▼                   ▼
      Data                MQTT
        │                   │
        │                   ▼
        │                 Device
        │
        ▼
   Database / Cache

As the system grows, additional components can be introduced:

API Gateway
Message Queue
Microservices
Object Storage
Monitoring
Logging
Tracing
Security

The architecture does not need to be overly complicated at the beginning. A practical evolution path is often:

Phase 1
APP
 ↓
Service
 ↓
MQTT
 ↓
Device
 ↓
Database


Phase 2
Add Redis
Add WebSocket
Add Message Queue


Phase 3
Split Services
Scale MQTT Consumers
Optimize Database
Add Monitoring


Phase 4
Kubernetes
Microservices
Multi-Region Deployment
High Availability
Elastic Scaling

The fundamental goal is not to follow one rigid architecture, but to establish clear boundaries between applications, business logic, communication infrastructure, devices, and data.

A well-designed IoT platform should allow these components to evolve independently while remaining reliable, secure, observable, and scalable as the number of devices and the complexity of the business increase.

Fighting Wildfires with IoT and AI: A Father-Son Open Source Mission

John Cheng — Sat, 05 Sep 2026 05:28:39 +0000

🌲 Falling in Love with Nature—and Facing Wildfires

A while ago, my family and I moved to Canada.

Coming from a bustling industrial background in China, the first thing that struck us was the raw, breathtaking nature. The air is crisp, the lakes and tap water are crystal clear, and humans live side-by-side with wildlife in total harmony. In fact, in our quiet neighborhood, black bears occasionally drop by! They wander through the yard looking for berries or fallen apples, and then quietly move on without ever disrupting our daily routine.

It felt like paradise.

Except for one devastating problem that returns every summer: wildfires.

Year after year, millions of hectares of majestic forests are consumed by roaring fires. Smoke blankets the skies for hundreds of miles, turning the sun into an eerie orange disc, shutting down schools, and forcing families to evacuate.

Watching this repeat year after year broke our hearts. We asked ourselves: Can we use technology to make a difference before fires rage out of control?

🤝 The Father & Son Team: IoT Meets Computer Vision

I have spent decades working in industrial IoT hardware and embedded systems in China—designing rugged field sensors, remote telemetry units, and ultra-reliable communications.

Meanwhile, my son is currently studying Computer Science with a focus on Artificial Intelligence and Computer Vision.

One evening, looking out at the haze during wildfire season, we had an idea: What if we combined our strengths?

The Father (Edge Hardware & Firmware): Build an ultra-reliable, off-grid, low-power IoT sensing node capable of surviving years in remote wilderness.
The Son (Cloud Backend & Multi-Modal AI): Ingest real-time telemetry, filter sensor drift, and run lightweight AI vision models on remote camera snapshots to verify smoke columns and suppress false alarms.

That is how ForestFireGuard was born.

🎥 The Story in Video

I recently recorded a short video sharing our journey, our motivation, and why we decided to make this project 100% open source:

(If the embed doesn't load, you can watch it directly on YouTube.)

🛠️ The Engineering Challenge: Why Wildfire IoT is Hard

Building a wildfire detector for a warm living room is easy. Building one for an isolated boreal forest is an entirely different engineering beast:

Zero External Power: The node must run off a single primary battery and small solar cells for years.
Fragile Cellular Connectivity: In deep forests, 4G signals fluctuate or drop for days. Zero data loss is mandatory.
Severe Hardware RAM Constraints: Running on low-cost, low-power microcontrollers (STM32F030 with only 32KB SRAM) means you cannot allocate massive memory buffers for camera image payloads without risking stack overflows or fragmentation.

🏗️ How ForestFireGuard Works

Here is a look at the system architecture we engineered:

+-----------------------------------------------------------------------------------+
|                            EDGE NODE SUBSYSTEM (mcu/)                             |
|                                                                                   |
|  +--------------------+         +---------------------+                           |
|  | ESP32 Camera Node  |--UART-->|  SPI RAM (64KB)     | (High-Speed Image Cache)  |
|  | (VGA 640x480 JPEG) |         +----------+----------+                           |
|  +--------------------+                    |                                      |
|                                            v                                      |
|  +--------------------+         +---------------------+         +--------------+  |
|  |  Primary Sensors   |--I2C/-->|  SPI Flash (1MB)    |<--FRAM--|  Mainboard   |  |
|  | (GPS, AQ, Temp/Humi|  ADC    | (Append-Only Queue) |  (8KB)  | (STM32F030)  |  |
|  +--------------------+         +----------+----------+         +-------+------+  |
|                                            |                            |         |
+--------------------------------------------|----------------------------|---------+
                                             v                            | TLV
                                  +---------------------+                 v
                                  |   Network Module    |<----------------+
                                  | (EC800CN 4G Cat.1)  |
                                  +----------+----------+
                                             |
                                    MQTT / TCP Cellular
                                             v
+-----------------------------------------------------------------------------------+
|                        SERVER & AI SUBSYSTEM (service/ & Cloud)                   |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  |                       IoT Ingress Service (service/)                        |  |
|  |                     (Receives 4G Payload & Stores to DB)                    |  |
|  +--------------------------------------+--------------------------------------+  |
|                                         |                                         |
|  =======================================|=======================================  |
|                 DATABASE LAYER (Delivery & Separation Boundary)                   |
|  =======================================|=======================================  |
|                                         v                                         |
|  +-----------------------------------------------------------------------------+  |
|  |                   Cloud AI Engine (Son's Pipeline)                          |  |
|  |  * Fetches real-time sensor metrics & VGA photos from DB                   |  |
|  |  * Runs Lightweight CV models (Smoke & Flame detection)                    |  |
|  |  * Dispatches verified high-confidence alerts to emergency channels        |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Key Technical Highlights:

3-Tier Storage for Zero Data Loss:
- 64KB SPI RAM caches incoming JPEG images from the ESP32 camera without touching the STM32's tiny 32KB internal SRAM.
- 1MB SPI Flash operates in append-only mode as an indestructible flight data recorder.
- 8KB Ferroelectric RAM (FRAM) records non-volatile read/write pointers with near-infinite endurance. If the battery dies or connection drops, transmission resumes seamlessly upon reboot.
Time-Multiplexed Power Gating: All power-hungry modules (4G modem, camera, gas sensors) sit behind independent MOSFET switches. The STM32 powers them on sequentially, preventing destructive peak current surges.
Clean Architecture Boundary: The PostgreSQL/Timescale database serves as the strict contract boundary. I ensure rock-solid data ingestion from the field into the DB; my son pulls from the DB for AI training, validation, and real-time inference.

🌍 Why We Are Making It 100% Open Source

Wildfires don't respect borders or patents. Communities in Canada, the US, Southern Europe, Australia, and South America are all battling the same threat.

We believe that by sharing our schematics, PCB gerbers, STM32 firmware, and cloud code with the global developer community, we can help local forestry teams, researchers, and volunteers deploy affordable, early-warning sensor meshes.

💻 Join Us on GitHub!

The project source code is available now on GitHub:
👉 Check out AI part on GitHub (Alex)
👉 Check out Hardware on GitHub(John)

We would love to hear your feedback:

What sensors would you prioritize for forest deployment?
Have you tackled similar memory and power constraints on Cortex-M0 devices?
Want to help with PCB design, embedded C refactoring, or computer vision models?

Drop a comment below or open an issue on GitHub. Let’s build technology that makes a real-world difference together! 🌲🔥🚒

Building an Images Repository - Harbor

John Cheng — Sun, 19 Nov 2023 13:53:59 +0000

Harbor is designed for storing container images and is generally deployed in an intranet environment to provide image services for containers running within the intranet.

Harbor is open source and can be found on GitHub at: https://github.com/goharbor/harbor

Step 1: Testing Environment

For testing purposes, set up a new virtual machine with CentOS 7.9 and make some modifications to the test environment:

Disable the firewall, install iptables, and temporarily disable it:

systemctl stop firewalld
systemctl disable firewalld
yum install iptables-services -y
systemctl stop iptables
systemctl disable iptables
iptables -F

Disable SELinux:

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Synchronize system time:

ntpdata cn.pool.ntp.org
crontab -e

Insert the following entry to synchronize every hour:

* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org

Install common components:

yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack

Set up the repository:

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Enable system forwarding:

echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p

Step 2: Install Docker and Download Docker Compose

Install the latest version of Docker:

yum install -y docker-ce

Enable Docker:

systemctl enable docker --now

Download Docker Compose:

wget https://github.com/docker/compose/releases/download/v2.23.1/docker-compose-linux-x86_64
mv docker-compose-linux-x86_64 /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

Step 3: Create Certificates

During the certificate creation process, ensure consistency in country code and city code. The hostname must match the local machine.

mkdir /data/ssl -p
cd /data/ssl/
openssl genrsa -out ca.key 3072
openssl req -new -x509 -days 3650 -key ca.key -out ca.pem
openssl genrsa -out harbor.key 3072
openssl req -new -key harbor.key -out harbor.csr
openssl x509 -req -in harbor.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out harbor.pem -days 3650

Step 4: Download and Install Harbor

Create a directory:

mkdir /data/install
cd /data/install

Download and extract files:

wget https://github.com/goharbor/harbor/releases/download/v2.3.0-rc3/harbor-offline-installer-v2.3.0-rc3.tgz
tar zxvf harbor-offline-installer-v2.3.0-rc3.tgz -C /data/install

Modify the configuration file:

cp /data/install/harbor/harbor.yml.tmpl /data/install/harbor/harbor.yml
vim /data/install/harbor/harbor.yml

Modify the hostname, certificate, and private_key to actual values:

hostname: harbor # Adjust as needed

certificate: /data/ssl/harbor.pem
private_key: /data/ssl/harbor.key

Install Harbor:

/data/install/harbor/install.sh

Step 5: Web User Management on the Client

Access the IP or domain of this host to open the login page. Initial admin account information:
Username: admin
Password: Harbor12345

Step 6: Configure Docker on the Client

Modify /etc/docker/daemon.json:

vim /etc/docker/daemon.json

Add the following field in the client machine's /etc/docker/daemon.json:

"insecure-registries": "192.168.xxx.xxx", "hostname"

Restart the Docker service:

systemctl daemon-reload
systemctl restart docker

Step 7: Testing

Upload Test: Tag the image on the local machine in the format: ip/project_name/image_name

docker tag tomcat:latest 192.168.xxx.xxx/test/tomcat:v1
docker push 192.168.xxx.xxx/test/tomcat:v1

Download Test:

docker pull 192.168.xxx.xxx/test/tomcat:v1

Building a Private Repository

John Cheng — Sun, 19 Nov 2023 13:50:55 +0000

In the architecture of cloud services, multiple network planes are generally deployed, and the management plane is usually in the intranet environment, accessible through a jump server or VPN technology. This brings significant inconvenience to the installation, upgrade, or system upgrade of applications for the infrastructure of the server. Today, I will introduce a simple method to build a self-built Linux private repository.

Before we begin, you need to prepare a virtual host with dual network cards, or a physical host will work as well. The host should run the Linux operating system; here, we'll use CentOS 7.9 as an example. If you are using a Debian-based Linux system, just replace the commands with apt commands; the principle is exactly the same.

Once the system is installed, connect one network card to the external network and another to the intranet, complete the relevant configurations and tests. Now let's officially start building the private repository.

Step 1: Install and Configure Nginx

Install Nginx

sudo yum install -y nginx

This is a basic operation, and installing Nginx may require additional steps, such as installing the EPEL repository. If you encounter any issues installing Nginx, please search online for related tutorials.

Configure Nginx

Configuring Nginx is straightforward. Just add three lines under the server section:

vim /etc/nginx/nginx.conf

    server {
        autoindex on;    # Add this line
        autoindex_exact_size on;    # Add this line
        autoindex_localtime on;    # Add this line
        listen       80;    # Listening port

Then restart Nginx:

sudo systemctl restart nginx

Step 2: Create a Directory for Sync

This directory is configured in the /etc/nginx/nginx.conf file from the previous step. Here, we create it in the default working directory of Nginx:

mkdir -p /usr/share/nginx/html/rsync/

Step 3: Download Sources

You can download applications individually or synchronize downloads in batches.

Download a Single Application

As an example, let's download haproxy:

mkdir -p /usr/share/nginx/html/rsync/haproxy
sudo yum -y install --downloadonly --downloaddir=/usr/share/nginx/html/rsync/haproxy/ haproxy

Note: This method downloads only applications that have not been installed on the local machine before. If you try to download an application that has been previously installed, such as nginx, you will get nothing.

Batch Download

Here we use the EPEL repository as an example:

mkdir -p /usr/share/nginx/html/rsync/epel/7/x86_64/
rsync -vrt --bwlimit=3000 --exclude=debug/ rsync://rsync.mirrors.ustc.edu.cn/epel/7/x86_64/ /usr/share/nginx/html/rsync/epel/7/x86_64/

Create an Index

When creating the index for the first time, install createrepo:

sudo yum install -y createrepo

Create the index; you need to use this command every time you update the content.

createrepo /usr/share/nginx/html/rsync/

Usage

After these steps, the sync source is successfully installed. On the intranet host, modify the repo file's baseurl:

baseurl = http://xxx.xxx.xxx.xxx/rsync

Alternatively, transfer it to the relevant host and install using the following command:

sudo yum install -y http://xxx.xxx.xxx.xxx/rsync/haproxy/haproxy-1.5.18-9.el7_9.1.x86_64.rpm