DEV Community: Mesak

Building a Composer Proxy Cache with OpenResty

Mesak — Tue, 23 Dec 2025 06:51:38 +0000

Building a Composer Proxy Cache with OpenResty

In enterprise environments or CI/CD workflows, frequent execution of composer install or composer update generates numerous external network requests. This article explains how to build an intelligent Composer proxy cache service using OpenResty (Nginx + Lua), while integrating with private Satis package repositories.

Goals and Motivation

Problems Solved

Redundant Downloads: Every CI/CD run re-downloads the same packages from Packagist and GitHub
Network Latency: Cross-region requests to Packagist (Netherlands) and GitHub (USA)
External Dependencies: Deployment pipeline breaks when GitHub or Packagist experiences downtime
Private Package Integration: Need to use both public packages and internal private packages

Why Nginx Cache?

Compared to other caching solutions (Redis, Varnish), Nginx proxy cache offers these advantages:

Feature	Nginx Cache	Other Solutions
Zero Extra Dependencies	Only Nginx needed	Requires additional services
Filesystem Cache	Stored on disk, survives restarts	Memory cache needs warm-up
Automatic Expiration	Built-in inactive cleanup	Requires custom implementation
Large File Handling	Native streaming support	Potential memory issues
Operational Complexity	Low	Medium to High

Using OpenResty extends Nginx with Lua, enabling complex URL rewriting logic while maintaining high performance.

Why Build Your Own? Limitations of Existing Solutions

Several Composer proxy solutions exist, but each has limitations:

Solution	Limitations
Packagist Mirror	Full mirror requires TB-scale storage, long sync times
Toran Proxy	Commercial software, requires paid license
Satis	Only supports private packages, cannot proxy public Packagist
Artifactory	Enterprise solution, complex architecture, expensive licensing
Pure Nginx Proxy	Cannot modify response body, dist URLs still point externally

Advantages of This Approach:

On-Demand Caching: Only cache packages actually used, not the entire Packagist (300,000+ packages)
Zero License Fees: Entirely based on open-source software
Lightweight Deployment: Single Docker container, low resource requirements
Complete Proxy: Both metadata and dist files are proxied and cached
Private Integration: Seamless integration with Satis private repositories

Architecture Overview

┌─────────────────┐     ┌──────────────────────┐     ┌────────────────┐
│   Composer      │────▶│   OpenResty Proxy    │────▶│ Packagist      │
│   Client        │     │                      │     │ (repo.packagist.org)
└─────────────────┘     └──────────────────────┘     └────────────────┘
                               │
                               ├─▶ Local Satis Static Files
                               └─▶ GitHub/GitLab Download Proxy

Request Flow

Composer sends metadata request to proxy
Proxy checks: Does local Satis have this package?
- Yes → Return local file directly
- No → Proxy to Packagist, cache result
Composer parses metadata, requests actual dist files
Proxy forwards dist requests to GitHub/GitLab, caches result

Satis Integration

What is Satis?

Satis is the official static package repository generator for Composer, creating Packagist-compatible JSON format for private packages.

Standard Satis Workflow

# 1. Configure satis.json
{
    "name": "My Private Packages",
    "homepage": "https://packages.example.com",
    "repositories": [
        { "type": "vcs", "url": "git@gitlab.example.com:team/package-a.git" },
        { "type": "vcs", "url": "git@gitlab.example.com:team/package-b.git" }
    ],
    "require-all": true
}

# 2. Run Satis build
php bin/satis build satis.json ./output

# 3. Output structure
output/
├── packages.json           # Main entry point
├── include/                # Split metadata
│   └── all$xxx.json
└── dist/                   # Package archives
    └── vendor/
        └── package/
            └── package-version-hash.tar

How This Project Integrates Satis

Mount the Satis output directory to /var/www/packagist. The proxy will:

Prioritize Local Files: Serve private packages directly from Satis output
Fallback to Packagist: Proxy public packages to official Packagist
Unified Entry Point: Composer only needs one repository URL

Core Implementation

1. Smart Routing Logic

-- proxy_logic.lua
local root_path = "/var/www/packagist"
local uri = ngx.var.uri

local function file_exists(name)
    local f = io.open(name, "r")
    if f ~= nil then
        io.close(f)
        return true
    end
    return false
end

local file_path = root_path .. uri
if string.sub(uri, -1) == "/" then
    file_path = file_path .. "index.html"
end

if file_exists(file_path) then
    -- Local file exists, serve directly
    return ngx.exec("@satis_local")
else
    -- Proxy to Packagist
    return ngx.exec("@packagist")
end

2. Dist URL Rewriting

JSON metadata from Packagist contains download URLs pointing to GitHub. To cache these downloads, we rewrite URLs to point to our proxy:

-- body_filter.lua (core logic)
local scheme = ngx.var.http_x_forwarded_proto or ngx.var.scheme
local host = ngx.var.http_host
local proxy_base = scheme .. "://" .. host .. "/dist_proxy/"

local new_body = ngx.re.gsub(body, '"url"\\s*:\\s*"(https?)://([^"]+)"', 
    function(m)
        local proto = m[1]  -- "http" or "https"
        local rest = m[2]   -- domain/path

        -- Only process domains we want to proxy
        if string.find(rest, "^api%.github%.com") or 
           string.find(rest, "^codeload%.github%.com") or
           string.find(rest, "^gitlab%.com") then

           return '"url": "' .. proxy_base .. proto .. '/' .. rest .. '"'
        end

        return m[0]  -- Keep other URLs unchanged
    end, "jo")

Rewrite Example:

Original: https://api.github.com/repos/vendor/package/zipball/abc123
Proxied:  https://proxy.example.com/dist_proxy/https/api.github.com/repos/vendor/package/zipball/abc123

3. Removing available-packages

Satis generates packages.json with an available-packages field, which limits Composer to only query listed packages. Remove this field to allow the proxy to handle all packages:

body = ngx.re.gsub(body, '"available-packages"\\s*:\\s*\\[.*?\\]\\s*,?', '', "jos")

4. Nginx Configuration Highlights

# Cache configuration
proxy_cache_path /var/cache/nginx 
    levels=1:2 
    keys_zone=composer_cache:50m 
    max_size=10g 
    inactive=7d 
    use_temp_path=off;

# Dist file proxy
location ~ ^/dist_proxy/(https?)/(.*) {
    set $target_proto $1;
    set $target_url $2;

    proxy_pass $target_proto://$target_url;

    # Long-term cache (versioned files are immutable)
    proxy_cache composer_cache;
    proxy_cache_valid 200 1y;

    # Handle GitHub API 302 redirects
    header_filter_by_lua_block {
        local loc = ngx.header.Location
        if loc then
            local proto, rest = string.match(loc, "^(https?)://(.*)")
            if proto then
                ngx.header.Location = scheme .. "://" .. host .. "/dist_proxy/" .. proto .. "/" .. rest
            end
        end
    }
}

5. SSL Termination Support

When deployed behind a reverse proxy, correctly detect the original protocol to satisfy Composer's secure-http requirement:

local scheme = ngx.var.http_x_forwarded_proto or ngx.var.scheme

Frontend Nginx must pass this header:

proxy_set_header X-Forwarded-Proto $scheme;

Caching Strategy

Type	Cache Duration	Notes
Metadata (JSON)	10 minutes	Balance freshness and cache benefits
Dist files (zip/tar)	1 year	Versioned files are immutable
Max cache size	10 GB	Adjust based on disk space
Inactive cleanup	7 days	Free space from unused files

Docker Compose Configuration

services:
  composer-proxy:
    image: openresty/openresty:alpine
    container_name: composer-proxy
    ports:
      - "8000:80"
    volumes:
      - ./conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro
      - ./cache:/var/cache/nginx
      - ./satis-output:/var/www/packagist:ro
      - ./lua:/usr/local/openresty/nginx/lua:ro
    restart: unless-stopped

Host Directory Permissions

OpenResty container runs as nobody user (UID 65534). Ensure host directory permissions are correct:

Cache Directory

Nginx needs write access to /var/cache/nginx (maps to host ./cache):

# Create directory and set permissions
mkdir -p ./cache
chmod 777 ./cache

# Or more secure: set owner to container's nobody user
chown 65534:65534 ./cache
chmod 755 ./cache

Satis Output Directory

Satis outputs to /var/www/packagist (maps to host ./satis-output), container only needs read access:

# Ensure Satis output directory exists
mkdir -p ./satis-output

# Run Satis build to this directory
php bin/satis build satis.json ./satis-output

# Ensure files are readable
chmod -R 755 ./satis-output

Directory Structure

./
├── cache/              # Nginx cache (requires write permission)
├── satis-output/       # Satis output (read-only mount)
│   ├── packages.json
│   ├── include/
│   └── dist/
├── conf/
│   └── nginx.conf
└── lua/
    ├── proxy_logic.lua
    └── body_filter.lua

Usage

Configure Composer

# Global configuration
composer config -g repo.packagist composer http://localhost:8000

# Revert
composer config -g --unset repo.packagist

Clear Cache

docker exec -it composer-proxy rm -rf /var/cache/nginx/*
docker-compose restart

Performance Improvements

Scenario	Without Cache	With Cache	Improvement
First Laravel install	~45s	~45s	-
Second install	~45s	~12s	73%
10 CI runners simultaneously	450s total requests	~50s total requests	89%

Important Notes

Accept-Encoding: Must be set to empty string, otherwise upstream returns gzip content that cannot be modified in Lua
Content-Length: Must clear this header after modifying body, use chunked encoding

GitHub Rate Limit & Token Authentication

Problem: 504 Timeout Errors

When CI/CD runs frequently, you may encounter GitHub API 504 timeout errors. The main cause is GitHub API Rate Limit:

Authentication	Scope	Requests per Hour
Unauthenticated	Per IP	60
Personal Access Token	Per Token	5,000
GitHub App	Per App	15,000

How Proxy Caching Helps

Using this proxy service significantly reduces GitHub requests:

Without Cache:
  10 CI runners executing composer install simultaneously
  → 10 x 50 packages = 500 GitHub requests
  → Exceeds 60 limit → Rate limit triggered → 504 errors

With Cache:
  First CI runner executes, caches 50 packages
  CI runners 2-10 → Read directly from cache
  → Actual GitHub requests: only 50
  → No rate limit triggered ✓

Best Solution: Cache + Token

Even with caching, setting a GitHub Token is recommended as a safeguard:

Cache Miss: New packages or expired cache still require GitHub requests
Initial Cache Build: Large projects may exceed 60 requests on first run
Traffic Spikes: Multiple projects updating simultaneously

Token Authentication Implementation

Add authentication logic to the dist_proxy location in nginx.conf:

# GitHub Token Authentication (increases rate limit from 60 to 5000/hour)
set_by_lua_block $github_auth {
    local host = ngx.var.target_host or ""
    if string.find(host, "github.com") or string.find(host, "githubusercontent.com") then
        local token = os.getenv("GITHUB_TOKEN") or ""
        if token ~= "" then
            return "token " .. token
        end
    end
    return ""
}
proxy_set_header Authorization $github_auth;

Workflow:

┌─────────────────┐     ┌──────────────────────────────────────┐     ┌─────────────────┐
│   Composer      │────▶│   OpenResty Proxy                    │────▶│   GitHub API    │
│   Client        │     │                                      │     │                 │
└─────────────────┘     │  1. Receive /dist_proxy/https/api... │     │                 │
                        │  2. Check target_host                 │     │                 │
                        │  3. Is github.com?                    │     │                 │
                        │     ├─ Yes → Add Authorization header │     │                 │
                        │     └─ No  → No header                │     │                 │
                        │  4. Proxy request to GitHub           │────▶│ Rate Limit:     │
                        └──────────────────────────────────────┘     │ 5000/hr (w/token)│
                                                                     │ 60/hr (no token) │
                                                                     └─────────────────┘

Configuration

Docker Compose:

services:
  composer-proxy:
    environment:
      - GITHUB_TOKEN=${GITHUB_TOKEN:-}

Create Token:

Go to GitHub Settings → Tokens
Create Fine-grained token with Public repositories (read-only) permission only

Conclusion

By leveraging OpenResty's Lua extension capabilities, we implement complex proxy logic at the Nginx layer, maintaining high performance while gaining flexibility. This solution is particularly suitable for enterprise environments requiring private package integration, and can significantly improve CI/CD pipeline performance.

Docker Environment Composer Local Package Development Tips

Mesak — Mon, 22 Dec 2025 06:27:04 +0000

When developing Laravel applications in a Docker environment, you often need to simultaneously develop custom Composer packages. This article shares an elegant solution using symbolic links.

Problems

Docker containers cannot access host machine absolute paths
Packages scattered across projects, difficult to manage centrally
Package modifications require reinstallation to take effect

Solution

1. Create Unified Package Development Directory

mkdir -p /home/developer/packages/php
mkdir -p /home/developer/project/my-laravel-app/plugins

# Create symbolic links
ln -s /home/developer/packages/php/my-custom-package /home/developer/project/my-laravel-app/plugins/my-custom-package

2. Docker Volume Mounting

# docker-compose.yml
volumes:
    - ./:/var/www/html  # Laravel project mounted to /var/www/html, NGINX usually points to /var/www/html
    - /home/developer/packages/php:/home/developer/packages/php  # Key: Make the same path available inside container

3. Composer Configuration

{
    "repositories": [
        {
            "type": "path",
            "url": "plugins/my-custom-package",
            "options": {
                "symlink": true
            }
        }
    ]
}

4. Independent Version Control for Packages

Each package should have its own Git repository:

cd /home/developer/packages/php/my-custom-package
git init
git remote add origin https://github.com/your-org/my-custom-package.git

Why Independent Git is Needed?

Version Management: Packages have their own evolution history
Cross-project Sharing: Multiple projects can reference different versions
Release Process: Easy to tag and publish to Packagist
Permission Control: Different packages can have different developer permissions
CI/CD: Packages can have independent testing and deployment processes

The main project's .gitignore should exclude the symbolic link directory:

/plugins/

Advantages

Centralized Management: All packages developed in one place
Real-time Updates: Changes take effect immediately, no reinstallation needed
Multi-project Sharing: One codebase serves multiple projects
Container Friendly: Perfect for Docker environments
Independent Publishing: Each package can have independent version control and publishing

Core Principle

Through dual mounting strategy:

Host machine creates symbolic links pointing to unified package directory
Docker mounts the same path, making symbolic links work inside containers
Composer uses relative paths, ensuring consistency between host and container

Restoration Strategy After Development

After package development is complete and published, switch back to official package sources:

1. Publish Package to Official Repository

# In package directory
cd /home/developer/packages/php/my-custom-package
git tag v1.0.0
git push origin v1.0.0
# Publish to Packagist or private Composer repository

2. Update Project's composer.json

{
    "repositories": [
        // Remove or comment out local path
        // {
        //     "type": "path",
        //     "url": "plugins/my-custom-package",
        //     "options": {
        //         "symlink": true
        //     }
        // }
    ],
    "require": {
        "vendor/my-custom-package": "^1.0"  // Change to official version number
    }
}

3. Clean Up Local Symbolic Links

# Remove symbolic links
rm /home/developer/project/my-laravel-app/plugins/my-custom-package

# Reinstall official version
composer update vendor/my-custom-package

4. Restore Docker Configuration (Optional)

If no longer developing other packages, you can remove the package directory mounting:

# docker-compose.yml
volumes:
    - ./:/var/www/html
    # Remove package mounting
    # - /home/developer/packages/php:/home/developer/packages/php

This completes the transition from development mode to official release mode, ensuring the project uses stable official versions!

This way you can enjoy the convenience of local package development in a Docker environment!

gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

Mesak — Wed, 17 Dec 2025 03:17:05 +0000

[AI Collaboration Note] gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

This article documents an optimization strategy adopted during the development of a high-performance database middleware (hypool) to address gRPC transmission efficiency and type limitations. The solution, proposed with the assistance of AI, draws inspiration from low-level database storage principles to resolve two major challenges faced when transmitting large database result sets via traditional gRPC.

1. Background and Technical Challenges

When designing APIs for database middleware, we typically need to return multi-row query results. Using traditional gRPC definitions can lead to the following performance and implementation issues.

1.1 Payload Redundancy (Key Repetition)

The most intuitive Protobuf definition is to map each row of data to a Map or Object:

message Row {
    map<string, string> data = 1;
}
message Response {
    repeated Row rows = 1;
}

Problem Analysis:
This structure leads to severe payload bloating. Assuming a query result has 10,000 records containing fields customer_id, created_at, and status, the field names (Keys) are transmitted repeatedly 10,000 times, consuming significant bandwidth resources.

1.2 Protobuf NULL Value Limitations

The design philosophy of Protobuf (proto3) treats scalar types as non-nullable.

If a string field is NULL, it is serialized as an empty string "".
The Client cannot distinguish between an "empty value" and a "NULL value in the original database".

Although Wrapper types like google.protobuf.StringValue can solve this, they add extra message nesting levels and processing overhead.

2. Solution: Database-Like Low-Level Architecture

Addressing these challenges, the AI suggested breaking away from traditional API object thinking and referencing Database Columnar Storage or ODBC/JDBC driver implementations.

The core optimization strategy consists of two parts:

A. Flattening

Primarily used to solve the Key-Value redundancy problem.

Instead of transmitting an "Array of Objects", we flatten the data structure:

Header (Metadata): Transmit field definitions (columns) only once.
Body (Values): Flatten all data values into a massive one-dimensional array (values).

This design completely removes Key transmission from each row, significantly reducing payload size.

B. Bitset (Bitmap) Mechanism

Primarily used to solve the NULL value marking problem.

Since string is used to transmit "Values", we introduce an extra binary field (bytes) to specifically record "State":

Use a Bitset to mark whether each value is NULL.
1 bit corresponds to 1 value.
- Bit = 1: Indicates the value is NULL.
- Bit = 0: Indicates the value is valid.

Space Efficiency:
Every 8 field values consume only 1 byte of extra space. For 1,000 rows x 8 columns, only about 1 KB of overhead is needed to accurately record all NULL states.

3. Implementation Essentials

This section demonstrates key code for "Compression" on the Server side and "Restoration" on the Client side.

3.1 Protobuf Definition (`proto/query.proto`)

message QueryResponse {
    repeated string values = 3;   // Flattened values
    repeated Column columns = 4;  // Field definitions
    bytes null_bitmap = 7;        // NULL marker bitstream
    int32 row_count = 6;
}

3.2 Server Side: Encoding and Compression

The Server's task is to iterate through database results once, simultaneously completing "Value Flattening" and "Bitmap Generation".

// $result['rows'] is the 2D array returned by the database
$values = [];
$packedBytes = "";
$currentByte = 0;
$bitIndex = 0;

foreach ($result['rows'] as $row) {
    foreach ($row as $value) {
        if ($value === null) {
            $values[] = "";             // Placeholder for empty string
            $currentByte |= (1 << $bitIndex); // Mark as NULL (Set bit to 1)
        } else {
            $values[] = (string)$value; // Store actual value
            // Non-NULL, Bit remains 0
        }

        // Write a byte every 8 bits
        if (++$bitIndex === 8) {
            $packedBytes .= chr($currentByte);
            $currentByte = 0;
            $bitIndex = 0;
        }
    }
}
// Write remaining bits
if ($bitIndex > 0) {
    $packedBytes .= chr($currentByte);
}

3.3 Client Side: Decoding and Restoration

Upon receiving data, the Client needs to slice it according to the number of columns and refer to the null_bitmap to restore NULLs.

$fetchedRows = [];
$columns = $response->getColumns();
$colCount = count($columns);
$values = $response->getValues();     // Get flattened array
$bitmap = $response->getNullBitmap(); // Get Bitmap string

$rowCount = $response->getRowCount();

for ($r = 0; $r < $rowCount; $r++) {
    $row = [];
    for ($c = 0; $c < $colCount; $c++) {
        // Calculate absolute index in the 1D array
        $flatIndex = ($r * $colCount) + $c;

        // [Core] Bitset Lookup
        // 1. Find corresponding Byte position
        $bytePos = (int)($flatIndex / 8); 
        // 2. Find Bit position within the Byte
        $bitPos = $flatIndex % 8;

        // 3. Check if the Bit is 1
        $isNull = (ord($bitmap[$bytePos]) >> $bitPos) & 1;

        if ($isNull) {
            $row[] = null; // Accurately restore NULL
        } else {
            $row[] = $values[$flatIndex];
        }
    }
    $fetchedRows[] = $row;
}

Through the symmetric logic above, we achieve compression and restoration of data with extremely low computational cost.

4. Optimization Benefits Analysis

Adopting this architecture yields the following specific benefits:

Extreme Transmission Efficiency
- Through the flattening design, payload size grows linearly with data volume, unaffected by field name length. In large data query scenarios, bandwidth savings are extremely significant.
Precise Type Restoration
- The Client can accurately restore database NULL states by reading the null_bitmap, solving the limitations of gRPC default types.
Parsing Performance Improvement
- For PHP and other languages, processing flat one-dimensional arrays (Indexed Arrays) generally offers better CPU Cache hit rates and lower memory fragmentation compared to processing massive complex nested objects.

5. Conclusion

This optimization case demonstrates the importance of moderately introducing low-level system design thinking in modern distributed systems. Through collaboration with AI, we moved beyond the framework of simple API design, utilizing Bitwise Operations and Data Structure Optimization to solve inherent limitations of gRPC/Protobuf in database application scenarios at a very low cost.

[AI 協作筆記] gRPC 傳輸優化：基於 Flattening 與 Bitset 的高效方案

Mesak — Wed, 17 Dec 2025 03:13:08 +0000

[AI 協作筆記] gRPC 傳輸優化：基於 Flattening 與 Bitset 的高效方案

本篇文章記錄了在開發高效能資料庫中間件 (hypool) 時，針對 gRPC 傳輸效率與型別限制所採用的優化方案。該方案由 AI 協助提出，借鑑了資料庫底層儲存原理，解決了傳統 gRPC 在傳輸大量資料庫結果集時面臨的兩個主要挑戰。

1. 背景與技術挑戰

在設計資料庫中間件的 API 時，我們通常需要回傳多行的查詢結果。若採用傳統的 gRPC 定義方式，會面臨以下效能與實作上的問題。

1.1 Payload 冗餘問題 (Key Repetition)

最直觀的 Protobuf 定義通常是將每一行資料定義為一個 Map 或 Object：

message Row {
    map<string, string> data = 1;
}
message Response {
    repeated Row rows = 1;
}

問題分析：
這種結構會導致嚴重的 Payload 膨脹。假設查詢結果有 10,000 筆資料，且包含欄位 customer_id, created_at, status。在傳輸過程中，這些欄位名稱 (Key) 會被重複傳輸 10,000 次，佔用了大量的頻寬資源。

1.2 Protobuf 對 NULL 值的限制

Protobuf (proto3) 的設計哲學將純量型別 (Scalar Types) 視為不可為空。

string 欄位若為 NULL，傳輸時會被序列化為空字串 ""。
Client 端無法區分這是「空值」還是「原始資料庫中的 NULL」。

雖然可以使用 google.protobuf.StringValue 等 Wrapper 類型解決，但這會增加額外的 Message 嵌套層級與處理開銷。

2. 解決方案：類資料庫底層架構

針對上述挑戰，AI 建議跳脫傳統的 API 物件思維，轉而參考 資料庫底層 (Columnar Storage) 或 ODBC/JDBC 驅動 的實作方式。

核心優化策略包含以下兩個部分：

A. 陣列扁平化 (Flattening)

主要用於解決 Key-Value 冗餘問題。

我們不再傳輸「物件陣列 (Array of Objects)」，而是將資料結構扁平化：

Header (Metadata)：單獨傳輸一次欄位定義 (columns)。
Body (Values)：將所有資料值攤平成一個巨大的一維陣列 (values)。

這種設計完全移除了每行資料中的 Key 傳輸，顯著降低 Payload 大小。

B. Bitset (Bitmap) 機制

主要用於解決 NULL 值標記問題。

既然 string 用來傳輸「值」，我們便引入一個額外的二進位欄位 (bytes) 來專門記錄「狀態」：

使用 Bitset 來標記每一個值是否為 NULL。
1 個 bit 對應 1 個值。
- Bit = 1: 表示該值為 NULL。
- Bit = 0: 表示該值有效。

空間效率：
每 8 個欄位值僅需消耗 1 byte 的額外空間。對於 1,000 行 x 8 欄的資料，僅需約 1 KB 的 Overhead 即可精確記錄所有 NULL 狀態。

3. 實作細節 (Implementation Essentials)

本節將展示 Server 端如何「壓縮」與 Client 端如何「還原」的關鍵程式碼。

3.1 Protobuf 定義 (`proto/query.proto`)

message QueryResponse {
    repeated string values = 3;   // 扁平化數值
    repeated Column columns = 4;  // 欄位定義
    bytes null_bitmap = 7;        // NULL 標記位元流
    int32 row_count = 6;
}

3.2 Server 端：編碼與壓縮 (Encoding)

Server 端的任務是一次性遍歷資料庫結果，同時完成「數值扁平化」與「Bitmap 生成」。

// $result['rows'] 是資料庫回傳的二維陣列
$values = [];
$packedBytes = "";
$currentByte = 0;
$bitIndex = 0;

foreach ($result['rows'] as $row) {
    foreach ($row as $value) {
        if ($value === null) {
            $values[] = "";             // 值放空字串佔位
            $currentByte |= (1 << $bitIndex); // 標記 NULL (Bit設為1)
        } else {
            $values[] = (string)$value; // 存入實際值
            // 非 NULL，Bit 保持 0
        }

        // 每滿 8 個 bits 就寫入一個 byte
        if (++$bitIndex === 8) {
            $packedBytes .= chr($currentByte);
            $currentByte = 0;
            $bitIndex = 0;
        }
    }
}
// 寫入剩餘的 bits
if ($bitIndex > 0) {
    $packedBytes .= chr($currentByte);
}

3.3 Client 端：解碼與還原 (Decoding)

Client 端收到資料後，需要根據 columns 數量進切割，並參考 null_bitmap 將 NULL 還原回來。

$fetchedRows = [];
$columns = $response->getColumns();
$colCount = count($columns);
$values = $response->getValues();     // 取得扁平化陣列
$bitmap = $response->getNullBitmap(); // 取得 Bitmap string

$rowCount = $response->getRowCount();

for ($r = 0; $r < $rowCount; $r++) {
    $row = [];
    for ($c = 0; $c < $colCount; $c++) {
        // 計算在一維陣列中的絕對索引
        $flatIndex = ($r * $colCount) + $c;

        // [精華] Bitset 查表法
        // 1. 找到對應的 Byte 位置
        $bytePos = (int)($flatIndex / 8); 
        // 2. 找到 Byte 中的 Bit 位置
        $bitPos = $flatIndex % 8;

        // 3. 檢查該 Bit 是否為 1
        $isNull = (ord($bitmap[$bytePos]) >> $bitPos) & 1;

        if ($isNull) {
            $row[] = null; // 精確還原 NULL
        } else {
            $row[] = $values[$flatIndex];
        }
    }
    $fetchedRows[] = $row;
}

透過上述兩段對稱的邏輯，我們即可以極低的運算成本完成資料的壓縮與還原。

4. 優化效益分析

採用此架構後，我們獲得了以下具體效益：

極致的傳輸效率
- 透過扁平化設計，Payload 大小與資料量呈線性單純增長，不受欄位名稱長度影響。在大數據量查詢中，頻寬節省效果極為顯著。
精確的型別還原
- Client 端可透過讀取 null_bitmap 精確還原資料庫的 NULL 狀態，解決了 gRPC 預設型別的限制。
解析效能提升
- 對於 PHP 與其他語言而言，處理扁平的一維陣列 (Indexed Array) 通常比處理大量複雜的巢狀物件 (Nested Objects) 擁有更好的 CPU Cache 命中率與更低的記憶體碎片。

5. 總結

這個優化案例展示了在現代分散式系統中，適度引入底層系統設計思維的重要性。透過與 AI 的協作，我們跳脫了單純的 API 設計框架，利用 位元運算 與 資料結構優化，以極低的成本解決了 gRPC/Protobuf 在資料庫應用場景下的先天限制。

Vue Transitions can't use for nested routes's KeepAlive

Mesak — Thu, 10 Aug 2023 01:44:33 +0000

The following scenario is what I encountered while using the soybean-admin frontend UI framework.

When using nested routes, the KeepAlive component caches routed elements. However, when reusing them, the transition animation component cannot place the elements.

This issue arises because the frontend UI framework automatically retains components of the secondary menu, leading to this problem. I am not certain whether the root cause is the KeepAlive or the Transition component, but in any case, excluding components with nested routes from entering the cache can resolve this problem.

The following is the original code where the problem occurred
soybean-admin/blob/v0.10.3

Laravel package api-response

Mesak — Sat, 31 Dec 2022 10:13:29 +0000

I mentioned a method before, which is to use the unified processing of the controller return value to output the API response. For this reason, I wrote my method into a package to make it more convenient to use.

package in :
https://github.com/mesak/laravel-api-response

in Laravel 8 or 9

use command :

composer require mesak/laravel-api-response

your can install this package

then use two step

change Controller and change Exception\Handler

Exception\Handler

change your app/Exceptions/Handler.php file to extend Mesak\LaravelApiResponse\Exceptions\Handler

namespace App\Exceptions;

use Mesak\LaravelApiResponse\Exceptions\Handler  as ExceptionHandler;

Controller

change your api controller, to extend Mesak\LaravelApiResponse\Http\Controllers\Controller

namespace App\Http\Controllers;

use Mesak\LaravelApiResponse\Http\Controllers\ApiController as BaseController;

class MainController extends BaseController
{

}

then enjoy this

Laravel FormRequest With Opis Validator

Mesak — Sun, 14 Aug 2022 17:25:16 +0000

I published a package for using Opis JSON Schema to FormRequest on laravel

https://github.com/mesak/laravel-opis-validator

installation

composer require mesak/laravel-opis-validator

Example

Requests

<?php

namespace App\Http\Requests;

use Mesak\LaravelOpisValidator\JsonSchemaRequest;

class JsonSchema extends JsonSchemaRequest
{
    protected $extendValidatorMessage = true;

    /**
     * Get the validation rules that apply to the request.
     *
     * @return array
     */
    public function rules()
    {
        return [
            '$schema' => "http://json-schema.org/draft-07/schema#",
            "type" => "object",
            "title" => "Base Preference",
            "description" => "Base Preference Setting",
            "properties" => [
                "limit" => [
                    "type" => "integer",
                    "minimum" => 5,
                    "maximum" => 15,
                    "title" => "limit",
                    "attrs" => [
                        "placeholder" => "limit (limit)"
                    ]
                ],
                "page" => [
                    "type" => "object",
                    "title" => "Page",
                    "attrs" => [
                        "placeholder" => "Page ( Page )"
                    ],
                    "properties" => [
                        "limit" => [
                            "type" => "integer"
                        ]
                    ]
                ]
            ],
            "additionalProperties" => false,
            "required" => [
                "limit",
                "page"
            ]
        ];
    }

Controller


use App\Http\Requests\JsonSchema as JsonSchemaRequest;

    public function update(JsonSchemaRequest $request)
    {
        dd($request->validated());
    }

Laravel Controller use callAction to meke ApiController

Mesak — Fri, 29 Jul 2022 14:00:00 +0000

In most projects, the controller will be used to return the response to specify JSON to determine the content.

I found that in laravel's routing, ControllerDispatcher uses callback function to call the corresponding Method

Illuminate\Routing\ControllerDispatcher

    public function dispatch(Route $route, $controller, $method)
    {
        $parameters = $this->resolveClassMethodDependencies(
            $route->parametersWithoutNulls(), $controller, $method
        );

        if (method_exists($controller, 'callAction')) {
            return $controller->callAction($method, $parameters);
        }

        return $controller->{$method}(...array_values($parameters));
    }

You can see that the Controller will check whether your Controller has a function called callAtion before executing it. If not, it will return and call the Controller method specified by the route.

This led me to an idea that when Controller implements API or WEB routing, it must return a response to display content

Among them, most people will unify json for API return, but the implementation methods are quite diverse.

I have seen the Response::macro method, or the ApiResponseTrait method, but this method will make me feel difficult to maintain later,

You will see that your Controller will have a lot of judgments to return 200 or 400 or 200 + content of http status code.

API Controller

In my own project, I built an ApiController, let all the content under the API folder in the route inherit ApiController, intercept all results, and convert them into API Response

ApiController

    public function callAction($method, $parameters)
    {
        $result =  $this->{$method}(...array_values($parameters));
        $httpStatus = ($method  === 'store') ? 201 : 200;
        if ($result instanceof ResponseSchema) {
            $response = &$result;
        } else {
            $response = new ResponseSchema($result);
        }
        $response->setStatus($httpStatus);
        return response()->json($response, $httpStatus);
    }

I created a ResponseSchema Class, specifically defined the JSON Schema format, and then returned it uniformly. The content returned by the Controller will be the result of successful execution. The only small difference is that the newly added method is unified and I call it store, which will put http status code returns 201.

Since only successful information is returned here, the Controller does not need to write a bunch of try catches, and can focus on returning the necessary information.

Error Handler

How to handle error handling? According to the-exception-handler mentioned on the official website, there is a special error handling Handler for errors, so only App\Exceptions\Handler , This program focuses on handling errors. Loading ResponseSchema in the render function and spit out the corresponding method can unify the JSON Response Schema.

    public function render($request, Throwable $exception)
    {
        $responseSchema = new ResponseSchema();
        $statusCode = 500;
        $responseSchema->setException($exception);
        $responseSchema->setMessage('Server Error');
        $responseSchema->setStatus($statusCode);
        if ($request->is(['api/*', 'oauth/*'])) {
            $request->headers->set('Accept', 'application/json');
            return response($responseSchema->toArray(), $statusCode);
        }
        return parent::render($request, $exception);
    }

At present, this method is my current method of implementing API Response, so that I can focus on Controller writing and achieve the purpose of hierarchical division of labor

Simple REST API Use Vue Version

Mesak — Thu, 26 May 2022 16:35:42 +0000

DEMO: Simple REST API Use Vue 3 Version

This example is mainly to present, use the functions of ES6 Classes to achieve API encapsulation and access of the same type ...

this is use Vue Framework, JavaScript Version

Source & Dependencies

Simple REST API

Mesak — Wed, 25 May 2022 17:41:00 +0000

DEMO: Simple REST API

This example is mainly to present, use the functions of ES6 Classes to achieve API encapsulation and access of the same type ...

Source & Dependencies

REST API
template7
lodash
axios
sweetalert2 (css include fail :( )

傳送物件/陣列到 FORMDATA

Mesak — Sun, 08 May 2022 03:31:48 +0000

JS 自從有了打包工具之後，很多工具造福不少開發者，以往要送個 AJAX 到後端，大多都是一個頁面處理一個 method

框架化後，每個程式碼開始有了物件化、封裝方式來做，現在大多都用 axios 來送 AJAX

但是遇到一個問題就是送檔案的時候，需要使用到 FormData 的物類別物件。

但是整個像後端送的程式碼都封裝起來了，要針對一個檔案上傳的方法，做出額外處理，當檔案上傳處理頁面多了之後維護除錯方法就有點麻煩。

所以比較好的方式還是修改 API 送出前，先把物件轉換好判定是否有 File 類型，然後轉換成 FormData 就可以一勞永逸

問題來了，以往直接用 axios 傳送到後端，一個 object 就可以搞定 value is array 的問題，開發者似乎不用煩惱太多後續處理問題

頂多就是遇到 GET 網址參數需要轉換純文字的狀態，但也也只需要一個 qs 套件就可以解決解決

但如果採用 FormData 這個類型的時候，整個 object 就必須自己處理，如果遇到傳送一個陣列的時候，就必須額外針對 key = array 的狀態去處理，然後如果 value = array 也是很麻煩的事情。

舉個例子，往後端送 array1 = [1,2,3] ，如果是 axios 就可以直接送，如果是送 FormData 就必須處理成

array1[]:1;
array1[]:2;
array1[]:3;

這點可以利用 PostMan 來做測試看看後端可以吃到怎樣結構的陣列

這邊原本的想法是，把物件的每個 value 歷遍，只要 value instanceof File 就成立條件，把 key 轉換成陣列

不過找了很多遞迴解法，都沒有 qs 轉換陣列來的快速有效，最終還是爬了一下 qs 的實際操作用法，

  const files = new Map()
  const queryString = qs.stringify(data, {
    arrayFormat: 'brackets',
    encode: false,
    filter: (name, value) => {
      if (value instanceof File) {
        let id = lodash.uniqueId('__FILE__.')
        files.set(id, value)
        return id
      } else {
        return value
      }
    }
  })

利用 qs 的 filter method 把所有 value 檔案移到暫存變數 files 裡面，利用 lodash 取得目前唯一的數值，將原先的 data (object) 的 File 類型 value，全部轉換成 file 的 uniqueId，最終就可以得到一串處理過的字串，快速的將物件陣列轉換成字串

a=__FILE__.1&b[a]=1&b[b]=2 ...

只要有了這個純字串的東西，要拆解成陣列 + 物件就方便多拉，拆完之後遇到 FILE 的 value 記得去把 files 裡面的檔案給領回來，這樣塞入 FormData 就可以解決陣列 key 或是 value = array 的問題了

完整範例放在 codepen.io

DEV Community: Mesak

Building a Composer Proxy Cache with OpenResty

Building a Composer Proxy Cache with OpenResty

Goals and Motivation

Problems Solved

Why Nginx Cache?

Why Build Your Own? Limitations of Existing Solutions

Architecture Overview

Request Flow

Satis Integration

What is Satis?

Standard Satis Workflow

How This Project Integrates Satis

Core Implementation

1. Smart Routing Logic

2. Dist URL Rewriting

3. Removing available-packages

4. Nginx Configuration Highlights

5. SSL Termination Support

Caching Strategy

Docker Compose Configuration

Host Directory Permissions

Cache Directory

Satis Output Directory

Directory Structure

Usage

Configure Composer

Clear Cache

Performance Improvements

Important Notes

GitHub Rate Limit & Token Authentication

Problem: 504 Timeout Errors

How Proxy Caching Helps

Best Solution: Cache + Token

Token Authentication Implementation

Configuration

Conclusion

Docker Environment Composer Local Package Development Tips

Problems

Solution

1. Create Unified Package Development Directory

2. Docker Volume Mounting

3. Composer Configuration

4. Independent Version Control for Packages

Advantages

Core Principle

Restoration Strategy After Development

1. Publish Package to Official Repository

2. Update Project's composer.json

3. Clean Up Local Symbolic Links

4. Restore Docker Configuration (Optional)

gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

[AI Collaboration Note] gRPC Transmission Optimization: An Efficient Solution Based on Flattening and Bitset

1. Background and Technical Challenges

1.1 Payload Redundancy (Key Repetition)

1.2 Protobuf NULL Value Limitations

2. Solution: Database-Like Low-Level Architecture

A. Flattening

B. Bitset (Bitmap) Mechanism

3. Implementation Essentials

3.1 Protobuf Definition (proto/query.proto)

3.2 Server Side: Encoding and Compression

3.3 Client Side: Decoding and Restoration

4. Optimization Benefits Analysis

5. Conclusion

[AI 協作筆記] gRPC 傳輸優化：基於 Flattening 與 Bitset 的高效方案

[AI 協作筆記] gRPC 傳輸優化：基於 Flattening 與 Bitset 的高效方案

1. 背景與技術挑戰

1.1 Payload 冗餘問題 (Key Repetition)

1.2 Protobuf 對 NULL 值的限制

2. 解決方案：類資料庫底層架構

A. 陣列扁平化 (Flattening)

B. Bitset (Bitmap) 機制

3. 實作細節 (Implementation Essentials)

3.1 Protobuf 定義 (proto/query.proto)

3.2 Server 端：編碼與壓縮 (Encoding)

3.3 Client 端：解碼與還原 (Decoding)

4. 優化效益分析

5. 總結

Vue Transitions can't use for nested routes's KeepAlive

3.1 Protobuf Definition (`proto/query.proto`)

3.1 Protobuf 定義 (`proto/query.proto`)

傳送物件/陣列到 FORMDATA