Introduction to HarmonyOS Next Build Tool Lycium Principles

Background Introduction

Many system APIs in HarmonyOS Next are provided as C++ interfaces. To use C++ interfaces, NAPI must be used for interaction between ArkTS and C++. In such scenarios, the cross-compilation tools integrated in DevEco-Studio and the CMake build tool are fully sufficient. However, for scenarios involving third-party library migration, such as FFmpeg and OpenSSL, configuring the compilation environment and scripts independently can be cumbersome. Key concerns during cross-compilation include: how to perform cross-compilation with different build methods, how to configure cross-compilation environments for different build platforms, how to configure different cross-compilation architectures, and how to test and verify artifacts after cross-compilation. Currently, open-source C/C++ third-party libraries have diverse compilation methods, with the following mainstream cross-compilation approaches:

• CMake build
• Configure build
• Make build

The official cross-compilation tool Lycium helps quickly build third-party libraries.

Introduction to Lycium Tool

Lycium is a compilation framework tool that assists developers in achieving rapid cross-compilation of C/C++ third-party libraries through shell scripts and quick verification on the OpenHarmony system. Developers only need to set the compilation method and parameters for the corresponding C/C++ third-party library, and Lycium can quickly build binary files operable on the OpenHarmony system.

The construction principle of Lycium is that the transplantation process must not modify source code (i.e., no patching of C/C++ files or build scripts). If patching is necessary for transplantation, it must undergo review with sufficient justification. (Business patches are not accepted.)

Lycium build tool address: https://gitee.com/openharmony-sig/tpc_c_cplusplus

Usage Example

1. Compilation Environment Preparation: The Lycium framework supports third-party libraries with various build methods. To ensure normal compilation, the environment must include basic compilation commands: gcc, cmake, make, pkg-config, autoconf, autoreconf, automake. If any command is missing, download the corresponding toolset from the official website or install it via commands on the build machine. For example, install CMake on Ubuntu with: sudo apt install cmake.
2. Modify Third-Party Library Compilation Methods and Parameters: The Lycium framework provides an HPKBUILD file for developers to configure compilation settings for corresponding C/C++ third-party libraries. Specific steps:
   • Create a new directory for the third-party library named pkgname under the thirdparty directory.
   • Copy the HPKBUILD template file to the new third-party library directory.
   • Modify the HPKBUILD template based on the actual situation of the third-party library, referring to the minizip co-construction for file modification.
3. Quick Compilation of Third-Party Libraries: After configuring the compilation method parameters for the third-party library, execute ./build.sh pkgname in the Lycium directory to automatically compile the library and package it into usr/pkgname/ARCH/ in the current directory.
   • ARCH directory: ./build.sh (compiles all libraries in the thirdparty directory by default).
   • ./build.sh aaa bbb ccc ... (compiles specified libraries aaa, bbb, ccc, etc., in the thirdparty directory; if aaa has dependencies, ensure they are included in the parameters, or aaa will not compile).

The Lycium framework is written in Linux shell scripts. Next, we analyze the shell code of the build tool to understand the build process, which helps locate compilation failures.

Introduction to Build Script Principles

The Lycium framework mainly consists of the following components:

1. HPKBUILD build configuration
2. Top-level build script build.sh
3. Cross-compilation toolchain
4. Test verification environment

Build Process

1. Build Configuration Preparation

Developers need to create a directory for the third-party library to be compiled under the thirdparty directory and write an HPKBUILD build configuration file. The HPKBUILD file defines:

• Source code acquisition method
• Compilation parameter configuration
• Dependency declaration
• Installation rules

HPKBUILD build configuration file example:

# Contributor: Jeff Han <[e-mail]>
# Maintainer: Jeff Han <[e-mail]>
pkgname=FFmpeg
pkgver=n6.0
pkgrel=0
pkgdesc="FFmpeg is a collection of libraries and tools to process multimedia content such as audio, video, subtitles and related metadata."
url="https://github.com/FFmpeg/FFmpeg/"
archs=("armeabi-v7a" "arm64-v8a")
license=("GPL2" "GPL3" "LGPL3" "MIT" "X11" "BSD-styl")
depends=("rtmpdump" "openssl_1_0_2u")
makedepends=()
source="https://github.com/FFmpeg/$pkgname/archive/refs/tags/$pkgver.tar.gz"

autounpack=false
downloadpackage=true
buildtools="configure"

builddir=$pkgname-${pkgver}
packagename=$builddir.tar.gz
source envset.sh
buildhost=true
arch=
ldflags=

prepare() {
    if [ "$LYCIUM_BUILD_OS" == "Linux" ]
    then
        hostosname=linux
    elif [ "$LYCIUM_BUILD_OS" == "Darwi" ]
    then
        hostosname=darwin
    else
        echo "System cannot recognize, exiting"
        return -1
    fi
    if [ $buildhost == true ]
    then
        tar -zxf $packagename
        cd $builddir
        ./configure --enable-static --enable-shared --disable-doc --disable-htmlpages \
            --target-os=$hostosname --disable-optimizations --prefix=`pwd`/hostbuild > $publicbuildlog 2>&1
        $MAKE >> $publicbuildlog 2>&1
        $MAKE install >> $publicbuildlog 2>&1
        export LD_LIBRARY_PATH=`pwd`/hostbuild/lib:$LD_LIBRARY_PATH
        sed -i.bak 's/include $(SRC_PATH)\/tests\/fate\/source.mak/#include $(SRC_PATH)\/tests\/fate\/source.mak/g' tests/Makefile
        $MAKE check >> $publicbuildlog 2>&1
        ret=$?
        buildhost=false
        cd $OLDPWD
    fi

    mkdir $pkgname-$ARCH-build
    tar -zxf $packagename -C $pkgname-$ARCH-build
    cd  $pkgname-$ARCH-build/$builddir
    patch -p1 < ../../FFmpeg_oh_test.patch
    cd $OLDPWD

    if [ $ARCH == "armeabi-v7a" ]
    then
        setarm32ENV
        arch=arm
        ldflags="-L${OHOS_SDK}/native/sysroot/usr/lib/arm-linux-ohos"
    elif [ $ARCH == "arm64-v8a" ]
    then
        setarm64ENV
        arch=aarch64
        ldflags="-L${OHOS_SDK}/native/sysroot/usr/lib/aarch64-linux-ohos"
    else
        echo "${ARCH} not support"
        return -1
    fi

    return $ret
}

build() {
    cd $pkgname-$ARCH-build/$builddir
    PKG_CONFIG_LIBDIR="${pkgconfigpath}" ./configure "$@" --enable-neon --enable-asm --enable-network \
    --disable-vulkan --enable-cross-compile --enable-librtmp --disable-x86asm --enable-openssl --enable-protocols \
    --enable-static --enable-shared --disable-doc --disable-htmlpages --target-os=linux --arch=$arch \
    --cc=${CC} --ld=${CC} --strip=${STRIP} --host-cc="${CC}" --host-ld="${CC}" --host-os=linux \
    --host-ldflags=${ldflags} --sysroot=${OHOS_SDK}/native/sysroot > $buildlog 2>&1
    $MAKE >> $buildlog 2>&1
    ret=$?
    cd $OLDPWD
    return $ret
}

package() {
    cd $pkgname-$ARCH-build/$builddir
    $MAKE install >> $buildlog 2>&1
    cd $OLDPWD
}

checktestfiles() {
    cd $pkgname-$ARCH-build/$builddir/tests/ref

    tmpdir=("fate" "acodec" "lavf" "lavf-fate" "pixfmt" "seek" "vsynth")
    for dir in ${tmpdir[*]}
    do
        for file in `ls $dir`
        do
            if [ ! -f $dir/$file ]; then
                continue
            fi
            str=`cat $dir/$file | grep "\*tests"`
            if [ ! -z "$str" ]
            then
                sed -i.bak 's/\*tests/tests/g' $dir/$file
            fi
        done
    done

    cd $OLDPWD
}

copyhostbin() {
    file=$1
    if [[ -f tests/$file ]] && [[ ! -f tests/$file.${ARCH} ]]
    then
        mv tests/$file tests/$file.${ARCH}
        cp ../../$builddir/tests/$file tests/$file
    fi
}

check() {
    cd $pkgname-$ARCH-build/$builddir
    # disable running cmd
    sed -i.bak 's/  $(Q)$(SRC_PATH)\/tests\/fate-run.sh/#   $(Q)$(SRC_PATH)\/tests\/fate-run.sh/g' tests/Makefile
    # disable check git sources
    sed -i.bak 's/include $(SRC_PATH)\/tests\/fate\/source.mak/#include $(SRC_PATH)\/tests\/fate\/source.mak/g' tests/Makefile
    # disable check ffprobe,this use xmllint command, which ohos is not support
    sed -i.bak 's/include $(SRC_PATH)\/tests\/fate\/ffprobe.mak/#include $(SRC_PATH)\/tests\/fate\/ffprobe.mak/g' tests/Makefile

    # change x86 cmd for generate test target
    mv ffmpeg ffmpeg.${ARCH}
    cp ../../$builddir/ffmpeg ./
    retrytimes=0
    ret=0
    while true
    do
        $MAKE check >> $buildlog 2>&1
        if [ $? -eq 0 ]
        then
            break;
        fi

        copyhostbin base64
        copyhostbin audiomatch
        copyhostbin audiogen
        copyhostbin videogen
        copyhostbin tiny_psnr
        copyhostbin tiny_ssim
        copyhostbin rotozoom

        let retrytimes=$retrytimes+1
        if [ $retrytimes -gt 4 ]
        then
            ret=1
            break
        fi
    done

    mv ffmpeg.${ARCH} ffmpeg
    for file in `ls tests/*.${ARCH}`
    do
        tmpfile=${file%.*}
        mv $file $tmpfile
    done

    # reduction running cmd for real test
    sed -i.bak 's/# $(Q)$(SRC_PATH)\/tests\/fate-run.sh/    $(Q)$(SRC_PATH)\/tests\/fate-run.sh/g' tests/Makefile
    cd $OLDPWD
    checktestfiles

    echo "The test must be on an OpenHarmony device!"
    # skip running test on host
    # real test CMD
    # make check

    return $ret
}

recoverpkgbuildenv() {
    unset arch
    unset ldflags
    if [ $ARCH == "armeabi-v7a" ]
    then
        unsetarm32ENV
    elif [ $ARCH == "arm64-v8a" ]
    then
        unsetarm64ENV
    else
        echo "${ARCH} not support"
        return -1
    fi
}

# 清理环境
cleanbuild() {
    rm -rf ${PWD}/${builddir} ${PWD}/$pkgname-arm64-v8a-build ${PWD}/$pkgname-armeabi-v7a-build #${PWD}/$packagename
}

2. Build Process

The main entry build.sh script executes the following steps:

1. Parse command-line parameters to determine the target library to compile.
2. Check the compilation environment (compilation toolchain, etc.).
3. Read the HPKBUILD configuration of the target library.
4. Compile each library in the order of dependencies.
5. For each library, execute:
   • Acquire source code
   • Configure compilation parameters
   • Perform compilation
   • Install to the specified directory

Compilation environment check code:

# 检测操作系统类型
unames=`uname -s`
osname=${unames:0:5}

# 设置根目录
LYCIUM_ROOT=$(cd $(dirname ${BASH_SOURCE[0]}); pwd)

# 检查 OHOS_SDK 环境
if [ -z ${OHOS_SDK} ]
then
    echo "OHOS_SDK 未设置..."
    exit 1
fi

Dependency management check:

# 依赖库暂存文件
depend_tmp_file="/tmp/$USER-lycium_deps-$build_time"
export LYCIUM_DEPEND_PKGNAMES=$depend_tmp_file

# 已完成库列表
donelist=()
donelibs=()

The core function buildhpk() implements build process control:

1. Execute tasks in rounds.
2. Handle dependency relationships.
3. Implement error handling mechanisms.

Main variables:

• nextroundlist: Next round of tasks to build
• notdonelist: List of incomplete tasks
• buildfalselist: List of build failures

Key function descriptions:

• checkbuildenv(): Check if necessary build tools (e.g., gcc, cmake, make, autoconf, automake, git, curl) are installed.
• prepareshell(): Prepare necessary scripts for each build directory (e.g., build_hpk.sh for project building, envset.sh for environment setup).
• makelibsdir(): Manage build directories (check validity, filter built projects, and add to the build queue).

3. Cross-Compilation Support

The framework achieves cross-compilation through:

1. Using the cross-compilation toolchain provided by the OpenHarmony NDK.
2. Configuring cross-compilation parameters in HPKBUILD.
3. Supporting multi-architecture compilation (arm32/arm64/x86, etc.).

4. Artifact Output

Compilation artifacts are organized as follows:

usr/
  └── ${pkgname}/
       └── ${ARCH}/
            ├── lib/      # Library files
            ├── include/  # Header files
            └── bin/      # Executable files

build_hpk.sh Build Script Description

Core build function descriptions:

1. prepare(): Prepare the build environment (host build when buildhost=true), extract the source package, apply patches, and set up the cross-compilation environment.

2. build(): Execute the build process (configure build parameters, run configure, execute make, and return build results).

3. package(): Install and package (run make install and generate the final installation package).

4. check(): Test and verify (modify test configurations, prepare the test environment, execute test cases, and process results).

5. Environment management functions:

• recoverpkgbuildenv(): Clean up compilation environment variables and restore the original environment.
• cleanbuild(): Clean up build directories and delete temporary files.

Summary

This article introduces the functionality, usage, and principles of the HarmonyOS Next cross-platform build script, as well as the related shell code of the build script.