DEV Community: project-laguardia

LuCI on MGMT - Wk 2 - Day 08

project-laguardia — Thu, 17 Jul 2025 05:22:16 +0000

Hurdle 11: Prioritizing Ports and Setting Up VCS

Previously:

https://github.com/project-laguardia/lumi.working/blob/9b6835f4b7f116db9fdc83a63b6ee7a868e53b3a/porting/week/0/DAY%207.md

Task 1: Addressing Tracking Challenges:

We can identify what files were modified by the LuMI project what was modified upstream with:

git diff --name-only upstream/main..HEAD > upstream-changes.txt

We can also use cherry, cherry-pick -n, and tagging to help automate staging upstream changes.

However, in order for that to work, the root of our repo must also be the same root as LuCI. So we will be renaming this repo to lumi.working and then creating a new repo called lumi containing our ported files.

We may need to develop our own solution to ensure commits are properly staged. In order to better enforce this, we must also protect the main branch of the lumi repo.

For now, the tag fork has been created on lumi to mark the point where we forked from LuCI.

git cherry looks like it is going to be the most useful tool for tracking changes. However, it is still limited. It tells us all of the commits that haven't been applied to our branch. It doesn't provide us with a way to mark which commits have already been reviewed.

There are 2 ways we can handle this:

We can use git notes to mark commits that have been reviewed.
We can use tags to mark the last reviewed commit.

Notes seem like a decent option, but they are not as visible as tags. I think I'm going to use tags. One problem will be that these get mixed in with release tags, so we will need to use a naming convention that makes it clear these are not release tags.

To solve this, we can use a prefix like reviewed/ for these tags.
We can also make sure they are lightweight tags (they don't need metadata).

Though, after thinking about it and deciding on review tags, I think the simplest solution is the best one. Instead of using git cherry and dealing with its limitations, we can just track the changes between LuCI's head and the last reviewed commit. We will also need an in-progress tag though if we do this, to denote the last commit reviewed, but not yet merged/implemented.

Task 2: Setting Up VCS

For convenience, it may be best to dump the list of commits between reviewed, in-progress, and upstream head into a file. So we don't have to run the command every time we want to see what has changed.

function Get-UnmergedCommits {
    param (
        [string] $Upstream = "https://github.com/openwrt/luci",
        [string] $Repo = "./lumi",
        [switch] $InProgress, # reviewed .. in-progress
        [switch] $Pending, # in-progress .. upstream head
        [switch] $Verbose
    )

    pushd $Repo
    # ensure upstream is set

    # Hack to reset $LASTEXITCODE to 0
    & (Resolve-Path "$PSHome\p*w*sh*" | Where-Object {
        $Path = "$_"

        if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) {
            return $false
        }

        if ($IsWindows) {
            $ext = [IO.Path]::GetExtension($Path)
            $pathext = ($env:PATHEXT -split ';') -replace '^\.', ''
            return $pathext -contains $ext.TrimStart('.').ToUpperInvariant()
        } else {
            $escapedPath = $Path.Replace("'", "''")

            # Prefer bash, fallback to sh
            $shell = if (Get-Command bash -ErrorAction SilentlyContinue) { "bash" } else { "sh" }

            & $shell -c "test -x '$escapedPath'"
            return ($LASTEXITCODE -eq 0)
        }
    } | Select-Object -First 1) -v | Out-Null

    $remotes = @{}
    git remote -v 2>$null | ForEach-Object {
        if ($_ -match '^(\S+)\s+(\S+)') {
            $remotes[$matches[1]] = $matches[2]
        }
    }
    $remote = $remotes.Keys | ForEach-Object { $_ } | Where-Object {
        $remotes."$_" -eq $Upstream
    }
    If( "$remote".Trim() -eq "" ){
        $remote = "upstream"
        git remote set-url $remote $Upstream 2>&1 | ForEach-Object {
            If( $LASTEXITCODE -ne 0 ){
                throw "$_"
            }
            If( $_ -is [System.Management.Automation.ErrorRecord] ) {
                throw $_.Exception
            }
            If( $Verbose ) {
                If ( $_ -is [System.Management.Automation.WarningRecord] ) {
                    Write-Host $_.Message -ForegroundColor Yellow
                } else {
                    Write-Host "$Repository`: $_" -ForegroundColor DarkGray
                }
            }
        }
    }

    git remote set-url upstream $Upstream

    git fetch --tags

    $range = if (([bool]$InProgress) -eq ([bool]$Pending)) {
        "full"
    } elseif ($InProgress) {
        "in-progress"
    } else {
        "pending"
    }

    $reviewed = git tag --list "reviewed/*" | Select-Object -Last 1
    $in_progress = git tag --list "in-progress/*" | Select-Object -Last 1

    $start = if( ($range -eq "pending") -and ("$in_progress".Trim() -ne "") ) {
        $in_progress
    } else {
        if ("$reviewed".Trim() -ne "") {
            $reviewed
        } else {
            "fork"
        }
    }
    $end = if( ($range -eq "in-progress") -and ("$in_progress".Trim() -ne "")) {
        $in_progress
    } else {
        "upstream/main"
    }

    $commits = git rev-list --reverse "$start..$end"

    $diffs = [ordered]@{}
    # get the diffs
    foreach ($commit in $commits) {
        $diff = git show --format= --no-prefix $sha
        if ("$diff".Trim() -ne "") {
            $diffs[$commit] = $diff
        }
    }
    popd
    return @{
        Start = $start
        End = $end
        Range = $range
        Commits = $commits
        Diffs = $diffs
    }
}

This script should give me a serializable hashtable that I can write to file for roadmapping upstream changes.

Task 2: Cleaning Up My Devtools

Since my repository now has a slew of new tools, I feel a need to clean them up and organize them into their own directory. I will create a sdk directory here and move the ~search.ps1 script into it. This will help keep the root directory clean and make it easier to develop more tools in the future.

I also need to update my scripts to accept a repo and upstream as parameters at the file level. This should be an easy fix... and done.

My 2 tools are now in the sdk directory:

unmerged.ps1: This script will help me track unmerged commits between my fork and the upstream LuCI repository.
search.ps1: This script will help me search through the LuCI codebase for specific patterns or files. Useful for building out the initial roadmap

Last remark for Hurdle 11: unmerged.ps1 will likely not be needed for some time.

I will continue tomorrow with Task 3: Prioritizing Ports.

LuCI on MGMT - Day 7

project-laguardia — Wed, 16 Jul 2025 03:58:01 +0000

Hurdle 7: Backtracking on `ucode` Files

Previously:

https://github.com/project-laguardia/lumi/blob/main/porting/DAY%206.md

After some realization, it was discovered that ucode files were being omitted from the search results.

Luckily, LuCI on ucode is still in its early stages, so only a minimal amount of files were missed:

uci results:
- modules/luci-base/ucode/zoneinfo.uc
- Result by mistake. (does not need to be ported)
- modules/luci-base/ucode/dispatcher.uc
- This is the current HTTP Router for LuCI
- Will definitely have to be ported to LuMI
- modules/luci-base/ucode/runtime.uc
- Handles .ut template file generation
- Adds a Lua bridge to the ucode runtime
- modules/luci-base/ucode/controller/admin/uci.uc
- Adds low-level uci and ubus interop to LuCI's ucode runtime
- Will definitely have to be ported to LuMI
/etc/config results: none noteworthy
luci.config results: none noteworthy

Hurdle 8: Identifying Any `rpc` Interop

Here are the results of the rpc code search:

libs/luci-lib-nixio/axTLS/www/lua/test_lib.lua
- This is a legacy test file that can probably be dropped
libs/luci-lib-px5g/src/library/timing.c
- This was returned by mistake. It references assembly instructions for the px5g library, which has rpcc as part of the instruction set.
libs/rpcd-mod-luci/src/luci.c and libs/rpcd-mod-rrdns/src/rrdns.c
- rpc-mod-luci is a rpcd plugin written by LuCI developers that exposes some functions that LuCI was worth sharing with rpcd.
- This is already planned for porting.
modules/luci-base/ucode/dispatcher.uc
- This is the current HTTP Router for LuCI
- This is already planned for porting.
modules/luci-base/ucode/controller/admin/index.uc
- This serves as the LuCI API gateway.
- This is a high-value target for porting.
modules/luci-base/ucode/controller/admin/uci.uc
- Adds low-level uci and ubus interop to LuCI's ucode runtime
- Already planned for porting. modules/luci-lua-runtime/luasrc/dispatcher.lua
- This is the older HTTP Router for LuCI
- This is already planned for porting.
modules/luci-lua-runtime/luasrc/model/uci.lua
- This is a wrapper around the uci module that provides a more Lua-like interface.
- Already planned for porting. modules/luci-mod-rpc/luasrc/jsonrpc.lua
- This is the JSON-RPC implementation for LuCI.
- High-value target for porting.
modules/luci-mod-rpc/luasrc/controller/rpc.lua
- This is a utility for doing RPC calls in LuCI.
- Already planned for porting. modules/luci-mod-rpc/luasrc/jsonrpcbind/uci.lua
- A convenience wrapper around uci. Honestly seems kind of redundant with luci.model.uci.
- Already planned for porting.

Hurdle 9: Identifying Any `ubus` Interop

Here are the results of the ubus code search:

libs/luci-lib-nixio/axTLS/config/scripts/config/mconf.c
- Resulted by mistake. For some reason axTLS includes menuconfig.
libs/luci-lib-nixio/axTLS/crypto/bigint.c
- Also resulted by mistake.
libs/luci-lib-nixio/src/poll.c
- Also resulted by mistake.
modules/luci-compat/luasrc/model/network/proto_modemmanager.lua
- Also resulted by mistake.
modules/luci-compat/luasrc/model/network/proto_qmi.lua
- Also resulted by mistake.
libs/luci-lib-base/luasrc/util.lua
- provides string manipulation and serialization functions as well as methods to interact with the ubus
- High-value target for porting.
libs/rpcd-mod-luci/src/luci.c and libs/rpcd-mod-rrdns/src/rrdns.c
- rpc-mod-luci is a rpcd plugin written by LuCI developers that exposes some functions that LuCI was worth sharing with rpcd.
- This is already planned for porting.
modules/luci-base/ucode/dispatcher.uc
- This is the current HTTP Router for LuCI
- This is already planned for porting.
modules/luci-base/ucode/controller/admin/index.uc
- This serves as the LuCI API gateway.
- This is already planned for porting.
modules/luci-base/ucode/controller/admin/uci.uc
- Adds low-level uci and ubus interop to LuCI's ucode runtime
- Already planned for porting.
modules/luci-compat/luasrc/model/network.lua
- Provides a model for configuring network interfaces, devices, and L2 (sometimes L3) protocols.
- Already planned for porting.
modules/luci-compat/luasrc/tools/webadmin.lua
- Minimally interacts with ubus
- Provides a small set of utility tools for LuCI.
- Already planned for porting.
modules/luci-lua-runtime/luasrc/dispatcher.lua
- This is the older HTTP Router for LuCI
- Already planned for porting.
modules/luci-lua-runtime/luasrc/sys.lua
- This module provides a set of system utilities for LuCI.
- It minimally interacts with uci. In fact its only interaction appears to be pulling IPs from it.
- Already planned for porting.
modules/luci-lua-runtime/luasrc/model/uci.lua
- This is a wrapper around the uci module that provides a more Lua-like interface.
- Already planned for porting.
modules/luci-mod-rpc/luasrc/controller/rpc.lua
- This is a utility for doing RPC calls in LuCI.
- I believe that mgmt uses a different RPC mechanism, so any ubus and RPC functionality may also have to be ported.
- Already planned for porting.

Hurdle 10: Identifying Any Other Languages Used

At this point, we need to ensure that we have identified all languages used in LuCI.

For this, we will be adding Enry to our dev chain. We will be using temporary installs to avoid polluting the system with Go dependencies.

Task 1: Setup Enry

$languages = & {
    $tmp = New-TemporaryFile | % { rm $_; ni $_ -ItemType Directory }
    & {
        pushd $tmp
        go mod init tempmod
        go get github.com/go-enry/enry
        go build -o ./enry github.com/go-enry/enry
        popd
    } | Write-Host
    & {
        pushd luci
        & "$tmp/enry" -all -json
        popd
    }
    rm -r $tmp -Force | Out-Null
} | ConvertFrom-Json

When enry is used like this, it is only good for summarizing the languages within cwd. I will have to iterate over each file in order to create the language mapping that I want.

... (6 hours later) ...

Took a lot to get language detection working, but ~search.ps1 has been updated to use Enry to detect languages.

This should be the final hurdle in the research phase of porting LuCI to LuMI. A quick review of the updated results will be needed to make that determination.

Task 2: Reviewing the new results

A few new results were found:

At this point, we have a pretty good idea of what needs to be ported. We can even treat the search results as a sort of "porting roadmap."

The next 2 challenges are to organize the ports then also ensure that changes from LuCI can be propagated to LuMI.

LuCI on MGMT - Day 6

project-laguardia — Mon, 14 Jul 2025 09:06:05 +0000

Hurdle 5: Identifying Any `/etc/config` Interop

Previously:

https://github.com/project-laguardia/lumi/blob/ac0df8855bf9e0a1d94e9ab3193fbfaefdbf09ca/porting/DAY%205.md

function Find-InSource {
    param(
        [string] $Pattern,
        [string] $Repository = ".\luci",
        [string[]] $Extensions = @(
            ".lua",
            ".c",
            ".js",
            ".mjs"
        ),
        [switch] $Verbose
    )

    # ...
}

& { # Search for `uci` in the source code
    $hits = Find-InSource -Pattern '(?<!l)uci' -Verbose
    $hits | ConvertTo-Json -Depth 5 | Out-File "porting/searches/uci/hits.json" -Encoding UTF8
    $hits.Keys | Out-File "porting/searches/uci/hits.txt" -Encoding UTF8
}

Task 1: Reviewing Search Results

Some new searches have been added to porting/~search.ps1. /etc/config interop search is saved to porting/searches/config/etc.hits.*

Only noteable hit was:

modules/luci-lua-runtime/luasrc/model/uci.lua
- This is a Lua abstraction for UCI
- Already planned to be ported to LuMI

Hurdle 6: Identifying Any `luci.config` interop

Task 1: Reviewing Search Results

This search has also been added to porting/~search.ps1. The results are saved to porting/searches/module.hits.*

A few noteable hits:

modules/luci-compat/luasrc/cbi.lua
- This is a part of LuCI's Configuration Bind Interface (CBI)
- Helps map UCI configuration files to HTML forms.
- Already planned to be ported to LuMI
modules/luci-lua-runtime/luasrc/cacheloader.lua
- This is a new one.
- It's a small cache enabler for LuCI. It appears to be an artifact of the old LuCI architecture (pre-ucode)
modules/luci-lua-runtime/luasrc/config.lua
- Quite literally the "luci.config" module.
- Already planned to be ported to LuMI
modules/luci-lua-runtime/luasrc/template.lua
- Modifies luci.config.template and reads luci.config.main to provide a template engine for LuCI.
modules/luci-lua-runtime/luasrc/model/uci.lua
- This is a wrapper around the uci module that provides a more Lua-like interface.
- Already planned to be ported to LuMI
modules/luci-mod-rpc/luasrc/controller/rpc.lua
- Primarily referenced for getting uci session time setting.
- Already planned to be ported to LuMI

Today's load will be light, as yesterday's took quite a bit of time to go through. Tomorrow, we will be backstepping as we missed reviewing search results within ucode files.

LuCI on MGMT - Day 05

project-laguardia — Mon, 14 Jul 2025 00:55:17 +0000

Hurdle 4: Identifying Where `uci` Interop Actually Occurs

Previously:

https://github.com/project-laguardia/lumi/blob/24344c6562d3d719f0bd03d97e515eb88f3a54ab/porting/DAY%204.md

Task 1: Code-Search

I believe uci interop occurs in the following modules:

luci.model.uci

Like we've discussed before, LuCI uses an older version of Lua that allows the module function to be used. This allows LuCI to name modules consistently regradless of whether they are Lua or C modules.

After searching for "luci.model.uci", we discover that it is defined in Lua:

modules/luci-lua-runtime/luasrc/model/uci.lua

The API doc for luci.model.uci is available at:

API docs: https://openwrt.github.io/luci/api/modules/luci.model.uci.html

One can see how uci may be interacted with by using regex search (?<!l>)uci in code search. Unfortunately, Github code search does not support regex, so we will have to do this locally. Here is a PowerShell script that does that for us:

function Find-InSource {
    param(
        [string] $Pattern,
        [string] $Repository = ".\luci",
        [string[]] $Extensions = @(
            ".lua",
            ".c",
            ".js",
            ".mjs"
        ),
        [switch] $Verbose
    )

    $gitRoot = (git rev-parse --show-toplevel 2>$null)

    $output = [ordered]@{}

    If( $Verbose ) {
        Write-Host "Aggregating source files with extensions: $($Extensions -join ', ')" -ForegroundColor Yellow
        Write-Host "- Wait a moment, this may take a bit..." -ForegroundColor DarkGray
    }

    $files = Get-ChildItem -Path $Repository -File -Recurse | Where-Object {
        $_.Extension -in $Extensions
    }

    if ($files.Count -eq 0) {
        Write-Host "No files found with the specified extensions." -ForegroundColor Yellow
        return $output
    } elseif ( $Verbose ) {
        Write-Host "Searching in $($files.Count) files with extensions: $($Extensions -join ', ')" -ForegroundColor Black -BackgroundColor DarkYellow
    }

    $files | ForEach-Object {
        $filename = $_.FullName.Replace($gitRoot, '').TrimStart('\/')
        $fileContent = Get-Content $_.FullName

        $hits = for ($ln = 0; $ln -lt $fileContent.Count; $ln++) {
            $line = $fileContent | Select-Object -Index $ln
            if ($line -match $pattern) {
                @{
                    Line = $ln + 1
                    Content = $line.Trim()
                }
            }
        }

        if ($hits.Count -gt 0) {
            $output."$filename" = [ordered]@{}

            $hits | ForEach-Object {
                $output."$filename"."$($_.Line)" = $_.Content
            }

            If( $Verbose ){
                Write-Host "$filename`:" -ForegroundColor Cyan

                $hits | ForEach-Object {
                    Write-Host "L$($_.Line)" -NoNewline -ForegroundColor Gray
                    Write-Host ": $($_.Content)"
                }
                Write-Host ""
            }
        }
    }

    If( $Verbose ) {
        $total_files = $output.Keys.Count
        $total_hits = ($output.Values | ForEach-Object { $_.Count }) -as [int[]] | Measure-Object -Sum | Select-Object -ExpandProperty Sum

        Write-Host "Found $total_hits hits in $total_files files." -ForegroundColor Green
    }

    return $output
}

& { # Search for `uci` in the source code
    $hits = Find-InSource -Pattern '(?<!l)uci' -Verbose
    $hits | ConvertTo-Json -Depth 5 | Out-File "porting/searches/uci/hits.json" -Encoding UTF8
    $hits.Keys | Out-File "porting/searches/uci/hits.txt" -Encoding UTF8
}

I will be making this script available in the porting directory of the repository, so that it can be used for future searches.

see the README for usage instructions

After running our first search, I've aggregated the following results:

Task 2: Reviewing the Results

For this we are going scope it down to just libs, and modules. Applications are third-party. We may want to port them later, but for now we will focus on the core LuCI codebase. Contrib was matched by mistake. The referenced C file is a html special character parser for ucode. The search script matched the û which is html for û, which is not what we are looking for.

Libs:

libs/luci-lib-nixio/axTLS/crypto/aes.c
- This only had one match, and it was a mistake: "/* Perform doubling in Galois Field GF(2^8) using the irreducible polynomial"
modules/luci-base/src/contrib/lemon.c - also a mistake
- modules/luci-lua-runtime/src/contrib/lemon.c - also a mistake
modules/luci-lua-runtime/luasrc/sys/zoneinfo/tzdata.lua - also a mistake
libs/rpcd-mod-luci/src/luci.c
- This C file is a plugin for rpcd. It shares certain RPC functions useful for LuCI with the entire ubus. Not entirely sure why LuCI does this, because I believe LuCI is the only user of these functions.
- These will have to be ported to lumi. We may not use the same mechanism, but we can use some of the logic.
modules/luci-compat/luasrc/cbi.lua and modules/luci-compat/luasrc/cbi/datatypes.lua
- This is a part of LuCI's Configuration Bind Interface (CBI)
- This is a high-value target for porting. It helps map UCI configuration files to HTML forms.
modules/luci-compat/luasrc/model/firewall.lua
- Provides a model for the configuring the firewall in LuCI.
- This is a valuable target for porting.
modules/luci-compat/luasrc/model/network.lua
- Provides a model for configuring network interfaces, devices, and L2 (sometimes L3) protocols.
- This is a valuable target for porting.
modules/luci-compat/luasrc/tools/webadmin.lua
- Provides a small set of utility tools for LuCI.
- Low value target for porting, but a target nonetheless.
modules/luci-lua-runtime/luasrc/config.lua
- This is a convenience wrapper around the uci module that provides a more Lua-like interface.
- Codebase and changes required are small, but a very high-value target for porting.
- It is called 'luci.config' in LuCI,
modules/luci-lua-runtime/luasrc/dispatcher.lua
- This is the HTTP Router for LuCI
- It mainly provides auth protection for uci
- Valuable target for porting
modules/luci-lua-runtime/luasrc/sys.lua
- This module provides a set of system utilities for LuCI.
- It minimally interacts with uci. In fact its only interaction appears to be pulling IPs from it.
- We need to port this, but low priority
modules/luci-lua-runtime/luasrc/model/uci.lua
- This is a wrapper around the uci module that provides a more Lua-like interface.
- Definitely a high-value target for porting.
modules/luci-mod-rpc/luasrc/controller/rpc.lua
- This is a utility for doing RPC calls in LuCI.
- I believe that mgmt uses a different RPC mechanism, so any ubus and RPC functionality may also have to be ported.
modules/luci-mod-rpc/luasrc/jsonrpcbind/uci.lua
- A convenience wrapper around uci. Honestly seems kind of redundant with luci.model.uci.
- Will still be ported, but may iterate over LuCI packages and drop its usage in favor of luci.model.uci.

LuCI on MGMT - Day 04

project-laguardia — Sat, 12 Jul 2025 03:47:18 +0000

Hurdle 3: Scoping OpenWRT's Build System Down (Continued)

Previously:

https://github.com/project-laguardia/lumi/blob/main/porting/DAY%203.md

When using the SDK (or the full buildroot), you will often be instructed to use feeds quite rigorously mainly for healing your SDK, buildroot, or if you are building using the OS itself, your OpenWRT installation. It can also be used to install dependencies for your project as well.
...
LuCI instructs that you run:
./scripts/feeds update
./scripts/feeds install -a -p luci
make menuconfig
...
(from SDK container > setup.sh):
if [ -z "$PACKAGES" ]; then
  # compile all packages in feed
  for FEED in $ALL_CUSTOM_FEEDS; do
      group "feeds install -p $FEED -f -a"
      ./scripts/feeds install -p "$FEED" -f -a
      endgroup
  done

  RET=0

  make \
      BUILD_LOG="$BUILD_LOG" \
      CONFIG_SIGNED_PACKAGES="$CONFIG_SIGNED_PACKAGES" \
      IGNORE_ERRORS="$IGNORE_ERRORS" \
      CONFIG_AUTOREMOVE=y \
      V="$V" \
      -j "$(nproc)" || RET=$?
else
...
fi
...
Now that we understand how they build the packages automatically, we can now use this knowledge for building our own way. For now, we will use the recommended make menuconfig as described in Task 1, but we will likely switch to using the GH Actions way of building in the future.

Task 1: Digesting `./scripts/feeds` (continued)

https://gitlab.com/openwrt/openwrt/openwrt/-/blob/9ea174c7bf64ec34e96871ce223d7a597ca80d26/scripts/feeds

To continue from where we left off, we will now look at how ./scripts/feeds works in more detail. The feeds system is a way to manage packages and their dependencies in OpenWRT. It allows you to update, install, and manage packages easily.

First thing we'll notice is the used libraries:

use Getopt::Std;
use FindBin;
use Cwd;
use lib "$FindBin::Bin";
use metadata;
use warnings;
use strict;
use Cwd 'abs_path';

All of these are standard Perl libraries, except for metadata, which is another part of the OpenWRT build system, and I believe we will need it to learn how the OpenWRT SDK makes config files for menuconfig. For now, we will continue digesting ./scripts/feeds and come back to it later.

The first thing feeds does is setup the working environment and verify make:


chdir "$FindBin::Bin/..";
$ENV{TOPDIR} //= getcwd();
chdir $ENV{TOPDIR};
$ENV{GIT_CONFIG_PARAMETERS}="'core.autocrlf=false'";
$ENV{GREP_OPTIONS}="";

my $mk=`command -v gmake 2>/dev/null`;  # select the right 'make' program
chomp($mk);     # trim trailing newline
$mk or $mk = "make";    # default to 'make'

# check version of make
my @mkver = split /\s+/, `$mk -v`, 4;
my $valid_mk = 1;
$mkver[0] =~ /^GNU/ or $valid_mk = 0;
$mkver[1] =~ /^Make/ or $valid_mk = 0;

my ($mkv1, $mkv2) = split /\./, $mkver[2];
($mkv1 >= 4 || ($mkv1 == 3 && $mkv2 >= 81)) or $valid_mk = 0;

$valid_mk or die "Unsupported version of make found: $mk\n";

Then we are provided with a good handful of different functions:

sub parse_file($$);
sub parse_config();
sub update_location($$);
sub update_index($);
sub update_feed_via($$$$$$$);
sub get_targets($);
sub get_feed($);
sub get_installed();
sub search_feed;
sub list_feed;
sub do_install_src($$);
sub do_install_target($);
sub lookup_src($$);
sub lookup_package($$);
sub lookup_target($$);
sub is_core_src($);
sub install_target;
sub install_src;
sub install_package;
sub install_target_or_package;
sub refresh_config;
sub uninstall_target($);
sub update_feed($$$$$$);

sub feed_config();
sub search;
sub uninstall;
sub install;
sub list;
sub update;
sub usage();

And a few useful hashtables:

my %update_method = (
    'src-svn' => {
        'init'      => "svn checkout '%s' '%s'",
        'update'    => "svn update",
        'controldir'    => ".svn",
        'revision'  => "svn info | grep 'Revision' | cut -d ' ' -f 2 | tr -d '\n'"},
    'src-cpy' => {
        'init'      => "cp -Rf '%s' '%s'",
        'update'    => "",
        'revision'  => "echo -n 'local'"},
    'src-link' => {
        'init'      => "ln -s '%s' '%s'",
        'update'    => "",
        'revision'  => "echo -n 'local'"},
    'src-dummy' => {
        'init'      => "true '%s' && mkdir '%s'",
        'update'    => "",
        'revision'  => "echo -n 'dummy'"},
    'src-git' => {
        'init'          => "git clone --depth 1 '%s' '%s'",
        'init_branch'   => "git clone --depth 1 --branch '%s' '%s' '%s'",
        'init_commit'   => "git clone '%s' '%s' && cd '%s' && git checkout -b '%s' '%s' && cd -",
        'update'    => "git pull --ff-only",
        'update_rebase' => "git pull --rebase=merges",
        'update_stash'  => "git pull --rebase=merges --autostash",
        'update_force'  => "git pull --ff-only || (git reset --hard HEAD; git pull --ff-only; exit 1)",
        'post_update'   => "git submodule update --init --recursive",
        'controldir'    => ".git",
        'revision'  => "git rev-parse HEAD | tr -d '\n'"},
    'src-git-full' => {
        'init'          => "git clone '%s' '%s'",
        'init_branch'   => "git clone --branch '%s' '%s' '%s'",
        'init_commit'   => "git clone '%s' '%s' && cd '%s' && git checkout -b '%s' '%s' && cd -",
        'update'    => "git pull --ff-only",
        'update_rebase' => "git pull --rebase=merges",
        'update_stash'  => "git pull --rebase=merges --autostash",
        'update_force'  => "git pull --ff-only || (git reset --hard HEAD; git pull --ff-only; exit 1)",
        'post_update'   => "git submodule update --init --recursive",
        'controldir'    => ".git",
        'revision'  => "git rev-parse HEAD | tr -d '\n'"},
    'src-gitsvn' => {
        'init'  => "git svn clone -r HEAD '%s' '%s'",
        'update'    => "git svn rebase",
        'controldir'    => ".git",
        'revision'  => "git rev-parse HEAD | tr -d '\n'"},
    'src-bzr' => {
        'init'      => "bzr checkout --lightweight '%s' '%s'",
        'update'    => "bzr update",
        'controldir'    => ".bzr"},
    'src-hg' => {
        'init'      => "hg clone '%s' '%s'",
        'update'    => "hg pull --update",
        'controldir'    => ".hg"},
    'src-darcs' => {
        'init'    => "darcs get '%s' '%s'",
        'update'  => "darcs pull -a",
        'controldir' => "_darcs"},
);
my %commands = (
    'list' => \&list,
    'update' => \&update,
    'install' => \&install,
    'search' => \&search,
    'uninstall' => \&uninstall,
    'feed_config' => \&feed_config,
    'clean' => sub {
        system("rm -rf ./feeds ./package/feeds ./target/linux/feeds");
    }
);

Some important things to note about the update methods:

src-cpy, src-link, and src-dummy are used for local sources that basically do not require any updates.
- src-cpy does, however it is just a copy of an already local resource.
src-git, src-git-full, src-gitsvn, src-bzr, src-hg,src-svn and src-darcs are all used for version control systems.
- src-git is the most common one, and it is used for Git repositories.
- src-git-full is used for full Git repositories, which may include all branches and history.
- src-gitsvn is used for Git repositories that are backed by a Subversion repository.
- src-bzr, src-hg, and src-darcs are used for Bazaar, Mercurial, and Darcs repositories respectively.
- src-svn is used for Subversion repositories.

We have already gone over introducing the commands, but as a refresher, here they are once more:

sub usage() {
    print <<EOF;
Usage: $0 <command> [options]

Commands:
    list [options]: List feeds, their content and revisions (if installed)
    Options:
        -n :            List of feed names.
        -s :            List of feed names and their URL.
        -r <feedname>:  List packages of specified feed.
        -d <delimiter>: Use specified delimiter to distinguish rows (default: spaces)
        -f :            List feeds in opkg feeds.conf compatible format (when using -s).

    install [options] <package>: Install a package
    Options:
        -a :           Install all packages from all feeds or from the specified feed using the -p option.
        -p <feedname>: Prefer this feed when installing packages.
        -d <y|m|n>:    Set default for newly installed packages.
        -f :           Install will be forced even if the package exists in core OpenWrt (override)

    search [options] <substring>: Search for a package
    Options:
        -r <feedname>: Only search in this feed

    uninstall -a|<package>: Uninstall a package
    Options:
        -a :           Uninstalls all packages.

    update -a|<feedname(s)>: Update packages and lists of feeds in feeds.conf .
    Options:
        -a :           Update all feeds listed within feeds.conf. Otherwise the specified feeds will be updated.
        -r :           Update by rebase. (git only. Useful if local commits exist)
        -s :           Update by rebase and autostash. (git only. Useful if local commits and uncommited changes exist)
        -i :           Recreate the index only. No feed update from repository is performed.
        -f :           Force updating feeds even if there are changed, uncommitted files.

    clean:             Remove downloaded/generated files.

EOF
    exit(1);
}

After some review of feeds source code, most of the "installing" is just linking and copying source files to the appropriate directories. Build doesn't actually occur here, so I will have to dig into make invocation during make menuconfig/make defconfig to see how it is done. I suspect that include/toplevel.mk and scripts/package-metadata.pl are the two files that actually make the necessary config files for menuconfig. Though, I'm not too sure. OpenWRT's SDK is quite opaque and minimally documented, so it is hard to tell what is actually going on even if you can read the source code.

Before moving on there is one thing I do want to point out. feeds has a refresh_config function that calls make defconfig/make oldconfig and that function is called every time the commands install, uninstall, and update are called.

Task 2: Digesting `make menuconfig`

The menuconfig make target is defined in the "Top Level" Makefile, which is a highly referenced file in the OpenWRT build system.

./include/toplevel.mk is referenced here
./include/toplevel.mk defines the menuconfig target here

This target is really just a wrapper for ./scripts/config/mconf ./Config.in

toplevel.mk is run once every time make is called. From the SDK root Makefile, this is ensured using this mechanism:

ifneq ($(OPENWRT_BUILD),1)
  override OPENWRT_BUILD=1
  export OPENWRT_BUILD

Where OPENWRT_BUILD is acting like a flag to indicate that make is already running. Note that at any time during the make process, you can clear OPENWRT_BUILD if you want to run make inside your build script in a way that it thinks it isn't running. This has a handful of uses across the SD, even inside ./scripts/feeds:

sub update_index($)
{
    my $name = shift;

    -d "./feeds/$name.tmp" or mkdir "./feeds/$name.tmp" or return 1;
    -d "./feeds/$name.tmp/info" or mkdir "./feeds/$name.tmp/info" or return 1;

    system("$mk -s prepare-mk OPENWRT_BUILD= TMP_DIR=\"$ENV{TOPDIR}/feeds/$name.tmp\"");
    system("$mk -s -f include/scan.mk IS_TTY=1 SCAN_TARGET=\"packageinfo\" SCAN_DIR=\"feeds/$name\" SCAN_NAME=\"package\" SCAN_DEPTH=5 SCAN_EXTRA=\"\" TMP_DIR=\"$ENV{TOPDIR}/feeds/$name.tmp\"");
    system("$mk -s -f include/scan.mk IS_TTY=1 SCAN_TARGET=\"targetinfo\" SCAN_DIR=\"feeds/$name\" SCAN_NAME=\"target\" SCAN_DEPTH=5 SCAN_EXTRA=\"\" SCAN_MAKEOPTS=\"TARGET_BUILD=1\" TMP_DIR=\"$ENV{TOPDIR}/feeds/$name.tmp\"");
    system("ln -sf $name.tmp/.packageinfo ./feeds/$name.index");
    system("ln -sf $name.tmp/.targetinfo ./feeds/$name.targetindex");

    return 0;
}

sub get_installed() {
    system("$mk -s prepare-tmpinfo OPENWRT_BUILD=");
    clear_packages();
    parse_package_metadata("./tmp/.packageinfo");
    %installed_pkg = %vpackage;
    %installed = %srcpackage;
    %installed_targets = get_targets("./tmp/.targetinfo");
}

Now, we need to address how the SDK generates the config files for menuconfig. toplevel.mk defines another target called prepare-tmpinfo. This target invokes the aforementioned ./scripts/package-metadata.pl script, which is used to generate the .in files for menuconfig.

Another interesting discovery is that it may also invoke ./scripts/target-metadata.pl.

I think this is enough information to start working on porting LuCI now. If there is any questions about what parts of the SDK variables or scripts may refer to, we now have the files for looking them up. One last step for me is to organize my findings and post them in the README.

LuCI on MGMT - Day 03

project-laguardia — Fri, 11 Jul 2025 02:27:47 +0000

Hurdle 3: Scoping OpenWRT's Build System Down (Continued)

Previously:

https://github.com/project-laguardia/lumi/blob/92ec839f81b08e7dab9e785dd73dc0fb5b2b27f7/porting/DAY%202.md

./scripts/feeds

feeds is the main script for OpenWRT's package feed management written in Perl, abstracting all the repository operations and package install/uninstall logic in a uniform manner, so users don't have to worry themselves with details of where packages come from or how they are versioned.

...

(About understanding their build system:) I've just realized that another way to approach this is to look at their GH Actions. Upon inspection, it appears that they do have a build flow used to test LuCI, and we even have some example runs to look at

Task 1: Digesting `./scripts/feeds` and `./scripts/feeds.conf.default`

Previously, I failed to mention why we are going over feeds and feeds.conf.default. When using the SDK (or the full buildroot), you will often be instructed to use feeds quite rigorously mainly for healing your SDK, buildroot, or if you are building using the OS itself, your OpenWRT installation. It can also be used to install dependencies for your project as well.

Some, but not all instances of it being referenced in OpenWRT/LuCI documentation:

LuCI instructs that you run:

./scripts/feeds update
./scripts/feeds install -a -p luci
make menuconfig

...but as we have discovered previously, it may be better for us to use feeds the way their GH Actions do

Task 2: Reviewing SDK Usage in GH Actions:

LuCI's interaction with the SDK at the workflow level is quite minimal:

https://github.com/openwrt/luci/blob/4678d6a82c5c36626edb88cdfb93edbb7e93e0fa/.github/workflows/build.yml#L52-L68

- name: Build
  uses: openwrt/gh-action-sdk@v7
  env:
    ARCH: ${{ matrix.arch }}-${{ env.BRANCH }}
    FEEDNAME: packages_ci
    V: s

# nothing actually happens between invoking the SDK and moving the packages around:
- name: Move created packages to project dir
  run: cp bin/packages/${{ matrix.arch }}/packages_ci/* . || true

It seems that most of the building is done automatically by the SDK action:

https://github.com/openwrt/gh-action-sdk/blob/b8cc97d1072dedff455e2945a73fc43f5c7e1749/entrypoint.sh

The available environment variables are documented here:

https://github.com/openwrt/gh-action-sdk/tree/main?tab=readme-ov-file#environmental-variables

ARCH determines the used OpenWRT SDK Docker container. E.g. x86_64 or x86_64-22.03.2

ARTIFACTS_DIR determines where the built packages and build logs are saved. Defaults to the default working directory (GITHUB_WORKSPACE).

BUILD_LOG stores the build logs in ./logs.

CONTAINER can set other SDK containers than openwrt/sdk

EXTRA_FEEDS are added to the feeds.conf, where | are replaced by white spaces.

FEED_DIR used in the created feeds.conf for the current repo. Defaults to the default working directory (GITHUB_WORKSPACE).

FEEDNAME is used in the created feeds.conf for the current repo. Defaults to action.

IGNORE_ERRORS can ignore failing package builds.

INDEX makes the action build the package index. Default is 0. Set to 1 to enable.

KEY_BUILD can be a private Signify/usign key to sign the packages (ipk) feed.

PRIVATE_KEY can be a private key to sign the packages (apk) feed.

NO_DEFAULT_FEEDS disable adding the default SDK feeds

NO_REFRESH_CHECK disable check if patches need a refresh

NO_SHFMT_CHECK disable check if init files are formatted

PACKAGES (Optional) specify the list of packages (space separated) to be built

V changes the build verbosity level.

setup.sh Breakdown:
First, it calls setup.sh from ghcr.io/openwrt/sdk

group "bash setup.sh"
# snapshot containers don't ship with the SDK to save bandwidth
# run setup.sh to download and extract the SDK
[ ! -f setup.sh ] || bash setup.sh
endgroup

Then, it sets up the feed configuration:

FEEDNAME="${FEEDNAME:-action}"
...
echo "src-link $FEEDNAME /feed/" >> feeds.conf

Which we know from before is packages_ci:

It appears they use openwrt/gh-action-sdk with a few env vars:

ARCH - <arch>-<branch>

FEEDNAME - packages_ci

Noteworthy (but unused by LuCI) is the fact that you can optionally disable the default feeds:

if [ -z "$NO_DEFAULT_FEEDS" ]; then
    sed \
        -e 's,https://git.openwrt.org/feed/,https://github.com/openwrt/,' \
        -e 's,https://git.openwrt.org/openwrt/,https://github.com/openwrt/,' \
        -e 's,https://git.openwrt.org/project/,https://github.com/openwrt/,' \
        feeds.conf.default > feeds.conf
fi

Then the script starts adding in the extra feeds:

ALL_CUSTOM_FEEDS="$FEEDNAME "
#shellcheck disable=SC2153
for EXTRA_FEED in $EXTRA_FEEDS; do
    echo "$EXTRA_FEED" | tr '|' ' ' >> feeds.conf
    ALL_CUSTOM_FEEDS+="$(echo "$EXTRA_FEED" | cut -d'|' -f2) "
done

Finally, it updates the feeds:

group "feeds.conf"
cat feeds.conf
endgroup

group "feeds update -a"
./scripts/feeds update -a
endgroup

Then it builds defconfig:

group "make defconfig"
make defconfig
endgroup

We're not gonna go over defconfig, unless I later determine we need it, but more on it can be found here:

https://gitlab.com/openwrt/openwrt/openwrt/-/blob/master/include/toplevel.mk#L120-124
It is a makefile target that generates a .config file used in kconfig build systems.
It is used in place of make menuconfig to generate a default configuration file for the build system.

The majority of the rest of the script is determined by whether $PACKAGES is set or not. LuCI does not set it, so it uses the default behavior:

if [ -z "$PACKAGES" ]; then
    # compile all packages in feed
    for FEED in $ALL_CUSTOM_FEEDS; do
        group "feeds install -p $FEED -f -a"
        ./scripts/feeds install -p "$FEED" -f -a
        endgroup
    done

    RET=0

    make \
        BUILD_LOG="$BUILD_LOG" \
        CONFIG_SIGNED_PACKAGES="$CONFIG_SIGNED_PACKAGES" \
        IGNORE_ERRORS="$IGNORE_ERRORS" \
        CONFIG_AUTOREMOVE=y \
        V="$V" \
        -j "$(nproc)" || RET=$?
else
...
fi

Then it wraps up the build:

if [ "$INDEX" = '1' ];then
    group "make package/index"
    make package/index
    endgroup
fi

if [ -d bin/ ]; then
    mv bin/ /artifacts/
fi

if [ -d logs/ ]; then
    mv logs/ /artifacts/
fi

exit "$RET"

Now that we understand how they build the packages automatically, we can now use this knowledge for building our own way. For now, we will use the recommended make menuconfig as described in Task 1, but we will likely switch to using the GH Actions way of building in the future.

I think the next 2 steps is to finish digesting feeds, then play around with make menuconfig to get a better understanding of the available options.

DEV Community: project-laguardia

LuCI on MGMT - Wk 2 - Day 08

Hurdle 11: Prioritizing Ports and Setting Up VCS

Previously:

Task 1: Addressing Tracking Challenges:

Task 2: Setting Up VCS

Task 2: Cleaning Up My Devtools

LuCI on MGMT - Day 7

Hurdle 7: Backtracking on ucode Files

Previously:

Hurdle 8: Identifying Any rpc Interop

Hurdle 9: Identifying Any ubus Interop

Hurdle 10: Identifying Any Other Languages Used

Task 1: Setup Enry

Task 2: Reviewing the new results

LuCI on MGMT - Day 6

Hurdle 5: Identifying Any /etc/config Interop

Previously:

Task 1: Reviewing Search Results

Hurdle 6: Identifying Any luci.config interop

Task 1: Reviewing Search Results

LuCI on MGMT - Day 05

Hurdle 4: Identifying Where uci Interop Actually Occurs

Previously:

Task 1: Code-Search

Task 2: Reviewing the Results

LuCI on MGMT - Day 04

Hurdle 3: Scoping OpenWRT's Build System Down (Continued)

Previously:

Task 1: Digesting ./scripts/feeds (continued)

Task 2: Digesting make menuconfig

LuCI on MGMT - Day 03

Hurdle 3: Scoping OpenWRT's Build System Down (Continued)

Previously:

Task 1: Digesting ./scripts/feeds and ./scripts/feeds.conf.default

Task 2: Reviewing SDK Usage in GH Actions:

Hurdle 7: Backtracking on `ucode` Files

Hurdle 8: Identifying Any `rpc` Interop

Hurdle 9: Identifying Any `ubus` Interop

Hurdle 5: Identifying Any `/etc/config` Interop

Hurdle 6: Identifying Any `luci.config` interop

Hurdle 4: Identifying Where `uci` Interop Actually Occurs

Task 1: Digesting `./scripts/feeds` (continued)

Task 2: Digesting `make menuconfig`

Task 1: Digesting `./scripts/feeds` and `./scripts/feeds.conf.default`