Create SPDX with Yocto

AngryMane — Sun, 19 Feb 2023 06:07:25 +0000

Overview

Poky(yocto) has support to generate spdx files.(create-spdx.bbclass). This article explains how to use it.

There are other ways to generate spddx files with using meta-spdxscanner, but they are not described here.

How to generate spdx files

At first, you have to do this as usual.

$ source oe-init-build-env

Then, edit conf/local.conf as follows.

+ INHERIT += "create-spdx"      # required. enable to generate spdx files.
+ SPDX_PRETTY = "1"             # optional. if "1", output spdx files will be formatted.
+ SPDX_INCLUDE_SOURCES = "1"    # optional. if "1", output spdx files includes [file-information section](https://spdx.github.io/spdx-spec/v2.3/file-information/).
+ SPDX_ARCHIVE_SOURCES = "1"    # optional. if "1", bitbake will create source files archive for each package.
+ SPDX_ARCHIVE_PACKAGED = "1"   # optional. if "1", bitbake will create output binary archive for each package.

Finally, run build.

# for example, `bitbake core-image-minimal`
$ bitbake ${TARGET_IMAGE_NAME}

If you want to generate one spdx file for specific package.
Please note that this command output spdx files to ${WORKDIR}/spdx of target package.

# for example, `bitbake -c create_spdx gcc`
$ bitbake -c create_spdx ${TARGET_PACKAGE_NAME}

# if you want to confirm ${WORKDIR} of ${TARGET_PACKAGE_NAME}
# for example, `bitbake -e gcc | grep ^WORKDIR=`
$ bitbake -e ${TARGET_PACKAGE_NAME} | grep ^WORKDIR=

Outputs

bitbake outputs as follows.

# if you want to confirm ${DEPLOY_DIR}
# for example, `bitbake -e core-image-minimal | grep ^DEPLOY_DIR=`
$ bitbake -e ${TARGET_IMAGE_NAME} | grep ^DEPLOY_DIR= 

# for example, `cd tmp/deploy`
$ cd ${DEPLOY_DIR}
$ tree -d 
.
├── images # Outputs
│   └── ${MACHINE}
│       ├── ${IMAGE_NAME}-${MACHINE}.spdx.index.json
│       ├── ${IMAGE_NAME}-${MACHINE}.spdx.json
│       └── ${IMAGE_NAME}-${MACHINE}.spdx.tar.zst
│
└── spdx   # intermediate product. With the exception of some ARCHIVE files, they are just intermediate generated files and are not explained here.

${IMAGE_NAME}-${MACHINE}.spdx.json

This is a spdx file for target image.

${IMAGE_NAME}-${MACHINE}.spdx.index.json

This file includes a spdx file list that is correspondint to packages following below conditions.

the package installed into target image
the package depended by packages installed into target image

${IMAGE_NAME}-${MACHINE}.spdx.tar.zst

This archive file contains spdx files listed in ${IMAGE_NAME}-${MACHINE}.spdx.index.json.
There are 3-types of spdx file in this archive.

${PN}.spdx.json
recipe-${PN}.spdx.json
runtime-${PN}.spdx.json

Only file-information section and relationship information section of these files are described below.

The other sections are not so distinctive and should be checked by actually looking at the files.

If you don't know file-information section or relationship information section, see following links.

${PN}.spdx.json

file-information section(files)
- files created by the target package and installed in the target image
relationship information section(relationships)
- reference to the file-information section of the source files of this package
- reference to the file-information section of the header files of external libraries

recipe-${PN}.spdx.json

file-information section(files)
- the source files of the target package.
relationship information section(relationships)
- reference to the file-information section of the source files of this package
- reference to the file-information section of the recipe files of external libraries

runtime-${PN}.spdx.json

file-information section(files)
- Nothing
relationship information section(relationships)
- reference to the package information section of the packages this package depends on

Which information is where?

installed packages list

# for example, `cat tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.manifest`
$ cat  ${DEPLOY_DIR}/images/${MACHINE}/${TARGET_IMAGE_NAME}-${MACHINE}.manifest

license info of package

# for example, `tar -I zstd -xvf core-image-minimal-qemux86-64.spdx.tar.zst`
$ tar -I zstd -xvf ${IMAGE_NAME}-${MACHINE}.spdx.tar.zst

# for example, `cat tmp/deploy/images/qemux86-64/libc6.spdx.json`
$ cat ${DEPLOY_DIR}/images/${MACHINE}/${TARGET_PACKAGE_NAME}.spdx.json
...
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-libc6",
      "copyrightText": "NOASSERTION",
      "downloadLocation": "NOASSERTION",
      "hasFiles": [
        "SPDXRef-PackagedFile-libc6-1",
        "SPDXRef-PackagedFile-libc6-2",
        "SPDXRef-PackagedFile-libc6-3",
        "SPDXRef-PackagedFile-libc6-4",
        "SPDXRef-PackagedFile-libc6-5",
        "SPDXRef-PackagedFile-libc6-6",
        "SPDXRef-PackagedFile-libc6-7",
        "SPDXRef-PackagedFile-libc6-8",
        "SPDXRef-PackagedFile-libc6-9",
        "SPDXRef-PackagedFile-libc6-10",
        "SPDXRef-PackagedFile-libc6-11",
        "SPDXRef-PackagedFile-libc6-12",
        "SPDXRef-PackagedFile-libc6-13",
        "SPDXRef-PackagedFile-libc6-14",
        "SPDXRef-PackagedFile-libc6-15",
        "SPDXRef-PackagedFile-libc6-16"
      ],
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "GPL-2.0-only AND LGPL-2.1-only",
      "licenseInfoFromFiles": [
        "NOASSERTION"
      ],
      "name": "libc6",
      "packageFileName": "libc6.tar.zst",
      "packageVerificationCode": {
        "packageVerificationCodeValue": "dc1d4ff83689b1922ab1a0b8dfaa80e40986c3b2"
      },
      "supplier": "Organization: OpenEmbedded ()",
      "versionInfo": "2.35"
    }
  ],
...

license info of source files

# for example, `tar -I zstd -xvf core-image-minimal-qemux86-64.spdx.tar.zst`
$ tar -I zstd -xvf ${IMAGE_NAME}-${MACHINE}.spdx.tar.zst

# for example, `cat tmp/deploy/images/qemux86-64/recipe-libffi.spdx.json`
$ cat ${DEPLOY_DIR}/images/${MACHINE}/recipe-${TARGET_PACKAGE_NAME}.spdx.json
...
  "files": [
    {
      "SPDXID": "SPDXRef-SourceFile-libffi-1",
      "checksums": [
        {
          "algorithm": "SHA1",
          "checksumValue": "1541a69f581bcdf9405369d32fc834cb0d6aa621"
        },
        {
          "algorithm": "SHA256",
          "checksumValue": "e99c283675c59ad08620044e9b8758acb318dc1c93944009470b594db31e961b"
        }
      ],
      "copyrightText": "NOASSERTION",
      "fileName": "not-win32.patch",
      "fileTypes": [
        "SOURCE"
      ],
      "licenseConcluded": "NOASSERTION",
      "licenseInfoInFiles": [
        "NOASSERTION"
      ]
    },
    {
      "SPDXID": "SPDXRef-SourceFile-libffi-2",
      "checksums": [
        {
          "algorithm": "SHA1",
          "checksumValue": "245bae99b2b5916865ee2e5c9dc9f817cf8771bb"
        },
        {
          "algorithm": "SHA256",
          "checksumValue": "328c7c2908c43ce42eefda072307d2fc7fe8f5ae1d21f7f9f0b832e469e21be9"
        }
      ],
      "copyrightText": "NOASSERTION",
      "fileName": "0001-arm-sysv-reverted-clang-VFP-mitigation.patch",
      "fileTypes": [
        "SOURCE"
      ],
      "licenseConcluded": "NOASSERTION",
      "licenseInfoInFiles": [
        "NOASSERTION"
      ]
    },
    {
      "SPDXID": "SPDXRef-SourceFile-libffi-3",
      "checksums": [
        {
          "algorithm": "SHA1",
          "checksumValue": "88f1017994a6dc390903d73474fd193b86038564"
        },
        {
          "algorithm": "SHA256",
          "checksumValue": "e4d016c7690f82469d7eb52487e56c3c26429c5400f7f86e056b12e54a746ad3"
        }
      ],
      "copyrightText": "NOASSERTION",
      "fileName": "libffi-3.4.4/fficonfig.h.in",
      "fileTypes": [
        "SOURCE"
      ],
      "licenseConcluded": "NOASSERTION",
      "licenseInfoInFiles": [
        "NOASSERTION"
      ]
    },
    ...

source files of the package

To do this, please set SPDX_INCLUDE_SOURCES and SPDX_ARCHIVE_SOURCES as "1".

$ tar -I zstd -xvf ${DEPLOY_DIR}/spdx/${MACHINE}/recipes/recipe-${PN}.yst.zst

installed binary of the package

To do this, please set SPDX_ARCHIVE_PACKAGED as "1".

$ tar -I zstd -xvf  ${DEPLOY_DIR}/spdx/${MACHINE}/packages/${PN}.yst.zst

Let's use bitbake commands from python!

AngryMane — Sat, 20 Aug 2022 12:36:37 +0000

First of all

I'm working on building a custom linux build using yocto.
In this job, I often get requests like

I need a list of files that inherit from a specific bbclass (like core-image.bbclass).
I need to list the variables (e.g. PE, PV, PR) with all packages
I need to list the bbappend files for a specific package (e.g. gcc).

Isn't that a pain in the ass? I don't want to grep, and I don't want to build because it takes too long...

Let's use Bitbake from python.

It's a pain in the ass I think, so let's investigate how to do something about it. First of all, the build mechanism by bitbake command is roughly like this.

This Tinfoil is a little bit complicated, so it's not easy to use. The actual code is here. On the other hand, IF between server_client is defined somewhat nicely here. Considering these circumstances, it seems like I should do like below.

If I make this bbclient in python, it will be easy to use from language server and so on.

What I made

Based on the results of the study so far, the bbclient I made is shown in below.

https://pypi.org/project/bbclient/

I've also prepared some documentation.

What you can do

There are so many IFs that I can't show you all of them, but for now, let's try to implement the use case I mentioned at the beginning of this article.
First of all, the way to initialize the bbclient instance is shown in below.

import bbclient

# Initialize bbclient instance
project_path: str = "/PATH/TO/POKY"
init_command: str = ". oe-init-build-env"
client: BBClient = BBClient(project_path, init_command)

# launch server 
server_adder: str = "localhost"
server_port: int = 8081
client.start_server(server_adder, server_port)

# set event filter
ui_handler: int = client.get_uihandler_num()
client.set_event_mask(ui_handler, logging.DEBUG, {}, ["*"])

# paser recipes and create caches
ret = client.parse_files()
client.wait_done_async()

You can use this client to implement each of the following use cases.

I need a list of recipe files that inherit a specific bbclass (like core-image.bbclass).

ret: List[GetRecipeInheritsResult] = client.get_recipe_inherits()
recipes_inherit = [i for i in ret if "/PATH/TO/core-image.bbclass" in i.inherit_file_paths]
for i in recipes_inherit:
    print(i.recipe_file_path)

I need you to list the variables (e.g. PE, PV, PR) with all packages

ret: List[GetRecipesResult] = client.get_recipes()
for i in ret:
    data_store_index: int = client.parse_recipe_file(i.recipe_files)
    pe: Any = client.data_store_connector_cmd(data_store_index, "getVar", "PE")
    pv: Any = client.data_store_connector_cmd(data_store_index, "getVar", "PE")
    pr: Any = client.data_store_connector_cmd(data_store_index, "getVar", "PE")
    print(pe, pv, pr)

I need you to list the bbappend files for a specific package (e.g. gcc).

ret: List[str] = client.find_best_provider("gcc")
target_recipe_file_path: str = ret[3]
ret: List[str] = client.get_file_appends(target_recipe_file_path)
print(ret)

You can also build images from this client and generate dependency taskdepends files.

Finally

If you would like to try it, please use it!

DEV Community: AngryMane