Experimentations on Bazel: version.txt

We'll try to create a useful rule that create a version.txt file with the output of git describe --always --dirty:

• If you use git annotated tag for version number, this command will display only the annotated tag when you are on it, else it will provide more info, like the short hash (if no annotated tag), or the number of commit since the last annotated tag, the short hash (prefixed by g) and maybe the suffix dirty if you have uncommitted changes.
• The version.txt could be used as input of other rule to include this metainfo into the build

# commit previous changes then
❯ git describe --always --dirty
5472a43
# add the first annoted tag
❯ git tag -a "0.1.0" -m ":bookmark: 0.1.0"
❯ git describe --always --dirty
0.1.0

For version as git tag: use annotated tag, not 'v' prefix

David Bernard ・ Mar 28 '21

#git

Now create the rule, git describe need to access the content of folder .git to run and bazel require to be explicit about inputs and input files should be under the package. So we will create the rule into ./BUILD.bazel into the parent folder, aka the workspace root folder.

genrule(
    name = "version",
    srcs = [".git"],
    outs = ["version.txt"],
    cmd_bash = "git --git-dir=$(location :.git) describe --always --dirty |tee $@",
)

❯ bazel build //:version
INFO: Analyzed target //:version (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
WARNING: /home/david/src/github.com/davidB/sandbox_bazel/BUILD.bazel:1:8: input '.git' to //:version is a directory; dependency checking of directories is unsound
INFO: From Executing genrule //:version:
0.1.0-dirty
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.046s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

Except the warning seems to work. Commit this code and rerun bazel.

❯ git add BUILD.bazel
❯ git commit -m ":alembic: create version.txt"
[development b5fea4c] :alembic: create version.txt
 1 file changed, 6 insertions(+)
 create mode 100644 BUILD.bazel

❯ git describe --always --dirty
0.1.0-1-gb5fea4c

❯ bazel build //:version
INFO: Analyzed target //:version (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.029s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

:-( bazel did NOT detect the change.

What is instead of declare .git as input we declare all the content of the folder with the function glob ?

genrule(
    name = "version",
    srcs = glob([".git/**"]),
    outs = ["version.txt"],
    cmd_bash = "git --git-dir=$(location :.git) describe --always --dirty |tee $@",
)

❯ bazel build //:version
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/BUILD.bazel:1:8: in cmd_bash attribute of genrule rule //:version: label '//:.git' in $(location) expression is not a declared prerequisite of this rule
ERROR: Analysis of target '//:version' failed; build aborted: Analysis of target '//:version' failed
INFO: Elapsed time: 0.116s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 111 targets configured)

Damned location if for a single file, and .git is not longer listed into srcs. Also because only the files under .git folder are symlinked and not the git working directory the command will always detect the working directory as "dirty". In this case maybe the we should request bazel to NOT run the rule into a sandbox, with local = True (as an exercice try to create the rule envinfo_local by duplicate envinfo and adapt it).

genrule(
    name = "version",
    srcs = glob([".git/**"]),
    outs = ["version.txt"],
    cmd_bash = "git --git-dir=.git --no-pager describe --always --dirty |tee $@",
    local = True,
)

And test it

❯ bazel build //:version
INFO: Analyzed target //:version (1 packages loaded, 112 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:version:
0.1.0-1-gb5fea4c-dirty
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.073s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

❯ git add BUILD.bazel
❯ git commit -m ":alembic: tune version"
❯ git describe --always --dirty
0.1.0-2-ge453fae

❯ bazel build //:version
INFO: Analyzed target //:version (1 packages loaded, 115 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:version:
0.1.0-2-ge453fae-dirty
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.057s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

❯ git describe --always --dirty
0.1.0-2-ge453fae

It's better but we have the unexpected -dirty when running through bazel.

If we modify the cmd_bash to run ls -al before git we'll see that the folder where the git command is running is always a sandbox with symlinks to the folder (more symlinks than without local = True but symlinks plus other file and folders.

❯ bazel build //:version
INFO: Analyzed target //:version (1 packages loaded, 118 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:version:
total 28
drwxr-xr-x 5 david david 4096 Mar 28 17:51 .
drwxr-xr-x 3 david david 4096 Mar 14 11:08 ..
lrwxrwxrwx 1 david david   61 Mar 28 17:51 .bazelversion -> /home/david/src/github.com/davidB/sandbox_bazel/.bazelversion
lrwxrwxrwx 1 david david   52 Mar 28 17:51 .git -> /home/david/src/github.com/davidB/sandbox_bazel/.git
lrwxrwxrwx 1 david david   55 Mar 28 17:51 .github -> /home/david/src/github.com/davidB/sandbox_bazel/.github
lrwxrwxrwx 1 david david   58 Mar 28 17:51 .gitignore -> /home/david/src/github.com/davidB/sandbox_bazel/.gitignore
lrwxrwxrwx 1 david david   59 Mar 28 17:51 BUILD.bazel -> /home/david/src/github.com/davidB/sandbox_bazel/BUILD.bazel
lrwxrwxrwx 1 david david   63 Mar 28 17:51 WORKSPACE.bazel -> /home/david/src/github.com/davidB/sandbox_bazel/WORKSPACE.bazel
drwxr-xr-x 5 david david 4096 Mar 28 17:51 bazel-out
lrwxrwxrwx 1 david david   59 Mar 28 17:51 exp_genrule -> /home/david/src/github.com/davidB/sandbox_bazel/exp_genrule
drwxr-xr-x 2 david david 4096 Mar 28 17:51 external
drwx------ 3 david david 4096 Mar 28 17:51 local-spawn-runner.1529500632871132789
0.1.0-2-ge453fae-dirty
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.098s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

Maybe it's time to find an other solution. We can remove the flags --dirty, but it means that uncommitted change will be able to generate code with invalid VERSION info, and also to destroy part of the trust into this information to debug,...

Workspace status

After back in the doc of basel, we can find a section Workspace status

Use these options to "stamp" Bazel-built binaries: to embed additional information into the binaries, such as the source control revision or other workspace-related information. You can use this mechanism with rules that support the stamp attribute, such as genrule, cc_binary, and more...

Seams to match our goal, let's try it. For details, why,... please read the documentation linked above, but as overlook we can provide a script to bazel that should generate key/value pair(s) the key starting by STABLE_ will be stored into a file bazel-out/stable-status.txt and other into bazel-out/volatile-status.txt. Both files are available to action/rules. Action with attribe stamp = True are retriggered if change is detected into bazel-out/stable-status.txt.

# check if the files already exists
❯ cat bazel-out/stable-status.txt
BUILD_EMBED_LABEL 
BUILD_HOST xxxxxxxxx
BUILD_USER david

❯ cat bazel-out/volatile-status.txt
BUILD_TIMESTAMP 1616946664

Like for the rest of the tutorial, use a linux/macOs only bash script for workspace_status, tools/workspace_status.sh (do not forgot the set it as executable):

mkdir tools
touch tools/workspace_status.sh
chmod +x tools/workspace_status.sh

Define the content of tools/workspace_status.sh as:

#!/bin/bash

echo "STABLE_BUILD_GIT_DESCRIBE $(git --no-pager describe --always --dirty)"

For quick test run with the tools and visual check of stable-status.txt:

❯ bazel build //:version --workspace_status_command "tools/workspace_status.sh"
INFO: Analyzed target //:version (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.049s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

❯ cat bazel-out/stable-status.txt
BUILD_EMBED_LABEL 
BUILD_HOST dwaynebox6
BUILD_USER david
STABLE_BUILD_GIT_DESCRIBE 0.1.0-2-ge453fae

The version number doesn't include the -dirty because dirty flag ignores untracked files like the new file we created (it's not perfect).

So no let modify the version rule to use info from bazel-out/stable-status.txt:

genrule(
    name = "version",
    outs = ["version.txt"],
    # alternative: "sed -n 's/STABLE_BUILD_GIT_DESCRIBE //p' bazel-out/stable-status.txt"
    cmd_bash = "grep -Po '^STABLE_BUILD_GIT_DESCRIBE\\s+\\K.*' bazel-out/stable-status.txt |tee $@",
    stamp = True,
)

❯ bazel build //:version --workspace_status_command "tools/workspace_status.sh"
INFO: Analyzed target //:version (1 packages loaded, 1 target configured).
INFO: Found 1 target...
INFO: From Executing genrule //:version:
0.1.0-2-ge453fae-dirty
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.079s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

Before doing a test without uncommitted change, we'll add register --workspace_status_command "tools/workspace_status.sh" as a default flags for every bazel build calls via .bazelrc:

echo 'build --workspace_status_command "tools/workspace_status.sh"' >>.bazelrc

No its time to commit and test if dirty go away.

❯ git add BUILD.bazel tools/* .bazelrc
❯ git commit -m ":recycle: rewrite version using workspace status"

❯ bazel build //:version
INFO: Analyzed target //:version (1 packages loaded, 1 target configured).
INFO: Found 1 target...
INFO: From Executing genrule //:version:
0.1.0-3-g95b84e9
Target //:version up-to-date:
  bazel-bin/version.txt
INFO: Elapsed time: 0.087s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

🎉

Conclusion

It was a long experimentation to have this version.txt file that we can (re)use in real project. We gain better understanding and experience with genrule and we discover local and stamp attribute.

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: article/3_versiontxt.

Comments

[Jack]

Thank you for posting this. Can this version be passed as a parameter to a python wheel label within the build.bzl? I have a process where I generate a wheel using bazel and then a separate script to replace the wheel version with my git tag. I would like to remove the second step and let bazel pull the version.

[David Bernard]

I don't know, it depends how to generate the wheel. Maybe in the command that generate the wheel you can read the content of version.txt and use it as parameter, or to change the content of the config file you use to generate the wheel (eg: setup.py, pyptoject.toml,...)

PS if your question is "how to cinvert content of the file (like version.txt) into a parameter of a bazel function (from the caller side)?", Sorry I don't know. But I'm interested by the answer if you find it.