DEV Community: Nioan

How to Use Unreleased Features in Bazel Rule Sets with git_override

Nioan — Mon, 30 Dec 2024 12:43:06 +0000

Sometimes, you’ll encounter a feature in a rule set that’s already been completed, but the new release isn’t available yet. It’s frustrating to see that shiny feature sitting unused in the repo, just out of reach. Thankfully, with git_override, you can check out any commit hash you want and start using it without waiting for a new release.

I recently faced this issue while wanting to use the unreleased pyi file generation feature in the rules_proto_grpc_python rule set.

In this post I'll explain how to override Bazel modules in a slightly complex rule set repo, using this example.

To use a specific commit hash for this rule set, you can add the following to your MODULE.bazel file:

bazel_dep(name = "rules_proto_grpc", version = "5.0.1")

git_override(
    module_name = "rules_proto_grpc",
    remote = "https://github.com/rules-proto-grpc/rules_proto_grpc.git",
    commit = "d17b5b16c8b12143c6f1b78dabd6bbc228e89b58",
    strip_prefix = "modules/core",
)

bazel_dep(name = "rules_proto_grpc_python", version = "5.0.1")

git_override(
    module_name = "rules_proto_grpc_python",
    remote = "https://github.com/rules-proto-grpc/rules_proto_grpc.git",
    commit = "d17b5b16c8b12143c6f1b78dabd6bbc228e89b58",
    strip_prefix = "modules/python",
)

There are two main complications with this rule set:

Multiple rule set modules exist under the modules/ directory of the repo.
rules_proto_grpc_python depends on rules_proto_grpc, which lives in the same repo.

1. Multiple Modules in the repo

The first complication arises from the multiple modules under the modules/ directory. When using git_override, Bazel needs to know where to find the appropriate MODULE.bazel for the override. If the module names don’t match, it will throw an error. That means that if you don’t use the strip_prefix attribute, Bazel will try to use the root MODULE.bazel (which doesn’t have the same name), and you’ll encounter an error like this:

ERROR: Error computing the main repository mapping: in module dependency 
chain <root> -> rules_proto_grpc_python@_: the MODULE.bazel file of 
rules_proto_grpc_python@_ declares a different name ()

The solution is to use the strip_prefix parameter to define the root of the module. This works because, under the hood, Bazel is using the git_repository rule. A bit cryptic, but it’s how Bazel rolls :).

2. The in-repo dependency

The second complication is that rules_proto_grpc_python depends on rules_proto_grpc, that lives in the same repo. They also both are released toghether and use the same version. Even though you don’t need to add this dependency when not using git_override, you must do so in this case. If you omit it, Bazel will throw an error like this:

ERROR: Error computing the main repository mapping: module not found
in registries: rules_proto_grpc@0.0.0.rpg.version.placeholder

This happens because the MODULE.bazel for the Python ruleset contains a reference like this:

bazel_dep(name = "rules_proto_grpc", version = "0.0.0.rpg.version.placeholder")

The override registers rules_proto_grpc with the placeholder version, allowing the dependency to resolve correctly.

Resolving Python Import Failures in Bazel with `proto_library` Targets

Nioan — Fri, 27 Dec 2024 20:22:06 +0000

TL;DR:

If Python imports fail when a package depends on multiple proto_library targets, it's because Bazel generates non-namespace packages by default.

Add the following flag to your .bazelrc file to resolve this issue:

build --incompatible_default_to_explicit_init_py

At some point in the future it might be enabled by default

Problem Overview

While working with Bazel to build Python code from Protocol Buffers, I encountered a perplexing runtime error:

Importing the first package in PYTHONPATH works, but subsequent imports from other proto_library targets fail.
Reversing the dependency order changes which package is successfully imported.

This issue arises because Bazel-generated Python packages for Protocol Buffers are not namespace packages by default. When two packages share the same top-level path (e.g., proto.common), Python loads only the first one it encounters, masking the others.

Environment and Setup

We're using Bazel with bzlmod in our (auxillis.ai) monorepo, which has the following structure:

proto/
   common/
      some_lib/
         some_lib.proto
         BUILD.bazel
   service/
       some_service/
            some_service.proto
            BUILD.bazel

OurMODULE.bazel file includes dependencies for Python and Protocol Buffers:

# Protobuf
bazel_dep(name = "protobuf", version = "27.3", repo_name = "com_google_protobuf")
bazel_dep(name = "rules_proto", version = "6.0.2")
bazel_dep(name = "rules_proto_grpc_python", version = "5.0.0")

# Python
PYTHON_VERSION = "3.12.4"
bazel_dep(name = "rules_python", version = "0.35.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(python_version = PYTHON_VERSION, is_default = True)

Reproducing the Issue

The following example shows how the issue manifests. If you run the main.py with bazel run, the imports would fail.

`BUILD.bazel` for `common/some_lib`

proto_library(
    name = "auxillis_ai_common_some_lib_proto",
    srcs = ["some_lib.proto"],
    visibility = ["//visibility:public"],
    deps = ["@com_google_protobuf//:struct_proto"],
)

python_proto_library(
    name = "py",
    protos = [":auxillis_ai_common_some_lib_proto"],
    visibility = ["//visibility:public"],
)

`BUILD.bazel` for `service/some_service`

proto_library(
    name = "auxillis_ai_service_some_service_proto",
    srcs = ["some_service.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//proto/common/some_lib:auxillis_ai_common_some_lib_proto",
        "@com_google_protobuf//:any_proto",
    ],
)

python_grpc_library(
    name = "py",
    protos = [":auxillis_ai_service_some_service_proto"],
    visibility = ["//visibility:public"],
)

Python Script (`main.py`)

import os

def main():
    python_path = os.getenv("PYTHONPATH")

    python_path = python_path.split(":")
    common_prefix = os.path.commonprefix(python_path)
    print("Common prefix: ", common_prefix)
    for path in python_path:
        print(path.replace(common_prefix + "/", ""))
    print("-------------------")
    try:
        from proto.common.some_lib import some_lib_pb2
        from proto.service.some_service import some_service_pb_2, some_service_pb_2_grpc
        print(some_lib_pb2)
        print(some_service_pb_2)
        print(some_service_pb_2_grpc)
    except ImportError:
        raise 

if __name__ == "__main__":
    main()

`BUILD.bazel` for `main.py`

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        "//proto/common/some_lib:py",
        "//proto/service/some_service:py",
        "@pip//grpcio",
    ],
)

Root Cause

Inspecting the PYTHONPATH environment variable reveals that Bazel adds generated Python packages to PYTHONPATH in the order specified in the deps argument. However:

Each proto_library generates a directory with an __init__.py file, making it a regular Python package.
When Python loads a package (e.g., proto.common.some_lib), any other packages sharing the same namespace are masked.

Solution

To fix this, you need to configure Bazel to generate namespace packages. Add the following to your .bazelrc:

build --incompatible_default_to_explicit_init_py

This ensures Bazel creates namespace packages, allowing multiple directories to contribute to the same Python package hierarchy. After adding the flag, all imports work as expected.

Key Takeaways

Bazel’s default behavior for Python Protocol Buffer packages can lead to runtime import issues when dependencies overlap.
Adding --incompatible_default_to_explicit_init_py makes Bazel generate namespace packages, resolving the issue.
Use flag if your monorepo has shared Protocol Buffer dependencies.