DEV Community

Nioan
Nioan

Posted on

Resolving Python Import Failures in Bazel with `proto_library` Targets

TL;DR:

If Python imports fail when a package depends on multiple proto_library targets, it's because Bazel generates non-namespace packages by default.

Add the following flag to your .bazelrc file to resolve this issue:

build --incompatible_default_to_explicit_init_py
Enter fullscreen mode Exit fullscreen mode

At some point in the future it might be enabled by default

Problem Overview

While working with Bazel to build Python code from Protocol Buffers, I encountered a perplexing runtime error:

  • Importing the first package in PYTHONPATH works, but subsequent imports from other proto_library targets fail.
  • Reversing the dependency order changes which package is successfully imported.

This issue arises because Bazel-generated Python packages for Protocol Buffers are not namespace packages by default. When two packages share the same top-level path (e.g., proto.common), Python loads only the first one it encounters, masking the others.

Environment and Setup

We're using Bazel with bzlmod in our (auxillis.ai) monorepo, which has the following structure:

proto/
   common/
      some_lib/
         some_lib.proto
         BUILD.bazel
   service/
       some_service/
            some_service.proto
            BUILD.bazel
Enter fullscreen mode Exit fullscreen mode

OurMODULE.bazel file includes dependencies for Python and Protocol Buffers:

# Protobuf
bazel_dep(name = "protobuf", version = "27.3", repo_name = "com_google_protobuf")
bazel_dep(name = "rules_proto", version = "6.0.2")
bazel_dep(name = "rules_proto_grpc_python", version = "5.0.0")

# Python
PYTHON_VERSION = "3.12.4"
bazel_dep(name = "rules_python", version = "0.35.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(python_version = PYTHON_VERSION, is_default = True)
Enter fullscreen mode Exit fullscreen mode

Reproducing the Issue

The following example shows how the issue manifests. If you run the main.py with bazel run, the imports would fail.

BUILD.bazel for common/some_lib 

proto_library(
    name = "auxillis_ai_common_some_lib_proto",
    srcs = ["some_lib.proto"],
    visibility = ["//visibility:public"],
    deps = ["@com_google_protobuf//:struct_proto"],
)

python_proto_library(
    name = "py",
    protos = [":auxillis_ai_common_some_lib_proto"],
    visibility = ["//visibility:public"],
)
Enter fullscreen mode Exit fullscreen mode

BUILD.bazel for service/some_service 

proto_library(
    name = "auxillis_ai_service_some_service_proto",
    srcs = ["some_service.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//proto/common/some_lib:auxillis_ai_common_some_lib_proto",
        "@com_google_protobuf//:any_proto",
    ],
)

python_grpc_library(
    name = "py",
    protos = [":auxillis_ai_service_some_service_proto"],
    visibility = ["//visibility:public"],
)
Enter fullscreen mode Exit fullscreen mode

Python Script (main.py) 

import os

def main():
    python_path = os.getenv("PYTHONPATH")

    python_path = python_path.split(":")
    common_prefix = os.path.commonprefix(python_path)
    print("Common prefix: ", common_prefix)
    for path in python_path:
        print(path.replace(common_prefix + "/", ""))
    print("-------------------")
    try:
        from proto.common.some_lib import some_lib_pb2
        from proto.service.some_service import some_service_pb_2, some_service_pb_2_grpc
        print(some_lib_pb2)
        print(some_service_pb_2)
        print(some_service_pb_2_grpc)
    except ImportError:
        raise 

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

BUILD.bazel for main.py 

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        "//proto/common/some_lib:py",
        "//proto/service/some_service:py",
        "@pip//grpcio",
    ],
)
Enter fullscreen mode Exit fullscreen mode

Root Cause

Inspecting the PYTHONPATH environment variable reveals that Bazel adds generated Python packages to PYTHONPATH in the order specified in the deps argument. However:

  1. Each proto_library generates a directory with an __init__.py file, making it a regular Python package.
  2. When Python loads a package (e.g., proto.common.some_lib), any other packages sharing the same namespace are masked.

Solution

To fix this, you need to configure Bazel to generate namespace packages. Add the following to your .bazelrc:

build --incompatible_default_to_explicit_init_py
Enter fullscreen mode Exit fullscreen mode

This ensures Bazel creates namespace packages, allowing multiple directories to contribute to the same Python package hierarchy. After adding the flag, all imports work as expected.

Key Takeaways

  • Bazel’s default behavior for Python Protocol Buffer packages can lead to runtime import issues when dependencies overlap.
  • Adding --incompatible_default_to_explicit_init_py makes Bazel generate namespace packages, resolving the issue.
  • Use flag if your monorepo has shared Protocol Buffer dependencies.

Top comments (0)