Resolving Python Import Failures in Bazel with `proto_library` Targets

TL;DR:

If Python imports fail when a package depends on multiple proto_library targets, it's because Bazel generates non-namespace packages by default.

Add the following flag to your .bazelrc file to resolve this issue:

build --incompatible_default_to_explicit_init_py

At some point in the future it might be enabled by default

---

Problem Overview

While working with Bazel to build Python code from Protocol Buffers, I encountered a perplexing runtime error:

• Importing the first package in PYTHONPATH works, but subsequent imports from other proto_library targets fail.
• Reversing the dependency order changes which package is successfully imported.

This issue arises because Bazel-generated Python packages for Protocol Buffers are not namespace packages by default. When two packages share the same top-level path (e.g., proto.common), Python loads only the first one it encounters, masking the others.

---

Environment and Setup

We're using Bazel with bzlmod in our (auxillis.ai) monorepo, which has the following structure:

proto/
   common/
      some_lib/
         some_lib.proto
         BUILD.bazel
   service/
       some_service/
            some_service.proto
            BUILD.bazel

OurMODULE.bazel file includes dependencies for Python and Protocol Buffers:

# Protobuf
bazel_dep(name = "protobuf", version = "27.3", repo_name = "com_google_protobuf")
bazel_dep(name = "rules_proto", version = "6.0.2")
bazel_dep(name = "rules_proto_grpc_python", version = "5.0.0")

# Python
PYTHON_VERSION = "3.12.4"
bazel_dep(name = "rules_python", version = "0.35.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(python_version = PYTHON_VERSION, is_default = True)

Reproducing the Issue

The following example shows how the issue manifests. If you run the main.py with bazel run, the imports would fail.

BUILD.bazel for common/some_lib

proto_library(
    name = "auxillis_ai_common_some_lib_proto",
    srcs = ["some_lib.proto"],
    visibility = ["//visibility:public"],
    deps = ["@com_google_protobuf//:struct_proto"],
)

python_proto_library(
    name = "py",
    protos = [":auxillis_ai_common_some_lib_proto"],
    visibility = ["//visibility:public"],
)

BUILD.bazel for service/some_service

proto_library(
    name = "auxillis_ai_service_some_service_proto",
    srcs = ["some_service.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//proto/common/some_lib:auxillis_ai_common_some_lib_proto",
        "@com_google_protobuf//:any_proto",
    ],
)

python_grpc_library(
    name = "py",
    protos = [":auxillis_ai_service_some_service_proto"],
    visibility = ["//visibility:public"],
)

Python Script (main.py)

import os

def main():
    python_path = os.getenv("PYTHONPATH")

    python_path = python_path.split(":")
    common_prefix = os.path.commonprefix(python_path)
    print("Common prefix: ", common_prefix)
    for path in python_path:
        print(path.replace(common_prefix + "/", ""))
    print("-------------------")
    try:
        from proto.common.some_lib import some_lib_pb2
        from proto.service.some_service import some_service_pb_2, some_service_pb_2_grpc
        print(some_lib_pb2)
        print(some_service_pb_2)
        print(some_service_pb_2_grpc)
    except ImportError:
        raise 

if __name__ == "__main__":
    main()

BUILD.bazel for main.py

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        "//proto/common/some_lib:py",
        "//proto/service/some_service:py",
        "@pip//grpcio",
    ],
)

---

Root Cause

Inspecting the PYTHONPATH environment variable reveals that Bazel adds generated Python packages to PYTHONPATH in the order specified in the deps argument. However:

1. Each proto_library generates a directory with an __init__.py file, making it a regular Python package.
2. When Python loads a package (e.g., proto.common.some_lib), any other packages sharing the same namespace are masked.

---

Solution

To fix this, you need to configure Bazel to generate namespace packages. Add the following to your .bazelrc:

build --incompatible_default_to_explicit_init_py

This ensures Bazel creates namespace packages, allowing multiple directories to contribute to the same Python package hierarchy. After adding the flag, all imports work as expected.

---

Key Takeaways

• Bazel’s default behavior for Python Protocol Buffer packages can lead to runtime import issues when dependencies overlap.
• Adding --incompatible_default_to_explicit_init_py makes Bazel generate namespace packages, resolving the issue.
• Use flag if your monorepo has shared Protocol Buffer dependencies.