How to Manage Python Dependencies Natively inside C# Build Pipelines

Python has a massive ecosystem of AI, machine learning, and data science libraries like PyTorch, NumPy, and Transformers. While libraries like PythonNET allow us to call Python from C#, managing the underlying environment setup can be tedious:

• Installing CPython runtimes across different target environments
• Restoring pip packages and virtual environments
• Managing module search paths (sys.path) and native .dll/.so binaries

To streamline this workflow, I built DotNetPip, an open-source MSBuild integration that handles Python dependency resolution directly inside your standard .csproj build target.

How It Works

Instead of running external environment setup scripts, you declare your Python packages directly inside your project file:

<ItemGroup>
  <PythonPackage Include="numpy" Version="1.26.4"/>
  <PythonPackage Include="transformers" Version="4.38.0"/>
</ItemGroup>

When you run dotnet build, custom MSBuild tasks trigger behind the scenes to:
Ensure the Python environment and pip dependencies are restored.
Expose the environment in-process to your C# runtime.

Getting Started

1. Add the DotNetPip package from NuGet: dotnet add package DotNetPip
2. Declare your required Python dependencies in your .csproj.
3. Run dotnet build

The goal is to make using Python's AI/ML ecosystem feel like a first-class citizen inside the .NET toolchain without manual sidecar scripts.

Links & Resources

GitHub Repository: https://github.com/jaime-olivares/DotNetPip

Feedback, edge cases, and contributions are welcome!