Calling Python's Async World from C#

Calling Python's Async World from C#: A Deep Dive into PyDotNet's Async Bridge

The problem everyone runs into eventually

At some point, nearly every serious .NET project ends up needing Python. It might be a machine learning model that only has a Python SDK, a data science library that simply doesn't exist on NuGet, or a team that's been doing data engineering in Python for years and isn't about to rewrite it all. The question then becomes: how do you make the two languages talk to each other without the whole thing becoming a maintenance nightmare?

The standard answer for a long time was to spin up a subprocess. Run python script.py, capture stdout, parse it. It works, and it's simple, but it has real costs: startup time measured in hundreds of milliseconds, data that has to be serialized to text and back, and no practical way to handle streaming results or async code. For anything performance-sensitive or interactive, it quickly becomes a bottleneck.

The more sophisticated approach is to use a proper in-process embedding — load the Python shared library directly and call into it via its C API. This is what Python.NET (pythonnet) has been doing for years, and it deserves credit for making .NET/Python interop possible at all. But pythonnet has accumulated some rough edges over time. Its type marshaling layer uses COM-style reflection internally, which adds latency — benchmarks typically show 5–20 µs per call versus the raw C API cost. More critically for modern applications, it has no built-in support for Python's async/await model. There's no way to await a Python coroutine from C# without significant boilerplate, and Python async generators — the backbone of streaming APIs — are simply not addressable. Memory management is also non-deterministic: Py_DecRef gets called from .NET finalizers, which means Python objects can live far longer than expected and collection pauses can surface at unpredictable times.

Where PyDotNet fits in

PyDotNet takes a different approach. It embeds CPython in-process too — that part is the same — but it was designed from the start around three ideas that pythonnet compromised on.

Explicit, deterministic ownership. Every Python object you hold from C# is a using variable. When the using block ends, Py_DecRef is called immediately. There are no finalizer races, no surprise GC pauses, and no lingering Python objects. If you're debugging a refcount issue, the stack tells you exactly where the object was released.

Zero-copy memory. NumPy arrays, PyTorch tensors, Python bytearrays — anything that implements Python's buffer protocol — can be accessed from C# as Span<T> or Memory<T> without copying. DLPack tensor exchange goes further: you can share GPU tensors with PyTorch, JAX, and TensorFlow without any host-side copy at all, even when the data lives on CUDA. pythonnet doesn't do either of these.

A real async bridge. Python's asyncio coroutines become C# Tasks. Python async generators become IAsyncEnumerable<T>. CancellationToken works. Task.WhenAll works. If you're writing await in C# today, calling Python's async world feels like a first-class citizen, not a workaround.

The call latency difference is also worth knowing: PyDotNet benchmarks at roughly 1–3 µs per call, compared to 5–20 µs for pythonnet and 1–50 ms for the subprocess approach.

Approach	Call latency	Zero-copy memory	Async coroutines
PyDotNet	~1–3 µs	✓ Span<T> / DLPack	✓ native Task
pythonnet	~5–20 µs	✗	✗
Subprocess	~1–50 ms	✗	✗
REST / gRPC	~0.5–10 ms	✗	via HTTP/2

---

The async ecosystem problem

Python has quietly built one of the richest async ecosystems in any language. Whether it's asyncio, httpx, LangChain's streaming APIs, or virtually every modern data pipeline library — Python's async/await model is everywhere. The problem is that most of these libraries were designed to live inside a Python event loop, not to be called from a .NET process.

PyDotNet's async bridge handles this. Python coroutines look like ordinary .NET Tasks, and Python async generators look like IAsyncEnumerable<T>. If you're already writing await in C#, consuming Python's async code feels surprisingly natural.

This article walks through the full picture: from the simplest coroutine call to producer/consumer queues, structured concurrency, and proper cancellation.

---

The simplest case: awaiting a coroutine

Say you have a Python function that fetches data asynchronously. It doesn't matter what it actually does — it could call httpx, query a database, or just await asyncio.sleep(...) for demonstration. From C#, you call it with CallAsync<T>():

interp.Execute("""
    import asyncio

    async def slow_add(a, b):
        await asyncio.sleep(0.05)
        return a + b
    """);

using var module = interp.ImportModule("__main__");
using var slowAdd = module.GetFunction("slow_add");

var result = await slowAdd.CallAsync<int>(17, 25);
// result == 42

That's it. The coroutine runs on a .NET thread-pool thread using its own asyncio event loop, and the result comes back as a proper Task<int>. The C# await suspends the calling context without blocking the thread, exactly as you'd expect.

If you don't need the return value — a fire-and-forget log write, for example — there's a non-generic overload:

await log.CallAsync("System started successfully");

---

Running coroutines in parallel

Because CallAsync<T>() returns a real Task<T>, you can feed it straight into Task.WhenAll:

using var greet = module.GetFunction("fetch_greeting");

var tasks = new[]
{
    greet.CallAsync<string>("Alice"),
    greet.CallAsync<string>("Bob"),
    greet.CallAsync<string>("Charlie"),
};

var greetings = await Task.WhenAll(tasks);

Three Python coroutines run concurrently. Each one gets its own event loop on a thread-pool thread, so there's no shared-loop bottleneck. For independent I/O-bound tasks — think fetching from multiple APIs, running a batch of model inferences — this is a straightforward speedup without any manual threading on your side.

---

Keyword arguments

Python functions often have keyword-only arguments, and async functions are no different. PyDotNet passes kwargs via an IDictionary<string, object?>:

var result = await module.CallAsync<string>(
    "compute_stats",
    new object?[] { new[] { 10, 20, 30, 40 } },
    new Dictionary<string, object?> { ["scale"] = 2.0 });

You can also skip GetFunction entirely and call by name through module.CallAsync(...) when you don't need to hold a reference to the function object.

---

Async generators as IAsyncEnumerable<T>

This is where it gets genuinely useful. Python's async generators — functions that yield inside async def — map directly to C#'s IAsyncEnumerable<T>, which means you can iterate them with await foreach:

interp.Execute("""
    import asyncio

    async def fibonacci_stream(count):
        a, b = 0, 1
        for _ in range(count):
            await asyncio.sleep(0)
            yield a
            a, b = b, a + b
    """);

await foreach (var n in module.CallAsyncEnumerable<long>("fibonacci_stream", 8))
{
    Console.Write($"{n} ");
}
// 0 1 1 2 3 5 8 13

Each value is pulled lazily. The Python generator suspends at each yield, and the C# side gets a ValueTask<bool> for each MoveNextAsync. You're not materialising the whole sequence upfront — if the generator produces a continuous feed (prices, sensor readings, log events), your C# code processes items as they arrive.

Kwargs in async generators

For generators that take keyword arguments, use the overload that accepts both positional and keyword arguments:

using var rangeStream = module.GetFunction("range_stream");

await foreach (var v in rangeStream.CallAsyncEnumerable<int>(
    Array.Empty<object?>(),
    new Dictionary<string, object?> { ["start"] = 2, ["stop"] = 10, ["step"] = 3 }))
{
    Console.Write($"{v} ");
}
// 2 5 8

---

Early exit and resource cleanup

One thing that can trip you up with async generators is the finally block. If your Python generator holds a resource — a file, a connection, a lock — it's supposed to release it in finally, which runs when the generator is closed. But if you break out of the loop early, Python needs to be explicitly told to close the generator by calling aclose() on it.

PyDotNet handles this automatically. When you break out of an await foreach loop, DisposeAsync() on the enumerator is called, which calls Python's aclose() coroutine. The generator's finally block runs:

async def resource_stream():
    try:
        for i in range(1000):
            await asyncio.sleep(0)
            yield i
    finally:
        # This runs even if the C# side breaks early
        cleanup()

await foreach (var item in resourceStream.CallAsyncEnumerable<int>())
{
    if (item >= 3)
        break;  // aclose() fires, generator finally-block runs
}

If the generator exhausts naturally, aclose() is skipped — StopAsyncIteration already means there's nothing to close.

---

CancellationToken support

CallAsync<T> accepts a CancellationToken for timeout and cancellation scenarios:

// Already cancelled before the call — throws immediately
using var cts = new CancellationTokenSource();
cts.Cancel();

try
{
    await slowValue.CallAsync<int>(new object?[] { 21 }, cts.Token);
}
catch (OperationCanceledException)
{
    // thrown without ever calling into Python
}

// Timeout-based cancellation
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
var result = await slowValue.CallAsync<int>(new object?[] { 21 }, cts.Token);

When cancellation fires, the returned Task transitions to Canceled. The Python coroutine may still complete on the thread-pool thread it was running on — there's no way to interrupt CPython mid-execution from the outside — but the C# caller gets the cancellation exception promptly.

---

PyAsyncQueue<T>: producer/consumer across the boundary

Sometimes you want a genuine producer/consumer setup where both the .NET side and the Python side are independently active. PyAsyncQueue<T> wraps Python's asyncio.Queue and exposes it as a .NET queue with PutAsync / GetAsync / ReadAllAsync:

using var queue = PyAsyncQueue<string>.Create(interp);

var producer = Task.Run(async () =>
{
    foreach (var msg in new[] { "alpha", "beta", "gamma", "delta", "epsilon" })
    {
        await queue.PutAsync(msg);
        await Task.Delay(20);
    }
});

using var cts = new CancellationTokenSource();
var items = new List<string>();

var consumer = Task.Run(async () =>
{
    await foreach (var item in queue.ReadAllAsync(cts.Token))
    {
        items.Add(item);
        if (items.Count == 5) cts.Cancel();
    }
});

await Task.WhenAll(producer, consumer.ContinueWith(_ => { }));

The queue lives on the Python side (asyncio.Queue), and both .NET tasks interact with it through PyDotNet's GIL management. You can optionally pass a maxsize to Create(interp, maxsize: 10) to get backpressure: PutAsync will block until space is available.

---

PyTaskGroup: concurrent coroutines with a shared result set

When you want to run a batch of coroutines and collect their results together, PyTaskGroup is cleaner than manually managing a list of tasks:

using var computeFunc = module.GetFunction("compute");
using var group = new PyTaskGroup(interp);

group.Add(computeFunc, 3)
     .Add(computeFunc, 4)
     .Add(computeFunc, 5);

var results = await group.RunAsync<int>();
// [9, 16, 25]

Under the hood it uses asyncio.gather(), so all three coroutines run on the same event loop iteration — proper cooperative concurrency, not separate threads. On Python 3.11+ you can also use RunWithTaskGroupAsync() which routes through asyncio.TaskGroup for structured concurrency semantics (exceptions propagate correctly from any member of the group).

---

EvaluateAsync<T>: driving a coroutine you already have

Sometimes Python code creates a coroutine object before you can call it from C#. EvaluateAsync<T> on the interpreter handles this:

interp.Execute("""
    async def async_pow(base, exp):
        await asyncio.sleep(0)
        return base ** exp

    _pending_coro = async_pow(3, 10)
    """);

var result = await interp.EvaluateAsync<long>("_pending_coro");
// 59049

The string is evaluated in Python's __main__ scope, the result is treated as a coroutine object, and it's driven to completion on a thread-pool event loop.

---

How it actually works

A few implementation details are worth knowing.

Each CallAsync<T> call runs the coroutine on a thread-pool thread using a dedicated asyncio event loop — asyncio.new_event_loop(), loop.run_until_complete(coro), loop.close(). This means coroutines that use asyncio.sleep, asyncio.gather, or any standard asyncio primitive work correctly. It also means there's no shared event loop that could become a bottleneck.

The GIL is acquired by the thread-pool thread before any Python call and released after. The calling C# thread never holds the GIL, so .NET's thread pool keeps running normally while Python executes.

For async generators, each MoveNextAsync() dispatches to a thread-pool thread that acquires the GIL, calls __anext__() on the Python async iterator, drives it to the next yield using a mini event loop, and returns the value. The iterator object is owned by the IAsyncEnumerator<T> implementation and released (with aclose() if needed) in DisposeAsync.

---

Putting it together

If your application already does await in C#, PyDotNet's async bridge is the path of least resistance to Python's async ecosystem. You don't need to redesign your architecture, spin up a sidecar process, or learn a new IPC protocol. The Python coroutine looks like a Task. The Python async generator looks like IAsyncEnumerable<T>. Cancellation works with the tokens you already have.

The library is on NuGet:

dotnet add package PyDotNet

Python 3.11–3.14, .NET 8/9/10, Windows/Linux/macOS, x64 and ARM64. The async features don't require any optional packages — they're in the core library.

The sample projects have runnable examples for every scenario covered here.