DEV Community: Jung Hyun, Nam

Speech, search, and Stable Diffusion — calling HuggingFace from C#

Jung Hyun, Nam — Mon, 11 May 2026 12:16:30 +0000

TL;DR. I wrote DotNetPy, a small C# library that calls into CPython via the C API.
Version 0.6.0 ships three working samples — semantic search with sentence-transformers, speech-to-text with Whisper, and text-to-image with Stable Diffusion Turbo — and a verification matrix that runs them under classic CPython 3.13 and the new free-threaded builds (3.13t / 3.14t).
The whole thing is Native AOT-compatible, and the per-call isolation that PEP 703 needs is exposed as a one-liner: Python.CreateIsolated().
Code: https://github.com/rkttu/dotnetpy.
NuGet: dotnet add package DotNetPy.

The problem

Every few months I run into the same pattern: I need a HuggingFace model — Whisper for transcripts, a sentence-transformer for retrieval, sometimes Stable Diffusion — and I'm working in C#.

The usual escape hatches all have downsides:

Convert to ONNX. Works for many vision/encoder models. Doesn't work for newer architectures, doesn't work for diffusion pipelines without a lot of effort, and the conversion itself is a separate project.
Stand up a Python micro-service. Now you've got two processes, two deployment stories, and a network hop in your hot path.
Call an external API. Costs money, requires internet, and your data leaves the box.
Use pythonnet or CSnakes. Solid choices. But pythonnet doesn't currently support Native AOT, and CSnakes pushes you into a Source Generator workflow. Neither has a public story for the free-threaded CPython builds yet.

I wanted a thinner option: write Python inline as a string from C#, pass arrays in, get JSON-shaped results back, and have the whole thing AOT-compile to a single binary. That's what DotNetPy is. The three samples below all run end-to-end on a Windows 11 laptop with no GPU.

Sample 1 — Semantic search with `sentence-transformers`

The first sample encodes a small corpus, embeds a query, and returns the top-K most similar sentences as a DotNetPyValue — a JSON-document wrapper that gets you back into the .NET world with GetString(), GetInt32(), GetDouble(), and path-based property access.

using DotNetPy;
using DotNetPy.Uv;

using var project = PythonProject.CreateBuilder()
    .WithProjectName("dotnetpy-ml-embeddings")
    .WithPythonVersion("==3.12.*")
    .AddDependencies(
        "sentence-transformers==2.7.0",
        "transformers==4.40.2",
        "torch>=2.2,<2.5")
    .Build();

await project.InitializeAsync();
var executor = project.GetExecutor();

executor.Execute(@"
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
");

var corpus = new[]
{
    "Python is a popular programming language for data science.",
    "C# and .NET are great for building enterprise applications.",
    "Rust offers memory safety without garbage collection.",
    "Pizza is delicious with various toppings.",
    // …
};
var query = "Tell me about programming languages";

using var hits = executor.ExecuteAndCapture(@"
corpus_emb = model.encode(corpus, normalize_embeddings=True)
query_emb = model.encode([query], normalize_embeddings=True)[0]
sims = corpus_emb @ query_emb
top_idx = np.argsort(-sims)[:3]
result = [
    {'rank': int(rank + 1), 'score': float(sims[i]), 'text': corpus[int(i)]}
    for rank, i in enumerate(top_idx)
]
", new Dictionary<string, object?> { { "corpus", corpus }, { "query", query } });

foreach (var hit in hits!.RootElement.EnumerateArray())
{
    Console.WriteLine($"  {hit.GetProperty("rank").GetInt32()}. " +
                      $"[{hit.GetProperty("score").GetDouble():F3}] " +
                      $"{hit.GetProperty("text").GetString()}");
}

Actual output:

  1. [0.578] Python is a popular programming language for data science.
  2. [0.370] C# and .NET are great for building enterprise applications.
  3. [0.203] Rust offers memory safety without garbage collection.

The interesting bit is the boundary: corpus is a .NET string[] and query is a .NET string, both arriving in Python as native lists/strings. The scored results come back as a JSON document that the .NET side reads with the usual JsonElement API.

Sample 2 — Speech-to-text with Whisper

Same shape, different modality. Drop a .wav/.flac file in, get back text plus chunk-level timestamps. The audio bytes never cross the .NET ↔ Python boundary — Python opens the file directly, the boundary only carries the structured transcript.

var executor = project.GetExecutor();
executor.Execute(@"
from transformers import pipeline
import torch
asr = pipeline(
    'automatic-speech-recognition',
    model='openai/whisper-base.en',
    chunk_length_s=30,
    return_timestamps=True,
    torch_dtype=torch.float32,
)
");

using var transcript = executor.ExecuteAndCapture(@"
out = asr(audio_path)
chunks = [
    {'start': float(c['timestamp'][0]), 'end': float(c['timestamp'][1]),
     'text': c['text'].strip()}
    for c in out.get('chunks', [])
    if c['timestamp'][0] is not None and c['timestamp'][1] is not None
]
result = {'text': out['text'].strip(), 'chunks': chunks}
", new Dictionary<string, object?> { { "audio_path", audioPath } });

Console.WriteLine($"\"{transcript!.GetString("text")}\"");
foreach (var c in transcript.RootElement.GetProperty("chunks").EnumerateArray())
    Console.WriteLine($"  [{c.GetProperty("start").GetDouble():F2}s → " +
                      $"{c.GetProperty("end").GetDouble():F2}s] " +
                      $"{c.GetProperty("text").GetString()}");

Run against a public-domain JFK clip that ships with the sample:

"And so my fellow Americans, ask not what your country can do for you,
 ask what you can do for your country."

  [0.00s → 11.00s] And so my fellow Americans, ask not what your country can
                   do for you, ask what you can do for your country.

whisper-base.en is 290 MB, transcription of an 11-second clip takes about 7 seconds on CPU on my machine. Subsequent runs reuse the cached model and the venv — the only first-run cost is the initial download.

Sample 3 — Text-to-image with Stable Diffusion Turbo

stabilityai/sd-turbo is a one-step diffusion model. On CPU you get a 512×512 image in ~30 seconds; on a recent GPU you get one in ~2 seconds. The .NET side never sees the image bytes — Python writes the PNG to disk and hands back metadata.

executor.Execute(@"
import torch
from diffusers import AutoPipelineForText2Image
pipe = AutoPipelineForText2Image.from_pretrained(
    'stabilityai/sd-turbo',
    torch_dtype=torch.float32,
    safety_checker=None, requires_safety_checker=False,
)
pipe.set_progress_bar_config(disable=True)
");

using var meta = executor.ExecuteAndCapture(@"
import time, os
t0 = time.time()
img = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images[0]
elapsed = time.time() - t0
out_path = os.path.join(out_dir, 'generated.png')
img.save(out_path)
result = {
    'path': out_path,
    'width': img.size[0],
    'height': img.size[1],
    'size_bytes': os.path.getsize(out_path),
    'elapsed_seconds': elapsed,
}
", new Dictionary<string, object?>
{
    { "prompt", "a serene mountain lake at sunset, oil painting style" },
    { "out_dir", outDir },
});

Console.WriteLine($"  Saved:   {meta!.GetString("path")}");
Console.WriteLine($"  Size:    {meta.GetInt32("width")}×{meta.GetInt32("height")} px, " +
                  $"{meta.GetInt32("size_bytes"):N0} bytes");
Console.WriteLine($"  Inference: {meta.GetDouble("elapsed_seconds"):F2}s");

Output:

  Saved:   .../samples/ml-image-gen/output/generated.png
  Size:    512×512 px, 434,242 bytes
  Inference: 31.19s

This is the pattern I want to highlight: only structured data crosses the boundary. The PNG bytes (~400 KB), the embedding matrices, the float32 tensors — they all stay Python-side. The .NET side sees prompts going in and small JSON objects coming out.

Install + first run

The library is just a NuGet package:

dotnet add package DotNetPy --version 0.6.0

If you want to follow the samples literally, samples/ in the repo has the three above plus a native-aot consumer that drives the AOT-published native DLL through its C exports — the path for embedding DotNetPy into C/C++/Rust hosts.

The ML samples use uv to provision Python + the HuggingFace stack declaratively from C#:

using var project = PythonProject.CreateBuilder()
    .WithProjectName("my-app")
    .WithPythonVersion("==3.12.*")
    .AddDependencies("transformers==4.40.2", "torch>=2.2,<2.5")
    .Build();

await project.InitializeAsync();

That's enough to get a working executor. No separate Python install, no manual venv juggling.

The part I actually care about: PEP 703 free-threaded Python

The interesting cliff for an interop library lands in 2025–26. CPython 3.13 introduced free-threaded builds (the t suffix: python3.13t). The GIL goes away, and concurrent threads can really run Python code in parallel for the first time. This is fantastic for ML serving — you want multiple inference workers sharing one process — and it breaks a lot of implicit invariants in libraries written against the classic GIL.

Specifically, pythonnet has been working through this. PR #2721 catalogues five categories of work needed:

Refcount layout changes (ob_refcnt is now a split structure)
Concurrent type/object cache races
Reflection.Emit thread safety
GCHandle slot ownership atomicity
Finalizer / Py_Finalize race

When DotNetPy 0.6.0 happened, I used pythonnet's PR as the audit lens. Four of those five categories don't apply to DotNetPy by design — it doesn't bridge .NET and Python type systems, doesn't subclass Python types from CLR, doesn't use Reflection.Emit, doesn't expose GCHandle slots to Python, and doesn't call Py_Finalize. The fifth (finalizer / shutdown) was mitigated with an explicit PyGILState_Ensure guard around Py_DecRef in SafeHandle.ReleaseHandle.

What did surface from the audit, and got fixed in 0.6.0:

Internal scratch names in shared __main__ globals. Every helper variable (_json_result, _is_valid, …) is now minted per-call via Interlocked.Increment, so two concurrent callers don't race on the same slot.
Evaluate leaking a shared result global. Same fix — per-call unique sink, cleaned up in finally.
The two __main__-globals fixes interact in a subtle way: even after they shipped, the existing user-variable injection (the variables: parameter on Execute / ExecuteAndCapture) still wrote into shared __main__ globals. Two concurrent callers using the same user name would still collide.

The fix for that — and the most user-visible 0.6.0 addition — is a factory:

using var iso = Python.CreateIsolated();
iso.Execute("import json");
iso.Execute("data = {'k': 1}");   // only this executor sees `data`

CreateIsolated() produces an executor that owns its own Python dict, pre-populated with __builtins__. Each isolated executor coexists with the shared singleton and with other isolated executors; nothing leaks between them.

That makes the concurrent ML pattern obvious:

Parallel.For(0, Environment.ProcessorCount, threadId =>
{
    using var iso = Python.CreateIsolated();
    iso.Execute("import torch; from transformers import pipeline");
    iso.Execute(@"
asr = pipeline('automatic-speech-recognition',
               model='openai/whisper-base.en')
");
    using var r = iso.ExecuteAndCapture(@"
out = asr(audio_path)
result = {'text': out['text']}
", new Dictionary<string, object?> { { "audio_path", path } });

    Console.WriteLine(r?.GetString("text"));
});

On a free-threaded CPython build, that loop runs truly in parallel — every worker has its own asr pipeline and its own Python namespace. On the classic GIL build the same code is correct but serializes at the interpreter (and you'd hit the same wall regardless of which interop library you used).

I verified the matrix on three builds:

Python build	Unit tests	Native AOT consumer
CPython 3.13 (GIL, auto-discovered)	209 / 1 / 0	8 / 8 ✅
CPython 3.13.13t (free-threaded)	205 / 5 / 0	8 / 8 ✅
CPython 3.14.4t (free-threaded)	205 / 5 / 0	8 / 8 ✅

The full audit lives in docs/FREETHREADED-AUDIT.md. It's deliberately a public document — when I claim "verified", you can read what that means.

Caveats, honestly

A few things to be straight about:

DotNetPy is 0.6.0. Experimental, not production-stable yet. Lots of patterns are still being worked out.
The Python ML stack itself isn't fully free-threaded yet. Torch's FT support is in active migration. NumPy 2.1+ supports PEP 703. transformers and diffusers work, but their underlying C extensions are mixed. Until the upstream stack catches up, you'll get correctness under free-threaded Python from DotNetPy's interop layer but Python-side ML performance may still serialize through library locks.
Native AOT publishing requires the platform C toolchain. On Windows that means the Visual Studio C++ build tools; on Linux you need clang/lld. Same constraint as any AOT'd .NET app.
JSON marshalling is the data plane. Every result variable is serialized in Python and deserialized in .NET via System.Text.Json. This is a deliberate trade-off for Native AOT compatibility. For workloads where this dominates (very large result objects), batch results into a single capture call and only return the small structured summary.

Where to go from here

Code: https://github.com/rkttu/dotnetpy
NuGet: dotnet add package DotNetPy --version 0.6.0
Samples: samples/ml-embeddings, samples/ml-whisper, samples/ml-image-gen — each is a single dotnet run sample.cs
Free-threaded audit: docs/FREETHREADED-AUDIT.md
Comparison with pythonnet / CSnakes / IronPython: docs/COMPARISON.md with a decision tree

If you've been wondering "how do I run a current HuggingFace model from C#" — I hope this is a useful answer. Comments, issues, and PRs welcome.

What if calling Python from C# took 4 lines, supported Native AOT, and didn't need a project file? Just shipped DotNetPy v0.5.0 — a lightweight Python interop library built for modern .NET. 📦 https://github.com/rkttu/dotnetpy

Jung Hyun, Nam — Thu, 19 Mar 2026 07:01:43 +0000

GitHub - rkttu/dotnetpy: Lightweight and AOT-compatible Python Interop Library · GitHub

Lightweight and AOT-compatible Python Interop Library - rkttu/dotnetpy

github.com

Introducing DotNetPy: Python Interop for Modern .NET, Reimagined

Jung Hyun, Nam — Thu, 19 Mar 2026 06:58:50 +0000

The Problem

.NET and Python are two of the most widely used platforms in enterprise software today, yet bringing them together has always been painful. Existing interop libraries either require heavy runtime dependencies, lack support for modern .NET features like Native AOT, or demand complex project configurations with Source Generators.

If you've ever wanted to leverage Python's rich data science and ML ecosystem from a C# application — without leaving your .NET toolchain — you know the friction.

DotNetPy is my answer to that problem. Version 0.5.0 is now available on NuGet.

What Is DotNetPy?

DotNetPy (pronounced dot-net-pie) is a .NET library for executing Python code directly from C#. It wraps the Python C API behind a clean, minimal interface.

Here's the entire "hello world":

var executor = Python.GetInstance();
var temperatures = new[] { 23.5, 19.2, 31.8, 27.4, 22.1 };

using var result = executor.ExecuteAndCapture(@"
    import statistics
    result = {'mean': statistics.mean(data), 'stdev': statistics.stdev(data)}
", new Dictionary<string, object?> { { "data", temperatures } });

Console.WriteLine($"Mean: {result?.GetDouble("mean"):F1}°C ± {result?.GetDouble("stdev"):F1}");

No separate .py files. No Source Generators. No complex setup.

Why Another Interop Library?

There are established options — pythonnet, CSnakes, IronPython. Each has strengths, but none was designed for where .NET is heading in 2025 and beyond. DotNetPy fills that gap with three design pillars:

1. Native AOT Support

DotNetPy is the only .NET-Python interop library that works with PublishAot=true. If you're building self-contained, ahead-of-time compiled applications, DotNetPy won't hold you back. Neither pythonnet nor CSnakes support this today.

2. File-based App Ready (.NET 10+)

.NET 10 introduces file-based apps — run a single .cs file with dotnet run script.cs, no project file required. DotNetPy is designed to work in this scenario from day one.

# No csproj needed
dotnet run my-analysis.cs

This makes it ideal for scripting, prototyping, and lightweight automation tasks where spinning up a full project is overkill.

3. Declarative uv Integration

Managing Python environments from .NET has always been a manual, error-prone process. DotNetPy integrates with uv, the fast Python package manager, so you can declare your entire Python environment in C#:

using var project = PythonProject.CreateBuilder()
    .WithProjectName("my-analysis")
    .WithPythonVersion(">=3.10")
    .AddDependencies("numpy>=1.24.0", "pandas>=2.0.0")
    .Build();

await project.InitializeAsync();  // Downloads Python, creates venv, installs packages

var executor = project.GetExecutor();
executor.Execute("import numpy as np; print(np.mean([1,2,3]))");

One call to InitializeAsync() handles Python download, virtual environment creation, and dependency installation. No more "works on my machine" issues.

How Does It Compare?

Here's a quick comparison with the existing ecosystem:

	DotNetPy	pythonnet	CSnakes	IronPython
Native AOT	✅	❌	❌	❌
File-based Apps	✅	❌	❌	❌
Source Generator Required	No	No	Yes	No
uv Integration	Built-in	None	Supported	None
Bidirectional Calls	C#→Py	C#↔Py	C#→Py	C#↔Py
Learning Curve	Very Low	Medium	High	Low

DotNetPy deliberately focuses on the C# → Python direction. If you need Python calling back into .NET objects, pythonnet remains the right choice. DotNetPy is for scenarios where .NET is the host and Python is the tool.

Built-in Security

Executing dynamic code always carries risk. DotNetPy ships with a Roslyn analyzer that detects potential code injection at compile time.

// ❌ The analyzer flags this — user input as code
executor.Execute(userInput);

// ✅ Safe — user data passed as variables
executor.Execute(
    "result = sum(numbers)",
    new Dictionary<string, object?> { { "numbers", userNumbers } }
);

This is a deliberate design choice: make the safe path easy and the dangerous path visible.

Features at a Glance

Automatic Python Discovery — cross-platform detection of installed Python distributions
Data Marshaling — pass .NET arrays, dictionaries, and primitives to Python and back
Variable Management — capture, check, delete, and clear Python variables from C#
Free-threaded Python Support — detects Python 3.13+ builds with --disable-gil
Thread Safety — automatic GIL management for safe concurrent access

Getting Started

Install from NuGet:

dotnet add package DotNetPy --version 0.5.0

Initialize and run:

using DotNetPy;

Python.Initialize("/path/to/python313.dll");
var executor = Python.GetInstance();

var sum = executor.Evaluate("sum([1,2,3,4,5])")?.GetInt32();
Console.WriteLine(sum); // 15

Or let DotNetPy find Python automatically:

Python.Initialize(PythonDiscovery.FindPython());

Use Cases I'm Targeting

AI/ML integration: Call scikit-learn, PyTorch, or Hugging Face models from a .NET service
Data analysis scripts: Run pandas/numpy workloads inline without a separate Python service
Automation & scripting: Use .NET 10 file-based apps as Python-capable scripts
Legacy modernization: Gradually introduce Python capabilities into existing .NET applications

What's Next

The roadmap includes embeddable Python support on Windows (for simplified deployment) and specialized optimizations for AI and data science workflows. This is v0.5.0 — the API is stabilizing, and feedback at this stage is especially valuable.

Make Your WSL Environment Programmable

Jung Hyun, Nam — Sat, 10 Jul 2021 16:48:28 +0000

Original Post: https://cloudeveloper.net/make-your-wsl-environment-programmable-72be5907d1ff

I have been fascinated by WSL architecture since its debut. It has the elegant and beautiful architecture to achieve interoperability between Windows and the Linux world. Eventually, I dive into the WSL Win32 APIs, its registry model, and internal too.

But there was only slight WSL API documentation and sample. Later, I realized that Microsoft does not intend to use the WSL API for general purposes but WSL distribution developers. Also, the CLI tool known as WSL.exe and distro launchers is moving parts (because each Windows 10 major release has different features, command-line options). So these situations make it hard to automate the WSL environment.

After many trials and errors, I developed a small but effective automation tool called WSL SDK. This tool is an out-of-process style COM server so that you can access the SDK with any COM-supported language.
In this article, I want to share the walkthrough of the WSL SDK to overcome the difficulties of using official Win32 WSL APIs.

Hidden treasures of WSL

If you are interested in the WSL APIs, you might get a hint about the API. For example, the API WslLaunch returns an HRESULT code. A WSL-related COM interface is in its internal composition known as LxssUserSession, and this API wraps around the COM object.

Sadly, the internal COM object was completely hiding by Microsoft, and it looks like it was pretty intended. I guess that there are reasonable decisions about this direction. However, its internal COM object also does not have documentation well.

Unkindness of the WSL APIs

Because of that, there is well-known and by-design behavior about the WSL APIs. If you call the WSL APIs with P/Invoke via PowerShell, LINQPad, or any COM-enabled environment, you cannot reach any WSL APIs. Many enthusiasts tried the APIs, but there is no luck.
And why is that? Those environments I mentioned already initiated with another CoInitializeSecurity call. Sadly, WSL APIs require a particular initialization parameter. And the CoInitializeSecurity called in somewhere; you can never invoke the CoInitializeSecurity again. To overcome this issue, inevitably, I should choose the out-of-process model.
Excavating the old samples
However, the out-of-process model makes cumbersome steps to use the API. You should check the existence of the process. You will have to define how to communicate with the external process such as pipe, internal networking, or any marshaling protocols. Moreover, this approach makes it hard to extend and maintain the API calls and functionalities.

I stuck at this point so a long time. But, thanks to the old project named All-In-One Code Framework developed by Microsoft, I found a beautiful solution. Yes. The out-of-process COM server model! So I adapted the sample out-of-process COM server code, and it works like a charm.

Under the hood

When the client application requests a WSL SDK service object via COM API, Windows automatically launches the executable file to obtain an appropriate object reference. Then, wrap it with a proxy interface, and the client application retrieves that reference. For better understanding, please look at the below picture.

Justly, WSL SDK executable file invokes CoInitializeAPI with a correct parameter to communicate with WSL APIs. Then start a message pump to handle external RPC requests and any Windows GUI-related requests. When an object reference is requested, its reference count will increase or decrease. Then the count reaches zero; the executable process will shut down. And again, another request arrived, the same round will occur again until unregistering the COM information from the registry.

So the WSL SDK decouples the COM security model between the application's one and WSL's requirement. And process lifecycle management handled by the operating system's infrastructure and a reference count mechanism. Every WSL SDK client does not need to care about any details. They request a WSL SDK interface as usual, and all things are going well.

Comparing direct P/Invoke to the WSL API and using WSL SDK

I will show a simple demonstration.

A PowerShell sample code looks like below. I excerpted a sample code from the GitHub issue WSL API does not work in PowerShell · Issue #4058 · microsoft/WSL (github.com).

# Excerpted from https://github.com/microsoft/WSL/issues/4058
Write-Host 'Calling WslIsDistributionRegistered directrly (Ubuntu-20.04):'
Add-Type -TypeDefinition @'
using System.Runtime.InteropServices;
public class wslutil
{
 [DllImport("wslapi.dll", CharSet = CharSet.Unicode)]
 public static extern uint WslIsDistributionRegistered([In, MarshalAs(UnmanagedType.LPWStr)] string distributionName);
public static void Main(string[] args)
 {
  System.Console.WriteLine(WslIsDistributionRegistered("Ubuntu-20.04"));
 }
}
'@
[wslutil]::Main({})
Write-Host

The PowerShell code references a C-function from the WSLAPI.dll. Seemingly, it makes sense and should work well. But the code returns zero, does not reflect the current status.

However, WSL SDK returns the correct value.

Write-Host 'Calling WSL SDK API (Ubuntu-20.04):'
$DistroName = 'Ubuntu-20.04'
$obj = New-Object -ComObject 'WslSdk.WslService'
$Result = $obj.IsDistroRegistered($DistroName)
Write-Host "$Result"
Write-Host

Essentially, both code depends on the WslIsDistributionRegistered function, but the PowerShell already calls the CoInitializeSecurity which does not meet requirements for WSL APIs. The first example will not work because of that.

A Demo: Sandboxing WSL distro

I will show another, more complex example script. The below code will automatically download the Alpine Linux root filesystem image from the official mirror. Then, add the VI improved editor to the distribution.

$ErrorActionPreference = "Stop"
$obj = New-Object -ComObject 'WslSdk.WslService'
Write-Output 'A WslSdk.WslService object is created.'
Pause

# Get installed distro list
Write-Output 'Currently installed WSL distro list: '
$list = $obj.GetDistroList()
Write-Output $list
Pause

# Generate Random Name
$RandomName = $obj.GenerateRandomName($false)
Write-Output "We will use $RandomName as a new distro"

# Download Alpine Linux RootFS Image
Write-Output 'Downloading alpine linux root file system image'
$TargetUrl = 'https://dl-cdn.alpinelinux.org/alpine/v3.14/releases/x86_64/alpine-minirootfs-3.14.0-x86_64.tar.gz'
$RootfsFilePath = "$env:TEMP\alpine.tar.gz"
$InstallPath = "C:\Distro\$RandomName"
Invoke-WebRequest -UseBasicParsing -Uri $TargetUrl -OutFile $RootfsFilePath
Pause

# Register Distro
Write-Output "Distro installation begins"
Write-Output " - Distro Name: $RandomName"
Write-Output " - Source RootFS File Path: $RootfsFilePath"
Write-Output " - Destination Install Path: $InstallPath"
$obj.RegisterDistro($RandomName, $RootfsFilePath, $InstallPath)
Pause

# Distro Register Check
$Result = $obj.IsDistroRegistered($RandomName)
Write-Output "Distro Name $RandomName Installed: $Result"
Pause

# Metadata Query
Write-Output "Querying $RandomName metadata..."
$o = $obj.QueryDistroInfo($RandomName)
Write-Output " - Distro ID: $($o.DistroId())"
Write-Output " - Distro Name: $($o.DistroName())"
Write-Output " - Environment Variabls: $($o.DefaultEnvironmentVariables())"
Write-Output " - Default Uid: $($o.DefaultUid())"
Write-Output " - Flags: $($o.DistroFlags())"
Write-Output " - Win32 Interop Enabled: $($o.EnableInterop())"
Write-Output " - Drive Mounting Enabled: $($o.EnableDriveMounting())"
Write-Output " - NT Path Append Enabled: $($o.AppendNtPath())"
Write-Output " - WSL Version: $($o.WslVersion())"
Pause

# Run WSL command
Write-Output "Installing vim..."
$res = $obj.RunWslCommand($o.DistroName(), "apk add vim")
Write-Output $res
Pause

# Revealing launcher executable
Write-Output "Revealing launcher executable file"
Start-Process -FilePath "$env:windir\explorer.exe" -ArgumentList "/select,$InstallPath\$RandomName.exe"
Pause

# Unregister Distro
Write-Output "Unregister $RandomName distro..."
$obj.UnregisterDistro($RandomName)
Pause

# Get installed distro list
Write-Output 'Currently installed WSL distro list: '
$list = $obj.GetDistroList()
Write-Output $list
Pause
$obj = $null

It is a basic root filesystem image does not design for the WSL. As you already know, WSL supports importing any Linux root filesystem image which meets exact processor architecture.

WSL SDK handles dynamic distribution registration and manipulation even in the PowerShell environment.

Just for fun, even in Microsoft Excel, you can interact with the WSL environment.

You can access a variety of sample codes of WSL SDKs here: https://github.com/wslhub/wsl-sdk-com/tree/main/sample.

Future Roadmap

I recently created a GitHub action pipeline thanks to the https://github.com/marketplace/actions/setup-wsl plugin and the GitHub team's decision to enable the WSL component in their Windows Server workload.

Because of that, I could create a continuous integration for WSL SDK, which makes more confident releases. (https://github.com/wslhub/wsl-sdk-com/actions/workflows/wsl-sdk-com-build.yml) This work is a significant achievement of the WSL SDK roadmap.

Moreover, my bucket list contains those goals.

Registration-free COM server (If possible)
ARM64 native support
Adopting WSL SDK to another of my previous project (WSL Manager)
Various language wrappers (C#, Python, Go-lang, PowerShell, or any COM supported languages)

If you are interested in the WSL SDK project, please come and contribute to the GitHub repo. (https://github.com/wslhub/wsl-sdk-com)

Set up your SSH server in Windows 10 native way

Jung Hyun, Nam — Sun, 02 Feb 2020 07:50:53 +0000

Continuing from the last post, we'll look at how to set up a built-in SSH server starting with Windows 10 and Windows Server 1709. This method allows Windows Server to connect remotely using SSH, just like a traditional Linux server. We will also look at how you can use Remote Desktop securely without modifying your firewall settings using SSH port tunneling.

Installing and configuring OpenSSH Server

You can install OpenSSH Server the same way you installed the SSH client in the previous article.

$OpenSSHServer = Get-WindowsCapability -Online | ? Name -like ‘OpenSSH.Server*’
Add-WindowsCapability -Online -Name $OpenSSHServer.Name

After installing the OpenSSH server program, start and stop the NT service once to create the necessary initial configuration files.

$SSHDaemonSvc = Get-Service -Name ‘sshd’
Start-Service -Name $SSHDaemonSvc.Name
Stop-Service -Name $SSHDaemonSvc.Name

Apply asymmetric key authentication

It is highly recommended that you use the public key authentication method and disable the password authentication method to prevent attacks through password assignment. To enable this authentication feature, start PowerShell as an administrator and open the file in the path below with notepad. (Or you can use another text editor of your choice.)

notepad.exe $env:PROGRAMDATA\ssh\sshd_config

For the following items, uncomment the items and apply the value as follows:

PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no

Then choose your preferred method of managing SSH public keys. Starting with Windows Server 2019 (or 1809), there are two ways to describe SSH public keys. One of which is the traditional way of creating an authorized_keys file in the user's home directory.

Using `$HOME\.ssh\authorized_keys`

To use this method, comment out the following block of code at the bottom of the configuration file:

Match Group administrators
  AuthorizedKeysFile
  __PROGRAMDATA__/ssh/administrators_authorized_keys

Then go to the user home directory you want to log in to and create a .ssh directory.

mkdir "$HOME\.ssh"

Create an authorized_keys file (without the extension) inside the newly created directory and open it with your favorite text editor.

$authorizedKeyFilePath = “$HOME\.ssh\authorized_keys”
New-Item $authorizedKeyFilePath
notepad.exe $authorizedKeyFilePath

Add the SSH public key value you are using here.

When you save the file, you must change the file permission settings as described in the section Setting File Permissions with Authentication Key Information. If this setting is missing, the SSH connection will fail.

Using `administrators_authorized_keys`

This is the default used by OpenSSH included in Windows Server 2019 (1809). Instead of registering a new SSH key for each user, you can manage your files in one place.

If you use this method, all public keys must be stored in the $env: PROGRAMDATA\ssh\administrators_authorized_keys file, except for non-administrative users (that is, users who do not belong to the Administrators group). If you try, this setting will be used instead of the setting in your home directory, so if there is no key here, the connection will fail.

The administrators_authorized_keys file does not exist by default and must be created.

$authorizedKeyFilePath = “$env:ProgramData\ssh\administrators_authorized_keys”
New-Item $authorizedKeyFilePath
notepad.exe $authorizedKeyFilePath

Add the SSH public key value you are using here.

Setting File Permissions with Authentication Key Information

A common and very tough problem that you will face about using the OpenSSH server for Windows is this. SSH key file permission should have correct and limited file permission. Windows version of SSH also follows this rule, but especially in Windows, configuring file permission can be unintuitive.

Depending on the method you chose in the previous step, you must verify the path of the authorized_keys file or administrators_authorized_keys file and change the permissions so that only the system account can access it using the icacls.exe utility and the Get-Acl and Set-Acl commands.

$authorizedKeyFilePath = "..."
icacls.exe $authorizedKeyFilePath /remove “NT AUTHORITY\Authenticated Users”
icacls.exe $authorizedKeyFilePath /inheritance:r
Get-Acl “$env:ProgramData\ssh\ssh_host_dsa_key” | Set-Acl $authorizedKeyFilePath

Changing the SSH Default Shell

Basically, for compatibility reasons, the Windows operating system has provided a shell-based interpreter that recognizes DOS commands for a long time. But now, with more and more features than working with DOS commands, PowerShell is becoming a good alternative.

If necessary, you can specify that PowerShell as the default shell for SSH instead of the DOS interpreter. However, the settings here are specific to SSH sessions, not for the Remote Desktop or console session.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "$env:WINDIR\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

Staring SSH Server

You are now ready to start your SSH server. The SSH server is set to manual run by default, so you can change the startup mode to automatic. Then starts the service.

$SSHDaemonSvc = Get-Service -Name ‘sshd’
Set-Service -Name $SSHDaemonSvc.Name -StartupType Automatic
Start-Service -Name $SSHDaemonSvc.Name

Congratulations! From now, you can connect to Windows with SSH-key authentication.

How to Secure Remote Desktops

Unlike Linux, Windows still runs much of the system on a graphical interface rather than on the command line. So if you try to do something with a terminal like this, you may not have much to do as you might expect.

Remote Desktop, however, is a well-known food for many hackers and script kiddies as is well known. You may encounter the dilemma of choosing between convenience and security.

Fortunately, SSH provides the concept of tunneling, supporting the ability to relay other network connections securely. Remote desktop connections can also be protected in this way so that you can use them with confidence.

Start by blocking the TCP 3389 and UDP 3389 ports. You can do this because you will use Remote Desktop only with SSH tunneling.

Set-NetFirewallRule -DisplayName “Remote Desktop - User Mode (UDP-In)” -Action Block
Set-NetFirewallRule -DisplayName “Remote Desktop - User Mode (TCP-In)” -Action Block

Next, you must change the registry flag value so that the remote desktop server can accept the connection.

Set-ItemProperty -Path ‘HKLM:\System\CurrentControlSet\Control\Terminal Server’ -Name “fDenyTSConnections” -Value 0

And the part that I'm going to explain right now is really cool. As you saw earlier, you will not be asked for your password when you connect to OpenSSH, so you can set up a randomly generated strong password each time you use it. It is very useful to have a simple script in the system directory that can do this.

Choose the type of script you want to create a file called ChangePassword.ps1. We will keep the file in the system directory for your convenience.

$change_pwd_script_path = “$env:WINDIR\ChangePassword.ps1”
Clear-Content $change_pwd_script_path -ErrorAction SilentlyContinue
notepad.exe $change_pwd_script_path

Set your desired password

Create the contents of the ChangePassword.ps1 file as follows:

$Password = Read-Host -Prompt "Provide your new account password" -AsSecureString
Set-LocalUser -Name "$env:UserName" -Password $Password
Clear-Variable "Password"
Write-Host "Detailed settings such as remote desktop settings, WinRM connection settings, and Windows Update can be controlled using the sconfig command."

This script allows you to enter your own password. However, unless you change the policy, you can only use passwords that pass the Windows Server enhanced default password rules. You must specify a password that must meet all of the following conditions.

English capital letters (A through Z)
Lowercase English letters (a through z)
Arabic numerals (0 to 9)
Special symbols (e.g.!, $, #,%)

Generate a new password every time

Create the contents of the ChangePassword.ps1 file as follows:

Add-Type -AssemblyName System.Web
$Password = [System.Web.Security.Membership]::GeneratePassword(30,10)
Set-LocalUser -Name "$env:UserName" -Password ($Password | ConvertTo-SecureString -AsPlainText -Force)
Write-Host "Your New Password is:`r`n`r`n$Password"
Write-Host "Detailed settings such as remote desktop settings, WinRM connection settings, and Windows Update can be controlled using the sconfig command."
Clear-Variable "Password"

This way, you can set a strong password every time. If you forget your password, you can rest assured that you can still use public key authentication as a secondary means of authentication.

Try logging in to Remote Desktop with Tunneling

Now enter the following command to run the above script. After that, just set the password as guided by the script and verify that the remote desktop connection is working.

ChangePassword

To log in to the remote desktop, run SSH as follows:

ssh -L 3389:localhost:3389 <user_id>@<host_address>

The first 3389 is the port number on the server-side, and the second address is the port number you want to use locally. If you have changed the remote desktop's port number from the registry to another number on the server, you can enter the changed port number instead of 3389.

If you try to connect to a remote desktop using only the <host_address> part as before, the firewall will block the connection as previously set up. So no one can access the remote desktop directly unless the user has registered a public key that matches the SSH secret key with that server.

Using secure file sending and receiving

Not surprisingly, it is possible to use SSH based SFTP. This feature can be used to securely handle large file transfers in place of the remote desktop's folder sharing feature.

Any client that supports the SFTP feature, such as FileZilla, is compatible and has an management advantage, as there is no need to apply complex firewall open policies like traditional FTP.

Wrap-up

This walks you through all the new SSH features that have been added since Windows 10 1709. Both articles are available for Windows 10 and Windows Server 2019, so please take a moment to set them up for even more security.

Credits

The following articles helped me as I wrote this article.

Set up SSH Key and Git integration in Windows 10 native way

Jung Hyun, Nam — Sun, 27 Oct 2019 16:18:36 +0000

Did you know that Windows 10 comes with an OpenSSH client?

Starting with the Windows 10 Fall Creators Update (1709), OpenSSH clients are starting to be offered as Windows add-ons. However, it is easy to misunderstand that it is only provided by unfamiliar usage that differs from Linux, or that it is still not supported properly.

In this article, we'll look at how to set up the commonly used client environment related to OpenSSH in a way that's built into Windows 10, and I'll give you some useful tips.

Configure Windows OpenSSH

Start PowerShell as an administrator and use the PowerShell commands below to add Windows components.

Microsoft's current installation of OpenSSH is an add-on package, Feature-On-Demand, not an item in the Add/Remove Windows Components dialog box of the classic Control Panel control.exe. You can install it only by using the following command:

$OpenSSHClient = Get-WindowsCapability -Online | ? Name -like ‘OpenSSH.Client*’
Add-WindowsCapability -Online -Name $OpenSSHClient.Name

Normally, no system restart is required after installation.

After completing the installation, you may enable the ssh-agent service. This service is used to register to not ask for the SSH key password every time. Initially, the service is disabled and stopped, so set the service to autostart, and start it now.

$SSHAgentSvc = Get-Service -Name ‘ssh-agent’
Set-Service -Name $SSHAgentSvc.Name -StartupType Automatic
Start-Service -Name $SSHAgentSvc.Name

You should now close the PowerShell window in administrator mode and work with the PowerShell window open as normal.

Since we are setting up a new system, let's create a new SSH key. Some common utilities have been added along with the OpenSSH client package. Run the ssh-keygen command and answer questions.

ssh-keygen

This creates a key pair from the $HOME\.ssh\id_rsa file and the $HOME\.ssh\id_rsa.pub file.

Now run the ssh-add command to add this key pair to the ssh-agent service.

ssh-add

It automatically registers the $HOME\.ssh\id_rsa key pair, and now you can authenticate with that key pair.

Note: Sometimes the system may have several ssh.exe binary files installed, and the ssh.exe binary path may be duplicated in the PATH environment variable. To debug this problem, review the contents of the path command or the PATH environment variable and change the folder path containing the ssh.exe binary to be used first, or keep only one.

Registering SSH Keys on Github

You need to register the public key of this SSH Key Pair to Github or your Git repository.

Enter the following PowerShell command to copy the public key value to register other systems.

Get-Content -Path $HOME\.ssh\id_rsa.pub | Set-Clipboard

With this command, public key automatically entered on the clipboard.

Then enter the following command to open the GitHub configuration page. (Or you can open the URL below directly in your preferred browser instead of your default browser.)

Start-Process ‘https://github.com/settings/ssh/new'

After that, paste the public key from the clipboard and register it by adding a clear description of the key.

Install Git Client and SSH Client

There are many ways to install the Git client, but I personally recommend the Chocolatey Package Manager as the most intuitive and easy way.

The official Git client installation package exposes a lot of options that can cause side effects, so if you install it incorrectly, you may run into difficulties due to unintended features.

First install the Chocolatey Package Manager if it is not already installed. Because this is a system-level addition of software, allow a few minutes to open a new PowerShell window as an administrator.

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1'))

Now enter the command to install the Git for Windows client.

Note: Often, if the Chocolatey.org website enters a regular checkout period, the installation may not proceed properly. In this case, check the Chocolatey.org website and try again later.

choco install git -y

Back in the regular PowerShell window, set the GIT_SSH environment variable. You must specify this environment variable so that Git clients can properly recognize SSH clients on Windows 10.

$SSHPath = (Get-Command -Name ‘ssh.exe’).Source
[Environment]::SetEnvironmentVariable(‘GIT_SSH’, $SSHPath, ‘User’)

Powering PowerShell with oh-my-posh

As the name suggests, oh-my-posh is a Windows PowerShell version of oh-my-zsh that is popular on macOS and Linux these days. And interestingly, it supports some of the features that oh-my-zsh provides.

First run the oh-my-posh installation script. It's listed in PowerShell's official module repository, so the commands are not complicated and simple.

Install-Module posh-git -Scope CurrentUser
Install-Module oh-my-posh -Scope CurrentUser

If you are installing on PowerShell Core for Windows (6.x or later), run the following command.

Install-Module -Name PSReadLine -AllowPrerelease -Scope CurrentUser -Force -SkipPublisherCheck

Edit the Profile file so that the oh-my-posh shell can be loaded automatically when PowerShell starts. If you run the command below, the file will be created if it doesn't exist.

if (!(Test-Path -Path $PROFILE )) { New-Item -Type File -Path $PROFILE -Force }
notepad.exe $PROFILE

Add the following code at the end of the script and save it.

Import-Module posh-git
Import-Module oh-my-posh
Set-Theme Paradox

As a side note, using a programming font with powerline patching looks pretty and doesn't break the glyphs. Run the following command again in an elevated PowerShell and update the console window's font settings with the D2Coding font.

Import-Module posh-git
Import-Module oh-my-posh
Set-Theme Paradox

When all the settings are applied, you will see something like the picture below, and if you move the directory to the Git repository, you will see the branch name look good. This seems to have some assortment. 😎

Appendix 1: Installing the Windows Terminal App

You can go directly to the Windows Terminal app store page by running the following command in PowerShell: As is well known, using Windows Terminal gives you all the benefits of a modern CLI development environment.

Start-Process 'https://www.microsoft.com/store/productId/9N0DX20HK701'

After installation, you can launch the Windows Terminal app directly with wt.exe or wt shortcuts. That is, on Windows, remember wt instead of cmd, powershell or pwsh. 😏

Appendix 2: For SourceTree

Unfortunately, the Git client used by SourceTree does not work with the SSH Agent service provided by Windows. Instead, you can use the keys you created.
In the SourceTree Options window, change the SSH client to OpenSSH as shown below.

At this point, verify that the SSH key is the same as the $HOME\.ssh\id_rsa file created in the previous step. If it is different, specify it again.
When done, press the OK button to save the settings.

Appendix 3: Integrating with Visual Studio Code

If GIT_SSH environment variable is registered properly, integration is completed without any special setting. However, even though you have completed the configuration, if you are still in progress without any message when performing git pull, you can run the ssh-add -l command built-in terminal to check the connection status with the ssh-agent service.