[Devlog - InDepth] Context Packer: Architecture, Privacy Engines & Engine Trade-offs

This technical overview breaks down the multi-host architecture, dynamic module resolution, non-blocking stream I/O, privacy scrubbing, and output transformations behind Context Packer.

---

1. Why Standard Copy-Pasting Fails

When feeding code context to Large Language Models (Claude, ChatGPT, Gemini) or building knowledge graphs in Obsidian, manual copy-pasting breaks down quickly:

1. File Locks & IDE Crashes: Game engines (Unreal Engine 5), Roslyn analyzers, and MSBuild hold active write handles during background compilation or hot-reloading. Standard high-level APIs like File.ReadAllText() request restrictive sharing permissions, throwing IOException: The process cannot access the file because it is being used by another process.
2. Workstation Privacy Leaks: Absolute local paths (C:\Users\Emil\Source\Repos\...) leak private usernames and machine directory structures into public AI model prompts.
3. Context Drift: AI models lose track of module boundaries and layer hierarchies when source code is provided as disjointed snippets without structural map headers.

---

2. Real-World Benchmark: The UE5 Test

To test the Packer.Core pipeline under extreme conditions, the engine was benchmarked against a full production Unreal Engine 5 repository while the UE5 Editor was actively running and compiling in the background:

Metric	Target Repository Scope
Total Disk Footprint	13.9 GB (14,932,377,600 bytes)
Total Workspace Entities	30,277 files, 1,740 directories
Source Modules Filtered	5 sub-folders / plugins (374 source files, ~60.7 MB)
Environment State	Unreal Engine 5 Editor Open & Active
Execution Speed	0.42 seconds

Despite scanning over 30,000 files, filtering build artifacts (Intermediate, Saved, bin, obj), redacting local usernames, and handling heavy third-party headers (emitting size warnings for large SDL3 files), the engine completed the entire packing job in 420 milliseconds without a single lock exception in the VSIX tool.

---

3. Multi-Host Ecosystem & Architecture

To support different developer workflows without requiring a single monolithic tool, the core packaging logic is split across three distinct tiers that share the same configuration contracts:

Architectural Rationale & Design Choices

• Decoupled Engine (Packer.Core): Multi-targeted for .NET 8.0 and .NET Standard 2.0. Decoupling IPackerConfig and IPackerEngine from UI frameworks ensures that core logic remains headless, lightweight, and fully testable via xUnit.

• Visual Studio 2022 Extension (Packer.VSIX): Integrated into Visual Studio as a dockable tool window (.NET Framework 4.7.2). Uses COM APIs (EnvDTE) to read staged files directly from Solution Explorer tree nodes.

• WinUI 3 Standalone App (Packer.WinUI3): High-performance desktop client running unpackaged. Ideal for quick, visual file batching and context creation outside the IDE without needing Visual Studio running.

• PowerShell CLI (ks pack): Zero-dependency terminal script that mirrors the engine contract natively in pure PowerShell. Perfect for CI/CD automation pipelines, automated drive staging, or quick terminal workflows.

---

4. Zero-Dependency PowerShell CLI (ks pack)

For terminal environments where compiling C# binaries is unnecessary, the entire packing pipeline—including directory traversal, username scrubbing, module anchor parsing, and cache management—is implemented in pure PowerShell:

param(
    [string]$OutputName = 'Kadmium_Context',
    [string]$Mission = '',
    [switch]$Cache,
    [switch]$Obsidian
)

[Console]::OutputEncoding = [System.Text.Encoding]::UTF8

$cacheDir = [System.IO.Path]::Combine($env:LOCALAPPDATA, 'KadmiumCache')
if (-not (Test-Path $cacheDir)) { New-Item -ItemType Directory -Path $cacheDir -Force | Out-Null }

# --- 1. Cache Browser Mode ('ks pack -Cache') ---
if ($Cache) {
    Write-Host '===========================================================' -ForegroundColor Cyan
    Write-Host ' Kadmium Context Packer - Cache Browser' -ForegroundColor Cyan
    Write-Host '===========================================================' -ForegroundColor Cyan

    $cachedFiles = Get-ChildItem -Path $cacheDir -Filter '*.txt' | Sort-Object LastWriteTime -Descending

    if ($cachedFiles.Count -eq 0) {
        Write-Host ' [!] No cached packages found in:' -ForegroundColor Yellow
        Write-Host "     $cacheDir`n" -ForegroundColor DarkGray
        Write-Host '  [O] Open Cache Folder | [Q] Exit' -ForegroundColor Yellow
        $emptyChoice = Read-Host ' Select option'
        if ($emptyChoice -eq 'o') { Start-Process 'explorer.exe' $cacheDir }
        exit
    }

    Write-Host " Found $($cachedFiles.Count) cached package(s):`n" -ForegroundColor DarkCyan
    for ($i = 0; $i -lt$cachedFiles.Count; $i++) {$num = $i + 1$size = [math]::Round($cachedFiles[$i].Length / 1KB, 1)
        Write-Host "  [$num] " -NoNewline -ForegroundColor Green
        Write-Host "$($cachedFiles[$i].Name)" -NoNewline -ForegroundColor White
        Write-Host " (${size} KB -$($cachedFiles[$i].LastWriteTime.ToString('yyyy-MM-dd HH:mm')))" -ForegroundColor DarkGray
    }

    Write-Host "`n  [O] Open Cache Folder" -ForegroundColor Cyan
    Write-Host '  [C] Clear Cache | [Q] Exit' -ForegroundColor Yellow
    Write-Host '-----------------------------------------------------------' -ForegroundColor DarkCyan

    $selection = Read-Host ' Select package (number to copy, O to open folder, C to clear, Q to exit)'

    if ($selection -eq 'q' -or [string]::IsNullOrWhiteSpace($selection)) { exit }

    if ($selection -eq 'o') {
        Start-Process 'explorer.exe' $cacheDir
        Write-Host ' [OPENED] Cache folder opened in Explorer.' -ForegroundColor Green
        exit
    }

    if ($selection -eq 'c') {
        Remove-Item "$cacheDir\*.txt" -Force -ErrorAction SilentlyContinue
        Write-Host ' [CLEANED] Cache cleared!' -ForegroundColor Red
        exit
    }

    if ($selection -match '^\d+$' -and [int]$selection -le $cachedFiles.Count) {
        $selectedFile = $cachedFiles[[int]$selection - 1]

        Write-Host "`n Selected: $($selectedFile.Name)" -ForegroundColor White
        $action = Read-Host ' [C] Copy text to Clipboard | [O] Open & Highlight File in Explorer'

        if ($action -eq 'o') {
            Start-Process 'explorer.exe' "/select,`"$($selectedFile.FullName)`""
            Write-Host " [HIGHLIGHTED] File selected in Explorer! Just drag and drop it into chat." -ForegroundColor Green
        } else {
            Get-Content $selectedFile.FullName -Raw | Set-Clipboard
            Write-Host " [COPIED] '$($selectedFile.Name)' copied to clipboard!" -ForegroundColor Green
        }
        exit
    } else {
        Write-Host ' Invalid choice.' -ForegroundColor Red
        exit
    }
}

# --- 2. Interactive Menu Mode ---
if ([string]::IsNullOrWhiteSpace($Mission) -and$args.Count -eq 0) {
    Write-Host '===========================================================' -ForegroundColor Cyan
    Write-Host ' Kadmium Context Packer (Packer.Core CLI)' -ForegroundColor Cyan
    Write-Host '===========================================================' -ForegroundColor Cyan
    Write-Host '  [1] Pack Current Repository' -ForegroundColor Green
    Write-Host '  [2] Load Cache (Browse / Drag & Drop files)' -ForegroundColor Green
    Write-Host '  [3] Open Cache Folder Directly' -ForegroundColor Cyan
    Write-Host '  [4] Clear All Cache' -ForegroundColor Red
    Write-Host '-----------------------------------------------------------' -ForegroundColor DarkCyan

    $choice = Read-Host ' Select option (1, 2, 3, or 4)'

    if ($choice -eq '2') {
        & $MyInvocation.MyCommand.Path -Cache
        exit
    }
    if ($choice -eq '3') {
        Start-Process 'explorer.exe' $cacheDir
        Write-Host ' [OPENED] Cache folder opened in Explorer.' -ForegroundColor Green
        exit
    }
    if ($choice -eq '4') {
        Remove-Item "$cacheDir\*.txt" -Force -ErrorAction SilentlyContinue
        Write-Host ' [CLEANED] Cache cleared!' -ForegroundColor Red
        exit
    }

    $Mission = Read-Host ' Enter optional Mission Context for LLM (press Enter to skip)'
}

# --- 3. Packing Pipeline Execution ---
$currentDir = (Get-Location).Path
$allowedExtensions = @('.cs', '.cpp', '.h', '.json', '.xaml', '.txt', '.md', '.uplugin', '.uproject')
$ignoredFolders = @('bin', 'obj', '.vs', '.git', 'build', 'out', 'node_modules', 'Intermediate', 'Saved')
$anchors = @('Plugins', 'repos', 'Source', 'Projects')
$userName =$env:USERNAME

function Get-KadmiumModuleName {
    param([string]$FilePath)
    $parts =$FilePath.Split([System.IO.Path]::DirectorySeparatorChar)
    foreach ($anchor in $anchors) {$idx = [array]::IndexOf($parts,$anchor)
        if ($idx -ge 0) {
            if ($anchor -eq 'Source' -and$idx -gt 0) { return $parts[$idx - 1] }
            if ($parts.Length -gt ($idx + 1)) { return $parts[$idx + 1] }
        }
    }
    if ($parts.Length -gt 1) { return $parts[$parts.Length - 2] }
    return 'UnknownProject'
}

Write-Host "`nScanning repository: $currentDir..." -ForegroundColor Cyan

$files = Get-ChildItem -Path $currentDir -Recurse -File | Where-Object {
    $ext = $_.Extension.ToLower()
    $pathParts = $_.FullName.Split([System.IO.Path]::DirectorySeparatorChar)
    ($allowedExtensions -contains $ext) -and -not ($pathParts | Where-Object { $ignoredFolders -contains $_ })
}

if ($files.Count -eq 0) {
    Write-Host 'No matching source files found to pack.' -ForegroundColor Yellow
    exit
}

$lines = [System.Collections.Generic.List[string]]::new()
$lines.Add('# Project Context')
$lines.Add('Generated: ' + (Get-Date).ToString('yyyy-MM-dd HH:mm'))

if (-not [string]::IsNullOrWhiteSpace($Mission)) {
    $lines.Add('')
    $lines.Add('## Mission Context')
    $lines.Add($Mission)
}

$groupedFiles = $files | Group-Object { Get-KadmiumModuleName $_.FullName }

$lines.Add('')
$lines.Add('## Directory Structure')
foreach ($group in $groupedFiles) {
    $lines.Add('### Module: ' + $group.Name)
    foreach ($file in $group.Group) {
        $displayPath = $file.FullName.Replace($currentDir, '').TrimStart('\', '/')
        if ($displayPath.Contains($userName)) { $displayPath = $displayPath.Replace($userName, '[REDACTED]') }

        if ($Obsidian) {
            $lines.Add('- [[#File: ' + $file.Name + '|' + $displayPath + ']]')
        } else {
            $lines.Add('- `' + $displayPath + '`')
        }
    }
}

$lines.Add('')
$lines.Add('## Source Code')

$currentModule = ''
foreach ($file in $files) {
    $module = Get-KadmiumModuleName $file.FullName
    if ($module -ne $currentModule) {
        $lines.Add('')
        $lines.Add('# --- MODULE: ' + $module.ToUpper() + ' ---')
        $lines.Add('')
        $currentModule = $module
    }

    $displayPath = $file.FullName.Replace($currentDir, '').TrimStart('\', '/')
    $ext = $file.Extension.TrimStart('.').ToLower()
    $lang = switch ($ext) {
        'cs' { 'csharp' }
        'cpp' { 'cpp' }
        'h' { 'cpp' }
        'xaml' { 'xml' }
        'xml' { 'xml' }
        'json' { 'json' }
        default { '' }
    }

    $lines.Add('### File: ' + $file.Name)
    $lines.Add('```

' + $lang)
    try {
        $content = [System.IO.File]::ReadAllText($file.FullName)
        $lines.Add($content)
    } catch {
        $lines.Add('// ERROR: Could not read file.')
    }
    $lines.Add('

```')
    $lines.Add('')
}

$finalText = $lines -join [Environment]::NewLine

# Save to local cache directory
$fileName = $OutputName + '.txt'
$outputPath = [System.IO.Path]::Combine($cacheDir, $fileName)
[System.IO.File]::WriteAllText($outputPath, $finalText)
Set-Clipboard -Value $finalText

$fileCount = $files.Count
Write-Host ('[SUCCESS] Packed ' + $fileCount + ' files across ' + $groupedFiles.Count + ' module(s)!') -ForegroundColor Green
Write-Host ('[COPIED] Context copied to clipboard & saved in cache: ' + $outputPath) -ForegroundColor DarkGray

$openNow = Read-Host ' Open cache folder in Explorer now to drag file? (y/N)'
if ($openNow -eq 'y') {
    Start-Process 'explorer.exe' "/select,`"$outputPath`""
}
Write-Host ''

---

5. Stream I/O & Non-Blocking Read Architecture

To achieve zero-lock file ingestion during active compilation sessions, PackerEngine avoids high-level methods like File.ReadAllText(). Instead, it uses explicit stream sharing and isolated per-file exception handlers:

string content = string.Empty;
try
{
    content = await Task.Run(() =>
    {
        // FileShare.ReadWrite permits concurrent reads while IDEs/Compilers hold write locks
        using (var fs = new FileStream(file.FilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
        using (var sr = new StreamReader(fs))
        {
            return sr.ReadToEnd();
        }
    });

    if (content.Length > config.LargeFileWarningThreshold)
    {
        warnings.Add($"File '{file.FileName}' is very large ({content.Length} chars). This might consume a lot of tokens!");
    }
}
catch (Exception ex)
{
    content = $"// ERROR: File read failed. Locked by another process.\n// Exception: {ex.Message}";
    warnings.Add($"Could not read '{file.FileName}': It might be locked by another program.");
}

Key Stream I/O Rationale

• Explicit FileShare.ReadWrite Flags: Opening the FileStream with FileAccess.Read combined with FileShare.ReadWrite allows the engine to read the file's current byte stream on disk even if an IDE process or compiler thread is concurrently modifying or write-locking it. This builds the document package without blocking any active build threads.
• Cross-Framework Stream Consistency: Because Packer.Core multi-targets .NET 8.0 and .NET Standard 2.0, relying on explicit FileStream + StreamReader constructs guarantees uniform non-blocking behavior across modern WinUI 3 environments and legacy Visual Studio VSIX (.NET Framework 4.7.2) hosts.
• Fault-Tolerant Batch Isolation: In a staging queue containing dozens or hundreds of files, a single permission error or exclusive lock never aborts the entire packing job. Instead of crashing the pipeline, PackerEngine catches exceptions per file, injects an inline error comment (// ERROR: File read failed...) into the generated code block, and appends a message to the warning list in the returned tuple (generatedFiles, warnings). The developer still gets 99% of their staged workspace context along with clear feedback on what was skipped.

---

6. Native OS Interop: WinUI 3 Cross-Process Drag-Out

A standout technical capability of the WinUI 3 desktop client is dragging generated .txt context packages directly out of the application window into web browser DOM inputs (Claude, ChatGPT, or local LLM web UI uploads).

This requires converting internal data models into native Windows OS OLE storage items using StorageFile:

private async void GeneratedFilesList_DragItemsStarting(object sender, DragItemsStartingEventArgs e)
{
    if (e.Items.FirstOrDefault() is PackerModel fileModel && File.Exists(fileModel.FullPath))
    {
        // Wrap the cached text file path in a native Windows StorageFile handle
        StorageFile file = await StorageFile.GetFileFromPathAsync(fileModel.FullPath);

        // Populate OS DataPackage so Chromium/DOM browser drag handlers accept the payload
        e.Data.SetStorageItems(new[] { file });
        e.Data.RequestedOperation = DataPackageOperation.Copy;
    }
}

---

7. Dynamic Module Resolution Algorithm (GetModuleName)

Instead of requiring project manifests or AST parsers, PackerEngine dynamically categorizes files into module namespaces using directory anchors (Plugins, repos, Source, Projects):

public string GetModuleName(string path, string anchorsSetting)
{
    var parts = path.Split(Path.DirectorySeparatorChar);
    var anchors = anchorsSetting.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)
                                .Select(a => a.Trim()).ToList();

    foreach (var anchor in anchors)
    {
        int idx = Array.IndexOf(parts, anchor);
        if (idx != -1)
        {
            // Unreal / C++ convention: 'Source' anchor uses parent directory as module name
            if (anchor.Equals("Source", StringComparison.OrdinalIgnoreCase) && idx > 0)
                return parts[idx - 1];

            // Standard .NET / Git convention: uses directory immediately following anchor
            if (parts.Length > idx + 1)
                return parts[idx + 1];
        }
    }

    // Fallback to parent directory if no anchor matches
    return parts.Length > 1 ? parts[parts.Length - 2] : "UnknownProject";
}

---

8. Configuration System Reference (IPackerConfig)

All host implementations share the IPackerConfig interface contract for exclusions, chunking limits, and file discovery:

public interface IPackerConfig
{
    string AllowedExtensions { get; }
    string IgnoredFolders { get; }
    string ModuleFolderAnchors { get; }
    bool RedactPrivateInformation { get; }
    string CacheLocation { get; }
    int LargeFileWarningThreshold { get; }
    int MaxCharsPerFile { get; }

    string GetActualCachePath();
}

Setting Property	Default Value	Technical Purpose
AllowedExtensions	.cs,.xaml,.xml,.json,.h,.cpp,...	Whitelist filter during recursive folder traversal.
IgnoredFolders	bin,obj,.git,.vs,node_modules,...	Skipping build artifacts and hidden directories.
ModuleFolderAnchors	Plugins,repos,Source,Projects	Path anchors for dynamic module resolution.
RedactPrivateInformation	true	Replaces workstation username with [REDACTED].
LargeFileWarningThreshold	50000	Emits token-consumption warning for large individual files.
MaxCharsPerFile	3000000	Multi-part document chunking limit (_Part1.txt, _Part2.txt).
CacheLocation	%LocalAppData%\KadmiumCache	Staging folder for generated markdown context packages.

---

9. Workstation Privacy & Path Redaction

To prevent sensitive workstation layouts or personal account paths (C:\Users\johndoe\repos\...) from leaking into public LLM prompts, PackerEngine implements path sanitization before writing to disk:

string userName = Environment.UserName;

if (config.RedactPrivateInformation && displayPath.Contains(userName))
{
    displayPath = displayPath.Replace(userName, "[REDACTED]");
}

This transformation scrubs file headers, module tree maps, and inline code annotations before exporting context documents.

---

10. End-to-End Output Transformation

Here is how staged workspace files, settings, and prompt injections are formatted into output packages.

Input Settings & Staged Items

• Staged File 1: C:\Users\Emil\Source\Repos\Aether\Source\Private\PlayerMovementComponent.cpp
• Staged File 2: C:\Users\Emil\Source\Repos\PackerTool\Packer.Core\Services\PackerEngine.cs
• Active Settings: RedactPrivateInformation = true, MissionContext = "Refactor movement component to be thread-safe."

Mode A: Standard LLM Tree Output (.txt)

# Project(s) Context
Generated: 2026-09-08 16:00

## Mission Context
Refactor movement component to be thread-safe.

## Directory Structure
### Module: Aether
- `C:\Users\[REDACTED]\Source\Repos\Aether\Source\Private\PlayerMovementComponent.cpp`
### Module: PackerTool
- `C:\Users\[REDACTED]\Source\Repos\PackerTool\Packer.Core\Services\PackerEngine.cs`

## Source Code


# --- MODULE: AETHER ---

### File: PlayerMovementComponent.cpp
```cpp
#include "PlayerMovementComponent.h"

void UPlayerMovementComponent::TickComponent(float DeltaTime) {
    Super::TickComponent(DeltaTime);
}
```


# --- MODULE: PACKERTOOL ---

### File: PackerEngine.cs
```csharp
namespace Packer.Core.Services {
    public class PackerEngine {
        // Core packing pipeline
    }
}
```

Mode B: Obsidian Note Tree Output

When isObsidianFormat = true is selected, directory maps transform into bi-directional wiki-links pointing to in-document header anchors:

## Directory Structure

### Module: Aether
- [[#File: PlayerMovementComponent.cpp|C:\Users\[REDACTED]\Source\Repos\Aether\Source\Private\PlayerMovementComponent.cpp]]
### Module: PackerTool
- [[#File: PackerEngine.cs|C:\Users\[REDACTED]\Source\Repos\PackerTool\Packer.Core\Services\PackerEngine.cs]]

---

11. Key Technical Strengths & Limitations

Key Technical Strengths:

• Ultra-Fast Execution: Scans 30,000+ files and packs 300+ source items in 0.42 seconds.
• Zero-Lock File Ingestion: Non-blocking FileShare.ReadWrite streams handle active compilation locks gracefully.
• AOT & Trim Safe: WinUI 3 utilizes JsonSerializerContext source generators for reflection-free startup times.
• Native OS Interop: Supports dragging generated files directly into browser chats via Windows OLE.
• Automatic Safeguards: Multi-part document chunking (> 3,000,000 chars) protects tokenizer context windows from out-of-memory errors.

Current Limitations & Trade-offs:

• Character-Based Limits: Document chunking and file warnings use raw character length rather than Byte Pair Encoding (BPE) tokenizers (e.g. tiktoken).
• Full-File Ingestion: Ingests complete source files without AST-based method pruning or semantic filtering.
• Local Workspace Scope: Requires local disk paths (remote Git repositories must be cloned before packing).

Note on Large Codebase Chunking: Currently, payload chunking (>3M characters) uses sequential streaming based on file size rather than an AST dependency graph. If a caller and its dependency end up split across different chunk files, no warning is triggered. Thanks to @aliakseizelianouski for highlighting this edge case—dependency-aware graph chunking is now on the roadmap for a future release!

That's it, thank you for reading! A GitHub repository with the compiled binaries will be up soon, along with a read-only version of the source code for the curious. or visit https://www.kadmium.dev/dev-tech/context-packer for a live browser demo

If you want to discuss the project or have any questions, feel free to drop a comment below or reach out at [e-mail] and I'll get back to you when I have time.

I will be editing and refining this devlog over the coming weeks if I missed any details.

---

AI Disclosure: The underlying codebase, architecture, and logic are 100% handcrafted and tested in C# by Kadmium. Product copy and documentation phrasing were refined with AI assistance to keep technical descriptions clear and concise.