DEV Community: Nick

Streaming Large HTTP Responses Without Killing Your Memory

Nick — Fri, 04 Sep 2026 16:10:00 +0000

Hey memory-conscious developers! 👋

Ever downloaded a 500MB file with HttpClient and watched your app's memory balloon? Today we'll fix that with streaming!

The Problem: Buffering Everything

By default, HttpClient buffers the entire response in memory:

// 😱 This loads EVERYTHING into memory!
var content = await client.GetStringAsync("https://example.com/huge-file.json");

For a 500MB response, that's 500MB of RAM. For 10 concurrent downloads? 5GB!

🧠 Fun Fact: The default HttpClient.MaxResponseContentBufferSize is 2GB on 64-bit systems. You can hit OutOfMemoryException before you even realize what's happening!

The Solution: HttpCompletionOption.ResponseHeadersRead

This magic option tells HttpClient to return as soon as headers arrive — before the body downloads:

using var response = await client.GetAsync(
    "https://example.com/huge-file.json",
    HttpCompletionOption.ResponseHeadersRead);

response.EnsureSuccessStatusCode();

// Now stream the content
await using var stream = await response.Content.ReadAsStreamAsync();

Streaming to a File

public async Task DownloadFileAsync(string url, string outputPath, CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url, 
        HttpCompletionOption.ResponseHeadersRead, 
        ct);

    response.EnsureSuccessStatusCode();

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = new FileStream(
        outputPath, 
        FileMode.Create, 
        FileAccess.Write, 
        FileShare.None, 
        bufferSize: 81920, // 80KB buffer
        useAsync: true);

    await contentStream.CopyToAsync(fileStream, ct);
}

Streaming JSON with System.Text.Json

For large JSON arrays, you can deserialize while streaming:

public async IAsyncEnumerable<User> StreamUsersAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/users/export",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);

    await foreach (var user in JsonSerializer.DeserializeAsyncEnumerable<User>(
        stream, 
        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
        ct))
    {
        if (user != null)
            yield return user;
    }
}

Consuming Streamed Data

await foreach (var user in userClient.StreamUsersAsync(cancellationToken))
{
    Console.WriteLine($"Processing: {user.Name}");
    // Process one at a time - memory stays flat!
}

Progress Reporting

Want a download progress bar? Here's how:

public async Task DownloadWithProgressAsync(
    string url, 
    string outputPath,
    IProgress<double> progress,
    CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    var totalBytes = response.Content.Headers.ContentLength ?? -1L;
    var canReportProgress = totalBytes > 0;

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = File.Create(outputPath);

    var buffer = new byte[81920];
    long totalBytesRead = 0;
    int bytesRead;

    while ((bytesRead = await contentStream.ReadAsync(buffer, ct)) > 0)
    {
        await fileStream.WriteAsync(buffer.AsMemory(0, bytesRead), ct);
        totalBytesRead += bytesRead;

        if (canReportProgress)
        {
            progress.Report((double)totalBytesRead / totalBytes);
        }
    }
}

💡 Pro Tip: Chunk Processing for APIs

Some APIs return newline-delimited JSON (NDJSON). Process line by line:

public async IAsyncEnumerable<LogEntry> StreamLogsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/logs/stream",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    while (!reader.EndOfStream)
    {
        ct.ThrowIfCancellationRequested();

        var line = await reader.ReadLineAsync(ct);
        if (string.IsNullOrWhiteSpace(line)) continue;

        var entry = JsonSerializer.Deserialize<LogEntry>(line);
        if (entry != null)
            yield return entry;
    }
}

Server-Sent Events (SSE)

SSE is HTTP streaming in action. Here's a client:

public async IAsyncEnumerable<ServerEvent> SubscribeToEventsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var request = new HttpRequestMessage(HttpMethod.Get, "/api/events/stream");
    request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/event-stream"));

    using var response = await _client.SendAsync(
        request,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    string? eventType = null;
    var dataBuilder = new StringBuilder();

    while (!reader.EndOfStream)
    {
        var line = await reader.ReadLineAsync(ct);

        if (string.IsNullOrEmpty(line))
        {
            // Empty line = event complete
            if (dataBuilder.Length > 0)
            {
                yield return new ServerEvent(eventType ?? "message", dataBuilder.ToString());
                eventType = null;
                dataBuilder.Clear();
            }
            continue;
        }

        if (line.StartsWith("event:"))
            eventType = line[6..].Trim();
        else if (line.StartsWith("data:"))
            dataBuilder.AppendLine(line[5..].Trim());
    }
}

public record ServerEvent(string Type, string Data);

Memory Comparison

Approach	100MB File Memory
`GetStringAsync`	~100MB + overhead
`GetByteArrayAsync`	~100MB
`ResponseHeadersRead` + Stream	~80KB (buffer only)

That's a 1000x+ reduction!

When to Stream

✅ Use streaming for:

File downloads
Large JSON arrays (1000+ items)
Server-Sent Events
WebSocket-like patterns
Export endpoints
Log tailing

❌ Don't bother for:

Small API responses (< 1MB)
Single objects
When you need the whole payload anyway

Wrapping Up

Streaming is your secret weapon for handling large HTTP responses. The key insight: HttpCompletionOption.ResponseHeadersRead gives you control over when and how content downloads.

Your app's memory will thank you, and your users won't see mysterious OutOfMemoryException crashes!

Happy streaming! 🚀

Building Bulletproof HTTP Calls with Polly in C#

Nick — Thu, 03 Sep 2026 13:09:00 +0000

Hey friends! 👋

Ever had your app crash because a third-party API had a hiccup? Or watched users get errors because of a temporary network blip? Today we're going to make your HTTP calls bulletproof with resilience patterns!

The Problem: Networks Are Unreliable

The internet is held together by hope and duct tape. Services go down, networks get congested, and that API you depend on? It's going to fail eventually.

🎲 Fun Fact: Amazon found that even 100ms of latency costs them 1% in sales. Netflix reports that 0.1% of their requests fail due to transient errors. If the big players deal with this, so will you!

Enter Polly (and the New .NET 8+ Standard)

For years, Polly has been the go-to library for resilience in .NET. But with .NET 8, Microsoft introduced Microsoft.Extensions.Http.Resilience — a first-party solution built on Polly v8!

Option 1: Microsoft.Extensions.Http.Resilience (.NET 8+)

The new standard way — minimal config, sensible defaults:

// Install: dotnet add package Microsoft.Extensions.Http.Resilience

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler();

That's it! You get:

⏱️ Timeout: 30 seconds total
🔄 Retry: 3 attempts with exponential backoff
⚡ Circuit Breaker: Opens after 10% failure rate
🎚️ Rate Limiter: Prevents overwhelming the target

Customize It

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 5;
        options.Retry.Delay = TimeSpan.FromMilliseconds(500);
        options.CircuitBreaker.FailureRatio = 0.1; // 10% failures trips it
        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(10);
    });

Option 2: Raw Polly (Full Control)

Need more control? Use Polly directly:

// Install: dotnet add package Microsoft.Extensions.Http.Polly

var retryPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => !r.IsSuccessStatusCode)
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
        onRetry: (outcome, delay, attempt, ctx) =>
        {
            Console.WriteLine($"Retry {attempt} after {delay.TotalSeconds}s");
        });

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddPolicyHandler(retryPolicy);

The Circuit Breaker Pattern

This is my favorite pattern. Think of it like an electrical circuit breaker:

Closed (normal): Requests flow through
Open (tripped): Requests fail immediately — don't waste time on a dead service
Half-Open (testing): Let one request through to check if service recovered

var circuitBreaker = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
    .CircuitBreakerAsync(
        handledEventsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (result, duration) =>
        {
            Console.WriteLine($"Circuit OPEN for {duration.TotalSeconds}s!");
        },
        onReset: () => Console.WriteLine("Circuit CLOSED - back to normal"),
        onHalfOpen: () => Console.WriteLine("Circuit HALF-OPEN - testing..."));

💡 Pro Tips

1. Add Jitter to Retries

Without jitter, all your retrying clients hit the recovering server at the same moment. Add randomness:

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.UseJitter = true; // Adds randomness to retry delays
    });

2. Be Idempotency-Aware

Only retry operations that are safe to repeat! A GET is fine, but retrying a POST that creates an order? You might create duplicates.

// Good: Retry GETs
var response = await _client.GetAsync("/api/orders/123");

// Careful: Don't blindly retry POSTs
// Use idempotency keys or check if operation already completed

3. Timeout Correctly

Set both per-attempt AND total timeouts:

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);  // Per attempt
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(30); // Overall
    });

Real-World Example

services.AddHttpClient<IPaymentGateway, PaymentGateway>(client =>
{
    client.BaseAddress = new Uri("https://api.stripe.com/");
})
.AddStandardResilienceHandler(options =>
{
    // Payment APIs need quick failures
    options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(15);

    // Retry on transient errors only
    options.Retry.MaxRetryAttempts = 2;
    options.Retry.ShouldHandle = args => ValueTask.FromResult(
        args.Outcome.Exception is HttpRequestException ||
        args.Outcome.Result?.StatusCode == HttpStatusCode.TooManyRequests);
});

Wrapping Up

Stop hoping your HTTP calls will work. Plan for failure, and your app will gracefully handle the chaos of distributed systems.

Start with AddStandardResilienceHandler() in .NET 8+ — it's an excellent default. Then customize as you learn your services' failure patterns.

Your users will never know how many retries and circuit breaker trips happen behind the scenes. They'll just see a reliable app! 🎉

Happy resilient coding! 🚀

Modern JSON Handling in HttpClient: Ditch the Manual Serialization!

Nick — Wed, 02 Sep 2026 08:05:00 +0000

Hey developers! 👋

I see way too much code that looks like this:

// 😱 The old way - please stop doing this!
var json = JsonConvert.SerializeObject(myObject);
var content = new StringContent(json, Encoding.UTF8, "application/json");
var response = await client.PostAsync(url, content);
var responseJson = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<MyType>(responseJson);

There's a MUCH better way. Let's modernize your HTTP + JSON code!

The System.Net.Http.Json Way

Since .NET 5, System.Net.Http.Json gives us beautiful extension methods:

// ✨ The modern way
var response = await client.PostAsJsonAsync(url, myObject);
var result = await response.Content.ReadFromJsonAsync<MyType>();

That's it. Two lines. No manual serialization, no StringContent, no encoding worries.

📦 Fun Fact: System.Net.Http.Json is included by default in .NET 5+. For .NET Standard 2.0 or .NET Core 3.1, just add the NuGet package!

All the Extension Methods

Sending JSON

// POST with JSON body
await client.PostAsJsonAsync("/api/users", newUser);
await client.PostAsJsonAsync("/api/users", newUser, cancellationToken);

// PUT with JSON body
await client.PutAsJsonAsync("/api/users/123", updatedUser);

// PATCH with JSON body  
await client.PatchAsJsonAsync("/api/users/123", patchDocument);

Reading JSON

// Read response body as JSON
var user = await response.Content.ReadFromJsonAsync<User>();

// One-liner: GET and deserialize
var users = await client.GetFromJsonAsync<List<User>>("/api/users");
var user = await client.GetFromJsonAsync<User>("/api/users/123");

With Cancellation

var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));

var user = await client.GetFromJsonAsync<User>(
    "/api/users/123", 
    cts.Token);

Custom Serialization Options

Need custom JSON settings? Pass JsonSerializerOptions:

var options = new JsonSerializerOptions
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    PropertyNameCaseInsensitive = true,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
    Converters = { new JsonStringEnumConverter() }
};

// Use everywhere
var user = await client.GetFromJsonAsync<User>("/api/users/123", options);
await client.PostAsJsonAsync("/api/users", newUser, options);

Pro Tip: Configure Once

Create a shared options instance:

public static class JsonDefaults
{
    public static JsonSerializerOptions Api { get; } = new()
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        PropertyNameCaseInsensitive = true,
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
        Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) }
    };
}

// Usage
var user = await client.GetFromJsonAsync<User>(url, JsonDefaults.Api);

💡 Source Generators for AOT

Building for Native AOT or want faster serialization? Use source generators!

[JsonSerializable(typeof(User))]
[JsonSerializable(typeof(List<User>))]
[JsonSerializable(typeof(CreateUserRequest))]
public partial class AppJsonContext : JsonSerializerContext { }

// Use the generated context
var user = await client.GetFromJsonAsync(
    "/api/users/123",
    AppJsonContext.Default.User);

await client.PostAsJsonAsync(
    "/api/users",
    newUser,
    AppJsonContext.Default.CreateUserRequest);

⚡ Performance: Source generators are ~2-3x faster than reflection-based serialization and support Native AOT compilation!

Error Handling Done Right

public async Task<User?> GetUserAsync(int id)
{
    var response = await _client.GetAsync($"/api/users/{id}");

    if (response.StatusCode == HttpStatusCode.NotFound)
        return null;

    if (!response.IsSuccessStatusCode)
    {
        var error = await response.Content.ReadFromJsonAsync<ApiError>();
        throw new ApiException(error?.Message ?? "Unknown error", response.StatusCode);
    }

    return await response.Content.ReadFromJsonAsync<User>();
}

Handling Empty Responses

public async Task<User?> GetUserSafeAsync(int id)
{
    var response = await _client.GetAsync($"/api/users/{id}");
    response.EnsureSuccessStatusCode();

    // Handle empty body gracefully
    if (response.Content.Headers.ContentLength == 0)
        return null;

    return await response.Content.ReadFromJsonAsync<User>();
}

Real-World Typed Client Example

public class UserApiClient : IUserApiClient
{
    private readonly HttpClient _client;
    private static readonly JsonSerializerOptions JsonOptions = new()
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        PropertyNameCaseInsensitive = true
    };

    public UserApiClient(HttpClient client)
    {
        _client = client;
    }

    public async Task<IReadOnlyList<User>> GetAllAsync(CancellationToken ct = default)
    {
        return await _client.GetFromJsonAsync<List<User>>(
            "/api/users", 
            JsonOptions, 
            ct) ?? [];
    }

    public async Task<User?> GetByIdAsync(int id, CancellationToken ct = default)
    {
        try
        {
            return await _client.GetFromJsonAsync<User>(
                $"/api/users/{id}", 
                JsonOptions, 
                ct);
        }
        catch (HttpRequestException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
        {
            return null;
        }
    }

    public async Task<User> CreateAsync(CreateUserRequest request, CancellationToken ct = default)
    {
        var response = await _client.PostAsJsonAsync("/api/users", request, JsonOptions, ct);
        response.EnsureSuccessStatusCode();
        return (await response.Content.ReadFromJsonAsync<User>(JsonOptions, ct))!;
    }

    public async Task<bool> DeleteAsync(int id, CancellationToken ct = default)
    {
        var response = await _client.DeleteAsync($"/api/users/{id}", ct);
        return response.IsSuccessStatusCode;
    }
}

Migration Cheat Sheet

Old Way (Newtonsoft)	New Way (System.Text.Json)
`JsonConvert.SerializeObject(obj)`	`JsonSerializer.Serialize(obj)`
`JsonConvert.DeserializeObject<T>(json)`	`JsonSerializer.Deserialize<T>(json)`
`new StringContent(json, ...)`	`PostAsJsonAsync(url, obj)`
`ReadAsStringAsync()` + Deserialize	`ReadFromJsonAsync<T>()`

Wrapping Up

Stop writing boilerplate serialization code. System.Net.Http.Json makes HTTP + JSON a breeze:

Cleaner code
Better performance (especially with source generators)
Built-in cancellation support
Native AOT compatible

Your future self will thank you for the cleaner code! 🎉

Happy serializing! 🚀

HttpClientFactory: Why new HttpClient() Is Killing Your App

Nick — Tue, 01 Sep 2026 08:03:12 +0000

Hey there, fellow C# developers! 👋

Let me tell you a horror story. You're building a nice web service, everything works great in development, and then... production hits. Suddenly your app is throwing SocketException errors left and right. What happened?

You probably did this:

// DON'T DO THIS!
using (var client = new HttpClient())
{
    var response = await client.GetAsync("https://api.example.com/data");
    // ...
}

The Socket Exhaustion Problem

Here's the thing: HttpClient implements IDisposable, so your instincts tell you to wrap it in a using statement. Makes sense, right? Wrong!

When you dispose of an HttpClient, the underlying socket doesn't close immediately. It enters a TIME_WAIT state for about 240 seconds (4 minutes!). Under load, you'll exhaust all available sockets before the old ones recycle.

🤯 Fun Fact: A single Windows machine has roughly 16,000 ephemeral ports available by default. At 100 requests/second with disposed clients, you'll hit socket exhaustion in under 3 minutes!

Enter HttpClientFactory

.NET Core 2.1 introduced IHttpClientFactory to solve this elegantly. It pools and reuses HttpMessageHandler instances while still allowing proper DNS rotation.

Basic Setup

// In Program.cs or Startup.cs
services.AddHttpClient();

// In your service
public class MyService
{
    private readonly IHttpClientFactory _factory;

    public MyService(IHttpClientFactory factory)
    {
        _factory = factory;
    }

    public async Task<string> GetDataAsync()
    {
        using var client = _factory.CreateClient();
        var response = await client.GetAsync("https://api.example.com/data");
        return await response.Content.ReadAsStringAsync();
    }
}

Named Clients

Want pre-configured clients for different APIs? Use named clients:

services.AddHttpClient("github", client =>
{
    client.BaseAddress = new Uri("https://api.github.com/");
    client.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");
    client.Timeout = TimeSpan.FromSeconds(30);
});

services.AddHttpClient("weather", client =>
{
    client.BaseAddress = new Uri("https://api.weather.com/");
});

Then retrieve by name:

var githubClient = _factory.CreateClient("github");
var weatherClient = _factory.CreateClient("weather");

Typed Clients (My Favorite!)

The cleanest approach — inject a fully configured HttpClient directly into your typed service:

services.AddHttpClient<IGitHubClient, GitHubClient>(client =>
{
    client.BaseAddress = new Uri("https://api.github.com/");
    client.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");
});

public class GitHubClient : IGitHubClient
{
    private readonly HttpClient _client;

    // HttpClient is injected, pre-configured!
    public GitHubClient(HttpClient client)
    {
        _client = client;
    }

    public async Task<User> GetUserAsync(string username)
    {
        var response = await _client.GetAsync($"users/{username}");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<User>();
    }
}

💡 Pro Tip: Handler Lifetime

By default, handlers are recycled every 2 minutes to handle DNS changes. You can customize this:

services.AddHttpClient("long-lived")
    .SetHandlerLifetime(TimeSpan.FromMinutes(10));

But be careful with long lifetimes — if your target service's IP changes (common with cloud load balancers), you'll keep hitting the old address!

Wrapping Up

Stop creating HttpClient instances manually. Use IHttpClientFactory and let .NET manage the connection pooling for you. Your ops team will thank you when socket exhaustion errors disappear from your logs! 🎉

Happy coding! 🚀

The Anatomy of a Modern Payment Gateway: Demystifying the Black Box

Nick — Tue, 14 Jul 2026 06:39:55 +0000

Processing payments seems incredibly easy today — just drop in Stripe or Adyen and you are done. But things get interesting when your transaction volume scales and those Payment Service Provider (PSP) fees start taking a massive bite out of your margins. Eventually, you realize there is a vast ecosystem of smaller, regional PSPs that charge less, cover local payment methods the major players ignore, and can significantly boost your conversion rates.

In this article, I will explain how modern Payment Gateways (PG) actually work, specifically when you outgrow a single provider and need to build your own system to route payments across multiple PSPs.

The Illusion of Simplicity

Looking at the Stripe API, integrating a new provider seems as easy as it gets: a simple HTTP request with a JSON payload and a 200 OK in response. Reality is rarely so kind.

While many PSPs try to offer simple interfaces, customer demands and local niches force them to require highly specific additional data. Mix in strict country regulations and regional fraud protection, and in the blink of an eye, you are dealing with a massive, messy protocol with outdated documentation. Some PSPs even invent their own authorization methods or enforce government-approved encryption standards — usually just slightly modified versions of existing algorithms, but enough to make your head hurt.

To a frontend engineer, a payment is just a JSON payload and a success screen. The truly interesting and complicated parts happen on the backend. After that initial request, we don’t just call a PSP and wait for a callback. That is never the end of it. The business always wants more: intelligent routing, cascading fallbacks, fee calculations, currency conversions, and — holly electron forbid — ledger balances with full financial reconciliation. Add legacy XML, SOAP, or plain TCP socket protocols to the mix, and you have a real engineering challenge.

So, what actually happens between the user clicking “Submit” and the payment finishing? It generally follows this lifecycle:

Security & Fraud Detection: First and foremost, the user and their credentials are screened for fraudulent activity.
Routing & Pre-processing: The Payment Gateway selects the optimal payment method and PSP (routing based on cost, region, and reliability). It verifies all required parameters are present and calculates any necessary fees (e.g., subtracting your internal fee from a withdrawal amount before sending it out).
The Request: The PG sends the payment payload to the chosen PSP.
PSP Processing: The external PSP does its own processing (their own anti-fraud, fund reservation, and settlement) and eventually sends a callback with a transaction status: processing, success, or failure.
Post-processing & Conversion: The PSP might process or settle the transaction in a different currency, meaning you must handle currency conversion on your side. If it’s a deposit, you calculate the final fee and settle a slightly smaller amount into the merchant’s internal account.
Cascading (On Failure): If the transaction fails, the flow isn’t necessarily over. Through cascading, the gateway can seamlessly choose a backup PSP and loop back to the request stage to save the transaction.
There are countless nuances at every step, and the edge cases can go on forever. But for the user, this all happens in 10 to 15 seconds (and we fight constantly to reduce that time). For the backend, it is a very long and complex story.

Meet The Crew

From an architectural standpoint, here are the core components you need to build to manage this complexity.

The Payment Gateway (The Core Engine and Internal Protocol): This is the orchestrator, the accountant, and the primary validator. This is the API your merchants or frontends actually talk to. Most of the magic happens here: validating user input, routing to the most cost-effective and reliable PSP, deducting fees, handling local government fiscalization, and executing cascading logic on failures.

The Adaptor (The Translator and External Protocol): Almost every external PSP uses its own protocol — JSON, XML, SOAP, etc. A highly viable architectural strategy is to use the Adaptor Pattern. You build microservices whose sole job is to translate these weird external protocols into your Gateway’s standard internal protocol. I can’t help but mention the old engineering joke: “We have 15 different standards, we need to unite them all. Great, now we have 16 different standards.” But in this case, it is strictly necessary. It separates the workload: one part of your engineering team can grind through external integrations, while the core team focuses on improving the main engine.

The Payment Service Provider (PSP): To tell you the truth, a PSP is basically just another Gateway developed by a different team. They have their own internal routing, methods, and core engines. The only difference is that you communicate with them through their External Protocol. To your internal Gateway, a PSP is simply a provider of one or more payment methods. Even if a PSP is a direct integration with a tier-one bank via PCI-DSS, architecturally, it’s just another method in your pool.

There is a hidden beauty to defining your own strict, standardized external protocol: once your transaction volume is high enough, you can flip the script. Instead of your team building an Adaptor for a new PSP, you can hand the PSP your API documentation and force them to integrate with your protocol.

But beware this trap. While it drastically reduces your initial development costs and workload, it shifts the burden directly onto your support and operations teams. It looks great on paper, but integrations require maintenance. If your traffic to that specific PSP drops and your business becomes less relevant to their bottom line, they will stop maintaining the integration. When their code breaks, it becomes a massive headache for your support team to get them to fix it.

Next time i plan to talk about the lifecycle of a transaction in detail. When it is created, why we do neet such entity and for what purpose we can use it. Hope to see you!

Part 15: Workflow Patterns and Recipes - Data Transformation

Nick — Fri, 26 Jun 2026 08:03:00 +0000

As we conclude this 15-part series on Vyshyvanka, we want to leave you with practical tools. One of the most common challenges in workflow automation is not moving data, but transforming it. Today, we look at the patterns we use to map, clean, and reshape data as it flows through your pipelines.

The Transformation Challenge

In a real-world workflow, you rarely get the data exactly in the format you need. Service A might return a deeply nested JSON object, while Service B expects a flat structure with different field names. Vyshyvanka handles this through its expression engine and Code Node — giving you both declarative and programmatic transformation options.

Pattern 1: Path Projection with Expressions

When you have a large JSON response and only need specific values, use expressions in your node configuration to extract exactly what you need:

{{ nodes.api.data.items[0].user.id }}

This plucks a deeply nested ID from an API response and passes it as a clean input to your next node. You can use this directly in any node's configuration field.

For more complex extraction, combine with built-in functions:

{{ toUpper(nodes.api.data.items[0].user.name) }}
{{ concat(nodes.user.data.firstName, " ", nodes.user.data.lastName) }}
{{ coalesce(nodes.api.data.nickname, nodes.api.data.name, "Anonymous") }}

Pattern 2: Structural Remapping with Code Nodes

When you need to transform the entire shape of the data, the Code Node is your best tool. It supports two runtimes:

JavaScript (via Jint engine):

General-purpose JavaScript for complex transformations. You have access to:

input — the full input data from upstream nodes
executionId — current execution ID
workflowId — current workflow ID
log(message) — logging that appears in execution output
getItems() — returns input as an array (wraps single values)
toJson(value) — serializes a value to a JSON string

// Remap an API response to a different structure
log("Transforming user data: " + input.email);
return {
    email: input.email.toLowerCase(),
    fullName: input.firstName + " " + input.lastName,
    isVerified: input.status === "active",
    createdYear: new Date(input.createdAt).getFullYear()
};

JSONata:

A declarative query and transformation language, ideal for complex path selection and restructuring:

{
    "users": items.{
        "id": userId,
        "name": firstName & " " & lastName,
        "active": status = "active"
    },
    "total": $count(items)
}

JSONata shines when you need to reshape deeply nested structures without imperative loop code.

Pattern 3: Batch Processing with runForEachItem

Both JavaScript and JSONata support two execution modes:

runOnce: Executes your code against the full input object. Use this for single-item transformations or when you need access to the entire dataset.
runForEachItem: If your input is an array, the engine executes your logic for every item, collecting the results into a new array.

// Mode: runForEachItem
// 'input' here is a single item from the array
return {
    id: input.id,
    label: input.name.toUpperCase(),
    processed: true
};

This is the equivalent of a .map() operation, but managed by the engine with proper error handling and logging for each item.

Pattern 4: Conditional Routing with Logic Nodes

Not all transformation is about reshaping data — sometimes you need to route data differently based on its content. The Switch node evaluates a value and routes to different output ports:

Configure the switch value: {{ nodes.api.data.status }}
Define cases: "active" → port A, "suspended" → port B, default → port C

Each downstream branch can then apply its own transformation logic appropriate to that case.

The If node handles binary decisions:

Condition: {{ nodes.check.data.amount }} > threshold
True branch: process normally
False branch: send alert

Pattern 5: Aggregation with Loop + Merge

For workflows that need to process a collection and then aggregate the results:

Loop Node: Iterates over an array, executing a subgraph for each item.
Subgraph nodes: Transform/enrich each item (API calls, lookups, etc.).
Loop completion: The loop node collects all outputs from its "done" port.
Merge Node: Combines data from multiple branches into a single object.

The loop node exposes per-iteration context:

{{ nodes.loop.data.index }}        // Current iteration index
{{ nodes.loop.data.item }}         // Current item
{{ nodes.loop.data.isFirst }}      // Boolean
{{ nodes.loop.data.isLast }}       // Boolean
{{ nodes.loop.data.totalCount }}   // Total items

Pattern 6: Data Enrichment Pipeline

A common pattern is fetching additional data for each item in a collection:

HttpRequest → Get list of orders from API
Loop → For each order:
- HttpRequest → Fetch customer details by {{ nodes.loop.data.item.customerId }}
- Code Node → Merge order + customer into enriched object
DatabaseQuery → Store enriched results

Each step references the previous step's output through expressions, creating a data pipeline that transforms raw API responses into enriched, business-ready objects.

Best Practices

Keep transformations close to their consumer: Transform data in the node configuration or a Code Node immediately before the node that needs it.
Use expressions for simple extractions: {{ nodes.api.data.user.name }} is clearer than a Code Node for simple path access.
Use Code Nodes for complex logic: Multi-field remapping, conditional logic, and calculations belong in Code Nodes where they are testable and readable.
Prefer JSONata for pure data reshaping: When you are only restructuring JSON without side effects, JSONata expressions are more concise than JavaScript.
Use runForEachItem for array transformations: It handles iteration, error collection, and result aggregation automatically.

Wrapping Up the Series

We hope this series has shown you that Vyshyvanka is built for the real world — secure, extensible, and performant. We started with basic triggers and ended with complex data transformation patterns. The platform gives you the building blocks; how you combine them is up to your creativity.

The community is the final piece of the puzzle. Now it is your turn. Go build something, share your custom nodes, and let us know what you think.

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 14: Community and Ecosystem - Contributing to Vyshyvanka

Nick — Thu, 25 Jun 2026 09:02:00 +0000

It is clear that Vyshyvanka is more than just code — it is an ecosystem. The true power of an open-source workflow engine lies in its community. Today, we want to talk about how you can get involved, whether you are interested in pushing the boundaries of the core engine or building specialized solutions with custom plugins.

The Core Engine vs. The Plugin Ecosystem

A common question we get is: 'Should I contribute a PR to the core engine, or should I build a separate plugin?' The answer depends entirely on the scope of your contribution.

When to Contribute to Core

The core engine (Vyshyvanka.Core, Vyshyvanka.Engine, Vyshyvanka.Api, Vyshyvanka.Designer) should be reserved for changes that benefit every user of the platform. Good candidates for core contributions:

Performance improvements to the execution pipeline
New fundamental port types or expression functions
Bug fixes in the engine, validation, or persistence layers
Enhancements to the Designer UI (canvas, node editor, property editors)
Improvements to the API surface (new endpoints, better error responses)
Documentation improvements

These changes require careful review and testing because they impact every installation. We encourage PRs here, but we also ask that you open an issue first so we can discuss the architectural impact.

When to Build a Plugin

Plugins (./plugins/) are the best way to extend functionality without increasing the maintenance burden of the core. Good candidates for plugins:

Integration with a specific third-party SaaS tool (CRM, CI/CD, monitoring)
Custom nodes specific to your industry or use case
Experimental node behaviors that are not yet ready for core
Proprietary integrations you want to keep separate from the open source project

Plugins are independent, versionable, and can be maintained outside the core release cycle. They empower you to solve your specific problems immediately without waiting for a core release.

Project Structure at a Glance

Understanding where things live is key to contributing effectively:

Vyshyvanka/
├── src/
│   ├── Vyshyvanka.Core/        # Domain layer (zero dependencies)
│   ├── Vyshyvanka.Engine/      # Execution engine, persistence, plugins
│   ├── Vyshyvanka.Api/         # REST API
│   ├── Vyshyvanka.Designer/    # Blazor WASM UI
│   ├── Vyshyvanka.AppHost/     # .NET Aspire orchestration
│   └── Vyshyvanka.ServiceDefaults/
├── plugins/
│   ├── Vyshyvanka.Plugin.AdvancedHttp/
│   ├── Vyshyvanka.Plugin.GitLab/
│   ├── Vyshyvanka.Plugin.Jira/
│   └── Vyshyvanka.Plugin.Tmplt/  # Starter template!
├── tests/
│   └── Vyshyvanka.Tests/
└── docs/

Dependencies flow strictly downward. Core has zero dependencies. Engine depends on Core. Api depends on Core and Engine. Plugins depend only on Core.

Getting Started with Plugin Development

The fastest way to start is with the template plugin:

Copy the template: Duplicate plugins/Vyshyvanka.Plugin.Tmplt/ and rename it.
Update PluginInfo.cs: Set your plugin ID, name, version, and author.
Create your nodes: Inherit from BasePluginNode, add [NodeDefinition] and [ConfigurationProperty] attributes.
Build and test: The test project can reference your plugin directly.
Package: Publish as a NuGet package for easy distribution.

Here is the minimal structure:

// PluginInfo.cs
[assembly: Plugin(
    "com.yourorg.myplugin",
    Name = "My Plugin",
    Version = "1.0.0",
    Description = "Does something useful",
    Author = "Your Name")]

// Nodes/MyCustomNode.cs
[NodeDefinition(
    Name = "My Custom Action",
    Description = "Does the thing",
    Icon = "fa-solid fa-star")]
[ConfigurationProperty("apiUrl", "string", Description = "API endpoint", IsRequired = true)]
public class MyCustomNode : BasePluginNode
{
    public override string Type => "my-custom-action";
    public override NodeCategory Category => NodeCategory.Action;

    public override async Task<NodeOutput> ExecuteAsync(NodeInput input, IExecutionContext context)
    {
        var apiUrl = GetRequiredConfigValue<string>(input, "apiUrl");
        // Your logic here...
        return SuccessOutput(JsonSerializer.SerializeToElement(new { result = "done" }));
    }
}

Contributing to Core: Guidelines

If you want to contribute to the core:

Open an issue first: Describe the problem and your proposed solution. This helps us maintain consistency and prevents wasted effort.
Follow conventions: File-scoped namespaces, primary constructors for DI, CancellationToken in every async method, AwesomeAssertions in tests.
Write tests: Every change needs appropriate test coverage — unit tests for logic, integration tests for API changes.
Respect the dependency rules: Never introduce an upward or circular reference between projects.
Check the ripple effect: If you change an interface in Core, update all implementations in Engine.

Running the Full Stack

# Build everything
dotnet build

# Run all tests
dotnet test

# Start the full application (API + Designer)
dotnet run --project src/Vyshyvanka.AppHost

# Start just the API
dotnet run --project src/Vyshyvanka.Api

A Growing Ecosystem

Every plugin you build and every bug you fix makes Vyshyvanka more valuable for everyone else. Whether you are an enterprise developer building mission-critical workflows or a hobbyist automating your home server, your contribution helps shape the future of open-source workflow automation.

We are excited to see what you build.

In the final part of this series, we will discuss Part 15: Workflow Patterns and Recipes - Data Transformation. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 13: Deployment Strategies - Orchestrating Vyshyvanka with .NET Aspire

Nick — Wed, 24 Jun 2026 08:02:00 +0000

As we move from development to production, deployment complexity often becomes the biggest bottleneck. For Vyshyvanka, we chose .NET Aspire as our development orchestration framework and as the foundation for production-ready deployments. Today, we look at how Aspire simplifies running a multi-service workflow engine.

Why Aspire?

Traditional development setup means manually configuring your API, your Blazor frontend, your database, and ensuring they can discover each other. If one service's port changes, everything breaks. Aspire solves this by treating the entire application as a single, orchestrated 'app model'. This app model describes how your services relate to each other — what resources they need, what ports they share, and how they discover each other via service discovery.

Our AppHost

The Vyshyvanka.AppHost project defines our application topology in C#:

var builder = DistributedApplication.CreateBuilder(args);

// Check if PostgreSQL mode is enabled
var usePostgres = builder.Configuration["UsePostgres"]?
    .Equals("true", StringComparison.OrdinalIgnoreCase) == true;

IResourceBuilder<ProjectResource> api;

if (usePostgres)
{
    // Production: PostgreSQL with persistent volume
    var postgres = builder.AddPostgres("postgres")
        .WithDataVolume("vyshyvanka-postgres-data");
    var database = postgres.AddDatabase("vyshyvankadb");

    api = builder.AddProject<Projects.Vyshyvanka_Api>("api")
        .WithReference(database)
        .WaitFor(database);
}
else
{
    // Development: SQLite (no external dependencies)
    api = builder.AddProject<Projects.Vyshyvanka_Api>("api");
}

// Designer discovers API via service discovery
builder.AddProject<Projects.Vyshyvanka_Designer>("designer")
    .WithReference(api)
    .WaitFor(api);

builder.Build().Run();

This is actual production code from our repository. A few things to note:

Conditional infrastructure: The same AppHost supports both SQLite (for quick local dev) and PostgreSQL (for production-like environments). A single environment variable flips the switch.
Service discovery: The Designer automatically discovers the API endpoint — no hard-coded URLs in configuration files.
Dependency ordering: WaitFor ensures the API doesn't start until the database is ready, and the Designer doesn't start until the API is accepting requests.

Development Experience

Running the full stack locally is a single command:

dotnet run --project src/Vyshyvanka.AppHost

This starts:

The PostgreSQL container (if configured) with a persistent data volume
The API service with proper connection strings injected
The Blazor WebAssembly Designer with API endpoint configured

The Aspire dashboard gives you a unified view of all services, their logs, and distributed traces — all without any additional configuration.

Service Defaults

The Vyshyvanka.ServiceDefaults project provides shared configuration that every service gets automatically:

Health checks — standardized readiness and liveness probes
OpenTelemetry — distributed tracing and metrics collection
Resilience — default HTTP client resilience policies
Service discovery — automatic endpoint resolution

Each service opts in with a single line: builder.AddServiceDefaults().

Containerization

Aspire is built on a container-first mindset. In development, it manages Docker containers for infrastructure (PostgreSQL). For production deployment, the same application model can generate deployment manifests for container orchestrators.

Because our application is already decomposed into well-defined services with explicit dependencies, the transition from local development to containerized production is straightforward:

The API and Designer are standard .NET web applications — they containerize with the default .NET SDK container support.
Database connection strings are injected via environment variables, whether running locally or in a container.
Service discovery works the same way in both environments.

Environment-Specific Configuration

We handle the dev/prod split cleanly through the AppHost configuration:

Environment	Database	Auth Provider	Credential Storage
Development	SQLite (file)	Built-in (seeded users)	Built-in (AES-256)
Staging	PostgreSQL (container)	Keycloak/Authentik	Built-in (AES-256)
Production	PostgreSQL (managed)	OIDC provider	Vault/OpenBao

The application code is identical across all environments. Only the infrastructure wiring changes.

Observability Built In

One of the biggest wins with Aspire is built-in observability. Through ServiceDefaults, every HTTP request, database query, and cross-service call is automatically traced with OpenTelemetry. When a workflow fails in production, you can trace the entire request path through the system using standard observability tools (Jaeger, Zipkin, Application Insights, etc.).

Deployment Consistency

The 'it works on my machine' problem is eliminated. With Aspire, your infrastructure is defined as code alongside your logic. The topology you test locally is the same topology that runs in production. You don't manage shell scripts to wire up services — you manage the C# model that defines your application architecture.

By using Aspire as our orchestration layer, we have drastically reduced the time it takes to go from a local feature to a testable deployment.

In the next part, we will discuss Part 14: Community and Ecosystem - Contributing to Vyshyvanka. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 12: Performance Optimization - High-Throughput Concurrency

Nick — Tue, 23 Jun 2026 07:01:00 +0000

In the world of workflow automation, efficiency is not just a nice-to-have; it is a fundamental requirement. A single workflow engine instance might need to manage hundreds of concurrent executions, each waiting on I/O, external APIs, or database operations. Today, we explore how Vyshyvanka achieves high-throughput performance using .NET's powerful concurrency model.

The Async-First Architecture

Vyshyvanka was built with an async-first philosophy. We recognized early on that workflow engines are inherently I/O-bound. You are constantly waiting: waiting for a database row to lock, waiting for an API to respond, or waiting for a disk write to confirm.

If we were to use traditional synchronous threads, we would quickly exhaust the thread pool and bring the system to a halt. By using async/await everywhere, we ensure that while a workflow waits for I/O, the underlying thread is returned to the .NET thread pool to handle other work. This allows a relatively small number of threads to manage hundreds of concurrent workflow executions.

Concurrency Control with SemaphoreSlim

The engine uses SemaphoreSlim to limit the degree of parallelism at each execution level. This prevents a single large workflow from saturating all available resources:

private async Task<NodeExecutionResult[]> ExecuteLevelWithThrottlingAsync(
    Workflow workflow, List<string> level, IExecutionContext context,
    ConcurrentBag<NodeExecutionResult> nodeResults, int maxParallelism,
    CancellationToken cancellationToken)
{
    using var semaphore = new SemaphoreSlim(maxParallelism, maxParallelism);

    var tasks = level.Select(async nodeId =>
    {
        await semaphore.WaitAsync(cancellationToken);
        try
        {
            return await ExecuteNodeInWorkflowAsync(
                workflow, nodeId, context, nodeResults, cancellationToken);
        }
        finally
        {
            semaphore.Release();
        }
    }).ToList();

    return await Task.WhenAll(tasks);
}

The default parallelism is Environment.ProcessorCount * 2, configurable per workflow via workflow.Settings.MaxDegreeOfParallelism. This gives you fine-grained control — a lightweight data-sync workflow might run with parallelism of 1, while a fan-out notification workflow might use 20.

Thread Safety Through Design

Managing concurrency requires careful handling of shared state. Here is our approach:

Thread-Safe Collections: The engine tracks active executions and node results using ConcurrentDictionary and ConcurrentBag:

private readonly ConcurrentDictionary<Guid, CancellationTokenSource> _activeExecutions = new();

// Node results are collected concurrently from parallel branches
var nodeResults = new ConcurrentBag<NodeExecutionResult>();

Scoped Execution Context: Each execution gets its own IExecutionContext instance. The context stores node outputs and execution variables. While the context is mutated during execution (nodes write their outputs into it), each execution is isolated — one workflow's context never touches another's.
Linked Cancellation: Each execution creates a linked CancellationTokenSource so that cancellation can come from either the caller or the engine's CancelExecutionAsync method:

using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
_activeExecutions[context.ExecutionId] = cts;

Minimizing Thread Pool Starvation

A common trap in .NET is 'sync-over-async' — calling .Result or .Wait() on an asynchronous method. This blocks a thread, preventing it from doing other work, and is the primary cause of thread pool starvation. We enforce a strict rule in our codebase: No blocking calls. Every call from the engine core through the plugin layer must be properly awaited.

This rule extends to plugin nodes. Plugin execution goes through IPluginHost.ExecuteNodeInIsolationAsync with a configurable timeout. If a plugin blocks or hangs, the timeout fires and the node is marked as failed without affecting other executions.

Memory Efficiency

We apply several techniques to reduce allocations in the hot path:

Cached empty objects: A single JsonDocument.Parse("{}").RootElement.Clone() is reused across all nodes that need an empty input, avoiding repeated allocations.
StringBuilder reuse: Expression evaluation uses StringBuilder with pre-estimated capacity for string interpolation.
Compiled regex: The expression pattern matching uses source-generated [GeneratedRegex] for zero-allocation matching.

// Cached once, shared across all executions
private static readonly JsonElement EmptyObjectElement =
    JsonDocument.Parse("{}").RootElement.Clone();

Plugin Timeout Protection

External plugin nodes get an additional layer of protection. The engine wraps plugin execution with a timeout to prevent runaway code from blocking the pipeline:

private static readonly TimeSpan DefaultPluginTimeout = TimeSpan.FromSeconds(30);

private async Task<NodeOutput> ExecuteNodeInstanceAsync(
    INode nodeInstance, NodeInput input, IExecutionContext context, TimeSpan timeout)
{
    if (_pluginHost is not null && _pluginHost.IsPluginNode(nodeInstance.Type))
        return await _pluginHost.ExecuteNodeInIsolationAsync(nodeInstance, input, context, timeout);

    return await nodeInstance.ExecuteAsync(input, context);
}

Leveraging the .NET Thread Pool

The .NET Thread Pool is incredibly good at its job, but it needs the engine to be a 'good citizen'. By being truly asynchronous, we allow the .NET runtime to:

Dynamically scale the number of threads based on CPU load
Optimize task scheduling to keep CPU caches warm
Avoid context-switching overhead that comes with excessive multi-threading

The Results

By combining non-blocking I/O with granular concurrency control, Vyshyvanka can scale horizontally. Because our execution state is persisted separately from the workflow definition, you can spin up multiple engine instances connected to the same database. Your throughput is limited by your infrastructure, not by a single process.

Performance is a journey of constant refinement. By respecting the asynchronous nature of .NET, isolating execution contexts, and avoiding thread-blocking patterns, we have built an engine that is as fast as it is flexible.

In the next part, we will discuss Part 13: Deployment Strategies - Orchestrating Vyshyvanka with .NET Aspire. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 11: Testing Strategies - Guaranteeing Rock-Solid Workflows

Nick — Mon, 22 Jun 2026 07:01:00 +0000

In the world of workflow automation, the cost of failure is high. A bug in a workflow can lead to lost data, broken integrations, or halted business processes. To prevent this, we have built a comprehensive testing strategy for Vyshyvanka that covers every layer — from isolated unit tests to full API integration tests.

The Philosophy: Testable by Design

Before we write a single test, we ensure our code is testable. We accomplish this by:

Injecting dependencies (like HttpClient or ICredentialProvider) into constructors
Using interfaces everywhere so we can substitute real implementations in tests
Keeping node logic pure — given input and configuration, produce output

This simple architectural choice allows us to easily swap real services for test doubles.

Our Testing Stack

Tool	Purpose
xUnit	Test framework with async support
AwesomeAssertions	Fluent assertion library (`.Should().Be(...)`)
NSubstitute	Interface mocking (`Substitute.For<IService>()`)
CsCheck	Property-based testing for validation and serialization
bUnit	Blazor component testing
WebApplicationFactory	Full API integration tests

Example: Testing a Node

Let us look at how we test the HttpRequestNode. We do not make real network calls — we inject a custom MockHttpHandler that returns preconfigured responses:

public class HttpRequestNodeTests
{
    private static ExecutionContext CreateContext() =>
        new(Guid.NewGuid(), Guid.NewGuid(), NullCredentialProvider.Instance);

    private static HttpRequestNode CreateNodeWithHandler(HttpMessageHandler handler)
    {
        var client = new HttpClient(handler);
        return new HttpRequestNode(client);
    }

    [Fact]
    public async Task WhenGetRequestSucceedsThenReturnsSuccessWithResponseData()
    {
        var handler = new MockHttpHandler(new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = new StringContent(
                """{"message":"hello"}""",
                Encoding.UTF8,
                "application/json")
        });
        var sut = CreateNodeWithHandler(handler);
        var config = JsonSerializer.SerializeToElement(new
        {
            url = "https://api.example.com/data",
            method = "GET"
        });
        var input = new NodeInput { Data = default, Configuration = config };
        var context = CreateContext();

        var result = await sut.ExecuteAsync(input, context);

        result.Success.Should().BeTrue();
        result.Data.GetProperty("statusCode").GetInt32().Should().Be(200);
    }
}

Key patterns here:

MockHttpHandler is a custom HttpMessageHandler that captures requests and returns preset responses.
NullCredentialProvider.Instance provides a no-op credential provider for tests that don't need auth.
Configuration is passed as JsonElement — matching exactly how the engine delivers it.
Assertions use AwesomeAssertions' fluent syntax.

Mocking with NSubstitute

For services behind interfaces, we use NSubstitute. Here is how we test the PluginLoadingService:

public class PluginLoadingServiceTests
{
    private readonly IPluginLoader _pluginLoader = Substitute.For<IPluginLoader>();
    private readonly IPluginValidator _pluginValidator = Substitute.For<IPluginValidator>();
    private readonly INodeRegistry _nodeRegistry = Substitute.For<INodeRegistry>();
    private readonly IPackageCache _packageCache = Substitute.For<IPackageCache>();
    private readonly PluginLoadingService _sut;

    public PluginLoadingServiceTests()
    {
        _sut = new PluginLoadingService(
            _pluginLoader, _pluginValidator, _nodeRegistry, _packageCache);
    }

    [Fact]
    public async Task WhenPluginLoadsSuccessfullyThenRegistersNodes()
    {
        var assembly = typeof(PluginLoadingServiceTests).Assembly;
        _pluginLoader.TryLoadAssembly(Arg.Any<string>())
            .Returns(assembly);
        _pluginValidator.ValidatePlugin(assembly)
            .Returns(new PluginValidationResult { IsValid = true });

        var result = await _sut.LoadAndValidatePluginsAsync(
            "test-pkg", "1.0.0", "/install/path");

        result.Failure.Should().BeNull();
        _nodeRegistry.Received(1).RegisterFromAssembly(assembly);
    }
}

NSubstitute gives us:

Substitute.For<T>() — creates a test double for any interface
Arg.Any<T>() — matches any argument
.Returns(value) — configures return values
.Received(n) — verifies a method was called n times

Property-Based Testing with CsCheck

For logic that should hold true for all inputs (not just hand-picked examples), we use CsCheck:

[Fact]
public void WhenSerializedThenDeserializationRoundTrips()
{
    Gen.String.Sample(original =>
    {
        var json = JsonSerializer.Serialize(original);
        var result = JsonSerializer.Deserialize<string>(json);
        Assert.Equal(original, result);
    });
}

[Fact]
public void WhenWorkflowIdGeneratedThenAlwaysValid()
{
    Gen.Guid.Sample(id =>
    {
        id.Should().NotBe(Guid.Empty);
        Guid.TryParse(id.ToString(), out _).Should().BeTrue();
    });
}

CsCheck generates hundreds of random inputs and verifies your invariants hold for all of them. We use this extensively for:

Serialization round-trips
Expression evaluator edge cases
Validation logic completeness

Integration Tests

For end-to-end API testing, we use WebApplicationFactory:

public class ExecutionApiTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient _client;

    public ExecutionApiTests(WebApplicationFactory<Program> factory)
    {
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task WhenWorkflowNotFoundThenReturns404()
    {
        var request = new { workflowId = Guid.NewGuid() };
        var response = await _client.PostAsJsonAsync("/api/execution", request);

        response.StatusCode.Should().Be(HttpStatusCode.NotFound);
    }
}

This spins up the full ASP.NET Core pipeline in-process — middleware, controllers, DI, database — giving us confidence that the entire stack works together.

Test Organization

All tests live in Vyshyvanka.Tests/, organized by type:

tests/Vyshyvanka.Tests/
├── Unit/                  # Isolated tests, no I/O
│   ├── Nodes/            # Node-specific tests
│   ├── Components/       # Blazor component tests (bUnit)
│   └── ...
├── Property/             # CsCheck property-based tests
├── Integration/          # WebApplicationFactory tests
│   └── Fixtures/         # Shared test fixtures
└── E2E/                  # End-to-end scenario tests

Test Naming Convention

We follow the pattern When{Condition}Then{ExpectedResult}:

WhenWorkflowHasNoTriggerThenValidationFails()
WhenGetRequestSucceedsThenReturnsSuccessWithResponseData()
WhenPluginValidationFailsThenReturnsError()

This makes test output self-documenting — you can read the test names as a specification of the system's behavior.

The Benefits

Speed: Unit tests run in milliseconds. We can run the full suite on every commit.
Confidence: By mocking external dependencies, we verify our code's logic in isolation.
Documentation: Tests are the ultimate specification — they show exactly how every component behaves under specific conditions.
Regression Protection: Property-based tests catch edge cases that hand-written tests miss.

Testing is not a chore; it is a safety net. By making our nodes modular, injecting our dependencies, and rigorously testing every layer, we ensure that Vyshyvanka remains as reliable as the systems it automates.

In the next part, we will discuss Part 12: Performance Optimization - High-throughput processing and concurrency. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 10: Plugin System Architecture - Extensibility by Design

Nick — Fri, 19 Jun 2026 08:02:00 +0000

Our workflow engine is designed to be more than just a pre-defined set of nodes. We recognize that every business, every API, and every integration is unique. That is why we built Vyshyvanka around a robust, assembly-based plugin architecture that allows you to extend the engine without touching the core codebase.

The Foundation: Assembly Loading

At the heart of our extensibility model is .NET's AssemblyLoadContext. When the engine starts, the PluginLoader scans a designated plugin directory for compiled .dll files. Each plugin gets its own collectible PluginLoadContext, which provides full isolation from the core application — preventing version conflicts between your dependencies and the engine's dependencies.

internal class PluginLoadContext : AssemblyLoadContext
{
    private readonly AssemblyDependencyResolver _resolver;

    public PluginLoadContext(string pluginPath) : base(isCollectible: true)
    {
        _resolver = new AssemblyDependencyResolver(pluginPath);
    }

    protected override Assembly? Load(AssemblyName assemblyName)
    {
        // Try to resolve from plugin directory first
        var assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName);
        if (assemblyPath is not null)
            return LoadFromAssemblyPath(assemblyPath);

        // Fall back to default context for shared assemblies (like Vyshyvanka.Core)
        return null;
    }
}

The isCollectible: true flag means plugins can be unloaded at runtime — enabling hot-reload scenarios and safe removal of installed packages.

Declaring Plugins

Each plugin is identified by a PluginAttribute on the assembly. This tells the engine who you are, what version you are on, and provides discovery metadata:

// PluginInfo.cs
using Vyshyvanka.Core.Interfaces;

[assembly: Plugin(
    "vyshyvanka.plugin.httpclient",
    Name = "Advanced HTTP Client",
    Version = "1.0.0",
    Description = "Advanced HTTP client nodes with retry, polling, and batch request capabilities.",
    Author = "Vyshyvanka")]

If an assembly does not have this attribute, the loader skips it silently — only intentional Vyshyvanka plugins get registered.

Creating Custom Nodes

The power of the system lies in how we define nodes. A node class inherits from BasePluginNode and is decorated with attributes that describe its UI representation and configuration schema:

[NodeDefinition(
    Name = "GraphQL Request",
    Description = "Execute GraphQL queries and mutations against a GraphQL endpoint",
    Icon = "fa-solid fa-diagram-project")]
[NodeInput("input", DisplayName = "Input")]
[NodeOutput("output", DisplayName = "Response", Type = PortType.Object)]
[ConfigurationProperty("endpoint", "string", Description = "GraphQL endpoint URL", IsRequired = true)]
[ConfigurationProperty("query", "string", Description = "GraphQL query or mutation string", IsRequired = true)]
[ConfigurationProperty("variables", "object", Description = "Query variables as key-value pairs")]
[ConfigurationProperty("headers", "object", Description = "Additional HTTP headers")]
[ConfigurationProperty("timeout", "number", Description = "Request timeout in seconds (default: 30)")]
public class GraphQLNode : BasePluginNode
{
    public override string Type => "graphql-request";
    public override NodeCategory Category => NodeCategory.Action;

    public override async Task<NodeOutput> ExecuteAsync(NodeInput input, IExecutionContext context)
    {
        var endpoint = GetRequiredConfigValue<string>(input, "endpoint");
        var query = GetRequiredConfigValue<string>(input, "query");
        var variables = GetConfigValue<Dictionary<string, object>>(input, "variables");
        var timeoutSeconds = GetConfigValue<int?>(input, "timeout") ?? 30;

        // ... execute the GraphQL request ...

        return SuccessOutput(responseData);
    }
}

Notice that [ConfigurationProperty] attributes are placed on the class itself (not on individual properties). Configuration values are accessed at runtime through the GetConfigValue<T>() and GetRequiredConfigValue<T>() helper methods inherited from the base class. This design means the engine and Designer UI can discover the configuration schema via reflection without instantiating the node.

Plugin Validation

Before a plugin is registered, the PluginValidator checks it for correctness:

Does it have the required [Plugin] attribute?
Do its node types properly implement INode?
Are there any dependency conflicts?

If validation fails, the plugin is rejected but the engine continues operating. You get a clear error in the logs telling you exactly what went wrong.

Package Management

Beyond dropping DLLs into a directory, Vyshyvanka has a full NuGet-based package management system. The NuGetPackageManager orchestrates the lifecycle:

PackageSearchService — Searches configured NuGet sources, resolves versions, checks for updates.
PluginLoadingService — Loads plugin assemblies, validates them, and registers their node types with the NodeRegistry.
NuGetPackageManager — Orchestrates install/update/uninstall using the above services.

This means you can install plugins from private NuGet feeds, manage versions centrally, and update plugins without restarting the engine.

Why This Design?

Strong Typing: By leveraging C# attributes, we automatically generate the UI schema for your nodes. When you add a new [ConfigurationProperty], it appears in the Designer without writing frontend code.
Isolation: Each plugin runs in its own AssemblyLoadContext. If a plugin fails to load or crashes during execution, the rest of the engine remains operational. Plugin nodes are executed through IPluginHost with a timeout wrapper.
Discoverability: The NodeRegistry automatically picks up all INode implementations from loaded plugin assemblies, making them available in the Designer immediately.
Hot Unload: Because load contexts are collectible, you can uninstall a plugin at runtime. The GC reclaims the memory once all references are released.

Exploring Real Examples

If you look into our ./plugins/ directory, you can see this in action:

Vyshyvanka.Plugin.AdvancedHttp — HTTP retry with exponential backoff, HTTP polling with success/failure conditions, batch requests with concurrency control, and GraphQL.
Vyshyvanka.Plugin.GitLab — Full GitLab integration: issues, merge requests, pipelines, releases, files, tags, and webhook triggers.
Vyshyvanka.Plugin.Jira — Jira issues, comments, search (JQL), users, and versions.
Vyshyvanka.Plugin.Tmplt — A starter template for building your own plugin.

Getting Started with Your Own Plugin

Create a new class library targeting the same .NET version.
Add a reference to Vyshyvanka.Core (the only allowed dependency from core).
Add your PluginInfo.cs with [assembly: Plugin(...)].
Create node classes inheriting BasePluginNode (or implement INode directly).
Decorate with [NodeDefinition] and [ConfigurationProperty] attributes.
Build, and either drop the DLL into the plugins directory or publish as a NuGet package.

Building for extensibility ensures that as your business needs evolve, your automation platform evolves with you.

In the next part, we will discuss Part 11: Testing Strategies - How to guarantee your workflows are rock-solid. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka

Part 9: Security First - Credentials, Auth, and Secrets Management

Nick — Thu, 18 Jun 2026 08:01:00 +0000

Building a workflow engine that interacts with external APIs, databases, and internal systems means you are constantly handling sensitive data. If you get security wrong here, the consequences are severe. Today, we are discussing how we handle credentials, authentication, and secrets management in Vyshyvanka.

The Problem: Where do secrets live?

The biggest mistake developers make when building automation tools is hard-coding credentials. Whether it is an API key in a config file or a database password in an environment variable, these secrets eventually leak. We needed a way to manage secrets that is both developer-friendly and secure enough for production environments.

The Credential Store

In Vyshyvanka, we do not store raw secrets in the workflow definition. Instead, we use a dedicated Credential Store. When you add a new service integration to your workflow, you create a Credential object through the Credential Manager UI. This object holds the encrypted access data for that service.

Each node that needs authentication is assigned a CredentialId in its configuration. At runtime, the engine resolves that reference through the ICredentialProvider, decrypts the credential in-memory, and provides it to the node instance. The raw secrets never appear in workflow JSON, API responses, or execution logs.

// How a node receives its credential at execution time
var input = new NodeInput
{
    Data = inputData,
    Configuration = evaluatedConfig,
    CredentialId = node.CredentialId  // Reference, not the actual secret
};

Three Storage Backends

We support three credential storage providers, configurable via appsettings.json:

Provider	Config Value	How Secrets Are Stored
Built-in	`BuiltIn`	AES-256 encrypted in the database
HashiCorp Vault	`HashiCorpVault`	Vault KV v2 secret engine
OpenBao	`OpenBao`	OpenBao KV v2 secret engine

All three share the same ICredentialService interface and CredentialValidator logic. The choice of backend is transparent to the rest of the system — nodes never know where their credentials are stored.

Encryption at Rest (Built-in Provider)

For the built-in provider, we use AesCredentialEncryption with AES-256. The master encryption key is managed via environment variables — never stored in appsettings.json or committed to source control.

public sealed class AesCredentialEncryption : ICredentialEncryption
{
    // Encrypts credential data before DB persistence
    // Decrypts on-demand when a node needs access
}

Even if a database backup is compromised, the actual credentials remain encrypted and unusable without the master key.

External Secrets Managers (Vault / OpenBao)

For enterprises that already manage secrets centrally, the VaultCredentialService delegates all secret storage to HashiCorp Vault or OpenBao. Metadata (name, type, associations) stays in the database, but the actual sensitive values live in the vault's KV v2 engine.

This gives you:

Centralized secret rotation without touching Vyshyvanka
Audit trails from the vault itself
Integration with existing enterprise key management

Authentication Providers

Vyshyvanka supports four authentication strategies, selectable at deployment time:

Provider	Session Tokens	Login Flow
Built-in	Local JWT	Username/password via API
Keycloak	External OIDC	Redirect to Keycloak
Authentik	External OIDC	Redirect to Authentik
LDAP	Local JWT	Verify against directory

For OIDC providers, the API validates external tokens and an OidcClaimsTransformation middleware provisions local users on first login. For LDAP, the LdapAuthenticationService verifies credentials against the directory server.

Additionally, API key authentication (X-API-Key header) is always available regardless of the primary provider — useful for CI/CD integrations and automated workflow triggers.

Node-Level Auth Strategies

Our node architecture supports various authentication strategies per credential type:

Basic Auth: Username/password for legacy services
Bearer Tokens: The standard for most modern REST APIs
API Keys: Header or query parameter based
Custom Headers: For services with non-standard auth schemes

Each node that requires credentials declares this with the [RequiresCredential] attribute, and the Designer UI automatically shows the credential picker for that node.

Security Rules We Enforce

Always:

Encrypt credentials at rest (AES-256 or delegate to Vault/OpenBao)
Validate resource ownership via ICurrentUserService before any operation
Use parameterized queries for all database operations
Sanitize user input in expressions to prevent injection

Never:

Return credential values in any API response
Log credentials or sensitive data at any log level
Allow cross-user workflow access without explicit sharing
Store Vault/OpenBao tokens in plain text in config files

Best Practices

Rotate your keys: The credential store allows you to update a secret without modifying workflow logic. The CredentialId reference stays the same.
Use scoped permissions: If a service supports it, create API keys with minimum required permissions.
Choose the right backend: Use Vault/OpenBao for production environments where compliance and audit trails matter. The built-in provider works well for development and small deployments.

Security is not a feature you add at the end; it is the foundation on which everything else is built.

In the next part, we will discuss Part 10: Plugin System Architecture - Extensibility by Design. Stay tuned!

Check out the project source code here: https://github.com/homolibere/Vyshyvanka