DEV Community: Arash Zand

How I Fixed a Silent Bug in Microsoft's Semantic Kernel and Got Merged into Main

Arash Zand — Thu, 30 Jul 2026 22:32:16 +0000

If you've been running reasoning models like Qwen3, phi4-reasoning, or DeepSeek-R1 locally through Semantic Kernel's .NET Ollama connector, you may have hit this: GetTextContentsAsync returns an empty string. No exception. No error. Just silence.

This is the story of how I tracked down that bug, built the fix, and got it merged into microsoft/semantic-kernel main.

The Problem

When a reasoning model has thinking enabled by default, its chain-of-thought output lands in a separate thinking stream — not in the standard response field. Semantic Kernel reads the standard response field. So when thinking is active, GetTextContentsAsync comes back empty even though the model ran successfully.

The only workaround was passing think=false in the Ollama request. But OllamaPromptExecutionSettings had no such property — there was simply no API surface to set it.

This was filed as issue #14078.

The Fix

Three parts: add the Think property, wire it through to the request, add full test coverage.

1. Add `Think` to `OllamaPromptExecutionSettings`

/// <summary>
/// Controls thinking behavior for Ollama reasoning models
/// (deepseek-r1, qwen3, phi4-reasoning).
/// When null, the model's default is used.
/// Set to false to suppress thinking and receive output
/// in the standard response field.
/// </summary>
[JsonPropertyName("think")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public bool? Think
{
    get => this._think;
    set
    {
        this.ThrowIfFrozen();
        this._think = value;
    }
}
private bool? _think;

bool? is intentional — null means "don't send the parameter at all" (let the model decide). WhenWritingNull keeps the serialized request clean when it isn't set. ThrowIfFrozen() maintains the standard SK immutability contract. Clone() was updated to carry the value across.

2. Wire through to `OllamaTextGenerationService`

The change required bumping OllamaSharp from 5.4.12 to 5.4.25 — the version that introduced GenerateRequest.Think and the ThinkValue type. This was a necessary dependency bump, not a convenience one.

3. Tests

Full coverage across both paths:

ThinkPropertyRoundTripsViaSerialization — true and false survive JSON round-trip
ThinkPropertyIsPreservedByClone — Clone() carries the value
ThinkPropertyThrowsWhenFrozen — setter throws after Freeze()
GetTextContentsShouldSendThinkSettingAsync — serialized payload contains the correct think field
GetTextContentsShouldNotSendThinkWhenNotSetAsync — payload omits think when null
GetStreamingTextContentsShouldSendThinkSettingAsync — streaming path also propagates Think

All 115 existing unit tests continued to pass.

How to Use It

Once this ships in a Semantic Kernel release:

var kernel = Kernel.CreateBuilder()
    .AddOllamaTextGeneration("qwen3", new Uri("http://localhost:11434"))
    .Build();

var settings = new OllamaPromptExecutionSettings
{
    Think = false  // suppress thinking, get output in standard response
};

var result = await kernel.InvokePromptAsync(
    "Explain the difference between IEnumerable and IQueryable in C#",
    new KernelArguments(settings)
);

Console.WriteLine(result); // No longer empty!

Set Think = true to explicitly enable thinking (e.g., if you're consuming the thinking stream separately), or leave it null to use the model's default.

What the Contribution Process Taught Me

Read the issue before the code. Issue #14078 was already filed with a clear reproduction case. An existing, acknowledged issue is a green light — the maintainers have validated the problem. You're delivering a solution, not arguing for one.

Match the existing pattern. OllamaPromptExecutionSettings had other nullable properties (TopP, Temperature) with the same JSON serialization pattern. Matching that convention made review smooth — reviewers could see immediately the new property followed the established contract.

Tests aren't optional in SK. The Copilot reviewer flagged two minor things: a nullable simplification and ToLowerInvariant() instead of ToLower(). Both were addressed in a follow-up commit. Having thorough tests from the start meant the logic was never questioned — only the style.

The dependency bump needs justification. Bumping OllamaSharp wasn't just a version change — it was required specifically because 5.4.25 introduced the GenerateRequest.Think API surface. Explaining that in the PR description helped reviewers trust the bump.

The PR

microsoft/semantic-kernel #14122
Merged July 9, 2026 · Reviewed by @rogerbarreto and @westey-m

Semantic Kernel has 28.3k ⭐ — if you're hitting the empty-response issue with reasoning models via Ollama, this fix will be available in the next SK release. And if you find something similar that doesn't work — file the issue, or better yet, fix it.

Originally published on Medium.

The DI Coupling Trap: When .NET Libraries Block Their Own Capabilities

Arash Zand — Thu, 30 Jul 2026 22:29:57 +0000

A pattern that shows up repeatedly in .NET library code — where a public convenience method hardcodes DI resolution and silently blocks entire categories of real-world use.

// General overload — caller supplies the monitor directly
context.EnableFeature(myCustomMonitor);
// ✓ Works. No DI required.

// Library convenience method — hardcodes DI resolution
context.EnableFeature<MyOptions>();
// ✗ InvalidOperationException: No service for type 'IOptionsMonitor`1[MyOptions]'

You can see the feature in the docs. You can find it in the source. But when you try to use it outside the library's preferred infrastructure pattern, you hit an exception. Not a missing feature. An artificial wall built into the public surface while the implementation sits quietly capable behind it.

The Pattern

A library exposes a method like this:

// What callers see — the only public path
public void EnableFeature<TOptions>(string? name = null)
    => _context.EnableFeatureCore(
           _serviceProvider.GetRequiredService<IOptionsMonitor<TOptions>>(),
           name);

// Internal implementation — already general
internal static void EnableFeatureCore<TOptions>(
    IOptionsMonitor<TOptions> monitor,  // accepts anything
    string? name = null)
{
    // doesn't care where monitor came from
}

The public method resolves IOptionsMonitor<TOptions> from DI and passes it to an internal method. The internal method accepts any monitor. But callers can't reach the internal method. They only see the public one, and it demands DI.

How this happens: an engineer writes a general internal implementation, then adds a DI-resolving convenience wrapper for the common case. The convenience method is documented and shipped. The internal method stays internal. A user with a non-standard setup hits the wall.

How to detect it: when a library method doesn't work outside DI, look for this combination:

The public method calls GetRequiredService<T>()
It passes the result to another method that accepts T as a parameter
That other method doesn't care where T came from

If all three are true, there's a missing public overload.

The Four Scenarios It Blocks

1. Third-party configuration sources

Feature flag SDKs (LaunchDarkly, ConfigCat), remote config services (AWS AppConfig, Consul), or custom config stores all have their own change-notification models. Using them forces a workaround:

// ❌ Forced: must register wrapper in DI just to satisfy the library
public class FeatureFlagMonitor<TOptions> : IOptionsMonitor<TOptions> { /* ... */ }
services.AddSingleton<IOptionsMonitor<MyOptions>, FeatureFlagMonitor<MyOptions>>();
context.EnableFeature<MyOptions>(); // DI resolves your wrapper

// ✅ With the general overload:
var monitor = new FeatureFlagMonitor<MyOptions>(featureFlagClient);
context.EnableFeature(monitor); // DI container untouched

The problem isn't implementing IOptionsMonitor<T> — that's reasonable. The problem is being forced to register it in DI when you have no other reason to.

2. Multi-tenant applications

Tenants are discovered at runtime. DI containers are designed for startup-time registration. If a library forces DI resolution, per-tenant configuration becomes genuinely difficult:

// ❌ DI has one global registration — not per-tenant
context.EnableFeature<TenantOptions>(); // gets the GLOBAL monitor — wrong

// ✅ With the general overload:
public SomeFeature GetForTenant(string tenantId)
{
    return _cache.GetOrAdd(tenantId, id =>
    {
        // Create per-tenant monitor at runtime — no startup registration needed
        var monitor = new TenantOptionsMonitor<TenantOptions>(_store, id);
        context.EnableFeature(monitor);
        return BuildFeature(monitor.CurrentValue);
    });
}

3. Shared instances across multiple consumers

Sometimes two components must react to the same change event together — not coincidentally, but by design:

// ✅ One monitor, passed explicitly to both consumers
var sharedMonitor = new RemoteConfigMonitor<PaymentConfig>(configService);

featureA.Initialize("payment-read", (builder, ctx) => ctx.EnableFeature(sharedMonitor));
featureB.Initialize("payment-write", (builder, ctx) => ctx.EnableFeature(sharedMonitor));

// When PaymentConfig changes:
// - Both featureA and featureB receive the same event from the same listener list
// - Guaranteed atomic update — no race, no missed events

4. Testing without infrastructure

This is the most universally felt consequence. A DI-only public API turns unit tests into integration tests:

// ❌ With DI coupling — integration test, timing-dependent, fragile
[Fact]
public void Feature_UpdatesOnConfigChange_WithDI()
{
    var services = new ServiceCollection();
    var config = new ConfigurationBuilder()
        .AddInMemoryCollection(new Dictionary<string, string?> { ["MaxRetries"] = "3" })
        .Build();
    services.Configure<MyOptions>(config);
    // ... more setup ...
    var sp = services.BuildServiceProvider();
    var feature = sp.GetRequiredService<IFeature>();

    ((IConfigurationRoot)config).Reload(); // may or may not fire synchronously
    feature.CurrentValue.MaxRetries.ShouldBe(5); // flaky
}

// ✅ With the general overload — true unit test, synchronous, deterministic
[Fact]
public void Feature_UpdatesOnConfigChange_WithFakeMonitor()
{
    var monitor = new FakeOptionsMonitor<MyOptions>(new() { MaxRetries = 3 });

    var services = new ServiceCollection();
    services.AddFeature("test", (builder, ctx) =>
    {
        ctx.EnableFeature(monitor);
        builder.Configure(monitor.CurrentValue);
    });
    var feature = services.BuildServiceProvider().GetRequiredService<IFeature>();

    monitor.TriggerChange(new() { MaxRetries = 5 }); // synchronous, exact control
    feature.CurrentValue.MaxRetries.ShouldBe(5);     // always passes
}

The Fix: General Form First

Expose the general overload publicly. Make the DI-resolving version a thin wrapper over it:

// 1. The general form — real implementation, accepts anything
public void EnableFeature<TOptions>(IOptionsMonitor<TOptions> monitor, string? name = null)
{
    if (monitor is null) throw new ArgumentNullException(nameof(monitor));
    _context.EnableFeatureCore(monitor, name);
}

// 2. The convenience form — wrapper, not capability
public void EnableFeature<TOptions>(string? name = null)
    => EnableFeature(_serviceProvider.GetRequiredService<IOptionsMonitor<TOptions>>(), name);

Null guard placement: put it in the public method, not the internal one. When the caller passes null, they should see "monitor cannot be null" — not "optionsMonitor cannot be null" from an internal parameter name they never used.

RS0026: adding a second overload with optional parameters triggers the Roslyn analyzer warning "Do not add multiple public overloads with optional parameters." The two overloads have distinct mandatory parameters — no actual ambiguity. Suppress it with a pragma:

#pragma warning disable RS0026
public void EnableFeature<T>(IOptionsMonitor<T> monitor, string? name = null) { /* ... */ }
public void EnableFeature<T>(string? name = null) { /* ... */ }
#pragma warning restore RS0026

The FakeOptionsMonitor Test Helper

Once the general overload exists, a small helper unlocks deterministic testing across your entire test suite:

public sealed class FakeOptionsMonitor<TOptions> : IOptionsMonitor<TOptions>
{
    private readonly List<Action<TOptions, string?>> _listeners = new();

    public FakeOptionsMonitor(TOptions initialValue) => CurrentValue = initialValue;

    public TOptions CurrentValue { get; private set; }
    public TOptions Get(string? name) => CurrentValue;

    public IDisposable? OnChange(Action<TOptions, string?> listener)
    {
        _listeners.Add(listener);
        return new Reg(() => _listeners.Remove(listener));
    }

    /// <summary>Simulates a config change. Synchronous — no timing issues.</summary>
    public void TriggerChange(TOptions newValue, string? name = null)
    {
        CurrentValue = newValue;
        foreach (var listener in _listeners.ToList())
            listener(newValue, name);
    }

    private sealed class Reg(Action onDispose) : IDisposable
    {
        public void Dispose() => onDispose();
    }
}

No DI, no configuration stack, no timing uncertainty. TriggerChange fires synchronously, so your assertions run immediately after.

Before vs. After

Scenario	DI-only API	General overload exposed
Standard IOptions / appsettings	✓ Works	✓ Still works
Feature flag SDKs	✗ Forced DI registration	✓ Pass directly
Remote config (push refresh)	✗ Same forced registration	✓ Instantiate, pass in
Multi-tenant (runtime discovery)	✗ Can't register at startup	✓ Create at runtime
Shared instance across consumers	△ DI singleton (implicit)	✓ Explicit, guaranteed
Unit testing	✗ Integration test required	✓ FakeOptionsMonitor
Existing callers	—	✓ No changes needed

The fix is purely additive. Zero breaking changes.

The Real-World Case: Polly v8

This exact pattern appeared in Polly v8. AddResiliencePipelineContext<TKey>.EnableReloads<TOptions> only accepted DI-resolved monitors. The internal extension method ConfigureBuilderContextExtensions.EnableReloads already accepted a monitor directly. The public surface was narrower than the implementation.

The fix — add one overload, add a null guard, suppress RS0026, update the public API surface file, write two tests:

// New overload — Polly v8, App-vNext/Polly PR #3140
public void EnableReloads<TOptions>(IOptionsMonitor<TOptions> monitor, string? name = null)
{
    Guard.NotNull(monitor);
    RegistryContext.EnableReloads(monitor, name);
}

// Original overload — unchanged, all existing callers unaffected
public void EnableReloads<TOptions>(string? name = null)
    => RegistryContext.EnableReloads(
           ServiceProvider.GetRequiredService<IOptionsMonitor<TOptions>>(),
           name);

Three files changed. 111 existing tests continued to pass. All four blocked scenarios above were unblocked.

Principles for Library API Design

General form first. Write the parameterized implementation, then add convenience wrappers on top. Never let a wrapper become the only path to the capability it wraps.

Infrastructure is opt-in. DI, configuration, logging — these should be integration points, not prerequisites. A library feature shouldn't require a DI container to be usable.

If your method resolves, expose an overload that accepts. Any time a public method calls GetRequiredService<T>() and passes the result elsewhere, ask whether callers should be able to supply T directly. The answer is almost always yes.

Testability is correctness. If you can't test a feature without assembling infrastructure, the API surface is incorrectly coupled to its integration points.

Document the fix, not the workaround. "Register IOptionsMonitor as a singleton pointing to your custom implementation" is a workaround for a broken API surface. When the fix is one overload, add the overload.

Originally published on Medium.

Migrating 200 Million Records in .NET: From Row-by-Row to a Crash-Safe Batched Pipeline

Arash Zand — Thu, 30 Jul 2026 22:28:38 +0000

A real-world walkthrough of how we rewrote a data migration four times before it was production-ready.

We had to move email history records out of a legacy SQL Server system (16 regional databases, ~200 million rows total) into a PostgreSQL privacy service already running in production. The job had to be idempotent, restartable without duplicates, survive network drops, and finish fast enough to matter.

We went through five versions. Here's what changed each time and why.

The Setup

Source (SQL Server — 16 regional DBs):

CREATE TABLE [dbo].[PlayerEmailHistory] (
    [HistoryId] INT           NOT NULL IDENTITY(1,1) PRIMARY KEY,
    [PlayerId]  INT           NOT NULL,
    [Email]     NVARCHAR(500) NULL,   -- nullable, some accounts wiped
    [ChangedAt] DATETIME      NULL    -- nullable, early records have no timestamp
);

CREATE TABLE [dbo].[PlayerProfileMap] (
    [PlayerId]  INT         NOT NULL PRIMARY KEY,
    [ProfileId] VARCHAR(50) NULL  -- format: "EU|12345"
);

Target (PostgreSQL):

CREATE TABLE "EmailHistory" (
    "Id"        uuid                     NOT NULL PRIMARY KEY,
    "LoginId"   text                     NOT NULL,
    "ProfileId" text                     NOT NULL,
    "Email"     text                     NOT NULL,
    "ChangedBy" text                     NOT NULL,
    "DateUtc"   timestamp with time zone NOT NULL
);
-- Already had a unique index from a previous migration:
CREATE UNIQUE INDEX "UX_EmailHistory_ProfileId_Email_DateUtc"
    ON "EmailHistory" ("ProfileId", "Email", "DateUtc");

Column mapping:

Target	Source	Resolution
`Id`	—	New UUID v7
`LoginId`	`PlayerProfileMap.ProfileId`	JOIN on PlayerId
`ProfileId`	`PlayerProfileMap.ProfileId`	Same as LoginId
`Email`	`PlayerEmailHistory.Email`	Validate
`ChangedBy`	—	`"legacy-{region}:{HistoryId}"`
`DateUtc`	`ChangedAt`	Cast to UTC; `UnixEpoch` if null

V1: Load Everything, Insert Row by Row

The obvious first attempt. Read all rows into memory, loop, insert each one with WHERE NOT EXISTS.

What goes wrong immediately:

Memory — 200M rows in a List<> = >20 GB heap. Process dies with OutOfMemoryException before the first insert.
Speed — at 1 ms per row-insert round trip: 200,000 seconds ≈ 7 days.
Wrong dedup key — WHERE NOT EXISTS checks (ProfileId, Email, DateUtc). But DateUtc is UnixEpoch for all rows with null ChangedAt. Three rows for the same player with no timestamp → same tuple → two silently dropped.
No resume — crash at row 15M, start over from zero.
Empty ChangedBy — no way to trace which rows came from which source.

V2: Fix the Dedup Key

Before touching performance, fix correctness.

Key insight: HistoryId is the source PK — unique, stable, never null. Embed it in ChangedBy as "legacy-{region}:{HistoryId}". Now the dedup check becomes:

WHERE NOT EXISTS (SELECT 1 FROM "EmailHistory" WHERE "ChangedBy" = @ChangedBy)

This handles the null-timestamp collision and gives full auditability — any row in the target traces back to an exact source row.

Why the region prefix matters: HistoryId starts at 1 in every regional DB. DK row HistoryId=1 and SE row HistoryId=1 go into the same EU PostgreSQL cluster. Without "legacy-DK:1" vs "legacy-SE:1", one silently disappears.

V2 fixes correctness. Memory and speed are unchanged.

V3: Batched Read + Batch Upsert (~37x faster)

Two changes that together give an order-of-magnitude speedup.

Cursor-based read instead of loading everything:

SELECT TOP (@BatchSize)
    eh.HistoryId, eh.PlayerId, eh.Email, eh.ChangedAt, pm.ProfileId
FROM dbo.PlayerEmailHistory eh WITH (NOLOCK)
LEFT JOIN dbo.PlayerProfileMap pm WITH (NOLOCK)
    ON pm.PlayerId = eh.PlayerId
WHERE eh.HistoryId > @LastHistoryId
ORDER BY eh.HistoryId

Start at @LastHistoryId = 0, advance after each batch. O(1) per batch, no OFFSET penalty.

Batch upsert instead of row-by-row:

private static async Task<ulong> UpsertChunkAsync(
    NpgsqlConnection conn, TargetRecord[] chunk)
{
    var sql        = new StringBuilder();
    var parameters = new DynamicParameters();

    sql.Append("""
        INSERT INTO "EmailHistory"
            ("Id","LoginId","ProfileId","Email","ChangedBy","DateUtc")
        VALUES
        """);

    for (var i = 0; i < chunk.Length; i++)
    {
        if (i > 0) sql.Append(',');
        sql.Append($"(@id{i},@li{i},@pi{i},@em{i},@cb{i},@du{i})");
        parameters.Add($"id{i}", chunk[i].Id);
        parameters.Add($"li{i}", chunk[i].LoginId);
        parameters.Add($"pi{i}", chunk[i].ProfileId);
        parameters.Add($"em{i}", chunk[i].Email);
        parameters.Add($"cb{i}", chunk[i].ChangedBy);
        parameters.Add($"du{i}", chunk[i].DateUtc);
    }
    sql.Append(" ON CONFLICT DO NOTHING;");

    return (ulong)await conn.ExecuteAsync(
        sql.ToString(), parameters, commandTimeout: 600);
}

Why ON CONFLICT DO NOTHING without a target?
The table already has UX_EmailHistory_ProfileId_Email_DateUtc from a previous migration. ON CONFLICT ("ChangedBy") DO NOTHING only handles conflicts on that index — any conflict on the existing index still throws 23505. Omitting the target tells PostgreSQL to silently skip any constraint violation.

Chunk size: 1,000 rows. Very large chunks can exceed SQL string limits; 500–1,500 is a safe range.

V4: Checkpoint (The Production Version)

V3 takes ~4.6 hours per full run. A mid-run crash means starting over. For 16 regions that's painful.

Add a checkpoint table to save the last processed HistoryId after every batch:

CREATE TABLE IF NOT EXISTS "MigrationCheckpointLegacy" (
    "MigrationName" text PRIMARY KEY,
    "LastHistoryId" int  NOT NULL,
    "UpdatedAtUtc"  timestamp with time zone NOT NULL DEFAULT NOW()
);

Name it specifically. The privacy service already had EmailMigrationCheckpoint from a MySQL migration with different columns. CREATE TABLE IF NOT EXISTS silently succeeds — the table exists — then your column query fails with 42703: column "LastHistoryId" does not exist. Naming it MigrationCheckpointLegacy avoids the collision.

Add a partial unique index instead of a full one:

CREATE UNIQUE INDEX IF NOT EXISTS "UX_EmailHistory_ChangedBy_Legacy"
ON "EmailHistory" ("ChangedBy")
WHERE "ChangedBy" LIKE 'legacy-%';

A full unique index on ChangedBy fails if any two existing rows (from the previous migration) share the same value. A partial index scoped to our own rows always succeeds.

Checkpoint save — upsert after every batch:

await conn.ExecuteAsync("""
    INSERT INTO "MigrationCheckpointLegacy"
        ("MigrationName", "LastHistoryId", "UpdatedAtUtc")
    VALUES (@MigrationName, @LastHistoryId, NOW())
    ON CONFLICT ("MigrationName")
    DO UPDATE SET
        "LastHistoryId" = EXCLUDED."LastHistoryId",
        "UpdatedAtUtc"  = EXCLUDED."UpdatedAtUtc"
    """, new { MigrationName = migrationKey, LastHistoryId = lastHistoryId });

The --region flag — run one region at a time:

// Program.cs
var regionFilter = GetArg(args, "--region");
if (regionFilter is not null)
    options.Regions = [options.Regions.First(
        r => r.Name.Equals(regionFilter, StringComparison.OrdinalIgnoreCase))];

dotnet run -- --region DK
dotnet run -- --region SE

V5: Channel-Based Producer-Consumer Pipeline

V4 is sequential: read batch N → write batch N → read batch N+1. Both I/O operations never overlap.

System.Threading.Channels (built into .NET, no NuGet required) lets us pipeline them:

V4: [read N]──[write N]──[read N+1]──[write N+1]──...
V5: [read N]──[read N+1]──[read N+2]──...
         [write N]──[write N+1]──[write N+2]──...

Channel setup — bounded to prevent lookahead memory explosion:

private record BatchPayload(List<TargetRecord> Records, int LastHistoryId);

private static Channel<BatchPayload> CreateChannel() =>
    Channel.CreateBounded<BatchPayload>(
        new BoundedChannelOptions(capacity: 3)
        {
            SingleWriter = true,
            SingleReader = true,
            FullMode = BoundedChannelFullMode.Wait
        });

Capacity 3 = 3 pre-read batches maximum. Without the bound, the producer runs arbitrarily far ahead — same memory problem as V1.

Producer — finally block is critical:

private async Task ProduceAsync(
    RegionConfig region, int startCheckpoint,
    ChannelWriter<BatchPayload> writer, Action<ulong> onSkipped)
{
    var checkpoint = startCheckpoint;
    try
    {
        while (true)
        {
            var batch = await ReadSourceBatchAsync(region, checkpoint);
            if (batch.Count == 0) break;

            var (valid, skipped) = FilterAndTransform(batch, region.Name);
            onSkipped(skipped);
            checkpoint = batch[^1].HistoryId;

            await writer.WriteAsync(new BatchPayload(valid, checkpoint));
        }
    }
    finally
    {
        writer.Complete(); // critical: consumer must exit even if producer throws
    }
}

Consumer:

private async Task<(ulong inserted, ulong duplicates)> ConsumeAsync(
    NpgsqlConnection conn, string migrationKey,
    ChannelReader<BatchPayload> reader)
{
    var inserted = 0UL; var duplicates = 0UL;

    await foreach (var payload in reader.ReadAllAsync())
    {
        var n = await UpsertBatchAsync(conn, payload.Records);
        inserted   += n;
        duplicates += (ulong)payload.Records.Count - n;
        await SaveCheckpointAsync(conn, migrationKey, payload.LastHistoryId);
    }
    return (inserted, duplicates);
}

Wiring:

var channel      = CreateChannel();
var producerTask = ProduceAsync(region, checkpoint, channel.Writer,
                                count => Interlocked.Add(ref skipped, count));
var consumerTask = ConsumeAsync(conn, migrationKey, channel.Reader);

await Task.WhenAll(producerTask, consumerTask);

Interlocked.Add on the skipped counter — producer and consumer run concurrently, shared state needs atomic updates.

Benchmark Comparison

Version	Peak Memory	Insert Rate	Est. Time (200M)	Crash Recovery
V1: row-by-row, all in memory	>20 GB	~320 rows/sec	~7 days	Start over
V2: row-by-row, fixed dedup	>20 GB	~290 rows/sec	~8 days	Start over
V3: batch read + batch upsert	~50 MB	~12,000 rows/sec	~4.6 hours	Start over
V4: batched + checkpoint	~50 MB	~11,500 rows/sec	~4.8 hours	Resume from last batch
V5: channel pipeline + checkpoint	~55 MB	~17,000 rows/sec	~3.3 hours	Resume from last batch

Our production run across all 16 regions took ~47 hours total. The checkpoint system recovered from two mid-run network failures without data loss or manual intervention.

Production War Stories

1. SSL Certificate Not Trusted
Microsoft.Data.SqlClient v4+ enforces encrypted connections and validates server certs. RDS self-signed cert → connection fails. Fix: TrustServerCertificate=True in the SQL Server connection string (fine for an internal migration script, not for application code).

2. Checkpoint Table Collision
The privacy service already had a migration checkpoint table with different columns. CREATE TABLE IF NOT EXISTS silently succeeded, then our column query failed with 42703. Fix: name your checkpoint table specifically for your migration (MigrationCheckpointLegacy).

3. Pre-Existing Unique Constraint
ON CONFLICT ("ChangedBy") DO NOTHING only handles conflicts on that index. A conflict on the existing UX_EmailHistory_ProfileId_Email_DateUtc still throws 23505. Fix: use ON CONFLICT DO NOTHING without specifying a target.

4. Cross-Region HistoryId Collision
HistoryId starts at 1 in every regional DB. Two regions, same HistoryId, same cluster → one row silently lost. Fix: include region in ChangedBy ("legacy-DK:1" vs "legacy-SE:1").

5. Error 4060 from SQL Server
The server connection succeeded but the login doesn't have access to that specific database. Not a network issue — a GRANT issue. Update credentials per region.

6. Network Drop Mid-Migration
System.IO.IOException: An existing connection was forcibly closed by the remote host. Without checkpoint: restart from scratch. With checkpoint: resume from last saved batch, ~1,000 rows re-processed as duplicates (silently skipped), re-run completed in under a minute.

Key Takeaways

Performance: Row-by-row insert doesn't scale. Batch upsert with ON CONFLICT DO NOTHING is an order of magnitude faster. Read with a cursor (WHERE id > @lastId), not OFFSET.

Resilience: Save a checkpoint after every batch. Name checkpoint tables specifically so they don't collide with other migrations. Use partial unique indexes when sharing a table with other writers.

Pipelining: If read and write are both I/O-bound, System.Threading.Channels lets them overlap. Bound the channel. Call writer.Complete() in a finally block. Embed the checkpoint value in the batch payload to avoid shared mutable state.

Operations: Run regions one at a time with a --region flag. Add a --progress command. Set explicit command timeouts (commandTimeout: 600) — Dapper's default 30s isn't enough for batch inserts into busy tables.

Originally published on Medium. The full article includes complete V3 and V4 migrator code, configuration models, and the --progress CLI command.

Architecting a GenAI FAQ System with AWS Bedrock, Python, TypeScript, C# .NET and Blazor

Arash Zand — Thu, 30 Jul 2026 22:25:57 +0000

This is a step-by-step guide to building a production-ready Generative AI FAQ system using AWS Bedrock for LLM + embeddings, OpenSearch for semantic search (RAG), Lambda for processing, a C# .NET API, and a Blazor UI. Bedrock Guardrails enforce compliance throughout.

Architecture Overview

User → Blazor UI → C# .NET API → API Gateway → TypeScript Lambda
                                                      ↓
                                              AWS Bedrock (Claude)
                                                      ↓
                                         OpenSearch (k-NN semantic search)
                                                      ↑
                             S3 → Python Lambda (clean) → Python Lambda (embed)

Components:

S3 — stores raw and cleaned FAQ documents
Python Lambda (CleanDocuments) — triggered on S3 upload, cleans text, publishes to SNS
Python Lambda (GenerateEmbeddings) — reads cleaned docs, generates Titan embeddings, stores in OpenSearch
OpenSearch — k-NN index for semantic document retrieval
TypeScript Lambda (ProcessQuestion) — embeds user question, searches OpenSearch (RAG), calls Claude with Guardrails
C# .NET API — RESTful layer between Blazor and API Gateway
Blazor UI — web interface for asking questions
Bedrock Guardrails — blocks off-topic or prohibited responses

Step 1: AWS Infrastructure Setup

S3 Bucket

Create bucket faq-documents-2025 (us-east-1), private, versioning on, SSE-S3 encryption. Create two folders: faqs/ (raw) and processed/ (cleaned).

Sample FAQ document for faqs/product_faq.txt:

Q: What is the product warranty?
A: The product comes with a one-year warranty covering manufacturing defects.
Q: How do I contact support?
A: Email support@company.com or call 1-800-555-1234.

OpenSearch Domain

Create domain faq-embeddings-2025, t3.small.search, fine-grained access control. Then create the k-NN index:

curl -X PUT "https://<domain>/faq_index" \
  -H 'Content-Type: application/json' \
  -u admin:password \
  -d '{
    "settings": { "index": { "knn": true, "knn.space_type": "cosinesimil" } },
    "mappings": {
      "properties": {
        "text":       { "type": "text" },
        "embedding":  { "type": "knn_vector", "dimension": 1536 },
        "source_key": { "type": "keyword" }
      }
    }
  }'

The dimension: 1536 matches Bedrock's Titan Embeddings model output.

Bedrock

In the Bedrock Console (us-east-1), request access to Anthropic Claude v2 and Amazon Titan Embeddings G1 — Text.

Create Guardrail FaqGuardrail:

Content filters: hate speech, violence, profanity — threshold: Medium
Topical filter: allow keywords product, support, warranty, features, faq; block unrelated topics

IAM Role for Lambda

Create FaqLambdaRole with policies for S3 (s3:GetObject, s3:PutObject, s3:ListBucket), OpenSearch (es:ESHttpPut, es:ESHttpGet, es:ESHttpPost), Bedrock (bedrock:InvokeModel, bedrock:ApplyGuardrail), and AWSLambdaBasicExecutionRole.

API Gateway

Create REST API FaqApi, resource /faq, POST method with Lambda proxy integration pointing to ProcessQuestion. Deploy to stage prod.

Step 2: Data Cleaning Lambda (Python)

Triggered by S3 upload to faqs/. Cleans text, saves to processed/, notifies GenerateEmbeddings via SNS.

import boto3, re, json, logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
s3_client = boto3.client('s3')

def clean_text(text):
    text = re.sub(r'\s+', ' ', text.strip())
    text = re.sub(r'[^\w\s.,!?]', '', text)
    return text.lower()

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key    = event['Records'][0]['s3']['object']['key']
    logger.info(f"Processing: {key}")

    raw_text     = s3_client.get_object(Bucket=bucket, Key=key)['Body'].read().decode('utf-8')
    cleaned_text = clean_text(raw_text)
    cleaned_key  = f"processed/{key.split('/')[-1]}"

    s3_client.put_object(Bucket=bucket, Key=cleaned_key, Body=cleaned_text.encode('utf-8'))

    boto3.client('sns').publish(
        TopicArn='arn:aws:sns:us-east-1:YOUR_ACCOUNT:FaqProcessingTopic',
        Message=json.dumps({'bucket': bucket, 'cleaned_key': cleaned_key})
    )
    return {'statusCode': 200, 'body': json.dumps({'cleaned_key': cleaned_key})}

Deploy: Runtime Python 3.9, role FaqLambdaRole, S3 trigger on faqs/ prefix, timeout 30s, memory 256MB.

Step 3: Embedding Generation Lambda (Python)

Triggered by SNS from the cleaning Lambda. Generates Titan embeddings and stores them in OpenSearch.

import boto3, json, logging
from opensearchpy import OpenSearch, RequestsHttpConnection
from aws_requests_auth.boto_utils import BotoAWSRequestsAuth

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

s3_client      = boto3.client('s3')
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')

DOMAIN   = 'search-faq-embeddings-2025-XXXXXXXXXX.us-east-1.es.amazonaws.com'
auth     = BotoAWSRequestsAuth(aws_host=DOMAIN, aws_region='us-east-1', aws_service='es')
os_client = OpenSearch(
    hosts=[{'host': DOMAIN, 'port': 443}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

def generate_embedding(text):
    response  = bedrock_client.invoke_model(
        modelId='amazon.titan-embed-text-v1',
        body=json.dumps({'inputText': text})
    )
    return json.loads(response['body'].read())['embedding']

def lambda_handler(event, context):
    msg         = json.loads(event['Records'][0]['Sns']['Message'])
    bucket      = msg['bucket']
    cleaned_key = msg['cleaned_key']

    text      = s3_client.get_object(Bucket=bucket, Key=cleaned_key)['Body'].read().decode('utf-8')
    embedding = generate_embedding(text)

    os_client.index(index='faq_index', body={
        'text':       text,
        'embedding':  embedding,
        'source_key': cleaned_key
    })
    logger.info(f"Stored embedding for {cleaned_key}")
    return {'statusCode': 200}

Deploy: Runtime Python 3.9, SNS trigger on FaqProcessingTopic, timeout 60s, memory 512MB. Package with pip install opensearch-py aws-requests-auth -t ..

Step 4: Question Processing Lambda (TypeScript)

Embeds the user question, runs k-NN search on OpenSearch, builds a RAG prompt, and calls Claude with Guardrails.

import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { Client } from '@opensearch-project/opensearch';
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import { BedrockRuntimeClient, InvokeModelCommand } from '@aws-sdk/client-bedrock-runtime';

const DOMAIN = 'https://search-faq-embeddings-2025-XXXXXXXX.us-east-1.es.amazonaws.com';

const osClient = new Client({
  node: DOMAIN,
  auth: { credentials: defaultProvider() },
});

const bedrockClient = new BedrockRuntimeClient({ region: 'us-east-1' });

export const handler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  const { question } = JSON.parse(event.body || '{}');
  if (!question) return { statusCode: 400, body: JSON.stringify({ error: 'Question required' }) };

  try {
    // Embed the question
    const questionEmbedding = await generateEmbedding(question);

    // Semantic search — retrieve top 3 relevant documents
    const searchResult = await osClient.search({
      index: 'faq_index',
      body: {
        query: {
          knn: { embedding: { vector: questionEmbedding, k: 3 } }
        }
      }
    });

    const context = searchResult.body.hits.hits
      .map((hit: any) => hit._source.text)
      .join(' ');

    // Call Claude with Guardrails applied
    const response = await bedrockClient.send(new InvokeModelCommand({
      modelId: 'anthropic.claude-v2',
      body: JSON.stringify({
        prompt: `Answer based on context. Be concise.\nQuestion: ${question}\nContext: ${context}`,
        max_tokens_to_sample: 500,
        guardrailIdentifier: 'guardrail-12345', // replace with your guardrail ID
        guardrailVersion: '1',
      }),
    }));

    const answer = JSON.parse(Buffer.from(response.body).toString()).completion;
    return { statusCode: 200, body: JSON.stringify({ answer }) };

  } catch (error) {
    console.error(error);
    return { statusCode: 500, body: JSON.stringify({ error: 'Internal server error' }) };
  }
};

async function generateEmbedding(text: string): Promise<number[]> {
  const response = await bedrockClient.send(new InvokeModelCommand({
    modelId: 'amazon.titan-embed-text-v1',
    body: JSON.stringify({ inputText: text }),
  }));
  return JSON.parse(Buffer.from(response.body).toString()).embedding;
}

Deploy: Runtime Node.js 18.x, compile with npx tsc, package with zip -r function.zip dist node_modules, timeout 60s, memory 512MB.

Step 5: C# .NET API

Sits between Blazor and API Gateway. Validates input, forwards to Lambda, returns the answer.

[ApiController]
[Route("api/faq")]
public class FaqController : ControllerBase
{
    private readonly HttpClient _httpClient;
    private readonly string _lambdaUrl = "https://YOUR_API_GW_URL/prod/faq";

    public FaqController(IHttpClientFactory httpClientFactory)
        => _httpClient = httpClientFactory.CreateClient();

    [HttpPost("ask")]
    public async Task<IActionResult> AskQuestion([FromBody] QuestionRequest request)
    {
        if (string.IsNullOrEmpty(request.Question))
            return BadRequest(new { Error = "Question is required" });

        var content = new StringContent(
            JsonSerializer.Serialize(new { question = request.Question }),
            Encoding.UTF8, "application/json");

        var response = await _httpClient.PostAsync(_lambdaUrl, content);
        response.EnsureSuccessStatusCode();

        var result = JsonSerializer.Deserialize<QuestionResponse>(
            await response.Content.ReadAsStringAsync());
        return Ok(result);
    }
}

public record QuestionRequest(string Question);
public record QuestionResponse(string Answer);

Program.cs — CORS for Blazor:

builder.Services.AddHttpClient();
builder.Services.AddCors(o => o.AddPolicy("AllowBlazor", b =>
    b.WithOrigins("https://localhost:5001").AllowAnyMethod().AllowAnyHeader()));

app.UseCors("AllowBlazor");

Deploy to Elastic Beanstalk: eb init -p dotnetcore FaqApi && eb create faq-api-env && eb deploy.

Step 6: Blazor UI

@page "/faq"
@using System.Net.Http.Json
@inject HttpClient Http

<h3>Frequently Asked Questions</h3>

<div class="form-group">
    <label>Your Question:</label>
    <input class="form-control" @bind="question" placeholder="Enter your question" />
</div>
<button class="btn btn-primary mt-2" @onclick="SubmitQuestion">Submit</button>

@if (!string.IsNullOrEmpty(answer))
{
    <div class="mt-4"><h4>Answer:</h4><p>@answer</p></div>
}
@if (!string.IsNullOrEmpty(error))
{
    <div class="mt-4 text-danger"><p>@error</p></div>
}

@code {
    private string question = "", answer = "", error = "";

    private async Task SubmitQuestion()
    {
        error = "";
        try
        {
            var response = await Http.PostAsJsonAsync(
                "http://YOUR_API_URL/api/faq/ask",
                new { Question = question });
            response.EnsureSuccessStatusCode();
            var result = await response.Content.ReadFromJsonAsync<QuestionResponse>();
            answer = result!.Answer;
        }
        catch (Exception ex) { error = $"Error: {ex.Message}"; }
    }

    record QuestionResponse(string Answer);
}

Step 7: Security & Production Checklist

IAM least privilege on all roles — no wildcards in production
SSE-S3 on the bucket, HTTPS everywhere
Update FaqGuardrail regularly with new prohibited terms
Enable CloudWatch logs for all Lambdas and API Gateway
S3 lifecycle rule: archive processed/ to Glacier after 30 days
OpenSearch: VPC access in production, not public endpoint
Scale OpenSearch index replicas: "number_of_replicas": 2

Step 8: IaC with CloudFormation

Automate the entire setup:

Resources:
  FaqBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: faq-documents-2025
      VersioningConfiguration:
        Status: Enabled

  FaqLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: FaqLambdaRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: { Service: lambda.amazonaws.com }
            Action: sts:AssumeRole
      Policies:
        - PolicyName: S3Access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: [s3:GetObject, s3:PutObject, s3:ListBucket]
                Resource:
                  - arn:aws:s3:::faq-documents-2025
                  - arn:aws:s3:::faq-documents-2025/*

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name FaqStack \
  --region us-east-1

Common Troubleshooting

Symptom	Fix
No embeddings in OpenSearch	Check CloudWatch logs for `GenerateEmbeddings`
Guardrail blocks valid questions	Widen topical filter keywords in Bedrock Console
API Gateway 502	Verify proxy integration is enabled on the POST method
Blazor CORS error	Check `WithOrigins` in `Program.cs`
Bedrock access denied	Confirm model access is approved in Bedrock Console
OpenSearch connection fails	Verify domain endpoint and IAM auth credentials

Originally published on Medium.

Implementing a Hierarchical Cache for Tagging in C# .NET

Arash Zand — Thu, 30 Jul 2026 22:23:43 +0000

Tagging is everywhere — CMS platforms, e-commerce, social media. A flat tagging system works fine at small scale, but as soon as you need hierarchical relationships ("C#" is under "Programming" which is under "Technology"), the basic approach falls apart: multiple queries, redundant data, and cascading maintenance headaches.

This article walks through the problem, then builds a hierarchical cache that organizes tags in a tree structure using IMemoryCache — turning multi-query reads into single in-memory traversals.

The Problem: Basic Tagging

A flat system has three tables — Item, Tag, ItemTag (junction). Simple enough:

public class Tag
{
    [Key] public int Id { get; set; }
    public string Name { get; set; }
    public List<ItemTag> ItemTags { get; set; }
}

public class ItemTag
{
    public int ItemId { get; set; }
    public int TagId  { get; set; }
}

CRUD operations work, but as soon as you want "all items tagged with 'Programming' or any of its children", you need multiple queries:

// Three separate DB hits just to get all programming-related items
var programmingItems = manager.GetItemsForTag(2); // "Programming"
var csharpItems      = manager.GetItemsForTag(3); // "C#"
var pythonItems      = manager.GetItemsForTag(4); // "Python"
var allItems = programmingItems.Union(csharpItems).Union(pythonItems).ToList();

And tagging a new item with "C#" means manually also tagging it with "Programming" and "Technology" to keep things consistent. This breaks down fast.

Root causes:

No native hierarchy — relationships are manual
Read-heavy queries hit the DB every time
Write operations don't propagate up the tree
Deleting a parent tag requires careful cascading logic

The Solution: Hierarchical Cache

The idea: load the tag tree into memory as a graph of TagNode objects. Reads are tree traversals (microseconds). Writes update both the DB and the cache. Parent nodes automatically accumulate items from their children.

Tag Model — add `ParentId`

public class Tag
{
    [Key] public int Id { get; set; }
    public string Name { get; set; }
    public int? ParentId { get; set; }
    public Tag Parent { get; set; }
    public List<Tag> Children { get; set; }
    public List<ItemTag> ItemTags { get; set; }
}

TagNode — the in-memory tree node

public class TagNode
{
    public int Id { get; set; }
    public string Name { get; set; }
    public TagNode Parent { get; set; }
    public List<TagNode> Children { get; set; }
    public List<int> AssociatedItems { get; set; }

    public TagNode(int id, string name)
    {
        Id = id;
        Name = name;
        Children = new List<TagNode>();
        AssociatedItems = new List<int>();
    }
}

TagCacheManager — the full implementation

public class TagCacheManager
{
    private readonly IMemoryCache _cache;
    private readonly TagContext _context;
    private const string CacheKey = "TagHierarchy";

    public TagCacheManager(IMemoryCache cache, TagContext context)
    {
        _cache = cache;
        _context = context;
    }

    // Load entire tag tree from DB into memory
    public void LoadFromDatabase()
    {
        var tags = _context.Tags
            .Include(t => t.Children)
            .Include(t => t.ItemTags)
            .ToList();

        var root = new TagNode(0, "Root");
        var nodeMap = new Dictionary<int, TagNode>();

        foreach (var tag in tags)
        {
            var node = new TagNode(tag.Id, tag.Name);
            node.AssociatedItems = tag.ItemTags.Select(it => it.ItemId).ToList();
            nodeMap[tag.Id] = node;
        }

        foreach (var tag in tags)
        {
            var node = nodeMap[tag.Id];
            if (tag.ParentId.HasValue && nodeMap.ContainsKey(tag.ParentId.Value))
            {
                node.Parent = nodeMap[tag.ParentId.Value];
                nodeMap[tag.ParentId.Value].Children.Add(node);
            }
            else
            {
                root.Children.Add(node);
                node.Parent = root;
            }
        }

        SetCache(root);
    }

    // Create tag — persists to DB, updates cache
    public void CreateTag(int id, string name, int? parentId)
    {
        _context.Tags.Add(new Tag { Id = id, Name = name, ParentId = parentId });
        _context.SaveChanges();

        if (!_cache.TryGetValue(CacheKey, out TagNode root))
        {
            LoadFromDatabase();
            return;
        }

        var newNode = new TagNode(id, name);
        if (parentId.HasValue)
        {
            var parent = FindNode(root, parentId.Value);
            if (parent != null)
            {
                newNode.Parent = parent;
                parent.Children.Add(newNode);
            }
        }
        else
        {
            root.Children.Add(newNode);
            newNode.Parent = root;
        }

        SetCache(root);
    }

    // Associate item — propagates up to all ancestors automatically
    public void AssociateItem(int tagId, int itemId)
    {
        _context.ItemTags.Add(new ItemTag { TagId = tagId, ItemId = itemId });
        _context.SaveChanges();

        if (!_cache.TryGetValue(CacheKey, out TagNode root)) return;

        var node = FindNode(root, tagId);
        if (node == null) return;

        // Add to this node
        if (!node.AssociatedItems.Contains(itemId))
            node.AssociatedItems.Add(itemId);

        // Propagate up the tree — parent and all ancestors get this item
        var current = node.Parent;
        while (current != null)
        {
            if (!current.AssociatedItems.Contains(itemId))
                current.AssociatedItems.Add(itemId);
            current = current.Parent;
        }

        SetCache(root);
    }

    // Read — single in-memory traversal, includes all descendants
    public List<int> GetItemsForTag(int tagId)
    {
        if (!_cache.TryGetValue(CacheKey, out TagNode root))
            return new List<int>();

        var node = FindNode(root, tagId);
        if (node == null) return new List<int>();

        var result = new List<int>();
        CollectItems(node, result);
        return result.Distinct().ToList();
    }

    // Update name — DB + cache
    public void UpdateTag(int id, string newName)
    {
        var tag = _context.Tags.Find(id);
        if (tag == null) return;

        tag.Name = newName;
        _context.SaveChanges();

        if (!_cache.TryGetValue(CacheKey, out TagNode root)) return;
        var node = FindNode(root, id);
        if (node != null) { node.Name = newName; SetCache(root); }
    }

    // Delete — removes from DB, detaches from parent in cache
    public void DeleteTag(int id)
    {
        var tag = _context.Tags
            .Include(t => t.ItemTags)
            .FirstOrDefault(t => t.Id == id);

        if (tag == null) return;

        _context.ItemTags.RemoveRange(tag.ItemTags);
        _context.Tags.Remove(tag);
        _context.SaveChanges();

        if (!_cache.TryGetValue(CacheKey, out TagNode root)) return;
        var node = FindNode(root, id);
        if (node != null)
        {
            node.Parent?.Children.Remove(node);
            SetCache(root);
        }
    }

    private void SetCache(TagNode root) =>
        _cache.Set(CacheKey, root, new MemoryCacheEntryOptions
        {
            SlidingExpiration = TimeSpan.FromMinutes(30)
        });

    private TagNode FindNode(TagNode node, int id)
    {
        if (node.Id == id) return node;
        foreach (var child in node.Children)
        {
            var found = FindNode(child, id);
            if (found != null) return found;
        }
        return null;
    }

    private void CollectItems(TagNode node, List<int> result)
    {
        result.AddRange(node.AssociatedItems);
        foreach (var child in node.Children)
            CollectItems(child, result);
    }
}

Registration

services.AddDbContext<TagContext>(options =>
    options.UseSqlServer("YourConnectionString"));
services.AddMemoryCache();
services.AddSingleton<TagCacheManager>();

Usage Example

manager.LoadFromDatabase();

// Build hierarchy: Technology > Programming > C#, Python
//                             > Hardware > CPUs
manager.CreateTag(1, "Technology",   null);
manager.CreateTag(2, "Programming",  1);
manager.CreateTag(3, "C#",           2);
manager.CreateTag(4, "Python",       2);
manager.CreateTag(5, "Hardware",     1);
manager.CreateTag(6, "CPUs",         5);

// Associate items — propagation happens automatically
manager.AssociateItem(3, 101); // post 101 → "C#" → also "Programming" + "Technology"
manager.AssociateItem(4, 102); // post 102 → "Python" → also "Programming" + "Technology"
manager.AssociateItem(6, 103); // post 103 → "CPUs" → also "Hardware" + "Technology"

Console.WriteLine(string.Join(", ", manager.GetItemsForTag(2))); // Programming: 101, 102
Console.WriteLine(string.Join(", ", manager.GetItemsForTag(1))); // Technology:  101, 102, 103

Performance Comparison

Operation	Basic System	Hierarchical Cache
Read "Technology" (3 subtrees)	~3 DB queries, ~100ms	1 tree traversal, ~1ms
Tag item with "C#"	Manual inserts for each related tag	Single insert + in-memory propagation
Delete parent tag	Manual cascade logic required	Detach node, DB handles the rest

Scaling Beyond In-Memory

Redis for distributed environments:

services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "localhost:6379";
});

Serialize the tag tree to JSON, store in Redis. All app instances share the same cache.

Thread safety for high concurrency:
Replace List<int> in AssociatedItems with ConcurrentBag<int> or use a read-write lock when mutating the tree.

Cache invalidation via message queue:
Publish a TagChanged event to RabbitMQ/Service Bus when the DB changes. Each instance subscribes and refreshes its cache slice.

Unit Test

[Fact]
public void GetItemsForTag_IncludesDescendantItems()
{
    var cache   = new MemoryCache(new MemoryCacheOptions());
    var context = new TagContext(
        new DbContextOptionsBuilder<TagContext>()
            .UseInMemoryDatabase("Test")
            .Options);
    var manager = new TagCacheManager(cache, context);

    manager.CreateTag(1, "Technology",  null);
    manager.CreateTag(2, "Programming", 1);
    manager.CreateTag(3, "C#",          2);
    manager.AssociateItem(3, 101);

    var items = manager.GetItemsForTag(1); // Technology
    Assert.Contains(101, items);           // should include the C# item
}

Conclusion

A hierarchical cache transforms multi-query reads into single tree traversals and makes parent-child propagation automatic. The TagCacheManager keeps the DB as the source of truth while the in-memory tree handles all reads. For production systems, pair it with Redis for distribution and a message queue for cache invalidation.

Originally published on Medium.

Mastering AutoMapper in .NET 8+: Best Practices, Pitfalls, and Real-World Examples

Arash Zand — Thu, 30 Jul 2026 22:22:24 +0000

AutoMapper eliminates the boilerplate of copying properties between domain models and DTOs. This guide covers everything from setup to advanced techniques, performance, testing, and the common pitfalls that trip up teams in production.

Setup

dotnet add package AutoMapper
dotnet add package AutoMapper.Extensions.Microsoft.DependencyInjection

Register in Program.cs:

builder.Services.AddAutoMapper(typeof(UserProfile).Assembly);

AutoMapper scans the assembly, finds all classes inheriting from Profile, and registers them automatically.

Always validate at startup in development:

var config = new MapperConfiguration(cfg => cfg.AddProfile<UserProfile>());
config.AssertConfigurationIsValid(); // throws if anything is misconfigured

Basic Mapping

When property names and types match, no configuration is needed:

public class UserProfile : Profile
{
    public UserProfile()
    {
        CreateMap<User, UserDto>(); // convention-based, zero config
    }
}

var dto = _mapper.Map<UserDto>(user);

Different property names:

CreateMap<Customer, CustomerDto>()
    .ForMember(dest => dest.FirstName, opt => opt.MapFrom(src => src.GivenName))
    .ForMember(dest => dest.LastName,  opt => opt.MapFrom(src => src.Surname));

Reverse mapping:

CreateMap<Employee, EmployeeDto>().ReverseMap();
// now both Employee→EmployeeDto and EmployeeDto→Employee work

Transformations:

CreateMap<Product, ProductDto>()
    .ForMember(dest => dest.DisplayPrice,
               opt => opt.MapFrom(src => $"${src.Price:F2}"));

Nested objects — just define both mappings and AutoMapper wires them:

CreateMap<Address, AddressDto>();
CreateMap<Person, PersonDto>(); // Address→AddressDto applied automatically

Collections:

var dtos = _mapper.Map<List<UserDto>>(users); // maps each element

Advanced Features

Conditional Mapping

CreateMap<Order, OrderDto>()
    .ForMember(dest => dest.Discount,
               opt => opt.Condition(src => src.IsPremiumCustomer));

Multiple conditions:

.ForMember(dest => dest.Discount, opt => opt.Condition(
    (src, dest) => src.IsPremiumCustomer && src.Discount > 0));

Null Substitution

CreateMap<UserProfile, UserProfileDto>()
    .ForMember(dest => dest.Bio,
               opt => opt.NullSubstitute("No biography provided."));

Custom Value Resolver

Encapsulates complex logic in a reusable, independently testable class:

public class AgeResolver : IValueResolver<User, UserDto, int>
{
    public int Resolve(User source, UserDto destination,
                       int destMember, ResolutionContext context)
        => (int)((DateTime.UtcNow - source.DateOfBirth).TotalDays / 365.25);
}

CreateMap<User, UserDto>()
    .ForMember(dest => dest.Age, opt => opt.MapFrom<AgeResolver>());

Custom Type Converter

For when you need full control over object creation:

public class StringToDateTimeConverter : ITypeConverter<string, DateTime>
{
    public DateTime Convert(string source, DateTime destination, ResolutionContext context)
        => DateTime.TryParse(source, out var result) ? result : DateTime.MinValue;
}

CreateMap<UserInputDto, User>()
    .ForMember(dest => dest.BirthDate,
               opt => opt.ConvertUsing(new StringToDateTimeConverter(),
                                       src => src.BirthDateString));

Flattening

AutoMapper matches AddressStreet → Address.Street automatically by naming convention:

// Source: Customer { Address { Street, City } }
// Dest:   CustomerDto { AddressStreet, AddressCity }
CreateMap<Customer, CustomerDto>(); // zero config needed

AfterMap

CreateMap<Order, OrderDto>()
    .AfterMap((src, dest) => dest.TotalWithTax = src.Total * 1.2m);

Polymorphic Mapping

CreateMap<Animal, AnimalDto>()
    .Include<Dog, DogDto>()
    .Include<Cat, CatDto>();

CreateMap<Dog, DogDto>();
CreateMap<Cat, CatDto>();
// AutoMapper resolves the correct derived type at runtime

DI-Injected Resolvers

public class PremiumDiscountResolver : IValueResolver<Order, OrderDto, decimal>
{
    private readonly IDiscountService _discountService;

    public PremiumDiscountResolver(IDiscountService discountService)
        => _discountService = discountService;

    public decimal Resolve(Order source, OrderDto destination,
                           decimal destMember, ResolutionContext context)
        => _discountService.CalculateDiscount(source);
}

Performance

AutoMapper compiles mapping plans once at startup and caches them. Subsequent calls for the same type pair are fast. The main cost is the cold start.

Precompile at startup to shift that cost away from user requests:

var config = new MapperConfiguration(cfg => cfg.AddProfile<UserProfile>());
config.CompileMappings(); // compile all mappings now
var mapper = config.CreateMapper();

Use ProjectTo for EF Core — this is the most important performance tip:

// ❌ Loads full entities into memory, then maps
var users = _dbContext.Users.ToList();
var dtos = _mapper.Map<List<UserDto>>(users);

// ✅ Translates mapping to SQL, only fetches DTO fields
var dtos = _dbContext.Users
    .ProjectTo<UserDto>(_mapper.ConfigurationProvider)
    .ToList();

Relative performance (indicative, 100k mappings):

Method	Relative speed
Manual mapping	Fastest
Mapster (code-gen)	Very close to manual
AutoMapper	Slightly slower

AutoMapper trades some raw speed for flexibility. For most enterprise workloads this is irrelevant. For batch processing millions of records, profile first.

Testing

Never skip mapping tests. Silent property mismatches are the hardest bugs to find.

Configuration validation:

public class MappingTests
{
    private readonly IMapper _mapper;

    public MappingTests()
    {
        var config = new MapperConfiguration(cfg =>
        {
            cfg.AddProfile<UserProfile>();
            cfg.AddProfile<OrderProfile>();
        });
        config.AssertConfigurationIsValid(); // fail fast if anything is wrong
        _mapper = config.CreateMapper();
    }
}

Property mapping:

[Fact]
public void User_ShouldMapFullName()
{
    var user = new User { FirstName = "John", LastName = "Doe" };
    var dto = _mapper.Map<UserDto>(user);
    Assert.Equal("John Doe", dto.FullName);
}

Conditional mapping:

[Fact]
public void Order_DiscountNotMappedForNonPremium()
{
    var order = new Order { Id = 1, Discount = 50, IsPremiumCustomer = false };
    var dto = _mapper.Map<OrderDto>(order);
    Assert.Equal(0, dto.Discount);
}

Custom resolver in isolation:

[Fact]
public void AgeResolver_CalculatesCorrectAge()
{
    var resolver = new AgeResolver();
    var user = new User { DateOfBirth = new DateTime(2000, 1, 1) };
    var age = resolver.Resolve(user, null, 0, null);
    Assert.Equal(25, age); // assuming 2025
}

Add mapping tests to your CI pipeline — they catch regressions before deployment.

Common Pitfalls

1. Using Map on IQueryable — loads everything into memory. Use ProjectTo instead.

2. Skipping AssertConfigurationIsValid — new properties added to entities silently go unmapped until a user hits the endpoint.

3. One giant profile — breaks the Single Responsibility Principle and makes navigation painful. One profile per domain area.

4. Overusing AfterMap — it's a code smell if you're doing it everywhere. Prefer inline expressions or resolvers.

5. AutoMapper in simple projects — if you have 3 DTOs, just write the manual mapping. AutoMapper adds a dependency without payoff.

6. Not testing custom resolvers — they contain business logic and deserve their own unit tests.

Recommended Project Structure

Application/
├── DTOs/
│   ├── UserDto.cs
│   └── OrderDto.cs
├── Profiles/
│   ├── UserProfile.cs
│   ├── OrderProfile.cs
│   └── ProductProfile.cs
└── Resolvers/
    ├── AgeResolver.cs
    └── PremiumDiscountResolver.cs

One profile per domain. Resolvers alongside profiles. Tests in a dedicated Tests/ project.

When to Use AutoMapper

✅ Multiple DTOs with repeated mapping patterns

✅ Nested objects, flattening, or polymorphic types

✅ Complex transformations that should be centralized and tested

✅ EF Core projections with ProjectTo

❌ 2-3 simple one-to-one mappings — just write them manually

❌ Performance-critical batch processing of millions of records

❌ Mapping logic so complex it's clearer as explicit code

Alternatives: Mapster (faster, code-gen), Mapperly (source generator, compile-time safe), manual mapping.

Originally published on Medium.

Modern Design Patterns in .NET 10 and C# 12: 10 Essential Patterns

Arash Zand — Thu, 30 Jul 2026 22:20:56 +0000

.NET 10 and C# 12 have reshaped how design patterns are implemented — minimal APIs, record types, improved pattern matching, and async streams reduce boilerplate and make patterns more expressive. This article covers 10 patterns with practical code examples tailored to modern .NET development.

1. Minimal API Pipeline Pattern

Groups related endpoints to share filters, authentication, and OpenAPI metadata — replacing bloated controllers with concise, functional definitions.

var userEndpoints = app.MapGroup("/api/users")
    .WithTags("Users")
    .WithOpenApi()
    .RequireAuthorization();

userEndpoints.MapGet("/{id:int}", async (int id, HttpContext context) =>
{
    if (id <= 0) return Results.BadRequest("Invalid user ID");
    await Task.Delay(100);
    return Results.Ok(new User(id, $"User {id}", $"user{id}@example.com"));
}).WithName("GetUser")
  .WithDescription("Retrieves a user by ID");

userEndpoints.MapPost("/", async (UserInput input, HttpContext context) =>
{
    if (string.IsNullOrEmpty(input.Name)) return Results.BadRequest("Name is required");
    await Task.Delay(100);
    var newUser = new User(1, input.Name, input.Email);
    return Results.Created($"/api/users/{newUser.Id}", newUser);
}).WithName("CreateUser");

record User(int Id, string Name, string Email);
record UserInput(string Name, string Email);

RequireAuthorization and WithOpenApi apply to all endpoints in the group. C# 12 records make DTOs immutable and concise.

2. Result Pattern

Replaces exception-based error handling with a typed result object — explicit, predictable, and easy to unit test.

public abstract record Result<T>
{
    private Result() { }
    public record Success(T Value) : Result<T>;
    public record Failure(string Error, int? StatusCode = null) : Result<T>;
}

public class UserService
{
    private readonly Dictionary<int, User> _users = new()
    {
        { 1, new User(1, "Alice", "alice@example.com") }
    };

    public async Task<Result<User>> GetUserAsync(int id)
    {
        if (id <= 0) return new Result<User>.Failure("Invalid user ID", 400);
        await Task.Delay(100);
        return _users.TryGetValue(id, out var user)
            ? new Result<User>.Success(user)
            : new Result<User>.Failure("User not found", 404);
    }
}

// Usage — C# 12 pattern matching
var result = await service.GetUserAsync(1);
switch (result)
{
    case Result<User>.Success(var user):
        Console.WriteLine($"Found: {user.Name}");
        break;
    case Result<User>.Failure(var error, var statusCode):
        Console.WriteLine($"Error {statusCode}: {error}");
        break;
}

No try-catch blocks. Outcomes are explicit in the return type. The StatusCode field makes it HTTP-friendly.

3. Functional Command Pattern

Encapsulates business logic as immutable pure functions — deterministic, side-effect-free, and trivially testable.

public record Order(int Id, decimal Amount, string Customer);

public static class OrderProcessor
{
    public static Func<Order, Task<Result<string>>> CreateProcessOrderCommand(
        decimal discount, ILogger logger)
    {
        return async order =>
        {
            if (order.Amount <= 0)
                return new Result<string>.Failure("Invalid order amount");

            logger.LogInformation("Processing order {OrderId}", order.Id);
            var finalAmount = order.Amount - (order.Amount * discount);
            await Task.Delay(100);
            return new Result<string>.Success(
                $"Order {order.Id} processed: {finalAmount:C}");
        };
    }
}

// Usage
var processOrder = OrderProcessor.CreateProcessOrderCommand(0.15m, logger);
var result = await processOrder(new Order(1, 100m, "Alice"));

The function closes over discount and logger but keeps Order immutable. Behavior is deterministic — mock the inputs, predict the outputs.

4. Event Sourcing with Async Streams

Captures state as a sequence of events. Current state is reconstructed by replaying events. .NET 10's IAsyncEnumerable enables efficient async streaming of event history.

public record UserEvent(string Type, int UserId, string Data, DateTime Timestamp);
public record UserState(string Name = "", bool IsActive = false);

public class UserEventStore
{
    private readonly List<UserEvent> _events = [];

    public async Task AppendEventAsync(UserEvent evt)
    {
        await Task.Delay(10);
        _events.Add(evt);
    }

    public async IAsyncEnumerable<UserEvent> GetEventsAsync(int userId)
    {
        foreach (var evt in _events.Where(e => e.UserId == userId)
                                   .OrderBy(e => e.Timestamp))
        {
            await Task.Delay(10);
            yield return evt;
        }
    }

    public async Task<UserState> RebuildStateAsync(int userId)
    {
        var state = new UserState();
        await foreach (var evt in GetEventsAsync(userId))
        {
            state = evt.Type switch
            {
                "UserCreated"     => state with { Name = evt.Data, IsActive = true },
                "UserDeactivated" => state with { IsActive = false },
                _                 => state
            };
        }
        return state;
    }
}

Full audit trail built in. Add new event types without changing existing logic. Integrate with Kafka or EventStoreDB for production.

5. Dependency Injection Scoping Pattern

Manages service lifetimes correctly — transient, scoped, or singleton — preventing memory leaks and concurrency bugs.

public interface IOperationService
{
    string GetOperationId();
    Task ProcessAsync();
}

public class OperationService : IOperationService
{
    private readonly Guid _operationId = Guid.NewGuid();
    public string GetOperationId() => _operationId.ToString();
    public async Task ProcessAsync()
    {
        await Task.Delay(100);
        Console.WriteLine($"Processing: {_operationId}");
    }
}

// Registration
services.AddScoped<IOperationService, OperationService>();

// Usage — each scope gets a unique instance
async Task ProcessRequestAsync(IServiceProvider provider, int requestId)
{
    using var scope = provider.CreateScope();
    var service = scope.ServiceProvider.GetRequiredService<IOperationService>();
    await service.ProcessAsync();
    Console.WriteLine($"Request {requestId}: {service.GetOperationId()}");
}

Each CreateScope() call produces a fresh OperationService. The using declaration ensures disposal. Critical for DbContext, unit-of-work, and any stateful per-request service.

6. Strategy Pattern

Encapsulates interchangeable algorithms behind a common interface — swap at runtime without touching callers.

public interface ISortingStrategy
{
    void Sort<T>(List<T> data) where T : IComparable<T>;
}

public class BubbleSortStrategy : ISortingStrategy
{
    public void Sort<T>(List<T> data) where T : IComparable<T>
    {
        for (int i = 0; i < data.Count - 1; i++)
            for (int j = 0; j < data.Count - i - 1; j++)
                if (data[j].CompareTo(data[j + 1]) > 0)
                    (data[j], data[j + 1]) = (data[j + 1], data[j]); // C# 12 tuple swap
    }
}

public class QuickSortStrategy : ISortingStrategy { /* ... */ }

// Usage
ISortingStrategy strategy = new BubbleSortStrategy();
await Task.Run(() => strategy.Sort(data));

strategy = new QuickSortStrategy();
await Task.Run(() => strategy.Sort(data));

Add a MergeSortStrategy without touching any caller code — Open/Closed Principle in practice.

7. Observer Pattern

Notifies subscribers of state changes without tight coupling — essential for event-driven systems, dashboards, and real-time feeds.

public class NewsAgency
{
    private readonly List<IObserver<string>> _observers = [];

    public void Subscribe(IObserver<string> observer) => _observers.Add(observer);
    public void Unsubscribe(IObserver<string> observer) => _observers.Remove(observer);

    public async Task PublishNewsAsync(string news)
    {
        await Task.Delay(100);
        foreach (var observer in _observers.ToList())
        {
            try { observer.OnNext(news); }
            catch (Exception ex) { observer.OnError(ex); }
        }
    }
}

public class NewsSubscriber : IObserver<string>
{
    private readonly string _name;
    public NewsSubscriber(string name) => _name = name;
    public void OnNext(string value) => Console.WriteLine($"{_name}: {value}");
    public void OnError(Exception error) => Console.WriteLine($"{_name} error: {error.Message}");
    public void OnCompleted() => Console.WriteLine($"{_name} completed");
}

ToList() before iterating prevents modification-during-enumeration bugs. Individual subscriber failures don't crash the publisher.

8. Decorator Pattern

Wraps objects to add responsibilities at runtime — cross-cutting concerns like logging, encryption, or caching without modifying the original class.

public interface IStream
{
    Task WriteAsync(string data);
}

public class BasicStream : IStream
{
    public async Task WriteAsync(string data)
    {
        await Task.Delay(100);
        Console.WriteLine($"BasicStream wrote: {data}");
    }
}

public class EncryptedStream(IStream inner) : IStream
{
    public async Task WriteAsync(string data)
    {
        var encrypted = Convert.ToBase64String(
            System.Text.Encoding.UTF8.GetBytes(data));
        await inner.WriteAsync(encrypted);
    }
}

public class LoggingStream(IStream inner, ILogger<LoggingStream> logger) : IStream
{
    public async Task WriteAsync(string data)
    {
        logger.LogInformation("Writing: {Data}", data);
        await inner.WriteAsync(data);
        logger.LogInformation("Write complete");
    }
}

// Stack decorators
IStream stream = new LoggingStream(
    new EncryptedStream(new BasicStream()), logger);
await stream.WriteAsync("Sensitive Data");

Add compression, validation, or retry decorators by wrapping further — no changes to existing classes.

9. Factory Method Pattern

Delegates object creation to subclasses — centralises creation logic, enables extension without modification.

public abstract class Vehicle
{
    public abstract string GetTypeName();
    public virtual async Task InitializeAsync()
    {
        await Task.Delay(100);
        Console.WriteLine($"{GetTypeName()} initialized");
    }
}

public class Car : Vehicle { public override string GetTypeName() => "Car"; }
public class Bike : Vehicle { public override string GetTypeName() => "Bike"; }

public abstract class VehicleFactory
{
    public abstract Vehicle CreateVehicle();
}

public class CarFactory : VehicleFactory
{
    public override Vehicle CreateVehicle() => new Car();
}

// Usage
VehicleFactory factory = new CarFactory();
var vehicle = factory.CreateVehicle();
await vehicle.InitializeAsync();

Add TruckFactory without touching existing code. Pairs naturally with DI registration for factory selection at runtime.

10. Singleton Pattern

One instance, globally accessible — use .NET's DI container rather than static fields for thread safety and testability.

public class ConfigurationService
{
    private static readonly Lazy<ConfigurationService> _instance =
        new(() => new ConfigurationService());

    public static ConfigurationService Instance => _instance.Value;

    private readonly Dictionary<string, string> _settings = new()
    {
        { "ApiKey", "12345-abcde" },
        { "Timeout", "30" }
    };

    private ConfigurationService() { }

    public string GetConfig(string key) =>
        _settings.TryGetValue(key, out var value) ? value : "Not found";

    public async Task SaveConfigAsync(string key, string value)
    {
        await Task.Delay(100);
        _settings[key] = value;
    }
}

// Register via DI
services.AddSingleton<ConfigurationService>(_ => ConfigurationService.Instance);

Lazy<T> ensures thread-safe initialisation without explicit locking. Registering via DI keeps it testable — inject a mock in tests rather than hitting the static instance.

Summary

Pattern	Best for
Minimal API Pipeline	Grouping endpoints with shared config
Result	Explicit, exception-free error handling
Functional Command	Pure, testable business logic
Event Sourcing + Async Streams	Audit trails, state reconstruction
DI Scoping	Correct service lifetime management
Strategy	Swappable algorithms at runtime
Observer	Decoupled event notifications
Decorator	Composable cross-cutting concerns
Factory Method	Centralised, extensible object creation
Singleton	Shared resources with thread-safe init

Start with the patterns that address your current pain points. Test in isolated feature branches before rolling out across a codebase.

Originally published on Medium.

Common Patterns in Software Architecture

Arash Zand — Thu, 30 Jul 2026 22:18:14 +0000

Software architecture defines the structure and behavior of systems, ensuring they meet requirements like scalability, maintainability, and performance. This guide covers 15 widely used patterns with explanations, trade-offs, and tools across Node.js/TypeScript, C#/.NET, Go, and Java.

1. Layered Architecture

Organizes a system into horizontal layers — presentation, business logic, data access — each depending only on the layer below. Promotes separation of concerns and makes layers independently testable.

Trade-offs: Inter-layer communication adds latency. Over-layering creates unnecessary complexity.

Tools: ASP.NET Core + EF Core + Blazor (C#/.NET) · Express.js + TypeORM + React (Node) · Spring Boot + JPA + Thymeleaf (Java) · Gin + GORM (Go)

2. Microservices Architecture

Divides a system into small, independently deployable services, each owning its data and business capability. Services communicate via APIs or messaging.

Trade-offs: Independent scaling and deployment, but significant distributed systems complexity — network latency, data consistency, service discovery, and DevOps overhead.

Tools: NestJS + MassTransit + Kafka (C#/.NET) · Spring Cloud + Hibernate + Kafka (Java) · Go Micro + gRPC + NATS (Go) · Docker + Kubernetes + Prometheus

3. Event-Driven Architecture (EDA)

Components communicate by producing and consuming events via a message broker. Producers and consumers are fully decoupled — each scales independently.

Trade-offs: High responsiveness and scalability, but eventual consistency and complex error handling (dead-letter queues, retries, schema evolution).

Tools: MassTransit + Azure Event Hubs (C#/.NET) · KafkaJS + EventStoreDB (Node) · Axon + Apache Kafka (Java) · Watermill + NATS (Go)

Example pattern — event sourcing:
Instead of updating a record directly, store events like OrderPlaced, ItemShipped. Replay events to reconstruct current state. Enables auditability and time-travel debugging.

4. Service-Oriented Architecture (SOA)

Organizes a system as a collection of independent, reusable services that communicate over a network via SOAP or REST. An ESB or API Gateway handles orchestration.

Trade-offs: Reusability and interoperability across heterogeneous systems. More complex governance than microservices — service contracts, versioning, security.

Tools: MuleSoft ESB · WSO2 · Kong API Gateway · Spring Boot + Eureka (Java) · ASP.NET Core + Steeltoe (C#/.NET)

5. Monolithic Architecture

All functionality in a single codebase, deployed as one unit. Simple to develop, test, and deploy. Suitable for small teams or early-stage products.

Trade-offs: Fast to start, but tight coupling makes scaling and independent deployment hard as the system grows. Consider a modular monolith — loosely coupled internal modules — as a middle ground before microservices.

Tools: ASP.NET Core + EF Core + Razor Pages (C#/.NET) · Express.js + Sequelize + Next.js (Node) · Spring Boot + JPA (Java) · Go standard library + GORM

6. Client-Server Architecture

Clients request services; servers provide them. The foundation of every networked application.

Key decisions:

Stateless vs stateful — stateless servers scale horizontally (REST APIs); stateful servers maintain connections (WebSockets)
Caching — CDNs cache static assets; Redis caches session data
Load balancing — NGINX/AWS ELB distribute traffic across server instances

Tools: ASP.NET Core + SignalR (C#/.NET) · Express.js + Socket.IO (Node) · Spring Boot + WebSocket (Java) · Go net/http + Gorilla WebSocket

7. Model-View-Controller (MVC)

Separates an application into Model (data/logic), View (UI), and Controller (coordination). Promotes testability — each component can be tested in isolation.

Server-side MVC: ASP.NET Core MVC, Spring MVC, Rails — renders HTML on the server.
Client-side MVC: Angular, React — updates the DOM in the browser.

When it gets complex: Controllers become bloated in large apps. MVVM or Clean Architecture addresses this.

Tools: ASP.NET Core MVC + EF Core (C#/.NET) · Angular/React + Express (Node) · Spring MVC + JPA (Java) · Revel/Beego + GORM (Go)

8. Component-Based Architecture

Structures a system as reusable, self-contained UI components that interact through well-defined props and events. Central to React, Angular, Blazor, and Vue.

Trade-offs: Fast development, parallel team work, easy testing of isolated components. Shared state management adds complexity — Redux (React), NgRx (Angular), Fluxor (Blazor).

Tools: React + Redux + Storybook (Node) · Blazor + Fluxor (C#/.NET) · Vaadin/GWT (Java) · Templ (Go)

9. Domain-Driven Design (DDD)

Aligns software design with the business domain. Key building blocks:

Concept	Description
Entity	Object with a unique identity (e.g., `Customer`)
Value Object	Describes an attribute, no identity (e.g., `Address`)
Aggregate	Cluster of entities with a root (e.g., `Order` + `OrderItems`)
Repository	Abstracts data access for aggregates
Bounded Context	Clear boundary for a domain model
Domain Event	Signals a meaningful state change (e.g., `OrderPlaced`)

Trade-offs: Ideal for complex domains (finance, healthcare, logistics). Significant upfront investment in domain modeling. Overkill for simple CRUD apps.

Tools: MediatR + EF Core + Ardalis.Specification (C#/.NET) · NestJS + TypeORM (Node) · Axon Framework + JPA (Java) · Go structs + SQLx + Watermill

10. API Gateway Pattern

A single entry point for all client requests in a distributed system. Routes requests to services and handles cross-cutting concerns centrally.

Responsibilities: Authentication · Rate limiting · Request/response transformation · Response aggregation · Caching · Load balancing

Trade-offs: Simplifies client interactions but becomes a single point of failure — cluster it. Adds a network hop — minimize processing overhead.

Tools: Ocelot (C#/.NET) · Spring Cloud Gateway / Zuul (Java) · KrakenD / Traefik (Go) · Kong · Amazon API Gateway · NGINX · Envoy

11. Circuit Breaker Pattern

Prevents cascading failures by halting requests to a failing service and returning fallbacks until the service recovers.

Three states:

Closed — normal operation, requests flow through
Open — failure threshold exceeded, requests blocked, fallback returned
Half-Open — timeout elapsed, one test request allowed; closes on success, reopens on failure

Example fallbacks: Return cached data · Redirect to alternative service · Return a degraded response

Tools: Polly (C#/.NET) · Resilience4j / Hystrix (Java) · Opossum (Node) · GoBreaker (Go) · Envoy (built-in)

12. Saga Pattern

Manages distributed transactions across microservices without two-phase commit. Breaks a transaction into local steps, each emitting an event to trigger the next. Failed steps trigger compensating transactions to undo previous work.

Two coordination styles:

Choreography — services react to events, no central coordinator. Simple but hard to trace.
Orchestration — a Saga Coordinator sends commands and tracks state. Easier to reason about, more coupling.

Example — order placement:

Order Service: create order → emit OrderCreated
Inventory Service: reserve items → emit ItemsReserved
Payment Service: charge card → emit PaymentProcessed
If payment fails → compensate: release items, cancel order

Tools: MassTransit + NServiceBus (C#/.NET) · Axon Framework (Java) · Eventuate Tram (Node) · Watermill (Go) · AWS Step Functions

13. CQRS (Command Query Responsibility Segregation)

Separates write operations (Commands) from read operations (Queries) into distinct models — each optimized for its workload.

Command side: POST /orders → validates → persists to write DB → emits event
Query side:   GET /orders  → reads from pre-built read model (Redis, MongoDB, Elasticsearch)

Trade-offs: Independent scaling of reads and writes. Paired with Event Sourcing, the read model rebuilds from events. Eventual consistency — read model may lag slightly behind write model. Significant complexity overhead — not worth it for simple CRUD.

Tools: MediatR + EventStoreDB + Marten (C#/.NET) · Axon + Elasticsearch (Java) · NestJS + Redis (Node) · Watermill + MongoDB (Go)

14. Publish-Subscribe Pattern

Publishers send messages to topics on a broker; subscribers consume from those topics. Producers and consumers have no direct knowledge of each other.

Ideal for: Real-time feeds · Stock price updates · IoT sensor data · Notification systems · Log aggregation

Trade-offs: Dynamic, decoupled scaling. Requires schema management (Confluent Schema Registry), dead-letter queues for failures, and monitoring for queue health.

Tools: MassTransit + Azure Service Bus (C#/.NET) · Spring Cloud Stream + Kafka (Java) · KafkaJS + Redis Pub/Sub (Node) · Watermill + NATS (Go)

15. API-First Approach

Design and publish the API specification before writing implementation code. The spec (OpenAPI/Swagger or GraphQL schema) becomes the contract between teams.

Benefits:

Frontend and backend teams work in parallel against the same contract
Mock servers simulate responses during frontend development
Third-party integrations have clear, stable contracts
Consistent error codes, schemas, and versioning from day one

Versioning strategy: Use URI versioning (/v1/payments, /v2/payments) to avoid breaking consumers.

Tools: Swashbuckle + HotChocolate (C#/.NET) · Swagger-jsdoc + Apollo Server (Node) · OpenAPI Generator + Spring GraphQL (Java) · oapi-codegen + gqlgen (Go) · Postman · SwaggerHub · Stoplight

Choosing the Right Pattern

Need	Consider
Simple app, small team	Monolithic or Layered
Independent team scaling	Microservices
Real-time, async workflows	Event-Driven + Pub/Sub
Complex business domain	DDD
High read/write disparity	CQRS
Distributed transactions	Saga
Resilience in dependencies	Circuit Breaker
Multiple client types	API Gateway
Parallel frontend/backend	API-First

No pattern is universally correct. The best architecture is the simplest one that satisfies your actual constraints.

Originally published on Medium.

Learning AWS CloudFormation: A Practical Introduction

Arash Zand — Thu, 30 Jul 2026 22:16:25 +0000

AWS CloudFormation lets you manage your AWS infrastructure using code. Instead of setting up resources by hand, you write a template in JSON or YAML and let CloudFormation handle the rest. This guide covers the core concepts, stack types, lifecycle states, and real troubleshooting patterns from hands-on experience.

Why CloudFormation?

Automation — define once, deploy anywhere, no manual clicking
Consistency — same template = same environment every time
Version control — templates are code, store them in Git
Cost control — delete a whole stack with one command, no orphaned resources
Scalability — update templates to grow infrastructure as the project grows

Types of Stacks

1. Single Stack

All resources in one template. Good for small projects.

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  BlogSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for blog app
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0

  BlogInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

  BlogEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-0c55b159cbfafe1f0
      SecurityGroupIds:
        - !Ref BlogSecurityGroup
      IamInstanceProfile: !Ref BlogInstanceProfile

  BlogStaticBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub blog-static-files-${AWS::StackName}

2. Nested Stacks

Split large templates into modular child stacks. A parent stack references child stacks stored in S3. Good for separating networking, compute, and storage concerns.

Parent template:

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  VpcCidr:
    Type: String
    Default: 10.1.0.0/16

Resources:
  NetworkStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/my-bucket/ecommerce-network.yaml
      Parameters:
        VpcCidr: !Ref VpcCidr

  AppServerStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/my-bucket/ecommerce-app.yaml
      Parameters:
        SubnetId: !GetAtt NetworkStack.Outputs.PublicSubnetId
        SecurityGroupId: !GetAtt NetworkStack.Outputs.AppSecurityGroupId

  DatabaseStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/my-bucket/ecommerce-database.yaml
      Parameters:
        SubnetId: !GetAtt NetworkStack.Outputs.PrivateSubnetId
        SecurityGroupId: !GetAtt NetworkStack.Outputs.DBSecurityGroupId

Child network template (ecommerce-network.yaml):

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  VpcCidr:
    Type: String

Resources:
  EcommerceVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref EcommerceVPC
      CidrBlock: 10.1.1.0/24

  AppSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for app servers
      VpcId: !Ref EcommerceVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0

  DBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for database
      VpcId: !Ref EcommerceVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 3306
          ToPort: 3306
          SourceSecurityGroupId: !Ref AppSecurityGroup

Outputs:
  PublicSubnetId:
    Value: !Ref PublicSubnet
  PrivateSubnetId:
    Value: !Ref PrivateSubnet
  AppSecurityGroupId:
    Value: !Ref AppSecurityGroup
  DBSecurityGroupId:
    Value: !Ref DBSecurityGroup

Lesson learned: A parent stack with 8 child stacks took 25 minutes to deploy vs 12 minutes for a single stack. Keep nesting manageable.

3. Stacks with Exports

Share values between independent stacks using Outputs + Export.

Network stack exports a subnet ID:

Outputs:
  SubnetIdExport:
    Value: !Ref AppSubnet
    Export:
      Name: AppSubnetExport

Storage stack imports it:

Resources:
  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: Subnet group for RDS
      SubnetIds:
        - !ImportValue AppSubnetExport

Stack Lifecycle States

Understanding states helps you debug what's happening and why.

State	Meaning
`CREATE_IN_PROGRESS`	Resources are being provisioned
`CREATE_COMPLETE`	Stack created successfully
`CREATE_FAILED`	Something went wrong during creation
`UPDATE_IN_PROGRESS`	A change is being applied
`UPDATE_COMPLETE`	Update succeeded
`UPDATE_ROLLBACK_IN_PROGRESS`	Update failed, rolling back
`UPDATE_ROLLBACK_COMPLETE`	Rollback finished, back to previous state
`DELETE_IN_PROGRESS`	Stack deletion in progress
`DELETE_COMPLETE`	All resources removed
`REVIEW_IN_PROGRESS`	Change set being validated
`ROLLBACK_IN_PROGRESS`	Initial creation failed, cleaning up
`ROLLBACK_COMPLETE`	Cleanup done after failed creation

Troubleshooting Common Issues

1. Syntax errors

Validate before deploying:

aws cloudformation validate-template \
  --template-body file://template.yaml \
  --region us-west-2

2. Wrong resource settings

Check the AWS CloudFormation docs for valid property names. A single typo lands you in CREATE_FAILED.

3. Stuck in REVIEW_IN_PROGRESS

Cancel the update or delete the change set:

aws cloudformation cancel-update-stack \
  --stack-name AppBase-US-West-2 \
  --region us-west-2

aws cloudformation delete-change-set \
  --stack-name AppBase-US-West-2 \
  --change-set-name FailedChange \
  --region us-west-2

Nuclear option if needed:

aws cloudformation delete-stack \
  --stack-name AppBase-US-West-2 \
  --region us-west-2

4. Pipeline failures

If a CI/CD pipeline (GitHub Actions, Azure DevOps) fails mid-deploy, add a cleanup step that deletes failed change sets before retrying. Otherwise the next run will fail immediately on the stale change set.

5. Dependency issues

Use DependsOn when a resource needs another to exist first:

Resources:
  MyLambda:
    Type: AWS::Lambda::Function
    DependsOn: MyDynamoDBStream
    Properties:
      ...

Tips for Getting Started

Start simple — one EC2 instance, get it deploying cleanly before adding complexity
Use parameters — make templates reusable across environments from day one
Test on a disposable stack — never test changes directly on production
Watch the Events tab — it's the most useful debugging surface in the console
Plan for rollback — always include cleanup steps in your pipeline

Conclusion

CloudFormation is foundational infrastructure-as-code on AWS. Single stacks for simplicity, nested stacks for modularity, exports for cross-stack sharing. Know the lifecycle states, validate templates before deploying, and build rollback handling into your pipelines from the start.

Originally published on Medium.

Optimizing API Latency in C# .NET Applications

Arash Zand — Thu, 30 Jul 2026 22:07:23 +0000

API latency plays a crucial role in performance and user experience. High latency frustrates users, reduces scalability, and increases infrastructure costs. This guide dives deep into causes, measurement, and optimization strategies for C# .NET APIs.

1. Understanding API Latency

API latency is the total time from a client sending a request to receiving a response — covering network transmission, server-side processing, and database interactions.

Types of latency:

Network latency — distance, bandwidth, congestion. Fix: CDNs.
Processing latency — inefficient code, blocking operations. Fix: async programming.
Database latency — slow queries, missing indexes. Fix: query optimization, caching, connection pooling.

Client → (Network) → API Gateway → (Processing) → Database → (DB latency) → API Gateway → Client

2. Measuring API Latency

Stopwatch — quick and surgical:

var stopwatch = Stopwatch.StartNew();
await ProcessRequestAsync();
stopwatch.Stop();
Console.WriteLine($"Elapsed: {stopwatch.ElapsedMilliseconds} ms");

Middleware timing — covers every request:

public class LatencyMiddleware
{
    private readonly RequestDelegate _next;

    public LatencyMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        var sw = Stopwatch.StartNew();
        await _next(context);
        sw.Stop();
        Console.WriteLine($"Request latency: {sw.ElapsedMilliseconds} ms");
    }
}

app.UseMiddleware<LatencyMiddleware>();

Application Insights — production-grade:

public void ConfigureServices(IServiceCollection services)
{
    services.AddApplicationInsightsTelemetry(
        Configuration["ApplicationInsights:InstrumentationKey"]);
}

Other tools: Postman (endpoint testing), JMeter (load testing), Grafana + Prometheus (dashboards), Jaeger / OpenTelemetry (distributed tracing).

3. Latency in C# .NET — How Requests Flow

Client → Web Server (Kestrel/IIS) → Middleware Pipeline → Controller → DB/Logic → Middleware → Response

Synchronous vs asynchronous — the single biggest lever:

// ❌ Synchronous — blocks the thread
public IActionResult GetData()
{
    var data = _service.GetData();
    return Ok(data);
}

// ✅ Asynchronous — frees the thread for other requests
public async Task<IActionResult> GetDataAsync()
{
    var data = await _service.GetDataAsync();
    return Ok(data);
}

Common bottlenecks:

Bottleneck	Fix
Blocking I/O	`async/await` throughout
Slow DB queries	Indexes, `AsNoTracking()` for reads
Heavy middleware	Remove unnecessary steps, async logging
Large serialization	`System.Text.Json`, smaller payloads

4. Optimizing API Latency — Best Practices

Async/Await

public async Task<IActionResult> GetUserDataAsync()
{
    var data = await _databaseService.GetUserDataAsync();
    return Ok(data);
}

In-Memory Caching

public async Task<User> GetUserByIdAsync(int userId)
{
    if (!_memoryCache.TryGetValue(userId, out User user))
    {
        user = await _databaseService.GetUserByIdAsync(userId);
        _memoryCache.Set(userId, user, TimeSpan.FromMinutes(10));
    }
    return user;
}

Response Compression

public void Configure(IApplicationBuilder app)
{
    app.UseResponseCompression();
}

Efficient Serialization

private readonly JsonSerializerOptions _jsonOptions = new JsonSerializerOptions
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase
};

[HttpGet]
public IActionResult GetUser(int userId)
{
    var user = _databaseService.GetUserById(userId);
    return Content(JsonSerializer.Serialize(user, _jsonOptions), "application/json");
}

5. Advanced Techniques

Message Queues — offload non-urgent work

public async Task<IActionResult> ProcessOrderAsync(Order order)
{
    await _messageQueue.SendAsync(order);
    return Accepted(); // responds immediately
}

Distributed Caching with Redis

public async Task<User> GetUserByIdAsync(int userId)
{
    var cache = _redis.GetDatabase();
    var cached = await cache.StringGetAsync(userId.ToString());

    if (!cached.IsNullOrEmpty)
        return JsonSerializer.Deserialize<User>(cached);

    var user = await _databaseService.GetUserByIdAsync(userId);
    await cache.StringSetAsync(
        userId.ToString(),
        JsonSerializer.Serialize(user),
        TimeSpan.FromMinutes(10));

    return user;
}

Database Sharding & HTTP/2

For high-volume systems, sharding distributes data across DB instances. Upgrading to HTTP/2 or HTTP/3 enables multiplexing — multiple requests over a single connection, reducing handshake overhead.

6. Case Study: E-Commerce Checkout API

Problem: Checkout averaging 2–3 seconds. Identified via Application Insights + SQL Profiler:

Synchronous inventory service calls
Missing indexes on the orders table

Fixes applied:

Refactored inventory check to async/await
Added indexes on orders table, optimized inventory queries
Redis caching for inventory data that rarely changes

Result: Average checkout response dropped from 3 seconds → under 500ms.

7. Monitoring in Production

Azure Monitor — API performance and resource utilization
Prometheus + Grafana — real-time metrics and dashboards
New Relic — end-to-end latency per endpoint
Serilog — structured async logging

Set up alerts for latency thresholds so the team can respond to regressions before users notice.

Conclusion

Reducing API latency is not a one-time task. The core levers are: async programming, database optimization, caching (in-memory and distributed), payload compression, and middleware hygiene. Pair those with ongoing monitoring and load testing, and your .NET API stays fast as traffic scales.

Originally published on Medium.

Understanding How Normalizing Tables Can Affect Performance in C# .NET

Arash Zand — Thu, 30 Jul 2026 21:56:45 +0000

In software development, we're always chasing that sweet spot of building apps that run smoothly and perform well. When it comes to relational databases, one big decision keeps coming up: normalization. It's a technique that tidies up a database, cutting down on duplicate data and keeping everything consistent.

But here's the catch — while normalization makes your data clean and organized, it can throw a wrench in performance, especially when you're dealing with large datasets or complex queries. In this article, I'll dive into how normalizing tables impacts C# .NET apps running on PostgreSQL or SQL Server, and share practical ways to stay fast despite the trade-offs.

The Basics of Database Normalization
How Normalization Messes with Performance
Normalization Tricks in SQL Databases
When Normalization Starts Slowing You Down
Tips to Speed Up Normalized Databases
A Real-Life Example: Balancing Normalization and Performance
Keeping an Eye on Performance in Production
Best Practices I've Picked Up Along the Way
Next-Level Optimization for Normalized Databases
Testing and Benchmarking Performance
Applying This Stuff in a C# .NET CRM Project
Tuning Up a CRM System
Wrapping It Up

1. The Basics of Database Normalization

What's Database Normalization All About?

Database normalization splits data into separate tables to cut down on repeats and keep things consistent. The goals:

Ditch the duplicates — no more storing the same data in multiple places
Stop anomalies — updates, inserts, and deletes stay clean
Keep things consistent — data stays accurate across the entire database

Breaking It Down: Normal Forms (1NF, 2NF, 3NF, BCNF)

First Normal Form (1NF): Every column has simple, single values — no lists or groups crammed into one row
Second Normal Form (2NF): All non-key columns fully depend on the primary key, not just part of it
Third Normal Form (3NF): Non-key fields don't depend on other non-key fields — everything ties straight to the primary key
Boyce-Codd Normal Form (BCNF): Every "determiner" must be a candidate key — the strictest form

Pros and Cons

✅ Data integrity, less storage, cleaner updates

❌ Query complexity, join overhead, more disk I/O

2. How Normalization Messes with Performance

When you go all-in on normalization, performance starts to feel the squeeze.

The Join Problem

Normalized databases spread data across tables — Users, UserProfiles, UserOrders. To get the full picture you're stuck joining them back together:

SELECT u.UserName, p.ProfilePicture, o.OrderDate  
FROM Users u  
JOIN UserProfiles p ON u.UserID = p.UserID  
JOIN UserOrders o ON u.UserID = o.UserID  
WHERE u.UserID = 123;

If those tables grow large or indexes aren't on point, this query starts moving like molasses.

Disk I/O Woes

More tables means more hopping around. Instead of one Users table with everything, your data is spread across Users, UserProfiles, and UserOrders. A single query might poke around all three, piling up disk reads. Without solid indexes: hello, full table scans.

Caching and Indexing as Lifelines

Stash frequently accessed data — like user profiles — in memory, and you're not pinging the database constantly. Indexes make lookups fast, but too many will slow down your writes. It's about finding the sweet spot.

3. Normalization in SQL Databases

PostgreSQL Example

-- Step 1: Users (1NF)
CREATE TABLE Users (
  UserID SERIAL PRIMARY KEY,
  UserName VARCHAR(50) NOT NULL,
  Email VARCHAR(100) UNIQUE NOT NULL
);

-- Step 2: UserProfiles (2NF)
CREATE TABLE UserProfiles (
  ProfileID SERIAL PRIMARY KEY,
  UserID INT REFERENCES Users(UserID),
  ProfilePicture VARCHAR(255),
  Bio TEXT
);

-- Step 3: UserOrders (3NF)
CREATE TABLE UserOrders (
  OrderID SERIAL PRIMARY KEY,
  UserID INT REFERENCES Users(UserID),
  OrderDate DATE,
  TotalAmount DECIMAL(10, 2)
);

SQL Server Equivalent

Same structure, different syntax — IDENTITY instead of SERIAL, NVARCHAR instead of VARCHAR, FOREIGN KEY explicitly declared.

4. When Normalization Starts Slowing You Down

Join overload — juggling many tables tanks performance at scale
Real-time pressure — all those joins feel like a traffic jam when sub-second response is required
Over-splitting — breaking a simple entity into a dozen tiny tables creates unnecessary complexity

5. Tips to Speed Up Normalized Databases

Smart Indexing

Add indexes on columns you join or filter on frequently — like UserID. Don't over-index or your inserts and updates will crawl.

Denormalize Where It Fits

Merge tables or add duplicate columns to skip expensive joins. For read-heavy workloads, this is a significant win.

Async Queries in C# .NET

public async Task<UserDetails> GetUserDetailsAsync(int userId)
{
    return await _context.Users
        .Where(u => u.UserID == userId)
        .Include(u => u.UserProfile)
        .Include(u => u.UserOrders)
        .FirstOrDefaultAsync();
}

6. A Real-Life Example: Balancing Normalization and Performance

The Slow Query

SELECT u.UserName, u.Email, o.OrderDate, p.ProductName, od.Quantity, od.Price
FROM Users u
JOIN UserOrders o ON u.UserID = o.UserID
JOIN OrderDetails od ON o.OrderID = od.OrderID
JOIN Products p ON od.ProductID = p.ProductID
WHERE u.UserID = 123;

Before optimization:

Query time: 2.5 seconds
Disk I/O: 15,000 reads
CPU: 85%
App response: 3 seconds

What we did:

Added indexes on UserID, OrderID, ProductID
Cached user profiles and order details in memory
Denormalized ProductNames and TotalAmount into UserOrders
Rewrote the query to use the denormalized columns — 2 joins instead of 4

After optimization:

Query time: 0.5 seconds
Disk I/O: 2,500 reads
CPU: 30%
App response: 1 second

7. Keeping an Eye on Performance in Production

PostgreSQL EXPLAIN ANALYZE

EXPLAIN ANALYZE
SELECT u.UserName, u.Email, o.OrderDate, p.ProductName, od.Quantity, od.Price
FROM Users u
JOIN UserOrders o ON u.UserID = o.UserID
JOIN OrderDetails od ON o.OrderID = od.OrderID
JOIN Products p ON od.ProductID = p.ProductID
WHERE u.UserID = 123;

Shows you the execution plan, row estimates, and where time is being spent.

Application Insights in C# .NET

public void TrackQueryPerformance(string queryName, TimeSpan duration)
{
    var telemetry = new DependencyTelemetry
    {
        Type = "SQL",
        Target = "PostgreSQL",
        Data = queryName,
        Duration = duration
    };
    _telemetryClient.TrackDependency(telemetry);
}

Set alerts for queries exceeding a threshold — catch regressions before users notice.

8. Best Practices

Don't over-normalize — if speed matters, denormalize selectively
Index the join columns — UserID, OrderID, ProductID are your starting points
Cache read-heavy data — user profiles, product catalogs, anything that doesn't change constantly
Think about partitioning — for massive tables, partition by date or region to scope queries

Normalize when: data integrity is the priority, you're optimizing for storage, updates need to stay consistent.

Denormalize when: reads dominate, joins are killing you at scale, query complexity is spiraling.

9. Next-Level Optimization

Materialized Views

Pre-compute expensive joins and store the results:

CREATE MATERIALIZED VIEW UserOrderDetails AS
SELECT u.UserID, u.UserName, o.OrderID, o.OrderDate, p.ProductName, od.Quantity, od.Price
FROM Users u
JOIN UserOrders o ON u.UserID = o.UserID
JOIN OrderDetails od ON o.OrderID = od.OrderID
JOIN Products p ON od.ProductID = p.ProductID;

-- Query it directly
SELECT * FROM UserOrderDetails WHERE UserID = 123;

-- Refresh when needed
REFRESH MATERIALIZED VIEW UserOrderDetails;

Partitioning

CREATE TABLE UserOrders (
  OrderID SERIAL PRIMARY KEY,
  UserID INT,
  OrderDate DATE,
  TotalAmount DECIMAL(10, 2)
) PARTITION BY RANGE (OrderDate);

CREATE TABLE UserOrders_2024 PARTITION OF UserOrders
  FOR VALUES FROM ('2024-01-01') TO ('2024-12-31');

Queries scoped to 2024 skip all historical data entirely.

Redis Caching

public async Task<string> GetUserProfileAsync(int userId)
{
    string cacheKey = $"user:{userId}:profile";
    var cached = await _cache.StringGetAsync(cacheKey);

    if (cached.HasValue) return cached;

    var profile = GetUserProfileFromDatabase(userId);
    await _cache.StringSetAsync(cacheKey, profile);
    return profile;
}

Bulk Inserts with Dapper

connection.Execute(
    "INSERT INTO UserOrders (UserID, OrderDate, TotalAmount) VALUES (@UserID, @OrderDate, @TotalAmount)",
    orders
);

10. Testing and Benchmarking with BenchmarkDotNet

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

public class DatabaseQueryBenchmark
{
    [Benchmark]
    public void GetUserOrders()
    {
        var result = GetUserOrdersFromDatabase(123);
    }
}

public class Program
{
    public static void Main(string[] args) =>
        BenchmarkRunner.Run<DatabaseQueryBenchmark>();
}

Run against production-like data volumes to validate that optimizations hold under real load.

11. & 12. CRM Case Study

A C# .NET CRM with customers, contacts, opportunities, products, and orders. Heavy normalization worked fine at small scale — then the data grew.

Before optimization:

Query time: 4 seconds
Disk I/O: 20,000 reads
CPU: 90%
App response: 5 seconds

Fixes applied:

Indexes on CustomerID, OrderID, ProductID
Denormalized ProductName and Quantity into the Orders table — eliminated 2 joins from the core query
Redis caching for customer profiles and sales opportunities
Materialized view for the customer sales report
Async queries throughout with Dapper/Npgsql

After optimization:

Query time cut by 80%+
CPU load under 30%
App response under 1 second

The materialized view for reporting deserves a special mention:

CREATE MATERIALIZED VIEW CustomerSalesReport AS
SELECT c.CustomerID, c.CustomerName, SUM(o.TotalAmount) AS TotalSales
FROM Customers c
JOIN Orders o ON c.CustomerID = o.CustomerID
GROUP BY c.CustomerID, c.CustomerName;

Reports that previously ran aggregations on the fly now return instantly.

13. Wrapping It Up

Normalization is essential for data integrity and clean architecture — but as systems scale, the join overhead becomes real. The trick is knowing when to stay normalized and when to selectively denormalize, and pairing that with the right mix of indexing, caching, materialized views, and async patterns.

Tools like PostgreSQL's EXPLAIN ANALYZE, Application Insights, and BenchmarkDotNet give you the visibility to make those decisions based on evidence rather than guesswork.

The sweet spot: normalization for correctness, selective denormalization and caching for speed. Neither at the expense of the other.

Originally published on Medium.

DEV Community: Arash Zand

How I Fixed a Silent Bug in Microsoft's Semantic Kernel and Got Merged into Main

The Problem

The Fix

1. Add Think to OllamaPromptExecutionSettings

2. Wire through to OllamaTextGenerationService

3. Tests

How to Use It

What the Contribution Process Taught Me

The PR

The DI Coupling Trap: When .NET Libraries Block Their Own Capabilities

The Pattern

The Four Scenarios It Blocks

1. Third-party configuration sources

2. Multi-tenant applications

3. Shared instances across multiple consumers

4. Testing without infrastructure

The Fix: General Form First

The FakeOptionsMonitor Test Helper

Before vs. After

The Real-World Case: Polly v8

Principles for Library API Design

Migrating 200 Million Records in .NET: From Row-by-Row to a Crash-Safe Batched Pipeline

The Setup

V1: Load Everything, Insert Row by Row

V2: Fix the Dedup Key

V3: Batched Read + Batch Upsert (~37x faster)

V4: Checkpoint (The Production Version)

V5: Channel-Based Producer-Consumer Pipeline

Benchmark Comparison

Production War Stories

Key Takeaways

Architecting a GenAI FAQ System with AWS Bedrock, Python, TypeScript, C# .NET and Blazor

Architecture Overview

Step 1: AWS Infrastructure Setup

S3 Bucket

OpenSearch Domain

Bedrock

IAM Role for Lambda

API Gateway

Step 2: Data Cleaning Lambda (Python)

Step 3: Embedding Generation Lambda (Python)

Step 4: Question Processing Lambda (TypeScript)

Step 5: C# .NET API

Step 6: Blazor UI

Step 7: Security & Production Checklist

Step 8: IaC with CloudFormation

Common Troubleshooting

Implementing a Hierarchical Cache for Tagging in C# .NET

The Problem: Basic Tagging

The Solution: Hierarchical Cache

Tag Model — add ParentId

TagNode — the in-memory tree node

TagCacheManager — the full implementation

Registration

Usage Example

Performance Comparison

Scaling Beyond In-Memory

Unit Test

Conclusion

Mastering AutoMapper in .NET 8+: Best Practices, Pitfalls, and Real-World Examples

Setup

Basic Mapping

Advanced Features

Conditional Mapping

Null Substitution

Custom Value Resolver

Custom Type Converter

Flattening

AfterMap

Polymorphic Mapping

DI-Injected Resolvers

Performance

Testing

Common Pitfalls

Recommended Project Structure

When to Use AutoMapper

Modern Design Patterns in .NET 10 and C# 12: 10 Essential Patterns

1. Minimal API Pipeline Pattern

2. Result Pattern

1. Add `Think` to `OllamaPromptExecutionSettings`

2. Wire through to `OllamaTextGenerationService`

Tag Model — add `ParentId`