DEV Community: ZèD

Configuring GitHub Container Registry with Kubernetes for Private Images

ZèD — Thu, 14 May 2026 15:36:42 +0000

Configuring GitHub Container Registry with Kubernetes for private images

Private container images represent standard production deployment artifacts. When container images reside in GitHub Container Registry (GHCR) at ghcr.io, Kubernetes requires valid registry credentials to authenticate image pull requests during Pod scheduling.

Configuring declarative image pull Secrets and ServiceAccounts enables automated, decoupled authentication against GitHub Container Registry without manual node-level logins.

The problem and production context

Kubernetes cluster nodes pull container images anonymously by default. When referencing private registry artifacts, anonymous requests fail.

Failure scenario: A Deployment references a private image at ghcr.io/my-org/private-api:1.0.0. During pod scheduling, the kubelet issues an unauthenticated pull request to ghcr.io. The registry responds with HTTP 401 Unauthorized, causing the Pod to enter ErrImagePull and loop indefinitely in ImagePullBackOff.
Why default approaches fall short: Executing manual docker login commands on individual cluster nodes violates declarative infrastructure principles, fails on dynamically autoscaled node groups, and introduces configuration drift. Inlining raw registry secrets directly inside every workload manifest duplicates credential references and exposes base64-encoded tokens in source control.
Production impact: Workloads fail to boot during deployments or node drain events, deployment pipelines stall, and hardcoded registry credentials leaked into Git repositories compromise container registry access.

Declarative credential management provides key operational advantages:

Restricts application container images from public access.
Enables automated private image pulls directly from ghcr.io.
Eliminates manual node-level Docker logins.
Decouples credential management from workload manifests using ServiceAccounts.
Integrates cleanly with GitOps workflows, Helm charts, and Kustomize overlays.

Mental model and core concepts

Pulling private container images from GHCR requires understanding registry URI formatting, Docker credential structures, and Kubernetes identity inheritance.

1. GitHub Container Registry image naming and permissions

Private GHCR image references follow a standardized naming convention:

ghcr.io/OWNER/IMAGE_NAME:TAG

Example:

ghcr.io/my-org/private-api:1.0.0

Pulling private images requires authentication with a personal access token (classic or fine-grained) or a GitHub Actions token configured with at least the read:packages permission scope. When credentials are absent or invalid, the Pod terminates with:

ImagePullBackOff
ErrImagePull
unauthorized

2. Registry credential Secret structure

Kubernetes authenticates against private container registries using Secrets of type kubernetes.io/dockerconfigjson. Prior to base64 encoding, the Docker configuration payload contains:

{
  "auths": {
    "ghcr.io": {
      "username": "YOUR_GITHUB_USERNAME",
      "password": "YOUR_GITHUB_TOKEN",
      "email": "you@example.com",
      "auth": "BASE64_USERNAME_COLON_TOKEN"
    }
  }
}

The auth value represents the base64-encoded string of USERNAME:TOKEN:

base64(YOUR_GITHUB_USERNAME:YOUR_GITHUB_TOKEN)

The entire JSON structure is base64-encoded and embedded in the .dockerconfigjson field of the Secret manifest. The Secret must reside in the exact namespace of the workloads consuming it.

3. ServiceAccount-based credential decoupling

Rather than defining imagePullSecrets across every Deployment, StatefulSet, or Job manifest, attaching the Secret to a ServiceAccount enables automatic credential inheritance. When a Pod specifies serviceAccountName, the kubelet automatically inherits the referenced image pull secrets during container initialization.

4. Direct workload credential binding

As an alternative to ServiceAccount binding, individual Pod specifications can declare imagePullSecrets directly. While functional for single-manifest setups, it couples workload specifications to namespace secret names and requires repetitive declarations across services.

5. GitOps secret management pipelines

Raw Kubernetes Secrets use base64 serialization, not cryptographic encryption. Committing raw .dockerconfigjson manifests to version control leaks access tokens. Production GitOps workflows synchronize credentials from external vaults using operators:

Secret Manager
  -> ExternalSecret
  -> Kubernetes Secret
  -> ServiceAccount
  -> Deployment

Dedicated tools such as External Secrets Operator, Sealed Secrets, Mozilla SOPS, or HashiCorp Vault manage the lifecycle, rotation, and injection of registry secrets.

Production implementation

The following declarative manifests provide configurations for namespaces, Secrets, ServiceAccounts, Deployments, and ExternalSecret integrations.

Complete declarative configuration defining the namespace, registry Secret, ServiceAccount, and private container Deployment:

apiVersion: v1
kind: Namespace
metadata:
  name: demo

---
apiVersion: v1
kind: Secret
metadata:
  name: ghcr-secret
  namespace: demo
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: BASE64_DOCKER_CONFIG_JSON

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: demo
imagePullSecrets:
  - name: ghcr-secret

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: private-api
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-api
  template:
    metadata:
      labels:
        app: private-api
    spec:
      serviceAccountName: app-service-account
      containers:
        - name: private-api
          image: ghcr.io/my-org/private-api:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Production configuration referencing a versioned image path (ghcr.io/octocat/order-service:2026.05.14) in a dedicated namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production

---
apiVersion: v1
kind: Secret
metadata:
  name: ghcr-secret
  namespace: production
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: BASE64_DOCKER_CONFIG_JSON

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-service-sa
  namespace: production
imagePullSecrets:
  - name: ghcr-secret

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      serviceAccountName: order-service-sa
      containers:
        - name: order-service
          image: ghcr.io/octocat/order-service:2026.05.14
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Direct imagePullSecrets reference in the Pod specification for standalone workloads:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: private-api
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-api
  template:
    metadata:
      labels:
        app: private-api
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: private-api
          image: ghcr.io/my-org/private-api:1.0.0

Automated secret synchronization using External Secrets Operator:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ghcr-secret
  namespace: demo
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: cluster-secret-store
    kind: ClusterSecretStore
  target:
    name: ghcr-secret
    creationPolicy: Owner
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "ghcr.io": {
                "username": "{{ .githubUsername }}",
                "password": "{{ .githubToken }}",
                "email": "you@example.com",
                "auth": "{{ printf "%s:%s" .githubUsername .githubToken | b64enc }}"
              }
            }
          }
  data:
    - secretKey: githubUsername
      remoteRef:
        key: github-username
    - secretKey: githubToken
      remoteRef:
        key: github-token

Architectural trade-offs and edge cases

Managing private image authentication across Kubernetes environments involves operational trade-offs regarding credential scope and secret distribution.

Latency versus consistency: Attaching image pull secrets to ServiceAccounts centralizes credential maintenance and eliminates manifest duplication. ServiceAccount secret inheritance applies at Pod creation time; updating a Secret does not invalidate already running images on nodes until a restart or tag re-evaluation occurs.
Failure recovery: If a GitHub personal access token expires or is revoked, kubelet nodes cannot pull images when scheduling new Pods or scaling Deployments. Clusters should configure monitoring on ImagePullBackOff events and automate token rotations using external secret operators.
Scale limitations: Kubernetes Secrets are namespace-scoped. Sharing registry credentials across hundreds of isolated tenant namespaces requires automated secret replication controllers (such as Reflector or External Secrets Operator) or node-level credential providers, as Secrets cannot be cross-referenced across namespace boundaries.

Common anti-patterns and gotchas

Cross-namespace Secret referencing: Creating ghcr-secret in the default namespace and attempting to reference it from workloads in demo fails because Kubernetes Secrets are strictly namespace-scoped.
Insufficient token scope: Generating a GitHub personal access token without the read:packages permission scope causes authentication to fail with HTTP 401 Unauthorized.
Unconfigured package repository visibility: Failing to grant the token access to the specific organization or repository package settings blocks pull operations even with valid user credentials.
Committing base64 Secret manifests to Git: Storing base64-encoded Secret YAMLs in version control exposes plaintext GitHub tokens to anyone with repository read access.
Divergent ServiceAccount mappings: Specifying imagePullSecrets on a Deployment while assigning a ServiceAccount that references a missing Secret causes pull failures.
Malformed registry image URIs: Misspelling the registry domain (ghcr.io), organization name, or repository slug produces immediate image resolution errors.
Mutable tag references: Relying on mutable tags like :latest instead of immutable, versioned tags or sha256 digests leads to non-deterministic deployments across cluster nodes.

Implementation checklist

Generate a GitHub personal access token or deploy token configured with the read:packages scope.
Grant the token read permissions within the target GitHub organization or repository settings.
Construct the Docker configuration JSON with base64-encoded username:token auth strings.
Deploy the kubernetes.io/dockerconfigjson Secret into the destination Kubernetes namespace.
Create a ServiceAccount referencing the registry Secret under imagePullSecrets.
Configure workload Deployments to declare serviceAccountName pointing to the authenticated ServiceAccount.
Integrate an external secret manager such as External Secrets Operator or Sealed Secrets to automate credential rotation.
Define dedicated ServiceAccounts per workload or namespace according to the principle of least privilege.
Automate image digest pinning and tag promotion through your CI/CD delivery pipeline.
Establish an automated rotation policy for GitHub personal access tokens and deploy keys.
Configure alerting and monitoring for ImagePullBackOff and ErrImagePull events.

SQL-First-in-Migrations: Governing Every Database Artifact Through EF Core Migrations

ZèD — Tue, 12 May 2026 11:12:11 +0000

SQL-first in migrations: governing every database artifact through EF Core migrations

Entity Framework Core code-first workflows frequently treat migrations as if they only cover entity classes and relational table mappings. In practice, production databases rely heavily on views, stored procedures, user-defined functions, triggers, vendor-specific indexes, and operational baseline data that cannot be modeled cleanly through the Fluent API. Managing these non-entity artifacts outside the migration pipeline creates configuration drift, untracked manual schema mutations, and broken deployment pipelines across environments.

Governing every database artifact through MigrationBuilder.Sql with embedded, versioned SQL files establishes the EF Core migration timeline as the single source of truth for all schema structures, procedural logic, and baseline reference data.

The problem and production context

Treating database artifacts as secondary scripts executed manually or via out-of-band tooling disrupts deterministic schema deployment.

Failure scenario: An application deployment runs automated EF Core table migrations, but an unmigrated dependent stored procedure or view is missing in the production database, causing runtime relational query failures when procedures reference non-existent columns.
Why default approaches fall short: Embedding multiline raw SQL strings directly inside C# migration classes causes escaping syntax bugs, impairs code reviews, lacks SQL syntax highlighting, and frequently results in omitted or untested Down() rollback logic.
Production impact: Rollbacks fail during automated CI/CD deployment aborts, environment provisioning requires manual database administrator interventions, and database logic drifts across staging and production instances.

Extending the MigrationBuilder timeline guarantees that views, stored procedures, functions, triggers, custom indexes, and baseline seed records execute in an auditable, deterministic sequence across every environment.

Mental model and core concepts

A SQL-first migration pattern separates C# migration orchestration from raw SQL scripts while binding their lifecycles to the unified migration history.

1. Unified migration timeline

EF Core tracks applied migrations in the __EFMigrationsHistory table. By wrapping raw DDL scripts inside MigrationBuilder.Sql(), custom SQL artifacts share the exact transactional and ordering guarantees as table alterations. Spinning up fresh environments from source control applies entity schemas and custom database logic in lockstep.

2. Immutable versioned script hierarchy

SQL scripts reside in dedicated subdirectories organized by artifact category and version. Once merged and deployed, a version directory (such as v1) remains strictly immutable. Subsequent alterations introduce new version folders (v2), preserving historical progression and preventing retroactive history tampering:

YourApp.Persistence/
  Migrations/
    20260421062959_SeedBaseCategories.cs
    20260501000000_AddCalculateOrderTotalsStoredProc.cs
    20260505120000_AddGetUserFullNameFunction.cs
    20260510120000_AddActiveUsersView.cs
  SqlScripts/
    StoredProcedures/
      CalculateOrderTotals/
        v1/
          Up.sql
          Down.sql
    Functions/
      GetUserFullName/
        v1/
          Up.sql
          Down.sql
    Views/
      ActiveUsers/
        v1/
          Up.sql
          Down.sql
    SeedData/
      BaseCategories/
        v1/
          Up.sql
          Down.sql
  Helpers/
    SqlFileLoader.cs

3. Embedded resource extraction

Embedding SQL files as assembly resources ensures scripts are compiled directly into the persistence assembly. At runtime during migration execution, SqlFileLoader resolves scripts from the manifest resource stream using predictable dot-delimited namespaces, avoiding external file path dependencies in containerized runners.

4. Deterministic bidirectional execution

Every migration pairing an Up.sql with a matching Down.sql ensures full reversibility. Rollback logic is codified and verified in continuous integration pipelines, eliminating manual cleanup steps when reversing a release.

Production implementation

The following implementation defines the embedded resource loader and coordinates migrations for stored procedures, functions, views, concurrent indexes, and baseline reference data.

The resource loader extracts SQL text from embedded assembly resources:

using System.Reflection;

namespace YourApp.Persistence.Helpers;

public static class SqlFileLoader
{
    private const string RootNamespace = "YourApp.Persistence.SqlScripts";

    public static string Read(string relativePath)
    {
        var resourceName = $"{RootNamespace}.{relativePath.Replace('\\', '.').Replace('/', '.')}";

        using var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName)
            ?? throw new FileNotFoundException($"Embedded SQL script not found: {resourceName}");

        using var reader = new StreamReader(stream);

        return reader.ReadToEnd();
    }
}

The C# migration class orchestrates deployment by loading versioned scripts:

using Microsoft.EntityFrameworkCore.Migrations;
using YourApp.Persistence.Helpers;

public partial class AddCalculateOrderTotalsStoredProc : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.Sql(
            SqlFileLoader.Read("StoredProcedures/CalculateOrderTotals/v1/Up.sql"));
    }

    protected override void Down(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.Sql(
            SqlFileLoader.Read("StoredProcedures/CalculateOrderTotals/v1/Down.sql"));
    }
}

The corresponding stored procedure scripts calculate line item totals and update order records:

StoredProcedures/CalculateOrderTotals/v1/Up.sql:

CREATE OR REPLACE PROCEDURE calculate_order_totals()
LANGUAGE plpgsql
AS $$
BEGIN
    UPDATE orders o
    SET total_amount = sub.total
    FROM (
        SELECT order_id, SUM(price * quantity * (1 - discount)) AS total
        FROM order_items
        GROUP BY order_id
    ) sub
    WHERE o.id = sub.order_id;
END;
$$;

StoredProcedures/CalculateOrderTotals/v1/Down.sql:

DROP PROCEDURE IF EXISTS calculate_order_totals();

Scalar function scripts for reporting expressions that LINQ cannot translate efficiently:

Functions/GetUserFullName/v1/Up.sql:

CREATE OR REPLACE FUNCTION get_user_full_name(first_name text, last_name text)
RETURNS text
LANGUAGE sql
IMMUTABLE
AS $$
    SELECT TRIM(first_name || ' ' || last_name);
$$;

Functions/GetUserFullName/v1/Down.sql:

DROP FUNCTION IF EXISTS get_user_full_name(text, text);

Database view scripts standardizing soft-delete filtering across reporting tools:

Views/ActiveUsers/v1/Up.sql:

CREATE VIEW active_users AS
SELECT id, email, first_name, last_name, created_at
FROM users
WHERE is_deleted = false;

Views/ActiveUsers/v1/Down.sql:

DROP VIEW IF EXISTS active_users;

For materialized views, Up.sql executes CREATE MATERIALIZED VIEW alongside supporting indexes, while Down.sql drops the view cleanly.

Custom DDL for vendor-specific index creation not supported by Fluent API modeling:

migrationBuilder.Sql("CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_events_timestamp ON events (timestamp);");

Baseline reference data scripts applying static lookup records:

SeedData/BaseCategories/v1/Up.sql:

INSERT INTO product_categories (id, name)
VALUES
    (1, 'Electronics'),
    (2, 'Books'),
    (3, 'Clothing')
ON CONFLICT (id) DO NOTHING;

SELECT setval('product_categories_id_seq', (SELECT MAX(id) FROM product_categories));

SeedData/BaseCategories/v1/Down.sql:

DELETE FROM product_categories WHERE id IN (1, 2, 3);

Architectural trade-offs and edge cases

Governing raw SQL through migrations introduces trade-offs between database-native optimizations and provider portability.

Latency versus consistency: Executing procedural SQL and complex views inside migrations enforces strict transactional consistency during schema deployment, but large table backfills or unindexed view definitions can block schema migration transactions.
Failure recovery: Migrations wrapped in a transaction roll back automatically on DDL failure where supported. Non-transactional statements such as PostgreSQL CREATE INDEX CONCURRENTLY cannot run inside an ambient migration transaction and require suppressTransaction: true in migrationBuilder.Sql().
Scale limitations: When supporting multiple database engines (such as PostgreSQL and SQL Server), SQL files must be duplicated per dialect or selected dynamically via migrationBuilder.ActiveProvider (e.g., Up.psql.sql and Up.sqlserver.sql).

Common anti-patterns and gotchas

Embedding raw multiline SQL strings in C# classes: Inlining raw SQL inside C# strings breaks IDE syntax validation, causes string escaping errors, and produces noisy Git pull request diffs. Keep SQL isolated in external .sql files.
Mutating existing versioned scripts after deployment: Modifying an already-merged v1/Up.sql file causes new environments to deploy different logic than existing production instances. Always create a new version folder (v2) for schema modifications.
Omitting Down.sql rollback scripts: Failing to define matching drop or delete logic in Down() prevents automated CI/CD rollback recovery. Always pair every Up.sql with a tested Down.sql.
Forgetting the embedded resource build action: Leaving SQL files configured as content or none results in runtime FileNotFoundException during assembly execution. Ensure the project file sets the build action to EmbeddedResource.
Neglecting primary key sequence synchronization: Inserting hardcoded primary key identifiers in seed scripts without synchronizing database sequences causes subsequent application inserts to crash on primary key conflict. Always call setval() or equivalent sequence reset logic.

Implementation checklist

Scaffold an empty migration using dotnet ef migrations add AddArtifactName.
Clear auto-generated entity operations inside Up() and Down() methods.
Create versioned directories under SqlScripts/{Category}/{ArtifactName}/v1/.
Create Up.sql containing the forward DDL or DML logic.
Create Down.sql containing the reverse drop or rollback logic.
Set the Build Action of all .sql files to EmbeddedResource in the project file.
Call SqlFileLoader.Read() inside Up() and Down() passing the relative script paths.
Execute dotnet ef database update against a local test database.
Execute a rollback to the previous migration using dotnet ef database update <PreviousMigrationName> to verify the Down() script.
Re-apply the migration and verify that views, functions, procedures, and seed records function correctly.
Review references including Microsoft EF Core Custom Migrations Operations, GitHub dotnet/efcore issue 34469, EntityFramework.Docs issue 694, Npgsql EF Core Provider schema documentation, Redgate Flyway and EF Core guides, SSW script versioning rules, and EFCore.MigrationExtensions.PostgreSQL.

Atomic Redis Value Replacement Without Downtime: The Temporary Key Pattern

ZèD — Wed, 29 Apr 2026 17:41:50 +0000

Atomic Redis value replacement without downtime: the temporary key pattern

When Redis stores sensitive runtime data such as authorization rules, feature flags, or live configuration, naive key updates can cause service outages. A simple DELETE followed by SET creates a window where the key does not exist, while running HSET on a live hash leaves stale fields behind.

The temporary key pattern uses the atomic RENAME command in Redis to swap data structures without downtime, guaranteeing zero missing-key windows and zero stale fields.

The problem and production context

Updating live data structures in place creates race conditions between write operations and concurrent client reads.

Failure scenario: An application updates configuration by executing KeyDeleteAsync("config:site") followed by StringSetAsync("config:site", newJson). During the interval between the two network operations, concurrent reader requests receive null or fallback defaults. If StringSetAsync fails or the network connection drops, the key remains missing until an administrator backfills it.
Why default approaches fall short: While cache entries tolerate transient misses, runtime configuration, permissions, and routing tables serve as sources of truth. Similarly, updating hashes in place with HSET modifies supplied fields without removing omitted fields. If an earlier record contained beta_access: true, that field persists after an update intended to revoke privileges. Adding members to a set shows the same defect: obsolete entries are never purged. Calling KeyDelete prior to HashSet or SetAdd reintroduces the empty key window.
Production impact: Downgraded accounts retain revoked permissions, critical microservices run with empty configuration fallbacks, and concurrent requests crash during routine deployments or cache warm-ups.

Mental model and core concepts

The temporary key pattern replicates filesystem atomic swaps at the Redis key-space level.

1. Two-step atomic swap mechanism

The replacement pattern consists of two sequenced phases:

Write the new dataset completely to a temporary key (such as key:tmp).
Run RENAME to replace the production key with the temporary key.

RENAME executes atomically on the Redis single-threaded execution pipeline. It removes the destination key if it exists and moves the temporary key into place within a single step. Readers observe either the old structure or the new structure, with no intermediate state or missing key interval.

2. Transaction batching with MULTI/EXEC

Executing writes and RENAME as individual network calls risks orphaning temporary keys if a network disconnect occurs mid-sequence. StackExchange.Redis transactions wrap writes, deletions, and renames inside a Redis MULTI/EXEC block. The transaction guarantees that either all commands execute together or the production key remains untouched.

3. Redis cluster hash tagging

In a clustered Redis environment, RENAME throws a cross-slot error if the source and destination keys map to different hash slots across cluster nodes. Enclosing the variable shard token in curly braces forces Redis to hash only the enclosed string:
{user:123}:permissions and {user:123}:permissions:tmp share the identical hash slot and execute on the same node without cross-slot errors.

4. Distributed lock coordination

The rename pattern prevents read inconsistency but does not prevent concurrent background workers from overwriting each other's temporary keys before swapping. Coordinating updates with a distributed lock acquired via LockTakeAsync with a unique GUID token ensures that only one worker writes and renames at any moment.

Production implementation

The following implementation provides generic atomic string replacement, set replacement, hash replacement, and distributed lock synchronization using StackExchange.Redis.

using System;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
using StackExchange.Redis;

public class RedisAtomicReplacer
{
    public async Task<bool> ReplaceAsync<T>(
        IDatabase db, string key, T value, TimeSpan? expiry = null)
    {
        string tempKey = $"{key}:tmp";
        TimeSpan ttl = expiry ?? TimeSpan.FromHours(1);

        var txn = db.CreateTransaction();
        var setTask = txn.StringSetAsync(
            tempKey, JsonSerializer.Serialize(value), ttl);
        var renameTask = txn.KeyRenameAsync(tempKey, key);

        bool committed = await txn.ExecuteAsync();
        await Task.WhenAll(setTask, renameTask);

        return committed;
    }

    public async Task ReplaceSetAsync(
        IDatabase db, string key, string[] members)
    {
        string tempKey = $"{key}:tmp";

        var txn = db.CreateTransaction();
        txn.KeyDeleteAsync(tempKey);
        txn.SetAddAsync(tempKey, members.Select(m => (RedisValue)m).ToArray());
        txn.KeyRenameAsync(tempKey, key);

        await txn.ExecuteAsync();
    }

    public async Task ReplaceHashAsync(
        IDatabase db, string key, HashEntry[] entries)
    {
        string tempKey = $"{key}:tmp";

        var txn = db.CreateTransaction();
        txn.KeyDeleteAsync(tempKey);
        txn.HashSetAsync(tempKey, entries);
        txn.KeyRenameAsync(tempKey, key);

        await txn.ExecuteAsync();
    }

    public async Task<bool> ExecuteWithLockAsync(
        IDatabase db, string lockKey, TimeSpan lockExpiry, Func<Task> action)
    {
        string token = Guid.NewGuid().ToString("N");
        bool acquired = await db.LockTakeAsync(lockKey, token, lockExpiry);

        if (!acquired)
        {
            return false;
        }

        try
        {
            await action();
            return true;
        }
        finally
        {
            await db.LockReleaseAsync(lockKey, token, CommandFlags.FireAndForget);
        }
    }

    public async Task SafeReplaceSetAsync(
        IDatabase db, string key, string[] members)
    {
        string lockKey = $"lock:{key}";
        TimeSpan lockExpiry = TimeSpan.FromSeconds(10);

        bool lockAcquired = await ExecuteWithLockAsync(
            db, lockKey, lockExpiry, () => ReplaceSetAsync(db, key, members));

        if (!lockAcquired)
        {
            throw new InvalidOperationException("Could not acquire lock for replacement");
        }
    }
}

Architectural trade-offs and edge cases

The temporary key pattern introduces specific operational trade-offs and concurrency requirements.

Latency versus consistency: Atomic renames guarantee read consistency and eliminate stale or missing states. However, building the entire data structure under a temporary key requires double the memory allocation in Redis during the swap phase. For large collections, this transient memory overhead can trigger key eviction policies or out-of-memory errors.
Failure recovery: If ExecuteAsync returns false, the transaction aborted and the production key remains unchanged. When an application process crashes after writing a :tmp key but before issuing RENAME, the orphaned temporary key consumes memory until expired. Configuring a TTL via KeyExpireAsync or running a periodic janitor sweep over *:tmp keys recovers orphaned storage.
Scale limitations: When replacing massive sets or hashes containing millions of entries, creating the complete temporary structure in a single transaction can block the Redis event loop. In massive dataset scenarios, versioned keys (config:v1, config:v2) combined with a pointer swap provide an alternative to bulk in-memory copies.

Common anti-patterns and gotchas

Unsynchronized concurrent writers: Executing temporary key writes without a distributed lock allows concurrent workers to interleave writes into the same :tmp key, causing corrupted payloads to be swapped into production.
Missing hash tags in cluster mode: Swapping keys in Redis Cluster without wrapping the shard key in curly braces ({user:123}) causes cross-slot exceptions because the main key and the temporary key hash to different physical slots.
In-place hash updates for complete replacements: Using HSET to replace a user profile or configuration record without clearing missing fields preserves deprecated or revoked attributes indefinitely.
Delete-then-set workflows: Deleting a live key before writing the replacement introduces an unhedged latency window where concurrent reader threads encounter null references.

Implementation checklist

Identify critical keys that serve as sources of truth where missing or stale data causes outages.
Ensure clustered environments use hash tags (e.g., {tenant:id}:config) on all main and temporary keys.
Replace direct KeyDelete and StringSet sequences with transactional temporary key writes and atomic renames.
Clean up leftover temporary keys at the start of replacement transactions to prevent stale merges.
Set explicit TTLs on temporary keys to prevent orphaned memory usage in the event of unexpected worker process crashes.
Guard multi-worker update paths with LockTakeAsync using unique tokens to serialize writes.
Verify that fallback configurations or null checks do not trigger during background payload swaps.

Implementing Soft Delete with Filtered Indexes in Entity Framework Core

ZèD — Wed, 22 Apr 2026 16:32:18 +0000

Implementing soft delete with filtered indexes in Entity Framework Core

Soft delete implementations often fail in production because adding a simple IsDeleted boolean flag introduces unintended side effects across queries, unique constraints, and foreign key cascades. When applications execute standard delete operations, downstream relations can break unexpectedly, or unique index violations can block new active records from reusing identifying values like email addresses or usernames.

This pattern couples explicit domain-level entity contracts with Entity Framework Core change tracking interception, automated global query filters, and database-level filtered unique indexes. The architecture guarantees that standard deletion APIs safely convert into state updates, active reads systematically exclude soft-deleted rows, and unique constraints remain enforceable on active data without sacrificing historical associations.

The problem and production context

In production databases, naive soft delete implementations create friction between domain models, persistence layers, and schema constraints. When a system sets a boolean flag without infrastructure support, developers must remember to filter out deleted rows in every single query. Standard unique database constraints also reject valid insert operations when new entities share unique keys with inactive, soft-deleted records.

Failure scenario: An application attempts to register a new user using an email address that previously belonged to a deleted account. The database raises a unique constraint violation error because the naive unique index includes soft-deleted rows. Alternatively, a developer writes an entity query and forgets to append .Where(x => !x.IsDeleted), exposing logically deleted data in production user interfaces and reports.
Why default approaches fall short: Default unique indexes enforce uniqueness across all rows regardless of deletion status. In addition, manual filtering across ad-hoc queries is prone to human error, and relying on database cascading hard-deletes strips historical associations and breaks soft-delete audit trails.
Production impact: Preventable unique constraint violations block customer registrations and data entry. Accidental data leaks occur when omitted query filters expose sensitive soft-deleted rows, and untracked database cascades physically purge records intended for regulatory audit and operational recovery.

A complete soft delete implementation requires:

Default queries that systematically hide deleted rows across all application read operations.
A write path that transparently converts deletes into updates without breaking service-layer abstractions.
Database-level unique constraints that ignore soft-deleted rows.
An explicit restore path that maintains internal entity state consistency.

Mental model and core concepts

1. Domain encapsulation via explicit contracts

Soft delete is a domain concern rather than a database-only trick. The domain model should explicitly express that an entity can be deleted and restored. An interface defines the minimal contract without forcing entities into a rigid inheritance tree:

public interface ISoftDeletable
{
    bool IsDeleted { get; }
    DateTimeOffset? DeletedAt { get; }

    void Delete();
    void Restore();
}

A base entity provides a default implementation that encapsulates state transitions and prevents invalid mutations:

public abstract class BaseEntity : ISoftDeletable
{
    public Guid Id { get; private set; } = Guid.NewGuid();
    public bool IsDeleted { get; private set; }
    public DateTimeOffset? DeletedAt { get; private set; }

    public void Delete()
    {
        if (IsDeleted)
        {
            return;
        }

        IsDeleted = true;
        DeletedAt = DateTimeOffset.UtcNow;
    }

    public void Restore()
    {
        if (!IsDeleted)
        {
            return;
        }

        IsDeleted = false;
        DeletedAt = null;
    }
}

This encapsulates deletion and restoration logic within the entity itself.

2. Transparent delete interception via change tracking

To preserve the standard EF Core delete API, intercept entities marked as Deleted before changes persist to the database. Overriding SaveChanges and SaveChangesAsync converts entries into Modified state and invokes the domain Delete() method:

using System.Linq;
using Microsoft.EntityFrameworkCore;

public class ApplicationDbContext : DbContext
{
    private void ApplySoftDelete()
    {
        foreach (var entry in ChangeTracker.Entries<ISoftDeletable>()
                     .Where(e => e.State == EntityState.Deleted))
        {
            entry.State = EntityState.Modified;
            entry.Entity.Delete();
        }
    }

    public override int SaveChanges()
    {
        ApplySoftDelete();
        return base.SaveChanges();
    }

    public override Task<int> SaveChangesAsync(CancellationToken cancellationToken = default)
    {
        ApplySoftDelete();
        return base.SaveChangesAsync(cancellationToken);
    }
}

The service layer continues using the standard deletion call:

context.Users.Remove(user);
await context.SaveChangesAsync();

From the perspective of calling code, the entity is removed. In the database, the row remains with its deletion flags updated. If a project already uses EF Core interceptors for auditing or multi-tenancy, this logic can run inside an interceptor instead.

3. Global query filters and automated model conventions

Global query filters keep deleted rows out of standard queries. Configuring filters manually on individual entities becomes repetitive and error-prone across large models. A reflection-driven convention pass in OnModelCreating automatically binds query filters, configures filtered unique indexes, and sets foreign key delete behaviors to DeleteBehavior.NoAction across all ISoftDeletable entities:

using System.Linq;
using System.Linq.Expressions;

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.ApplyConfigurationsFromAssembly(typeof(SomeEntityConfiguration).Assembly);

    foreach (var entityType in modelBuilder.Model.GetEntityTypes())
    {
        if (typeof(ISoftDeletable).IsAssignableFrom(entityType.ClrType))
        {
            var parameter = Expression.Parameter(entityType.ClrType, "e");
            var isDeleted = Expression.Property(parameter, nameof(ISoftDeletable.IsDeleted));
            var isActive = Expression.Equal(isDeleted, Expression.Constant(false));
            var filter = Expression.Lambda(isActive, parameter);

            modelBuilder.Entity(entityType.ClrType).HasQueryFilter(filter);

            foreach (var index in entityType.GetIndexes().Where(i => i.IsUnique && i.GetFilter() is null))
            {
                index.SetFilter($"[{nameof(ISoftDeletable.IsDeleted)}] = 0");
            }
        }

        foreach (var foreignKey in entityType.GetForeignKeys())
        {
            foreignKey.DeleteBehavior = DeleteBehavior.NoAction;
        }
    }

    base.OnModelCreating(modelBuilder);
}

This centralizes persistence conventions in OnModelCreating instead of configuring each entity manually. Normal reads exclude deleted records without manual .Where(u => !u.IsDeleted) conditions.

When deleted records are required, disable the filter using IgnoreQueryFilters():

var deletedUser = await context.Users
    .IgnoreQueryFilters()
    .FirstOrDefaultAsync(u => u.Id == userId && u.IsDeleted);

This provides a clear escape hatch for audit logs, administrative portals, and restore routines.

4. Database filtered unique indexes

Soft-delete designs encounter constraint issues with standard unique database indexes. If an Email column has an unconditional unique constraint, a soft-deleted row prevents a new user from registering with that same email address. Applying a partial index filter restricts uniqueness checks to active records.

For SQL Server:

index.SetFilter($"[{nameof(ISoftDeletable.IsDeleted)}] = 0");

For PostgreSQL, the equivalent filter is:

index.SetFilter($"\"{nameof(ISoftDeletable.IsDeleted)}\" = false");

Key considerations for unique indexes:

Restrict uniqueness checks strictly to active records.
Avoid relying on application-level checks to enforce uniqueness.
Verify generated migration scripts against your database provider.

Setting foreign key relationships to DeleteBehavior.NoAction during the same pass prevents soft deletes from triggering unintended database cascades.

Production implementation

The following workflow restores soft-deleted records through explicit domain entity methods combined with bypassing global query filters.

var user = await context.Users
    .IgnoreQueryFilters()
    .FirstOrDefaultAsync(u => u.Id == userId);

if (user is null)
{
    return;
}

user.Restore();
await context.SaveChangesAsync();

This guarantees that entity state transitions remain encapsulated within the domain model while EF Core tracks the update.

Architectural trade-offs and edge cases

Latency versus consistency: Global query filters inject an additional boolean condition into every generated SQL query. In return, the system guarantees that soft-deleted rows never leak into standard queries without requiring manual filter enforcement across thousands of application queries.
Failure recovery: Restoring a soft-deleted record when another active record has since taken the same unique value (such as an email address) causes a database unique constraint violation. Restoration services must catch unique constraint exceptions or perform validation before invoking Restore().
Scale limitations: When tables accumulate millions of soft-deleted records over time, table size and scan performance degrade if queries touch unindexed columns. A dedicated partitioning strategy or background archival job is required to purge or move long-deleted rows into cold storage.
Bulk operations: Bulk operations such as EF Core's ExecuteDelete or raw SQL commands bypass the change tracker entirely and do not trigger SaveChanges overrides or interceptors. Physical deletes occur unless separate database triggers or explicit update scripts run.
Relationship navigation: Navigating required relationships (Include) where the target related record is soft-deleted can cause queries to return null or filter out parent records entirely. Relationships must be tested to verify intended join behavior under global query filters.
Hard delete requirements: Soft delete handles recovery, audit trails, and referential integrity. However, when compliance rules (such as GDPR or CCPA) mandate physical data destruction, a dedicated hard-delete administrative routine remains necessary.

Common anti-patterns and gotchas

Indexing IsDeleted in isolation: Developers add a standalone index on IsDeleted. Because boolean flags possess very low selectivity, database query planners ignore this index during query execution. Filtered indexes on candidate keys should be used instead.
Relying on application-level uniqueness checks: Developers check for duplicate entries in application code before inserting rather than using filtered database indexes. Concurrent write operations bypass application checks, producing duplicate active records and corrupting data.
Allowing default cascade behaviors on foreign keys: Default EF Core cascade rules may attempt to delete child records or throw constraint exceptions when parent records are marked deleted. Setting foreign key relationships to DeleteBehavior.NoAction prevents unintended cascading deletes.
Assuming bulk operations trigger soft delete: Developers assume calling context.Users.Where(...).ExecuteDeleteAsync() executes soft delete logic. Bulk operations bypass the change tracker and permanently delete records from the physical table.
Scattering manual filter clauses: Writing .Where(x => !x.IsDeleted) manually across individual queries inevitably results in omitted filters and production data leaks. Global query filters must be enforced by convention.

Implementation checklist

Define the ISoftDeletable interface and implement it on base or domain entities with encapsulated state mutators.
Override SaveChanges and SaveChangesAsync in DbContext (or implement an ISaveChangesInterceptor) to intercept EntityState.Deleted and convert to EntityState.Modified.
Register automated global query filters in OnModelCreating using expression trees across all ISoftDeletable entities.
Configure filtered partial unique indexes for your target database engine (SQL Server or PostgreSQL) across all candidate keys.
Set foreign key delete behaviors to DeleteBehavior.NoAction to prevent unintended database cascades.
Verify generated database migration scripts to confirm filtered index definitions match the target database syntax.
Test queries that use Include on required relationships to confirm join behaviors under global query filters.
Establish an explicit administrative archival path for GDPR compliance and cold storage data purging.

Building Dynamic Filters in Clean Architecture (CQRS) using ExpressionBuilder

ZèD — Wed, 25 Mar 2026 16:01:12 +0000

Building dynamic filters in Clean Architecture with ExpressionBuilder

Dynamic filtering in CQRS applications frequently leads to architectural leaks when handlers accept or return IQueryable interfaces. In Clean Architecture, query compilation and execution belong inside the infrastructure layer, while filter definitions belong in application contracts.

Using an ExpressionBuilder<T> utility composes dynamic predicate expressions into unified expression trees using parameter-rewriting visitors, ensuring complete architectural isolation without leaking database dependencies into domain handlers.

The problem and production context

Constructing optional query filters across multiple API parameters introduces condition branching and parameter coupling.

Failure scenario: An application handler naive-chains lambda expressions by combining expressions without rewriting parameter references, or by passing an IQueryable query across layer boundaries. EF Core throws an unhandled InvalidOperationException at runtime when the LINQ expression provider attempts to translate mismatched lambda parameter nodes into SQL queries.
Why default approaches fall short: Direct variable reassignment overwrites previously evaluated filters instead of composing them. Alternatively, passing IQueryable<T> through the application layer exposes database engine abstractions, prevents unit testing handlers in isolation, and allows domain layers to trigger arbitrary deferred queries.
Production impact: Unhandled runtime query translation failures crash API search endpoints, while leaky abstractions force EF Core package references into domain and application projects, violating dependency inversion boundaries.

Applying dynamic expression composition provides key structural benefits:

Keeps query filtering logic inside the application layer without referencing EF Core packages.
Allows repository methods to accept a single predicate expression instead of numerous optional filter parameters.
Eliminates repetitive conditional statements across query handlers and services.
Produces valid expression trees that LINQ providers can translate into parameterized SQL.

Mental model and core concepts

Building composable predicate expressions across architectural layers requires parameter normalization and expression tree rewriting.

1. Passing expression predicates across layer boundaries

The application layer defines what data is requested via strongly typed lambda expressions without dictating how the infrastructure layer executes or indexes the query:

Task<IReadOnlyList<TEntity>> ListAsync(
    Expression<Func<TEntity, bool>>? filter = null,
    CancellationToken cancellationToken = default);

This contract establishes a strict boundary: the application layer constructs the expression tree, and the infrastructure repository evaluates it against the database provider.

2. Failure modes of naive expression assignment

Naive optional filtering typically reassigns expression variables in sequence:

Expression<Func<User, bool>>? filter = null;
if (request.IsActive.HasValue)
    filter = x => x.IsActive == request.IsActive.Value;
if (!string.IsNullOrWhiteSpace(request.Name))
    filter = x => x.Name.Contains(request.Name);

In this pattern, subsequent evaluations overwrite earlier criteria rather than joining them with logical AND operators.

3. Expression collection and composition

ExpressionBuilder<T> maintains a sequence of independent predicate expressions and merges them using logical AND (Expression.AndAlso):

var builder = new ExpressionBuilder<User>()
    .And(x => !x.IsDeleted)
    .And(x => x.Age >= 18 && x.Age <= 60)
    .And(x => x.Role == "Admin" || x.Role == "Manager");

Internal grouped precedence is preserved within each clause while separate clauses are joined sequentially at the root.

4. Parameter rewriting via ExpressionVisitor

Each compiled lambda expression instantiates its own parameter symbol (x => ..., y => ...). Merging distinct lambda bodies directly fails because the expressions evaluate different parameter instances. ExpressionBuilder<T> uses an ExpressionVisitor subclass to rewrite all parameter nodes to reference a single unified ParameterExpression.

Production implementation

The following implementation provides the complete ExpressionBuilder<T>, parameter rewriting visitor, CQRS query handler, and EF Core repository.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;

public sealed class ExpressionBuilder<T>
{
    private readonly List<Expression<Func<T, bool>>> _conditions = new();

    public ExpressionBuilder<T> And(Expression<Func<T, bool>> condition)
    {
        ArgumentNullException.ThrowIfNull(condition);

        _conditions.Add(condition);
        return this;
    }

    public ExpressionBuilder<T> AndIf(bool shouldAdd, Expression<Func<T, bool>> condition)
    {
        if (shouldAdd)
        {
            And(condition);
        }

        return this;
    }

    public Expression<Func<T, bool>>? Build()
    {
        if (_conditions.Count == 0)
        {
            return null;
        }

        var parameter = Expression.Parameter(typeof(T), "x");
        Expression combinedBody = RewriteBody(_conditions[0], parameter);

        for (var i = 1; i < _conditions.Count; i++)
        {
            var nextBody = RewriteBody(_conditions[i], parameter);
            combinedBody = Expression.AndAlso(combinedBody, nextBody);
        }

        return Expression.Lambda<Func<T, bool>>(combinedBody, parameter);
    }

    private static Expression RewriteBody(
        Expression<Func<T, bool>> sourceExpression,
        ParameterExpression targetParameter)
    {
        return new ReplaceParameterVisitor(sourceExpression.Parameters[0], targetParameter)
            .Visit(sourceExpression.Body)
            ?? throw new InvalidOperationException("Failed to rewrite filter expression.");
    }

    private sealed class ReplaceParameterVisitor : ExpressionVisitor
    {
        private readonly ParameterExpression _source;
        private readonly ParameterExpression _target;

        public ReplaceParameterVisitor(ParameterExpression source, ParameterExpression target)
        {
            _source = source;
            _target = target;
        }

        protected override Expression VisitParameter(ParameterExpression node)
        {
            return node == _source ? _target : base.VisitParameter(node);
        }
    }
}

public sealed record SearchUsersQuery(bool? IsActive, string? Name);

public sealed class SearchUsersQueryHandler
{
    private readonly IUserRepository _userRepository;

    public SearchUsersQueryHandler(IUserRepository userRepository)
    {
        _userRepository = userRepository;
    }

    public Task<IReadOnlyList<User>> Handle(SearchUsersQuery request, CancellationToken cancellationToken)
    {
        var filter = new ExpressionBuilder<User>()
            .And(x => !x.IsDeleted)
            .AndIf(request.IsActive.HasValue, x => x.IsActive == request.IsActive!.Value)
            .AndIf(!string.IsNullOrWhiteSpace(request.Name), x => x.Name.Contains(request.Name!))
            .Build();

        return _userRepository.ListAsync(filter, cancellationToken);
    }
}

public interface IUserRepository
{
    Task<IReadOnlyList<User>> ListAsync(
        Expression<Func<User, bool>>? filter = null,
        CancellationToken cancellationToken = default);
}

public sealed class UserRepository : IUserRepository
{
    private readonly AppDbContext _dbContext;

    public UserRepository(AppDbContext dbContext)
    {
        _dbContext = dbContext;
    }

    public async Task<IReadOnlyList<User>> ListAsync(
        Expression<Func<User, bool>>? filter = null,
        CancellationToken cancellationToken = default)
    {
        IQueryable<User> query = _dbContext.Users;

        if (filter is not null)
        {
            query = query.Where(filter);
        }

        return await query.ToListAsync(cancellationToken);
    }
}

The builder cleanly handles various predicate compositions:

A single required condition:

var filter = new ExpressionBuilder<User>()
    .And(x => !x.IsDeleted)
    .Build();

Combining required and optional conditions:

var filter = new ExpressionBuilder<User>()
    .And(x => !x.IsDeleted)
    .AndIf(request.IsActive.HasValue, x => x.IsActive == request.IsActive!.Value)
    .AndIf(!string.IsNullOrWhiteSpace(request.Name), x => x.Name.Contains(request.Name!))
    .Build();

Grouped logic inside one block:

var filter = new ExpressionBuilder<User>()
    .And(x => !x.IsDeleted)
    .And(x => x.Role == "Admin" || x.Role == "Manager")
    .And(x => x.Age >= 18 && x.Age <= 60)
    .Build();

Architectural trade-offs and edge cases

Composing expressions programmatically involves trade-offs between layer purity and query translation flexibility.

Latency versus consistency: Building and visiting expression trees adds negligible CPU overhead in the application layer compared to the database query latency. Emitting null when no conditions exist allows the query planner to omit WHERE clauses completely, avoiding suboptimal SQL execution plans.
Failure recovery: If invalid member accesses or unmappable custom functions are inserted into expression trees, EF Core throws a translation exception during SQL generation. Handlers should validate input bounds prior to building expressions to prevent malformed queries from reaching infrastructure.
Scale limitations: When queries require complex multi-table navigations (Include), explicit projections (Select), or full-text search functions, simple predicate expressions become insufficient. Specification patterns or dedicated query handler queries should be utilized when operations exceed simple filter predicates.

Common anti-patterns and gotchas

Expression reassignment overwrite: Overwriting an expression variable sequentially instead of combining predicates drops earlier filter parameters.
Default true expressions: Returning a fallback x => true expression instead of null causes EF Core to append WHERE 1 = 1, which can hinder query plan compilation on certain database providers.
Mismatched lambda parameters: Combining lambda bodies without rewriting parameter references to a single ParameterExpression causes EF Core expression translation failures at runtime.
Leaking IQueryable across boundaries: Exposing IQueryable<T> from repository contracts enables handlers to execute arbitrary, unindexed database queries and couples application layers directly to ORM assemblies.
Embedding database mechanics in handlers: Writing direct SQL operations or EF Core-specific execution calls inside CQRS command or query handlers violates Clean Architecture boundaries.

Implementation checklist

Implement ExpressionBuilder<T> and the ReplaceParameterVisitor inside an application or core utility project.
Update repository interfaces to accept Expression<Func<TEntity, bool>>? filter = null with explicit CancellationToken support.
Write unit tests covering empty filter cases, single predicates, and multi-condition scenarios.
Extend ExpressionBuilder<T> with an Or method if your domain requires root-level disjunction.
Inspect the SQL queries generated by EF Core in logging outputs to verify index usage across combined conditions.
Consult documentation on expression trees in C# for advanced visitor patterns when extending tree rewriting.

Cursor vs Offset Pagination in EF Core: What Actually Works in Production

ZèD — Thu, 19 Mar 2026 10:18:33 +0000

Cursor vs offset pagination in EF Core: what works in production

Pagination is simple to implement at small scales, but query performance and data consistency degrade as tables grow into millions of rows. Under heavy read workloads, naive pagination patterns create database bottlenecks that exhaust database I/O, increase latency, and return inconsistent page windows to clients.

Keyset cursor pagination replaces arbitrary offset scans with index-seek lookups based on deterministic sort keys, guaranteeing constant-time page traversal and stable read windows under concurrent write traffic.

The problem and production context

Most applications begin with Skip and Take. While this pattern functions adequately for basic administrative pages with small datasets, it introduces structural performance degradation and data drift when datasets scale.

Failure scenario: A client queries deep pages such as page 5,000 with a page size of 20. Under offset pagination, the database engine must scan, sort, and discard the first 100,000 matching rows before streaming the requested 20 rows, which produces linear degradation in execution time. Concurrently, if a new record is inserted or deleted while a user navigates from page 1 to page 2, the window shifts by one row, surfacing duplicate records or skipping records across page boundaries.
Why default approaches fall short: Offset pagination relies on row offsets rather than immutable record identifiers. The database engine cannot jump directly to an arbitrary row number within a non-clustered index; it must traverse all preceding index entries up to the offset count.
Production impact: As query depth grows, database engines suffer excessive memory pressure, CPU saturation, and disk I/O exhaustion. Concurrent write activity causes data drift, corrupting user-facing feeds and downstream data export pipelines.

Mental model and core concepts

Selecting between offset and keyset pagination requires understanding how the database engine traverses ordered data and handles concurrent mutations.

1. Offset traversal mechanics

Offset pagination defines a query window using the current page number and page size:

var items = await context.Posts
    .AsNoTracking()
    .OrderByDescending(x => x.CreatedAt)
    .ThenByDescending(x => x.Id)
    .Skip((pageNumber - 1) * pageSize)
    .Take(pageSize)
    .ToListAsync();

The database executes this by ordering the entire matched result set, counting up to (pageNumber - 1) * pageSize rows, discarding them, and returning the requested count. It accepts simple page and pageSize parameters, supports direct random-access jumps (such as jumping directly to page 5 or 20), and integrates into frontend grid components. Its execution time degrades linearly as the offset increases.

2. Keyset cursor seek mechanics

Cursor pagination (also called keyset pagination) tracks the specific column values of the last retrieved item instead of a numeric offset:

var items = await context.Posts
    .AsNoTracking()
    .OrderByDescending(x => x.CreatedAt)
    .ThenByDescending(x => x.Id)
    .Where(x => x.CreatedAt < lastCreatedAt
        || (x.CreatedAt == lastCreatedAt && x.Id < lastId))
    .Take(pageSize)
    .ToListAsync();

Instead of counting and discarding rows, the database engine uses the cursor values to seek directly into the B-tree index. Query latency remains constant between page 1 and page 10,000. Because the seek points directly to immutable keys, insertions or deletions ahead of the cursor do not shift the current result set window.

3. Deterministic sorting and tiebreakers

Both offset and keyset pagination require deterministic sorting. If the primary sort column contains duplicate values (such as identical timestamps), the database ordering becomes non-deterministic:

.OrderBy(x => x.CreatedAt)

To establish a stable order, a guaranteed-unique column such as the primary key must be appended to the sort clause:

.OrderBy(x => x.CreatedAt)
.ThenBy(x => x.Id)

Appending a unique secondary column prevents records with identical timestamps from shifting erratically across page boundaries.

4. Index alignment

Keyset queries require composite index alignment matching both the ORDER BY and WHERE filter predicates. Without an index covering the exact sort and tiebreaker columns, the database engine falls back to a full table scan and in-memory sort, eliminating the performance advantage of keyset seek logic.

Production implementation

Configure the composite index in Entity Framework Core and execute deterministic keyset pagination against the dataset.

public class PostConfiguration : IEntityTypeConfiguration<Post>
{
    public void Configure(EntityTypeBuilder<Post> builder)
    {
        builder.HasKey(x => x.Id);

        builder.Property(x => x.CreatedAt)
            .IsRequired();

        builder.HasIndex(x => new { x.CreatedAt, x.Id });
    }
}

public sealed class PostService
{
    private readonly AppDbContext _context;

    public PostService(AppDbContext context)
    {
        _context = context;
    }

    public async Task<List<Post>> GetPostsCursorAsync(
        DateTime? lastCreatedAt,
        int? lastId,
        int pageSize,
        CancellationToken cancellationToken)
    {
        if (pageSize <= 0 || pageSize > 100)
        {
            throw new ArgumentOutOfRangeException(nameof(pageSize), "Page size must be between 1 and 100.");
        }

        IQueryable<Post> query = _context.Posts
            .AsNoTracking()
            .OrderByDescending(x => x.CreatedAt)
            .ThenByDescending(x => x.Id);

        if (lastCreatedAt.HasValue && lastId.HasValue)
        {
            DateTime cursorCreatedAt = lastCreatedAt.Value;
            int cursorId = lastId.Value;

            query = query.Where(x => x.CreatedAt < cursorCreatedAt
                || (x.CreatedAt == cursorCreatedAt && x.Id < cursorId));
        }

        return await query
            .Take(pageSize)
            .ToListAsync(cancellationToken);
    }
}

Architectural trade-offs and edge cases

Choosing between offset and cursor pagination involves distinct trade-offs across query complexity, UI requirements, and data stability.

Scenario	Offset pagination	Cursor pagination
Small datasets	Fast	Fast
Deep page offsets	Degrades linearly	Constant time
High write concurrency	Susceptible to duplicate/skipped items	Stable pagination window
Sequential navigation	Moderate	Optimal

Latency versus consistency: Offset pagination provides simple random-access navigation at the expense of linear query degradation and window drift under concurrent writes. Cursor pagination provides constant latency and window consistency across deep traversals, but requires clients to retain state and supply the last record's sort keys for each subsequent page.
Failure recovery: Cursor pagination tolerates client-side disconnects and retries because passing the same cursor token returns the exact subsequent page slice, regardless of how many rows were inserted prior to the cursor point.
Scale limitations: Offset pagination is bounded by table size and read volume; it remains appropriate for back-office admin grids, customer support ticket lists, and small configuration tables where users require explicit numbered page controls and direct random-access jumps. Keyset cursor pagination scales to millions of records and fits infinite-scroll interfaces, message streams, audit logs, and high-throughput public API endpoints.
Bi-directional navigation: Cursor pagination does not support jumping directly to arbitrary page numbers (such as navigating directly to page 14). Bi-directional navigation requires reversing the sort predicates and reordering the result slice in application memory.

Common anti-patterns and gotchas

Missing composite index: Developers implement keyset WHERE clauses matching CreatedAt and Id without creating a matching composite index (CreatedAt, Id). The database engine performs a full table scan and in-memory sort, eliminating all keyset performance advantages. Always define matching composite indexes in database migrations.
Non-deterministic sort ordering: Developers sort exclusively by non-unique fields like CreatedAt or Status. When duplicate values span across page boundaries, rows are duplicated or omitted between queries. Always append a unique tiebreaker column such as the primary key.
Applying keyset pagination to arbitrary page jumping: Developers attempt to compute arbitrary page numbers with keyset pagination by chaining multiple offset operations. Keyset pagination inherently requires sequential navigation; if random jumping to page N is an absolute business requirement, use bounded offset pagination with strict maximum page limits.
State shift in offset pagination: Developers rely on Skip and Take for high-frequency activity feeds. New insertions push older items into subsequent offsets, forcing users to see duplicate items on every page scroll. Replace offset feeds with cursor pagination for all append-heavy event streams.

Implementation checklist

Audit target entity queries to determine whether the UI requires random page access or sequential infinite scrolling.
Add a composite index covering both the primary sort column and the unique tiebreaker column in EF Core model configurations.
Replace Skip with keyset filter expressions using tuple comparison or compound WHERE clauses.
Append the entity primary key as the final deterministic tiebreaker in all OrderBy chains.
Validate that API input constraints enforce upper bounds on requested page size.
Verify database execution plans via EXPLAIN or SQL Server execution plans to confirm an index seek replaces index scans across deep page queries.

Understanding EF.Functions.Like and ILike in Entity Framework Core

ZèD — Sat, 28 Feb 2026 05:21:15 +0000

Understanding EF.Functions.Like and ILike in Entity Framework Core

Pattern-based string search in Entity Framework Core often leads developers to apply standard .NET methods like string.ToLower() or LINQ string operations that wrap table columns in SQL scalar functions. This suppresses database index utilization, forces sequential table scans, or accidentally triggers client-side evaluation that transfers entire tables across the network.

Using EF.Functions.Like and PostgreSQL-specific EF.Functions.ILike translates pattern matching directly into native SQL relational operators, preserving index sargability and ensuring filtering executes entirely within the database engine.

The problem and production context

Writing string search filters in LINQ without understanding relational expression translation degrades database engine throughput.

Failure scenario: A search query executes .Where(u => u.Email.ToLower().Contains(input.ToLower())); the database applies the LOWER() function to every row in a multi-million row table, causing a full table scan that consumes all database I/O bandwidth and times out API endpoints.
Why default approaches fall short: Wrapping column references in functions renders queries non-sargable (unable to utilize standard B-tree indexes), while using EF.Functions.ILike on non-PostgreSQL database providers throws runtime translation exceptions.
Production impact: Unindexed wildcard queries trigger database CPU spikes, API response latencies surge from milliseconds to seconds, and unescaped wildcard characters allow arbitrary wildcards in user search inputs.

Ensuring string filtering executes inside the database engine, preventing accidental client evaluation, providing explicit control over case sensitivity across providers, and keeping wildcard patterns maintainable within LINQ establishes scalable query patterns.

Mental model and core concepts

Expression trees in EF Core map high-level C# expressions to provider-specific SQL abstract syntax tree nodes.

1. SQL LIKE translation mechanics

EF.Functions.Like(column, pattern) translates directly to the SQL LIKE operator across all relational providers. The pattern supports standard SQL wildcard operators:

%: Matches any sequence of zero or more characters.
_: Matches exactly one character.

2. PostgreSQL ILIKE operator

EF.Functions.ILike(column, pattern) is an extension provided by Npgsql.EntityFrameworkCore.PostgreSQL. It compiles directly to PostgreSQL's native ILIKE operator, providing case-insensitive pattern matching natively without wrapping columns in LOWER() and without modifying database-level collations.

3. Comparison with standard string methods

Standard LINQ string methods map to specific SQL patterns:

Contains("term") translates to LIKE '%term%'
StartsWith("term") translates to LIKE 'term%'
EndsWith("term") translates to LIKE '%term'

While standard methods handle fixed positions, EF.Functions.Like allows custom, composite wildcard combinations (such as _a% or %prefix%suffix%) inside a single parameterized expression.

4. Provider and collation sensitivity

Like conforms to the underlying column collation. On Microsoft SQL Server with default case-insensitive collations (e.g., SQL_Latin1_General_CP1_CI_AS), Like is case-insensitive. On PostgreSQL, default LIKE is strictly case-sensitive.
ILike is strictly PostgreSQL-specific. Invoking ILike against SQL Server, SQLite, MySQL, or Oracle throws an InvalidOperationException during query translation.

Production implementation

The following implementation demonstrates case-sensitive and case-insensitive pattern matching, wildcard queries, and safe repository search routines with input sanitization.

Standard wildcard queries using EF.Functions.Like:

using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;

public static async Task<List<User>> QueryUsersWithLikeAsync(AppDbContext context)
{
    var users = await context.Users
        .Where(u => EF.Functions.Like(u.Name, "%John%"))
        .ToListAsync();

    var usersByPrefix = await context.Users
        .Where(u => EF.Functions.Like(u.Name, "J%"))
        .ToListAsync();

    var usersByPattern = await context.Users
        .Where(u => EF.Functions.Like(u.Name, "_a%"))
        .ToListAsync();

    return users;
}

PostgreSQL queries using EF.Functions.ILike:

public static async Task<List<Product>> QueryProductsWithILikeAsync(AppDbContext context)
{
    var products = await context.Products
        .Where(p => EF.Functions.ILike(p.Description, "%widget%"))
        .ToListAsync();

    var productsByCategory = await context.Products
        .Where(p => EF.Functions.ILike(p.Category, "electronics"))
        .ToListAsync();

    return products;
}

Repository search method implementing wildcard escaping and case-insensitive search:

using System;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;

public sealed class UserRepository(AppDbContext context)
{
    private static string EscapeLikePattern(string input)
    {
        return Regex.Replace(input, @"([%_\[\]])", @"\$1");
    }

    public async Task<List<User>> SearchUsersAsync(string query, CancellationToken cancellationToken)
    {
        if (string.IsNullOrWhiteSpace(query))
            return new List<User>();

        var sanitized = EscapeLikePattern(query.Trim());
        var pattern = $"%{sanitized}%";

        return await context.Users
            .Where(u => EF.Functions.ILike(u.Name, pattern) || EF.Functions.ILike(u.Email, pattern))
            .ToListAsync(cancellationToken);
    }
}

Architectural trade-offs and edge cases

Choosing between native pattern matching, standard string methods, and full-text search introduces distinct architectural trade-offs.

Latency versus consistency: EF.Functions.Like executes in-database with strong transactional read consistency, but leading wildcards (%term) force full table or index scans unless specialized indexes are created.
Failure recovery: Queries executed against unsupported database providers fail at translation time. Applications supporting multiple database backends must abstract pattern generation or guard provider-specific calls via context.Database.IsNpgsql().
Scale limitations: For tables exceeding hundreds of thousands of rows, leading wildcard queries (%term%) degrade significantly on standard B-tree indexes. Scaling requires PostgreSQL pg_trgm GIN/GiST indexes or full-text search engines.

Common anti-patterns and gotchas

Calling ToLower or ToUpper in LINQ queries: Wrapping columns in .ToLower() causes EF Core to emit LOWER(column), invalidating standard B-tree indexes and forcing full table scans. Use EF.Functions.ILike on PostgreSQL or provider collations.
Invoking ILike on non-PostgreSQL providers: Calling EF.Functions.ILike on SQL Server, SQLite, or Oracle throws runtime translation exceptions because the operator is unique to the Npgsql provider. Use EF.Functions.Like with appropriate collations on non-PostgreSQL engines.
Unindexed leading wildcards on large tables: Searching with "%term%" cannot use standard B-tree index prefixes and triggers sequential scans across large tables. Create trigram indexes (pg_trgm) or dedicated full-text catalogs.
Failing to escape wildcard characters in user input: Passing raw user input directly into pattern interpolations allows users to enter % or _ to execute arbitrary wildcard scans. Always sanitize or escape reserved wildcard symbols.

Implementation checklist

Inspect existing LINQ queries for .ToLower() and .ToUpper() invocations and replace with EF.Functions.Like or EF.Functions.ILike.
Confirm the target database provider supports EF.Functions.ILike before deploying Npgsql-specific calls.
Profile search query performance using EXPLAIN ANALYZE on PostgreSQL or SQL Server execution plans.
Implement wildcard escaping routines to neutralize literal % and _ characters in user input strings.
Create PostgreSQL pg_trgm GIN indexes on columns requiring leading wildcard searches (LIKE '%term%').
Evaluate full-text search engines if requirements expand beyond substring and wildcard pattern matching.

BCrypt vs Argon2: Password Hashing in .NET – A Practical Deep Dive

ZèD — Sat, 28 Feb 2026 04:41:09 +0000

BCrypt vs Argon2 password hashing in .NET: a practical comparison

Password storage security depends on computationally expensive, intentionally slow cryptographic functions. Fast cryptographic hashes verify data integrity at high throughput, but using them for passwords exposes systems to rapid offline credential recovery attacks.

Choosing between BCrypt and Argon2 in .NET requires balancing CPU and memory hardness, hardware-assisted cracking resistance, interactive login latency, and progressive credential migration strategies.

The problem and production context

General-purpose hashing algorithms compromise user credentials when database storage layers are breached.

Failure scenario: An application stores user credentials using SHA-256 or SHA-512. Following an unauthorized database extraction, an attacker deploys consumer GPU or ASIC clusters capable of calculating billions of SHA evaluations per second. Weak and moderately complex passwords crack within minutes via offline dictionary and rainbow table lookups.
Why default approaches fall short: General-purpose cryptographic hash functions prioritize execution throughput and low resource utilization. When applied to passwords, this efficiency works entirely in favor of the attacker. Defending against offline attacks requires introducing artificial delays and hardware resource constraints that degrade GPU parallelism.
Production impact: Compromised databases result in account takeover across systems, credential stuffing cascades, and compliance violations. Conversely, misconfigured password hashers with excessively high cost parameters can trigger denial-of-service conditions during concurrent login spikes.

Key production considerations when selecting an algorithm include:

Password hashing algorithms must be intentionally slow and resource-intensive.
Attackers run parallelized brute-force attacks on specialized GPU and ASIC hardware.
The hashing algorithm you choose determines how effectively your system withstands compromised database dumps.
Safe default configurations reduce the risk of implementation flaws in authentication systems.

Mental model and core concepts

Modern password hashing functions prevent mass parallel cracking by exhausting specific hardware resource classes.

1. The failure of fast hashes for credential storage

Cryptographic hash functions such as SHA-256 and SHA-512 are engineered for throughput. Modern GPUs feature thousands of small execution cores capable of parallelizing billions of independent SHA computations per second. Dedicated password hashers counter this hardware advantage by requiring tunable iterations and substantial operational memory.

2. BCrypt mechanics and CPU-bound stretching

BCrypt derives from the Blowfish cipher and uses an expensive key setup phase to enforce computational cost.

Operates primarily as a CPU-bound key derivation function.
Tuned via a single logarithmic work factor parameter (cost 10 to 14).
Maintains a fixed, minimal memory footprint under 4 KB.

3. Argon2 and memory-hard computational resistance

Argon2, winner of the Password Hashing Competition, introduces memory hardness in addition to iteration counts.

The Argon2id variant is recommended for general password storage because it resists both side-channel cache-timing attacks and GPU-assisted cracking.
Tunable across three distinct axes: memory size (m), iteration count (t), and parallelism threads (p).
Allocating tens or hundreds of megabytes per computation prevents GPUs from running thousands of concurrent hashing threads, as GPU video memory quickly exhausts.

4. Memory footprint divergence between BCrypt and Argon2

The primary architectural difference lies in memory consumption:

BCrypt enforces high CPU overhead while keeping memory usage under 4 KB.
Argon2 enforces both CPU overhead and substantial memory allocation (often tens or hundreds of megabytes per hash).

Because GPUs possess thousands of lightweight processing cores with limited memory per core, memory-hard algorithms like Argon2 scale poorly for attackers. An attacker cannot easily run thousands of concurrent Argon2 computations if each thread requires 64 MB of RAM.

5. Production parameter tuning

Benchmark hashing performance on hardware that matches the production environment. A standard target latency for interactive user logins ranges between 400 ms and 800 ms per operation, adjusted according to peak authentication throughput and SLA constraints. Setting parameters too low weakens protection, while setting them too high exposes authentication endpoints to denial-of-service vulnerabilities under sudden traffic spikes.

6. Transparent rehash-on-login migration

Upgrading an existing authentication system to a stronger algorithm does not require resetting user passwords. When a user logs in successfully using the legacy hash format (such as BCrypt or PBKDF2), the server verifies the plaintext password against the old hash, immediately rehashes the password using the new algorithm (such as Argon2id), and updates the database record.

Production implementation

The following implementation provides decoupled BCrypt and Argon2 hashing implementations with an abstract authentication contract in .NET.

using System;
using BCrypt.Net;
using Isopoh.Cryptography.Argon2;

public interface IPasswordHasher
{
    string Hash(string password);
    bool Verify(string password, string hash);
}

public sealed class BcryptPasswordHasher : IPasswordHasher
{
    private const int WorkFactor = 13;

    public string Hash(string password)
    {
        if (string.IsNullOrWhiteSpace(password))
        {
            throw new ArgumentException("Password cannot be empty", nameof(password));
        }
        return BCrypt.Net.BCrypt.HashPassword(password, WorkFactor);
    }

    public bool Verify(string password, string hash)
    {
        if (string.IsNullOrWhiteSpace(password) || string.IsNullOrWhiteSpace(hash))
        {
            return false;
        }
        return BCrypt.Net.BCrypt.Verify(password, hash);
    }
}

public sealed class Argon2PasswordHasher : IPasswordHasher
{
    public string Hash(string password)
    {
        if (string.IsNullOrWhiteSpace(password))
        {
            throw new ArgumentException("Password cannot be empty", nameof(password));
        }
        return Argon2.Hash(password);
    }

    public bool Verify(string password, string hash)
    {
        if (string.IsNullOrWhiteSpace(password) || string.IsNullOrWhiteSpace(hash))
        {
            return false;
        }
        return Argon2.Verify(hash, password);
    }
}

public sealed class PasswordService
{
    private readonly IPasswordHasher _hasher;

    public PasswordService(IPasswordHasher hasher)
    {
        _hasher = hasher ?? throw new ArgumentNullException(nameof(hasher));
    }

    public string CreateHash(string password)
    {
        return _hasher.Hash(password);
    }

    public bool Validate(string password, string storedHash)
    {
        return _hasher.Verify(password, storedHash);
    }
}

Architectural trade-offs and edge cases

Choosing between BCrypt and Argon2 involves balancing memory allocations against resistance to specialized cracking hardware.

Latency versus consistency: Higher iteration counts and memory parameters exponentially increase attacker resistance, but directly increase CPU thread holding times and HTTP request latencies. Systems handling thousands of concurrent logins per second must budget thread pool utilization to avoid thread starvation.
Failure recovery: Authentication endpoints must handle thread pool saturation. If memory pressure spikes due to concurrent Argon2 evaluations, authentication services must reject requests or apply backpressure before triggering process out-of-memory aborts.
Scale limitations: Argon2 requires substantial dedicated memory (e.g., 64 MB per hash). Under high concurrent login volume (e.g., thousands of simultaneous requests), total memory consumption can exceed available RAM on lightweight container instances, making BCrypt with an elevated work factor more suitable in memory-constrained environments.

Common anti-patterns and gotchas

Direct use of general-purpose digests: Storing credentials using SHA-256, SHA-512, or MD5 without key stretching allows GPU arrays to crack passwords via precomputed tables and parallel dictionaries.
Stale default work factors: Retaining library default work factors configured for decade-old hardware reduces computational barriers against modern hardware.
Forced credential invalidation during migrations: Forcing all users to reset passwords during algorithm upgrades degrades user experience. Instead, employ transparent rehash-on-login verification pipelines.
Unconstrained Argon2 memory settings: Configuring memory allocations without evaluating peak login concurrency can exhaust container memory and trigger operating system out-of-memory kills.
Isolated password hashing without perimeter defenses: Relying entirely on password hashing while neglecting complementary protections such as rate limiting, account lockouts, and multi-factor authentication leaves endpoints vulnerable to credential stuffing.

Implementation checklist

Measure execution times for both BCrypt and Argon2 on staging hardware that mirrors production specs.
Balance login latency against expected throughput to select appropriate cost factors (target 400 ms to 800 ms).
Validate non-empty password inputs prior to executing key stretching functions.
Abstract hashing logic behind an IPasswordHasher interface to decouple algorithms from domain handlers.
Build a rehash-on-login workflow to upgrade legacy hashes gradually without user password resets.
Protect authentication endpoints with rate limiting, account lockouts, and multi-factor authentication to defend against credential stuffing.

Implementing Event-Driven Order Processing with Kafka using KafkaFlow in .NET

ZèD — Fri, 30 Jan 2026 20:34:10 +0000

Implementing event-driven order processing with Kafka using KafkaFlow in .NET

In distributed systems, producer and consumer workloads scale along completely different resource profiles. Web APIs require sub-millisecond dispatch times with minimal memory footprints, while background event consumers execute long-running I/O, database writes, and third-party integrations. Combining producer and consumer lifecycles within the same application process creates resource contention and prevents independent horizontal scaling.

Structuring KafkaFlow with decoupled producer and consumer registrations backed by a shared configuration contract guarantees isolated resource allocation, high API responsiveness, and scalable background message processing.

The problem and production context

Event-driven architectures often begin by registering Kafka producers and consumers in the same application host. As transaction volume scales, this monolithic configuration leads to operational failure.

Failure scenario: A surge in client order submissions causes consumer threads within an API process to execute heavy database queries and external payment gateway calls. Thread pool starvation ensues, leading to API request timeouts, high consumer lag, and inability to auto-scale API web pods without redundantly spinning up consumer partitions.
Why default approaches fall short: Monolithic registrations bind consumer group rebalancing, thread distribution, and broker connection pools to web application lifecycles. Scaling the web layer to absorb HTTP traffic multiplies consumer instances beyond the partition count of the topic, causing idle consumer threads and rebalance storms.
Production impact: API latency spikes under consumer load, background processing backs up across partitions, and deployment rollouts take longer due to coordinated consumer rebalances.

Mental model and core concepts

Separating event publishing from message processing relies on modular dependency injection, typed event routing, and consumer worker pools.

1. Shared configuration model

Centralizing Kafka cluster endpoints, topic names, consumer groups, and client identifiers in a strongly typed configuration model ensures configuration parity between publishing APIs and processing workers:

public sealed class KafkaFlowConfiguration
{
    public string ServerUrl { get; set; } = string.Empty;
    public OrderConfig Orders { get; set; } = new();
}

public sealed class OrderConfig
{
    public string TopicName { get; set; } = string.Empty;
    public string ConsumerGroup { get; set; } = string.Empty;
    public string ConsumerName { get; set; } = string.Empty;
}

2. Producer-only registration pattern

API hosts register only Kafka producer instances and topic definitions, avoiding the overhead of background worker pools, partition listeners, and message deserializers:

public static class KafkaFlowProducerExtensions
{
    public static IServiceCollection AddKafkaFlowProducer(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        var config = configuration
            .GetSection("KafkaConfiguration")
            .Get<KafkaFlowConfiguration>()
            ?? throw new InvalidOperationException("KafkaConfiguration is missing.");

        services.AddKafka(kafka => kafka
            .AddCluster(cluster => cluster
                .WithBrokers(config.ServerUrl.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
                .CreateTopicIfNotExists(config.Orders.TopicName, 10, 1)
                .AddProducer("order-producer", producer =>
                    producer.DefaultTopic(config.Orders.TopicName))));

        return services;
    }
}

3. Consumer worker balancing and typed handlers

Worker services register consumers configured with explicit worker pools, lag balancers, and typed message handlers. Using FreeWorkerDistributionStrategy alongside WithConsumerLagWorkerBalancer allows the worker process to dynamically scale concurrent message processing threads between defined minimum and maximum thresholds based on consumer lag.

4. Shared event contracts

Business events represent immutable facts within the domain and are defined as pure record types shared between producers and consumers:

public record OrderCreated(Guid OrderId, Guid CustomerId, decimal TotalAmount, DateTime CreatedAt);
public record OrderPaid(Guid OrderId, string PaymentId, DateTime PaidAt);
public record OrderShipped(Guid OrderId, string TrackingNumber, DateTime ShippedAt);

5. Strongly typed event publisher

Encapsulating KafkaFlow's IMessageProducer inside a domain publisher service enforces partition key usage to guarantee message ordering across topic partitions without reflection overhead.

6. Architecture configuration topology

The producer service (Web API) and consumer service (Background Worker) consume the same configuration structure while remaining isolated processes:

{
  "KafkaConfiguration": {
    "ServerUrl": "kafka-1:9092,kafka-2:9092",
    "Orders": {
      "TopicName": "orders",
      "ConsumerGroup": "order-processing",
      "ConsumerName": "order-events-handler"
    }
  }
}

Production implementation

Implement the configuration extensions, typed domain handlers, publisher service, and application startup configuration.

using KafkaFlow;
using KafkaFlow.Consumers.DistributionStrategies;
using KafkaFlow.Producers;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

public sealed class KafkaFlowConfiguration
{
    public string ServerUrl { get; set; } = string.Empty;
    public OrderConfig Orders { get; set; } = new();
}

public sealed class OrderConfig
{
    public string TopicName { get; set; } = string.Empty;
    public string ConsumerGroup { get; set; } = string.Empty;
    public string ConsumerName { get; set; } = string.Empty;
}

public record OrderCreated(Guid OrderId, Guid CustomerId, decimal TotalAmount, DateTime CreatedAt);
public record OrderPaid(Guid OrderId, string PaymentId, DateTime PaidAt);
public record OrderShipped(Guid OrderId, string TrackingNumber, DateTime ShippedAt);

public interface IInventoryService
{
    Task ReserveStockAsync(Guid orderId);
}

public interface IShippingService
{
    Task InitiateAsync(Guid orderId);
}

public interface INotificationService
{
    Task SendOrderUpdateAsync(Guid orderId, string status);
}

public sealed class OrderCreatedHandler : IMessageHandler<OrderCreated>
{
    private readonly IInventoryService _inventory;

    public OrderCreatedHandler(IInventoryService inventory)
    {
        _inventory = inventory;
    }

    public Task Handle(IMessageContext context, OrderCreated message)
    {
        return _inventory.ReserveStockAsync(message.OrderId);
    }
}

public sealed class OrderPaidHandler : IMessageHandler<OrderPaid>
{
    private readonly IShippingService _shipping;

    public OrderPaidHandler(IShippingService shipping)
    {
        _shipping = shipping;
    }

    public Task Handle(IMessageContext context, OrderPaid message)
    {
        return _shipping.InitiateAsync(message.OrderId);
    }
}

public sealed class OrderShippedHandler : IMessageHandler<OrderShipped>
{
    private readonly INotificationService _notification;

    public OrderShippedHandler(INotificationService notification)
    {
        _notification = notification;
    }

    public Task Handle(IMessageContext context, OrderShipped message)
    {
        return _notification.SendOrderUpdateAsync(message.OrderId, "shipped");
    }
}

public sealed class OrderEventPublisher
{
    private readonly IMessageProducer _producer;

    public OrderEventPublisher(IMessageProducer producer)
    {
        _producer = producer;
    }

    public Task PublishAsync<T>(string key, T eventPayload) where T : class
    {
        return _producer.ProduceAsync(
            key,
            eventPayload);
    }
}

public static class KafkaFlowRegistrationExtensions
{
    public static IServiceCollection AddKafkaFlowProducer(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        KafkaFlowConfiguration config = configuration
            .GetSection("KafkaConfiguration")
            .Get<KafkaFlowConfiguration>()
            ?? throw new InvalidOperationException("KafkaConfiguration is missing.");

        string[] brokers = config.ServerUrl.Split(
            ',',
            StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);

        services.AddKafka(kafka => kafka
            .AddCluster(cluster => cluster
                .WithBrokers(brokers)
                .CreateTopicIfNotExists(config.Orders.TopicName, 10, 1)
                .AddProducer("order-producer", producer =>
                    producer.DefaultTopic(config.Orders.TopicName))));

        services.AddSingleton<OrderEventPublisher>();

        return services;
    }

    public static IServiceCollection AddKafkaFlowConsumer(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        KafkaFlowConfiguration config = configuration
            .GetSection("KafkaConfiguration")
            .Get<KafkaFlowConfiguration>()
            ?? throw new InvalidOperationException("KafkaConfiguration is missing.");

        string[] brokers = config.ServerUrl.Split(
            ',',
            StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);

        services.AddKafka(kafka => kafka
            .AddCluster(cluster => cluster
                .WithBrokers(brokers)
                .CreateTopicIfNotExists(config.Orders.TopicName, 10, 1)
                .AddConsumer(consumer => consumer
                    .Topic(config.Orders.TopicName)
                    .WithGroupId(config.Orders.ConsumerGroup)
                    .WithName(config.Orders.ConsumerName)
                    .WithBufferSize(10)
                    .WithConsumerLagWorkerBalancer(maxWorkers: 100, minWorkers: 10, lagThreshold: 25)
                    .WithWorkerDistributionStrategy<FreeWorkerDistributionStrategy>()
                    .AddMiddlewares(middlewares => middlewares
                        .AddTypedHandlers(handlers => handlers
                            .AddHandler<OrderCreatedHandler>()
                            .AddHandler<OrderPaidHandler>()
                            .AddHandler<OrderShippedHandler>())))));

        return services;
    }
}

In the Web API Program.cs:

builder.Services.AddKafkaFlowProducer(builder.Configuration);

In the Worker service Program.cs:

builder.Services.AddKafkaFlowConsumer(builder.Configuration);

Architectural trade-offs and edge cases

Decoupling Kafka producers from consumers introduces operational considerations across partition assignment, ordering semantics, and delivery guarantees.

Latency versus consistency: Publishing asynchronously to Kafka minimizes HTTP API response latency. However, it introduces eventual consistency: downstream database projections and order fulfillment states update milliseconds or seconds after the initial HTTP request finishes.
Failure recovery: In the event of temporary broker disconnections, KafkaFlow producers buffer outbound messages in memory according to configured retry thresholds. For background consumers, transient handler exceptions must trigger exponential backoff retries and route poison-pill messages to a dead-letter queue (DLQ) to prevent consumer loop blockage.
Scale limitations: Maximum parallelism within a single consumer group is bounded by the topic's partition count. If an orders topic has 10 partitions, allocating more than 10 active consumer process instances leaves redundant pods idle. The internal worker balancer mitigates this by distributing messages across multiple worker threads per partition using FreeWorkerDistributionStrategy.

Common anti-patterns and gotchas

Registering consumers in API hosts: Accidentally invoking AddKafkaFlowConsumer within public-facing Web API deployments causes background message processing to compete with HTTP request threads for CPU and database connections.
Uncoordinated broker lists: Maintaining separate, out-of-sync broker connection strings across services causes intermittent communication failures during broker cluster rotations. Always share configuration structures across services.
Mismatched topic configurations: Defining differing topic names between producer configurations and consumer subscriptions causes events to be published into unmonitored topics.
Neglecting partition keys: Publishing order events with null or static partition keys distributes all messages to a single partition or breaks ordering. Always provide an explicit entity identifier (such as OrderId) as the message key to preserve event ordering per aggregate.
Non-idempotent consumer handlers: Assuming messages will arrive exactly once. Network retries or consumer group rebalances can cause duplicate delivery. All consumer handlers must implement idempotent database updates or maintain deduplication logs.

Implementation checklist

Define a shared KafkaFlowConfiguration binding cluster endpoints and order topic parameters.
Implement AddKafkaFlowProducer in a reusable extension for Web API services.
Implement AddKafkaFlowConsumer in worker services with typed handlers and worker balancing.
Define immutable event contracts using C# record types (OrderCreated, OrderPaid, OrderShipped).
Ensure message publishing uses domain entity keys (OrderId) for deterministic partition routing.
Implement idempotent handling or deduplication verification within each consumer handler.
Configure retry middleware and dead-letter queues for unprocessable messages.
Validate broker connectivity and consumer group assignment using integration tests.

Runtime Environment Variables for React Apps with Nginx and Docker

ZèD — Thu, 25 Dec 2025 12:38:37 +0000

Runtime environment variables for React apps with Nginx and Docker

React environment variables are typically inlined at build time by bundlers like Vite or Webpack. In containerized environments, this model forces a complete container rebuild whenever a configuration variable changes between deployment stages, preventing engineering teams from promoting identical container images across staging, UAT, and production.

Injecting runtime configuration into a static React application served by Nginx solves this operational limitation without requiring server-side rendering. Generating a dynamic browser script during container startup via standard shell substitution gives the application a true build-once, deploy-anywhere deployment lifecycle.

The problem and production context

Single-page applications execute purely within client browser runtimes and have no native access to the host operating system's environment variables. Bundlers circumvent this by statically replacing references like import.meta.env.VITE_API_URL or process.env.REACT_APP_API_URL with string literals during the build step.

Failure scenario: A container image built for staging is promoted to production. Because backend API URLs and telemetry keys were baked into the compiled JavaScript chunks during the staging build, the production deployment sends client requests to staging endpoints, causing data contamination and service disruption.
Why default approaches fall short: Maintaining separate container builds per environment violates continuous delivery principles, wastes CI/CD compute time, and invalidates artifact verification. Adopting full server-side rendering (SSR) merely to inject environment variables adds unnecessary server overhead, operational complexity, and maintenance costs.
Production impact: Development teams suffer from long deployment lead times, risk configuration mismatch outages during rollouts, and struggle to manage Kubernetes ConfigMap updates dynamically across environments.

Injecting runtime configuration at container startup addresses these challenges:

Enables the standard build once, deploy anywhere workflow.
Allows configuration changes (such as API endpoints or feature toggles) without rebuilding container images.
Keeps configuration externalized and environment-driven.
Integrates cleanly with Docker, Kubernetes ConfigMaps, and CI/CD pipelines.

Mental model and core concepts

1. Runtime template file definition

Create a template file at public/runtime-env.template.js containing placeholders for your configuration:

window.RUNTIME_ENV = {
  API_URL: "${API_URL}",
  FEATURE_FLAG_ANALYTICS: "${FEATURE_FLAG_ANALYTICS}",
  FEATURE_FLAG_NEW_DASHBOARD: "${FEATURE_FLAG_NEW_DASHBOARD}",
  SENTRY_DSN: "${SENTRY_DSN}",
};

2. Loading runtime configuration in HTML head

Reference the generated script inside <head> in public/index.html before the application bundles load:

<script src="/runtime-env.js"></script>

3. Application code consumption with development fallback

Create a configuration module that reads from window.RUNTIME_ENV with sensible fallbacks for local development:

const config = {
  API_URL: "https://fallback.com",
  FEATURE_FLAG_ANALYTICS: false,
  FEATURE_FLAG_NEW_DASHBOARD: false,
  SENTRY_DSN: "",
  ...(window.RUNTIME_ENV || {}),
};

export default config;

4. Multi-stage Docker build pipeline

Build the static assets in a build stage, then copy them into an Nginx image alongside an entrypoint script:

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginx:alpine
RUN apk add --no-cache gettext
COPY --from=builder /app/build /usr/share/nginx/html
COPY public/runtime-env.template.js /usr/share/nginx/html/runtime-env.template.js
COPY entrypoint.sh /docker-entrypoint.d/40-runtime-env.sh
RUN chmod +x /docker-entrypoint.d/40-runtime-env.sh

5. Startup substitution via Nginx entrypoint hooks

The official Nginx image runs scripts in /docker-entrypoint.d/ on container startup. Use envsubst to replace variables and generate the final runtime-env.js:

#!/bin/sh
set -eu

export API_URL=${API_URL:-}
export FEATURE_FLAG_ANALYTICS=${FEATURE_FLAG_ANALYTICS:-}
export FEATURE_FLAG_NEW_DASHBOARD=${FEATURE_FLAG_NEW_DASHBOARD:-}
export SENTRY_DSN=${SENTRY_DSN:-}

envsubst '${API_URL} ${FEATURE_FLAG_ANALYTICS} ${FEATURE_FLAG_NEW_DASHBOARD} ${SENTRY_DSN}' \
  < /usr/share/nginx/html/runtime-env.template.js \
  > /usr/share/nginx/html/runtime-env.js

rm -f /usr/share/nginx/html/runtime-env.template.js

6. Variable whitelisting and security boundaries

Only whitelist specific variables within the envsubst parameter list to avoid leaking sensitive host environment variables into browser-accessible files.

Production implementation

The following operational workflow executes the containerized React application with dynamic environment overrides injected at launch.

Run the container by passing environment variables directly:

docker run -p 80:80 \
  -e API_URL=https://api.prod.com \
  -e FEATURE_FLAG_ANALYTICS=true \
  -e FEATURE_FLAG_NEW_DASHBOARD=false \
  -e SENTRY_DSN=https://example@sentry.io/123 \
  your-image:latest

At container launch, the entrypoint script generates /usr/share/nginx/html/runtime-env.js. The browser loads the script before booting the bundle, making the active environment values immediately available. This allows you to deploy the exact same image across multiple environments without rebuilding.

Architectural trade-offs and edge cases

Latency versus consistency: Loading an unbundled /runtime-env.js script synchronously inside <head> introduces an additional blocking HTTP round-trip prior to bundle parsing. To minimize latency impact, ensure Nginx serves runtime-env.js with Cache-Control: no-cache, no-store, must-revalidate so updates apply immediately on pod restart without stale client caching.
Failure recovery: If the container starts with missing or invalid environment variables, the shell entrypoint should fail fast if required parameters are missing rather than generating corrupted placeholder strings.
Scale limitations: Because values in window.RUNTIME_ENV are exposed publicly to any client inspecting browser memory or network tabs, this architecture must never be used for private secrets, database connection strings, or server private keys. It is strictly limited to public client configuration.

Common anti-patterns and gotchas

Reading process.env in client browser code: Attempting to read process.env dynamically in client-side code after compilation yields undefined or runtime errors because the Node.js runtime does not exist in browsers.
Placing the runtime-env.js tag after application bundles: Loading the configuration script asynchronously or placing it after bundle <script> tags causes the React application to initialize before window.RUNTIME_ENV is populated, falling back to default values.
Running unconstrained envsubst without variable filters: Executing bare envsubst without an explicit parameter list replaces any dollar signs present in the template file, corrupting JavaScript syntax. Always pass explicit variable names to envsubst.
Overriding the default Nginx entrypoint: Replacing the official Nginx entrypoint binary with a custom entrypoint script often breaks standard configuration parsing and signal handling. Use the /docker-entrypoint.d/ lifecycle directory instead.
Omitting local development fallbacks: Failing to provide default values when window.RUNTIME_ENV is absent breaks local development workflows running outside of Docker containers.

Implementation checklist

Create public/runtime-env.template.js containing public configuration keys.
Add <script src="/runtime-env.js"></script> to public/index.html inside <head> before bundle scripts.
Implement src/config.js reading from window.RUNTIME_ENV with fallback defaults.
Create entrypoint.sh executing envsubst with explicit variable arguments.
Configure multi-stage Dockerfile mounting the entrypoint into /docker-entrypoint.d/40-runtime-env.sh.
Add schema validation for window.RUNTIME_ENV values during app initialization.
Tag client-side error telemetry with environment identifiers populated at runtime.
Map environment variables to Kubernetes ConfigMaps and Secrets in deployment manifests.
Add entrypoint assertions to halt container boot if required environment variables are missing.

Named Configuration Binding in .NET Using the Modern Options Pattern

ZèD — Sat, 29 Nov 2025 18:51:53 +0000

Named configuration binding in .NET using the modern options pattern

When multi-tenant or multi-client applications rely on fixed tenant identities defined in code, managing heterogeneous endpoints, credentials, and regional settings through raw dictionaries or string lookups introduces configuration drift and runtime failures.

Named options provide a structured approach to binding per-tenant settings. This pattern separates shared global infrastructure settings from tenant-specific configurations while using native dependency injection, compile-time enum keys, and startup validation.

The problem and production context

Applications servicing multiple known clients or tenants frequently maintain separate credentials, endpoints, and regional parameters. Developers often implement configuration access by querying IConfiguration directly across services using raw string paths or untyped dictionary representations.

Failure scenario: A developer adds a new tenant key in appsettings.json with a typographical error in the connection string key name. The application starts without warnings. Later, an inbound user request targets the client, triggering a null reference exception deep within the persistence layer during production traffic.
Why default approaches fall short: Direct IConfiguration injection provides no compile-time verification, no automatic schema validation, and no structured lifecycle integration. Standard IOptions<T> registrations only map a single instance per type, forcing teams to create separate wrapper classes for each tenant or resort to unvalidated nested dictionary mappings.
Production impact: Missing or invalid tenant configurations escape into runtime execution, causing partial outages, untyped runtime errors, cross-tenant configuration contamination, and tedious debugging across distributed microservices.

Using strongly typed named options resolves these problems by:

Keeping per-tenant configuration strongly typed, validated, and predictable.
Using built-in Microsoft dependency injection and options abstractions.
Eliminating manual string-based configuration parsing across services.
Maintaining clarity as new clients are added to the fixed tenant registry.

Mental model and core concepts

1. Hierarchical configuration structure

Organize settings into distinct top-level sections for global configuration and tenant-specific entries in appsettings.json.

{
  "GlobalSettings": {
    "FeatureXEnabled": true,
    "ApiEndpoint": "https://example.com"
  },
  "Clients": {
    "ClientA": {
      "ConnectionString": "Server=A",
      "Region": "EU"
    },
    "ClientB": {
      "ConnectionString": "Server=B",
      "Region": "US"
    }
  }
}

2. Fixed tenant enumeration

Define tenant identifiers as enum members to ensure compile-time safety and consistent naming across configuration files.

public enum Client
{
    ClientA,
    ClientB
}

3. Strongly typed options models

Create dedicated, immutable options classes for global settings and per-tenant settings.

public sealed class GlobalSettings
{
    public bool FeatureXEnabled { get; init; }
    public string ApiEndpoint { get; init; } = string.Empty;
}

public sealed class ClientSettings
{
    public string ConnectionString { get; init; } = string.Empty;
    public string Region { get; init; } = string.Empty;
}

4. Global options registration and validation

Bind and validate the global options section during application startup using startup validation.

builder.Services
    .AddOptions<GlobalSettings>()
    .Bind(builder.Configuration.GetSection("GlobalSettings"))
    .Validate(settings => !string.IsNullOrWhiteSpace(settings.ApiEndpoint), "ApiEndpoint is required")
    .ValidateOnStart();

5. Named options per tenant iteration

Iterate through the tenant enum members to bind and validate individual named option instances for each tenant section.

foreach (Client client in Enum.GetValues<Client>())
{
    var clientName = client.ToString();

    builder.Services
        .AddOptions<ClientSettings>(clientName)
        .Bind(builder.Configuration.GetSection($"Clients:{clientName}"))
        .Validate(settings => !string.IsNullOrWhiteSpace(settings.ConnectionString), $"{clientName} ConnectionString is required")
        .Validate(settings => !string.IsNullOrWhiteSpace(settings.Region), $"{clientName} Region is required")
        .ValidateOnStart();
}

6. Tenant options resolution

Inject IOptionsSnapshot<T> and retrieve tenant-specific configurations via Get(name) during request execution.

public sealed class ClientConsumer
{
    private readonly IOptionsSnapshot<ClientSettings> _clientSettings;

    public ClientConsumer(IOptionsSnapshot<ClientSettings> clientSettings)
    {
        _clientSettings = clientSettings;
    }

    public ClientSettings Get(Client client)
    {
        return _clientSettings.Get(client.ToString());
    }
}

Production implementation

The following host initialization binds global options, registers named options for all configured tenants, and verifies validity at host startup.

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddOptions<GlobalSettings>()
    .Bind(builder.Configuration.GetSection("GlobalSettings"))
    .ValidateOnStart();

foreach (Client client in Enum.GetValues<Client>())
{
    var name = client.ToString();
    builder.Services
        .AddOptions<ClientSettings>(name)
        .Bind(builder.Configuration.GetSection($"Clients:{name}"))
        .ValidateOnStart();
}

var app = builder.Build();
app.Run();

This pattern keeps tenant configuration deterministic, explicit, and integrated with the host lifecycle.

Architectural trade-offs and edge cases

Latency versus consistency: IOptionsSnapshot<T> recomputes option instances once per request scope, enabling immediate reloading when configuration providers reload without restarting the process. This introduces a slight per-request object allocation overhead compared to singleton IOptions<T>.
Failure recovery: Using ValidateOnStart() causes configuration errors to fail fast during the host startup phase. If a tenant entry is missing or fails validation predicates, the container aborts before opening HTTP listener sockets, preventing misconfigured pods from accepting traffic.
Scale limitations: The enum-driven named options pattern suits fixed, known tenant registries (such as dedicated partner integrations or regional deployments). It is not suited for multi-tenant architectures requiring dynamic, unbounded tenant registration at runtime without application redeployment.

Common anti-patterns and gotchas

Using arbitrary string literals for tenant names: Hardcoding string values instead of strongly typed enum identifiers introduces typo-induced configuration lookup mismatches. Always drive registration and retrieval via compile-time constants or enums.
Skipping startup validation: Omitting ValidateOnStart() allows misconfigured tenant sections to start cleanly, deferring missing configuration errors until production requests hit the specific tenant code path.
Merging global settings and client-specific settings: Combining global infrastructure flags and tenant-specific connection details into a single monolithic options model breaks single-responsibility principles and complicates tenant isolation.
Injecting IConfiguration directly into business logic: Reading raw keys directly from IConfiguration bypasses typed validation, hides dependencies, and tightly couples domain services to configuration schemas.
Omitting the named options parameter on resolution: Calling _clientSettings.Value instead of _clientSettings.Get(name) returns the default un-named instance, which will contain uninitialized default values.

Implementation checklist

Separate configuration sections into GlobalSettings and client-specific Clients objects.
Define a fixed enum containing all supported client or tenant identifiers.
Create immutable, strongly typed POCO classes for global and per-tenant settings.
Register global options with .Bind() and .ValidateOnStart().
Iterate enum members to register named client options bound to their respective configuration sections.
Inject IOptionsSnapshot<ClientSettings> into client consumers and access options using .Get(client.ToString()).
Integrate external vault providers to load sensitive per-tenant connection strings.
Implement health checks verifying connectivity for all configured tenants at startup.
Write unit and integration tests confirming proper option binding for every registered enum value.
Establish a migration strategy if tenant definitions need to become dynamic in the future.

.NET 9 dropped Swashbuckle — but you can still keep Swagger UI backed by OpenAPI

ZèD — Fri, 24 Oct 2025 03:23:25 +0000

.NET 9 dropped Swashbuckle generator, but you can still keep Swagger UI backed by OpenAPI

In .NET 9, ASP.NET Core removed the default dependency on Swashbuckle.AspNetCore in favor of built-in, native OpenAPI document generation. Upgrading applications without adapting schema generation leads to broken documentation endpoints, missing security definitions, and deprecated package conflicts.

Developers can retain the familiar Swagger UI frontend while delegating specification generation entirely to the native Microsoft.AspNetCore.OpenApi engine. Combining native OpenAPI registration with custom document and operation transformers produces interactive API documentation supporting JWT Bearer authentication and RFC 7807 Problem Details schemas without obsolete third-party generators.

The problem and production context

Prior to .NET 9, ASP.NET Core templates relied on Swashbuckle for OpenAPI document generation and UI hosting. As Swashbuckle experienced slower maintenance cycles and architectural drift from official OpenAPI specifications, Microsoft built first-party OpenAPI generation directly into the framework.

Failure scenario: Upgrading an existing ASP.NET Core Web API to .NET 9 while retaining legacy AddSwaggerGen registrations creates package dependency friction, deprecated API warnings, and schema generation discrepancies with modern System.Text.Json polymorphs. Removing Swashbuckle entirely removes interactive UI documentation, leaving consumers without an in-browser request sandbox.
Why default approaches fall short: Removing Swashbuckle generator packages (Swashbuckle.AspNetCore.SwaggerGen) leaves teams without an interactive UI if they do not decouple UI rendering from schema generation. Conversely, relying solely on app.MapOpenApi() serves raw JSON without an exploratory browser console.
Production impact: API consumers lose interactive sandbox exploration, developers struggle to verify authorization headers against secured endpoints, and client code generators receive non-standardized error schemas across divergent controllers.

Adopting native OpenAPI document generation paired with standalone Swagger UI addresses these challenges:

Uses native OpenAPI document generation in ASP.NET Core.
Retains the familiar Swagger UI developer experience.
Avoids third-party generator plumbing and obsolete dependencies.
Handles JWT authentication and problem details response contracts cleanly.

Mental model and core concepts

1. Built-in OpenAPI document registration

public static IServiceCollection AddOpenApiDocumentation(this IServiceCollection services)
{
    services.AddOpenApi("v1", options =>
    {
        options.AddDocumentTransformer<JwtBearerSecurityDocumentTransformer>();
        options.AddOperationTransformer<ProblemDetailsOperationTransformer>();
    });

    return services;
}

2. OpenAPI and Swagger UI pipeline

Map the OpenAPI JSON endpoint and configure Swagger UI to load /openapi/v1.json:

public static void UseOpenApiDocumentation(this WebApplication app)
{
    app.MapOpenApi();

    app.UseSwaggerUI(options =>
    {
        options.SwaggerEndpoint("/openapi/v1.json", "v1");
        options.EnablePersistAuthorization();
        options.DisplayRequestDuration();
        options.EnableTryItOutByDefault();
        options.EnableFilter();
        options.DocExpansion(DocExpansion.List);
        options.DefaultModelsExpandDepth(0);
    });
}

3. JWT security scheme transformer

Add the JWT Bearer scheme and a default security requirement to the OpenAPI document:

public sealed class JwtBearerSecurityDocumentTransformer : IOpenApiDocumentTransformer
{
    private const string SecuritySchemeId = JwtBearerDefaults.AuthenticationScheme;

    public Task TransformAsync(OpenApiDocument document, OpenApiDocumentTransformerContext context, CancellationToken cancellationToken)
    {
        document.Info = new OpenApiInfo
        {
            Title = "DevInsightForge.API",
            Version = "v1",
            Description = "The DevInsightForge API built with ASP.NET Core, it ensures secure and efficient communication through JSON Web Tokens (JWT) for authentication."
        };

        document.Components ??= new OpenApiComponents();
        document.Components.SecuritySchemes ??= new Dictionary<string, IOpenApiSecurityScheme>();
        document.Components.SecuritySchemes[SecuritySchemeId] = new OpenApiSecurityScheme
        {
            BearerFormat = "JWT",
            Name = "JWT Authentication",
            In = ParameterLocation.Header,
            Type = SecuritySchemeType.Http,
            Scheme = JwtBearerDefaults.AuthenticationScheme,
            Description = "Put **_ONLY_** your JWT Bearer token on the textbox below!"
        };

        document.Security ??= [];
        document.Security.Add(new OpenApiSecurityRequirement
        {
            [new OpenApiSecuritySchemeReference(SecuritySchemeId, document, null)] = []
        });

        return Task.CompletedTask;
    }
}

4. Operation transformer for problem details

Inject a reusable schema for 4xx and 5xx problem details responses across endpoints:

public sealed class ProblemDetailsOperationTransformer : IOpenApiOperationTransformer
{
    public async Task TransformAsync(OpenApiOperation operation, OpenApiOperationTransformerContext context, CancellationToken cancellationToken)
    {
        operation.Responses ??= [];

        var errorResponseSchema = await context.GetOrCreateSchemaAsync(typeof(ErrorResponse), null, cancellationToken);

        operation.Responses["4xx/5xx"] = new OpenApiResponse
        {
            Description = typeof(ErrorResponse).Name,
            Content = new Dictionary<string, OpenApiMediaType>
            {
                ["application/problem+json"] = new OpenApiMediaType
                {
                    Schema = errorResponseSchema
                }
            }
        };

        return;
    }
}

Production implementation

The following complete extension module encapsulates service registration, pipeline middleware, document-level security schemas, and standard error response operation transformers.

using DevInsightForge.WebAPI.Contracts;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.OpenApi;
using Microsoft.OpenApi;
using Swashbuckle.AspNetCore.SwaggerUI;

namespace DevInsightForge.WebAPI.Extensions;

public static class OpenApiExtensions
{
    public static IServiceCollection AddOpenApiDocumentation(this IServiceCollection services)
    {
        services.AddOpenApi("v1", options =>
        {
            options.AddDocumentTransformer<JwtBearerSecurityDocumentTransformer>();
            options.AddOperationTransformer<ProblemDetailsOperationTransformer>();
        });

        return services;
    }

    public static void UseOpenApiDocumentation(this WebApplication app)
    {
        app.MapOpenApi();

        app.UseSwaggerUI(options =>
        {
            options.SwaggerEndpoint("/openapi/v1.json", "v1");
            options.EnablePersistAuthorization();
            options.DisplayRequestDuration();
            options.EnableTryItOutByDefault();
            options.EnableFilter();
            options.DocExpansion(DocExpansion.List);
            options.DefaultModelsExpandDepth(0);
        });
    }
}

public sealed class JwtBearerSecurityDocumentTransformer : IOpenApiDocumentTransformer
{
    private const string SecuritySchemeId = JwtBearerDefaults.AuthenticationScheme;

    public Task TransformAsync(OpenApiDocument document, OpenApiDocumentTransformerContext context, CancellationToken cancellationToken)
    {
        document.Info = new OpenApiInfo
        {
            Title = "DevInsightForge.API",
            Version = "v1",
            Description = "The DevInsightForge API built with ASP.NET Core, it ensures secure and efficient communication through JSON Web Tokens (JWT) for authentication."
        };

        document.Components ??= new OpenApiComponents();
        document.Components.SecuritySchemes ??= new Dictionary<string, IOpenApiSecurityScheme>();
        document.Components.SecuritySchemes[SecuritySchemeId] = new OpenApiSecurityScheme
        {
            BearerFormat = "JWT",
            Name = "JWT Authentication",
            In = ParameterLocation.Header,
            Type = SecuritySchemeType.Http,
            Scheme = JwtBearerDefaults.AuthenticationScheme,
            Description = "Put **_ONLY_** your JWT Bearer token on the textbox below!"
        };

        document.Security ??= [];
        document.Security.Add(new OpenApiSecurityRequirement
        {
            [new OpenApiSecuritySchemeReference(SecuritySchemeId, document, null)] = []
        });

        return Task.CompletedTask;
    }
}

public sealed class ProblemDetailsOperationTransformer : IOpenApiOperationTransformer
{
    public async Task TransformAsync(OpenApiOperation operation, OpenApiOperationTransformerContext context, CancellationToken cancellationToken)
    {
        operation.Responses ??= [];

        var errorResponseSchema = await context.GetOrCreateSchemaAsync(typeof(ErrorResponse), null, cancellationToken);

        operation.Responses["4xx/5xx"] = new OpenApiResponse
        {
            Description = typeof(ErrorResponse).Name,
            Content = new Dictionary<string, OpenApiMediaType>
            {
                ["application/problem+json"] = new OpenApiMediaType
                {
                    Schema = errorResponseSchema
                }
            }
        };

        return;
    }
}

The application entry point registers authentication, applies the OpenAPI extensions, and mounts HTTP routes:

builder.Services.AddControllers();
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = builder.Configuration["Jwt:Issuer"],
            ValidAudience = builder.Configuration["Jwt:Audience"],
            IssuerSigningKey = new SymmetricSecurityKey(
                Encoding.UTF8.GetBytes(builder.Configuration["Jwt:SecurityKey"]!))
        };
    });

builder.Services.AddOpenApiDocumentation();

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();
app.UseOpenApiDocumentation();
app.MapControllers();

app.Run();

Once the application starts, check these endpoints in the browser:

OpenAPI JSON: /openapi/v1.json
Swagger UI: /swagger

The authorize dialog in Swagger UI will accept a JWT bearer token, and responses will reflect the standardized problem details contract.

Architectural trade-offs and edge cases

Latency versus consistency: Native OpenAPI documents are generated dynamically on first request and cached by default in memory. Schema transformations execute during generation; extensive reflection or deep schema traversal in transformers adds small startup latency to the initial /openapi/v1.json load.
Failure recovery: If a transformer throws an unhandled exception during document generation, requests to /openapi/v1.json return an HTTP 500 error, disabling Swagger UI without crashing the host process. Validation during build pipelines prevents broken transformers from reaching production.
Scale limitations: Swagger UI is a client-side single-page application rendering raw OpenAPI schemas. For APIs with hundreds of endpoints and complex model graphs, rendering large JSON documents inside the browser can cause client memory spikes and UI lag. In massive architectures, split schemas across versioned documents.

Common anti-patterns and gotchas

Leaving obsolete AddSwaggerGen configurations in place: Retaining legacy Swashbuckle generator configurations alongside the native .NET 9 OpenAPI pipeline creates conflicting middleware and duplicate schema definitions. Remove AddSwaggerGen and retain only Swashbuckle.AspNetCore.SwaggerUI.
Calling UseSwaggerUI without mapping OpenApi endpoint: Invoking app.UseSwaggerUI() without calling app.MapOpenApi() causes Swagger UI to fail with an HTTP 404 error when fetching the specification.
Registering authentication middleware without declaring OpenAPI security scheme: Configuring JWT authentication in ASP.NET Core without adding a document transformer leaves Swagger UI without an Authorize button, preventing developers from testing protected endpoints.
Applying mismatched problem details schemas across controllers: Using custom error response contracts without registering a global operation transformer creates inconsistent documentation schemas between endpoints.
Pulling in incompatible versions of Swashbuckle UI: Using outdated versions of Swashbuckle.AspNetCore.SwaggerUI that depend on older Microsoft schema types can cause type resolution conflicts. Use the standalone UI package compatible with .NET 9.

Implementation checklist

Remove Swashbuckle.AspNetCore.SwaggerGen from project dependencies.
Add Microsoft.AspNetCore.OpenApi and Swashbuckle.AspNetCore.SwaggerUI package references.
Register native OpenAPI services using services.AddOpenApi("v1", options => ...).
Implement JwtBearerSecurityDocumentTransformer to define the security scheme and requirement.
Implement ProblemDetailsOperationTransformer to standardize 4xx and 5xx error responses.
Map OpenAPI endpoints using app.MapOpenApi() and configure app.UseSwaggerUI().
Verify /openapi/v1.json and /swagger endpoints in the browser.
Add separate named OpenAPI documents for versioned API endpoints.
Group operations using tags to organize larger controller surfaces.
Validate the generated OpenAPI specification during CI builds.
Restrict Swagger UI access based on hosting environments.