DEV Community: Suresh Thotakura

Your Subscriber Passed the POC. Production Has a Different Test.

Suresh Thotakura — Sat, 05 Sep 2026 18:32:29 +0000

Your subscriber passed the POC. Production has a different test: volume.

A message subscriber that does its work inline on the delivery callback is capped at one message per handler-time. At POC volume you never hit that ceiling. At production volume it is the only thing you hit. Same broker, same handler, same 1000 queued messages: 636 seconds to drain inline, 44 seconds off the callback.

Why the POC never shows it

The pattern is easy to write and easy to sign off on. The broker calls you with a message. You deserialize it, do the work, ack it, return. It is correct, it is simple, and at a few messages a minute it is indistinguishable from any other design. That is exactly why it survives review: the POC never disagrees with you.

Common mistake: "We load tested it and it kept up." A load test at POC volume proves the handler is fast enough for the callback to keep pace, not that the design scales. Below the ceiling, inline and hand-off look identical. The test that matters queues more messages than one handler-time per second can clear, and then measures how long the backlog takes to drain.

The math is what the POC hides. A Solace context drives delivery for every flow bound to it from one thread, and the next message is not handed to you until your callback returns. So throughput is capped at 1 divided by your average handler time. Not roughly. Exactly. A bigger box does not move it. A bigger backlog does not move it, it just makes the wait longer. Solace's own docs say not to block that thread, but a warning without a number reads as theory.

So I built the number.

The two callbacks
Both of these are the real OnMessageReceived from messaging-lab, a small ports-and-adapters layer over the native Solace .NET SDK with two subscriber implementations that share everything except this method. The first is shown without its try/catch and logger; the second is verbatim.

// SolaceSequentialSubscriber<T>: work inline on the delivery callback
void OnMessageReceived(object? sender, MessageEventArgs args)
{
    using var message = args.Message;
    var json = Encoding.UTF8.GetString(message.BinaryAttachment ?? []);
    var payload = _deserializer.Deserialize(json);
    if (_handler.Handle(payload))
    {
        _flow.Ack(message.ADMessageId);
    }
}

Deserialize, handle, ack, all on the broker's single delivery thread. The next message waits for this one.

// SolaceConcurrentSubscriber<T>: hand off and return
void OnMessageReceived(object? sender, MessageEventArgs args) =>
    _ingress.Writer.WriteAsync(args.Message).AsTask().GetAwaiter().GetResult();

The callback only writes the message to a bounded Channel and returns. The channel is sized to the flow's WindowSize, so if the workers fall behind, the write blocks the callback and the broker's own window flow control takes over. Nothing is buffered without bound.

Behind the channel, a single router task reads messages in delivery order, deserializes each one, and hashes an ordering key into one of N lanes:

var lane = _lanes![unchecked((uint)_keySelector!.GetKey(payload).GetHashCode()) % (uint)_lanes.Length]; await lane.Writer.WriteAsync((message, payload));

Each lane is a single-reader channel drained by exactly one worker, which calls the handler and acks on success. Same key, same lane, same order. Different keys, different lanes, in parallel.

The numbers
The lab publishes 1000 OrderPlaced messages spread across 100 order ids, each carrying a per-key sequence number, then binds one subscriber or the other and measures how long the backlog takes to drain. The handler blocks for a uniform random 250 to 1000 ms per message, mean 625 ms, standing in for an HTTP call or a database write. Everything else is identical between runs.

First the ceiling, on paper: 1 divided by 0.625 seconds is 1.6 messages per second. Then the ceiling, measured:

Configuration	Elapsed	Throughput	Speedup	p50 wait	p99 wait
Inline on the callback	635.9 s	1.6/s	1.00x	320.8 s	632.9 s
Off the callback, 2 lanes	328.7 s	3.0/s	1.88x	162.7 s	323.9 s
Off the callback, 4 lanes	223.6 s	4.5/s	2.81x	90.5 s	218.4 s
Off the callback, 8 lanes	102.5 s	9.8/s	6.12x	47.2 s	102.1 s
Off the callback, 32 lanes	43.6 s	23.0/s	14.38x	20.6 s	41.5 s

The inline subscriber landed on 1.6 messages per second. The ceiling is not a metaphor, it is the measurement. Off the callback, 2 lanes halves the drain time, 8 lanes gives about 6x, 32 lanes gives about 14x, and an earlier sweep on the same setup kept scaling through 62 and 93 lanes. None of that was available to the inline design at any hardware size.

p50 and p99 here are backlog-drain waits, not steady-state request latency. Every message was already sitting in the queue before the subscriber connected, so they mostly measure how long a message waited its turn.

The ordering objection
The reason a subscriber usually stays sequential is "we need messages for the same record in order." That is a real requirement and it does not require a single thread. Route by key. Same-key messages always land in the same lane and are handled in delivery order. Different keys run in parallel. The subscriber in the lab checks this on every message with a per-key sequence number, and across all five runs above, 5000 messages, it recorded zero ordering violations.

Two things to watch when you do this
Lane count needs key cardinality behind it. A key never spans lanes, so a lane with no keys assigned does nothing. The lab's first sweep used only 16 keys, and 8 lanes reached just 3.25x. The same 8 lanes with 100 keys, in the table above, reached 6.12x. Before picking a lane count, check how many distinct keys your real stream actually has.

Blocking handlers above the ThreadPool minimum need help. The lab's handler uses Thread.Sleep, so each lane holds a real ThreadPool thread for the whole call. The pool's default minimum is Environment.ProcessorCount and it grows past that only through a throttled injection algorithm, so a lane count well above the core count ramps up over tens of seconds instead of running at full concurrency immediately. The lab's subscriber calls ThreadPool.SetMinThreads from the configured lane count before it binds the flow. An awaited async handler would not need this.

And a caution in the other direction: lane count is not a CPU question for an I/O-shaped handler. It is how many calls you are willing to have in flight against whatever is downstream. That dependency's capacity, not your core count, is what should set it.

Takeaway
If the delivery callback does the work, throughput is 1 divided by handler time, and no POC will ever show you that. Hand the message off, return, and let a keyed lane keep the order you actually need. Then put the number in front of the design before production does.

Code, both subscribers, the load generator, and the benchmark reports: github.com/sthotakura/messaging-lab. The run behind the table above is reports/benchmark-20260905-191257.html.

Verified against a local Solace PubSub+ Standard Docker broker via SolaceSystems.Solclient.Messaging on .NET 10. All numbers are from a single trial per configuration on one machine (32 logical CPUs); the same 32-lane configuration measured 19.8/s in an earlier sweep and 23.0/s in this one, so expect roughly 15% run-to-run variance. Speedup figures are the report's own ratios of rounded throughputs; the elapsed-time ratio for 32 lanes is 14.6x. The "one context thread, do not block the callback" claim is from Solace's current API developer guide, linked below.

A Rate Limiter That Knows Who's Calling

Suresh Thotakura — Sat, 29 Aug 2026 19:47:21 +0000

ASP.NET Core's rate limiter partitions however you key it. Key it on the caller, and one client's burst stops being every other client's problem.

I built a small lab to get that wiring right end to end.

Four stages:

Authenticate. A custom header scheme validates the client and issues a client_id claim. Nothing downstream can partition per client until this exists.
Partition. The policy reads that claim and keys the partition by client id plus HTTP verb. A client's read budget and write budget are separate buckets, and so are two different clients'.
Limit. Fixed window: 10 requests per 10 seconds, queue limit 0. No waiting in line, you either get a permit or you don't.
Enforce. Request 11 in the window gets HTTP 429.

The client's profile decides which partition it lands in:

Standard: limited on every verb.
Premium: reads skip the limiter, writes still get 10 per 10 seconds.
Unlimited: no limiter at all.

The part worth remembering is the ordering.
UseRateLimiter() has to run after UseAuthentication() and UseAuthorization(). Register it earlier and the policy executes before the client_id claim exists, so there is nothing to partition on. In this implementation that throws rather than quietly limiting on nothing, which is the failure mode you want: loud, not silent.

Three calls do the wiring: AddRateLimiter() to register the policy, UseRateLimiter() to put it in the pipeline, RequireRateLimiting() to attach it to an endpoint group.

Code, including the in-memory client fixtures and the Todo endpoints it protects: https://github.com/sthotakura/rate-limiter-lab

Sealed Isn't a Restriction, It's a Promise

Suresh Thotakura — Fri, 28 Aug 2026 18:53:06 +0000

Leaving a class open to inheritance is a design decision, not a default you can ignore.

The core idea

An unsealed class is a promise: every virtual member can be overridden without breaking what the class guarantees. Most classes never meant to make that promise. They're just unsealed by default, because that's what class gives you unless you say otherwise.

Common mistake: treating sealed as "I don't want to think about subclassing" rather than "this type's invariants would break if someone could."

One override breaks the promise

Here's the promise, a BankAccount that refuses to go negative:

public class BankAccount
{
    public decimal Balance { get; protected set; }

    public virtual void Withdraw(decimal amount)
    {
        if (amount > Balance)
            throw new InvalidOperationException();
        Balance -= amount;
    }
}

And here's the override that breaks it:

public class RiskyAccount : BankAccount
{
    public override void Withdraw(decimal amount)
    {
        Balance -= amount; // no check
    }
}

Nothing here is exotic. It compiles cleanly, and RiskyAccount is a
perfectly legal BankAccount as far as the type system is concerned. Open
one with a balance of 100 and withdraw 500:

BankAccount account = new RiskyAccount(100m);
account.Withdraw(500m);
Console.WriteLine($"Balance: {account.Balance:F2}");

Real dotnet run output:

Balance: -400.00

The check on the left never ran. virtual was an open invitation, and
RiskyAccount took it.

Sealing turns a silent bug into a compile error

Without sealed, the code above compiles and produces a wrong answer at
runtime; nothing points you at the problem until it's already in
production. With sealed, the same mistake becomes something the compiler
catches before the code ever runs:

public sealed class BankAccount
{
    public decimal Balance { get; protected set; }

    public void Withdraw(decimal amount)
    {
        if (amount > Balance) throw new InvalidOperationException();
        Balance -= amount;
    }
}

public class RiskyAccount : BankAccount { }

// error CS0509: 'RiskyAccount': cannot derive
// from sealed type 'BankAccount'

Sealing also makes virtual on Withdraw pointless, and the compiler
enforces that too:

public sealed class BankAccount
{
    public virtual void Withdraw(decimal amount) { /* ... */ }
}

// error CS0549: 'BankAccount.Withdraw(decimal)' is a new virtual
// member in sealed type 'BankAccount'

Both of those are real compiler errors, not paraphrased, verified against
a live build.

The takeaway

Seal by default. Unseal only when you intend to support inheritance. Every
unsealed class is an implicit promise to every future subclass that
overriding won't break its invariants. Most classes never meant to make
that promise, they're just unsealed because nobody had to actively choose
not to be.