Your Subscriber Passed the POC. Production Has a Different Test.

Your subscriber passed the POC. Production has a different test: volume.

A message subscriber that does its work inline on the delivery callback is capped at one message per handler-time. At POC volume you never hit that ceiling. At production volume it is the only thing you hit. Same broker, same handler, same 1000 queued messages: 636 seconds to drain inline, 44 seconds off the callback.

Why the POC never shows it

The pattern is easy to write and easy to sign off on. The broker calls you with a message. You deserialize it, do the work, ack it, return. It is correct, it is simple, and at a few messages a minute it is indistinguishable from any other design. That is exactly why it survives review: the POC never disagrees with you.

Common mistake: "We load tested it and it kept up." A load test at POC volume proves the handler is fast enough for the callback to keep pace, not that the design scales. Below the ceiling, inline and hand-off look identical. The test that matters queues more messages than one handler-time per second can clear, and then measures how long the backlog takes to drain.

The math is what the POC hides. A Solace context drives delivery for every flow bound to it from one thread, and the next message is not handed to you until your callback returns. So throughput is capped at 1 divided by your average handler time. Not roughly. Exactly. A bigger box does not move it. A bigger backlog does not move it, it just makes the wait longer. Solace's own docs say not to block that thread, but a warning without a number reads as theory.

So I built the number.

The two callbacks
Both of these are the real OnMessageReceived from messaging-lab, a small ports-and-adapters layer over the native Solace .NET SDK with two subscriber implementations that share everything except this method. The first is shown without its try/catch and logger; the second is verbatim.

// SolaceSequentialSubscriber<T>: work inline on the delivery callback
void OnMessageReceived(object? sender, MessageEventArgs args)
{
    using var message = args.Message;
    var json = Encoding.UTF8.GetString(message.BinaryAttachment ?? []);
    var payload = _deserializer.Deserialize(json);
    if (_handler.Handle(payload))
    {
        _flow.Ack(message.ADMessageId);
    }
}

Deserialize, handle, ack, all on the broker's single delivery thread. The next message waits for this one.

// SolaceConcurrentSubscriber<T>: hand off and return
void OnMessageReceived(object? sender, MessageEventArgs args) =>
    _ingress.Writer.WriteAsync(args.Message).AsTask().GetAwaiter().GetResult();

The callback only writes the message to a bounded Channel and returns. The channel is sized to the flow's WindowSize, so if the workers fall behind, the write blocks the callback and the broker's own window flow control takes over. Nothing is buffered without bound.

Behind the channel, a single router task reads messages in delivery order, deserializes each one, and hashes an ordering key into one of N lanes:

var lane = _lanes![unchecked((uint)_keySelector!.GetKey(payload).GetHashCode()) % (uint)_lanes.Length];
await lane.Writer.WriteAsync((message, payload));

Each lane is a single-reader channel drained by exactly one worker, which calls the handler and acks on success. Same key, same lane, same order. Different keys, different lanes, in parallel.

The numbers
The lab publishes 1000 OrderPlaced messages spread across 100 order ids, each carrying a per-key sequence number, then binds one subscriber or the other and measures how long the backlog takes to drain. The handler blocks for a uniform random 250 to 1000 ms per message, mean 625 ms, standing in for an HTTP call or a database write. Everything else is identical between runs.

First the ceiling, on paper: 1 divided by 0.625 seconds is 1.6 messages per second. Then the ceiling, measured:

Configuration	Elapsed	Throughput	Speedup	p50 wait	p99 wait	Ordering violations
Inline on the callback	635.9 s	1.6/s	1.00x	320.8 s	632.9 s	0
Off the callback, 2 lanes	328.7 s	3.0/s	1.88x	162.7 s	323.9 s	0
Off the callback, 4 lanes	223.6 s	4.5/s	2.81x	90.5 s	218.4 s	0
Off the callback, 8 lanes	102.5 s	9.8/s	6.12x	47.2 s	102.1 s	0
Off the callback, 32 lanes	43.6 s	23.0/s	14.38x	20.6 s	41.5 s	0

The inline subscriber landed on 1.6 messages per second. The ceiling is not a metaphor, it is the measurement. Off the callback, 2 lanes halves the drain time, 8 lanes gives about 6x, 32 lanes gives about 14x, and an earlier sweep on the same setup kept scaling through 62 and 93 lanes. None of that was available to the inline design at any hardware size.

p50 and p99 here are backlog-drain waits, not steady-state request latency. Every message was already sitting in the queue before the subscriber connected, so they mostly measure how long a message waited its turn.

The ordering objection
The reason a subscriber usually stays sequential is "we need messages for the same record in order." That is a real requirement and it does not require a single thread. Route by key. Same-key messages always land in the same lane and are handled in delivery order. Different keys run in parallel. The subscriber in the lab checks this on every message with a per-key sequence number, and across all five runs above, 5000 messages, it recorded zero ordering violations.

Two things to watch when you do this
Lane count needs key cardinality behind it. A key never spans lanes, so a lane with no keys assigned does nothing. The lab's first sweep used only 16 keys, and 8 lanes reached just 3.25x. The same 8 lanes with 100 keys, in the table above, reached 6.12x. Before picking a lane count, check how many distinct keys your real stream actually has.

Blocking handlers above the ThreadPool minimum need help. The lab's handler uses Thread.Sleep, so each lane holds a real ThreadPool thread for the whole call. The pool's default minimum is Environment.ProcessorCount and it grows past that only through a throttled injection algorithm, so a lane count well above the core count ramps up over tens of seconds instead of running at full concurrency immediately. The lab's subscriber calls ThreadPool.SetMinThreads from the configured lane count before it binds the flow. An awaited async handler would not need this.

And a caution in the other direction: lane count is not a CPU question for an I/O-shaped handler. It is how many calls you are willing to have in flight against whatever is downstream. That dependency's capacity, not your core count, is what should set it.

Takeaway
If the delivery callback does the work, throughput is 1 divided by handler time, and no POC will ever show you that. Hand the message off, return, and let a keyed lane keep the order you actually need. Then put the number in front of the design before production does.

Code, both subscribers, the load generator, and the benchmark reports: github.com/sthotakura/messaging-lab. The run behind the table above is reports/benchmark-20260905-191257.html.

Verified against a local Solace PubSub+ Standard Docker broker via SolaceSystems.Solclient.Messaging on .NET 10. All numbers are from a single trial per configuration on one machine (32 logical CPUs); the same 32-lane configuration measured 19.8/s in an earlier sweep and 23.0/s in this one, so expect roughly 15% run-to-run variance. Speedup figures are the report's own ratios of rounded throughputs; the elapsed-time ratio for 32 lanes is 14.6x. The "one context thread, do not block the callback" claim is from Solace's current API developer guide, linked below.