DEV Community: lepp-sd

When a node loses its lease anyway: a takeover, second by second

lepp-sd — Wed, 04 Nov 2026 21:57:45 +0000

I am building a cloud trade copier for MetaTrader. Each follower account has one broker session on one node, and that node copies the master's trades onto it. In an earlier post I described how a node leases its accounts from Postgres: a node_id and a lease_until on the account row, FOR UPDATE SKIP LOCKED to claim, and a node that stops sending orders 15 seconds before its lease ends.

That post was about keeping ownership. This one is about losing it: what happens, second by second, when a node is cut off from the database, another node takes its accounts, and both have to end up agreeing with what is actually open at the broker.

The defaults

	default
lease	60 s
renewal cycle	every 20 s
fence (stop sending before the lease ends)	15 s
retry after a failed cycle	2 s

These are configuration values, not measurements.

The timeline

Say node A last renewed its leases at t = 0. Then its connection to the database breaks. The broker sessions and the message bus are still fine, which is the uncomfortable part: A can still trade, it just can no longer prove that it should.

t = 0. The renewal query wrote lease_until = now() + 60 s, so in the database the lease runs until at least t = 60. A recorded its own deadline as started + 60 s - 15 s, where started was read before the query.

t = 20. The next cycle fails. Nothing changes yet. A logs the error and tries again every 2 seconds. It keeps its broker sessions and keeps copying, because its lease is still confirmed.

t = 45. HoldsLease turns false. From this moment every unit of work for every follower on A is dropped at the gate, right before it would execute:

if (!_engine.HoldsLease)
{
    // Cut off from the database long enough that another node may own this slave now:
    // sending orders could duplicate its copies. The reconcile after the next lease
    // renewal (or on the new owner) catches up.
    _log.LogWarning("[slave {Id}] Lease not confirmed, {Work} not executed", _slaveId, work.GetType().Name);
    return;
}

The same check guards the manual path: an internal "close this position" request answers 503 while the lease is not confirmed. The node's uptime record marks those minutes as degraded, with the reason "no account lease".

t = 60. lease_until passes by the database's clock. From now on, any node's claim query may take A's accounts.

t = 60 to 80. Node B's next cycle claims them. Its cycle runs every 20 seconds, so the claim lands somewhere in that window. B opens a broker session for each account, and when a follower's session is connected, B queues a reconcile for each of its links.

Later, when A reaches the database again, one of two things happens:

B has taken the account. A's cycle returns the accounts A still owns, and this one is not among them, so A logs "Account no longer owned by this node, disconnecting" and closes its session.
Nobody has taken it (the whole database was down, so nobody could claim). A's renewal goes through (WHERE node_id = <A>, whatever the expiry). Because A held back orders while it was unsure, its first cycle with a confirmed lease runs a reconcile of every link, with the reason lease_restored.

Between t = 45 and the new owner's reconcile, nobody trades that follower. That gap is the price. A duplicated order on a real account is worse than a late one, so the design pays it.

Dropped work is not replayed

During the gap the master keeps trading. A receives those events and drops them. B was not listening yet. Neither of them keeps a queue of "things to do once I am sure". Replaying would mean trusting a history that is, by definition, incomplete.

Instead, the reconcile compares two states: what is open on the master now, and what is open on the follower now. The event history does not matter. It is a pure function, which made it the easiest part of the system to test:

public static IReadOnlyList<CopyAction> Plan(ReconcileInput input, ISlaveContext slave)

The input holds the master's positions and pending orders, the follower's positions and pending orders, the link's settings, and the pair rows the database knows about (master ticket ↔ follower ticket). The output is a list of actions. Nothing runs inside it.

For each pair it knows:

The follower's copy is gone (closed by its own stop, by hand, by a stop-out): mark the pair closed, do not open a new one.
The master's position is gone: close the copy.
The master was partially closed while nobody listened: close the same share of the copy.
Stop loss or take profit differ: modify the copy.

For each master position without a pair:

Look for our own copy at the broker first (see below). If there is one, adopt it.
Otherwise open it late, but only if the price is still near the master's open price. "Near" is the link's own slippage limit. If the price has moved further, it records a skip (late_open_price_moved), and the trade is not copied. A copy at a much worse price is not the trade the master took.
A master ticket that already completed on this link is never opened again.

The broker decides what was already done

The hard case is a trade that was copied just before the gap.

On the copy path, the order goes to the broker first. The database write follows in a batch, about every 100 ms. If A opened a copy at t = 44 and its database connection was already gone, the pair row never arrived. B's reconcile then sees an open master position with no pair. Without anything else, B would open it a second time.

What prevents that is a tag on the order itself. Every copy carries a per-link magic number and a comment with the master's ticket, mc<masterTicket>. Before planning a late open, the reconcile looks for it among the follower's positions:

private static PositionLink? FindTagged(CopyLink link, long masterTicket, IEnumerable<Position> positions,
    IEnumerable<PendingOrder> pendings, HashSet<long> claimed)
{
    var tag = CopyLink.SlaveComment(masterTicket);
    foreach (var p in positions)
    {
        if (p.Magic == link.SlaveMagic && !claimed.Contains(p.Ticket) && TagMatches(p.Comment, tag))
        {
            return new PositionLink { /* ... */ SlaveTicket = p.Ticket, Status = MapStatus.Open };
        }
    }
    // the same for pending orders
}

A tagged position becomes the pair (AdoptAction) instead of a reason to open. The same lookup resolves a pair row that was written as "sending" but never learned its follower ticket: found by tag, it is adopted, and if it is not found, the send never reached the broker.

So the database is not the record of what the copier did. It is a fast cache of it that may lag behind. The record is the order at the broker, and it carries enough to be matched back. The tests that pin this down are small and named after the case:

Tagged_slave_order_without_a_row_is_adopted_not_reopened
Sending_row_without_ticket_is_resolved_by_tag
Untracked_tagged_orphan_is_closed

An order already on its way

The gate runs before a unit of work starts, not while it waits for the broker. An order that passed the gate at t = 44.9 is still sent, and its answer comes back after t = 45. This is what the 15 seconds are for: the order has to be out before the lease can move at t = 60. An ordinary pause fits into that. A process frozen for longer does not, and the earlier post says so.

A bug I found while writing this

Going through the timeline for this post, I followed A's buffered writes to the end. When A reaches the database again, it flushes everything it kept, including the pair row of the copy it opened at t = 44. By then B has adopted that copy and written its own live pair for the same master ticket. A partial unique index allows only one live pair per link and master ticket, so A's row is refused. The rule is right, but the writer that hit it was not ready for it. A refused copy event was written row by row and only the bad row is dropped (I learned that lesson once already). A refused pair row was retried with the whole batch, forever, and it held back the node's other writes with it.

It needs a rare combination: at least two nodes per platform, and one of them cut off from the database for a minute while it keeps copying. A clean shutdown writes everything before it lets go of the accounts. A crash loses the buffered rows, and the new owner adopts the copy from its tag. The bug never happened in production. It was still a stalled writer waiting to happen, and I had already learned that this kind costs a whole node its writes.

The fix has two parts:

Live pair rows are fenced by the owner too, like the account writes. A row that says a copy is open only goes in while this node owns the follower. A row that ends a pair always goes in, because a close is true whoever writes it.

WHERE s.status NOT IN ('Sending', 'Open')
   OR EXISTS (SELECT 1 FROM copy_links l JOIN trading_accounts t ON t.id = l.slave_account_id
              WHERE l.id = s.link_id AND t.node_id = $15)

A refused pair batch is written row by row, the same way as the copy events. Only the refused row is dropped, with what it was in the log.

I tested it against a throwaway Postgres with the real writer: B writes its adopted pair, A's stale live row is left out, A's own rows go in, a forced refusal is dropped, and both writers are empty and healthy afterwards.

What this does not cover

The gap is real. For the seconds between the fence and the new owner's reconcile, nothing is copied to that follower. A trade the master opens in that window is copied late at the current price, or skipped if the price has moved past the link's limit.
A reconcile repairs state, not timing. It gets the follower back in line with the master. It cannot make the copy as early as it would have been.
There is still no test with two real nodes racing for the same rows. The reconcile is covered by unit tests because it is pure. The handover itself has run in production, not in a test.
The tag lives in a broker-side comment. If a broker rewrote or truncated order comments, the lookup would not find the copy. The reconcile also closes an order with our magic number that no pair and no tag explains, so that case leans towards fewer positions, not more.

Takeaways

Plan the failover the same way as the normal path. Write it down second by second, with the defaults filled in.
A node that is not sure it owns something should stop acting and keep nothing to replay. Repair from state, not from a history it might have missed.
Make the reconcile a pure function: state in, actions out. It is the part that runs in every bad case, so it should be the easiest part to test.
If the write comes after the action, the action has to carry its own identity. Then the external system, not your database, tells you what you already did.
Follow every buffered write of a node that lost ownership to the end. Mine found a bug, and the fix was the same fence again.

Everything here comes from SnapCopier, the cloud trade copier for MT4 and MT5 that I am building. From the trader's side, all of the above shrinks to two promises: an account is traded by one node at a time, and a follower that missed something is compared with its master and repaired. I wrote that side up in how a cloud trade copier works without a VPS.

810 blocking calls at once: what the .NET thread pool did to the first burst

lepp-sd — Sat, 31 Oct 2026 20:48:04 +0000

I am building a cloud trade copier for MetaTrader. When a trader closes a position on the master account, the same close has to go out on every follower account, as fast as possible. For MT4 the vendor library is a .NET DLL with a synchronous API: OrderClose blocks its thread until the broker answers. In my setup that answer takes about 130 ms.

That is fine for one close. It stops being fine when 27 positions close in the same second and each of them has 30 followers.

What I measured

I measure "own time": the time from the master's callback to the moment the follower's order call is made. It excludes the broker's answer, so it is the part my code is responsible for. In a quiet minute it is 1-3 ms.

During a burst of 27 simultaneous closes (810 copies, MT4 follower accounts, a real broker with demo accounts) it looked like this, grouped by how many threads the pool had when each copy started:

thread pool threads at start	own time, median
33-36	5.5-14 ms
40-69	19 ms
73-102	45 ms
134-163	71 ms
344-486	125-190 ms (max 244 ms)

The pool had grown from 33 to 486 threads during those twelve seconds, and the delay grew with it. A staircase like that has a cause, and it is not one of the usual suspects.

What it was not

GC. The longest pause was 6-34 ms, none above 200 ms, and the staircase was too regular to be GC.
Channel wake-ups and fan-out. A microbenchmark writing one event to 30 channels showed the last reader waking about 20 µs later.
Eager listing of open positions before the send. It looked suspicious, but warm it costs 2 µs.

One note on method, because I got it wrong three times: drop the first five minutes after a restart (JIT, cold caches, a small pool), and never write log lines on the path you are measuring. A log line before the send added 2-3 ms. My measuring code only stored timestamps and a background thread printed them.

What it was

Every in-flight order call holds a pool thread for the broker's whole answer. During a burst, all of them are blocked at once, and the next piece of work (the next follower's dispatcher continuation) needs a new thread. The pool creates threads gradually, and creating a thread costs about 0.2-0.5 ms. With hundreds of blocked threads and hundreds of new ones to create, those fractions of a millisecond add up to the staircase.

ThreadPool.SetMinThreads did not help the way I expected, for a dull reason: it silently rejects the whole call if either value is above the maximum. The I/O thread maximum was 1000 and I had asked for 1024, so the minimum had never taken effect and stayed at the core count (4 on my server). A warning on rejection and a log line with the effective minimum at startup fixed that. It removes the "two new threads per second" throttle, but not the cost of creating the thread.

The fix: threads that never retire

The calls that block for the broker's answer now run on their own threads, created once, kept forever, and reused:

public Task<T> Run<T>(Func<T> work, CancellationToken ct = default)
{
    var job = new Job<T>(work, ct, () => Interlocked.Decrement(ref _inFlight));
    var wanted = Interlocked.Increment(ref _inFlight);
    _queue.Add(job);

    // One thread per outstanding call, up to the cap: never fewer threads than calls to run at once.
    EnsureThreads(wanted);
    return job.Task;
}

Each worker is a plain loop over a BlockingCollection. A few details turned out to matter:

The task's continuation must not run on these threads (TaskCreationOptions.RunContinuationsAsynchronously), or an awaiting caller can hold one.
The in-flight counter is decremented before the task completes. Whoever continues from the task and starts the next burst then sees the true number of running calls, and no thread is created for a call that has already finished.
A call still waiting for a thread is cancelled at once, like Task.Run, not when a thread finally gets to it.
Past the cap, calls wait in a FIFO queue.

My first fix was not enough

I started the executor with 64 ready threads and let it grow on demand. A burst of 12 closes (360 MT4 copies) then made it create about 296 threads right in the copy path, and the own time barely improved:

	closes	own time p50 / p95 / max
before the fix, warm pool	12	2.8 / 20.7 / 25.2 ms
executor, 64 ready threads	12	25.9 / 55.3 / 61.1 ms
executor, 512 ready threads	12	1.2 / 5.0 / 7.9 ms

The cost was never the pool's. It was thread creation, and it is paid wherever it happens in the burst. The fix was to create the threads before the burst: the worst case is followers × per-follower parallelism (30 × 16 = 480 here), so that is how many threads the executor starts with. With 512 ready threads the burst created none, and the own time stayed at about 1 ms regardless of the burst size.

The three waves are different bursts at different times, not a controlled A/B, but they have the same size (12 closes), and the thread counts show the mechanism: 296 created in the second, 0 in the third.

What a ready thread costs

Sleeping threads are cheap. On Linux with .NET 10, 1024 idle threads cost about 35 MiB of resident memory (35 KiB each) and 186 ms to create at startup. The whole node stayed at 318-380 MiB resident with 452 threads, the same as before the change.

A side effect I did not prove

During the 64-thread run, my MT5 node, which has no blocking calls and was unchanged, also slowed down: p95 went from about 2 ms to 28.6 ms. The two nodes share a machine, and the MT4 node was creating 300-450 threads. With 512 ready threads the MT5 numbers went back to normal. The correlation is measured; the cause is not proven.

Takeaways

A synchronous call on the thread pool costs a thread for as long as it blocks. In a burst, that is a lot of threads, created exactly when you can least afford it.
Warming up the pool is not enough, and neither is SetMinThreads: creating a thread costs time wherever it happens. Create the threads before the burst.
Size them from the worst case you can compute, not from the average. Mine is one multiplication.
Check that SetMinThreads actually returned true.

If you are curious about the project: I build SnapCopier, a cloud trade copier for MT4 and MT5. The numbers above are from its own node logs.