When a node loses its lease anyway: a takeover, second by second

I am building a cloud trade copier for MetaTrader. Each follower account has one broker session on one node, and that node copies the master's trades onto it. In an earlier post I described how a node leases its accounts from Postgres: a node_id and a lease_until on the account row, FOR UPDATE SKIP LOCKED to claim, and a node that stops sending orders 15 seconds before its lease ends.

That post was about keeping ownership. This one is about losing it: what happens, second by second, when a node is cut off from the database, another node takes its accounts, and both have to end up agreeing with what is actually open at the broker.

The defaults

	default
lease	60 s
renewal cycle	every 20 s
fence (stop sending before the lease ends)	15 s
retry after a failed cycle	2 s

These are configuration values, not measurements.

The timeline

Say node A last renewed its leases at t = 0. Then its connection to the database breaks. The broker sessions and the message bus are still fine, which is the uncomfortable part: A can still trade, it just can no longer prove that it should.

t = 0. The renewal query wrote lease_until = now() + 60 s, so in the database the lease runs until at least t = 60. A recorded its own deadline as started + 60 s - 15 s, where started was read before the query.

t = 20. The next cycle fails. Nothing changes yet. A logs the error and tries again every 2 seconds. It keeps its broker sessions and keeps copying, because its lease is still confirmed.

t = 45. HoldsLease turns false. From this moment every unit of work for every follower on A is dropped at the gate, right before it would execute:

if (!_engine.HoldsLease)
{
    // Cut off from the database long enough that another node may own this slave now:
    // sending orders could duplicate its copies. The reconcile after the next lease
    // renewal (or on the new owner) catches up.
    _log.LogWarning("[slave {Id}] Lease not confirmed, {Work} not executed", _slaveId, work.GetType().Name);
    return;
}

The same check guards the manual path: an internal "close this position" request answers 503 while the lease is not confirmed. The node's uptime record marks those minutes as degraded, with the reason "no account lease".

t = 60. lease_until passes by the database's clock. From now on, any node's claim query may take A's accounts.

t = 60 to 80. Node B's next cycle claims them. Its cycle runs every 20 seconds, so the claim lands somewhere in that window. B opens a broker session for each account, and when a follower's session is connected, B queues a reconcile for each of its links.

Later, when A reaches the database again, one of two things happens:

• B has taken the account. A's cycle returns the accounts A still owns, and this one is not among them, so A logs "Account no longer owned by this node, disconnecting" and closes its session.
• Nobody has taken it (the whole database was down, so nobody could claim). A's renewal goes through (WHERE node_id = <A>, whatever the expiry). Because A held back orders while it was unsure, its first cycle with a confirmed lease runs a reconcile of every link, with the reason lease_restored.

Between t = 45 and the new owner's reconcile, nobody trades that follower. That gap is the price. A duplicated order on a real account is worse than a late one, so the design pays it.

Dropped work is not replayed

During the gap the master keeps trading. A receives those events and drops them. B was not listening yet. Neither of them keeps a queue of "things to do once I am sure". Replaying would mean trusting a history that is, by definition, incomplete.

Instead, the reconcile compares two states: what is open on the master now, and what is open on the follower now. The event history does not matter. It is a pure function, which made it the easiest part of the system to test:

public static IReadOnlyList<CopyAction> Plan(ReconcileInput input, ISlaveContext slave)

The input holds the master's positions and pending orders, the follower's positions and pending orders, the link's settings, and the pair rows the database knows about (master ticket ↔ follower ticket). The output is a list of actions. Nothing runs inside it.

For each pair it knows:

• The follower's copy is gone (closed by its own stop, by hand, by a stop-out): mark the pair closed, do not open a new one.
• The master's position is gone: close the copy.
• The master was partially closed while nobody listened: close the same share of the copy.
• Stop loss or take profit differ: modify the copy.

For each master position without a pair:

• Look for our own copy at the broker first (see below). If there is one, adopt it.
• Otherwise open it late, but only if the price is still near the master's open price. "Near" is the link's own slippage limit. If the price has moved further, it records a skip (late_open_price_moved), and the trade is not copied. A copy at a much worse price is not the trade the master took.
• A master ticket that already completed on this link is never opened again.

The broker decides what was already done

The hard case is a trade that was copied just before the gap.

On the copy path, the order goes to the broker first. The database write follows in a batch, about every 100 ms. If A opened a copy at t = 44 and its database connection was already gone, the pair row never arrived. B's reconcile then sees an open master position with no pair. Without anything else, B would open it a second time.

What prevents that is a tag on the order itself. Every copy carries a per-link magic number and a comment with the master's ticket, mc<masterTicket>. Before planning a late open, the reconcile looks for it among the follower's positions:

private static PositionLink? FindTagged(CopyLink link, long masterTicket, IEnumerable<Position> positions,
    IEnumerable<PendingOrder> pendings, HashSet<long> claimed)
{
    var tag = CopyLink.SlaveComment(masterTicket);
    foreach (var p in positions)
    {
        if (p.Magic == link.SlaveMagic && !claimed.Contains(p.Ticket) && TagMatches(p.Comment, tag))
        {
            return new PositionLink { /* ... */ SlaveTicket = p.Ticket, Status = MapStatus.Open };
        }
    }
    // the same for pending orders
}

A tagged position becomes the pair (AdoptAction) instead of a reason to open. The same lookup resolves a pair row that was written as "sending" but never learned its follower ticket: found by tag, it is adopted, and if it is not found, the send never reached the broker.

So the database is not the record of what the copier did. It is a fast cache of it that may lag behind. The record is the order at the broker, and it carries enough to be matched back. The tests that pin this down are small and named after the case:

Tagged_slave_order_without_a_row_is_adopted_not_reopened
Sending_row_without_ticket_is_resolved_by_tag
Untracked_tagged_orphan_is_closed

An order already on its way

The gate runs before a unit of work starts, not while it waits for the broker. An order that passed the gate at t = 44.9 is still sent, and its answer comes back after t = 45. This is what the 15 seconds are for: the order has to be out before the lease can move at t = 60. An ordinary pause fits into that. A process frozen for longer does not, and the earlier post says so.

A bug I found while writing this

Going through the timeline for this post, I followed A's buffered writes to the end. When A reaches the database again, it flushes everything it kept, including the pair row of the copy it opened at t = 44. By then B has adopted that copy and written its own live pair for the same master ticket. A partial unique index allows only one live pair per link and master ticket, so A's row is refused. The rule is right, but the writer that hit it was not ready for it. A refused copy event was written row by row and only the bad row is dropped (I learned that lesson once already). A refused pair row was retried with the whole batch, forever, and it held back the node's other writes with it.

It needs a rare combination: at least two nodes per platform, and one of them cut off from the database for a minute while it keeps copying. A clean shutdown writes everything before it lets go of the accounts. A crash loses the buffered rows, and the new owner adopts the copy from its tag. The bug never happened in production. It was still a stalled writer waiting to happen, and I had already learned that this kind costs a whole node its writes.

The fix has two parts:

• Live pair rows are fenced by the owner too, like the account writes. A row that says a copy is open only goes in while this node owns the follower. A row that ends a pair always goes in, because a close is true whoever writes it.

WHERE s.status NOT IN ('Sending', 'Open')
   OR EXISTS (SELECT 1 FROM copy_links l JOIN trading_accounts t ON t.id = l.slave_account_id
              WHERE l.id = s.link_id AND t.node_id = $15)

• A refused pair batch is written row by row, the same way as the copy events. Only the refused row is dropped, with what it was in the log.

I tested it against a throwaway Postgres with the real writer: B writes its adopted pair, A's stale live row is left out, A's own rows go in, a forced refusal is dropped, and both writers are empty and healthy afterwards.

What this does not cover

• The gap is real. For the seconds between the fence and the new owner's reconcile, nothing is copied to that follower. A trade the master opens in that window is copied late at the current price, or skipped if the price has moved past the link's limit.
• A reconcile repairs state, not timing. It gets the follower back in line with the master. It cannot make the copy as early as it would have been.
• There is still no test with two real nodes racing for the same rows. The reconcile is covered by unit tests because it is pure. The handover itself has run in production, not in a test.
• The tag lives in a broker-side comment. If a broker rewrote or truncated order comments, the lookup would not find the copy. The reconcile also closes an order with our magic number that no pair and no tag explains, so that case leans towards fewer positions, not more.

Takeaways

1. Plan the failover the same way as the normal path. Write it down second by second, with the defaults filled in.
2. A node that is not sure it owns something should stop acting and keep nothing to replay. Repair from state, not from a history it might have missed.
3. Make the reconcile a pure function: state in, actions out. It is the part that runs in every bad case, so it should be the easiest part to test.
4. If the write comes after the action, the action has to carry its own identity. Then the external system, not your database, tells you what you already did.
5. Follow every buffered write of a node that lost ownership to the end. Mine found a bug, and the fix was the same fence again.

Everything here comes from SnapCopier, the cloud trade copier for MT4 and MT5 that I am building. From the trader's side, all of the above shrinks to two promises: an account is traded by one node at a time, and a follower that missed something is compared with its master and repaired. I wrote that side up in how a cloud trade copier works without a VPS.

Comments

[Gautam K]

Recording the deadline from a time read before the query is the detail that makes the fence safe: A always believes its lease ends earlier than the database does. One question: A's 45 s is on its own clock while lease_until is on the database clock. Is the 15 s fence also meant to cover clock drift between the node and Postgres, or do you check skew separately?

[lepp-sd]

Thanks, good catch to ask. The fence isn’t really there for skew, because the two clocks never get compared. The node only measures how much time passed on its own clock since it read the timestamp before the renewal query. Postgres only checks lease_until against its own now(), and the other node’s claim runs inside the database too. So if the node’s clock is off by a few seconds, it doesn’t matter, the offset cancels out.

Drift in speed would matter in theory, but that’s parts per million, a few milliseconds over a minute. The 15 seconds are mostly for stalls like a slow query or a GC pause.

Where it can actually break is a clock jump. Right now the node uses DateTime.UtcNow, so if NTP steps the clock back, it would trust its lease longer than it should. The fix is to measure that with a monotonic clock (Stopwatch) instead. It’s on my list, your question just moved it up a bit :)