A deposit watcher carries two numbers that look interchangeable and are not. The confirmation depth decides when an incoming payment gets credited. The reorg window decides how long that credit can still be taken back. Neither one answers the third question, which block a poll should start reading from, and if you reuse the window floor as the read offset you get a watcher that is correct only while it runs at least once per block.
Nothing guarantees that. A poller runs on a timer, inside a process that gets deployed, restarted, killed for memory and occasionally throttled by its own node. A chain with ten-second blocks will mine several of them while a container image pulls. If the read starts at the window floor and the window is six blocks deep, a transfer that landed eight blocks ago is not in the answer at all, and you get no second chance to see it: a chain source returns the chain as it looks now, not a log of what was ever seen.
This post is about the three heights a poll has to consider, and why the lowest of them wins. What that buys you comes later. The code is in chain-watch, a small package I maintain for exactly this bookkeeping: https://github.com/polycratia/chain-watch
The window is a promise about reversal
The reorg window is a policy statement, not an I/O hint. Its floor is tip.height - reorg_depth + 1, never below start_height. Everything at or above that line is provisional, and everything below it is history. That is the only reversibility the watcher actually has: a deposit inside the window can be handed back, and a deposit below it is final, so it will not be reverted and will not be reported twice.
The window therefore tells you which blocks you must re-examine to honour that promise. It says nothing about which blocks you have already examined. Those are two different facts, and the moment a poll can be late they stop coinciding.
Three floors, and the minimum wins
So poll computes the window floor first, and then lowers it:
def poll(self) -> PollResult:
tip = self._source.tip()
floor = max(self._start_height, tip.height - self._reorg_depth + 1)
addresses = sorted(self._addresses)
def _read_floor(self, floor: int) -> int:
since = floor
if self._last_tip is not None:
since = min(since, self._last_tip + 1)
for transfer in self._pending.values():
since = min(since, transfer.block.height)
return max(self._start_height, since)
That is the whole idea, in seven lines: three candidate heights, one minimum, clamped so a read never goes below the height the watcher was told to start at. The result is the since_height handed to the source's transfers method.
The window floor is there for the revert check. The second floor, self._last_tip + 1, is there for the gap: whatever the last poll saw, this poll reads from the block after it, however long ago that was. The third one is easy to miss.
The pending floor: let the chain confirm, not memory
A transfer can wait longer than the window is deep. By default the window matches the policy's deepest requirement, but the window stays a separate argument precisely because you may want a tighter one:
policy = ConfirmationPolicy(default=6, per_asset={"ETH": 12, "USDT": 12})
watcher = DepositWatcher(source, ["addr-1"], policy=policy, reorg_depth=6)
Now take a USDT transfer that needs twelve confirmations. Depth counts the block itself:
def confirmations_for(block: BlockRef, tip: BlockRef) -> int:
"""Depth of ``block`` under ``tip``, counting the block itself as one."""
if tip.height < block.height:
return 0
return tip.height - block.height + 1
By the time that transfer has eight confirmations it is still pending, and its block already sits two blocks below a six-deep window floor. If the read stopped at the window floor, the source would stop returning it. The watcher would then be promoting a deposit on the strength of a Transfer it is holding in a dictionary: crediting money because of what it remembers, not because of what the chain currently says. Worse, under the drop rule below, it would delete the transfer instead.
So the read reaches down to transfer.block.height for every transfer still pending. A payment waiting outside the window gets confirmed by the chain or taken away by it, and either way the decision comes from the current answer.
What the wide read licenses: dropping a missing pending transfer
The three floors together guarantee one specific thing: every transfer the watcher still has an opinion about sits inside the range it just read. That is what makes absence meaningful, and absence is otherwise the hardest signal to interpret in this kind of service, I think.
Because of it, poll does its deletions first. Before any confirmation work, it removes every pending key the answer no longer holds, and reports nothing for them. Nothing is reported because nothing was credited: a pending transfer never moved a balance, so there is no reversal to announce. reverted is kept for deposits that were actually reported and are still inside the window. Only after that pass does the poll check window deposits against what was seen, hold newly seen transfers as pending unless they are already reported or settled, promote pending ones whose confirmations_for satisfies the policy, and settle deposits that have fallen below the floor. The last thing it does is record tip.height in _last_tip.
Ordering matters here. Deleting first means a transfer that vanished and a transfer that was re-mined at a different height go through the same mechanism: the key is dropped if the answer does not contain it, and re-added from the answer if it does, carrying whatever block it now sits in. Identity is the DepositKey, the (tx_id, output_index) pair, which names what was paid rather than where it was mined, so a replaced block leaves the key alone.
The snapshot has to carry the tip
All of this collapses on restart unless the last tip survives the process. So it is part of the serialised state, next to the pending transfers, the reported deposits and the settled keys:
@dataclass(frozen=True, slots=True)
class WatcherState:
start_height: int = 0
pending: tuple[Transfer, ...] = ()
reported: tuple[Deposit, ...] = ()
settled: tuple[DepositKey, ...] = ()
last_tip: int | None = None
to_dict writes it under STATE_VERSION, currently 2, and from_dict accepts None or an integer, rejecting a bool on the way through the integer coercion and raising InvalidState for any other version. The constructor refuses a non-zero start_height passed beside a state (two sources of truth for the same number is not something to resolve at runtime) and adopts the snapshot's start_height and last_tip, so a resumed watcher reads from the block after the one it stopped at.
The test that pins this down is the clearest statement of the whole argument:
def test_a_snapshot_carries_the_last_polled_tip():
chain = ScriptedChain()
watcher = DepositWatcher(chain, ["addr-1"], policy=2, reorg_depth=2)
chain.mine(count=3)
assert not watcher.poll()
snapshot = json.loads(json.dumps(watcher.state.to_dict()))
assert snapshot["last_tip"] == 3
chain.mine(pay("tx-a"))
chain.mine(count=4)
resumed = DepositWatcher(
chain,
["addr-1"],
policy=2,
reorg_depth=2,
state=WatcherState.from_dict(snapshot),
)
result = resumed.poll()
assert chain.last_query == (("addr-1",), 4)
assert keys(result.confirmed) == [("tx-a", 0)]
The snapshot was taken at tip 3. The chain moved on to 8 while the process was down. A two-deep window would have the resumed watcher reading from height 7, and the payment in block 4 would never get credited. It reads from 4, and the assertion on last_query is there so that a future refactor cannot quietly narrow the range.
The cost, stated honestly
The price is a wider read. A process that slept for a day asks its source for a range proportional to how long it slept, and one stuck pending transfer keeps the floor low until it resolves. That cost lands on the ChainSource implementation, which is where you can pay it properly: paginated, cached, or served from an indexer that was going to scan that range anyway. The alternative is cheaper per poll and wrong in exactly the situation you cannot design away, a late poll.
What I would do differently
My first version of this used the window floor as the read offset, because when you are writing the reorg logic the window is the number on your mind. It was correct in a single long-running process that never fell behind, which is the one operating condition nobody can promise you... The bug it produces is also the worst kind in a payments path: silent and intermittent, visible only as a customer telling you a deposit never arrived.
Two things I would do from the first version rather than later. First, treat the last polled tip as part of the public contract instead of internal bookkeeping: a state object without it cannot resume honestly, only approximately. Second, log the computed since_height on every poll next to the tip height. When a deposit goes missing, that one number tells you right away whether the watcher never looked or looked and saw nothing, and those two failures have nothing in common.
The window governs how long the watcher may change its mind. Where it starts reading is a different question, and the answer is the lowest block it still has an opinion about.
Originally published on polycratia.com — where I write about payment systems, crypto rails and marketplace backends.
Top comments (17)
the distinction between what the reorg window promises and which blocks you must re-examine is the one that keeps getting collapsed in real integrations, usually because the watcher was written against a demo that polls every block. the pending floor lowering the read to the oldest unconfirmed transfer is the quietly load-bearing detail here, since a missing transfer is only evidence of a reorg if the read actually covered the height it was last seen at. the snapshot carrying last_tip is the same argument one layer up: a restart that loses the tip silently narrows the read, and the missing-transfer inference becomes wrong without anything erroring.
Exactly — the key distinction is between “not found” and “not found in the range we actually checked.”
Without tracking
last_tip, a restart can silently turn a missed range into a false assumption about the chain’s state. And the pending floor matters for the same reason: a transfer can outlive the reorg window while still waiting for enough confirmations, so its block must remain within the read range.I’d also make
since_heightpart of the poll’s structured logs, alongsidetip.heightand the oldest pending block. That makes it much easier to distinguish a genuine disappearance from a watcher that simply never looked far enough back.Thanks for highlighting this — it’s exactly the kind of edge case that can pass every happy-path test and still cause missed deposits in production.
logging since_height on every poll is the cheap half; the expensive half is the test that replays a restart against a chain that moved between snapshot and reconnect, which almost nobody writes because the happy path never exercises it. the pending floor also interacts with checkpointing: if the watcher persists its cursor after confirming transfers but before the floor advances, the safest restart policy is to resume from min(cursor, oldest pending block) and re-read the overlap idempotently rather than trust the gap.
The restart test is already there:
test_a_snapshot_carries_the_last_polled_tipsnapshots at tip 3, mines to 8 with nothing running, and assertschain.last_query == (("addr-1",), 4). It asserts on the query rather than the credit, so a refactor cannot quietly narrow the range and still pass.test_a_poller_asleep_past_the_window_still_confirms_what_was_minedis the same shape, minus the round trip through JSON.On checkpointing, there is no separate cursor to go stale against the pending floor.
_read_floortakes the minimum of the window floor,_last_tip + 1andtransfer.block.heightacross everything in_pending, clamped up to_start_height, and it does that on every poll, not only the first one after a restart. Pending transfers ride insideWatcherStateright next tolast_tip, so a resumed watcher computes your min(cursor, oldest pending) from the same inputs, with the window floor as a third term.Your hazard is real one layer out.
_last_tipgets assigned at the end ofpoll, afterconfirmedis built: if the caller credits a balance and then dies before persisting the snapshot, the resumed watcher holds noreportedorsettledentry for that key and reports it again. Once and only once holds across a snapshot that was written, not across a crash between reporting and writing it, andDepositKeyis stable across exactly that restart, which is what makes the caller's credit dedupable.I would add a test that pins that re-report down as documented behavior, and a read-only accessor for the computed read floor so a caller can log it without reaching into
poll. Both go on the list...Fair correction on both halves. The gap I was reaching for is the one you named: once-and-only-once holds across a written snapshot, and the window between confirmed being built and the snapshot persisting is where the re-report fires, with the caller's stable DepositKey absorbing the duplicate. Pinning that as documented behavior matters because a caller reading the type names will assume exactly-once across the crash too, then file the duplicate as a bug; a test plus a changelog line makes the integration side design for the dedup instead. The read-only accessor matters more than it first looks: the min(cursor, oldest pending, window floor) computation is the number that answers "how far back did this restart actually re-read", and nobody should have to reach into poll internals just to log it.
Both halves go on the list, and the wording fix goes first: it is the part that sets the wrong expectation. The README headline promises to "notify once and only once" with no qualifier, and that is the line a caller reads before any type name. The honest form is once-and-only-once per persisted snapshot, at-least-once if the caller credits a balance and dies before writing one. There is no CHANGELOG in the repo yet, so that sentence belongs in the README next to the status note, and in the
WatcherStatedocstring, which right now only warns that memory alone would replay.The test is the easy half: credit a deposit, drop the watcher without taking a snapshot, build a new one from the older snapshot, then assert the key comes back in
confirmeda second time. That pins down a window that is currently undefined.One detail for the integration side you are designing for. The replayed report is not identical to the first.
Deposit.confirmationsis computed against the tip at report time, so the second one carries a deeper count, and if the transfer was re-mined theblockdiffers too.DepositKeyis the only part that holds still, which is why dedup has to key on(tx_id, output_index)and not on the deposit value.On the accessor I agree with the need and not with the shape of a plain property. You cannot compute the floor without a tip: the window term is
tip.height - reorg_depth + 1, so an attribute read would have to calltip(), and I do not want network I/O hiding behind one. What fits islast_since_height, recorded where_last_tipis assigned at the end ofpolland exposed next to it, reporting the floor the last poll actually used rather than predicting the next one. It stays out ofWatcherState: pending blocks andlast_tipare both in the snapshot already, and the number is derivable from them.last_since_height is the right shape. hiding a tip() call behind a property read is exactly the kind of surprise that gets found in production, and the floor the last poll actually used is the honest number anyway, since it is the one the watcher already acted on. on the replay drift, that makes the README note do double duty: consumers should match on (tx_id, output_index) and treat confirmations and block as advisory, because a replayed report can legitimately carry a deeper count or a re-mined block.
misunderstanding
last_since_heightholds whatever_read_floorhanded back, assigned inpollright next to_last_tipand read through a plain property that touches nothing. It sits atNoneuntil this process has polled once, snapshot or not: the floor some previous process used is not a floor this watcher ever acted on, and the first poll recomputes it from_last_tip,_pendingand the window floor anyway. That is also why it stays out ofWatcherState. Persisting it would cost you aSTATE_VERSIONbump for a number the next poll derives for free.The advisory wording goes into the README's
Depositparagraph, which today says only thatconfirmationsis the count the deposit had when it was reported. The addition: a replayed report can carry a deeper count and, after a re-mine, a differentblock, so(tx_id, output_index)is the only pair a consumer should match on.fair, that is the case I glossed over: until this process has polled once there is no floor it acted on, so "the one the watcher already acted on" is empty at cold start, snapshot or not. the None-until-first-poll behavior is the honest version of what I was reaching for, and the first-poll recompute from _last_tip, _pending and the window floor is what makes the first report trustworthy instead of inherited. agreed on keeping it out of WatcherState too: persisting a derived value buys a STATE_VERSION bump for a number the next poll derives for free. thanks for untangling it.
Worth pulling one thing out of that agreement: at cold start
last_since_heightisNonebecause no poll has happened yet, but on a resume the first floor gets computed instead of inherited._read_floorpulls_last_tip + 1out of the snapshot along with the block height of every transfer in_pending, takes the minimum against the window floor, then clamps up to_start_height. So a resumed first read can legitimately start below the block after the snapshot tip, any time a pending transfer sits lower than that. The inputs are persisted. The number is not.That pairing is also why the log line is worth reading. A
Nonefloor sitting next to a reallast_tiptells you this process has not polled yet, which is a different fact from a poll that ran and read a narrow range, and the two want different responses from whoever is looking.the resumed-first-read detail is the sharp one: the inputs are persisted and the number is derived, so a resume can legitimately start below the snapshot tip block whenever a pending transfer sits lower, and anyone who assumed resume means resuming from the persisted tip would call that a bug. the clamp up to _start_height is what keeps that first read honest. the log-line point is worth copying too: None next to a real last_tip is a state, not a missing value, and treating it as one would send someone debugging a narrow first poll as if it were a process that never polled.
One more input gets persisted than it looks: the clamp floor itself. The constructor refuses a non-zero
start_heightpassed beside astateand adoptsstate.start_height, so the lower bound_read_floorclamps up to on a resume comes out of the snapshot, not from the call that rebuilt the watcher.The clamp also isn't a resume guard. It runs on every poll, and you only see it when
_last_tip + 1or a pending block height drops below it: same arithmetic whether the process has been up for a week or 3 seconds.On the log line, I'll put that distinction in the
last_since_heightdocstring instead of leaving it to be inferred.Nonemeans this process hasn't polled yet, not that the floor is missing.the docstring is the right home for it, and the every-poll framing is the one worth carrying into tests: assert the clamp against any tip regression, not a resume fixture, and the snapshot-vs-argument question stops mattering. the part I'd add a log to is the silent adoption: a non-zero start_height passed beside a state being ignored reads like an override to most callers, so a restart against a new snapshot can think it moved the floor when it moved nothing. a line when the passed value loses to the snapshot's catches that at the rebuild instead of at the first poll below the floor.
There's nothing to log for the silent adoption. The constructor raises
ValueErrorwhen a non-zerostart_heightarrives beside a state, so a caller who thinks they moved the floor gets stopped at the rebuild, not at a first poll below it. Adoption only fires whenstart_heightis zero, and a zero that was passed explicitly looks the same as the default: there is no passed value to compare the snapshot's against. My earlier wording said refuses, and refuses there means raises.The test framing stands. I'll add one to
tests/test_watcher.pythat polls, then forks the chain back belowstart_height, and assertslast_queryclamps up to it, with no snapshot in the fixture.got it, the raise at rebuild is the guard and 'refuses' meaning 'raises' was my misread. the explicit-zero case collapsing into the default is an ambiguity worth a docstring line rather than a fix: a caller passing zero means it anyway. and the fork-below-start_height test is the right pin, it fails exactly where a naive reader would replay from genesis instead of clamping.
Agreed, and that docstring line belongs on the constructor's
start_heightparameter instead of sitting near the raise: that is where the ambiguity lives. An explicit zero and the default are the same value, so the snapshot'sstart_heightwins and nothing a caller meant gets lost. The raise covers only the non-zero case.One constraint on the test, now that I've written out what it asserts: the pin only bites with a non-zero
start_height, because_read_floorends inmax(self._start_height, since)and a default of zero clamps nothing. A fork back to height 1 withstart_height=0reads from 1 and is correct to. So the fixture mines past astart_heightabove the fork point, polls, forks below it, and assertslast_querycomes back clamped.