Why a Scheduler Cannot Trust Its Own Database

If your scheduler submits transactions, its database is not a record of current chain state. It is a local projection assembled from RPC reads, transaction submissions, receipts, logs, and assumptions about what happened between those events.

That projection is useful. Scheduling would be impractical if every decision required rebuilding state from chain history. But the projection is always slightly wrong. The important engineering question is not whether it becomes stale; it is how the system behaves when it does.

There are two failure directions: stale-permissive state and stale-restrictive state. They have different costs and need different controls.

The database is authoritative about intent, not chain facts

A scheduler can authoritatively say, “I created this job,” “I attempted this transaction,” or “I assigned this idempotency key.” Those are facts owned by the application.

It cannot authoritatively say, “This wallet still has enough balance,” “This nonce is unused,” “This contract remains unpaused,” or “That transaction is final.” Those are chain facts. The database can cache observations of them, but an observation has a block context and an age.

Even an RPC read is not an timeless answer. A read against a recent block can be displaced by a reorganization. Different RPC nodes can briefly disagree. A transaction visible in the pending pool may disappear, be replaced, or execute with different surrounding state than the scheduler expected.

Treating cached chain data as ordinary application state hides all of this uncertainty.

Stale-permissive: the database says yes

Stale-permissive state means the local model allows an action that current chain state rejects or makes unsafe.

Common examples include:

• A cached balance still includes funds already spent elsewhere.
• A nonce has been consumed by another transaction.
• An authorization was removed after the last sync.
• A contract parameter changed, invalidating the scheduler’s assumptions.
• A previous transaction succeeded, but the local worker timed out and scheduled the same intent again.

The obvious cost is a reverted transaction and wasted gas. The less obvious cost is incorrect execution. A transaction can succeed while doing something different from what the scheduler intended because price, liquidity, contract configuration, or account state changed before inclusion.

Preflight simulation helps, but it is not a lock on future state. Other transactions can land between simulation and execution. The right interpretation is “this action was valid against this observed state,” not “this action will remain valid.”

Safety-critical predicates therefore need checks as close to submission as practical. The scheduler should also encode constraints in the transaction itself when the contract interface supports them. A database check can reduce bad attempts; transaction-level constraints can stop execution when state moves beyond accepted bounds.

Stale-restrictive: the database says no

Stale-restrictive state is the mirror case. The database blocks an action that chain state would allow.

This often happens when:

• A worker marks an intent as pending and crashes before clearing its lease.
• A transaction is replaced, but the database keeps watching the old hash.
• A receipt or log is missed because an RPC request failed.
• A transaction succeeds while the local record remains in a submitted state.
• A temporary chain condition clears, but the scheduler never revisits the job.

This direction usually preserves funds but harms liveness. Jobs remain stuck, valid execution windows pass, and operators have to repair records manually. Over time, conservative patches can turn transient uncertainty into permanent blockage.

A common mistake is to treat “unknown” as “failed” or “still pending.” Unknown should be its own state. It tells the reconciliation process to gather evidence rather than blindly submit again or wait forever.

A simple failure sequence

Consider a scheduler that records an intended transaction and then broadcasts it:

intent: created
transaction: submitted
local state: pending
worker: crashes
chain: transaction succeeds
receipt update: never written

On restart, resubmitting may duplicate the action. Refusing to resubmit may leave the job stuck indefinitely.

The scheduler needs a third path: inspect the account nonce, transaction receipt, relevant logs, and resulting contract state. It can then classify the intent using chain evidence. That process is reconciliation.

Reconciliation is a control loop

Reconciliation should run continuously, not as an incident script. A practical design separates command handling from observation:

• Persist an intent before submission, with an idempotency key tied to the logical action.
• Record every submission attempt and transaction hash.
• Advance state through explicit transitions rather than overwriting a generic status field.
• Scan receipts and logs from a durable checkpoint.
• Re-read critical contract state for records that remain uncertain.
• Expire worker leases independently from transaction status.
• Revisit recently confirmed records when the chain can still reorganize.
• Rewind derived state when a previously observed block leaves the canonical chain.

Logs are efficient for broad discovery, while direct reads are useful for checking current state. Neither is sufficient alone. A log scanner can miss data because of checkpoint bugs or provider failures. A direct read can show the current value without explaining which local intents produced it.

Reconciliation also needs bounded concurrency and rate controls. An outage can create a large backlog, and an eager recovery worker can overwhelm the same RPC provider it depends on. The recovery path should be designed as normal production traffic, not exceptional traffic with unlimited resources.

Decide where freshness is worth its cost

Reading chain state before every action reduces some stale-permissive failures, but adds latency and consumes RPC budget. It can also create a false sense of certainty because the state can change after the read.

Skipping fresh reads improves throughput, but shifts more responsibility to transaction constraints, idempotency, and later reconciliation. For low-risk observations, cached state may be acceptable. For actions involving balances, permissions, nonces, or irreversible contract calls, fresher evidence is usually worth more.

There is no universal freshness policy. The honest trade-off is latency and RPC usage versus correctness risk. Make that decision per predicate rather than applying one cache duration to the whole scheduler.

Disclosure: I build Tradevo, software that compiles plain-English crypto strategies into deterministic rules, tests and paper-trades them, supports review and marketplace publishing, and executes subscribed rules on Base from wallets users control.

A reliable scheduler does not eliminate stale state. It makes staleness visible, limits what stale assumptions can authorize, and continuously compares its local model with chain evidence. Reconciliation is not cleanup after the system fails. It is part of the system’s normal operation.