A strategy game that keeps running while nobody is online

I make Last Thaw, a free browser strategy game where up to a hundred players start with five settlers on a real map of Earth and have 30 real days to build an empire before an ice age, advancing from both poles on a public schedule, takes the land. It's in early access and I'm looking for people to test it.

The design rule that shaped the whole server is simple: there is no pause. You queue a building, start a research, send an army across the Atlantic, close the tab and go to work. When you come back eight hours later, all of it has happened, in the right order, whether or not anybody was logged in.

That sounds like a game loop. It isn't, and the first thing I learned was that treating it like one would have been the wrong call. This post walks through how the server actually does it: a jobs table ordered by finish time, regions that are settled lazily instead of ticked, and the locking mistakes I made along the way.

Why not just tick everything?

The obvious approach is a fixed tick: every N seconds, walk every region in every match, add production, advance timers, resolve whatever finished.

A match has 408 regions. With a handful of matches running at once you're doing thousands of row updates per tick, and almost all of them are doing nothing interesting. A region of forest in Siberia that nobody owns does not need to be rewritten every 30 seconds just so its wood number goes up.

So I split time into two kinds:

1. Things that happen at a moment. A building finishes, a tech completes, an army arrives, the ice moves one step south. These are discrete and they have an exact time.
2. Things that accrue continuously. Food, wood, stone and population growth. These are just rates times elapsed time.

The first kind goes in a jobs table. The second kind is never stored as "current", only as "the value at the last time someone looked".

The jobs table

Every timed thing in the game is a row with an ends_at. Queuing a building in a region is an insert:

await tx
  .insert(jobs)
  .values({
    matchId: m.id,
    playerId: p.id,
    regionId,
    kind: "build",          // build, train, research, mission, ice_step, storm, match_end...
    refId: buildingId,      // which building
    level,                  // the level it reaches
    payload: { cost, paid },
    startsAt,
    endsAt,                 // the only thing the worker cares about
  })
  .returning();

The ice is the same. When a match starts, every step of the ice schedule is inserted as an ice_step job, so the advancing glacier is just more rows. The end of the match is a match_end job. Nothing is special-cased.

A separate worker process polls once a second for anything due and completes it, strictly in ends_at order:

const due = await db
  .select({ id: jobs.id, matchId: jobs.matchId })
  .from(jobs)
  .innerJoin(matches, eq(matches.id, jobs.matchId))
  .where(
    sql`${jobs.completedAt} is null and ${jobs.cancelledAt} is null
        and ${jobs.attempts} < ${MAX_ATTEMPTS}
        and ${jobs.waitingFor} is null
        and ${matches.status} = 'running'
        and ${jobs.endsAt} <= now()`,
  )
  // Same instant: the match end goes last, so the final ice step
  // and arrivals of that moment still resolve.
  .orderBy(jobs.endsAt, sql`case when ${jobs.kind} = 'match_end' then 1 else 0 end`, jobs.id)
  .limit(500);

That ordering comment came from a real bug. Two jobs can share an ends_at to the millisecond (the last ice step and the end of the match are both scheduled from the same start time). With only ORDER BY ends_at, Postgres was free to finish the match first, and the final ice step then landed on a finished match. The tie-breaker makes the rule explicit.

Each job completes in its own transaction, and the completion is stamped with the job's own ends_at, not with now():

await db.transaction(async (tx) => {
  const [job] = await tx.select().from(jobs)
    .where(eq(jobs.id, id))
    .for("update", { skipLocked: true });   // another worker has it? skip
  if (!job || job.completedAt || job.cancelledAt) return;
  const [m] = await tx.select().from(matches)
    .where(eq(matches.id, job.matchId))
    .for("no key update");
  await complete(tx, m, job, (fn) => after.push(fn));
  await tx.update(jobs).set({ completedAt: job.endsAt }).where(eq(jobs.id, id));
});
for (const fn of after) await fn().catch(() => {});   // publish only after commit

Using job.endsAt as "now" inside the handler matters more than it looks. If the worker was down for ten minutes (a deploy, say), it wakes up to a backlog. Each job still resolves as of the moment it was supposed to, so a building that finished at 09:02 starts producing from 09:02, not from 09:12 when the worker got round to it. A late worker never changes the outcome, only when you hear about it.

The after list is the other half of that. Websocket messages ("your Granary is finished") are queued during the transaction and only sent once it commits. Early on I published inside the transaction, and a rollback would leave players with a notification for a building that didn't exist.

Poisoned jobs

If a handler throws, the transaction rolls back and the job is still due, so the next pass a second later tries it again. And again. One bad row was enough to fill the logs and, worse, it sat at the front of the queue in ends_at order.

The fix is an attempts counter written outside the failed transaction:

} catch (err) {
  log.error({ err, jobId: id }, "job failed");
  await db.update(jobs)
    .set({
      attempts: sql`${jobs.attempts} + 1`,
      lastError: String((err as Error).message ?? err).slice(0, 500),
    })
    .where(eq(jobs.id, id))
    .catch(() => {});
}

After five tries the job is parked (the attempts < 5 in the query above) and the error text is on the row for me to read in the morning. In practice nearly every parked job so far was a bug I wanted to know about, not something to retry harder.

I also found jobs left open on matches that had already finished, still sitting there as "due" forever. Now a separate loop cancels anything still open on a finished match once a minute.

Regions are settled, not ticked

Now the continuous half. A region row stores its stock (food, wood, stone, population) and a settled_at timestamp. That's it. The current value is a projection:

/** Pure projection of a row to `asOf`. Does not touch the database. */
export function projectRegion(row: MatchRegionRow, asOf: Date, speed: number, tech: TechEffects) {
  const traits = traitsOf(row.regionId);
  const effects = applyTechEffects(effectsOf(row.buildings), tech);
  const economy = settle(traits, economyOf(row, effects), elapsedSeconds(row.settledAt, asOf), speed);
  return { economy, rates: computeRates(traits, economy), effects };
}

settle lives in a shared package, so the browser runs the exact same function. When you look at a region, the client takes the row from the API and animates the numbers forward locally. No polling for "how much wood do I have now".

The rule that keeps it correct: settle before you change a rate. Anything that changes production (a building finishing, units being trained out of the workforce, a region changing hands) first locks the row and writes the projected values up to that instant:

export async function settleAndLockRegion(tx: Tx, matchId: string, regionId: string, asOf: Date, speed: number) {
  const [row] = await tx.select().from(matchRegions)
    .where(and(eq(matchRegions.matchId, matchId), eq(matchRegions.regionId, regionId)))
    .for("update");
  if (asOf.getTime() <= row.settledAt.getTime()) return row;   // already current
  const { economy } = projectRegion(row, asOf, speed, await ownerTechEffects(tx, row.ownerPlayerId));
  const [updated] = await tx.update(matchRegions)
    .set({ ...economyColumns(economy), settledAt: asOf })
    .where(and(eq(matchRegions.matchId, matchId), eq(matchRegions.regionId, regionId)))
    .returning();
  return updated;
}

When a building finishes, its handler calls this with job.endsAt. Everything before that moment is accrued at the old rate, then the building is added and the new rate applies from there. The unowned Siberian forest is never written until something actually happens to it.

Where it went wrong: lock order

Settling is cheap, but it takes a row lock, and once more than one region is involved you're in deadlock territory.

The case that bit me was a training order. Training can be supplied from other regions of the same player (the stock gets carried in). The first version locked the region you were training in, worked out what was short, then locked the supplying regions. Meanwhile an army coming home locked its destination region and then others, and the periodic loyalty pass walked a player's regions in its own order.

In one overnight match that combination deadlocked 14 times. Postgres detected each one and killed a transaction, the retry succeeded, and nobody lost anything, but it was a sign the design was wrong rather than unlucky.

The fix was boring and it worked: when an order might touch several regions, it locks every region of that player up front, in ascending region id, before doing anything else. Same order everywhere, no cycles. There's now a test that holds a lower region locked from another connection and checks the order waits on it without having grabbed a higher one first.

The match row has a smaller version of the same problem. Completing a job locks the match with FOR NO KEY UPDATE rather than FOR UPDATE. Inserting a job, an army or an order takes a key-share lock on the match through its foreign key, and FOR UPDATE conflicts with those. NO KEY UPDATE still serialises job completions against other match updates without fighting every insert.

Testing a 30-day game

You can't wait 30 days to see whether the ice reaches the equator on time. Every piece of simulation code asks one helper for "now":

export function matchNow(match: { clockOffsetSeconds: number }, realNow = new Date()): Date {
  return new Date(realNow.getTime() + match.clockOffsetSeconds * 1000);
}

In development a match can be pushed forward by bumping its offset, and the job query compares ends_at against now() plus that offset. Because jobs complete as of their own ends_at and regions settle to an explicit instant, skipping a day produces the same state as living through it. The full test plays a whole 30-day game this way in a few minutes.

What I'd tell myself at the start

• Store moments as rows with a finish time. Store rates as a value plus a timestamp. Don't store "now".
• Complete each event as of its own time, so a late or restarted worker can't change the result.
• Publish after commit.
• Count failures and park poisoned work instead of retrying forever.
• Pick one lock order for multi-row writes on day one. You will need it.

If you want to see what this looks like from the player's side, the how-to-play page covers the timers and the ice schedule, and a new match opens every week at lastthaw.com. It's early access, so if you play it and something feels off, I'd really like to hear about it. Feel free to borrow any of this for your own timer-driven game.