Learning the Heater From Its Own Logs

The first code post in this series had a predict function whose body I typed by hand. Heat leaks towards outside at a tenth of the difference per step, the heater adds a degree and a half, and I made both of those numbers up. That was fine for showing the shape of a world model. It is useless for a real room, because a real room does not tell you its constants. What it gives you instead is a log.

This post throws the hand-written body away and fits a new one from logged state and action pairs. The model is four numbers. The training loop is twenty lines. At the end it predicts a room it has never seen more closely than my guess did, and you can run the whole thing in your browser.

A room that keeps its constants to itself

I don't have a thermostat wired up to this blog, so a simulator stands in for the hardware. It has the same shape as the 101 predict, with two changes: the constants are different from the ones I guessed, and there is a little noise, because someone keeps opening the door.

const REAL_LEAK = 0.07;   // better insulated than I guessed
const REAL_HEAT = 1.2;    // and a weaker heater
const NOISE = 0.1;        // draughts, in °C per step

function realRoom(state: State, action: Action): State {
  const heaterOn =
    action === "turnHeaterOn" ? true :
    action === "turnHeaterOff" ? false :
    state.heaterOn;
  const leak = (state.outsideTemp - state.roomTemp) * REAL_LEAK;
  const heat = heaterOn ? REAL_HEAT : 0;
  const draught = gauss() * NOISE;
  return {
    roomTemp: state.roomTemp + leak + heat + draught,
    outsideTemp: state.outsideTemp,
    heaterOn,
  };
}

The rule for the rest of the post is that nothing below this line gets to look inside realRoom. The model only sees what a logger would see.

Recording the logs

The log comes from running a thermostat against the room. Each episode picks an outside temperature between 2 and 12 degrees, a starting room temperature between 10 and 22, a setpoint between 18 and 21, and runs sixty steps. One step in five the thermostat does something random instead, because a log with only sensible decisions in it never shows the model what a daft decision does. Eighty episodes gives 4,800 transitions, each one the Transition type from the 101 post.

function record(episodes: number, steps: number): Transition[] {
  const log: Transition[] = [];
  for (let e = 0; e < episodes; e++) {
    let state: State = {
      roomTemp: 10 + rand() * 12,
      outsideTemp: 2 + rand() * 10,
      heaterOn: rand() < 0.5,
    };
    const setpoint = 18 + rand() * 3;
    for (let t = 0; t < steps; t++) {
      const action = thermostat(state, setpoint);
      const nextState = realRoom(state, action);
      log.push({ state, action, nextState });
      state = nextState;
    }
  }
  return log;
}

const log = record(80, 60);

Here are the first two rows, which is all of the data the model ever gets, four thousand eight hundred times over.

{ state: { roomTemp: 16.51, outsideTemp: 7.62, heaterOn: false },
  action: "turnHeaterOn",
  nextState: { roomTemp: 17.09, outsideTemp: 7.62, heaterOn: true } }
{ state: { roomTemp: 17.09, outsideTemp: 7.62, heaterOn: true },
  action: "wait",
  nextState: { roomTemp: 17.75, outsideTemp: 7.62, heaterOn: true } }

If you run energy hardware, this is your telemetry table with the commands joined on, which was the whole point of the first post.

The WHERE clause runs out

The last post computed a world model with a filter and a count: keep the rows where the state and action matched, tally what happened next. It worked because temperatures were bucketed into cold, mild and warm. Try it on the real log and it falls over immediately.

const probe = log[1234];
const matches = log.filter(t =>
  t.state.roomTemp === probe.state.roomTemp &&
  t.state.outsideTemp === probe.state.outsideTemp &&
  t.state.heaterOn === probe.state.heaterOn &&
  t.action === probe.action);

matches.length;  // 1, and it's row 1234 itself

Loosening the match to within a twentieth of a degree finds two rows. A continuous state almost never recurs, so the condition on the right of the bar almost never matches, which is the same wall the LLM equation hit in that post. The fix there was a very large neural network. The fix here is a very small one, because the room is a very small world.

The model is four numbers

What I want is a function with the predict signature whose body I did not write. The simplest body that could work is a weighted sum. Take the readings the logger can see, multiply each by a weight, add them up, and call the result the change in room temperature for this step.

type Weights = { room: number; outside: number; heater: number; bias: number };

function features(state: State, action: Action) {
  const heaterOn =
    action === "turnHeaterOn" ? true :
    action === "turnHeaterOff" ? false :
    state.heaterOn;
  return { room: state.roomTemp, outside: state.outsideTemp, heater: heaterOn ? 1 : 0, bias: 1, heaterOn };
}

function predictWith(w: Weights, state: State, action: Action): State {
  const f = features(state, action);
  const delta = w.room * f.room + w.outside * f.outside + w.heater * f.heater + w.bias * f.bias;
  return { roomTemp: state.roomTemp + delta, outsideTemp: state.outsideTemp, heaterOn: f.heaterOn };
}

Predicting the change rather than the next temperature outright is the one design decision in here, and it matters more than it looks. The next temperature is dominated by the current temperature, so a model asked to predict it spends its capacity learning "next is roughly the same as now" and has little left for the interesting part. The change is the interesting part on its own.

Notice what the model is not given. It does not know that the leak depends on the difference between inside and outside. It gets the two temperatures as separate inputs and has to discover the relationship, if there is one.

The training loop

Training means finding the four weights that make predictWith agree with the log. Gradient descent does that the slow and honest way: start at zero, measure how wrong the model is across every row, nudge each weight in the direction that reduces the error, repeat.

function train(log: Transition[], epochs: number, lr: number) {
  const w: Weights = { room: 0, outside: 0, heater: 0, bias: 0 };
  const losses: number[] = [];
  for (let epoch = 0; epoch < epochs; epoch++) {
    const g: Weights = { room: 0, outside: 0, heater: 0, bias: 0 };
    let loss = 0;
    for (const t of log) {
      const f = features(t.state, t.action);
      const target = t.nextState.roomTemp - t.state.roomTemp;
      const guess = w.room * f.room + w.outside * f.outside + w.heater * f.heater + w.bias * f.bias;
      const err = guess - target;
      loss += err * err;
      g.room += err * f.room; g.outside += err * f.outside;
      g.heater += err * f.heater; g.bias += err * f.bias;
    }
    const n = log.length;
    w.room -= lr * g.room / n; w.outside -= lr * g.outside / n;
    w.heater -= lr * g.heater / n; w.bias -= lr * g.bias / n;
    losses.push(loss / n);
  }
  return { w, losses };
}

const { w, losses } = train(log, 10000, 0.003);

The err * f.room lines are the entire derivative. For a weighted sum with a squared error, the gradient of the loss with respect to each weight is the error times that weight's input, which is why no library is needed. Ten thousand passes over 4,800 rows takes about 300 milliseconds in Node 23.

{{< chart name="heater-loss" csv="heater-loss" caption="Mean squared error per epoch. It flattens at 0.01, which is the variance of the draughts: the model has learned everything in the log that is not noise." >}}

Ten thousand epochs is a lot for four weights, and the reason is worth knowing. Room temperature and outside temperature move together in the log (warm days make warm rooms), so the loss surface is a long thin valley and plain gradient descent zigzags down it. Scaling the inputs or using a smarter optimiser would fix that in a few hundred epochs. I left it slow because the loop is easier to read this way, and because the playground is more fun when you can watch it converge.

Here are the weights it found, next to the constants realRoom was hiding.

weight	learned	truth
room	-0.074	-0.07
outside	0.069	0.07
heater	1.19	1.2
bias	0.077	0

The room and outside weights came out equal and opposite. Nobody told the model that heat flow depends on the difference between the two temperatures. It found that from the logs, which is the small version of the thing world models are for. The bias absorbed a little of the room weight's share, because the two inputs are correlated and the noise lets them trade off. That is what a fit looks like when the data is real rather than typed in.

And predict now has a learned body with the same signature as before.

const predict = (state: State, action: Action) => predictWith(w, state, action);

A room it has never seen

The test that matters is a room outside the training range. Every episode in the log had an outside temperature between 2 and 12 degrees. The test room is at minus four, starts at 22, and follows a script the thermostat never produced: heater off for ten steps, on for ten, off for ten. The same script runs through three functions: the real room, the learned model, and the hand-written guess from 101.

const unseen: State = { roomTemp: 22, outsideTemp: -4, heaterOn: false };

// rollout from 101, with the step function passed in instead of baked in
const truth = rollout(unseen, script, realRoom);
const learned = rollout(unseen, script, predict);
const guessed = rollout(unseen, script, predictGuess);

{{< chart name="heater-rollout" csv="heater-rollout" caption="Thirty steps at minus four outside. The learned model tracks the real room to within a few tenths of a degree the whole way. The hand-written guess leaks too fast and heats too hard, and is three degrees adrift by step thirty." >}}

The numbers at the three turning points:

step	real room	learned	hand-written
10, heater goes on	8.78	8.79	5.07
20, heater goes off	11.02	11.28	8.93
30	3.69	3.80	0.51

Root mean square error over the thirty steps is 0.20 degrees for the learned model and 2.77 for my guess. The model extrapolated to an outside temperature six degrees colder than anything in its logs, because what it learned was the physics rather than the trajectories. A lookup table could not have done that, and neither could I, since I had the constants wrong.

What four weights cannot hold

The playground at /heater/ runs all of the above. Change the room's constants, record fresh logs, press train, and watch the weights settle while the prediction on the right pulls into line with the real room. Turn the draughts up and the loss floor rises to match. Set the leak rate to 0.10 and the heater to 1.5 and the hand-written guess becomes correct, which is a nice reminder that my guess was only ever a model with its weights typed in.

It also shows where this model stops. The weighted sum can only learn relationships that are a straight line through the inputs it can see. The room I built is linear and fully observed, so four weights are enough. Give the real room a door that opens and closes on a schedule the logger never records, and the best a linear model can do is average over the door, predicting a room that is always half open. The field it needs is one that is not in the log. The 102 post called that a private field, and the next post gives the model one and watches it work out the door on its own.