A wrong integer can come with a rising bit score. This saved experiment starts from one fixed addition, then shows the same accounting trap across a larger fixed inventory.
The network reads two operands one bit at a time, least significant bit first. Each real input step receives only the current two bits. The target is the matching bit of a + b. After w real input steps there is one terminal step (0, 0), so a width-w case has w + 1 output positions, with room for an overflow bit. The hidden state is the vector carried from step to step, starting at zero.
The network has two inputs for the current bits, eight tanh hidden coordinates, an 8 by 8 recurrent matrix, and two softmax outputs. With both bias vectors that is 106 parameters. It was trained on operands at widths 1 through 6. Unordered numeric-pair groups were assigned as a whole to training, validation or test roles, so reverse orientations and leading-zero views cannot cross into another role. Checkpoint 80 was the earliest epoch with the lowest validation bit NLL.
There are two inference modes for the same trained weights. Recurrent mode passes the previous hidden state into the next step. Reset mode clears the hidden state to zero before every input bit while all weights stay exactly the same. Reset is a post-training inference intervention, not a second fit and not a separately optimized memoryless baseline.
The fixed 1 + 15 case
The default was chosen before any model prediction. I took the first held-out width-6 ordered pair whose exact reference has at least four incoming carries equal to 1. That pair is 1 + 15.
At width 6, recurrent mode predicts 16 and gets all 7 output positions right. Reset mode predicts 14 and gets 3 of 7. The target bits, least significant first, are 0000100. The reset output is 0111000. The wrong positions are 1, 2, 3 and 4.
Those positions match the carry structure. Position 0 sees (1, 1) with no incoming carry and outputs 0. Positions 1 through 3 each see (0, 1), but the exact incoming carry is 1, so the target bit is 0 each time. Reset mode cannot use the hidden state, and this trained network outputs 1 at each of those three steps. Position 4 sees (0, 0) while the target bit is 1 because of the carry; reset outputs 0. This saved mapping shows how much recurrence matters for this particular network on this particular input. It does not describe every possible local rule.
The exact reference computes the next bit from the current operand bits and the previous carry. In this least-significant-first representation, the mathematical recurrence needs only the carry from the prior column. That is a property of the reference rule. It does not imply the trained hidden state encodes only carry.
Padding changes the denominator
Keep the values 1 and 15 and represent them at width 64. The integers do not change. Recurrent mode still predicts 16 and gets 65 of 65 positions. Reset mode still predicts 14 and gets 61 of 65. The wrong positions remain 1, 2, 3 and 4.
The added high leading-zero steps are easy for reset mode because most see (0, 0) with no incoming carry. Padding therefore raises bit accuracy without fixing the wrong sum. Bit accuracy is correct positions divided by w + 1, so the denominator includes the terminal step and every added zero step. Integer accuracy is different: the whole predicted bit string must convert to the exact sum. One wrong high-order position can still break the integer.
Inventory and the padding control
The complete fixed inventory has 64,856 unique ordered operand and width cases. Recurrent mode predicts all of them correctly. Reset mode predicts 6,290 correct integers and fails on 58,566, with every failure preserved. Its integer accuracy is 9.6984 percent and bit accuracy is 59.5060 percent.
The 64,882 suite memberships point at the same 64,856 unique cases. The 129,712 mode-case rows are two inference modes over one inventory, not two fits and not independent new problems. The declared primary union takes the small-group held-out views, the exhaustive new-range suites at widths 7 and 8, the fixed random new-range suites, and the padding views after deduplication. It has 64,499 unique cases, and reset gets 6,114 integers correct there, or 9.4792 percent. The complete inventory also keeps 17 exact training representations inside stress views, explicitly tagged. Those three accounting views mean different things.
Widths 7 and 8 are exhaustive only under the stated condition that at least one operand sets its top bit. They contain 12,288 and 49,152 cases. Widths 16, 32 and 64 each contain 500 fixed random cases, not all 64-bit operand pairs. Wider operands change both magnitude and sequence length; the padding control is the separate view that holds values fixed and adds zero steps.
The padding control starts from 64 held-out unordered numeric groups, both orientations, repeated at widths 8, 16, 32 and 64: 500 dependent views in total. At each padding width, reset mode gets exactly 21 of 125 integers right and 324 bits wrong. Its bit score climbs from 71.2 percent at width 8 to 96.01 percent at width 64, while integer accuracy stays at 16.8 percent at each width. The combined 84 correct integers is 21 times four widths. The same numeric groups repeating across widths are not independent observations.
Figures
The study figure shows one training curve and one validation curve, each with 80 epoch points. It also keeps nine overlapping suite integer scores, the padding-width split, and 127 carry probes with mean wrong-bit counts. It supports the denominator and dependent-view limits, not a general proof.
The default-states figure shows the two modes' uncalibrated p(bit 1) curves, the exact target bits, the reference carry trace that never enters the network, and the raw 8 by 7 tanh activations for each mode. The hidden coordinates are numeric values, not identified carry neurons.
Reset mode changes the hidden-state distribution and removes recurrence while leaving the operand bits and weights unchanged. That shows sensitivity to the intervention across finite saved cases. It does not identify a causal carry neuron. Softmax probabilities are not calibrated, and NLL comes from stable log-softmax rather than from displayed rounded values. The exact XOR/AND carry reference is independent and authoritative. It never becomes a model input and never repairs a wrong neural answer.
Using the page
The browser demo takes canonical decimal strings for operands from 0 through 2^64 minus 1, with a width from 1 through 64 that fits both operands. The sum may need 65 bits. Operands and decoded integers never pass through JavaScript Number. Current edits live in page memory; clear or refresh removes them. A deliberate download keeps the full inputs, both mode outputs, hidden states and attribution at the location you choose. Hashes check consistency only. For real calculations, use exact integer arithmetic.
Rechecking the saved numbers
Obtain the fixed-commit source files and keep their relative repository paths. Downloading the static app bundle alone is not enough. The compressed artifacts must be actual gzip JSON payloads. Relevant files include evaluation-001/results.json, evaluation-001/all-case-predictions.json.gz, data/evaluation-cases.json.gz, and run-001/model.json. To recompute logits from weights instead of reading published predictions, also obtain core.py from the same commit.
Run the following from the repository root. It uses only the Python 3 standard library, reads the fixed results and the real gzip predictions, and writes no files.
python3 - <<'PY'
import gzip, json
from pathlib import Path
root = Path.cwd()
results = json.loads((root / "evaluation-001/results.json").read_text())
with gzip.open(root / "evaluation-001/all-case-predictions.json.gz", "rt", encoding="ascii") as f:
preds = json.load(f)
reset_all = next(
r for r in results["overall"]
if r["mode"] == "reset-each-bit" and r["suite"] == "all" and r["width"] == "all"
)
rows = {
(p["mode"], p["width"]): p
for p in preds
if p["a"] == "1" and p["b"] == "15" and p["width"] in (6, 64)
}
for width in (6, 64):
for mode in ("recurrent", "reset-each-bit"):
p = rows[(mode, width)]
assert p["bits"] == width + 1
assert p["width"] == width
print(f"1+15 width={width} {mode}: predicted={p['predicted_sum']}, bits={p['bits']}, correct_bits={p['correct_bits']}, wrong_positions={p['wrong_positions']}")
reset_failures = sum(1 for p in preds if p["mode"] == "reset-each-bit" and not p["integer_correct"])
assert reset_all["cases"] == 64856
assert 64856 - reset_all["integer_correct"] == reset_failures
print(f"reset total integer failures: {reset_failures}")
PY
The script outputs the two width-6 rows, the two width-64 rows, and 58,566 reset integer failures.
The one fit in this experiment belongs to the frozen November 4-5 batch. This November 20 release publishes that saved experiment and adds no training. Original synthetic data, code, weights, results and figures are by Renchao Wu under Apache 2.0. The fixed PROTOCOL.md and NOTICE describe the training, evaluation and licensing boundaries. The method reference is Dive into Deep Learning, Section 9.7, used for BPTT principles only; no textbook code, prose or figures are copied. The project page is Carry Memory Lab on Hugging Face Spaces, and the demo runs at https://widechaos-carry-memory-lab.static.hf.space/index.html. The 64-bit space is not exhausted, and finite passing probes do not promise reliable arbitrary-length addition.


Top comments (0)