Concatenated context selected linear mappings

Sean O'Connor — Wed, 02 Sep 2026 10:40:53 +0000

Imagine a chain of linear mappings each selected by something.
If the same mapping was selected each time that would be boring. Matrices such as A and B can be collapsed together to a single matrix M by matrix multiply, M=AB.

And M is actually very different from its parents A and B.

If you select A from a pool of 10 and likewise B you have 100 different possible M's from 20 matrices. They are like rabbits aren't they?
If you have 10 of A and B and C you have 1000 different matrices from 30.

This is exponentiation of expressiveness with depth similar to a neural network.

The something that is doing the matrix selection at each layer you can call the context. And that can be liberal in meaning.

That's nice but what if you now apply pressure from backpropagation on the matrix chain?
For a data channel input x and the co-selected linear mapping (eg A₂B₇C₅...) backpropagation drifts each chosen linear mapping to be an expert for the slice of the training data it sees.
Concatenated Context Selected Linear Mappings
ReLU neural networks as CCSML

Linear Mapping have a Memory

Sean O'Connor — Wed, 02 Sep 2026 07:00:52 +0000

Linear Mappings Have a Memory

I've been looking at something surprisingly simple: what happens when a linear mapping is treated explicitly as a linear associative memory.

One particularly interesting observation is that when a weighted sum is trained beyond capacity, new examples don't simply erase old ones. Instead, the weight vector is continually perturbed, gradually adding noise to previously stored associations. Recent examples are recalled better, but older examples can persist statistically for a surprisingly long time.

Even more interestingly, below capacity, removing a training example may produce no change whatsoever in the weight vector—until weight decay is introduced. Then the mapping can move to a different, lower-norm solution while preserving the remaining associations.

These elementary observations have some interesting consequences for initialization, weight decay, SGD dynamics, and ultimately CCSLM, where the local experts can themselves be viewed as factorized associative memories.

I've written up the technical details and some implications for CCSLM here:
Linear Mappings have a Memory

DEV Community: Sean O'Connor

Concatenated context selected linear mappings

Linear Mapping have a Memory