Linear Mapping have a Memory

Linear Mappings Have a Memory

I've been looking at something surprisingly simple: what happens when a linear mapping is treated explicitly as a linear associative memory.

One particularly interesting observation is that when a weighted sum is trained beyond capacity, new examples don't simply erase old ones. Instead, the weight vector is continually perturbed, gradually adding noise to previously stored associations. Recent examples are recalled better, but older examples can persist statistically for a surprisingly long time.

Even more interestingly, below capacity, removing a training example may produce no change whatsoever in the weight vector—until weight decay is introduced. Then the mapping can move to a different, lower-norm solution while preserving the remaining associations.

These elementary observations have some interesting consequences for initialization, weight decay, SGD dynamics, and ultimately CCSLM, where the local experts can themselves be viewed as factorized associative memories.

I've written up the technical details and some implications for CCSLM here:
Linear Mappings have a Memory