Eight years ago I had an idea. Four weeks ago I decided to start implementing that idea. Last weekend I started running an experiment to validate this idea.
This is real data:
The model has a fixed set of 5.6B parameters. It does not expand parameters. It does not rely on an external memory system or replay buffers. This chart is the result of running four continuous learning sessions with the parameter weights at the end of one phase used to start the next. The first phase ran on a curriculum of 6 domains, the second phase on 8 domains, the third on 10, and the fourth cut back to 4 (one from phase 1, one from phase 2, and two added in phase 3). In all cases both order and duration of domains presented to the model were purely randomized. Evaluation was carried out against a held-out subset of each of the 10 domains every 500 steps.
So, what would you do with an AI model that was capable of continuously learning new things (without forgetting the old)?
Top comments (0)