Rethinking Learning Dynamics in AI Models: An Early Theory from Experimentation

Observing Representation Instability During Neural Network Training

While experimenting with neural network training behaviors, I started noticing a recurring pattern that does not seem to be explicitly discussed in most mainstream AI literature. This article is not meant to present a finalized solution, but rather a working theory that emerged during development.

I am sharing this to ask: does this interpretation make sense, or am I misunderstanding the dynamics at play?

Background Assumption

Most modern AI systems, particularly deep learning models, rely on gradient-based optimization. The underlying assumption is relatively straightforward:

Minimize loss → improve performance

However, during a series of experiments, I observed that loss minimization alone does not always correlate with meaningful representation learning, especially in early training phases.

This led me to hypothesize that:

AI models may pass through a “representation instability phase” where gradients optimize surface-level patterns before stable internal abstractions emerge.

I am not fully confident whether this is already well-known under a different name, or whether I am misinterpreting training noise as structure.

Initial Observation

While training a small transformer-like model on synthetic data, I logged intermediate layer activations and noticed something interesting:

• Early epochs show highly volatile embeddings
• Mid-training shows sudden clustering behavior
• Late training stabilizes, even when loss improvement slows down

Here is a simplified snippet of the training loop I used:

for epoch in range(num_epochs):
    optimizer.zero_grad()

    outputs = model(inputs)
    loss = criterion(outputs, targets)

    loss.backward()
    optimizer.step()

    with torch.no_grad():
        embedding_norm = model.encoder.weight.norm().item()

    print(f"Epoch {epoch} | Loss: {loss.item():.4f} | Embedding Norm: {embedding_norm:.2f}")

`

What surprised me is that embedding norms and cosine similarities changed more drastically than loss values, especially early on.

Tentative Theory

My current theory (and this is where I’m unsure):

Gradient descent initially prioritizes optimization shortcuts rather than semantic structure, and only later converges toward representations that are robust and generalizable.

If this is true, then:

• Early stopping might prevent meaningful abstraction
• Some overfitting phases might actually be necessary
• Regularization might delay, not prevent, representation collapse

But this raises questions:

• Is this just an artifact of small datasets?
• Is this already explained by concepts like loss landscape flatness or mode connectivity?
• Am I confusing emergent structure with random alignment?

A Small Diagnostic Experiment

To test whether representations actually stabilize, I added a simple cosine similarity tracker:

`python
def cosine_similarity(a, b):
return torch.nn.functional.cosine_similarity(a, b, dim=0)

prev_embedding = None

for epoch in range(num_epochs):
# training step...

current_embedding = model.encoder.weight.clone().detach()

if prev_embedding is not None:
    similarity = cosine_similarity(
        prev_embedding.view(-1),
        current_embedding.view(-1)
    )
    print(f"Epoch {epoch} | Embedding Stability: {similarity.item():.4f}")

prev_embedding = current_embedding

`

The similarity score jumps erratically at first, then begins converging toward ~0.98–0.99, even when loss improvement becomes marginal.

This makes me wonder:

Is loss the wrong primary signal for understanding learning progress?

Open Questions for Discussion

I’m genuinely unsure whether this line of thinking is insightful or redundant, so I’d like to open this up:

1. Is this “instability → abstraction → stabilization” pattern formally recognized?
2. Could this be explained by information bottleneck theory, or is that a stretch?
3. Are there better metrics than loss for tracking learning quality?
4. Am I over-interpreting noise due to small-scale experiments?

Closing Thoughts

I feel like I’m observing something real, but I’m not convinced I’m explaining it correctly yet.

• If this theory is flawed, I’d like to know where the reasoning breaks.
• If it’s valid, I’d like to understand how others frame it more rigorously.

At this stage, I’m treating this less as a claim and more as a question:

What exactly is an AI model learning before it learns what we want it to learn?