Separation Before Depth

Why continual learning requires more than temporal structure

An earlier essay argued that catastrophic forgetting is a consequence of temporal flatness. All knowledge in a standard neural network lives at the same depth with the same status, so any update that serves new learning can disrupt old knowledge indiscriminately.

Temporal depth addresses this by graduating commitment. Knowledge confirmed over time becomes more resistant to revision. Provisional knowledge remains easy to update. This is a real and necessary property.

But temporal depth alone does not resolve a more basic problem. In a system where all knowledge shares the same parameter space, new information and old information are entangled at the point of storage. Every update that encodes something new must simultaneously preserve compatibility with everything already encoded.

This entanglement is not a problem of depth. It is a problem of architecture. And it may need to be solved before temporal depth can work at all.

The entanglement problem

When a neural network learns from new data, it adjusts parameters that already encode prior knowledge. The system must solve two problems at once: represent the new information and maintain the integrity of existing representations.

These two objectives compete for the same resources. A gradient update that captures a new pattern necessarily shifts parameters away from configurations that captured old patterns. The system has no way to encode the new without disturbing the old, because both occupy the same representational space.

Standard mitigation strategies work within this constraint. Elastic weight consolidation penalizes changes to important parameters. Replay buffers re-expose the system to old data. Progressive expansion allocates new parameters for new tasks. Each approach reduces the damage, but none eliminates the underlying cause: new and old knowledge are stored in the same substrate and cannot be addressed independently.

Separation as an architectural principle

A different approach is to store new information apart from existing knowledge.

When the system encounters new data, it does not immediately merge that data into its existing parameter space. Instead, it places the new information in a separate structure where it can be held, evaluated, and if necessary revised without affecting what the system already knows.

The system then requires a generic mechanism for relating new information to old. This mechanism can compare, distinguish, and evaluate relevance across the boundary between the two stores. The comparison capacity is a stable capability of the system, not something that must be re-learned each time new information arrives.

This separation has several direct consequences.

New information can be evaluated on its own terms. The system can assess whether a new observation is consistent with existing knowledge, extends it, or contradicts it, without needing to modify existing knowledge to make that assessment.

Individual pieces of new knowledge can be updated or discarded independently. Because they are not entangled with other representations, revising one does not create collateral damage elsewhere.

Integration becomes a deliberate process rather than an automatic side effect. New information does not need to be merged into core representations immediately. It can remain separate until its relevance and reliability have been established.

The relationship between separation and temporal depth

Separation and temporal depth are different properties, but they depend on each other.

Temporal depth governs how committed the system should be to a given piece of knowledge. It answers the question: how much evidence would be needed to revise this?

Separation governs whether commitment can be managed at all. It answers a prior question: can the system store and evaluate new knowledge without disrupting the knowledge it already holds?

In a system without separation, temporal depth is difficult to implement. Even if the system assigns different levels of commitment to different knowledge, the act of encoding new information still propagates through the shared parameter space. Depth becomes a policy imposed on an architecture that resists it.

In a system with separation, temporal depth becomes natural. The separate store is inherently provisional. Consolidation into deeper layers occurs as evidence accumulates. The generic comparison mechanism provides the basis for evaluating whether consolidation is warranted.

Separation is the enabling condition. Temporal depth is the governance layer built on top of it.

A biological parallel

This distinction appears in biological memory systems, though it is often described in purely temporal terms.

The hippocampus and the neocortex do not differ only in how long they hold information. They differ in how they encode it. Hippocampal representations are sparse, episodic, and individually addressable. Neocortical representations are distributed, overlapping, and structurally integrated.

New experiences are encoded in the hippocampus in a format that is structurally separate from existing cortical knowledge. This separation is what allows rapid learning of new information without disrupting established understanding. Consolidation into cortical structure happens later, gradually, and selectively.

The biological solution to catastrophic forgetting is not just temporal. It is architectural. The two systems encode information differently precisely so that new learning and old knowledge do not compete for the same representational space.

Plasticity as a consequence of separation

The practical benefit of separation is plasticity. A system that can store new information without disturbing old information is free to learn at whatever rate the environment demands.

In a system without separation, there is a fundamental tradeoff between plasticity and stability. Learning quickly means disrupting old knowledge. Preserving old knowledge means learning slowly. Every system must choose a point along this tradeoff.

Separation dissolves the tradeoff. The provisional store can be highly plastic because its contents are not entangled with existing knowledge. The deep store can be highly stable because it is not directly exposed to new updates. Plasticity and stability coexist because they operate in different structures rather than competing within one.

This is the sense in which separation may be the more fundamental mechanism. Temporal depth governs the dynamics of how knowledge matures. Separation is what makes that maturation possible without self-destruction.

Implications

If this analysis is correct, then addressing catastrophic forgetting requires two architectural properties, not one.

Separation: new information is stored apart from existing knowledge and can be evaluated, revised, or discarded independently.

Temporal depth: knowledge that is repeatedly confirmed over time consolidates into more stable representations, while provisional knowledge remains easy to revise.

Neither property alone is sufficient. Separation without depth produces a system that accumulates new information indefinitely but never integrates it. Depth without separation produces a system that tries to graduate commitment within a shared space that still entangles everything.

Together, they describe an architecture in which the system can learn continuously, hold new knowledge provisionally, evaluate it against what it already knows, and consolidate it selectively as evidence warrants.

The question is no longer how to prevent forgetting. It is how to build systems that can learn without entanglement and forget without catastrophe.