Neural Ordinary Differential Equations: Why Depth Can Be a Continuous Variable

A residual network is built from layers. Each layer takes a hidden state, adds a small change, and passes the result on. Write that update down, shrink the step size, and something familiar appears: the network starts to look like a differential equation. Neural ordinary differential equations, usually called Neural ODEs, take that resemblance seriously. Instead of choosing a fixed number of layers in advance, you define a vector field and let a numerical solver decide how the hidden state evolves.

The result is a model whose "depth" is not an integer you pick before training. It is a continuous quantity, governed by the solver, the tolerance you set, and the dynamics the network learns.

From residual layers to a differential equation

A residual block updates a hidden state $h$ by

$$h_{t+1} = h_t + f(h_t, \theta_t)$$

where $f$ is a small neural network and $\theta_t$ are the parameters of layer $t$ . Compare that with the Euler method for an ordinary differential equation (ODE). For a state $h(t)$ evolving under

$$\frac{dh(t)}{dt} = f(h(t), t, \theta)$$

one Euler step of size $\Delta t$ gives

$$h(t + \Delta t) = h(t) + \Delta t \cdot f(h(t), t, \theta)$$

The residual update is exactly this step with $\Delta t = 1$ . The difference is the point of view. A residual network fixes the step size and the number of steps. An ODE formulation keeps the dynamics $f$ and hands the stepping to a solver.

In a Neural ODE, the model is the function $f$ , parameterised by $\theta$ . Given an input $h(t_0)$ , the output is the solution of the initial value problem at some later time $t_1$ :

$$h(t_1) = h(t_0) + \int_{t_0}^{t_1} f(h(t), t, \theta) \, dt$$

The integral is not computed symbolically. A black-box ODE solver evaluates $f$ as many times as it needs to reach the requested accuracy. A simple problem may need few evaluations. A stiff or rapidly changing problem may need many. That is the sense in which depth becomes adaptive: the solver, not the architect, decides how much computation the forward pass uses.

This also changes what the parameters mean. In a residual network, layer 7 has its own parameters, distinct from layer 8. In the basic Neural ODE, the same parameters $\theta$ define the vector field at every time, and time $t$ is given to $f$ as an extra input so the dynamics can change along the trajectory. The model is therefore parameter-efficient in the same way a recurrent network is: one function, reused across time, rather than a fresh copy per step.

A concrete picture: a trajectory, not a stack

It helps to stop picturing layers as boxes in a row. Picture a particle instead. The input places the particle at a starting point in hidden space. The learned vector field assigns an arrow to every point in that space, saying which way to move and how fast. The solver follows the arrows. The output is where the particle ends up.

Classification with a Neural ODE then looks like this: embed the input into hidden space, let the flow carry each example along its trajectory, and read out a prediction from the final position. Examples from the same class should be carried to the same region, while examples from different classes should be carried apart. Training adjusts the vector field until the flow does that job.

This picture also explains a striking property of the basic model. Solutions of an ODE with a well-behaved vector field cannot cross. If two trajectories started at different points and crossed, they would share a point and then have to follow the same future, which contradicts uniqueness of solutions. In practical terms, the flow is a homeomorphism: it can stretch, twist, and fold space continuously, but it cannot tear it or glue two separate regions together.

That constraint has consequences. A plain Neural ODE cannot separate some simple configurations that a residual network can, because separating them would require trajectories to cross. The standard remedy is to lift the state into a higher-dimensional space, appending extra dimensions (often initialised to zero) that give trajectories room to pass around each other. This is called augmentation, and it restores much of the lost expressiveness. If you use Neural ODEs and hit a wall on a task that seems too easy to fail, the crossing constraint is one of the first suspects.

Training without storing every step: the adjoint method

The obvious way to train a Neural ODE would be to backpropagate through the solver, step by step. That works, but it defeats one of the main attractions of the model: memory efficiency. A solver may take hundreds of steps, and storing the state at every step for the backward pass costs memory proportional to the number of steps.

The adjoint method avoids that. Instead of storing the forward trajectory, it defines a second ODE, running backward in time, for the sensitivity of the loss to the hidden state. Call that sensitivity the adjoint state $a(t)$ :

$$a(t) = \frac{\partial L}{\partial h(t)}$$

where $L$ is the loss. The adjoint obeys its own differential equation:

$$\frac{da(t)}{dt} = -a(t)^T \frac{\partial f(h(t), t, \theta)}{\partial h(t)}$$

and the gradient with respect to the parameters is an integral along the trajectory:

$$\frac{dL}{d\theta} = -\int_{t_1}^{t_0} a(t)^T \frac{\partial f(h(t), t, \theta)}{\partial \theta} \, dt$$

In practice, the backward pass solves an augmented ODE that reconstructs $h(t)$ by running the forward dynamics backward, while simultaneously integrating the adjoint and the parameter gradient. The memory cost is then constant in the number of solver steps, because no intermediate states are stored. The trade-off is numerical: reconstructing $h(t)$ backward is not always exact, especially for dynamics that are not reversible in practice, and small numerical errors in the reconstruction can bias the gradient. Modern implementations often checkpoint some states or use a solver with tighter tolerances on the backward pass to control this. Constant memory is a real advantage, but it is bought with numerical care, not for free.

Where the continuous view pays off

Neural ODEs are not a drop-in replacement for every network, and they are usually slower per forward pass than a fixed-depth network of comparable size, because the solver may evaluate $f$ many times. They earn their place in settings where continuity itself is the point.

Irregularly sampled time series. Real measurements rarely arrive on a neat grid. A patient may be observed at hours 0, 3, and 11; a sensor may drop out for a day. A recurrent network has to decide what a "step" means when the gaps vary. A Neural ODE handles it naturally: the state evolves continuously between observations, and an observation updates the state when it arrives. Models in this family, such as ODE-RNNs, combine a continuous evolution between observations with a discrete update at each observation, and they are a natural fit for clinical and sensor data.

Continuous normalizing flows. The same change-of-variables idea that underlies normalizing flows has a continuous version. If a state evolves under an ODE, the log density along the trajectory changes according to the trace of the Jacobian of $f$ :

$$\frac{d \log p(h(t))}{dt} = -\mathrm{Tr}\left( \frac{\partial f}{\partial h(t)} \right)$$

Computing a trace is far cheaper than computing a determinant, which is one reason continuous flows scale to higher dimensions than many discrete flows. The trace is often estimated stochastically rather than computed exactly, which introduces its own variance to manage.

Learning dynamics from data. When the data really come from a dynamical system, such as a physical process or a controlled system, parameterising the derivative directly is a strong inductive bias. The model is asked to learn the law of motion, not just a mapping from inputs to outputs, and it can then be evaluated at times it never saw during training, or integrated further into the future.

Adaptive computation. Because the solver adapts its step size to the local difficulty of the trajectory, easy examples cost less than hard ones, at least in principle. In practice the saving is uneven: the solver tolerance, not the example difficulty alone, dominates the cost, and a badly conditioned vector field can make every example expensive.

A minimal implementation sketch

The sketch below shows the structure, using a generic ODE solver interface. The network defines the derivative; the solver does the integration; the loss is computed from the final state.

import torch
import torch.nn as nn
from torchdiffeq import odeint

class VectorField(nn.Module):
    def __init__(self, hidden_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(hidden_dim + 1, 64),
            nn.Tanh(),
            nn.Linear(64, hidden_dim),
        )

    def forward(self, t, h):
        # Append time so the dynamics can vary along the trajectory.
        t_feature = torch.full_like(h[:, :1], t)
        return self.net(torch.cat([h, t_feature], dim=1))

class NeuralODE(nn.Module):
    def __init__(self, hidden_dim):
        super().__init__()
        self.func = VectorField(hidden_dim)

    def forward(self, h0):
        t = torch.tensor([0.0, 1.0])
        trajectory = odeint(self.func, h0, t)
        return trajectory[-1]

Two details in this sketch deserve attention. First, time is an input to the vector field, so a single network can define different dynamics at different points along the trajectory. Without that, the model would be autonomous and strictly less expressive. Second, the solver tolerance, which is not shown here, is a training hyperparameter in its own right: too loose and the gradients are noisy, too tight and training crawls.

Limitations and common pitfalls

Speed. A Neural ODE forward pass can be several times slower than a residual network with a similar number of parameters, because the solver evaluates the vector field repeatedly and the number of evaluations is not fixed in advance. If latency matters, measure function evaluations per forward pass, not just parameter count.

Stiffness and solver choice. Some learned dynamics are stiff: they contain fast and slow timescales at once. Explicit solvers such as Runge-Kutta then need tiny steps to stay stable, and training slows to a crawl. Implicit solvers handle stiffness better but are more expensive per step and harder to differentiate through. Solver choice is part of the model, not an afterthought.

The crossing constraint. As discussed above, trajectories of an ODE cannot cross. If a task requires the flow to separate interleaved classes, a plain Neural ODE will struggle no matter how large the vector field is. Augmenting the state dimension is the usual fix, and it should be tried before concluding that the approach fails.

Numerical error in the backward pass. The adjoint method reconstructs the forward trajectory backward in time. For dynamics that amplify small differences, that reconstruction drifts, and the gradient drifts with it. If training is unstable in a way that does not respond to learning rate changes, try tightening solver tolerances, checkpointing, or backpropagating through a fixed-step solver for comparison.

Tolerance as a hidden hyperparameter. Two runs with the same architecture but different solver tolerances are effectively different models. Report the solver and tolerance alongside the architecture, or results will not reproduce.

Do not confuse depth with time. The continuous variable $t$ in a Neural ODE is an integration variable, not necessarily physical time. In an image classifier it is an abstract depth variable. In a time-series model it may correspond to real elapsed time. Conflating the two leads to wrong intuitions about what the model can extrapolate.

The takeaway

Neural ODEs reframe a familiar object. A residual network is a discretisation of a differential equation, and once you see that, the number of layers stops being a structural choice and becomes a numerical one. The model is the vector field; the solver decides how far and how carefully to follow it.

That reframing buys real things: constant-memory gradients through the adjoint method, a natural treatment of irregularly sampled data, and a direct way to learn dynamical laws. It also imposes real constraints: trajectories cannot cross, solvers can be slow, and numerical tolerance becomes part of the model specification.

The deeper lesson is about levels of description. Much of deep learning is built from discrete steps because hardware and software make steps convenient. But the functions those steps approximate are often continuous, and sometimes naming the continuous object first, and letting the discretisation follow, gives a cleaner model and a clearer account of what can go wrong.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial