Gaussian Processes: Covariance Functions (Kernels)

Adapted from an appendix of my MS thesis.

Covariance Functions

A covariance function encodes our assumptions about the function which we wish to learn. In supervised learning, the notion of similarity between data points is also crucial. It is a basic assumption that points $\boldsymbol{x}_ p$ and $\boldsymbol{x}_ q$ that are close are likely to have a similar target value $y$ . Under the Gaussian process view it is the covariance function that defines nearness or similarity [1].

A stationary covariance function is a function of $\boldsymbol{x}-\boldsymbol{x}'$ . Thus it is invariant to translations in the input space. If further the covariance function is a function only of $|\boldsymbol{x}-\boldsymbol{x}'|$ then it is called isotropic, and is thus invariant to all rigid motions. The squared exponential covariance function is isotropic. These functions are also known as radial basis functions (RBFs). Its formulation is as follows where $\ell$ is the characteristic length scale [1].

$$k(\boldsymbol{x},\boldsymbol{x}') = \exp\left(-\frac{|\boldsymbol{x}-\boldsymbol{x}'|^ 2}{2\ell^ 2}\right).$$

Furthermore, if a covariance function depends only on $\boldsymbol{x}$ and $\boldsymbol{x}'$ through $\boldsymbol{x}\cdot\boldsymbol{x}'$ it is called a dot product covariance function. The linear covariance function is a special case of the dot product covariance function. The linear covariance function is written as follows where $\boldsymbol{c}$ is constant [1].

$$k(\boldsymbol{x},\boldsymbol{x}') = (\boldsymbol{x}-\boldsymbol{c})(\boldsymbol{x'}-\boldsymbol{c}).$$

Example functions drawn from a GP prior with a given covariance function. (Left) The squared exponential covariance function. (Right) The linear covariance function [2].

Samples from additional covariance functions. (Top Left) The Matérn 5/2 kernel. (Top Right) The Matérn 3/2 kernel. (Bottom Left) The exponential kernel. (Bottom Right) The polynomial kernel [2].

A general name for a function $k$ of two arguments mapping a pair of inputs $\boldsymbol{x}\in\mathcal{X},\boldsymbol{x}'\in\mathcal{X}$ into $\mathbb{R}$ is a kernel. A real kernel is said to be symmetric if $k(\boldsymbol{x},\boldsymbol{x}')=k(\boldsymbol{x}',\boldsymbol{x})$ . Covariance functions must be symmetric by definition [1].

References

1. Rasmussen, Carl Edward, Williams, Christopher K. I. (2005) Gaussian Processes for Machine Learning. The MIT Press.
2. Bill Engels, Oriol Abril Pla, Juan Orduz (2023) Mean and Covariance Functions. Zenodo.