Gaussian Processes: Covariance Functions (Kernels)

Adapted from an appendix of my MS thesis.

Covariance Functions

A covariance function encodes our assumptions about the function which we wish to learn. In supervised learning, the notion of similarity between data points is also crucial. It is a basic assumption that points $\boldsymbol{x}_ p$ and $\boldsymbol{x}_ q$ that are close are likely to have a similar target value $y$ . Under the Gaussian process view it is the covariance function that defines nearness or similarity [1].

A stationary covariance function is a function of $\boldsymbol{x}-\boldsymbol{x}'$ . Thus it is invariant to translations in the input space. If further the covariance function is a function only of $|\boldsymbol{x}-\boldsymbol{x}'|$ then it is called isotropic, and is thus invariant to all rigid motions. The squared exponential covariance function is isotropic. These functions are also known as radial basis functions (RBFs). Its formulation is as follows where $\ell$ is the characteristic length scale [1].

$$k(\boldsymbol{x},\boldsymbol{x}') = \exp\left(-\frac{|\boldsymbol{x}-\boldsymbol{x}'|^ 2}{2\ell^ 2}\right).$$

Furthermore, if a covariance function depends only on $\boldsymbol{x}$ and $\boldsymbol{x}'$ through $\boldsymbol{x}\cdot\boldsymbol{x}'$ it is called a dot product covariance function. The linear covariance function is a special case of the dot product covariance function. The linear covariance function is written as follows where $\boldsymbol{c}$ is constant [1].

$$k(\boldsymbol{x},\boldsymbol{x}') = (\boldsymbol{x}-\boldsymbol{c})(\boldsymbol{x'}-\boldsymbol{c}).$$

Squared exponential covariance function — samples from a GP prior [2].

Linear covariance function — samples from a GP prior [2].

Matérn 5/2 kernel — samples from a GP prior [2].

Matérn 3/2 kernel — samples from a GP prior [2].

Exponential kernel — samples from a GP prior [2].

Polynomial kernel — samples from a GP prior [2].

A general name for a function $k$ of two arguments mapping a pair of inputs $\boldsymbol{x}\in\mathcal{X},\boldsymbol{x}'\in\mathcal{X}$ into $\mathbb{R}$ is a kernel. A real kernel is said to be symmetric if $k(\boldsymbol{x},\boldsymbol{x}')=k(\boldsymbol{x}',\boldsymbol{x})$ . Covariance functions must be symmetric by definition [1].

References

1. Rasmussen, Carl Edward, Williams, Christopher K. I. (2005) Gaussian Processes for Machine Learning. The MIT Press.
2. Bill Engels, Oriol Abril Pla, Juan Orduz (2023) Mean and Covariance Functions. Zenodo.