Deep Learning notes

An unusual intro to deep learning.

The only primitive operation that one needs to understand to get started on in Deep Learning is dot product: vectors r and x of same dimension d can be dotted to produce a scalar $\langle r,x \rangle = \sum_{i\in [d]}{r_i x_i}$ .

To do this operation in parallel across multiple vectors $r$ , one can write it as a matrix vector product. One way to think about these dot product is that each vector r is a measurement of (an unknown) signal x. For example if vector r is [1, 0, 0, 0], then it is measuring the first coordinate of x. Likewise vector r [0.25, 0.25, 0.25, 0.25] is measuring the average value of coordinates of x. Notice we can make many measurements of x by doing this matrix vector multiplication where each measurement is independent (and possibly redundant) and done in parallel.

As an aside, are you wondering what exactly is a dot product geometrically? It is simply a (scaled) projection of the high dimensional signal along a "line". The line could be x-axis (e.g. when r i s [1, 0, 0, 0]) or some 45-degree line passing through origin.

So given an "unknown" x, one is interested in asking the question, hey which direction should I measure x in to get maximum bang for buck. More on this later.

Suppose we have an image represented as a vector x. To make things concrete, imagine it is a 10x10 image with each value in [0, 1] representing the grayscale measure normalized to lie in [0, 1]. This image can be represented as a 100-dim column vector. We want to deduce if it is an image of a dog or a cat.