DEV Community: Terra

Mathematics for Machine Learning - Day 18

Terra — Fri, 26 Jul 2024 15:40:44 +0000

Honestly, this is because I forgot that the book shouldn't need a bachelors in mathematics to read :(

A more mathematical proof

So, remember the long breakdown I did before? So, turns out the author has a more fancy way (and probably the correct way) of finding out why there's an inverse in our formula before hand.

We are going to look at

T^{-1}A_\Phi = A_\Phi S \in \reals^{m \times n}

With:

\text{1. }S \in \reals^{n \times n} \text{ being the transformation of idv } \\ \text{with respect to }\tilde{B} \text{ onto coordinates with respect to } B \\

and

\text{2. }T \in \reals^{m \times m} \text{ being the transformation of idv } \\ \text{with respect to }\tilde{C} \text{ onto coordinates with respect to } C

What does this mean?

We'll get to that, before that though, remember this?

(b_1, \dots, b_n), \tilde{B} = (\tilde{b}_1, \dots, \tilde{b}_n) \to V \\ C = (c_1, \dots, c_n), \tilde{C} = (\tilde{c}_1, \dots, \tilde{c}_n) \to W

The linear mapping can be notated as:

\tilde{b_j} = s_{1j} b_1, \dots, s_{nj} b_n = \Sigma_{i = 1}^n s_{ij} b_i, j = 1, \dots, n

With the beggining of the quation bj being the vectors of the new basis B (with a weird eyebrow) of V.

And,

\tilde{c_k} = t_{1k} c_1, \dots, s_{mk} c_m = \Sigma_{l = 1}^m t_{lk} c_l, k = 1, \dots, m

With the beggining of the quation ck being the vectors of the new basis C (with a weird eyebrow) of W.

Finally

((s_{ij})) \in \reals^{n \times n} \text{ and } T = ((t_{kl})) \in \reals^{m \times m}

being the transformation matrix with respect to B (eyebrow) onto coordinates with respect to B and being the transformation matrix with respect to C (eyebrow) onto coordinates with respect to C respectively

Now, let's find out about the first equation

\Phi \tilde{B_j} = \tilde{C_j}

Note:

The first half of the equation above is what's said in the book, but if it's a transformation of B after a basis change, then it's C

First method:

Apply Mapping

\text{Applying the mapping } (\Phi) (j = 1, \ldots, n)

Which results in

\Phi(\tilde{b_j}) = \sum_{k=1}^{m} \tilde{a_{kj}} \tilde{c_k} = \sum_{k=1}^{m} \tilde{a_{kj}} \sum_{l=1}^{m} t_{lk} c_l = \sum_{l=1}^{m} \left( \sum_{k=1}^{m} t_{lk} \tilde{a_{kj}} \right) c_l

where we first expressed the new basis vectors

(\tilde{c_k} \in W) \text{ as linear combinations of the basis vectors } (c_l \in W)

and then swapped the order of summation.

Second method

We express it as:

(\tilde{b_j} \in V) \text{ is a linear combinations of the basis vectors } (b_j \in V)

Which we'll arrive at

\Phi(\tilde{b_j}) = \Phi \left( \sum_{i=1}^{n} s_{ij} b_i \right) = \sum_{i=1}^{n} s_{ij} \Phi(b_i) = \sum_{i=1}^{n} s_{ij} \sum_{l=1}^{m} a_{li} c_l

\sum_{l=1}^{m} \left( \sum_{i=1}^{n} a_{li} s_{ij} \right) c_l , \quad j = 1, \ldots, n

Finally

where we exploited the linearity of matrix transformation which result in

\sum_{k=1}^{m} t_{lk} \tilde{a_{kj}} = \sum_{i=1}^{n} a_{li} s_{ij},(j = 1, \ldots, n), (l = 1, \ldots, m)

and therefore,

\tilde{A_\Phi} = A_\Phi S \in \reals^{m \times n}

such that

\tilde{A_\Phi} = T^{-1} A_\Phi S

But Terra, does that mean the method in the last blog is wrong?

A simple answer is no, it's still stays true since it's working :D

also because the example stated that the matrix is a homomorphism, making the V and W the same.

What does that mean?

It means it's when bj and ck is the same, making T and S the same. The use of P is just arbitrary since we can just change it to T and S and it's still the same result.

What's clear is that if the matrix isn't a homomorphism, then the formula to use is this one, where it differentiates each of the basis change to ensure if the shape of the matrix is different the formula can still be used.

Acknowledgement

I can't overstate this: I'm truly grateful for this book being open-sourced for everyone. Many people will be able to learn and understand machine learning on a fundamental level. Whether changing careers, demystifying AI, or just learning in general, this book offers immense value even for fledgling composer such as myself. So, Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, thank you for this book.

Source:
Axler, Sheldon. 2015. Linear Algebra Done Right. Springer
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 17

Terra — Thu, 25 Jul 2024 16:22:04 +0000

What an absolute clown. When it's just separate examples, it's working fine, but when combined I'm suddenly confused like a headless chicken. I need to stop finding shortcuts only to just re-learn it again the right way.

Well, another lesson learned. No use dwelling on yesterday's idiocy.

Basis Change.

It's combining a few sections we've discussed before into one. So far what I've known to be related directly are:

Transformation Matrix
Linear mapping

Which combined will become... Transformation mapping. I know, so original.

So what is it?

It's transforming matrices while taking into consideration linear mapping if the basis is changed.

So, with

\Phi : V \to W

Consider 2 ordered basis of V and W.

(b_1, \dots, b_n), \tilde{B} = (\tilde{b}_1, \dots, \tilde{b}_n) \to V \\ C = (c_1, \dots, c_n), \tilde{C} = (\tilde{c}_1, \dots, \tilde{c}_n) \to W

Note:
If you forgot, here's what the above basis are for.

\to \text{ Basis Change } \to \tilde{B} \\ C \to \text{ Basis Change } \to \tilde{C} \\ B \to \text{ Linear Mapping } \to C \\ \tilde{B} \to \text{ Linear Mapping } \to \tilde{C} \\

With both basis change and linear mapping can be inversed. So anything can become anything, you just gotta believe in yourself.

Oh yeah, also

A_{\Phi} \in \reals^{m \times n} \text{ and } \tilde{A_{\Phi}} \in \reals^{m \times n}

are the transformation matrix for the basis change.

So what?

So, consider this transformation matrix:

\left[\begin{array}{cc} 2 & 1 \\ 1 & 2 \\ \end{array}\right]

with respect to the standard basis, if we define a new basis.

\left[\begin{array}{c} 1 \\ 1 \\ \end{array}\right], \left[\begin{array}{c} 1 \\ -1 \\ \end{array}\right] )

we obtain the diagonal transformation matrix

\tilde{A} = \left[\begin{array}{cc} 3 & 0 \\ 0 & 1 \\ \end{array}\right]

which is easier to work with than A.

Huh... How?

I hope that's what you're thinking right now, because I spent well over an hour knowing how the hell did this happen.

If you already know how... Please review to see if it's really correct or not :D I'm new in this.

Step 1: Transform the new basis into a matrix

\left[\begin{array}{c} 1 \\ 1 \\ \end{array}\right], \left[\begin{array}{c} 1 \\ -1 \\ \end{array}\right] ) = \left[\begin{array}{cc} 1 & 1 \\ 1 & -1 \\ \end{array}\right]

Step 2: Find Inverse B

B^{-1} = \frac{1}{\Vert{B}\Vert} adj(A) \\ B^{-1} = \frac{1}{-2} \left[\begin{array}{cc} -1 & -1 \\ -1 & 1 \\ \end{array}\right] \\ B^{-1} = \left[\begin{array}{cc} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} \\ \end{array}\right]

Step 3: Calculate (Inverse B) AB

B^{-1}AB = \left[\begin{array}{cc} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} \\ \end{array}\right] \left[\begin{array}{cc} 2 & 1 \\ 1 & 2 \\ \end{array}\right] \left[\begin{array}{cc} 1 & 1 \\ 1 & -1 \\ \end{array}\right] \\

Calculate AB

B^{-1}AB = \left[\begin{array}{cc} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} \\ \end{array}\right] \left[\begin{array}{cc} 3 & 3 \\ 3 & -1 \\ \end{array}\right]

Calculate Inverse B to AB

B^{-1}AB = \left[\begin{array}{cc} 3 & 0 \\ 0 & 1 \\ \end{array}\right]

For some of you, this is enough.

You might already understand the steps and what does each step mean, to which I say, congratulations the other texts below aren't need for you so feel free to leave.

But honestly, I still don't get it :D.

How did I read through a bunch of sections and only now inverse is used again, what's happening here?. So I asked myself a second time.

Huh... How?

It all stems back to linear mapping, the bastard that's laughing maniacally feeling superior as I use an ineffective method on my journey to learning this.

Consider this:

T_{(v){std}} = P T{(v){B}} \\ T{(v){std}} = \text{Linear transformation on standard basis} \\ P = \text{Change of Basis matrix} \\ T{(v)_{B}} = \text{Linear transformation on new basis} \\

P is what we defined as B beforehand.

That's easy, then just inverse it to get the answer right?

T_{(v){B}} = P^{-1} T{(v)_{std}}

That's right. P is the B that we used on the example.

But what about the linear transformation T, what's that?

T_{(v){std}} = A{std} \left[v_{std}\right] \\ A_{std} = \text{The matrix representation of T with respect to the standard basis} \\ \left[v_{std}\right] = \text{Vectors with respect to the standard basis} \\

And since the vectors any vector inside of T on the standard basis, we need to map the vector to the new basis as well.

How?

\left[v_{std}\right] = P \left[v_{b}\right] \\

Notice it's familiar with the T formula for changing basis right? that's because it's the same.

Add it to the formula

T_{(v){std}} = A{std} P \left[v_{b}\right]

Combine with the new basis linear transformation.

T_{(v){B}} = P^{-1} A{std} P \left[v_{b}\right]

Boom. This gets the formula.

Sidenote:

I absolutely have no clue why some are able to be subscript but some aren't. I think there's something wrong with the notation or maybe there's a parameter that's messing it up. But either way, just incase, here's a clarification.

Astd = A_{std} \\ T(v)std = T_{(v)_{std}}

Acknowledgement

Mathematics for Machine Learning - Day 16

Terra — Wed, 24 Jul 2024 13:25:18 +0000

Another day another question at the bottom.

As always, if there's a question that means I stumbled upon an solution that's different, this time however, it isn't the solution, but the way to get there itself. But since I've read till the end of the section this might be a solution I'll use until proven otherwise.

I'll break down the steps with so much scrutiny that even my mental state will join the the fun... The mental break down type of fun :D. So, let's begin.

Transformation matrix

Transformation matrix is much like the last topic, but if the previous day was regarding vectors this will be with matrices.

So Why do you need to make this if it's just from vectors to matrices?

So I'll explain the method that works in the book's example and I'll explain what's really going on.

Explanation from the book

Consider a vector space with the corresponding ordered bases.

\text{ vector space and an ordered basis } {b_1, \dots, b_n} \\ W \text{ vector space and an ordered basis } {c_1, \dots, c_n}

We're also considering a linear mapping

\Phi : V \to W \text{ For } j \in {1, \dots, n}

Which will result in the following unique representation

\Phi (b_j) = \alpha_{1j}c_1 + \dots + \alpha_{mj}c_m = \Sigma_{i = 1}^m \alpha_{ij}c_i

As I've said, this is the unique representationwith respect to C. Then we call the mxn-matrix whose element are given my

A_{\Phi} (i,j) = \alpha_{ij}

is the transformation matrix with respect to the ordered bases B:V and C:W.

Example

Consider a homomorphism with an ordered bases B of V and C of W.

\Phi : V \to W \\ B = {b_1, \dots, b_n} \\ C = {c_1, \dots, c_n}

With

\Phi (b_1) = C_1 - C_2 + 3C_3 - C_4 \\ \Phi (b_2) = 2C_1 + C_2 + 7C_3 + 2C_4 \\ \Phi (b_3) = 3C_2 + 1C_3 + 4C_4

The transformation matrix with respect to B and C satisfies the following

\Phi (b_k) = \Sigma_{i = 1}^4 \alpha_{ik} c_i \text{ for } k = 1, \dots, 3

and is given as :

A_\Phi = [\alpha_1, \alpha_2, \alpha_3] = \left[\begin{array}{ccc} 1 & 2 & 0 \\ -1 & 1 & 3 \\ 3 & 7 & 1 \\ 1 & 2 & 4 \\ \end{array}\right] \text{ where } j = 1, 2, 3

You get all that?

Honestly, I sure don't. So let's try going in even more depth than the book intended.

So in order to transform the matrix, we need to map the result that we want to the transformation matrix. Which is why we use C instead of B.

\Phi (b_1) = C_1 - C_2 + 3C_3 - C_4 \\ \Phi (b_2) = 2C_1 + C_2 + 7C_3 + 2C_4 \\ \Phi (b_3) = 3C_2 + 1C_3 + 4C_4

If we lie them in a transposed vector type of way, it'll look something like this

[\Phi (b_1), \Phi (b_2), \Phi (b_3)] = \left[\begin{array}{ccc} C_1 & 2C_1 & 0 \\ -C_2 & C_2 & 3C_2 \\ 3C_3 & 7C_3 & 1C_3 \\ 1C_4 & 2C_4 & 4C_4 \\ \end{array}\right]

then just remove the variable and you have the same formula as before.

But Terra, I get how it's made now but how do you use it?

So if the matrix we're trying to achieve is W. Given there's three variables, we can write it like this. Remember, A is the transformation and V is the vector beforehand.

A_{\Phi} V

Then we expand the formula

\left[\begin{array}{ccc} 1 & 2 & 0 \\ -1 & 1 & 3 \\ 3 & 7 & 1 \\ 1 & 2 & 4 \\ \end{array}\right] \left[\begin{array}{c} x_1 \\ x_2 \\ x_3 \\ \end{array}\right]

And that's it, just multiply the vectors before with the transformation you want and you're all set!

A question.

I haven't found the answer and probably never will considering it took me way longer than I imagined. So I'll just leave it here :D

The equation for finding transformation matrix.

\Phi (b_k) = \Sigma_{i = 1}^4 \alpha_{ik} c_i \text{ for } k = 1, \dots, 3

The solution I thought and that haven't been disproven (a.k.a not scientifically proven)

Just... transpose it.

Here's the formula

\Phi (b_1) = C_1 - C_2 + 3C_3 - C_4 \\ \Phi (b_2) = 2C_1 + C_2 + 7C_3 + 2C_4 \\ \Phi (b_3) = 3C_2 + 1C_3 + 4C_4

Split it and remove the constants

\left[\begin{array}{cccc} 1 & -1 & 3 & -1 \\ 2 & 1 & 7 & 2 \\ 0 & 3 & 1 & 4 \end{array}\right]

Notice that with the result before, each row corresponds to a column?

\left[\begin{array}{cccc} 1 & -1 & 3 & -1 \\ 2 & 1 & 7 & 2 \\ 0 & 3 & 1 & 4 \end{array}\right]^T = \left[\begin{array}{ccc} 1 & 2 & 0 \\ -1 & 1 & 3 \\ 3 & 7 & 1 \\ 1 & 2 & 4 \\ \end{array}\right]

So change the row into columns and vice versa!

\left[\begin{array}{ccc} 1 & 2 & 0 \\ -1 & 1 & 3 \\ 3 & 7 & 1 \\ 1 & 2 & 4 \\ \end{array}\right]

Seriously, what's math?? What am I doing wrong? I know the guy from Cambridge who wrote this is much smarter than I am so he must have thought someone will think this is just a transposed version. So why :(

Acknowledgement

Mathematics for Machine Learning - Day 15

Terra — Mon, 22 Jul 2024 17:13:18 +0000

We really do, it's getting more and more confusing yet that's a sign of progress! :D

Matrix Representation of Linear Mappings

Congratulations! Now you can't just write a basis without worrying if the basis is ordered or not. You've reached a stage where it's getting more and more technical. Now:

\mathscr{B} = {b_1, \dots, b_n} \text{ is an unordered basis} \\ \text{ and } \\ \mathscr{B} = (b_1, \dots, b_n) \text{ is an ordered basis}

The topic that's going to be discuss later on will matter if the vector are in order or not.

Coordinates

Consider a vector space and an ordered basis

\mathscr{B} = (b_1, \dots, b_n) \text{ of } V \\ \text{For any } x \in V

We obtain a unique representation (linear combination)

(\alpha_1 b_1, \dots, \alpha_n b_n)

of x with respect to B. Then:

(b_1, \dots, b_n) \text{ are the coordinates of x with respect to B.} \\ \text{And the vector } \left[\begin{array}{c} y_1 \\ \vdots \\ y_2 \end{array}\right] \in \reals^n

is the coordinate vector / representation of x with respect to the ordered basis B.

Coordinate vector?

Yup, a basis effectively defines a coordinate system, much like the cartesian system.

There's a slight difference though, in this coordinate system, a vector:

\in \reals^2

has a representation that tells us how to linearly combine

(e_1 \text{ and } e_2)

to obtain x.

What's the difference?

Yeah, cartesian is also a coordinate system that requires two points to make a two dimensional plain. But there's a twist, any basis of the vector defines a valid coordinate representation.

Example:

ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …2) of \reals^2

This means we can write it as:

x = 2e_1 + 3e_2

However, we don't have to use the standard basis! be creative, for example, we can use:

b_1 = \left[\begin{array}{c} 1 \\ -1 \end{array}\right] \text{ and } b_2 = \left[\begin{array}{c} 1 \\ 1 \end{array}\right] \\ \text{This will obtain the coordinates } \frac{1}{2} \left[\begin{array}{c} -1 \\ 5 \end{array}\right]

Proof

What? how does changing one of the value into a minus affect it? Then let me change it into something that's easier to digest.

\\ -1a + 1b = 3

Then we use addition on it to find b

\\ \therefore b = \frac{5}{2} \\ \text{Adding into any of the two functions well obtain } a = -\frac{1}{2}

Then to get the author's version, we just put the half on the front!

\left[\begin{array}{c} -\frac{1}{2} \\ \frac{5}{2} \end{array}\right] = \frac{1}{2}\left[\begin{array}{c} -1 \\ 5 \end{array}\right]

See! it's the same value, just from a different perspective.

Acknowledgement

Mathematics for Machine Learning - Day 14

Terra — Sun, 21 Jul 2024 16:25:43 +0000

Second Weekly Summary

Another week another summary. For this week, I felt most of it follow one another, from sets, to groups, to abelian groups, and so on. So keeping track of the progress is easy.

Then came ranks and vector mapping. Though these topic in itself isn't the problem, I still don't understand the bigger picture for the relationship with other subjects that has been discussed. So, today it's a review on Rank and Linear Mapping, while starting off briefly regarding the other topic first.

Vector space and the whole shebang.

Set, a list / collection of unique numbers, this can range from numbers, vectors, types of cats, or anything else, so long as they're unique then they're a set
Group, a set with an operation that defined the set to keep some structure in the set.
Vector space, a group that fulfill certain conditions of a group such as closure, neutral element, etc.
Linear Independence, if each column in a vector is a pivot column, then they're linearly independent.
Basis, a minimal subset of a vector where the basis can create the original vector space by linear combination but a basis cannot be created by the same method.

Rank

The amount of linearly dependent column in a matrix, if the matrix are all linearly dependent, then that matrix is a full rank.

I know what I rank is, I know how to identify one, but...

What's the use of it?

Right now, there's a few correlations, such as:

Invertibility, a matrix is invertible if and only if the matrix is a full rank
Linear independence, technically this is the same as the first point, but this alone is also true, that if each column is a pivot column and all of it is unique, then that matrix is a full rank.

Since this is one of the things that will be explain later and the impact isn't really known right now, I can't say much about it ヽ(ヅ)ノ

Linear mapping

Linear mapping is mapping to a vector that will also result in a vector as well.

Mapping itself is fine since it's applying a function to a vector, this function can be a matrix or a more complex formula to the vector.

But my problem and want to try to explain more is:

1. Injective

Ensures that each element in the vector mapped will result in a unique value for the resulting vector as well.

2. Surjective

From what I understand, this just means that it applies to more than one element, so, for example, a mapping of the function squared.

When a vector is squared element wise, the resulting vector with the value 4 can come from 2 and -2, that means it doesn't result in a unique value.

3. Bijective

When injective meets with surjective.

How? so, we take the unique value from injective and we take the more than one element from surjective. So, more than one element are mapped into the same element to the resulting vector.

Butt, the resulting map will return unique values. So this means that surjective may contain duplicates in the result so long as it maps more than one element.

Special cases for linear mapping

I'll be honest, aside from isomorphism, I not keen on learning more since this will be explained later on and it feels like a spoiler more than anything :D

So, I'll only summarize isomorphism. Isomorphism is a case of linear mapping when the dimension of the previous vector is the same as the resulting vector (dim(V) = dim(W)).

What I always find surprising is what the author said "intuitive", which means that technically isomorphic linear mapping is the same vector before and after since they can be transformed from one another without losing any information from the vector.

That's it

There isn't much that I can say explained fully where I can grasp it with other topics, but this doesn't mean there isn't any benefit from it. So I can't wait to start moving forward again tomorrow, hope this week is useful!

Acknowledgement

Mathematics for Machine Learning - Day 13

Terra — Sat, 20 Jul 2024 19:25:45 +0000

Linear Mappings

Let me slap you with a formula first.

Consider

\in \reals \\ \text{A mapping } \Phi : V \to W \text{ preserves the vector space }

\Phi (x+y) = \Phi(x) + \Phi(y) \\ \Phi (\lambda x) = \lambda \Phi (x) \\ \forall x,y \in V \text{ and } \lambda \in \reals

What a mapping? and why did you start with a formula?

Good question. It's because you might realize this from the property of distributivity but with a different symbol also if I'm being jumped by an equation, you're going to get jumped as well.

It's preserving the vector while being able to scale it and don't forget with mathematical notations, Phi can be anything so long as it doesn't violate any rules stated, it can be a scalar, a vector, a matrix, even a set!... Actually it can't be a set, I'd get slapped by a mathematician for trying to map a vector with a set.

What use is mapping?

I don't know :D but much like in Panda's map (It's a python library) a mapping can be a an important tool and I see similarities from this type of mapping.

Also, the mapping itself isn't the main topic, it's:

Linear mapping

A mapping is called linear mapping if:

\forall x,y \in V, \forall \lambda \text{ and } \Psi \in \reals : \\ \Phi(\lambda x + \Psi y) = \lambda \Phi (x) + \Psi \Phi (y)

That means that it's defined as linear mapping when a mapping is done to a vector and will result in a vector. Linear mapping can also be called linear transformation / vector space homomorphism

Consider a mapping

\text{A mapping } \Phi : V \to W \\ \text{Where } V, W \text{ can be arbitrary sets}

Phi will be called differently depending on the conditions

Injective

\forall x,y \in V : \Phi (x) = \Phi (y) \to x=y

Surjective

\Phi (V) = W

This means that every element in W can be reached from V using Phi

Bijective

\Psi \circ \Phi(x) = x

Bijective will fulfill both injective and surjective. This can be undone by mapping the inverse of the previous mapping

\Psi : W \to V \\ text{Where } \Psi = \Phi^{-1}

Special cases of linear mapping

\Phi : V \to W \text{Linear and Bijective} \\ text{2. Endomorphism: } \Phi : V \to V \text{Linear} \\ text{3. Automorphism: } \Psi : V \to V \text{Linear and Bijective}

Example

\Phi : \reals \to \Complex, \Phi(x) = x_1 + i x_2

\Phi ( \left[\begin{array}{c} x_1 \\ x_2 \end{array}\right] + \left[\begin{array}{c} y_1 \\ y_2 \end{array}\right] ) = x_1 + ix_2 + y_1 + iy_2 \\ = \Phi ( \left[\begin{array}{c} x_1 \\ x_2 \end{array}\right]) + \Phi ( \left[\begin{array}{c} y_1 \\ y_2 \end{array}\right] )

On the other hand :

\Phi ( \lambda \left[\begin{array}{c} x_1 \\ x_2 \end{array}\right]) = \lambda x_1 + i \lambda x_2 \\ = \lambda ( x_1 + i x_2 ) = \Phi \lambda ( \left[\begin{array}{c} x_1 \\ x_2 \end{array}\right] )

Finite-dimensional vector

(From theorem 3.59 in Axler, 2015)

V and W is isomorphic if and only if dim(V) = dim(W)

Intuition

This means that V and W are kind of the same thing, since they can be transformed to one another without incurring any loss.

Consider the vector V, W, X

For linear mappings:

\Phi : V \to W \text{ and } \Psi : W \to X

the mapping that is also linear will be

\Phi \circ \Phi V \to X

For isomorphism

If:

\Phi : V \to W \text{ is an isomorphism}

Then:

\Phi^{-1} : W \to V \text{ is an isomorphism too}

For linear mapping (2)

If:

\Phi : V \to W, \Psi : V \to W \text{ are linear}

Then:

\Phi + \Psi \text{ and } \lambda \Phi, \lambda \in \reals \text{ are linear too}

Acknowledgement

Mathematics for Machine Learning - Day 12

Terra — Fri, 19 Jul 2024 15:34:52 +0000

An offer you literally can refuse.

You're not gonna like today, it's going to be really short since I'm only going to cover one section and it's been a long day. Though I'd rather be consistent than anything else, it doesn't negate the fact that this is a poor time management from me. Since I'm not a politician, I won't sugarcoat that this will happen and I can't guarantee this won't happen again but I'll keep moving forward.

Rank

So, you're stuck in low ELO hell, be it in chess, Valorant or Dota 2, we've all been there but how do you get from low ELO (or dare I say deficient rank) to a full rank? But we're going too far, let's start all the way back.

What's a rank?

A rank is the number of linearly independent columns of a matrix

\in \reals^{m\times n}

equals the number of linearly independent rows. The rank of a is denoted by:

\text{Rank of A } A = rk(A)

Is it really that simple?

Of course not.

Properties of rank

1. The column rank equals the row rank

rk(A) = rk(A^T)

2. The columns of

\in \reals^{m\times n} \text{ span a subspace } U \subseteq \reals^m

with the dimensions of U equal the rank of A. Later on, this subspace will be called the image/range.

3. The rows of

\in \reals^{m\times n} \text{ span a subspace } W \subseteq \reals^n

with the dimensions of W equal the rank of A. A basis of W can be found by applying Gaussian elimination to A.

4. For all

\in \reals^{m\times n} \text{ and all } b \in \reals^m

it holds that the linear equation system Ax = b can be solved if and only if the rank of A equals the rank of A|b, where A|b denotes the augmented system.

5. For all

\in \reals^{m\times n} \text{ the subspace of solutions for } Ax = 0

posesses dimensions

n - r k (A)

Later on we'll call ths subspace the kernel or the null space.

6. A matrix

\in \reals^{m\times n}

has full rank if its rank equals the largest possible rank for a matrix of the same dimensions. This means that the rank of a full rank matrix is the lesser of the number of rows and columns. i.e. rk(A) = min(m,n). A matrix is said to be rank deficient if it doesn't have full rank.

Conclusion.

If you're friend is asking for tips to climb the rank? Just tell them to be a matrix with each column being a pivot column.

Because honestly, a full rank rank is just a basis but in matrices and not vectors.

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 11

Terra — Thu, 18 Jul 2024 15:39:43 +0000

So, turns out, the answer I searched was correct... It was on the next page. Well, lesson learned, I'll finish a section from now on if possible instead of using a time measured learning system.

The answer to my previous post (Day 10)

let's say I have these vectors

x_1 = \begin{pmatrix} 1 \\ 2 \\ -3 \\ 4 \end{pmatrix}; x_2 = \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \end{pmatrix}; x_3 = \begin{pmatrix} -1 \\ -2 \\ 1 \\ 1 \end{pmatrix} \\

To know if they're linearly dependent:

\lambda_1 x_1 + \lambda_2 x_2 + \lambda_3 x_3 = 0

Equals:

\lambda_1 \begin{pmatrix} 1 \\ 2 \\ -3 \\ 4 \end{pmatrix} + \lambda_2 \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \end{pmatrix} + \lambda_3 \begin{pmatrix} -1 \\ -2 \\ 1 \\ 1 \end{pmatrix} = 0

Sooo

If we use the RREF it'll be:

\left[\begin{array}{cccc} 1 & 1 & 1 \\ 2 & 1 & -1 \\ -3 & 0 & 1 \\ 4 & 2 & 1 \end{array}\right] \to \dots \to \left[\begin{array}{cccc} 1 & 1 & -1 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{array}\right]

And since every element of the column is a pivot column, there's no trivial solution making every single column a unique column which means :

\lambda_1 = \lambda_2 = \lambda_3 x_3 = 0

and these vectors are linearly independent from one another.

Generating Set and Basis

Finally, from a long time ago I didn't know what a basis was and it's going to be discussed today!

Consider the vector space:

(\nu, +, \bullet)

and a set of vectors

\mathscr{A} = {x_1, \dots, x_k} \subseteq \nu

If every vector set of the vector is a part of the vector space, meaning:

\nu \in V

and can be expressed as a linear combination of A (Yes, that absolute horrendous letter is an A but using \mathscr). Then A is a generating set of V

The set of all linear combination of the vectors A is called the span of A.

\text{If } \mathscr{A} \text{ spans the vector space } V. \\ \text{} V = \text{span} [\mathscr{A}] = \text{span} [x_1, \dots, x_k]

Consider a vector space:

(\nu, +, \bullet) \text{ and } \mathscr{A} \subseteq \nu

If there's no smaller set, then the last generating set is called a minimal\basis.

\tilde{\mathscr{A}} \subseteq \mathscr{A} \subseteq A \subseteq \nu \text{ that spans } V \\ \text{Then } \tilde{\mathscr{A}} \text{ is a minimal or basis}

One more example

Let

(\nu, +, \bullet)

and

\mathscr{B} \subseteq \nu, \mathscr{B} \not = \emptyset

Then,

B is a basis of V
B is a minimal generating set
B is a maximal linearly independent set of vectors in V, i.e. adding any other vectors to this set will make it linearly dependent
Every vector

\in V \text{ is a linearly combination of vector from } \mathscr{B}

and every linear combinations are unique, i.e. with:

\sigma_{i = 1}^k \lambda_i b_i = \sigma_{i = 1}^k \psi_i b_i

with

\lambda_i, \psi_i \in \reals, b_i \in B \\ \text{ it follows that } \lambda_i = \psi_i, i = 1, \dots, k

Thoughts

Honestly, today's section is pretty great to read, since today is focused on answering the previous chapter's question / confusion and finally learn what the hell is a basis, it almost feels like eating a dessert after a buffet, not too difficult but enough since it completes a few gaps in the previous sections.

A little TL;DR:

A generating set is a subset of the vector space that can recreate the vector subspace with linear combination
A generating set can be linearly dependent, since it's not a smaller chunk of the vector space, but can be an entirely different vector all together, but can recreate the subspace
A basis on the other hand is a linearly independent vector subspace, since this is the smallest known generating set that the vector space can create. So removing even a single vector from this set will remove the ability to recreate a portion of the vector space.

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 10

Terra — Wed, 17 Jul 2024 15:53:44 +0000

Linear Independence

The closure property guarantees that we end up with another vector in the same vector space. It's possible to find a set of vector which we can represent every vector space by adding them together and scaling them. This set of vectors is called a basis. (Honestly, I don't see when this paragraph is related to the topic, but I'll add it in just in case I'm too dumb to understand)

Linear combinations

Consider

\text{Vector space } (\nu) \\ \text{Infinite number of vectors } x1,x2,\dots,x_k \in \nu

Then:

\nu = \lambda_1 x_1, \lambda_2 x_2, \dots, \lambda_k x_k = \Sigma_{i=1}^k \lambda_i x_i \in \nu

With:

\text{Linear combination =} \lambda_1, \lambda_2 x_2, \dots, \lambda_k \in \reals \text{ of the vector } x_1, x_2, \dots, x_k

Dependence and Independence

If:

\Sigma_{i=1}^k 0 x_i = 0

Then the result will always be true. This is the same as before, which in the last post is called "Trivial subspace", given the neutral element is required in a vector space.

So, if:

C_1, C_2, \dots, C_k = 0 \\ \nu = C_1 x_1 + C_2 x_2 + \dots + C_k x_k \\ \nu = 0

Then the vector space is linearly independent.

But, if:

\Sigma_{i=1}^k \lambda_i x_i = 0 \\ \text{With any } \lambda_i \not = 0

Then the vector space is linearly dependent.

Notes:

Intuitively, a set of linearly independent vectors consist of no redundant vectors. i.e. If we remove any of the vectors from the set, we will lose something. Like my hope of finishing this book in under a year... Feels bad man :(

Discussion:

Honestly, the note from the author is quite confusing, is it really intuitive considering it's the requirement of a set which leads to a group, which leads to a vector space that a set needs to have distinct elements? So why add the note?

Example

This is where I spent an hour just on an example. I'll add a question in the end for you to see if, again..., I'm an idiot or math is mathing right now.

To obtain the information on the route from Nairobi to Kigali, we need either:

A. 506 km northwest and 374 km southwest
B. 751 km west

Both A and B are linearly independent but if we use both A and B then it becomes linearly dependent, where B is linearly dependent from A given it's a linear combination.

So, from reading this, it's safe to assume that one of the condition for being linearly independent is:

If any change on a single vector in the vector space x1, x2, ..., xn will affect more than one element, then the vector space is linearly dependent.

What are the other properties/conditions?

K vectors can only be either linearly dependent or linearly independent.
If at least one of the vector x1, x2, ..., xk is 0, then they're linearly dependent.
If at least one of the vector x1, x2, ..., xk is a duplicate of another vector, then they're linearly dependent. (Look at the Discussion above, this is the same thing)
The vectors

{x_1, x_2, \dots, x_k : x \not = 0, i = 1, 2, \dots, k} k \geqslant 2

Are:
a. Linearly dependent if (at least) one is a multiple of another vector.
b. Linearly dependent if (at least) one is a linear combination of the other.

Tips

This is actually the fifth point but it feels more like a tip more than the properties

Firstly, transform the vectors into matrices

From a format like this:

\lambda_1 x_1, \lambda_2 x_2, \dots, \lambda_k x_k

Into something like this

\begin{pmatrix} \lambda_1 \lambda_2 \dots \lambda_k \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_5 \end{pmatrix}

Then, ignore the x vector like in an augmented matrix and we have this:

\begin{pmatrix} \lambda_1 \\ \lambda_2 \\ \vdots \\ \lambda_m \end{pmatrix} = \begin{pmatrix} \begin{pmatrix} v_{11} & v_{12} & \cdots & v_{1k} \end{pmatrix} \\ \begin{pmatrix} v_{21} & v_{22} & \cdots & v_{2k} \end{pmatrix} \\ \vdots \\ \begin{pmatrix} v_{m1} & v_{m2} & \cdots & v_{mk} \end{pmatrix} \end{pmatrix} = \\ \left[\begin{array}{cccc} v_{11} & v_{12} & \cdots & v_{1k} \\ v_{21} & v_{22} & \cdots & v_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ v_{m1} & v_{m2} & \cdots & v_{mk} \end{array}\right]

What kind of tip is this Terra?

Good question, honestly, I'm not sure if this is a tip or another explanation.

Notice that this is eerily familiar with the topic regarding particular and general solutions, I hope you enjoyed that part since after transforming the vectors into a matrix, we're going to transform it again into a Reduced-Row Echelon Form (RREF).

Only after in the form of RREF these are the tips I've summarized.

If there exist a column that's empty on the main diagonal, then it's linearly dependent
If there exist a column that's empty on the main diagonal but on the have non-zero values on the other rows, then it's linearly dependent

Meaning:

I'll give you a few matrices and I'll tell you why each of them are considered linearly dependent or independent.

Matrix 1:

\left[\begin{array}{cccc} 4 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{array}\right]

Obviously this is linearly independent. Each non-zero element is on the main diagonal making them unique to each row.

Matrix 2:

\left[\begin{array}{cccc} 4 & 2 & 0 \\ 0 & 0 & 2 \\ 0 & 0 & 3 \end{array}\right]

Okay, so this is a linearly dependent vector, but why? is it because of the 2nd column or the 3rd column? Found your answer yet? it's the second column, given that the second column is 1/2 of the first column, this means it's a multiplication making it redundant.

P.S. I made the question column in numbers because people usually focus on numbers better, making them not see the answer text below if they really focus on answering the question. That's why only this part I made it in numbers :D Hope it worked for you

Matrix 3:

\left[\begin{array}{cccc} 4 & 0 & 0 \\ 0 & 2 & 2 \\ 0 & 0 & 3 \end{array}\right]

What about this? Well, it's linearly independent. Because though the third column contains non-zero value on the second row, there still isn't any linear combination that can make the third column.

Matrix 4:

\left[\begin{array}{cccc} X & O & O \\ X & O & X \\ O & X & X \end{array}\right]

This is a tic-tac-toe and the one playing circle won.

My Question

To the person reading this:

Hi, hope you're doing well :D

Let's say I have two vectors

x_1 = \begin{pmatrix} 1 \\ 5 \end{pmatrix} \\ x_2 = \begin{pmatrix} 5 \\ 1 \end{pmatrix}

Using the closure property:

\forall x,y \in \nu : x+y \in \nu

This means that the addition between the vector x and y from the vector space v is also inside of the vector space.

With linear dependence being:

\Sigma_{i=1}^k \lambda_i x_i = 0 \\ \text{With any } \lambda_i \not = 0

If I have the equation:

\lambda_1 x_1 + \lambda_2 x_2 = x_3 \text{ for } \lambda_1, \lambda_2 \not = 0

If both lambda = 1

1 x_1 + 1 x_2 = x_3 \\ x_1 + x_2 = x_3

Doesn't that mean it fulfills the linear dependence equation because of the closure property?

Then doesn't that mean all vector space are linear dependent?? I'm absolutely confused.

I have found that it's said to determine if it's linearly dependent or not, I need to solve this instead:

\lambda_1 x_1 + \lambda_2 x_2 = 0

I think I'm confused because the example is regarding the redundancy and not much strengthening the concept of vector space.

If you can answer this, please do, I really need the explanation further :( Thank you!

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 9

Terra — Tue, 16 Jul 2024 18:39:38 +0000

Get it? because they're in space, but if they're on earth the vector (in) space disappears and only 0 remains :D

Vector Space

With your knowledge regarding groups we can start to discuss regarding vector spaces! and if you read till the end, the mathematical notations if vector space will be a breeze!

What's a vector space?

With the four conditions from group hold true, we can start to add operations outside of the group. Such as:

\text{The multiplication of a vector } x \in \mathscr{G} \text{ by a scalar } \lambda \in \reals

A real valued vector space:

\nu = (\nu, +, \bullet) \text{ is a set } \nu \text{with two operations}

Those operations are:

\nu \times \nu \to \nu) \\ (\bullet : \reals \times \nu \to \nu)

Side note: The brackets doesn't mean anything, the + sign doesn't work at the beginning of KaTeX

What does that?

Good question, much like my mental health, let's have a breakdown!

Abelian Group

\nu = (\nu, +) \text{ is an Abelian group }

An Abelian group is the same as a normal group but with an additional condition:

\forall x,y \in \mathscr{G} : x \bigotimes y = y \bigotimes x

Then

(\mathscr{G}, \bigotimes )

In the book, there's a lot of examples to ensure we have a good understanding on what Abelian and groups in general are, but I won't go too deep in it here. I feel so long as we understood the conditions for a group and the additional one for Abelian, we're good to go.

For example:

(\natnums_0 , +)

is not a group, because though all natural number additions are still part of the natural number and with the addition of 0 (Which is why there's a subscript 0) means there's a neutral number.

The "group" lacks inverse numbers, in addition and subtraction means it lacks negative numbers.

Distributivity

Yeah, yeah, this is the third time I've mentioned distributivity so I'll just show the formula and continue on.

\text{A. } \forall \lambda \in \reals, x,y \in \nu : \lambda (x+y) = \\ \lambda x + \lambda y

\text{B. } \forall \lambda , \psi \in \reals, x \in \nu : (\lambda + \psi) x = \\ \lambda x + \psi y

Associativity

Ditto (I really hope you're not skipping chapters, there's a lot of same conditions that would be better to start from vectors than here straight away)

\forall \lambda ,\psi \in \reals, x \in \nu : \lambda(x \psi) = (\lambda \psi) x

Neutral elements

... Hi.

\forall x \in \nu : I x = x

A few remarks

\text{The elements } x \in \nu \text{ are called vectors} \\ \reals^n , \reals^{n \times 1}, \reals^{1 \times n} \text{ are only different ways vectors can be written} \\

The only difference for the vectors is that Rn is for vertical matrices so it's the same as Rnx1 but R1xn is for horizontal matrices.

\reals^n , \reals^{n \times 1} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_5 \end{pmatrix} \\ \reals^{1 \times n} = \begin{pmatrix} x_1, x_2, \dots, x_5 \end{pmatrix}

Vector Subspaces.

The author reminded in us the importance of vector subspaces as it will be revisited later on (i.e. Chapter 10 Dimensionality Reduction)

Let:

\nu = (\nu, +, \bullet)

be a vector space, and

\upsilon \subseteq \nu : \nu \not = \empty

Then:

\nu = (\nu, +, \bullet) \text{ is a vector subspace of } \nu \text { or linear subspace}

Subsets

To determine:

\upsilon \subseteq \nu : \nu \not = \empty

The subset needs to inherit many properties from the vector space

So what?

\text{(1.) } \upsilon \not = \empty \\ \text{(2.) } \forall \lambda \in \reals, \forall x \in \upsilon : \lambda \times \in \upsilon \text{ (Outer operations)} \\ \text{(3.) } \forall x,y \in \upsilon : x + y \in \upsilon \text{ (Inner operations)}

Point two and three are the Closure condition from V

Example

For every vector space, the trivial subspace are the vector space itself and {0}
Only example D is a subspace of R (with the usual inner/outer operations). In A and C, the closure property is violated and B doesn't contain 0.
The solution set of a hogenous system of linear equations Ax = 0 with n unknowns x = [x1, x2, ..., xn] transposed is a subspace of Rn
The solution of an inhomogenous system of linear equations Ax = b, b not equaling 0 is not a subspace of Rn.
The intersection of arbitrary many subspaces is a subspace itself.

If you're confused, so am I, let me try to break it down.

All subspace have at least two subspaces. The vector space itself and 0.
A violates the closure condition since we can use additions to find elements outside of the box
B violates the neutral element condition since it doesn't intersect with (0,0)
C violates the closure condition since using conditions we can find elements outside the weird shape as well.
D fulfills all conditions, no violation for associativity, distributivity, neutral element nor inverse element.
Ax = 0 is a homogenous system. This is a subspace

Why? because it fulfills the neutral element again, the value returns to zero and if it's all zero (Just like example D) it doesn't violate any conditions

Ax = b with b not 0 is a in-homogenous system. This isn't a subspace.
Let's say we use example A and C. Assuming both of them are from the same vector space. There are areas where example A overlap with example C and in the areas where those subspaces overlap/intersect (Intersect is for points) will also be a subspace.

Ending Note

Honestly, yesterday's short summary was worth it. It was so much faster understanding this concept after fully understanding groups and sets. So we covered two topics today, vector spaces and their subspaces, horray! :D

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 8

Terra — Mon, 15 Jul 2024 15:40:39 +0000

Dear mathmaticians.

I love you math and mathematicians... but why? Today I spent the most time trying to understand and the least amount page read... And so, it has become the meme of the day.

What happened? It all began with:

Algorithms for Solving System of Linear Equations

Until now, the examples and matrices chosen always has an answer, but if we don't make the assumption that there isn't a solution, we can make an approximation to the solution.

One way is using the approach of linear regression.

A x = B

if A is a square matrix and is invertible (non-zero determinant)

x = A^{-1}B

If A does have a determinant of zero, we are using something called:

Moore-Penrose Pseudo-inverse

Ax = B \\ A^{T}Ax = A^{T}B \\ (A^{T}A)^{-1}A^{T}Ax = (A^{T}A)^{-1}A^{T}B \\ x = (A^{T}A)^{-1}A^{T}B

Though we can solve using these methods. There are some disadvantages when it comes to it:

Requiring many computations for matrix-matrix product
Requiring the important role of Gaussian elimination, what for?

Compute determinant
Compute inverse
Check for linearly independancy
Compute the rank of a matrix
Find basis of a vector space

Well... Technically

In practice there are many linear equations that can be solved indirectly,

Stationary iterative method

Richardson method
Jacobi method
Gauss-Seidel method
Successive over-relaxation method

Krylov subspace method

Conjugative gradients
Generalized minimal residual
Biconjugate gradients

Vector Space

A structured space in which vectors live. I know, very helpful explanation.

But before we can learn about vector space, there's something that needs to be established.

Group

A set of elements and an operation defined on these elements that keep some structure of the set intact.

This is where the fun begins.

\text{A set } \mathscr{G} \text{ and an operation } \bigotimes : \mathscr{G} \times \mathscr{G} \to \mathscr{G} \\ \text{then} \\ G := (\mathscr{G},\bigotimes) \text{ is called a group (with some conditions)}

Conditions

Now, to make sure you understand how confused I was, I'll notate the equations first then continue with the explanation.

\text{1. Closure of } \mathscr{G} \text{ under } \bigotimes : \forall x, y \in \mathscr{G} : \\ x \bigotimes y \in \mathscr{G}

\text{2. Associativity: } \forall x, y, z \in \mathscr{G} : \\ (x \bigotimes y) \bigotimes z = x \bigotimes (y \bigotimes z)

\text{3. Neutral element: } \exist e \in \mathscr{G} \forall x \in \mathscr{G} : \\ x \bigotimes e \And e \bigotimes x = x

\text{4. Inverse element: } \forall x \in \mathscr{G} \exist y \in \mathscr{G} : \\ x \bigotimes y = e \And y \bigotimes x = e \text{ where e is the neutral number}

Where's the numbers?

Good question. I don't know, even the inverse portion doesn't have any -1 on it. Which I have to say... Impressive. Let's start with the very first thing.

G and G

I won't lie, neither KaTeX nor LaTeX have this G and further search leads me to no where. So I assume it's the same thing, which is a group. Here's what the custom-GPT math said

So what's a set?

If you're like me, you basically forgot this even existed aside from inside programming languages. It's very similar.

A set is a collection of unique/distinct objects. For example:

{1,2,3,...} is a set of natural numbers
{2,4,6,...} is a set of even numbers

Now you might think that "Oh, that's easy. Let's move on". Then let me ask you this:

Which one of these is a set?

{0,1,4,9,16,...} \\ {0,1,7,9,16,...}

Yup, it's both. What about this?

{2,3,5,7,11, 13...} \\ {34,1,6,7,3,4,9,10}

Still both. Finally, this:

\\ {\spades, \hearts, \diamonds, \clubs }

I don't have to say it right? The Fibonacci sequence isn't a set.

\bigotimes or ⨂ (This is the KaTeX notation)

\bigotimes or Big'O as the kids say these days is a placeholder for binary operations. Not to be confused with or which also uses ⨂ and I spent so much time thinking it was these.

What do I mean by placeholder?

Let me give you an example

\bigotimes y = 4 \\ \text{if x = 2, what is the value of y?} \\ \text{Answer} = \text{Yes}

Y can be 2, 1/2 and -2. Since Big'O can be an addition, a subtraction, a multiplication. All three answer is correct depending on further information.

Big'O can also be used in matrices or sets with the same concept. Here are the things Big'O can replace:

Addition
Subtraction
Multiplication
Union
Intersect

With that out of the way, let's continue with the conditions!

Closure

\text{1. Closure of } \mathscr{G} \text{ under } \bigotimes : \forall x, y \in \mathscr{G} : \\ x \bigotimes y \in \mathscr{G}

Closure means, with x and y being an element of the set G, the result of the operation is also an element from the set G.

That means, if G is a set of natural numbers, with X ⨂ Y being elements of the set G, we can deduce that:

X and Y > 0
X > Y
If X ⨂ Y = Z, Z is an element from the set G.

Associativity

\text{2. Associativity: } \forall x, y, z \in \mathscr{G} : \\ (x \bigotimes y) \bigotimes z = x \bigotimes (y \bigotimes z)

This is something we've already discussed extensively, so I won't go too long, it just means that the order of the operation doesn't matter.

Neutral element

\text{3. Neutral element: } \exist e \in \mathscr{G} \forall x \in \mathscr{G} : \\ x \bigotimes e \And e \bigotimes x = x

Neutral element is an element that when operated with any other value, the result will said other value.

Here are what counts as neutral elements:

\text{Integers} = 0 \\ \text{Matrices} = \text{Identity Matrix} \\ \text{Sets} = \emptyset

Inverse element

\text{4. Inverse element: } \forall x \in \mathscr{G} \exist y \in \mathscr{G} : \\ x \bigotimes y = e \And y \bigotimes x = e \text{ where e is the neutral number}

This condition goes hand in hand with the last condition, with the value x there exist y that when operated against x will return a neutral number.

Here's what I mean:

\bigotimes y = e \\ \text{if x = 3} \\ \therefore \text{y = -3}

This means that the inverse of x is -3. What's interesting, the author notes that though inverse is often noted with -1 on the top of the element, it doesn't always mean to be divided by said value.

X^{-1} \not = \frac{1}{X}

And that's it for today.

If you think it's pretty short and without any mathematical background you can understand very quickly, you're welcome. I'm kidding, sort of. I took me so long to understand those notations and what it really means to be a set, operation, etc.

Since the author said this part is very important in computer science, I want to make sure I understood as clearly as possible and It'll take a while :D

P.S.

I'm starting to understand more and more regarding mathematical notations but whenever I feel like I can breeze a few pages, math just slaps me in the face and kick me in the nuts. Feels bad man.

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com

Mathematics for Machine Learning - Day 7

Terra — Sun, 14 Jul 2024 17:28:41 +0000

A weekly review

Today I'm just going to summarize and translate some into code regarding the properties and regarding the particular solution so there isn't anything new today nor will there be much mathematical notation, aside from the last section.

A fun fact about today is, remember the first ever equation regarding the particular solution? I used gradient descent and found another different way :D so fun.

Matrices

Matrix comparison function

The reason I'm using this function instead of a built-in function is for you (the readers) to know how the comparison is made (because honestly, I don't know how np.array_equal works).

import numpy as np

def compare_two_matrices(matrixA:np.ndarray, matrixB:np.ndarray)->np.ndarray:
    # Ensuring matrix A and B is the same
    if matrixA.shape != matrixB.shape:
        return "Matrix A and B should have the same shape"

    # Comparing each index of the matrices and returning a list
    result = [i for j in matrixA==matrixB for i in j]

    #Return text if all inside the list is true
    if all(result):
        return "Both matrices are exactly the same"

    #Return text if not all inside the list is true
    return "Matrices are not the same"

Addition and subtraction

m = 5
n = 3

Amn = np.random.randint(low=0,high=100,size=(m,n))
Bmn = np.random.randint(low=0,high=100,size=(m,n))
Cmn = Amn+Bmn
Dmn = Amn-Bmn

print(Amn.shape,Bmn.shape, Cmn.shape, Dmn.shape)
# (5, 3) (5, 3) (5, 3) (5, 3)

Multiplication

m = 5
n = 3
k = 9

Amn = np.random.randint(low=0,high=100,size=(m,n))
Bnk = np.random.randint(low=0,high=100,size=(n,k))
Cmk = np.dot(Amn,Bnk)

print(Amn.shape,Bnk.shape, Cmk.shape)
# (5, 3) (3, 9) (5, 9)

Associativity

m = 9
n = 3
k = 5
l = 7

Amn = np.random.randint(low=0,high=100,size=(m,n))
Bnk = np.random.randint(low=0,high=100,size=(n,k))
Ckl = np.random.randint(low=0,high=100,size=(k,l))

Left_section = np.dot(np.dot(Amn,Bnk), Ckl)
Right_section = np.dot(Amn, np.dot(Bnk,Ckl))

compare_two_matrices(Left_section, Right_section)

# 'Both matrices are exactly the same'

Distributivity

Test 1

m = 9
n = 3
k = 5

Amn = np.random.randint(low=0,high=100,size=(m,n))
Bmn = np.random.randint(low=0,high=100,size=(m,n))
Cnk = np.random.randint(low=0,high=100,size=(n,k))

Left_section = np.dot((Amn+Bmn),Cnk)
Right_section = np.dot(Amn, Cnk) + np.dot(Bmn, Cnk)

compare_two_matrices(Left_section, Right_section)

# 'Both matrices are exactly the same'

Test 2

m = 9
n = 3
k = 5

Amn = np.random.randint(low=0,high=100,size=(m,n))
Bnk = np.random.randint(low=0,high=100,size=(n,k))
Cnk = np.random.randint(low=0,high=100,size=(n,k))

Left_section =  np.dot(Amn, (Bnk + Cnk))
Right_section = np.dot(Amn, Bnk) + np.dot(Amn, Cnk)

compare_two_matrices(Left_section, Right_section)

# 'Both matrices are exactly the same'

Inverse

identity_matrix = np.identity(2)
wrong_matrix = np.array([[4,8],[1,2]])
right_matrix = np.array([[4,8],[0.5,2]])

I hope you remember why the wrong matrix won't work when I try to inverse the matrix while the right matrix works just fine even when it's just a one value difference!

Inverse Function

def create_inverse(matrix:np.ndarray)->np.ndarray:
    # Confirming square matrix
    if matrix.shape[0] != matrix.shape[1]:
        return "Matrix needs to be square to be inversed."

    # Confirming shape
    if matrix.shape != (2,2):
        return "I'm not smart enough to code more complex matrices and don't ask for an inverse of 1x1."

    # Creating the adjoint matrix
    adj_matrix = [[matrix[-1][-1],-matrix[0][-1]],\
            [-matrix[-1][0],matrix[0][0]]]

    # Creating the determinant of the matrix
    det_matrix = np.dot(matrix[0][0],matrix[-1][-1])-np.dot(matrix[0][-1],matrix[-1][0])

    #Calculating the inverse of the matrix
    inverse_matrix = adj_matrix/det_matrix

    return inverse_matrix


create_inverse(wrong_matrix)

# RuntimeWarning: divide by zero encountered in divide (inverse_matrix = adj_matrix/det_matrix)

Damn... my function spoiled the fun. So always remember! not all matrices can be inversed, aside from the rule that it must be a square matrix, it also needs to have a non-zero determinant.

create_inverse(right_matrix)

"""
array([[ 0.5  , -2.   ],
       [-0.125,  1.   ]])
"""

multiply_with_inverse = np.dot(create_inverse(right_matrix), right_matrix)

compare_two_matrices(multiply_with_inverse, identity_matrix)

# 'Both matrices are exactly the same'

This also proves the formula that of a matrix is multiplied by the inverse of said matrix, the result is an identity matrix!

Particular Solution

This is where it gets fun. So let me ask you reader, if you've read the previous days, you know that aside from having a sort of identity matrix inside the matrix, the formula to find the particular solution that I used is more like guessing or iterating values to find the answer.

So what?

That means, when translating it into code, I also need to make it iterative and change the value of x until it matches the value we know as the result.

So what?

That means I need to skip a few chapters. (Gradient Descent)

\theta := \theta - \alpha \left( \frac{2}{n} X^T (X\theta - y) \right)

A name so famous and so badass that I can't help to learn it quicker. This is what we'll use in determining x. Now bear in mind, today is more coding than mathematics, so I won't go to too much detail regarding why do they use theta or why is a nabla there.

Today, I'll just explain that what we did is just gradient descent but in our brain, so this is how I'm translating it.

Mean Squared Error (MSE)

MSE=1n∑i=1n(yi−y^i)2 \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

I'm going to be using Mean Square Error (MSE), because since there'll be some negative values, I need to ensure what's being calculated is the difference in value not accounting if it's negative or positive.

def MSE_function(predicted:np.ndarray, expected:np.ndarray)->np.ndarray: # Mean squared error
    if len(predicted) != len(expected):
        return "Predicted output and expected output should be the same"

    #The amount of values in prediction and expected values
    n = len(predicted)

    # Calculating the difference squared of the expected and predicted value
    total_square = np.array([(predicted[i]-expected[i]) for i in range(n)])

    # The mean squared of the difference
    mse_value = total_square/n

    return mse_value

Gradient Descent

\theta := \theta - \alpha \left( \frac{2}{n} X^T (X\theta - y) \right)

def finding_gradient(input_matrix:np.ndarray, mse_error:np.ndarray)->np.ndarray:

    #Multiplying matrix A (transposed) by the MSE vector
    At_E = np.dot(input_matrix.T, mse_error)

    # Dividing it by m and multiplying it by 2
    gradient = (2/len(mse_error))*At_E

    return gradient

def update_x(current_x:np.ndarray, gradient:np.ndarray, descent:int = 0.1)->np.ndarray:
"""
I'm using assert because I'd rather have an error on this section rather than outputing a string that'll be an error
somewhere else down the line :D
"""
    assert len(current_x)==len(gradient)

    # The amount of value
    n = len(current_x)

    # Multiplying the gradient by the learning rate
    gradient = gradient*descent

    # Iterating against x and subtracting it by alpha*gradient
    update_values = [(current_x[i]-gradient[i]) for i in range(n)]

    return update_values

Yes, it's split into two functions. I try my best to ensure each function play only one specific role (from what I know this is best practice in coding, S in SOLID principle).

So, you can see from my code and the function that there's similarities. Here's what my code mean in the mathematical notation.

X^T \\ gradient = \frac{2}{n} X^T

\text{current x} = X\theta \\ descent = \alpha

\text{updated values} = \theta - \alpha \left( \frac{2}{n} X^T (X\theta - y) \right)

P.S. I can't use underscore (_) inside of katex text, so it should've been At_E, current_x and updated_values which refer to my variables and not just some random name.

P.P.S. I'll never change from snake case so katex can fight me. I'm a python developer, the snake god might hate me if I don't use snake case and if you don't know what I'm talking about... It's been a long day, I'm sorry for rambling.

Full code

And that's it! we can use it to calculate the system equation from the previous days.

A = np.array([[1, 0, 8, 0, -4], [0, 1, 2, 0, -12], [0, 0, 4, 1, 7]])
B = np.array([42, 8, 12])
x = np.array([0, 0, 0, 0, 0])
descent_value = 0.041

for i in range(10000):
    prediction = np.dot(A,x)

    mse_value = MSE_function(prediction, B)

    gradient = finding_gradient(A, mse_value)

    x = update_x(x, gradient, descent_value)

    # Generating final report
    if np.abs(sum(mse_value))<0.0001:
        print("Generation finish after {} iteration".format(i))
        print("A total mean square error value of {} or an average of {}\n".format(round(sum(mse_value),5),np.mean(mse_value)))
        print("With x:",x)
        print("With Ax:",np.dot(A,x))
        break
"""
Generation finish after 1077 iteration
A total mean square error value of 0.0001 or an average of 3.303279802955059e-05

With x: [4.024892733606064, -4.136195147684619, 4.626743232157506, -4.825002085330449, -0.24024375952185326]
With Ax: [41.99981363  8.00021643 12.00026453]
"""

And that's it! You've made a basic model that learns from previous data making it a machine learning model!

Acknowledgement

Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com