LINEAR ALGEBRA

Machine learning has a strong connection with mathematics. Each machine learning
algorithm is based on the concepts of mathematics & also with the help of mathematics, one
can choose the correct algorithm by considering training time, complexity, number of
features, etc. Linear Algebra is an essential field of mathematics, which defines the study of
vectors, matrices, planes, mapping, and lines required for linear transformation.

Linear algebra plays a vital role and key foundation in machine learning, and it enables ML
algorithms to run on a huge number of datasets.

The concepts of linear algebra are widely used in developing algorithms in machine learning.
Although it is used almost in each concept of Machine learning, specifically, it can perform
the following task:

 Optimization of data.
 Applicable in loss functions, regularisation, covariance matrices, Singular Value
Decomposition (SVD), Matrix Operations, and support vector machine classification.
 Implementation of Linear Regression in Machine Learning .

USECASES OF LINEAR ALGEBRA IN MACHINE LEARNING

Linear Algebra is just similar to the flour of bakery in Machine Learning. As the cake is
based on flour similarly, every Machine Learning Model is also based on Linear Algebra.
Further, the cake also needs more ingredients like egg, sugar, cream, soda. Similarly,
Machine Learning also requires more concepts as vector calculus, probability, and
optimization theory. So, we can say that Machine Learning creates a useful model with
the help of the above-mentioned mathematical concepts.

o Better Graphic experience

o Improved Statistics
o Creating better Machine Learning algorithms
o Estimating the forecast of Machine Learning
o Easy to Learn

Below are some linear Algebra concepts which are mostly used in
Machine Learning implementation: Associative Property: It is a
property in Mathematics which states that if a, b and c are mathematical
objects than a + (b + c) = (a + b) + c in which + is a binary operation.
 Commutative Property: It is a property in Mathematics which states that
if a and b are mathematical objects then a + b = b + a in which + is a
binary operation.
  Distributive Property: It is a property in Mathematics which states that
if a, b and c are mathematical objects then a * (b + c)= (a * b) + (a * c) in
which * and + are binary operators.

Below are some linear Algebra concepts which are mostly used in
Machine Learning implementation:

Scalar: It is a physical quantity described using a single element, It has

only magnitude and not direction. Basically, a scalar is just a single
number.
Example: 17 and 256

Vector: It is a geometric object having both magnitude and direction, it is

an ordered number array, and are always in a row or column. A Vector has
just one index, which can refer to a particular value within the Vector.

Here V is a vector in which e1, e2, e3 and e4 are its elements, and V[2] is
e3.
Example:
[2, 3, 4] and [34, 53]

 Operations:

Scalar-Vector Multiplication:

p = [e1, e2, e3]

The product of a scalar with a vector gives the below result-

p * 2 = [2 * e1, 2 * e2, 2 * e3]

when the scalar 2 is multiplied by a vector p then all the elements of the vector p is multiplied
by that scalar. This operation satisfies commutative property.

Example:

x = [1, 2]

x * 4 = [4, 8]
MATRICES:

It is an ordered 2D array of numbers, symbols or expressions arranged in rows and

columns. It has two indices, the first index points to the row, and the second
index points to the column. A Matrix can have multiple numbers of rows and

columns.
Above M is a 2D matrix having e1, e2, e3 and e4 as elements, and M [1][0] is e3.
Example:
2 3 6 and 56 12
458 45 78
34 67

A matrix having its left diagonal elements as 1 and other elements 0 is an Identity
matrix.
Example:
10
0 1 is 2D Identity Matrix.

100
010
0 0 1 is 3D Identity Matrix.

Operations:
Scalar-Vector Multiplication:
p = [e1, e2, e3]
The product of a scalar with a vector gives the below result-

p * 2 = [2 * e1, 2 * e2, 2 * e3]

when the scalar 2 is multiplied by a vector p then all the elements of the vector p is
multiplied by that scalar. This operation satisfies commutative property.
Example:
x = [1, 2]
x * 4 = [4, 8]

Matrix: It is an ordered 2D array of numbers, symbols or expressions arranged in

rows and columns. It has two indices, the first index points to the row, and the
second index points to the column. A Matrix can have multiple numbers of rows and
columns. Above M is a 2D matrix
having e1, e2, e3 and e4 as elements, and M [1][0] is e3.
Let x1, x2 and x3 be matrices such that x1 has a number of rows and b number of
columns and x2 has b number of rows and c number of columns and x3 is the
product of x1 and x2 so, x3 will have a number of rows and c number of
columns. x1 * x2 = x3 in which x3 has a rows and c columns.
Transpose: The transpose of a matrix generates a new matrix in which the rows
become columns and columns become rows of the original matrix.
m=ab
cd
Transpose(m) = a c
bd
If A is a matrix and B is the transpose of matrix A then, transpose of matrix B is
the original matrix A. B = Transpose(A) then, Transpose(B)=Transpose
(Transpose(A)) = A
Example:
x=123
Transpose of matrix x is 1
2
3
Transpose of an m*n matrix will give a n*m matrix.
Inverse: The inverse of a matrix is the matrix when multiplied with the original
matrix gives the Identity matrix as the product. If m is a matrix and n is the
inverse matrix of m, then m*n = I, in which I represent Identity matrix.
Example:
m = 4 7 and inverse(m) = 0.6 -0.7
26 -0.2 0.4
4 7 * 0.6 -0.7 = 1 0
2 6 -0.2 0.4 0 1
al. The addition of matrix m1 and m2 gives the below result-
m1 + m2 = (a + p) (b + q)
(c + r) (d + s)
Each element of the first matrix is added with the respective element of
another matrix both having same row and column value. Matrix-Matrix
addition is associative, distributive and commutative.
Example:
12 55 67
21+55=76
12 55 67
132 3 10
4 0 1 * 1 = 14
2
 Let x1, x2 and x3 be matrices such that x1 has a number of rows and b number
of columns and x2 has b number of rows and c number of columns and x3 is
the product of x1 and x2 so, x3 will have a number of rows and c number of
columns. x1 * x2 = x3 in which x3 has a rows and c columns.
Transpose: The transpose of a matrix generates a new matrix in which the
rows become columns and columns become rows of the original matrix.
m=ab
cd
Transpose(m) = a c
bd
 If A is a matrix and B is the transpose of matrix A then, transpose of
matrix B is the original matrix A. B =
Transpose(A) then, Transpose(B)=Transpose (Transpose(A)) = A
Example:
x=123
Transpose of matrix x is 1
2
3
 Transpose of an m*n matrix will give a n*m matrix.
 Inverse: The inverse of a matrix is the matrix when multiplied with the
original matrix gives the Identity matrix as the product. If m is a matrix
and n is the inverse matrix of m, then m*n = I, in which I represent Identity
matrix.
5
1 3 2 3 10
4 0 1 * 1 = 14
2
 Let x1, x2 and x3 be matrices such that x1 has a number of rows and b number
of columns and x2 has b number of rows and c number of columns and x3 is
the product of x1 and x2 so, x3 will have a number of rows and c number of
columns. x1 * x2 = x3 in which x3 has a rows and c columns.
 Transpose: The transpose of a matrix generates a new matrix in which the
rows become columns and columns become rows of the original matrix.
m=ab
cd
Transpose(m) = a c
bd
  If A is a matrix and B is the transpose of matrix A then, transpose of
matrix B is the original matrix A. B =
Transpose(A) then, Transpose(B)=Transpose (Transpose(A)) = A
Example:
x=123
Transpose of matrix x is 1
2
3
 Transpose of an m*n matrix will give a n*m matrix.
 Inverse: The inverse of a matrix is the matrix when multiplied with the
original matrix gives the Identity matrix as the product. If m is a matrix
and n is the inverse matrix of m, then m*n = I, in which I represent Identity
matrix.
Example:
m = 4 7 and inverse(m) = 0.6 -0.7
26 - 0.2 0.4
4 7 * 0.6 -0.7 = 1 0
2 6 -0.2 0.4 0 1
1 3 2 3 10
4 0 1 * 1 = 14
2
 Let x1, x2 and x3 be matrices such that x1 has a number of rows and b number
of columns and x2 has b number of rows and c number of columns and x3 is
the product of x1 and x2 so, x3 will have a number of rows and c number of
columns. x1 * x2 = x3 in which x3 has a rows and c columns.
 Transpose: The transpose of a matrix generates a new matrix in which the
rows become columns and columns become rows of the original matrix.
m=ab
cd
Transpose(m) = a c
bd
 If A is a matrix and B is the transpose of matrix A then, transpose of
matrix B is the original matrix A. B =
Transpose(A) then, Transpose(B)=Transpose (Transpose(A)) = A
Example:
x=123
Transpose of matrix x is 1
2
3
 Transpose of an m*n matrix will give a n*m matrix.
 Inverse: The inverse of a matrix is the matrix when multiplied with the
original matrix gives the Identity matrix as the product. If m is a matrix
and n is the inverse matrix of m, then m*n = I, in which I represent Identity
matrix.
Example:
m = 4 7 and inverse(m) = 0.6 -0.7
26 -0.2 0.4
4 7 * 0.6 -0.7 = 1 0
2 6 -0.2 0.4 0 1

MATRIX MULTIPLICATION PROPERTIES:

1. Associative Property:
The associative property of matrix multiplication states that the order in which you multiply
matrices doesn't affect the result. In mathematical terms:
(A * B) * C = A * (B * C)
Here's a more detailed explanation:
 Suppose you have three matrices, A, B, and C, and you want to multiply them
together in two different orders: (A * B) * C and A * (B * C).
 No matter which orders you choose, the result will be the same.
Numerical Example:
Let's consider matrices A, B, and C:
A=|12| B=|34| C=|56|
|78| | 9 10 | |11 12 |

We'll compute both (A * B) * C and A * (B * C):

1. (A * B) * C:
2. (A * B) * C = | 1*3+2*9 1*4+2*10 | * | 5 6 |
| 7*3+8*9 7*4+8*10 | |11 12 |
= | 21+18 4+20 | * | 5 6 |
| 63+72 28+80 | |11 12 |
= | 39 24 | * | 5 6 |
| 135 108 | |11 12 |
3. A * (B * C):
A * (B * C) = | 1 2 | * | 3*5+4*11 3*6+4*12 |
| 7 8 | | 7*5+8*11 7*6+8*12 |
= | 1 2 | * | 73 78 |
| 7 8 | | 107 118 |
 = | 1*73+2*107 1*78+2*118 |
| 7*73+8*107 7*78+8*118 |
= | 253 314 |
| 581 766 |

As you can see, both (A * B) * C and A * (B * C) yield the same result:

(A * B) * C = | 39 24 | * | 5 6 | = | 441 516 |
| 135 108 | |11 12 | |1221 1440 |

A * (B * C) = | 253 314 |
| 581 766 |

2. Distributive Property:
The distributive property of matrix multiplication states that you can distribute the
multiplication of a matrix across a sum (or difference) of matrices. In mathematical
terms:
A * (B + C) = (A * B) + (A * C)
Here's a more detailed explanation:
 Suppose you have three matrices, A, B, and C, and you want to multiply matrix A by
the sum of matrices B and C.
 You can achieve the same result by separately multiplying A by B and A by C and
then adding the results together.
Numerical Example:
Using matrices, A, B, and C from the previous example:

A=|12| B=|34| C=|56|

|78| | 9 10 | |11 12 |

We'll compute both A * (B + C) and (A * B) + (A * C):

1. A * (B + C):

A * (B + C) = | 1 2 | * (| 3 4 | + | 5 6 |)
| 7 8 | | 9 10 | |11 12 |
= | 1 2 | * | 3+5 4+6 |
| 7 8 | | 9+11 10+12 |
= | 1 2 | * | 8 10 |
| 7 8 | |20 22 |

2. (A * B) + (A * C):
(A * B) + (A * C) = | 1*3+2*9 1*4+2*10 | + | 1*5+2*11 1*6+2*12 |
| 7*3+8*9 7*4+8*10 | | 7*5+8*11 7*6+8*12 |
= | 21+18 4+20 | + | 5+22 6+24 |
| 63+72 28+80 | | 35+88 42+96 |
= | 39 24 | + | 27 30 |
 | 135 108 | |123 138 |

Now, let's add the results of A * (B + C) and (A * B) + (A * C):

A * (B + C) = | 1 2 | * | 8 10 | = | 8 12 |
| 7 8 | |20 22 | | 36 50 |

(A * B) + (A * C) = | 39 24 | + | 27 30 | = | 66 54 |
| 135 108 | |123 138 | |258 246 |

As you can see, both A * (B + C) and (A * B) + (A * C) yield the same result:

A * (B + C) = | 8 12 |
| 36 50 |

(A * B) + (A * C) = | 66 54 |
| 258 246 |

INVERSE OF MATRIX:
Inverse of a Matrix is a matrix that on multiplying with the original matrix result in
the identity matrix. It is required to solve complex problems using matrix operations.
For any matrix A its inverse is denoted as A -1.

A.A-1 = A-1A = I
Where I is the identity matrix.

Matrix Inverse Example:

Let's work with a 2x2 matrix B:

B=|21|

|34|

To find the inverse of B, we can use the formula for a 2x2 matrix:

B^ (-1) = (1 / det(B)) * | d -b |

| -c a |

Where:

- a, b, c, and d are the elements of the original matrix B.

- det(B) is the determinant of B, calculated as det(B) = ad – bc

In our case:

-a=2

-b=1

-c=3

-d=4

Calculate the determinant:

det(B) = (2 * 4) - (1 * 3) = 8 - 3 = 5

Now, plug these values into the formula:

B^ (-1) = (1 / 5) * | 4 -1 |

| -3 2 |

Now, perform the multiplication by (1/5):

B^ (-1) = | 4/5 -1/5 |

 | -3/5 2/5 |

So, B^ (-1) is the inverse of B:

B^ (-1) = | 4/5 -1/5 |

| -3/5 2/5 |

This is how you find the inverse of a 2x2 matrix. For larger matrices, you may need to use

specialized software or programming libraries to calculate the inverse, as manual

calculation becomes more complex. In machine learning, matrix inverses are often used in

operations like solving systems of linear equations or in certain optimization algorithms.

TRANSPOSE OF A MATRIX

1. Matrix Transpose:

- The transpose of a matrix A, denoted as A^T or A', is obtained by switching its rows

with columns.

- If A is an m × n matrix, then A^T is an n × m matrix.

- The (i, j)-th element of A^T is the same as the (j, i)-th element of A: (A^T)_ij = A_ji.

Transpose is used in various machine learning operations, such as when you need to

convert feature vectors into column vectors or when dealing with covariance matrices.
Matrix Transpose Example:

Suppose we have the following matrix A:

A=|123|

|456|

To find the transpose of A, we simply switch its rows with columns:

A^T = | 1 4 |

|25|

|36|

So, A^T is the transpose of A, and it's a 3x2 matrix.