Introduction to Linear

Algebra (Part 1)

Jing Li, Assistant Professor

Department of Computing
The Hong Kong Polytechnic University

1
 Week Topic Instructor
1 Data Analytics: An Introduction Jing/Lotto
2 Introduction to Linear Algebra (Part 1) Jing/Lotto
3 Introduction to Linear Algebra (Part 2) Jing/Lotto
4 Introduction to Calculus (Part 1) Jing/Lotto
(+ Quiz 1)
5 Introduction to Calculus (Part 2) Jing/Lotto

Teaching 6
7
In-class Midterm Test
Programming with R (Part 1)
Jing/Lotto
Jibin
Plan 8 Programming with R (Part 2) Jibin
9 Data Visualization Jibin
10 Monto-Carlo Simulation Jibin
11 Linear Regression Jibin
(Assignment out)
12 Time-series Analysis Jibin
(+ Quiz 2)
13 Review and Exam Q&A Jibin & Jing/Lotto
(Assignment due)
Course Structure
Simulation Time-Series
Advanced Data Analytics (3 lectures) Analysis
Regression

Mathematical Basics R programming

(4 lectures) (3 lectures)

Linear Algebra Environment

Data Manipulation
Calculus
Data Analytics

3
The roles of
Linear • Understanding Multidimensional Data
Algebra in • The language for describing and manipulating
multidimensional data structures.
Data • Key Concepts
Analytics • Vectors: Data points, features, or even functions
can be represented as vectors.
• Matrices: Datasets are often represented as
matrices, with rows corresponding to individual
records and columns to features.
4
More Examples about data representations

Computer Vision

Product Recommendation

5
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
6
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
7
What is a Vector?
A vector is an ordered list of numbers, such as
−1
Default:
(−1 0 3.6 7.2) or 0 column Elements or entries,
3.6 vectors e.g., the 3rd entry is 3.6
7.2
Seen as a directed line segment in 𝑛-dimensions.
Count of entries: dimension.
Algebra in a
Vector above has dimension 4
y higher dimension.
Vectors of dimension 𝑛: 𝑛-vector.
Numbers are called scalars. v

Denoted as symbols, such as 𝑎, 𝑏, 𝑐, …, or x

formally as 𝑎,
Ԧ 𝑏, 𝑐,
Ԧ… 8
Example: Word Count Vector
A short sentence.

Word count vectors are used in computer based

document analysis.

word 1
in 1 Word
number 0 Count
Dictionary
house 0 Vector
the 0
document 1

9
Vector’s Norm (Length)
The Euclidean norm (or norm) of an 𝑛-vector 𝑥 is:

𝑥 = 𝑥12 + 𝑥22 + ⋯ + 𝑥𝑛2 , or simplified as 𝑥

Used to measure the length (or magnitude) of a vector.

y The straight-line distance from the origin

v to the point represented by the vector.

10
Vectors’ Distance
The Euclidean distance (or distance) of two 𝑛-vectors 𝑥 and 𝑦 is:
𝑥−𝑦 = 𝑥1 − 𝑦1 2 + 𝑥2 − 𝑦2 2 + ⋯ + 𝑥𝑛 − 𝑦𝑛 2

Length of the subtraction of the two vectors.

The straight-line distance from

one point to the other point
represented by their vectors.

11
Example: Document Distance

5 Wikipedia articles:
Veterans Day, Memorial Day, Academy Awards, Golden
Globe Awards, Super Bowl
Word count vectors with 4,423 words in dictionary.
Pairwise
Distance

12
 What is a Matrix?

• A matrix is a rectangular array of numbers.

• They are typically represented by a capitalized symbol
with a double-sided arrow above, e.g., 𝐵 ി, or simply 𝐵.
• Its size is given by the (row dimension)×(column 0 1 −2.3 0.1
dimension), say 3 × 4 for the above example. 1.3 4 −0.1 0
• Elements are also called entries. 4.1 −1 0 1.7
• 𝐵𝑖,𝑗 is the entry at the 𝑖-th row and 𝑗-th column.
• Two matrices are the equal (=) if they have the same
size and all corresponding entries are equal.
13
 Two sentences in a dataset

Word count vectors are used in computer Example: Word

based document analysis.
Count Matrix
Each entry of the word count vector is the
number of times the associated dictionary
word appears in the document. • We are examining are 𝑛 sentences.
• The dictionary size is 𝑚 words.
Sen 1 Sen 2 • How can you represent the count
word of each word in different sentences?
1 2
in • We define a 𝑚 × 𝑛 matrix 𝐴
Dictionary 1 1
number
0 1 • 𝐴𝑖,𝑗 denotes the count of the 𝑖-
house 0 0 th word in dictionary occurring
the 0 4 in the 𝑗-th sentence.
document 1 1
14
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
15
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
16
Vector and Matrix Addition
0 1 1 To add corresponding
7 + 2 = 9 entries to get the sum
3 0 3

17
Example: Word Count Vector Addition
Two sentences in a dataset

Word count vectors are used in computer based document analysis.

Each entry of the word count vector is the number of times the
associated dictionary word appears in the document.

word 1 2 3
Dictionary in 1 1 2 Word
number 0 1 1 Count
+ = Vector
house 0 0 0
the 0 4 4 Addition
document 1 1 2 18
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
19
 To stretch or shrink the vector's length
Scalar times vector without changing its direction, like pulling or
compressing a spring along its line of action.

20
 Product of 2 Vectors
Three ways to multiply

• Element-by-element
• Inner product
• Outer product

21
Element-by-element • Element-wise multiplication
product (Hadamard
• Sometimes written as ⊙
product)

22
 Multiplication: A single number that represents
the magnitude of one vector
Dot product (inner product) projected onto the other.

𝑥Ԧ 𝑇 𝑦Ԧ or

𝑦:
Ԧ column vector

𝑥:
Ԧ row vector (transpose,
T, of column vector)

1XN NX1

‘inner matrix dimensions must agree’

23
Example: Dot Product
• Pick out the i-th entry: 𝑒𝑖𝑇 𝑎 = 𝑎𝑖
• Sum of entries: 1𝑇 𝑎 = 𝑎1 + 𝑎2 + ⋯ + 𝑎𝑛
• Sum of squares of entries:
• 𝑎𝑇 𝑎 = 𝑎12 + 𝑎22 + ⋯ + 𝑎𝑛2
• More examples.
• 𝑤 is a weight vector, 𝑓 is feature vector,
𝑤 𝑇 𝑓 is weighted score.
• 𝑝 is vector of prices, 𝑞 is vector of
quantities, 𝑝𝑇 𝑞 is total cost.
24
Dot product geometric intuition:
“Overlap” of 2 vectors

Measuring how much

one vector goes in the
direction of the other.

25
The angle of two vectors
• Angle 𝜃 between two non-zero vectors 𝑎 and 𝑏
𝑎𝑇 𝑏
• 𝑐𝑜𝑠𝜃 = where 0≤𝜃≤𝜋
𝑎 ⋅| 𝑏 |

• Several cases of 𝜃:
𝜋
• 𝜃 = = 90°: 𝑎 and 𝑏 are orthogonal, i.e., 𝑎 ⊥ 𝑏
2
• 𝜃 = 0: 𝑎 and 𝑏 are aligned. Here 𝑎𝑇 𝑏 = 𝑎 ⋅ | 𝑏 |
• 𝜃 = 𝜋 = 180°: 𝑎 and 𝑏 are anti-aligned. 𝑎𝑇 𝑏 = − 𝑎 ⋅ | 𝑏 |
𝜋
• 𝜃 ∈ (0, ): 𝑎 and 𝑏 make an acute angle. 𝑎𝑇 𝑏 > 0.
2
𝜋
• 𝜃 ∈ ( , 𝜋): 𝑎 and 𝑏 make an obtuse angle. 𝑎𝑇 𝑏 < 0.
2

26
Example: Document Dissimilarity

5 Wikipedia articles:
Veterans Day, Memorial Day, Academy Awards, Golden Globe Awards, Super Bowl
Word count vectors with 4,423 words in dictionary.

Pairwise Angles in
Degrees

27
Example: Dot product and cosine similarity

• The dot product between two vectors is a scalar:

• The dot product tends to be high when the two
vectors have large values in the same dimensions
• E.g., two word count vectors sharing many
words in common.
• Dot product can be a similarity metric between
vectors, e.g., to quantify the similarity of two
documents!

28
 Example: A Bias Problem
with raw dot-product
• Dot product favors long vectors
• Dot product is higher if a vector is longer (has higher values in many
dimension)
• Vector length:

• For example, a document with frequent words (of, the, you) have
long vectors (since these words tend to occur many times)…
• So dot product overly favors document sizes…
29
 Dot Product

We encode the punishment to

Alternative: document sizes via
cosine for normalizing the dot product
with the vector length!
computing
word Cauchy-Schwarz inequality

similarity

30
 Input neurons’
Example: Firing rates

linear feed- r1

forward
network r2
Output neuron’s
firing rate
ri

rn

31
Multiplication: Outer product

𝑥Ԧ 𝑦Ԧ 𝑇 or 𝑥Ԧ ⊗ 𝑦Ԧ

NX1 1XM NXM

• Each element is the product of a pair of elements from the two vectors
• The results measure all possible products between the components of the two vectors.
32
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
33
Matrix times a vector

MX1 MXN NX1

34
Matrix times a vector:
• Rule: the ith element of y is the dot
inner product product of the ith row of W with x
interpretation

35
Matrix times a vector:
outer product interpretation

( 𝑊 (1) 𝑊 (2) 𝑊 (𝑁) )

• The product is a weighted sum of the columns

of W, weighted by the entries of x
36
Example of the outer product method

(3,5)
• Note: different
(0,4)
combinations of the
(0,2) columns of M can give you
(3,1) any vector in the plane
(we say the columns of M
“span” the plane)
37
 Rank of a Matrix
Are there special matrices whose columns don’t span the full plane?

(1,2)

(-2, -4)

• You can only get vectors along the (1,2) direction (i.e.,
outputs live in 1 dimension, so we call the matrix rank 1)

38
Example: 2-layer linear network

Wij is the connection strength (weight)

onto neuron yi from neuron xj.

39
Example: 2-layer linear network:
inner product point of view

What is the response of cell yi of the second layer?

• The response is the dot

product of the ith row of
W with the vector x

40
Example: 2-layer linear network:
outer product point of view
•How does cell xj contribute to the pattern of firing of layer 2?

1st column
of W

Contribution
of xj to
network output

41
Product of 2 Matrices

NXP PXM NXM

• inner matrix dimensions must agree

• Note: Matrix multiplication doesn’t (generally) commute, AB  BA
42
Matrix times Matrix:
by inner products

• Cij is the inner product of the ith

row of A with the jth column of B
43
Matrix times Matrix:
by outer products

C is a sum of outer products of the columns of A with the rows of B

44
Roadmap
• Concepts of the Vector and Matrix
• Vector and Matrix arithmetic
• Addition
• Vector Product
• Matrix Product
• Some special matrices
45
Special matrices: diagonal matrix
This acts like scalar multiplication

46
Special matrices: • for all
identity matrix

47
Special matrices:
inverse matrix
Does the inverse always exist?
We’ll explore in the next class.

48
 • A vector is a collection of numbers arranged in
order, representing a point in space with direction
and length, while a matrix is a rectangular array
of numbers arranged in rows and columns, used
to represent and perform operations on linear
transformations and data sets.
• The inner product of vectors is a scalar that
A slide to quantifies the extent to which one vector aligns
with another, whereas the outer product
take away produces a matrix representing all pairwise
multiplications of the elements from two vectors.
• The matrix product combines the inner product
through row-by-column multiplications to
determine each individual element, and can be
viewed as a series of outer products that are
summed to form the final matrix.

49