Scribd
Upload
50 views14 pages

Lecture 2

dddd

Uploaded by

01689373477
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
50 views14 pages

Lecture 2

dddd

Uploaded by

01689373477
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

ECE5602: Neural Networks

(Deep Learning)

Taesup Moon
Lecture 2

M.IN.D Lab ECE 5602: Neural Networks


Outline

• Review
– Probability
(http://web.stanford.edu/class/cs224n/readings/cs229-prob.pdf)
– Linear algebra
(http://web.stanford.edu/class/cs224n/readings/cs229-linalg.pdf)
– Convex optimization
(http://web.stanford.edu/class/cs224n/readings/cs229-cvxopt.pdf)
– Information theory

M.IN.D Lab ECE 5602: Neural Networks 2 / 24


Probability

• Axioms of probability

• Joint probability
– Sum rule

– Product rule

M.IN.D Lab ECE 5602: Neural Networks 3 / 24


Probability

• Conditional probability

• Bayes’ rule

M.IN.D Lab ECE 5602: Neural Networks 4 / 24


Probability

• Random variables
– Discrete

– Continuous

M.IN.D Lab ECE 5602: Neural Networks 5 / 24


Probability

• Independence

• Conditional independence

M.IN.D Lab ECE 5602: Neural Networks 6 / 24


Probability

• Mean & variance

• Covariance

M.IN.D Lab ECE 5602: Neural Networks 7 / 24


Information theory

• Entropy
– Measure of uncertainty
– Lower limit of data compression

M.IN.D Lab ECE 5602: Neural Networks 8 / 24


Information theory

• Relative entropy
– Also known as Kullback-Leibler (KL) divergence
– Often used as a distance between two distributions
à Rigorously, not a metric, though.

M.IN.D Lab ECE 5602: Neural Networks 9 / 24


Linear algebra

• Matrix, vector

• Norms

• Eigenvalues / eigenvectors

M.IN.D Lab ECE 5602: Neural Networks 10 / 24


Linear algebra

• Matrix calculus
– Gradient, Jacobian matrix

– Hessian

M.IN.D Lab ECE 5602: Neural Networks 11 / 24


Convex functions / Optimization

• Convex set

M.IN.D Lab ECE 5602: Neural Networks 12 / 24


Convex functions / Optimization

• Convex function

Jensen’s Inequality

M.IN.D Lab ECE 5602: Neural Networks 13 / 24


Convex functions / Optimization

• Convex optimization

– f(x) is a convex function


– C is a convex set
• Good thing about convex optimization
– All locally optimal points are globally optimal

M.IN.D Lab ECE 5602: Neural Networks 14 / 24

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy