Lecture 11 Recap

I2DL: Dai 1
 Transfer Learning
Distribution Distribution

P1 P2
Large dataset Small dataset
Use what has been
learned for another
setting
I2DL: Prof. Dai 2
 Transfer Learning
TRAIN
Trained on ImageNet New dataset with C
classes

FROZEN

Source : http://cs231n.stanford.edu/slides/2016/winter1516_lecture11.pdf
[Donahue et al., ICML’14] DeCAF,
[Razavian et al., CVPRW’14] CNN Features off-the-shelf
I2DL: Prof. Dai 3
 Representation Learning

[Chen et al., ICML’20] SimCLR,

I2DL: Prof. Dai https://amitness.com/2020/03/illustrated-simclr/ 4
 Basic Structure of RNN
• We want to have notion of “time” or “sequence”

𝑨𝑡 = 𝜽𝑐 𝑨𝑡−1 + 𝜽𝑥 𝒙𝑡

Hidden state
Previous Input
hidden state

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

I2DL: Prof. Dai 5

 Long-Term Dependencies

I moved to Germany … so I speak German fluently.

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

I2DL: Prof. Dai 6

Long-Short Term Memory Units(LSTM)

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

I2DL: Prof. Dai 7

 Long-Short Term Memory Units
• Key ingredients
• Cell = transports the information through the unit

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

I2DL: Prof. Dai 8

 LSTM
• Highway for the gradient to flow

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

I2DL: Prof. Dai 9

 Attention
Intuition: Take the query Q, find the most similar
key K, and then find the value V that
corresponds to the key.

In other words, learn V, K, Q where:

V – here is a bunch of interesting things.
K – here is how we can index some things.
Q – I would like to know this interesting thing.

Loosely connected to Neural Turing Machines

(Graves et al.).

I2DL: Prof. Dai 10

 Attention
Index the values Multiply queries
via a differentiable with keys
operator.
Get the values

𝑄𝐾 𝑇
Attention 𝑄, 𝐾, 𝑉 = softmax 𝑉
𝑑𝑘

To train them well, divide by 𝑑𝑘 , “probably” because for

large values of the key’s dimension, the dot product grows
large in magnitude, pushing the softmax function into regions
where it has extremely small gradients.
I2DL: Prof. Dai 11
 Transformers

I2DL: Prof. Dai 12

 Lecture 12:
Advanced DL topics

I2DL: Prof. Dai 13

 Graph Neural
Networks

I2DL: Prof. Dai 21

 A graph
• Node: a concept
• Edge: a connection between concepts

Nodes

Edges

I2DL: Prof. Dai 22

 Deep learning on graphs
• Generalizations of neural networks that can operate
on graph-structured domains:
– Scarselli et al. “The Graph Neural Network Model”. IEEE Trans. Neur. Net 2009.
– Kipf et al. “Semi-Supervised Classification with Graph Convolutional Networks. ICLR
2016.
– Gilmer et al. “Neural Message Passing for Quantum Chemistry”. ICML 2017
– Battaglia et al. “Relational inductive biases, deep learning, and graph networks”. arXiv
2018 (review paper)
• Key challenges:
– Variable sized inputs (number of nodes and edges)
– Need invariance to node permutations

I2DL: Prof. Dai 23

 General Idea
Graph with optional node and edge feature vectors

Graph with updated context-

Graph with optional node
aware node and (possibly
and edge feature vectors
Information propagation across edge) feature vector(s)
the graph for several iterations

I2DL: Prof. Dai Figure credit: https://tkipf.github.io/graph-convolutional-networks/ 24

 General Idea
Graph with optional node and edge feature vectors

Graph with updated context-

Graph with optional node
aware node and (possibly
and edge feature vectors
Information propagation across edge) feature vector(s)
the graph for several iterations

I2DL: Prof. Dai Figure credit: https://tkipf.github.io/graph-convolutional-networks/ 25

 Message Passing Networks
• We can divide the propagation process in two steps:
‘node to edge’ and ‘edge to node’ updates.

Initial Graph ‘Node to Edge’ Update ‘Edge to Node’ Update

Node embeddings
Edge embeddings

I2DL: Prof. Dai Battaglia et al. “Relational inductive biases, deep learning, and graph networks”. 2018 26
 ‘Node to edge’ updates
• At every message passing step , first do:

Embedding of node i in Embedding of Embedding of node

the previous message edge (i,j) in the j in the previous
passing step previous message message passing
passing step step

I2DL: Prof. Dai 27

 ‘Node to edge’ updates
• At every message passing step , first do:

I2DL: Prof. Dai 28

 ‘Node to edge’ updates
• At every message passing step , first do:

Learnable function (e.g.

MLP) with shared
weights across the
entire graph

I2DL: Prof. Dai 29

 ‘Edge to node’ updates
• After a round of edge updates, each edge
embedding contains information about its pair of
incident nodes
• Then, edge embeddings are used to update nodes:
message

message
Order invariant Neighbors of message
operation (e.g. node i
sum, mean, max)

I2DL: Prof. Dai 30

 ‘Edge to node’ updates
• After a round of edge updates, each edge
embedding contains information about its pair of
incident nodes
• Then, edge embeddings are used to update nodes:
The aggregation
provides each node
embedding with
contextual information
about its neighbors
Learnable function (e.g. MLP) with shared
weights across the entire graph
I2DL: Prof. Dai 31
 GNN Applications
• Node or edge classification
– identifying anomalies such as spam, fraud
– Relationship discovery for social networks, search
networks

I2DL: Prof. Dai https://gm-neurips-2020.github.io/master-deck.pdf 32

 GNN Applications
• Modeling epidemiology
– Spatio-temporal graph

I2DL: Prof. Dai https://gm-neurips-2020.github.io/master-deck.pdf 33

 GNN Applications
• Traffic forecasting

https://www.deepmind.com/blog/traffic-prediction-with-advanced-graph-neural-networks

I2DL: Prof. Dai 34

 GNN Applications
• Scene graph generation

[Xu et al. ‘17] Scene Graph Generation by Iterative Message Passing

I2DL: Prof. Dai 35
 GNN Applications
• 3D mesh generation

[Dai and Niessner] Scan2Mesh: From Unstructured Range Scans to 3D Meshes

I2DL: Prof. Dai 36
 Generative Models

I2DL: Prof. Dai 37

 Semantic Segmentation (FCN)
• Recall the Fully Convolutional Networks

Can we
do
better?

[Long et al., CVPR’15] : Fully Convolutional Networks for Semantic Segmentation

I2DL: Prof. Dai 38
 SegNet

[Badrinarayanan et al., TPAMI‘16] SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
I2DL: Prof. Dai 39
 Generative Models
• Given training data, how to generate new samples
from the same distribution

Generated Images
Real Images

I2DL: Prof. Dai Source: https://openai.com/blog/generative-models/ 40

 Generative Models

Explicit Density Implicit Density

Tractable Density Approximate Density Markov Chain Direct

Fully Visible Belief Nets GSN GAN

Variational Markov Chain

Variational Autoencoder Boltzmann Machine

Denoising Diffusion Probabilistic Models (DDPM)

Figure copyright and adapted from Ian Goodfellow, Tutorial on Generative Adversarial Networks, 2017
I2DL: Prof. Dai 41
 Variational
Autoencoders

I2DL: Prof. Dai 42

 Autoencoders
• Encode the input into a representation (bottleneck)
and reconstruct it with the decoder
Encoder Decoder

𝑥 𝑥෤

Conv Transpose Conv

I2DL: Prof. Dai 43
 Autoencoders
• Encode the input into a representation (bottleneck)
and reconstruct it with the decoder

Latent space learned

by autoencoder on MNIST

Source: https://bit.ly/37ctFMS

I2DL: Prof. Dai 44

 Variational Autoencoder

𝑞𝜙 𝑧 𝑥 𝑝𝜃 𝑥෤ 𝑧)
Encoder Decoder

𝑥 𝜙 𝜃 𝑥෤

Conv Transpose Conv

I2DL: Prof. Dai 45
 Variational Autoencoder
Goal: Sample from the latent distribution to generate new outputs!

𝑥 𝜙 𝜃 𝑥෤

Conv Transpose Conv

I2DL: Prof. Dai 46
 Variational Autoencoder
• Latent space is now a distribution
• Specifically it is a Gaussian
Encoder Decoder
𝜇𝑧|𝑥

𝜙 Sample 𝜃
𝑥 𝑥෤

Σ𝑧|𝑥 𝑧

𝑧|𝑥 ∼ 𝒩(𝜇𝑧|𝑥 , Σ𝑧|𝑥 )

I2DL: Prof. Dai 47

 Variational Autoencoder
• Latent space is now a distribution
• Specifically it is a Gaussian
Encoder
𝜇𝑧|𝑥 Mean

𝑥 𝜙 𝑧|𝑥 ∼ 𝒩(𝜇𝑧|𝑥 , Σ𝑧|𝑥 )

Σ𝑧|𝑥 Diagonal covariance

I2DL: Prof. Dai 48

 Variational Autoencoder
• Training: loss makes sure the latent space is close to a
Gaussian and the reconstructed output is close to the
input
Encoder Decoder
𝜇𝑧|𝑥

𝜙 Sample 𝜃
𝑥 𝑥෤

Σ𝑧|𝑥 𝑧

𝑧|𝑥 ∼ 𝒩(𝜇𝑧|𝑥 , Σ𝑧|𝑥 )

I2DL: Prof. Dai 49

 Variational Autoencoder
• Test: Sample from the latent space

Decoder
𝜇𝑧|𝑥

Sample 𝜃 𝑥෤

Σ𝑧|𝑥 𝑧

𝑧|𝑥 ∼ 𝒩(𝜇𝑧|𝑥 , Σ𝑧|𝑥 )

I2DL: Prof. Dai 52

 Autoencoder vs VAE

Autoencoder Variational Autoencoder Ground Truth

Source: https://github.com/kvfrans/variational-autoencoder

I2DL: Prof. Dai 53

 Generating data

Degree of smile

I2DL: Prof. Dai

Head pose 54
 Autoencoder Overview
• Autoencoders (AE)
– Reconstruct input
– Unsupervised learning

• Variational Autoencoders (VAE)

– Probability distribution in latent space (e.g., Gaussian)
– Interpretable latent space (head pose, smile)
– Sample from model to generate output

I2DL: Prof. Dai 55

Generative Adversarial
Networks (GANs)

I2DL: Prof. Dai 56

 Generative Adversarial Networks (GANs)

Source: https://github.com/hindupuravinash/the-gan-zoo

I2DL: Prof. Dai 57

 Autoencoder

Conv Transposed conv

I2DL: Prof. Dai 58
 Decoder as Generative Model
Reconstruction
Loss (often L2)

Test time:
-> reconstruction from
‘random’ vector

Output Image
Latent space 𝑧
dim 𝑧 < dim(𝑥)

I2DL: Prof. Dai 59

 Decoder as Generative Model
Reconstruction Loss
Often L2, i.e., sum of squared dist.
-> L2 distributes error equally
-> mean is opt.
-> res. Is blurry
“Test time”:
-> reconstruction from
‘random’ vector

Instead of L2, can we

Latent space z
dim (z) < dim (x)
“learn” a loss function?

I2DL: Prof. Dai 60

 Generative Adversarial Networks (GANs)

𝐺
𝑧
𝐺(𝑧)

𝐷(𝐺(𝑧))

[Goodfellow et al., NIPS‘14] Generative Adversarial Networks (slide from McGuinness)

I2DL: Prof. Dai 61
 Generative Adversarial Networks (GANs)
𝑥

𝐷(𝑥)
𝐷

𝐺
𝑧
𝐺(𝑧)

𝐷(𝐺(𝑧))

[Goodfellow et al., NIPS‘14] Generative Adversarial Networks (slide from McGuinness)

I2DL: Prof. Dai 62
 Generative Adversarial Networks (GANs)

real data fake data

I2DL: Prof. Dai [Goodfellow, NIPS‘16] Tutorial: Generative Adversarial Networks 63

 GANs: Loss Functions
• Discriminator loss
𝐷
1 1
𝐽 =− 𝔼𝐱∼𝑝𝑑𝑎𝑡𝑎 log 𝐷 𝒙 − 𝔼𝒛 log 1 − 𝐷 𝐺 𝒛
2 2

binary cross entropy

• Generator loss
𝐽(𝐺) = −𝐽 𝐷

• Minimax Game:
– G minimizes probability that D is correct
– Equilibrium is saddle point of discriminator loss
• D provides supervision (i.e., gradients) for G
[Goodfellow et al., NIPS‘14] Generative Adversarial Networks
I2DL: Prof. Dai 64
 GAN Applications

I2DL: Prof. Dai 65

 BigGAN: HD Image Generation

[Brock et al., ICLR‘18] BigGAN : Large Scale GAN Training for High Fidelity Natural Image Synthesis
I2DL: Prof. Dai 66
 StyleGAN: Face Image Generation

[Karras et al., ‘18] StyleGAN : A Style-Based Generator Architecture for Generative Adversarial Networks
[Karras et al., ‘19] StyleGAN2 : Analyzing and Improving the Image Quality of StyleGAN
I2DL: Prof. Dai 67
Cycle GAN: Unpaired Image-to-Image Translation

[Zhu et al., ICCV‘17] Cycle GAN : Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
I2DL: Prof. Dai 68
 SPADE: GAN-Based Image Editing

[Park et al., CVPR‘19] SPADE : Semantic Image Synthesis with Spatially-Adaptive Normalization
I2DL: Prof. Dai 69
 Texturify: 3D Texture Generation

I2DL: Prof. Dai 𝑧0 𝑧1 𝑧2 70

 Diffusion

I2DL: Prof. Dai 71

 Diffusion – Search Interest

Source: Google Trends

I2DL: Prof. Dai 72

 Diffusion Models
• Class of generative models

• Achieved state-of-the-art image generation (DALLE-

2, Imagen, StableDiffusion)

• What is diffusion?

I2DL: Prof. Dai 73

 Diffusion Process
• Gradually add noise to input image 𝑥0 in a series of 𝑇
time steps

• Neural network trained to recover original data

[Ho et al. ’20] Denoising Diffusion Probabilistic Models

I2DL: Prof. Dai 74
 Forward Diffusion
• Markov chain of 𝑇 steps
– Each step depends only on previous
• Adds noise to 𝑥0 sampled from real distribution 𝑞 𝑥
𝑞 𝑥𝑡 𝑥𝑡−1 = 𝒩(𝑥𝑡 ; 𝝁𝑡 = 1 − 𝛽𝑡 𝑥𝑡−1 , 𝚺𝑡 = 𝛽𝑡 𝐈)
mean variance
identity matrix

[Ho et al. ’20] Denoising Diffusion Probabilistic Models

I2DL: Prof. Dai 75
 Forward Diffusion
• Go from 𝑥0 to 𝑥𝑇 :
𝑇

𝑞 𝑥1:𝑇 𝑥0 = ෑ 𝑞(𝑥𝑡 |𝑥𝑡−1 )

𝑡=1

• Efficiency?

I2DL: Prof. Dai 76

 Reparameterization
• Define 𝛼𝑡 = 1 − 𝛽𝑡 , 𝛼𝑡 = ς𝑡𝑠=0 𝛼𝑠 , 𝜖0 , … , 𝜖𝑡−1 ~𝒩(𝟎, 𝐈)

𝑥𝑡 = 1 − 𝛽𝑡 𝑥𝑡−1 + 𝛽𝑡 𝜖𝑡−1
= 𝛼𝑡 𝑥𝑡−2 + 1 − 𝛼𝑡 𝜖𝑡−2
=⋯
= 𝛼𝑡 𝑥0 + 1 − 𝛼𝑡 𝜖0

𝑥𝑡 ~𝑞 𝑥𝑡 𝑥0 = 𝒩(𝑥𝑡 ; 𝛼𝑡 𝑥0 , 1 − 𝛼𝑡 𝐈)
I2DL: Prof. Dai 77
 Reverse Diffusion
• 𝑥𝑇→∞ becomes a Gaussian distribution

• Reverse distribution 𝑞(𝑥𝑡−1 |𝑥𝑡 )

– Sample 𝑥𝑇 ~𝒩 𝟎, 𝐈 and run reverse process
– Generates a novel data point from original distribution

• How to model reverse process?

I2DL: Prof. Dai 78

 Approximate Reverse Process
• Approximate 𝑞(𝑥𝑡−1 |𝑥𝑡 ) with parameterized model 𝑝𝜃
(e.g., deep network)
𝑝𝜃 𝑥𝑡−1 𝑥𝑡 = 𝒩(𝑥𝑡−1 ; 𝜇𝜃 𝑥𝑡 , 𝑡 , Σ𝜃 𝑥𝑡 , 𝑡 )
𝑇

𝑝𝜃 (𝑥0:𝑇 ) = 𝑝𝜃 (𝑥𝑇 ) ෑ 𝑝𝜃 𝑥𝑡−1 𝑥𝑡

𝑡=1

I2DL: Prof. Dai [Ho et al. ’20] Denoising Diffusion Probabilistic Models 79
 Training a Diffusion Model
• Optimize negative log-likelihood of training data

𝐿𝑉𝐿𝐵
= 𝔼𝑞 [𝐷𝐾𝐿 𝑞(𝑥𝑇 |𝑥0 ||𝑝𝜃 (𝑥𝑇 ) 𝐿𝑇
𝑇

+ ෍ 𝐷𝐾𝐿 𝑞 𝑥𝑡−1 𝑥𝑡 , 𝑥0 ||𝑝𝜃 𝑥𝑡−1 𝑥𝑡 − log 𝑝𝜃 (𝑥0 |𝑥1 )]

𝑡=2
𝐿𝑡−1 𝐿0
• Nice derivations: https://lilianweng.github.io/posts/2021-07-
11-diffusion-models
I2DL: Prof. Dai 80
 Training a Diffusion Model
• 𝐿𝑡−1 = 𝐷𝐾𝐿 𝑞 𝑥𝑡−1 𝑥𝑡 , 𝑥0 ||𝑝𝜃 𝑥𝑡−1 𝑥𝑡
• Comparing two Gaussian distributions
• 𝐿𝑡−1 ∝ 𝜇෥𝑡 𝑥𝑡 , 𝑥0 − 𝜇𝜃 (𝑥𝑡 , 𝑡) 2

• Predicts diffusion posterior mean

I2DL: Prof. Dai 81

 Diffusion Model Architecture
• Input and output dimensions must match

• Highly flexible to architecture design

• Commonly implemented with U-Net architectures

I2DL: Prof. Dai 83

 Applications for Diffusion Models
• Text-to-image

I2DL: Prof. Dai 84

 Applications for Diffusion Models
• Image inpainting & outpainting

I2DL: Prof. Dai https://github.com/lkwq007/stablediffusion-infinity 85

 Applications for Diffusion Models
• Text-to-3D Neural Radiance Fields

I2DL: Prof. Dai https://dreamfusion3d.github.io/ 86

 Reinforcement
Learning

I2DL: Prof. Dai 87

 Learning Paradigms in ML

Supervised
Unsupervised Reinforcement
Learning
Learning Learning
E.g., classification,
E.g., clustering,
regression
anomaly detection Sequential data
Labeled data
Unlabeled data Learning by
Find mapping from interaction with
Find structure in data the environment
input to label

I2DL: Prof. Dai 88

 In a Nutshell

• RL-agent is trained using the

“carrot and stick“ approach
• Good behavior is
encouraged by rewards
• Bad behavior is discouraged
by punishment
Source: quora.com

I2DL: Prof. Dai 89

 Examples of RL: Learning to Walk

Source: Deepmind.com

I2DL: Prof. Dai 90

 Agent and Environment
Action, at

Agent Environment

Observation, ot

Reward, rt
I2DL: Prof. Dai 91
 Characteristics of RL
• Sequential, non i.i.d. data (time matters)

• Actions have an effect on the environment

-> Change future input

• No supervisor, target is approximated by the reward

signal

I2DL: Prof. Dai 92

 History and State
• The agent makes decisions based on the history h of
observations, actions and rewards up to time-step t
ℎ𝑡 = 𝑜1 , 𝑎1 , 𝑟1 , … , 𝑎𝑡−1 , 𝑟𝑡−1 , 𝑜𝑡

• The state s contains all the necessary information

from h -> s is a function of h

𝑠𝑡 = 𝑓 ℎ𝑡

I2DL: Prof. Dai Source: UCL Reinforcement Learning 93

 Markov Assumption
• Problem: History grows linearly over time
• Solution: Markov Assumption
• A state St is Markov if and only if:

ℙ 𝑠𝑡+1 |𝑠𝑡 = ℙ 𝑠𝑡+1 |𝑠1 , … 𝑠𝑡

• “The future is independent of the past given the

present“

I2DL: Prof. Dai Source: UCL Reinforcement Learning 94

 Agent and Environment
• Reward and next Action, at
state are functions
of current
observation ot
Agent Environment
and action at only

Observation, ot

Reward, rt

I2DL: Prof. Dai 95

 Mathematical Formulation
• The RL problem is a Markov Decision Process (MDP)
defined by: 𝒮, 𝒜, ℛ, ℙ, 𝛾

𝒮 : Set of possible states

𝒜 : Set of possible actions
ℛ : Distribution of reward given (state, action) pair
ℙ : Transition probability of a (state, action) pair
𝛾 : Discount factor (discounts future rewards)

I2DL: Prof. Dai Source: Stanford cs231n 96

 Components of an RL Agent

• Policy 𝜋 : Behavior of the agent

-> Mapping from state to action: 𝑎 = 𝜋(𝑠)

• Value-, Q-Function: How good is a state or (state,

action) pair
-> Expected future reward

I2DL: Prof. Dai Source: UCL Reinforcement Learning 97

 Taxonomy of RL Algorithms
RL Algorithms

Model-Free RL Model-Based RL

Policy Optimization Q-Learning Learn the Model Given the Model

Policy Gradient DQN World Models Alpha Zero

DDPG
A2C / A3C C51 I2A
TD3
PPO QR-DQN MBMF
SAC
TRPO HER MBVE

I2DL: Prof. Dai Source: spinningup.openai.com 98

 RL Milestones: Playing Atari

• Mnih et al. 2013, first appearance of DQN

• Successfully learned to play different Atari games
like Pong, Breakout, Space Invaders, Seaquest and
Beam Rider

I2DL: Prof. Dai [Mnih et al., NIPS’13] Playing Atari with Deep Reinforcement Learning 99
 RL Milestones: AlphaZero (StarCraft)
• Model: Transformer network with a LSTM core
• Trained on 200 years of StarCraft play for 14 days
• 16 Google v3 TPUs
• December 2018:
Beats MaNa, a
professional StarCraft
player (world rank 13)

I2DL: Prof. Dai 100

 I2DL Overview

I2DL: Prof. Dai 101

 Machine Learning Basics
• Unsupervised vs • Data splitting
Supervised Learning

• Linear vs logistic
regression

I2DL: Prof. Dai 102

 Intro to Neural Networks
• Backpropagation • Activation functions

• Loss functions
– Comparison & effects

I2DL: Prof. Dai 103

 Training Neural Networks
• Gradient Descent/ SGD • Regularization

• Parameter search
& interpretation

Source: http://ruder.io/optimizing-gradient-descent/,
https://srdas.github.io/DLBook/ImprovingModelGeneralization.html,
http://cs231n.github.io/neural-networks-3/
I2DL: Prof. Dai 104
 Typology of Neural Networks
• CNNs • Autoencoder

𝑁+2⋅𝑃−𝐹
+1
𝑆
𝑁+2⋅𝑃−𝐹
× +1
𝑆

• RNNs • GANs

real or
fake pair?

I2DL: Prof. Dai 105

 Other DL Courses

I2DL: Prof. Dai 106

 Deep Learning at TUM
DL in DL for
Robotics Medical
(Bäuml) Applicat.
(Menze)
DL for Intro to Machine
Physics Deep Learning
(Thuerey) Learning (Günneman)

ML for DL for
3D Vision
Geometry (Niessner)
(Dai)

I2DL: Prof. Dai 107

 Deep Learning at TUM
• Keep expanding the courses on Deep Learning

• This Introduction to Deep Learning course is the basis

for a series of Advanced DL lectures on different
topics

• Advanced topics are only for Master students

– Preparation for MA theses, etc.

I2DL: Prof. Dai 108

 Advanced DL for Computer Vision
• Deep Learning for Vision (Profs. Niessner)

• Syllabus
– Advanced architectures, e.g., Siamese neural networks
– Variational Autoencoders
– Generative models, e.g. GAN,
– Multi-dimensional CNN
– Graph neural networks
– Domain adaptation

I2DL: Prof. Dai 109

 Advanced DL for Computer Vision
• Deep Learning for Vision
– 2V+3P
– Must have attended the Intro to DL

– Practical part is a project that will last the whole semester

– Please do not sign up unless you are willing to spend a
lot of time on the project!

I2DL: Prof. Dai 110

 ML for 3D Geometry
• Lectures + Practical Project
– Geometric foundations
– Shape descriptors, similarity, segmentation
– Shape modeling, reconstruction, synthesis
– Deep learning for multi-view, volumetric, point cloud, and
graph data

– Prof. Dai

I2DL: Prof. Dai 111

 Next Dates and Exam
• Friday (31.01): Guest Lecture, Prof. Björn Ommer
– Here in HS1!

• Exam date: February 10th at 18:30-20:00

• There will NOT be a retake exam
• Neither cheat sheet nor calculator during the exam

I2DL: Prof. Dai 112

 Good Luck
in the Exam ☺

I2DL: Prof. Dai 113

 References for Further Reading
• https://towardsdatascience.com/intuitively-
understanding-variational-autoencoders-1bfe67eb5daf

• https://phillipi.github.io/pix2pix/

• http://cs231n.stanford.edu/slides/2017/cs231n_2017_le
cture13.pdf

I2DL: Prof. Dai 114