Convolutional Neural Network

1
 Preview [From recent Yann
LeCun slides]

Fei-Fei Li & Andrej Karpathy & Justin Johnson

ImageNet Large Scale Visual Recognition Challenge, 2012

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep
convolutional neural networks." Advances in Neural Information Processing Systems. 2012
Convolution layer

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

1 -1 -1 0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

-1 1 -1 0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-1 -1 1 -0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33

-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1 -0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
-1
-1
-1
-1
1
-1
-1
1
-1
-1
-1
1
1
-1
-1
-1
-1
-1
1 -1 1 0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11

-1
-1
-1
-1
-1
-1
-1
1
1
-1
-1
1
-1
-1
-1
-1
-1
-1
-1 1 -1 -0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11

0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11

-1
-1
-1
1
1
-1
-1
-1
-1
-1
-1
-1
1
-1
-1
1
-1
-1
1 -1 1 -0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55

0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33

-1 -1 -1 -1 -1 -1 -1 -1 -1

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

-1 -1 1 0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-1 1 -1 0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

1 -1 -1 -0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33

Pooling: Shrinking the image stack

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Pooling
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

1.00
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Pooling
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

1.00 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Pooling
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

1.00 0.33 0.55 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 1.00 0.33 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 0.33 1.00 0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.55 0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Rectified Linear Units (ReLUs)

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33 0.77

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Rectified Linear Units (ReLUs)

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33 0.77 0

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Rectified Linear Units (ReLUs)

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33 0.77 0 0.11 0.33 0.55 0 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11

0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Rectified Linear Units (ReLUs)

0.77 -0.11 0.11 0.33 0.55 -0.11 0.33 0.77 0 0.11 0.33 0.55 0 0.33

-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11 0 1.00 0 0.33 0 0.11 0

0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55 0.11 0 1.00 0 0.11 0 0.55

0.33 0.33 -0.33 0.55 -0.33 0.33 0.33 0.33 0.33 0 0.55 0 0.33 0.33

0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11 0.55 0 0.11 0 1.00 0 0.11

-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11 0 0.11 0 0.33 0 1.00 0

0.33 -0.11 0.55 0.33 0.11 -0.11 0.77 0.33 0 0.55 0.33 0.11 0 0.77
Layers get stacked

1.00 0.33 0.55 0.33

0.33 1.00 0.33 0.55

-1 -1 -1 -1 -1 -1 -1 -1 -1 0.55 0.33 1.00 0.11

-1 1 -1 -1 -1 -1 -1 1 -1 0.33 0.55 0.11 0.77

-1 -1 1 -1 -1 -1 1 -1 -1
0.55 0.33 0.55 0.33
-1 -1 -1 1 -1 1 -1 -1 -1
0.33 1.00 0.55 0.11
-1 -1 -1 -1 1 -1 -1 -1 -1
0.55 0.55 0.55 0.11
-1 -1 -1 1 -1 1 -1 -1 -1
0.33 0.11 0.11 0.33
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1 0.33 0.55 1.00 0.77

-1 -1 -1 -1 -1 -1 -1 -1 -1 0.55 0.55 1.00 0.33

1.00 1.00 0.11 0.55

0.77 0.33 0.55 0.33

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Deep stacking

1.00 0.55

-1 -1 -1 -1 -1 -1 -1 -1 -1 0.55 1.00
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1 1.00 0.55
-1 -1 -1 -1 1 -1 -1 -1 -1
0.55 0.55
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1 0.55 1.00
-1 -1 -1 -1 -1 -1 -1 -1 -1
1.00 0.55

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Fully connected layer

1.00

0.55

1.00 0.55 0.55

0.55 1.00 1.00

1.00

1.00 0.55 0.55

0.55 0.55 0.55

0.55
0.55 1.00
0.55
1.00 0.55
1.00

1.00

0.55

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Fully connected layer

0.55

1.00

1.00

0.55

0.55
X
0.55

0.55

0.55

1.00

0.55

0.55
O
1.00

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Images

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Sound

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

Text

Brandon Rohrer https://brohrer.github.io/how_convolutional_neural_networks_work.html

8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1
8x8 image, 3x3 filter, Stride 1

Total Connections?

6x6x1 x 3x3x1

Total Unique
Parameters?

3x3x1
+ bias
8x8 image, 3x3 filter, Stride 1 2x2 pooling, Stride 2
8x8 image, 3x3 filter, Stride 1 2x2 pooling, Stride 2
 (not counting biases)

Model Memory Parameter

INPUT: [224x224x3]
CONV3-64: [224x224x64]
CONV3-64: [224x224x64]
POOL2: [112x112x64]
CONV3-128: [112x112x128]
CONV3-128: [112x112x128]
POOL2: [56x56x128]
CONV3-256: [56x56x256]
CONV3-256: [56x56x256]
CONV3-256: [56x56x256]
POOL2: [28x28x256]
CONV3-512: [28x28x512]
CONV3-512: [28x28x512]
CONV3-512: [28x28x512]
POOL2: [14x14x512]
CONV3-512: [14x14x512]
CONV3-512: [14x14x512]
CONV3-512: [14x14x512]
POOL2: [7x7x512]
FC: [1x1x4096]
FC: [1x1x4096]
FC: [1x1x1000]

Fei-Fei Li & Andrej Karpathy & Justin Johnson

 (not counting biases)
INPUT: [224x224x3] memory: 224*224*3=150K params: 0
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864
POOL2: [112x112x64] memory: 112*112*64=800K params: 0
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456
POOL2: [56x56x128] memory: 56*56*128=400K params: 0
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824
POOL2: [28x28x256] memory: 28*28*256=200K params: 0
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296
POOL2: [14x14x512] memory: 14*14*512=100K params: 0
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
POOL2: [7x7x512] memory: 7*7*512=25K params: 0
FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448
FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216
FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000

Fei-Fei Li & Andrej Karpathy & Justin Johnson

 two more layers to go: POOL/FC

48
Fei-Fei Li & Andrej Karpathy & Justin Johnson
Resources: Datasets
• UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html
• UCI KDD Archive: http://kdd.ics.uci.edu/summary.data.application.html
• Statlib: http://lib.stat.cmu.edu/
• Delve: http://www.cs.utoronto.ca/~delve/

49
Resources: Journals
• Journal of Machine Learning Research www.jmlr.org
• Machine Learning
• IEEE Transactions on Neural Networks
• IEEE Transactions on Pattern Analysis and Machine Intelligence
• Annals of Statistics
• Journal of the American Statistical Association
• ...

50
Resources: Conferences
• International Conference on Machine Learning (ICML)
• European Conference on Machine Learning (ECML)
• Neural Information Processing Systems (NIPS)
• Computational Learning
• International Joint Conference on Artificial Intelligence (IJCAI)
• ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)
• IEEE Int. Conf. on Data Mining (ICDM)

51
 Acknowledgements
 Emily Fox & Carlos Guestrin, Machine Learning Courses, University of Washington, Coursera
 Introduction to Machine Learning, Alphaydin
Material in these slides has been taken from, the following resources

 Statistical Pattern Recognition: A Review – A.K Jain et al., PAMI (22) 2000
 Pattern Recognition and Analysis Course – A.K. Jain, MSU
 Pattern Classification” by Duda et al., John Wiley & Sons.

52
MATLAB for Deep Learning
Department of Computer & Software Engineering
NUST College of E&ME
Outline
• Introduction
• Installation
• Deep Learning for MATLAB
• Convolutional Neural Networks (CNNs)
• CNN Architecture
• CNN Hidden Layers
• CNN Applications
• Training a CNN
• Transfer Learning
• Training from Scratch
• Hands-on Exercises
Matrix Laboratory (MATLAB)
• Developed by MathWorks Inc. in 1984
• MATLAB was the leading tool in industry for more than 10 years
• The script written in MATLAB are wrapped for commercial products
• It’s widely adapted in engineering schools worldwide for teaching
purposes
• However, the reign of MATLAB has been taken by Python now. Why?
• Still MathWorks are trying hard to compete with the changing
dynamics
 Deep Learning in MATLAB
• Starting from MATLAB R2016b, the support for deep learning has
been officially added in MATLAB
• For deep learning in MATLAB, the deep learning toolbox should be
installed
• Deep learning toolbox contains pre-trained:
• Convolutional Neural Networks Models
• Long Term Short Memory (LSTM) Models
• Furthermore, users can create their own deep architectures such as:
• Series Networks
• Directed Acyclic Graph Networks
• Recurrent Networks

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
Convolutional Neural Networks
• One of the most popular deep neural network
• Widely applied for visual analysis and recognition
• Consists of an input layer, output layer and multiple hidden layers
• Each hidden layer typically consists of:
• Convolution layers
• Rectified Linear Units (ReLUs) layers
• Pooling layers
• Normalization layers
• Dropout layers
• Fully connected layers
• Does not require handcrafted feature set
 2D Convolution Layer
• A 2-D convolutional layer applies sliding convolutional filters to the
input
• The layer convolves the input by moving the filters along the input
vertically and horizontally
• Afterwards, the dot product of the weights and the input are
computed along with adding the bias term
• layer = convolution2dLayer(filterSize,numFilters)

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Rectified Linear Unit (ReLU)
• A ReLU layer performs a threshold operation to each element of the
input, where any value less than zero is set to zero.
• This operation is equivalent to
𝑥 𝑥≥0
𝑓 𝑥 =ቊ
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
• layer = reluLayer
• layer = reluLayer('Name',Name)

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Pooling Layers
• Down samples the feature map
• Have different criteria for down-sampling
• Max pooling
• Average pooling
• layer = maxPooling2dLayer(poolSize)
• layer = averagePooling2dLayer(poolSize)

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Dropout Layer
• A dropout layer randomly sets input elements to zero with a given
probability
• At prediction time the output of a dropout layer is equal to its input
• layer = dropoutLayer('Name','drop1')

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Batch Normalization Layer
• A batch normalization layer normalizes each input channel across a
mini-batch
• Normalization speed up the training of CNNs and reduce the
sensitivity to network initialization
• Batch normalization layers are used between convolutional layers and
nonlinearities, such as ReLU layers
• layer = batchNormalizationLayer

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Fully Connected Layer
• A fully connected layer multiplies the input by a weight matrix and
then adds a bias vector
• layer = fullyConnectedLayer(outputSize)
• layer = fullyConnectedLayer(outputSize,Name,Value)

“MATLAB for Deep Learning”, MATLAB & Simulink, MathWorks, Accessed: 19/12/18
 Typical feed forward CNN architecture

Leyuan Fang, David Cunefare, Chong Wang, Robyn H. Guymer, Shutao Li, Sina Farsiu, “Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD
patients using deep learning and graph search”, BIOMEDICAL OPTICS EXPRESS, Vol. 8, No. 5, May 2017.
 Recurrent Networks

“Recurrent Neural Networks and Long Short Term Memory”, Towards Data Science, Accessed: 19/12/18
 Encoder-Decoder CNNs

Stefania Raimondo, “Semantic Segmentation”, Visual Perception for Autonomous Driving, 2015
 Regional CNNs

“R-CNN, Fast R-CNN, Faster R-CNN, YOLO — Object Detection Algorithms”, Towards Data Science, Accessed: 19/12/18
CNN Applications
CNNs are used in variety of applications for:
• Classification
• Feature Extraction
• Segmentation
• Object Detection
• Regression
Training a CNN
• All the pre-trained models are trained on ImageNet dataset (which is a
huge dataset)
Transfer Learning
• Remove the classification layer from the pre-trained model and only
generate the CNN codes
• Pass the codes to the classifier trained on new dataset for classification
• Or fine tune the layers of a pre-trained model using the new dataset, add
the new classification layer for classification
Training from Scratch
• If you have large dataset and enough time, you can train the CNN from
scratch as well (not a recommended way)
Hands-on Exercises
MATLAB for Deep Learning
Exercise 1 (Vehicle Recognition)
• Develop a deep framework that is able to classify the vehicle within
the image
Exercise 2 (Number Plate Detection)
• Develop a deep framework that can detect the number plate from the
input image
Exercise 3 (CNN for Regression)
• Use CNN to find the angle of rotation from the handwritten text
images