Introductio

n to
Deep Learn
ing
Ismini Loure
ntzou
11-30-2017
 Outline
 Machine Learning basics
 Introduction to Deep Learning
 what is Deep Learning
 why is it useful
 Main components/hyper-parameters:
 activation functions
 optimizers, cost functions and training

x
 regularization methods
 tuning Backpropagation
GANs & Adversarial training
 classification vs. regression tasks
Bayesian Deep Learning
 DNN basic architectures: Generative models
 convolutional Unsupervised / Pretraining
 recurrent
 attention mechanism
 Application example: Relation Extraction

Most material from

 Machine Learning Basics
Machine learning is a field of computer science that gives computers the ability to
learn without being explicitly programmed

Machine Learning
Labeled Data
algorithm

Training
Prediction

Learned model
Labeled Data Prediction

Methods that can learn from and make predictions on data

 Types of Learning
Supervised: Learning with a labeled training set
Example: email classification with already labeled emails

Unsupervised: Discover patterns in unlabeled data

Example: cluster similar documents based on text

Reinforcement learning: learn to act based on feedback/reward

Example: learn to play Go, reward: win or lose

class A

class A

Classification Regression Clustering

Anomaly Detection
Sequence labeling
http://mbjoseph.github.io/2013/11/27/measure.html
…
 ML vs. Deep Learning
Most machine learning methods work well because of human-designed
representations and input features
ML becomes just optimizing weights to best make a final prediction
 What is Deep Learning (DL) ?
A machine learning subfield of learning representations of data. Exceptional effective
at learning patterns.
Deep learning algorithms attempt to learn (multiple levels of) representation by using a
hierarchy of multiple layers
If you provide the system tons of information, it begins to understand it and respond
in useful ways.

https://www.xenonstack.com/blog/static/public/uploads/media/machine-learning-vs-deep-learning.png
 Why is DL useful?
o Manually designed features are often over-specified, incomplete and take a long time
to design and validate
o Learned Features are easy to adapt, fast to learn
o Deep learning provides a very flexible, (almost?) universal, learnable framework for
representing world, visual and linguistic information.
o Can learn both unsupervised and supervised
o Effective end-to-end joint system learning
o Utilize large amounts of training data

In ~2010 DL started outperforming other

ML techniques
first in speech and vision, then NLP
 State of the art in …

Several big improvements in recent years in NLP

 Machine Translation
 Sentiment Analysis Leverage different levels of representation
 Dialogue Agents o words & characters
 Question Answering o syntax & semantics
 Text Classification …
Neural Network Intro
Weights

Activation functions

How do we train?

4 + 2 = 6 neurons (not counting inputs)

[3 x 4] + [4 x 2] = 20 weights
4 + 2 = 6 biases
26 learnable parameters
 Training
Sample labeled Forward it Back-
data through the Update the
network, get
propagate the network weights
(batch) errors
predictions

Optimize (min. or max.) objective/cost function

Generate error signal that measures difference between
predictions and target values

Use error signal to change the weights and get more

accurate predictions
Subtracting a fraction of the gradient moves you towards
the (local) minimum of the cost function
https://medium.com/@ramrajchandradevan/the-evolution-of-gradient-descend-optimization-algorithm-4106a6702d39
 Gradient Descent
objective/cost function

Update each element of θ

Matrix notation for all parameters

learning rate

Recursively apply chain rule though each node

 One forward pass
Text (input) representation
TFIDF
Word embeddings
….

0.2 -0.5 0.1 0.1 1.0 0.95 very positive

2.0 1.5 1.3 3.0 3.89 positive
0.2 0.025
0.5 0.0 0.25 0.15 negative
-0.3 2.0 0.0 0.37 very negative
0.3 0.0
 Activation functions
Non-linearities needed to learn complex (non-linear) representations of data, otherwise
the NN would be just a linear function

http://cs231n.github.io/assets/nn1/layer_sizes.jpeg

More layers and neurons can approximate more complex functions

Full list:
 Activation: Sigmoid
Takes a real-valued number and
“squashes” it into range between 0 and
1.

http://adilmoujahid.com/images/activation.png

+ Nice interpretation as the firing rate of a neuron

• 0 = not firing at all
• 1 = fully firing

- Sigmoid neurons saturate and kill gradients, thus NN will barely learn

� gradient at these regions almost zero

• when the neuron’s activation are 0 or 1 (saturate)

� almost no signal will flow to its weights

� if initial weights are too large then most neurons would saturate
 Activation: Tanh
Takes a real-valued number and
“squashes” it into range between -1 and
1.

http://adilmoujahid.com/images/activation.png

- Like sigmoid, tanh neurons saturate

- Unlike sigmoid, output is zero-centered
- Tanh is a scaled sigmoid:
 Activation: ReLU
Takes a real-valued number and
thresholds it at zero

http://adilmoujahid.com/images/activation.png

Most Deep Networks use ReLU nowadays

� Trains much faster

• accelerates the convergence of SGD

� Less expensive operations

• due to linear, non-saturating form

• compared to sigmoid/tanh (exponentials etc.)

� More expressive
• implemented by simply thresholding a matrix at zero

� Prevents the gradient vanishing problem

 Overfitting

http://wiki.bethanycrane.com/overfitting-of-data

Learned hypothesis may fit the

training data very well, even
outliers (noise) but fail to generalize
to new examples (test data)

https://www.neuraldesigner.com/images/learning/selection_error.svg
 Regularization
Dropout
• Randomly drop units (along with their
connections) during training
• Each unit retained with fixed probability p,
independent of other units
• Hyper-parameter p to be chosen (tuned)
Srivastava, Nitish, et al. Journal of machine learning research (2014)

L2 = weight decay
• Regularization term that penalizes big weights, added to the
objective
• Weight decay value determines how dominant regularization is during
gradient computation
• Big weight decay coefficient  big penalty for big weights

Early-stopping
• Use validation error to decide when to stop training
• Stop when monitored quantity has not improved after n subsequent epochs
• n is called patience
 Tuning hyper-parameters
g(x) ≈ g(x) + h(y)

g(x) shown in green

h(y) is shown in yellow
Bergstra, James, and Yoshua Bengio. "" Journal
of Machine Learning Research, Feb (2012)

“Grid and random search of 9 trials for optimizing function g(x) ≈ g(x) + h(y)
With grid search, nine trials only test g(x) in three distinct places.
With random search, all nine trials explore distinct values of g. ”

Both try configurations randomly and blindly

Next trial is independent to all the trials done before

Make smarter choice for the next trial, minimize the number of trials
1. Collect the performance at several configurations
2. Make inference and decide what configuration to try next
 Loss functions and output
Classification Regression

Training Rn x {class_1, ..., class_n} Rn x R m

examples (one-hot encoding)

Output Soft-max Linear (Identity)

Layer [map R to a probability distribution]
n
or Sigmoid

f(x)=x

Cost (loss) Cross-entropy Mean Squared Error

function

Mean Absolute Error

 Convolutional Neural
Networks (CNNs)
Main CNN idea for text:
Compute vectors for n-grams and group them afterwards

Example: “this takes too long” compute vectors for:

This takes, takes too, too long, this takes too, takes too long, this takes too long

Convolutional
Input matrix 3x3 filter
http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution
 Convolutional Neural
Networks (CNNs)
Main CNN idea for text:
Compute vectors for n-grams and group them afterwards

max pool
2x2 filters
and stride 2

https://shafeentejani.github.io/assets/images/pooling.gif
 CNN for text classification

Severyn, Aliaksei, and Alessandro Moschitti. "UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment
Classification." SemEval@ NAACL-HLT. 2015.
CNN with multiple filters

Kim, Y. “Convolutional Neural Networks for Sentence Classification”, EMNLP (2014)

sliding over 3, 4 or 5 words at a time

 Recurrent Neural Networks
(RNNs)
Main RNN idea for text:
Condition on all previous words
Use same set of weights at all time steps

https://pbs.twimg.com/media/C2j-8j5UsAACgEK.jpg

� Stack them up, Lego fun!

https://discuss.pytorch.org/uploads/default/original/1X/6415da0424dd66f2f5b134709b92baa59e604c55.jpg
 Bidirectional RNNs
Main idea: incorporate both left and right context
output may not only depend on the previous elements in the sequence, but also future
elements.

past and future around a single token

http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-
part-1-introduction-to-rnns/

two RNNs stacked on top of each other

output is computed based on the hidden state of both RNNs
Sequence 2 Sequence or
Encoder Decoder model

Cho, Kyunghyun, et al. "Learning phrase

representations using RNN encoder-decoder for
statistical machine translation." EMNLP 2014
 Gated Recurrent Units
(GRUs)
Main idea:
keep around memory to capture long dependencies
Allow error messages to flow at different strengths depending on the inputs
Standard RNN computes hidden layer at next time step
directly

Compute an update gate based on current input word

vector and hidden state
http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-
4-implementing-a-grulstm-rnn-with-python-and-theano/

Controls how much of past state should matter now

If z close to 1, then we can copy information in that unit through many steps!
 Gated Recurrent Units
(GRUs)
Main idea:
keep around memory to capture long dependencies
Allow error messages to flow at different strengths depending on the inputs
Standard RNN computes hidden layer at next time step
directly

Compute an update gate based on current input word

vector and hidden state
http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-
4-implementing-a-grulstm-rnn-with-python-and-theano/

Compute a reset gate similarly but with different weights

If reset close to 0, ignore previous hidden
state (allows model to drop information that is
irrelevant in the future)

Units with short-term dependencies often have reset gates very active
Units with long-term dependencies have active update gates z
 Gated Recurrent Units
(GRUs)
Main idea:
keep around memory to capture long dependencies
Allow error messages to flow at different strengths depending on the inputs
Standard RNN computes hidden layer at next time step
directly

Compute an update gate based on current input word

vector and hidden state
http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-
4-implementing-a-grulstm-rnn-with-python-and-theano/

Compute a reset gate similarly but with different weights

are a more complex form, but basically

New memory same intuition
GRUs are often more preferred than
Final memory LSTMs
combines current & previous time steps
 Attention Mechanism
Pool of source states

Bahdanau D. et al. "Neural machine translation by jointly learning to align and translate." ICLR (2015)

Main idea: retrieve as needed

 Attention - Scoring

Compare target and source hidden states

 Attention - Normalization

Convert into alignment weights

 Attention - Context

Build context vector: weighted average

 Attention - Context

Compute next hidden state

 Application Example:
IMDB Movie reviews
sentiment classification
https://uofi.box.com/v/cs510DL

Binary Classification
Dataset of 25,000 movies reviews from IMDB, labeled by
sentiment (positive/negative)
 Application Example:
Relation Extraction from text

http://www.mathcs.emory.edu/~dsavenk/slides/relation_extraction/img/distant.png
Useful for:
• knowledge base completion
• social media analysis
• question answering
• …
 Task: binary (or multi-class)
classification
sentence S = w1 w2 .. e1 .. wj .. e2 .. wn e1 and e2 entities

“The new iPhone 7 Plus includes an improved camera to take amazing pictures”

Component-Whole(e1 , e2 ) ?
YES / NO

It is also possible to include more than two entities as well:

“At codons 12, the occurrence of point mutations from G to T were observed”
 point mutation(codon, 12, G, T)
 Features / Input
representation
1) context-wise split of Embeddings Embeddings
the sentence Embeddings Left
Middle Right

The new iPhone 7 Plus includes an improved camera that takes amazing pictures

2) word sequences Word indices Position indices e1 Position indices e2

concatenated with [5, 7, 12, 6, 90 …] [-1, 0, 1, 2, 3 …] [-4, -3, -2 -1, 0]
positional features
Word Embeddings Positional emb. e1 Positional emb. e2

3) concatenating
embeddings of two entities
with average of word Embeddings e1 Embeddings e2 context embeddings
embeddings for rest of the
words

The new iPhone 7 Plus includes an improved camera that takes amazing pictures
 Models: MLP
Component-Whole(e1 , e2 ) ?
Sigmoid YES / NO

Dense Layer n
…

Dense Layer 1

Embeddings e1 context embeddings Embeddings e2

The new iPhone 7 Plus includes an improved camera that takes …

Simple fully-connected multi-layer perceptron

 Models: CNN
Component-Whole(e1 , e2 ) ?
Sigmoid
YES / NO

Max Pooling Max Pooling Max Pooling

Convolutional Layer Convolutional Layer Convolutional Layer

Embeddings Left Embeddings Middle Embeddings Right

The new iPhone 7 Plus includes an improved camera that takes …

OR
Word indices Position indices e1 Position indices e2
[5, 7, 12, 6, 90 …] [-1, 0, 1, 2, 3 …] [-4, -3, -2 -1, 0]

Word Embeddings Positional emb. e1 Positional emb. e2

Zeng, D.et al. “Relation classication via convolutional deep neural network”.COLING
(2014)
 Models: CNN (2)
Component-Whole(e1 , e2 ) ?
Sigmoid
YES / NO

Max Max Max

Pooling Pooling Pooling

Convoluti Convoluti Convoluti

on on on CNN with multiple CNN with multiple
filter=2 filter=3 filter=k filter sizes filter sizes

Embeddings Left Embeddings Middle Embeddings Right

The new iPhone 7 Plus includes an improved camera that takes …

OR
Word indices Position indices e1 Position indices e2
[5, 7, 12, 6, 90 …] [-1, 0, 1, 2, 3 …] [-4, -3, -2 -1, 0]

Word Embeddings Positional emb. e1 Positional emb. e2

Nguyen, T.H., Grishman, R. “Relation extraction: Perspective from convolutional neural networks.” VS@ HLT-NAACL. (2015)
 Models: Bi-GRU
Component-Whole(e1 , e2 ) ?
Sigmoid
YES / NO

Attention or
Max Pooling

Bi-GRU

Embeddings Left Embeddings Middle Embeddings Right

The new iPhone 7 Plus includes an improved camera that takes …

OR
Word indices Position indices e1 Position indices e2
[5, 7, 12, 6, 90 …] [-1, 0, 1, 2, 3 …] [-4, -3, -2 -1, 0]

Word Embeddings Positional emb. e1 Positional emb. e2

Zhang, D., Wang, D. “Relation classication via recurrent neural network.” -arXiv preprint arXiv:1508.01006 (2015)
Zhou, P. et al. “Attention-based bidirectional LSTM networks for relation classication. ACL (2016)
 Distant Supervision
Circumvent the annotation problem – create large dataset
Exploit large knowledge bases to automatically label entities and their relations in text

Assumption:
when two entities co-occur in a sentence, a certain relation is expressed

knowledge base
Relation Entity 1 Entity 2 text
place of birth Michael Gary Barack Obama moved from Gary ….
Jackson Michael Jackson met … in Hawaii
place of birth Barack Hawaii
Obama
place of birth
… … …

For many ambiguous relations, mere co-occurrence does not guarantee the existence of the
relation  Distant supervision produces false positives
 Attention over Instances
s representation of the sentence set

ai weight given by sentence-level attention

xi sentence vector representation

xi sentence for an entity pair (e1,e2)

n sentences for relation r(e1,e2)

Lin et al. “Neural Relation Extraction with Selective Attention over Instances” ACL (2016)
 Sentence-level ATT results
NYT10 Dataset
Align Freebase relations with New
York Times corpus (NYT)
53 possible relationships
+NA (no relation between entities)

Data sentences entity pairs

Training 522,611 281,270

Test 172,448 96,678

Lin et al. “Neural Relation Extraction with Selective Attention over Instances” ACL (2016)
 References
 Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." Journal of machine
learning research (2014)
 Bergstra, James, and Yoshua Bengio. "Random search for hyper-parameter optimization." Journal of Machine
Learning Research, Feb (2012)
 Kim, Y. “Convolutional Neural Networks for Sentence Classification”, EMNLP (2014)
 Severyn, Aliaksei, and Alessandro Moschitti. "UNITN: Training Deep Convolutional Neural Network for
Twitter Sentiment Classification." SemEval@ NAACL-HLT (2015)
 Cho, Kyunghyun, et al. "Learning phrase representations using RNN encoder-decoder for statistical machine
translation." EMNLP (2014)
 Ilya Sutskever et al. “Sequence to sequence learning with neural networks.” NIPS (2014)
 Bahdanau et al. "Neural machine translation by jointly learning to align and translate." ICLR (2015)
 Gal, Y., Islam, R., Ghahramani, Z. “Deep Bayesian Active Learning with Image Data.” ICML (2017)
 Nair, V., Hinton, G.E. “Rectified linear units improve restricted boltzmann machines.” ICML (2010)
 Ronan Collobert, et al. “Natural language processing (almost) from scratch.” JMLR (2011)

 Kumar, Shantanu. "A Survey of Deep Learning Methods for Relation Extraction." arXiv preprint arXiv:1705.03645 (2017)
 Lin et al. “Neural Relation Extraction with Selective Attention over Instances” ACL (2016) [code]
 Zeng, D.et al. “Relation classification via convolutional deep neural network”. COLING (2014)
 Nguyen, T.H., Grishman, R. “Relation extraction: Perspective from CNNs.” VS@ HLT-NAACL. (2015)
 Zhang, D., Wang, D. “Relation classification via recurrent NN.” -arXiv preprint arXiv:1508.01006 (2015)
 Zhou, P. et al. “Attention-based bidirectional LSTM networks for relation classification . ACL (2016)
 Mike Mintz et al. “Distant supervision for relation extraction without labeled data.” ACL- IJCNLP (2009)
 References & Resources
 http://web.stanford.edu/class/cs224n
 https://www.coursera.org/specializations/deep-learning
 https://chrisalbon.com/#Deep-Learning
 http://www.asimovinstitute.org/neural-network-zoo
 http://cs231n.github.io/optimization-2
 https
://medium.com/@ramrajchandradevan/the-evolution-of-gradient-descend-optimization-algorithm-4106a6702d39
 https://arimo.com/data-science/2016/bayesian-optimization-hyperparameter-tuning
 http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow
 http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp
 https://
medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad
5df8

 http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/
 http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python
-and-theano/
 http://colah.github.io/posts/2015-08-Understanding-LSTMs
 https://github.com/hyperopt/hyperopt
 https://github.com/tensorflow/nmt
https://giphy.com/gifs/thanks-thank-you-thnx-3o6ozuHcxTtVWJJn32/download