Data Science

Unit-V
From: Tom M. Mitchell, Machine Learning, India Edition 2013, McGraw Hill
Education.

ARTIFICIAL NEURAL
NETWORKS 1
CONTEN
T
• Introduction
• Neural Network Representation
• Appropriate Problems for Neural Network Learning
• Perceptrons
• Multilayer Networks and BACKPROPAGATION Algorithms
• Remarks on the BACKPROPAGATION Algorithms
• Advanced Topics in Artificial Neural Networks.
• Case Study: Defeating a CAPTCHA

2
Introduction: How can we answer this problems?

• Pattern recognition: Does that image contain a face?

•
• Classification problems: Is this product defective?

• Prediction: Given these symptoms, the patient has disease X

• Forecasting: predicting behavior of stock market

• Handwriting: is character recognized?

• Optimization: Find the shortest path for the .

Neural Networks can be used to solve these problems

When to consider Neural Networks ?

• Input is a high-dimensional discrete or real-valued (e.g., sensor

input)
• Output is discrete or real-valued
• Output is a vector of values
• Possibly noisy data
• Form of target function is unknown
• Human readability of result is unimportant

Examples:
1. Speech phoneme recognition
2. Image classification
3. Financial perdition
4
INTRODUCTION

Artificial neural networks (ANNs) provide a general, practical method

for learning real-valued, discrete-valued, and vector-valued target
functions from examples.

5
Biological Motivation

• The study of artificial neural networks (ANNs) has been inspired by the
observation that biological learning systems are built of very complex webs of
interconnected Neurons
• Human information processing system consists of brain neuron: basic building
block cell that communicates information to and from various parts of body
• Simplest model of a neuron: considered as a threshold unit –a processing element
(PE)
• Collects inputs & produces output if the sum of the input exceeds an internal
threshold value

6
Facts of Human Neurobiology

• Number of neurons ~ 1011

• Connection per neuron ~ 10 4 – 5
• Neuron switching time ~ 0.001 second or 10 -3
• Scene recognition time ~ 0.1 second
• 100 inference steps doesn’t seem like enough
• Highly parallel computation based on distributed representation

Properties of Neuron:
• Many neuron-like threshold switching units
• Many weighted interconnections among units
• Highly parallel, distributed process
• Emphasis on tuning weights automatically
• Input is a high-dimensional discrete or real-valued (e.g, sensor input)
7
Neuron

8
What Is an Artificial Neural Network (ANN)?

• Artificial Neural Network is a Deep Learning model that draws inspiration

from the neural structure of the human brain.
• An artificial neural network (ANN) is the piece of a computing system
designed to simulate the way the human brain analyzes and processes
information.
• ANNs have self-learning capabilities that enable them to produce better
results as more data becomes available.

Structure of ANN:

9
Topological units of ANN:

 Neural Network.
 Is
a Mathematical model.
 Inspired by Human biological neurons and now leading in
solving many real world problems
 ANN topology Units include:
 Neurons.
 Layers.
 Connecting Links.
 Initial weights(on links)

and bias and activation function

 Inputs/outputs

10
Properties of ANNs

• Supports many neuron- which act like threshold switching units.

• Has many weighted interconnections among units, to improve the NN

learning process.

• Emphasis on tuning network weights automatically, to reduce the

learning error.

• supports robustness, non-linearity, high parallelism, fault and failure

tolerance, ability to handle imprecise and fuzzy information, and
generalizing ability.

• Input is a high-dimensional discrete or real-valued (e.g, sensor input)

11
ANN Architecture:
Bias
b
x1 w1
Activation
Local function
Field
Output
x2 w2 v
y
Input
values
Summing
function

xm wm
weights

12
Activation Functions

13
Activation Functions:

• Activation function is one of the building blocks on Neural Network

• Activation function is added for Hidden and O/P layers.
• The purpose of an activation function is to add some kind of non-linear
property to the input.
• Without the activation functions, the only mathematical operation during
the forward propagation would be dot-products between an input vector
and a weight matrix: which is linear
• The activation functions help the network use the important information
and suppress the irrelevant data points.
• Can we do without an activation function?
• A neural network without an activation function is essentially just a
linear regression model.

14
Why nonlinear Activation function: (ex: sigmoid)
• Linear functions always do exhibit a constant slope.
• Slope as we know pictures the rate of change of the predictor variable (y)
with respect to the independent variable (x).
• Real-world problems emphasize on finding the points of minimal or
maximal change: may be minimization of loss or maximization of gain.
• If we take linear activation functions, these changes are constant and
cannot be distinguishable between points of minimal and maximal
changes.
• Nonlinear activation functions have gradients that vary between various
points.
• Based on these gradients descent or accent, we can address all the
minimization and maximization problems.
.

15
Why nonlinear Activation function: (ex: sigmoid)

• The nonlinear exponential function eX has a range between [0, ∞] and has
an infinite range.
• The gradients for the function eX exist between [0, ∞].
• To find the optimal gradients, it is quite difficult within this infinite range.
• Hence, such functions though nonlinear cannot be taken as activation
functions.
• Hence, the needs for activation functions which show active gradients
within shorter intervals are needed.
• One such is the sigmoid activation function which will analyze the
gradients between a small [0, 1] interval

16
• Input to neuron: x = (weight * input) + bias

Post that, an activation function is applied on the above result, x

• Imagine a neural network without the activation functions.

• In that case, every neuron will only be performing a linear transformation on the
inputs using the weights and biases.
• Although linear transformations make the neural network simpler, but this
network would be less powerful and will not be able to learn the complex
patterns from the data.
• Thus we use a non linear transformation to the inputs of the neuron and his non-
linearity in the network is introduced by an activation function.

17
Role of Bias:

18
Types of Activation functions

19
Types of Activation functions

20
Neural Networks representation

21
• Figure illustrates the neural network representation.
• The network is shown on the left side of the figure, with the input camera image depicted below it.
• Each node (i.e., circle) in the network diagram corresponds to the output of a
single network unit, and the lines entering the node from below are its inputs.
• There are four units that receive inputs directly from all of the 30 x 32 pixels in the image. These are called
"hidden" units because their output is available only within the network and is not available as part of the global
network output. Each of these four hidden units computes a single real-valued output based on a weighted
combination of its 960 inputs
• These hidden unit outputs are then used as inputs to a second layer of 30 "output"
units.
• Each output unit corresponds to a particular steering direction, and the output values of these units determine
which steering direction is recommended most strongly.
• The diagrams on the right side of the figure depict the learned weight values associated with one of the
four hidden units in this ANN.
• The large matrix of black and white boxes on the lower right depicts the weights from the 30 x 32
pixel inputs into the hidden unit. Here, a white box indicates a positive weight, a black box a negative weight,
and the size of the box indicates the weight magnitude.
• The smaller rectangular diagram directly above the large matrix shows the weights from this hidden unit
to each of the 30 output units.

22
APPROPRIATE PROBLEMS FOR NEURAL NETWORK LEARNING

ANN is appropriate for problems with the following characteristics :

• Instances are represented by many attribute-value pairs.
• The target function output may be discrete-valued, real-valued, or a vector
of several real- or discrete-valued attributes.
• The training examples may contain errors.
• Long training times are acceptable.
• Fast evaluation of the learned target function may be required
• The ability of humans to understand the learned target function is not important

23
Architectures of Artificial Neural Networks

An artificial neural network can be divided into three parts (layers), which are
known as:
• Input layer: This layer is responsible for receiving information (data), signals,
features, or measurements from the external environment. These inputs are usually
normalized within the limit values produced by activation functions
• Hidden, intermediate, or invisible layers: These layers are composed of neurons
which are responsible for extracting patterns associated with the process or system
being analysed. These layers perform most of the internal processing from a
network.
• Output layer : This layer is also composed of neurons, and thus is responsible for
producing and presenting the final network outputs, which result from the
processing performed by the neurons in the previous layers.
24
 Architectures of Artificial Neural Networks

The main architectures of artificial neural networks, considering the neuron

disposition, how they are interconnected and how its layers are composed, can be
divided as follows:

1. Single-layer feedforward network

2. Multi-layer feedforward networks
3. Recurrent or Feedback networks
4. Mesh networks

25
Single-Layer Feedforward Architecture

• This artificial neural network has just one input layer and a single neural layer, which is also the
output layer.
• Figure illustrates a simple-layer feedforward network composed of n inputs and m outputs.
• The information always flows in a single direction (thus, unidirectional), which is from the input
layer to the output layer

26
Multi-Layer Feedforward Architecture
• This artificial neural feedforward networks with multiple layers are composed of one or more
hidden neural layers.
• Figure shows a feedforward network with multiple layers composed of one input layer with n
sample signals, two hidden neural layers consisting of n1 and n2 neurons respectively, and, finally,
one output neural layer composed of m neurons representing the respective output values of the
problem being analyzed.

27
Recurrent or Feedback Architecture
• In these networks, the outputs of the neurons are used as feedback inputs for other neurons.
• Figure illustrates an example of a Perceptron network with feedback, where one of its
output
signals is fed back to the middle layer.

28
Mesh Architectures

• The main features of networks with mesh structures reside in considering the spatial arrangement
of neurons for pattern extraction purposes, that is, the spatial localization of the neurons is directly
related to the process of adjusting their synaptic weights and thresholds.
• Figure illustrates an example of the Kohonen network where its neurons are arranged within a two-
dimensional space

29
PERCEPTRONS
• Perceptron is a single layer neural network.
• A perceptron takes a vector of real-valued inputs, calculates a linear combination
of these inputs, then outputs a 1 if the result is greater than some threshold and -
1 otherwise
• Given inputs x1 through xn, the output O(x1, . . . , xn) computed by the perceptron
is

• where each wi is a real-valued constant, or weight, that determines the contribution

of input xi to the perceptron output.
• -w0 is a threshold that the weighted combination of inputs w1x1 + . . . + wnxn must
surpass in order for the perceptron to output a 1.

30
Sometimes, the perceptron function is written
as,

Learning a perceptron involves choosing values for the weights w0 , . . . , wn .

Therefore, the space H of candidate hypotheses considered in perceptron learning
is the set of all possible real-valued weight vectors

Why do we need Weights and Bias?

Weights shows the strength of the particular node.
A bias value allows you to shift the activation function curve up or down
31
32
33
Representational Power of Perceptrons

*The perceptron can be

viewed as representing a
hyperplane decision surface
in the n-dimensional space
of instances.
*The perceptron outputs a 1
for instances lying on one
side of the hyperplane and
outputs a -1 for instances
lying on the other side

34
The Perceptron Training Rule

The learning problem is to determine a weight vector that causes the perceptron to
produce the correct + 1 or - 1 output for each of the given training examples.

To learn an acceptable weight vector

• Begin with random weights, then iteratively apply the perceptron to each training
example, modifying the perceptron weights whenever it misclassifies an example.
• This process is repeated, iterating through the training examples as many times as
needed until the perceptron classifies all training examples correctly.
• Weights are modified at each step according to the perceptron training
rule, which revises the weight wi associated with input xi according to the rule.

35
• The role of the learning rate is to moderate the degree to which weights are
changed at each step. It is usually set to some small value (e.g., 0.1) and is
sometimes made to decay as the number of weight-tuning iterations increases
Drawback: The perceptron rule finds a successful weight vector when the training
examples are linearly separable, it can fail to converge if the examples are not
linearly separable. This can overcome with Gradient decent Delta rule
36
Perceptron Algorithm

37
38
39
40
41
42
Design perceptron model for Exclusive OR Gate

43
44
45
46
47
48
49
Exercises:

Design perceptron model for

1.A implies B (Negation A OR B)

2.Negation A
3.Negation A AND B

50
Gradient Descent and the Delta Rule

51
Gradient Descent and the Delta Rule

• If the training examples are not linearly separable, the

delta rule converges toward a best-fit approximation to the
target concept.
• OR,AND=Linearly separable, XOR = not linearly separable
• The key idea behind the delta rule is to use gradient descent
to search the hypothesis space of possible weight vectors to
find the weights that best fit the training examples.
To understand the delta training rule, consider the task of
training an un thresholded perceptron. That is, a linear unit for
which the output O is given by

52
Gradient Descent and the Delta Rule

To derive a weight learning rule for linear units, specify a measure for the training
error of a hypothesis (weight vector), relative to the training examples.

Where,
• D is the set of training examples,
• td is the target output for training example d,
• od is the output of the linear unit for training example d
• E [ w ] is simply half the squared difference between the target
output td and the linear unit output od, summed over all training
examples.

53
Visualizing the Hypothesis Space

• To understand the gradient descent algorithm, it is helpful to visualize the entire

hypothesis space of possible weight vectors and their associated E values as
shown in below figure.
• Here the axes w0 and wl represent possible values for the two weights of a simple
linear unit. The w0, wl plane therefore represents the entire hypothesis space.
• The vertical axis indicates the error E relative to some fixed set of training
examples.
• The arrow shows the negated gradient at one particular point, indicating
the direction in the w0, wl plane producing steepest descent along the error
surface.
• The error surface shown in the figure thus summarizes the desirability of every
weight vector in the hypothesis space

54
Visualizing the Hypothesis Space

• Given the way in which we chose to define E, for linear units this error surface must always
be parabolic with a single global minimum.
Gradient descent search determines a weight vector that minimizes E by starting with an
arbitrary
initial weight vector, then repeatedly modifying it in small steps.
At each step, the weight vector is altered in the direction that produces the steepest descent along
the error surface depicted in above figure. This process continues until the global minimum error is
55
reached.
Derivation of the Gradient Descent Rule

How to calculate the direction of steepest descent along the error surface?
The direction of steepest can be found by computing the derivative of E with respect
to each component of the vector w . This vector derivative is called the gradient of
E with respect to w , written as

56
• The gradient specifies the direction of steepest increase of E, the training rule for
gradient descent is

• Here η is a positive constant called the learning rate, which determines the
step
size in the gradient descent search.
• The negative sign is present because we want to move the weight vector in
the direction that decreases E
• This training rule can also be written in its component form

57
 𝜕𝐸
Calculate the gradient at each step. The vector derivatives that form the gradient can be obtained
𝜕𝑤𝑖
of differentiating E from Equation (2), as by

58
GRADIENT DESCENT algorithm for training a linear unit

To summarize, the gradient descent algorithm for training linear units is as follows:
• Pick an initial random weight vector.
• Apply the linear unit to all training examples, then compute Δwi for each weight according to Equation (7).
• Update each weight wi by adding Δwi, then repeat this process 59
Features of Gradient Descent Algorithm

Gradient descent is an important general paradigm for learning. It is a strategy for

searching through a large or infinite hypothesis space that can be applied whenever
1. The hypothesis space contains continuously parameterized hypotheses
2. The error can be differentiated with respect to these hypothesis parameters

The key practical difficulties in applying gradient descent are

3. Converging to a local minimum can sometimes be quite slow
4. If there are multiple local minima in the error surface, then there is no guarantee
that the procedure will find the global minimum

60
Stochastic Approximation to Gradient Descent

• The gradient descent training rule presented in Equation (7) computes weight
updates after summing over all the training examples in D
• The idea behind stochastic gradient descent is to approximate this gradient descent
search by updating weights incrementally, following the calculation of the error
for each individual example

where t, o, and xi are the target value, unit output, and ith input for the training
example in question

61
One way to view this stochastic gradient descent
is to consider a distincterror function Ed ( w ) for each
individual training example d as follows

Where, td and od are the target value and the unit output value for training example d.
• Stochastic gradient descent iterates over the training examples d in D, at each
iteration altering the weights according to the gradient with respect to Ed( w )
• The sequence of these weight updates, when iterated over all training examples,
provides a reasonable approximation to descending the gradient with respect to
our original error function Ed ( w )
• By making the value of η sufficiently small, stochastic gradient descent can
be made to approximate true gradient descent arbitrarily closely

62
The key differences between standard gradient descent and stochastic gradient
descent are
• In standard gradient descent, the error is summed over all examples before
updating weights, whereas in stochastic gradient descent weights are updated upon
examining each training example.
• Summing over multiple examples in standard gradient descent requires more
computation per weight update step. On the other hand, because it uses the true
gradient, standard gradient descent is often used with a larger step size per weight
update than stochastic gradient descent.
• In cases where there are multiple local minima with respect to stochastic gradient
descent can sometimes avoid falling into these local minima because it uses the
various 𝛻Ed ( w ) rather than 𝛻E ( w ) to guide its search

63
64
Types of Gradient descent

Batch Gradient Descent

In batch gradient descent, we use the complete dataset available to compute the
gradient of the cost function. Batch gradient descent is very slow because we need
to calculate the gradient on the complete dataset to perform just one update, and if
the dataset is large then it will be a difficult task.
1.Cost function is calculated after the initialization of parameters.
2.It reads all the records into memory from the disk.
3.After calculating sigma for one iteration, we move one step further, and repeat
the process.
65
Mini-batch Gradient Descent
It is a widely used algorithm that makes faster and accurate results. The dataset,
here, is clustered into small groups of ‘n’ training datasets. It is faster because it
does not use the complete dataset. In every iteration, we use a batch of ‘n’
training datasets to compute the gradient of the cost function. It reduces the
variance of the parameter updates, which can lead to more stable convergence. It
can also make use of a highly optimized matrix that makes computing of the
gradient very efficient.

Stochastic Gradient Descent

We use stochastic gradient descent for faster computation. The first step is to
randomize the complete dataset. Then, we use only one training example in every
iteration to calculate the gradient of the cost function for updating every
parameter. It is faster for larger datasets also because it uses only one training
example in each iteration.
66
67
What is CAPTCHA?
CAPTCHA is no longer an alien term to web users. It’s the annoying human validation check added on
many websites. An acronym for Completely Automated Public Turing test to tell Computers and
Humans Apart. CAPTCHA can be defined as a computer program developed to distinguish between the
human and machine to prevent any type of illicit activity on websites. The assumption that underlies the
concept of CAPTCHA is that only a human would pass this test and bot or automated scripts would fail

Why do we need to break CAPTCHA?

Now, people use automated CAPTCHA solving for different use cases, some of these are illegal while
others are legitimate purposes. Spammers use CAPTCHA solving to extract email addresses of users to
be able to generate as many spams as possible. The legit examples are scenarios where a new client or
business partner has joined you and needs access to your Application Programming Interface(API)
which is not ready or can’t be shared with due to some security issue or abuse it may give rise to. Thus,
we are only left with CAPTCHA bypassing using automated scripts.
There are different types of CAPTCHA, text-based CAPTCHA, image-based CAPTCHA, reCAPTCHA,
and mathematical CAPTCHA. Solving one can get challenging sometimes as the technology used in
CAPTCHA and reCAPTCHA is getting smarter as we speak with updates coming at regular intervals.

68
69