2/12/24, 8:24 AM Transfer Learning Alexnet.

ipynb - Colaboratory

keyboard_arrow_down Transfer Learning Alexnet using Keras

Why does AlexNet achieve better results?

1. Relu activation function is used.

Relu function: f (x) = max (0, x)

alex1

ReLU-based deep convolutional networks are trained several times faster than tanh and sigmoid-
based networks. The following figure shows the number of iterations for a four-layer
convolutional network based on CIFAR-10 that reached 25% training error in tanh and ReLU:

alex1

2. Standardization ( Local Response Normalization )

After using ReLU f (x) = max (0, x), you will find that the value after the activation function has no
range like the tanh and sigmoid functions, so a normalization will usually be done after ReLU,
and the LRU is a steady proposal (Not sure here, it should be proposed?) One method in
neuroscience is called "Lateral inhibition", which talks about the effect of active neurons on its
surrounding neurons.

alex1

3. Dropout

Dropout is also a concept often said, which can effectively prevent overfitting of neural networks.
Compared to the general linear model, a regular method is used to prevent the model from
overfitting. In the neural network, Dropout is implemented by modifying the structure of the
neural network itself. For a certain layer of neurons, randomly delete some neurons with a
defined probability, while keeping the individuals of the input layer and output layer neurons
unchanged, and then update the parameters according to the learning method of the neural
network. In the next iteration, rerandom Remove some neurons until the end of training.

alex1

4. Enhanced Data ( Data Augmentation )

In deep learning, when the amount of data is not large enough, there are generally 4 solutions:

Data augmentation- artificially increase the size of the training set-create a batch
of "new" data from existing data by means of translation, flipping, noise

Regularization——The relatively small amount of data will cause the model to

overfit, making the training error small and the test error particularly large. By
https://colab.research.google.com/drive/1RdWJDFMqC0DWAQMNyXduPe-ySuL-odUx#printMode=true 1/5
2/12/24, 8:24 AM Transfer Learning Alexnet.ipynb - Colaboratory

adding a regular term after the Loss Function , the overfitting can be suppressed.
The disadvantage is that a need is introduced Manually adjusted hyper-parameter.

Dropout- also a regularization method. But different from the above, it is achieved
by randomly setting the output of some neurons to zero

Unsupervised Pre-training- use Auto-Encoder or RBM's convolution form to do

unsupervised pre-training layer by layer, and finally add a classification layer to do
supervised Fine-Tuning

keyboard_arrow_down Code Of Alexnet Structure Using Keras

import tensorflow as tf

tf.__version__

'2.2.0'

https://colab.research.google.com/drive/1RdWJDFMqC0DWAQMNyXduPe-ySuL-odUx#printMode=true 2/5
2/12/24, 8:24 AM Transfer Learning Alexnet.ipynb - Colaboratory

import tensorflow.keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout, Flatten,\
Conv2D, MaxPooling2D,BatchNormalization

# (3) Create a sequential model

model = Sequential()

# 1st Convolutional Layer

model.add(Conv2D(filters=96, input_shape=(227,227,3), kernel_size=(11,11),\
strides=(4,4), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# 2nd Convolutional Layer

model.add(Conv2D(filters=256, kernel_size=(11,11), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# 3rd Convolutional Layer

model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# 4th Convolutional Layer

model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# 5th Convolutional Layer

model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Passing it to a dense layer

model.add(Flatten())
# 1st Dense Layer
model.add(Dense(4096, input_shape=(224*224*3,)))
model.add(Activation('relu'))
# Add Dropout to prevent overfitting
model.add(Dropout(0.4))
https://colab.research.google.com/drive/1RdWJDFMqC0DWAQMNyXduPe-ySuL-odUx#printMode=true 3/5
2/12/24, 8:24 AM Transfer Learning Alexnet.ipynb - Colaboratory

# Batch Normalisation
model.add(BatchNormalization())

# 2nd Dense Layer

model.add(Dense(4096))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# 3rd Dense Layer

model.add(Dense(1000))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(17))
model.add(Activation('softmax'))

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 54, 54, 96) 34944
_________________________________________________________________
activation (Activation) (None, 54, 54, 96) 0
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 27, 27, 96) 0
_________________________________________________________________
batch_normalization (BatchNo (None, 27, 27, 96) 384
_________________________________________________________________
conv2d_1 (Conv2D) (None, 17, 17, 256) 2973952
_________________________________________________________________
activation_1 (Activation) (None, 17, 17, 256) 0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 8, 8, 256) 0
_________________________________________________________________
batch_normalization_1 (Batch (None, 8, 8, 256) 1024
_________________________________________________________________
conv2d_2 (Conv2D) (None, 6, 6, 384) 885120
_________________________________________________________________
activation_2 (Activation) (None, 6, 6, 384) 0
_________________________________________________________________
batch_normalization_2 (Batch (None, 6, 6, 384) 1536
_________________________________________________________________
conv2d_3 (Conv2D) (None, 4, 4, 384) 1327488
_________________________________________________________________
https://colab.research.google.com/drive/1RdWJDFMqC0DWAQMNyXduPe-ySuL-odUx#printMode=true 4/5
2/12/24, 8:24 AM Transfer Learning Alexnet.ipynb - Colaboratory
activation_3 (Activation) (None, 4, 4, 384) 0
_________________________________________________________________
batch_normalization_3 (Batch (None, 4, 4, 384) 1536
_________________________________________________________________
conv2d_4 (Conv2D) (None, 2, 2, 256) 884992
_________________________________________________________________
activation_4 (Activation) (None, 2, 2, 256) 0
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 1, 1, 256) 0
_________________________________________________________________
batch_normalization_4 (Batch (None, 1, 1, 256) 1024
_________________________________________________________________
flatten (Flatten) (None, 256) 0
_________________________________________________________________
dense (Dense) (None, 4096) 1052672
_________________________________________________________________
activation_5 (Activation) (None, 4096) 0
_________________________________________________________________
dropout (Dropout) (None, 4096) 0
_________________________________________________________________
batch_normalization_5 (Batch (None, 4096) 16384

https://colab.research.google.com/drive/1RdWJDFMqC0DWAQMNyXduPe-ySuL-odUx#printMode=true 5/5