Introduction to AutoEncoders

• Autoencoder is a special type of neural

network where the output layer has the
same dimensionality as the input layer.
• Autoencoders are used to reduce the
size of our inputs into a smaller
representation.
• The number of output units in the
output layer is equal to the number of
input units in the input layer.
• An autoencoder replicates the data
from the input to the output in an
unsupervised manner and is therefore
sometimes referred to as a replicator
neural network.
Architecture of Autoencoders

• Data-specific &Lossy
• HyperparametersCode size, Number of layers, Number of nodes
per layer & Loss function
Hyperparameters
• Code size
• Number of layers
• Number of nodes per layer
• Loss function
Applications contd.,
• Data Compression
• Feature Extraction
• Image Generation
• Image colourisation
• Watermark removal
AutoEncoder-Code Segment for data compression
import numpy as np encoder = keras.Model(input_img, encoded)
from keras.layers import Input, Dense encoded_input = Input(shape=(encoding_dim,))
from keras.models import Model decoder_layer = autoencoder.layers[-1]
from keras.datasets import mnist decoder = Model(encoded_input,
decoder_layer(encoded_input))
import matplotlib.pyplot as plt
autoencoder.compile(optimizer=‘Adam',
encoding_dim = 32 # 32 floats -> compression of loss='binary_crossentropy')
factor 24.5, assuming the input is 784 floats
(x_train, _), (x_test, _) = mnist.load_data()
input_img = Input(shape=(784,))
x_train = x_train.astype('float32') / 255.
encoded = Dense(encoding_dim,
activation='relu')(input_img) x_test = x_test.astype('float32') / 255.
decoded = Dense(784, x_train = x_train.reshape((len(x_train),
activation='sigmoid')(encoded) np.prod(x_train.shape[1:])))
autoencoder = Keras.Model(input_img, x_test = x_test.reshape((len(x_test),
decoded) np.prod(x_test.shape[1:])))
print x_train.shape
print x_test.shape
autoencoder.fit(x_train, x_train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
ax = plt.subplot(2, n, i + 1 + n)
plt.imshow(decoded_imgs[i].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()
Output
Sparse Autoencoders
Types of AutoEncoders- Convolution
AutoEncoders
Dimensionality Reduction Denoising Image
Convolution Operation
Transpose Convolution
Cont.,
Cont.,
AutoEncoders to color the
gray-scale images
AutoEncoders to color the gray-scale images
• an Auto-encoder is a neural h-hidden layer
network that is trained to h=f(x) -encoder function
attempt to copy its input to its
output. r=g(h)- decoder function
• encoder — transforms the
input data into a lower- The goal of the convolutional
dimensional representation. layers is to extract potential
• decoder — recovers the input features by applying convolutional
from the low-dimensional filtering.
representation.
Example
Test Images and Results
Generative Adversarial
Networks (GANs)
Introduction
Generative modeling is an unsupervised learning approach.
It was developed and introduced by Ian J. Goodfellow in 2014.
It automatically discovers and learn patterns in input data.
The model can be used to generate new examples from the original
dataset.
Generative: To learn a generative model, which describes how data is
generated in terms of a probabilistic model.
Adversarial: The training of a model is done in an adversarial setting.
Networks: Use deep neural networks as the artificial intelligence (AI)
algorithms for training purpose.
Cont.,
• Generator: It is trained to generate new dataset, for example in
computer vision it generates new images from existing real world
images.
• Discriminator: It compares those images with some real world
examples and classify real and fake images.
Types of GAN
1) DC GAN- Deep Convolutional Generative Adversarial Networks
2) Conditional GAN and Unconditional GAN (CGAN)
3) Least Square GAN(LSGAN)
4) Cycle GAN
Fake Image Generated
After 10 epochs
After 90 epochs
After 200 epochs
from numpy import zeros, ones, expand_dims,
asarray (X_train, _), (_, _) = fashion_mnist.load_data()
from numpy.random import randn, randint X_train = X_train.astype(np.float32) / 127.5 - 1
from keras.datasets import fashion_mnist X_train = np.expand_dims(X_train, axis=3)
from keras.optimizers import Adam print(X_train.shape)
from keras.models import Model, load_model def generate_latent_points(latent_dim,
from keras.layers import Input, Dense, Reshape, n_samples):
Flatten x_input = randn(latent_dim * n_samples)
from keras.layers import Conv2D, z_input = x_input.reshape(n_samples,
Conv2DTranspose, Concatenate latent_dim)
from keras.layers import LeakyReLU, Dropout, return z_input
Embedding
def generate_fake_samples(generator,
from keras.layers import BatchNormalization, latent_dim, n_samples):
Activation
z_input = generate_latent_points(latent_dim,
from keras import initializers n_samples)
from keras.initializers import RandomNormal images = generator.predict(z_input)
from keras.optimizers import Adam, RMSprop, y = zeros((n_samples, 0))
SGD
return images, y
from matplotlib import pyplot
import numpy as np
def summarize_performance(step, def save_plot(examples, n_examples):
g_model, latent_dim, n_samples=100): for i in range(n_examples):
X, _ = generate_fake_samples(g_model, pyplot.subplot(sqrt(n_examples),
latent_dim, n_samples) sqrt(n_examples), 1 + i)
X = (X + 1) / 2.0 pyplot.axis('off')
for i in range(100): pyplot.imshow(examples[i, :, :, 0],
pyplot.subplot(10, 10, 1 + i) cmap='gray_r')
pyplot.axis('off') pyplot.show()
pyplot.imshow(X[i, :, :, 0],
cmap='gray_r')
filename2 = 'model_%04d.h5' % (step+1)
g_model.save(filename2)
print('>Saved: %s' % (filename2))
 def define_generator(latent_dim):
def define_discriminator(in_shape=(28, 28, 1)): init = RandomNormal(stddev=0.02)
init = RandomNormal(stddev=0.02) in_lat = Input(shape=(latent_dim,))
in_image = Input(shape=in_shape) gen = Dense(256, kernel_initializer=init)(in_lat)
fe = Flatten()(in_image) gen = LeakyReLU(alpha=0.2)(gen)
fe = Dense(1024)(fe) gen = Dense(512, kernel_initializer=init)(gen)
fe = LeakyReLU(alpha=0.2)(fe) gen = LeakyReLU(alpha=0.2)(gen)
fe = Dropout(0.3)(fe) gen = Dense(1024, kernel_initializer=init)(gen)
fe = Dense(512)(fe) gen = LeakyReLU(alpha=0.2)(gen)
fe = LeakyReLU(alpha=0.2)(fe) gen = Dense(28 * 28 * 1, kernel_initializer=init)(gen)
fe = Dropout(0.3)(fe) out_layer = Activation('tanh')(gen)
fe = Dense(256)(fe) out_layer = Reshape((28, 28, 1))(gen)
fe = LeakyReLU(alpha=0.2)(fe) model = Model(in_lat, out_layer)
fe = Dropout(0.3)(fe) return model
out = Dense(1, activation='sigmoid')(fe) generator = define_generator(100)
model = Model(in_image, out) def define_gan(g_model, d_model):
opt = Adam(lr=0.0002, beta_1=0.5) d_model.trainable = False
model.compile(loss='binary_crossentropy', optimizer=opt, gan_output = d_model(g_model.output)
metrics=['accuracy'])
return model model = Model(g_model.input, gan_output)

discriminator = define_discriminator() opt = Adam(lr=0.0002, beta_1=0.5)

model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
return model
gan_model = define_gan(generator, discriminator)
def define_gan(g_model, d_model): def train(g_model, d_model, gan_model, X_train, latent_dim,
n_epochs=100, n_batch=64):
d_model.trainable = False
bat_per_epo = int(X_train.shape[0] / n_batch)
gan_output = d_model(g_model.output)
n_steps = bat_per_epo * n_epochs
model = Model(g_model.input, gan_output)
for i in range(n_steps):
opt = Adam(lr=0.0002, beta_1=0.5)
X_real, y_real = generate_real_samples(X_train, n_batch)
model.compile(loss='binary_crossentropy', optimizer=opt,
metrics=['accuracy']) d_loss_r, d_acc_r = d_model.train_on_batch(X_real, y_real)
return model X_fake, y_fake = generate_fake_samples(g_model, latent_dim,
n_batch)
gan_model = define_gan(generator, discriminator)
d_loss_f, d_acc_f = d_model.train_on_batch(X_fake, y_fake)
z_input = generate_latent_points(latent_dim, n_batch)
y_gan = ones((n_batch, 1))
g_loss, g_acc = gan_model.train_on_batch(z_input, y_gan)
print('>%d, dr[%.3f,%.3f], df[%.3f,%.3f], g[%.3f,%.3f]' % (i+1,
d_loss_r,d_acc_r, d_loss_f,d_acc_f, g_loss,g_acc))
if (i+1) % (bat_per_epo * 1) == 0:
summarize_performance(i, g_model, latent_dim)
Output
MNIST Digit Dataset
def genG_model():
modelG = tf.keras.Sequential()
modelG.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
modelG.add(layers.BatchNormalization())
modelG.add(layers.LeakyReLU())
modelG.add(layers.Reshape((7, 7, 256)))
assert modelG.output_shape == (None, 7, 7, 256) # Note: None is the batch size
modelG.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
assert modelG.output_shape == (None, 7, 7, 128)
modelG.add(layers.BatchNormalization())
modelG.add(layers.LeakyReLU())
modelG.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
assert modelG.output_shape == (None, 14, 14, 64)
modelG.add(layers.BatchNormalization())
modelG.add(layers.LeakyReLU())
modelG.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False,
activation='tanh'))
assert modelG.output_shape == (None, 28, 28, 1)
return modelG
# Input to discriminator = 28*28*1 grayscale image
# Output binary prediction (image is real (class=1) or fake (class=0))
# no pooling layers
# single node in the output layer with the sigmoid activation function to predict whether the input sample is
real or fake.
# Downsampling from 28×28 to 14×14, then to 7×7, before the model makes an output prediction
def make_discriminator_model():
modelD = tf.keras.Sequential()
modelD.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same',input_shape=[28, 28, 1])) #2×2 stride to
downsample
modelD.add(layers.LeakyReLU())
modelD.add(layers.Dropout(0.3))
modelD.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same')) #downsampling 2×2 stride to
downsample
modelD.add(layers.LeakyReLU())
modelD.add(layers.Dropout(0.3))
modelD.add(layers.Flatten()) # classifier real (class=1) or fake (class=0))
modelD.add(layers.Dense(1, activation='sigmoid'))
return modelD
model.add(layers.Flatten()) # classifier real (class=1) or fake (class=0))
model.add(layers.Dense(1, activation='sigmoid'))
return model
Output