Boltzmann Machines

A Boltzmann machine is a type of neural network that is used for unsupervised learning. It is a
probabilistic model that can be used to represent the joint probability distribution of a set of
variables. Boltzmann machines are made up of a set of nodes where every node is connected
to every other node and the nodes are undirected.
Boltzmann Machine is not a deterministic DL model but a stochastic or generative DL model
(which have one of the two possible states, either 1 or 0.).

At the core of a Boltzmann Machine is a network of stochastic binary units called neurons or
nodes. These nodes are organized into two main layers: visible units and hidden units. The
visible units (which we can measure) correspond to the observed variables, while the hidden
units (which we can not measure) capture the latent variables that influence the visible units.

The behavior of a Boltzmann Machine is governed by an energy function, which assigns an

energy value to each possible configuration of the units. The energy function is defined by the
weights associated with the connections between the units. Once the system is trained and the
weights are set, the system always tries to find the lowest energy state for itself by adjusting the
weights.

Although the node types are different, the Boltzmann machine considers them as the same and
everything works as one single system.

The training data is fed into the Boltzmann Machine and the weights of the system are adjusted
accordingly. Boltzmann machines help us understand abnormalities by learning about the
working of the system in normal conditions.

During training, Boltzmann Machines learn to model the joint probability distribution over the
visible and hidden units. Once trained, they can generate new samples by iteratively updating
the state of the units based on the learned probabilities. The generated samples can be used for
tasks such as data generation, denoising, or dimensionality reduction.

Boltzmann Distribution is used in the sampling distribution of the Boltzmann Machine.

Boltzmann Distribution describes different states of the system and thus Boltzmann machines
create different states of the machine using this distribution.

Deep Belief Networks

We create Deep Belief Networks (DBNs) to address issues with classic neural networks in deep
layered networks. For example – slow learning, becoming stuck in local minima owing to poor
parameter selection, and requiring a large number of training datasets.
Deep Belief Networks (DBNs) are a class of generative models that combine the power of deep
architectures and the learning principles of Boltzmann Machines. DBNs were introduced by
Geoffrey Hinton and his colleagues in the mid-2000s as a way to learn hierarchical
representations from unlabeled data.

A DBN consists of multiple layers of Restricted Boltzmann Machines (RBMs) stacked on top of
each other. The RBMs in a DBN are trained in a greedy layer-wise manner. Each RBM learns to
model the distribution of its layer given the activities of the layer below it. This unsupervised
pre-training phase initializes the weights of the DBN to capture useful features in the data.

Once the RBMs are pre-trained, a process called fine-tuning is performed to adjust the weights
of the entire network using a supervised learning algorithm, such as backpropagation. During
fine-tuning, labeled data is used to train the DBN as a deep neural network, enabling it to learn
discriminative representations that are useful for classification or regression tasks.

The key idea behind DBNs is that by combining multiple layers of RBMs, the network can learn
increasingly abstract and hierarchical representations of the input data. Lower layers capture
low-level features, such as edges or textures, while higher layers capture more complex and
abstract features that are built upon the lower-level representations.

The generative process in a DBN involves initializing the visible units and then sampling the
activities of the hidden units in each layer, from the bottom layer to the top. This process allows
the DBN to generate new samples by propagating activity patterns up the layers, and sampling
from the learned distributions at each layer.

DBNs have been successful in various applications, including image recognition, speech
recognition, natural language processing, and recommendation systems. The hierarchical
nature of DBNs enables them to automatically learn useful representations from raw data, which
can lead to better performance on tasks that involve complex and high-dimensional inputs.

Advantage:
- DBNs can learn features from the data in an unsupervised manner
- DBNs have proven to be resistant to overfitting
- DBNs can also be employed for generative activities like the creation of text and images.
- They can learn hierarchical representations of data, which can be used to improve the
performance of tasks such as classification and regression.

Generative Adversarial Network (GAN)

A Generative Adversarial Network (GAN) is a deep learning architecture that consists of two
neural networks, a generator and a discriminator, which are trained together in a competitive
setting. The goal of GANs is to generate new, synthetic data that resembles some known data
distribution.
Generative Adversarial Networks (GANs) can be broken down into three parts:
- Generative: To learn a generative model, which describes how data is generated in
terms of a probabilistic model.
- Adversarial: The training of a model is done in an adversarial setting.
- Networks: Use deep neural networks as artificial intelligence (AI) algorithms for training
purposes.
The generator network in a GAN is responsible for producing new synthetic samples. It takes as
input random noise or a latent vector and transforms it into a sample that resembles the training
data. The goal of the generator is to learn to generate samples that are indistinguishable from
the real data.

The discriminator network, on the other hand, is a binary classifier that distinguishes between
real and generated samples. It receives both real data samples from the training dataset and
generated samples from the generator. The discriminator is trained to correctly classify the
samples as real or fake.

The training process of a GAN involves a game-like scenario where the generator and
discriminator networks compete against each other. The generator tries to produce samples that
the discriminator cannot distinguish from real data, while the discriminator tries to improve its
ability to differentiate between real and fake samples.

During training, the generator and discriminator networks are updated alternately. First, the
discriminator is trained on a batch of real and generated samples, adjusting its weights to
improve its discrimination capability. Then, the generator is trained to fool the discriminator by
generating samples that are more realistic, updating its weights to produce better samples.

This back-and-forth training process continues iteratively, with the hope that the generator
improves over time, generating samples that are increasingly difficult for the discriminator to
differentiate. The ultimate goal is for the generator to learn the underlying distribution of the
training data and generate realistic samples that capture its characteristics.

The GANs are formulated as a minimax game, where the Generator is trying to minimize the
Discriminator’s reward or in other words, maximize its loss. It is given formula: min max V(D, G)
Discriminator Network
The discriminator network, also known as the critic, is a key component of a Generative
Adversarial Network (GAN). It is responsible for distinguishing between real and generated
samples, acting as a binary classifier. The primary objective of the discriminator is to learn to
accurately classify whether a given input sample is real (drawn from the training dataset) or fake
(generated by the generator network).

The discriminator network typically consists of one or more layers of artificial neurons, which
take the input sample and produce an output indicating the probability that the sample is real or
fake. The output is often a single scalar value, representing the probability of being real (usually
in the range of 0 to 1). For example, a value close to 0.9 may indicate a high probability of being
real, while a value close to 0.1 may suggest a high probability of being fake.

The training process of the discriminator involves providing it with labeled examples of both real
and generated samples. Real samples are drawn from the training dataset, while generated
samples are produced by the generator network. The discriminator is trained using supervised
learning techniques, aiming to optimize its parameters to correctly classify real and fake
samples.

During training, the discriminator's weights are adjusted using gradient descent optimization
algorithms, such as backpropagation, to minimize the classification error. It learns to identify
distinguishing features in the data that help it differentiate between real and fake samples. The
goal is for the discriminator to become increasingly accurate in classifying samples as the
training progresses.

The discriminator's role in a GAN is crucial for the overall training process. By providing
feedback to the generator network, it guides the generator to produce more realistic samples.
The generator's objective is to generate samples that fool the discriminator into classifying them
as real. As the generator improves, it poses a more challenging task for the discriminator,
leading to a competitive and adversarial training process.

The discriminator's architecture can vary depending on the specific GAN application. It can
range from simple feedforward neural networks to more complex architectures such as
convolutional neural networks (CNNs) for image-related tasks. The choice of architecture
depends on the nature of the input data and the complexity of the classification problem.
Generator Network
The generator network is a fundamental component of a Generative Adversarial Network
(GAN). It is responsible for producing synthetic samples that resemble the data from the training
set. The main goal of the generator is to learn the underlying distribution of the training data and
generate new samples that are indistinguishable from real data.

The generator network takes as input a random noise vector or a latent representation and
transforms it into a sample that resembles the training data. The latent vector acts as a random
seed that the generator uses to generate unique samples. The size and structure of the input
noise vector depend on the specific GAN architecture and the nature of the data being
generated.

The generator network is typically composed of one or more layers of artificial neurons, which
may include fully connected layers, convolutional layers, or other types of layers based on the
task and the type of data being generated. The structure and complexity of the generator
architecture can vary based on the complexity of the data and the desired level of detail in the
generated samples.

During training, the generator network is updated to improve its ability to generate realistic
samples. The training process is performed in an adversarial manner, where the generator
competes against the discriminator network. The generator's objective is to produce samples
that fool the discriminator into classifying them as real.

The Generator is trained while the Discriminator is idle. After the Discriminator is trained by the
generated fake data of the Generator, we can get its predictions and use the results for training
the Generator and get better from the previous state to try and fool the Discriminator.

As the GAN training progresses, the generator gradually improves its ability to produce
high-quality samples that resemble the training data. The generator's success is measured by
its ability to generate diverse and realistic samples that fool the discriminator and capture the
underlying distribution of the real data.

We train the generator with the following procedure:

- sample random noise
- produce generator output from sampled random noise.
 - Get discriminator "Real" or "Fake" classification of generator output.
- calculate loss from discriminator classification.
- Use gradients to change only the generator weights.

Different Types of GAN Models

1. Vanilla GAN: This is the simplest type of GAN. Here, the Generator and the
Discriminator are simple multi-layer perceptrons. In vanilla GAN, the algorithm is really
simple, it tries to optimize the mathematical equation using stochastic gradient descent.
2. Conditional GAN (CGAN): CGAN can be described as a deep learning method in which
some conditional parameters are put into place. In CHAN, an additional parameter ‘y’ is
added to the Generator for generating the corresponding data. Labels are also put into
the input to the Discriminator in order for the Discriminator to help distinguish the real
data from the fake generated data.
3. Deep Convolutional GAN (DCGAN): DCGAN is one of the most popular and also the
most successful implementations of GAN. It is composed of ConvNets in place of
multi-layer perceptrons. The ConvNets are implemented without max pooling, which is in
fact replaced by convolutional stride. Also, the layers are not fully connected.
4. Laplacian Pyramid GAN (LAPGAN): Laplacian Pyramid GAN (LAPGAN) is a variant of
the Generative Adversarial Network (GAN) that generates high-resolution images from
low-resolution inputs. It utilizes a pyramid structure of generators and discriminators,
where each level of the pyramid works on a progressively higher resolution. The
generators at each level produce a residual image, which is added to an upsampled
version of the previous level's output, creating a refined image. LAPGAN generates
high-resolution images by iteratively improving the details from coarse to fine levels,
resulting in more realistic and detailed image generation.
5. Super Resolution GAN (SRGAN): SRGAN as the name suggests is a way of designing
a GAN in which a deep neural network is used along with an adversarial network in order
to produce higher-resolution images. This type of GAN is particularly useful in optimally
up-scaling native low-resolution images to enhance their details, minimizing errors while
doing so. The generator network takes a low-resolution image as input and generates a
corresponding high-resolution image, while the discriminator network tries to distinguish
between the generated high-resolution images and real high-resolution images. Through
adversarial training, SRGAN learns to generate high-quality and realistic high-resolution
images, enhancing the level of detail and sharpness compared to the original
low-resolution inputs.

Applications of GAN Networks

- Predicting the next frame in a video: GAN can be trained on video sequences and let
it predict what would occur next.
- Increasing Resolution of an image: Generate a high resolution photo from a
comparatively low resolution.
- Text to Image Generation: GANs can be used to generate images from text
descriptions. This can be used for a variety of applications, such as creating new forms
of art, generating realistic images for use in movies or video games, and even creating
personalized avatars for users
- Audio synthesis: creating new forms of music, generating realistic sound effects for use
in movies or video games, and even creating personalized voice assistants for users
- 3D modeling: GAN can create new forms of art, generating realistic 3D models for use
in movies or video games, and even creating personalized 3D avatars for users.
- Image Denoising: GANs can be used to remove noise from images. This can be useful
for restoring old photographs or for improving the quality of images that have been taken
in noisy conditions.