PART-B

1. What is Unfolding Computational Graphs?

Unfolding computational graphs refers to the process of "unrolling" or expanding a computation
graph in time, especially in the context of Recurrent Neural Networks (RNNs). In an RNN, the
same computation is applied at each time step, with the output at each step depending on the
current input and the previous hidden state. When you "unfold" an RNN, you visualize it as a
series of computations over time, where each time step corresponds to a layer in the graph.
This process helps in understanding the flow of data and how dependencies are handled across
time steps.

2. List out some applications of Bidirectional RNN.

● language translation
● sentiment analysis,
● Time series forecasting tasks, such as predicting stock prices or weather patterns,
● Speech Recognition
● Handwriting Recognition
● NLP tasks

3.Design an Encoder-Decoder model with RNN.

An Encoder-Decoder model with RNNs is commonly used in sequence-to-sequence tasks, such
as machine translation.
● Encoder: An RNN (typically LSTM or GRU) processes the input sequence one token at a
time and outputs a context vector (final hidden state).
● Decoder: The decoder is another RNN that takes the context vector from the encoder as
its initial hidden state and generates the output sequence token by token.

4.What are the limitations of Bidirectional RNN?

While Bidirectional RNNs have advantages, they also have some limitations:

● Computational Complexity: Since BiRNNs process data in both directions, they require
more computational resources and time compared to unidirectional RNNs.
● Memory Usage: The need to store both forward and backward states increases memory
usage, which can be a concern for large datasets or long sequences.
● Not Suitable for Real-time Processing: BiRNNs are less effective in real-time
applications where the future context is not available at the time of prediction.
● Training Difficulty: Like all RNNs, BiRNNs can still suffer from vanishing or
exploding gradient problems, making training difficult on long sequences.
5. What are the applications of RNN?
Recurrent Neural Networks (RNNs) have a wide range of applications in tasks involving
sequential data:

● Speech Recognition
● Natural Language Processing (NLP)
● Time Series Prediction
● Video Processing
● Music Generation
● Handwriting Recognition
● Anomaly Detection

6.What are Recurrent Neural Networks?

● Recurrent neural networks are a class of artificial neural network commonly used for
sequential data processing. Unlike feedforward neural networks, which process data in a
single pass, RNNs process data across multiple time steps, making them well-adapted
for modelling and processing text, speech, and time series.
● This architecture enables RNNs to maintain a "memory" of previous inputs, making them
ideal for tasks where context or temporal dependencies are important, such as speech
recognition, language modeling, and time series prediction. At each time step, an RNN
takes the current input and combines it with the previous hidden state to produce the
output and the next hidden state.

7. Outline the issues faced while training in Recurrent Networks

● Vanishing and Exploding Gradients: During backpropagation, gradients can either
become very small (vanishing gradients) or grow exponentially (exploding gradients),
making it difficult for the network to learn long-range dependencies.
● Long-term Dependencies: RNNs struggle to capture long-term dependencies due to
the difficulty in preserving information over many time steps
● Slow Convergence: Due to the sequential nature of RNNs, training can be slow and
computationally expensive, especially for long sequences.
● Memory Issues: RNNs require storing all hidden states for backpropagation through
time (BPTT), leading to high memory consumption.

8. Mention the advantages and drawbacks of RNN:

Advantages:
● Sequence Modeling: RNNs are capable of handling sequential data, making them ideal
for time series prediction, text, and speech.
● Contextual Learning: RNNs can retain information about previous inputs, which is
crucial for tasks where past inputs influence future predictions.
● Versatility: RNNs can be applied to various tasks, such as classification, generation,
and prediction in sequences.

Drawbacks:
● Struggles with long-term dependencies.
● Training is computationally expensive.
● Suffers from vanishing and exploding gradients.

9.How does backpropagation differ in RNN compared to ANN?

● Backpropagation in RNNs is different from traditional artificial neural networks (ANNs)
 due to the temporal nature of the data and the recurrence in RNNs. While both use
backpropagation to update weights, RNNs employ Backpropagation Through Time
(BPTT).
● RNNs backpropagate errors through time, which means the gradients must be
propagated not only through the layers but also across time steps (via Backpropagation
Through Time - BPTT). ANNs only propagate errors through the layers in a feedforward
manner, as there are no temporal dependencies.
● RNNs are more prone to the vanishing and exploding gradient problems due to the
repeated multiplication of gradients over many time steps.

10.Why do RNNs work better with text data?

● Textdata is sequential, and RNNs are specifically designed to capture temporal patterns
and dependencies in sequences.
● They Can Remember Context from earlier parts of the sequence, enabling better
predictions of subsequent words or characters.
● RNNs can capture contextual relationships between words in a sentence or between
characters in a word. For example, they can understand how the meaning of a word may
depend on its previous words or characters.

11. Define LSTM

LSTM is a special type of Recurrent Neural Network (RNN) designed to address the issues of
vanishing and exploding gradients in traditional RNNs, particularly when learning long-term
dependencies. An LSTM unit consists of a memory cell that can store information over time and
three gates:
● Forget Gate: Decides what portion of the previous memory to forget.
● Input Gate: Controls what new information is added to the memory cell.
● Output Gate: Determines what the output of the cell should be, based on the current
memory state.
The architecture of LSTM enables it to capture long-range dependencies more effectively,
making it suitable for tasks such as language modeling, machine translation, and time series
prediction.

12. Differentiate exploding gradients and vanishing gradients:

Criteria Vanishing Gradients Exploding Gradients

Problem Gradients become very small, causing Gradients grow exponentially,

Description slow or no updates to the weights. causing instability in training.

Cause Deep networks or long sequences Large weights or deep networks

cause gradients to shrink as they are cause gradients to grow
propagated back. uncontrollably.

Impact on Model fails to learn long-range Training becomes unstable, and

Training dependencies. weights may diverge.
 solution LSTM, GRU Gradient clipping

Typical More common in deep networks or More common in networks with

Occurrence RNNs with long sequences. large weights or improper
initialization.

13. What are Deep Recurrent Networks

Deep Recurrent Networks (DRNs) are a type of RNN where multiple layers of recurrent units
(such as LSTMs or GRUs) are stacked on top of each other to form a deep architecture. These
models aim to capture more complex patterns in sequential data by learning hierarchical
representations.

● Advantage: Capture both low-level and high-level dependencies.

● Challenge: Higher risk of vanishing gradients

14. How do Deep Recurrent Networks differ from RNA

Criteria Deep Recurrent Networks Recurrent Neural Networks

(DRNs) (RNNs)

Number of Layers Multiple layers of recurrent units Single layer of recurrent units.
stacked together.

Complexity Can capture more complex Simpler, captures patterns at a

patterns due to hierarchical feature single level.
learning.

Ability to Learn Better at learning long-term Struggles with long-term

Dependencies dependencies and complex dependencies due to single-layer
relationships in data. architecture.

Application Suitable for complex tasks like Suitable for simpler sequential
machine translation, speech tasks.
recognition.

15.Give the architectural benefit of Deep Recurrent Networks

● Improved Feature Learning: Multiple layers allow hierarchical feature extraction,

capturing complex patterns in sequential data.
● Better Handling of Long-Term Dependencies: Deeper architectures can more
effectively model long-range dependencies compared to shallow RNNs.
● Enhanced Performance on Complex Tasks: DRNs are more suited for tasks like
machine translation and speech recognition due to their ability to understand deeper
contextual relationships.
PART-C
1.RNN
A Recurrent Neural Network (RNN) is a type of neural network designed to handle sequential
data, where the output from previous time steps is fed back into the network to influence the
current time step.
The defining feature of RNNs is their hidden state—also called the memory state—which
preserves essential information from previous inputs in the sequence.

Key Components of RNN

● Recurrent unit
The fundamental processing unit in a Recurrent Neural Network (RNN) is a Recurrent Unit, hold
a hidden state that maintains information about previous inputs in a sequence. Recurrent units
can “remember” information from prior steps by feeding back their hidden state, allowing them to
capture dependencies across time.

● RNN Unfolding
RNN unfolding, or “unrolling,” is the process of expanding the recurrent structure over time
steps. During unfolding, each step of the sequence is represented as a separate layer in a
series, illustrating how information flows across each time step. This unrolling enables
backpropagation through time (BPTT), a learning process where errors are propagated across
time steps to adjust the network’s weights, enhancing the RNN’s ability to learn dependencies
within sequential data

Types of RNN Architectures:

RNNs can be configured in various ways to suit different tasks:
● One-to-One: A single input produces a single output, similar to traditional feedforward
neural networks.
● One-to-Many: A single input generates multiple outputs, useful in tasks like image
captioning.
● Many-to-One: Multiple inputs produce a single output, as seen in sentiment analysis
where a sequence of words determines a class label.
 ● Many-to-Many: Multiple inputs generate multiple outputs, applicable in tasks like
machine translation.

The architecture of a Recurrent Neural Network (RNN)

1. Input Layer:
The input layer receives sequential data, where each input at time step t, denoted as (x_t) can
be a vector representing a feature (e.g., a word in NLP, or a value in a time series).

2. Hidden Layer:
● The hidden layer consists of recurrent units (like simple RNN cells, LSTM cells, or GRU
cells).
● The key characteristic of the RNN is the presence of loops in the hidden layer, which
allows the network to maintain a hidden state (h_t)
● The activation function (e.g., tanh or ReLU) introduces non-linearity to the model.

3. Output Layer:
● The output layer produces the result based on the hidden state at each time step.
● For tasks like sequence prediction, the output can be a probability distribution over
possible next states or labels.

4. Backpropagation Through Time (BPTT):

● In RNNs, training is done using Backpropagation Through Time (BPTT), which involves
unfolding the network across time steps and applying standard backpropagation to
update the weights.
● During BPTT, gradients are propagated backward from the output to the input through
each time step, adjusting the weights of the hidden states at each step.

Varieties of RNN:
● Vanilla RNN: The basic form of RNN with simple recurrent connections
● Long Short-Term Memory (LSTM): A specialized RNN designed to overcome the
vanishing gradient problem. It uses memory cells and gates (forget, input, output) to
 maintain and control information over long sequences
● Gated Recurrent Unit (GRU): Similar to LSTM, but with a simplified architecture. GRUs
combine the forget and input gates into a single update gate
● Bidirectional RNN (BiRNN): Processes sequences in both forward and backward
directions, allowing the network to capture context from both past and future information,
enhancing performance in tasks like machine translation.
● Deep RNN: An architecture where multiple layers of RNNs are stacked together,
allowing for more complex representations and better performance on tasks requiring
deeper temporal context.

2.DEEP RNN
A deep RNN is simply an RNN with multiple hidden layers stacked on top of each other. This
stacking allows the network to learn more complex patterns and representations from the data.
Each layer in a deep RNN can capture different levels of abstraction, making it more powerful
than a single-layer RNN.

Architecture of Deep Recurrent Networks:

● Multiple Layers: DRNs are built by stacking several layers of RNNs (such as LSTM or
GRU cells) on top of each other. Each layer learns different levels of abstraction from the
input data, with higher layers capturing more complex patterns.
● Input to Hidden Layers: The input is processed by the first layer of recurrent units. The
output of this layer is then fed into the next layer, which processes it further, and so on
until the final output layer.
● Complex Hierarchical Features: Lower layers of DRNs typically capture basic patterns
and short-term dependencies, while higher layers can learn more abstract and long-term
dependencies in the data.
Examples of Deep Recurrent Networks:
1. Speech Recognition:
○ In speech recognition, DRNs can be used to model the sequence of audio
features over time. The deep architecture helps in learning complex patterns in
the audio sequence, such as phonemes and words, by capturing both short-term
and long-term dependencies.
2. Machine Translation:
○ Deep Recurrent Networks are commonly used in sequence-to-sequence tasks
like machine translation. For instance, in translating a sentence from English to
French, lower layers might capture syntactical structures, while higher layers
learn more abstract semantic representations. DRNs help improve the translation
accuracy by learning better contextual relationships between words.
3.BiRNN

A Bidirectional Recurrent Neural Network (BiRNN) is an extension of the traditional Recurrent

Neural Network (RNN) that processes input data in both directions—from the past to the future
and from the future to the past—simultaneously. This architecture allows the network to capture
context from both the past and the future, making it more powerful for sequence-based tasks,
especially when the full context is important for making predictions.

Architectural Design of Bidirectional RNN:

Output Layer:
● The final output of the BiRNN can be passed to an output layer, such as a fully
connected (dense) layer, depending on the task (e.g., classification, regression,
sequence generation).
● The output can either be a sequence of predictions (e.g., for sequence labeling tasks) or
a single prediction (e.g., for classification).
4.LSTM
LSTM is a specialized type of Recurrent Neural Network (RNN) designed to handle the problem
of long-term dependencies in sequential data. Unlike traditional RNNs, LSTMs have a more
complex architecture with mechanisms to control the flow of information, enabling them to retain
relevant information over long periods and forget irrelevant details. This is achieved using gates.

Key Features of LSTM:

● Memory Cells: Store information over time and act as a memory.
● Gates: Mechanisms to control the flow of information into, out of, and within the memory
cell.
○ Forget Gate: Decides what information to discard.
○ Input Gate: Decides what new information to store.
○ Output Gate: Decides what part of the memory to output.

Complete LSTM Workflow

1. The forget gate removes irrelevant
information from the previous cell state Ct−1
2. The input gate determines what new
information to add to the memory, forming an
updated cell state Ct
3. The output gate decides which part of
the updated cell state Ct​to pass on as the
hidden state hth for the next time step and
output.
5. Encoder-Decoder Sequence-to-Sequence Architecture

The Encoder-Decoder architecture is a neural network design commonly used for

sequence-based tasks, such as machine translation, speech recognition, and text
summarization. It maps an input sequence to an output sequence, where the lengths of the
input and output sequences can differ.

Overview of the Architecture

The architecture consists of two main components:

1. Encoder:
○ Processes the input sequence and compresses it into a fixed-length context
vector (also called a thought vector or latent representation).
○ Captures the information and semantics of the entire input sequence.
2. Decoder:
○ Takes the context vector as input and generates the output sequence step by
step.
○ Decodes the compressed information into the target sequence.
Workflow:

1. The encoder processes the input sequence and produces the context vector CCC.
2. The decoder uses CCC as input to generate the output sequence one token at a time.
3. The process continues until a predefined end-of-sequence token is generated.

Encoder-Decoder Sequence-to-Sequence (Seq2Seq) Architectures

The Encoder-Decoder architecture, commonly used for sequence-to-sequence (Seq2Seq)
tasks, is a neural network architecture designed to transform one sequence into another, often
with different lengths. This is useful in applications like machine translation, text summarization,
and chatbots.

How it Works:

1. Encoder:
○ The encoder is typically an RNN (or LSTM/GRU) that reads the input sequence
and compresses it into a fixed-size vector called the context vector or hidden
state. This vector represents the entire input sequence in a summarized form.
○ The encoder processes the input sequence one step at a time, updating its
hidden state based on the current input and previous hidden state.
2. Decoder:
○ The decoder is another RNN that generates the output sequence based on the
context vector received from the encoder. At each time step, the decoder predicts
the next element in the output sequence, taking the previous hidden state and its
 previous output as input.
○ In a translation task, for instance, the encoder reads the source sentence (e.g.,
English), and the decoder generates the target sentence (e.g., French) one word
at a time.

Key Components:

● Context Vector: A fixed-size representation (output of the encoder) that encapsulates

the entire input sequence and serves as the initial hidden state for the decoder.
● Training: The model is trained using pairs of input-output sequences (e.g., source
language sentences and their translations). The encoder learns to represent the input in
a way that the decoder can use to generate the corresponding output.

Limitations of Basic Encoder-Decoder Architecture:

● Bottleneck Problem: The entire input sequence is compressed into a single fixed-size
vector (context vector). If the sequence is long or complex, this vector may not capture
all the relevant information, leading to performance degradation, especially for long
sequences.

Attention Mechanism in Seq2Seq Models

The attention mechanism was introduced to address the bottleneck problem in the
Encoder-Decoder architecture. Instead of relying solely on the fixed-size context vector, the
decoder "attends" to different parts of the input sequence while generating the output.

How Attention Works:

● At each step of the decoding process, the model computes a set of attention weights
that determine which parts of the input sequence (from the encoder) are most relevant
for generating the current output token.
● This allows the model to focus on specific parts of the input, dynamically adjusting the
attention as it generates the output sequence.