NLU CHATBOT

A PROJECT REPORT

Submitted by

A.BRITTO RAJ

In partial fulfillment for the award of the degreeof

BACHELOR OF ENGINEERING

in

COMPUTER SCEINECE AND ENGINEERING

CHRISTIAN COLLEGE OF ENGINEERING AND TECHNOLOGY,

ODDANCHATRAM-624619

ANNA UNIVERSITY::CHENNAI–600025

MAY 2024

1
 CHRISTIAN COLLEGE OF ENGINEERING AND TECHNOLOGY

ODDANCHATRAM-624619

NM1022–EXPERIENCE BASED PROJECT LEARNING

RECORD NOTE BOOK

REGISTER NO:

Certified that it is a bonafide record of practical lab

work done by during the year

STAFF – INCHARGE HEAD OF THE DEPARTMENT

Submitted for the practical exam held on .

Internal Examiner External Examiner

5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
Initial Inspection: Load the datasets into a pandas DataFrame and perform an initial inspection to understand their structure and content (e.g., number of
records, columns, and data types).

Visualization: Create visualizations (e.g., histograms, word clouds, and scatter plots) to explore the distribution of the data and understand patterns and
relationships within the dataset.

2. Data Cleaning:
Handling Missing Values: Identify and handle missing values through methods such as imputation, removal, or lling with placeholder values, depending on
the context.

Removing Duplicates: Check for and remove duplicate records to ensure data quality.

Outlier Detection: Detect and handle outliers that may skew the model training, using statistical methods or visualization techniques.
Standardizing Formats: Standardize data formats (e.g., date formats, text casing) to ensure consistency across the dataset.

3. Data Transformation:
Tokenization: Split text data into tokens (words or subwords) using tokenizers from NLP libraries like SpaCy or Hugging Face Transformers.

Normalization: Normalize text data by converting it to lowercase, removing punctuation, special characters, and stop words.
Lemmatization/Stemming: Apply lemmatization or stemming to reduce words to their base or root form, aiding in reducing dimensionality and improving
model performance.

Vectorization: Convert text data into numerical format using techniques like TF-IDF, word embeddings (Word2Vec, GloVe), or transformer-based
embeddings (BERT, GPT).

4. Feature Engineering:
Contextual Features: Extract contextual features such as dialogue context, speaker information, and conversation history to enhance model understanding.

Sentiment Analysis: Incorporate sentiment analysis to capture the emotional tone of the conversations.

Custom Features: Create custom features relevant to the chatbot’s domain, such as named entity recognition (NER) tags or topic modeling outputs.

Feature Selection: Select and prioritize features that are most relevant to the task, using techniques like correlation analysis, mutual information, or feature
importance from tree-based models.

5. Model Selection and Training: Model Architecture: Choose appropriate model architectures for the chatbot, such as transformer-based models (e.g.,
BERT, GPT) for their superior performance in NLP tasks.

Hyperparameter Tuning: Perform hyperparameter tuning using techniques like grid search or random search to optimize model performance.

Training Process: Train the model on the preprocessed data, ensuring to use appropriate loss functions, optimizers, and learning rate schedules. Data
Augmentation: Apply data augmentation techniques if necessary to arti cially expand the training dataset and improve model robustness.

6. Model Evaluation:
Validation Split: Split the dataset into training, validation, and test sets to evaluate model performance.

Evaluation Metrics: Use relevant metrics such as accuracy, precision, recall, F1-score, and confusion matrix to assess model performance.

Cross-Validation: Implement cross-validation to ensure the model generalizes well to unseen data.

Error Analysis: Conduct error analysis to identify common failure modes and areas for improvement.

Iterative Improvement: Based on evaluation results, iterate on the model by re ning preprocessing steps, adjusting features, or exploring alternative model
architectures.

NLU chatbot development has evolved through:

Rule-Based Systems: Early systems like ELIZA and PARRY set the stage for automated dialogue.

Statistical Models: Techniques like HMM and CRF improved tasks with machine learning.

Sequence-to-Sequence Models: Models like GNMT brought coherence to responses using RNNs.

Transformer Models: Transformers like BERT and GPT revolutionized NLP with attention mechanisms.

Pre-trained Language Models: BERT and GPT excel in context comprehension and response generation.

Conversational AI Platforms: Tools like Dialog ow streamline chatbot development.

Multimodal Chatbots: Emerging chatbots integrate text, voice, and visual inputs for richer interactions.

Evaluation and Benchmarks: GLUE and SuperGLUE set standards for assessing model performance

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 2/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
FLOW CHART

FUTURE ENHANCEMENT
Consider adding multimodal capabilities to your chatbot for a richer user experience. This includes:

Image Recognition: Let the chatbot interpret images for tasks like product information or recommendations.

Voice Input: Enable users to interact with voice commands, enhancing accessibility and natural conversation.

Emotion Detection: Incorporate sentiment analysis to tailor responses based on user emotions.

Interactive Visual Output: Provide visual aids like graphs or charts for better information conveyance.

Personalization: Offer personalized responses based on user history and preferences.

These enhancements broaden user engagement and cater to diverse needs, making your chatbot more versatile and user-friendly.

CONCLUSION:

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 3/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
In conclusion, our NLU chatbot project signi es a signi cant advancement in conversational AI. Leveraging Kaggle's dataset and advanced ML techniques,
we've created a robust chatbot pro cient in understanding and responding to natural language inputs. Overcoming challenges like data preprocessing and
model training, our chatbot showcases impressive performance.

This project isn't just a standalone chatbot but a precursor to future AI-driven conversational systems. With potential enhancements like multimodal
capabilities and personalization, our chatbot paves the way for more interactive human-computer interactions.

Re ecting on our journey, we acknowledge the collaborative effort and dedication involved. Moving forward, we're excited about deploying our chatbot
across various domains, revolutionizing user experiences. Overall, our project underscores AI and ML's transformative potential, offering innovative
solutions for human-centric interactions.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig, TrainingArguments, Trainer

import torch import time import pandas as pd import numpy as np import string from nltk.corpus import
stopwords import pandas as pd from sklearn.feature_extraction.text import CountVectorizer from sklearn.tree
import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier from
sklearn.feature_extraction.text import TfidfTransformer,TfidfVectorizer from sklearn.pipeline import
Pipeline import evaluate

from google.colab import drive

drive.mount('/content/drive')

Mounted at /content/drive

columns = ['question', 'answer'] df = pd.read_csv('/content/drive/MyDrive/NLU chatbot

/dialogs.txt', sep='\t', names=columns)

# Now df will have columns named 'question' and 'answer'

df.head()

question answer

0 hi, how are you doing? i'm fine. how about yourself?

1 i'm fine. how about yourself? i'm pretty good. thanks for
asking.

2 i'm pretty good. thanks for asking.no problem. so how have you
been?

3 no problem. so how have you been? i've been great.

what about you? 4 i've been great. what about you? i've

been good. i'm in school right now.

import matplotlib.pyplot as plt

from wordcloud import WordCloud

print("\nData types of columns:")

print(df.dtypes) print("\nShape
of the dataset:") print(df.shape)
print("\nMissing values:")
print(df.isnull().sum())

Data types of columns:

question object answer
object dtype: object

Shape of the dataset:

(3725, 2)

Missing values:
question 0
answer 3
dtype: int64

Text Analysis:

Analyze the length distribution of questions and answers. Check for any unusual characters or patterns in the text. Explore the most common words or
phrases in questions and answers (word frequency analysis).

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 4/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
df['question_length'] = df['question'].apply(len)

# Convert all entries in the 'answer' column to strings before applying len()
df['answer_length'] = df['answer'].astype(str).apply(len)

Visualization:

Create visualizations to better understand the data distribution (e.g., histograms, word clouds). Plot the distribution of question and answer lengths.
Visualize word frequency using bar plots or word clouds.

plt.figure(figsize=(8, 6)) plt.hist(df['question_length'], bins=30, alpha=0.7,

color='red', label='Question Length') plt.hist(df['answer_length'], bins=30, alpha=0.7,
color='blue', label='Answer Length') plt.title('Distribution of Question and Answer
Lengths') plt.xlabel('Length') plt.ylabel('Frequency') plt.legend() plt.show()

question_text = ' '.join(df['question']) wordcloud = WordCloud(width=600, height=200,

background_color ='black').generate(question_text) plt.figure(figsize=(12, 6))
plt.imshow(wordcloud, interpolation='bilinear') plt.title('generating Word Cloud for
Questions') plt.axis('off') plt.show()

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 5/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab

# Replace NaN values with an empty string, then convert all entries to strings
df['answer'] = df['answer'].fillna('').astype(str)

# Join all the answers into a single string

answer_text = ' '.join(df['answer'])

# Generate the word cloud wordcloud = WordCloud(width=800, height=400,

background_color='white').generate(answer_text)

# Plot the word cloud plt.figure(figsize=(10,

6)) plt.imshow(wordcloud,
interpolation='bilinear') plt.title('generating
Word Cloud for Answers') plt.axis('off') #
Hide axes plt.show()

Topic Modeling:

Use techniques like Latent Dirichlet Allocation (LDA) to identify underlying topics in the questions and answers. Cluster similar questions and answers based
on topic distributions.

from collections import Counter from

sklearn.feature_extraction.text import CountVectorizer from
sklearn.decomposition import LatentDirichletAllocation from
textblob import TextBlob

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 6/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
vectorizer = CountVectorizer(max_df=0.95, min_df=2, stop_words='english') X =
vectorizer.fit_transform(df['question']) lda = LatentDirichletAllocation(n_components=7, random_state=50)
lda.fit(X) # Display the top words for each topic for idx, topic in enumerate(lda.components_):
print(f"Topic {idx}:", ", ".join([vectorizer.get_feature_names_out()[i] for i in topic.argsort()[:-11:-1]]))

Topic 0: yes, good, need, ll, course, don, thank, hope, talk, just
Topic 1: like, lot, make, people, don, got, great, fun, doing, sounds
Topic 2: didn, day, sure, just, tell, come, maybe, nice, bad, eat
Topic 3: going, mean, does, tv, heard, party, long, happened, idea, told
Topic 4: think, right, time, ll, money, okay, today, school, look, love
Topic 5: want, know, don, really, ve, movie, weather, yeah, buy, best
Topic 6: did, let, say, new, wrong, ll, enjoy, smell, school, phone

Language Complexity:

Measure the complexity of language used in questions and answers (e.g., average word length, vocabulary richness). Explore readability scores or linguistic
features.

# Ensure there are no NaN values and convert to strings

df['answer'] = df['answer'].fillna('').astype(str)

# Function to calculate average word length, handling empty strings

def avg_word_length(text): words = text.split() if
len(words) == 0:
return 0 return sum(len(word) for word in
words) / len(words)

# Analyze language complexity avg_question_word_length = df['question'].apply(lambda

x: avg_word_length(x)).mean() avg_answer_word_length = df['answer'].apply(lambda x:
avg_word_length(x)).mean()

print("\nAverage word length in questions:", avg_question_word_length)

print("Average word length in answers:", avg_answer_word_length)

Average word length in questions: 4.166629246606172

Average word length in answers: 4.171673641983572

Data Preprocessing

Data Augmentation (UPSampling)

new_dialogue_data =
[["Hi", "Hello"],
["How are you?", "I'm good, thanks for asking. How about you?"],
["I'm doing well too.", "That's great to hear. What have you been up to lately?"],
["Not much, just working and spending time with family.", "That sounds nice. Have you watched any good movies recently?"],["Yeah,
I saw a really good one last weekend.", "It was a thriller, right? I heard good things about it."],
["Yes, it was.", "Do you want to watch it together sometime?"],
["Sure, that sounds like a plan.", "Awesome! Let's plan it for this weekend."],
["Sounds good to me.", "Alright then, it's a plan. What time works for you?"],
["How about Saturday evening?", "Perfect! Saturday evening it is. I'll book the tickets."],
["Great! Looking forward to it.", "Me too. It'll be fun."]
] new_df = pd.DataFrame(new_dialogue_data,
columns=columns)

# Concatenate the new DataFrame with the existing DataFrame

df = pd.concat([df, new_df], ignore_index=True)

# Print the updated DataFrame

df

question answer question_length answer_length

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 7/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
0 hi, how are you doing? i'm fine. how about yourself? 22.0 29.0
i'm fine. how about i'm pretty good. thanks for
1 yourself? asking. 29.0 35.0

i'm pretty good. thanks no problem. so how have you

2 for asking. been? 35.0 33.0

no problem. so how i've been great. what about

3 have you been? you? 33.0 32.0

i've been great. what i've been good. i'm in school

4 about you? right now. 32.0 40.0

... ... ... ... ...

Do you want to watch it

3730 Yes, it was. together sometime? NaN NaN

Sure, that sounds like Awesome! Let's plan it for

3731 a plan. this weekend. NaN NaN

3732 Sounds good to me. Alright then, it's a plan. What NaN NaN
ti kf
Preprocessing

Lowercase

Tokenization

Stop words removal Lemmization

import nltk from nltk.corpus import

stopwords from nltk.tokenize import
word_tokenize nltk.download('punkt')

# Function for preprocessing text

def preprocess_text(text): #
Convert text to lowercase text
= text.lower() # tokens =
word_tokenize(text)
# tokens = [word for word in tokens if word.isalnum()]
# preprocessed_text = ' '.join(tokens)

return text

# Apply preprocessing to question and answer columns preprocessed_df = df.copy()

preprocessed_df['question'] = preprocessed_df['question'].apply(preprocess_text)
preprocessed_df['answer'] = preprocessed_df['answer'].apply(preprocess_text)

# Print the preprocessed DataFrame

preprocessed_df

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 8/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Package punkt is already up-to-date!
question answer question_length answer_length

0 hi, how are you doing? i'm fine. how about yourself? 22.0 29.0
i'm fine. how about i'm pretty good. thanks for
1 yourself? asking. 29.0 35.0

i'm pretty good. thanks no problem. so how have

2 for asking. you been? 35.0 33.0

no problem. so how i've been great. what about

3 have you been? you? 33.0 32.0

i've been great. what i've been good. i'm in school

4 about you? right now. 32.0 40.0

... ... ... ... ...

do you want to watch it

3730 yes, it was. together sometime? NaN NaN

sure, that sounds like a awesome! let's plan it for this

3731 plan. weekend. NaN NaN

3732 sounds good to me. alright then, it's a plan. what NaN NaN
ti kf

Classical ML ChatBot

from sklearn.model_selection import train_test_split

# Splitting the dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(preprocessed_df['question'], preprocessed_df['answer'], test_size=0.2, random_state=42,

Model Pipline and training

Pipe = Pipeline([
('tfidf', TfidfVectorizer()),
('classifier', RandomForestClassifier())
])

Pipe.predict(['where are you going'])[0]

'we went to a nice restaurant.' Model

Evaluation

predicted_text = Pipe.predict(X_test)

# Creating a DataFrame to compare the first 10 results comparison_df = pd.DataFrame({'Real Question ': X_test[:10],'Real Generated
Text': y_test[:10], 'Predicted Text': predicted_text[:10]}) comparison_df.head(10)

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 9/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
Real Question Real Generated Text Predicted Text

3253 how are you doing that? i started shopping at the dollar store. i don't know. i think i'm average.
3190 the pants are fine, but the pocket has a huge ... you shouldn't carry your keys and pens in your... i'm pretty good. thanks for asking.

2194 uh-oh. that means that she's fat and ugly. she's cute. men singers don't have to look good.

3303 that's great. we won't have neighbors on both ... no pets are allowed. if they don't like it, they can move.

642 i really wanted you to come, but i understand. yeah, maybe next time. this friday? sorry, i already have plans.

3214 that's a good deal. and a one-pound tub of soft butter was the sam... yes, even though some of the potatoes had eyes.

184 she's one of the prettiest girls at the school. what does she look like? maybe we should learn some good jokes.

3515 no, that's not the problem. maybe it will go away in a little while. did you need something?

3185 why not? i didn't want to pay for the holes. you might hit someone in the head.

2206 eight o'clock. that sounds great. i have the dvd.

import seaborn as sns

# Visualize Feature Importance if isinstance(Pipe.named_steps['classifier'],

RandomForestClassifier): feature_importances =
Pipe.named_steps['classifier'].feature_importances_ feature_names =
Pipe.named_steps['tfidf'].get_feature_names_out()

# Sort feature importances sorted_indices =

feature_importances.argsort()[::-1] top_feature_importances =
feature_importances[sorted_indices][:10] top_feature_names =
np.array(feature_names)[sorted_indices][:10]

plt.figure(figsize=(6, 4))
sns.barplot(x=top_feature_importances, y=top_feature_names)
plt.xlabel('Token Importance') plt.ylabel('Token Name')
plt.title('Top 10 Tokenze Importance') plt.show()

Chat with you ChatBot :D

def get_response(question):
response = Pipe.predict([question])[0]
return response

# Function to interact with the user

def chat(): while True:

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 10/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
question = input("You: ")
if question.lower() == 'quit':
print("Chatbot: Goodbye!")
break response =
get_response(question)
print("Chatbot:", response)

# Start the chat

chat()

You: hello Chatbot: okay. i'll return your pen

when i'm done.
You: i am really excited for you
Chatbot: are you really?
You: yes Chatbot: see
those stains?
You: what are you doing?
Chatbot: i'm going to change the light bulb. it burnt out.
You: is that a lot of work?
Chatbot: babies cry all the time.
You: yes Chatbot: see
those stains?
You: no Chatbot: how do
you know?
You: no Chatbot: how do
you know?
You: come Chatbot: you can see the stars so much more clearly after
it rains.
You: Goodbye!
Chatbot: when's that?
You: dead Chatbot:
when's that?
You: die Chatbot: yes, he did. his cat
died, too.
KeyboardInterrupt
Traceback (most recent call last)
<ipython-input-22-22eefa91ce41> in <cell line: 16>()
14
15 # Start the chat
---> 16 chat()

2 frames
/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py in _input_request(self, prompt, ident, parent, password)
893 except KeyboardInterrupt:
894 # re-raise KeyboardInterrupt, to truncate
traceback
--> 895 raise KeyboardInterrupt("Interrupted by user") from None
896 except Exception as e:
897 self.log.warning("Invalid Message:", exc_info=True)

KeyboardInterrupt: Interrupted by user

Encoder - Decoder Model with Attention and LSTMs Chatbot from scratch

import tensorflow as tf from tensorflow.keras.preprocessing.text import Tokenizer from

tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.models
import Model from tensorflow.keras.layers import Input, LSTM, Dense, Embedding,
Attention, Concatenate

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 11/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
import nltk from nltk.corpus import
stopwords from nltk.tokenize import
word_tokenize import re import
unicodedata

nltk.download('punkt')

def unicode_to_ascii(s):
return ''.join(c for c in unicodedata.normalize('NFD', s)
if unicodedata.category(c) != 'Mn')

# Function for preprocessing text def

preprocess_text(text): # Convert text to
lowercase text =
unicode_to_ascii(text.lower().strip()) text =
re.sub("(\\W)"," ",text) text =
re.sub('\S*\d\S*\s*','', text) text = "<sos>
" + text + "<eos>" return text

# Apply preprocessing to question and answer columns preprocessed_df = df.copy()

preprocessed_df['question'] = preprocessed_df['question'].apply(preprocess_text)
preprocessed_df['answer'] = preprocessed_df['answer'].apply(preprocess_text)

# Print the preprocessed DataFrame

preprocessed_df
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Package punkt is already up-to-date!
question answer question_length answer_length

0 <sos> hi how are you doing <eos> <sos> i m fine how about yourself <eos> 22.0 29.0
1 <sos> i m fine how about yourself <eos> <sos> i m pretty good thanks for asking <eos> 29.0 35.0

2 <sos> i m pretty good thanks for asking <eos> <sos> no problem so how have you been <eos> 35.0 33.0

3 <sos> no problem so how have you been <eos> <sos> i ve been great what about you <eos> 33.0 32.0

4 <sos> i ve been great what about you <eos> <sos> i ve been good i m in school right now ... 32.0 40.0

... ... ... ... ...

3730 <sos> yes it was <eos> <sos> do you want to watch it together sometim... NaN NaN

3731 <sos> sure that sounds like a plan <eos> <sos> awesome let s plan it for this weekend ... NaN NaN

3732 <sos> sounds good to me <eos> <sos> alright then it s a plan what time wor... NaN NaN

3733 <sos> how about saturday evening <eos> <sos> perfect saturday evening it is i ll bo... NaN NaN

3734 <sos> great looking forward to it <eos> <sos> me too it ll be fun <eos> NaN NaN
3735 rows × 4 columns

questions = preprocessed_df['question'].values.tolist()
answers = preprocessed_df['answer'].values.tolist()

# Tokenizing the data tokenizer = Tokenizer(filters='')

tokenizer.fit_on_texts(np.concatenate((questions, answers), axis=0))
vocab_size = len(tokenizer.word_index) + 1

# Convert text to sequences question_seqs =

tokenizer.texts_to_sequences(questions) answer_seqs =
tokenizer.texts_to_sequences(answers)

# Padding sequences for equal length # Pad sequences for equal length
max_len_question = max(len(seq) for seq in question_seqs) max_len_answer =
max(len(seq) for seq in answer_seqs) max_len = max(max_len_question,
max_len_answer) print(max(max_len_question, max_len_answer)) # Pad sequences
separately for questions and answers question_seqs =
pad_sequences(question_seqs, maxlen=max_len, padding='post') answer_seqs =
pad_sequences(answer_seqs, maxlen=max_len, padding='post')

24

tokenizer.texts_to_sequences("<sos>")

[[], [9], [490], [9], []]

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 12/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
tokenizer.word_index["<sos>"]

Model Architecture

from tensorflow.keras.models import Model from tensorflow.keras.layers import Input, LSTM, Dense,
Embedding, Attention, Concatenate, Dropout

# Define the model architecture latent_dim = 256 #

Dimensionality of the encoding space

# Encoder encoder_inputs = Input(shape=(max_len,)) encoder_embedding = Embedding(vocab_size, latent_dim,

input_shape=(max_len,)) encoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,
dropout=0.1, recurrent_dropout=0.1) encoder_outputs, state_h, state_c =
encoder_lstm(encoder_embedding(encoder_inputs)) encoder_states = [state_h, state_c]

# Decoder decoder_inputs = Input(shape=(max_len-1,)) decoder_embedding = Embedding(vocab_size, latent_dim,

input_shape=(max_len-1,)) decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,
dropout=0.1, recurrent_dropout=0.1) decoder_outputs, _, _ = decoder_lstm(decoder_embedding(decoder_inputs),
initial_state=encoder_states)

# Attention mechanism attention_layer = Attention() attention_output =

attention_layer([decoder_outputs, encoder_outputs])

# Concatenate attention output and decoder LSTM output decoder_concat_input =

Concatenate(axis=-1)([decoder_outputs, attention_output])

# Add dropout layer for regularization

decoder_concat_input = Dropout(0.1)(decoder_concat_input)

# Output layer decoder_dense = Dense(vocab_size,

activation='softmax') decoder_outputs =

decoder_dense(decoder_concat_input) Model Training

model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

# Compile the model model.compile(optimizer='adam',

loss='sparse_categorical_crossentropy')

# Print model summary

model.summary()

# Train the model model.fit([question_seqs, answer_seqs[:, :-1]],

answer_seqs[:, 1:], batch_size=64, epochs=32,
validation_split=0.2)

Model: "model"

Layer (type) Output Shape Param # Connected to

======================== ========================================================================== input_1 (InputLayer)
[(None, 24)] 0 []
input_2 (InputLayer) [(None, 23)] 0 []
embedding (Embedding) (None, 24, 256) 618496 ['input_1[0][0]']
embedding_1 (Embedding) (None, 23, 256) 618496 ['input_2[0][0]']
lstm (LSTM) [(None, 24, 256), 525312 ['embedding[0][0]']

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 13/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
(None, 256),
(None, 256)]

lstm_1 (LSTM) [(None, 23, 256), 525312 ['embedding_1[0][0]',

(None, 256), 'lstm[0][1]',
(None, 256)] 'lstm[0][2]']

attention (Attention) (None, 23, 256) 0 ['lstm_1[0][0]',

'lstm[0][0]']

concatenate (Concatenate) (None, 23, 512) 0 ['lstm_1[0][0]',

'attention[0][0]']

dropout (Dropout) (None, 23, 512) 0 ['concatenate[0][0]']

dense (Dense) (None, 23, 2416) 1239408 ['dropout[0][0]']

==================================================================================================
Total params: 3527024 (13.45 MB)
Trainable params: 3527024 (13.45 MB)
Non-trainable params: 0 (0.00 Byte)

Epoch 1/32
47/47 [==============================] - 63s 1s/step - loss: 3.0299 - val_loss: 2.2781
Epoch 2/32
47/47 [==============================] - 49s 1s/step - loss: 1.9036 - val_loss: 2.0941
Epoch 3/32
47/47 [==============================] - 49s 1s/step - loss: 1.7891 - val_loss: 2.0648
Epoch 4/32
47/47 [==============================] - 55s 1s/step - loss: 1.7495 - val_loss : 2.0476
Epoch 5/32
47/47 [==============================] - 47s 1s/step - loss: 1.7074 - val_loss: 2.0242
Epoch 6/32
47/47 [==============================] - 48s 1s/step - loss: 1.6609 - val_loss: 2.0007
Epoch 7/32
47/47 [==============================] - 49s 1s/step - loss: 1.6212 - val_loss: 1.9783
Epoch 8/32
47/47 [==============================] - 47s 1s/step - loss: 1.5819 - val_loss: 1.9536
Epoch 9/32
47/47 [==============================] - 48s 1s/step - loss: 1.5394 - val_loss : 1.9287
Epoch 10/32
47/47 [==============================] - 50s 1s/step - loss: 1.4969 - val_loss: 1.9095
Epoch 11/32
47/47 [==============================] - 47s 1s/step - loss: 1.4552 - val_loss: 1.8925
Epoch 12/32

def generate_response(input_text):
# Tokenize the input text
input_sequence = tokenizer.texts_to_sequences([input_text])
# Pad the input sequence
input_sequence = pad_sequences(input_sequence, maxlen=max_len, padding='post')

# Initialize the decoder input sequence with start token

decoder_input_sequence = np.zeros((1, max_len-1))
decoder_input_sequence[0, 0] = tokenizer.word_index['<sos>']

# Generate response using the trained model

for i in range(max_len - 1):
predictions = model.predict([input_sequence, decoder_input_sequence])
predicted_id = np.argmax(predictions[0, i, :])
if predicted_id == tokenizer.word_index['<eos>']:
break
decoder_input_sequence[0, i+1] = predicted_id

# Convert output sequence to text

output_text = ''
for token_index in decoder_input_sequence[0]:
if token_index == tokenizer.word_index['<eos>'] or token_index == 0:
break
output_text += tokenizer.index_word[token_index] + ' '

return output_text.strip()

# Test the function with input "how are you"

input_text = "how do you do"
response = generate_response(input_text)
print("Response:", response[5:])

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 14/15
5/30/24, 1:16 PM NLU chat bot phase 5.ipynb - Colab
1/1 [==============================] - 2s 2s/step
1/1 [==============================] - 0s 175ms/step
1/1 [==============================] - 0s 175ms/step
1/1 [==============================] - 0s 172ms/step
1/1 [==============================] - 0s 171ms/step
Response: i m not sure model_name='google/flan-t5-

small'

original_model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

tokenizer = AutoTokenizer.from_pretrained(model_name)

/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will b

warnings.warn( /usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: The secret `HF_TOKEN` does not
exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secre
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
warnings.warn(
config.json: 100% 1.40k/1.40k [00:00<00:00, 21.9kB/s]
model.safetensors: 100% 308M/308M [00:03<00:00, 79.4MB/s]

generation_config.json: 100% 147/147 [00:00<00:00, 3.48kB/s]

tokenizer_config.json: 100% 2.54k/2.54k [00:00<00:00, 53.9kB/s]

spiece.model: 100% 792k/792k [00:00<00:00, 19.5MB/s]

tokenizer.json: 100% 2.42M/2.42M [00:00<00:00, 5.05MB/s]

special_tokens_map.json: 100% 2.20k/2.20k [00:00<00:00, 95.9kB/s]

def print_number_of_trainable_model_parameters(model):
trainable_model_params = 0
all_model_params = 0 for _, param in
model.named_parameters():
all_model_params += param.numel()
if param.requires_grad:
trainable_model_params += param.numel() return f"trainable model parameters: {trainable_model_params}\nall model parameters:

{all_model_params}\npercentage of trainable model p print(print_number_of_trainable_model_parameters(original_model))

trainable model parameters: 76961152 all model

parameters: 76961152 percentage of trainable
model parameters: 100.00%

Test the Model with Zero Shot Inferencing

Test the model with the zero shot inferencing. You can see that the model struggles to summarize the dialogue compared to the baseline summary, but it does
pull out some important information from the text which indicates the model can be ne-tuned to the task at hand.

index = 0

question = df['question'][index]
answer = df['answer'][index]

https://colab.research.google.com/drive/1B9vUz86zGJkw5auKFSUskzS0U5-BmGJE#scrollTo=wrcO4Fj0vveQ&printMode=true 15/15