In [ ]: 1.

Creating a NumPy Array

a. Basic ndarray
b. Array of zeros
c. Array of ones
d. Random numbers in ndarray
e. An array of your choice
f. Imatrix in NumPy
g. Evenly ced """ndarray

In [20]: # 1. Creating a NumPy Array

import numpy as np

# a. Basic ndarray
a = np.array([1, 2, 3, 4, 5])
print("Basic ndarray:", a)

Basic ndarray: [1 2 3 4 5]

In [5]: # b. Array of zeros

zeros = np.zeros((3, 3))
print("Array of zeros:\n", zeros)

Array of zeros:
[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]

In [6]: # c. Array of ones

ones = np.ones((2, 4))
print("Array of ones:\n", ones)

Array of ones:
[[1. 1. 1. 1.]
[1. 1. 1. 1.]]

In [7]: # d. Random numbers in ndarray

random_array = np.random.rand(3, 3)
print("Random numbers:\n", random_array)

Random numbers:
[[0.95286609 0.88055076 0.84394613]
[0.05645231 0.58839302 0.30319334]
[0.60025703 0.63418513 0.869445 ]]

In [8]: # e. An array of your choice

custom_array = np.array([[10, 20], [30, 40]])
print("Custom array:\n", custom_array)

Custom array:
[[10 20]
[30 40]]

In [9]: # f. Identity matrix in NumPy

identity_matrix = np.eye(3)
print("Identity Matrix:\n", identity_matrix)

Identity Matrix:
[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]

In [12]: # g. Evenly spaced ndarray

even_spaced = np.linspace(1, 10, 5)
even_spaced

Out[12]: array([ 1. , 3.25, 5.5 , 7.75, 10. ])

In [ ]: 2. The Shape and Reshaping of NumPy Array

a. Dimensions of NumPy array
b. Shape of NumPy array
c. Size of NumPy array
d. Reshaping a NumPy array
e. Flattening a NumPy array
f. Transpose of a NumPy arra

In [21]: # 2. The Shape and Reshaping of NumPy Array

# a. Dimensions of NumPy array
dim = a.ndim
print("Dimensions of array:", dim)

Dimensions of array: 1

In [22]: # b. Shape of NumPy array

shape = random_array.shape
print("Shape of array:", shape)

Shape of array: (3, 3)

In [23]: # c. Size of NumPy array

size = random_array.size
print("Size of array:", size)

Size of array: 9

In [24]: # d. Reshaping a NumPy array

reshaped = a.reshape(5, 1)
print("Reshaped array:\n", reshaped)

Reshaped array:
[[1]
[2]
[3]
[4]
[5]]

In [25]: # e. Flattening a NumPy array

flattened = random_array.flatten()
print("Flattened array:", flattened)

Flattened array: [0.95286609 0.88055076 0.84394613 0.05645231 0.58839302 0.30319334

0.60025703 0.63418513 0.869445 ]

In [26]: # f. Transpose of a NumPy array

transposed = random_array.T
print("Transposed array:\n", transposed)

Transposed array:
[[0.95286609 0.05645231 0.60025703]
[0.88055076 0.58839302 0.63418513]
[0.84394613 0.30319334 0.869445 ]]

In [ ]: Expanding and Squeezing a NumPy Array

a. Expanding a NumPy array
b. Squeezing a NumPy array
c. Sorting in NumPy Arrays

In [27]: # 3. Expanding and Squeezing a NumPy Array

# a. Expanding a NumPy array
expanded = np.expand_dims(a, axis=0)
print("Expanded array:\n", expanded, "\nShape:", expanded.shape)

Expanded array:
[[1 2 3 4 5]]
Shape: (1, 5)

In [28]: # b. Squeezing a NumPy array

squeezed = np.squeeze(expanded)
print("Squeezed array:\n", squeezed, "\nShape:", squeezed.shape)

Squeezed array:
[1 2 3 4 5]
Shape: (5,)

In [29]: # c. Sorting in NumPy Arrays

unsorted_array = np.array([3, 1, 4, 1, 5, 9, 2, 6])
sorted_array = np.sort(unsorted_array)
print("Sorted array:", sorted_array)

Sorted array: [1 1 2 3 4 5 6 9]

In [ ]: 4Indexing and Slicing of NumPy Array

a. Slicing 1-D NumPy arrays
b. Slicing 2-D NumPy arrays
c. Slicing 3-D NumPy arrays
d. Negative slicing of NumPy arrays

In [30]: # 4. Indexing and Slicing of NumPy Array

# a. Slicing 1-D NumPy arrays
one_d_array = np.array([10, 20, 30, 40, 50, 60])
sliced_1d = one_d_array[1:4] # Extract elements from index 1 to 3
print("Sliced 1-D array:", sliced_1d)

Sliced 1-D array: [20 30 40]

In [31]: # b. Slicing 2-D NumPy arrays

two_d_array = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
sliced_2d = two_d_array[:2, 1:] # Extract first two rows and columns from index 1 onwards
print("Sliced 2-D array:\n", sliced_2d)

Sliced 2-D array:

[[2 3]
[5 6]]

In [32]: # c. Slicing 3-D NumPy arrays

three_d_array = np.arange(27).reshape(3, 3, 3)
sliced_3d = three_d_array[:2, :2, :2] # Extract a 2x2x2 block
print("Sliced 3-D array:\n", sliced_3d)

Sliced 3-D array:

[[[ 0 1]
[ 3 4]]

[[ 9 10]
[12 13]]]

In [33]: # d. Negative slicing of NumPy arrays

negative_sliced = one_d_array[-4:-1] # Extract elements from index -4 to -2
print("Negative sliced array:", negative_sliced)

Negative sliced array: [30 40 50]

In [ ]: 5.Stacking and Concatenating Numpy Arrays

a. Stacking ndarrays
b. Concatenating ndarrays
c. Broadcasting in Numpy Arrays

In [34]: arr1 = np.array([[1, 2], [3, 4]])

arr2 = np.array([[5, 6], [7, 8]])
stacked_vertically = np.vstack((arr1, arr2))
stacked_horizontally = np.hstack((arr1, arr2))
print("Vertically Stacked Array:\n", stacked_vertically)
print("Horizontally Stacked Array:\n", stacked_horizontally)

Vertically Stacked Array:

[[1 2]
[3 4]
[5 6]
[7 8]]
Horizontally Stacked Array:
[[1 2 5 6]
[3 4 7 8]]

In [35]: # b. Concatenating ndarrays

concatenated_axis0 = np.concatenate((arr1, arr2), axis=0)
concatenated_axis1 = np.concatenate((arr1, arr2), axis=1)
print("Concatenated along axis 0:\n", concatenated_axis0)
print("Concatenated along axis 1:\n", concatenated_axis1)

Concatenated along axis 0:

[[1 2]
[3 4]
[5 6]
[7 8]]
Concatenated along axis 1:
[[1 2 5 6]
[3 4 7 8]]

In [36]: # c. Broadcasting in NumPy Arrays

broadcast_arr = arr1 + np.array([10, 20]) # Broadcasting a 1D array to match 2D
print("Broadcasted Addition:\n", broadcast_arr)

Broadcasted Addition:
[[11 22]
[13 24]]

In [ ]: 6.Perform following operations using pandas

a. Creating dataframe
b. concat()
c. Setting conditions
d. Adding a new column

In [39]: # 6. Performing operations using pandas

# a. Creating dataframe
import pandas as pd
data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print("DataFrame:\n", df)

DataFrame:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35

In [40]: # b. Using concat()

df2 = pd.DataFrame({'Name': ['David', 'Eve'], 'Age': [40, 45]})
concatenated_df = pd.concat([df, df2], ignore_index=True)
print("Concatenated DataFrame:\n", concatenated_df)

Concatenated DataFrame:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
3 David 40
4 Eve 45

In [41]: # c. Setting conditions

adults = df[df['Age'] > 25]
print("Filtered DataFrame (Age > 25):\n", adults)

Filtered DataFrame (Age > 25):

Name Age
1 Bob 30
2 Charlie 35

In [42]: # d. Adding a new column

df['City'] = ['New York', 'Los Angeles', 'Chicago']
print("DataFrame with new column:\n", df)

DataFrame with new column:

Name Age City
0 Alice 25 New York
1 Bob 30 Los Angeles
2 Charlie 35 Chicago

In [ ]: 7. Perform following operations using pandas

a. Filling NaN with string
b. Sorting based on column values
c. groupby()

In [44]: import pandas as pd

import numpy as np

# Creating a sample DataFrame with NaN values

data = {'Name': ['Alice', 'Bob', 'Charlie', np.nan, 'Eve'],
'Age': [25, 30, np.nan, 40, 22],
'City': ['New York', np.nan, 'Los Angeles', 'Chicago', 'Houston']}

df = pd.DataFrame(data)

# Filling NaN values with a custom string

df_filled = df.fillna("Missing")

print(df_filled)

Name Age City

0 Alice 25.0 New York
1 Bob 30.0 Missing
2 Charlie Missing Los Angeles
3 Missing 40.0 Chicago
4 Eve 22.0 Houston

In [45]: # Sorting by Age column in ascending order

df_sorted = df.sort_values(by='Age', ascending=True)
print(df_sorted)

Name Age City

4 Eve 22.0 Houston
0 Alice 25.0 New York
1 Bob 30.0 NaN
3 NaN 40.0 Chicago
2 Charlie NaN Los Angeles

In [46]: df_sorted_desc = df.sort_values(by='Age', ascending=False)

print(df_sorted_desc)

Name Age City

3 NaN 40.0 Chicago
1 Bob 30.0 NaN
0 Alice 25.0 New York
4 Eve 22.0 Houston
2 Charlie NaN Los Angeles

In [47]: # Creating a sample DataFrame

data = {'Department': ['HR', 'IT', 'HR', 'Finance', 'IT', 'Finance'],
'Employee': ['Alice', 'Bob', 'Charlie', 'David', 'Eve', 'Frank'],
'Salary': [50000, 60000, 55000, 65000, 70000, 62000]}

df = pd.DataFrame(data)

# Grouping by Department and calculating mean salary

grouped_df = df.groupby('Department')['Salary'].mean()
print(grouped_df)

Department
Finance 63500.0
HR 52500.0
IT 65000.0
Name: Salary, dtype: float64

In [48]: pip install pandas openpyxl

Requirement already satisfied: pandas in c:\users\khadersha\anaconda3\lib\site-packages (2.2.2)

Requirement already satisfied: openpyxl in c:\users\khadersha\anaconda3\lib\site-packages (3.1.2)
Requirement already satisfied: numpy>=1.26.0 in c:\users\khadersha\anaconda3\lib\site-packages (from pandas) (1.26.4)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\khadersha\anaconda3\lib\site-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in c:\users\khadersha\anaconda3\lib\site-packages (from pandas) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in c:\users\khadersha\anaconda3\lib\site-packages (from pandas) (2023.3)
Requirement already satisfied: et-xmlfile in c:\users\khadersha\anaconda3\lib\site-packages (from openpyxl) (1.1.0)
Requirement already satisfied: six>=1.5 in c:\users\khadersha\anaconda3\lib\site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Note: you may need to restart the kernel to use updated packages.

In [49]: import pandas as pd

# Sample DataFrame
data = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'City': ['New York', 'Los Angeles', 'Chicago']
}

df = pd.DataFrame(data)

In [50]: df.to_csv('data.txt', sep='\t', index=False) # Tab-separated format

In [51]: df_txt = pd.read_csv('data.txt', sep='\t')

print(df_txt)

Name Age City

0 Alice 25 New York
1 Bob 30 Los Angeles
2 Charlie 35 Chicago

In [52]: df.to_csv('data.csv', index=False) # Saving without row indices

In [53]: df_csv = pd.read_csv('data.csv')

print(df_csv)

Name Age City

0 Alice 25 New York
1 Bob 30 Los Angeles
2 Charlie 35 Chicago

In [54]: df.to_excel('data.xlsx', index=False, engine='openpyxl')

In [55]: df_excel = pd.read_excel('data.xlsx', engine='openpyxl')

print(df_excel)

Name Age City

0 Alice 25 New York
1 Bob 30 Los Angeles
2 Charlie 35 Chicago

In [56]: df.to_json('data.json', orient='records', indent=4)

In [57]: df_json = pd.read_json('data.json')

print(df_json)

Name Age City

0 Alice 25 New York
1 Bob 30 Los Angeles
2 Charlie 35 Chicago

In [58]: import pickle

# Sample data
data = {'Name': 'Alice', 'Age': 25, 'City': 'New York'}

# Save to pickle file

with open('data.pkl', 'wb') as file:
pickle.dump(data, file)

In [59]: # Read from pickle file

with open('data.pkl', 'rb') as file:
loaded_data = pickle.load(file)

print(loaded_data)

{'Name': 'Alice', 'Age': 25, 'City': 'New York'}

In [60]: pip install pillow

Requirement already satisfied: pillow in c:\users\khadersha\anaconda3\lib\site-packages (10.3.0)

Note: you may need to restart the kernel to use updated packages.

In [62]: from PIL import Image

# Open an image file

img = Image.open('image.jpg')

# Show the image

img.show()

# Save the image in a different format

img.save('image.png')

In [63]: import glob

# List all .txt files in the current directory

txt_files = glob.glob('*.txt')

# Print the names of the text files

print(txt_files)

# Reading files
for file in txt_files:
with open(file, 'r') as f:
print(f.read())

['data.txt', 'test.txt']
Name Age City
Alice 25 New York
Bob 30 Los Angeles
Charlie 35 Chicago

import java.utile;

class test{

public static void main(str args[]){

System.out.println("hello world")}

In [64]: import sqlite3

# Connect to SQLite database (or create one if it doesn't exist)

conn = sqlite3.connect('example.db')

# Create a cursor object to interact with the database

cursor = conn.cursor()

# Create a sample table (if it doesn't already exist)

cursor.execute('''
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY,
name TEXT,
age INTEGER
)''')

# Insert sample data into the table

cursor.execute("INSERT INTO users (name, age) VALUES ('Alice', 25)")
cursor.execute("INSERT INTO users (name, age) VALUES ('Bob', 30)")

# Commit the transaction

conn.commit()

# Query the database

cursor.execute("SELECT * FROM users")

# Fetch all records

rows = cursor.fetchall()

# Print the results

for row in rows:
print(row)

# Close the connection

conn.close()

(1, 'Alice', 25)

(2, 'Bob', 30)

In [65]: pip install beautifulsoup4 requests

Requirement already satisfied: beautifulsoup4 in c:\users\khadersha\anaconda3\lib\site-packages (4.12.3)Note: you may need to restart the kernel to use updated packages.

Requirement already satisfied: requests in c:\users\khadersha\anaconda3\lib\site-packages (2.32.2)

Requirement already satisfied: soupsieve>1.2 in c:\users\khadersha\anaconda3\lib\site-packages (from beautifulsoup4) (2.5)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\khadersha\anaconda3\lib\site-packages (from requests) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in c:\users\khadersha\anaconda3\lib\site-packages (from requests) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\khadersha\anaconda3\lib\site-packages (from requests) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\khadersha\anaconda3\lib\site-packages (from requests) (2024.8.30)

In [66]: import requests

from bs4 import BeautifulSoup

# URL of the webpage to scrape

url = 'https://www.example.com'

# Sending a GET request to the webpage

response = requests.get(url)

# Parse the webpage content with BeautifulSoup

soup = BeautifulSoup(response.text, 'html.parser')

# Find and print the title of the webpage

title = soup.title.string
print(f'Webpage Title: {title}')

Webpage Title: Example Domain

In [67]: # Find all <h1> tags on the page

headings = soup.find_all('h1')

# Print each heading

for heading in headings:
print(heading.text)

Example Domain

In [71]: import pandas as pd

# Load the dataset

df = pd.read_csv('loan.csv')

# Display first few rows

print(df.head())

Loan_ID Gender Married Dependents Education Self_Employed \

0 LP001002 Male No 0 Graduate No
1 LP001003 Male Yes 1 Graduate No
2 LP001005 Male Yes 0 Graduate Yes
3 LP001006 Male Yes 0 Not Graduate No
4 LP001008 Male No 0 Graduate No

ApplicantIncome CoapplicantIncome LoanAmount Loan_Amount_Term \

0 5849 0.0 NaN 360.0
1 4583 1508.0 128.0 360.0
2 3000 0.0 66.0 360.0
3 2583 2358.0 120.0 360.0
4 6000 0.0 141.0 360.0

Credit_History Property_Area Loan_Status

0 1.0 Urban Y
1 1.0 Rural N
2 1.0 Urban Y
3 1.0 Urban Y
4 1.0 Urban Y

In [72]: from sklearn.preprocessing import MinMaxScaler

# Select the columns for scaling (e.g., 'LoanAmount', 'ApplicantIncome')

scaler = MinMaxScaler()

# Apply Min-Max scaling

df[['LoanAmount', 'ApplicantIncome']] = scaler.fit_transform(df[['LoanAmount', 'ApplicantIncome']])

print(df[['LoanAmount', 'ApplicantIncome']].head())

LoanAmount ApplicantIncome
0 NaN 0.070489
1 0.172214 0.054830
2 0.082489 0.035250
3 0.160637 0.030093
4 0.191027 0.072356

In [73]: from sklearn.preprocessing import StandardScaler

# Select the columns for standardization

scaler = StandardScaler()

# Apply standardization
df[['LoanAmount', 'ApplicantIncome']] = scaler.fit_transform(df[['LoanAmount', 'ApplicantIncome']])

print(df[['LoanAmount', 'ApplicantIncome']].head())

LoanAmount ApplicantIncome
0 NaN 0.072991
1 -0.215309 -0.134412
2 -0.940328 -0.393747
3 -0.308860 -0.462062
4 -0.063289 0.097728

In [76]: from sklearn.preprocessing import LabelEncoder

# Assuming 'LoanStatus' is a categorical column (e.g., 'Y' or 'N')

encoder = LabelEncoder()

# Encode the 'LoanStatus' column (Y=1, N=0)

df['Loan_Status'] = encoder.fit_transform(df['Loan_Status'])

print(df[['Loan_Status']].head())

Loan_Status
0 1
1 0
2 1
3 1
4 1

In [83]: pip install matplotlib

Requirement already satisfied: matplotlib in c:\users\khadersha\anaconda3\lib\site-packages (3.8.4)

Requirement already satisfied: contourpy>=1.0.1 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (1.2.0)
Requirement already satisfied: cycler>=0.10 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (4.51.0)
Requirement already satisfied: kiwisolver>=1.3.1 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (1.4.4)
Requirement already satisfied: numpy>=1.21 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (1.26.4)
Requirement already satisfied: packaging>=20.0 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (23.2)
Requirement already satisfied: pillow>=8 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (10.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\khadersha\anaconda3\lib\site-packages (from matplotlib) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in c:\users\khadersha\anaconda3\lib\site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Note: you may need to restart the kernel to use updated packages.

In [84]: import matplotlib.pyplot as plt

import pandas as pd
import numpy as np

In [86]: # Sample DataFrame

data = {
'Gender': ['Male', 'Male', 'Female', 'Female', 'Male', 'Female'],
'Age': [23, 35, 29, 42, 55, 60],
'Income': [50000, 60000, 45000, 70000, 80000, 75000],
'LoanAmount': [200, 250, 150, 300, 350, 400],
'LoanStatus': ['Y', 'N', 'Y', 'Y', 'N', 'Y']
}

df = pd.DataFrame(data)

In [87]: # Bar graph for 'Gender' count

gender_count = df['Gender'].value_counts()

plt.bar(gender_count.index, gender_count.values, color=['blue', 'orange'])

plt.xlabel('Gender')
plt.ylabel('Count')
plt.title('Gender Distribution')
plt.show()

In [88]: # Pie chart for 'LoanStatus'

loan_status_count = df['LoanStatus'].value_counts()

plt.pie(loan_status_count, labels=loan_status_count.index, autopct='%1.1f%%', colors=['green', 'red'])

plt.title('Loan Status Distribution')
plt.show()

In [89]: # Histogram for 'LoanAmount'

plt.hist(df['LoanAmount'], bins=5, color='skyblue', edgecolor='black')
plt.title('Loan Amount Distribution')
plt.xlabel('Loan Amount')
plt.ylabel('Frequency')
plt.show()

In [90]: # Line chart for 'Income' vs 'Age'

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.plot(df['Age'], df['Income'], marker='o', color='purple')
plt.title('Income vs Age')
plt.xlabel('Age')
plt.ylabel('Income')

# Line chart for 'LoanAmount' vs 'Income'

plt.subplot(1, 2, 2)
plt.plot(df['Income'], df['LoanAmount'], marker='x', color='orange')
plt.title('Loan Amount vs Income')
plt.xlabel('Income')
plt.ylabel('Loan Amount')

plt.tight_layout()
plt.show()

In [91]: # Scatter plot for 'Income' vs 'LoanAmount'

plt.scatter(df['Income'], df['LoanAmount'], color='red')
plt.title('Income vs Loan Amount')
plt.xlabel('Income')
plt.ylabel('Loan Amount')
plt.show()

In [92]: !pip install nltk

Requirement already satisfied: nltk in c:\users\khadersha\anaconda3\lib\site-packages (3.8.1)

Requirement already satisfied: click in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (8.1.7)
Requirement already satisfied: joblib in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (1.4.2)
Requirement already satisfied: regex>=2021.8.3 in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (2023.10.3)
Requirement already satisfied: tqdm in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (4.66.4)
Requirement already satisfied: colorama in c:\users\khadersha\anaconda3\lib\site-packages (from click->nltk) (0.4.6)

In [93]: import nltk

# Download popular NLTK datasets and corpora (like stopwords, punkt tokenizer, etc.)
nltk.download('punkt') # Tokenizer models
nltk.download('stopwords') # Common stopwords
nltk.download('wordnet') # WordNet corpus for lemmatization

[nltk_data] Downloading package punkt to

[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Unzipping tokenizers\punkt.zip.
[nltk_data] Downloading package stopwords to
[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Unzipping corpora\stopwords.zip.
[nltk_data] Downloading package wordnet to
[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
Out[93]: True

In [94]: from nltk.tokenize import word_tokenize

text = "Natural language processing with NLTK is fun!"

tokens = word_tokenize(text)

print(tokens)

['Natural', 'language', 'processing', 'with', 'NLTK', 'is', 'fun', '!']

In [95]: !pip install nltk scikit-learn

Requirement already satisfied: nltk in c:\users\khadersha\anaconda3\lib\site-packages (3.8.1)

Requirement already satisfied: scikit-learn in c:\users\khadersha\anaconda3\lib\site-packages (1.4.2)
Requirement already satisfied: click in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (8.1.7)
Requirement already satisfied: joblib in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (1.4.2)
Requirement already satisfied: regex>=2021.8.3 in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (2023.10.3)
Requirement already satisfied: tqdm in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (4.66.4)
Requirement already satisfied: numpy>=1.19.5 in c:\users\khadersha\anaconda3\lib\site-packages (from scikit-learn) (1.26.4)
Requirement already satisfied: scipy>=1.6.0 in c:\users\khadersha\anaconda3\lib\site-packages (from scikit-learn) (1.13.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\khadersha\anaconda3\lib\site-packages (from scikit-learn) (2.2.0)
Requirement already satisfied: colorama in c:\users\khadersha\anaconda3\lib\site-packages (from click->nltk) (0.4.6)

In [96]: import nltk

from nltk.corpus import movie_reviews
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

In [97]: nltk.download('movie_reviews')

[nltk_data] Downloading package movie_reviews to

[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Unzipping corpora\movie_reviews.zip.
Out[97]: True

In [98]: # Load the movie reviews data from NLTK

from nltk.corpus import movie_reviews

# Get the fileids (positive and negative categories)

file_ids = movie_reviews.fileids()

# Extract features (text) and labels (sentiment)

texts = [movie_reviews.raw(file_id) for file_id in file_ids]
labels = [file_id.split('/')[0] for file_id in file_ids] # 'pos' or 'neg'

# Split the data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2, random_state=42)

print(f"Number of training samples: {len(X_train)}")

print(f"Number of test samples: {len(X_test)}")

Number of training samples: 1600

Number of test samples: 400

In [99]: from nltk.corpus import stopwords

from nltk.tokenize import word_tokenize

# Download necessary data

nltk.download('punkt')
nltk.download('stopwords')

# Tokenize and remove stopwords

stop_words = set(stopwords.words('english'))

def preprocess_text(text):
# Tokenize the text
tokens = word_tokenize(text)

# Remove stopwords
filtered_tokens = [word for word in tokens if word.lower() not in stop_words]

return ' '.join(filtered_tokens)

# Apply preprocessing
X_train_processed = [preprocess_text(text) for text in X_train]
X_test_processed = [preprocess_text(text) for text in X_test]

[nltk_data] Downloading package punkt to

[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Package stopwords is already up-to-date!

In [100… # Convert text data into a bag-of-words representation

vectorizer = CountVectorizer()

# Fit and transform the training data

X_train_vec = vectorizer.fit_transform(X_train_processed)

# Transform the test data (using the same vectorizer)

X_test_vec = vectorizer.transform(X_test_processed)

In [101… # Initialize and train the Naive Bayes classifier

model = MultinomialNB()
model.fit(X_train_vec, y_train)

Out[101… ▾ MultinomialNB i ?

MultinomialNB()

In [102… # Make predictions on the test data

y_pred = model.predict(X_test_vec)

# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)

print(f"Accuracy: {accuracy * 100:.2f}%")

Accuracy: 81.00%

In [103… import nltk

from nltk.corpus import movie_reviews
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Download necessary NLTK data

nltk.download('movie_reviews')
nltk.download('punkt')
nltk.download('stopwords')

# Load the movie reviews data from NLTK

file_ids = movie_reviews.fileids()
texts = [movie_reviews.raw(file_id) for file_id in file_ids]
labels = [file_id.split('/')[0] for file_id in file_ids]

# Split the data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2, random_state=42)

# Tokenize and remove stopwords

stop_words = set(stopwords.words('english'))

def preprocess_text(text):
tokens = word_tokenize(text)
filtered_tokens = [word for word in tokens if word.lower() not in stop_words]
return ' '.join(filtered_tokens)

# Apply preprocessing
X_train_processed = [preprocess_text(text) for text in X_train]
X_test_processed = [preprocess_text(text) for text in X_test]

# Convert text data into a bag-of-words representation

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train_processed)
X_test_vec = vectorizer.transform(X_test_processed)

# Train the Naive Bayes classifier

model = MultinomialNB()
model.fit(X_train_vec, y_train)

# Make predictions
y_pred = model.predict(X_test_vec)

# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy * 100:.2f}%")

[nltk_data] Downloading package movie_reviews to

[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Package movie_reviews is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data] C:\Users\KHADERSHA\AppData\Roaming\nltk_data...
[nltk_data] Package stopwords is already up-to-date!
Accuracy: 81.00%

In [106… !pip install nltk spacy

Requirement already satisfied: nltk in c:\users\khadersha\anaconda3\lib\site-packages (3.8.1)

Collecting spacy
Using cached spacy-3.8.4-cp312-cp312-win_amd64.whl.metadata (27 kB)
Requirement already satisfied: click in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (8.1.7)
Requirement already satisfied: joblib in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (1.4.2)
Requirement already satisfied: regex>=2021.8.3 in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (2023.10.3)
Requirement already satisfied: tqdm in c:\users\khadersha\anaconda3\lib\site-packages (from nltk) (4.66.4)
Collecting spacy-legacy<3.1.0,>=3.0.11 (from spacy)
Downloading spacy_legacy-3.0.12-py2.py3-none-any.whl.metadata (2.8 kB)
Collecting spacy-loggers<2.0.0,>=1.0.0 (from spacy)
Downloading spacy_loggers-1.0.5-py3-none-any.whl.metadata (23 kB)
Collecting murmurhash<1.1.0,>=0.28.0 (from spacy)
Downloading murmurhash-1.0.12-cp312-cp312-win_amd64.whl.metadata (2.2 kB)
Collecting cymem<2.1.0,>=2.0.2 (from spacy)
Downloading cymem-2.0.11-cp312-cp312-win_amd64.whl.metadata (8.8 kB)
Collecting preshed<3.1.0,>=3.0.2 (from spacy)
Downloading preshed-3.0.9-cp312-cp312-win_amd64.whl.metadata (2.2 kB)
Collecting thinc<8.4.0,>=8.3.4 (from spacy)
Downloading thinc-8.3.4-cp312-cp312-win_amd64.whl.metadata (15 kB)
Collecting wasabi<1.2.0,>=0.9.1 (from spacy)
Downloading wasabi-1.1.3-py3-none-any.whl.metadata (28 kB)
Collecting srsly<3.0.0,>=2.4.3 (from spacy)
Downloading srsly-2.5.1-cp312-cp312-win_amd64.whl.metadata (20 kB)
Collecting catalogue<2.1.0,>=2.0.6 (from spacy)
Downloading catalogue-2.0.10-py3-none-any.whl.metadata (14 kB)
Collecting weasel<0.5.0,>=0.1.0 (from spacy)
Downloading weasel-0.4.1-py3-none-any.whl.metadata (4.6 kB)
Collecting typer<1.0.0,>=0.3.0 (from spacy)
Downloading typer-0.15.2-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: numpy>=1.19.0 in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (1.26.4)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (2.32.2)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (2.5.3)
Requirement already satisfied: jinja2 in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (3.1.4)
Requirement already satisfied: setuptools in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (69.5.1)
Requirement already satisfied: packaging>=20.0 in c:\users\khadersha\anaconda3\lib\site-packages (from spacy) (23.2)
Collecting langcodes<4.0.0,>=3.2.0 (from spacy)
Downloading langcodes-3.5.0-py3-none-any.whl.metadata (29 kB)
Collecting language-data>=1.2 (from langcodes<4.0.0,>=3.2.0->spacy)
Downloading language_data-1.3.0-py3-none-any.whl.metadata (4.3 kB)
Requirement already satisfied: annotated-types>=0.4.0 in c:\users\khadersha\anaconda3\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (0.6.0)
Requirement already satisfied: pydantic-core==2.14.6 in c:\users\khadersha\anaconda3\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (2.14.6)
Requirement already satisfied: typing-extensions>=4.6.1 in c:\users\khadersha\anaconda3\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.11.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\khadersha\anaconda3\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in c:\users\khadersha\anaconda3\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\khadersha\anaconda3\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\khadersha\anaconda3\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (2024.8.30)
Collecting blis<1.3.0,>=1.2.0 (from thinc<8.4.0,>=8.3.4->spacy)
Downloading blis-1.2.0-cp312-cp312-win_amd64.whl.metadata (7.9 kB)
Collecting confection<1.0.0,>=0.0.1 (from thinc<8.4.0,>=8.3.4->spacy)
Downloading confection-0.1.5-py3-none-any.whl.metadata (19 kB)
Requirement already satisfied: colorama in c:\users\khadersha\anaconda3\lib\site-packages (from tqdm->nltk) (0.4.6)
Collecting shellingham>=1.3.0 (from typer<1.0.0,>=0.3.0->spacy)
Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: rich>=10.11.0 in c:\users\khadersha\anaconda3\lib\site-packages (from typer<1.0.0,>=0.3.0->spacy) (13.3.5)
Collecting cloudpathlib<1.0.0,>=0.7.0 (from weasel<0.5.0,>=0.1.0->spacy)
Downloading cloudpathlib-0.21.0-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in c:\users\khadersha\anaconda3\lib\site-packages (from weasel<0.5.0,>=0.1.0->spacy) (5.2.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\khadersha\anaconda3\lib\site-packages (from jinja2->spacy) (2.1.3)
Collecting marisa-trie>=1.1.0 (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy)
Downloading marisa_trie-1.2.1-cp312-cp312-win_amd64.whl.metadata (9.3 kB)
Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in c:\users\khadersha\anaconda3\lib\site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (2.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\users\khadersha\anaconda3\lib\site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (2.15.1)
Requirement already satisfied: mdurl~=0.1 in c:\users\khadersha\anaconda3\lib\site-packages (from markdown-it-py<3.0.0,>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (0.1.0)
Downloading spacy-3.8.4-cp312-cp312-win_amd64.whl (11.8 MB)
---------------------------------------- 0.0/11.8 MB ? eta -:--:--
---------------------------------------- 0.1/11.8 MB 3.2 MB/s eta 0:00:04
---------------------------------------- 0.1/11.8 MB 1.8 MB/s eta 0:00:07
--------------------------------------- 0.3/11.8 MB 2.2 MB/s eta 0:00:06
- -------------------------------------- 0.4/11.8 MB 2.7 MB/s eta 0:00:05
- -------------------------------------- 0.5/11.8 MB 2.9 MB/s eta 0:00:04
-- ------------------------------------- 0.8/11.8 MB 3.3 MB/s eta 0:00:04
--- ------------------------------------ 1.0/11.8 MB 3.5 MB/s eta 0:00:04
---- ----------------------------------- 1.3/11.8 MB 4.2 MB/s eta 0:00:03
----- ---------------------------------- 1.5/11.8 MB 4.1 MB/s eta 0:00:03
------ --------------------------------- 1.8/11.8 MB 4.6 MB/s eta 0:00:03
------- -------------------------------- 2.1/11.8 MB 4.8 MB/s eta 0:00:03
-------- ------------------------------- 2.4/11.8 MB 5.2 MB/s eta 0:00:02
--------- ------------------------------ 2.7/11.8 MB 5.3 MB/s eta 0:00:02
---------- ----------------------------- 3.0/11.8 MB 5.3 MB/s eta 0:00:02
----------- ---------------------------- 3.3/11.8 MB 5.5 MB/s eta 0:00:02
------------ --------------------------- 3.6/11.8 MB 5.6 MB/s eta 0:00:02
------------- -------------------------- 3.9/11.8 MB 5.7 MB/s eta 0:00:02
-------------- ------------------------- 4.2/11.8 MB 5.9 MB/s eta 0:00:02
--------------- ------------------------ 4.6/11.8 MB 6.0 MB/s eta 0:00:02
---------------- ----------------------- 4.9/11.8 MB 6.1 MB/s eta 0:00:02
----------------- ---------------------- 5.2/11.8 MB 6.1 MB/s eta 0:00:02
------------------- -------------------- 5.6/11.8 MB 6.3 MB/s eta 0:00:01
------------------- -------------------- 5.9/11.8 MB 6.3 MB/s eta 0:00:01
--------------------- ------------------ 6.3/11.8 MB 6.5 MB/s eta 0:00:01
---------------------- ----------------- 6.7/11.8 MB 6.6 MB/s eta 0:00:01
----------------------- ---------------- 7.0/11.8 MB 6.7 MB/s eta 0:00:01
----------------------- ---------------- 7.0/11.8 MB 6.7 MB/s eta 0:00:01
----------------------- ---------------- 7.0/11.8 MB 6.7 MB/s eta 0:00:01
------------------------ --------------- 7.2/11.8 MB 6.0 MB/s eta 0:00:01
------------------------ --------------- 7.3/11.8 MB 5.9 MB/s eta 0:00:01
------------------------- -------------- 7.5/11.8 MB 5.9 MB/s eta 0:00:01
------------------------- -------------- 7.5/11.8 MB 5.9 MB/s eta 0:00:01
-------------------------- ------------- 7.7/11.8 MB 5.7 MB/s eta 0:00:01
-------------------------- ------------- 7.8/11.8 MB 5.6 MB/s eta 0:00:01
-------------------------- ------------- 7.8/11.8 MB 5.4 MB/s eta 0:00:01
-------------------------- ------------- 7.9/11.8 MB 5.3 MB/s eta 0:00:01
---------------------------- ----------- 8.3/11.8 MB 5.4 MB/s eta 0:00:01
---------------------------- ----------- 8.5/11.8 MB 5.4 MB/s eta 0:00:01
------------------------------ --------- 8.9/11.8 MB 5.5 MB/s eta 0:00:01
------------------------------ --------- 9.0/11.8 MB 5.5 MB/s eta 0:00:01
------------------------------ --------- 9.0/11.8 MB 5.5 MB/s eta 0:00:01
------------------------------ --------- 9.0/11.8 MB 5.5 MB/s eta 0:00:01
------------------------------- -------- 9.3/11.8 MB 5.3 MB/s eta 0:00:01
-------------------------------- ------- 9.5/11.8 MB 5.2 MB/s eta 0:00:01
-------------------------------- ------- 9.6/11.8 MB 5.2 MB/s eta 0:00:01
--------------------------------- ------ 9.7/11.8 MB 5.1 MB/s eta 0:00:01
--------------------------------- ------ 9.8/11.8 MB 5.1 MB/s eta 0:00:01
---------------------------------- ----- 10.0/11.8 MB 5.1 MB/s eta 0:00:01
---------------------------------- ----- 10.2/11.8 MB 5.0 MB/s eta 0:00:01
----------------------------------- ---- 10.3/11.8 MB 5.2 MB/s eta 0:00:01
----------------------------------- ---- 10.5/11.8 MB 5.2 MB/s eta 0:00:01
----------------------------------- ---- 10.6/11.8 MB 5.1 MB/s eta 0:00:01
------------------------------------ --- 10.8/11.8 MB 5.2 MB/s eta 0:00:01
------------------------------------- -- 10.9/11.8 MB 5.1 MB/s eta 0:00:01
------------------------------------- -- 11.1/11.8 MB 5.1 MB/s eta 0:00:01
-------------------------------------- - 11.3/11.8 MB 5.1 MB/s eta 0:00:01
-------------------------------------- - 11.5/11.8 MB 5.0 MB/s eta 0:00:01
--------------------------------------- 11.7/11.8 MB 5.0 MB/s eta 0:00:01
--------------------------------------- 11.7/11.8 MB 5.0 MB/s eta 0:00:01
--------------------------------------- 11.7/11.8 MB 4.9 MB/s eta 0:00:01
--------------------------------------- 11.8/11.8 MB 4.9 MB/s eta 0:00:01
---------------------------------------- 11.8/11.8 MB 4.8 MB/s eta 0:00:00
Downloading catalogue-2.0.10-py3-none-any.whl (17 kB)
Downloading cymem-2.0.11-cp312-cp312-win_amd64.whl (39 kB)
Downloading langcodes-3.5.0-py3-none-any.whl (182 kB)
---------------------------------------- 0.0/183.0 kB ? eta -:--:--
-------------------------- ------------- 122.9/183.0 kB 7.5 MB/s eta 0:00:01
---------------------------------------- 183.0/183.0 kB 3.8 MB/s eta 0:00:00
Downloading murmurhash-1.0.12-cp312-cp312-win_amd64.whl (25 kB)
Downloading preshed-3.0.9-cp312-cp312-win_amd64.whl (122 kB)
---------------------------------------- 0.0/122.4 kB ? eta -:--:--
---------------------------------------- 122.4/122.4 kB 7.5 MB/s eta 0:00:00
Downloading spacy_legacy-3.0.12-py2.py3-none-any.whl (29 kB)
Downloading spacy_loggers-1.0.5-py3-none-any.whl (22 kB)
Downloading srsly-2.5.1-cp312-cp312-win_amd64.whl (632 kB)
---------------------------------------- 0.0/632.6 kB ? eta -:--:--
----------- ---------------------------- 174.1/632.6 kB 3.5 MB/s eta 0:00:01
---------------- ----------------------- 256.0/632.6 kB 3.2 MB/s eta 0:00:01
-------------------------- ------------- 419.8/632.6 kB 3.3 MB/s eta 0:00:01
--------------------------------- ------ 532.5/632.6 kB 3.3 MB/s eta 0:00:01
--------------------------------------- 624.6/632.6 kB 3.0 MB/s eta 0:00:01
---------------------------------------- 632.6/632.6 kB 2.7 MB/s eta 0:00:00
Downloading thinc-8.3.4-cp312-cp312-win_amd64.whl (1.5 MB)
---------------------------------------- 0.0/1.5 MB ? eta -:--:--
--- ------------------------------------ 0.1/1.5 MB 3.3 MB/s eta 0:00:01
----- ---------------------------------- 0.2/1.5 MB 2.5 MB/s eta 0:00:01
---------- ----------------------------- 0.4/1.5 MB 3.3 MB/s eta 0:00:01
------------ --------------------------- 0.5/1.5 MB 2.8 MB/s eta 0:00:01
---------------- ----------------------- 0.6/1.5 MB 3.0 MB/s eta 0:00:01
------------------- -------------------- 0.7/1.5 MB 3.2 MB/s eta 0:00:01
----------------------- ---------------- 0.9/1.5 MB 3.1 MB/s eta 0:00:01
------------------------- -------------- 0.9/1.5 MB 3.1 MB/s eta 0:00:01
----------------------------- ---------- 1.1/1.5 MB 3.1 MB/s eta 0:00:01
--------------------------------- ------ 1.2/1.5 MB 3.1 MB/s eta 0:00:01
-------------------------------------- - 1.4/1.5 MB 3.2 MB/s eta 0:00:01
---------------------------------------- 1.5/1.5 MB 3.1 MB/s eta 0:00:00
Downloading typer-0.15.2-py3-none-any.whl (45 kB)
---------------------------------------- 0.0/45.1 kB ? eta -:--:--
---------------------------------------- 45.1/45.1 kB ? eta 0:00:00
Downloading wasabi-1.1.3-py3-none-any.whl (27 kB)
Downloading weasel-0.4.1-py3-none-any.whl (50 kB)
---------------------------------------- 0.0/50.3 kB ? eta -:--:--
---------------------------------------- 50.3/50.3 kB ? eta 0:00:00
Downloading blis-1.2.0-cp312-cp312-win_amd64.whl (6.3 MB)
---------------------------------------- 0.0/6.3 MB ? eta -:--:--
- -------------------------------------- 0.2/6.3 MB 5.9 MB/s eta 0:00:02
- -------------------------------------- 0.3/6.3 MB 3.8 MB/s eta 0:00:02
-- ------------------------------------- 0.4/6.3 MB 3.9 MB/s eta 0:00:02
--- ------------------------------------ 0.6/6.3 MB 3.3 MB/s eta 0:00:02
---- ----------------------------------- 0.7/6.3 MB 3.6 MB/s eta 0:00:02
----- ---------------------------------- 0.9/6.3 MB 3.7 MB/s eta 0:00:02
------ --------------------------------- 1.1/6.3 MB 3.8 MB/s eta 0:00:02
------- -------------------------------- 1.2/6.3 MB 3.7 MB/s eta 0:00:02
-------- ------------------------------- 1.3/6.3 MB 3.5 MB/s eta 0:00:02
--------- ------------------------------ 1.5/6.3 MB 3.6 MB/s eta 0:00:02
---------- ----------------------------- 1.6/6.3 MB 3.6 MB/s eta 0:00:02
----------- ---------------------------- 1.8/6.3 MB 3.5 MB/s eta 0:00:02
------------ --------------------------- 1.9/6.3 MB 3.6 MB/s eta 0:00:02
------------- -------------------------- 2.1/6.3 MB 3.6 MB/s eta 0:00:02
-------------- ------------------------- 2.2/6.3 MB 3.5 MB/s eta 0:00:02
-------------- ------------------------- 2.3/6.3 MB 3.4 MB/s eta 0:00:02
--------------- ------------------------ 2.5/6.3 MB 3.5 MB/s eta 0:00:02
---------------- ----------------------- 2.7/6.3 MB 3.5 MB/s eta 0:00:02
----------------- ---------------------- 2.8/6.3 MB 3.6 MB/s eta 0:00:01
------------------ --------------------- 2.9/6.3 MB 3.5 MB/s eta 0:00:01
------------------- -------------------- 3.1/6.3 MB 3.5 MB/s eta 0:00:01
-------------------- ------------------- 3.2/6.3 MB 3.5 MB/s eta 0:00:01
--------------------- ------------------ 3.4/6.3 MB 3.5 MB/s eta 0:00:01
---------------------- ----------------- 3.5/6.3 MB 3.4 MB/s eta 0:00:01
----------------------- ---------------- 3.6/6.3 MB 3.4 MB/s eta 0:00:01
----------------------- ---------------- 3.7/6.3 MB 3.4 MB/s eta 0:00:01
------------------------ --------------- 3.9/6.3 MB 3.4 MB/s eta 0:00:01
------------------------- -------------- 4.0/6.3 MB 3.4 MB/s eta 0:00:01
-------------------------- ------------- 4.1/6.3 MB 3.4 MB/s eta 0:00:01
--------------------------- ------------ 4.3/6.3 MB 3.4 MB/s eta 0:00:01
---------------------------- ----------- 4.4/6.3 MB 3.4 MB/s eta 0:00:01
----------------------------- ---------- 4.6/6.3 MB 3.4 MB/s eta 0:00:01
----------------------------- ---------- 4.6/6.3 MB 3.4 MB/s eta 0:00:01
------------------------------- -------- 4.9/6.3 MB 3.5 MB/s eta 0:00:01
------------------------------- -------- 5.0/6.3 MB 3.4 MB/s eta 0:00:01
-------------------------------- ------- 5.1/6.3 MB 3.4 MB/s eta 0:00:01
--------------------------------- ------ 5.2/6.3 MB 3.4 MB/s eta 0:00:01
---------------------------------- ----- 5.4/6.3 MB 3.5 MB/s eta 0:00:01
----------------------------------- ---- 5.6/6.3 MB 3.5 MB/s eta 0:00:01
------------------------------------ --- 5.7/6.3 MB 3.5 MB/s eta 0:00:01
------------------------------------- -- 5.9/6.3 MB 3.4 MB/s eta 0:00:01
-------------------------------------- - 6.0/6.3 MB 3.5 MB/s eta 0:00:01
--------------------------------------- 6.2/6.3 MB 3.5 MB/s eta 0:00:01
--------------------------------------- 6.3/6.3 MB 3.5 MB/s eta 0:00:01
---------------------------------------- 6.3/6.3 MB 3.4 MB/s eta 0:00:00
Downloading cloudpathlib-0.21.0-py3-none-any.whl (52 kB)
---------------------------------------- 0.0/52.7 kB ? eta -:--:--
---------------------------------------- 52.7/52.7 kB 2.8 MB/s eta 0:00:00
Downloading confection-0.1.5-py3-none-any.whl (35 kB)
Downloading language_data-1.3.0-py3-none-any.whl (5.4 MB)
---------------------------------------- 0.0/5.4 MB ? eta -:--:--
- -------------------------------------- 0.2/5.4 MB 5.9 MB/s eta 0:00:01
-- ------------------------------------- 0.3/5.4 MB 4.1 MB/s eta 0:00:02
--- ------------------------------------ 0.5/5.4 MB 4.4 MB/s eta 0:00:02
---- ----------------------------------- 0.6/5.4 MB 3.9 MB/s eta 0:00:02
----- ---------------------------------- 0.8/5.4 MB 4.1 MB/s eta 0:00:02
------ --------------------------------- 0.9/5.4 MB 3.9 MB/s eta 0:00:02
------- -------------------------------- 1.0/5.4 MB 3.7 MB/s eta 0:00:02
--------- ------------------------------ 1.2/5.4 MB 3.7 MB/s eta 0:00:02
--------- ------------------------------ 1.3/5.4 MB 3.7 MB/s eta 0:00:02
----------- ---------------------------- 1.5/5.4 MB 3.8 MB/s eta 0:00:02
------------ --------------------------- 1.6/5.4 MB 3.7 MB/s eta 0:00:01
------------- -------------------------- 1.8/5.4 MB 3.8 MB/s eta 0:00:01
-------------- ------------------------- 1.9/5.4 MB 3.6 MB/s eta 0:00:01
--------------- ------------------------ 2.1/5.4 MB 3.7 MB/s eta 0:00:01
---------------- ----------------------- 2.3/5.4 MB 3.7 MB/s eta 0:00:01
------------------ --------------------- 2.4/5.4 MB 3.7 MB/s eta 0:00:01
------------------- -------------------- 2.6/5.4 MB 3.7 MB/s eta 0:00:01
-------------------- ------------------- 2.7/5.4 MB 3.7 MB/s eta 0:00:01
--------------------- ------------------ 2.9/5.4 MB 3.7 MB/s eta 0:00:01
---------------------- ----------------- 3.0/5.4 MB 3.7 MB/s eta 0:00:01
----------------------- ---------------- 3.2/5.4 MB 3.8 MB/s eta 0:00:01
------------------------ --------------- 3.3/5.4 MB 3.7 MB/s eta 0:00:01
------------------------- -------------- 3.5/5.4 MB 3.7 MB/s eta 0:00:01
-------------------------- ------------- 3.6/5.4 MB 3.7 MB/s eta 0:00:01
--------------------------- ------------ 3.7/5.4 MB 3.6 MB/s eta 0:00:01
---------------------------- ----------- 3.8/5.4 MB 3.7 MB/s eta 0:00:01
---------------------------- ----------- 3.9/5.4 MB 3.6 MB/s eta 0:00:01
----------------------------- ---------- 4.0/5.4 MB 3.6 MB/s eta 0:00:01
------------------------------- -------- 4.2/5.4 MB 3.6 MB/s eta 0:00:01
-------------------------------- ------- 4.3/5.4 MB 3.5 MB/s eta 0:00:01
--------------------------------- ------ 4.5/5.4 MB 3.6 MB/s eta 0:00:01
---------------------------------- ----- 4.6/5.4 MB 3.6 MB/s eta 0:00:01
----------------------------------- ---- 4.7/5.4 MB 3.5 MB/s eta 0:00:01
------------------------------------ --- 4.9/5.4 MB 3.5 MB/s eta 0:00:01
------------------------------------- -- 5.1/5.4 MB 3.5 MB/s eta 0:00:01
-------------------------------------- - 5.2/5.4 MB 3.5 MB/s eta 0:00:01
--------------------------------------- 5.4/5.4 MB 3.5 MB/s eta 0:00:01
---------------------------------------- 5.4/5.4 MB 3.4 MB/s eta 0:00:00
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading marisa_trie-1.2.1-cp312-cp312-win_amd64.whl (150 kB)
---------------------------------------- 0.0/150.8 kB ? eta -:--:--
-------------------------------- ------- 122.9/150.8 kB 7.0 MB/s eta 0:00:01
---------------------------------------- 150.8/150.8 kB 3.0 MB/s eta 0:00:00
Installing collected packages: cymem, wasabi, spacy-loggers, spacy-legacy, shellingham, murmurhash, marisa-trie, cloudpathlib, catalogue, blis, srsly, preshed, language-data, typer, langcodes, confection, weasel, thinc, spacy
Successfully installed blis-1.2.0 catalogue-2.0.10 cloudpathlib-0.21.0 confection-0.1.5 cymem-2.0.11 langcodes-3.5.0 language-data-1.3.0 marisa-trie-1.2.1 murmurhash-1.0.12 preshed-3.0.9 shellingham-1.5.4 spacy-3.8.4 spacy-legacy-
3.0.12 spacy-loggers-1.0.5 srsly-2.5.1 thinc-8.3.4 typer-0.15.2 wasabi-1.1.3 weasel-0.4.1

In [ ]: import nltk
import spacy
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from spacy import displacy

# Ensure you have the necessary NLTK resources

nltk.download('punkt')
nltk.download('stopwords')

# Initialize spaCy model

nlp = spacy.load('en_core_web_sm')

# Text for analysis

text = "Apple is looking at buying U.K. startup for $1 billion. " \
"Barack Obama was the 44th President of the United States."

# Step 1: Tokenization using NLTK

tokens = word_tokenize(text)
print("Tokens using NLTK:", tokens)

# Step 2: Stopwords Removal using NLTK

stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word.lower() not in stop_words]
print("Tokens after removing stopwords using NLTK:", filtered_tokens)

# Step 3: Named Entity Recognition (NER) using spaCy

doc = nlp(text)
print("Named Entities using spaCy:")
for ent in doc.ents:
print(f"Text: {ent.text}, Label: {ent.label_}")

# Visualize the named entities (Uncomment to run in a Jupyter Notebook or similar environment)
# displacy.render(doc, style="ent")

# Step 4: Lemmatization using spaCy (alternative to PyNLPI)

lemmatized_tokens = [token.lemma_ for token in doc]
print("Lemmatized Tokens using spaCy:", lemmatized_tokens)

# Step 5: Part-of-Speech (POS) Tagging using spaCy

print("POS Tags using spaCy:")
for token in doc:
print(f"Text: {token.text}, POS: {token.pos_}")

In [109… !pip install --upgrade pydantic

Requirement already satisfied: pydantic in c:\users\khadersha\anaconda3\lib\site-packages (2.5.3)

Collecting pydantic
Downloading pydantic-2.10.6-py3-none-any.whl.metadata (30 kB)
Requirement already satisfied: annotated-types>=0.6.0 in c:\users\khadersha\anaconda3\lib\site-packages (from pydantic) (0.6.0)
Collecting pydantic-core==2.27.2 (from pydantic)
Downloading pydantic_core-2.27.2-cp312-cp312-win_amd64.whl.metadata (6.7 kB)
Collecting typing-extensions>=4.12.2 (from pydantic)
Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Downloading pydantic-2.10.6-py3-none-any.whl (431 kB)
---------------------------------------- 0.0/431.7 kB ? eta -:--:--
---------------------------------------- 0.0/431.7 kB ? eta -:--:--
---------------------------------------- 0.0/431.7 kB ? eta -:--:--
--- ------------------------------------ 41.0/431.7 kB 1.9 MB/s eta 0:00:01
----- ---------------------------------- 61.4/431.7 kB 1.6 MB/s eta 0:00:01
-------- ------------------------------ 92.2/431.7 kB 871.5 kB/s eta 0:00:01
--------- ---------------------------- 102.4/431.7 kB 737.3 kB/s eta 0:00:01
---------- --------------------------- 122.9/431.7 kB 654.9 kB/s eta 0:00:01
------------ ------------------------- 143.4/431.7 kB 708.1 kB/s eta 0:00:01
--------------- ---------------------- 174.1/431.7 kB 615.9 kB/s eta 0:00:01
---------------- --------------------- 184.3/431.7 kB 556.2 kB/s eta 0:00:01
------------------ ------------------- 204.8/431.7 kB 593.2 kB/s eta 0:00:01
------------------- ------------------ 225.3/431.7 kB 573.4 kB/s eta 0:00:01
-------------------- ----------------- 235.5/431.7 kB 553.5 kB/s eta 0:00:01
----------------------- -------------- 266.2/431.7 kB 565.1 kB/s eta 0:00:01
------------------------- ------------ 286.7/431.7 kB 553.0 kB/s eta 0:00:01
--------------------------- ---------- 307.2/431.7 kB 542.9 kB/s eta 0:00:01
----------------------------- -------- 337.9/431.7 kB 551.6 kB/s eta 0:00:01
-------------------------------- ----- 368.6/431.7 kB 533.3 kB/s eta 0:00:01
---------------------------------- --- 389.1/431.7 kB 550.7 kB/s eta 0:00:01
------------------------------------ - 419.8/431.7 kB 546.1 kB/s eta 0:00:01
-------------------------------------- 431.7/431.7 kB 550.2 kB/s eta 0:00:00
Downloading pydantic_core-2.27.2-cp312-cp312-win_amd64.whl (2.0 MB)
---------------------------------------- 0.0/2.0 MB ? eta -:--:--
--------------------------------------- 0.0/2.0 MB 660.6 kB/s eta 0:00:03
--------------------------------------- 0.0/2.0 MB 653.6 kB/s eta 0:00:03
- -------------------------------------- 0.1/2.0 MB 660.6 kB/s eta 0:00:03
- -------------------------------------- 0.1/2.0 MB 585.1 kB/s eta 0:00:04
-- ------------------------------------- 0.1/2.0 MB 656.4 kB/s eta 0:00:03
-- ------------------------------------- 0.1/2.0 MB 610.6 kB/s eta 0:00:04
--- ------------------------------------ 0.2/2.0 MB 618.3 kB/s eta 0:00:03
--- ------------------------------------ 0.2/2.0 MB 620.6 kB/s eta 0:00:03
---- ----------------------------------- 0.2/2.0 MB 626.9 kB/s eta 0:00:03
---- ----------------------------------- 0.2/2.0 MB 600.7 kB/s eta 0:00:03
----- ---------------------------------- 0.3/2.0 MB 605.3 kB/s eta 0:00:03
----- ---------------------------------- 0.3/2.0 MB 609.2 kB/s eta 0:00:03
----- ---------------------------------- 0.3/2.0 MB 571.2 kB/s eta 0:00:03
----- ---------------------------------- 0.3/2.0 MB 571.2 kB/s eta 0:00:03
------- -------------------------------- 0.4/2.0 MB 587.1 kB/s eta 0:00:03
------- -------------------------------- 0.4/2.0 MB 560.0 kB/s eta 0:00:03
-------- ------------------------------- 0.4/2.0 MB 566.4 kB/s eta 0:00:03
-------- ------------------------------- 0.4/2.0 MB 556.2 kB/s eta 0:00:03
-------- ------------------------------- 0.4/2.0 MB 548.8 kB/s eta 0:00:03
--------- ------------------------------ 0.5/2.0 MB 554.9 kB/s eta 0:00:03
--------- ------------------------------ 0.5/2.0 MB 548.4 kB/s eta 0:00:03
--------- ------------------------------ 0.5/2.0 MB 550.0 kB/s eta 0:00:03
---------- ----------------------------- 0.5/2.0 MB 548.0 kB/s eta 0:00:03
---------- ----------------------------- 0.5/2.0 MB 548.0 kB/s eta 0:00:03
---------- ----------------------------- 0.5/2.0 MB 532.5 kB/s eta 0:00:03
----------- ---------------------------- 0.6/2.0 MB 536.3 kB/s eta 0:00:03
----------- ---------------------------- 0.6/2.0 MB 534.0 kB/s eta 0:00:03
------------ --------------------------- 0.6/2.0 MB 537.0 kB/s eta 0:00:03
------------ --------------------------- 0.6/2.0 MB 541.7 kB/s eta 0:00:03
------------- -------------------------- 0.7/2.0 MB 522.8 kB/s eta 0:00:03
------------- -------------------------- 0.7/2.0 MB 532.5 kB/s eta 0:00:03
-------------- ------------------------- 0.7/2.0 MB 530.7 kB/s eta 0:00:03
-------------- ------------------------- 0.7/2.0 MB 527.4 kB/s eta 0:00:03
-------------- ------------------------- 0.7/2.0 MB 523.0 kB/s eta 0:00:03
--------------- ------------------------ 0.8/2.0 MB 525.7 kB/s eta 0:00:03
--------------- ------------------------ 0.8/2.0 MB 530.1 kB/s eta 0:00:03
---------------- ----------------------- 0.8/2.0 MB 527.2 kB/s eta 0:00:03
---------------- ----------------------- 0.8/2.0 MB 528.4 kB/s eta 0:00:03
----------------- ---------------------- 0.9/2.0 MB 528.2 kB/s eta 0:00:03
----------------- ---------------------- 0.9/2.0 MB 529.3 kB/s eta 0:00:03
----------------- ---------------------- 0.9/2.0 MB 526.7 kB/s eta 0:00:03
------------------ --------------------- 0.9/2.0 MB 530.5 kB/s eta 0:00:03
------------------- -------------------- 1.0/2.0 MB 533.8 kB/s eta 0:00:02
------------------- -------------------- 1.0/2.0 MB 533.8 kB/s eta 0:00:02
------------------- -------------------- 1.0/2.0 MB 527.8 kB/s eta 0:00:02
-------------------- ------------------- 1.0/2.0 MB 531.0 kB/s eta 0:00:02
-------------------- ------------------- 1.0/2.0 MB 537.2 kB/s eta 0:00:02
--------------------- ------------------ 1.0/2.0 MB 529.6 kB/s eta 0:00:02
--------------------- ------------------ 1.1/2.0 MB 532.5 kB/s eta 0:00:02
---------------------- ----------------- 1.1/2.0 MB 538.7 kB/s eta 0:00:02
---------------------- ----------------- 1.1/2.0 MB 537.1 kB/s eta 0:00:02
----------------------- ---------------- 1.2/2.0 MB 539.7 kB/s eta 0:00:02
------------------------ --------------- 1.2/2.0 MB 546.9 kB/s eta 0:00:02
------------------------ --------------- 1.2/2.0 MB 547.8 kB/s eta 0:00:02
------------------------ --------------- 1.2/2.0 MB 541.6 kB/s eta 0:00:02
------------------------- -------------- 1.3/2.0 MB 547.7 kB/s eta 0:00:02
------------------------- -------------- 1.3/2.0 MB 546.3 kB/s eta 0:00:02
-------------------------- ------------- 1.3/2.0 MB 551.3 kB/s eta 0:00:02
--------------------------- ------------ 1.4/2.0 MB 558.1 kB/s eta 0:00:02
--------------------------- ------------ 1.4/2.0 MB 555.2 kB/s eta 0:00:02
---------------------------- ----------- 1.4/2.0 MB 561.9 kB/s eta 0:00:02
----------------------------- ---------- 1.4/2.0 MB 562.9 kB/s eta 0:00:01
----------------------------- ---------- 1.5/2.0 MB 568.0 kB/s eta 0:00:01
------------------------------ --------- 1.5/2.0 MB 569.5 kB/s eta 0:00:01
------------------------------ --------- 1.5/2.0 MB 574.5 kB/s eta 0:00:01
------------------------------- -------- 1.6/2.0 MB 572.7 kB/s eta 0:00:01
-------------------------------- ------- 1.6/2.0 MB 580.9 kB/s eta 0:00:01
-------------------------------- ------- 1.6/2.0 MB 582.1 kB/s eta 0:00:01
--------------------------------- ------ 1.7/2.0 MB 586.6 kB/s eta 0:00:01
---------------------------------- ----- 1.7/2.0 MB 587.7 kB/s eta 0:00:01
---------------------------------- ----- 1.7/2.0 MB 588.7 kB/s eta 0:00:01
----------------------------------- ---- 1.8/2.0 MB 589.5 kB/s eta 0:00:01
----------------------------------- ---- 1.8/2.0 MB 593.7 kB/s eta 0:00:01
------------------------------------ --- 1.8/2.0 MB 601.2 kB/s eta 0:00:01
------------------------------------- -- 1.9/2.0 MB 602.2 kB/s eta 0:00:01
-------------------------------------- - 1.9/2.0 MB 606.0 kB/s eta 0:00:01
-------------------------------------- - 1.9/2.0 MB 606.7 kB/s eta 0:00:01
--------------------------------------- 2.0/2.0 MB 613.4 kB/s eta 0:00:01
--------------------------------------- 2.0/2.0 MB 614.1 kB/s eta 0:00:01
---------------------------------------- 2.0/2.0 MB 606.1 kB/s eta 0:00:00
Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Installing collected packages: typing-extensions, pydantic-core, pydantic
Attempting uninstall: typing-extensions
Found existing installation: typing_extensions 4.11.0
Uninstalling typing_extensions-4.11.0:
Successfully uninstalled typing_extensions-4.11.0
Attempting uninstall: pydantic-core
Found existing installation: pydantic_core 2.14.6
Uninstalling pydantic_core-2.14.6:
Successfully uninstalled pydantic_core-2.14.6
Attempting uninstall: pydantic
Found existing installation: pydantic 2.5.3
Uninstalling pydantic-2.5.3:
Successfully uninstalled pydantic-2.5.3
Successfully installed pydantic-2.10.6 pydantic-core-2.27.2 typing-extensions-4.12.2

In [110… !pip install pydantic==1.8.2

Collecting pydantic==1.8.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
openai 1.64.0 requires pydantic<3,>=1.9.0, but you have pydantic 1.8.2 which is incompatible.
Downloading pydantic-1.8.2-py3-none-any.whl.metadata (103 kB)
---------------------------------------- 0.0/103.1 kB ? eta -:--:--
--- ------------------------------------ 10.2/103.1 kB ? eta -:--:--
--- ------------------------------------ 10.2/103.1 kB ? eta -:--:--
--- ------------------------------------ 10.2/103.1 kB ? eta -:--:--
----------- ------------------------- 30.7/103.1 kB 131.3 kB/s eta 0:00:01
----------- ------------------------- 30.7/103.1 kB 131.3 kB/s eta 0:00:01
-------------- ---------------------- 41.0/103.1 kB 140.9 kB/s eta 0:00:01
----------------------------- ------- 81.9/103.1 kB 241.3 kB/s eta 0:00:01
----------------------------------- 102.4/103.1 kB 295.4 kB/s eta 0:00:01
------------------------------------ 103.1/103.1 kB 270.8 kB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\khadersha\anaconda3\lib\site-packages (from pydantic==1.8.2) (4.12.2)
Downloading pydantic-1.8.2-py3-none-any.whl (126 kB)
---------------------------------------- 0.0/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
--------- ------------------------------ 30.7/126.0 kB ? eta -:--:--
------------------- ------------------- 61.4/126.0 kB 172.4 kB/s eta 0:00:01
------------------- ------------------- 61.4/126.0 kB 172.4 kB/s eta 0:00:01
------------------- ------------------- 61.4/126.0 kB 172.4 kB/s eta 0:00:01
------------------- ------------------- 61.4/126.0 kB 172.4 kB/s eta 0:00:01
------------------------------------- 122.9/126.0 kB 248.7 kB/s eta 0:00:01
-------------------------------------- 126.0/126.0 kB 231.6 kB/s eta 0:00:00
Installing collected packages: pydantic
Attempting uninstall: pydantic
Found existing installation: pydantic 2.10.6
Uninstalling pydantic-2.10.6:
Successfully uninstalled pydantic-2.10.6
Successfully installed pydantic-1.8.2

In [ ]: !pip uninstall spacy

!pip install spacy
In [ ]: python -m spacy download en_core_web_sm

In [ ]: