Understanding Algorithms, Time &

Space Complexity, and Big O

Notation
A foundational look into computer science
What is an Algorithm?
•Definition:
An algorithm is a step-by-step procedure or formula for solving a problem.
•Key Characteristics:
• Input
• Output
• Definiteness
• Finiteness
• Effectiveness

•Why it Matters in AI
• Algorithms define how models learn, optimize, and generalize
• Underpin efficiency, accuracy, and fairness

•Example:
Sorting a list, finding the shortest path, or performing calculations.

Real-Life Analogy
•Example: A recipe
• Ingredients = Input
• Instructions = Steps of the algorithm
• Final Dish = Output
Time and Space Complexity
•Time Complexity: How much time an algorithm takes based on the
input size.
•Space Complexity: How much memory an algorithm uses during
execution.
•Goal: Optimize for both, though sometimes there's a trade-off.

Why Complexity Matters

•Efficiency is critical:
• Speed (Time Complexity)
• Memory usage (Space Complexity)
•Good algorithms scale well with increasing data.

What is Big O Notation?

•A mathematical way to describe the upper bound of an algorithm’s
growth rate.
•Focuses on worst-case performance.
 Common Big O Notations

Big O Name Example

Accessing array
O(1) Constant time
element
O(log n) Logarithmic time Binary search
O(n) Linear time Loop through array
O(n log n) Linearithmic Merge sort, quicksort
Bubble sort, nested
O(n²) Quadratic time
loops
O(2ⁿ) Exponential time Recursive Fibonacci
Best vs Worst Case
•Best case: Minimum time (e.g., already sorted list)
•Worst case: Maximum time (e.g., reverse-sorted list)
•Average case: Expected time over all inputs

Practical Tips
•Always analyze the algorithm before implementation.
•Consider edge cases and scalability.
•Choose the right algorithm based on your problem and data size.

Conclusion
•Algorithms are the core of problem-solving in computer science.
•Time & Space Complexity help assess efficiency.
•Big O Notation is a key tool for comparing algorithms.
 Why Algorithms Are Critical for
Understanding the Performance of AI
Models and Training Pipelines
Unlocking the Logic Behind Intelligent Systems
The Foundation of AI Models
•Algorithms define:
• How data is processed
• How models learn patterns (e.g., backpropagation in neural
networks)
•Without algorithms, there is no structured way to train models

Understanding Model Performance

•Performance metrics (accuracy, F1 score, etc.) are shaped by:
• Algorithmic decisions (e.g., optimization method, regularization)
•Example:
• Gradient Descent vs. Adam → Different convergence behavior

Algorithm Choice Affects Results

•Different algorithms lead to different:
• Training speeds
• Generalization capabilities
• Resource efficiency
•Real-world impact:
• Trade-offs in precision vs. computational cost
Training Pipelines and Automation
•Pipelines automate:
• Data preprocessing, model training, evaluation
•Algorithms orchestrate:
• Feature selection
• Hyperparameter tuning (e.g., grid search, Bayesian optimization)

Debugging and Optimization

•Poor performance Algorithm design can be the root cause
• Learning rate too high?
• Poor regularization?
•Algorithmic analysis helps identify bottlenecks

Real-World Example
•Case Study: Image classification with CNNs
• Algorithm: Convolutional layers, ReLU, softmax
• Optimizer: SGD vs. Adam
• Impact: Faster convergence, better accuracy with Adam
Ethical and Responsible AI
•Algorithms impact:
• Fairness
• Bias
• Interpretability
•Transparent algorithmic choices are key for trust

Summary
•Algorithms are:
• Essential for model design, training, evaluation
• Crucial for debugging and improving pipelines
• Central to responsible and high-performance AI
Analyzing the Complexity of Simple
Search and Sort Algorithms
Understanding Time and Space Efficiency
What is Algorithm Complexity?
•Time Complexity: Measures how execution time grows with input size.
•Space Complexity: Measures how memory usage grows with input
size.
•Big-O Notation: Used to express upper bounds of an algorithm’s
growth.
Big-O Notation Examples

Big-O Name Example

O(1) Constant Time Accessing array index
O(log n) Logarithmic Binary Search
O(n) Linear Linear Search
O(n log n) Log-linear Merge Sort
O(n²) Quadratic Bubble Sort
Linear Search
•Description: Scans each element one by one.
•Time Complexity: O(n)
•Space Complexity: O(1)
•Best Case: O(1) (if element is first)
•Worst Case: O(n) (element not found)

Binary Search
•Description: Searches sorted array by repeatedly dividing in half.
•Time Complexity: O(log n)
•Space Complexity: O(1) for iterative, O(log n) for recursive
•Limitation: Works only on sorted arrays.

Bubble Sort
•Description: Repeatedly swaps adjacent elements if they are in the
wrong order.
•Time Complexity:
• Best Case (sorted): O(n)
• Average/Worst Case: O(n²)
•Space Complexity: O(1)
Selection Sort
•Description: Selects the smallest element and places it in order.
•Time Complexity: O(n²) in all cases
•Space Complexity: O(1)
•Stability: Not stable

Insertion Sort
•Description: Builds sorted array one item at a time.
•Time Complexity:
• Best Case (nearly sorted): O(n)
• Average/Worst: O(n²)
•Space Complexity: O(1)
•Stability: Stable

Key Takeaways
•Choose wisely: Simpler algorithms may be fine for small data
sets.
•Understand trade-offs: Time vs. space, best vs. worst case.
•Binary search is faster but requires sorted input.
•Sorting complexity greatly impacts large datasets.
 Comparison Table

Algorithm Best Average Worst Space Stable?

Linear
O(1) O(n) O(n) O(1) Yes
Search

Binary
O(1) O(log n) O(log n) O(1) N/A
Search
Bubble
O(n) O(n²) O(n²) O(1) Yes
Sort
Selection
O(n²) O(n²) O(n²) O(1) No
Sort

Insertion
O(n) O(n²) O(n²) O(1) Yes
Sort
 Understanding
Recursion
Basics, Divide-and-Conquer Approach, and Recursion vs Iteration
What is Recursion?
Definition:
Recursion is a programming technique where a function calls
itself to solve a problem.

Key Features:
•Problem broken into smaller subproblems
•Each recursive call has its own execution context
•Base case and recursive case required

Example Use Cases:

•Factorial calculation
•Fibonacci sequence
•Tree traversal
 Anatomy of a Recursive Function

def factorial(n):
if n == 0: # Base case
return 1
else:
return n * factorial(n - 1) # Recursive case

Components:
•Base Case: Condition to stop recursion
•Recursive Case: Function calls itself
Types of Recursion
•Direct Recursion – Function calls itself directly
•Indirect Recursion – Function A calls B, which calls A
•Tail Recursion – Recursive call is the last operation
•Non-Tail Recursion – Further computation after recursion call

The Divide-and-Conquer Strategy

Concept:
Divide the problem into smaller subproblems, solve them recursively, and combine
results.
Steps:
1.Divide: Break problem into subproblems
2.Conquer: Recursively solve each subproblem
3.Combine: Merge solutions to form the final result
Examples:
•Merge Sort
•Quick Sort
•Binary Search
•Matrix Multiplication (Strassen’s Algorithm)
 Example – Merge Sort

def merge_sort(arr):
if len(arr) > 1:
mid = len(arr) // 2
left = merge_sort(arr[:mid])
right = merge_sort(arr[mid:])
return merge(left, right)
return arr

Visual Flow:
Array → Divide → Sort Left & Right → Merge
 Recursion vs Iteration – Conceptual Comparison

Feature Recursion Iteration

Looping constructs
Structure Function calls itself
(for, while)
Termination Base case Loop condition
Uses call stack (more Constant space
Memory Usage
overhead) (usually)
More elegant and
Readability Verbose but efficient
concise
Slower, risk of stack
Performance Faster and safer
overflow
 Example – Factorial (Recursion vs Iteration)
Recursive:

def factorial(n):
if n == 0:
return 1
return n * factorial(n - 1)

Iterative:

def factorial_iter(n):
result = 1
for i in range(1, n + 1):
result *= i
return result
When to Use Recursion vs Iteration
Use Recursion When:
•Problem is naturally recursive (e.g., trees)
•Readability and maintainability matter
•Divide-and-conquer is suitable
Use Iteration When:
•Performance is critical
•Stack space is a concern
•Simple repetitive tasks

Summary
•Recursion simplifies problem-solving using self-calls
•Divide-and-conquer is a powerful recursive paradigm
•Iteration is generally faster and more memory efficient
•Choose based on problem structure and performance needs
 The Key to Understanding
Recursive Model Structures
Exploring Recursive Logic in Decision Trees
What Are Recursive Model Structures?
Definition:
Recursive models break down problems into smaller parts by
applying the same logic repeatedly at each level of a structure.
Common Use Cases:
•Decision Trees
•Recursive Neural Networks
•Hierarchical Models
Core Idea:
The same model structure is applied at each level of the hierarchy.

Real-World Analogy
Example:
"20 Questions" game — Ask yes/no questions recursively to
narrow down the answer.
Key Characteristics:
•Start from a general question
•Each answer leads to a more specific question
•Process repeats until a conclusion is reached
Decision Trees as Recursive Models
What is a Decision Tree?
A tree-like model used for classification and regression.
Recursive Nature:
•Each node represents a decision rule
•Each branch represents the outcome
•Each leaf node represents a final output
•Tree construction is recursive: split → apply rule →
repeat
Example of a Decision Tree
Sample Problem: Classify whether someone will play tennis

Outlook?
/ \
Sunny Overcast
/ \ \
Humidity? Play
/ \
High Normal
No Yes

Recursive Flow:
•Check "Outlook"
•If "Sunny", check "Humidity"
•Each decision rule leads to another rule or final decision
Building a Decision Tree – Recursive Algorithm
Common Algorithm: ID3, CART, C4.5
Steps (Recursively Applied):
1.Check for a stopping condition (e.g., all data has same label)
2.Select the best feature to split on (e.g., based on Information
Gain)
3.Split dataset based on the selected feature
4.Recursively apply the above steps to each subset

Pseudocode of Recursive Tree Building

def build_tree(data):
if stopping_condition(data):
return LeafNode()
best_feature = choose_best_feature(data)
branches = {}
for value in unique_values(data[best_feature]):
subset = filter_data(data, best_feature, value)
branches[value] = build_tree(subset)
return DecisionNode(feature=best_feature,
branches=branches)
Why Recursion Works Well for Trees
•Tree structures are inherently recursive
•Natural way to divide data at each node
•Each subtree is an independent problem
•Simplifies model building and prediction logic

Recursion in Prediction
Prediction Algorithm:
•Start at root node
•Test the input feature
•Follow the branch matching the input value
•Recurse down until reaching a leaf
Example:
def predict(node, input):
if node is LeafNode:
return node.label
value = input[node.feature]
return
predict(node.branches[value], input)

Benefits and Limitations

Benefits of Recursive Models:
•Clear and interpretable
•Handle hierarchical and conditional logic
•Easy to implement recursively

Limitations:
•Can overfit (if tree is too deep)
•Recursive depth can be computationally expensive
•Need pruning techniques
Implementing Fibonacci with
Recursion and Memoization
From Naive to Efficient Solutions
What is the Fibonacci Sequence?
Definition:
A series of numbers where each number is the sum of
the two preceding ones.
Formula:
F(n) = F(n - 1) + F(n - 2)
Base Cases:
F(0) = 0, F(1) = 1
Example:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …
Recursive Implementation
(Naive)
def fib(n):
if n <= 1:
return n
return fib(n - 1) + fib(n - 2)

Pros:
•Simple and elegant
•Directly follows the mathematical definition
Cons:
•Exponential time complexity: O(2^n)
•Many redundant calculations
 Problem with Naive Recursion

Example: fib(5)

fib(5)
/ \
fib(4) fib(3)
/ \ / \
fib(3) fib(2) fib(2) fib(1)
...

Redundant Calls:
•fib(3) is calculated twice
•fib(2) is calculated three times
Introducing Memoization
Definition:
An optimization technique to store results of expensive
function calls and reuse them.
Approach:
•Store intermediate results in a dictionary or array
•Check before computing

Fibonacci with Memoization (Top-

Down)
def fib_memo(n, memo={}):
if n in memo:
return memo[n]
if n <= 1:
return n
memo[n] = fib_memo(n - 1, memo) + fib_memo(n - 2,
memo)
return memo[n]
Efficiency:
•Time Complexity: O(n)
•Space Complexity: O(n)
Fibonacci with LRU Cache (Pythonic
Way)
from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
if n <= 1:
return n
return fib(n - 1) + fib(n - 2)

Advantages:
•Cleaner code
•Automatic caching
•Ideal for Python 3.2+
Bottom-Up Dynamic Programming
(Tabulation)
def fib_dp(n):
if n <= 1:
return n
dp = [0] * (n + 1)
dp[1] = 1
for i in range(2, n + 1):
dp[i] = dp[i - 1] + dp[i - 2]
return dp[n]

Pros:
•No recursion
•Time: O(n), Space: O(n)
Optimized Bottom-Up (Constant
Space)
def fib_optimized(n):
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return a

Space Complexity:
•Reduced to O(1)
•Ideal for large inputs
 Performance Comparison
Space
Method Time Complexity Notes
Complexity

Naive Recursion O(2^n) O(n) Very inefficient

Recursion + Great
O(n) O(n)
Memo improvement
Pythonic,
LRU Cache O(n) O(n)
concise
Bottom-Up DP O(n) O(n) No recursion
Optimized Most efficient for
O(n) O(1)
Iterative large n

When to Use What?

•Naive Recursion: For teaching or very small n
•Memoization / LRU Cache: When recursion is required with
performance
•Bottom-Up DP: For iterative environments (embedded systems, large
scale)
Sorting & Searching Algorithms
Quick Sort, Merge Sort, Binary Search, Linear Search
What are Sorting and Searching Algorithms?
Sorting algorithms are methods used to arrange data
in a specific order, usually in ascending or
descending order. Examples include:
•Bubble Sort
•Selection Sort
•Insertion Sort
•Merge Sort
•Quick Sort
•Heap Sort
Searching algorithms are techniques for finding a
specific item or value within a data structure. Examples
include:
•Linear Search
•Binary Search
•Hashing (e.g., Hash Tables)
Why Are They Important in Computer Science?
Sorting and searching algorithms are fundamental building blocks in
computer science for several reasons:
1. Efficiency in Data Management
•Well-sorted data allows for faster search operations.
•Efficient algorithms reduce computational time and resource usage,
crucial for large-scale applications.
2. Foundational Knowledge
•Understanding these algorithms helps in grasping core principles of
algorithm design and analysis (e.g., time and space complexity).
3. Real-world Applications
•Used in databases, search engines, data analysis, e-commerce (e.g.,
filtering or ranking), and more.
4. Basis for More Complex Algorithms
•Many advanced algorithms (e.g., in machine learning or graphics) use
sorting and searching as subroutines.
5. Algorithm Analysis Practice
•They provide a clear way to study performance trade-offs between
different approaches (e.g., comparison-based sorting vs. non-comparison-
based sorting).
What is Sorting?
Sorting is the process of arranging data in a particular
format or order—typically in ascending or descending
order. It is a fundamental operation in computer science
and is used to organize data for efficient processing,
searching, and presentation.
•Ascending Order: From smallest to largest (e.g., 1, 3,
7, 9)
•Descending Order: From largest to smallest (e.g., 9,
7, 3, 1)
Common Use Cases of Sorting
1.Data Organization
1. Helps in arranging records (e.g., names, numbers, dates) for better
readability and usability.
2. Example: Sorting contacts alphabetically in a phonebook app.
2.Search Optimization
1. Many search algorithms (e.g., binary search) require sorted data to
function efficiently.
2. Improves the speed and accuracy of data retrieval.
3.Data Analysis and Reporting
1. Useful in ranking, finding top-k elements (like highest sales), and
summarizing data.
2. Example: Sorting products by price to analyze sales performance.
4.User Interface Enhancements
1. Allows users to sort lists by criteria (e.g., sort emails by date or
sender).
2. Improves UX by providing flexible views.
5.Database Management
1. Essential for indexing and optimizing query performance.
2. Enables efficient JOIN operations and range queries.
6.Algorithms and Problem Solving
1. Sorting is often a preliminary step in complex algorithms like divide-
Quick Sort Concept
Quick Sort is a divide-and-conquer algorithm. The basic steps are:
1.Choose a pivot element.
2.Partition the array: elements less than the pivot go to the left, elements
greater go to the right.
3.Recursively apply the same process to the left and right subarrays.

Example: Sorting [7, 2, 1, 6, 8, 5, 3, 4]

Let's walk through how it would sort this list.
Step 1: Choose a Pivot
Let's say we pick the last element, 4, as the pivot.
Partition:
•Elements less than 4: [2, 1, 3]
•Pivot: 4
•Elements greater than 4: [7, 6, 8, 5]
Now recursively apply Quick Sort to [2, 1, 3] and [7, 6, 8, 5].
Continue this process until each subarray is sorted.
Python Code Example

def quick_sort(arr):
if len(arr) <= 1:
return arr # Base case
pivot = arr[-1]
left = [x for x in arr[:-1] if x <= pivot]
right = [x for x in arr[:-1] if x > pivot]
return quick_sort(left) + [pivot] + quick_sort(right)

# Test
arr = [7, 2, 1, 6, 8, 5, 3, 4]
sorted_arr = quick_sort(arr)
print("Sorted array:", sorted_arr)

Output:

Sorted array: [1, 2, 3, 4, 5, 6, 7, 8]

Merge Sort is a classic divide and conquer
algorithm used for sorting. It works by:
1.Dividing the input array into two halves.
2.Recursively sorting each half.
3.Merging the sorted halves back together.

Merge Sort Steps:

1.Divide: Split the array into two
halves.
2.Conquer: Sort each half recursively
using merge sort.
3.Combine: Merge the two sorted
halves into one sorted array.
Merge Sort Algorithm in Python
def merge_sort(arr):
if len(arr) > 1:
mid = len(arr) // 2 # Finding the middle of the array
left_half = arr[:mid] # Dividing the array elements into 2 halves
right_half = arr[mid:]

merge_sort(left_half) # Sorting the first half

merge_sort(right_half) # Sorting the second half

# Merging the sorted halves

i=j=k=0

# Copy data to temp arrays left_half[] and right_half[]

while i < len(left_half) and j < len(right_half):
if left_half[i] < right_half[j]:
arr[k] = left_half[i]
i += 1
else:
arr[k] = right_half[j]
j += 1
k += 1

# Checking for any remaining elements

while i < len(left_half):
arr[k] = left_half[i]
i += 1
k += 1

while j < len(right_half):

Time Complexity:
•Best Case: O(n log n)
•Average Case: O(n log n)
•Worst Case: O(n log n)
•Space Complexity: O(n) – due to the temporary
arrays used in merging.

Key Properties:
•Stable sort (doesn’t change the relative order of
equal elements)
•Not in-place (requires extra space)
•Good for sorting linked lists or large datasets that
don't fit in memory
 sort the array:[38, 27, 43, 3, 9, 82, 10]
Step 1: Divide Phase (Recursive
Splitting)
We keep dividing the array until we reach
subarrays of size 1.
[38, 27, 43, 3, 9, 82, 10]
↓
Left: [38, 27, 43] Right: [3, 9, 82, 10]
↓ ↓
[38] [27, 43] [3, 9] [82, 10]
↓ ↓ ↓ ↓
[27] [43] [3] [9] [82] [10]
Step 2: Merge Phase (Combine Back
Sorted)
Now, we start merging the sorted parts back
together.
Merge [27] and [43]:
[27] + [43] → [27, 43]
 Merge [38] and [27, 43]:
Compare 38 with 27 → 27 is smaller
Compare 38 with 43 → 38 is smaller
Only 43 remains

[38] + [27, 43] → [27, 38, 43]

Merge [3] and [9]:

[3] + [9] → [3, 9]

Merge [82] and [10]:

[82] + [10] → [10, 82]

Merge [3, 9] and [10, 82]:

Compare 3 with 10 → 3
Compare 9 with 10 → 9
Then 10 and 82 remain

[3, 9] + [10, 82] → [3, 9, 10, 82]

 Final Merge
Now merge the two large halves:
Left: [27, 38, 43]
Right: [3, 9, 10, 82]

Merge step-by-step:
1.Compare 27 with 3 → 3
2.Compare 27 with 9 → 9
3.Compare 27 with 10 → 10
4.Compare 27 with 82 → 27
5.Compare 38 with 82 → 38
6.Compare 43 with 82 → 43
7.Only 82 remains
[27, 38, 43] + [3, 9, 10, 82] → [3, 9, 10, 27, 38, 43,
82]
Final Sorted Array:

[3, 9, 10, 27, 38, 43, 82]

What is Searching?
Searching is the process of finding a specific
element or value in a collection of data (like an
array, list, or database).
For example, finding a phone number in your
contacts list or looking up a customer record in a
database are searching operations.

Why is Searching Important?

Searching is foundational in computer science and
real-world applications. It allows systems to retrieve
information quickly and accurately, which is
essential for performance, usability, and decision-
making.
 Applications of Searching

Area How Searching Is Used

Finding records by ID, name, or
Databases date. Efficient search is key to
performance.
Searching files, processes,
Operating Systems
memory locations.
Retrieving relevant websites from
Search Engines
billions of indexed pages.
Searching for products, filtering by
E-Commerce
attributes.
Games & AI Searching optimal moves or paths.
Scanning for vulnerabilities or
Cybersecurity
malicious patterns.
Types of Searching Algorithms
1. Linear Search (Sequential Search)
•Description: Check each element one by one.
•Time Complexity: O(n)
•When to Use: Small or unsorted data sets.
2. Binary Search
•Description: Repeatedly divide the sorted array in half.
•Time Complexity: O(log n)
•When to Use: Data must be sorted.
3. Hashing
•Description: Direct access using a hash function (e.g., dictionaries).
•Time Complexity: O(1) average case
•When to Use: Fast lookups with key-value pairs.
4. Search Trees (BST, AVL, B-Trees)
•Used in: Databases, memory-efficient searches.
•Time Complexity: O(log n) in balanced trees.
5. Graph Search Algorithms
•DFS (Depth-First Search)
•BFS (Breadth-First Search)
•Used in networking, AI, route finding.
Linear Search, also known as sequential search, is a simple
searching algorithm used to find the position of a target value within a
list. It works by checking each element in the list one by one until the
desired element is found or the list ends.

How Linear Search Works

1.Start from the first element of the array/list.
2.Compare the current element with the target value.
3.If they match, return the index of the element.
4.If they don't match, move to the next element.
5.Repeat steps 2–4 until the element is found or the end of the list is
reached.
6.If the target is not found, return -1 (or a similar "not found" indicator).
Example:
Let’s say you have the following list:
numbers = [5, 8, 12, 3, 9]
target = 3

Comparison
Element at
Step Index with Target Result
Index
(3)
Keep
1 0 5 5≠3
searching
Keep
2 1 8 8≠3
searching
Keep
3 2 12 12 ≠ 3
searching
Found at
4 3 3 ✅3=3
index 3

Final Result:
The algorithm stops at index 3 because that’s
where the target (3) is located.
If Target Not Found:
Suppose the target was 7 instead. Here's what would happen:

Element at Comparison
Step Index Result
Index with 7
Keep
1 0 5 5≠7
searching
Keep
2 1 8 8≠7
searching
Keep
3 2 12 12 ≠ 7
searching
Keep
4 3 3 3≠7
searching
Keep
5 4 9 9≠7
searching
❌ Return -1
6 - End of list Not found or "Not
Found"
Python Code: Linear Search
def linear_search(arr, target):
# Step 1: Loop through the array
for index in range(len(arr)):
# Step 2: Compare current element with target
if arr[index] == target:
# Step 3: If match is found, return the index
return index
# Step 4: If not found, return -1
return -1

# Example usage:
numbers = [5, 8, 12, 3, 9]
target = 3

# Call the function

result = linear_search(numbers, target)

# Output the result

if result != -1:
print(f"Element {target} found at index {result}.")
else:
print(f"Element {target} not found in the list.")
 Output:

Element 3 found at index 3.

with an element not in the list:

Change target = 3 to target = 7 and run it again:

target = 7

Output:

Element 7 not found in the list.

When to Use Linear Search

•When the list is unsorted.
•When the list is small and performance isn’t a major concern.
•When you want a simple and easy-to-implement solution.
What Is Binary Search?
Binary Search is an efficient search algorithm used to find the
position of a target element in a sorted array (ascending or
descending order). It works by repeatedly dividing the search interval
in half.
•Time Complexity: O(log n)
•Prerequisite: The array must be sorted
How Binary Search Works (Step-by-Step)
Let’s say you have a sorted array, and you want to find a particular value.
1.Start with two pointers: low at the beginning and high at the end of the array.
2.Find the middle element: mid = (low + high) // 2
3.Compare the mid value with the target:
•If it's a match, return the index.
•If the target is less, search the left half.
•If the target is more, search the right half.
4.Repeat steps 2–3 until the value is found or the interval is empty.
Example
Problem: Find 22 in the array
arr = [3, 8, 15, 22, 31, 45, 50]
Step-by-Step:
1.Initial Setup
low = 0, high = 6
mid = (0 + 6) // 2 = 3
arr[3] = 22 ✅ Match found!
✔️Found 22 at index 3.
Example: Not Found Case
Find 20 in the same array
arr = [3, 8, 15, 22, 31, 45, 50]
1.low = 0, high = 6 → mid = 3 → arr[3] = 22
20 < 22, so search left → high = 2
2.low = 0, high = 2 → mid = 1 → arr[1] = 8
20 > 8, so search right → low = 2
3.low = 2, high = 2 → mid = 2 → arr[2] = 15
20 > 15, so search right → low = 3
Now low > high, so the search ends.
❌ 20 is not in the array.
Iterative Binary Search (Non-Recursive)

def binary_search_iterative(arr, target):

low = 0
high = len(arr) - 1

while low <= high:

mid = (low + high) // 2

if arr[mid] == target:
return mid # Found the target
elif arr[mid] < target:
low = mid + 1 # Search right half
else:
high = mid - 1 # Search left half

return -1 # Target not found

 Recursive Binary Search
def binary_search_recursive(arr, target, low, high):
if low > high:
return -1 # Base case: target not found

mid = (low + high) // 2

if arr[mid] == target:
return mid
elif arr[mid] < target:
return binary_search_recursive(arr, target, mid + 1, high)
else:
return binary_search_recursive(arr, target, low, mid - 1)
Example Usage

arr = [3, 8, 15, 22, 31, 45, 50]

target = 22

# Iterative
index_iter = binary_search_iterative(arr, target)
print(f"Iterative: Found at index {index_iter}" if index_iter != -1 else "Not
found")

# Recursive
index_rec = binary_search_recursive(arr, target, 0, len(arr) - 1)
print(f"Recursive: Found at index {index_rec}" if index_rec != -1 else "Not
found")
Iterative Binary Search with Tracing
def binary_search_iterative(arr, target):
low = 0
high = len(arr) - 1

while low <= high:

mid = (low + high) // 2
print(f"Checking range [{low}, {high}], mid={mid}, arr[mid]={arr[mid]}")

if arr[mid] == target:
print(f"✅ Found {target} at index {mid}")
return mid
elif arr[mid] < target:
print(f"{target} > {arr[mid]} → Searching right half")
low = mid + 1
else:
print(f"{target} < {arr[mid]} → Searching left half")
high = mid - 1

print("❌ Target not found")

return -1
Recursive Binary Search with Tracing
def binary_search_recursive(arr, target, low, high):
print(f"Checking range [{low}, {high}]")

if low > high:

print("❌ Target not found in current range")
return -1

mid = (low + high) // 2

print(f"mid={mid}, arr[mid]={arr[mid]}")

if arr[mid] == target:
print(f"✅ Found {target} at index {mid}")
return mid
elif arr[mid] < target:
print(f"{target} > {arr[mid]} → Recursing into right half")
return binary_search_recursive(arr, target, mid + 1, high)
else:
print(f"{target} < {arr[mid]} → Recursing into left half")
return binary_search_recursive(arr, target, low, mid - 1)
Example Usage

arr = [3, 8, 15, 22, 31, 45, 50]

target = 22

print("=== Iterative Binary Search ===")

index_iter = binary_search_iterative(arr, target)
print(f"Result: Found at index {index_iter}" if index_iter != -1 else
"Result: Not found")

print("\n=== Recursive Binary Search ===")

index_rec = binary_search_recursive(arr, target, 0, len(arr) - 1)
print(f"Result: Found at index {index_rec}" if index_rec != -1 else "Result:
Not found")
Output (for target = 22)

=== Iterative Binary Search ===

Checking range [0, 6], mid=3, arr[mid]=22
✅ Found 22 at index 3
Result: Found at index 3

=== Recursive Binary Search ===

Checking range [0, 6]
mid=3, arr[mid]=22
✅ Found 22 at index 3
Result: Found at index 3

Efficient data manipulation and preprocessing are critical steps in

machine learning (ML) workflows because they directly impact model
performance, training time, and scalability. Here’s a breakdown of the
best practices, tools, and techniques for efficient data handling:
Data Loading & I/O Efficiency
•Use optimized file formats: Prefer formats like Parquet, Feather, or HDF5 over
CSV for large datasets due to better read/write speeds and compression.
•Chunking: For large files, load data in chunks using pandas.read_csv(...,
chunksize=...) to avoid memory overload.
•Parallel I/O: Libraries like Dask, Vaex, and Modin parallelize data reading and
operations.

Data Cleaning & Wrangling

•Missing values: Use vectorized operations (e.g., df.fillna(), df.dropna())
instead of loops.
•Categorical encoding:
•Low cardinality: Use One-Hot Encoding (pd.get_dummies()).
•High cardinality: Use Target Encoding, Frequency Encoding, or libraries
like CategoryEncoders.
•Datetime features: Extract useful components (df['date'].dt.month,
dt.weekday) for time series or seasonal data.
Scaling and Normalization
•Standardization: Use StandardScaler for Gaussian-like distributions.
•Min-Max Scaling: Useful when features are bounded.
•RobustScaler: Resistant to outliers.
•Use sklearn.preprocessing or sklearn.compose.ColumnTransformer for
scalable pipelines.

Feature Engineering
•Interaction terms: Multiply or combine features (f1 * f2) to capture interactions.
•Polynomial features: Use PolynomialFeatures for non-linear relationships.
•Dimensionality reduction:
•PCA: Principal Component Analysis.
•UMAP / t-SNE: For visualization or unsupervised learning.
omation with Pipelines
sklearn.pipeline.Pipeline or sklearn.compose.ColumnTransformer to:
Automate and reproduce preprocessing steps.
Avoid data leakage.
Streamline training and cross-validation.

Working with Large Datasets

•Dask/Modin: Pandas-like APIs for out-of-core and parallel processing.
•Polars: A blazing-fast DataFrame library optimized in Rust.
•Spark: Use PySpark for distributed processing in big data
environments.

Data Imbalance Handling

•Resampling: Use SMOTE, RandomOverSampler, RandomUnderSampler.
•Class weighting: Use model-specific options (e.g., class_weight='balanced' in
sklearn models).
•Stratified sampling: Maintain class proportions during train-test splits.
 Tools & Libraries Summary

Task Tools/Libraries
pandas, NumPy, Polars, Dask,
General Processing
Modin

sklearn, Feature-engineering
Feature Engineering
packages

Encoding sklearn, CategoryEncoders

Scaling sklearn.preprocessing

Pipelines sklearn.pipeline, Featuretools

Big Data PySpark, Dask, Vaex, Polars

compare the time performance of different sorting
techniques
Common Sorting Algorithms
Here are the typical algorithms used in comparisons:

Average
Algorithm Time Best Case Worst Case Stable?
Complexity
Bubble Sort O(n²) O(n) O(n²) Yes
Selection
O(n²) O(n²) O(n²) No
Sort
Insertion
O(n²) O(n) O(n²) Yes
Sort
Merge Sort O(n log n) O(n log n) O(n log n) Yes
Quick Sort O(n log n) O(n log n) O(n²) No
Heap Sort O(n log n) O(n log n) O(n log n) No
TimSort
O(n log n) O(n) O(n log n) Yes
(Python)
Dataset Considerations
Time performance depends heavily on:
•Size: e.g., 100, 1,000, 10,000, 1,000,000 elements
•Order: sorted, reverse, random, nearly sorted
•Type: integers, floats, strings, complex objects
import time
import random
import pandas as pd

# Sorting algorithm implementations

def bubble_sort(arr):
n = len(arr)
for i in range(n):
for j in range(0, n-i-1):
if arr[j] > arr[j+1]:
arr[j], arr[j+1] = arr[j+1], arr[j]
return arr

def insertion_sort(arr):
for i in range(1, len(arr)):
key = arr[i]
j = i-1
while j >= 0 and key < arr[j]:
arr[j + 1] = arr[j]
j -= 1
arr[j + 1] = key
return arr

def selection_sort(arr):
for i in range(len(arr)):
min_idx = i
for j in range(i+1, len(arr)):
if arr[j] < arr[min_idx]:
min_idx = j
arr[i], arr[min_idx] = arr[min_idx], arr[i]
return arr
def merge_sort(arr):
if len(arr) > 1:
mid = len(arr) // 2
L = arr[:mid]
R = arr[mid:]
merge_sort(L)
merge_sort(R)

i=j=k=0
while i < len(L) and j < len(R):
if L[i] < R[j]:
arr[k] = L[i]
i += 1
else:
arr[k] = R[j]
j += 1
k += 1
while i < len(L):
arr[k] = L[i]
i += 1
k += 1
while j < len(R):
arr[k] = R[j]
j += 1
k += 1
def quick_sort(arr):
if len(arr) <= 1:
return arr
pivot = arr[0]
less = [x for x in arr[1:] if x <= pivot]
greater = [x for x in arr[1:] if x > pivot]
return quick_sort(less) + [pivot] + quick_sort(greater)

# Timing wrapper
def time_sorting_algorithms(data):
results = {}
algorithms = {
'Bubble Sort': bubble_sort,
'Insertion Sort': insertion_sort,
'Selection Sort': selection_sort,
'Merge Sort': merge_sort,
'Quick Sort': quick_sort,
'Python TimSort': sorted
}
for name, func in algorithms.items():
data_copy = data.copy()
start = time.time()
func(data_copy)
end = time.time()
results[name] = end - start

return results

# Create dataset
dataset_size = 1000
random_data = [random.randint(0, 10000) for _ in range(dataset_size)]

# Run benchmarks
timing_results = time_sorting_algorithms(random_data)
timing_df = pd.DataFrame(list(timing_results.items()),
columns=["Algorithm", "Time (s)"])
timing_df.sort_values(by="Time (s)", inplace=True)
print(timing_df)
Hash Functions
A hash function takes an input (or "key") and returns a fixed-
size string of bytes. The output is typically a number, which is
used as an index in a hash table.
Properties of a good hash function:
•Deterministic: Same input always gives the same output.
•Fast: Should compute quickly.
•Uniform distribution: Spreads keys evenly across the hash
table.
•Minimizes collisions: Different keys should ideally have
different hash values.
Example:

hash("apple") → 99162322 (example value)

Hash Tables
A hash table is a data structure that maps keys to values
using a hash function. It allows for fast data retrieval.
Basic idea:
1.Use a hash function to compute an index from the key.
2.Store the key-value pair at that index.
Example (Python dictionary):

phone_book = {
"Alice": "123-4567",
"Bob": "987-6543"
}
Collision Handling
A collision occurs when two keys hash to the same
index.
Common collision resolution techniques:
1.Chaining
1.Each table index holds a list of entries.
2.Colliding items are stored in the same list.
table[3] = [("apple", 100), ("grape", 200)]

Open Addressing
•If a spot is taken, find the next open one using a probing
sequence.
•Types of probing:
• Linear probing: Try the next slot.
• Quadratic probing: Try i² slots away.
• Double hashing: Use a second hash function.
 Summary Table

Concept Description

Hash Function Maps a key to an index

Stores data using hash function-

Hash Table
generated keys

Collision Two keys hash to the same index

Use a list to store multiple items per

Chaining
index

Open Addressing Find another place in the array

Real-World Applications of Hash Functions and Hash
Tables
1. Dictionaries / Maps
•Use case: Fast lookup, insertion, and deletion.
•Real-world example: Phone books, caches, symbol
tables in compilers.
Code Example:
# Phone book using dictionary (hash table under the hood)
phone_book = {
"Alice": "123-4567",
"Bob": "987-6543"
}

print(phone_book["Alice"]) # Output: 123-4567

Implementing Sets
•Use case: Fast membership testing.
•Real-world example: Checking if a user has already logged
in, set operations in data processing.
Code Example:

logged_in_users = {"alice", "bob", "charlie"}

# Check if a user is logged in

if "bob" in logged_in_users:
print("Bob is logged in.") # Fast lookup due to hash
table
Caching / Memoization
•Use case: Store results of expensive function calls to avoid
recomputation.
•Real-world example: Web page caching, dynamic
programming optimizations.
Code Example:
# Memoization using a dictionary
cache = {}

def fibonacci(n):
if n in cache:
return cache[n]
if n <= 1:
return n
cache[n] = fibonacci(n-1) + fibonacci(n-2)
return cache[n]

print(fibonacci(10)) # Output: 55
Databases / Indexing
•Use case: Quickly locate records.
•Real-world example: Hash indexes in databases like
MongoDB, PostgreSQL.

Password Hashing
•Use case: Store passwords securely.
•Real-world example: User authentication systems.
Code Example:

import hashlib

# Store a hashed password

password = "my_secret_password"
hashed =
hashlib.sha256(password.encode()).hexdigest()
print("Stored hash:", hashed)

# Verify password
input_password = "my_secret_password"
if
hashlib.sha256(input_password.encode()).hexdiges
t() == hashed:
print("Password is correct.")
Data Deduplication
•Use case: Identify and remove duplicates quickly.
•Real-world example: File systems (like ZFS), backup
software.
Code Example:
file_hashes = set()

def is_duplicate(file_content):
file_hash =
hashlib.md5(file_content.encode()).hexdigest()
if file_hash in file_hashes:
return True
file_hashes.add(file_hash)
return False

print(is_duplicate("hello world")) # False

print(is_duplicate("hello world")) # True
. Efficient Word Embedding Lookup
In NLP, word embeddings (like Word2Vec, GloVe, FastText, or learned
embeddings in transformers) are stored as large matrices where:
•Each row corresponds to a word or token.
•Each column is a dimension of the embedding.
Lookup is usually an index operation:

embedding_vector =
embedding_matrix[word_index]

This is highly efficient because it's just a slice operation in NumPy or a

torch.nn.Embedding lookup in PyTorch.

Tip: Use torch.nn.Embedding or tf.nn.embedding_lookup—

they're optimized for speed and memory.
Embedding Caching Techniques
For large vocabularies or when embeddings are stored remotely or
sparsely accessed, caching becomes crucial.
a. In-Memory Cache
Use LRU (Least Recently Used) cache when memory is limited and
the vocabulary is large.
•Python Example:

from functools import lru_cache

@lru_cache(maxsize=10000)
def get_embedding(word):
return load_embedding_from_disk(word)
 b. Faiss for Fast Similarity Search
If you're doing similarity or nearest neighbor search (e.g., semantic
search):
•Use Faiss (Facebook AI Similarity Search).
•It indexes large embedding spaces efficiently.
c. Memory-Mapped Files
For large embeddings (e.g., GloVe), use memory-mapped files via
NumPy:
embeddings = np.memmap("glove.dat",
dtype='float32', mode='r', shape=(vocab_size,
emb_dim))

This avoids loading everything into RAM

Caching in Production Systems

For serving ML models:
•Use a Redis or Memcached backend to cache frequently queried
embeddings or model features.
•Embedding lookups can be batched and cached per user or session.
Batched Lookups
Always perform lookups in batches instead of loops for higher
throughput:

indices = [word_to_index[word] for word in words]

batch_embeddings = embedding_matrix[indices]

Summary Table

Method Use Case Pros Cons

torch.nn.Embeddi Training deep Fast, GPU-
Requires PyTorch
ng models optimized
Small dynamic
lru_cache Easy to use RAM-limited
cache
Large static Read-only, slower
NumPy memmap Low RAM use
embedding files access
Scalable, fast
Redis/Memcached Model serving Needs infra setup
access
Nearest neighbor Complex to set up
Faiss Very fast lookup
search initially
PyTorch: Basic Embedding Model
A simple PyTorch model that learns embeddings for items or
tokens.
import torch
import torch.nn as nn

class EmbeddingModel(nn.Module):
def __init__(self, num_items, embedding_dim):
super(EmbeddingModel, self).__init__()
self.embeddings = nn.Embedding(num_items,
embedding_dim)

def forward(self, item_ids):

return self.embeddings(item_ids)

# Example usage
model = EmbeddingModel(num_items=1000,
embedding_dim=64)
item_ids = torch.LongTensor([1, 5, 42])
embeddings = model(item_ids)
print(embeddings.shape) # (3, 64)
TensorFlow: Basic Embedding Layer
Embedding layer using TensorFlow / Keras.

import tensorflow as tf

embedding_layer =
tf.keras.layers.Embedding(input_dim=1000,
output_dim=64)

item_ids = tf.constant([1, 5, 42])

embeddings = embedding_layer(item_ids)
print(embeddings.shape) # (3, 64)
Production-Ready Embedding Service with Redis
Serve embeddings using a Redis-backed API (e.g., FastAPI +
Redis).
Dependencies:
pip install fastapi uvicorn redis numpy
Code:
# embedding_service.py
from fastapi import FastAPI, HTTPException
import redis
import numpy as np
import json

app = FastAPI()
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

@app.get("/embedding/{item_id}")
def get_embedding(item_id: str):
emb_json = r.get(item_id)
if emb_json is None:
raise HTTPException(status_code=404, detail="Embedding not found")
return {"item_id": item_id, "embedding": json.loads(emb_json)}

@app.post("/embedding/{item_id}")
def store_embedding(item_id: str, embedding: list):
r.set(item_id, json.dumps(embedding))
return {"status": "stored"}

Run it:
uvicorn embedding_service:app --reload
For a recommender system using PyTorch, you can implement a neural
collaborative filtering (NCF) model. The model can learn embeddings for
users and items and then predict the interaction score between them (which
could represent a user's preference for an item).
Additionally, we'll implement similarity search using the learned
embeddings (typically cosine similarity).

1. Recommender System with Neural Collaborative Filtering (NCF)

We'll create a model that learns user and item embeddings and uses a neural
network to predict interactions.
Step-by-Step Code:

2. Similarity Search using Learned Embeddings

After training, we can perform a similarity search by computing the cosine
similarity between the learned embeddings of items (or users). Here's how
you can implement that:
import torch.nn.functional as F

def cosine_similarity(embedding_a, embedding_b):

# Compute cosine similarity between two vectors
sim = F.cosine_similarity(embedding_a, embedding_b)
return sim

def get_most_similar_items(model, item_id, top_k=5):

# Get the embedding of the given item
item_embedding =
model.item_embedding(torch.LongTensor([item_id]))

# Get all item embeddings

all_item_embeddings = model.item_embedding.weight

# Compute cosine similarity between the given item and all items
similarities = F.cosine_similarity(item_embedding,
all_item_embeddings)

# Get the top_k most similar items

_, top_k_indices = torch.topk(similarities, top_k)

return top_k_indices

# Example: Get the top 5 most similar items to item_id=10

Explanation:
1.Dataset: We simulate random user-item pairs and ratings. In a
real-world case, this would be replaced by actual interaction data.
2.NCF Model: The model learns separate embeddings for users
and items. These embeddings are passed through a fully connected
network to predict the interaction score.
3.Training: We train the model using Mean Squared Error
(MSE) loss because the ratings are continuous (i.e., predicting
ratings).
4.Similarity Search: After training, we use cosine similarity to
find the most similar items to a given item based on their learned
embeddings.
Next Steps:
•Data Preparation: Instead of random ratings, use actual data
from your recommender system.
•Evaluation: Use metrics like Root Mean Squared Error
(RMSE) to evaluate the prediction accuracy.
•Optimizations: Experiment with different neural network
architectures and hyperparameters (e.g., embedding dimensions,
learning rate).
Trees and Graphs
Binary trees, BFS, DFS, graph representations
Binary Trees
A binary tree is a tree data structure where each
node has at most two children, referred to as the
left and right child.
Properties:
•Full binary tree: Every node has 0 or 2 children.
•Complete binary tree: All levels are filled except
possibly the last, which is filled from left to right.
•Balanced binary tree: Height difference between
left and right subtrees is minimal.
•Binary Search Tree (BST): Left child < parent <
right child.
DFS (Depth-First Search)
DFS explores as far as possible along one branch before
backtracking.
In Binary Trees:
DFS comes in 3 common variants (recursive or iterative using
a stack):
1.In-order: Left → Node → Right
2.Pre-order: Node → Left → Right
3.Post-order: Left → Right → Node
In Graphs:
DFS uses a stack (explicit or via recursion) and visits nodes
deeply before going wide.
BFS (Breadth-First Search)
BFS explores all nodes at the current depth before moving to
the next level.
In Binary Trees:
Uses a queue to process nodes level by level.
In Graphs:
Also uses a queue. Often used to find the shortest path in
unweighted graphs.
Graph Representations
There are multiple ways to represent a graph, each with trade-offs.
1. Adjacency Matrix
•2D array graph[V][V]
•Space: O(V²)
•Fast edge lookup: O(1)
•Good for dense graphs.
2. Adjacency List
•Array or dict of lists {node: [neighbors]}
•Space: O(V + E)
•Efficient for sparse graphs.
•Used in most practical applications.
 BFS vs DFS Summary

Feature BFS DFS

Data Structure Queue Stack (or recursion)
High (can store
Space Complexity many nodes at Lower in some cases
once)
Yes (in unweighted
Finds Shortest Path? No
graphs)
Shortest paths, Path existence,
Good For
levels topological sort
 1. Binary Tree Structure in Python

class TreeNode:
def __init__(self, val):
self.val = val
self.left = None
self.right = None

# Sample binary tree:

# 1
# /\
# 2 3
# /\
# 4 5

root = TreeNode(1)
root.left = TreeNode(2)
root.right = TreeNode(3)
root.left.left = TreeNode(4)
root.left.right = TreeNode(5)
 2. DFS Traversals on Binary Tree

# In-order: Left -> Node -> Right

def in_order(root):
if root:
in_order(root.left)
print(root.val, end=' ')
in_order(root.right)

# Pre-order: Node -> Left -> Right

def pre_order(root):
if root:
print(root.val, end=' ')
pre_order(root.left)
pre_order(root.right)

# Post-order: Left -> Right -> Node

def post_order(root):
if root:
post_order(root.left)
post_order(root.right)
print(root.val, end=' ')
 3. BFS (Level Order Traversal) on Binary Tree

from collections import deque

def bfs_tree(root):
if not root:
return
queue = deque([root])
while queue:
node = queue.popleft()
print(node.val, end=' ')
if node.left:
queue.append(node.left)
if node.right:
queue.append(node.right)
Graph Representations + BFS & DFS
a) Graph Using Adjacency List
graph = {
'A': ['B', 'C'],
'B': ['D', 'E'],
'C': ['F'],
'D': [],
'E': ['F'],
'F': []
}

b) DFS on a Graph (Recursive)

def dfs_graph(node, visited=set()):

if node not in visited:
print(node, end=' ')
visited.add(node)
for neighbor in graph[node]:
dfs_graph(neighbor, visited)
c) BFS on a Graph (Using Queue)

from collections import deque

def bfs_graph(start):
visited = set()
queue = deque([start])
while queue:
node = queue.popleft()
if node not in visited:
print(node, end=' ')
visited.add(node)
for neighbor in graph[node]:
queue.append(neighbor)
d) Graph Using Adjacency Matrix

# A - 0, B - 1, C - 2, D - 3
adj_matrix = [
[0, 1, 1, 0], # A
[0, 0, 0, 1], # B
[0, 0, 0, 1], # C
[0, 0, 0, 0], # D
]

Example Usage

print("Binary Tree In-order:")

in_order(root)
print("\n\nBinary Tree BFS:")
bfs_tree(root)

print("\n\nGraph DFS from 'A':")

dfs_graph('A')

print("\n\nGraph BFS from 'A':")

bfs_graph('A')
1. Neural Networks
🔹 Core Idea:
Neural networks are function approximators inspired by
biological neurons. They consist of layers of
interconnected nodes (neurons), which learn patterns
from data through backpropagation and gradient
descent.
🔹 Structure:
•Input layer: Takes in raw features.
•Hidden layers: Apply weights, biases, and activation
functions (e.g., ReLU, sigmoid).
•Output layer: Produces predictions (e.g., probabilities or
classifications).
Use Cases:
•Image recognition (CNNs)
•Text processing (RNNs, Transformers)
•Tabular data (MLPs)
•Games and control (RL)
Example (using PyTorch):

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(4, 10) # 4 input features, 10
hidden units
self.relu = nn.ReLU()
self.fc2 = nn.Linear(10, 3) # 3 output classes

def forward(self, x):

x = self.relu(self.fc1(x))
return self.fc2(x)

model = SimpleNN()
print(model)
2. Decision Trees
🔹 Core Idea:
Decision trees are supervised learning models that split
data into subsets based on feature values. They're
interpretable and good for classification/regression on
tabular data.
🔹 Structure:
•Nodes represent features/conditions.
•Edges are decision outcomes (e.g., True/False).
•Leaves are output labels or values.

Use Cases:
•Tabular classification problems
•Loan approval, fraud detection
•As base models for ensembles (Random
Forest, XGBoost)
Example (using scikit-learn):

from sklearn.tree import DecisionTreeClassifier

from sklearn.datasets import load_iris
from sklearn.tree import plot_tree
import matplotlib.pyplot as plt

X, y = load_iris(return_X_y=True)
clf = DecisionTreeClassifier(max_depth=3)
clf.fit(X, y)

plot_tree(clf, filled=True)
plt.show()
3. Knowledge Graphs
🔹 Core Idea:
A knowledge graph is a structured
representation of entities and relationships.
It's like a graph database (nodes and edges)
with semantics.
🔹 Structure:
•Nodes: Entities (e.g., "Einstein", "Physics")
•Edges: Relationships (e.g., "studied",
"invented")
•Often based on triples: (subject, predicate,
object)
Use Cases:
•Semantic search (e.g., Google Knowledge
Graph)
•Question answering systems
•Data integration in enterprise systems
Example (using RDFLib):

from rdflib import Graph, URIRef, Literal,

Namespace

g = Graph()
EX = Namespace("http://example.org/")

g.add((EX.Einstein, EX.studied, EX.Physics))

g.add((EX.Physics, EX.is_a, EX.Science))
g.add((EX.Einstein, EX.born_in,
Literal("Germany")))

for s, p, o in g:
print(s, p, o)
DFS Implementation for a Graph
We'll use an adjacency list to represent the graph and implement a
recursive DFS function.
🔹 Graph + DFS Code
# Define the graph using an adjacency list
graph = {
'A': ['B', 'C'],
'B': ['D', 'E'],
'C': ['F'],
'D': [],
'E': ['F'],
'F': []
}

# DFS implementation
def dfs(node, visited=None):
if visited is None:
visited = set()
if node not in visited:
print(node, end=' ')
visited.add(node)
for neighbor in graph[node]:
dfs(neighbor, visited)

# Run DFS from node 'A'

print("DFS Traversal from 'A':")
 Output:

DFS Traversal from 'A':

ABDEFC

(Note: Output order may vary depending on Python's dictionary ordering unless you use OrderedDict.)

Simple Decision Tree (using Scikit-learn)

We’ll use the classic Iris dataset, fit a decision
tree classifier, and visualize it.
🔹 Decision Tree Code
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

# Load sample dataset

iris = load_iris()
X, y = iris.data, iris.target

# Build a decision tree

clf = DecisionTreeClassifier(max_depth=3, random_state=42)
clf.fit(X, y)

# Visualize the tree

plt.figure(figsize=(12, 6))
plot_tree(clf, filled=True, feature_names=iris.feature_names,
class_names=iris.target_names)
plt.show()

What You’ll See:

A decision tree showing how features like petal
length and sepal width are used to classify iris
flowers (e.g., setosa, versicolor, virginica).
Implementing a Simple Decision Tree from Scratch
A basic decision tree works by splitting data based on the feature that
gives the best separation of classes. We'll focus on binary
classification for simplicity and use a basic greedy algorithm to
build the tree by repeatedly choosing the feature that gives the
highest information gain.

Building the Decision Tree from Scratch

1.Select the best feature to split the data (based on Gini impurity or entropy).
2.Recursively split the data into subsets.
3.Stop when we meet the stopping criterion (e.g., a max depth, minimum sample size,
or pure leaf).

Combining DFS with the Decision Tree

Now, let's incorporate DFS in traversing the decision tree. While
the decision tree is binary, we can treat it like a graph where each
node represents a decision. DFS can be used to explore the tree or
even search for a particular path.
DFS on the Decision Tree
DFS traversal function to print out the path to
each leaf:
# DFS function to traverse the decision tree
def dfs_tree(tree, path=[]):
if 'label' in tree: # Leaf node, print path and label
print(f"Path: {' -> '.join(map(str, path))}, Label: {tree['label']}")
return

# Traverse left and right subtrees

dfs_tree(tree['left'], path + [f"Feature {tree['feature_index']} <=
{tree['value']}"])
dfs_tree(tree['right'], path + [f"Feature {tree['feature_index']} >
{tree['value']}"])

# Run DFS traversal on the built decision tree

print("\nDFS Traversal of the Decision Tree:")
dfs_tree(tree)
Output:

DFS Traversal of the Decision Tree:

Path: Feature 0 <= 5, Label: 0
Path: Feature 0 > 5 -> Feature 1 <= 10, Label: 0
Path: Feature 0 > 5 -> Feature 1 > 10, Label: 1
How This Integrates:
•DFS is used to explore the tree from the root to each leaf,
effectively printing the conditions that lead to a particular
prediction.
•The decision tree is built with a greedy algorithm (using
Gini impurity to choose splits) and recursively splits the
dataset based on feature values.
This example shows both a handcrafted decision tree and
how we can apply DFS to explore its structure.
Greedy Strategy

When It Works
A greedy algorithm builds up a solution piece by piece, always choosing the
locally optimal option at each step, with the hope that this leads to a globally
optimal solution.
Key requirement: The problem must exhibit greedy choice property and
optimal substructure.

Examples Where It Works

1.Activity Selection Problem
2.Huffman Encoding
3.Minimum Spanning Tree (Prim’s, Kruskal’s)
4.Fractional Knapsack
5.Dijkstra’s Algorithm (with non-negative edge weights)

Why It Can Fail

•Greedy choices may ignore future consequences.
•It can’t "go back" to fix mistakes.
•Example: In 0/1 Knapsack, greedy fails because the locally best choice may
prevent a better overall solution.
Dynamic Programming

When It Works
Dynamic programming solves problems by breaking them into overlapping subproblems,
solving each just once, and storing the solutions.
Key requirement: The problem must have optimal substructure and overlapping
subproblems.

Examples Where It Works

1.0/1 Knapsack Problem
2.Fibonacci Sequence
3.Longest Common Subsequence (LCS)
4.Edit Distance
5.Matrix Chain Multiplication
6.Coin Change
How greedy algorithms apply in feature selection, clustering, and
resource allocation:

1. Feature Selection
Greedy algorithms are used to iteratively select features that contribute
the most to a model's predictive power.
•Forward Selection: Starts with no features and adds one feature at a
time that improves model performance the most.
•Backward Elimination: Starts with all features and removes the least
useful one at each step.
•Recursive Feature Elimination (RFE): A popular greedy approach
that fits a model and removes the weakest features recursively.
Why greedy works here: While not guaranteed to find the global
optimal subset, greedy methods offer a good trade-off between
performance and computational cost.
2. Clustering
Greedy methods can be used in clustering when building clusters
incrementally.
•Agglomerative Hierarchical Clustering: A bottom-up greedy
approach that merges the closest pair of clusters step-by-step.
•K-means++ Initialization: Greedily chooses initial cluster centers to
improve clustering performance over random initialization.
Why greedy works here: It builds solutions in a step-by-step fashion,
often leading to faster convergence and better initialization.
3. Resource Allocation
Greedy algorithms are frequently used in problems like:
•Knapsack Problem (Greedy version): Allocate resources to
maximize value without exceeding capacity, picking items with the best
value-to-weight ratio.
•Task Scheduling: Assign tasks to resources (e.g., CPUs) based on
shortest processing time or earliest deadline.
•Bandwidth Allocation: Assign bandwidth to users/applications
greedily based on priority or demand.
Why greedy works here: These problems often require fast
decisions, and greedy strategies can give near-optimal solutions
efficiently.
 Graph Algorithms for AI

Dijkstra's algorithm, A*, PageRank, shortest path problems

Key Concepts
•Input: A graph (usually represented by an adjacency list or matrix),
a starting node (source), and edge weights.
•Output: Shortest path distances from the source to all other
vertices.

How It Works (Step-by-Step)

1.Initialization:
•Set the distance to the source node as 0.
•Set the distance to all other nodes as ∞ (infinity).
•Use a priority queue (min-heap) to store and select the next node with the
smallest tentative distance.
2.Relaxation:
•Repeatedly select the unvisited node with the smallest tentative distance.
•For each of its neighbors, update the neighbor's distance only if the new path is
shorter.
3.Repeat:
•Continue until all nodes have been visited (or the queue is empty).
 Example (Graph)
Let’s say we have a graph:

A --(1)--> B
A --(4)--> C
B --(2)--> C
B --(5)--> D
C --(1)--> D

Starting from A, the shortest paths would be:

•A to A = 0
•A to B = 1
•A to C = 1 (A→B→C)
•A to D = 2 (A→B→C→D)
import heapq

def dijkstra(graph, start):

distances = {node: float('inf') for node in graph}
distances[start] = 0
priority_queue = [(0, start)]

while priority_queue:
curr_distance, curr_node = heapq.heappop(priority_queue)

if curr_distance > distances[curr_node]:

continue

for neighbor, weight in graph[curr_node]:

distance = curr_distance + weight
if distance < distances[neighbor]:
distances[neighbor] = distance
heapq.heappush(priority_queue, (distance, neighbor))

return distances
Graph Input Format:

graph = {
'A': [('B', 1), ('C', 4)],
'B': [('C', 2), ('D', 5)],
'C': [('D', 1)],
'D': []
}
Run:

print(dijkstra(graph, 'A'))

Key Notes
•Efficient with binary heap: O((V + E) log V)
•Works only for non-negative edge weights
•Great for GPS, network routing, and AI pathfinding
Problem: Predicting Customer Churn
We want to predict whether a customer will cancel their subscription
based on usage behavior and demographic data.

Step-by-Step Solution Using Combined Algorithmic

Concepts
1. Data Preprocessing (ETL and Feature Engineering)
Raw Data Might Include:
•Customer demographics (age, location)
•Usage statistics (login frequency, session duration)
•Customer service interactions
•Subscription type
Algorithmic Concepts Applied:
•Missing value imputation (mean, median, or model-based)
•Feature encoding:
• One-hot encoding for categorical variables (e.g., subscription type)
• Label encoding for ordinal features
•Scaling:
• StandardScaler for models sensitive to scale (e.g., SVM, logistic regression)
•Outlier detection:
• Use Isolation Forest or Z-score filtering
•Temporal features:
• Derive new features like “days since last login” from timestamps
2. Feature Selection

Algorithmic Concepts:
•Univariate feature selection (e.g., SelectKBest using mutual information)
•Recursive Feature Elimination (RFE)
•Tree-based feature importance (from Random Forest or XGBoost)

3. Model Building
Models to Consider:
•Logistic Regression (for interpretability)
•Random Forest (for non-linear interactions)
•Gradient Boosting (e.g., XGBoost or LightGBM for performance)
•Neural Networks (if you have high-dimensional data)
Algorithmic Concepts:
•Cross-validation (e.g., Stratified K-Fold)
•Hyperparameter tuning using:
• Grid Search
• Random Search
• Bayesian Optimization (e.g., with Optuna or Hyperopt)
4. Model Decision-Making (Explainability & Threshold Tuning)
Concepts Used:
•ROC-AUC curve to select the best classification threshold
•Precision-Recall trade-off analysis (important for imbalanced
datasets)
•SHAP (SHapley Additive exPlanations) or LIME for model
explainability
•Cost-sensitive learning (e.g., adjusting weights for false positives vs.
false negatives)
5. Post-Model Actions (Deployment & Feedback Loop)

Concepts:
•Model inference pipeline: Wrap preprocessing + model in a pipeline (e.g., with
sklearn.pipeline)
•Monitor model drift using statistical tests (e.g., Kolmogorov-Smirnov)
•Online learning or retraining schedule based on new data (e.g., weekly or
monthly retraining)
Combined Concepts in Action:
Example Pipeline:
python
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score

# Define preprocessing for numerical and categorical features

numeric_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler())
])

categorical_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="most_frequent")),
("onehot", OneHotEncoder(handle_unknown="ignore"))
])

# Combine preprocessing
preprocessor = ColumnTransformer(
transformers=[
("num", numeric_transformer, numeric_features),
("cat", categorical_transformer, categorical_features)
]
)

# Combine preprocessing and modeling in one pipeline

model = Pipeline(steps=[
("preprocessor", preprocessor),
("classifier", RandomForestClassifier())
])
 Summary of Combined Algorithmic Concepts

Concept Applied To
Data Cleaning & Encoding Preprocessing
Feature Selection Dimensionality Reduction
ML Modeling Random Forest, XGBoost
Hyperparameter Tuning Optimization
Threshold Adjustment Decision-making
Explainability (SHAP) Trust & Audit
Pipelines Deployment
 Building a simple spam filter using decision trees and hashing is a classic
example combining feature engineering and classification.

Goal:
•Use hashing trick to convert email text into fixed-length numerical
features.
•Train a decision tree classifier to distinguish spam from non-spam
emails.
Step-by-step:

1. Hashing Trick for Feature Extraction

Instead of using traditional vectorizers that build a vocabulary, hashing
converts words into a fixed number of buckets (features) by applying a
hash function. This is memory-efficient and good for large vocabularies.
2. Decision Tree Classifier
A simple, interpretable classifier that splits data based on feature values
to classify spam vs. ham.
 Code Implementation (Python, sklearn):
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Sample dataset (replace with your real emails)

emails = [
"Win money now!!!", # spam
"Hello friend, how are you?", # not spam
"Limited offer, buy today", # spam
"Meeting at 10am tomorrow", # not spam
"Congratulations, you've won a prize", # spam
"Lunch plans for today?" # not spam
]

labels = [1, 0, 1, 0, 1, 0] # 1 = spam, 0 = not spam

# 1. Convert text to hashed features

vectorizer = HashingVectorizer(n_features=20, alternate_sign=False)
# n_features = number of hash buckets; keep small for demo

X = vectorizer.transform(emails)

# 2. Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.3, random_state=42)

# 3. Train decision tree

clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)

# 4. Predict and evaluate

y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))
Explanation:
•HashingVectorizer:
•Converts text into fixed-size sparse feature vectors.
•n_features=20 means the text features are mapped into 20
buckets.
•alternate_sign=False to avoid signed hashes (simpler for decision
trees).
•DecisionTreeClassifier:
•Learns simple decision rules on the hashed features.
•Classification report:
•Shows precision, recall, and F1-score for spam/not spam classes.

Possible improvements:
•Increase n_features to get finer text representation.
•Use more training data.
•Use tree ensembles like Random Forests or Gradient Boosting for better accuracy.
•Add preprocessing like removing stop words or stemming before hashing.