CZ3005

Artificial Intelligence

Intelligent Agents and Search

A/P Hanwang Zhang

https://personal.ntu.edu.sg/hanwangzhang/
Email: hanwangzhang@ntu.edu.sg
Office: N4-02c-87
Lesson Outline

• How can one describe the task/problem for the agent?

• What are the properties of the task environment for the agent?
• Problem formulation
• Uninformed search strategies
• Informed search strategies: greedy search, A * search
Agent
An agent is an entity that
• Perceives through sensors (e.g. eyes, ears, cameras, infrared range
sensors)
• Acts through effectors (e.g. hands, legs, motors)
Sensors

Percepts

Environment ?

Agent
Actions

Effectors
Rational Agents
• A rational agent is one that does the right thing
• Rational action: action that maximises the expected value
of an objective performance measure given the percept
sequence to date
• Rationality depends on/subject to:
• performance measure
• everything that the agent has perceived so far
• built-in knowledge about the environment
• actions that can be performed
Example: Google X2: Driverless Taxi
• Percepts: video, speed, acceleration, engine status, GPS,
radar, …
• Actions: steer, accelerate, brake, horn, display, …
• Goals (Measure): safety, reach destination, maximise
profits, obey laws, passenger comfort,…
• Environment: Singapore urban
streets, highways, traffic,
pedestrians, weather, customers, …
Image source: https://en.wikipedia.org/wiki/Waymo#/media/File:Waymo_Chrysler_Pacifica_in_Los_Altos,_2017.jpg
More Examples
Agent Type Percepts Actions Goals Environment

Medical diagnosis Symptoms, findings, Questions, tests, Healthy patient, Patient, hospital
system patient’s answers treatments minimize costs

Satellite image Pixels of varying Print a Correct Images from orbiting

analysis system intensity, color categorization of categorization satellite
scene

Part-picking robot Pixels of varying Pick up parts and Place parts in Conveyor belt with
intensity sorts into bins correct bins parts

Refinery controller Temperature, Open, close valves; Maximize purity, Refinery

pressure readings adjust temperature yield, safety

Interactive English Typed words Print exercises, Maximize student’s Set of students
tutor suggestions, score on test
corrections
Types of Environment
Accessible (vs Agent's sensory apparatus gives it access to the
inaccessible) complete state of the environment
Deterministic (vs The next state of the environment is completely determined
nondeterministic) by the current state and the actions selected by the agent
Episodic (vs Each episode is not affected by the previous taken
Sequential) actions
Static (vs Environment does not change while an agent is
dynamic) deliberating
Discrete (vs A limited number of distinct percepts and actions
continuous)
Example: Driverless Taxi
Accessible? No. Some traffic information on road is missing
Deterministic? No. Some cars in front may turn right suddenly
Episodic? No. The current action is based on previous driving
actions
Static? No. When the taxi moves, Other cars are moving as
well
Discrete? No. Speed, Distance, Fuel consumption are in real
domains
Example: Chess
Accessible? Yes. All positions in chessboard can be observed
Deterministic? Yes. The outcome of each movement can be
determined
Episodic? No. The action depends on previous movements
Static? Yes. When there is no clock, when are you considering
the next step, the opponent can't move; Semi. When
there is a clock, and time is up, you will give up the
movement
Discrete? Yes. All positions and movements are in discrete
domains
More Examples
Environment Accessible Deterministic Episodic Static Discrete

Chess with a clock Yes Yes No Semi Yes

Chess without a clock Yes Yes No Yes Yes
Poker No No No Yes Yes
Backgammon Yes No No Yes Yes
Taxi driving No No No No No
Medical diagnosis system No No No No No
Image-analysis system Yes Yes Yes Semi No
Part-picking robot No No Yes No No
Refinery controller No No No No No
Interactive English tutor No No No No Yes
Design of Problem-Solving Agent

Idea
• Systematically considers the expected outcomes of
different possible sequences of actions that lead to
states of known value
• Choose the best one
• shortest journey from A to B?
• most cost effective journey from A to B?
Design of Problem-Solving Agent
Steps
1. Goal formulation
2. Problem formulation (states, actions, goal test)
3. Solution (a sequence of legal action execution)
4. Search (algorithm of how to find a good solution)
• No knowledge  uninformed search
• Knowledge  informed search
Example: Romania
 Goal: be in Bucharest
 Problem:
– states: various cities
– actions: drive between cities
– goal test: city == Bucharest?
 Solution:
– sequence of cities, e.g., Arad, Sibiu, Fagaras, Bucharest
Oradea Neamt

Zerind
lasi
Arad
Sibiu Fagaras
Vasliu

Timisoara Rimnicu Vilcea

Lugoj Pitesti Hirsova

Urziceni
Mehadia Bucharest
Eforie
Dobreta
Craiova Giurgiu
Example: Vacuum Cleaner Agent
o
Robotic vacuum cleaners
move autonomously
o
Some can come back to
a docking station to
charge their batteries
o
A few are able to empty
their dust containers into
the dock as well
Example: A Simple Vacuum World
 Two locations, each location may or may not contain dirt, and the agent may
be in one location or the other
1 2

3 4

5 6

7 8


States: 8 possible world states

Actions: left, right, and suck

Goal: clean up all dirt Two goal states, i.e. {7, 8}
Single-State Problem

– Accessible world state (sensory information is available)

– Known outcome of action (deterministic)
1 2

3 4

5 6

7 8

– e.g.: start in #5
– Solution: right, suck
Multiple-State Problem
– Inaccessible world state (with limited sensory
information):

agent only knows which sets of states it is in
– Known outcome of action (deterministic)
1 2

3 4

5 6

7 8

– e.g.: start in {1, 2, 3, 4, 5, 6, 7, 8}


Action right goes to {2, 4, 6, 8}

Solution: right, suck, left, suck
Well-Defined Formulation
Definition of a Re-state the goal with more specification (rules, resource, see below)
problem
Formulation • Initial state
• Action set, i.e. available actions (successor functions)
• State space, i.e. states reachable from the initial state
• Solution path: sequence of actions from one state to
another
• Goal test predicate
• Single state, enumerated list of states, abstract properties
• Cost function
• Path cost g(n), sum of all (action) step costs along the path
Solution A path (a sequence of operators leading) from the Initial-State to a
(Legal play) state that satisfies the Goal-Test
Measuring Problem-Solving Performance
Search Cost
• What does it cost to find the solution?
• e.g. How long (time)? How many resources used (memory)?

Total cost of problem-solving

• Search cost (“offline") + Execution cost (“online")
• Trade-offs often required
• Search a very long time for the optimal solution, or
• Search a shorter time for a “good enough" solution
 Single-State Problem Example
• Initial state: e.g., “at Arad“
Oradea Neamt
• Set of possible actions and the
Zerind
Arad
lasi corresponding next states
Sibiu
Fagaras Vasliu • e.g., Arad  Zerind
Rimnicu
Vilcea
• Goal test:
Urziceni
Timisoara
Lugoj Hirsova • explicit (e.g., x = “at Bucharest")
Pitesti
Mehadia Bucharest
Eforie
• Path cost function
Dobreta
• e.g., sum of distances, number
Craiova Giurgiu of operators executed solution:
a sequence of operators leading
from the initial state to a goal
state
Example: Vacuum World (Single-state
Version)
• Initial state: one of the eight states shown previously
• Actions: left, right, suck
• Goal test: no dirt in any square
• Path cost: 1 per action
Multiple-State Problem Formulation
 Initial state set
 Set of possible actions and the corresponding sets of next states
 Goal test
 Path cost function

• Solution:
 a path (connecting sets of states) that leads to a set of states all of which are
goal states
Example: Vacuum World (Multiple-state
Version)
 States: subset of the eight states
 Operators: left, right, suck
 Goal test: all states in state set have no dirt
 Path cost: 1 per operator
Example: Vacuum World (Multiple-state
Version)
Example: 8-puzzle
• States: integer locations of tiles Start state
• number of states = 9!
• Actions: move blank left, right, up, down
• Goal test: = goal state (given)
• Path cost: 1 per move

Goal state
Real-World Problems
Route finding problems:
• Routing in computer networks
• Robot navigation
• Automated travel advisory
• Airline travel planning

Touring problems:
• Traveling Salesperson problem
• “Shortest tour": visit every city exactly once
Question to Think About
AlphaGo
https://forms.gle/dmFLBxgx7kBktFPdA
Search Algorithms
• Exploration of state space by
generating successors of Arad
already-explored states Timisoara
Zerind
• Frontier: candidate nodes Sibiu
for expansion
Arad Rimnicu
• Explored set Vilcea
Fagaras
Oradea

Sibiu Bucharest
Search Strategies
• A strategy is defined by picking the order of node expansion.
• Strategies are evaluated along the following dimensions:

Completeness Does it always find a solution if one exists?

Time How long does it take to find a solution: the
Complexity number of nodes generated
Space Maximum number of nodes in memory
Complexity
Optimality Does it always find the best (least-cost) solution?
Uninformed vs Informed
Uninformed search strategies Informed search strategies
• Use only the information • Use problem-specific
available in the problem knowledge to guide the
definition search
1. Breadth-first search • Usually more efficient
2. Uniform-cost search
3. Depth-first search
4. Depth-limited search
5. Iterative deepening search
 Breadth-First Search
Expand shallowest unexpanded node which can be implemented by a
First-In-First-Out (FIFO) queue

1 A 2 A 3 A 4 A

B C B C B C

D E D E F G

Denote • b: maximum branching factor of the search tree

• d: depth of the solution
• Complete: Yes
• Optimal: Yes when all step costs equally
Complexity of BFS
• Hypothetical state-space, where every node can be expanded into b new
nodes, solution of path-length d
• Time: ++ ... + =
• Space: (keep every frontier node in memory)
Depth Nodes Time Memory
0 1 millisecond 100 bytes
2 0.1 seconds 1 kilobytes
4 11 seconds 11 kilobytes
6 18 minutes 111 megabyt
e
8 31 hours 11 gigabytes
10 128 days 1 terabyte
12 35 years 111 terabytes
14 3500 years 11111 terabytes
Uniform-Cost Search
To consider edge costs, expand
unexpanded node with the least path A
cost g
1 10
• Modification of breath-first search
5 B 5
• Instead of First-In-First-Out (FIFO) S G
queue, using a priority queue with 15 5
path cost g(n) to order the elements
C
• BFS = UCS with g(n) = Depth(n)
Uniform-Cost Search

A 1 2
S S
0
1 10
A B C
5 B 5 1 5 15
S G
S S
15 4 3
5
A B C A B C
C
15 5 15
G G G
11 10 11
Here we do not expand nodes that have been expanded.
Uniform-Cost Search
Complete Yes
Time # of nodes with path cost g <= cost of optimal solution
(eqv. # of nodes pop out from the priority queue)
Space # of nodes with path cost g <= cost of optimal solution
Optimal Yes
Depth-First Search
Expand deepest unexpanded node which can be implemented by a Last-In-
First-Out (LIFO) stack, Backtrack only when no more expansion
1 A 2 A 3 A 4 A 5 A 6 A

B C B C B C B C B C

D E D E D E E

7 A 8 A H I I

9 A 10 A 11 A 12 A
B C B C

E C C C C
E
F G F G F G
J K K
L M M
Depth-First Search
Denote
• d: maximum depth of the state space

Complete • infinite-depth spaces: No

No
• finite-depth spaces with loops: No
No
• with repeated-state checking: Yes
Ye
• finite-depth spaces
s without loops: Yes
Ye
s
Time
If solutions are deep, may be much faster than breadth-first
Space
No
Optimal No
 Depth-Limited Search
To avoid infinite searching, Depth-first search with a cutoff on the
max depth / of a path
Complete Yes, if L

Time
Space
Optimal No
Iterative Deepening Search
Iteratively estimate the max depth / of DLS one-by-one

Limit = 0

Limit = 1
Iterative Deepening Search
Iteratively estimate the max depth / of DLS one-by-one

Limit = 2
Iterative Deepening Search
Iteratively estimate the max depth / of DLS one-by-one

Limit = 3
Iterative Deepening Search...
Function ITERATIVE-DEEPENING-SEARCH(problem) returns a solution
sequence
inputs: problem, a problem

for depth to ∞ do
if DEPTH-LIMITED-SEARCH(problem, depth) succeeds then return its
result
end
Complete
return failureYes

Time
Space
Optimal Yes
 CZ3005
Artificial Intelligence

Intelligent Agents and Search

Assoc/P Hanwang Zhang

https://personal.ntu.edu.sg/hanwangzhang/
Email: hanwangzhang@ntu.edu.sg
Office: N4-02c-87
 Summary (we make assumptions for optimality)
Breadth- Uniform- Depth-First Depth- Iterative Bidirectional
Criterion first Cost Limited Deepening (if applicable)

Time
Space
Optimal Yes Yes No No Yes Yes
Complete Yes Yes No Yes, if Yes Yes
General Search
Uninformed search strategies Informed search strategies
• Systematic generation of new • Use problem-specific knowledge
states (Goal Test) • To decide the order of node
expansion
• Inefficient (exponential space
• Best First Search: expand the
and time complexity)
most desirable unexpanded node
• Use an evaluation function to
estimate the “desirability" of
each node
Evaluation function
• Path-cost function
• Cost from initial state to current state (search-node)
• Information on the cost toward the goal is not required.
• “Heuristic” function
• Estimated cost of the cheapest path from to a goal state:
• Exact cost cannot be determined
• depends only on the state at that node (may not include history)
• is not larger than the real cost (admissible)
Greedy Search
Expands the node that appears to be closest to goal
• Evaluation function :estimate of cost from to
• Function Greedy-Search(problem) returns solution
• Return Best-First-Search(problem, ) //

Question: How to estimation the cost from to goal?

Answer: Recall that we want to use problem-specific

knowledge
 Example: Route-finding from Arad to Bucharest
= straight-line distance from to Bucharest Straight-line distance to Bucharest
Arad 366
Oradea Neamt Bucharest 0
71
87 Craiova 160
75 Zerind 151 Dobreta 242
Lasi Efoire 161
Arad 140 Sibiu 92 Fagaras 176
99 Fagaras
Vaslui Giurgiu 77
118
80 Rimnicu Hirsova 151
142
Vilcea Lasi 226
Timisoara 211
Lugoj 244
Pitesti 98 Hirsova Mehadia 241
111 Lugoj 97 85
Neamt 234
70 146 Urziceni 86
101 Oradea 380
Mehadia 138 Bucharest Pitesti 98
75 90 Eforie Rimnicu Vilcea 193
Sibiu 253
Dobreta Craiova
120 Giurgiu Timisoara 329
Urziceni 80
• Useful but potentially fallible (heuristic) Vaslui 199
Zerind 374
• Heuristic functions are problem-specific
Example
a) The initial Straight-line distance to Bucharest

state Arad
Arad
Bucharest
366
0
Craiova 160
366 Dobreta 242
Efoire 161
Fagaras 176
Giurgiu 77
Hirsova 151
Lasi 226
Lugoj 244
Mehadia 241
Neamt 234
Oradea 380
Pitesti 98
Rimnicu Vilcea 193
Sibiu 253
Timisoara 329
Urziceni 80
Vaslui 199
Zerind 374
Example
b) After Straight-line distance to Bucharest

expanding Arad
Arad
Bucharest
366
0
Arad Craiova
Dobreta
160
242
Efoire 161
Sibiu Timisoara Zerind Fagaras 176
Giurgiu 77
253 329 374 Hirsova 151
Lasi 226
Lugoj 244
Mehadia 241
Neamt 234
Oradea 380
Pitesti 98
Rimnicu Vilcea 193
Sibiu 253
Timisoara 329
Urziceni 80
Vaslui 199
Zerind 374
Example
c) After Straight-line distance to Bucharest

expanding Arad Arad

Bucharest
366
0
Sibiu Craiova
Dobreta
160
242
Efoire 161
Sibiu Timisoara Zerind Fagaras 176
Giurgiu 77
329 374 Hirsova 151
Lasi 226
Arad Lugoj 244
Oradea Mehadia 241
366 Fagaras Rimnicu Neamt 234
380 Vilcea Oradea 380
176 193 Pitesti 98
Rimnicu Vilcea 193
Sibiu 253
Timisoara 329
Urziceni 80
Vaslui 199
Zerind 374
Example
d) After Straight-line distance to Bucharest

expanding Arad Arad

Bucharest
366
0
Fagaras Craiova
Dobreta
166
242
Efoire 161
Sibiu Timisoara Zerind Fagaras 176
Giurgiu 77
329 374 Hirsova 151
Lasi 226
Arad Lugoj 244
Oradea Mehadia 241
366 Fagaras Rimnicu Neamt 234
380 Vilcea Oradea 380
Pitesti 98
193
Rimnicu Vilcea 193
Sibiu 253
Sibiu Bucharest Timisoara 329
Urziceni 80
253 0 Vaslui 199
Zerind 374
Complete?
Question: Is this approach complete?
Example: Find a path from Lasi to Fagaras
Oradea
Neamt
Zerind
Lasi
Arad
Sibiu Fagaras
R Vaslui
Rimnicu
Timisoara Vilcea
Lugoj Pitesti Hirsova
Urziceni
Mehadia Bucharest

Dobreta Eforie
Craiova
Giurgiu

Answer: No
Greedy Search...
• m: maximum depth of the search space
Complete No
Time
Space
Optimal No
A * Search
• Uniform-cost search
• g(n): cost to reach n (Past Experience)
• optimal and complete, but can be very
inefficient
• Greedy search
• h(n): cost from n to goal (Future Prediction)
• neither optimal nor complete, but cuts search
space considerably
A * Search
Idea: Combine Greedy search with Uniform-Cost search
Evaluation function:

• f (n): estimated total cost of path through n to goal (Whole Life)

• If g = 0  greedy search; If h = 0  uniform-cost search
• Function A* Search(problem) returns solution
• Return Best-First-Searchproblem,
 Example: Route-finding from Arad to Bucharest
Best-first-search with evaluation function

(a) The initial state (b) After expanding Arad (c) After expanding Sibiu
Arad Arad Arad
366 = 0 + 366
Sibiu Zerind Sibiu Zerind
Timisoara
393 = 140 + 253 449 = 75 + 374 449 = 75 + 374
447 = 118 + 329
Timisoara Arad
Oradea
447 = 118 + 329 646 = 280 + 366 Rimnicu
671 = 291+ 380
Vilcea
Fagaras
413 = 220 + 193
415 = 239 + 176
Example: Route-finding from Arad to Bucharest
Best-first-search with evaluation function

(d) After expanding

Rimnicu Vilcea Arad

Zerind
Sibiu
Timisoara 449 = 75 + 374
Arad 447 = 118 + 329
646 = 280 + 366 Oradea Rimnicu
Fagaras 671 = 291+ 380 Vilcea

415 = 239 + 176 Sibiu

Craiova Pitesti 553 = 300 + 253
526 = 360 + 166
415 = 317 + 98
Example: Route-finding from Arad to Bucharest
Best-first-search with evaluation function
Arad
(e) After expanding
Fagaras Zerind
Sibiu
Timisoara 449 = 75 + 374
Arad 447 = 118 + 329
646 = 280 + 366 Oradea Rimnicu
Fagaras 671 = 291+ 380 Vilcea

Sibiu
Sibiu Craiova Pitesti 553 = 300 + 253
Bucharest
591 = 338 + 253 526 = 360 + 166
415 = 317 + 98
450 = 450 + 0
Example: Route-finding from Arad to Bucharest
Best-first-search with evaluation function
Arad

(f) After expanding Zerind

Sibiu
Pitesti Timisoara 449 = 75 + 374
Arad 447 = 118 + 329
646 = 280 + 366 Oradea Rimnicu
Fagaras 671 = 291+ 380 Vilcea

Sibiu
Sibiu Craiova Pitesti 553 = 300 + 253
Bucharest
591 = 338 + 253 526 = 360 + 166
450 = 450 + 0
Rimnicu
Bucharest Craiova Vilcea

418 = 418 + 0 615 = 455 + 160 607 = 414 + 193

Complexity of A*
Time
Space

With a good heuristic, significant

savings are still possible
compared to uninformed search
methods
Another Example: Alpha Go

#turns

#Moves = 250150

#stone positions
Another Example: Monte Carlo Search Tree (MCST)
MCST Key Steps
#Wins/#Games
MCST for Go is very expensive & slow

Why? The deeper you move, the wider you are


#number of simulation is exploded

Solution:
1. For selection/expansion, no longer stats
(xx/xx) but f = g+h
2. For simulation, no longer real self-play but f =
g+h
 MCST: Update f = g() + h()

Policy Network Value Network f f

𝑔() h ()
f f f f f f

f f f f f f f f

f f f f f f f f

No real play,
just an estimation