Chapter 2

The discipline matures: Three milestones

Materials taken from Cognitive Science, José Luis Bermúdez / Cambridge University Press 2019
 Overview
1. Language and micro-worlds
– Natural language processing & SHRDLU
(Winograd, 1972)
2. How do mental images represent?
– Mental rotation: Shepard & Metzler (1971)
3. An interdisciplinary model of vision
– Levels of explanation: Marr’s Vision (1982)

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 1. Language and micro-worlds
• Chomsky’s transformational linguistics tells us about what
we know when we understand a language, but does not tell
us how this knowledge is stored and used.

• One solution: Build a machine that is

capable of linguistic understanding

• Early attempts in AI was just to simulate conversations:

e.g., ELIZA
Eliza as a chatbot:
https://www.youtube.com/watch?v=RMK9AphfLco
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Natural language processing &
SHRDLU (Winograd, 1972)
- It is capable of using
language to report on its
environment, plan actions, and
reason about the implications
of what is being said to it.

- It is programmed to deal with

a very limited micro-world.

- Actions can be carried out

through a virtual robot arm (on
a computer screen).
 Why was SHRDLU important in the
development of cognitive science?
• It illustrates how abstract rules (e.g., grammar) can be practically
implemented.

• It illustrates the general approach by breaking down a cognitive

system into information processing components.

• It is based on the assumption that understanding language is an

algorithmic process.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 SHRDLU in action
• Three systems, operate concurrently, represented in the form
of procedure:
– Syntactic system - decodes
– Semantic system – assigns meanings
– Cognitive deductive system – integrates acquired
information with existing information

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
SHRDLU: Syntactic system

(Assume dealing with a very

simple language that only
contains nouns, intransitive
verbs, transitive verbs, and
determiners such as “the” or
“a”).
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
Procedures for testing noun
and verb phrases:

© Cambridge University Press 2014

 SHRDLU: Semantic system
• Information about the micro-world and
actions the system can perform.
– E.g., blocks and their properties, locations, relationships
with other blocks, etc.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 How does SHRDLU follow
instructions and answer questions?
1. Pick up a big, red block
OK
2. Grasp the pyramid
I DON’T UNDERSTAND WHICH
PYRAMID YOU MEAN.

PARSE an no
ADJECITVE
yes

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 How does SHRDLU follow
instructions and answer questions?

3. Find a block which is taller than the one

you are holding and put it in the box.

BY “IT” I ASSUME YOU MEAN THE

BLOCK WHICH IS TALLER THAN THE
ONE I AM HOLDING.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 How does SHRDLU follow
instructions and answer questions?
4. What does the box
contain?
THE BLUE PYRAMID AND THE
BLUE BLOCK

5. What is the pyramid

supported by?
THE BOX
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Limitation of SHRDLU
• It does not explain how conversations work in
real life (pragmatics of conversation)
– Pragmatics: the branch of linguistics dealing with
language in use and the contexts in which it is used
– Social interactions require both knowledge of the
world and of each other simultaneously.
• It only models a very basic form of linguistic
understanding.
– However, it demonstrates how a complex cognitive
process can be performed by the interaction of many
simpler processes and understood algorithmically.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Discussion question 1
• Compare ELISA and SHRDLU, and most
recently GPT - what are the differences
between them in terms of purpose,
functionality, and contribution to the cognitive
science literature? How do we modify the VP
parser to parse the sentence “put it in the
box”?
 2. How do images represent?
• Artificial intelligence is not the only way of developing and
testing hypotheses about cognitive science. E.g., cognitive
psychology.

• Imagery debate – one of the first occasions when cognitive

scientists got seriously to grips with the nature of mental
representation.

• Related concept: Artificial Cognition (Taylor & Taylor, 2021)

– “a branch of the Machine Behavior movement toward XAI, unique in
its emphasis on cognitive models that are inferred from data elicited
via experimentation rather than directly observed”
(https://link.springer.com/article/10.3758/s13423-020-01825-5).
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Mental rotation:
Shepard & Metzler (1971)
• Which pairs depicted the same
figure at different degrees of
rotation?

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Mental rotation:
Shepard & Metzler (1971)
• A linear (positive correlation) relationship between response time
and the degree of rotation between the two figures.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
Information processing in mental imagery
• Using digital computer as a model (digital information
storage)
– Binary digit (0 and 1) carries a single unit of information (a
bit)
– A byte is an 8-bit word – carries 256 units of information

• How can mental rotation images be

represented and processed digitally?
– E.g., write a function to map an input
image to a rotated image.
• Why does one transformation take longer
than the other?
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
Information processing in mental imagery
• Imagistic representation: representation
is secured through resemblance.

• Kosslyn’s experiment (1973):

Length of time to judge
whether a part belongs to an
visualized object depends on
the point of focus.

• Are there other possible explanations? There are competing

models of how information is stored and processed.
 Discussion question 2
• Use your own words/examples to describe the
differences between digital representations
and imagistic representations. Can you think
of a way of explaining the results of Kosslyn’s
experiments without the hypothesis of
imagistically encoded information? Try to
google the imagery debate.
 3. An interdisciplinary model of vision
• The mind can be studied at different levels – how do we integrate
across different levels of explanations?
• Levels of explanation: Marr’s vision (1982)
– Computational level
• Translate a general description of the cognitive system into
an information processing problem (input & output).
• Identify the constraints that hold upon any solution.
– Algorithmic level: How input is transformed to output
(algorithm – detailed information processing instructions)
– Implementational level: Physical realization for the algorithm.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
Levels of explanation: Marr’s
vision (top-down analysis)

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Applying top-down analysis to the
visual system
• Observations from research into brain-damaged patients carried
out by clinical neuropsychologists:
– Patients with right parietal lesions are able to recognize
familiar objects only from familiar perspectives.
– Patients with left parietal lesions have language problems but
no problem in matching shapes in either conventional or
unconventional conditions.
Parietal lobe

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Applying top-down analysis to the
visual system
• Shape of an object is processed separately from information about
what the object is for and what it is called.

• Visual system can deliver a specification of object shape even

when the object is not recognized.

• Marr’s analysis of the visual system

– Computational level: derive a representation of the 3-
dimensional shape and spatial arrangement of an object in a
form that can be recognized.
– Object-centered (not affected by view-point) vs. egocentric.
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Applying top-down analysis to the
visual system

• Marr’s analysis of the visual system

– Algorithmic level:
• What are the system’s representational primitives and the
operations performed on them?
• Psychophysics: the experimental study of perceptual
systems to understand the relations between physical
stimuli and mental phenomena.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Applying top-down analysis to the
visual system
• Primal sketch: basic elements such as information from light
intensity across the retinal image (edges, blobs, segments,
etc.).

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Applying top-down analysis to the
visual system

• 2.5D sketch: add information of

the depth and orientation of
visible surfaces from the viewer’s
perspective.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
Applying top-down analysis to the
visual system
• 3D sketch: viewer
independent
representation.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Discussion question 3
• Do later findings in cognitive science generally
support the claims made in Marr’s theory of
vision? Try to google structural description
model and image based model of visual object
recognition.
 Marr’s Analysis
(1982)
• Implementational level:
Neural mechanisms

Figure 2.15: The place of the

implementational level within Marr’s
overall theory.

Psychophysics – Studies how perceptual

systems react to different physical stimuli

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
A more recent version of
implementational level

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Discussion question 4
• If we do use imagistic representations in
mental rotation and visual imagery tasks, how
can these imagistic representations be
implemented in the brain? Check Figure 2.16
in the textbook and get more information
about the functions of the different regions in
the visual system.
 Applying top-down analysis to the
visual system
• This analysis illustrates:
– How a single cognitive phenomenon can be studied at
different levels of explanation
– How the different levels of explanation can come together
to provide a unified analysis.

Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010
 Summary

• Three milestones, which demonstrate how researchers

brought together some of the basic tools discussed in
Chapter 1 to understand specific cognitive capacities:
– SHRDLU (Winograd): modeling natural language
understanding.
– Mental rotation experiments (Shepard): explorations into
the representational formats.
– Multilevel analysis of the early visual system (Marr)

-the end-
Cognitive Science
Ó José Luis Bermúdez / Cambridge
University Press 2010