CS 483/487

Week 03 – Week 04
Understanding the Problem (Part 2)

Hamza Ghandorh, Ph.D.

Winter Term 2022 / 1443H

CA14 Section

Courtesy by Alex Dan, Keith Edwards, Samer Al-Salah

The Computer
a computer system is made up of various elements

each of these elements affects the interaction

◦ input devices – text entry and pointing
◦ output devices – screen (small&large), digital paper
◦ virtual reality – special interaction and display devices
◦ physical interaction – e.g. sound, haptic, bio-sensing
◦ paper – as output (print) and input (scan)
◦ memory – RAM & permanent media, capacity & access
◦ processing – speed of processing, networks
 Interacting with computers

to understand human–computer interaction

… need to understand computers!

what goes in and out

devices, paper,
sensors, etc.

what can it do?

memory, processing,
networks
 A ‘typical’ computer system
? —
—
screen, or monitor, on which there are windows
keyboard
— mouse/trackpad window 1

— variations window 2

◦ desktop
◦ laptop
◦ PDA
12-37pm

the devices dictate the styles of interaction that the system supports
If we use different devices, then the interface will support a different style
of interaction
How many computers …
in your house? in your pockets?

◦ PC ◦ PDA
◦ TV,VCR, DVD, HiFi, ◦ phone, camera
cable/satellite TV ◦ smart card, card with
◦ microwave, cooker, washing magnetic strip?
machine ◦ electronic car key
◦ central heating ◦ USB memory
◦ security system

try your pockets and bags

can you think of more?
Interactivity?
Long ago in a galaxy far away … batch processing
◦ punched card stacks or large data files prepared
◦ long wait ….
◦ line printer output
… and if it is not right …

Now most computing is interactive

◦ rapid feedback
◦ the user in control (most of the time)
◦ doing rather than thinking …

Is faster always better?

Richer interaction

sensors
and devices
everywhere
Keyboards
— Most common text input device
— Allows rapid entry of text by experienced users

— Keypress closes connection, causing a character code to

be sent
— Usually connected by cable, but can be wireless
layout – QWERTY
— Standardised layout
but …
◦ non-alphanumeric keys are placed differently
◦ accented symbols needed for different scripts
◦ minor differences between UK and USA keyboards

— QWERTY arrangement not optimal for typing

– layout to prevent typewriters jamming!
— Alternative designs allow faster typing but large social base of
QWERTY typists produces reluctance to change.
QWERTY (ctd)

1 2 3 4 5 6 7 8 9 0
Q W E R T Y U I O P
A S D F G H J K L
Z X C V B N M , .
SPACE
alternative keyboard layouts
Alphabetic
◦ keys arranged in alphabetic order
◦ not faster for trained typists
◦ not faster for beginners either!

Dvorak
◦ common letters under dominant fingers
◦ biased towards right hand
◦ common combinations of letters alternate between hands
◦ 10-15% improvement in speed and reduction in fatigue
◦ But - large social base of QWERTY typists produce market pressures
not to change
special keyboards
— designs to reduce fatigue for RSI
— for one handed use
e.g. the Maltron left-handed keyboard
Chord keyboards
only a few keys - four or 5
letters typed as combination of keypresses
compact size
– ideal for portable applications
short learning time
– keypresses reflect letter shape
fast
– once you have trained

BUT - social resistance, plus fatigue after extended use

NEW – niche market for some wearables
phone pad and T9 entry
— use numeric keys with
multiple presses
2–abc 6-mno
3-def 7-pqrs
4-ghi 8-tuv
5-jkl 9-wxyz
hello = 4433555[pause]555666
surprisingly fast!

— T9 predictive entry
◦ type as if single key for each letter
◦ use dictionary to ‘guess’ the right word
◦ hello = 43556 …
◦ but 26 -> menu ‘am’ or ‘an’
Handwriting recognition
— Text can be input into the computer, using a pen and a
digesting tablet
◦ natural interaction

— Technical problems:
◦ capturing all useful information - stroke path, pressure, etc. in a
natural manner
◦ segmenting joined up writing into individual letters
◦ interpreting individual letters
◦ coping with different styles of handwriting

— Used in PDAs, and tablet computers …

… leave the keyboard on the desk!
Speech recognition
— Improving rapidly

— Most successful when:

◦ single user – initial training and learns peculiarities
◦ limited vocabulary systems

— Problems with
◦ external noise interfering
◦ imprecision of pronunciation
◦ large vocabularies
◦ different speakers
Numeric keypads
— for entering numbers quickly:
◦ calculator, PC keyboard
— for telephones
1 2 3 7 8 9
not the same!! 4 5 6 4 5 6

7 8 9 1 2 3
ATM like phone 0 # 0 . =
*
telephone calculator
the Mouse
— Handheld pointing device
◦ very common
◦ easy to use

— Two characteristics
◦ planar movement
◦ buttons
(usually from 1 to 3 buttons on top, used for
making a selection, indicating an option, or to
initiate drawing etc.)
the mouse (ctd)
Mouse located on desktop
◦ requires physical space
◦ no arm fatigue

Relative movement only is detectable.

Movement of mouse moves screen cursor
Screen cursor oriented in (x, y) plane,
mouse movement in (x, z) plane …

… an indirect manipulation device.

◦ device itself doesn’t obscure screen, is accurate and fast.
◦ hand-eye coordination problems for novice users
How does it work?
Two methods for detecting motion

— Mechanical
◦ Ball on underside of mouse turns as mouse is moved
◦ Rotates orthogonal potentiometers
◦ Can be used on almost any flat surface

— Optical
◦ light emitting diode on underside of mouse
◦ may use special grid-like pad or just on desk
◦ less susceptible to dust and dirt
◦ detects fluctuating alterations in reflected light intensity to calculate
relative motion in (x, z) plane
Even by foot …
— some experiments with the footmouse
◦ controlling mouse movement with feet …
◦ not very common :-)

— but foot controls are common elsewhere:

◦ car pedals
◦ sewing machine speed control
◦ organ and piano pedals
Touchpad
— small touch sensitive tablets
— ‘stroke’ to move mouse pointer
— used mainly in laptop computers

— good ‘acceleration’ settings important

◦ fast stroke
– lots of pixels per inch moved
– initial movement to the target
◦ slow stroke
– less pixels per inch
– for accurate positioning
Positioning in 3D space
— cockpit and virtual controls
◦ steering wheels, knobs and dials … just like real!
— the 3D mouse
◦ six-degrees of movement: x, y, z + roll, pitch, yaw
— data glove
◦ fibre optics used to detect finger position
— VR helmets
◦ detect head motion and possibly eye gaze
— whole body tracking
◦ accelerometers strapped to limbs or reflective dots and video
processing
 pitch, yaw and roll

yaw

roll
pitch
3D displays
— desktop VR
◦ ordinary screen, mouse or keyboard control
◦ perspective and motion give 3D effect
— seeing in 3D
◦ use stereoscopic vision
◦ VR helmets
◦ screen plus shuttered specs, etc.

also see extra slides on 3D vision

VR headsets
— small TV screen for each eye
— slightly different angles
— 3D effect
VR motion sickness
— time delay
◦ move head … lag … display moves
◦ conflict: head movement vs. eyes
— depth perception
◦ headset gives different stereo distance
◦ but all focused in same plane
◦ conflict: eye angle vs. focus
— conflicting cues => sickness
◦ helps motivate improvements in technology
3D Human Interface Guidelines
(ARKit, ARCore)
— Let people use the entire display.
— Use audio and haptics to enhance the immersive
experience.
— Minimize text in the environment.
— Consider using indirect controls when you need to
provide persistent controls.
— Be mindful of people's comfort and safety.
— Make a clear transition into AR
— Always consider your users’ spaces.
— Allow the user to reset the experience if they need
to.
— Use visual or audio cues to encourage offscreen
exploration.
The Interaction
— interaction models
◦ translations between user and system
— ergonomics
◦ physical characteristics of interaction
— interaction styles
◦ the nature of user/system dialog
— context
◦ social, organizational, motivational
What is interaction?
communication

user system

but is that all … ?

Some terms of interaction
domain – the area of work under study
e.g. graphic design
goal – what you want to achieve
e.g. create a solid red triangle
task – how you go about doing it
– ultimately in terms of operations or actions
e.g. … select fill tool, click over triangle

Note …
◦ traditional interaction …
◦ use of terms differs a lot especially task/goal !!!
Interaction Task Types
Donald Norman’s model
— Seven stages
◦ user establishes the goal
◦ formulates intention
◦ specifies actions at interface
◦ executes action
◦ perceives system state
◦ interprets system state
◦ evaluates system state with respect to goal

— Norman’s model concentrates on user’s view of the

interface
Execution/evaluation loop
goal
execution evaluation
system
– user establishes the goal
– formulates intention
– specifies actions at interface
– executes action
– perceives system state
– interprets system state
– evaluates system state with respect to goal
Ergonomics
— Study of the physical characteristics of interaction

— Also known as human factors – but this can also be

used to mean much of HCI!

— Ergonomics good at defining standards and guidelines

for constraining the way we design certain aspects of
systems
Ergonomics - examples
— arrangement of controls and displays
e.g. controls grouped according to function or frequency of use,
or sequentially
— surrounding environment
e.g. seating arrangements adaptable to cope with all sizes of
user
— health issues
e.g. physical position, environmental conditions (temperature,
humidity), lighting, noise,
— use of colour
e.g. use of red for warning, green for okay,
awareness of colour-blindness etc.
 Indirect manipulation

— office– direct manipulation

◦ user interacts
with artificial world
system

— industrial – indirect manipulation

◦ user interacts
with real world
through interface
interface plant
— issues ..
◦ feedback immediate
feedback
◦ delays
instruments
Common interaction styles
— command line interface
— menus
— natural language
— question/answer and query dialogue
— form-fills and spreadsheets
— WIMP (Windows, Icons, Menus, Pointers)
— point and click
— three–dimensional interfaces
Command line interface
— Way of expressing instructions to the computer directly
◦ function keys, single characters, short abbreviations, whole
words, or a combination

— suitable for repetitive tasks

— better for expert users than novices
— offers direct access to system functionality
— command names/abbreviations should be meaningful!

Typical example: the Unix system

Menus
— Set of options displayed on the screen
— Options visible
◦ less recall - easier to use
◦ rely on recognition so names should be meaningful
— Selection by:
◦ numbers, letters, arrow keys, mouse
◦ combination (e.g. mouse plus accelerators)
— Often options hierarchically grouped
◦ sensible grouping is needed
— Restricted form of full WIMP system
Natural language
— Familiar to user
— speech recognition or typed natural language
— Problems
◦ vague
◦ ambiguous
◦ hard to do well!
— Solutions
◦ try to understand a subset
◦ pick on key words
Query interfaces
— Question/answer interfaces
◦ user led through interaction via series of questions
◦ suitable for novice users but restricted functionality
◦ often used in information systems

— Query languages (e.g. SQL)

◦ used to retrieve information from database
◦ requires understanding of database structure and language
syntax, hence requires some expertise
Form-fills
— Primarily for data entry or data retrieval
— Screen like paper form.
— Data put in relevant place
— Requires
◦ good design
◦ obvious correction
facilities
Spreadsheets
— first spreadsheet VISICALC, followed by
Lotus 1-2-3
MS Excel most common today
— sophisticated variation of form-filling.
◦ grid of cells contain a value or a formula
◦ formula can involve values of other cells
e.g. sum of all cells in this column
◦ user can enter and alter data spreadsheet
maintains consistency
WIMP Interface
Windows
Icons
Menus
Pointers
… or windows, icons, mice, and pull-down menus!

— default style for majority of interactive computer

systems, especially PCs and desktop machines
Point and click interfaces
— used in ..
◦ multimedia
◦ web browsers
◦ hypertext

— just click something!

◦ icons, text links or location on map

— minimal typing
Three dimensional interfaces
— virtual reality
— ‘ordinary’ window systems
◦ highlighting
◦ visual affordance
flat buttons …
◦ indiscriminate use
just confusing!
— 3D workspaces click me!
◦ use for extra virtual space
… or sculptured
◦ light and occlusion give depth
◦ distance effects
Windows
— Areas of the screen that behave as if they were
independent
◦ can contain text or graphics
◦ can be moved or resized
◦ can overlap and obscure each other, or can be laid out next to
one another (tiled)

— scrollbars
◦ allow the user to move the contents of the window up and
down or from side to side
— title bars
◦ describe the name of the window
Icons
— small picture or image
— represents some object in the interface
◦ often a window or action
— windows can be closed down (iconised)
◦ small representation fi many accessible
windows
— icons can be many and various
◦ highly stylized
◦ realistic representations.
Pointers
— important component
◦ WIMP style relies on pointing and selecting things
— uses mouse, trackpad, joystick, trackball, cursor keys or
keyboard shortcuts
— wide variety of graphical images
 Menus

— Choice of operations or services offered on the screen

— Required option selected with pointer

problem – take a lot of screen space

solution – pop-up: menu appears when needed
Kinds of Menus
— Menu Bar at top of screen (normally), menu drags down
◦ pull-down menu - mouse hold and drag down menu
◦ drop-down menu - mouse click reveals menu
◦ fall-down menus - mouse just moves over bar!

— Contextual menu appears where you are

◦ pop-up menus - actions for selected object
◦ pie menus - arranged in a circle
– easier to select item (larger target area)
– quicker (same distance to any option)
… but not widely used!
Menus extras
— Cascading menus
◦ hierarchical menu structure
◦ menu selection opens new menu
◦ and so in ad infinitum

— Keyboard accelerators
◦ key combinations - same effect as menu item
◦ two kinds
– active when menu open – usually first letter
– active when menu closed – usually Ctrl + letter
usually different !!!
Menus design issues
— which kind to use
— what to include in menus at all
— words to use (action or description)
— how to group items
— choice of keyboard accelerators
Buttons
— individual and isolated regions within a
display that can be selected to invoke an
action

— Special kinds
◦ radio buttons
– set of mutually exclusive choices
◦ check boxes
– set of non-exclusive choices
Toolbars
— long lines of icons …
… but what do they do?

— fast access to common actions

— often customizable:
◦ choose which toolbars to see
◦ choose what options are on it
Palettes and tear-off menus
— Problem
menu not there when you want it

— Solution
palettes – little windows of actions
◦ shown/hidden via menu option
e.g. available shapes in drawing package
tear-off and pin-up menus
◦ menu ‘tears off’ to become palette
Dialogue boxes
— information windows that pop up to
inform of an important event or request
information.

e.g: when saving a file, a dialogue box is

displayed to allow the user to specify the
filename and location. Once the file is saved,
the box disappears.
Physical design
— many constraints:
◦ ergonomic – minimum button size
◦ physical – high-voltage switches are big
◦ legal and safety – high cooker controls
◦ context and environment – easy to clean
◦ aesthetic – must look good
◦ economic – … and not cost too much!
Design trade-offs
constraints are contradictory … need trade-offs

within categories:
e.g. safety – cooker controls
front panel – safer for adult
rear panel – safer for child

between categories
e.g. ergonomics vs. physical – MiniDisc remote
ergonomics – controls need to be bigger
physical – no room!
solution – multifunction controls & reduced functionality