A Semiotic Approach To Investigating Presence

Shaleph O’Neill David Benyon

The HCI group The HCI group
Napier University Napier University
10 Colinton Road, Edinburgh 10 Colinton Road, Edinburgh
EH10 5DT EH10 5DT
s.o’neil@napier.ac.uk d.benyon@napier.ac.uk

ABSTRACT
Computers these days are highly complex devices that consist not 2. INTERACTIVE SYSTEMS AS MEDIUM
only of simple computational forms but also of complex cultural Malcolm McCullough in his engaging book Abstracting Craft
forms derived from other media. A good interaction designer [20] devotes a whole chapter to medium. He says a “medium may
understands this media and how combinations of components be a material such as plaster, or a means, an agency or an
result in engaging interaction. Presented here is our semiotic instrumentality. It may be an intervening person or thing or some
model of interaction that considers the computer as a medium. As other kind of carrier. It may be a pervasive environment”(p. 193).
part of the EU Presence initiative we are contributing to McCullough gives the example of wood as a medium. The
developing measures of presence that will provide designers with artist/designer works within the medium using his or her tools
a pattern language for designing presence. This new medium such as chisels. But he also acknowledges how the idea of a tool
needs new approaches to assist designers and the semiotics of can soon become a medium in its own right as the artist expresses
interactive systems is such an approach. ideas in wood through the chiselling. This leads to three
Keywords Semiotics, Medium, Embodiment, Interaction, important features of media; affordances, constraints and
engagement.
Presence
Wood affords chiselling. The medium has certain characteristics
1. INTRODUCTION that interact with the person through the tool/medium of the
Changes are afoot and concepts about new media are becoming chisel that provide possibilities for action. You can’t chisel metal.
increasingly important. These new media are interactive systems But just as the medium affords possibilities so it constrains
activities as well. The wood will break if it is chiselled too thin.
worn or embedded in environments, with physical tangible
The third feature of medium is ‘engagement’. A medium is
interfaces augmented by graphics or virtual environments
engaging if it draws the person in, if it seems to surround the
augmented by physical objects. New media demand new
approaches to interaction design and new foundations upon which activity, if it stimulates the imagination. McCullough argues that
to build our understanding. an engaging medium allows for continuity and variety, for ‘flow’
In this paper we explore some of these new foundations. Our own and movement between many subtle differentiations of
work is to look at semiotics as a new foundation for interactive conditions. “Thus the attuned craftsman asks ‘what can this
systems design. Of course semiotics is very old but if brought up medium do?’ as much as ‘what can I do with this medium’” (p.
198). A medium establishes a world of actions (p. 120).
to date it offers a level of discourse for discussing design issues
Digital Media or New Media, as it has become known [19], can
that seems appropriate. We do not want to discuss the details of
buttons and menus or the efficiency of tasks, as has been the be considered to be an extension of Marshall McLuhan’s ideas of
focus of traditional HCI. We want to phrase discussions in terms Mass Media [21]. His statement “The Medium is the Message”, is
of the interpretations, meanings and significances that people seen as an attempt to address the way in which new mediums
experience living in a world of interactive systems. affect the messages that we use to communicate.
The paper is organised as follows. Section 2 introduces the notion In terms of trying to understand the computer as a medium it can
of interactive systems as a medium and what it means to think easily be argued that these digital media are simply elements
constructed from the 1’s and 0’s of binary code. However,
about interactive technologies in this way. Section 3 introduces
semiotic analysis and the model of semiotics that we have computers these days are highly complex devices that consist not
developed to help understand interactive systems. Section 4 only of simple computational forms but also of complex cultural
describes some preliminary empirical work and in Section 5 these forms derived from the other media that they are now able to
and other features of our analysis are related to other concepts of support such as. video, sound, graphics, haptics and so on. [19].
embodiment. Section 6 introduces the concept of presence The parameters, and qualities of the computer as medium have
including work on the BENOGO project and Section 7 provides a increased dramatically as they have taken these new forms on
conclusion and some considerations about future work on board. Indeed the very nature of these older forms of media have
presence as non-mediated interaction. been subject to change by the qualities of the computer and must
now be understood within this new context [20].
First published at COSIGN-2003, The media that the interaction designer has to work with consists
09 – 12 September 2003, University of Teesside (UK), of all the different forms and functions of input and output and all
School of Computing and Mathematics, Virtual Environments the manipulations that can be performed on the content. The
Group interaction designer has software tools and hardware devices,
screen displays, sounds and haptics (touch) with which to create

81
an engaging, effective and efficient experience for the people from the work of Peter Bogh Andersen [2-4], SERG
involved in the interaction. A good interaction designer will [6,11,12,25]and the HCI group at Napier University[7-9,23]). It
understand this media and how combinations of components will consists of four main parts that are discussed below.
result in an engaging interaction. The good designer will
understand what the medium affords and the constraints that it
imposes.

3. SEMIOTICS AND INTERACTIVE

MEDIA
Research by the Semiotic Engineering Group (SERG) takes the
viewpoint that a user interface is a “one shot message”
[24,25]sent from a designer to a user, which can be seen as a
representation of the users needs as defined by the designer.
Furthermore, this message contains a number of smaller messages
that constitute the functional aspects of the interface, which are
delivered and articulated by the signs that the designer has chosen
to represent them. In Manovich’s terms the important thing about
the one shot message is that it contains other messages and is
therefore a metamessage. That is to say that the messages sent by
designers to users contain more messages that are to be used by
the users for their own ends. In the case of the computer as
medium it is no longer sufficient to say that the “medium is the Figure 2 A Semiotic Model of Interaction with a Digital
message” as McLuhan did. The “emergence of a new cultural
Medium
metalanguage” [19] has turned the messages into the medium.
In our paper “Semiotics and Interaction Analysis” [23] the focus 3.1 Sequential and Concurrent Syntagms
is on Umberto Eco’s “A Theory of Semiotics” [15]in relation to a Peter Bogh Andersen’s semiotic notions of concurrent and
semiotic analysis of mobile phone interfaces. Starting from the sequential syntagms [2] provide an insight into HCI by
SERG perspective Eco’s revised KF model (Figure 1) is applied abstracting a point of view from different media. Drawing on
as a tool by which to analyse interactions looking at the meanings semiotic concepts from Theatre and Dance, Andersen focuses on
associated with the signs within mobile phone interfaces. the notions of the sequence of events in relation to the actors and
props present on the stage. For Andersen computer based signs
exist as two-dimensional objects that occupy both sequential and
c1 ,c2 e tc [circ X ] - c 3 concurrent planes. During interaction computer based signs
(co ntA ) - d 3 .d 4 occupy a place in the interface, which is relative to other signs on
[circ X ] - c 4
the screen. As they are interacted with, they are brought into
(co ntB ) - d 5 ,d 6 - c5 ,c 6 e tc
relation sequentially to other signs in the interface that occur as a
result of system response.
/s- v / - sm .. .=< <se m e m e >> - d 1 ,d 2 Andersen proposes a model here that looks at the process of
(c o ntC ) - d 7 ,d 8 - c 7 ,c 8 e tc. interaction based on the notion that it takes place through the
[circ Y]
manipulation of the signs within an interface over a period of
time. What is unique in this description of interaction is that it can
(co ntD ) - d 9 ,d 1 0 - c9 ,c1 0 e tc . be viewed as a kind of discourse that takes place between the
computer and the user in terms of the meanings each one can
[c ircZ ] - d 1 1 .d 1 2 - c1 1 ,c1 2 e tc . attribute to the signs as they are activated during the interaction.

3.2 The Umwelt

F Jacob Von Uexkull’s conception of the Umwelt [13,16,26] is
igure 1 The Revised KF model (Eco, 1976) built upon the unique notion that all significations take place
within the bounds of firstly, our genetic codes in terms of
The revised KF model is built around the notion that meanings hereditary aspects of species, and secondly, the social codes
can be extrapolated from signs as either denotations (labelled ‘d’ within which we live as aspects of our environment into which
in Figure 1) or connotations (c) that are dependent on the context we become indoctrinated as we develop and grow as people.
(cont) and circumstances (circ) in which the signs are There can be no signification outside these constraints because
encountered. The revised KF model then is a dynamic tool that they are what give us a) the need to communicate and b) the
looks at the way the meanings of signs change depending on means by which to do it. The Umwelt then is effectively the mass
where they are encountered. Applying these ideas to mobile of knowledge that we carry around with us into every interaction,
phone interfaces uncovered how the meanings of individual signs which has been formed and continues to form as a result of those
are dependent on the context provided by the concurrent and interactions.
sequential framing of other signs in the interface [1].
In our paper “Semiotics and Interactive media” [22] we present 3.3 The Perception/Action Loop
our semiotic model of interaction that considers the computer as a Contained within Uexkull’s conception of the Umwelt is a model
medium from a semiotic point of view (Figure 2). It is derived of the relationship between organism and environment, which is a

82
perception/action model that is at the heart of every interaction. It and to determine, through the observation and interpretation [18]
has some similarities to ideas proposed in HCI by information of user interactions, how messages in an interface are defined;
processing psychology but is perhaps closer to a how the various forms of messages inform the user about the
phenomenological perspective that we will get to later. The system; and what users do with these messages.
fundamentally important thing about Uexkull’s perception/action The secondary, but equally important aim, was to develop an
loop is that Uexkull characterises its operation in terms of signs appropriate method for examining the model in the field. This
rather than in terms of processing raw sensory data. This is an consisted of taking the basis of a method already tested in
important shift in perspective. experiments with a semiotic perspective and adapting it to the
Linked to Andersen’s concurrent and sequential syntagms, field environment. The work of the Semiotic Engineering Group
Uexkull’s perception/action loop can really be seen as the Human (SERG) provided us with the most suitable basis for a semiotic
part of the interaction that makes sense of and manipulates the approach to fieldwork. Their concern with usability and system
‘Information artefacts’ [7] that exist in an interface. This activity, communicability [6,11,12,24] provides a well-established
which produces sequential chains throughout an interaction, framework for conducting observational experiments that support
occurs between the two aspects of the sign i.e. the signifier and a semiotic viewpoint. SERG have concentrated their efforts
the signified, or here, the system and the user’s Umwelt. largely on ‘one to one’ interface interactions, which focus on the
communicability of the interface in order to establish usability
3.4 Information Artefacts problems. Our work is somewhat different in focus as we are
The traditional signs or information artefacts that make up an exploring aspects of our model and notions of mediation but we
interface are the buttons, graphics and words that Andersen used the same talk-aloud principles employing a video camera
categorised in his book “Computer Semiotics” [2]. Since then rather than screen based capture equipment.
however many new forms have come to be included in an
interface to the point where we now have a ‘new media interface’
[19]. So the information artefacts in our model are considered to
be all of the elements that now go into an interface which
constitute the beginnings of this new metalanguage [19].

3.5 Medium
A sequence of actions, which are traditionally viewed as system
state changes in HCI, can also be viewed as changes within a
medium [20]. More specifically they are transformations within
the medium that occur through the processes of the
perception/action loop. The idea of medium proposed here in
relation to our model, places the properties of the system in the
hands of the messages or signs that communicate the system
state. That is, the messages are the medium which are
manipulated by both designers and users in a similar fashion in
order to produce the object of their interaction, be it a piece of
artwork, a selection of tunes on a media player or a new piece of
software for somebody else to use.
McCullough’s notion of craft [20] is applied to interaction here,
framing the computer as a medium in which the user becomes Figure 3 Information Artefacts
expert in handling its specific properties. Much like the sculptor
With this type of approach we were able to analyse the video
who is expert in understanding the medium of wood or stone or
footage from a semiotic perspective looking at the signs
clay, the digital art worker is an expert in handling and
manipulated in the environment in relation to the tasks that
manipulating the signs that construct the medium of that program.
Conceptually s/he is aware of the systems properties and knows participants performed and in relation to what they said about
how to manipulate them through the signs to get the desired what they were doing. The types of data we got then were screen
result. images showing the concurrent and sequential nature of sign use
(Figure 3) and transcriptions of utterances by the participants
4. THE MODEL AND MEDIATION (Figure 4) that correspond to the time coded screen shots from the
video.
4.1 Preliminary Studies
Turning to the practical aspects of our research [22], we have
been using a qualitative methodological approach in order to Time Actions Participant
explore the particular aspects of our proposed model, and the
more general notions of messages as medium that underpin it. [20:48:46] Making a sign. “He’s still there but
Three preliminary studies were undertaken which focused on the Selects the ‘path he’s not got that
notion of medium in different environments. The first two studies again. Chooses natural light shadow
looked at subjects involved in using computer interfaces the dodge burn that’s coming round
(Photoshop, and others) to design both print and electronic media. tool. Opens the here, so I’m gonna
The third study was to provide a comparison to the software- brush palette. recreate that and just
controlled interfaces by looking at the ‘real interface’ of an artist Selects shape take him right
working on a painting. The main aim was to explore the model and size. Back back…make my

83
 up to tool bar to selection, shadow 6, 5. Embedded technologies participate in the world they
set exposure. take a hundred, he’s represent
quite big, exposure… 6. Embodied interaction turns action into meaning.
start at 10 and work The reason for exploring Dourish’s notion of embodied
back from there” interaction here is to note the similarities between his analysis
and our own; in particular the concept of computation, or
Figure 4 Transcriptions interactive systems in our terminology, as a medium. Where a
semiotic analysis can go further, we think, is in the idea of
What was interesting about the studies was that they not only meaning. Semiotics recognises that meaning itself is a complex
confirmed the usefulness of concurrent/sequential paradigms in web of significances. It is not simply the things that some
analysing data, it also linked them to notions about the Umwelt information artefact denotes that is important, it is all the
where sense making is a direct result of the perception/action connotations that flow from the denotations, turning infinitely
loop as the user engages with the world [22]. This gives weight to back on themselves, that characterises our understandings and
the idea that even expert users perform work tasks in an feelings.
exploratory way. Moreover, different ‘zones’ or ‘modes’ of The zones of medium uncovered from the analysis of expert users
activity were uncovered in these studies which users switched of Photoshop correspond to the ideas of ‘interacting with’ and
between throughout their interaction. Each one seemed to affect, ‘interacting through’ that Dourish describes and, indeed, are
if not at least overlap with the other. characteristic of Winograd and Flores’s earlier introduction of
phenomenology to HCI [27]. Our analysis suggests a third zone –
5. EMBODIED INTERACTION the larger medium in which the interaction that is the focus of
In Where the Action Is Paul Dourish develops his ideas on the attention takes place.
foundations of embodied interaction [14]. The embodied
perspective considers interaction ‘with the things themselves’. 6. VIRTUAL ENVIRONMENTS
Dourish draws on the phenomenological philosophy of such 6.1 Presence
writers as Heidegger, Husserl and Merleau-Ponty and recent In research into tele-presence (or ‘presence’, as it is usually
developments in tangible computing and social computing to abbreviated) there is a topic known as ‘the book problem’. This
develop a theory of embodied interaction. For Dourish, characterises the problem that we can feel really immersed and
phenomenology is about the tight coupling of action and involved when reading a book. The medium though which we are
meaning. interacting is apparently very impoverished compared, say, to
Embodied interaction is concerned with two main features; cinema or virtual reality and yet the feeling of presence that we
meaning and coupling. Within meaning Dourish finds three types: can experience can be quite considerable. You can really get
ontology, intersubjectivity and intentionality. Ontology is transported to another place. The mistake, of course, is to think
concerned with how we describe the world, with the entities and that the book is the medium. It is the words and skills of the
relationships that we perceive – or rather with which we interact. storyteller that is the medium through which we interact with the
Dourish is concerned with how we understand the computational significances that the story has for us. In the cinema the medium
world. Intersubjectivity is about how meaning can be shared with is very rich and much more realistic – at least in terms of visual
others. This involves both the communication of meaning from fidelity. Even when the objects on the screen are impossible
designer to user, so that the system can reveal its purpose, and the spacecraft, the hi-fidelity representation is characteristic of the
communication between users through the system. The task- medium. Of course you can ‘drift off’ at the cinema and lose the
artefact cycle is a familiar concept to HCI people; designers sense of presence (or feel presence in the new found reverie), just
design some technology to support some task, but then the as you can when reading a book.
technology inevitably changes the nature of the task. Dourish is Designing for presence is about designing the illusion of non-
concerned with the ways in which we use technologies in our mediation. When you put on a head mounted display you are
activities and how these affect the decisions we take, expectations immediately transported into the computed world beyond the
we have and so on. The third aspect of meaning is intentionality. headset. You are not aware that there are two tiny displays sitting
This is to do with the directedness of meaning and how it relates close to your eyes; that part of the interaction is apparently
one thing to another. unmediated. For remote tele-operation of vehicles and tools a
Thus actions take on meaning for people. Coupling is concerned feeling of non-mediation, or embodied interaction, would be an
with making that relationship effective. If objects and advantage. The person controlling the Mars Lander wants,
relationships are coupled then the effect of actions can be passed ideally, to feel that he or she is really picking up the rock to
through the system. Dourish uses the familiar example of a examine it. The headset, the gloves, the transmitters, the robot
hammer (also used by Heidegger) to illustrate coupling. When arms all need to disappear into a single medium so that the
you use a hammer is becomes an extension to your arm (it is controller feels that the interaction is unmediated, that it is
coupled) and you act through the hammer onto the nail. You are embodied.
engaged in the activity of hammering.
From this theory of embodied interaction – ‘not just how we act
on technology, but how we act through it’ [14] – Dourish goes on
to develop some high-level design principles:
1. Computation is a medium
2. Meaning arises on multiple levels
3. Users, not designers, create and communicate meaning
4. Users, not designers, manage coupling

84
6.2 The BENOGO Project restrictive nature of the first Demo where no movement or
interaction was actually possible within the virtual environment.
We could not effectively observe interactions with the virtual
environment; therefore this first study was focused on a
description of the significant aspects of the environment
highlighting technical problems with the VE but limited in
exploring the concept of presence.
The videos were first of all transferred from tape to hard disk for
storage and easy access. They were then viewed a number of
times to promote immersion in the data and a transcript of
participants comments was rendered along with notes on aspects
of visual, audio and timing of events. Analysis of the talk aloud
method resulted in a number of factors that consistently recurred
in all of the sessions. Broadly speaking these were grouped into
the three categories that the talk aloud questions were designed to
enquire about.

1. Description: The descriptive level of the environment:

Figure 5 BENOGO Demo 1 Equipment recognisable objects and features of the environment,
trees, plants, bridge etc (6.4).
The BENOGO project is a European funded project that
2. Significance: The personal subjective engagement with
concentrates on this notion of presence. We are contributing to
this project by including a semiotic analysis of participant’s the environment. Feelings of calmness, pleasantness,
interactions within these environments. Based on the model lack of atmosphere and humidity, memories of holidays
presented here our aim is to look at the types of meanings that etc (6.5).
3. Realness: The technical limitations of the environment:
people generate as they interact with these virtual environments
cables, HMD, resolution etc (6.6).
(Figures 5 and 6) and to compare them with the types of
meanings generated in similar real environments.
Additionally there were two other main areas of interest that
The BENOGO project is unique in that it uses real time Image
arose from the talk aloud sessions that were not considered before
Based Rendering (IBR) technology to create the visual
component of the virtual environment. At this stage in the the tests, Movement (6.7) and Sound (6.8).
development of the technology The Region of Exploration (REX) 6.4 Descriptions, things and objects
has been restricted to one point of view and no movement within
In the BENOGO environment, the types of elements that people
the environment is possible. However the singular point of view
could see were obviously identifiable despite the resolution
does provide full 360-degree head movement and stereo depth of problems that arose due to the technical limitations. Interestingly
field.
enough participants often identified these technical problems as
things that they could see in this descriptive section as if they
were objects in the environment (e.g. “I can see stereo”). Every
single participant commented on a computer generated sculpture
that had been added to the world and how odd it looked.
In the real botanical garden in Edinburgh similar types of
description occurred where participants identified particular
objects such as trees, plants, water, the building etc. As well as
these a number of other things were highlighted in the
descriptions of the real world. Fish, birds, signposts, heat,
humidity and people were all existent in the real world but not
present in the BENOGO environment. The only time any of these
things was mentioned in the BENOGO environment was to point
out their absence. (Note: participants are labelled r-real, b-
BENOGO)

“I see a garden, with a bridge and an object, looks like coming

from a leaf, staying in the middle, then I see the sun on the leaves.
Figure 6 The BENOGO Botanical Environment I hear some water. I see the roof.” Participant b2
6.3 Talk-aloud analysis “There is no moisture in the air, in my breathing or sensing on my
The talk-aloud approach we used was part of a raft of techniques skin. That’s one of the things I’m missing” Participant b10
employed during data gathering in demo 1. 10 participants took
part in this section of the BENOGO test while only half that 6.5 Significance and Memory
amount took part in the benchmarking activities, which are an on In terms of personal responses to the environments it was in the
going process. The talk-aloud method we used for this study was real world that much more reference to significance and memory
to a certain extent treated more like a live interview than the ones occurred. Participants were often reminded of other places that
we had done previously [22]. This was largely due to the

85
they had been. Other botanical gardens, gardens in general or that the sound was not necessarily connected to the visuals even
places with hot climates. Cultural references to films such as the although there was something directional about it. Comments
Jungle Book, Tarzan and general jungle films were mentioned often arose about cars outside, birds and the noise of water in the
and personal memories of holidays, family members and in the environment. These were sometimes accompanied by comments
case of two Greek participants, home were also mentioned. In the about the water not moving visually while it sounded like it was
BENOGO environment very little of this type of data was or no movement in the trees where birds might be. In the real
uncovered. There were some mentions of memories of other environment sound comments were restricted to comments on the
botanical gardens and holidays but very little that was as vivid as water, the humidifier being turned off and on, and the sense of
those in the real environment. In the BENOGO environment there quiet in the space.
were a few mentions of games and gaming related comments that
were not present in the real environment. “Sound, sound is very spatial it’s location based.” Participant b8

“It reminds me of Kew Gardens I went there when I was younger, “I can hear this bird’s cry somewhere in the soundscape. So I, for
the other thing is the heat and the condensation it reminds me of a a while, actually try to locate the bird. It seems to be impossible
shower. Its very relaxing and quiet” Participant r1 for me.” Participant b10

“It reminds me of a place, a museum in Copenhagen which has a 6.9 Discussion

kind of indoor garden like this. It’s not the same actually but it In terms of the model presented earlier this BENOGO study
sounds very much the same… it reminds me of being on a points towards some interesting aspects of a semiotics of Virtual
holiday in a different place. Actually it doesn’t remind me of a Environments. Although unlike our other studies [22], due to
rain forest although it could be but there’s too much light in restricted interaction, it is still possible to see relationships with
here.” Participant b10 aspects of the model in our data. For example in the descriptive
level we get a sense of artefacts in the environment trees, plants,
6.6 Realness bridge etc. These are examples of simple denotation active in the
In the BENOGO environment comments about realness were representational images of the environment. Furthermore there
almost always couched in relative terms. Most people understood are examples of a more connotative semiotics present in the
or pinpointed resolution problems that made the visuals seem environments. These are captured through focusing on elements
unreal. At the same time most of the participants said that it of significance and memory. In short this is the territory of the
‘looked pretty real’ particularly in relation to other types of VR. Umwelt. Here an environment such as a botanical garden can
In the real environment the same thing happened but this time in trigger a sense-making semiosis that allows cultural references to
reverse. Everybody understood that they were in a real jungle movies or memories of holidays to take place. A link then
environment and that they could see real things but the man-made is established between encountering the phenomena of the
construction of the physical environment brought out comments environment and the signification process within the Umwelt.
such as ‘fake’ or ‘unnatural’ that seemed to impinge upon Although we are still early on in the BENOGO project, we
participants sense of realness. consider that this approach has been useful in exploring the
differences between real and virtual places. What it has
“It is an artificially created real place. Everything around me is highlighted, in a similar way to the Photoshop studies, is the
real I can touch it. It is tangible.” Participant r1 relative difference in the richness between real and virtual
environments. Denotative aspects come through strongly in both
“I think the way I see through the glass in here or whatever, is a but connotative aspects are much stronger in the real world. This
bit blurry especially when I move quickly, but I think that it looks may have something to do with embodiment in terms of extra
like a place that is here and I am looking through something.” channels of sensory input that only occur through actually being
Participant b14 in a place. It may also have something to do with the technical
limitations of the technology that interferes with this feeling
6.7 Movement presence e.g. low resolution heavy HMD, etc. As the BENOGO
Participants in the real world had much more freedom to move project develops these are areas that we are keen to explore in
around the environment. In the BENOGO environment attempts terms of the concept of mediation and the semiotic model
to move and mentions of wanting to move were quite common presented.
across most participants but these were physically restricted by
the cables of the HMD, and technically by the 1 point of view 7. CONCLUSION
REX (Region of Exploration) of the environment. In this paper we have presented our approach to understanding
interactive systems as new media. The reason why books such as
“I get the feeling of being attracted to walking over the bridge or those by Dourish and McCullogh and Manovich are appearing is
trying to step down on some other place maybe walk round, to because human-computer interaction is changing in the light of
explore it even more. This possibility of being able to move new media. Dourish refers to tangible computing and to social
around this place would enhance the feeling of being there.” computing and the changes that these are bringing to the ways we
Participant b10 think about HCI. We are currently working with photo-realistic
virtual reality in order to investigate the notion of presence [5].
6.8 Sound How can we design systems so that people feel they are
Sound also featured quite highly in both environments. In the somewhere else? As part of BENOGO we are contributing to
BENOGO environment many participants commented on the developing measures of presence and aim to provide designers
sound and its suitability to the visuals. However many realised with a pattern language for designing for presence. The work

86
presented here contributes to this notion as we seek ways of Representations Of Real Places., In Proceedings of
measuring the ‘amount’ of mediation in an interaction on various PRESENCE 2002:, The 5th Annual International Workshop
dimensions such as fidelity, interest level, concentration level and on Presence, Universidade Fernando Pessoa Porto, Portugal -
so on. This is a radical departure from previous approaches to October 9,10,11, 2002.
presence that have concentrated on physiological measures. [6] Barbosa, S., et al, Direct and Indirect user-to-developer
We are also keen to explore the new information spaces that are messages through communicability evaluation.
being created through pervasive, distributed computing Representational Support for User Developer
environments. Here Benyon has already characterised a new HCI, Communication workshop, INTERACT’99, 1999.
concerned with the navigation of information spaces [8,10], from [7] Benyon, D., Beyond the Metaphor of Navigation in
a semiotic perspective. Looking to lessons from architectural Information Space, chi2000 (2000).
semiotics, interior and garden design we are looking to the design [8] Benyon, D., The new HCI? Navigation of information space,
of physical environments with many embedded information and Knowledge-Based Systems, Volume 14 (2001) 425-430.
communication devices. The information space is, thus, built into [9] Benyon, D., & Hook, K., Navigation in Information Spaces:
the environment and people are in a very real sense inside an supporting the individual. In S. Howard, Hammond, j.,
information space. Theirs is a zone 1 medium that they will shape Lingaard, G. (Ed.), Human Computer Interaction:
and form into an environment within which they can engage in INTERACT’97, Chapman & Hall, 1997.
activities. [10] Benyon, D.R., Cognitive Ergonomics as Navigation in
The semiotic analysis of information spaces provides an Information Space, Ergonomics 41 (2) Feb (1998) 153 -
alternative and, we believe, useful perspective on interaction with 156.
and through new media. Designers have this one-off chance, the [11] de Souza, C.S., et al, A Semiotic Engineering Approach to
‘one shot message’, to communicate with their users. This User Interface Design, Knowledge-Based Systems, 14 (2001)
message is the medium with which and through which people 461-465.
interact. This is part of the intersubjectivity that Dourish deals [12] De Souza, C.S.P., R.O.; Carey, T, Missing and Declining
with, seeing ‘communication between designer and user as Affordances: Are these Appropriate Concepts?, Journal of
medium’ [14]. But the medium is made of interactive systems and the Brazilian Computer Society, vol.6 (2000).
we have developed a semiotic model of interactive systems that [13] Deely, J., Umwelts Semiootika osakonna kodulehekulg,
captures the temporal as well as spatial relations between signs Semiotika 134, special volume about Jakob von Uexkull
(information artefacts), their denotations and their connotations. (2001) 125-135.
The individual also brings a host of background knowledge and [14] Dourish, P., Where the Action Is, MIT Press, Cambridge,
interpretations to the interactions in the form of his or her 2001.
‘Umwelt’. Here we see connections with the perception/action [15] Eco, U., A theory of Semiotics, Indiana University Press,
loop that characterises much phenomenology and with the Indiana, 1976.
interpretations through blends and metaphors suggested by [16] Kull, K., On semiosis, Umwelt, and semiosphere, Semiotica,
Lakoff and Johnson [17]. vol. 120 (1998) 299-310.
The semiotic analysis lets us go beyond the denoted meanings of [17] Lakoff, G.J., M, Philosophy of the Flesh, 1999.
things and asks us to consider the connotations and cultural [18] Manning, P., Semiotics and Fieldwork, Sage Publications,
effects that designs have. We are increasingly living in a physical California, 1987.
world augmented by virtual displays, and populated by [19] Manovich, L., The Language of New Media, MIT Press,
interconnected information and communication devices. This new 2001.
medium needs new approaches to assist designers and the [20] McCullogh, M., Abstracting Craft, The practiced digital
semiotics of interactive systems is such an approach. hand, MIT Press, Cambridge Massachusetts, 1996.
[21] McLuhan, M., "Understanding Media: The extensions of
8. ACKNOWLEDGEMENTS Man", Routledge, London, 1994.
Susan Turner has contributed much to these ideas during [22] O'Neill, S., Benyon, D. R. and Turner, S., The Semiotics of
discussions, as have the partners at BENOGO see Interactive Systems., To appear in Cognition, Technology
http://www.benogo.dk/. and Work (forthcoming).
[23] O'Neill, S., Benyon, D. R. and Turner, S., Semiotics and
9. REFERENCES Interaction Analysis. proceedings of ECCE 11, Catania
[1] Allot, R., Language and the origin of semiosis. In W. Noth Sicily, 2002.
(Ed.), Origins of Semiosis: sign Evolution in Nature and [24] Prates, R., de Souza, C, and Barbosa, S, A method for
Culture, Morton de Gruyter, Berlin, 1994. evaluating the communicability of User Interfaces,
[2] Andersen, P.B., A Theory of Computer Semiotics, Cambridge Interactions (2000) 31-38.
University Press, Cambridge, 1990. [25] Prates R, d.S.C., and Barbosa S, A case Study for Evaluating
[3] Andersen, P.B., A semiiotic approach to programming. In Interface Design through Communicability. ACM Designing
B.H. Peter bogh Andersen, Jens F. Jensen (Ed.), The Interactive Systems, DIS’2000., Brooklyn, NY., 2000.
Computer as Medium, Cambridge University Press, Aarhus, [26] Sebeok, T.A., The sign and its Masters, University of Texas
1993, pp. 16-67. Press, Austin, 1979.
[4] Andersen, P.B., What Semiotics can and cannot do for HCI, [27] Winograd T and Flores, F., Computers and Cognition., 1986.
Knowledge-Based Systems, Volume 14 (2001) 419-424.
[5] Arnspang, J., Benyon, D. R., Fahle, M. W., Granum, E.,
Madsen, C. W., Pajdla, T., Peleg, S., Smyth, M., Turner, P.,
Turner, S. and Weinshall, D., An Investigation Into Virtual

87