TMP 3720
TMP 3720
Abstract: The coming Internet of Things will usher in a is more or less smart depending on the standard protocols it
semantic world (analogous to the semantic web) where physical supports (which can change over time). It also differs in that
objects will be networked so that they can communicate with we use 3D virtual world technology to construct and
each other and with humans. This paper identifies protocols demonstrate the protocols in an understandable manner. Two
that smart objects will need to follow and how to use today’s 3D
potential advantages of using 3D virtual worlds to understand
virtual worlds to better simulate and better understand
protocols for tomorrow’s smart world. a future smart semantic world are: modeling is low cost when
compared to developing a deploying real world technologies,
Keywords: pervasive computing, semantic Web, Internet of and it may be that modular services that we develop for
Things, virtual world, smart objects, semantic world interoperating with virtual worlds will transfer more or less
directly to the real world.
way (for machines) to address each object uniquely. A download security updates. Many users (e.g., the elderly) are
legal ownership and an object’s type are additional challenged by this complexity and just want unintelligent,
interfaces related to aspects of identity. Nearly everyone simple, and reliable objects that lower maintenance
could afford hundreds of RFID tags (at $.07 each) to requirements. This means smart complex objects will
explicitly identify all the objects they own (though, of compete with unintelligent conventional objects on criteria
course, it is not yet cost effective to do this with like cost, reliability, functionality, and ease of use. In a world
inexpensive items at grocery stores). Local identities where every user controls hundreds to millions of smart
could be used within an enclave so only members know objects, having hundreds or millions of separate remote
the mapping to global identities – so the RFID tags in your controls (one per object) does not scale, so truly universal
home cannot be meaningful if read from outside. remotes (e.g. smarter smart phones) will be needed – we call
APIs supported – A smart object can support one or these soft controllers (think Star Trek communicators – see
multiple application program interfaces (APIs), and these [9]) because they import different object interfaces from the
different APIs might be available for different purposes objects and network. Also, different users may see the object
and to different personnel. The owner may be able to use differently – so one user has a simple controller and another
the object (turn the thermostat on or off, up or down), but a has a more sophisticated controller (solving the problem of
qualified repair person might be required to repair it. hitting the input button on a TV remote and not understanding
Security – Not just anyone should be able to command, how to reset it, a typical problem with edging today’s remote
control and communicate with my possessions. Access controls that plagues naïve users).
control could be used to specify any user’s digital rights
along with encryption to communicate securely over less III. Using Virtual Worlds to develop Smart
secure channels. Many objects will only communicate Object Protocols
with their owners or qualified repair person.
Object-to-object communication – A networked object is In the future, when people go to the store, buy a smart object
an object that humans or other objects can communicate and bring it home, a 3D model of the object will be installed
with. The network can be wired or wireless, LAN or into the virtual model of their smart home (another protocol).
WAN, use 802.11* or RFID, and some messaging Changes people make in the real world will or may affect the
language. There may be several messaging languages model and vice versa – a bidirectional mirror world [10].
such as SNMP or WSDL. In the meantime, before the real world converts to smart
Human-to-object communication – A person needs a way object protocols, we need to understand how such a world will
to command, control, and communicate with smart function. What will it be like to manage and maintain
objects. Assuming a person has a way to designate a thousands of smart objects, especially when today many of us
device and can upload information about that device (its have trouble maintaining under ten complex fairly dumb
ownership, API, …), then, a GUI or menu-based interface objects (laptops, stereo and TV, and a drawer of user
could be used to control or query the device (possibly from manuals). Certainly, we do not want to have to remember to
a remote location). set manual permissions on the TV channel by channel when a
Micropayments – There may be a cost to accessing, house guest visits (but every family member might still want
communicating with, or using an object which one does their own list of favorite channels via a personalization
not own. service). The world needs to become simpler, not more
Plugins – A basic device might be extendible with plugin complex. Therefore, we need uniform and simple ways to
behaviors. A simple thermostat might only be able to turn manage a smart world.
on or off and turn temperature up or down, but it could be
extended with a scheduler plugin for scheduling time of IV. “Intelligence” does not always reside in
day and days of the week and/or with a history logging objects
plugin for remembering all past settings (useful for
People tend to think that to make an object smart, the object
calculating energy usage, another plugin).
itself has to be smart. But, do all the smarts really have to be
Is this protocol list complete? No (other useful protocols
located inside the object? No, though some might be. Here is
are mentioned below). Each bullet needs refinement and one
a simple algorithm for making a smart world. Add item level
could argue about any or many of the characterizations. For
RFID to many or all objects. Add an RFID reader to a smart
instance, implicit identity is sufficient for many purposes –
phone (the way GPS was recently added to cell phones and RF
“buy me one of those lamps--I don’t care which one.”
plugins are now being added to control one’s TV and stereo).
Does an object have to support all the interfaces to be
Since the smart RFID enabled phone can now read the tags of
smart? Is there a core set? One possible answer is “no” – a
any object and since the phone is already connected to the
degenerate smart object might contain none of the additional
Internet, all information about the object can be downloaded
interfaces as long as it is possible to add additional interfaces
from the web-cloud. Chairs with RFID tags will immediately
from the list. The binding time for adding smart object
become smart.
protocols could be during design, assembly, or dynamically,
We have built a bookshelf to demonstrate this idea first
during use, on an as-needed basis. As interfaces are added (or
(see Figure 1). People usually do not consider a book itself to
removed), the object becomes increasingly smart.
be a smart object. However, in the real world, any published
A significant challenge for widespread adoption of smart
book is assigned a unique ISBN. We can use this identifier to
objects involves reducing complexity while increasing
retrieve more information about each book. This setting
functionality. Today, managing 5-10 network objects is
assumes that each book is labeled with some identifier like
challenging and requires humans to run virus scans, set up
RFID which stores ISBN number. Therefore, if you scan any
firewalls, change permissions, run defragmenters, and
Towards A Semantic World : Smart Objects In A Virtual World 907
book with a mobile RFID scanner, which is wirelessly To experiment with some of the smart object protocols, we
connected to the Internet, you can get additional relevant developed a collection of smart healthcare objects in the
information from the Internet. virtual world Second Life (http://secondlife.com). Second
Life is a reasonable platform for this experiment - but it has
limitations as a long-term virtual world platform, in part
because its server farm is owned by Linden Labs. But we
could have performed the same demonstrations in
OpenSimulator (http://opensimulator.org).
We visited the University of Arkansas School of
Nursing’s training facilities. One of our interests was to
determine how to overlay training scenarios on virtual world
architectures (considered briefly later in this paper). This
paper mainly focuses on the subproblem of how we built
smart objects needed for the training. Screenshots appear
below to give the idea of what we developed (see Figures 2-8),
and videos are available on the web to see the function of
these objects (videos at
http://vw.ddns.uark.edu/index.php?page=media: infant
warmer [14MB], device controller [6MB], IV Drip [3MB]).
We developed a collection of smart objects as follows:
A hospital bed has several functions to make the patient
feel comfortable. The angle of the bed is adjustable. The
Figure 1. Smart Shelf
bed has a table which users can pull out.
Following are capabilities that illustrate a smart bookshelf An air conditioner/heater is attached to a wall of the room
containing smart books. – it can be turned on or off or from cool to warm visually
A user’s avatar wanting to search for a book (using its title displayed with blue and red particle effects.
or ISBN number) can ask the book shelf for the book’s A nursing dummy the same scale as a human is used for
location. The bookshelf consults a remote database, finds training a nurse in real life. We developed an infant
the book and displays the location. dummy. The dummy can be opened to show its internal
A user asks a book for more information, its ISBN is used organs. Each organ displays its name when it is clicked so
to remotely query Amazon for basic information on the that the nurse-in-training can learn which organs are
book or reader recommendations. Received information is which.
displayed on the screen of the remote control smart phone An infant warmer is a machine which keeps a baby warm.
device. It has mechanical arms to give the infant oxygen and
The smart shelf can periodically inventory itself for smart measure his suction. It can display an X-ray from a nearby
books and then sends updates to the remote inventory portable X-ray machine.
database. A portable X-ray machine has a screen where the digital
Of course, to get full value, future devices will need to be picture of the X-ray taken by this machine is displayed.
manufactured with network controls so that people can When the machine is clicked, it moves its arm upward and
remotely control their behaviors. This is not to say that smart approaches the object. Then, it accesses the X-ray sheet in
objects will themselves contain no processing; rather, the infant warmer and displays it on the digital screen.
knowledge and processing that makes a smart object smart When clicked again, it goes back to its original state and
might be contained in the object, the universal controller (see turns the digital screen off.
below), the user, and/or various information sources on the A search robot roams the virtual healthcare clinic to search
Internet and different smart objects might distribute this for and catalog other smart objects - which are objects that
information differently (for instance the Internet might be obey our reflection protocol that will download their API
only intermittently available and smart objects might need to to the search robot from a database we keep in a web
cache some of their log history to upload it later. cloud. The robot has a remote control and is able to leave
the users sight to discover new smart objects as it traverses
V. Prototype of Smart Objects and Soft the clinic independently. A GPS control is a handheld
Controller device which has scheduling capabilities. It can store the
current location of the user as he enters the checkpoints
3D virtual worlds give us a way to learn to manage and which the robot will follow. This way the user can create
manipulate smart objects, and simulating those objects in a previously fixed paths which the robot will now be able to
virtual world helps people imagine how new devices would traverse by itself. An RFID tag is an identification tag
change the world. Also, since developing and testing in a which responds to an RFID source. It responds by
virtual world may eventually require less cost than in the real providing identification information and its location.
world, this approach to prototyping and testing could provide
advantages over real world prototyping and testing. In all
likelihood, the smart object interface protocols we develop
can be platform-agnostic, so they can operate either in the real
or virtual world.
908 Eguchi and Thompson
each smart object, which is retrieved from an online database already in place. There is a manifest destiny that more and
by sending a query with a set of unique ids associated with more object types will become smarter. So far, this is
each smart object (Figure 9), on the soft controller screen. happening application by application (for example, smart
Avatar users can choose from the menu and control the home entertainment, security systems, and washing machines
devices. The controller has text based instructions on its - but these are not typically not interoperable). To get the
screen. The names of smart objects in range are displayed on most value, interoperability standards will be needed to enable
the screen, and users are asked to choose one. Then, the plug-and-play so that all objects obey a suite of smart object
functions of the chosen device are displayed on the screen, protocols, possibly with many implementations.
and the avatar can choose from the list and send the command Understanding more about such a suite and testing it early can
to the device. Also, it is important to point out that since we accelerate progress toward a universally smart world. As we
use the same format of input for each smart object, there can said above, virtual worlds give us a way to design and test
be more than one controller. An example is a web-browser these protocols in the near term.
based controller that also uses the soft controller architecture
(Figure 10) [12]. VI. Training
One additional result from our work regards training. We
developed two training scenarios:
The first involved training to monitor the patient’s blood
oxygen supply. The infant mannequin has a function of
saturation of peripheral oxygen (SpO2).. When a trainer
avatar types “/5 start SpO2”, the infant starts with 100%
SpO2 but the level decreases. The infant’s face looks
paler as it loses SpO2, and if it goes under 75%, the infant
dies. As SpO2 goes under 95%, a monitor shows the
message “problem zone”, and as it goes below 85%, an
alarm message indicates “danger zone.” When the nurse
avatar in training administers oxygen, the infant
gradually stabilizes until it returns to 100%.
In the second training exercise is shown in Figure 11.
The training serves the purpose of virtually giving nurses
practice in setting the proper infusion bottles. Training is
begun by touching the console on the IV drip stand. A
prompt instructs the nurse to set a certain bottle. If
selected successfully, a new bottle is prompted, and this
Figure 10. Web browser based controller continues until training is complete. Once training is
complete, an overall score is produced.
Instead of a frequency band like infrared in the real world, Actual nursing mannequins are expensive and a
in Second Life, various channels are used to communicate nurse-in-training must go to a nursing school or similar
between devices and avatars or devices and other devices. All facility to train with them. We originally conjectured that
smart objects are in one sense connected to each other because using virtual worlds to simulate nursing dummies and
all the devices can be controlled by a soft controller. associated procedures could accelerate nurse training for
Although each device has a different use and different nurses anywhere in the world at any time at no cost. We still
commands and works independently, by implementing a believe our conjecture is valid but to a more limited extent –
common format of the input, they have a universally the virtual world can familiarize nurses-in-training with
formatted API. devices, their operation, and with procedures and thus can be
While the implementation described above is Second Life used for training. However, certain actions such as learning
specific, a similarly functioning implementation could be the physical action of administering a shot or the fine motor
implemented for other virtual worlds or for smart phones in skills needed to open a latch still require some hands on
the real world. For instance, we could have used a web experience. We should note that we did not actually train
service discover the API of smart objects. nurses using our two training scenarios - our interest was
Extending smart phones with RFID readers and smart whether we could involve smart objects in training protocols
objects with network actuators. Building phone apps for and how difficult it would be to set up such protocols since
smart objects. Interfacing phone apps not just to the real experience in setting up serious training simulations is still
world but to virtual worlds. very limited in virtual worlds (though some work has been
Arranging the hundreds of thousands of real world smart done at Imperial College London and others).
objects into “lower” or product ontologies to make it easier
to develop protocols using categories and inheritance [13].
Identifying additional smart object protocols (e.g., touch,
taste, smell).
Making objects even smarter [14].
Standards will be needed. By our description of smart
objects, some objects are already somewhat smart, and more
are becoming smarter every day, so a migration path is
910 Eguchi and Thompson
Author Biographies
Akihiro Eguchi is a senior at the University of Arkansas
double majoring in computer science and psychology with
a minor in mathematics. His interests include smart world
technologies, artificial intelligence, computational
linguistics, and image processing.