Lesson05

Emotional Interaction
- Affective Computing and Emotional AI

http://fpt.edu.vn 12/01/24 2
Affective computing is concerned with how to use computers
to recognize and express emotions in the same way as
humans do (Picard, 1998). It involves designing ways for
people to communicate their emotional states, through using
novel, wearable sensors and creating new techniques to
evaluate frustration, stress, and moods by analyzing
people’s expressions and conversations. It also explores how
affect influences personal health (Jacques et al., 2017). More
recently, emotional AI has emerged as a research area that
seeks to automate the measurement of feelings and
behaviors by using AI technologies that can analyze facial
expressions and voice in order to infer emotions.

http://fpt.edu.vn 12/01/24 3
A number of sens-ing technologies can be used to achieve
this and, from the data collected, predict aspects of a user’s
behavior, for example, forecasting what someone is most
likely to buy online when feeling sad, bored, or happy. The
main techniques and technologies that have been used to do
this are as follows:
• Cameras for measuring facial expressions
• Biosensors placed on fingers or palms to measure galvanic
skin response (which is used to infer how anxious or nervous
someone is as indicated by an increase in their sweat)
• Affective expression in speech (voice quality, intonation,
pitch, loudness, and rhythm)
http://fpt.edu.vn 12/01/24 4
• Body movement and gestures, as detected by motion
capture systems or accelerometer sensors placed on various
parts of the body
The use of automated facial coding is gaining popularity in
commercial settings, especially in marketing and e-
commerce. For example, Affdex emotion analytics software
from Affectiva (www.affectiva.com) employs advanced
computer vision and machine learning algorithms to catalog
a user’s emotional reactions to digital content, as captured
through a webcam, to analyze how engaged the user is with
digital online content, such as movies, online shopping sites,
and advertisements.

http://fpt.edu.vn 12/01/24 5
Six fundamental emotions are classified based on the facial
expressions that Aff-dex collects.
• Anger
• Contempt
• Disgust
• Fear
• Joy
• Sadness

http://fpt.edu.vn 12/01/24 6
These emotions are indicated as a percentage of what was
detected beside the emotion labels above the person’s face
appearing on a display. For example, Figure 6.10 shows a
label of 100 percent happiness and 0 percent for all the
other categories above the woman’s head on the
smartphone display. The white dots overlaying her face are
the markers used by the app when modeling a face. They
provide the data that determines the type of facial
expression being shown, in terms of detecting the presence
or absence of the following:
• Smiling
• Eye widening

http://fpt.edu.vn 12/01/24 7
http://fpt.edu.vn 12/01/24 8
• Brow raising
• Brow furrowing
• Raising a cheek
• Mouth opening
• Upper-lip raising
• Wrinkling of the nose
If a user screws up their face when an ad pops up, this
suggests that they feel disgust, whereas if they start smiling,
it suggests that they are feeling happy. The website can
then adapt its ad, movie storyline, or content to what it
perceives the person needs at that point in their emotional
state.
http://fpt.edu.vn 12/01/24 9
Affectiva has also started to analyze drivers’ facial
expressions when on the road with the goal of improving
driver safety. The emotional AI software perceives if a
driver is angry and then suggests an intervention. For
example, a virtual agent in the car might suggest to the
driver to take a deep breath and play soothing music to
help relax them. In addition to identifying particular
emotions through facial expressions (for example, joy,
anger, and sur-prise), Affectiva uses particular markers to
detect drowsiness.

http://fpt.edu.vn 12/01/24 10
These are eye closure, yawning, and blinking rate. Again,
upon detecting when a threshold has been reached for
these facial expressions, the software might trigger an
action, such as getting a virtual agent to suggest to the
driver that they pull over where it is safe to do so.
Other indirect methods that are used to reveal the
emotional state of someone include eye-tracking,
finger pulse, speech, and the words/phrases they use
when tweeting, chatting online, or posting to
Facebook (van den Broek, 2013).

http://fpt.edu.vn 12/01/24 11
The level of affect expressed by users, the language they
use, and the frequency with which they express
themselves when using social media can all indicate their
mental state, well-being, and aspects of their personality
(for instance, whether they are an extrovert or introvert,
neurotic or calm, and so on). Some companies may try to
use a combination of these measures, such as facial
expressions and the language that people use when
online, while others may focus on just one aspect, such
as the tone of their voice when answering questions over
the phone.

http://fpt.edu.vn 12/01/24 12
This type of indirect emotion detection is beginning to be
used to help infer or predict someone’s behavior, for
example, determining their suitability for a job or how they
will vote in an election.
Another application of biometric data is being used in
streaming video games where spectators watch players,
known as streamers, play video games. The most popular
site is Twitch; millions of viewers visit it each day to watch
others compete in games, such as Fortnite. The biggest
streamers have become a new breed of celebrity, like
YouTubers. Some even have millions of dedicated fans.

http://fpt.edu.vn 12/01/24 13
Various tools have been developed to enhance the viewers’
experience. One is called All the Feels, which provides an
overlay of biometric and webcamderived data of a streamer
onto the screen interface (Robinson et al., 2017). A
dashboard provides a visualization of the streamer’s heart
rate, skin conductance, and emotions. This additional layer
of data has been found to enhance the spectator experience
and improve the connection between the streamer and
spectators. Figure 6.11 shows the emotional state of a
streamer using the All the Feels interface.

http://fpt.edu.vn 12/01/24 14
http://fpt.edu.vn 12/01/24 15