Author: Tina Schart

As I want to use XR (extended reality includes AR, VR & MR) for my master thesis and since the current topic is prosopagnosia and I want to influence their lives for the better, I thought it would be useful to research if and how much VR or MR can influence our emotions. Actually there exist a few various studies on the subject by now and I looked into a few of them. But first it is important to clarify the terms mood and emotion because the two are not the same. Mood does not have to have a specific cause, but is long-term and has low intensity where emotions are triggered by specific things, are short-lived and much more intense.
Emotions can be differently measured or recognized because each person responds differently based on their lifestyle and culture. However, the main characteristics to be able to measure emotions are in most cases subjective evaluation and perception, actions, facial or vocal expressions, heart rate, skin reactions and a few others. This is where user experience comes into play, especially in VR. In order to ensure a good UX some areas need to be included. This includes the feeling of presence within the environment or immersion, the system’s ability to interact with user input, and the user’s involvement in the virtual environment. It can be both passive and active. However, the user must always be a part of the environment or able to do something with it.

But back to the studies.

The Virtual Counselling Environment

The first study deals with the extent to which virtual people can have an influence on the emotions of the user.

To investigate this, a computer-generated counselling environment was created. Furnishings were oriented on a normal therapy room, which means sofas, paintings, books, a table and a chair on which the user can sit down. The user should feel as if he is in this virtual world and sitting on a real chair. The patient is a virtual human called ‘Justin’. The therapist is the user wearing the headset. So a training environment has been set up where virtual patients can be treated by novice clinicians and those therapists can practice challenging situations. To make it more realistic, it is important that the complete process is included in the virtual environment. This includes coming in, reactions, statements and leaving the room. The virtual human has basic movements, human facial features and a human voice. The whole system works only if speech recognition is built in, because only then a positive response is possible and the user is not frustrated.
Surprisingly, the virtual human could simulate human emotions so well that the user could recognize them.

Study results

• VR has a strong influence on the emotions of the user
• A natural human computer interaction has an impact on the user
• Speech recognition is important, because only then it seems more natural and the emotional level is much stronger
• The use by controllers has less influence on the emotional response

Virtual Park Scenarios

The next study is about whether a scenario of a fictional VR park can evoke joy, sadness, boredom, anger and anxiety. 
The Velten Mood Induction Procedure method was chosen, which means that under controlled conditions images, film clips or music are shown which evoke temporary emotional states. This method tests effects of emotional states, memory or change in information processing.
In addition, electrodermal activity (EDA) was applied for this study. 

„Electrodermal activity (EDA) refers to the variation of the electrical properties of the skin in response to sweat secretion. By applying a low constant voltage, the change in skin conductance (SC) can be measured non-invasively (Fowles et al., 1981).“[3]

To be able to evoke emotions, the feeling of presence must be considered above all. Presence is often defined as the sense of being there in a Virtual Environment and because of this feeling, an emotion can occur. So, a greater presence would create a much greater reaction and the user would behave, feel or think as if he or she were in a similar situation, because it does not feel as if what he or she is experiencing is only from technology. Frame Rate actually affects the feeling of being present as well. 
For the study, 120 students were selected to first of all relax for 5 minutes and then explore the park in the VE environment in first-person view for 5 minutes.

1. Joy: sunny, calm, quit, daytime scenario with chirping birds, non-playing characters
2. Anger: constant, unnerving sound of heavy construction work
3. Boredom: dull and boring scenario with sound of distant traffic
4. Anxiety: gloomy night-time, dim light, owl sound, non-playing characters silhouettes
5. Sadness: grey, rainy day, rain dripping sound, non-playing characters carrying umbrellas and walking hastily 

Study results

• Joy, anger, anxiety and boredom have triggered the planned emotional states
• The Park vor sadness failed because boredom and sadness are often accompanied by one another
• Most negative emotions are accompanied to a small degree by other negative emotions

Feelings of fear and anger in VR

The purpose of this study was to find out to what extent the user’s emotions depend on the medium used, in this case VR and computer screen. To be even more specific, fear and anger were selected and compared with each other. Because especially from these two negative-valance emotions it is possible to see clear differences. The willingness to take risks increases with anger, while it decreases with fear. Both emotions have in common that they are negative and that we are strongly aroused by both. The differences are not only the willingness to take risks but also the control over the body and the awareness of the emotion.
To trigger anger, a short film clip from the movie ‘My Bodyguard’ was chosen, in which a group of teenagers bully weaker people. For the emotion fear, the subjects had to play the VR horror game ‘Play with me’. Both emotions were shown to the subjects on a computer screen as well as in VR.

Study results

• Decisions can change depending on the emotional state in which the person is at the time
• People take more risk when they are angry
• VR has a much higher impact on decision-making behavior compared to the use of a computer

Emotional reactions are intensified much more by VR

To get to the bottom of this, the GAPED picture database was used. GAPED stands for Geneva Affective Picture Database. In this database 730 pictures were collected, which are supposed to trigger different states of mind. These included negatively affected pictures such as snakes or pictures about mistreatment (injured animals or butchering), positive ones such as puppies, nature scenes or laughing faces, and neutral pictures such as non-living objects (bicycle spokes).
Participants had to answer three questions and rate their emotional state on a scale after viewing 24 selected images for 4 seconds.

Study results

• For all images, the arousal was much higher with VR than with a computer screen
• Photos with snakes or spiders where associated with phobias so the arousal was even higher
• By diving into the world and blocking out the reality, the much higher arousal of the user could be explained
• The attention span could be much higher when virtual reality is used

Sources:

1. The effect of two different types of human-computer interactions on user’s emotion in virtual counseling environment, Tu Ziqi, Weng Dongdong, Cheng Dewen, Shen Ruiying, Fang,Hui, Bao Yihua (10.2019), https://booksc.org/book/80763010/fcb00e
2. Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios, Anna Felnhofera, Oswald D.Kothgassnera, Mareike Schmidt, Anna-Katharina Heinzle, Leon Beutl, Helmut HlavacsbIls, Kryspin-Exner (10.2020), https://www.sciencedirect.com/science/article/abs/pii/S1071581915000981
3. A continuous measure of phasic electrodermal activity, Mathias Benedek, Christian Kaernbach (30.06.2010), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2892750/
4. The Feeling is Real: Emotion Elicitation in Virtual Reality, Sahinya Susindar, Mahnoosh Sadeghi, Lea Huntington, Andrew Singer, Thomas K. Ferris (11.2019), https://journals.sagepub.com/doi/pdf/10.1177/1071181319631509
5. Can Virtual Reality Increase Emotional Responses (Arousal and Valence)? A Pilot Study, Sergio Estupiñán, Francisco Rebelo, Paulo Noriega, Carlos Ferreira, Emília Duarte (2014), https://link.springer.com/chapter/10.1007/978-3-319-07626-3_51
6. Physiological Measures of Presence in Stressful Virtual Environments, Frederick Phillips Brooks Jr., Michael Meehan, Brent Insko, Mary C. Whitton (08.2002), https://www.researchgate.net/publication/2529722_Physiological_Measures_of_Presence_in_Stressful_Virtual_Environments

VR, AR or MR is already changing the way we think. The experiences, thoughts and emotions we have while using VR for example, keep us engaged long after we take off the headset and return to the real world. Because these experiences in the interactive, virtual world affect us just as much as face-to-face interactions, we feel the same way. VR can trigger emotions and feelings as if you were really physically there. This experiences are such powerful tools for building empathy because it makes little difference in our brains whether we perform an action ourselves or someone else does; the mirror neurons do the same. Our brains can’t tell the difference between real and virtual, which is why both bad and good experiences can have real, emotional effects. It may happen that our brain thinks that we have a different body. This can happen when sensory, visual and perceptual feedback matches. To give an example, it has been discovered that people who had an avatar with lighter or darker skin color while using VR had less racist prejudice after leaving the virtual world. Other advantages of it are, for example, trainings for very dangerous situations, because these virtual experiences trigger real emotional and physical reactions. However, it may be necessary to include real-world elements or constraints to remind the user of the virtual nature of their world or to ground the user over and over again. Should no limitations or assistances be built in and the brain becomes too accustomed to the virtual body, it can cause the user to lose control and make decisions that they would never actually make in the real environment. To prevent the brain from no longer recognizing boundaries, it can help for the user to put on a kind of vest and slip into the avatar’s body or walk across some kind of bridge. The goal, should VR be used for therapies, is for the user to benefit and learn from the scenario and fully embrace the experience as as his or her own.

Because VR can do both good and bad, Michael Madary and Thomas Metzinger drafted two lists and recommendations for the research ethics of VR and for the use of VR by the general public. These lists are at least first steps because the extent to which VR impacts us is very complex to understand. And because it is so complex, there is a need to think about ethics.

Super short summary of both lists into one.

1. No permanent or serious harm should be caused to the subject or user.
2. Every participant needs to be informed about lasting and serious behavioural impacts resulting from VR and that not every consequence is known.
3. Media and Researches should be transparent especially when virtual reality is being discussed as medical treatment and they should avoid overstating the benefits of VR.
4. There must be awareness of dual use, which is when the technology is used for something other than its original intent. „Torture in a virtual environment is still torture.“ (Madary & Metzinger, 2016, p. 19).
5. Responsible handling of sensitive data that can be recorded through VR (eye movement, emotions, body movements) must be ensured and the trust of users must not be abused by researchers or commercial companies.
6. When it comes to advertising, data protection must also be taken into account. Consumer behavior can be seriously influenced, since advertising can be precisely adapted to the user and could therefore influence entire mental mechanisms.

Sources:

1. VR changes how we think. Now what?, Artefact group (n.d.), https://www.artefactgroup.com/ideas/vr-changes-how-we-think/
2. We’re Already Violating Virtual Reality’s First Code of Ethics, Daniel Oberhaus (06.03.2016), https://www.vice.com/en/article/yp3va5/vr-code-of-ethics
3. Recommendations for Good Scientific Practice and the Consumers of VR-Technology, Michael Madary, Thomas Metzinger (02.2016), https://www.researchgate.net/publication/295083641_Recommendations_for_Good_Scientific_Practice_and_the_Consumers_of_VR-Technology