1. Introduction

This paper provides a systematic review of the use of immersive technologies in Empathic Computing experiences. According to Clark^[1], Alfred Adler described empathy as "seeing with the eyes of another, listening with the ears of another, and feeling with the heart of another". Empathic computing is an emerging field that explores how technology can be used to create shared understanding or empathy between users^[2]. According to Piumsomboon et al.^[2], empathic computing systems involve a combination of natural collaboration between users, user experience and environment capture, and implicit understanding of user emotion and context. The focus is on creating shared emotional experiences, and so it extends the earlier work by Picard on Affective Computing^[3] and similar systems where the focus is on creating interfaces that recognize emotion from individual users. Picard stated affective computing as a type of computing that "related to, arises from, or influence emotions". They have defined emotion as a pivotal mechanism for expressing human feelings, delineating perceptions and cognitive states, fostering empathy, and facilitating interactions among individuals. Subsequently, advancements in technological innovation within the realm of human-computer interaction have transformed affective computing, culminating in the emergence of empathic computing as the prospective evolution of this field. Empathic computing advances prior Human-Computer Interaction (HCI) research on emotion recognition by enhancing the integration of emotional data and promoting richer human-machine interactions.

Empathic computing encompasses several key elements, including Emotion Recognition, Context Awareness, Adaptive Interaction, Feedback Mechanisms, User Modelling, and Ethics and Privacy^[4]. In such systems, emotion can be implicitly recognized through a variety of different channels, including spoken words, facial expressions, behavioral patterns, and physiological responses^[5]. Virtual Reality (VR) can be used to enable users to feel "present" in a synthetic 3D scene^[6], and so is an ideal technology for sharing experiences. For example, a VR display can be used with live 360 video streaming to enable a remote person to feel like they are sharing the same location as a remote collaborator. Similarly, Augmented Reality (AR) seamlessly blends digital content with the physical environment and provides an intuitive way to share communication cues that enhance natural collaboration between remote people. For example, AR is used to enable a remote user to overlay gaze and gesture cues over a local user's workplace and so help them to significantly improve their performance on a physical task. Both VR and AR provide full immersion and presence to users to experience emotions more vividly and authentically compared to traditional emotion elicitation/recognition tools and methods. AR/VR systems can integrate real-time feedback loops, responding to users' emotional and physiological cues like heart rate and skin conductance. This dynamic interaction is far more advanced than static computational tools, enabling adaptive experiences that are critical for emotional elicitation and regulation. Moreover, users can navigate and interact within a 3D space, which creates a more engaging and contextually rich experience, that is often missing in other conventional methods. Thus, AR and VR technologies provide an ideal medium to induce, study, and understand emotions, moods, feelings, reasoning, deliberation, and planning^[7,8].

Systematic reviews for AR/VR research are crucial since they provide a comprehensive and unbiased synthesis of existing studies, identifying trends, gaps, and inconsistencies. They also help researchers understand the current state of knowledge, guide future research directions, and ensure that new studies build on a solid foundation of previous works. There have been previous systematic reviews and surveys of the use of AR/VR technologies for enhancing face-to-face and remote collaboration^[9] and evaluating collaborative AR systems^[10-14], but not of empathic computing. For example, Schäfer et al. conducted a literature survey from the viewpoint of the three pillars of the remote collaboration system: namely Environment, Avatars, and Interaction^[13]. Similarly, Ens et al. covered 30 years of collaborative AR or Mixed Reality (MR) research and integrated it with established theories from computer-supported cooperative work (CSCW), aiming to guide MR researchers toward productive future endeavors^[11]. These reviews have helped researchers understand previous work in the field, the technologies and methods typically used, and most importantly, the research gaps and opportunities for future work. However, Bernal et al. explained that empathy research in immersive virtual environments (IVE) using devices like Galea encompasses (1) emotion elicitation, (2) emotion recognition, and (3) fostering empathy^[15]. There has been a wide range of research using AR and VR focusing on each of these three aspects separately, however, there have been no previous systematic reviews that address all three elements together to address the role of immersive technologies in empathic computing.

Motivated by previous systematic reviews and relevant gaps in this field, our systematic review of the use of immersive technologies (AR/VR) for empathic computing is designed to identify research trends and knowledge gaps to advance the field. The novel contributions of the paper are to:

• Explore the research trends in empathic computing and immersive technologies from the last two decades.

• Explore the use of immersive technologies for emotion elicitation, recognizing emotions, and fostering empathy in various application domains.

• Investigate the various emotion models in immersive empathic computing.

• Explore the perception and cognition related to immersive empathic computing.

• Identify the research gaps and future opportunities for the use of immersive technologies in empathic computing.

2. Method

In this study, we followed the preferred reporting items for systematic reviews and meta-analyses (PRISMA) as a reporting guideline^[16]. PRISMA is widely used for systematic reviews and meta-analyses as it guides researchers in reporting their findings transparently and systematically by providing a checklist and a flow diagram. Following the PRISMA guidelines, we collected papers that integrated empathic computing research in immersive technologies from the leading VR/AR conferences and journals. This included IEEE Virtual Reality and 3D User Interfaces (VR), IEEE International Symposium on Mixed and Augmented Reality (ISMAR), ACM Virtual Reality Software and Technology (VRST), IEEE Transactions on Visualization and Computer Graphics (TVCG), and the ACM Conference on Human-computer Interaction (CHI). We restricted the search to the past two decades from 2000-2024 due to significant advances in AR/VR technologies during that time. The past two decades have seen key publications, frameworks, and innovations aligning with industry trends and ensuring a focus on contemporary technologies relevant to current and future developments. So, focusing on this period avoids outdated technologies, capturing relevant innovations aligned with present and future trends^[17]. We chose to search the conferences and journals for papers with the keywords ("Empathic" OR "Emotion") AND ("Virtual Reality" OR "Augmented Reality" OR "Mixed Reality" OR "Extended Reality") in their title or abstract. Doing this, we ended up with 681 papers in total, with 586 unique papers from the IEEE and ACM databases, and 95 from the Google Scholar search.

The papers were fully reviewed by two authors independently following the PRISMA approach. As shown in the PRISMA diagram (Figure 1), 467 records were excluded from the initial screening as they were not related to immersive technology or emotion/empathy-related research. Then, we excluded 12 more duplicate papers and ended up with 202 papers. Among them, 193 papers were chosen for screening in full. Following the approach of Dey et al.^[18], we then applied minimum citation criteria to ensure that we were reviewing papers that had an impact. For the years from 2000-2018, we required papers to have an Average Citation Count of greater than 2.0 per year, while for papers from 2019-2024, we removed the minimum citation requirement. We then excluded posters, workshop papers, and extended abstracts (EA) from the records, leaving 77 papers for full review. Among these 77 papers, 35 papers study emotion elicitation, 19 papers study emotion recognition, 19 papers employ fostering empathy, 1 paper on both emotion recognition and empathy building, 1 paper focuses on recognition and classification, 1 paper on emotion representation, and 1 paper on emotion influence in cybersickness in immersive environments. After multiple rounds of filtering, the final selection consisting of 77 papers was from the last decade, specifically from 2003 to 2024.

Display Full Size

Figure 1. PRISMA approach for selecting relevant papers from 2000-2024. PRISMA: preferred reporting items for systematic reviews and meta-analyses; VR: Virtual Reality; AR: Augmented Reality.

We categorized the collected data accordingly to capture the diverse facets of empathic computing in immersive technologies, reflecting both the technical and psychological dimensions. This multi-dimensional approach ensures that the review holistically examines key areas of immersive empathic computing research. The categories used are (1) Applications and interdisciplinary approach, (2) Empathy design strategies and assistive technologies, (3) Framework of emotions in immersive environments, and (4) Perception/Cognition in immersive environments. The summary table of 77 reviewed articles is provided in Table 1.

3. Applications and Interdisciplinary Approach

Research in empathic computing using immersive technologies has grown continuously over the last 10 years, with a higher volume of research being published in the last three years (Figure 2). This study can be categorized into the application areas of healthcare, education, training, entertainment, and remote collaboration.

Display Full Size

Figure 2. Number of publications per year in immersive empathic computing.

3.1 Healthcare and therapy

Immersive environments can be used for emotional regulation, exposure therapy, and treatment of conditions such as post-traumatic stress disorder (PTSD), anxiety disorders, and phobias. These applications allow for controlled simulations of stress-inducing scenarios that are used for therapeutic purposes. Guarese et al. experimented with participants, who were sighted and blindfolded and used various sonification methods to locate and place objects in a simulated kitchen setting^[19]. Feedback from the blind and visually impaired (BVI) community was collected to assess the perceived benefits of such empathy-inducing experiences, with results indicating the potential for new assistive technologies. Their findings include that immersive AR experiences could effectively increase empathy toward the BVI community.

Immersive empathic computing is an excellent platform for therapeutic applications as it helps measure stress levels and emotional states including increasing positive and releasing negative emotions^[20]. For example, Dietz et al. showed how to maintain a dynamic balance in stressful situations to improve postural control and reduce the risk of injury^[21]. They evaluated the stress response by using physiological measures of stress and workload and used a ghost avatar to help participants concentrate on tasks. The outcome of these studies conducted by Luong et al. and Weiß et al.^[22,23] can be applied to healthcare-related training^[19], rehabilitation, psychological well-being^[24], and therapeutic applications.

3.2 Education

Education is another prominent application area to enhance emotional engagement and empathy. Using perspective-taking, embodied learning, and increasing empathy through a narrative-based VR application, Ponto et al. demonstrated how immersive technologies can promote education and informal learning^[25]. Using a VR experience with visuomotor congruence, they allowed users to experience life as an Adélie penguin, experience dispositional empathy with nature scale, avatar embodiment questionnaire, and informal learning through open-ended questions. Another study conducted by Gu et al. involved 234 middle and high school students participating in an experiment, aiming to determine if a role-exchange strategy can enhance empathy, moral judgment, and a commitment to stop bullying^[26]. They found that experiencing both perspectives (bully and victim) helped participants develop a more comprehensive understanding of the impacts of bullying and fostered a more profound emotional and moral response. Schaper et al. ran an experimental case study focusing on enhancing students' social and emotional learning (SEL) through projective AR in an educational virtual heritage context^[27]. Bian et al. showed that a positive emotional feedback strategy led to a better learning experience and performance including the personality types having a significant effect on learning^[28].

3.3 Training

Virtual training aims to simulate real-world scenarios to practice and improve specific skills and competencies. Li et al. conducted an experiment to show that the immersive nature of VR and the ability to connect with the experience emotionally play critical roles in the effectiveness of VR training^[29]. Nam et al. reflected training intertwined with the pediatric nursing application in their study where the emotions (happiness, satisfaction, neutrality, anxiety, anger) of a responsive virtual parent are dynamically updated based on the trainee's actions^[30]. The outcome of the study also shows enhanced perceived realism, learning efficacy, and usefulness of the VR simulation compared to a static emotional model.

3.4 Entertainment and gaming

Video games, movies, music, and virtual concerts are using immersive technologies, especially VR to provide more engaging and emotionally resonant experiences. Ito et al. investigated how multisensory audio-visual stimuli in virtual environments (indoor and outdoor) affected the perception of wind comfort and emotional responses^[31]. Shijo et al. investigated the emotions expressed by the posture of "Kemo-mimi"; animal-like ears on humanoid characters^[32]. Kors et al. developed a VR game "Permanent" to foster empathy where coding of empathic states revealed that participants reported other-oriented empathic states more than twice as often as self-oriented empathic states^[33]. Peng et al. used the idea of the emotional challenge to understand the player's experience^[34]. The study developed by Bevan et al. encompasses both education and entertainment where a series of ethnographic and experimental studies were conducted to explore VR non-fiction for real-life storytelling to evoke empathy^[35]. Other studies that use the application of immersive empathic research in entertainment including interactive gaming, virtual embodiment, role-playing, music, and art include^[36-39].

3.5 Remote collaboration

While being interlaced with other major application fields, there are collaborative AR/VR applications that support emotion and empathy research. Choudhary et al. investigated how mismatched facial and vocal emotional expressions from virtual avatars in VR impact users' interpretations of emotions and consequent social influence^[40]. Jing et al. developed a bi-directional emotion-sharing system using nine visual representations to enhance the communication of emotions in virtual spaces^[41]. They explored the effect of interacting gaze behaviors, hand-pointing gestures, and heart rate visualizations to affect remote collaborations in empathic MR environments. Semertzidis et al. used MR to improve interpersonal communication of emotions using a neuro-responsive system called "Neo-Noumena"^[42]. Finally, empathy glasses provided empathic visual information exchanges between remote collaborators in the study conducted by Masai et al.^[43].

4. Empathy Design Strategies and Assistive Technologies

Immersive empathy research encompasses (1) Emotion Elicitation, (2) Emotion Recognition, and (3) Fostering Empathy. These components form a comprehensive empathy design strategy by providing a structured approach to provoke, measure, enhance, and accurately depict emotional experiences in immersive environments. We discuss each of the areas in this section.

4.1 Emotion elicitation

4.1.1 Illusion and engagement

One of the most effective ways to elicit empathy is by placing users in the "shoes" of others. In immersive platforms, this is achieved through embodiment, where users inhabit a virtual avatar representing someone else's perspective. Also, multiple users can also have the same avatar body. This can be particularly powerful when the avatar differs from the user in terms of race, gender, age, or ability. Bian et al. identified embodiment interactions for enhancing social and emotional learning through projective AR in an educational virtual heritage context. VR and AR can tell compelling stories where users interact with characters and influence the narrative^[28]. Engaging narratives that focus on social issues or personal experiences can evoke empathetic responses from users^[23,35].

4.1.2 Environmental immersion

4.1.2.1 Simulations

Simulating real-world situations where users can experience what it's like to face challenges or adversities faced by others can help elicit empathy. This could include scenarios like being a refugee, experiencing discrimination, or living with disabilities, as per the findings from Guarese et al.^[44]. However, a delicate balance is required when using simulations to evoke empathy. There is a risk that such simulations could lead to patronizing attitudes rather than genuine understanding and empathy^[45-47]. Ito et al. ran a simulation of natural environments with accompanying natural wind sounds that led to a reduction in mental stress and improvements in emotional responses^[31]. In another study by Dietz et al., emotion was elicited due to VR simulation along with other physiological stimuli^[21]. Nam et al. illustrated the usefulness of the VR simulation compared to a static emotional model^[30]. Additionally, Wang et al. developed a VR simulation where users interacted with virtual humans (VHs) under different emotional and language settings^[48]. Benvegnù et al. revealed that participants experience a high sense of presence in the simulation and showcase their different reactions to driving mode, greater arousal, and more negative valance^[49].

4.1.2.2 360-degree video

Immersive platforms can use 360-degree videos to present narratives from another person's viewpoint, allowing users to see what they see and hear, thereby promoting empathy through perspective-taking. Bindman et al. made use of 360 immersion in both high (industry VR headset) and low (smartphone) fidelity headsets, in terms of engagement and empathy^[36]. Tse et al. used 360 immersive mobile VR to arouse empathy in a fire Rescue scenario. They focused more on the immersive experience of 360 videos to elicit arousal emotions^[50].

4.1.2.3 Sensory feedback

Incorporating physical sensations like haptic feedback or environmental effects (e.g., temperature changes) can enhance the sense of presence and make experiences more relatable, thereby eliciting empathy. A positive emotional feedback strategy led to a better learning experience and performance^[28]. Guarese et al. explores the integration of audio cues and visual feedback through AR and VR devices, aiming to enhance the guidance experience and provide a more intuitive understanding of the physical space for both users^[44]. Kono et al. develops a novel system called 'in-pulse' for exploring the potential of bringing emotional feedback to users^[51]. Dey et al. showed that displaying heart rate feedback tended to enhance the observer's understanding of the active player's emotional state^[37]. Another study by Guarese et al. showed that feedback from the blind and visually impaired (BVI) community could be used to assess the perceived benefits of empathy-inducing experiences^[19].

4.2 Emotion recognition

4.2.1 Biometric monitoring

4.2.1.1 Physiological data analysis

The most common physiological cues for empathy research are Heart Rate Monitoring, electrodermal activity (EDA), and electroencephalography (EEG). heart rate (HR) and heart rate variability (HRV) are measured to indicate emotional arousal and empathetic engagement. Ito et al. measured HRV in response to multisensory audio-visual stimuli in virtual environments (indoor and outdoor) and found out how it affects the perception of wind comfort and emotional responses^[31]. The study conducted by Jing et al. used HR as a visual stimulus to affect remote collaborations in empathic MR environments^[41]. Also, Valente et al. used both HRV and EDA in their experiment to recognize participants' emotional states and visualize through the AR system^[52]. Voigt-Antons et al. used HR and HRV were used to elicit emotions by repeatedly playing horror games^[53]. Lee et al. visualized the HR and respiration rate data in a social VR experience to understand emotion^[54]. In another study by Prachyabrued et al., HR and Galvanic Skin Response (GSR) were measured to develop a story-driven virtual environment for stress inoculation training in the context of emergency healthcare personnel^[55]. Dey et al. used the Empatica device^[56] to monitor HR and other physiological signals and investigate the effects of sharing real-time HR data between players in collaborative VR games and if it enhanced emotional connection and gameplay experience^[37]. Li et al. investigated the effectiveness of using VR as a tool for emotion induction, by assessing both subjective (Self-Assessment Manikin (SAM) and In-group Presence Questionnaire) and neurophysiological responses (EEG data) in participants^[57]. Semertzidis et al. developed a neuroresponsive system called "Neo-Noumena" combining EEG data analysis with procedural content generation to visualize emotions as dynamic fractal patterns in an MR environment^[42]. Hou et al. used the Emotiv EEG sensor to develop a haptic-based serious game for post-stroke rehabilitation^[58].

4.2.1.2 Eye tracking systems

Several studies implement eye-tracking technology to determine emotional focus and engagement, aiding in empathy recognition through eye movements and attention. These studies mainly measure eye movements, gaze direction, and pupil dilation. For example, Jing et al. used eye-tracking data to investigate user perceptions of visual designs combining communication cues with heart rate biofeedback^[41]. The study conducted by Kourtesis et al. measured changes in pupil size to investigate the influence of emotion on cybersickness^[59]. The authors in their experiment^[22] utilized pulmonary and pupillary responses to physiological states such as fear, frustration, and insight and found the difference between fear and frustration pupillary respiratory along with EDA responses. The authors^[48] assessed how native and non-native language communication affects the non-verbal (eye gaze) behaviors of users interacting with emotionally expressive VH crowds in VR. Jhan et al. explored the presence of small talk affected users' non-verbal behaviors, such as gaze and interpersonal distance^[60]. Volonte et al. used eye gaze as a non-verbal cue to explore how language familiarity affects emotional responses in VR interactions with a crowd of digital humans displaying various emotions^[61]. Another study by Volonte et al. also used participants' eye gaze as a non-verbal cue while they interacted with VH crowds exhibiting different emotional expressions^[62]. In this study^[43], the authors used eye-tracking data for empathic visual information exchanges between the collaborators.

4.2.1.3 Facial expression analysis

Immersive technologies are effective mediums for recognizing emotions and empathetic reactions based on facial movements. Pan et al. used avatar facial expressions with audio stimuli to elicit specific emotions^[63]. Wang et al. developed a database for reconstructing the emotion reflected in facial expressions^[64]. Suzuki et al. proposed a method to map facial expressions from a user wearing a Head-Mounted Device (HMD) to a virtual avatar using embedded photo-reflective sensors^[65]. They also addressed the challenge of facial expression recognition when significant parts of the face are occluded by an HMD. Bekele et al. also utilized facial expression recognition using avatars for measuring arousal^[66]. Choudhary et al. investigated how mismatched facial and vocal emotional expressions from virtual avatars in VR impact users' interpretations of emotions and consequent social influence^[40]. Lohesara et al. developed a novel dataset for Extended Reality teleconferencing, facial expression recognition, and volumetric video^[67]. It included specific applications like facial expression classification, HMD removal, and visual quality assessment of 3D models. This study by Nam et al. integrated multimodal inputs including facial expressions to analyze the emotions of the virtual parent which are dynamically updated based on the trainee's actions^[30]. Wang et al. used some predefined emotional expressions for VHs to capture real human emotional responses^[48]. Other studies^[60-62] also used facial expressions with other verbal and non-verbal cues for emotion elicitation. Baloup et al. explored non-isomorphic methods that use input devices to control avatars' facial expressions in VR environments^[68]. In another study, Choudhary et al. utilized the emotional facial expressions (Happy, Sad, Angry, Skeptic) for manipulating the avatar head size in VR^[69]. Kono et al. also developed a novel system exploring the potential of bringing emotional feedback to users through EMS-induced facial expressions^[51]. The experiment by Bönsch et al. provided virtual agents with facial expressions to study how personal space is affected by virtual agents' emotions^[70]. Waltemate et al. also used avatar facial expressions for emotional arousal in their study^[71]. In another study by Norouzi et al., facial expression was analyzed for using a virtual pet (or non-humanoid robot) to study social emotion in AR^[72]. Though not a direct facial expression analysis, the study by Shijo et al. explored how different postures of the ears can convey various emotional states^[32].

4.2.2 Behavioral analysis

Several studies have used body gesture recognition and analysis to infer emotion and empathy. Such systems track body movements and gestures and analyze non-verbal cues related to empathy and emotion. This study by Ponto et al. used hand tracking as part of upper body gestures recognized to foster empathy by leveraging embodied learning, perspective-taking, and increasing empathy through narrative-based VR application^[25]. Another study used hand gesture analysis along with HR signals as part of emotion elicitation^[53]. Nam et al. also incorporated hand gestures along with facial expressions for emotion recognition^[30]. These authors utilized body postures in general along with other verbal, and non-verbal cues for emotion elicitation in their respective studies^[48,60]. This study by Volonte et al. also incorporated hand gestures and head movement as non-verbal cues for emotion elicitation^[61]. In another study, Volonte et al. used many non-verbal cues were used to elicit emotion in their study which tracked body postures, head movement, and hand pointing^[62]. Finally, Bian et al. conducted a physiological experiment involving two personality types and two emotional feedback strategies where they utilized upper body movement^[28].

4.3 Fostering Empathy

4.3.1 Repeated exposure

Repeated exposure to empathetic scenarios in immersive environments can enhance empathetic abilities over time. Iterative experiences with varying scenarios can deepen understanding and emotional connection. Immersive platforms can be used for empathy training, such as through role-playing exercises or guided interventions, which help foster empathy by building emotional intelligence and perspective-taking skills. The role-exchange method proves more effective in modifying participants' attitudes towards bullying compared to traditional role-playing techniques that do not involve switching perspectives^[26]. Providing feedback on users' empathetic behaviors or emotional responses, such as through virtual coaches or interactive feedback, can help foster and reinforce empathy.

4.3.2 Social interaction

Creating shared or collaborative experiences in immersive platforms, where users engage with others empathetically, can foster empathy through social connection and shared understanding. Moustafa et al. designed an exploratory study to understand how social groups use VR for collaborative activities^[73]. Dey et al. developed collaborative VR games to enhance emotional connection and gameplay experience among players^[37]. Guarese et al. proposed a novel immersive system for a collaborative, immersive tele-guidance experience between sighted and visually impaired individuals^[44]. Gupta et al. also promoted remote collaborations in empathic MR environments in their study^[74]. Enhancing social presence, or the sense of being with others in a virtual space, and measuring personal space in social interaction, and interpersonal distance in social environments can foster empathy by making interactions more realistic and emotionally engaging. This can be achieved through realistic avatars, voice chat, or shared tasks^{[60,70,73,75]}.

5. Emotion Models in Immersive Environments

In immersive technologies, emotion models provide frameworks for understanding and representing emotional states, which are essential for creating and interpreting emotional responses in AR/VR environments. Emotion models, such as dimensional or discrete models, provide insight into how users experience and interpret emotions in immersive environments. Understanding various emotion models lays the foundation for future research and innovation in empathic computing. These models play a pivotal role in empathic computing by facilitating emotion recognition, guiding emotion elicitation, supporting adaptation in immersive environments in real-time and designing experiences that foster empathy. For instance, Ekman's six basic emotions such as happiness, sadness, fear, anger, surprise, and disgust offer an outset for designing algorithms for facial emotion recognition.

5.1 Discrete emotion models

Discrete models are typically based on specific discrete emotions, e.g. Ekman's Six Basic Emotions^[76]. These models categorize emotions into distinct types, each associated with unique facial expressions and physiological responses. In our review, one study used Ekman's model to represent seven discrete emotions (Enjoyment, Surprise, Contempt, Sadness, Fear, Disgust, Anger) to evaluate the emotion recognition performance of real autism spectrum disorder (ASD) participants with the help of an avatar^[66].

5.2 Dimensional emotion models

Dimensional models such as the Circumplex Model of Affect or Valence-Arousal-Dominance (VAD) model categorize emotions based on continuous dimensions^[77,78]. Dimensional models can be sub-categorized into the following:

5.2.1 2D emotion models

The Circumplex Model of Affect^[79] is a popular 2D model used to gauge emotional responses based on valence (positive or negative) and arousal (activation or deactivation). This model has been used in immersive environments to monitor and influence users' emotional states, enhancing the effectiveness of immersive experiences. In our review, most 2D model types referred to the valence-arousal (VA) model to represent continuous emotions. Valente et al. conducted a study where the participants' emotional states were visualized through the AR system based on physiological signals (HRV and EDA), and the recognized emotions were mapped into the VA model^[52]. This study also utilized the VA model while inducing emotional states measured by subjective scales and heart rate parameters using virtual environments^[53]. While most of the studies^{[31,57,67,80]} addressed the VA model to represent various emotions, other distinct types of 2D emotions can be observed from our review. Moustafa et al. the authors mentioned the Geneva Emotion Wheel (GEW) model^[81] to assess the emotional states in their study^[73]. Although the GEW model is related to the VA model, these are two distinct models, with the GEW having control instead of arousal to represent the high-low dimension. This study addressed valence-dominance in 2D to represent the personalized avatar impacts on body ownership, presence, and dominance^[71]. Jicol et al. conducted a user study where they utilized the TAP-fear model to address fear in different types or stages in 2D^[82]. TAP stands for Threat, Anxiety, and Panic. These components are aligned in two dimensions: one related to the type of fear and the other related to the intensity or acuteness of the fear response. Bian et al. addressed a unique type of 2D model based on Eysenck's personality theory^[28]. Eysenck's model categorizes personality into four types based on the dimensions of Extraversion (E), Neuroticism (N), and Psychoticism (P). Melancholic (introverted and neurotic), Choleric (extroverted and neurotic), Phlegmatic (introverted and emotionally stable), and Sanguine (extroverted and emotionally stable) are the personalities classified based on the combination of these dimensions.

5.2.2 3D Emotion models

The VAD model adds one more dimension, called Dominance, to the 2D Valence-Arousal model. This model has been applied in AR/VR environments to capture complex emotional responses and tailor experiences accordingly. Freeling et al. addressed the VAD model in their study where a higher sense of embodiment impacted emotion both negatively and positively^[83]. Hou et al. addressed the VAD model and showed that a game could be used for patients suffering from stroke to recover even at home and without accessing a nurse^[58]. The Pleasure-Arousal-Dominance (PAD) model^[84] provides a more nuanced understanding of emotions and adds pleasure instead of valence in the 3D VAD model. These authors^[32] utilized the PAD model to investigate the emotions expressed by the posture of "Kemo-mimi," which refers to animal-like ears on humanoid characters. The Happy, Relaxed, Glad, Excited, Delighted, Serene, Sleepy, Astonished, Neutral, Tired, Angry, Disgustful, Sad, Afraid, Frustrated, Bored, and Miserable emotions were categorized using this model. Another study^[68] addressed Plutchik's Wheel of Emotions^[85] to explore non-isomorphic methods that use input devices to control avatars' facial expressions in VR environments.

Overall, the immersive empathic studies for eliciting, recognizing emotions, and fostering empathy utilized different kinds of emotion models to accurately represent the emotion states. We have demonstrated the emotion models mentioned in the studies and their correspondence with various human emotions in Figure 3. This figure provides a visual summary of how different emotion models map to specific human emotions in the selected papers over the years. The numbers represent the frequency of a specific emotion being addressed under an individual emotion model.

Display Full Size

Figure 3. Emotion Models and Emotion Types addressed in the selected publications over the years. VAD: Valence-Arousal-Dominance; TAP: Threat, Anxiety, and Panic; SAM: Self-Assessment Manikin; GEM: Geneva Emotion Wheel.

6. Perception/Cognition in Immersive Environments

Perception and Cognition are integral to the development and effectiveness of immersive empathic computing research as they contribute to enhancing empathy and emotional engagement. These domains utilize VR/AR technologies to provide seamless immersive empathic experiences by leveraging various sensory inputs and managing cognitive processes.

6.1 Perception

6.1.1 Visual perception

Immersive environments mostly rely on visual cues to create a sense of presence and realism. High-resolution displays and realistic rendering of virtual environments are crucial for conveying contextual and emotional information accurately. Findings in this study conducted by Jicol et al.^[86] show that visually sharing emotions leads to better emotional consensus and positive interdependent behaviors. These authors also used the perception of visualized AR/VR environments for recognizing and fostering empathy in their respective studies^[14,75].

6.1.2 Auditory perception

Sound design, including spatial audio, helps create immersive experiences. Auditory cues can enhance the perception of emotions and empathy by simulating realistic environments and interactions. Multi-sensory integration in AR environments for fostering empathy for visually impaired people is reflected in this study^[19] to assess the perceived benefits of such empathy-inducing experiences. Along with multi-sensory cues, they used various sonification methods to locate and place objects in a simulated kitchen setting. The authors^[44] developed a novel AR/VR system that incorporates the innovative use of audio cues for navigation, reflecting the effectiveness of audio-based guidance over visual cues. The authors^[31] investigated how multisensory audio-visual stimuli in indoor and outdoor virtual environments affect the perception of wind comfort and emotional responses.

6.1.3 Embodied perception

6.1.3.1 Sense of presence and immersion level

Jicol et al. performed a systemic investigation to find the correlation between VR and human factors in terms of the sense of presence^[82]. Their findings show that visual realism induces fear easily and thus higher presence in VR environments. Moreover, these studies^[72,83] demonstrate how the sense of presence and embodiment influence human emotion in VR. In addition, the study by Jicol et al. portrayed the impact of emotion and agency on the sense of presence in VR^[86].

6.1.3.2 Virtual avatar

Advanced systems use facial recognition technology to detect and respond to users' emotions in real-time. This study used emotional voice puppetry utilizing the virtual avatar's facial expression with audio to express several discrete emotions^[63]. Another study also utilized various facial expressions in the posture of "Kemo-Mimi" to see what emotion they perceived from different ear postures^[32]. The gestures and body movements of the virtual avatar are also crucial for conveying emotions, as this study finds. Nam et al. utilized both speech and hand gestures to demonstrate that the dynamic emotional responses of the virtual parent enhance perceived realism^[30].

6.1.4 Haptic feedback

Tactile sensations can be provided through haptic devices, allowing users to feel virtual objects or interactions, a direct measure of user perception. This feedback can enhance emotional engagement and empathy by making experiences more tangible. In the experiment conducted by Ju et al., 7 Participants created vibration patterns to represent four emotions, and the patterns were played back to them in a random order to see if they could recognize the intended emotions^[80].

6.2 Cognition

6.2.1 Cognitive load management

Immersive environments can be overwhelming due to the richness of stimuli. Effective design strategies aim to manage cognitive load, ensuring users can process information without becoming overwhelmed. Kourtesis et al. examined defective cybersickness on cognitive motor skills and reading performance^[59]. Kim et al. measured the stress levels and emotional states including increasing positive and decreasing negative emotions^[20]. The study by Dietz et al. evaluated the stress response by using physiological measures of stress workload and user experience in a VR environment^[21]. This study also implemented a virtual ICU patient scenario to learn about the effect of virtually replicated emotional stressors^[23]. Prachyabrued et al. developed a story-driven virtual environment for stress inoculation training in a healthcare setting^[55].

6.2.2 Attention and focus

Guiding user attention through design elements like lighting, sound, and movement helps focus on key aspects of the experience, enhancing the understanding of emotional and empathic cues. Schaper et al. employed a multi-method analysis including observations, questionnaires, and interviews to assess emotional and educational impacts to evaluate students' engagement and learning and qualitative analysis focusing on their interactions, emotional responses, and understanding of historical contexts^[27].

6.2.3 Empathy and perspective-taking

Immersive environments often involve role-playing scenarios where users can experience situations from different perspectives, thereby fostering empathy. Lowy et al. simulated the neurodivergent (ND) perspectives in neurotypical (NT)-structured environments to bridge the empathy gap and foster empathy between the two groups^[87]. Gu et al. showed that experiencing both perspectives (bully and victim) helps participants develop a more comprehensive understanding of the impacts of bullying and fosters a more profound emotional and moral response^[26]. The study by Jhan et al. also demonstrated how small talk influences more nuanced or complex emotions in virtual environments^[60]. Gil et al. developed a 3D AR storybook to foster the empathic behavior of children through roleplaying, storytelling, and perspective-taking^[39]. Moreover, Bian et al. showed that empathy is driven by 10 categories of cognitive perspectives: viewer role, point of view, visual composition, audio composition, gaze manipulation, evidence of embodiment, interaction, locomotion, interpersonal space, and the manipulation of time^[28].

6.2.4 Neurocognitive aspects

The use of brain-computer interface to monitor and influence cognitive and emotional states in real-time is correlated to the neurocognitive aspect of empathic immersive research. Immersive environments can activate specific brain regions associated with empathetic processing. Hou et al. developed a haptic-based game for post-stroke rehabilitation where they utilized EEG signals^[58]. Li et al. explored brain wave patterns associated with different emotional states, revealing cognitive processing of emotional stimuli^[57]. This paper represents appraisal theory which is central to understanding how participants cognitively interpret VR scenarios and subsequently experience emotions. However, this study is also related to perception as the participants in the experiment are exposed to VR videos, which are designed to elicit specific emotional responses based on their visual content. Moreover, the subjective measurement through SAM scales captures participants' perceived arousal, valence, and dominance, which are direct indicators of their emotional perception of the VR stimuli. Although the study conducted by Semertzidis et al. is mostly related to auditory and visual perception, participants engaged in introspection and reflection on their emotional states to better understand their own and their partners' emotions using this neuroresponsive system^[42].

7. Discussion

7.1 Applications

The applications of immersive emotion research highlight the transformative potential of immersive technologies. Notable studies in immersive empathic computing showcase some major application areas of this research realm^[88-91]. In healthcare, real-time emotion recognition enhances therapeutic outcomes through adaptive interventions. In this domain, we see the emerging trend of using VR/AR in exposure therapy for phobias, PTSD, and anxiety, and in rehabilitation for stroke patients. VR simulations are widely used to train healthcare professionals in empathic care by recreating patient experiences like dementia and autism. Regarding the education and training domains, immersive experiences foster empathy and understanding by engaging students with diverse perspectives and by allowing students to experience historical events or the lives of individuals from different backgrounds. Here we notice the upward trend of providing personalized feedback based on users' emotional responses in various emotion recognition systems. The entertainment industry pushes the boundaries of emotional engagement with VR experiences that adapt to users' emotions, creating personalized narratives. This domain is being taken over by VR games and experiences, adaptive storytelling that evolves based on players' emotional states, and raising awareness about social issues like anti-bullying. Finally, emotion recognition tools maintain human connections and effective communication in remote collaboration. Remote collaboration is becoming more popular over time, facilitated by virtual meetings with emotion recognition systems providing real-time feedback on participants' emotional states, improving communication and understanding. Moreover, the integration of AI and personalization trends reflects the evolving sophistication of these technologies, promising more effective applications. These advancements emphasize the importance of continued research and innovation in immersive empathic computing to develop environments that entertain, educate, and foster deeper human connections.

7.2 Empathy design strategies

We have explored various strategies for emotion elicitation, recognition, and fostering empathy using immersive technologies that can manipulate emotional states, using a combination of verbal and non-verbal cues, as well as diverse stimuli. The analysis of emotion recognition techniques involving deep and recurrent neural networks^[52,63,67] showcases the potential for future innovation in emotion classification and early emotion prediction. This could lead to the possibility of mitigating cybersickness^[92], stress, and cognitive load.

7.3 Emotion models for measuring emotions

From our review, the majority of the papers used the VA model as a prime 2D model to map the emotions elicited or recognized in the immersive empathic systems. However, some papers also introduced other 2D models like the TAP-fear model to map fear emotion, the GEW model to describe control instead of arousal-valence, and the valence-dominance model. Furthermore, we see the use of 3D models mentioned in some of the studies. For example, these two studies^[58,83] used the VAD model while Mehrabian et al.^[84] introduced PAD model and Plutchik et al.^[85] utilized Plutchik's wheel of emotions in their respective studies. Only one study^[66] utilized Ekman's discrete emotion model to represent emotions. This indicates a trend toward capturing a wider range of emotional states over time. The application of discrete emotion models, like Ekman's or Plutchik's, suggests an emerging interest in representing specific, distinct emotions within immersive environments. This diversity in emotion modeling reflects a broader trend toward tailoring emotional representations to specific applications and user experiences. Future research may continue to explore and develop context-specific emotion models that align with the unique characteristics and needs of different immersive environments. As empathic computing evolves, there may be an increased emphasis on hybrid models that combine both dimensional and discrete frameworks to offer even more comprehensive representations of human emotion.

7.4 Body postures for emotion representation: a new perspective

An intriguing aspect of emotion/empathy research using immersive technologies involves gesture recognition. While some studies have explored hand tracking^[25,30,53], head movements^[61,62], and upper body tracking^[48,60] for recognizing emotions, few have investigated lower body postures, particularly leg patterns, to determine how they correlate with emotional states. This highlights a potential avenue for future research, specifically focusing on gait pattern analysis and its relationship with emotional states in immersive environments. Notably, some researchers have addressed this aspect of body behavior analysis within the context of immersive empathic domains^[93-99]. However, rigorous empirical studies are needed to explore lower body behavior analysis for emotion recognition. Analyzing gait patterns in immersive environments could be valuable for applications such as rehabilitation, virtual therapy^[98] and assisting individuals with ASD.

7.5 Research trend in perception/cognition

Over the last decade, significant progress has been made in understanding the role of perception and cognition in immersive empathic computing^[89,100,101]. Recent research underscores the critical role of advanced sensory integration, visual and auditory enhancements, haptic feedback, and emotional cues in immersive empathic computing. Concurrently, optimizing cognitive load, managing attention, and leveraging neurocognitive insights have been pivotal in enhancing empathy and emotional engagement. These developments collectively advance the potential of immersive technologies in fostering empathy across diverse applications. We notice recent research highlighting the importance of synchronizing multiple sensory inputs (visual, auditory, haptic) to enhance user presence and emotional engagement^[14,80,86]. Advances in high-resolution displays, realistic rendering, and spatial audio have significantly improved immersion. These enhancements contribute to emotional engagement, with studies emphasizing first-person perspectives and dynamic audio for greater empathy. Haptic technology now offers sophisticated tactile feedback, enhancing emotional and empathic experiences in VR. In addition, incorporating real-time emotional cues and facial recognition enhances interactive and emotionally intelligent virtual agents. Regarding cognition, optimizing cognitive load through simplified interfaces and balanced sensory inputs ensures user engagement without overwhelming in recent research^[20,59]. Effective attention management, using visual and auditory cues, is crucial for directing user focus in immersive environments^[27]. Moreover, VR's potential to foster empathy by allowing users to experience different perspectives is well-addressed in recent research^[26,60,87]. VR has transcended traditional human interaction by enabling deeper connections with the animal world, effectively fostering empathy across species and its impact on human perception and cognition together. Recent VR simulation study on stray animals showed how to foster empathy for stray animals and promote awareness and behavioral change toward animal welfare by designing a Virtual Reality Perspective-Taking (VRPT) system^[102]. Finally, neuroimaging studies have provided insights into the brain's response to immersive experiences, informing the design of environments that maximize cognitive and emotional impact^{[42,57,58,103]}.

7.6 Limitations

The systematic review method we adopted limits our study by applying strict inclusion criteria, thereby excluding relevant research that falls outside those criteria. The reliance on PRISMA-based selection criteria has contributed to the dominance of VR studies over AR/MR, potentially overlooking relevant but unpublished or industry-based findings. Furthermore, this systematic review is inherently constrained by its dependence on published literature up to a certain point in time, which may exclude ongoing advancements in immersive empathic computing. Future reviews can incorporate a broader range of sources, including gray literature and preprints, and explore alternative review methodologies that allow for a more dynamic synthesis of emerging trends. The dominance of dimensional emotion models and variability in methodologies across studies points to both a trend and a challenge: while these models provide flexibility, their prevalence may overlook applications that benefit from discrete emotion frameworks. Additionally, publication biases and methodological inconsistencies complicate direct comparisons across studies. Furthermore, the role of perception and cognition in terms of sensory integration, cognitive load management, the interplay among visual, auditory, and haptic modalities, as well as neurocognitive processes have been identified as pivotal factors that influence user engagement and emotional regulation. While significant progress has been made in understanding these processes within VR, future research should extend these insights to AR and MR, investigating areas such as peripheral awareness, implicit emotional processing, and the long-term cognitive effects of immersion.

8. Conclusion

This systematic review offers a comprehensive overview of empathic computing in immersive technologies, highlighting key research trends, evaluating the effectiveness of current approaches, and identifying opportunities for future exploration. Over the last two decades, research has increasingly focused on VR environments, revealing their potential for emotion elicitation, recognition and empathy enhancement. However, we found that the articles selected through PRISMA were mostly in VR rather than AR/MR, and these immersive areas remain insufficiently explored, which limits insights into their unique affordances. Future immersive empathic computing research is strongly encouraged to explore the potential of AR and MR, refining emotion models, and integrating hybrid frameworks for emotion representation. Advancements in multimodal emotion recognition and real-time feedback systems hold promise for more personalized empathic computing experiences. In addition, the correlation between lower body postures and emotion in immersive environments should be rigorously studied to bring out new aspects of body behavior research in this emerging field. This review not only identifies these challenges but also provides actionable insights to guide future research, fulfilling its aim to advance the understanding and application of empathic computing across immersive technologies.

Authors contribution

Jinan UA: Ideation, methodology, investigation, data analysis, data curation, visualization, writing original draft.

Jung S: Ideation, methodology, investigation, writing, review, editing, supervision.

Heidarikohol N: Data analysis, data curation, visualization, writing original draft.

Borst CW, Billinghurst M: Writing, review, editing, supervision.

All authors approved the final version of the manuscript.

Conflicts of interest

Mark Billinghurst is the Editor-in-Chief of Empathic Computing. Other authors declared that there are no conflicts of interest.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable

Availability of data and materials

Not applicable.

Funding

None.

Copyright

© The Author(s) 2025.