1. Introduction

Advances in immersive technologies and artificial intelligence are enabling increasingly naturalistic forms of human–machine interaction. Virtual reality (VR), whether fully or semi-immersive, provides a controlled and low-confounding environment that has become a dynamic platform for education^[1], therapy^[2], training^[3], and social interaction^[4]. The effectiveness of VR experiences relies heavily on the realism and expressiveness of virtual agents, which are embodied characters capable of communicating through speech, gestures, and behavior. Synthetic speech plays a pivotal role in supporting these interactions by enabling scalable and adaptable voice interfaces.

Despite its advantages in control and customization, synthetic speech presents challenges in conveying emotional authenticity. Unlike natural human voices, synthetic voices often lack subtle prosodic features and affective nuances. They may also fail to align with the visual representation of an avatar, which can hinder emotional engagement, particularly in contexts that require empathy, trust, or affective resonance.

These limitations highlight the importance of synthetic speech in interdisciplinary domains such as affective computing and empathic computing. Affective computing emphasizes systems that can recognize and interpret human emotions, while empathic computing focuses on systems capable of responding in ways that mirror human emotion-sharing. Within VR, these paradigms are used not only to detect and adapt to users’ emotional states but also to design agents that elicit meaningful, emotionally appropriate, and socially intelligent responses.

Emotional and expressive speech synthesis refers to the generation of artificial speech that conveys affective states such as joy, sadness, or anger through prosodic features including pitch, rhythm, stress, and intonation^[5]. Unlike traditional text-to-speech (TTS) systems, expressive approaches extend beyond intelligibility and pronunciation to approximate the emotional nuances of human speech, thereby enhancing realism and user engagement in applications such as virtual agents. As highlighted by Pitrelli et al., expressive speech enables a system to “reinforce the message with paralinguistic cues, to communicate moods and other content beyond the text itself,” allowing it to “say the same words differently and appropriately for different situations”^[6].

Early research on expressive synthetic speech focused on understanding and modeling prosody (e.g., intonation, rhythm, stress duration) to achieve expressiveness^[7]. The first expressive TTS systems relied on formant synthesis^[8] and unit selection^[9,10]. However, their limited flexibility, dependence on speech databases, and constraints in computational power restricted both expressiveness and overall speech quality. In recent years, the advent of deep learning has transformed speech synthesis, enabling more natural and expressive outputs. The Tacotron 2 model^[11], based on an end-to-end neural architecture, was a breakthrough that allowed direct mapping from text to speech waveforms without hand-crafted linguistic features or manual alignment rules. The introduction of variational autoencoders, which disentangle content, speaker identity, and style, provided greater control over the synthesis of specific emotions^[12]. More recently, diffusion models have further advanced expressive TTS by offering finer control over prosody^[13]. Mori et al.^[14] proposed that as the similarity of a robot or artificial entity to a human increases, so does our affinity and empathy, but only up to a certain threshold. Beyond this point, if the entity appears almost human yet not entirely convincing, positive feelings shift abruptly to unease or discomfort, a phenomenon known as the “uncanny valley”. When the likeness continues to improve to the point of being indistinguishable from a human, affinity is restored. Although extensively studied in the context of graphics and visual realism, the uncanny valley remains underexplored in the auditory domain. A recent study by Ross Ross et al.^[15] reported a positive correlation between perceived voice realism and approval ratings, with no evidence of negative reactions to highly human-like synthetic voices.

Early theoretical frameworks for affective interaction via voice in VR also informed the development of emotion-aware systems employing synthetic speech. One example is MAUI^[16], a multimodal affective user interface proposed by Lisetti et al. This framework leverages advances in computer vision, prosody recognition, and biofeedback for real-time recognition of affect, integrating multimodal inputs to create a “multimodal perceived user’s state”. MAUI further introduced multicultural animated avatars that users could select or create, designed to mirror facial expressions in real time. Such mirroring provides feedback on users’ emotional states and can be applied to self-awareness or adaptive teaching strategies. Potential applications envisioned by the authors include driver safety, military training, and tele-home care.

Another illustrative example of adaptive intelligent agents is AutoTutor^[17], a system designed to adjust pedagogical strategies based on learners’ emotional states to enhance the learning process. This work emphasizes the intrinsic link between emotions and learning, particularly when students face challenging material that may trigger states such as confusion or frustration, which can eventually transition into more constructive states such as hope or enthusiasm. The study further envisioned the integration of multiple sensor channels into a unified emotion classifier, potentially achieved through sensor fusion techniques. The overarching goal of AutoTutor is to respond intelligently to learners’ emotions, fostering a “virtuous cycle” in which periods of confusion are resolved into productive engagement while avoiding boredom or prolonged frustration.

Only with recent advances in artificial intelligence have systems of this sophistication become feasible and increasingly relevant. Building on current research that integrates synthetic speech, affective experience, and VR, this article seeks to synthesize existing findings and contribute to the field by:

· Mapping the state of the art in VR-based applications of synthetic speech and affective experience.

· Developing a taxonomy and overview of existing research efforts and trends.

· Identifying common applications and methodological approaches.

· Highlighting research gaps and ethical considerations.

2. Method

To meet the research objectives and provide a transparent, rigorous overview of the topic, we conducted a scoping review following Arksey and O’Malley’s framework^[18] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines^[19]. The PRISMA-ScR flow diagram is presented in Figure 1, and the completed checklist is provided in Table S1.

Display Full Size

Figure 1. PRISMA-ScR approach for selecting relevant papers. PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews.

Relevant literature was identified by querying a combination of the terms “synthetic speech,” “virtual reality,” and “affective/emotional response”. Search terms were adapted according to database requirements, and the full list of queries is shown in Table 1. The asterisks (*) used in terms such as affect, emotion, and response denotes wildcard characters, enabling retrieval of variations such as affective and affect. All searches were completed in May 2025.

2.1 Search and selection

We conducted a manual search across six databases: ACM Digital Library, IEEE Xplore, Google Scholar, ScienceDirect, Scopus, and Web of Science. This process yielded 98 relevant publications from leading conferences and journals in the areas of human-computer interaction (HCI), affective computing, virtual agents, and immersive technologies.

In HCI and user experience, key sources included the International Journal of Human-Computer Interaction, the Journal of Computer Assisted Learning, and the ACM International Conference on Intelligent User Interfaces. For affective and multimodal systems, research was drawn from the International Conference on Affective Computing and Intelligent Interaction, as well as journals such as Multimodal Technologies and Interaction and Expert Systems with Applications. Core venues for virtual agents and autonomous systems included the ACM International Conference on Intelligent Virtual Agents and the International Conference on Autonomous Agents and Multiagent Systems. Foundational contributions in virtual reality and immersive graphics were published in the IEEE Transactions on Visualization and Computer Graphics, the IEEE Conference on Virtual Reality and 3D User Interfaces, and the IEEE International Symposium on Mixed and Augmented Reality. Additional contributions came from graphics-focused conferences such as SIGGRAPH Motion, Interaction and Games and journals such as Computers & Graphics and Proceedings of the ACM on Computer Graphics and Interactive Techniques. Speech-specific research was represented by the Annual Conference of the International Speech Communication Association and the ACM Transactions on Applied Perception. Bibliographic information, including author names, abstracts, publication year, and venue, was imported into Zotero. After removing five duplicates, 93 publications remained for screening. Both authors independently reviewed all articles using predefined inclusion and exclusion criteria. Consensus was reached to include 12 studies and exclude 81.

· Inclusion: Studies that examined the use of synthetic speech in virtual reality (fully or semi-immersive, i.e., immersive 3D environments) to elicit and measure users’ affective experiences and emotional responses (e.g., anxiety, comfort, empathy, trust).

· Exclusion: Studies that were not peer reviewed (e.g., doctoral theses, preprints), lacked empirical evaluation, or were works in progress without reported findings.

We subsequently applied forward and backward snowballing to the 12 included publications, which yielded an additional 10 papers that met the inclusion criteria. In total, 22 studies were included in the review. These studies are listed in Table 2 in reverse chronological order, based on their first appearance in a journal or conference.

2.2 Data extraction

Key data were extracted from the included publications regarding relational aspects (Table 3), intrinsic characteristics, and interaction outcomes and impact factors (Table 4). Relevant factors were initially identified through an individual semantic review and subsequently finalized in a joint discussion. Individual items are presented in alphabetical order from left to right, while publications are arranged in reverse chronological order from top to bottom. A filled dot indicates presence, a circle indicates partial presence, and an empty space indicates absence of the respective factor in a given study.

Relational aspects encompass affect and affinity, empathy, engagement, likeability, rapport, trust, and other related dimensions. Intrinsic characteristics include attributes such as believability, expressiveness, realism, and social presence. Interaction outcomes and impact factors primarily concern the observed and intended effects, including behavioral change, emotional response, and persuasive influence.

3. Results

The 22 included publications span the period from 2017 to 2025 (Table 2) and cover a range of application areas, including virtual counseling (Section 3.4), education (Section 3.3), public speaking support (Section 3.2), retail experiences (Section 3.5), and empathy elicitation (Section 3.1). As shown in Figure 2, most studies were initially published at conferences, with an increasing number appearing in journals over time. This trend reflects the growing interdisciplinary interest in the convergence of synthetic speech technologies, embodied virtual agents, and affective computing within immersive environments.

Display Full Size

Figure 2. Number of publications per year and type.

Most studies employed between-subjects experimental designs, with participant sample sizes ranging from 28 to over 1,000. Affective experiences were primarily assessed through self-report measures (e.g., Likert scales evaluating trust, empathy, warmth, presence, eeriness, or affect) and behavioral data (e.g., eye tracking, interaction duration, or task compliance). Physiological measures, such as heart rate and skin conductance, were occasionally included but used less frequently. Synthetic voices were generated using TTS engines with varying degrees of expressiveness, while a few studies implemented custom or controlled emotional voice synthesis pipelines. Table 3 and Table 4 provide a detailed overview of the qualitative and methodological characteristics of these studies, and the following sections further synthesize and categorize the findings.

3.1 Emotional engagement and empathy

A substantial body of research has investigated how synthetic speech, particularly prosody and affective expression, shapes users’ perceptions of virtual agents. Evidence indicates that emotionally congruent speech enhances the perceived naturalness, warmth, and believability of agents, especially when aligned with avatar embodiment and visual realism^[30,35,41]. Related studies have explored how agent personality, conveyed through synthetic speech, influences perceptions of trust, sympathy, and social presence^{[25,28,33,39,40]}.

Cabral et al.^[41] examined how the quality of synthetic voices affects evaluations of virtual characters. Participants judged characters speaking with high-quality, emotionally expressive synthetic voices as more appealing and believable than those using neutral or flat voices, highlighting the role of vocal prosody in fostering social connection with digital agents. Ehret et al.^[30] systematically manipulated both prosody and embodiment in a conversational agent to assess their effects on perceived naturalness. They found that emotionally rich prosody significantly increased the perceived naturalness and human-likeness of the agent’s speech, particularly when paired with a humanlike visual form. This emphasizes the importance of multimodal affective congruence in eliciting empathy and engagement. Bargum et al.^[35] investigated the impact of AI-generated speech on avatar perception in a VR environment. Their results showed that expressively modulated synthetic speech enhanced agents’ lifelikeness and emotional engagement, especially when facial expressions and lip movements were well-synchronized. These findings highlight the need for alignment between vocal expressiveness and visual behavior in emotionally credible agents.

Chang et al.^[25] demonstrated that synthetic speech emotionally aligned with an agent’s facial expressions fosters stronger rapport and greater emotional coherence, whereas mismatched affective cues disrupt the sense of natural interaction and reduce engagement. Similarly, Higgins et al.^[28] found that sympathy conveyed via voice and facial behavior enhanced users’ emotional attunement. Eye-tracking data revealed heightened attention to agents whose vocal expressions matched visual affect, reinforcing the role of vocal prosody in empathic perception. Gao et al.^[40] argued that trust in stylized agents depends more on emotional congruence than on voice realism. Agents using affectively appropriate synthetic voices were perceived as more trustworthy and competent, even when their appearance was highly stylized or abstract. Finally, the two studies by Zibrek et al.^[33,39] provide a nuanced perspective on voice-visual interactions. Their first study indicated that the appeal of photorealistic avatars depended on congruent vocal delivery, while the second showed that synthetic voices lacking emotional depth or alignment undermined the credibility and naturalness of realistic characters. Collectively, these findings highlight that both vocal realism and emotional expression are crucial for maintaining social believability in high-fidelity avatars.

3.2 Affective regulation

Several studies have examined how virtual agents using synthetic speech can support individuals in emotionally challenging situations, such as public speaking or mental health counseling. These studies indicate that affectively congruent synthetic speech can reduce anxiety, enhance self-confidence, and increase user comfort^{[26,31,32,36,38]}.

Murali et al.^[31] introduced Friendly Face, a virtual agent designed to reduce public speaking anxiety by acting as a supportive audience member. The agent senses the speaker’s behavior, including motion, speech, and prosody, and provides both emotional and instrumental support. Empirical evaluation against a text-only system showed that Friendly Face decreased public speaking anxiety, increased speaker confidence, and improved overall satisfaction. The approach of using a virtual agent as a calming and supportive audience member was well received and demonstrated considerable potential to enhance the public speaking experience. Similarly, Kimani et al.^[38] developed You’ll Be Great, a virtual coaching system aimed at reducing public speaking anxiety through automated Cognitive Behavioral Therapy (CBT) with a focus on cognitive restructuring. The system significantly reduced maladaptive thoughts, alleviated cognitive and somatic anxiety, and improved participants’ presentation experience. Users reported feeling a strong connection and understanding from the CBT agent and expressed willingness to continue using it for future presentations. Another line of research investigated affective regulation in health counseling contexts. Parmar et al.^[36] examined how animation quality, speech quality, and rendering style influence user perceptions and persuasive outcomes. They hypothesized that agents with consistent quality across animation, voice, and rendering would have the greatest impact, rather than a simple additive effect of high quality in each channel. The study found that while natural animations and recorded human voices generally enhance acceptance, trust, and credibility, they do not necessarily maximize persuasion. For tasks such as health counseling, minimal agent animation (using a static character) may be more effective for persuasive outcomes. This challenges the initial hypothesis and highlights the complex interplay between agent design and task goals. In a follow-up study, Parmar et al.^[26] explored simulated empathy as a design variable and its interaction with realism factors. High and consistent levels of empathy improved agent perception, but excessively high empathy could interfere with the formation of a trusting bond. In the mental health domain, Maxim et al.^[32] investigated how manipulating a virtual human’s vocal properties to convey different levels of extroversion affects user perceptions, rapport, and persuasion. They found that matching a virtual human’s vocal extroversion to the user’s personality does not reliably produce a similarity-attraction effect or improve persuasion. Instead, virtual humans exhibiting low extroversion traits, such as slower speech, lower pitch, and reduced volume, were generally more persuasive for promoting mental wellness, independent of the user’s personality. The study also emphasized that the user’s own personality plays a significant role in shaping perceptions and interactions with virtual agents.

3.3 Educational and cognitive engagement

Another prominent area of research concerns the role of synthetic voices in VR in supporting learning, cognitive engagement, and attention. Studies in educational and collaborative learning contexts have shown that expressive or emotionally stable voices can enhance student engagement, retention, and motivation during tasks^{[20,21,24,29]}.

Dai et al.^[21] investigated the use of TTS agents in VR primary school classrooms, comparing emotionally expressive and neutral voices. The study found that emotionally congruent synthetic speech significantly increased students’ enjoyment, attentiveness, and task performance, suggesting that affectively rich voices can enhance engagement and comprehension in immersive educational settings. Abdulrahman et al.^[29] evaluated the impact of natural versus synthetic speech in VR-based English language learning. Learners reported higher motivation and perceived helpfulness when guided by a synthetic agent with more natural-sounding prosody, providing further evidence that voice realism can facilitate second-language acquisition in immersive environments. Guo et al.^[20] explored the use of self-similar virtual agents in a puzzle-solving task, manipulating both voice and appearance to match participants’ gender and ethnicity. Agents with synthetic voices resembling the user led to increased perceived collaboration and social cohesion, suggesting that voice personalization can foster affective alignment and improve group performance in VR teamwork scenarios. Choi et al.^[24] examined team performance dynamics in a collaborative VR game, focusing on how the emotional stability of agents’ voices affected users’ experience. Agents with emotionally stable synthetic speech were perceived as more competent and reassuring, resulting in better coordination and higher player confidence. These findings highlight the importance of voice tone in shaping cooperative dynamics within educational simulations.

3.4 Persuasion, empathy, and bias reduction

Three studies have examined the impact of synthetic speech on persuasive effectiveness, interpersonal empathy, and bias-related responses. These studies focused on agents designed to reduce implicit or explicit racial bias, promote empathic reasoning, and persuade users to modify their attitudes or decisions^[23,27,37].

Obremski et al.^[37]investigated a VR intervention using Intelligent Virtual Agents (IVAs) to reduce implicit and explicit racial bias. The scenario involved a pub quiz in which a mixed-cultural IVA acted as a supportive teammate and positive out-group exemplar, while a mono-cultural IVA served as a rude, uncooperative in-group exemplar. Interaction with the virtual agents resulted in significant decreases in both implicit and explicit racial bias scores among participants. Llanes et al.^[23] developed virtual human (VH) characters designed to elicit social emotions through conversation. The system combined realistic avatars with a large language model (LLM) to generate contextually relevant responses, incorporating personality, mood, and attitudes. The VHs exhibited synchronized lip movements, voice synthesis, and facial expressions to reflect the intended emotion. In a study with 64 participants interacting with six different VHs, each targeting a specific basic emotion, the system successfully generated the intended emotional valence. Participants rated the conversations as relatively natural and realistic, although arousal was less effectively evoked, it was still detectable in the VHs. Do et al.^[27] examined how speech fidelity and listener gender influence social perceptions of a virtual human speaker. They compared standard concatenative TTS, advanced neural TTS, and recorded human speech. Results showed that neural TTS led to the VH being perceived as significantly less trustworthy by both male and female listeners compared with human speech. Male listeners also judged standard TTS as less trustworthy than human speech. These findings suggest that neural TTS may not be optimal for persuasive virtual humans and that listener gender plays a role in shaping perceptions of synthetic speech.

3.5 Commercial and aesthetic appeal

Several studies^[22,34] have investigated how synthetic voice characteristics, including tone, pacing, and affect, influence consumer trust, purchase intent, and perceived agent professionalism in commercial applications such as VR shopping. Zhu et al.^[22] examined the effects of speech modality and avatar realism on user perceptions in a VR shopping environment. Their findings indicated that integrating synthetic speech with highly anthropomorphic avatars enhanced perceived realism, trust, and shopping intent. The study highlights the importance of aligning voice characteristics with visual design to improve the consumer experience in VR retail. Morotti et al.^[34] studied voice assistants in fashion retail VR, demonstrating that affective tone and prosody influenced perceptions of professionalism and brand coherence. Expressive voices that matched the store’s aesthetic increased user satisfaction and contributed to a more immersive shopping experience.

4. Discussion

As with any scoping review, this study is limited by the selection criteria and scope of the included publications. Some relevant work may have been excluded due to language, indexing, or publication type. Nevertheless, the retained studies indicate a growing interest in using emotionally expressive synthetic voices to influence user perception and behavior in immersive environments. Synthetic speech was found to modulate core affective outcomes, including empathy, trust, discomfort, and anxiety regulation, across diverse applications ranging from emotional tutoring and social coaching to mental health support and retail interactions.

A key theme emerging from the studies is that prosody and emotional congruence between a synthetic voice and a virtual agent’s visual or behavioral design strongly shape users’ affective responses. Monotone or generic TTS voices often hinder emotional connection, whereas emotionally expressive synthetic voices, particularly when aligned with an agent’s visual embodiment, enhance engagement, empathy perception, and social presence. These effects were particularly pronounced in healthcare and counseling contexts, where affective alignment between voice and visual cues is critical for establishing trust and rapport. For example, Chang et al.^[25] demonstrated that aligning affective vocal tone with agent behavior substantially improves perceptions of naturalness and emotional consistency. Similarly, Parmar et al.^[26] found that agents using emotionally appropriate synthetic speech elicited higher empathy and interpersonal warmth ratings. Studies employing synthetic voices to reduce anxiety or support emotional regulation also reported positive outcomes. Murali et al.^[31] showed that a friendly synthetic voice in a supportive VR agent helped participants manage public speaking anxiety, enhancing both confidence and calmness. Obremski et al.^[37] similarly reported that interaction with emotionally aware virtual agents increased participants’ emotional openness.

However, the benefits of synthetic speech are not universal. Several studies examining trust and social presence found that flat or unnatural prosody undermines perceptions of authenticity and credibility. Some of these effects can be mitigated through affective tuning or multimodal alignment. Gao et al.^[40] found that stylized, emotionally congruent voices improved trust in stylized agents, even when speech was not fully human-like. Conversely, incongruent or low-fidelity synthetic voices can induce discomfort, particularly when paired with photorealistic avatars, supporting the uncanny valley hypothesis in vocal perception. Zibrek et al.^[33] and Higgins et al.^[28] demonstrated that mismatches between high visual realism and low vocal expressiveness increased user discomfort and reduced believability. This evidence highlights that incongruent multimodal cues, especially in emotionally salient contexts, can disrupt affective resonance. As noted in Section 1, there is a trade-off between fidelity and perceived pleasantness^[15]. Moreover, even advanced expressive TTS models often struggle to replicate the subtle affective nuances of natural speech, limiting their effectiveness in emotionally sensitive scenarios.

4.1 Ethical considerations

High-fidelity voice replication has become increasingly feasible thanks to large-scale neural TTS systems trained on thousands of hours of speech^[42]. As a result, virtual agents with high-quality synthetic voices are becoming more widespread, raising important ethical questions regarding their impact on users.

One key area of concern is trust and user agency. LLM-based agents that communicate through speech are often anthropomorphized and perceived as more trustworthy than non-verbal agents^[43]. Szekely et al. caution that increasing the realism, expressivity, and human-likeness of synthetic voices can have socio-affective effects, potentially influencing user perceptions and behavior^[44]. Emotional proximity with highly anthropomorphic agents may impair human judgment and affect willingness to follow their advice^[45]. Dubiel et al.^[46] provide an example of this influence, showing that expressive synthetic speech can subtly steer users’ choices, thereby undermining their agency.

Another area of concern involves relationships and well-being. Research indicates that users often interact with highly anthropomorphic AI companions as if in human-like relationships^[47], which can lead to dependence and emotional attachment^[48]. The long-term consequences of such interactions remain unclear. In a large-scale study (n = 981), Phang et al.^[49] investigated the impact of ChatGPT’s Advanced Voice Mode (AVM) on users’ emotional well-being and reported mixed outcomes. While AVM showed short-term benefits, prolonged use was associated with increased loneliness, greater emotional dependence, and compulsive usage. In response, Raedler et al.^[50] proposed actionable strategies to mitigate potential harm from AI companions, including encouraging real-world socialization, setting usage limits, and screening users for risk factors.

Given these potential risks, the use of expressive, human-like voices in virtual reality warrants careful investigation and broader ethical discussion. This is particularly important in immersive environments, where the perceived realism and emotional engagement of synthetic voices may amplify their impact on users.

4.2 Integration with affective and empathic computing

The findings highlight the central role of affective computing as a foundation for emotionally intelligent interactions. In this framework, synthetic speech functions both as a channel for emotional expression and as a cue for emotional inference. The reviewed studies indicate that affective congruence in voice is essential for empathic computing, enabling systems to not only detect user emotions but also respond in socially intelligent and emotionally appropriate ways. Consequently, voice serves not merely as a mode of interaction but as a critical component for eliciting perceived empathy in artificial agents.

4.3 Limitations of the Literature

Despite promising findings, the reviewed literature exhibits several limitations. First, most studies focused on short-term emotional responses, providing limited insight into the longitudinal effects of synthetic voices on user emotion or rapport. Second, sample populations were often restricted to university students or online participants, raising questions about the cultural and age-related generalizability of the results. Third, the heterogeneity of measurement approaches, ranging from single-item Likert scales to biometric data, complicates cross-study comparisons. Finally, many studies relied on off-the-shelf TTS engines with limited emotional control. Only a few employed state-of-the-art expressive TTS systems, such as emotion-conditioned models, while most used commercial platforms that constrain the granularity of experimental manipulations.

4.4 Implications for design and research

Designing emotionally credible virtual agents requires treating voice, appearance, and behavior as interconnected affective cues. Misalignment among these cues can undermine trust and perceived realism, whereas affective congruence enhances emotional credibility and social presence. Future research should focus on multimodal affect modeling, real-time adaptive voice synthesis, and cross-cultural validation to develop agents that are not only functional but also emotionally responsive and capable of supporting empathic computing. Additionally, transitioning from fully controlled VR environments to augmented reality (AR), which involves background noise, unpredictable acoustics, and dynamic user contexts, presents challenges for evaluating synthetic speech technologies. This shift necessitates careful examination of agent performance in less controlled and more ecologically valid conditions.

5. Conclusion

This scoping review demonstrates that although research on synthetic speech in VR is growing, its application for emotional and affective purposes remains underdeveloped. Existing studies indicate that features such as prosody, affective congruence, and voice-avatar alignment can significantly influence perceptions of realism, trust, empathy, and user engagement across diverse domains, including education, mental health, collaboration, retail, and social bias mitigation.

The review provides a structured synthesis of 22 peer-reviewed studies, organizing the literature around key affective outcomes and thematic application areas. It identifies patterns such as the importance of emotional alignment across modalities (e.g., voice and embodiment), the role of affectively expressive speech in enhancing learning and persuasion, and the potential of synthetic voices to shape social dynamics in immersive contexts. Nevertheless, the field remains fragmented, with considerable methodological variation and limited consensus on evaluating affective responses or comparing results across studies.

Notably, many studies rely primarily on subjective measures, with few incorporating physiological or behavioral data. Comparisons between synthetic and natural speech are rare, and the affective qualities of voices are often under described or inconsistently manipulated. Moreover, synthetic voice design seldom considers cultural, linguistic, or individual differences, despite evidence that perceived similarity and contextual appropriateness can influence emotional outcomes.

By identifying these trends and limitations, this review highlights both the potential and current boundaries of affective synthetic speech in VR, providing guidance for future research aimed at advancing synthetic speech technologies and enhancing affective experience in immersive environments for empathic computing.

Supplementary materials

The supplementary material for this article is available at: Supplementary materials.

Authors contribution

All authors made equal contributions to this work.

Conflicts of interest

Jean Botev is an Editorial Member of Empathic Computing. Another author has no conflicts of interest to declare.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Not applicable.

Funding

None.

Copyright

©The Author(s) 2025.