1. Introduction

As robotics has prospered in recent years, intelligent robots have been more integrated into everyday lives^[1]. In particular, human–robot interaction (HRI) has become a crucial field by intertwining humans with robots in human-centric environments^[2]. However, HRI has traditionally been rooted in the constraints of the physical world, prioritizing mechanical safety, real-time control, and spatial coordination between humans and embodied machines. These systems have yielded significant advances in domains such as manufacturing, logistics, and assistive robotics^[3-6], yet they remain bounded by several practical limitations, such as the cost of hardware, the risks of human–machine co-location, and the inflexibility of physical deployment in rapidly dynamic or hazardous environments^[7,8].

Meanwhile, extended reality (XR), an umbrella term encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR), has matured into a powerful paradigm for immersive interactions, multimodal interaction, and elevated cognition in the field of general human-computer interaction (HCI) over the past decades^[9]. By incorporating virtual information visualized either in a complete virtual world or the real-world, XR offers not only visual-spatial fidelity but also the capacity for dynamic feedback, co-presence, and more engaged interaction in many domains^[10,11]. There has been massive research that has delved into employing XR techniques to facilitate advanced HRI by bringing virtual setting, which makes it uniquely well-situated to complement and extend the principles of HRI beyond the physical workspace. Figure 1 provides representative examples of XR-native virtual robots spanning safety-oriented rehearsal/legibility, remote operation, and socially oriented humanoid interaction, motivating our later scenario framing rather than serving as a taxonomy.

Display Full Size

Figure 1. Representative examples of interacting with XR-based virtual robots across different domains, illustrating the breadth of mechanisms that motivate this paper’s focus. (a) Virtual robot arm in VR enabling safe task execution without physical co-location^[12]; (b) Virtual mobile robot in VR with motion details visualized to improve situational awareness^[13]; (c) Virtual humanoid robot in VR supporting social skill learning^[14]; (d) Virtual aerial robot (drone) in VR for teleoperation and training in environments where physical deployment is impractical^[15]; (e) Virtual medical robot in AR/MR where the visualization supports context-aware guidance and safer clinical workflow understanding^[16];(f) Virtual mobile robot in the format of AR/MR visualization to support shared scene coordination and spatial understanding^[17]. XR: extended reality; VR: virtual reality; AR: augmented reality; MR: mixed reality.

In parallel, we are witnessing an unprecedented shift in artificial intelligence (AI) through the development of large foundation models (FMs)^[18], particularly large language models (LLMs), large vision models (LVMs), and vision–language models (VLMs), that endow agents with context-aware reasoning, conversational competence, and multimodal perception^[19-21]. When integrated with HRI, these models enable the creation of more smooth interactions between humans and robots due to algorithmic advancements, for example, in communicating robot intent or learning from human feedback^[22,23]. When coupled with XR, the involvement of virtual elements would lead to another potential approach for interaction^[24]. For instance, intelligent virtual robots, which are agents that inhabit immersive spaces, can understand user intent and adaptively respond through natural language, gaze, gesture, or other inputs^[25-29].

The virtual robots are more than plain visualizations or surrogates^[16,30] of corresponding physical systems. Besides serving as virtual counterparts, they also represent a new category of cognitive agent capable of participating in collaborative tasks, simulating empathic engagement, and supporting experiential learning without the material constraints of conventional robotics. In industrial training scenarios, for example, XR-based robots have been shown to improve domain task performance, user experience, and procedural safety awareness^[31,32]. In specific therapeutic and educational contexts, socially responsive virtual agents powered by LLMs are already being explored as scalable tools for mental health interventions, language acquisition, and neurodiverse engagement^[33,34]. This evolution of immersive media technology and cognitive AI, alongside the intersection between them, encourages us to reflect: while the traditional robotic paradigm emphasized physical manipulation, like grasping, moving, and executing, more emerging paradigms emphasize co-presence and the co-construction of tasks through immersive interactions^[35,36] with the engagement of virtual elements^[37-39]. This shift reframes HRI from a primarily mechanical interface to a socio-cognitive encounter mediated by technology.

By witnessing the current landscape of interacting with virtual robots through the lens of XR-generated autonomous and adaptive agents that reside entirely within immersive environments, we argue that in the future, such virtual agentic loops will offer not only practical benefits—safety, scalability, and cognitive adaptability—but also open new directions for understanding human–machine interaction as a fundamentally relational process. We explore how humans engage with these virtual robots as social, instructional, and empathic partners across diverse domains, such as industrial safety training, education, therapeutic interventions, collaborative design, etc. By narrowing the scope to XR-native agents, we are able to examine the unique affordances and challenges inherent in this paradigm: embodiment without physical form, presence without mass, and cognition unconstrained by mechanical limitations. This framing also enables us to explore how emerging AI models (e.g., LLMs, VLMs) can endow these agents with contextual awareness, responsiveness, and the capacity for long-term adaptation. By synthesizing developments across XR, HRI, and large-scale FMs, we articulate a research agenda for the next generation of empathic and ethically grounded virtual robotics.

1.1 Positioning

This paper investigates the interaction between humans and intelligent virtual robots that exist entirely within XR environments. Unlike conventional literature reviews such as work^[40] identifying the methodologies of XR in HRI, or work^[41] discerning the interaction modalities for XR in HRI, in this paper, we position virtual robots as embodied, agentic entities, either humanoid or non-humanoid in form, rendered in immersive digital environments, and argue that these virtual robots can function with perception, reasoning, and action capabilities via AI. These agents are native to XR: they operate independently of physical actuators or hardware proxies and are designed for real-time, situated interaction within virtual contexts. Our focus extends beyond traditional robotic paradigms that employ XR techniques as visualization tools or user interface overlays. Apart from physical robots augmented by XR for specific task-oriented teleoperation or collaborative support, we concentrate on fully virtual robots capable of functioning as standalone agents with roles, such as instructors, collaborators, companions, or cognitive interfaces in immersive, spatial environments actuated by intelligent large FMs. A conceptual diagram of XR-empowered human-virtual robot interaction with FMs is illustrated in Figure 2: we treat the FM as the cognitive core that binds multimodal perception, contextual reasoning, and affect-sensitive response generation to the embodied XR interaction loops. Our paper is the first Perspective that argues virtual robots powered by XR and large FMs to be not only auxiliary but to be expected to become more intelligent agents.

Display Full Size

Figure 2. Conceptual interaction framework for XR-native virtual robots powered by multimodal FMs. User inputs in XR (e.g., speech, gesture, and contextual cues) are interpreted by an FM engine (e.g., LLM/VLM) for task-aware understanding, instruction parsing, and response generation; the XR-native virtual robots deliver real-time feedback and embodied interaction in the immersive environment. The right-hand block indicates optional but practical real-world integration: the same FM-driven policies or interaction strategies can be mapped onto physical robotics scenarios (e.g., via teleoperation). XR: extended reality; FM: foundation model; LLM: large language model; VLM: vision–language model.

1.2 Terminology disambiguation

To reduce ambiguity, we use the following terms consistently throughout the paper. We use robot to denote an embodied robotic system in the physical world (i.e., with sensors/actuators), and agent as a general term for an entity that perceives, reasons, and acts within an environment. virtual robot (our focus): a robot-like virtual agent that exists entirely within XR and is not dependent on a physical robot for its embodiment or behavior (although it may optionally be informed by external data). When referring to this entity, we primarily use virtual robot (or XR-native/XR-powered/XR-based virtual robot or similar) and avoid using robot alone where confusion could arise.

1.3 How does HRI with XR-native virtual robots differ from physical robot

Traditional HRI assumes a physically embodied robot whose interaction is constrained by mass, force, sensing placement, actuation limits, and co-location risks; as a result, much of HRI centers on mechanical safety, real-time control, and physical task feasibility. In contrast, XR-native virtual robots are embodied through perceptual and interaction cues rather than material form, and their action space is programmable rather than physically limited. This could result in a significant transition by unleashing the interaction paradigm from physical mass constraints. Table 1 summarizes the main differences between the two interaction formats and motivates why XR-powered virtual agents can enable different forms of HRI rather than merely simulating physical robots.

2. Current Advancements of HRI with XR and FMs

The intersection of XR and HRI has witnessed rapid growth in recent years, driven by advances in immersive technologies, intelligent agent architectures, and the democratization of AI models. Across both industrial and academic settings, XR platforms are increasingly being used to simulate robot behavior, mediate collaboration, and enable safer, more personalized forms of interaction. In this section, we examine recent related works to provide an argument-driven synthesis to motivate our Perspective’s central focus: XR-native virtual robots, when coupled with FMs, can unlock interactions that are (i) safer, (ii) smarter, and (iii) more empathic and emotionally intelligent. Accordingly, we discuss representative prior works by: (1) Safety-Aware XR for Human–Robot Collaboration; and (2) Cognitive and Empathic Virtual Robots: Large FMs with XR with specific sub-domains aligned with our pivotal concerns.

2.1 Safety-aware XR for human–robot collaboration

The use of XR (mostly VR and AR) has gained considerable traction in enabling safety-aware human–robot collaboration (HRC), especially in industrial, high-risk, or mission-critical domains. XR systems enable the development of virtual elements or digital twins of physical robots that support users in visualizing robot behaviors, simulating hazardous tasks, and reducing physical risks during planning and training phases.

2.1.1 Visualizing robot behavior for hazardousness

Multiple studies emphasize the power of XR to make robot behavior more transparent and predictable through visual overlays and real-time cues. Enayati et al.^[32] propose a human-in-the-loop XR system to visualize robot trajectories and operational zones. Similarly, MR interfaces evaluated by study^[42] use audio–visual cues to render dynamic hazard zones, increasing user awareness. Works^[30,43,44] explore AR projections or visualizations of robot motion intent or predicted paths, improving perceived safety, interaction naturalness, and reducing ambiguity during close-range collaborations with robots. Digital twins are increasingly used to visualize risk and monitor remote systems. As shown in works^[45-47], synchronizing virtual and physical robots allows users to anticipate unsafe interactions and maintain awareness during shared-space operations. There are several literature reviews^[40,48,49] which further highlight the role of digital twins and virtual agents in visualizing context and enhancing transparency.

2.1.2 Training and planning with simulated XR

XR-based training tools have demonstrated strong potential in improving procedural safety and knowledge retention in HRI^[50-52]. Dianatfar et al.^[31] show how VR welding simulations lead to increased user confidence and task proficiency. Similar systems^[53] offer immersive environments where users engage with simplified navigation alongside robot behavior. Surve et al.^[54] enabled HRI training to be conducted in photorealistic simulated environments, using XR wearables with virtual underwater robots before deployment in real-world scenarios where safety could be at risk. Virtual laboratories, allowing for reproducible and controlled studies of HRC under complex psychological and ergonomic conditions^[55,56], were developed for enhanced interaction safety. For example, study^[57] examines how robot motion affects human posture and stress using VR-based observation, while another^[58] combined AR with haptic feedback to reduce cognitive load during robot-assisted surgical situations. For task planning, some systems help reduce planning errors and allow iterative testing without physical execution, such as study^[59] which provided AR interfaces to preview and adjust robotic motion paths, or study^[60] which offered a virtual robot in AR for high-level requests before accurate motion planning. For industrial assembly contexts, AR-based tools assist in human-robot alignment and workflow optimization^[61,62] for improved task efficacy.

2.1.3 Remote and distributed safety-aware HRI

Remote interaction with robots or robotic teleoperation through XR is also gaining prominence^[63-66]. Previous studies^[67,68] combine XR and networked control for distributed robot supervision. Through immersive visualizations and overlays from VR and AR, users retain situational awareness and control without excessive direct physical presence in adaptive teleoperation, where intuitive visual, audio, or mid-air feedback^[69,70] is accompanied.

2.1.4 Adaptive, accessible, and inclusive XR-HRI

Despite broad applicability, many XR-HRI systems still lack adaptability or emotional responsiveness. Some reviews^[40,48] highlighted that most implementations are tailored to single-user setups or rigid scenarios. Recent works have begun to address these gaps through individualized or personalized safety concerns and adaptive communication modeling^[71,72]. XR is also proving to be a low-cost, accessible means of robot interaction. Mobile smartphone-based AR interfaces^[73] enable seamless robot manipulation in unconstrained settings. Wearable AR provides hands-free guidance, visual feedback, and improved interaction with higher effectiveness in dynamic environments^[74,75].

In summary, XR-based robot systems using additional visualized virtual elements contribute significantly to safety-aware HRI/HRC by enabling informative visualizations, safe training environments, and more intuitive interactions. Moving forward, research should continue to explore adaptive, intelligent, and inclusive XR interfaces to fully realize the potential of virtual robotic collaboration in complex and dynamic settings.

2.2 Cognitive and empathic virtual robots: large FMs with XR

The rise of large FMs during past years is enabling a new generation of cognitive and empathic norms of both advanced robotic and HRI loops. No longer limited to passive interfaces or scripted responses, virtual agents embedded in XR environments can now engage in contextual reasoning, adapt behavior in real time, and respond empathetically to users due to generative AI.

2.2.1 FMs as cognitive engines in XR-HRI

Numerous recently developed platforms and frameworks integrate LLMs into XR to support spatially and semantically grounded communications with agents. CUIfy^[76] demonstrates an open-source pipeline for embedding LLM-powered conversational agents in Unity XR environments, minimizing latencies during interactions. Konenkov et al.^[77] combine VR with VLMs, enabling reduction in task completion time and elevation in user comfort. Afzal et al.^[78] highlighted that LLMs with XR facilitate situational awareness while underlining ethical considerations. Li et al.^[79] showed that foundational VLMs can make robotic control more flexible while being a cost-effective and easy-to-use solution. Synthetically, study^[80] explored a GPT-based human-robot teaching through VR settings, which showed the functionality of simulated virtual robots powered by FMs.

2.2.2 Empathic and emotionally aware virtual robots

Empathy-focused HRI is a growing priority in XR systems. The review^[81] detailed how LLMs facilitate emotion recognition, conversational nuance, and contextual understanding in HRI. Social cues, such as haptics are also simulated in XR. For example, a study^[82] presented a haptic-enhanced VR handshake system with visual cues to enhance social presence for robotic partnership. Another study^[55] enabled empathetic testing of prosthetic interaction through a safe virtual laboratory. Understanding user stress, posture, and motion in XR has also been studied for adaptive and empathic robot control and interaction^[56,57]. Notably, some AR-related works in HRI are now also augmented with empathy-aware overlays. For example, Wang et al. explored user preferences of embodied virtual agents in AR with different design propositions^[83]. Systems from studies^[43,44] enhanced user comfort, trust, and efficiency by previewing robot motion intent through AR. Empathic engagement in high-risk environments can be further enabled by XR agents that visualize shared intentions^[49,84]. So far, many works have already investigated the utility of large FMs in robotic empathy. For instance, Lee et al.^[85] integrated LLM-driven generation of structured empathic non-verbal cues (e.g., speech prosody, gestures, and facial expressions) to augment social robots’ empathic capacities. Beyond cue generation, Darwesh et al.^[86] systematically examined how robotic embodiment and an LLM’s empathetic tone influence empathy elicitation during interaction, highlighting practical gaps between simulated empathic language and users’ prosocial responses. More recently, Chen et al.^[87] introduced EmpathyAgent, a large-scale multimodal benchmark and evaluation suite to assess whether embodied agents can plan and execute empathic actions across diverse scenarios, positioning empathy as a measurable capability for embodied FMs.

2.2.3 Human-centered and multimodal interaction

To support human-centered core, while retaining trustworthy and intuitive virtual agents in XR-based HRI, several works have proposed distinct taxonomies and design principles. These include the virtual element design taxonomy^[88], the dedicated AR-HRI classification^[84], and the key characteristics of robots^[89], which help researchers build user-centered, expressive, and empathic agentic interactions. Design-oriented XR systems^[90-93] explore proxemics, social communication, and information exchange, facilitating emotionally intelligent design of HRI systems. In terms of interaction modalities, gesture and non-verbal input are critical components of empathic HRI. Gesture recognition in VR^[94] and multimodal input^[95] via wearable MR^[25] enable virtual robots to interpret user intent and adjust behavior dynamically.

In summary, human-centered, cognitive, and empathic virtual robots especially powered by FMs are transforming XR into a shared-platform for more adaptive and emotionally aware HRI. For instance, through LLM integration, multimodal feedback, and empathic design, virtual agents now serve not only as tools, but as collaborators capable of sensing and understanding complex human needs.

3. Future Scenarios of XR-Enhanced Virtual Robot Interaction

By revisiting the current research and application status of XR-based HRI, as well as the surge of large foundational models, the next generation of XR-enabled virtual robots will go beyond mere simulation and visualization^[96]. These virtual robots are envisioned to be developed with agentic empathy^[97] to become different roles, such as adaptive collaborators and companions capable of reasoning, teaching, and responding to user needs across a range of real-world tasks. We outline here three forward-looking domains where virtual robots could deliver transformative impact, while the three corresponding scenarios are shown in Figure 3.

Display Full Size

Figure 3. Future scenes with XR-powered virtual robots. XR: extended reality.

3.1 Enhancing industrial collaboration in hazardous environments

Industrial domains such as manufacturing, construction, biochemical processing, and emergency cases frequently involve hazardous conditions that challenge conventional human–robot co-presence^[37,98]. Safety concerns, limited visibility, high-risk machinery, and unstable environments often preclude direct interaction with physical robots. XR technologies offer a transformative alternative: by simulating immersive digital twins of real-world operations, workers can engage with virtual counterparts of industrial robots to rehearse procedures, calibrate workflows, and develop collaborative pipelines with mitigating physical risks.

Within these environments, users can interactively visualize safety-critical zones, explore dynamic task trajectories, and manipulate virtual robots with high fidelity to their real-world equivalents^[99]. VR is particularly effective for simulating chaotic or unpredictable scenarios where physical testing is impractical. In such settings, virtual robots that are programmed with behaviors modeled after their physical counterparts can provide users with realistic and repeatable task rehearsals that improve coordination and situational awareness^[100,101].

AR or MR expand these capabilities into real-time field operations^[102]. For example, through optical see-through head-mounted displays, users can view superimposed virtual robots aligned with their physical environments, enabling spatially accurate remote teleoperation^[103-105]. This is especially valuable in mobile robotics deployed in the onsite hazardousness-aware situations, where AR overlays allow for user interaction and monitoring at a safe distance, while maintaining functional continuity with on-site deployments.

By integrating real-time sensory feedback and AI-driven reasoning (e.g., via LLMs), these virtual agents adaptively interpret human intent and respond to dynamic conditions. More than functional simulators, they serve as communicative intermediaries for visualizing robot intent, forecasting motion paths, and modeling shared control frameworks. Such transparency builds user trust, reduces cognitive load, and fosters safer, more effective collaboration during both training and live operations^[106-108]. As such, XR-enabled virtual robots act as critical mediators in hazardous environments, bridging the gap between simulation and safer deployment.

Illustrative vignette: FM-based virtual robot assistance in hazardous environment with XR. Consider a hazardous maintenance task (e.g., in welding or chemical processing) where standing next to an operating robot is highly risky. However, in VR, the team is able to rehearse the entire process in a virtually identical workspace, where the robot’s planned motions and prohibited areas are visualized clearly to help operators understand where it is safe to stand and how the robot will move^{[30,32,42,43,45,47]}. When an FM is integrated into the pipeline, it can anticipate the virtual robot’s next actions and explain the reasons behind them (e.g., why it slows down or pauses), answer “what happens if I do this” questions during rehearsal, and proactively highlight risky situations with timely safety reminders (e.g., if the operator is too close to the hazard)^[76,78]. Overall, the virtual setup reduces exposure during training and makes robot behavior easier to anticipate during real-world deployment.

3.2 Empowering socially and cognitively empathic interaction via virtual humanoid robots

In social, healthcare, and educational contexts, humanoid robots are increasingly deployed to support therapy, rehabilitation, companionship, and interactive learning^[109-112]. However in numerous cases, physical platforms remain expensive, difficult to adapt, and are often limited in their range of emotional expression. Virtual humanoid robots in XR alleviate many of these constraints by existing as fully programmable, reconfigurable avatars that can be tailored to individual users and scenarios. Within immersive environments, these agents can inhabit shared spaces with users, maintain functionality, modulate input signals such as voice and posture, and enact subtle social cues, practically achieving a richer empathic interaction than many current physical embodiments^[113].

The integration of large FMs further amplifies these capabilities. By combining LLMs, LVMs, and VLMs, virtual agents can interpret multimodal input such as speech, gaze, audios, facial expression, and gesture, and respond in a manner that is contextually appropriate and emotionally attuned. In VR, fully immersive environments populated with such agents can be used to rehearse or remaster challenging social situations, practice turn-taking with more empathy^[114,115]. The game-like learning experiences in VR with virtual robots can reward mutual collaboration and facilitate curiosity, improving overall learning efficiency in educational cases^[116]. In AR, virtual companionship robots can be overlaid onto ordinary environments, functioning as “always-available” conversational partners who accompany users in need through mentally therapeutic exercises^[117]. By enabling empathic conversations with target users, without the physical constraints of moving and operating hardware, virtual empathic agents can more effectively facilitate communicative interaction^[118-120].

These virtual companions are particularly well-suited for supporting heterogeneous user groups, ranging from language learners who benefit from personalized tutoring to patients undergoing exposure therapy for anxiety or mental trauma. Because their forms and behavior are reconfigurable, virtual robots can be adapted to different cultures, age groups, and appearance preferences without redesigning hardware. Their consistent and programmable empathy make them promising supports for vulnerable populations, including children on the autism phase or older adults experiencing cognitive decline. As AI-driven virtual agents continue to advance in empathy modeling and affective reasoning^[121-123], XR systems may increasingly sustain relationships that are practically meaningful, while dynamically adapting to users’ emotional states and evolving alongside therapeutic or educational goals.

Illustrative vignette: XR-native humanoid coaching with affective adaptation. Consider a VR (or AR) coaching or therapy session where a virtual humanoid robot helps a user practice a skill (e.g., coping with anxiety or learning a routine). Unlike scripted agents which reply merely to certain keywords, an FM-driven virtual humanoid robot can hold a natural, multi-turn conversation and adjust what it says based on what the user has already shared and what is happening in the context^[33,34,81]. Since the interaction happens in XR, the agent can also respond to simple and observable cues, such as long pauses and user emotional changes, by slowing the pace, offering more empathy, and giving the user more space rather than pushing forward^{[56,57,82,83]}. The empathic advantage is evident: with FM integration, the virtual robots can adapt to users’ affect in real-time without requiring the actuation on physical robots, and they can be easily personalized in appearance and interaction style for different users and contexts without changing any hardware.

3.3 Surpassing physical robot capabilities through XR and AI integration

Another revolutionary implication of XR-based virtual robots is envisioned to be their ability to transcend the limitations of physical embodiments^[124,125]. Unconstrained by friction, mass, structural rigidity, or mechanical complexity, virtual agents are not bound to a single physical body or configuration^[96]; they can transform roles instantaneously, adapt behaviors dynamically, and evolve across time and contexts. In VR environments, while the entire surroundings can be constructed virtually with immersion, designers are allowed either to mirror existing physical robots in appearance and shape or to create entirely novel embodiments tailored to specific interaction goals^[126,127]. Locomotion, morphology, and other functions are therefore determined by the design intent customized for concrete needs rather than hardware feasibility. In AR and MR, similar principles apply: virtual counterpart robots or purely virtual agentic avatars can be anchored into the physical environment^[128-131], moving and interacting according to the programmability that does not need to comply with the same constraints as physical robots.

These agents can seamlessly shift roles from tutor to collaborator to supervisor and beyond without the need for physical redesign, complex reconfiguration, or new hardware deployments. Through integration with LLMs or multimodal VLMs, they acquire the capacity to bear multi-turn dialogues, reason over task contexts, and anticipate user needs at a high level of abstraction^[77]. Such abilities position XR-native robots as empathic mediators unconstrained by physical hardware: they not only deliver instructions and guidance but also co-construct knowledge within specific contexts while sidestepping the limitations imposed by physical embodiments^[132-134]. In this sense, virtual robots act as adaptive, empathic partners that extend and complement human capabilities (Figure 4).

Display Full Size

Figure 4. XR virtual agentic robots surpass physical capabilities. XR: extended reality.

Illustrative vignette: Reconfigurable XR-native robot that exceeds physical embodiment constraints. Consider a participatory design session where the users’ needs shift over time: first they want a quick demonstration, then they switch to a different task, and finally they want a step-by-step guidance while trying it themselves. In the physical world, this could require multiple platforms (e.g., an arm for manipulation, a mobile robot for navigation, and another huamnoid robot for demonstration). In XR, one virtual robot can simply change its form and role to fit the situations by being customized into a robot arm, a mobile robot, a small pointer, or a floating agent that instructs next actions, since it is not limited by mass or hardware^[126-129]. With an FM in the loop, virtual agents can keep the interaction coherent across these changes by being grounded in the shared scene and generating explanations that match the users’ current goals^[76,77,79]. The advantage is practically straightforward: smart reconfigurations with contextual consistency which is difficult to achieve with one (or multiple) physical robot(s).

4. Discussion

After synthesizing representative works in Section 2 and articulating three envisioned future scenarios in Section 3. In this section, we consolidate our arguments by (i) discussing the advantages brought by virtual embodiment and how this embodiment and co-presence operate for XR-native robots despite the absence of physical mass; and (ii) distilling and surfacing critical concerns needed to be addressed for next generation interaction with virtual robots. We exemplify four key aspects expected to be resolved, including functional and cognitive advantages, challenges and risks, human-centered ethics, and empathy and adaption.

4.1 Virtual embodiment and user presence

Although virtual robots lack physical mass, they are not disembodied abstractions. In XR, virtual embodiment is achieved through consistent visual appearance, coherent spatial behavior, and responsiveness that aligns with human expectations for social interaction^[135,136]. Studies of MR hazard visualization and robot collaboration show that gaze synchrony, joint attention cues, and predictable proximity dynamics can substantially enhance users’ sense of safety, control, and shared situational awareness. Work on avatar responsiveness and motion timing similarly suggests that even relatively simple agents can evoke strong impressions of co-presence when they react promptly and contingently to user actions^[137].

However, this form of embodiment remains partial. The lack of rich haptic and tactile feedback limits the kinds of tasks that can be trained purely in XR, particularly those requiring fine manipulation, force sensing, or shared physical contact. Emerging haptic overlays and pseudo-physical resistance may narrow this gap, but they also introduce new layers of technical complexity and potential sensory mismatch. Designing virtual embodiment thus becomes a balancing act: providing sufficient realism to support user trust, learning, and empathy, without promising physical capabilities that the virtual environment cannot reliably deliver.

4.2 Toward next generation interaction with virtuality

To reframe the next generation of effective interactions between humans and virtual robots with XR, large FMs are expected to be employed for endowing virtual agents with situational awareness^[138-141]. This enables new forms of physical–virtual interaction but also introduces challenges and risks. Building on the instantiation of the three future scenarios before, we propose four aspects towards effective next generation interaction of human-virtual robot synergy, as illustrated in Figure 5.

Display Full Size

Figure 5. A new form of XR-based virtual robots powered by FMs. The upper left highlights functional and cognitive advantages, while the upper right summarizes key challenges. The lower bands illustrate two complementary design trajectories: human-centered and ethical XR agents (left) and embodied, empathic, and adaptive XR agents (right). XR: extended reality; FMs: foundation models.

4.2.1 Functional and cognitive advantages

The integration of XR with AI-powered virtual robots offers advantages that are not merely incremental extensions of traditional HRI, but qualitatively new affordances. Functionally, safety and scalability are the most immediate benefits: hazardous, rare, or high-stakes events can be rehearsed repeatedly without endangering humans or equipment, and training or evaluation can be replicated at low marginal cost across users and sites^[31,53,54]. For example, industrial collaborative welding, chemical processing, and other risk-aware scenarios involving robots can be practiced in VR with virtual counterparts before an operator is ever exposed to a physical system^[32,47]. XR also improves accessibility by lowering the barrier to entry for experimentation and learning: compared to purchasing and maintaining physical robots, XR-native agents can be instantiated on demand, updated rapidly, and deployed to settings where hardware access, space, or safety regulations would otherwise limit participation^[73-75].

On the cognitive side, XR-native agents supported by LLMs and VLMs can provide context-aware assistance that goes beyond scripted interaction^[77,80,81]. Because the agent has access to the shared virtual scene and to ongoing user input, it can explain intent, adapt guidance to the user’s current state, and adjust interaction pacing in real time^[43-45]. This supports empathic engagement in a practical way: the system can shift from a neutral, task-focused mode to a more supportive and reassuring mode when the user appears confused or stressed, which is difficult to achieve reliably with many physical platforms due to sensing and actuation constraints^[85,86,97]. Finally, XR-native embodiment enables agents to surpass physical constraints by being reconfigurable in form and capability: a virtual robot can change appearance, role, and interaction modality to match the task, allowing rapid prototyping and co-design of complex workflows and multi-agent interaction patterns in simulation before committing to physical hardware or interface designs^[126-128].

4.2.2 Potential challenges and risks

The same characteristics that make XR-based virtual robots powered by large FMs compelling also introduce significant risks: XR and FM integration with high immersion increases perceived realism and social presence, while FM-driven interaction can appear confident and socially fluent even when it is uncertain, biased, or misaligned with the user’s needs.

Trustworthiness and credibility are the most immediate concerns. FM-driven agents can generate persuasive explanations and recommendations that sound coherent while being incorrect, incomplete, or poorly grounded^[34,81]. In XR, where users may experience a virtual robot as a co-present partner, this fluency can amplify overtrust: users may follow guidance because it feels competent and empathic, not because it is verifiably correct^[30,44,49]. The risk is especially pronounced in safety training, clinical support, and education, where an erroneous suggestion delivered with high confidence may translate into unsafe actions, misplaced reliance, or inappropriate decisions^[31,58]. For XR-native virtual robots, a key challenge is therefore not only “making the agent smarter”, but helping users calibrate reliance through clear boundaries, uncertainty communication, and interaction designs that privilege justifiable guidance^[47,88].

Bias and cultural alignment constitute another critical, but often less visible risk. Because virtual robots in XR are embodied through appearance, gaze behavior, proxemics, and conversational style, bias can be expressed not only in language but also in social behavior^[84]. Models trained on unreliable or biased data may inadvertently reproduce stereotypes, misinterpret culturally specific gestures, or privilege certain communication styles over others^[34,81]. When such biases are encapsulated in smart or empathic manners, they can be harder to notice and may subtly shape users’ sense of belonging and legitimacy in the interaction^[83,92,93]. This makes cultural alignment an HRI problem as much as a model problem: the system must be designed and evaluated for diversity and inclusion, with attention to how embodiment and interaction choices coincide with social norms^[88].

Cognitive load and fatigue are practical risks that may be intensified by FM-driven interaction. XR already imposes perceptual and attentional demands; prolonged exposure can lead to motion sickness, eye strain, and attentional exhaustion^[10]. Adding an always-responsive conversational agent can further increase cognitive burden if the system delivers excessive prompts or frequent explanations that compete with the user’s primary task^[32,42]. These effects may disproportionately impact vulnerable populations, including children, older adults, and users with neurological sensitivities^[142]. For this reason, “more interaction” is not automatically better: XR-driven virtual agents must regulate when to intervene, how much information to present, and how to adapt pacing to the user’s workload^[25,40].

Finally, biometric data surveillance and the associated governance concerns are inherently tied to XR. Many XR systems rely on gaze tracking, head and hand motion, voice, and sometimes facial expression analysis for interaction and personalization^[25,28]. Even when collected for benign purposes, such signals can enable sensitive inference (e.g., stress, attention, and impairments)^[56,57]. Without careful design, governance, and transparency, these issues raise concerns about privacy, consent, and accessibility^[48,78]. The practicality and fairness of deploying XR-powered virtual robots to target groups therefore remains an open question, especially in domains such as healthcare and pediatric care where trust and vulnerability are heightened^[109,111]. Addressing this requires clearer privacy statements and transparency about what and why it is needed, and how it is governed.

4.2.3 Toward human-centered and ethical XR agents

The emergence of XR-based virtual robots powered by FMs introduces a new form of agency, autonomy, and perceived authenticity in HCI. As these agents begin to mirror human social cues, such as maintaining eye contact, modulating voice, and mirroring posture, they encourage users to treat them as intentional partners rather than mere tools^[28,97]. This blurring of boundaries raises practical questions for designers and practitioners: How should such agents be framed and introduced to users? How can we ensure that the data used for pre-training and adaptation is curated, documented, and governed in ethically robust and trustworthy ways^[34,48]

Agentic framing concerns how the system is introduced and what users are encouraged to believe about the agent’s competence and intent. Because XR embodiment can make a virtual robot feel socially present, users may interpret it as an intentional partner and attribute authority to it^[10,49]. A human-centered framing therefore requires explicit boundary setting: the system should disclose when it is FM-driven, what it is designed to do, what it is designed not to do, and how uncertainty should be interpreted^[78,81]. The aim is not to reduce engagement, but to support appropriate reliance and prevent users from confusing social fluency with verified capability.

Data governance and transparency are central because XR interaction often relies on multimodal sensing. Ethical deployment requires clarity about what is sensed (e.g., gaze, voice, motion), why it is needed, and whether it is used for personalization or model improvement^[25,28,48]. Consent must be acquired with granularity, and users should be able to disable sensitive modalities without losing basic access to the system^[48]. More broadly, governance should align with data minimization and purpose limitation: only the data required for the stated function should be collected, retention should be bounded, and secondary use should be constrained and monitored.

Context-sensitive design recognizes that “appropriate” embodiment and autonomy vary by domains and populations. In therapy and education, for example, highly anthropomorphic behavior may increase engagement for some users while increasing the risk of over-attachment for others^[109,112]. Similarly, official workplace deployments raise distinct concerns compared to home companionship, and pediatric settings differ from adult settings in both vulnerability and consent to a large extent^[48,111]. As a result, ethical design should be calibrated to the specific settings: the level of anthropomorphism, initiative, persuasion, and personalization should be chosen in relation to the contexts and the users’ agency, rather than harnessing universal design defaults.

Finally, long-term wellbeing and inclusion require thinking about out-of-the-box usability. XR-based agents can shape habits, expectations of social interaction, and patterns of reliance over time, especially for users who interact frequently or who are vulnerable (children, older adults, users with cognitive impairments). Human-centered design therefore needs longitudinal attention to whether these systems support dignity, independence, and psychological safety^[40,97,112]. In practice, this means designing for accessibility concerns, evaluating differential impacts across user groups, and ensuring that personalization supports long-term wellbeing and inclusion instead of only pursuing short-term task performance or technical novelty^[73-75].

4.2.4 Toward embodied, empathic, and adaptive XR agents

These concerns and opportunities point toward a reconceptualization of robots—from purely physical tools to relational, cognitively and emotionally engaged virtual partners. In XR, virtual robots do not primarily manipulate the physical world; instead, they shape users’ attention, scaffold their understanding, and participate in the co-construction of tasks and meanings^[92,93]. Their value lies in adaptive dialogue, emotional resonance, and the ability to inhabit shared virtual spaces in which roles, perspectives, and interaction scripts can be fluid and negotiable, supporting virtually embodied companionship that is experienced as socially situated rather than merely functional^[10,97,126].

Designing for this paradigm demands new forms of co-creation and evaluation. On the design side, achieving meaningful co-creation and co-presence requires XR–HRI systems to be developed through interdisciplinary collaboration that includes not only HCI and AI researchers, but also domain experts such as clinicians, educators, and accessibility specialists, together with end users themselves^[89]. On the evaluation side, a multifaceted evaluation is necessary: traditional performance metrics, such as task completion time or error rate, must be complemented by measures of psychological safety, perceived empathy, inclusion, and long-term behavioral and attitudinal change^[87]. Longitudinal studies are particularly needed to understand how repeated interaction with virtual agents affects user trust, skill retention, and social expectations over months or years^[109].

Reframing virtual robots as relational entities rather than passive tools shifts the focus of innovation from mechanical capability to transformative interaction quality. It requires asking not only what robots can do, but how they make people feel, what forms of understanding they enable, and whose needs they foreground or neglect^[97,112]. By making the interaction empirical, ethical, and collaborative, the transformative potential of XR-based HRI with virtual robots is most likely to be realized.

5. Future Research Agenda

To ensure that XR-based virtual robots evolve in ethically responsible and socially beneficial ways, several research priorities can be pursued in the coming years. Building on the perspectives outlined in this paper, we highlight four potential research directions.

5.1 Evaluation frameworks for trust, empathy, and transferability

Existing XR–HRI evaluations often foreground usability or task performance but rarely capture the affective, relational, and longitudinal dimensions that are central to XR-native virtual robots. Future work should therefore develop multilayered evaluation frameworks that address at least three questions: (i) how users calibrate trust in agents that may be communicatively fluent but epistemically unstable; (ii) how empathic relationships and perceived psychological safety emerge and are maintained over time; and (iii) how learning and behavioral patterns acquired in virtual settings transfer to physical contexts.

Such frameworks will likely need to incorporate and coordinate behavioral, physiological, and self-report measures. Transferability studies are particularly underdeveloped in XR-HRI works: it remains unclear when and how competencies developed with XR-native virtual robots can be generalized into interactions with physical robots or human teammates in real-world conditions. Addressing these gaps will require controlled virtuality-to-physicality experiments, longitudinal field deployments, and shared benchmarks that allow comparison across systems and domains.

5.2 Multi-agent and multi-user interactions

Real-world collaboration rarely occurs in dyads; it typically involves heterogeneous groups of humans and agents coordinating under uncertainty. Yet, most XR–HRI prototypes still assume a single user and a homogeneous agent or agent group. Next-generation platforms should therefore support synchronous, parallel interaction among multiple users and multiple virtual robots, with roles and behaviors that adapt dynamically over time. This calls for mechanisms that enable flexible role allocation and reallocation between humans and agents, and for systematic research on how such roles are negotiated and transformed—how participants shift between leader, peer, tutor, and observer, and how these dynamics shape system accountability, perceived empathy, and inclusion. Understanding and deliberately designing for these emergent, multi-party interaction patterns will be crucial for safe, equitable, and effective XR-mediated teamwork.

5.3 Societal and ethical design

As XR-based virtual robots might flourish in schools, clinics, workplaces, and homes, they will not only reflect but also participate in shaping social norms, expectations, and mutual relations. Future research must therefore move beyond ad hoc “ethics checklists” and toward empirically grounded, participatory approaches to socio-technical design. A first priority is the detection, characterization, and mitigation of bias in both reasoning processes and actual embodiments. This includes examining how cultural, gendered, racialized, or ability-related cues are encoded in appearance, voice, posture, and interaction style, and how these cues influence user trust, identification, and perceived legitimacy.

Second, the extensive biometric sensing often involved in XR (eye tracking, facial expression analysis, posture and gesture monitoring, and voice sentiment inference) demands consent frameworks and data governance models that are legible to non-experts and co-developed with the affected communities. Questions of data retention, secondary use, and the right to opt out of sensor-based personalization will be central, particularly in healthcare, education, and workplace surveillance contexts.

5.4 Mixed embodiment and XR–physical integration

Finally, the integration of XR-native agents with physical robots presents an important, yet comparatively underexplored frontier. XR can function as a cognitive and social twin of physical systems, providing a training and interaction layer that mirrors (or intentionally extends) the capabilities of real-world robots. Through shared autonomy and digital twin frameworks, users could engage with a virtual robot in XR for developing shared routines, understanding capabilities, and calibrating trust before completely transitioning to interaction with its physical counterpart.

Research in this area must examine how consistency, continuity, and synchronization are maintained across modalities, including: When does divergence between virtual and physical behavior support learning and creativity? When does it undermine trust or induce negative awareness? How should haptic overlays, audio cues, and other multisensory feedback be orchestrated so that transitions between virtual and physical embodiments remain intelligible and safe? Addressing these questions will require joint advances in control, interface design, perception, and systematic evaluation, as well as careful consideration of where hybrid systems are most appropriate versus where purely virtual or purely physical approaches suffice.

6. Conclusion and Outlook

Virtual robots in XR mark a significant evolution in the broader spectrum of HRI. More than simulations or visualization tools, these agents can act as collaborative partners, capable of learning from users, adapting to context, and engaging across cognitive, emotional, and spatial dimensions through the state-of-the-art large FMs. They offer safer environments for training and exploration, richer and more flexible educational and therapeutic experiences, and new spaces for participatory design and experimentation that are not constrained by the limits of physical hardware.

In this paper, we have argued that virtual robots in XR should be understood not only as technical artifacts, but as relational and cultural ones. They mediate how people learn, work, and relate to one another, and thus must be designed and evaluated with attention to empathy, inclusion, and long-term wellbeing. We call on researchers, designers, practitioners, and policymakers to treat XR-native virtual robots as part of a broader socio-technical ecosystem: one that spans AI, HRI, XR, ethics, accessibility, and domain expertise. By approaching these systems as sites of interdisciplinary co-creation, rather than as isolated engineering projects, we can better ensure that their evolution is aligned with human values and diverse forms of flourishing. The future of virtual robots, like our own, will be shaped by the commitments we adopt and the infrastructures we build in the present.

At the same time, this potential is inseparable from responsibility. Without appropriate design and governance, XR-based virtual robots with large FMs may inadvertently reproduce existing inequities, foster less transparency, or blur the boundaries in certain aspects. As XR technologies mature and FMs grow more capable and autonomous, we are approaching a decisive juncture: whether virtual robots will amplify human cognition, compassion, and creativity, or raise bias, erode agency, and normalize intrusive sensing will depend on choices made now.

Acknowledgements

We acknowledge that authors made limited use of generative AI tools for language polishing only.

Authors contribution

Zhang Y: Conceptualization, resources, writing-original draft.

Ma Y: Resources.

Kragic D: Supervision.

Conflicts of interest

The authors declare no conflicts of interest.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Not applicable.

Funding

None.

Copyright

© The Author(s) 2026.