Towards embodied smartphone AR: Improving avatar tracking accuracy via face tracking and hand IK on smartphones

Elhassan Makled

1,*

Wolfgang Broll

1,2

*Correspondence to: Elhassan Makled, Virtual Worlds and Digital Games Research Group, TU Ilmenau, Ilmenau 98693, Thuringia, Germany. E-mail: elhassan.makled@tu-ilmenau.de

Empath Comput. 2026;2:202625. 10.70401/ec.2026.0025

Received: June 30, 2026Accepted: September 24, 2026Published: September 29, 2026

This manuscript is made available in its unedited form to allow early access to the reported findings. Further editing will be completed before final publication. As such, the content may include errors, and standard legal disclaimers are applicable.

Abstract

Aims: Smartphone Augmented Reality (AR) enables social applications, but single-point Inverse Kinematics (IK) avatar tracking limits pose expressiveness and introduces positional errors. We introduce DPIK, a dual-point tracking system combining hand IK with face tracking, and evaluated whether it improves tracking accuracy, embodiment, and user experience over single-point IK.

Methods: DPIK uses the front and back cameras of ARKit to track the face of the user and hand/device pose, driving avatar hand IK and hip/head positioning. In a within-subjects study (N=20), we compared DPIK against our prior single-point IK system (ARIKA) and full-body motion capture (ground truth) across positional/rotational accuracy, embodiment, and user experience. We also examined temporal drift in tracking error and Spearman correlations between error and embodiment/user experience scores.

Results: DPIK significantly outperformed ARIKA in positional accuracy for 18 of 19 joints and reduced head rotational error by over . A temporal drift analysis found that neither ARIKA nor DPIK exhibited statistically significant growth in positional error over time. DPIK received significantly higher overall user experience ratings than ARIKA. Embodiment differences were not statistically significant, though effect sizes were moderate-to-large. Exploratory correlations suggested lower DPIK embodiment scores for Acceptance and Control with higher positional error, but these did not survive multiple-comparison correction; positional error did not predict user experience ratings for either technique.

Conclusion: DPIK improves tracking accuracy and user experience over ARIKA, with tracking fidelity selectively shaping embodiment rather than overall user experience, highlighting a distinction between perceptual ownership and holistic evaluation of smartphone AR avatar systems.

Keywords

User embodiment, tracking fidelity, virtual reality, augmented reality, user study

References

1. Makled E, Broll W. DPIK: User embodiment of dual-point tracked avatars using hand IK and face tracking for smartphone AR users. In: 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2025 Oct 8-12; Daejeon, Korea. Piscataway: IEEE; 2025. p. 921-931.

[DOI]
2. Eubanks JC, Moore AG, Fishwick PA, McMahan RP. The effects of body tracking fidelity on embodiment of an inverse-kinematic avatar for male participants. In: 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2020 Nov 9-13; Porto de Galinhas, Brazil. Piscataway: IEEE; 2020. p. 54-63.

[DOI]
3. Gonçalves G, Melo M, Barbosa L, Vasconcelos-Raposo J, Bessa M. Evaluation of the impact of different levels of self-representation and body tracking on the sense of presence and embodiment in immersive VR. Virtual Real. 2022;26(1):1-14.

[DOI]
4. Yoon B, Kim HI, Lee GA, Billinghurst M, Woo W. The effect of avatar appearance on social presence in an augmented reality remote collaboration. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR); 2019 Mar 23-27; Osaka, Japan. Piscataway: IEEE; 2019. p. 547-556.

[DOI]
5. Eichelberger P, Ferraro M, Minder U, Denton T, Blasimann A, Krause F, et al. Analysis of accuracy in optical motion capture—A protocol for laboratory setup evaluation. J Biomech. 2016;49(10):2085-2088.

[DOI]
6. Makled E, Weidner F, Broll W. Investigating user embodiment of inverse-kinematic avatars in smartphone augmented reality. In: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2022 Oct 17-21; Singapore, Singapore. Piscataway: IEEE; 2022. p. 666-675.

[DOI]
7. Ahuja K, Mayer S, Goel M. Pose-on-the-go: Approximating user pose with smartphone sensor fusion and inverse kinematics. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems; 2021 May 8-13; Yokohama, Japan. New York: Association for Computing Machinery; 2021. p. 1-12.

[DOI]
8. Murugan A, Vanukuru R, Pillai J. Towards avatars for remote communication using mobile augmented reality. In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 2021 Mar 27-Apr 1. Piscataway: IEEE; 2021. p. 135-139.

[DOI]
9. Weidner F, Boettcher G, Arboleda SA, Diao C, Sinani L, Kunert C, et al. A systematic review on the visualization of avatars and agents in AR & VR displayed using head-mounted displays. IEEE Trans Vis Comput Graph. 2023;29(5):2596-2606.

[DOI]
10. Genay A, Lecuyer A, Hachet M. Being an avatar “for real”: A survey on virtual embodiment in augmented reality. IEEE Trans Vis Comput Graph. 2022;28(12):5071-5090.

[DOI]
11. Wolf E, Fiedler ML, Döllinger N, Wienrich C, Latoschik ME. Exploring presence, avatar embodiment, and body perception with a holographic augmented reality mirror. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR); 2022 Mar 12-16; Christchurch, New Zealand. Piscataway: IEEE; 2022. p. 350-359.

[DOI]
12. Wu Y, Wang Y, Jung S, Hoermann S, Lindeman RW. Using a fully expressive avatar to collaborate in virtual reality: Evaluation of task performance, presence, and attraction. Front Virtual Real. 2021;2:641296.

[DOI]
13. Tarnec HL, Bevacqua E, Augereau O, De Loor P. Effect of avatar facial expressiveness on team collaboration in virtual reality. In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents; 2023 Sep 19-22; Würzburg Germany. New York: Association for Computing Machinery; 2023. p. 1-8.

[DOI]
14. Nimcharoen C, Zollmann S, Collins J, Regenbrecht H. Is that me?—Embodiment and body perception with an augmented reality mirror. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct); 2018 Oct 16-20; Munich, Germany.Piscataway: IEEE; 2018. p. 158-163.

[DOI]
15. Banakou D, Hanumanthu PD, Slater M. Virtual embodiment of white people in a black virtual body leads to a sustained reduction in their implicit racial bias. Front Hum Neurosci. 2016;10:601.

[DOI] [PubMed] [PMC]
16. Do TD, Protko CI, McMahan RP. Stepping into the right shoes: The effects of user-matched avatar ethnicity and gender on sense of embodiment in virtual reality. IEEE Trans Vis Comput Graph. 2024;30(5):2434-2443.

[DOI]
17. Suk H, Laine TH. Influence of avatar facial appearance on users’ perceived embodiment and presence in immersive virtual reality. Electronics. 2023;12(3):583.

[DOI]
18. Schott E, Makled EB, Zoeppig TJ, Muehlhaus S, Weidner F, Broll W, et al. UniteXR: Joint exploration of a real-world museum and its digital twin. In: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology; 2023 Oct 9-11; Christchurch New Zealand. New York: Association for Computing Machinery; 2023. p. 1-10.

[DOI]
19. Yassien A, Makled EB, Elagroudy P, Sadek N, Abdennadher S. Give-me-a-hand: The effect of partner’s gender on collaboration quality in virtual reality. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems; 2021 May 8-13; Yokohama, Japan. New York: Association for Computing Machinery; 2021. p. 1-6.

[DOI]
20. Benda B, Ragan ED. The effects of virtual avatar visibility on pointing interpretation by observers in 3D environments. In: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2021 Oct 4-8; Bari, Italy. Piscataway: IEEE; 2021. p. 50-59.

[DOI]
21. Lugrin JL, Ertl M, Krop P, Klupfel R, Stierstorfer S, Weisz B, et al. Any “body” there? Avatar visibility effects in a virtual reality game. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR); 2018 Mar 18-22; Tuebingen/Reutlingen, Germany. Piscataway: IEEE; 2018. p. 17-24.

[DOI]
22. Peck TC, Gonzalez-Franco M. Avatar embodiment. A standardized questionnaire. Front Virtual Real. 2021;1:575943.

[DOI]
23. Gonzalez-Franco M, Peck TC. Avatar embodiment. Towards a standardized questionnaire. Front Robot AI. 2018;5:74.

[DOI]
24. Roth D, Latoschik ME. Construction of the virtual embodiment questionnaire (VEQ). IEEE Trans Vis Comput Graph. 2020;26(12):3546-3556.

[DOI]
25. Botvinick M, Cohen J. Rubber hands ‘feel’ touch that eyes see. Nature. 1998;391(6669):756.

[DOI]
26. Slater M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Phil Trans R Soc B. 2009;364(1535):3549-3557.

[DOI]
27. Woodman OJ. An introduction to inertial navigation. Cambridge: University of Cambridge, Computer Laboratory; 2007.

[DOI]
28. Roetenberg D, Luinge H, Slycke P. Xsens MVN: Full 6DOF human motion tracking using miniature inertial sensors. Enschede: Xsens Motion Technologies BV. 2009. Available from: https://www.researchgate.net/profile/Per-Slycke/publication/239920367_Xsens_MVN_Full_6DOF_human_motion_tracking_using_miniature_inertial_sensors/links/0f31752f1f60c20b18000000/Xsens-MVN-Full-6DOF-human-motion-tracking-using-miniature-inertial-sensors.pdf
29. Borges M, Symington A, Coltin B, Smith T, Ventura R. HTC vive: Analysis and accuracy improvement. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2018 Oct 1-5; Madrid, Spain. Piscataway: IEEE; 2018. p. 2610-2615.

[DOI]
30. Lloyd DM. Spatial limits on referred touch to an alien limb may reflect boundaries of visuo-tactile peripersonal space surrounding the hand. Brain Cogn. 2007;64(1):104-109.

[DOI] [PubMed]
31. Kokkinara E, Slater M. Measuring the effects through time of the influence of visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion. Perception. 2014;43(1):43-58.

[DOI] [PubMed]
32. Waltemate T, Gall D, Roth D, Botsch M, Latoschik ME. The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response. IEEE Trans Vis Comput Graph. 2018;24(4):1643-1652.

[DOI]
33. Drakakis E, Goumopoulos C. Tracking systems and visualization devices in virtual, augmented, and mixed reality games for motor and cognitive rehabilitation and training: A scoping review. Appl Sci. 2026;16(6):2671.

[DOI]
34. Kim J, Jun H. Vision-based location positioning using augmented reality for indoor navigation. IEEE Trans Consum Electronics. 2008;54(3):954-962.

[DOI]
35. Singh M, Shankar RA, Jung B. Inside-out magnetic tracking for virtual/augmented reality applications. IEEE Sens J. 2021;21(24):28097-28106.

[DOI]
36. Kim P, Kim J, Song M, Lee Y, Jung M, Kim HG. A benchmark comparison of four off-the-shelf proprietary visual-inertial odometry systems. Sensors. 2022;22(24):9873.

[DOI] [PubMed] [PMC]
37. Barai S, Momin M. Outside-in electromagnetic tracking method for augmented and virtual reality 6-degree of freedom head-mounted displays. In: 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS); 2020 May 13-15; Madurai, India. Piscataway: IEEE; 2020. p. 467-476.

[DOI]
38. Vox JP, Weber A, Wolf KI, Izdebski K, Schüler T, König P, et al. An evaluation of motion trackers with virtual reality sensor technology in comparison to a marker-based motion capture system based on joint angles for ergonomic risk assessment. Sensors. 2021;21(9):3145.

[DOI] [PubMed] [PMC]
39. Monica R, Rizzini DL, Aleotti J. Adaptive complementary filter for hybrid inside-out outside-in hmd tracking with smooth transitions. IEEE Trans Vis Comput Graph. 2025;31(2):1598-1612.

[DOI]
40. Hu F, He P, Xu S, Li Y, Zhang C. FingerTrak: Continuous 3D hand pose tracking by deep learning hand silhouettes captured by miniature thermal cameras on wrist. Proc. Wearable Ubiquitous Technol. 2020;4(2):1-24.

[DOI]
41. Liu Y, Zhang S, Gowda M. NeuroPose: 3D hand pose tracking using EMG wearables. In: Proceedings of the Web Conference 2021; 2021 Apr 19-23; Ljubljana Slovenia. New York: Association for Computing Machinery; 2021:1471-1482.

[DOI]
42. Kim D, Park K, Lee G. Oddeyecam: A sensing technique for body-centric peephole interaction using wfov rgb and nfov depth cameras. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology; 2020 Oct 20-23; Virtual Event USA; New York: Association for Computing Machinery; 2020. p. 85-97.

[DOI]
43. Park K, Kim S, Yoon Y, Kim TK, Lee G. DeepFisheye: Near-surface multi-finger tracking technology using fisheye camera. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology; 2020 Oct 20-23; Virtual Event USA. New York: Association for Computing Machinery; 2020. p. 1132-1146.

[DOI]
44. Wu E, Yuan Y, Yeo HS; Quigley A; Koike H; Kitani KM. Back-hand-pose: 3D hand pose estimation for a wrist-worn camera via dorsum deformation network. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology; 2020 Oct 20-23; Virtual Event USA. New York: Association for Computing Machinery; 2020. p. 1147-1160.

[DOI]
45. Aristidou A, Lasenby J. FABRIK: A fast, iterative solver for the Inverse Kinematics problem. Graph Models. 2011;73(5):243-260.

[DOI]
46. Debarba HG, Chagué S, Charbonnier C. On the plausibility of virtual body animation features in virtual reality. IEEE Trans Vis Comput Graph. 2022;28(4):1880-1893.

[DOI]
47. Fribourg R, Argelaguet F, Lecuyer A, Hoyet L. Avatar and sense of embodiment: Studying the relative preference between appearance, control and point of view. IEEE Trans Vis Comput Graph. 2020;26(5):2062-2072.

[DOI] [PubMed]
48. Makled E, Gerhardt C, Schwandt T, Weidner F, Broll W. Investigating behavioral realism of single-point IK animated avatars of others in AR and VR. In: 2024 International Conference on Cyberworlds (CW); 2024 Oct 29-31; Kofu, Japan. Piscataway: IEEE; 2024. p. 1-8.

[DOI]
49. Fisher RA. Statistical methods for research workers. In: Kotz S, Johnson NL, editors. Breakthroughs in statistics: Methodology and distribution. New York: Springer; 1992. p. 66-70.

[DOI]
50. Schrepp M, Hinderks A, Thomaschewski J. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). Int J Interact Multimed Artif Intell. 2017;4(6):103-108.

[DOI]
51. Wilcoxon F. Individual comparisons by ranking methods. In: Kotz S, Johnson NL, editor. Breakthroughs in statistics: Methodology and distribution. New York: Springer; 1992. p. 196-202.

[DOI]
52. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New York: Routledge; 2013.
53. Tukey JW. Exploratory data analysis. Reading: Addison-Wesley Pub. Co; 1977. Available from: https://tocxten.com/wp-content/uploads/2025/01/Module2.pdf
54. Leys C, Ley C, Klein O, Bernard P, Licata L. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the Median. J Exp Soc Psychol. 2013;49(4):764-766.

[DOI]
55. Field A. Discovering statistics using IBM SPSS statistics. 6th ed. Thousand Oaks: SAGE Publications Limited; 2024. Available from: https://www.sagepub.com/shop/buy-a-book/discovering-statistics-using-ibm-spss-statistics-6-285130
56. Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Ser B Stat Methodol. 1995;57(1):289-300.

[DOI]

Copyright

© The Author(s) 2026. This is an Open Access article licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, sharing, adaptation, distribution and reproduction in any medium or format, for any purpose, even commercially, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Publisher’s Note

Science Exploration remains a neutral stance on jurisdictional claims in published maps and institutional affiliations. The views expressed in this article are solely those of the author(s) and do not reflect the opinions of the Editors or the publisher.

Share And Cite

Science Exploration Style

Makled E, Broll W. Towards embodied smartphone AR: Improving avatar tracking accuracy via face tracking and hand IK on smartphones. Empath Comput. 2026;2:202625. https://doi.org/10.70401/ec.2026.0025