A Virtual Spectator System for a Multi-User Video Game Environment

This is an exploratory project that began in Sep. 2021, through Dec. 2022.

Virtual experimentation in video game environments primarily focus on capturing user self-reported experiences towards virtually prototyped technologies because these environments have been primarily designed for users to interact within them. Although user self-reported experiences are important, direct observation of user behaviors should be utilized to complement self-reports because self-reported responses are not always congruent with users’ actual behaviors with technology and introduces individual biases. However, currently it is difficult for an observer to spectate users in video game environments because the spectating interfaces are often designed as an afterthought. Spectators can only either follow a single user from a first-person perspective, control a spectator camera to freely roam within the video game environment, or utilize a customized interface designed for a single game. Consequently, current spectating interfaces are not suitable for virtual experimentation because they do not consider the needs of an external observer such as spectating at different levels of organization; interpreting large quantities of data with complex interactions between users; and monitoring the intent, state, and actions of a user or a group of users.
The overall research goal of this project is to address this gap in virtual experimentations by enabling stakeholders to effectively spectate a virtual experiment within a multi-user gaming environment to gather observations on solider behaviors with new technological concepts. The specific research objectives for this project are to:

1. Identify the requirements from GVSC and ARC for a spectator interface during virtual experiments.
2. Design and develop a virtual spectator system for the unreal engine to address spectator requirements identified in objective one.
3. Evaluate spectator experiences for virtual experimentation on different hardware platforms.

This project feeds into the subsequent projects:

• Virtual Experimentation for Soldier Evaluation of Autonomous and Non-Autonomous Technologies Using Multi-User Immersive Gaming Environments
• Multi-Objective Rapid Generation of Virtual Environments and Data Analysis in Multi-User Immersive Gaming Environments for Testing New Crew Interface Concepts

Publications:

• W.-Y. G. Louie, M. AbuHijleh, S. Dallas, M. Brudnak, and G. Pappas, “A Virtual Spectator System for Virtual Experimentation in Multi-User Video Game Environments,” In Proceedings of the Ground Vehicle Systems Engineering and Technology Symposium (GVSETS), NDIA, Novi, MI, Aug. 16-18, 2023.

Publications from Prior Work closely related to this project:

1. R. Kulikovskiy, M. Sochanski, A. Hijaz, M. Eaton, J. Korneder, and W.-Y. G. Louie, “Can Therapists Design Robot-Mediated Interventions and Teleoperate Robots Using VR to Deliver Interventions for ASD?,” IEEE International Conference on Robotics and Autonomation, 2021, Under Review.
2. A. Hijaz, W. -Y. G. Louie, M. Bellafaire, O. Rawashdeh, and I. Mansour, “Driver Visual Focus of Attention Estimation in Autonomous Vehicles,” SAE Technical Paper, 2020, 2020-01-1037, Available: https://doi.org/10.4271/2020-01-1037.
3. W.-Y. G. Louie and R. Kulikovskiy, “Evaluating a Virtual Reality-based Interface for Teaching Humanoids,” MSGC Fall Conference, Ann Arbor, Michigan, October 2019.
4. A. Hijaz, W.-Y.G. Louie, and I. Mansour, “Towards a Driver Monitoring System for Estimating Driver Situational Awareness,” in IEEE International Conference on Robot & Human Interactive Communication, 2019, pp. 1-6, Available: http://dx.doi.org/10.1109/RO-MAN46459.2019.8956378.
5. W.-Y. G. Louie and G. Nejat, “A victim identification methodology for rescue robots operating in cluttered USAR environments,” Advanced Robotics, vol. 27, no. 5, pp. 373–384, 2013, Available: http://dx.doi.org/10.1080/01691864.2013.763743.

#2.A92