A Virtual Spectator System for a Multi-User Video Game Environment

Exploratory project #2.A92 began Sep. 2021 and completed Dec. 2022.

Virtual experimentation in video game environments primarily focus on capturing user self-reported experiences towards virtually prototyped technologies because these environments have been primarily designed for users to interact within them. Although user self-reported experiences are important, direct observation of user behaviors should be utilized to complement self-reports because self-reported responses are not always congruent with users’ actual behaviors with technology and introduces individual biases.

However, it is difficult for an observer to spectate users in video game environments because the spectating interfaces are often designed as an afterthought. Spectators can only either follow a single user from a first-person perspective, control a spectator camera to freely roam within the video game environment, or utilize a customized interface designed for a single game. Consequently, current spectating interfaces are not suitable for virtual experimentation because they do not consider the needs of an external observer such as spectating at different levels of organization; interpreting large quantities of data with complex interactions between users; and monitoring the intent, state, and actions of a user or a group of users.

Our research goal is to address this gap in virtual experimentations by enabling stakeholders to effectively spectate a virtual experiment within a multi-user gaming environment to gather observations on solider behaviors with new technological concepts. The specific research objectives for this project are to:

1. Identify the requirements from GVSC and ARC for a spectator interface during virtual experiments.
2. Design and develop a virtual spectator system for the unreal engine to address spectator requirements identified in objective one.
3. Evaluate spectator experiences for virtual experimentation on different hardware platforms.

We developed a spectator interface for data analysis and a virtual experimentation sandbox to evaluate its efficacy. Also, we developed a human-robot teaming security scenario with our virtual experiment sandbox and conducted a human-robot interaction study using the spectator system as a central tool for data analysis.

The project outcomes fed into an integrative effort that produced the 2024 case study “Through the Virtual Lens: Improving Human-Robot Team Performance via Training Reviews with a Virtual Spectator Interface”, and subsequent ARC exploratory projects 2.A97 and 2.A98:

• Virtual Experimentation for Soldier Evaluation of Autonomous and Non-Autonomous Technologies Using Multi-User Immersive Gaming Environments
• Multi-Objective Rapid Generation of Virtual Environments and Data Analysis in Multi-User Immersive Gaming Environments for Testing New Crew Interface Concepts

Publications:

• W.-Y. G. Louie, M. AbuHijleh, S. Dallas, M. Brudnak, and G. Pappas, “A Virtual Spectator System for Virtual Experimentation in Multi-User Video Game Environments,” In Proceedings of the Ground Vehicle Systems Engineering and Technology Symposium (GVSETS), NDIA, Novi, MI, Aug. 16-18, 2023.

2.A92