Virtual Experimentation for Soldier Evaluation of Autonomous and Non-Autonomous Technologies Using Multi-User Immersive Gaming Environments

Projects #2.A97 and #2.A98 began Q4 2022 and was completed Q3 2025.

Existing human-technology user studies primarily utilize physical prototypes of technology or occasionally utilize technologies with limited interaction capabilities that may not provide users with the necessary immersion and realism necessary to draw insights that predict future real-world user behaviors. Namely, current simulation technology lacks approaches for rapidly generating immersive and realistic human-technology interaction experiments with users and although there are some frameworks available, they remain unsuitable for military applications. It is also unclear within the literature what is the level of fidelity required in simulated human-technology interactions to produce comparable results as those found from physical experimentation. This project aimed to address these research gaps and answer the following broad research questions:

1. How can human-technology interactions be rapidly prototyped in video game engines to support soldier interactions during virtual experiments?
2. What is the level of virtual fidelity necessary to produce results comparable to corresponding physical experiments in different contexts?
3. How to effectively conduct and analyze large scale multi-user virtual experiments on new enabling technologies?

This project focused on developing a foundational platform for investigating large scale human-technology interactions via virtual experiments in video game environments. In this project, we developed an all-in-one virtual prototyping and experimentation platform that supports the rapid design, development, implementation, and analysis of human-technology interactions and experiments within video game environments. This project had the following three specific objectives:

1. Rapid prototyping of human-technology interactions for virtual experimentation
2. Characterizing the necessary level of fidelity for ecologically valid virtual experiments
3. Conducting and analyzing virtual experiments on new enabling technologies

These objectives were addressed in this project through six sub-projects:

1. Behavioral cloning of autonomous agents using low-cost heuristics (Objective 1) by creating an environment representative of GVSC-relevant scenario environments in Unreal Engine (UE) 5 and a communication framework enabling two-way communication between autonomy models and UE5. A dataset of low-dimensional expert demonstrations within the simulation environment was produced and use to train an autonomy model that can clone the expert’s behavior.
2. Rapid prototyping of immersive environments to replicate real-world settings (Objective 1) with a complete workflow for 3D reconstruction and VR deployment, automated indoor/outdoor image acquisition using robot/UAV means.
3. Rapid prototyping of crew stations to support manned-unmanned teaming (Objective 1 & 2) by answering the questions: a. How do varying levels of haptic fidelity impact user presence, embodiment, and system usability in gesture-based VR tasks? b. How do different haptic fidelity levels influence performance metrics such as accuracy, error rate, speed, and task completion time? c. Are continuous gestures (pan, pinch) more sensitive to differences in haptic fidelity compared to discrete gestures (tap, swipe)?
4. Semi-automated data analysis and spectating during virtual experimentation (Objective 3) by developing approaches to segment individual HRT sessions into discretized trajectories for analysis, unsupervised learning techniques to identify behaviors from the aggregated dataset of discretized trajectories, and a HRT virtual experiment and collecting as well as curating a representative dataset with ground truth behavior labels.
5. Scalable machine-assisted interviews of soldier experiences with combat vehicle technologies (Objective 3) by identifying and defining the verbal and nonverbal interviewer behaviors that contribute to high-quality interviews, translating these behaviors into system-level design principles, investigating how varying levels of communication richness influenced the effectiveness of the interview, and conducting a user study to compare the performance of the developed machine interviewer systems.
6. Multimodal sentiment analysis during virtual experimentation (Objective 3).

2.A97, 2.A98