After Action Reviews with a Virtual Spectator System for Improving Human-Robot Team Performance

Project Team

Principal Investigator

Dawn Tilbury, University of Michigan Lionel Robert, University of Michigan Wing-Yue Geoffrey Louie, Oakland University

Government

Kayla Riegner, Jon Smereka, Ben Haynes, Mark Brudnak, U.S. Army GVSC

Industry

Samantha Dubrow, The MITRE Corporation

Lilia Moshkina, May Mobility

Student

Sean Dallas (PhD), Motaz AbuHijleh (MS), Evan Dallas (UG), Belenus Carroll (UG), Oakland University

Hongjiao Qiang (MS), Jasmine Li (UG), Wonse Jo (Postdoc), University of Michigan

Project Summary

Project IE.01 began in 2023 and was completed June 2024.

This integration effort produced the case study “Through the Virtual Lens: Improving Human-Robot Team Performance via Training Reviews with a Virtual Spectator Interface” presented at the 2024 ARC Annual Program Review.

Case Study Abstract

In the military, a training review is often used to examine team member behavior and team outcomes. By reviewing what did and what did not go well and the reasons why, a training review has the potential to improve team performance in the future. Research has also shown that team performance can improve when the human has more situation awareness (SA) and SA is needed to accomplish team goals. We hypothesize that training reviews can improve future SA, and thereby also improve team performance. To test this hypothesis, this project integrates a virtual spectator interface (Project #2.A92) with a two-robot unmanned ground vehicle reconnaissance mission simulation (Project #2.17) to support training reviews for improving SA and human-robot team performance. Three types of training reviews varying in format (spectator interface, screen recording, verbal description) were tested in a between-subjects user experiment at two locations. The results provide implications on the design of training reviews, including what information may be most valuable for soldiers and the best format to present information.

In the military, an after action review (AAR) is often used to examine team member behavior and team outcomes. By reviewing what did and what did not go well and the reasons why, an AAR has the potential to improve team performance in the future. Research has shown that team performance can improve when the human has more situation awareness (SA) and SA is needed to accomplish team goals. We hypothesize that AAR (reviewing what happened and why) can improve future SA (understanding what will happen next), and thereby also improve team performance.

The goal of project 2.17 “Estimating and Calibrating Situation Awareness for Improving Human-Robot Teaming Performance” (SACO) is to develop methods for estimating and calibrating situational awareness (SA) in teams to improve team performance. The experimental design for the project has two UGVs performing a search and reconnaissance task, while a human operator supervises the UGVs and fills out spot reports. Different levels of communication from the UGVs to the operator are considered, and SA is measured (via survey). The eventual goal of the project is to develop models that can predict SA, and methods that can calibrate SA (through optimized communication).

The goal of project 2.A92 “A Virtual Spectator System for a Multi-User Video Game Environment” (VS) is to develop a virtual spectator interface to spectate virtual experiments, where soldiers are interacting with technology and teammates in a video game environment. The spectator interface currently consists of features which support spectators in processing, exploring, and visualizing data. The features include data visualization through graphs and map layers, event notifiers and automated camera directors to support tracking of key events, overlays to improve agent transparency, and replays for reviewing gameplay. Both SACO and VS projects are using the Unreal Engine as their simulation platform.

We aim to integrate the Virtual Spectator (VS) system into the SACO environment to construct an AAR, and measure the effects of AARs on SA and team performance.

The primary goal of this integration project is to understand how an AAR can improve future SA and overall team performance. A secondary goal is to understand what and how the information should be presented in an AAR to impact future SA. We hypothesize that an AAR will improve future SA, resulting in improved future team performance as well.

Supplementary video for ICRA2025 paper: https://youtu.be/lWdI5IEpxak

IE.01

Publications:

Dallas, S., Qiang, H., AbuHijleh, M., Jo, W., Riegner, K., Smereka, J., … & Tilbury, D. M. (2025, May). Training Human-Robot Teams by Improving Transparency Through a Virtual Spectator Interface. In 2025 IEEE International Conference on Robotics and Automation (ICRA) (pp. 4294-4300). IEEE.