Teleoperation with Semi-Autonomous Behaviors and Latency

Principal Investigator: Dawn Tilbury, University of Michigan,
Student: Justin Storms, University of Michigan
Government: Dave Daniszewski, Paul Muench, Ben Haynes, U.S. Army TARDEC
Industry: Mitch Rohde, Steve Rohde, Quantum Signal

Schematic of closed loop teleoperated robotic systemThe goal of this project is to improve the overall performance of the closed loop teleoperated robotic system as shown in the figure to the left. Based on current limitations of directly-controlled (e.g. human operator inputs of steering and throttle), teleoperated robots, we know that autonomous behaviors must be included. However, it is important to understand how the user interacts with these autonomous (or semi-autonomous) behaviors. Direct control of teleoperated robots extends the warfighter’s mobility into areas that are too difficult or dangerous to reach. The addition of autonomous behaviors adds intelligence to this mobility, increasing the capabilities of the human-robot team.

Building on a previous ARC project, where a mathematical model that can characterize user behavior in a direct-control teleoperation steering task with latency, we will expand our understanding of latency’s effect on user performance in teleoperation tasks, including under semi-autonomous behaviors. A good understanding of the user behavior remains important for characterizing the closed-loop performance.

The long-term research objective of this project is to model and characterize the closed-loop performance of a user teleoperating a mobile robot with significant latency, with the objective of developing design guidelines that can maximize the overall performance. Towards this end, we will develop and implement semi-autonomous behaviors on a simulated mobile robot (using the ANVEL software), develop different control blending methods to combine the user’s commands with the autonomy, and perform user tests to assess the performance. The major research questions that we will address include:

  1. How should the user and autonomous control inputs be blended to give the best overall “shared control” performance of the teleoperated mobile robot?
  2. How do factors unique to teleoperation (such as latency and highly unstructured environments) affect the optimal blending method?
  3. How should the system performance be defined for teleoperation in an unstructured environment? What is the objective that should be optimized?

This project will leverage related research underway in the ARC.


  • J.G. Storms, and D.M. Tilbury, “Reducing Rollover Risk of a High Speed Mobile Manipulator,” in ASME Dynamic Systems and Controls Conference, 2014.
  • J. G. Storms and D. M. Tilbury, “Blending of Human and Obstacle Avoidance Control for a High Speed Mobile Robot,” American Control Conference, 2014.
  • J. Storms, S. Vozar, and D. Tilbury. "Predicting Human Performance During Teleoperation," ACM/IEEE International Conference on Human-Robot Interaction, pp. 298-299, 2014.
  • J. Storms, and D. Tilbury. “Equating User Performance Among Communication Latency Distributions and Simulation Fidelities,” IEEE International Conference on Robotics and Automation, 2015.