Teleoperation with Semi-Autonomous Behaviors and Latency

This project began in 2014 and was completed in 2016.

The goal of this project is to improve the overall performance of the closed loop teleoperated robotic system as shown in the figure to the left. Based on current limitations of directly-controlled (e.g. human operator inputs of steering and throttle), teleoperated robots, we know that autonomous behaviors must be included. However, it is important to understand how the user interacts with these autonomous (or semi-autonomous) behaviors. Direct control of teleoperated robots extends the warfighter’s mobility into areas that are too difficult or dangerous to reach. The addition of autonomous behaviors adds intelligence to this mobility, increasing the capabilities of the human-robot team.

Building on a previous ARC project, where a mathematical model that can characterize user behavior in a direct-control teleoperation steering task with latency, we expand our understanding of latency’s effect on user performance in teleoperation tasks, including under semi-autonomous behaviors. A good understanding of the user behavior remains important for characterizing the closed-loop performance.

The research objective of this project is to model and characterize the closed-loop performance of a user teleoperating a mobile robot with significant latency, with the objective of developing design guidelines that can maximize the overall performance. Towards this end, we developed and implemented semi-autonomous behaviors on a simulated mobile robot (using the ANVEL software), developed different control blending methods to combine the user’s commands with the autonomy, and performed user tests to assess the performance. The major research questions addressed include:

1. How should the user and autonomous control inputs be blended to give the best overall “shared control” performance of the teleoperated mobile robot?
2. How do factors unique to teleoperation (such as latency and highly unstructured environments) affect the optimal blending method?
3. How should the system performance be defined for teleoperation in an unstructured environment? What is the objective that should be optimized?

This project leveraged related research in the ARC.

Deliverables:

• Human subject study (36 users) and simulations evaluating teleoperation performance with constant and time-varying latency. Conference publication [4].
• Development of dynamic weight shifting method for rollover prevention and collection of experimental data. Journal publication [5].
• Human subject study (20 users) investigating the impact of semi-autonomous control and its interaction with communication latency. Conference publication [6]. Journal publication [8].
• Human subject study (20 users) investigating the impact of environment configuration on UGV navigation performance. Conference publication [9].
• Human subject study (20 users) investigating the impact of communication delay and environment configuration on UGV navigation performance. Journal paper under review [10].
• Software and algorithms developed in ANVEL transferred to GVSC

Publications:

1. Storms, S. Vozar, and D. Tilbury, “Predicting Human Performance During Teleoperation,” ACM/IEEE International Conference on Human-Robot Interaction, pp. 298-299, 2014.
2. J. Storms and D. Tilbury, “Blending of Human and Obstacle Avoidance Control for a High Speed Mobile Robot,” American Control Conference, 2014.
3. J. Storms, and D. Tilbury, “Reducing Rollover Risk of a High Speed Mobile Manipulator,” in ASME Dynamic Systems and Controls Conference, 2014.
4. J. Storms, and D. Tilbury, “Equating User Performance Among Communication Latency Distributions and Simulation Fidelities,” IEEE International Conference on Robotics and Automation, 2015.
5. J. Storms, and D. Tilbury. “Dynamic Weight-Shifting for Improved Rollover Prevention and Handling Dynamics in High Speed Mobile Manipulators,” ASME Journal of Dynamic Systems, Measurement, and Control, 2016.
6. J. Storms, K. Chen, and D. Tilbury, “A Semi-Autonomous Control Method to Improve Performance of Small Unmanned Ground Vehicles with Communication Latency,” ASME International Design Engineering Technical Conferences, 2016.
7. S. Vozar, J. Storms, and D. Tilbury, “Development and Analysis of an Operator Steering Model for Teleoperated Mobile Robots Under Constant and Variable Latencies,” Robotica, 2016.
8. J. Storms, K. Chen, and D. Tilbury, “A Shared Control Method for Obstacle Avoidance with Mobile Robots and its Interaction with Communication Delay,” International Journal of Robotics Research, 2016.
9. J. Storms, K. Chen, D. Tilbury, “Modeling Teleoperated Robot Performance as a Function of Environment Difficulty,” IFAC Conference on Cyber Physical and Human Systems, 2016.
10. J. Storms, D. Tilbury, “A New Difficulty Index for Teleoperated Robots Driving through Obstacles: How Environment Difficulty, Communication Delay and Autonomy Level Affect Performance,” Journal of Intelligent and Robotic Systems, Accepted June 2, 2017.
11. J. Storms, “Modeling and Improving Teleoperation Performance of Semi-Autonomous Wheeled Robots,” PhD Dissertation, University of Michigan, 2016.

#2.7