Trust in autonomous systems to do what they are designed to do, and meant to do, is paramount before these systems can be confidently employed as an operational capability. In an era of Great Power Competition, where those who can field autonomous capabilities the fastest will have a distinct advantage, researchers at the Naval Postgraduate School (NPS) are exploring many of the fundamentals of autonomous systems, especially in the trust and confidence arena.  

On the eve of the COVID-19 pandemic, NPS researchers conducted a novel experiment with an autonomous robot in a simulated environment – with real Marines – seeking to understand just how confident the Marines were in the robot to complete its tasks in a combat environment.

The NPS research team, comprised of Assistant Professor Mollie McGuire, Research Associate Christian Fitzpatrick and NPS student Marine Corps Maj. Dan Yurkovich, traveled to Camp Lejeune, N.C., to research Yurkovich’s thesis, titled “Human-Machine Teaming,” using Camp Lajeune’s Combat Town Range and nearby Marines from the Advanced Infantry Training Battalion (AITB) as part of a testbed. This research was directly responsive to the Marine Corp’s 2018 Science and Technology Strategic Plan which has a key objective to pursue advanced robotic systems to support ground maneuvers.

The experiment put the robot in an urban environment in the Combat Town Range, calling for the robot to conduct room clearing, which is one of the most common of tasks in an urban warfare environment, but also one of the most dangerous.

For the experiment, the researchers employed the 310 Small Unmanned Ground Vehicle (SUGV), a man-portable robot intended for use by soldiers, combat engineers and mobile explosive ordnance disposal (EOD) technicians to gather data for situational awareness in critical conditions. The Marines interfaced with the robot through a virtual gaming environment, created by NPS’ Modeling Virtual Environments and Simulations Institute, where they “trained” the robot in its specific room clearing tasks and could then evaluate its effectiveness in accomplishing them from a “safe room” nearby.

“The experiment aimed to see if introducing and training with the robot in a virtual environment would aid in developing trust with the actual robot,” said McGuire, who served as thesis advisor. “The experiment explored ways to develop trust in robotic teammates in a more efficient manner by using virtual environments instead of having Marines spend hours training with the robot itself.”

McGuire noted that the objective was then to see if the Marines that “trained” the robot within the gaming environment had higher trust in the actual robot due to the potential forming of a “team dynamic” between man and machine – the robot learning human preferences, and the human learning robot capabilities.

“The young Marines in our experiment had a high level of technical competence especially in the understanding of machine learning,” said Fitzpatrick. “Some Marines [had questions about] our processes in the transfer of gaming data to the robot which indicated they were closely observing us as we were observing them.” 

According to Fitzpatrick, the team was able to collect good data across a range of measurements. 

“Gauging human-robot interaction with actual users was invaluable,” said Fitzpatrick. “Our conclusion was that if Marines are involved in the development and integration process from the start, they would trust the unmanned capability to a greater extent when using them during real-world operations.”

McGuire explained that if this trend continues, it might suggest a positive relationship between training time and trust development between humans and their robot teammates.

Yurkovich recently discovered that his original sponsor for this project, the Office of Naval Research, has incorporated some of his results into larger projects. He noted that other researchers are exploring the concept of rehearsals in virtual environment and how it transfers to live execution.

The experimentation appears to be promising as developers and researchers are evolving models to a next generation virtual environment game play and further resolving it to the real-world, he added.