James Staley - Gains on all Trajectories Demonstration agnostic learning with model based RL

Robots deployed in real-world environments must be able to effectively learn from humans to perform novel tasks or align with human preferences. One way humans can teach robots is with Learning from Demonstration (LfD) methods where an agent is shown example solutions, typically high-quality trajectories from expert users or policies. Despite the near-proverbial dependence of the performance of these algorithms on the quality of their demonstrations (``garbage in, garbage out''), little work in LfD has compared different demonstration sources' impact on learning. In this work we show that LfD algorithms like Diffusion Policy (DP) and Implicit Behavior Cloning (IBC) are sensitive to the source of their demonstrations and differ in performance when trained on demonstrations that come from trained models rather than human beings. Understanding this difference is critical because robots need to be able to learn from less frequent, high-value demonstrations that come from copresent humans in addition to their own behavior. We demonstrate this effect in two simulated environments and one real-world robot task where we collect data from inexperienced participants in-the-wild. In addition, we show how a world model based reinforcement learning (MBRL) system, DreamerV3 from Demonstrations (DfD), can effectively learn regardless of the demonstration source and can serve as an effective baseline. Baselines that are agnostic to demonstration source offer a neutral comparison and can aid in the development of LfD systems. James Staley is a PhD candidate in Human-Robot Interaction at Tuft University's AABL lab. He has been working on robotics and interactive AI on and off for the last decade. He likes the outdoors, books and games. This session is brought to you by the Cohere For AI Open Science Community - a space where ML researchers, engineers, linguists, social scientists, and lifelong learners connect and collaborate with each other. We'd like to extend a special thank y