The core innovation of this one-year initiative centers on a Generative Skill Training paradigm that utilizes internet-scale generative video tools (such as OpenAI's Sora and Google DeepMind's Veo) to synthesize rich visual data from basic human prompts. These generated interactions are subsequently reconstructed into 3D trajectories and simulated in IsaacGym to evaluate physical feasibility, safety, and failure recovery variables. Through collaboration with industrial partner Raise Robotics, Inc., the project intends to distill these simulated experiences into diffusion-based visuomotor policies, successfully elevating the underlying automated stack from lab-scale to testbed-scale. Ultimately, this effort advances MFI’s overarching framework for Human-Centric Agile Manufacturing by establishing a shareable "Skill Library" that enables industrial robots to effectively self-program.
Application/demonstration
The project will validate the end-to-end "Video-to-Policy" pipeline by successfully executing high-precision assembly and fastening workflows over three physical trials using multi-fingered dexterous hardware (e.g., Inspire or LEAP hands) in a Mill 19 Robotics Workcell.