Maintaining machines and executing state-of-the-art manufacturing processes require operators to follow a wide range of guidelines and instructions. Keeping track and executing such complex operations is highly challenging, and a major obstacle to advance manufacturing. We aim to overcome this challenge by creating an assistive guidance system for operators based on an end-to-end platform for the creation of and interaction with digital twins. The proposed pipeline consists of the following three innovations: 1) an approach for autonomous multi-robot exploration of manufacturing environments, and 2) an approach for creating digital twins and simulation environments directly from the multi-robot sensor data by extracting 3D representations and semantic information. This information is then processed by 3) an optimization-based Augmented Reality assistance system. The system presents operators with the digital twin as in-situ augmentation, which contains guides, requirements and next steps. The system will enable operators to perform complex tasks and keep track of multi-level manufacturing processes while increasing task performance, decreasing cognitive load and fostering situational awareness.
Pictured above: Infographic illustrating the workflow of the proposed research.
