Virtual reality partners become real world neighbors

by Lynn Shea

Global architectural products company opens YKK AP Technologies Lab at Mill 19 where they will continue to collaborate with Kenji Shimada to conduct digital twin research and develop a virtual factory for the production and installation of windows and doors.

Activity tracking for worker safety

Researchers showed how the deep learning detection system they developed can be used to track worker activity and issue safety violation alerts in real time. Alarms indicated when workers failed to follow safety guidelines, such as wearing helmets, using ladders properly, or correctly moving or installing windows.

Four men walking into the High Bay at Mill 19 — YKK AP executives arrive at Mill 19 for a technology demonstration by Kenji Shimada’s research team.

A man speaking into a microphone with a colorful display behind him — Mechanical Engineering Professor Kenji Shimada’s research team showcased how their machine learning and advanced robotics research is innovating the field of construction.

A man holds a drone while speaking to a crowd — Mechanical Engineering graduate student, Harvey Mei, shows how a UAV (unmanned aerial vehicle) is used to inspect the installation of windows and doors.

Two men in hardhats on a staged construction site — Ph.D. student researchers demonstrate how the deep learning detection system they developed is used to issue safety violation alerts.

Two men taking exaggerated steps — YKK executives help CMU researchers demonstrate how their detection system can be used to track worker activity and movement.

Robotic inspection for quality assurance

A second group of researchers demonstrated how robots could be used to inspect installed windows using computer vision to detect defects and collect image data that could be used to drive future quality improvements.

UAV and UGV for site inspection

A UAV (unmanned aerial vehicle) and a UGV (unmanned ground vehicle) were used to detect and avoid both static and dynamic obstacles while they explored, mapped, and inspected a mockup of a construction site environment.

“More than 50 students have been involved with the YKK AP projects since the partnership was established four years ago,” said Shimada. “We are grateful to be partnering with YKK AP on research that promises to innovate industry with technology that can generate cost savings and improve quality, efficiency, and worker safety.”

We are grateful to be partnering with YKK AP on research that promises to innovate industry with technology that can generate cost savings and improve quality, efficiency, and worker safety.
Kenji Shimada, Professor, Mechanical Engineering

“Our goal as an organization is to improve construction productivity and quality through the research and development of virtual reality, augmented reality, and mixed reality technologies,” said Shiori Fukada, chief information and digital officer, YKK AP Inc. “The advent of new technologies presents an opportunity for true innovation and the ability to fill crucial gaps presented by the ongoing skilled labor shortage.”

Fukada explained that after considering other top U.S. engineering programs, they were very pleased to find that Shimada was conducting robotics, engineering, and information technology research at Carnegie Mellon that was an ideal fit for their plans to automate the installation of core components of a building like windows and doors.

YKK AP’s decision to locate their new technology lab at Mill 19 is beneficial to both Shimada and CMU’s Manufacturing Futures Institute (MFI), which is located on the first two floors of Building A at Mill 19, where Shimada conducts much of his research.

YKK AP’s focus on the development of a “virtual factory” is well aligned with the digital twin work underway at the Manufacturing Futures Institute.
Gary Fedder, Faculty Director, Manufacturing Futures Institute

“YKK AP’s focus on the development of a “virtual factory” is well aligned with the digital twin work underway at the Manufacturing Futures Institute,” said Gary Fedder, MFI faculty director. MFI is currently building a robotics assembly/disassembly test bed at Mill 19 that will be used to demonstrate some of the most compelling use cases for digital twins in manufacturing. The institute is currently seed funding several projects to help grow and connect digital twin research across multiple departments within the university.

YKK AP Technologies Lab is a subsidiary of Japan-based YKK AP Inc., which manufactures entrances, store fronts, curtain wall, window wall, sunshades, windows and sliding doors for office buildings, residential high-rises, schools, stadiums, shopping centers and institutional structures across the globe.

YKK AP America Inc., based in Atlanta, Georgia, manufactures entrances, store fronts, curtain wall, window wall, sunshades, windows and sliding doors for office buildings, residential high-rises, schools, stadiums, shopping centers and institutional structures in the United States.

Pictured, top (from left to right): Hidemitsu Hori, chairperson of YKK AP Inc., Shiori Fukada, YKK AP Inc.’s chief information and digital officer, and Oliver Stepe, president and CEO of YKK AP America