A demonstrator that indicates the future of manufacturing in human-to-human collaboration combining artificial intelligence, virtual reality and a remote-controlled robotic arm. This was one of the technological highlights of the tenth anniversary of the Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague (CIIRC CTU), the host institution and coordinator of the RICAIP centre. On 30 June 2023, the Institute celebrated ten years of activity since its establishment in 2013. The RICAIP Centre enriched part of the celebration programme with a presentation of distributed modular manufacturing and a live demonstrator of the remote control system connecting testbeds in Prague, Saarbrücken, and Brno.
This RICAIP system demonstrates a multi-site and distributed manufacturing scenario in which a robotic arm is intuitively controlled remotely in real-time using VR glasses. Is is an example of cross-site human-human collaboration using a robot (in this particular case the UR-10 robot) as embodiment for the remote worker. It can help, for example, when performing tasks in hazardous environments or when a non-professional worker needs the assistance of an expert at a remote site.
Two RICAIP workplaces were remotely connected and thus through a microphone and virtual reality goggles, a worker in Testbed for Industry 4.0 in Prague communicated with a team of DFKI and ZeMA in Saarbrücken, Germany, where a workplace with a collaborative robot and a set of various assembly components was prepared.
Watch the recording from the 10th Anniversary CIIRC CTU to see how it works:
The robot in Saarbrücken was controlled by a person in Prague via a virtual interface that could see what was going on at the remote workplace through the hologlasses into which specific robot behaviors were projected. Visitors of the event could also see the same, with the view from the goggles and the German workplace projected side by side on a large screen in the Testbed Control Room. It is in virtual reality that the two places meet – the worker’s hand – which is only shown up to the wrist – manipulating objects in the virtual environment with a virtual robotic hand. The real robotic hand then does in real life in Saarbrücken exactly what it does in virtual reality operated by the researcher in Prague.
“Such applications can be used in environments that are otherwise dangerous for humans – for example, due to high temperatures, contamination with dangerous gases or even radiation. The robot will then do the precise manipulation for humans in an environment that is otherwise unsuitable or inaccessible for humans.”
Pavel Burget, Director, Testbed for Industry 4.0, CIIRC CTU in Prague
And what is behind this scenario? It is based on the work of the DFKI and ZeMA publication on Digital Twins for Distributed Collaborative Work in Shared Production. To provide the VR-Operator with the necessary information about the remote site, a digital twin is enriched through real-time data about the environment as provided by two RGBD camera modules. One camera is responsible for recognizing and locating the different assembly-parts and is transmitting their position over shared real-time data streams via ROS, which can then be consumed and displayed within VR. The other camera module sends a 3D point cloud to the VR-glasses, to provide the VR-operator with a better understanding of the remote environment, e.g. unknown objects, obstacles and the actions of the co-worker.
Directly from Saarbrücken, the co-authors, Caspar Jacob, Tim Schwartz, Andreas Luxenburger (all DFKI) and Khansa Rekik (ZeMA) also assisted Tomáš Jochman (CIIRC CTU) and Tilman Becker (RICAIP) in Prague during the demontrators. The scenario covers a distributed, collaborative assembly and in further development, a user-study is planned, to get a better insight into the planning space of case assembly task as well as the desired human-robot interaction modalities and verbal/nonverbal communication acts.
After the demonstrator, the live connection to Brno followed with a real-time presentation of equipment and specific technologies available in the Industry 4.0 testbed operated by CEITEC BUT.
During the ceremonial morning, the leading representatives of Czech science, industry and state administration saw also a new video presentation of the RICAIP distributed modular production concepts for advanced manufacturing, in which various technologies and remote workplaces are dynamically integrated into the production process using multi-agent approach. The aim is to be able to quickly adapt production to sudden changes and automatically ensure smooth operation in the event of sudden interruptions in supply chains or when product parameters change based on customer requirements.
Photos: Jiří Ryszawy (CTU), Roman Sejkot (CIIRC CTU)