The demonstrator shows multi-modal interaction with a 3D model of a production line (projected into the room by a Microsoft HoloLens). The modalities used include spoken language with natural dialogue, gaze, and gestures related to the projected 3D model.
The goal is to show remote access–based on an integrated semantic representation of all relevant data–to a production line. Possible applications range from design, manufacturing engineering, monitoring and control to maintenance use cases.
The demonstrator is built on a single common data representation in OSF (Open Semantic Framework) including 3D data and the dialogue system.
It shows how remote and distributed data can be brought to life in an interactive framework. The system has been built in cooperation with Siemens USA and is based on DFKI’s dialogue platform Siam-dp.
Currently, the system setup does not allow copying the image from the HoloLens screen to a regular monitor for a bigger audience. However, this video shows the audience what is going on.