BAUHAUS.INSIGHTS: Metaverse Made in Weimar
Imagine coming to Weimar, touring the Gropius Room and being able to invite your relatives in Hamburg to join you virtually. Imagine them experiencing the room not just on a video call, but as 3D avatars. You can talk to them live and – thanks to AI – even talk to the objects in the room.
It might sound like science fiction, but it’s possible and so far only in Weimar. The »MetaReal« project takes AI-supported knowledge transfer to an entirely new level. It combines virtual reality (VR) and augmented reality (AR) with embodied AI language agents to give physical and virtual visitors to the Gropius Room a shared interactive experience.
Like a walk-in Wikipedia, visitors can explore Walter Gropius' historic office at the Bauhaus-Universität Weimar physically and virtually at the same time. A VR tour guide, whose appearance is inspired by the work of Oskar Schlemmer, takes visitors through a realistic 3D digital replica of the office. The tour guide explains the history of the Gropius Room and answers questions. At the same time, visitors are able to interact with iconic objects in the room, such as the Wagenfeld lamp or the F51 armchair. The objects themselves come to life and respond with their own voices, reflecting their individual personalities. The tour guide and the objects in the room are embodied with AI voice agents that have been programmed with specific information about the Gropius Room.
Integrated AR technology allows physical visitors to the Gropius Room to use their smart phones to speak with the tour guide and interactive objects. They can also interact with virtual visitors experiencing the Gropius Room at the same time in VR. Virtual visitors appear in the room as avatars viewed through the smartphone cameras.
MetaReal blurs the lines between virtual and physical spaces and replaces traditional museum knowledge transfer with a shared, interactive experience.
Prof. Dr. Bernd Fröhlich from the Professorship of Virtual Reality and Visualisation is the spokesperson for the interdisciplinary team behind MetaReal. We spoke with him about the project.
Prof. Fröhlich, we have known about 360° tours on museum websites for some time, since at least the coronavirus pandemic. What makes a digital copy like MetaReal special?
Digital copies are true-to-life digital 3D reconstructions of actual places. Unlike traditional 360° tours, which are made up of panoramic images, digital copies allow authentic spatial interaction through virtual reality; you can walk through these spaces at any time, from anywhere, and with multiple other life-size visitors.
room360
Click the Play button to load and view external content from Vimeo.com.
Automatically load and view external content from Vimeo.com (You can change this setting at any time via our »Data protection policy«.)
Furthermore, areas of museums that are otherwise difficult or impossible to access, like the Gropius Room at the Bauhaus-Universität Weimar, are also made accessible. These virtual spaces permit visitors with limited mobility to share the museum experience with other visitors, either physically or through augmented reality.
In addition to free accessibility, the digital copy also allows visitors to interact with objects in the room. Our innovative animism approach brings the objects in the virtual space to life when they are touched. Visitors can enter into a dialogue with the objects and not only gain traditional knowledge, but also – if I may say so myself – learn about the objects’ personalities.
It's exactly through this connection between AR users, VR users, and intelligent language agents that MetaReal has raised the bar. The world has never seen a system like this that allows multiple AR and VR users to explore a real place and its digital copy simultaneously while also enabling natural communication between these groups and with the objects in the space. MetaReal has achieved a completely new experience in the field of interactive knowledge transfer.
MetaReal – Guide
Click the Play button to load and view external content from Vimeo.com.
Automatically load and view external content from Vimeo.com (You can change this setting at any time via our »Data protection policy«.)
How and where can visitors experience the MetaReal version of the Gropius Room? Will MetaReal be implemented in other museums?
Currently, the MetaReal version of the Gropius Room can only be experienced in our VR Lab in Weimar. The prototype there allows a number of people to get together in social VR at the same time. It is, of course, also on exhibit at events like summaery.
The prototype is continuously being developed as part of the project. In addition to the Gropius Room, the Goethe House in Weimar can now also be explored virtually. The Haus am Horn is next on the list.
MetaReal is a collaborative project between several partners. The Professorship of Virtual Reality and Visualisation at the Bauhaus-Universität Weimar is researching spatial interaction between VR and AR users in mixed social virtual and physical environments. It is also leading the project and integrating contributions from all other partners. The Professorship of Computer Vision in Engineering is creating digital copies of the Gropius Room and Haus am Horn, among other sites, and is conducting research into automated object recognition in these models. The Chair of Intelligent Information Systems conducts research into natural language knowledge queries and the accuracy of answers, while the Chair of Media Theory and History of Science prepares and scientifically classifies knowledge relating to the Bauhaus. The team is complemented by the Department of Virtual Worlds and Digital Games at TU Ilmenau, which deals with the representation of AR users and the combination of AR and VR technologies.
How do visitors access information located in the virtual space? Are there differences between the AR and VR experiences?
Actually, the experiences are very similar thanks to MetaReal. Both groups experience the Gropius Room in three dimensions and on a full scale. This alone ensures that the context is identical for both groups. The second similarity – and major advantage in comparison to the classic Wikipedia, 360° tours, or audio guides – is the natural dialogue; an AI language agent, in the form of a tour guide, takes over the tour of the Gropius Room on its own initiative. But visitors can ask it questions using microphones, other objects, or one another. This allows them to guide the knowledge transfer based on their own interests.
The biggest difference is that physical visitors can access information placed in the room via AR through their smartphones. This means that they can also interact with voice agents and other virtual visitors, but they can’t see them in 3D.
How exactly do physical and virtual visitors see one another in the room? And how do they interact?
The virtual visitors see each other and the physical visitors as human-like avatars. The avatars accurately depict where the physical visitors are in the real Gropius Room. The physical visitors, in turn, see the virtual visitors superimposed on the real environment through their smartphones. The virtual visitors appear in the room in the positions they occupy in the digital copy. The allows for a seamless connection between real and virtual space and interaction between both worlds.
Visitors can talk with one another and to the tour guide and objects using a microphone and headphones or headset. Communicating through body language or gestures is also possible.
An example of how this works would be grandparents physically visiting the Gropius Room and their grandchildren joining them virtually. Or a group of friends from all over the world could meet virtually and on site at the museum. The physical visit becomes an occasion for shared experiences where knowledge is shared in an interactive way.
How do you prevent people from losing sight of the room when there is a high number of avatars and many people are visiting the room?
In VR there are options for dealing high visitor numbers and the associated challenges, such as obscuring the view. One option is to not make all the visitors visible, or to make avatars that are blocking important exhibits partially transparent. But this can take away from the shared experience, which is why we are especially careful with such inventions.
We have looked at a number of approaches with our prototype. For instance, allowing VR visitors to see through others with a kind of x-ray vision, or automatically moving aside people who are standing in the way. VR also has the potential to ensure that all visitors are always in the »front row« and given the best view of the room and objects at all times. However, this solution restricts the social interaction between visitors.
You have to be flexible in choosing the right options to make the experience enjoyable and functional, even with large numbers of visitors.
What potential uses do you imagine for MetaReal technology outside of museums?
MetaReal’s potential extends far beyond museums. Digital copies can be used at trade fairs, exhibitions, and other events to enable shared experiences for physical and virtual visitors. Individuals who are physically in different locations could spontaneously join virtually as a kind of »3D Instant Sharing«. This would offer an alternative to conventional sharing and could take exchanges to a new level.
As 3D models become increasingly available and easily created thanks to modern computer vision technology using smart phones, digital copies of more and more places are being created. We expect these to be the building blocks of the metaverse – a networked, 3D world sometimes referred to as the »3D Internet«. The metaverse would allow users to meet in any virtual world, regardless of location or time.
AI language agents will also be increasingly common in the future. Acquiring new skills or knowledge through videos is already commonplace. I’m certain that embodied language agents in virtual environments will make this autodidactic learning process even more effective.
Prof. Fröhlich, thank you for exciting interview.
The BAUHAUS.INSIGHTS questions on the »MetaReal« project were asked by freelance editor Franz Löbling. Photos and videos: Irene Lopez Garcia / Ephraim Schott
The MetaReal project was funded from 2020 to 2024 by the Thüringer Ministerium für Wirtschaft, Wissenschaft und Digitale Gesellschaft as part of the state’s ProDigital programme (5575/10-5).
Metareal – Demonstrationsvideo zum Gropiuszimmer
Click the Play button to load and view external content from Vimeo.com.
Automatically load and view external content from Vimeo.com (You can change this setting at any time via our »Data protection policy«.)