Project ReVRse: Semantic Editing Tools for Immersive Recordings

Prof. Dr. Bernd Fröhlich
M.Sc. Anton Lammert
M.Sc. Tony Zöppig
M.Sc. Karoline Brehm

Immersive recordings capture events and interactions in virtual environments and allow users to explore them both temporally and spatially in virtual reality. Unlike 2D video and even 360 degree video, immersive recordings capture spatiotemporal data of a 3D virtual environment. Such recordings can be viewed from multiple perspectives and even interacted with during playback in VR. This unique capability makes immersive recordings a powerful tool, e.g. for psychotherapy. Recordings of reenactments of meaningful social interactions in VR could be collaboratively reviewed, analysed and modified according to therapeutic goals.

In this project, you will extend an existing recording system to capture semantic data, such as actions like 'picking up and moving a bottle'. This will involve designing a suitable data structure to capture and store semantic data. By incorporating this data, the system can be programmed to maintain logical consistency when changes are made to the recordings. For example, if an action such as grasping the bottle is removed, the playback will automatically be updated to keep the bottle stationary.

You will also investigate interaction techniques for allowing users to modify recorded interactions in virtual reality. This includes experimenting with playful approaches to engage users, such as developing a miniature 'dollhouse' version of the immersive environment. In this miniature, users can manipulate recorded actions by interacting with action figures or dolls that represent recorded characters.

During the project, you will learn to design virtual reality applications and implement these applications using Unity3D and C#. For participants with experience in C++ programming, the extension of the underlying C++ plug-ins for Unity3D is also a potential task.

The final assessment of your work will be conducted based on the project contributions of every team member, including:

• Quality and innovation of your practical results
• Active participation in the project during and in between weekly meetings
• Intermediate and final project presentations
• Documentation