Project Real-Time Avatars for 3D Telepresence in Unity

Prof. Dr. Bernd Fröhlich
Dr. rer. nat. Stephan Beck
M.Sc. Adrian Kreskowski
M.Sc. Gareth Rendle

3D telepresence enables participants at different locations to meet and collaborate in a shared virtual environment using realistic representations of themselves. We presented the first immersive group-to-group telepresence system in 2013. Since then, many fundamental technologies have been continuously improved; in particular, real-time 3D capturing and reconstruction of user representations. Such avatar representations are referred to as 3D video avatars or volumetric avatars in the literature.

Our server infrastructure for creating such volumetric avatars from colour and depth image streams implements state-of-the art 3D reconstruction techniques and supports the latest generation of Kinect colour and depth sensors for 3D capturing. In early 2020 we decided to use Unity as our virtual reality (VR) framework, to move closer to widespread adoption of our vision of social VR and 3D telepresence. In order to create VR applications in Unity, the capabability to stream compact avatar representations from remote locations to Unity clients for rendering is, of course, fundamental.

In this project, we will focus on the design and implementation of a C++ plugin for Unity that receives volumetric avatar streams from our existing capture server and subsequently renders avatars‘ geometry and texture natively in Vulkan. Since the transfer of such rich 3D data over the internet still remains a bottleneck, we will also investigate how to efficiently compress the avatar streams using existing libraries and different data reduction approaches, such as level-of-detail reconstruction, as well as geometry and image compression.

At the beginning of this project, we will study related research and learn to use our existing infrastructure and technologies. On this basis, we will define our requirements, before designing and structuring our tasks to achieve our goal with an efficient implementation.

In case students are not equipped with recent hardware, we are able to provide workstations, head-mounted displays (HMDs), and pre-recorded Kinect streams for students to work with during the project. Meetings, presentations, and general communication will be held online.

You will learn about and work on the following topics in your project team:

• Real-time rendering of avatar streams using state-of-the-art features of established graphics APIs with a focus on Vulkan & shader languages (Vulkan GLSL / GLSL)
• Native Render-Plugin Development in Unity
• Distributed multi-user virtual reality applications
• Design and implementation of a low-level rendering plugin for Unity clients that receives avatar streams over the internet from remote 3D reconstruction servers
• Design and implementation of one-to-many communication channels based on an existing network library (ZMQ)
• Compression techniques for geometry and texture streams, e.g. dimensionality reduction through principal component analysis
• Last, but certainly not least: Being a reliable team member in a complex software project

• Solid C++ skills, both conceptual and practical
• Basic analysis and linear algebra skills
• Previous experiences in computer graphics helpful
  • e.g. having worked with Vulkan, OpenGL or DirectX

If you are in doubt as to whether you fulfil the requirements, or if you have any further questions regarding the project, we are happy to have a discussion with you during the project fair on 6th of April. You can find us in our BigBlueButton room at the fair for the entire time between 5pm and 7pm.

The final assessment of your work will be conducted based on the project contributions of every team member, including:

• Active participation in the project during and in between weekly meetings
• Design, implementation and evaluation of a C++ Unity plugin for rendering avatars streamed from 3D reconstruction servers
• Intermediate talks
• Intermediate and final project presentations
• Documentation in form of a short paper

If you are excited about graphics programming, avatars and virtual reality, do not hesitate to come to our room during the project fair!