Project Efficient Gaussian Splatting for Virtual Reality Applications

Prof. Dr. Bernd Fröhlich
M.Sc. Adrian Kreskowski
M.Sc. Gareth Rendle
M. Sc. Simon Schneegans (Guest Researcher)
M.Sc. Anton Lammert

Novel-view synthesis techniques based on  Neural Radiance Fields [Mildenhall et al. 2020], Plenoxels [Fridovich-Keil et al. 2022], or, most recently and possibly most-well known, 3D Gaussian Splatting [Kerbl et al. 2023, Liu et al. 2024] enable the visually high-fidelity reconstruction of surfaces which are hard or even near-impossible to reconstruct using classic photogrammetric approaches. Examples of such surfaces include fur, vegetation, transparent or translucent objects and thin structures in general. The novel-view synthesis approaches perform faithful interpolation of existing color information contained in a set of high-quality input images. Novel views can be rendered in real-time, provided one has access to powerful graphics hardware.

First research [Lin et a. 2024] has emerged which aims at reducing the rendering workload of weaker mobile devices using foveated rendering techniques. However, to enable the exploration of high-quality datasets in virtual reality applications, it is necessary to design rendering algorithms with e.g. output-sensitivity in mind.  In the first part of this project, we will explore existing rendering and acceleration techniques for novel-view synthesis by example of 3D Gaussian Splatting. After a detailed analysis of the rendering algorithms, we will design, implement and evaluate our own acceleration techniques for enabling real-time 3D Gaussian Splatting at high visual fidelity for state-of-the-art virtual reality devices.

In order to optimize performance for real-world datasets in virtual reality applications, we plan to explore an ocean floor dataset in virtual reality using head-mounted displays. The dataset will be captured and provided to us by the MARUM - Center for Marine Environmental Sciences at the beginning of our project.

• Solid programming skills in C++ are required
• Successfully completed Computer Graphics course or equivalent qualifications
• Experience with GPGPU programming or algorithm design is welcome.

If you are in doubt as to whether you fulfil the requirements, or if you have any further questions regarding the project, we are happy to have a discussion with you before or after the project fair on 14th of October, or after that via email to adrian.kreskowski[at]uni-weimar.de.

The final assessment of your work will be conducted based on the project contributions of every team member, including:

• Active participation in the project during and in between weekly meetings
• Design, implementation and evaluation of Virtual Reality-based Optimizations for efficient rendering of Gaussian Splatting within Unity or an own real-time rendering framework
• Intermediate talks
• Intermediate and final project presentations
• Documentation in form of a short paper

If you are experienced or interested in real-time computer graphics and virtual reality, we would be excited to welcome you in our project. We will provide you with a Quest 3 for the duration of the project and together we will get our feet wet with our challenging real-world dataset and Efficient Gaussian Splatting for Virtual Reality!