Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality


Hannah Schieber 1,3, Jacob Young 2, Tobias Langlotz 2, Stefanie Zollmann 2, and Daniel Roth3

Friedrich-Alexander University (FAU) Erlangen-Nurnberg, Erlangen, Germany 1
Department of Computer Science, University of Otago, Dunedin, New Zealand 2
Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, Orthopaedics and Sports Orthopaedics, Munich, Germany 3

arixv

Code (coming soon)

Dataset


Abstract

Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.


Architecture



Evaluation


Novel View Synthesis

Outback Scene
Gaussian Grouping Before
Ours After
Outback Scene
Gaussian Grouping
(improved labels) Before
Ours After

Lake Scene
Gaussian Grouping Before
Ours After
Open Sea Scene
Gaussian Grouping Before
Ours After

Object removal examples on the Short Ride scene

Gaussian Grouping
(improved labels)
(convex hull) Before
Ours After
Gaussian Grouping
(improved labels)
(direct removal) Before
Ours After

Object Removal on other scenes

Street Scene shot with Insta360 X1
Full (Ours) Before
Removed (Ours) After
Kayak Scene shot with Insta360 X3
Full (Ours) Before
Removed (Ours) After





SCGS in Virtual Reality


SCGS allows the removal of large scene parts and the replacment of it with virtual 3D assets.




Citation

@misc{schieber2024semanticscontrolledgaussiansplattingoutdoor,
   title={Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality},
   author={Hannah Schieber and Jacob Young and Tobias Langlotz and Stefanie Zollmann and Daniel Roth},
   year={2024},
  eprint={2409.15959},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2409.15959},
}