V2: DynaMoN: Motion-Aware Fast And Robust Camera Localization for Dynamic Neural Radiance Fields

V1: DynaMoN: Motion-Aware Fast And Robust Camera Localization for Dynamic NeRF
Nicolas Schischka* (1), Hannah Schieber* (2,4), Mert Asim Karaoglu* (1,3), Melih Gorgulu* (1),
Florian Grotzner (1), Alexander Ladikos (3), Daniel Roth (4), Nassir Navab (1,5) and Benjamin Busam (1)

* indicates equal contribution

Technical University of Munich,
Munich, Germany 1
Friedrich-Alexander Universitat Erlangen-Nürnberg,
Erlangen, Germany 2
ImFusion GmbH,
Munich, Germany 3
Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, Orthopaedics and Sports Orthopaedics, Munich, Germany 4
Johns Hopkins University, Baltimore, MD, USA 4

arixv

Code (coming soon)


COLMAP + HexPlane
DynaMoN
Compare



We present DynaMoN, a motion aware fast and robust camera localization approach for novel view synthesis. DynaMoN can handle not only the motion of known objects using semantic segmentation masks but also that of unknown objects using a motion segmentation mask. Furthermore, it retrieves the camera poses faster and more robust compared to classical SfM approaches enabling a more accurate 4D scene representation. Compared to the state-of-the-art, DynaMoN outperforms other dynamic camera localization approaches and shows better results for novel view synthesis.


Abstract


Dynamic reconstruction with neural radiance fields (NeRF) requires accurate camera poses. These are often hard to retrieve with existing structure-from-motion (SfM) pipelines as both camera and scene content can change. We propose DynaMoN that leverages simultaneous localization and mapping (SLAM) jointly with motion masking to handle dynamic scene content. Our robust SLAM-based tracking module significantly accelerates the training process of the dynamic NeRF while improving the quality of synthesized views at the same time. Extensive experimental validation on TUM RGB-D, BONN RGB-D Dynamic and the DyCheck’s iPhone dataset, three real-world datasets, shows the advantages of DynaMoN both for camera pose estimation and novel view synthesis.



Architecture



Results

Visual Improvements

DynaMoN enables better visual results in dynamic scenes.



Citation

@misc{schischka2024dynamon,
   title={DynaMoN: Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields},
   author={Nicolas Schischka and Hannah Schieber and Mert Asim Karaoglu and Melih Görgülü and Florian Grötzner and Alexander Ladikos and Daniel Roth and Nassir Navab and Benjamin Busam},
   year={2024},
   eprint={2309.08927},
   archivePrefix={arXiv},
   primaryClass={cs.CV}
}