NeRFtrinsic Four: An End-To-End Trainable NeRF Jointly Optimizing Diverse Intrinsic and Extrinsic Camera Parameters

Hannah Schieber ^1,4, Fabian Deuser ², Bernhard Egger ³, Norbert Oswald ², and Daniel Roth⁴

Human-Centered Computing and Extended Reality, Friedrich-Alexander University (FAU) Erlangen-Nurnberg, Erlangen, Germany ¹

Institute for Distributed Intelligent Systems University of the Bundeswehr Munich Munich, Germany ²

Lehrstuhl fur Graphische Datenverarbeitung (LGDV) Friedrich-Alexander Universität (FAU) Erlangen-Nürnberg Erlangen, Germany ³

Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, Orthopaedics and Sports Orthopaedics, Munich, Germany ⁴

Paper (CVIU) open access

Preprint (arixv)

Code

Dataset

Abstract

Novel view synthesis using neural radiance fields (NeRF) is the state-of-the-art technique for generating high-quality images from novel viewpoints. Existing methods require a priori knowledge about extrinsic and intrinsic camera parameters. This limits their applicability to synthetic scenes, or real-world scenarios with the necessity of a preprocessing step. Current research on the joint optimization of camera parameters and NeRF focuses on refining noisy extrinsic camera parameters and often relies on the preprocessing of intrinsic camera parameters. Further approaches are limited to cover only one single camera intrinsic. To address these limitations, we propose a novel end-to-end trainable approach called NeRFtrinsic Four. We utilize Gaussian Fourier features to estimate extrinsic camera parameters and dynamically predict varying intrinsic camera parameters through the supervision of the projection error. Our approach outperforms existing joint optimization methods on LLFF and BLEFF. In addition to these existing datasets, we introduce a new dataset called iFF with varying intrinsic camera parameters. NeRFtrinsic Four is a step forward in joint optimization NeRF-based view synthesis and enables more realistic and flexible rendering in real-world scenarios with varying camera parameters.

Example on the iFF Dataset

Having the correct camera parameters is essential for NeRF. Our end-to-end optimzable approach learns the camera parameters and can optimize them. In the image of T1 we compare our appraoch with NerF--.

T1 one scene from iFF

Ours

Brick House from iFF

Ours

Room from LLFF

Ours

Architecture

Results for Novel View Synthesis

LLFF

PSNR

SSIM

We report the results of PSNR and SSIM on LLFF, highlighting the best approach using a ★.

BLEFF

PSNR

SSIM

We report the results of PSNR and SSIM on BLEFF, highlighting the best approach using a ★.

iFF

PSNR

SSIM

We report the results of PSNR and SSIM on BLEFF, highlighting the best approach using a ★.

Results for Camera Pose Estimation

LLFF

Rotation Error

Translation Error

Focal Error

We report the results of Translation,Rotation and Focal Length Error on LLFF, highlighting the best approach using a ★.

iFF

Rotation Error

Translation Error

Focal Error

We report the results of Translation,Rotation and Focal Length Error on iFF, highlighting the best approach using a ★.

iFF Dataset

Citation

@article{schieber2024nerftrinsic,
  title={Nerftrinsic four: An end-to-end trainable nerf jointly optimizing diverse intrinsic and extrinsic camera parameters},
  author={Schieber, Hannah and Deuser, Fabian and Egger, Bernhard and Oswald, Norbert and Roth, Daniel},
  journal={Computer Vision and Image Understanding},
  pages={104206},
  year={2024},
  publisher={Elsevier}
}