We consider the problem of 3D scene reconstruction and novel view synthesis from very sparse (i.e., as few as two) images in just one forward pass of atrained model. While remarkable progress has been made using neural scene rep_x0002_resentations, e.g., Scene Representation Networks (SRN) [34], Neural Radiance Fields (NeRF) [24] and Light Filed Networks (LFN) [33], these methods are still not satisfactory for practical applications due to expensive per-scene optimiza_x0002_tion [26, 39, 43], high memory cost [3, 18, 44] and slow rendering speed [40, 49]. Recently, 3D Gaussian Splatting (3DGS) [19] has emerged as an efficient and expressive 3D representation thanks to its fast rendering speed and high quality. Using rasterization-based rendering, 3DGS inherently avoids the expensive vol_x0002_umetric sampling process of NeRF, leading to highly efficient and high-quality 3D reconstruction and novel view synthesis. Very recently, several feed-forward Gaussian Splatting methods have been proposed to explore 3D reconstruction from sparse view images, notably Splat_x0002_ter Image [37] and pixelSplat [1]. Splatter Image regresses pixel-aligned Gaus_x0002_sian parameters using a standard image-to-image architecture, which achieves promising results for single-view object-level 3D reconstruction. However, recon_x0002_structing a 3D scene from a single image is inherently ill-posed and ambiguous, posing a significant challenge when applied to a more general and larger scene, which is the key focus of our paper. pixelSplat [1] proposes to regress Gaussian parameters for the binocular reconstruction problem. Specifically, it predicts a probabilistic depth distribution for each input view and then samples depths from that predicted distribution. Even though pixelSplat learns cross-view-aware features with an epipolar Transformer, it is still challenging to predict a reliable probabilistic depth distribution solely from image features, making pixelSplat’s geometry reconstruction of comparably low quality and exhibiting noisy arti_x0002_facts (see <PIC>). For improved geometry reconstruction results, slow depth finetuning with an additional depth regularization loss is required. To accurately localize the 3D Gaussian centers, our solution is to build a cost volume representation via plane sweeping [7,46,48] in the 3D space. Specifi_x0002_cally, the cost volume stores cross-view feature similarities for all potential depth candidates, where the similarities can provide valuable geometry cues to the lo_x0002_calization of 3D surfaces (i.e., high similarity more likely indicates a surface point). With our cost volume representation, the task is formulated as learn_x0002_ing to perform feature matching to identify the Gaussian centers, unlike the data-driven 3D regression from image features in previous works [1, 37]. Such a formulation reduces the task’s learning difficulty, enabling our method to achieve state-of-the-art performance with lightweight model size and fast speed. We obtain 3D Gaussian centers by unprojecting the multi-view-consistent depths estimated by our constructed multi-view cost volumes with a 2D network. Additionally, we also predict other Gaussian properties (covariance, opacity, and spherical harmonics coefficients), in parallel with the depths. This enables the rendering of novel view images using the predicted 3D Gaussians with the differ_x0002_entiable splatting operation [19].
Our full model MVSplat is trained end-to-end purely with the photometric loss between rendered and ground truth images. On the large-scale RealEstate10K [54] and ACID [21] benchmarks, MVS_x0002_plat achieves state-of-the-art performance with the fastest feed-forward infer-Sparse View Scene Reconstruction and Synthesis. The original NeRF and 3DGS methods are both designed for very dense views (e.g., 100) as inputs, which can be tedious to capture for real-world applications. Recently, there have been growing interests [1,3,5,12,26,36,39,43,44,51] in scene reconstruction and synthesis from sparse input views (e.g., 2 or 3). Existing sparse view methods can be broadly classified into two categories: per-scene optimization and cross_x0002_scene feed-forward inference methods. Per-scene approaches mainly focus on designing effective regularization terms [8, 11, 26, 39, 43, 50] to better constrain the optimization process. However, they are inherently slow at inference time due to the expensive per-scene gradient back-propagation process.
In contrast, feed-forward models [1,3,5,12,36,37,44,49,51] learn powerful priors from large_x0002_scale datasets, so that 3D reconstruction and view synthesis can be achieved via a single feed-forward inference by taking sparse views as inputs, which makes them significantly faster than those per-scene optimization methods. Feed-Forward NeRF. Early approaches used NeRF [24] for objects [6, 16, 18, 22, 27, 40, 49] and scenes [3, 5, 6, 10, 44] 3D reconstrution. pixelNeRF [49] pio_x0002_neered the paradigm of predicting pixel-aligned features from images for radi_x0002_ance field reconstruction. The performance of feed-forward NeRF models pro_x0002_gressively improved with the use of feature matching information [3, 5], Trans_x0002_formers [10, 25, 30] and 3D volume representations [3, 44]. The state-of-the-art feed-forward NeRF model MuRF [44] is based on a target view frustum vol_x0002_ume and a (2+1)D CNN for radiance field reconstruction, where the 3D volume and CNN need to be constructed and inferred for every target view. This makes MuRF expensive to train, with comparably slow rendering. Most importantly, all existing feed-forward NeRF models suffer from the expensive per-pixel volume sampling in the rendering process. Feed-Forward 3DGS. 3D Gaussian Splatting [4, 19] avoids NeRF’s expensive volume sampling via a rasterization-based splatting approach, where novel views can be rendered very efficiently from a set of 3D Gaussian primitives. Very re_x0002_cently, a growing number of feed-forward 3DGS models [1, 2, 36, 37, 42, 51, 53] have been proposed to solve the sparse-view-to-3D task. Among them, Splatter Image [37] proposes to regress pixel-aligned Gaussian parameters from a single view with a U-Net model. However, it mainly focuses on object-level reconstruc_x0002_tion, while we target the more general scene-level reconstruction. Although its follow-up work Flash3D [36] manages to extend to scene-level reconstruction, its performance on complex scenes is inherently non-satisfactory due to the limitedTo enable high-quality rendering and reconstruction, it is crucial to predict the position µj precisely since it defines the center of the 3D Gaussian [19].
 In this paper, we present MVSplat, a Gaussian-based feed-forward model for novel view synthesis. Unlike pixelSplat [1] that predicts probabilistic depth, we develop an efficient and high-performance multi-view depth estimation model that enables unprojecting predicted depth maps as the Gaussian centers, in parallel with another branch for prediction of other Gaussian parameters (αj , Σj and cj ). Our full model, illustrated in <PIC>, is trained end-to-end using only a simple rendering loss for supervision. Next, we discuss the key components.The qualitative comparisons of the top three best models are visualized in <PIC>. MVSplat achieves the highest quality on novel view results even un_x0002_der challenging conditions, such as these regions with repeated patterns (“win_x0002_dow frames” in 1st row), or these present in only one of the input views (“stair handrail” and “lampshade” in 2nd and 3rd rows), or when depicting large-scale outdoor objects captured from distant viewpoints (“bridge” in 4th row). The baseline methods exhibit obvious artifacts for these regions, while MVSplat shows no such artifacts due to our cost volume-based geometry representation. More evidence and detailed analysis regarding how MVSplat effectively infers the geometry structures are presented in Sec. 4.3.Assessing model efficiency. As reported in Tab. 1, apart from attaining su_x0002_perior image quality, MVSplat also shows the fastest inference time among all the compared models, accompanied by a lightweight model size, demonstrat_x0002_ing its efficiency and practical utility. It is noteworthy that the reported time encompasses both image encoding and rendering stages. For an in-depth time comparison with pixelSplat [1], our encoder runs at 0.043s, which is more than 2× faster than pixelSplat (0.102s). Besides, pixelSplat predicts 3 Gaussians per_x0002_pixel, while our MVSplat predicts 1 single Gaussian, which also contributes to our faster rendering speed (0.0015s vs. 0.0025s) due to the threefold reduction in the number of Gaussians. More importantly, equipped with the cost volume_x0002_based encoder, our MVSplat enables fast feed-forward inference of 3D Gaussians with a much light-weight design, resulting in 10× fewer parameters and more than 2× faster speed compared to pixelSplat [1]. Assessing geometry reconstruction. MVSplat also produces significantly higher-quality 3D Gaussian primitives compared to the latest state-of-the-art pixelSplat [1]. pixelSplat requires an extra 50,000 steps to fine-tune the Gaussians with an additional depth regularization to achieve reasonable geometry reconstruction results. Our MVSplat instead gen_x0002_erates high-quality geometries by training solely with photometric supervision. In contrast, our MVSplat reconstructs much higher-quality 3D Gaussians, demonstrating the effectiveness of our cost volume representation


Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments,
NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.Navigation is a fundamental skill for any organism with vision, playing a crucial role in survival by allowing agents to locate food, shelter, and avoid predators. In order to successfully navigate environments, smart agents primarily rely on vision, allowing them to construct representations of their surroundings to assess distances and capture landmarks in the environment, all useful for planning a navigation route. When human agents plan, they often imagine their future trajectories considering constraints and counter factuals. On the other hand, current state-of-the-art robotics navigation policies (Sridhar et al., 2024; Shahet al., 2023) are “hard-coded”, and after training, new constraints cannot be easily introduced (e.g, “no leftturns”). Another limitation of current supervised visual navigation models is that they cannot dynamicallyallocate more computational resources to address hard problems. We aim to design a new model that canmitigate these issues.In this work, we propose a Navigation World Model (NWM), trained to predict the future representation of avideo frame based on past frame representation(s) and action(s) (see <PIC>(a)). NWM is trained on videofootage and navigation actions collected from various robotic agents. After training, NWM is used to plannovel navigation trajectories by simulating potential navigation plans and verifying if they reach a target goal(see <PIC>(b)). To evaluate its navigation skills, we test NWM in known environments, assessing its abilityto plan novel trajectories either independently or by ranking an external navigation policy. In the planningsetup, we use NWM in a Model Predictive Control (MPC) framework, optimizing the action sequence thatenables NWM to reach a target goal. In the ranking setup, we assume access to an existing navigation policy,such as NoMaD (Sridhar et al., 2024), which allows us to sample trajectories, simulate them using NWM, andselect the best ones. Our NWM achieves competitive standalone performance and state-of-the-art resultswhen combined with existing methods.NWM is conceptually similar to recent diffusion-based world models for offline model-based reinforcementlearning, such as DIAMOND (Alonso et al.) and GameNGen (Valevski et al., 2024). However, unlike thesemodels, NWM is trained across a wide range of environments and embodiments, leveraging the diversityof navigation data from robotic and human agents. This allows us to train a large diffusion transformermodel capable of scaling effectively with model size and data to adapt to multiple environments. Ourapproach also shares similarities with Novel View Synthesis (NVS) methods like NeRF (Mildenhall et al.,2021), Zero-1-2-3 (Liu et al., 2023), and GDC (Van Hoorick et al., 2024), from which we draw inspiration.However, unlike NVS approaches, our goal is to train a single model for navigation across diverse environmentsand model temporal dynamics from natural videos, without relying on 3D priors.To learn a NWM, we propose a novel Conditional Diffusion Transformer (CDiT), trained to predict the nextimage state given past image states and actions as context. Unlike a DiT (Peebles and Xie, 2023), CDiT’scomputational complexity is linear with respect to the number of context frames, and it scales favorablyfor models trained up to 1B parameters across diverse environments and embodiments, requiring 4× fewerFLOPs compared to a standard DiT while achieving better future prediction results.In unknown environments, our results show that NWM benefits from training on unlabeled, action- andreward-free video data from Ego4D. Qualitatively, we observe improved video prediction and generationperformance on single images (see <PIC>(c)). Quantitatively, with additional unlabeled data, NWMproduces more accurate predictions when evaluated on the held-out Stanford Go (Hirose et al., 2018) dataset.Our contributions are as follows.
We introduce a Navigation World Model (NWM) and propose a novelConditional Diffusion Transformer (CDiT), which scales efficiently up to 1B parameters with significantlyreduced computational requirements compared to standard DiT. We train CDiT on video footage andnavigation actions from diverse robotic agents, enabling planning by simulating navigation plans independentlyor alongside external navigation policies, achieving state-of-the-art visual navigation performance. Finally,by training NWM on action- and reward-free video data, such as Ego4D, we demonstrate improved videoprediction and generation performance in unseen environments.Goal conditioned visual navigation is an important task in robotics requiring both perception and planningskills (Sridhar et al., 2024; Shah et al.; Pathak et al., 2018; Mirowski et al., 2022; Chaplot et al.; Fu et al.,2022). Given context image(s) and an image specifying the navigation goals, goal-conditioned visual navigationmodels (Sridhar et al., 2024; Shah et al.) aim to generate a viable path towards the goal if the environmentis known, or to explore it otherwise. Recent visual navigation methods like NoMaD (Sridhar et al., 2024)train a diffusion policy via behavior cloning and temporal distance objective to follow goals in the conditionalsetting or to explore new environments in the unconditional setting. Previous approaches like Active NeuralSLAM (Chaplot et al.) used neural SLAM together with analytical planners to plan trajectories in the 3Denvironment, while other approaches like (Chen et al.) learn policies via reinforcement learning. Here weshow that world models can use exploratory data to plan or improve existing navigation policies.Differently than in learning a policy, the goal of a world model (Ha and Schmidhuber, 2018) is to simulatethe environment, e.g, given the current state and action to predict the next state and an associated reward.Previous works have shown that jointly learning a policy and a world model can improve sample efficiencyon Atari (Hafner et al., b,a; Alonso et al.), simulated robotics environments (Seo et al., 2023), and evenwhen applied to real world robots (Wu et al., 2023). More recently, Hansen et al. proposed to use a singleworld model that is shared across tasks by introducing action and task embeddings while Yang et al.; Linet al. (2024b) proposed to describe actions in language, and Bruce et al. (2024) proposed to learn latentactions. World models were also explored in the context of game simulation. DIAMOND (Alonso et al.) andGameNGen (Valevski et al., 2024) propose to use diffusion models to learn game engines of computer gameslike Atari and Doom.
Our work is inspired by these works, and we aim to learn a single general diffusionvideo transformer that can be shared across many environments and different embodiments for navigation.In computer vision, generating videos has been a long standing challenge (Kondratyuk et al.; Blattmannet al., 2023; Girdhar et al., 2023; Yu et al., 2023; Ho et al., 2022; Tulyakov et al., 2018b; Bar-Tal et al.,2024). Most recently, there has been tremendous progress with text-to-video synthesis with methods likeSora (Brooks et al., 2024) and MovieGen (Polyak et al., 2024). Past works proposed to control video synthesisgiven structured action-object class categories (Tulyakov et al., 2018a) or Action Graphs (Bar et al., 2021).Video generation models were previously used in reinforcement learning as rewards (Escontrela et al., 2024),pretraining methods (Tomar et al., 2024), for simulating and planning manipulation actions (Finn and Levine,2017; Liang et al., 2024) and for generating paths in indoor environments (Hirose et al., 2019b; Koh et al.,2021). Interestingly, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are useful both for videotasks like generation (Voleti et al., 2022) and prediction (Lin et al., 2024a), but also for view synthesis (Chanet al., 2023; Poole et al.; Tung et al., 2025). Differently, we use a conditional diffusion transformer to simulatetrajectories for planning without explicit 3D representations or priors.As mentioned in the previous section, we design Fθ as a stochastic mapping so it can simulate stochasticenvironments. This is achieved using a Conditional Diffusion Transformer (CDiT) model, described next.Conditional Diffusion Transformer Architecture. The architecture we use is a temporally autoregressivetransformer model utilizing the efficient CDiT block (see <PIC>), which is applied ×N times over the inputsequence of latents with input action conditioning.CDiT enables time efficient autoregressive modeling by constraining the attention in the first attention blockonly to tokens from the target frame which is being denoised. To condition on tokens from past frames, weincorporate a cross attention layer, such that every query token from the current target attends to tokens frompast frames, which are used as keys and values. The cross attention then contextualizes the representationsusing a skip connection layerHere we experiment with adding unlabeled data, and ask whether NWM can make predictions in newenvironments using imagination. Clearly, In this experiment we train a model on all in-domain datasets, aswell as a susbet of unlabeled videos from Ego4D, where we only have access to the time-shift action. Wetrain a CDiT-XL model and test it on the Go Stanford dataset as well as other random images. We reportthe results in Table 4, finding that training on unlabeled data leads to significantly better video predictionsaccording to all metrics, including improved generation quality. We include qualitative examples in <PIC>.Compared to in-domain (<PIC>), the model breaks faster and expectedly hallucinates paths as it generatestraversals of imagined environments.


We introduce MVSplat360, a feed-forward approach for 360° novel view synthesis(NVS) of diverse real-world scenes, using only sparse observations. This settingis inherently ill-posed due to minimal overlap among input views and insufficientvisual information provided, making it challenging for conventional methodsto achieve high-quality results. Our MVSplat360 addresses this by effectivelycombining geometry-aware 3D reconstruction with temporally consistent videogeneration. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS)model to render features directly into the latent space of a pre-trained Stable VideoDiffusion (SVD) model, where these features then act as pose and visual cuesto guide the denoising process and produce photorealistic 3D-consistent views.Our model is end-to-end trainable and supports rendering arbitrary views withas few as 5 sparse input views. To evaluate MVSplat360’s performance, weintroduce a new benchmark using the challenging DL3DV-10K dataset, whereMVSplat360 achieves superior visual quality compared to state-of-the-art methodson wide-sweeping or even 360° NVS tasks. Experiments on the existing benchmarkRealEstate10K also confirm the effectiveness of our model.The rapid advancement in 3D reconstruction and NVS has been facilitated by the emergence ofdifferentiable rendering [29, 31, 41, 40, 21]. These methods, while fundamental and impressive, areprimarily tailored for per-scene optimization, requiring hundreds or even thousands of images tocomprehensively capture every aspect of the scene. Consequently, the optimization process for eachscene can be time-consuming, and collecting thousands of images is impractical for casual users.In contrast, we consider the problem of novel view synthesis in diverse real-world scenes using a lim-ited number of source views through a feed-forward network.
In particular, this work investigates thefeasibility ofrendering wide-sweeping or even 360° novel views using extremely sparse observations,like fewer than 5 images. This task is inherently challenging due to the complexity of scenes, wherethe limited views do not contain sufficient information to recover the whole 3D scene. Consequently,there is a necessity to ensemble visible information under minimal overlap accurately and generatemissing details reasonably.This represents a new problem setting in sparse-view feed-forward NVS. Existing feed-forwardmethods typically focus on two distinct scenarios: 360° NVS with extremely sparse observations, butonly at object-level [20, 52, 66, 59, 27, 72, 63, 56, 62, 50, 46, 47, 18], or generating reasonable resultsfor scene-level synthesis, but only for nearby viewpoints [53, 7, 19, 43, 9, 13, 6, 60, 10, 70, 42, 61].In contrast, we argue that the time is ripe to unify these previously distinct research directions.Our goal should be to develop systems capable of synthesizing wide-sweeping or even 360° novelviews of large, real-world scenes with complex geometry and significant occlusion. Specifically, thiswork explores synthesising 360° novel views from fewer than 5 input images. We show that in thischallenging setting, existing feed-forward scene synthesis approaches [9, 13, 6, 60, 10, 57] struggleto succeed. This failure arises from two main factors: i) the limited overlap among input views causesmany contents to appear in only a few views or even a single one, posing significant challenges for3D reconstruction; ii) the extremely sparse observations lack sufficient information to capture thecomprehensive details of the whole scene, resulting in regions unobserved from novel viewpoints.In this paper, we propose a simple yet effective framework to address these limitations and introducethe first benchmark for feed-forward 360° scene synthesis from sparse input views. Our key ideais to leverage prior knowledge from a large-scale pre-trained latent diffusion model (LDM) [35] to“imagine” plausible unobserved and disoccluded regions in novel views, which are inherently highlyambiguous. Unlike existing 360° object-level NVS approaches [27, 63, 49, 56, 25, 72, 50], large-scalereal-world scenes comprise multiple 3D assets with complex arrangements, heavy occlusions, andvarying rendering trajectories, which makes it particularly challenging to condition solely on cameraposes, as also verified by concurrent work ViewCrafter [68].To develop a performant framework for scene-level synthesis, we opt to treat the LDM as a refinementmodule, while relying on a 3D reconstruction model to process the complex geometric informa-tion. Broadly, we build upon the feed-forward 3DGS [21] model, MVSplat [10], to obtain coarsenovel views by matching and fusing multi-view information with the cross-view transformer andcost volume. Although these results are imperfect, exhibiting visual artifacts and missing regions(see <PIC>), they represent the reasonable geometric structure of the scene, as they are renderedfrom 3D representation. Furthermore, we choose Stable Video Diffusion (SVD) [3] over otherimage-based LDM as the refinement module, since its strong temporal consistency capabilities alignbetter with the view-consistent requirement of the NVS task, as also observed by concurrent work3DGS-Enhancer [28]. Conditioning SVD with the 3DGS rendered outputs, our MVSplat360 producesvisually appealing novel views that are multi-view consistent and geometrically accurate (see <PIC>).We conduct a series of experiments, mainly on two datasets. First, we establish a new benchmark onDL3DV-10K dataset [23], creating a new training and testing split for feed-forward wide-sweepingand 360° NVS. In this challenging setting, our MVSplat360 achieves photorealistic 360° NVSfrom sparse observations and demonstrates significantly better visual quality, where the previousscene-level feed-forward methods [9, 6, 60, 10] fail to achieve plausible results. Second, we deployMVSplat360 on the existing RealEstate10K [74] benchmark. Following latentSplat [57], we estimateboth interpolation and extrapolation NVS, and report state-of-the-art performance.Our main contributions can be summarized as follows. 1)
We introduce a crucial and pressingproblem for novel view synthesis, i.e., how to do wide-sweeping or even 360° NVS from sparseand widely-displaced observations of diverse in-the-wild scenes (not objects) in a feed-forwardmanner (no any per-scene optimization). 2) We propose an effective solution that nicely integrates thelatest feed-forward 3DGS and the pre-trained Stable Video Diffusion (SVD) model with meticulousintegration designs, where the former is for reconstructing coarse geometry and the latter is forrefining the noisy and incomplete coarse reconstruction. 3) Extensive results on the challengingDL3DV-10K and RealEstate10K datasets demonstrate the superior performance of our MVSplat360.We opt to go beyond per-scene optimisation [1, 2, 58, 15], and to deal with a more general feed-forward network capable of achieving 360° NVS for unseen scenes, yet without the need ofadditionalper-scene training. This requires effectively matching information between sparse views in 3Dspace, as well as generating sufficient content based on only partial observations. To achieve that,our MVSplat360 framework, illustrated in <PIC>, comprises two main components: a multi-viewgeometry reconstruction module (Section 3.1) and a multi-frame consistent appearance refinementnetwork (Section 3.2). The former is responsible for matching and fusing multi-view informationfrom sparse observations to create a coarse geometry reconstruction, whereas the latter is designedto refine the appearance with a pre-trained latent video diffusion model. While similar two-stepapproaches have been explored in recent related works, e.g., [4, 58, 38, 15], we are the first (to thebest of our knowledge) to explore it on wide-sweeping or even 360° NVS for large-scale scenes fromsparse views (as few as 5), in a feed-forward manner.bserved and novel viewpoints selection. To enable 360° scene synthesis, it is crucial to choose thecorrect camera viewpoints, so that they can cover most contents in diverse and complex scenes [15].It is impractical to assume a circular orbital camera trajectory like those object-level 360° viewsynthesis [27, 46, 72, 47], whereas it is suboptimal to randomly choose a video sequence like existingscene-level nearby viewpoint synthesis [46, 6, 10]. To this end, we propose to choose views evenlydistributed within a set of targeted viewpoints as input. Specifically, for a given set of candidate views,we apply farthest point sampling over the camera locations to identify the input views and randomlychoose from the rest as target views. The number of candidate views gradually increases throughoutthe training, stably improving the model’s capability toward handling 360° scene synthesis.Metrics. latentSplat [57] improves the results with anadditional decoder and adversarial training. However, its resulting object geometry and image qualityare still far from satisfactory, suggesting that the GAN-based framework cannot provide enoughprior knowledge for refining 360° NVS in diverse real-world scenes. Readers are referred to ourproject page for video results with more comprehensive comparisons, where our MVSplat360 shows


We propose DrivingForward, a feed-forward Gaussian Splat-ting model that reconstructs driving scenes from flexiblesurround-view input. Driving scene images from vehicle-mounted cameras are typically sparse, with limited overlap,and the movement of the vehicle further complicates the ac-quisition of camera extrinsics. To tackle these challenges andachieve real-time reconstruction, we jointly train a pose net-work, a depth network, and a Gaussian network to predictthe Gaussian primitives that represent the driving scenes. Thepose network and depth network determine the position of theGaussian primitives in a self-supervised manner, without us-ing depth ground truth and camera extrinsics during training.The Gaussian network independently predicts primitive pa-rameters from each input image, including covariance, opac-ity, and spherical harmonics coefficients. At the inferencestage, our model can achieve feed-forward reconstructionfrom flexible multi-frame surround-view input. Experimentson the nuScenes dataset show that our model outperforms ex-isting state-of-the-art feed-forward and scene-optimized re-construction methods in terms of reconstruction.
3D scene reconstruction is critical for understanding driv-ing scenes. Modern self-driving assistance cars are usu-ally equipped with several cameras to capture surround-ing scenes. Real-time reconstruction of driving scenesfrom sparse vehicle-mounted cameras contributes to variousdownstream tasks in autonomous driving, including onlinemapping (Li et al. 2022a), BEV perception (Li et al. 2022b;Liang et al. 2022; Liu et al. 2023), and 3D detection (Chenet al. 2023; Cai et al. 2023). However, the real-time comput-ing required by downstream tasks and the sparse surround-ing views challenge the driving scene reconstruction.Neural Radiance Fields (NeRF) (Mildenhall et al. 2021)and 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023) havesignificantly progressed the development of 3D scene recon-struction. DrivingGaussian (Zhou et al. 2024), StreetGaus-sian (Yan et al. 2024), and AutoSplat (Khan et al. 2024)further explore the reconstruction of driving scenes. Whilethese methods demonstrate strong capability in novel viewsynthesis, they are scene-optimized methods that requiredozens of images and expensive computing time to recon-struct just one scene. These offline reconstruction meth-ods are unsuitable for real-time downstream tasks in au-tonomous driving, thereby limiting their practicality.Our goal is to achieve online, generalizable driving scenereconstruction from sparse surrounding views. Several at-tempts, such as pixelSplat (Charatan et al. 2024) and MVS-plat (Chen et al. 2024), have explored the generalizable re-construction. They learn powerful priors from large-scaledatasets during training and achieve fast 3D reconstructionfrom sparse input views through a feed-forward inference.Unfortunately, these methods are difficult to apply in driv-ing scenes. Since the number of vehicle-mounted camerasis limited (usually 6 cameras), the overlap of adjacent viewsis extremely small (as low as 10%). While these existingmethods often require densely overlapping (usually over60%) input images. Additionally, acquiring camera extrin-sics for each view at various timesteps in driving scenes iscostly. These methods depend on such data during training,limiting their practical applicability. A recent NeRF-basedwork DistillNeRF (Wang et al. 2024) attempts to developa generalizable 3D representation for driving scenes. How-ever, it gains a suboptimal performance and relies on Li-DAR to train numerous NeRF models for distillation, whichis extremely computationally expensive. Besides, previousfeed-forward methods typically have a fixed mode of inputviews, either using stereo images (e.g., MVSplat, pixelSplat)or single-frame images of surrounding views (e.g., Distill-NeRF). However, as the vehicle moves forward and capturessurround-view images frame-by-frame, we aim to supportflexible multi-frame inputs for reconstruction, such as pre-dicting the next frame’s views from single-frame surround-ing views or synthesizing intermediate frame surroundingviews from two interval frames.
By unprojecting the consistent depth estimation, we ob-tain the position of Gaussian primitives. For other Gaus-sian parameters, we individually predict them from eachimage through a Gaussian network. The Gaussian networkis jointly trained with the pose network and depth net-work. It takes the depth map and image feature from thedepth network as input and outputs the covariance, opacity,and spherical harmonics coefficients of Gaussian primitives.Since Gaussian primitives are predicted independently fromsingle-frame images of surrounding views, our method is notconstrained by a fixed number of input frames. This allowsfor flexible multi-frame surround-view input, such as pre-dicting the next frames’ views from the current frame or syn-thesizing the intermediate frame from two interval frames.Extensive experiments on the nuScenes dataset demon-strate that our DrivingForward outperforms other feed-forward methods on the novel view synthesis under variousinputs. It also achieves higher reconstruction quality com-pared to scene-optimized methods with the same input. Afunctional comparison of our DrivingForward with the lat-est related works is present in <PIC>
DrivingForward learns powerful priors from large-scaledriving scene datasets during training and achieves real-timedriving scene reconstruction in a feed-forward manner fromsparse vehicle-mounted cameras at the inference stage.We begin with N sparse camera imagesas input and aim to predict Gaussian primitives from input view im-ages. The overview framework is illustrated in <PIC>. Apose network predict the vehiclemotion and estimate the scale-aware depth from the input.We assign each pixel to one Gaussian primitive and the po-sition is located through the estimated depth. Other parame-We unproject the Gaussian primitives from allviews into 3D space, render them to the target view in adifferentiable way, and jointly train the full model end-to-end. At the inference stage, the depth network and Gaussiannetwork are used for feed-forward reconstruction. Since thescale-aware localization and prediction of other parametersdo not depend on other frames, we can flexibly input differ-ent numbers of surround-view frames during inference.
In driving scenes, with multi-frame surrounding views,we take different inputs as the source image to compute thephotometric loss. First, we use the images from the samecamera at different frames, denoted as temporal contexts.Then, we use the images from adjacent cameras at the sameframe, denoted as spatial contexts. We also combined thetwo ways, using images from adjacent cameras at differentframes, denoted as spatial-temporal contexts. The key in-sight is to leverage the small overlap between spatially andtemporally neighboring images for matching, which pro-vides scale information and enables learning scale-awarecamera motion and depth maps during training.
Note that this camera-to-vehicle transformation matrix isfixed for each camera across all timesteps and is relativelyeasy to obtain in practice, whereas the general world-to-camera extrinsics vary at every timestep and thus are costlyto collect. Leveraging the fixed camera-to-vehicle transfor-mation matrix and the camera motion predicted by the posenetwork, we do not require the world-to-camera extrinsicsduring training, which is another advantage of our method.
Unlike pixelSplat (Charatan et al. 2024) and MVS-plat (Chen et al. 2024) that rely on paired views to predictGaussian parameters, we propose a Gaussian network to in-dependently predict these parameters from each image andaggregate the Gaussian primitives across all images, whichis more suitable for driving scenes with small overlap be-tween sparse vehicle-mounted cameras. To ensure the pre-dicted Gaussian parameters are consistent across all inputviews, we utilize the previous scale-aware depth and im-age feature from the depth network as input for the Gaus-sian network. Since the previous depth network learns scaleinformation from spatial and temporal context images, weargue that the scale-aware depth and image feature can en-hance the multi-view consistency of the remaining Gaussianparameters.By applying scale-aware localization and Gaussian param-eters prediction to each input view, we obtain the Gaussianprimitives for all images. These primitives are then aggre-gated into 3D space to form a 3D representation. Novel viewsynthesis can be achieved through the splat-based rasteriza-tion rendering in 3DGS (Kerbl et al. 2023).We jointly train the full model, including the depth net-work, the pose network, and the Gaussian network. For warpoperation of the depth and pose network, we use spatialtransformer network (Jaderberg et al. 2015) to sample thesynthesis image from the source image, which is fully differ-entiable (Godard, Mac Aodha, and Brostow 2017). For ren-dering novel views after obtaining the Gaussian primitivesin 3D space, the splat-based rasterization rendering is alsofully differentiable. These two operations along with otherdifferentiable parts enable the joint training end-to-end.We fuse image features from the Depth network into theGaussian network. This shared feature connects the scale-aware positions with predictions of other Gaussian parame-ters, allowing the Gaussian network to leverage scale infor-mation from temporal and spatial contexts. Additionally, itpromotes the convergence of the full model.We report the quantitative results of comparison with state-of-the-art feed-forward methods in Table 1. Although weadapt our method to align with the different settings of thebaselines, we outperform them across all metrics under thecorresponding configuration. The qualitative comparisons ofMVSplat, pixelSplat, and our method are visualized in <PIC>. Our DrivingForward achieves the highest quality onnovel view results even for challenging details, such as thetraffic sign in the front left view and the monument withwords in the back right view. Other methods show obviousartifacts in these regions, while our method synthesizes clearno
vel views without such artifacts.


We propose GGS, a Generalizable Gaussian Splatting methodfor Autonomous Driving which can achieve realistic render-ing under large viewpoint changes. Previous generalizable3D gaussian splatting methods are limited to rendering novelviews that are very close to the original pair of images, whichcannot handle large differences in viewpoint. Especially inautonomous driving scenarios, images are typically collectedfrom a single lane. The limited training perspective makesrendering images of a different lane very challenging. To fur-ther improve the rendering capability of GGS under largeviewpoint changes, we introduces a novel virtual lane gen-eration module into GSS method to enables high-quality laneswitching even without a multi-lane dataset. Besides, we de-sign a diffusion loss to supervise the generation of virtual laneimage to further address the problem of lack of data in the vir-tual lanes. Finally, we also propose a depth refinement mod-ule to optimize depth estimation in the GSS model. Extensivevalidation of our method, compared to existing approaches,demonstrates state-of-the-art performance.
Novel view synthesis is an essential task in the field of com-puter vision, with significant potential applications in au-tonomous driving (Yang et al. 2020;Wu et al. 2023; Liu et al.2023; Yang et al. 2023b, 2024a; Yu et al. 2024), object detec-tion, and digital human representations. To enhance the ro-bustness of autonomous driving systems, it is imperative toestablish a simulation environment for testing these systemseffectively. However, the majority of existing datasets arelimited to single-lane scenarios. This limitation presents sig-nificant challenges in inferring adjacent lane scenarios fromthe current viewpoint. If lane switching is not supported, thetest samples provided to the autonomous driving simulationsystem will be incomplete, making it impossible to conductbetter simulation testing and requiring a significant amountof data collection costs.The methods based on NeRF (Mildenhall et al. 2021;Yang et al. 2023a) often rely on lidar to better generate novelviews in autonomous driving scenarios. READ (Li, Li, andZhu 2023) introduces a new rendering method that adoptsa neural rendering approach different from NeRF. It learnsneural descriptors of the original point cloud with explicitgeometry to render images, instead of learning implicit ge-
ometry in NeRF methods. However, the efficiency of train-ing and rendering using these methods is very low.The efficiency in training and rendering speed, coupledwith the high reconstruction quality of 3D Gaussian Splat-ting (Kerbl et al. 2023), contributes to its widespread appli-cation of novel view synthesis in autonomous driving. Gaus-sianPro (Cheng et al. 2024) introduces multi-view stereoto improve the geometry of generated gaussian splats. DC-Gaussian(Cheng et al. 2023) introduces an adaptive imagedecomposition module to address the impact of glass reflec-tions on the quality of novel view synthesis. However, thesemethods still cannot have effective novel view synthesis inlane switching, as they do not address the main problem thatonly single lanes of data are collected.To address the problem of the sparse view synthesis, manymethods have sought to optimize this process using gener-ative models (Yu et al. 2021; Chen et al. 2024; Liu et al.2024; Wu et al. 2024; Tang et al. 2024). generative mod-els are trained across large amount of scenes to enhanceperformance in sparse view scenarios. However, the genera-tive model still lacks of data from multi-lanes to learn howto synthesize novel views for other lanes from single lanesdata.
Therefore, we propose a virtual lane module into genera-tive Gaussian splatting to address the synthesis of new viewsinvolving lane changes, despite the lack of multi-lane train-ing datasets for supervision. In the module, we first use 3Dgaussians generated from images in the single lanes using agenerative model to predict images from virtual lanes, thenuse 3D gaussians generated from the virtual images to pre-dict back the image collected in the single lanes. In this way,we can let generative model to learn how to generate thebest images in the other lanes even with only single lanes ofdata. In addition, we propose a diffusion loss from a latentdiffusion model (Sohl-Dickstein et al. 2015; Song, Meng,and Ermon 2020; Nichol and Dhariwal 2021; Rombach et al.2022) to virtual generated images to further improve the laneswitching of our GGS. Finally, as improving geometry ofgenerated 3D gaussians also improves novel view synthe-sis in sparse view collections, we employs points from tra-ditional multiview stereo reconstruction to refine the depthestimated in GGS. The main contributions of this paper can be summarizedas follows:• We propose a novel virtual lanes module into the gen-erative gaussian splatting to improve the quality of laneswitching novel view with only single lanes of data.• We introduce a diffusion loss to directly supervise theimage from virtual lanes predicted by GGS to furtherimprove the novel view synthesis from limited collectedviews.• We propose to fusing MVS geometry into the generative3D gaussian splating to improve geometry estimation.• We conduct extensive experiments on a wide range ofscenarios to validate the effectiveness of our algorithm,and achieve state-of-the-art street novel view synthesiseven without LiDAR.
Although generalized models can assist in synthesizingnovel views in sparse views, insufficient view informationleads to inaccurate depth estimation. Our method further op-timizes the generalization model. The overall framework di-agram of our GGS method is as shown in <PIC>. We inputfour different frame images and introduce neighborhood fea-ture in the Mutli-View Depth Refinement Module to bet-ter address scenes with occlusions. And we introduce moreglobal information to optimize the predicted depth map Byusing MVS. In the Virtual Lane Generation Module, weintroduce the concept of virtual lanes and solve the problemof not having a multi-lane dataset by switching back afterswitching, allowing the model to flexibly switch lanes. Inaddition, we introduce the Multi-Lane Diffusion Loss tosupervise the novel view synthesis.
Mutli-View Depth Refinement ModuleWe enhanced MVSplat by our Mutli-View Depth Refine-ment Module, i.e. Modified MVSplat. It can produce moreaccurate 3D gaussian primitives and improve the quality ofnovel view synthesis. To better infer unknown regions, weincorporated the color feature information of the neighbor-hood of this view. Our model takes this into consideration.We use the back-projected point cloud map reconstructedthrough Agisoft Metashape as an additional input color fea-ture for U-Net. The feature representation of the neighbor-hood is:
Previous generalizable 3D gaussian splatting methods arelimited to rendering novel views that are very close to theoriginal pair of images, which cannot handle large differ-ences in viewpoint. Especially in autonomous driving sce-narios, images are typically collected from a single lane. Thelimited training perspective makes rendering images of a dif-ferent lane very challenging. We have obtained a 3D Gaus-sian using our modified depth refinement module. To furtherimprove the rendering capability of GGS under large view-point changes, We introduce the virtual lane approach thatenables high-quality lane switching even without a multi-lane dataset, inspired by (Huang et al. 2023).The virtual lane converter is used to select the appropri-ate virtual lane, so that after lane switching, no informationcan be seen from the virtual perspective due to excessiveswitching amplitude. Then generate a pose for the virtuallane by performing a vertical translation along the lane. Fi-nally, generate a virtual perspective based on the pose of thevirtual lane. After introducing the virtual lane module, ourGGS module process mainly includes two stages.In the first stage, we input a set of N images:is a renderedimage without shifting the view, and the rendered view isconsistent with the ground truth. The current lane generatesa collection of virtual lane rendering images through laneconverters. The rendered image of the virtual lane is repre-sented as:
This forms a closed-loopprocess of switching to a new lane and then switching back.The advantage of doing so is that even without the groundtruth of the left and right lanes, we can still enhance the qual-ity of the model’s rendering of the left and right lanes by es-tablishing virtual lanes, allowing the model to improve thequality of lane switching, as shown in <PIC>.
There is no ground truth available for training when switch-ing lanes. When the lane switching amplitude is large, ob-stacles can obstruct the view during lane changes, making itimpossible to collect information about the new lane fromthe current lane, as shown in <PIC>. Therefore, in order tobetter address this issue, we use diffusion prior knowledgeto imagine color information from a novel lane view.The traditional diffusion model denoising method directlycompletes the generated image, but due to the diversity ofthe generated models, it can lead to inconsistent results be-tween frames. Our method calculates the loss of the de-noised image and the image before denoising, and generatesa new perspective supervised by diffusion. Additionally weconstruct multi-lane novel view images, instead of utilizingimage of the current lane as input for U-Net denoising. Thisapproach helps ensure that the autonomous driving lane re-mains visible in the image following a change in viewpoint.Specifically, we adapt the Stable Diffusion framework(Rombach et al. 2022), and use the Variational AutoEncoder(Kingma andWelling 2013) to encode the multi-lane imagesinto latent code, including the left lane, middle lane and right


The Bird’s-eye View (BeV) representation is widely usedfor 3D perception from multi-view camera images. It al-lows to merge features from different cameras into a com-mon space, providing a unified representation of the 3Dscene. The key component is the view transformer, whichtransforms image views into the BeV. However, actual viewtransformer methods based on geometry or cross-attentiondo not provide a sufficiently detailed representation of thescene, as they use a sub-sampling of the 3D space that isnon-optimal for modeling the fine structures of the envi-ronment. In this paper, we propose GaussianBeV, a novelmethod for transforming image features to BeV by finelyrepresenting the scene using a set of3D gaussians locatedand oriented in 3D space. This representation is then splat-tered to produce the BeV feature map by adapting recentadvances in 3D representation rendering based on gaus-sian splatting [12]. GaussianBeV is the first approach touse this 3D gaussian modeling and 3D scene rendering pro-cess online, i.e. without optimizing it on a specific scene anddirectly integrated into a single stage model for BeV sceneunderstanding. Experiments show that the proposed repre-sentation is highly effective and place GaussianBeV as thenew state-of-the-art on the BeV semantic segmentation taskon the nuScenes dataset [2].
Multi-camera 3D perception tasks, such as semantic seg-mentation, are crucial for autonomous navigation applica-tions. A common strategy involves projecting and merg-ing features from different cameras into a bird’s-eye view(BeV) representation, which is then analyzed by perceptionheads. The primary challenge in these approaches lies inaddressing the loss of 3D information during the projectionof the physical world into camera images, thus solving theinverse problem of transforming image views into the BeVRecent literature identifies three main subsets of methodsfor image-to-BeV transformation. First, depth-based meth-ods [8,10,13,18] achieve view transformation geometricallyby filling a 3D grid with features extracted from imagesbased on the prediction of discrete depth distribution. Thekey idea is to roughly localize the 3D position of each imagefeature and then accumulate them through a voxelizationstep. However, in these approaches, 3D feature localiza-tion depends on the depth discretization granularity and issub-optimal, as features are placed at the level of the visiblefaces of objects. Second, projection-based methods [3, 7]also use a geometric approach, project 3D grid points intothe cameras and gathers corresponding features from them.While straightforward, these methods do not yield accurate2D-to-3D back-projection, as all grid points along the samecamera ray receive the same feature. Third, transformer-based methods [15, 17, 27] utilize cross-attention to mergemulti-view features. Though effective for 3D object de-tection, their application to dense tasks like BeV semanticsegmentation incurs a high computational cost due to densespatial queries needed for BeV representation in the atten-tion process. Some works [17, 27] address this by reducingthe BeV resolution, leading to inevitable information loss.In this article, we propose a novel view transformationmethod called GaussianBeV, which enables fine 3D model-ing of scenes. Drawing on recent advances in explicit 3Drepresentation for rendering novel views based on GaussianSplatting (GS) [12], our method represents a scene using aset of 3D gaussians, each parameterized by a center, scale,rotation, opacity and semantic features (instead of colors inGS). Furthermore, unlike the original GS method, whichuses offline optimization for a specific scene to determinethe 3D gaussian representation, we propose to train a neuralnetwork to directly generate an online 3D gaussian repre-sentation of the scene from a set of multi-view images. Thisrepresentation is then rendered into a BeV feature map thatis then analyzed by semantic segmentation heads.The representation of a scene by a set of 3D gaussiansallows to model its entire content. Indeed, the geometricalproperties of the gaussians (position, size and rotation) en-able to cover 3D space with varying level of detail depend-ing on the structures encountered in the scene. Intuitively,a gaussian representing a lane marking will be rotated andelongated along its length. A gaussian representing a vehi-cle will be placed in the center of the vehicle and will followits shape. The <PIC>d illustrates the intuition behind therepresentation proposed in this paper. Our contributions canbe summarized as follows. (1) Introduction of Gaussian-BeV for BeV feature map generation from images throughan online image-to-3D gaussian representation of the scene,allowing fine 3D content modeling. This representation isthen splattered in BeV using a rasterizer module. To ourknowledge, this is the first time that a gaussian splattingrepresentation that is not scene-specific is proposed and in-tegrated into a perception model. (2) Experiments demon-strating the effectiveness of our method, establishing it asthe new state-of-the-art in BeV semantic segmentation.
2. Related workDepth-based. A series of models were built on the ex-plicit prediction of pixel-wise depth estimations along withimage features. Combined with camera calibration parame-ters, this enables the back-projection of 2D features into 3Dfeature point cloud which is finally aggregated in the BeVgrid. To accommodate for uncertainty in depth estimation,the features are actually propagated all along the ray thattraverses their pixel and modulated by a discrete depth prob-ability estimation [8, 10, 18]. To improves depth prediction,an explicit depth supervision scheme is proposed [13] usingLiDAR data during model training. However, depth-basedmethods are sensitive to the ray sampling strategy, usuallyback-projecting features along the ray and on object sur-faces (see <PIC>a).Projection-based. Using a thorough comparative study, [7]proposes to discard depth estimation in favor of a simplerprojection scheme: a predefined set of 3D points is usedto describe the scene and camera features are probed byprojecting the point on the camera feature maps using cal-ibration data. This projection disregards actual objects andbackground placement but returns a denser representationof the scene without void beyond the depth of objects sur-faces. The computational and memory overhead of the gen-erating the BeV grid is reduced by opting for a sparse gridrepresentation [3]. Projection-based view transformationmethods are simple but result in a coarse BeV represen-tation because all voxels along the optical ray receive thesame features (see <PIC>b).Attention-based. Capitalising on recent advances in Trans-former models, depth estimation is replaced by an attention-based feature modulation scheme [15–17, 22, 27]. Sev-eral optimization schemes are proposed to resolve compu-tational complexity of a pairwise matching between imageand BeV grid tokens: factorization of spatial and temporalattentions, deformable attention [15], injection of calibra-tion and timestamp priors [17]. For the segmentation task,attention-based view transformation is computationally andmemory intensive, due to the need to define a dense querymap [15]. This is why some methods [17,27] predict a low-resolution BeV (see <PIC>c), which is then upsampled bysuccessive deconvolutions.Gaussian splatting. Gaussian splatting (GS) [12] is a 3Dscene rendering technique which uses 3D gaussians to de-scribe a scene. Each gaussian is parameterized by its po-sition, scale, rotation, opacity and a Spherical Harmonicscolor model. The entire rendering pipeline is differentiableallowing the optimization of the gaussian parameters to aparticular scene based on a set of images. GS is both fastand parallel, allowing real-time operation on GPUs. More-over, it is possible to trade off details for rendering speedby pruning smaller and transparent gaussians. In com-parison to sparse voxel grids, gaussians offer more efficient representations of a scene since individual gaussianscan describe large volumes while smaller ones can accu-rately encode finer details with arbitrary resolution. Detailsand rasterization resolutions are parameters of the render-ing pipeline, not the scene description. Several extensionshave been proposed, allowing the management of dynamicobjects [14, 25] or the distillation of semantic features fromfoundation models in the representation [19, 20, 29]. In allcases, the gaussian representation is scene-specific.In our work, we propose to use gaussian representationto overcome the drawbacks of previous view transformermethods. Unlike previous gaussian splatting works, whichlearns gaussian representations offline, we propose to learna neural network capable of providing an online gaussianrepresentation of the scene.The <PIC> presents an overview of GaussianBeV.The model takes as input a set of multiview imagesextrinsic parameters are applied to transform 3D gaussiansfrom the camera to the world reference frame to finally con-catenate all the gaussians into the single set G.The third module is the BeV rasterizer (Sec 3.3) that per-forms a BeV rendering of the 3D gaussian set G to producethe BeV feature map
Given the input feature maps F, the 3D gaussian gen-erator predicts the 3D gaussian representation of the sceneusing several prediction heads. The <PIC> illustrates howit operates on the feature maps.Gaussian centers. The 3D positions of the gaussians inthe scene are estimated by a depth head and a 3D offsethead applied to F. The first predicts an initial position ofthe 3D centers along the optical rays. The second refinesthis 3D position by adding a small 3D displacement to it,giving more flexibility in the positioning of gaussians bynot freezing them along the optical rays.Given the input feature maps F, the 3D gaussian gen-erator predicts the 3D gaussian representation of the sceneusing several prediction heads. The <PIC> illustrates howit operates on the feature maps.Gaussian centers. The 3D positions of the gaussians inthe scene are estimated by a depth head and a 3D offsethead applied to F. The first predicts an initial position ofthe 3D centers along the optical rays. The second refinesthis 3D position by adding a small 3D displacement to it,giving more flexibility in the positioning of gaussians bynot freezing them along the optical rays.For each pixel, it calculates the corresponding 3D gaussian by passing through prediction heads(green boxes). Some of these predictions are then decoded (blue and red boxes) and transformed to be expressed in the world referenceframe (yellow boxes). All predicted gaussian parameters are then concatenated to produce the 3D gaussian representation G of the scene.Given the input feature maps F, the 3D gaussian gen-erator predicts the 3D gaussian representation of the sceneusing several prediction heads. The <PIC> illustrates howThe <PIC> shows qualita-tive results for vehicle and ground surface segmentation. Italso gives a visualization of the output of the BeV rasterizermodule. To visualize the BeV feature map, we appliedPrincipal Component Analysis (PCA) to the 3D gaussianfeatures before BeV rendering. This allows us to reduce thesize of the features to render the BeV features in color.BeV backbone. We conducted experiments to evaluate theinfluence of the choice of the BeV backbone. In particu-lar, GaussianBeV is trained (1) without using a BeV back-bone, by connecting the BeV feature directly to the seg-mentation heads (row 2 in the Table 4) and (2) with theit operates on the feature maps.


World models have demonstrated superiority in autonomousdriving, particularly in the generation of multi-view driving videos. How-ever, significant challenges still exist in generating customized drivingarXiv:2403.06845v2 [cs.CV] 11 Apr 2024videos. In this paper, we propose DriveDreamer-2, which builds uponthe framework of DriveDreamer and incorporates a Large LanguageModel (LLM) to generate user-defined driving videos. Specifically, anLLM interface is initially incorporated to convert a user’s query intoagent trajectories. Subsequently, a HDMap, adhering to traffic regula-tions, is generated based on the trajectories. Ultimately, we propose theUnified Multi-View Model to enhance temporal and spatial coherence inthe generated driving videos. DriveDreamer-2 is the first world model togenerate customized driving videos, it can generate uncommon drivingvideos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides,experimental results demonstrate that the generated videos enhance thetraining of driving perception methods (e.g., 3D detection and track-ing). Furthermore, video generation quality of DriveDreamer-2 surpassesother state-of-the-art methods, showcasing FID and FVD scores of 11.2and 55.7, representing relative improvements of ∼30% and ∼50%.World models for autonomous driving [23,25,59,62] have drawn extensive atten-tion from both the industry and academia in recent years. Benefiting from theirexcellent predictive capabilities, autonomous driving world models facilitate thegeneration of diverse driving videos, encompassing even long-tail scenarios. Thegenerated driving videos can be utilized to enhance the training of various driv-ing perception approaches, proving highly beneficial for practical applications inautonomous driving.
World modeling in autonomous driving presents a formidable challenge dueto its inherent complexity and large sampling space. Early approaches [8, 22]mitigate these problems by incorporating world modeling within the Bird’s EyeView (BEV) semantic segmentation space. However, these methods primarily ex-plore world models in simulated autonomous driving environments. In the recentevolution of autonomous driving technologies, there has been a substantial leapforward in the development of world models. This progress has been propelled bythe utilization of cutting-edge diffusion models [5, 17, 18, 39, 40, 46], exemplifiedby notable contributions such as DriveDreamer [59], Drive-WM [62], Magic-Drive [7], Panacea [64], and the integration of large language models like GAIA-1 [23], ADriver-I [25]. These sophisticated models have played a pivotal role inpushing the boundaries of world modeling capabilities, enabling researchers andengineers to delve into increasingly intricate and realistic driving scenarios. How-ever, it is important to note that a majority of these methods rely heavily onstructured information (e.g., 3D boxes, HDMaps, and optical flow) or real-worldimage frames as conditions. This dependence not only constrains interactivitybut also limits the diversity of generated videos.To tackle the aforementioned challenges, we propose DriveDreamer-2, whichis the first world model to generate diverse driving videos in a user-friendly man-ner. In contrast to previous methods [7,59,62] that rely on structured conditionseither from specific datasets or sophisticated annotations, DriveDreamer-2 em-phasizes generating customized driving videos by simulating various traffic con-ditions with user-friendly text prompts. Specifically, the traffic simulation taskhas been disentangled into the generation of foreground conditions (trajectoriesof the ego-car and other agents) and background conditions (HDMaps of laneboundary, lane divider, and pedestrian crossing). For foreground generation, afunctional library is constructed to finetune a Large Language Model (LLM), en-abling it to generate agent trajectories based on user text input. For backgroundconditions, we propose the HDMap generator that employs a diffusion model tosimulate road structures. In this process, the previously generated agent trajec-tories are involved as conditional inputs, which allows the HDMap generator tolearn the associations between foreground and background conditions in drivingscenes. Building upon the generated traffic structured conditions, we employ theDriveDreamer [59] framework to generate multi-view driving videos. It is notedthat we introduce the Unified Multi-view Video Model (UniMVM) within theDriveDreamer framework, which is designed to unify both intra-view and cross-view spatial consistency, enhancing the overall temporal and spatial coherencein the generated driving videos.
Extensive experiment results show that DriveDreamer-2 is capable of produc-ing diverse user-customized videos, including uncommon scenarios where vehiclesabruptly cut in (depicted in <PIC>). Besides, DriveDreamer-2 can generate high-quality driving videos with an FID of 11.2 and FVD of 55.7, relatively improvingprevious best-performing methods by ∼30% and ∼50%. Furthermore, experi-ments are conducted to verify that driving videos generated by DriveDreamer-2can enhance the training of various autonomous driving perception methods,where the performance of detection and tracking are relatively improved by∼4% and ∼8%.The main contributions of this paper can be summarized as follows:– We present DriveDreamer-2, which is the first world model to generate di-verse driving videos in a user-friendly manner.– We propose a traffic simulation pipeline employing only text prompts asinput, which can be utilized to generate diverse traffic conditions for drivingvideo generation.– UniMVM is presented to seamlessly integrate intra-view and cross-view spa-tial consistency, elevating the overall temporal and spatial coherence withinthe generated driving videos.– Extensive experiments are conducted to show that DriveDreamer-2 can craftdiverse customized driving videos. Besides, DriveDreamer-2 enhances theFID and FVD by ∼30% and ∼50% compared to previous best-performingmethods. Moreover, the driving videos generated by DriveDreamer-2 en-hance the training of various driving perception methods.
The primary objective of world methods is to establish dynamic environmen-tal models, endowing agents with predictive capabilities for the future. In theearly exploration, Variational Autoencoders (VAE) [31] and Long Short-TermMemory (LSTM) [19] are employed to capture transition dynamics and ren-dering functionality, showcasing remarkable success across diverse applications[9–13,13,29,36,48,65]. Constructing driving world models poses distinctive chal-lenges, primarily arising from the high sample complexity inherent in real-worlddriving tasks [3]. To address these challenges, ISO-Dream [42] introduces anexplicit disentanglement of visual dynamics into controllable and uncontrollablestates. MILE [22] strategically incorporates world modeling within the Bird’s EyeView (BEV) semantic segmentation space. Recently, DriveDreamer [59], GAIA-1 [23], ADriver-I [25], and Drive-WM [62] have explored the training of drivingworld models in the real world, leveraging powerful diffusion models or naturallanguage models. However, most of these methods heavily depend on structuredinformation (e.g., 3D boxes, HDMaps, and optical flow) as conditions. This de-pendency not only constrains interactivity but also limits generation diversity.
Video generation and prediction are pivotal techniques for understanding the vi-sual world. In the early stages of video generation, methods like VariationalAutoencoders (VAEs) [4, 21], flow-based model [33], and Generative Adver-sarial Networks (GANs) [38, 47, 53, 56] are explored. Language models [20, 26,32, 45, 51, 55, 60, 63] are also employed for intricate visual dynamics model-ing. Recent advancements have seen diffusion models [5, 17, 18, 39, 40, 46] ex-tending their influence to video generation. Notably, video diffusion models[1, 14, 16, 28, 49, 58, 67] exhibit superior capabilities in generating high-qualityvideos with realistic frames and smooth transitions, offering enhanced control-lability. These models adapt seamlessly to various input conditions, includingtext, canny, sketch, semantic maps, and depth maps. In the realm of autonomousdriving, DriveDreamer-2 leverages powerful diffusion models for learning visualdynamics.Driving simulators stand as a cornerstone in self-driving development, aimingto offer a controlled environment to mimic real-world conditions. LCTGen [52]utilizes an LLM to encode detailed language descriptions to a vector and sub-sequently employs a generator to produce corresponding simulated scenarios.This method requires highly detailed language descriptions, including informa-tion such as the speed and orientation of agents. TrafficGen [6] comprehendsthe inherent relationships within traffic scenarios, enabling the generation ofdiverse and legitimate traffic flows within the same map. CTG [70] generatestraffic simulations by employing manually designed loss functions that adhere totraffic constraints. CTG++ [69] further extends CTG by utilizing GPT-4 [41]to convert user language descriptions into a loss function, which guides thescene-level conditional diffusion model to generate the corresponding scenario.In DriveDreamer-2, we construct a functional library to finetune the LLM toachieve a user-friendly text-to-traffic simulation, which eliminates intricate lossdesign or complex text prompt inputs.
<PIC> illustrates the overall framework of DriveDreamer-2. A customized traf-fic simulation is first proposed to generate foreground agent trajectories andbackground HDMaps. Specifically, DriveDreamer-2 utilizes a finetuned LLM totranslate user prompts into agent trajectories, and the HDMap generator is thenintroduced to simulate road structures using the generated trajectories as con-ditions. Leveraging the customized traffic simulation pipeline, DriveDreamer-2is capable of generating diverse structured conditions for the subsequent videogeneration. Building upon the architecture of DriveDreamer [59], the UniMVMframework is proposed to unify both intra-view and cross-view spatial consis-tency, thereby enhancing the overall temporal and spatial coherence in the gen-erated driving videos. In the subsequent sections, we delve into the details of thecustomized traffic simulation and the UniMVM framework.
Finetuning LLM for Trajectory Generation Previous traffic simulationmethods [37, 69, 70] necessitate the intricate specification of parameters, involv-ing details such as the agent’s speed, position, acceleration, and mission goal.To simplify this intricate process, we propose to finetune LLM with the con-structed trajectory-generation function library, allowing for the efficient trans-formation of user-friendly language inputs into comprehensive traffic simulationscenarios. As depicted in <PIC>, the constructed function library encompasses 18functions, including agent functions (steering, constant speed, acceleration, andbraking), pedestrian functions (walking direction and speed), and other utilityfunctions such as saving trajectories. Building upon these functions, Text-to-Python-Scripts pairs are manually curated for finetuning LLM (GPT-3.5). Thescripts include a range of fundamental scenarios such as lane-changing, overtak-ing, following other vehicles, and executing U-turns. Additionally, we encompassmore uncommon scenarios like pedestrians abruptly crossing, and vehicles cut-ting into the lane. Taking the user input a vehicle cuts in as an example, thecorresponding script involves the following steps: initially generating a trajectoryof cutting in (agent.cut_in()), followed by generating the corresponding egocar trajectory (agent.forward()), and ultimately utilizing the saving functionfrom utilities to directly output the trajectory of the ego-car and other agents inarray format. For additional details, please refer to the supplementary materials.In the inference phase, we follow [37] to expand prompt inputs to a pre-definedtemplate, and the finetuned LLM can directly output the trajectory array.


World models, especially in autonomous driving, are trending and drawing extensive attention due to their ca- pacity for comprehending driving environments. The estab- lished world model holds immense potential for the gen- eration of high-quality driving videos, and driving poli- cies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gam- ing environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model en- tirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the
Spurred by insights from AGI (Artificial General Intelli-gence) and the principles of embodied AI, a profound trans-formation in autonomous driving is underway. Autonomousvehicles rely on sophisticated systems that engage with andcomprehend the real driving world. At the heart of this evo-lution is the integration of world models [15,17–19]. Worldmodels hold great promise for generating diverse and real-istic driving videos, encompassing even long-tail scenarios,which can be utilized to train various driving perception ap-proaches. Furthermore, the predictive capabilities in worldmodels facilitate end-to-end driving, ushering in a new eraof autonomous driving experiences.Deriving latent dynamics of world models fromvisual signals was initially introduced in video predic-tion [8, 11, 19]. By extrapolating from observed visualsequences, video prediction methods can infer future statesof the environment, effectively modeling how objects andentities within a scene will evolve over time. However,modeling the intricate driving scenarios in pixel space ischallenging due to the large sampling space [5, 7]. Toalleviate this problem, recent research endeavors havesought innovative strategies to enhance sampling efficiency.ISO-Dream [52] explicitly disentangles visual dynamicsinto controllable and uncontrollable states. MILE [29]strategically incorporates world modeling within theBird’s Eye View (BEV) semantic segmentation space,complementing world modeling with imitation learning.SEM2 [13] further extends the Dreamer framework intoBEV segmentation maps, utilizing Reinforce Learning(RL) for training. Despite the progress witnessed in worldmodels, a critical limitation in relevant research lies in itspredominant focus on simulation environments.In this paper, we propose DriveDreamer, which pioneersthe construction of comprehensive world models from realdriving videos and human driver behaviors. Considering theintricate nature of modeling real-world driving scenes, weintroduce the Autonomous-driving Diffusion Model (Auto-DM), which empowers the ability to create a comprehen-sive representation of the complex driving environment. Wepropose a two-stage training pipeline. In the first stage,we train Auto-DM by incorporating traffic structural infor-mation as intermediate conditions, which significantly en-hances sampling efficiency. Consequently, Auto-DM ex-hibits remarkable capabilities in comprehending real-worlddriving scenes, particularly concerning the dynamic fore-ground objects and the static background. In the second-stage training, we establish the world model through videoprediction. Specifically, driving actions are employed to it-eratively update future traffic structural conditions, whichenables DriveDreamer to anticipate variations in the drivingenvironment based on different driving strategies. More-over, DriveDreamer extends its predictive prowess to fore-see forthcoming driving policies, drawing from historicalobservations and Auto-DM features. Thus creating a exe-cutable, and predictable driving world model.The main contributions of this paper can be summa-rized as follows: (1) We introduce DriveDreamer, whichis the first world model derived from real-world drivingscenarios. DriveDreamer can jointly enable the generationof high-quality driving videos and reasonable driving poli-cies. (2) To enhance the comprehension of real-world driv-ing scenes and expedite the world model convergence, weintroduce the Autonomous-driving Diffusion Model and atwo-stage training pipeline. The first-stage training enablesthe comprehension of traffic structural information, and thesecond-stage video prediction training empowers the pre-dictive capacity. (3) DriveDreamer can controllably gener-ate driving scene videos that are highly aligned with trafficconstraints (see <PIC>), enhancing the training of drivingperception methods (e.g., 3D detection). Besides, Drive-Dreamer can generate future driving policies based on his-torical observations and Auto-DM features. Notably, Drive-Dreamer achieves promising planning results in open-loopassessments on the nuScenes dataset.
2. Related Work2.1. Diffusion ModelDiffusion models represent a family of probabilistic gen-erative models that progressively introduce noise to dataand subsequently learn to reverse this process for the pur-pose of generating samples [73]. These models have re-cently garnered significant attention due to their exceptionalperformance in various applications, setting new bench-marks in image synthesis [1, 14, 49, 55, 57], video gener-ation [21, 23, 35, 60, 67, 74], and 3D content generation[6, 43, 53, 69]. To enhance the controllable generation ca-pability, ControlNet [76], GLIGEN [42], T2I-Adapter [48]
and Composer [32] have been introduced to utilize variouscontrol inputs, including depth maps, segmentation maps,canny edges, and sketches. Concurrently, BEVControl [72],MagicDrive [12] and DrivingDiffuson [41] incorporate lay-out conditions to enhance image generation. The funda-mental essence of diffusion-based generative models lies intheir capacity to comprehend and understand the intricaciesof the world. Harnessing the power of these diffusion mod-els, DriveDreamer seeks to comprehend the complex realmof autonomous-driving scenarios.2.2. Video GenerationVideo generation and video prediction are effectiveapproaches to understanding the visual world. In therealm of video generation, several standard architectureshave been employed, including Variational Autoencoders(VAEs) [8,28], auto-regressive models [34,56,61,70], flow-based models [40], and Generative Adversarial Networks(GANs) [46, 58, 62, 65]. Recently, the burgeoning diffu-sion models [9, 24, 25, 49, 50, 57] have also been extendedto the domain of video generation. Video diffusion mod-els [21, 23, 35, 60, 67, 74] exhibit higher-quality video gen-eration capabilities, producing realistic frames and transi-tions between frames while offering enhanced controllabil-ity. They accommodate various input control conditionssuch as text, canny, sketch, semantic maps, and depth maps.Video prediction models represent a specialized formof video generation models, sharing numerous similari-ties. In particular, video prediction involves anticipatingfuture video changes based on historical video observa-tions [8, 11, 19, 27, 64]. DriveGAN [36] establishes asso-ciations between driving actions and pixels, predicting fu-ture driving videos by specifying future driving policies. Incontrast, DriveDreamer incorporates structured traffic con-ditions, text prompts, and driving actions as inputs, empow-ering precise, realistic video and action generation that arefaithfully aligned with real-world driving scenarios.2.3. World ModelsWorld models have been extensively explored in model-based imitation learning, demonstrating remarkable suc-cess in various applications [15–20, 37, 44, 59, 71]. Theseapproaches typically leverage Variational Autoencoders(VAE) [39] and Long Short-Term Memory (LSTM) [26]to model transition dynamics and rendering functionality.World methods target at establishing dynamic models of en-vironments, enabling agents to be predictive of the future.This aspect is of paramount importance in autonomous driv-ing, where precise predictions about the future are essentialfor safe maneuvering. However, constructing world mod-els in autonomous driving presents unique challenges, pri-marily due to the high sample complexity inherent in real-world driving tasks [5]. To address these problems, ISO-
Dream [52] introduces an explicit disentanglement of visualdynamics into controllable and uncontrollable states. MILE[29] strategically incorporates world modeling within theBEV semantic segmentation space, enhancing world mod-eling through imitation learning. SEM2 [13] extends theDreamer framework into BEV segmentation maps, em-ploying reinforcement learning for training. Despite theprogress witnessed in world models, a critical limitation inrelevant research lies in its predominant focus on simulationenvironments. The transition to real-world driving scenar-ios remains an under-explored frontier.3. DriveDreamerThe overall framework of DriveDreamer is depicted in<PIC>. The framework begins with an initial referenceframe I0 and its corresponding road structural informa-tion (i.e., HDMap H0 and 3D box B0). Within this con-text, DriveDreamer leverages the proposed ActionFormerto predict forthcoming road structural features in the latentspace. These predicted features serve as conditions andare provided to Auto-DM, which generates future drivingvideos. Simultaneously, the utilization of text prompts al-lows for dynamic adjustments to the driving scenario style(e.g., weather and time of the day). Moreover, Drive-Dreamer incorporates historical action information and themulti-scale latent features extracted from Auto-DM, whichare combined to generate reasonable future driving actions.In essence, DriveDreamer offers a comprehensive frame-work that seamlessly integrates multi-modal inputs to gen-erate future driving videos and driving policies, thereby ad-vancing the capabilities of autonomous-driving systems.Regarding the extensive search space of establishingworld models in real-world driving scenarios, we introducea two-stage training strategy for DriveDreamer. This strat-egy is designed to significantly enhance sampling efficiencyand expedite model convergence. The two-stage training isillustrated in <PIC>. There are two steps in the first-stagetraining. Step 1 involves utilizing the single-frame struc-tured condition, which guides DriveDreamer to generatedriving scene image, facilitating its comprehension of struc-tural traffic constraints. Step 2 extends its understandinginto video generation. The second-stage training enablesDriveDreamer to interact with the environment and predictfuture states effectively. This phase takes an initial frameimage along with its corresponding structured informationas input. Simultaneously, sequential driving actions are pro-vided, with the model expected to generate future drivingvideos and future driving actions. In the following sections,we delve into the specifics of the model architecture andtraining pipelines.


We introduce pixelSplat, a feed-forward model that learnsto reconstruct 3D radiance fields parameterized by 3D Gaus-sian primitives from pairs of images. Our model featuresreal-time and memory-efficient rendering for scalable train-ing as well as fast 3D reconstruction at inference time. Toovercome local minima inherent to sparse and locally sup-ported representations, we predict a dense probability dis-tribution over 3D and sample Gaussian means from thatprobability distribution. We make this sampling operationdifferentiable via a reparameterization trick, allowing us toback-propagate gradients through the Gaussian splattingrepresentation. We benchmark our method on wide-baselinenovel view synthesis on the real-world RealEstate10k andACID datasets, where we outperform state-of-the-art lightfield transformers and accelerate rendering by 2.5 ordersofmagnitude while reconstructing an interpretable and ed-itable 3D radiance field. Additional materials can be foundon the project website. 1
We investigate the problem of generalizable novel viewsynthesis from sparse image observations. This line ofwork has been revolutionized by differentiable render-ing [29, 40, 41, 50] but has also inherited its key weak-ness: training, reconstruction, and rendering are notoriouslymemory- and time-intensive because differentiable render-ing requires evaluating dozens or hundreds of points alongeach camera ray [58].This has motivated light-field transformers [10, 37, 43,47], where a ray is rendered by embedding it into a query to-ken and a color is obtained via cross-attention over image to-kens. While significantly faster than volume rendering, suchmethods are still far from real-time. Additionally, they donot reconstruct 3D scene representations that can be editedor exported for downstream tasks in vision and graphics.Meanwhile, recent work on single-scene novel view syn-thesis has shown that it is possible to use 3D Gaussian prim-itives to enable real-time rendering with little memory costvia rasterization-based volume rendering [19].We present pixelSplat, which brings the benefits ofa primitive-based 3D representation—fast and memory-efficient rendering as well as interpretable 3D structure—to generalizable view synthesis. This is no straightforwardtask. First, in real-world datasets, camera poses are onlyreconstructed up to an arbitrary scale factor. We addressthis by designing a multi-view epipolar transformer that re-liably infers this per-scene scale factor. Next, optimizingprimitive parameters directly via gradient descent suffersfrom local minima. In the single-scene case, this can beaddressed via non-differentiable pruning and division heuris-tics [19]. In contrast, in the generalizable case, we needto back-propagate gradients through the representation andthus cannot rely on non-differentiable operations. We thuspropose a method by which Gaussian primitives can im-plicitly be spawned or deleted during training, avoiding lo-cal minima, but which nevertheless maintains gradient flow.Specifically, we parameterize the positions (i.e., means) ofGaussians implicitly via dense probability distributions pre-dicted by our encoder. In each forward pass, we sample
Meanwhile, recent work on single-scene novel view syn-thesis has shown that it is possible to use 3D Gaussian prim-itives to enable real-time rendering with little memory costvia rasterization-based volume rendering [19].We present pixelSplat, which brings the benefits ofa primitive-based 3D representation—fast and memory-efficient rendering as well as interpretable 3D structure—to generalizable view synthesis. This is no straightforwardtask. First, in real-world datasets, camera poses are onlyreconstructed up to an arbitrary scale factor. We addressthis by designing a multi-view epipolar transformer that re-liably infers this per-scene scale factor. Next, optimizingprimitive parameters directly via gradient descent suffersfrom local minima. In the single-scene case, this can beaddressed via non-differentiable pruning and division heuris-tics [19]. In contrast, in the generalizable case, we needto back-propagate gradients through the representation andthus cannot rely on non-differentiable operations. We thuspropose a method by which Gaussian primitives can im-plicitly be spawned or deleted during training, avoiding lo-cal minima, but which nevertheless maintains gradient flow.Specifically, we parameterize the positions (i.e., means) ofGaussians implicitly via dense probability distributions pre-dicted by our encoder. In each forward pass, we sample
Gaussian primitive locations from this distribution. We makethe sampling operation differentiable via a reparameteriza-tion trick that couples the density of a sampled Gaussianprimitive to the probability of that location. When receivinga gradient that would increase the opacity of a Gaussian ata 3D location, our model increases the probability that theGaussian will be sampled at that location again in the future.We demonstrate the efficacy of our method by showcas-ing, for the first time, how a 3D Gaussian splatting represen-tation can be predicted in a single forward pass from just apair of images. In other words, we demonstrate how3D Gaus-sians can be integrated in an end-to-end differentiable system.We significantly outperform previous black-box based lightfield transformers on the real-world ACID and RealEstate10kdatasets while drastically reducing both training and render-ing cost and generating explicit 3D scenes.
2. Related WorkSingle-scene novel view synthesis. Advancements in neu-ral rendering [50] and neural fields [29, 42, 57] have rev-olutionized 3D reconstruction and novel view synthesisfrom collections of posed images. Recent approaches gen-erally create 3D scene representations by backpropagatingimage-space photometric error through differentiable ren-derers. Early methods employed voxel grids and learnedrendering techniques [27, 31, 40]. More recently, neuralfields [2, 28, 29, 57] and volume rendering [27, 29, 49]have become the de-facto standard. However, a key hurdleof these methods is their high computational demand, asrendering usually requires dozens of queries of the neuralfield per ray. Discrete data structures can accelerate render-ing [6, 12, 25, 30] but fall short of real-time rendering at highresolutions. 3D Gaussian splatting [19] solves this problemby representing the radiance field using 3D Gaussians thatcan efficiently be rendered via rasterization. However, allsingle-scene optimization methods require dozens of imagesto achieve high-quality novel view synthesis. In this work,we train neural networks to estimate the parameters of a3D Gaussian primitive scene representation from just twoimages in a single forward pass.Prior-based 3D Reconstruction and View Synthesis. Gen-eralizable novel view synthesis seeks to enable 3D recon-struction and novel view synthesis from only a handful ofimages per scene. If proxy geometry (e.g., depth maps) isavailable, machine learning can be combined with image-based rendering [1, 22, 36, 56] to produce convincing re-sults. Neural networks can also be trained to directly regressmulti-plane images for small-baseline novel view synthe-sis [45, 53, 60, 61]. Large-baseline novel view synthesis,however, requires full 3D representations. Early approachesbased on neural fields [32, 41] encoded 3D scenes in indi-vidual latent codes and were thus limited to single-objectscenes. Preserving end-to-end locality and shift equivariancebetween encoder and scene representation via pixel-alignedfeatures [14, 23, 39, 52, 58] or via transformers [35, 54]has enabled generalization to unbounded scenes. Inspiredby classical multi-view stereo, neural networks have alsobeen combined with cost volumes to match features acrossviews [5, 7, 18, 26]. While the above methods infer inter-pretable 3D representations in the form of signed distancesor radiance fields, recent light field scene representationstrade interpretability for faster rendering [10, 37, 43, 46, 47].Our method presents the best of both worlds: it infers aninterpretable 3D scene representation in the form of 3DGaussians while accelerating rendering by three orders ofmagnitude compared to light field transformers.Scale ambiguity in machine learning for multi-view ge-ometry. Prior work has recognized the challenge of scenescale ambiguity. In monocular depth estimation, state-of-the-art models rely on sophisticated scale-invariant depthlosses [11, 13, 33, 34]. In novel view synthesis, recent single-image 3D diffusion models trained on real-world data rescale3D scenes according to heuristics on depth statistics and con-dition their encoders on scene scale [4, 38, 51]. In this work,we instead build a multi-view encoder that can infer thescale of the scene. We accomplish this using an epipolartransformer that finds cross-view pixel correspondences andassociates them with positionally encoded depth values [16].In practice, this means that a neural network makingWe report quantitative results in Table 1. Our method out-performs the baselines on all metrics, with especially sig-nificant improvements in perceptual distance (LPIPS). Compared to the baselines,our method is better at capturing fine details and correctlyinferring 3D structure in portions of each scene that are onlyobserved by one reference view.Training and Inference cost. As shown in Table 1, ourmethod is significantly less resource-intensive than the base-lines. Compared to the next-fastest one, our method’s cost toinfer a single scene (encoding) and then render 100 images(decoding), the approximate number in a RealEstate10k orACID sequence, is about 650 times less. Our method alsouses significantly less memory per ray at training timpredictions about the geometry of a scene from a singleimage cannot possibly predict the depth that matches theposes reconstructed by structure-from-motion. In monocular<PIC>. Scale ambiguity. SfM does not reconstruct camera posesin real-world, metric scale—poses are scaled by an arbitrary scalefactor that is different for each scene. To render correct views, ourmodel’s 3D reconstruction needs to be consistent with this arbitraryscale. We illustrate how our epipolar encoder solves this problem.Features belonging to the ray’s corresponding pixel on the left arecompared with features sampled along the epipolar line on the right.Epipolar samples are augmented with their positionally-encodeddepths along the ray, which allows our encoder to record correctdepths. Recorded depths are later used for depth prediction.depth estimation, this has been addressed via scale-invariantlosses [11, 13, 33]. Our encoder similarly has to predictthe geometry of the scene, chiefly via the position of eachGaussian primitive, which depends on the per-scene scale si.Refer to <PIC> for an illustration of this challenge.each Gaussian. <PIC>illustrates the process by which we predict parameters ofpixel-aligned Gaussians from a single reference image.visualize the Gaussians it produces from views that are faroutside the training distribution in <PIC>. We visualizepoint clouds using the version of our model that has beenfine-tuned with a depth regularizer. Note that while the re-sulting Gaussians facilitate high-fidelity novel-view synthe-sis for in-distribution camera poses, they suffer from thesame failure modes as 3D Gaussians optimized using theoriginal 3D Gaussian splatting method [19]. Specifically, re-flective surfaces are often transparent, and Gaussians appea