initial commit

README.md

@@ -1,3 +1,172 @@
+<!-- # NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields -->
+
+<div align="center">
+    <img src="demo/nerf-mae_teaser.png" width="85%">
+    <img src="demo/nerf-mae_teaser.jpeg" width="85%">
+</div>
+<!-- <p align="center">
+<img src="demo/nerf-mae_teaser.jpeg" width="100%">
+</p> -->
+
+<br>
+<div align="center">
+
+[![arXiv](https://img.shields.io/badge/arXiv-2404.01300-gray?style=for-the-badge&logo=arxiv&logoColor=white&color=B31B1B)](https://arxiv.org/abs/2404.01300)
+[![Project Page](https://img.shields.io/badge/Project-Page-orange?style=for-the-badge&logoColor=white&labelColor=gray&link=https%3A%2F%2Fnerf-mae.github.io%2F)](https://nerf-mae.github.io)
+[![Pytorch](https://img.shields.io/badge/Pytorch-%3E1.12-gray?style=for-the-badge&logo=pytorch&logoColor=white&labelColor=gray&color=ee4c2c&link=https%3A%2F%2Fnerf-mae.github.io%2F)](https://pytorch.org/)
+[![Cite](https://img.shields.io/badge/Cite-Bibtex-gray?style=for-the-badge&logoColor=white&color=F7A41D
+)](https://github.com/zubair-irshad/NeRF-MAE?tab=readme-ov-file#citation)
+[![Video](https://img.shields.io/badge/youtube-video-CD201F?style=for-the-badge&logo=youtube&labelColor=grey
+)](https://youtu.be/D60hlhmeuJI?si=d4RfHAwBJgLJXdKj)
+
+
+
+</div>
+
+---
+
+<a href="https://www.tri.global/" target="_blank">
+ <img align="right" src="demo/GeorgiaTech_RGB.png" width="18%"/>
+</a>
+
+<a href="https://www.tri.global/" target="_blank">
+ <img align="right" src="demo/tri-logo.png" width="17%"/>
+</a>
+
+### [Project Page](https://nerf-mae.github.io/) | [arXiv](https://arxiv.org/abs/2308.12967) | [PDF](https://arxiv.org/pdf/2308.12967.pdf)
+
+
+
+**NeRF-MAE : Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields**
+
+<a href="https://zubairirshad.com"><strong>Muhammad Zubair Irshad</strong></a>
+·
+<a href="https://zakharos.github.io/"><strong>Sergey Zakharov</strong></a>
+·
+<a href="https://www.linkedin.com/in/vitorguizilini"><strong>Vitor Guizilini</strong></a>
+·
+<a href="https://adriengaidon.com/"><strong>Adrien Gaidon</strong></a>
+·
+<a href="https://faculty.cc.gatech.edu/~zk15/"><strong>Zsolt Kira</strong></a>
+·
+<a href="https://www.tri.global/about-us/dr-rares-ambrus"><strong>Rares Ambrus</strong></a>
+<br> **European Conference on Computer Vision, ECCV 2024**<br>
+
+<b> Toyota Research Institute &nbsp; | &nbsp; Georgia Institute of Technology</b>
+
+## 💡 Highlights
+- **NeRF-MAE**: The first large-scale pretraining utilizing Neural Radiance Fields (NeRF) as an input modality. We pretrain a single Transformer model on thousands of NeRFs for 3D representation learning.
+- **NeRF-MAE Dataset**: A large-scale NeRF pretraining and downstream task finetuning dataset.
+
+## Citation
+
+If you find this repository or our dataset useful, please star ⭐ this repository and consider citing 📝:
+
+```
+@inproceedings{irshad2024nerfmae,
+    title={NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields},
+    author={Muhammad Zubair Irshad and Sergey Zakharov and Vitor Guizilini and Adrien Gaidon and Zsolt Kira and Rares Ambrus},
+    booktitle={European Conference on Computer Vision (ECCV)},
+    year={2024}
+    }
+```
+
+### Contents
+ - [🌇  Environment](#-environment)
+ - [⛳ Model Usage and Checkpoints](#-model-usage-and-checkpoints)
+ - [🗂️ Dataset](#-dataset)
+
+ ## 🌇  Environment
+
+Create a python 3.7 virtual environment and install requirements:
+
+```bash
+cd $NeRF-MAE repo
+conda create -n nerf-mae python=3.9
+conda activate nerf-mae
+pip install --upgrade pip
+pip install -r requirements.txt
+pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
+```
+The code was built and tested on **cuda 11.3**
+
+Compile CUDA extension, to run downstream task finetuning, as described in [NeRF-RPN](https://github.com/lyclyc52/NeRF_RPN):
+
+```bash
+cd $NeRF-MAE repo
+cd nerf_rpn/model/rotated_iou/cuda_op
+python setup.py install
+cd ../../../..
+
+```
+
+## ⛳ Model Usage and Checkpoints
+
+NeRF-MAE is structured to provide easy access to pretrained NeRF-MAE models (and reproductions), to facilitate use for various downstream tasks. This is for extracting good visual features from NeRFs if you don't have resources for large-scale pretraining. Our pretraining provides an easy-to-access embedding of any NeRF scene, which can be used for a variety of downstream tasks in a straightforwaed way. 
+
+We have released 1.pretrained and 2. finetuned checkpoints to start using our codebase out-of-the-box. Below is a sample useage of our model with spelled out comments in a few lines of code:
+
+```
+import torch
+# Load data from the specified folder and filename with the given resolution.
+res, rgbsigma = load_data(folder_name, filename, resolution=args.resolution)
+
+# Build the model using provided arguments.
+model = build_model(args)
+
+# Load checkpoint if provided.
+if args.checkpoint:
+    model.load_state_dict(torch.load(args.checkpoint, map_location="cpu")["state_dict"])
+    model.eval()  # Set model to evaluation mode.
+
+# Run inference getting the features out for downsteam usage
+with torch.no_grad():
+    pred = model([rgbsigma], is_eval=True)[3]  # Extract only predictions.
+
+```
+
+### 1. How to plug these features for downstream 3D bounding detection from NeRFs (i.e. plug-and-play with a [NeRF-RPN](https://github.com/lyclyc52/NeRF_RPN) OBB prediction head)
+
+Please also see the section on [Finetuning](#-finetuning). Our released finetuned checkpoint achieves state-of-the-art on 3D object detection in NeRFs. To run evaluation using our finetuned checkpoint on the dataset provided by NeRF-RPN, please run the below script, after updating the paths to the pretrained checkpoint i.e. --checkpoint and  DATA_ROOT depending on evaluation done for ```Front3D``` or ```Scannet```:
+
+```
+bash test_fcos_pretrained.sh
+```
+
+Also see the cooresponding run file i.e. ```run_fcos_pretrained.py``` and our model adaptation i.e. ```SwinTransformer_FPN_Pretrained_Skip```. This is a minimal adaptation to plug and play our weights with a NeRF-RPN architecture and achieve significant boost in performance. 
+
+
+## 🗂️ Dataset
+
+Download the preprocessed datasets here. 
+
+- Pretraining dataset (comprising NeRF radiance and density grids). [Download link](https://s3.amazonaws.com/tri-ml-public.s3.amazonaws.com/github/nerfmae/NeRF-MAE_pretrain.tar.gz)
+- Finetuning dataset (comprising NeRF radiance and density grids and bounding box/semantic labelling annotations). [3D Object Detection (Provided by NeRF-RPN)](https://drive.google.com/drive/folders/1q2wwLi6tSXu1hbEkMyfAKKdEEGQKT6pj), [3D Semantic Segmentation (Coming Soon)](), [Voxel-Super Resolution (Coming Soon)]()
+
+
+Extract pretraining and finetuning dataset under ```NeRF-MAE/datasets```. The directory structure should look like this:
+
+```
+NeRF-MAE
+├── pretrain
+│   ├── features
+│   └── nerfmae_split.npz
+└── finetune
+    └── front3d_rpn_data
+        ├── features
+        ├── aabb
+        └── obb
+```
+
+
+Note: The above datasets are all you need to train and evaluate our method. Bonus: we will be releasing our multi-view rendered posed RGB images from FRONT3D, HM3D and Hypersim as well as Instant-NGP trained checkpoints soon (these comprise over 1M+ images and 3k+ NeRF checkpoints)
+
+Please note that our dataset was generated using the instruction from [NeRF-RPN](https://github.com/lyclyc52/NeRF_RPN) and [3D-CLR](https://vis-www.cs.umass.edu/3d-clr/). Please consider citing our work, NeRF-RPN and 3D-CLR if you find this dataset useful in your research. 
+
+Please also note that our dataset uses [Front3D](https://arxiv.org/abs/2011.09127), [Habitat-Matterport3D](https://arxiv.org/abs/2109.08238), [HyperSim](https://github.com/apple/ml-hypersim) and [ScanNet](https://www.scan-net.org/) as the base version of the dataset i.e. we train a NeRF per scene and extract radiance and desnity grid as well as aligned NeRF-grid 3D annotations. Please read the term of use for each dataset if you want to utilize the posed multi-view images for each of these datasets. 
+
+### For More details, please checkout out Paper, Github and Project Page!
+
 ---
 license: cc-by-nc-4.0
 ---

front3d_obb_finetuned.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7f0dee865f8629eee475cb87643c6af67fddefed8c9df9bf6d9414cd761e269
+size 314063759

nerf_mae_pretrained.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:713abb59691adbcc16e805cf6e42d8360e79113f0fd82ed8a8514350a7b60e05
+size 305663555

scannet_obb_finetuned.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bef943ef054ef0904e9fa5a0075a3f76be5274bf7dff11699b5a6f741dc987fa
+size 314063759