Skip to content

Releases: kwea123/nerf_pl

NeRF-W on brandenburg gate

27 Jan 00:13
cf7380f
Compare
Choose a tag to compare

Used for nerfw branch.

Train command (trained on 8 time downscaled images, just for proof of implementation):

python prepare_phototourism.py --root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ --img_downscale 8

python train.py \
  --root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ --dataset_name phototourism \
  --img_downscale 8 --use_cache \
  --N_importance 64 --N_samples 64 --encode_a --encode_t --beta_min 0.03 --N_vocab 1500 --N_emb_xyz 15 \
  --num_epochs 20 --batch_size 1024 \
  --optimizer adam --lr 5e-4 --lr_scheduler cosine \
  --exp_name brandenburg_scale8_nerfw

Profiler Report

Action                      	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-----------------------------------------------------------------------------------------------------------------------------
Total                       	|  -              	|_              	|  2.5398e+04     	|  100 %          	|
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch          	|  1269.8         	|20             	|  2.5396e+04     	|  99.991         	|
run_training_batch          	|  0.14633        	|170760         	|  2.4988e+04     	|  98.384         	|
optimizer_step_and_closure_0	|  0.12823        	|170760         	|  2.1896e+04     	|  86.212         	|
training_step_and_backward  	|  0.1241         	|170760         	|  2.1192e+04     	|  83.438         	|
model_backward              	|  0.099837       	|170760         	|  1.7048e+04     	|  67.124         	|
model_forward               	|  0.024055       	|170760         	|  4107.6         	|  16.173         	|
on_train_batch_end          	|  0.00052083     	|170760         	|  88.938         	|  0.35018        	|
get_train_batch             	|  0.00023393     	|170760         	|  39.946         	|  0.15728        	|
evaluation_step_and_end     	|  0.52576        	|21             	|  11.041         	|  0.043472       	|
cache_result                	|  1.2894e-05     	|854050         	|  11.012         	|  0.043357       	|
on_after_backward           	|  1.0743e-05     	|170760         	|  1.8345         	|  0.007223       	|
on_batch_start              	|  1.0535e-05     	|170760         	|  1.799          	|  0.0070832      	|
on_batch_end                	|  9.6894e-06     	|170760         	|  1.6546         	|  0.0065145      	|
on_before_zero_grad         	|  8.5198e-06     	|170760         	|  1.4548         	|  0.0057282      	|
training_step_end           	|  6.6891e-06     	|170760         	|  1.1422         	|  0.0044974      	|
on_train_batch_start        	|  5.9285e-06     	|170760         	|  1.0124         	|  0.003986       	|
on_validation_end           	|  0.027978       	|21             	|  0.58754        	|  0.0023133      	|
on_validation_batch_end     	|  0.00055518     	|21             	|  0.011659       	|  4.5904e-05     	|
on_epoch_start              	|  0.00054319     	|20             	|  0.010864       	|  4.2774e-05     	|
on_validation_start         	|  0.00024484     	|21             	|  0.0051417      	|  2.0244e-05     	|
on_validation_batch_start   	|  5.3095e-05     	|21             	|  0.001115       	|  4.3901e-06     	|
validation_step_end         	|  2.1799e-05     	|21             	|  0.00045779     	|  1.8024e-06     	|
on_train_epoch_start        	|  1.7319e-05     	|20             	|  0.00034637     	|  1.3638e-06     	|
on_epoch_end                	|  1.5776e-05     	|20             	|  0.00031551     	|  1.2423e-06     	|
on_train_end                	|  0.0002874      	|1              	|  0.0002874      	|  1.1316e-06     	|
on_validation_epoch_end     	|  1.1708e-05     	|21             	|  0.00024586     	|  9.6803e-07     	|
on_validation_epoch_start   	|  8.0324e-06     	|21             	|  0.00016868     	|  6.6415e-07     	|
on_train_start              	|  0.00015864     	|1              	|  0.00015864     	|  6.2463e-07     	|
on_train_epoch_end          	|  7.2367e-06     	|20             	|  0.00014473     	|  5.6986e-07     	|
on_fit_start                	|  1.4059e-05     	|1              	|  1.4059e-05     	|  5.5355e-08     	|

Eval command (used for scale2_epoch29 model):

python eval.py \
  --root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ \
  --dataset_name phototourism --scene_name brandenburg_test \
  --split test --N_samples 256 --N_importance 256 \
  --N_vocab 1500 --encode_a --encode_t \
  --ckpt_path ckpts/brandenburg/scale2/epoch\=29.ckpt \
  --chunk 16384 --img_wh 320 240

You can change the test camera path in eval.py.

NeRF-W with color perturbation and occluder

24 Jan 12:51
Compare
Choose a tag to compare

Used for nerfw branch.

Train command:

python train.py \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --dataset_name blender --img_wh 200 200 --data_perturb color occ \
  --N_importance 64 --N_samples 64 --noise_std 0 --encode_a --encode_t --beta_min 0.1 \
  --num_epochs 20 --batch_size 1024 \
  --optimizer adam --lr 5e-4 --lr_scheduler cosine \
  --exp_name lego_nerfw_all

Eval command:

python eval.py \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --dataset_name blender --split test --img_wh 200 200 \
  --N_importance 64 --encode_a --encode_t --beta_min 0.1 \
  --ckpt_path ckpts/lego_nerfw_all/epoch\=19.ckpt \
  --scene_name nerfw_all

Eval output: Mean PSNR : 24.86

Profiler Report

Action                      	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-----------------------------------------------------------------------------------------------------------------------------
Total                       	|  -              	|_              	|  1.1659e+04     	|  100 %          	|
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch          	|  582.57         	|20             	|  1.1651e+04     	|  99.931         	|
run_training_batch          	|  0.14307        	|78140          	|  1.1179e+04     	|  95.882         	|
optimizer_step_and_closure_0	|  0.12437        	|78140          	|  9718.4         	|  83.352         	|
training_step_and_backward  	|  0.12006        	|78140          	|  9381.8         	|  80.465         	|
model_backward              	|  0.095661       	|78140          	|  7475.0         	|  64.111         	|
model_forward               	|  0.024116       	|78140          	|  1884.5         	|  16.162         	|
evaluation_step_and_end     	|  1.8998         	|161            	|  305.86         	|  2.6233         	|
on_train_batch_end          	|  0.00053565     	|78140          	|  41.856         	|  0.35898        	|
get_train_batch             	|  0.00026832     	|78140          	|  20.966         	|  0.17982        	|
cache_result                	|  1.6708e-05     	|391370         	|  6.5391         	|  0.056084       	|
on_after_backward           	|  1.3945e-05     	|78140          	|  1.0897         	|  0.0093458      	|
on_batch_start              	|  1.1257e-05     	|78140          	|  0.87959        	|  0.007544       	|
on_batch_end                	|  1.0574e-05     	|78140          	|  0.82626        	|  0.0070866      	|
on_before_zero_grad         	|  9.9755e-06     	|78140          	|  0.77948        	|  0.0066854      	|
training_step_end           	|  7.3524e-06     	|78140          	|  0.57452        	|  0.0049275      	|
on_train_batch_start        	|  7.0481e-06     	|78140          	|  0.55074        	|  0.0047235      	|
on_validation_end           	|  0.025579       	|21             	|  0.53715        	|  0.004607       	|
on_validation_batch_end     	|  0.00039767     	|161            	|  0.064025       	|  0.00054912     	|
on_epoch_start              	|  0.00074399     	|20             	|  0.01488        	|  0.00012762     	|
on_validation_start         	|  0.00024646     	|21             	|  0.0051757      	|  4.439e-05      	|
on_train_end                	|  0.0033677      	|1              	|  0.0033677      	|  2.8884e-05     	|
on_validation_batch_start   	|  1.301e-05      	|161            	|  0.0020947      	|  1.7965e-05     	|
validation_step_end         	|  9.2702e-06     	|161            	|  0.0014925      	|  1.2801e-05     	|
on_epoch_end                	|  1.6658e-05     	|20             	|  0.00033316     	|  2.8575e-06     	|
on_validation_epoch_end     	|  1.4696e-05     	|21             	|  0.00030862     	|  2.6469e-06     	|
on_train_start              	|  0.00020975     	|1              	|  0.00020975     	|  1.799e-06      	|
on_validation_epoch_start   	|  9.7831e-06     	|21             	|  0.00020545     	|  1.7621e-06     	|
on_train_epoch_start        	|  9.096e-06      	|20             	|  0.00018192     	|  1.5603e-06     	|
on_train_epoch_end          	|  8.8208e-06     	|20             	|  0.00017642     	|  1.5131e-06     	|
on_fit_start                	|  1.3749e-05     	|1              	|  1.3749e-05     	|  1.1792e-07     	|

NeRF-A with color perturbation

24 Jan 04:26
Compare
Choose a tag to compare

Used for nerfw branch.

Train command:

python train.py \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --dataset_name blender --img_wh 200 200 --data_perturb color \
  --N_importance 64 --N_samples 64 --noise_std 0 --encode_a \
  --num_epochs 20 --batch_size 1024 \
  --optimizer adam --lr 5e-4 --lr_scheduler cosine \
  --exp_name lego_nerfa_color

Eval command:

python eval.py \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --dataset_name blender --split test --img_wh 200 200 \
  --N_importance 64 --encode_a \
  --ckpt_path ckpts/lego_nerfa_color/epoch\=19.ckpt \
  --scene_name nerfa_color

Eval output: Mean PSNR : 28.20

Profiler Report

Action                      	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-----------------------------------------------------------------------------------------------------------------------------
Total                       	|  -              	|_              	|  1.0174e+04     	|  100 %          	|
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch          	|  508.31         	|20             	|  1.0166e+04     	|  99.922         	|
run_training_batch          	|  0.12504        	|78140          	|  9770.7         	|  96.036         	|
optimizer_step_and_closure_0	|  0.10593        	|78140          	|  8277.7         	|  81.362         	|
training_step_and_backward  	|  0.10272        	|78140          	|  8026.6         	|  78.893         	|
model_backward              	|  0.081418       	|78140          	|  6362.0         	|  62.532         	|
model_forward               	|  0.021105       	|78140          	|  1649.1         	|  16.209         	|
evaluation_step_and_end     	|  1.6237         	|161            	|  261.41         	|  2.5694         	|
on_train_batch_end          	|  0.00040171     	|78140          	|  31.39          	|  0.30853        	|
get_train_batch             	|  0.0002557      	|78140          	|  19.981         	|  0.19639        	|
cache_result                	|  1.4961e-05     	|391370         	|  5.8553         	|  0.057551       	|
on_after_backward           	|  1.136e-05      	|78140          	|  0.88768        	|  0.008725       	|
on_batch_start              	|  1.0067e-05     	|78140          	|  0.78663        	|  0.0077318      	|
on_batch_end                	|  9.6172e-06     	|78140          	|  0.75149        	|  0.0073863      	|
on_before_zero_grad         	|  9.0155e-06     	|78140          	|  0.70447        	|  0.0069242      	|
on_validation_end           	|  0.026961       	|21             	|  0.56618        	|  0.005565       	|
training_step_end           	|  6.6222e-06     	|78140          	|  0.51746        	|  0.0050861      	|
on_train_batch_start        	|  6.3198e-06     	|78140          	|  0.49383        	|  0.0048539      	|
on_validation_batch_end     	|  0.00036434     	|161            	|  0.058659       	|  0.00057656     	|
on_epoch_start              	|  0.00047801     	|20             	|  0.0095601      	|  9.3966e-05     	|
on_validation_start         	|  0.00024532     	|21             	|  0.0051518      	|  5.0637e-05     	|
on_validation_batch_start   	|  1.2674e-05     	|161            	|  0.0020406      	|  2.0057e-05     	|
validation_step_end         	|  8.6672e-06     	|161            	|  0.0013954      	|  1.3716e-05     	|
on_epoch_end                	|  1.7733e-05     	|20             	|  0.00035466     	|  3.4859e-06     	|
on_train_end                	|  0.00025723     	|1              	|  0.00025723     	|  2.5283e-06     	|
on_validation_epoch_end     	|  1.1715e-05     	|21             	|  0.00024602     	|  2.4181e-06     	|
on_train_epoch_start        	|  1.1723e-05     	|20             	|  0.00023446     	|  2.3045e-06     	|
on_train_start              	|  0.00021311     	|1              	|  0.00021311     	|  2.0946e-06     	|
on_validation_epoch_start   	|  8.2239e-06     	|21             	|  0.0001727      	|  1.6975e-06     	|
on_train_epoch_end          	|  8.1054e-06     	|20             	|  0.00016211     	|  1.5934e-06     	|
on_fit_start                	|  1.3379e-05     	|1              	|  1.3379e-05     	|  1.315e-07      	|

NeRF-U with occluder

23 Jan 12:34
Compare
Choose a tag to compare

Used for nerfw branch.

Train command:

python train.py \
  --dataset_name blender --img_wh 200 200 \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --N_importance 64 --N_samples 64 --noise_std 0 \
  --num_epochs 20 --batch_size 1024 \
  --optimizer adam --lr 5e-4 --lr_scheduler cosine \
  --exp_name lego_nerfu_occ --beta_min 0.1 --data_perturb occ --encode_t

Eval command:

python eval.py \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
  --dataset_name blender --img_wh 200 200 --split test \
  --N_importance 64 \
  --ckpt_path ckpts/lego_nerfw_occ/epoch\=19.ckpt \
  --encode_t --beta_min 0.1 \
  --scene_name nerfu_occ

Eval output: Mean PSNR : 28.60

Note I use a very small image size (200x200) to speed up my experiments.

Profiler Report

Action                      	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-----------------------------------------------------------------------------------------------------------------------------
Total                       	|  -              	|_              	|  1.0901e+04     	|  100 %          	|
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch          	|  544.74         	|20             	|  1.0895e+04     	|  99.947         	|
run_training_batch          	|  0.13381        	|78140          	|  1.0456e+04     	|  95.921         	|
optimizer_step_and_closure_0	|  0.11587        	|78140          	|  9053.8         	|  83.057         	|
training_step_and_backward  	|  0.11173        	|78140          	|  8730.4         	|  80.09          	|
model_backward              	|  0.088715       	|78140          	|  6932.2         	|  63.595         	|
model_forward               	|  0.02281        	|78140          	|  1782.4         	|  16.351         	|
evaluation_step_and_end     	|  1.7842         	|161            	|  287.25         	|  2.6352         	|
on_train_batch_end          	|  0.00042159     	|78140          	|  32.943         	|  0.30221        	|
get_train_batch             	|  0.00025085     	|78140          	|  19.602         	|  0.17982        	|
cache_result                	|  1.4803e-05     	|391370         	|  5.7934         	|  0.053147       	|
on_batch_start              	|  1.0333e-05     	|78140          	|  0.80743        	|  0.0074072      	|
on_after_backward           	|  9.5508e-06     	|78140          	|  0.7463         	|  0.0068464      	|
on_batch_end                	|  9.4638e-06     	|78140          	|  0.7395         	|  0.006784       	|
on_before_zero_grad         	|  8.3572e-06     	|78140          	|  0.65303        	|  0.0059908      	|
on_validation_end           	|  0.025442       	|21             	|  0.53429        	|  0.0049014      	|
training_step_end           	|  6.2163e-06     	|78140          	|  0.48574        	|  0.0044561      	|
on_train_batch_start        	|  5.99e-06       	|78140          	|  0.46806        	|  0.0042939      	|
on_validation_batch_end     	|  0.00042104     	|161            	|  0.067788       	|  0.00062187     	|
on_epoch_start              	|  0.00079988     	|20             	|  0.015998       	|  0.00014676     	|
on_validation_start         	|  0.00024023     	|21             	|  0.0050449      	|  4.6281e-05     	|
on_validation_batch_start   	|  1.391e-05      	|161            	|  0.0022395      	|  2.0544e-05     	|
validation_step_end         	|  8.4167e-06     	|161            	|  0.0013551      	|  1.2431e-05     	|
on_train_end                	|  0.0003507      	|1              	|  0.0003507      	|  3.2172e-06     	|
on_epoch_end                	|  1.611e-05      	|20             	|  0.0003222      	|  2.9558e-06     	|
on_train_epoch_start        	|  1.5704e-05     	|20             	|  0.00031408     	|  2.8813e-06     	|
on_validation_epoch_end     	|  1.4037e-05     	|21             	|  0.00029477     	|  2.7041e-06     	|
on_validation_epoch_start   	|  8.7303e-06     	|21             	|  0.00018334     	|  1.6819e-06     	|
on_train_start              	|  0.00016846     	|1              	|  0.00016846     	|  1.5454e-06     	|
on_train_epoch_end          	|  7.923e-06      	|20             	|  0.00015846     	|  1.4537e-06     	|
on_fit_start                	|  1.3867e-05     	|1              	|  1.3867e-05     	|  1.2721e-07     	|

release succulent plant model

08 May 11:24
8189e3b
Compare
Choose a tag to compare

succulent plant model by @SpongeGirl
image size 496x372, spheric pose
1

release silica model

07 May 07:06
26745d0
Compare
Choose a tag to compare

release silica nerf model and reconstructed mesh (spheric poses)
link to the data
image size is 504x378 (original size 4032x3024)

Usage: place the poses_bounds.npy under a folder $DIR (anywhere you want), then you can run

python eval.py \
   --root_dir $DIR \
   --dataset_name llff --scene_name silica \
   --img_wh 504 378 --N_importance 64 --spheric_poses --ckpt_path $CKPT_PATH

as usual. To extract the mesh, follow README_mesh.

release fern model

02 May 01:03
Compare
Choose a tag to compare

Command:

python train.py \
  --dataset_name llff \
  --root_dir /home/ubuntu/data/nerf_example_data/nerf_llff_data/fern/ \
  --N_importance 64 --img_wh 504 378 \
  --batch_size 1024 --num_epochs 30 \
  --optimizer adam --lr 5e-4 \
  --lr_scheduler steplr --decay_step 10 20 --decay_gamma 0.5 \
  --exp_name fern

Profile

Profiler Report

Action              	|  Mean duration (s)	|  Total time (s) 
-----------------------------------------------------------------
on_train_start      	|  0.00023312     	|  0.00023312     
on_epoch_start      	|  0.00029521     	|  0.0088563      
get_train_batch     	|  0.00023997     	|  25.456         
on_batch_start      	|  6.1591e-06     	|  0.65317        
model_forward       	|  0.019652       	|  2084.1         
model_backward      	|  0.069537       	|  7374.4         
on_after_backward   	|  1.5543e-06     	|  0.16483        
optimizer_step      	|  0.0037302      	|  395.59         
on_batch_end        	|  0.00030407     	|  32.247         
on_epoch_end        	|  9.9102e-06     	|  0.00029731     
on_train_end        	|  0.00036468     	|  0.00036468     

release lego model

20 Apr 02:47
a070d07
Compare
Choose a tag to compare

lego model and reconstructed mesh.
Command:

python train.py \
   --dataset_name blender \
   --root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego/ \
   --N_importance 64 --img_wh 400 400 --noise_std 0 \
   --batch_size 1024 --num_epochs 16 \
   --optimizer adam --lr 5e-4 \
   --lr_scheduler steplr --decay_step 2 4 8 --decay_gamma 0.5 \
   --exp_name exp3

Detailed profile:

Action              	|  Mean duration (s)	|  Total time (s) 
-----------------------------------------------------------------
on_train_start      	|  1.2281e-05     	|  1.2281e-05     
on_epoch_start      	|  6.1691e-06     	|  9.8706e-05     
get_train_batch     	|  0.00023678     	|  59.198         
on_batch_start      	|  4.4245e-06     	|  1.1061         
model_forward       	|  0.041729       	|  1.0432e+04     
model_backward      	|  0.046964       	|  1.1741e+04     
on_after_backward   	|  1.5339e-06     	|  0.38347        
optimizer_step      	|  0.0035952      	|  898.81         
on_batch_end        	|  4.1799e-06     	|  1.045          
on_epoch_end        	|  5.4906e-06     	|  8.785e-05      
on_train_end        	|  9.583e-06      	|  9.583e-06