|
|
nohup: ignoring input |
|
|
W0908 09:31:40.935000 4132297 site-packages/torch/distributed/run.py:774] |
|
|
W0908 09:31:40.935000 4132297 site-packages/torch/distributed/run.py:774] ***************************************** |
|
|
W0908 09:31:40.935000 4132297 site-packages/torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
|
|
W0908 09:31:40.935000 4132297 site-packages/torch/distributed/run.py:774] ***************************************** |
|
|
wandb: Appending key for api.wandb.ai to your netrc file: /home/yitongli/.netrc |
|
|
wandb: Currently logged in as: liyitong (liyitong-Tsinghua University) to https://api.wandb.ai. Use `wandb login |
|
|
wandb: Tracking run with wandb version 0.21.3 |
|
|
wandb: Run data is saved locally in ./output/wandb/run-20250908_093154-g6k5a9jv |
|
|
wandb: Run `wandb offline` to turn off syncing. |
|
|
wandb: Syncing run fluent-field-871 |
|
|
wandb: ⭐️ View project at https://wandb.ai/liyitong-Tsinghua%20University/self-forcing |
|
|
wandb: 🚀 View run at https://wandb.ai/liyitong-Tsinghua%20University/self-forcing/runs/g6k5a9jv |
|
|
run dir: ./output/wandb/run-20250908_093154-g6k5a9jv/files |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
KV inference with 3 frames per block |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
ODERegression initialized. |
|
|
cache a block wise causal mask with block size of 3 frames |
|
|
BlockMask(shape=(1, 1, 32768, 32768), sparsity=42.52%, |
|
|
(0, 0) |
|
|
████░░ |
|
|
████░░ |
|
|
████░░░░░░░░ |
|
|
██████████░░ |
|
|
██████████░░ |
|
|
██████████░░░░░░░░ |
|
|
████████████████░░ |
|
|
████████████████░░ |
|
|
████████████████░░░░░░░░ |
|
|
██████████████████████░░ |
|
|
██████████████████████░░ |
|
|
██████████████████████░░░░░░░░ |
|
|
████████████████████████████░░ |
|
|
████████████████████████████░░ |
|
|
████████████████████████████████░░ |
|
|
████████████████████████████████░░ |
|
|
████████████████████████████████░░░░░░░░ |
|
|
████████████████████████████████████████ |
|
|
████████████████████████████████████████ |
|
|
████████████████████████████████████████ |
|
|
) |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_000000/model.pt |
|
|
training step 0... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:14, 5.45it/s]
Saving video: 1%| | 1/81 [00:00<00:14, 5.38it/s]
Saving video: 1%| | 1/81 [00:00<00:15, 5.13it/s]
Saving video: 1%| | 1/81 [00:00<00:14, 5.45it/s]
Saving video: 1%| | 1/81 [00:00<00:14, 5.61it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 111.27it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 110.34it/s]
Saving video: 1%| | 1/81 [00:00<00:15, 5.06it/s]
Saving video: 1%| | 1/81 [00:00<00:20, 3.98it/s]
Saving video: 1%| | 1/81 [00:00<00:16, 4.85it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 103.10it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 115.73it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 109.43it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 149.04it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 165.63it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 103.54it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 91.84it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 100.76it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 161.42it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 168.89it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 152.00it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 191.07it/s]
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 155.71it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 137.45it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 157.40it/s] |
|
|
Saving video: 53%|█████▎ | 43/81 [00:00<00:00, 129.36it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 139.19it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 176.48it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 185.88it/s]
Saving video: 80%|████████ | 65/81 [00:00<00:00, 153.41it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 153.51it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 138.28it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 145.59it/s] |
|
|
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 143.56it/s]
Saving video: 73%|███████▎ | 59/81 [00:00<00:00, 138.71it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 135.81it/s] |
|
|
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 144.20it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 145.59it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 144.56it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 126.04it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 119.46it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 123.71it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1... |
|
|
training step 2... |
|
|
training step 3... |
|
|
training step 4... |
|
|
training step 5... |
|
|
training step 6... |
|
|
training step 7... |
|
|
training step 8... |
|
|
training step 9... |
|
|
training step 10... |
|
|
training step 11... |
|
|
training step 12... |
|
|
training step 13... |
|
|
training step 14... |
|
|
training step 15... |
|
|
training step 16... |
|
|
training step 17... |
|
|
training step 18... |
|
|
training step 19... |
|
|
training step 20... |
|
|
training step 21... |
|
|
training step 22... |
|
|
training step 23... |
|
|
training step 24... |
|
|
training step 25... |
|
|
training step 26... |
|
|
training step 27... |
|
|
training step 28... |
|
|
training step 29... |
|
|
training step 30... |
|
|
training step 31... |
|
|
training step 32... |
|
|
training step 33... |
|
|
training step 34... |
|
|
training step 35... |
|
|
training step 36... |
|
|
training step 37... |
|
|
training step 38... |
|
|
training step 39... |
|
|
training step 40... |
|
|
training step 41... |
|
|
training step 42... |
|
|
training step 43... |
|
|
training step 44... |
|
|
training step 45... |
|
|
training step 46... |
|
|
training step 47... |
|
|
training step 48... |
|
|
training step 49... |
|
|
training step 50... |
|
|
training step 51... |
|
|
training step 52... |
|
|
training step 53... |
|
|
training step 54... |
|
|
training step 55... |
|
|
training step 56... |
|
|
training step 57... |
|
|
training step 58... |
|
|
training step 59... |
|
|
training step 60... |
|
|
training step 61... |
|
|
training step 62... |
|
|
training step 63... |
|
|
training step 64... |
|
|
training step 65... |
|
|
training step 66... |
|
|
training step 67... |
|
|
training step 68... |
|
|
training step 69... |
|
|
training step 70... |
|
|
training step 71... |
|
|
training step 72... |
|
|
training step 73... |
|
|
training step 74... |
|
|
training step 75... |
|
|
training step 76... |
|
|
training step 77... |
|
|
training step 78... |
|
|
training step 79... |
|
|
training step 80... |
|
|
training step 81... |
|
|
training step 82... |
|
|
training step 83... |
|
|
training step 84... |
|
|
training step 85... |
|
|
training step 86... |
|
|
training step 87... |
|
|
training step 88... |
|
|
training step 89... |
|
|
training step 90... |
|
|
training step 91... |
|
|
training step 92... |
|
|
training step 93... |
|
|
training step 94... |
|
|
training step 95... |
|
|
training step 96... |
|
|
training step 97... |
|
|
training step 98... |
|
|
training step 99... |
|
|
training step 100... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.54it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.36it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.53it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.11it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.56it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.47it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 151.66it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 151.40it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.26it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 135.50it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 116.33it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 135.36it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 127.81it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 199.87it/s]
Saving video: 1%| | 1/81 [00:00<00:14, 5.36it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 201.17it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 120.16it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 179.68it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 168.02it/s]
Saving video: 53%|█████▎ | 43/81 [00:00<00:00, 153.00it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 184.11it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 93.94it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 210.96it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 215.13it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 186.00it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 189.98it/s] |
|
|
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 170.67it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 197.20it/s]
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 195.20it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 183.45it/s]
Saving video: 84%|████████▍ | 68/81 [00:00<00:00, 184.81it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 169.10it/s] |
|
|
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 130.99it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.90it/s] |
|
|
Saving video: 80%|████████ | 65/81 [00:00<00:00, 172.68it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 156.31it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.23it/s] |
|
|
Saving video: 72%|███████▏ | 58/81 [00:00<00:00, 140.13it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 152.44it/s] |
|
|
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 141.76it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 123.35it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 101... |
|
|
training step 102... |
|
|
training step 103... |
|
|
training step 104... |
|
|
training step 105... |
|
|
training step 106... |
|
|
training step 107... |
|
|
training step 108... |
|
|
training step 109... |
|
|
training step 110... |
|
|
training step 111... |
|
|
training step 112... |
|
|
training step 113... |
|
|
training step 114... |
|
|
training step 115... |
|
|
training step 116... |
|
|
training step 117... |
|
|
training step 118... |
|
|
training step 119... |
|
|
training step 120... |
|
|
training step 121... |
|
|
training step 122... |
|
|
training step 123... |
|
|
training step 124... |
|
|
training step 125... |
|
|
training step 126... |
|
|
training step 127... |
|
|
training step 128... |
|
|
training step 129... |
|
|
training step 130... |
|
|
training step 131... |
|
|
training step 132... |
|
|
training step 133... |
|
|
training step 134... |
|
|
training step 135... |
|
|
training step 136... |
|
|
training step 137... |
|
|
training step 138... |
|
|
training step 139... |
|
|
training step 140... |
|
|
training step 141... |
|
|
training step 142... |
|
|
training step 143... |
|
|
training step 144... |
|
|
training step 145... |
|
|
training step 146... |
|
|
training step 147... |
|
|
training step 148... |
|
|
training step 149... |
|
|
training step 150... |
|
|
training step 151... |
|
|
training step 152... |
|
|
training step 153... |
|
|
training step 154... |
|
|
training step 155... |
|
|
training step 156... |
|
|
training step 157... |
|
|
training step 158... |
|
|
training step 159... |
|
|
training step 160... |
|
|
training step 161... |
|
|
training step 162... |
|
|
training step 163... |
|
|
training step 164... |
|
|
training step 165... |
|
|
training step 166... |
|
|
training step 167... |
|
|
training step 168... |
|
|
training step 169... |
|
|
training step 170... |
|
|
training step 171... |
|
|
training step 172... |
|
|
training step 173... |
|
|
training step 174... |
|
|
training step 175... |
|
|
training step 176... |
|
|
training step 177... |
|
|
training step 178... |
|
|
training step 179... |
|
|
training step 180... |
|
|
training step 181... |
|
|
training step 182... |
|
|
training step 183... |
|
|
training step 184... |
|
|
training step 185... |
|
|
training step 186... |
|
|
training step 187... |
|
|
training step 188... |
|
|
training step 189... |
|
|
training step 190... |
|
|
training step 191... |
|
|
training step 192... |
|
|
training step 193... |
|
|
training step 194... |
|
|
training step 195... |
|
|
training step 196... |
|
|
training step 197... |
|
|
training step 198... |
|
|
training step 199... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html |
|
|
warnings.warn( |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_000200/model.pt |
|
|
training step 200... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:17, 4.46it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.48it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.30it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 102.68it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.60it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.55it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.60it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.05it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 153.51it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.32it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 143.03it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 150.27it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 112.32it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 142.25it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.35it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 136.85it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 128.16it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 188.59it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 195.44it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 173.07it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 165.04it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 153.15it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 177.29it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 138.98it/s] |
|
|
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 172.06it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 159.03it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 192.57it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 179.65it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.57it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 176.23it/s]
Saving video: 73%|███████▎ | 59/81 [00:00<00:00, 158.67it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.84it/s] |
|
|
Saving video: 84%|████████▍ | 68/81 [00:00<00:00, 174.62it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 172.12it/s]
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 156.98it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 149.26it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 160.03it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 155.50it/s] |
|
|
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 166.83it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 152.12it/s] |
|
|
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 156.12it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 144.58it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 201... |
|
|
training step 202... |
|
|
training step 203... |
|
|
training step 204... |
|
|
training step 205... |
|
|
training step 206... |
|
|
training step 207... |
|
|
training step 208... |
|
|
training step 209... |
|
|
training step 210... |
|
|
training step 211... |
|
|
training step 212... |
|
|
training step 213... |
|
|
training step 214... |
|
|
training step 215... |
|
|
training step 216... |
|
|
training step 217... |
|
|
training step 218... |
|
|
training step 219... |
|
|
training step 220... |
|
|
training step 221... |
|
|
training step 222... |
|
|
training step 223... |
|
|
training step 224... |
|
|
training step 225... |
|
|
training step 226... |
|
|
training step 227... |
|
|
training step 228... |
|
|
training step 229... |
|
|
training step 230... |
|
|
training step 231... |
|
|
training step 232... |
|
|
training step 233... |
|
|
training step 234... |
|
|
training step 235... |
|
|
training step 236... |
|
|
training step 237... |
|
|
training step 238... |
|
|
training step 239... |
|
|
training step 240... |
|
|
training step 241... |
|
|
training step 242... |
|
|
training step 243... |
|
|
training step 244... |
|
|
training step 245... |
|
|
training step 246... |
|
|
training step 247... |
|
|
training step 248... |
|
|
training step 249... |
|
|
training step 250... |
|
|
training step 251... |
|
|
training step 252... |
|
|
training step 253... |
|
|
training step 254... |
|
|
training step 255... |
|
|
training step 256... |
|
|
training step 257... |
|
|
training step 258... |
|
|
training step 259... |
|
|
training step 260... |
|
|
training step 261... |
|
|
training step 262... |
|
|
training step 263... |
|
|
training step 264... |
|
|
training step 265... |
|
|
training step 266... |
|
|
training step 267... |
|
|
training step 268... |
|
|
training step 269... |
|
|
training step 270... |
|
|
training step 271... |
|
|
training step 272... |
|
|
training step 273... |
|
|
training step 274... |
|
|
training step 275... |
|
|
training step 276... |
|
|
training step 277... |
|
|
training step 278... |
|
|
training step 279... |
|
|
training step 280... |
|
|
training step 281... |
|
|
training step 282... |
|
|
training step 283... |
|
|
training step 284... |
|
|
training step 285... |
|
|
training step 286... |
|
|
training step 287... |
|
|
training step 288... |
|
|
training step 289... |
|
|
training step 290... |
|
|
training step 291... |
|
|
training step 292... |
|
|
training step 293... |
|
|
training step 294... |
|
|
training step 295... |
|
|
training step 296... |
|
|
training step 297... |
|
|
training step 298... |
|
|
training step 299... |
|
|
training step 300... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.69it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.49it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.25it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.20it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.51it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.55it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 154.86it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 147.46it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.46it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 137.84it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 141.70it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.44it/s]
Saving video: 1%| | 1/81 [00:00<00:12, 6.33it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 136.53it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 208.20it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 195.82it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 126.18it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 181.83it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 181.14it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 190.77it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 120.31it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 181.69it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 201.30it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 191.90it/s]
Saving video: 53%|█████▎ | 43/81 [00:00<00:00, 147.60it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 184.44it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 196.14it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.45it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 193.08it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 158.14it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 171.01it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 187.99it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.24it/s] |
|
|
Saving video: 74%|███████▍ | 60/81 [00:00<00:00, 154.06it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 168.47it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.76it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 157.19it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 154.74it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 152.46it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 139.24it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 143.27it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 301... |
|
|
training step 302... |
|
|
training step 303... |
|
|
training step 304... |
|
|
training step 305... |
|
|
training step 306... |
|
|
training step 307... |
|
|
training step 308... |
|
|
training step 309... |
|
|
training step 310... |
|
|
training step 311... |
|
|
training step 312... |
|
|
training step 313... |
|
|
training step 314... |
|
|
training step 315... |
|
|
training step 316... |
|
|
training step 317... |
|
|
training step 318... |
|
|
training step 319... |
|
|
training step 320... |
|
|
training step 321... |
|
|
training step 322... |
|
|
training step 323... |
|
|
training step 324... |
|
|
training step 325... |
|
|
training step 326... |
|
|
training step 327... |
|
|
training step 328... |
|
|
training step 329... |
|
|
training step 330... |
|
|
training step 331... |
|
|
training step 332... |
|
|
training step 333... |
|
|
training step 334... |
|
|
training step 335... |
|
|
training step 336... |
|
|
training step 337... |
|
|
training step 338... |
|
|
training step 339... |
|
|
training step 340... |
|
|
training step 341... |
|
|
training step 342... |
|
|
training step 343... |
|
|
training step 344... |
|
|
training step 345... |
|
|
training step 346... |
|
|
training step 347... |
|
|
training step 348... |
|
|
training step 349... |
|
|
training step 350... |
|
|
training step 351... |
|
|
training step 352... |
|
|
training step 353... |
|
|
training step 354... |
|
|
training step 355... |
|
|
training step 356... |
|
|
training step 357... |
|
|
training step 358... |
|
|
training step 359... |
|
|
training step 360... |
|
|
training step 361... |
|
|
training step 362... |
|
|
training step 363... |
|
|
training step 364... |
|
|
training step 365... |
|
|
training step 366... |
|
|
training step 367... |
|
|
training step 368... |
|
|
training step 369... |
|
|
training step 370... |
|
|
training step 371... |
|
|
training step 372... |
|
|
training step 373... |
|
|
training step 374... |
|
|
training step 375... |
|
|
training step 376... |
|
|
training step 377... |
|
|
training step 378... |
|
|
training step 379... |
|
|
training step 380... |
|
|
training step 381... |
|
|
training step 382... |
|
|
training step 383... |
|
|
training step 384... |
|
|
training step 385... |
|
|
training step 386... |
|
|
training step 387... |
|
|
training step 388... |
|
|
training step 389... |
|
|
training step 390... |
|
|
training step 391... |
|
|
training step 392... |
|
|
training step 393... |
|
|
training step 394... |
|
|
training step 395... |
|
|
training step 396... |
|
|
training step 397... |
|
|
training step 398... |
|
|
training step 399... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_000400/model.pt |
|
|
training step 400... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.76it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.74it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.82it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.48it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.02it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.59it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 152.73it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.68it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.56it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 148.57it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 130.55it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 140.24it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.81it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 195.52it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 198.27it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 127.46it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 175.38it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 191.46it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 183.90it/s]
Saving video: 1%| | 1/81 [00:00<00:20, 3.89it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 195.70it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 213.62it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 174.98it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 187.84it/s] |
|
|
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 177.39it/s]
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 178.64it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.97it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 177.19it/s]
Saving video: 25%|██▍ | 20/81 [00:00<00:00, 69.48it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 177.99it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 184.63it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 159.84it/s] |
|
|
Saving video: 84%|████████▍ | 68/81 [00:00<00:00, 169.03it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.38it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.25it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 167.69it/s] |
|
|
Saving video: 51%|█████ | 41/81 [00:00<00:00, 115.51it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 148.84it/s] |
|
|
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 141.17it/s]
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 155.99it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 120.41it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 401... |
|
|
training step 402... |
|
|
training step 403... |
|
|
training step 404... |
|
|
training step 405... |
|
|
training step 406... |
|
|
training step 407... |
|
|
training step 408... |
|
|
training step 409... |
|
|
training step 410... |
|
|
training step 411... |
|
|
training step 412... |
|
|
training step 413... |
|
|
training step 414... |
|
|
training step 415... |
|
|
training step 416... |
|
|
training step 417... |
|
|
training step 418... |
|
|
training step 419... |
|
|
training step 420... |
|
|
training step 421... |
|
|
training step 422... |
|
|
training step 423... |
|
|
training step 424... |
|
|
training step 425... |
|
|
training step 426... |
|
|
training step 427... |
|
|
training step 428... |
|
|
training step 429... |
|
|
training step 430... |
|
|
training step 431... |
|
|
training step 432... |
|
|
training step 433... |
|
|
training step 434... |
|
|
training step 435... |
|
|
training step 436... |
|
|
training step 437... |
|
|
training step 438... |
|
|
training step 439... |
|
|
training step 440... |
|
|
training step 441... |
|
|
training step 442... |
|
|
training step 443... |
|
|
training step 444... |
|
|
training step 445... |
|
|
training step 446... |
|
|
training step 447... |
|
|
training step 448... |
|
|
training step 449... |
|
|
training step 450... |
|
|
training step 451... |
|
|
training step 452... |
|
|
training step 453... |
|
|
training step 454... |
|
|
training step 455... |
|
|
training step 456... |
|
|
training step 457... |
|
|
training step 458... |
|
|
training step 459... |
|
|
training step 460... |
|
|
training step 461... |
|
|
training step 462... |
|
|
training step 463... |
|
|
training step 464... |
|
|
training step 465... |
|
|
training step 466... |
|
|
training step 467... |
|
|
training step 468... |
|
|
training step 469... |
|
|
training step 470... |
|
|
training step 471... |
|
|
training step 472... |
|
|
training step 473... |
|
|
training step 474... |
|
|
training step 475... |
|
|
training step 476... |
|
|
training step 477... |
|
|
training step 478... |
|
|
training step 479... |
|
|
training step 480... |
|
|
training step 481... |
|
|
training step 482... |
|
|
training step 483... |
|
|
training step 484... |
|
|
training step 485... |
|
|
training step 486... |
|
|
training step 487... |
|
|
training step 488... |
|
|
training step 489... |
|
|
training step 490... |
|
|
training step 491... |
|
|
training step 492... |
|
|
training step 493... |
|
|
training step 494... |
|
|
training step 495... |
|
|
training step 496... |
|
|
training step 497... |
|
|
training step 498... |
|
|
training step 499... |
|
|
training step 500... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.81it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.44it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.67it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 126.41it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.14it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.53it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.32it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.40it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 138.69it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 148.43it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 194.94it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 137.58it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 143.90it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 130.54it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.62it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 190.62it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 201.70it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 217.31it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 192.60it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 193.49it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 173.53it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 173.89it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 188.39it/s] |
|
|
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 208.64it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 179.02it/s] |
|
|
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 197.27it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 200.31it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 203.37it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 189.32it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 176.99it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 177.93it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 174.00it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 177.68it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 169.83it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 159.96it/s] |
|
|
Saving video: 1%| | 1/81 [00:00<00:22, 3.49it/s]
Saving video: 26%|██▌ | 21/81 [00:00<00:00, 68.17it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 138.41it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 181.52it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 133.91it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 501... |
|
|
training step 502... |
|
|
training step 503... |
|
|
training step 504... |
|
|
training step 505... |
|
|
training step 506... |
|
|
training step 507... |
|
|
training step 508... |
|
|
training step 509... |
|
|
training step 510... |
|
|
training step 511... |
|
|
training step 512... |
|
|
training step 513... |
|
|
training step 514... |
|
|
training step 515... |
|
|
training step 516... |
|
|
training step 517... |
|
|
training step 518... |
|
|
training step 519... |
|
|
training step 520... |
|
|
training step 521... |
|
|
training step 522... |
|
|
training step 523... |
|
|
training step 524... |
|
|
training step 525... |
|
|
training step 526... |
|
|
training step 527... |
|
|
training step 528... |
|
|
training step 529... |
|
|
training step 530... |
|
|
training step 531... |
|
|
training step 532... |
|
|
training step 533... |
|
|
training step 534... |
|
|
training step 535... |
|
|
training step 536... |
|
|
training step 537... |
|
|
training step 538... |
|
|
training step 539... |
|
|
training step 540... |
|
|
training step 541... |
|
|
training step 542... |
|
|
training step 543... |
|
|
training step 544... |
|
|
training step 545... |
|
|
training step 546... |
|
|
training step 547... |
|
|
training step 548... |
|
|
training step 549... |
|
|
training step 550... |
|
|
training step 551... |
|
|
training step 552... |
|
|
training step 553... |
|
|
training step 554... |
|
|
training step 555... |
|
|
training step 556... |
|
|
training step 557... |
|
|
training step 558... |
|
|
training step 559... |
|
|
training step 560... |
|
|
training step 561... |
|
|
training step 562... |
|
|
training step 563... |
|
|
training step 564... |
|
|
training step 565... |
|
|
training step 566... |
|
|
training step 567... |
|
|
training step 568... |
|
|
training step 569... |
|
|
training step 570... |
|
|
training step 571... |
|
|
training step 572... |
|
|
training step 573... |
|
|
training step 574... |
|
|
training step 575... |
|
|
training step 576... |
|
|
training step 577... |
|
|
training step 578... |
|
|
training step 579... |
|
|
training step 580... |
|
|
training step 581... |
|
|
training step 582... |
|
|
training step 583... |
|
|
training step 584... |
|
|
training step 585... |
|
|
training step 586... |
|
|
training step 587... |
|
|
training step 588... |
|
|
training step 589... |
|
|
training step 590... |
|
|
training step 591... |
|
|
training step 592... |
|
|
training step 593... |
|
|
training step 594... |
|
|
training step 595... |
|
|
training step 596... |
|
|
training step 597... |
|
|
training step 598... |
|
|
training step 599... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_000600/model.pt |
|
|
training step 600... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.94it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.71it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.67it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.69it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.79it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.63it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 160.25it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.30it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 154.59it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 137.49it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 141.59it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 144.41it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 115.89it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 209.45it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.38it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 205.83it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 188.50it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 193.38it/s]
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 199.83it/s]
Saving video: 1%| | 1/81 [00:00<00:21, 3.67it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 163.48it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 219.41it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 157.30it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 193.14it/s] |
|
|
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 210.05it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 195.64it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 188.88it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 187.97it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 168.23it/s] |
|
|
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 195.25it/s]
Saving video: 25%|██▍ | 20/81 [00:00<00:00, 67.30it/s]
Saving video: 80%|████████ | 65/81 [00:00<00:00, 170.80it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 174.11it/s] |
|
|
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 159.03it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.44it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 154.60it/s] |
|
|
Saving video: 48%|████▊ | 39/81 [00:00<00:00, 105.87it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 154.30it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 144.31it/s] |
|
|
Saving video: 70%|███████ | 57/81 [00:00<00:00, 127.78it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 133.16it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 111.54it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 601... |
|
|
training step 602... |
|
|
training step 603... |
|
|
training step 604... |
|
|
training step 605... |
|
|
training step 606... |
|
|
training step 607... |
|
|
training step 608... |
|
|
training step 609... |
|
|
training step 610... |
|
|
training step 611... |
|
|
training step 612... |
|
|
training step 613... |
|
|
training step 614... |
|
|
training step 615... |
|
|
training step 616... |
|
|
training step 617... |
|
|
training step 618... |
|
|
training step 619... |
|
|
training step 620... |
|
|
training step 621... |
|
|
training step 622... |
|
|
training step 623... |
|
|
training step 624... |
|
|
training step 625... |
|
|
training step 626... |
|
|
training step 627... |
|
|
training step 628... |
|
|
training step 629... |
|
|
training step 630... |
|
|
training step 631... |
|
|
training step 632... |
|
|
training step 633... |
|
|
training step 634... |
|
|
training step 635... |
|
|
training step 636... |
|
|
training step 637... |
|
|
training step 638... |
|
|
training step 639... |
|
|
training step 640... |
|
|
training step 641... |
|
|
training step 642... |
|
|
training step 643... |
|
|
training step 644... |
|
|
training step 645... |
|
|
training step 646... |
|
|
training step 647... |
|
|
training step 648... |
|
|
training step 649... |
|
|
training step 650... |
|
|
training step 651... |
|
|
training step 652... |
|
|
training step 653... |
|
|
training step 654... |
|
|
training step 655... |
|
|
training step 656... |
|
|
training step 657... |
|
|
training step 658... |
|
|
training step 659... |
|
|
training step 660... |
|
|
training step 661... |
|
|
training step 662... |
|
|
training step 663... |
|
|
training step 664... |
|
|
training step 665... |
|
|
training step 666... |
|
|
training step 667... |
|
|
training step 668... |
|
|
training step 669... |
|
|
training step 670... |
|
|
training step 671... |
|
|
training step 672... |
|
|
training step 673... |
|
|
training step 674... |
|
|
training step 675... |
|
|
training step 676... |
|
|
training step 677... |
|
|
training step 678... |
|
|
training step 679... |
|
|
training step 680... |
|
|
training step 681... |
|
|
training step 682... |
|
|
training step 683... |
|
|
training step 684... |
|
|
training step 685... |
|
|
training step 686... |
|
|
training step 687... |
|
|
training step 688... |
|
|
training step 689... |
|
|
training step 690... |
|
|
training step 691... |
|
|
training step 692... |
|
|
training step 693... |
|
|
training step 694... |
|
|
training step 695... |
|
|
training step 696... |
|
|
training step 697... |
|
|
training step 698... |
|
|
training step 699... |
|
|
training step 700... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.70it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.45it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.17it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.17it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.54it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.47it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 134.93it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.29it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.49it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 6.84it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 134.12it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 138.20it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.34it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 138.92it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 189.21it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 187.72it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 117.07it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 116.71it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 188.04it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 185.40it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 197.88it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 191.28it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 199.48it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 168.16it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 198.66it/s]
Saving video: 54%|█████▍ | 44/81 [00:00<00:00, 148.76it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 174.15it/s] |
|
|
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 191.81it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.87it/s] |
|
|
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 203.25it/s]
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 178.07it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 185.54it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.23it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 176.90it/s] |
|
|
Saving video: 80%|████████ | 65/81 [00:00<00:00, 163.59it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 162.51it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.50it/s] |
|
|
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 149.06it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 149.55it/s] |
|
|
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 147.33it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 134.61it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 701... |
|
|
training step 702... |
|
|
training step 703... |
|
|
training step 704... |
|
|
training step 705... |
|
|
training step 706... |
|
|
training step 707... |
|
|
training step 708... |
|
|
training step 709... |
|
|
training step 710... |
|
|
training step 711... |
|
|
training step 712... |
|
|
training step 713... |
|
|
training step 714... |
|
|
training step 715... |
|
|
training step 716... |
|
|
training step 717... |
|
|
training step 718... |
|
|
training step 719... |
|
|
training step 720... |
|
|
training step 721... |
|
|
training step 722... |
|
|
training step 723... |
|
|
training step 724... |
|
|
training step 725... |
|
|
training step 726... |
|
|
training step 727... |
|
|
training step 728... |
|
|
training step 729... |
|
|
training step 730... |
|
|
training step 731... |
|
|
training step 732... |
|
|
training step 733... |
|
|
training step 734... |
|
|
training step 735... |
|
|
training step 736... |
|
|
training step 737... |
|
|
training step 738... |
|
|
training step 739... |
|
|
training step 740... |
|
|
training step 741... |
|
|
training step 742... |
|
|
training step 743... |
|
|
training step 744... |
|
|
training step 745... |
|
|
training step 746... |
|
|
training step 747... |
|
|
training step 748... |
|
|
training step 749... |
|
|
training step 750... |
|
|
training step 751... |
|
|
training step 752... |
|
|
training step 753... |
|
|
training step 754... |
|
|
training step 755... |
|
|
training step 756... |
|
|
training step 757... |
|
|
training step 758... |
|
|
training step 759... |
|
|
training step 760... |
|
|
training step 761... |
|
|
training step 762... |
|
|
training step 763... |
|
|
training step 764... |
|
|
training step 765... |
|
|
training step 766... |
|
|
training step 767... |
|
|
training step 768... |
|
|
training step 769... |
|
|
training step 770... |
|
|
training step 771... |
|
|
training step 772... |
|
|
training step 773... |
|
|
training step 774... |
|
|
training step 775... |
|
|
training step 776... |
|
|
training step 777... |
|
|
training step 778... |
|
|
training step 779... |
|
|
training step 780... |
|
|
training step 781... |
|
|
training step 782... |
|
|
training step 783... |
|
|
training step 784... |
|
|
training step 785... |
|
|
training step 786... |
|
|
training step 787... |
|
|
training step 788... |
|
|
training step 789... |
|
|
training step 790... |
|
|
training step 791... |
|
|
training step 792... |
|
|
training step 793... |
|
|
training step 794... |
|
|
training step 795... |
|
|
training step 796... |
|
|
training step 797... |
|
|
training step 798... |
|
|
training step 799... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_000800/model.pt |
|
|
training step 800... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.57it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.49it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.05it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.73it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.93it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.62it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.32it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 128.64it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 120.80it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 132.15it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 137.79it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.29it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 118.25it/s]
Saving video: 22%|██▏ | 18/81 [00:00<00:00, 95.94it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 161.17it/s]
Saving video: 1%| | 1/81 [00:00<00:20, 3.92it/s]
Saving video: 49%|████▉ | 40/81 [00:00<00:00, 143.46it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 178.19it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 191.75it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 193.82it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 168.23it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 169.37it/s]
Saving video: 84%|████████▍ | 68/81 [00:00<00:00, 185.35it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 84.08it/s]
Saving video: 72%|███████▏ | 58/81 [00:00<00:00, 154.53it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 190.41it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 193.05it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 195.59it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 184.08it/s]
Saving video: 80%|████████ | 65/81 [00:00<00:00, 177.16it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 162.07it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 164.93it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 167.17it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 173.29it/s] |
|
|
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 115.42it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 151.78it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 154.89it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 150.66it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 137.61it/s] |
|
|
Saving video: 74%|███████▍ | 60/81 [00:00<00:00, 134.59it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 142.33it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 116.31it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 801... |
|
|
training step 802... |
|
|
training step 803... |
|
|
training step 804... |
|
|
training step 805... |
|
|
training step 806... |
|
|
training step 807... |
|
|
training step 808... |
|
|
training step 809... |
|
|
training step 810... |
|
|
training step 811... |
|
|
training step 812... |
|
|
training step 813... |
|
|
training step 814... |
|
|
training step 815... |
|
|
training step 816... |
|
|
training step 817... |
|
|
training step 818... |
|
|
training step 819... |
|
|
training step 820... |
|
|
training step 821... |
|
|
training step 822... |
|
|
training step 823... |
|
|
training step 824... |
|
|
training step 825... |
|
|
training step 826... |
|
|
training step 827... |
|
|
training step 828... |
|
|
training step 829... |
|
|
training step 830... |
|
|
training step 831... |
|
|
training step 832... |
|
|
training step 833... |
|
|
training step 834... |
|
|
training step 835... |
|
|
training step 836... |
|
|
training step 837... |
|
|
training step 838... |
|
|
training step 839... |
|
|
training step 840... |
|
|
training step 841... |
|
|
training step 842... |
|
|
training step 843... |
|
|
training step 844... |
|
|
training step 845... |
|
|
training step 846... |
|
|
training step 847... |
|
|
training step 848... |
|
|
training step 849... |
|
|
training step 850... |
|
|
training step 851... |
|
|
training step 852... |
|
|
training step 853... |
|
|
training step 854... |
|
|
training step 855... |
|
|
training step 856... |
|
|
training step 857... |
|
|
training step 858... |
|
|
training step 859... |
|
|
training step 860... |
|
|
training step 861... |
|
|
training step 862... |
|
|
training step 863... |
|
|
training step 864... |
|
|
training step 865... |
|
|
training step 866... |
|
|
training step 867... |
|
|
training step 868... |
|
|
training step 869... |
|
|
training step 870... |
|
|
training step 871... |
|
|
training step 872... |
|
|
training step 873... |
|
|
training step 874... |
|
|
training step 875... |
|
|
training step 876... |
|
|
training step 877... |
|
|
training step 878... |
|
|
training step 879... |
|
|
training step 880... |
|
|
training step 881... |
|
|
training step 882... |
|
|
training step 883... |
|
|
training step 884... |
|
|
training step 885... |
|
|
training step 886... |
|
|
training step 887... |
|
|
training step 888... |
|
|
training step 889... |
|
|
training step 890... |
|
|
training step 891... |
|
|
training step 892... |
|
|
training step 893... |
|
|
training step 894... |
|
|
training step 895... |
|
|
training step 896... |
|
|
training step 897... |
|
|
training step 898... |
|
|
training step 899... |
|
|
training step 900... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.61it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.24it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.63it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.08it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.52it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.58it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 154.98it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.66it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.13it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 139.06it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 131.79it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 143.77it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 126.90it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 197.66it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 190.86it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 124.40it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 196.55it/s]
Saving video: 1%| | 1/81 [00:00<00:16, 4.90it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 171.86it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 193.68it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 174.79it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 211.92it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 187.82it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 185.77it/s] |
|
|
Saving video: 54%|█████▍ | 44/81 [00:00<00:00, 149.32it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 186.56it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 87.98it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 168.80it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 167.15it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 199.32it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 165.86it/s] |
|
|
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 168.47it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 176.87it/s] |
|
|
Saving video: 80%|████████ | 65/81 [00:00<00:00, 169.12it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 158.74it/s] |
|
|
Saving video: 48%|████▊ | 39/81 [00:00<00:00, 118.06it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 149.19it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 146.73it/s] |
|
|
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 130.78it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 136.22it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 123.04it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 901... |
|
|
training step 902... |
|
|
training step 903... |
|
|
training step 904... |
|
|
training step 905... |
|
|
training step 906... |
|
|
training step 907... |
|
|
training step 908... |
|
|
training step 909... |
|
|
training step 910... |
|
|
training step 911... |
|
|
training step 912... |
|
|
training step 913... |
|
|
training step 914... |
|
|
training step 915... |
|
|
training step 916... |
|
|
training step 917... |
|
|
training step 918... |
|
|
training step 919... |
|
|
training step 920... |
|
|
training step 921... |
|
|
training step 922... |
|
|
training step 923... |
|
|
training step 924... |
|
|
training step 925... |
|
|
training step 926... |
|
|
training step 927... |
|
|
training step 928... |
|
|
training step 929... |
|
|
training step 930... |
|
|
training step 931... |
|
|
training step 932... |
|
|
training step 933... |
|
|
training step 934... |
|
|
training step 935... |
|
|
training step 936... |
|
|
training step 937... |
|
|
training step 938... |
|
|
training step 939... |
|
|
training step 940... |
|
|
training step 941... |
|
|
training step 942... |
|
|
training step 943... |
|
|
training step 944... |
|
|
training step 945... |
|
|
training step 946... |
|
|
training step 947... |
|
|
training step 948... |
|
|
training step 949... |
|
|
training step 950... |
|
|
training step 951... |
|
|
training step 952... |
|
|
training step 953... |
|
|
training step 954... |
|
|
training step 955... |
|
|
training step 956... |
|
|
training step 957... |
|
|
training step 958... |
|
|
training step 959... |
|
|
training step 960... |
|
|
training step 961... |
|
|
training step 962... |
|
|
training step 963... |
|
|
training step 964... |
|
|
training step 965... |
|
|
training step 966... |
|
|
training step 967... |
|
|
training step 968... |
|
|
training step 969... |
|
|
training step 970... |
|
|
training step 971... |
|
|
training step 972... |
|
|
training step 973... |
|
|
training step 974... |
|
|
training step 975... |
|
|
training step 976... |
|
|
training step 977... |
|
|
training step 978... |
|
|
training step 979... |
|
|
training step 980... |
|
|
training step 981... |
|
|
training step 982... |
|
|
training step 983... |
|
|
training step 984... |
|
|
training step 985... |
|
|
training step 986... |
|
|
training step 987... |
|
|
training step 988... |
|
|
training step 989... |
|
|
training step 990... |
|
|
training step 991... |
|
|
training step 992... |
|
|
training step 993... |
|
|
training step 994... |
|
|
training step 995... |
|
|
training step 996... |
|
|
training step 997... |
|
|
training step 998... |
|
|
training step 999... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_001000/model.pt |
|
|
training step 1000... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.35it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.82it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.03it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.70it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.64it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.81it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.53it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 142.95it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.99it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 132.22it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 154.57it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 129.80it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.16it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 141.14it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 173.74it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 192.96it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 190.91it/s]
Saving video: 1%| | 1/81 [00:00<00:20, 3.90it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 204.22it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 184.68it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 180.32it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 195.31it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 210.20it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 161.76it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 197.67it/s]
Saving video: 25%|██▍ | 20/81 [00:00<00:00, 70.17it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 179.02it/s] |
|
|
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 197.08it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.12it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 188.59it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 182.52it/s] |
|
|
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 182.89it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 188.60it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 148.61it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.45it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.04it/s] |
|
|
Saving video: 44%|████▍ | 36/81 [00:00<00:00, 100.30it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 160.23it/s] |
|
|
Saving video: 74%|███████▍ | 60/81 [00:00<00:00, 143.92it/s]
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 161.21it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 121.33it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1001... |
|
|
training step 1002... |
|
|
training step 1003... |
|
|
training step 1004... |
|
|
training step 1005... |
|
|
training step 1006... |
|
|
training step 1007... |
|
|
training step 1008... |
|
|
training step 1009... |
|
|
training step 1010... |
|
|
training step 1011... |
|
|
training step 1012... |
|
|
training step 1013... |
|
|
training step 1014... |
|
|
training step 1015... |
|
|
training step 1016... |
|
|
training step 1017... |
|
|
training step 1018... |
|
|
training step 1019... |
|
|
training step 1020... |
|
|
training step 1021... |
|
|
training step 1022... |
|
|
training step 1023... |
|
|
training step 1024... |
|
|
training step 1025... |
|
|
training step 1026... |
|
|
training step 1027... |
|
|
training step 1028... |
|
|
training step 1029... |
|
|
training step 1030... |
|
|
training step 1031... |
|
|
training step 1032... |
|
|
training step 1033... |
|
|
training step 1034... |
|
|
training step 1035... |
|
|
training step 1036... |
|
|
training step 1037... |
|
|
training step 1038... |
|
|
training step 1039... |
|
|
training step 1040... |
|
|
training step 1041... |
|
|
training step 1042... |
|
|
training step 1043... |
|
|
training step 1044... |
|
|
training step 1045... |
|
|
training step 1046... |
|
|
training step 1047... |
|
|
training step 1048... |
|
|
training step 1049... |
|
|
training step 1050... |
|
|
training step 1051... |
|
|
training step 1052... |
|
|
training step 1053... |
|
|
training step 1054... |
|
|
training step 1055... |
|
|
training step 1056... |
|
|
training step 1057... |
|
|
training step 1058... |
|
|
training step 1059... |
|
|
training step 1060... |
|
|
training step 1061... |
|
|
training step 1062... |
|
|
training step 1063... |
|
|
training step 1064... |
|
|
training step 1065... |
|
|
training step 1066... |
|
|
training step 1067... |
|
|
training step 1068... |
|
|
training step 1069... |
|
|
training step 1070... |
|
|
training step 1071... |
|
|
training step 1072... |
|
|
training step 1073... |
|
|
training step 1074... |
|
|
training step 1075... |
|
|
training step 1076... |
|
|
training step 1077... |
|
|
training step 1078... |
|
|
training step 1079... |
|
|
training step 1080... |
|
|
training step 1081... |
|
|
training step 1082... |
|
|
training step 1083... |
|
|
training step 1084... |
|
|
training step 1085... |
|
|
training step 1086... |
|
|
training step 1087... |
|
|
training step 1088... |
|
|
training step 1089... |
|
|
training step 1090... |
|
|
training step 1091... |
|
|
training step 1092... |
|
|
training step 1093... |
|
|
training step 1094... |
|
|
training step 1095... |
|
|
training step 1096... |
|
|
training step 1097... |
|
|
training step 1098... |
|
|
training step 1099... |
|
|
training step 1100... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.46it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.60it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 6.99it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.03it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.57it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.54it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.43it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.43it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 153.99it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 133.33it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 141.95it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 6.83it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 134.66it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.64it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 197.47it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 131.57it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 204.98it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 187.90it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 160.47it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 125.58it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 188.87it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 169.62it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 212.06it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 222.31it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 165.67it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 184.18it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 194.06it/s] |
|
|
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 206.38it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 155.74it/s]
Saving video: 78%|███████▊ | 63/81 [00:00<00:00, 158.52it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 173.34it/s] |
|
|
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 180.45it/s]
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 166.63it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 154.68it/s]
Saving video: 79%|███████▉ | 64/81 [00:00<00:00, 159.10it/s]
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 156.45it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 159.60it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 145.58it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 148.57it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 144.16it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 145.82it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1101... |
|
|
training step 1102... |
|
|
training step 1103... |
|
|
training step 1104... |
|
|
training step 1105... |
|
|
training step 1106... |
|
|
training step 1107... |
|
|
training step 1108... |
|
|
training step 1109... |
|
|
training step 1110... |
|
|
training step 1111... |
|
|
training step 1112... |
|
|
training step 1113... |
|
|
training step 1114... |
|
|
training step 1115... |
|
|
training step 1116... |
|
|
training step 1117... |
|
|
training step 1118... |
|
|
training step 1119... |
|
|
training step 1120... |
|
|
training step 1121... |
|
|
training step 1122... |
|
|
training step 1123... |
|
|
training step 1124... |
|
|
training step 1125... |
|
|
training step 1126... |
|
|
training step 1127... |
|
|
training step 1128... |
|
|
training step 1129... |
|
|
training step 1130... |
|
|
training step 1131... |
|
|
training step 1132... |
|
|
training step 1133... |
|
|
training step 1134... |
|
|
training step 1135... |
|
|
training step 1136... |
|
|
training step 1137... |
|
|
training step 1138... |
|
|
training step 1139... |
|
|
training step 1140... |
|
|
training step 1141... |
|
|
training step 1142... |
|
|
training step 1143... |
|
|
training step 1144... |
|
|
training step 1145... |
|
|
training step 1146... |
|
|
training step 1147... |
|
|
training step 1148... |
|
|
training step 1149... |
|
|
training step 1150... |
|
|
training step 1151... |
|
|
training step 1152... |
|
|
training step 1153... |
|
|
training step 1154... |
|
|
training step 1155... |
|
|
training step 1156... |
|
|
training step 1157... |
|
|
training step 1158... |
|
|
training step 1159... |
|
|
training step 1160... |
|
|
training step 1161... |
|
|
training step 1162... |
|
|
training step 1163... |
|
|
training step 1164... |
|
|
training step 1165... |
|
|
training step 1166... |
|
|
training step 1167... |
|
|
training step 1168... |
|
|
training step 1169... |
|
|
training step 1170... |
|
|
training step 1171... |
|
|
training step 1172... |
|
|
training step 1173... |
|
|
training step 1174... |
|
|
training step 1175... |
|
|
training step 1176... |
|
|
training step 1177... |
|
|
training step 1178... |
|
|
training step 1179... |
|
|
training step 1180... |
|
|
training step 1181... |
|
|
training step 1182... |
|
|
training step 1183... |
|
|
training step 1184... |
|
|
training step 1185... |
|
|
training step 1186... |
|
|
training step 1187... |
|
|
training step 1188... |
|
|
training step 1189... |
|
|
training step 1190... |
|
|
training step 1191... |
|
|
training step 1192... |
|
|
training step 1193... |
|
|
training step 1194... |
|
|
training step 1195... |
|
|
training step 1196... |
|
|
training step 1197... |
|
|
training step 1198... |
|
|
training step 1199... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_001200/model.pt |
|
|
training step 1200... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.71it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.68it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.01it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.14it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.48it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.56it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.39it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.36it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 150.04it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 133.02it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 137.41it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 140.56it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.21it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.21it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 197.54it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 199.69it/s]
Saving video: 1%| | 1/81 [00:00<00:17, 4.54it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 191.08it/s]
Saving video: 54%|█████▍ | 44/81 [00:00<00:00, 154.93it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 148.84it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 162.55it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 191.76it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 213.39it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 206.70it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 186.83it/s] |
|
|
Saving video: 20%|█▉ | 16/81 [00:00<00:01, 60.63it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 182.10it/s] |
|
|
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 211.81it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 177.59it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 177.31it/s]
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 168.10it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 182.80it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 194.08it/s]
Saving video: 42%|████▏ | 34/81 [00:00<00:00, 101.65it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 160.22it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.08it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.49it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 156.27it/s] |
|
|
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 129.25it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 139.96it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 115.46it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1201... |
|
|
training step 1202... |
|
|
training step 1203... |
|
|
training step 1204... |
|
|
training step 1205... |
|
|
training step 1206... |
|
|
training step 1207... |
|
|
training step 1208... |
|
|
training step 1209... |
|
|
training step 1210... |
|
|
training step 1211... |
|
|
training step 1212... |
|
|
training step 1213... |
|
|
training step 1214... |
|
|
training step 1215... |
|
|
training step 1216... |
|
|
training step 1217... |
|
|
training step 1218... |
|
|
training step 1219... |
|
|
training step 1220... |
|
|
training step 1221... |
|
|
training step 1222... |
|
|
training step 1223... |
|
|
training step 1224... |
|
|
training step 1225... |
|
|
training step 1226... |
|
|
training step 1227... |
|
|
training step 1228... |
|
|
training step 1229... |
|
|
training step 1230... |
|
|
training step 1231... |
|
|
training step 1232... |
|
|
training step 1233... |
|
|
training step 1234... |
|
|
training step 1235... |
|
|
training step 1236... |
|
|
training step 1237... |
|
|
training step 1238... |
|
|
training step 1239... |
|
|
training step 1240... |
|
|
training step 1241... |
|
|
training step 1242... |
|
|
training step 1243... |
|
|
training step 1244... |
|
|
training step 1245... |
|
|
training step 1246... |
|
|
training step 1247... |
|
|
training step 1248... |
|
|
training step 1249... |
|
|
training step 1250... |
|
|
training step 1251... |
|
|
training step 1252... |
|
|
training step 1253... |
|
|
training step 1254... |
|
|
training step 1255... |
|
|
training step 1256... |
|
|
training step 1257... |
|
|
training step 1258... |
|
|
training step 1259... |
|
|
training step 1260... |
|
|
training step 1261... |
|
|
training step 1262... |
|
|
training step 1263... |
|
|
training step 1264... |
|
|
training step 1265... |
|
|
training step 1266... |
|
|
training step 1267... |
|
|
training step 1268... |
|
|
training step 1269... |
|
|
training step 1270... |
|
|
training step 1271... |
|
|
training step 1272... |
|
|
training step 1273... |
|
|
training step 1274... |
|
|
training step 1275... |
|
|
training step 1276... |
|
|
training step 1277... |
|
|
training step 1278... |
|
|
training step 1279... |
|
|
training step 1280... |
|
|
training step 1281... |
|
|
training step 1282... |
|
|
training step 1283... |
|
|
training step 1284... |
|
|
training step 1285... |
|
|
training step 1286... |
|
|
training step 1287... |
|
|
training step 1288... |
|
|
training step 1289... |
|
|
training step 1290... |
|
|
training step 1291... |
|
|
training step 1292... |
|
|
training step 1293... |
|
|
training step 1294... |
|
|
training step 1295... |
|
|
training step 1296... |
|
|
training step 1297... |
|
|
training step 1298... |
|
|
training step 1299... |
|
|
training step 1300... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.46it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.59it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.05it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.75it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.47it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.27it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.69it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.47it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 118.38it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.08it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 128.78it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 127.91it/s]
Saving video: 25%|██▍ | 20/81 [00:00<00:00, 107.07it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 151.01it/s]
Saving video: 26%|██▌ | 21/81 [00:00<00:00, 105.33it/s]
Saving video: 60%|██████ | 49/81 [00:00<00:00, 175.40it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 167.01it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 135.22it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 173.30it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 181.64it/s]
Saving video: 49%|████▉ | 40/81 [00:00<00:00, 144.80it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 197.79it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 181.23it/s]
Saving video: 47%|████▋ | 38/81 [00:00<00:00, 128.42it/s]
Saving video: 78%|███████▊ | 63/81 [00:00<00:00, 164.62it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 178.59it/s]
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 180.89it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 190.64it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 204.65it/s]
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 166.55it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 159.00it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 158.80it/s] |
|
|
|
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 178.31it/s] |
|
|
Saving video: 70%|███████ | 57/81 [00:00<00:00, 149.74it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 167.05it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 151.16it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.33it/s] |
|
|
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 169.03it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 166.86it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 147.92it/s] |
|
|
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 155.77it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 152.72it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 136.30it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1301... |
|
|
training step 1302... |
|
|
training step 1303... |
|
|
training step 1304... |
|
|
training step 1305... |
|
|
training step 1306... |
|
|
training step 1307... |
|
|
training step 1308... |
|
|
training step 1309... |
|
|
training step 1310... |
|
|
training step 1311... |
|
|
training step 1312... |
|
|
training step 1313... |
|
|
training step 1314... |
|
|
training step 1315... |
|
|
training step 1316... |
|
|
training step 1317... |
|
|
training step 1318... |
|
|
training step 1319... |
|
|
training step 1320... |
|
|
training step 1321... |
|
|
training step 1322... |
|
|
training step 1323... |
|
|
training step 1324... |
|
|
training step 1325... |
|
|
training step 1326... |
|
|
training step 1327... |
|
|
training step 1328... |
|
|
training step 1329... |
|
|
training step 1330... |
|
|
training step 1331... |
|
|
training step 1332... |
|
|
training step 1333... |
|
|
training step 1334... |
|
|
training step 1335... |
|
|
training step 1336... |
|
|
training step 1337... |
|
|
training step 1338... |
|
|
training step 1339... |
|
|
training step 1340... |
|
|
training step 1341... |
|
|
training step 1342... |
|
|
training step 1343... |
|
|
training step 1344... |
|
|
training step 1345... |
|
|
training step 1346... |
|
|
training step 1347... |
|
|
training step 1348... |
|
|
training step 1349... |
|
|
training step 1350... |
|
|
training step 1351... |
|
|
training step 1352... |
|
|
training step 1353... |
|
|
training step 1354... |
|
|
training step 1355... |
|
|
training step 1356... |
|
|
training step 1357... |
|
|
training step 1358... |
|
|
training step 1359... |
|
|
training step 1360... |
|
|
training step 1361... |
|
|
training step 1362... |
|
|
training step 1363... |
|
|
training step 1364... |
|
|
training step 1365... |
|
|
training step 1366... |
|
|
training step 1367... |
|
|
training step 1368... |
|
|
training step 1369... |
|
|
training step 1370... |
|
|
training step 1371... |
|
|
training step 1372... |
|
|
training step 1373... |
|
|
training step 1374... |
|
|
training step 1375... |
|
|
training step 1376... |
|
|
training step 1377... |
|
|
training step 1378... |
|
|
training step 1379... |
|
|
training step 1380... |
|
|
training step 1381... |
|
|
training step 1382... |
|
|
training step 1383... |
|
|
training step 1384... |
|
|
training step 1385... |
|
|
training step 1386... |
|
|
training step 1387... |
|
|
training step 1388... |
|
|
training step 1389... |
|
|
training step 1390... |
|
|
training step 1391... |
|
|
training step 1392... |
|
|
training step 1393... |
|
|
training step 1394... |
|
|
training step 1395... |
|
|
training step 1396... |
|
|
training step 1397... |
|
|
training step 1398... |
|
|
training step 1399... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_001400/model.pt |
|
|
training step 1400... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.58it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.35it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.18it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.44it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.41it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.58it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 157.63it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.50it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.88it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 137.75it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 135.54it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 132.75it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 131.19it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 197.84it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 117.42it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 194.17it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 174.76it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 187.56it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 148.63it/s]
Saving video: 1%| | 1/81 [00:00<00:23, 3.38it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 166.22it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 197.12it/s]
Saving video: 58%|█████▊ | 47/81 [00:00<00:00, 168.90it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 201.06it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 187.38it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 179.39it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 174.19it/s] |
|
|
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 204.95it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 177.02it/s]
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 163.87it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 164.09it/s] |
|
|
Saving video: 23%|██▎ | 19/81 [00:00<00:01, 60.38it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 172.42it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 178.13it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 155.47it/s] |
|
|
Saving video: 43%|████▎ | 35/81 [00:00<00:00, 91.42it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 162.32it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 156.27it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 146.36it/s] |
|
|
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 127.12it/s]
Saving video: 95%|█████████▌| 77/81 [00:00<00:00, 149.72it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 110.87it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1401... |
|
|
training step 1402... |
|
|
training step 1403... |
|
|
training step 1404... |
|
|
training step 1405... |
|
|
training step 1406... |
|
|
training step 1407... |
|
|
training step 1408... |
|
|
training step 1409... |
|
|
training step 1410... |
|
|
training step 1411... |
|
|
training step 1412... |
|
|
training step 1413... |
|
|
training step 1414... |
|
|
training step 1415... |
|
|
training step 1416... |
|
|
training step 1417... |
|
|
training step 1418... |
|
|
training step 1419... |
|
|
training step 1420... |
|
|
training step 1421... |
|
|
training step 1422... |
|
|
training step 1423... |
|
|
training step 1424... |
|
|
training step 1425... |
|
|
training step 1426... |
|
|
training step 1427... |
|
|
training step 1428... |
|
|
training step 1429... |
|
|
training step 1430... |
|
|
training step 1431... |
|
|
training step 1432... |
|
|
training step 1433... |
|
|
training step 1434... |
|
|
training step 1435... |
|
|
training step 1436... |
|
|
training step 1437... |
|
|
training step 1438... |
|
|
training step 1439... |
|
|
training step 1440... |
|
|
training step 1441... |
|
|
training step 1442... |
|
|
training step 1443... |
|
|
training step 1444... |
|
|
training step 1445... |
|
|
training step 1446... |
|
|
training step 1447... |
|
|
training step 1448... |
|
|
training step 1449... |
|
|
training step 1450... |
|
|
training step 1451... |
|
|
training step 1452... |
|
|
training step 1453... |
|
|
training step 1454... |
|
|
training step 1455... |
|
|
training step 1456... |
|
|
training step 1457... |
|
|
training step 1458... |
|
|
training step 1459... |
|
|
training step 1460... |
|
|
training step 1461... |
|
|
training step 1462... |
|
|
training step 1463... |
|
|
training step 1464... |
|
|
training step 1465... |
|
|
training step 1466... |
|
|
training step 1467... |
|
|
training step 1468... |
|
|
training step 1469... |
|
|
training step 1470... |
|
|
training step 1471... |
|
|
training step 1472... |
|
|
training step 1473... |
|
|
training step 1474... |
|
|
training step 1475... |
|
|
training step 1476... |
|
|
training step 1477... |
|
|
training step 1478... |
|
|
training step 1479... |
|
|
training step 1480... |
|
|
training step 1481... |
|
|
training step 1482... |
|
|
training step 1483... |
|
|
training step 1484... |
|
|
training step 1485... |
|
|
training step 1486... |
|
|
training step 1487... |
|
|
training step 1488... |
|
|
training step 1489... |
|
|
training step 1490... |
|
|
training step 1491... |
|
|
training step 1492... |
|
|
training step 1493... |
|
|
training step 1494... |
|
|
training step 1495... |
|
|
training step 1496... |
|
|
training step 1497... |
|
|
training step 1498... |
|
|
training step 1499... |
|
|
training step 1500... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.89it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.44it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 6.99it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.80it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.33it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.29it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 159.71it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 144.88it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.57it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.60it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 132.60it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 127.81it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 128.27it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 129.96it/s]
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 211.63it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 194.81it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 133.16it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 133.32it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 187.53it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 170.84it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 188.81it/s]
Saving video: 49%|████▉ | 40/81 [00:00<00:00, 136.92it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 208.29it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 199.44it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 189.64it/s] |
|
|
Saving video: 60%|██████ | 49/81 [00:00<00:00, 171.16it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 179.90it/s] |
|
|
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 180.23it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 193.25it/s]
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 191.44it/s]
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 190.87it/s]
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 162.29it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 165.90it/s] |
|
|
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 180.07it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 164.32it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.86it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 187.83it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 165.62it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 154.23it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 146.81it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 164.05it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1501... |
|
|
training step 1502... |
|
|
training step 1503... |
|
|
training step 1504... |
|
|
training step 1505... |
|
|
training step 1506... |
|
|
training step 1507... |
|
|
training step 1508... |
|
|
training step 1509... |
|
|
training step 1510... |
|
|
training step 1511... |
|
|
training step 1512... |
|
|
training step 1513... |
|
|
training step 1514... |
|
|
training step 1515... |
|
|
training step 1516... |
|
|
training step 1517... |
|
|
training step 1518... |
|
|
training step 1519... |
|
|
training step 1520... |
|
|
training step 1521... |
|
|
training step 1522... |
|
|
training step 1523... |
|
|
training step 1524... |
|
|
training step 1525... |
|
|
training step 1526... |
|
|
training step 1527... |
|
|
training step 1528... |
|
|
training step 1529... |
|
|
training step 1530... |
|
|
training step 1531... |
|
|
training step 1532... |
|
|
training step 1533... |
|
|
training step 1534... |
|
|
training step 1535... |
|
|
training step 1536... |
|
|
training step 1537... |
|
|
training step 1538... |
|
|
training step 1539... |
|
|
training step 1540... |
|
|
training step 1541... |
|
|
training step 1542... |
|
|
training step 1543... |
|
|
training step 1544... |
|
|
training step 1545... |
|
|
training step 1546... |
|
|
training step 1547... |
|
|
training step 1548... |
|
|
training step 1549... |
|
|
training step 1550... |
|
|
training step 1551... |
|
|
training step 1552... |
|
|
training step 1553... |
|
|
training step 1554... |
|
|
training step 1555... |
|
|
training step 1556... |
|
|
training step 1557... |
|
|
training step 1558... |
|
|
training step 1559... |
|
|
training step 1560... |
|
|
training step 1561... |
|
|
training step 1562... |
|
|
training step 1563... |
|
|
training step 1564... |
|
|
training step 1565... |
|
|
training step 1566... |
|
|
training step 1567... |
|
|
training step 1568... |
|
|
training step 1569... |
|
|
training step 1570... |
|
|
training step 1571... |
|
|
training step 1572... |
|
|
training step 1573... |
|
|
training step 1574... |
|
|
training step 1575... |
|
|
training step 1576... |
|
|
training step 1577... |
|
|
training step 1578... |
|
|
training step 1579... |
|
|
training step 1580... |
|
|
training step 1581... |
|
|
training step 1582... |
|
|
training step 1583... |
|
|
training step 1584... |
|
|
training step 1585... |
|
|
training step 1586... |
|
|
training step 1587... |
|
|
training step 1588... |
|
|
training step 1589... |
|
|
training step 1590... |
|
|
training step 1591... |
|
|
training step 1592... |
|
|
training step 1593... |
|
|
training step 1594... |
|
|
training step 1595... |
|
|
training step 1596... |
|
|
training step 1597... |
|
|
training step 1598... |
|
|
training step 1599... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_001600/model.pt |
|
|
training step 1600... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.65it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.65it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.08it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.70it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.97it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.58it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.54it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 136.34it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 146.32it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 132.76it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 157.65it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 130.80it/s]
Saving video: 1%| | 1/81 [00:00<00:17, 4.63it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.38it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 131.72it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 192.32it/s]
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 180.43it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 187.51it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 207.67it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 144.04it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 100.32it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 187.39it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 164.86it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 178.44it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 185.61it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 214.74it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 182.64it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 189.24it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 169.40it/s] |
|
|
Saving video: 78%|███████▊ | 63/81 [00:00<00:00, 166.50it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 120.03it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 191.03it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 165.28it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 168.20it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.62it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 167.15it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 151.52it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 150.10it/s] |
|
|
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 140.38it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 152.94it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 125.57it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1601... |
|
|
training step 1602... |
|
|
training step 1603... |
|
|
training step 1604... |
|
|
training step 1605... |
|
|
training step 1606... |
|
|
training step 1607... |
|
|
training step 1608... |
|
|
training step 1609... |
|
|
training step 1610... |
|
|
training step 1611... |
|
|
training step 1612... |
|
|
training step 1613... |
|
|
training step 1614... |
|
|
training step 1615... |
|
|
training step 1616... |
|
|
training step 1617... |
|
|
training step 1618... |
|
|
training step 1619... |
|
|
training step 1620... |
|
|
training step 1621... |
|
|
training step 1622... |
|
|
training step 1623... |
|
|
training step 1624... |
|
|
training step 1625... |
|
|
training step 1626... |
|
|
training step 1627... |
|
|
training step 1628... |
|
|
training step 1629... |
|
|
training step 1630... |
|
|
training step 1631... |
|
|
training step 1632... |
|
|
training step 1633... |
|
|
training step 1634... |
|
|
training step 1635... |
|
|
training step 1636... |
|
|
training step 1637... |
|
|
training step 1638... |
|
|
training step 1639... |
|
|
training step 1640... |
|
|
training step 1641... |
|
|
training step 1642... |
|
|
training step 1643... |
|
|
training step 1644... |
|
|
training step 1645... |
|
|
training step 1646... |
|
|
training step 1647... |
|
|
training step 1648... |
|
|
training step 1649... |
|
|
training step 1650... |
|
|
training step 1651... |
|
|
training step 1652... |
|
|
training step 1653... |
|
|
training step 1654... |
|
|
training step 1655... |
|
|
training step 1656... |
|
|
training step 1657... |
|
|
training step 1658... |
|
|
training step 1659... |
|
|
training step 1660... |
|
|
training step 1661... |
|
|
training step 1662... |
|
|
training step 1663... |
|
|
training step 1664... |
|
|
training step 1665... |
|
|
training step 1666... |
|
|
training step 1667... |
|
|
training step 1668... |
|
|
training step 1669... |
|
|
training step 1670... |
|
|
training step 1671... |
|
|
training step 1672... |
|
|
training step 1673... |
|
|
training step 1674... |
|
|
training step 1675... |
|
|
training step 1676... |
|
|
training step 1677... |
|
|
training step 1678... |
|
|
training step 1679... |
|
|
training step 1680... |
|
|
training step 1681... |
|
|
training step 1682... |
|
|
training step 1683... |
|
|
training step 1684... |
|
|
training step 1685... |
|
|
training step 1686... |
|
|
training step 1687... |
|
|
training step 1688... |
|
|
training step 1689... |
|
|
training step 1690... |
|
|
training step 1691... |
|
|
training step 1692... |
|
|
training step 1693... |
|
|
training step 1694... |
|
|
training step 1695... |
|
|
training step 1696... |
|
|
training step 1697... |
|
|
training step 1698... |
|
|
training step 1699... |
|
|
training step 1700... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.26it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.62it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 6.93it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.82it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.47it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.45it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 144.82it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.52it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.33it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 107.37it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 140.08it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 127.99it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.41it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 131.32it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 195.01it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 197.32it/s]
Saving video: 26%|██▌ | 21/81 [00:00<00:00, 103.30it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 168.38it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 194.86it/s]
Saving video: 59%|█████▉ | 48/81 [00:00<00:00, 176.36it/s]
Saving video: 21%|██ | 17/81 [00:00<00:00, 84.05it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 179.17it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 199.25it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 214.20it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 184.43it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 178.73it/s] |
|
|
Saving video: 47%|████▋ | 38/81 [00:00<00:00, 128.49it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 180.40it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 166.25it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 196.03it/s]
Saving video: 42%|████▏ | 34/81 [00:00<00:00, 118.98it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 189.62it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 168.46it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 151.51it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 160.08it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.18it/s] |
|
|
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 145.67it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 147.59it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 149.98it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 163.87it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 134.97it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 136.38it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1701... |
|
|
training step 1702... |
|
|
training step 1703... |
|
|
training step 1704... |
|
|
training step 1705... |
|
|
training step 1706... |
|
|
training step 1707... |
|
|
training step 1708... |
|
|
training step 1709... |
|
|
training step 1710... |
|
|
training step 1711... |
|
|
training step 1712... |
|
|
training step 1713... |
|
|
training step 1714... |
|
|
training step 1715... |
|
|
training step 1716... |
|
|
training step 1717... |
|
|
training step 1718... |
|
|
training step 1719... |
|
|
training step 1720... |
|
|
training step 1721... |
|
|
training step 1722... |
|
|
training step 1723... |
|
|
training step 1724... |
|
|
training step 1725... |
|
|
training step 1726... |
|
|
training step 1727... |
|
|
training step 1728... |
|
|
training step 1729... |
|
|
training step 1730... |
|
|
training step 1731... |
|
|
training step 1732... |
|
|
training step 1733... |
|
|
training step 1734... |
|
|
training step 1735... |
|
|
training step 1736... |
|
|
training step 1737... |
|
|
training step 1738... |
|
|
training step 1739... |
|
|
training step 1740... |
|
|
training step 1741... |
|
|
training step 1742... |
|
|
training step 1743... |
|
|
training step 1744... |
|
|
training step 1745... |
|
|
training step 1746... |
|
|
training step 1747... |
|
|
training step 1748... |
|
|
training step 1749... |
|
|
training step 1750... |
|
|
training step 1751... |
|
|
training step 1752... |
|
|
training step 1753... |
|
|
training step 1754... |
|
|
training step 1755... |
|
|
training step 1756... |
|
|
training step 1757... |
|
|
training step 1758... |
|
|
training step 1759... |
|
|
training step 1760... |
|
|
training step 1761... |
|
|
training step 1762... |
|
|
training step 1763... |
|
|
training step 1764... |
|
|
training step 1765... |
|
|
training step 1766... |
|
|
training step 1767... |
|
|
training step 1768... |
|
|
training step 1769... |
|
|
training step 1770... |
|
|
training step 1771... |
|
|
training step 1772... |
|
|
training step 1773... |
|
|
training step 1774... |
|
|
training step 1775... |
|
|
training step 1776... |
|
|
training step 1777... |
|
|
training step 1778... |
|
|
training step 1779... |
|
|
training step 1780... |
|
|
training step 1781... |
|
|
training step 1782... |
|
|
training step 1783... |
|
|
training step 1784... |
|
|
training step 1785... |
|
|
training step 1786... |
|
|
training step 1787... |
|
|
training step 1788... |
|
|
training step 1789... |
|
|
training step 1790... |
|
|
training step 1791... |
|
|
training step 1792... |
|
|
training step 1793... |
|
|
training step 1794... |
|
|
training step 1795... |
|
|
training step 1796... |
|
|
training step 1797... |
|
|
training step 1798... |
|
|
training step 1799... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_001800/model.pt |
|
|
training step 1800... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.64it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.56it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.99it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.96it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.11it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.57it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.59it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 146.65it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.37it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 142.27it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 131.42it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 127.83it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 153.36it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.82it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 195.75it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 195.29it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 193.07it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 186.06it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 205.90it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 184.51it/s]
Saving video: 1%| | 1/81 [00:00<00:22, 3.59it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 186.42it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 202.24it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 197.85it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 175.52it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 175.70it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 194.53it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 200.13it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 207.82it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 182.69it/s]
Saving video: 22%|██▏ | 18/81 [00:00<00:01, 58.93it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 194.90it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 182.08it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.57it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.65it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 165.54it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 167.06it/s] |
|
|
Saving video: 41%|████ | 33/81 [00:00<00:00, 87.98it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 121.83it/s]
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 131.90it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 105.69it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1801... |
|
|
training step 1802... |
|
|
training step 1803... |
|
|
training step 1804... |
|
|
training step 1805... |
|
|
training step 1806... |
|
|
training step 1807... |
|
|
training step 1808... |
|
|
training step 1809... |
|
|
training step 1810... |
|
|
training step 1811... |
|
|
training step 1812... |
|
|
training step 1813... |
|
|
training step 1814... |
|
|
training step 1815... |
|
|
training step 1816... |
|
|
training step 1817... |
|
|
training step 1818... |
|
|
training step 1819... |
|
|
training step 1820... |
|
|
training step 1821... |
|
|
training step 1822... |
|
|
training step 1823... |
|
|
training step 1824... |
|
|
training step 1825... |
|
|
training step 1826... |
|
|
training step 1827... |
|
|
training step 1828... |
|
|
training step 1829... |
|
|
training step 1830... |
|
|
training step 1831... |
|
|
training step 1832... |
|
|
training step 1833... |
|
|
training step 1834... |
|
|
training step 1835... |
|
|
training step 1836... |
|
|
training step 1837... |
|
|
training step 1838... |
|
|
training step 1839... |
|
|
training step 1840... |
|
|
training step 1841... |
|
|
training step 1842... |
|
|
training step 1843... |
|
|
training step 1844... |
|
|
training step 1845... |
|
|
training step 1846... |
|
|
training step 1847... |
|
|
training step 1848... |
|
|
training step 1849... |
|
|
training step 1850... |
|
|
training step 1851... |
|
|
training step 1852... |
|
|
training step 1853... |
|
|
training step 1854... |
|
|
training step 1855... |
|
|
training step 1856... |
|
|
training step 1857... |
|
|
training step 1858... |
|
|
training step 1859... |
|
|
training step 1860... |
|
|
training step 1861... |
|
|
training step 1862... |
|
|
training step 1863... |
|
|
training step 1864... |
|
|
training step 1865... |
|
|
training step 1866... |
|
|
training step 1867... |
|
|
training step 1868... |
|
|
training step 1869... |
|
|
training step 1870... |
|
|
training step 1871... |
|
|
training step 1872... |
|
|
training step 1873... |
|
|
training step 1874... |
|
|
training step 1875... |
|
|
training step 1876... |
|
|
training step 1877... |
|
|
training step 1878... |
|
|
training step 1879... |
|
|
training step 1880... |
|
|
training step 1881... |
|
|
training step 1882... |
|
|
training step 1883... |
|
|
training step 1884... |
|
|
training step 1885... |
|
|
training step 1886... |
|
|
training step 1887... |
|
|
training step 1888... |
|
|
training step 1889... |
|
|
training step 1890... |
|
|
training step 1891... |
|
|
training step 1892... |
|
|
training step 1893... |
|
|
training step 1894... |
|
|
training step 1895... |
|
|
training step 1896... |
|
|
training step 1897... |
|
|
training step 1898... |
|
|
training step 1899... |
|
|
training step 1900... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.66it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.48it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.21it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.04it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.60it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.49it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.45it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 151.76it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 145.21it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.80it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 134.71it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 137.05it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 135.87it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 125.30it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 139.06it/s]
Saving video: 68%|██████▊ | 55/81 [00:00<00:00, 201.68it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 198.18it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 133.72it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 151.90it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 146.37it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 188.07it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 184.87it/s]
Saving video: 63%|██████▎ | 51/81 [00:00<00:00, 186.06it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 211.12it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 217.40it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 185.43it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 187.87it/s] |
|
|
Saving video: 62%|██████▏ | 50/81 [00:00<00:00, 178.79it/s]
Saving video: 83%|████████▎ | 67/81 [00:00<00:00, 176.38it/s]
Saving video: 78%|███████▊ | 63/81 [00:00<00:00, 168.33it/s]
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 189.07it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 157.03it/s] |
|
|
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 169.55it/s]
Saving video: 88%|████████▊ | 71/81 [00:00<00:00, 169.87it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.08it/s] |
|
|
Saving video: 85%|████████▌ | 69/81 [00:00<00:00, 171.43it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 154.69it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.49it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 158.07it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 156.16it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 1901... |
|
|
training step 1902... |
|
|
training step 1903... |
|
|
training step 1904... |
|
|
training step 1905... |
|
|
training step 1906... |
|
|
training step 1907... |
|
|
training step 1908... |
|
|
training step 1909... |
|
|
training step 1910... |
|
|
training step 1911... |
|
|
training step 1912... |
|
|
training step 1913... |
|
|
training step 1914... |
|
|
training step 1915... |
|
|
training step 1916... |
|
|
training step 1917... |
|
|
training step 1918... |
|
|
training step 1919... |
|
|
training step 1920... |
|
|
training step 1921... |
|
|
training step 1922... |
|
|
training step 1923... |
|
|
training step 1924... |
|
|
training step 1925... |
|
|
training step 1926... |
|
|
training step 1927... |
|
|
training step 1928... |
|
|
training step 1929... |
|
|
training step 1930... |
|
|
training step 1931... |
|
|
training step 1932... |
|
|
training step 1933... |
|
|
training step 1934... |
|
|
training step 1935... |
|
|
training step 1936... |
|
|
training step 1937... |
|
|
training step 1938... |
|
|
training step 1939... |
|
|
training step 1940... |
|
|
training step 1941... |
|
|
training step 1942... |
|
|
training step 1943... |
|
|
training step 1944... |
|
|
training step 1945... |
|
|
training step 1946... |
|
|
training step 1947... |
|
|
training step 1948... |
|
|
training step 1949... |
|
|
training step 1950... |
|
|
training step 1951... |
|
|
training step 1952... |
|
|
training step 1953... |
|
|
training step 1954... |
|
|
training step 1955... |
|
|
training step 1956... |
|
|
training step 1957... |
|
|
training step 1958... |
|
|
training step 1959... |
|
|
training step 1960... |
|
|
training step 1961... |
|
|
training step 1962... |
|
|
training step 1963... |
|
|
training step 1964... |
|
|
training step 1965... |
|
|
training step 1966... |
|
|
training step 1967... |
|
|
training step 1968... |
|
|
training step 1969... |
|
|
training step 1970... |
|
|
training step 1971... |
|
|
training step 1972... |
|
|
training step 1973... |
|
|
training step 1974... |
|
|
training step 1975... |
|
|
training step 1976... |
|
|
training step 1977... |
|
|
training step 1978... |
|
|
training step 1979... |
|
|
training step 1980... |
|
|
training step 1981... |
|
|
training step 1982... |
|
|
training step 1983... |
|
|
training step 1984... |
|
|
training step 1985... |
|
|
training step 1986... |
|
|
training step 1987... |
|
|
training step 1988... |
|
|
training step 1989... |
|
|
training step 1990... |
|
|
training step 1991... |
|
|
training step 1992... |
|
|
training step 1993... |
|
|
training step 1994... |
|
|
training step 1995... |
|
|
training step 1996... |
|
|
training step 1997... |
|
|
training step 1998... |
|
|
training step 1999... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_002000/model.pt |
|
|
training step 2000... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.40it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.78it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.19it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.87it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.59it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.68it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.54it/s]
Saving video: 28%|██▊ | 23/81 [00:00<00:00, 120.91it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 146.96it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 133.78it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 138.92it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 134.37it/s]
Saving video: 20%|█▉ | 16/81 [00:00<00:00, 84.35it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 136.62it/s]
Saving video: 1%| | 1/81 [00:00<00:18, 4.22it/s]
Saving video: 48%|████▊ | 39/81 [00:00<00:00, 136.72it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 198.96it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 182.17it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 189.56it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 190.69it/s]
Saving video: 48%|████▊ | 39/81 [00:00<00:00, 145.21it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 191.04it/s]
Saving video: 27%|██▋ | 22/81 [00:00<00:00, 81.27it/s]
Saving video: 74%|███████▍ | 60/81 [00:00<00:00, 163.95it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 206.07it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 180.92it/s] |
|
|
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 192.36it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 194.96it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 146.66it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 199.23it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 184.12it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 162.98it/s] |
|
|
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 175.52it/s]
Saving video: 52%|█████▏ | 42/81 [00:00<00:00, 119.85it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.33it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.03it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 151.37it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 160.10it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 156.66it/s]
Saving video: 75%|███████▌ | 61/81 [00:00<00:00, 141.44it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 137.82it/s] |
|
|
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 148.24it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 120.12it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 2001... |
|
|
training step 2002... |
|
|
training step 2003... |
|
|
training step 2004... |
|
|
training step 2005... |
|
|
training step 2006... |
|
|
training step 2007... |
|
|
training step 2008... |
|
|
training step 2009... |
|
|
training step 2010... |
|
|
training step 2011... |
|
|
training step 2012... |
|
|
training step 2013... |
|
|
training step 2014... |
|
|
training step 2015... |
|
|
training step 2016... |
|
|
training step 2017... |
|
|
training step 2018... |
|
|
training step 2019... |
|
|
training step 2020... |
|
|
training step 2021... |
|
|
training step 2022... |
|
|
training step 2023... |
|
|
training step 2024... |
|
|
training step 2025... |
|
|
training step 2026... |
|
|
training step 2027... |
|
|
training step 2028... |
|
|
training step 2029... |
|
|
training step 2030... |
|
|
training step 2031... |
|
|
training step 2032... |
|
|
training step 2033... |
|
|
training step 2034... |
|
|
training step 2035... |
|
|
training step 2036... |
|
|
training step 2037... |
|
|
training step 2038... |
|
|
training step 2039... |
|
|
training step 2040... |
|
|
training step 2041... |
|
|
training step 2042... |
|
|
training step 2043... |
|
|
training step 2044... |
|
|
training step 2045... |
|
|
training step 2046... |
|
|
training step 2047... |
|
|
training step 2048... |
|
|
training step 2049... |
|
|
training step 2050... |
|
|
training step 2051... |
|
|
training step 2052... |
|
|
training step 2053... |
|
|
training step 2054... |
|
|
training step 2055... |
|
|
training step 2056... |
|
|
training step 2057... |
|
|
training step 2058... |
|
|
training step 2059... |
|
|
training step 2060... |
|
|
training step 2061... |
|
|
training step 2062... |
|
|
training step 2063... |
|
|
training step 2064... |
|
|
training step 2065... |
|
|
training step 2066... |
|
|
training step 2067... |
|
|
training step 2068... |
|
|
training step 2069... |
|
|
training step 2070... |
|
|
training step 2071... |
|
|
training step 2072... |
|
|
training step 2073... |
|
|
training step 2074... |
|
|
training step 2075... |
|
|
training step 2076... |
|
|
training step 2077... |
|
|
training step 2078... |
|
|
training step 2079... |
|
|
training step 2080... |
|
|
training step 2081... |
|
|
training step 2082... |
|
|
training step 2083... |
|
|
training step 2084... |
|
|
training step 2085... |
|
|
training step 2086... |
|
|
training step 2087... |
|
|
training step 2088... |
|
|
training step 2089... |
|
|
training step 2090... |
|
|
training step 2091... |
|
|
training step 2092... |
|
|
training step 2093... |
|
|
training step 2094... |
|
|
training step 2095... |
|
|
training step 2096... |
|
|
training step 2097... |
|
|
training step 2098... |
|
|
training step 2099... |
|
|
training step 2100... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.71it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.30it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.63it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.83it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.59it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.49it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 147.85it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.56it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 137.80it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 131.82it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 138.66it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 150.08it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.08it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 130.59it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 200.80it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 124.86it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 188.95it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 184.12it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 191.52it/s]
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 204.32it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 123.44it/s]
Saving video: 64%|██████▍ | 52/81 [00:00<00:00, 184.55it/s]
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 215.48it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 188.32it/s] |
|
|
Saving video: 51%|█████ | 41/81 [00:00<00:00, 137.20it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 193.82it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 194.74it/s]
Saving video: 94%|█████████▍| 76/81 [00:00<00:00, 197.99it/s]
Saving video: 57%|█████▋ | 46/81 [00:00<00:00, 158.96it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 171.79it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.84it/s] |
|
|
Saving video: 96%|█████████▋| 78/81 [00:00<00:00, 191.11it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 170.75it/s] |
|
|
Saving video: 89%|████████▉ | 72/81 [00:00<00:00, 182.56it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.22it/s] |
|
|
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 163.37it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 161.54it/s] |
|
|
Saving video: 81%|████████▏ | 66/81 [00:00<00:00, 174.09it/s]
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 166.48it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 145.72it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 152.02it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 2101... |
|
|
training step 2102... |
|
|
training step 2103... |
|
|
training step 2104... |
|
|
training step 2105... |
|
|
training step 2106... |
|
|
training step 2107... |
|
|
training step 2108... |
|
|
training step 2109... |
|
|
training step 2110... |
|
|
training step 2111... |
|
|
training step 2112... |
|
|
training step 2113... |
|
|
training step 2114... |
|
|
training step 2115... |
|
|
training step 2116... |
|
|
training step 2117... |
|
|
training step 2118... |
|
|
training step 2119... |
|
|
training step 2120... |
|
|
training step 2121... |
|
|
training step 2122... |
|
|
training step 2123... |
|
|
training step 2124... |
|
|
training step 2125... |
|
|
training step 2126... |
|
|
training step 2127... |
|
|
training step 2128... |
|
|
training step 2129... |
|
|
training step 2130... |
|
|
training step 2131... |
|
|
training step 2132... |
|
|
training step 2133... |
|
|
training step 2134... |
|
|
training step 2135... |
|
|
training step 2136... |
|
|
training step 2137... |
|
|
training step 2138... |
|
|
training step 2139... |
|
|
training step 2140... |
|
|
training step 2141... |
|
|
training step 2142... |
|
|
training step 2143... |
|
|
training step 2144... |
|
|
training step 2145... |
|
|
training step 2146... |
|
|
training step 2147... |
|
|
training step 2148... |
|
|
training step 2149... |
|
|
training step 2150... |
|
|
training step 2151... |
|
|
training step 2152... |
|
|
training step 2153... |
|
|
training step 2154... |
|
|
training step 2155... |
|
|
training step 2156... |
|
|
training step 2157... |
|
|
training step 2158... |
|
|
training step 2159... |
|
|
training step 2160... |
|
|
training step 2161... |
|
|
training step 2162... |
|
|
training step 2163... |
|
|
training step 2164... |
|
|
training step 2165... |
|
|
training step 2166... |
|
|
training step 2167... |
|
|
training step 2168... |
|
|
training step 2169... |
|
|
training step 2170... |
|
|
training step 2171... |
|
|
training step 2172... |
|
|
training step 2173... |
|
|
training step 2174... |
|
|
training step 2175... |
|
|
training step 2176... |
|
|
training step 2177... |
|
|
training step 2178... |
|
|
training step 2179... |
|
|
training step 2180... |
|
|
training step 2181... |
|
|
training step 2182... |
|
|
training step 2183... |
|
|
training step 2184... |
|
|
training step 2185... |
|
|
training step 2186... |
|
|
training step 2187... |
|
|
training step 2188... |
|
|
training step 2189... |
|
|
training step 2190... |
|
|
training step 2191... |
|
|
training step 2192... |
|
|
training step 2193... |
|
|
training step 2194... |
|
|
training step 2195... |
|
|
training step 2196... |
|
|
training step 2197... |
|
|
training step 2198... |
|
|
training step 2199... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Start gathering distributed model states... |
|
|
Model saved to ./output/2025-09-08-09-31-54.546544_seed6944469/checkpoint_model_002200/model.pt |
|
|
training step 2200... |
|
|
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 0%| | 0/81 [00:00<?, ?it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.61it/s]
Saving video: 1%| | 1/81 [00:00<00:08, 9.93it/s]
Saving video: 1%| | 1/81 [00:00<00:11, 7.13it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.45it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 146.86it/s]
Saving video: 1%| | 1/81 [00:00<00:09, 8.45it/s]
Saving video: 1%| | 1/81 [00:00<00:10, 7.75it/s]
Saving video: 1%| | 1/81 [00:00<00:12, 6.61it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 160.85it/s]
Saving video: 35%|███▍ | 28/81 [00:00<00:00, 137.47it/s]
Saving video: 33%|███▎ | 27/81 [00:00<00:00, 143.98it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 198.86it/s]
Saving video: 31%|███ | 25/81 [00:00<00:00, 133.20it/s]
Saving video: 1%| | 1/81 [00:00<00:18, 4.22it/s]
Saving video: 32%|███▏ | 26/81 [00:00<00:00, 132.39it/s]
Saving video: 30%|██▉ | 24/81 [00:00<00:00, 114.81it/s]
Saving video: 69%|██████▉ | 56/81 [00:00<00:00, 211.90it/s]
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 185.88it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 193.94it/s]
Saving video: 49%|████▉ | 40/81 [00:00<00:00, 140.09it/s]
Saving video: 19%|█▊ | 15/81 [00:00<00:01, 55.28it/s]
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 193.12it/s]
Saving video: 65%|██████▌ | 53/81 [00:00<00:00, 188.13it/s]
Saving video: 56%|█████▌ | 45/81 [00:00<00:00, 151.00it/s]
Saving video: 99%|█████████▉| 80/81 [00:00<00:00, 220.77it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 196.25it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.06it/s] |
|
|
Saving video: 93%|█████████▎| 75/81 [00:00<00:00, 187.61it/s]
Saving video: 91%|█████████▏| 74/81 [00:00<00:00, 187.93it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 166.22it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 172.35it/s] |
|
|
Saving video: 74%|███████▍ | 60/81 [00:00<00:00, 159.32it/s]
Saving video: 46%|████▌ | 37/81 [00:00<00:00, 111.52it/s]
Saving video: 77%|███████▋ | 62/81 [00:00<00:00, 155.81it/s]
Saving video: 90%|█████████ | 73/81 [00:00<00:00, 183.73it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 163.59it/s] |
|
|
Saving video: 67%|██████▋ | 54/81 [00:00<00:00, 130.43it/s]
Saving video: 98%|█████████▊| 79/81 [00:00<00:00, 168.00it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 149.99it/s] |
|
|
Saving video: 100%|██████████| 81/81 [00:00<00:00, 145.91it/s] |
|
|
Saving video: 86%|████████▋ | 70/81 [00:00<00:00, 138.52it/s]
Saving video: 100%|██████████| 81/81 [00:00<00:00, 114.94it/s] |
|
|
in main process |
|
|
log video_0 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_1 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_2 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_3 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_4 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_5 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_6 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
log video_7 |
|
|
wandb: WARNING `fps` argument does not affect the frame rate of the video when providing a file path or raw bytes. |
|
|
training step 2201... |
|
|
training step 2202... |
|
|
training step 2203... |
|
|
training step 2204... |
|
|
training step 2205... |
|
|
training step 2206... |
|
|
training step 2207... |
|
|
training step 2208... |
|
|
training step 2209... |
|
|
training step 2210... |
|
|
training step 2211... |
|
|
training step 2212... |
|
|
training step 2213... |
|
|
training step 2214... |
|
|
training step 2215... |
|
|
training step 2216... |
|
|
training step 2217... |
|
|
training step 2218... |
|
|
training step 2219... |
|
|
training step 2220... |
|
|
training step 2221... |
|
|
training step 2222... |
|
|
training step 2223... |
|
|
training step 2224... |
|
|
training step 2225... |
|
|
training step 2226... |
|
|
training step 2227... |
|
|
training step 2228... |
|
|
training step 2229... |
|
|
training step 2230... |
|
|
training step 2231... |
|
|
training step 2232... |
|
|
training step 2233... |
|
|
training step 2234... |
|
|
training step 2235... |
|
|
training step 2236... |
|
|
training step 2237... |
|
|
training step 2238... |
|
|
training step 2239... |
|
|
training step 2240... |
|
|
training step 2241... |
|
|
training step 2242... |
|
|
training step 2243... |
|
|
training step 2244... |
|
|
training step 2245... |
|
|
training step 2246... |
|
|
training step 2247... |
|
|
training step 2248... |
|
|
training step 2249... |
|
|
training step 2250... |
|
|
training step 2251... |
|
|
training step 2252... |
|
|
training step 2253... |
|
|
training step 2254... |
|
|
training step 2255... |
|
|
training step 2256... |
|
|
training step 2257... |
|
|
training step 2258... |
|
|
training step 2259... |
|
|
training step 2260... |
|
|
training step 2261... |
|
|
training step 2262... |
|
|
training step 2263... |
|
|
training step 2264... |
|
|
training step 2265... |
|
|
W0908 21:58:13.132000 4132297 site-packages/torch/distributed/elastic/agent/server/api.py:723] Received Signals.SIGHUP death signal, shutting down workers |
|
|
W0908 21:58:13.135000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132394 closing signal SIGHUP |
|
|
W0908 21:58:13.136000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132395 closing signal SIGHUP |
|
|
W0908 21:58:13.137000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132396 closing signal SIGHUP |
|
|
W0908 21:58:13.150000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132397 closing signal SIGHUP |
|
|
W0908 21:58:13.151000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132398 closing signal SIGHUP |
|
|
W0908 21:58:13.181000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132399 closing signal SIGHUP |
|
|
W0908 21:58:13.183000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132400 closing signal SIGHUP |
|
|
W0908 21:58:13.190000 4132297 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4132401 closing signal SIGHUP |
|
|
Traceback (most recent call last): |
|
|
File "/home/yitongli/miniconda3/envs/causvid/bin/torchrun", line 7, in <module> |
|
|
sys.exit(main()) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper |
|
|
return f(*args, **kwargs) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main |
|
|
run(args) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run |
|
|
elastic_launch( |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 143, in __call__ |
|
|
return launch_agent(self._config, self._entrypoint, list(args)) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent |
|
|
result = agent.run() |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper |
|
|
result = f(*args, **kwargs) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 715, in run |
|
|
result = self._invoke_run(role) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 879, in _invoke_run |
|
|
time.sleep(monitor_interval) |
|
|
File "/home/yitongli/miniconda3/envs/causvid/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler |
|
|
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval) |
|
|
torch.distributed.elastic.multiprocessing.api.SignalException: Process 4132297 got signal: 1 |
|
|
|
