2025-05-06,11:15:52 | INFO | No latest resume checkpoint found in ./logs-lr1e-3-datacomp/clip_vit_b16_s512m_bs16k_mix0_8/checkpoints.
2025-05-06,11:15:53 | INFO | Running in distributed mode with multiple processes. Device: cuda:0.Process (global: 0, local 0), total 16.
2025-05-06,11:15:53 | INFO | Loaded ViT-B-16 model config.
2025-05-06,11:15:55 | INFO | Model:
2025-05-06,11:15:55 | INFO | CLIP(
  (visual): VisionTransformer(
    (conv1): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16), bias=False)
    (patch_dropout): Identity()
    (ln_pre): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (transformer): Transformer(
      (resblocks): ModuleList(
        (0-11): 12 x ResidualAttentionBlock(
          (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (attn): MultiheadAttention(
            (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
          )
          (ls_1): Identity()
          (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): Sequential(
            (c_fc): Linear(in_features=768, out_features=3072, bias=True)
            (gelu): GELU(approximate='none')
            (c_proj): Linear(in_features=3072, out_features=768, bias=True)
          )
          (ls_2): Identity()
        )
      )
    )
    (ln_post): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (transformer): Transformer(
    (resblocks): ModuleList(
      (0-11): 12 x ResidualAttentionBlock(
        (ln_1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (attn): MultiheadAttention(
          (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
        )
        (ls_1): Identity()
        (ln_2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (mlp): Sequential(
          (c_fc): Linear(in_features=512, out_features=2048, bias=True)
          (gelu): GELU(approximate='none')
          (c_proj): Linear(in_features=2048, out_features=512, bias=True)
        )
        (ls_2): Identity()
      )
    )
  )
  (token_embedding): Embedding(49408, 512)
  (ln_final): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
)
2025-05-06,11:15:55 | INFO | Params:
2025-05-06,11:15:55 | INFO |   NDR_patch_size: 16
2025-05-06,11:15:55 | INFO |   accum_freq: 1
2025-05-06,11:15:55 | INFO |   aug_cfg: {}
2025-05-06,11:15:55 | INFO |   batch_size: 1024
2025-05-06,11:15:55 | INFO |   beta1: 0.9
2025-05-06,11:15:55 | INFO |   beta2: 0.98
2025-05-06,11:15:55 | INFO |   checkpoint_path: ./logs-lr1e-3-datacomp/clip_vit_b16_s512m_bs16k_mix0_8/checkpoints
2025-05-06,11:15:55 | INFO |   coca_caption_loss_weight: 2.0
2025-05-06,11:15:55 | INFO |   coca_contrastive_loss_weight: 1.0
2025-05-06,11:15:55 | INFO |   copy_codebase: False
2025-05-06,11:15:55 | INFO |   csv_caption_key: title
2025-05-06,11:15:55 | INFO |   csv_img_key: filepath
2025-05-06,11:15:55 | INFO |   csv_separator: 	
2025-05-06,11:15:55 | INFO |   dataset_resampled: False
2025-05-06,11:15:55 | INFO |   dataset_type: webdataset
2025-05-06,11:15:55 | INFO |   ddp_static_graph: True
2025-05-06,11:15:55 | INFO |   debug: False
2025-05-06,11:15:55 | INFO |   delete_prev_step_ckpt: True
2025-05-06,11:15:55 | INFO |   delete_previous_checkpoint: False
2025-05-06,11:15:55 | INFO |   device: cuda:0
2025-05-06,11:15:55 | INFO |   dist_backend: nccl
2025-05-06,11:15:55 | INFO |   dist_url: env://
2025-05-06,11:15:55 | INFO |   distill: False
2025-05-06,11:15:55 | INFO |   distill_model: None
2025-05-06,11:15:55 | INFO |   distill_pretrained: None
2025-05-06,11:15:55 | INFO |   distributed: True
2025-05-06,11:15:55 | INFO |   epochs: 4
2025-05-06,11:15:55 | INFO |   epochs_cooldown: None
2025-05-06,11:15:55 | INFO |   eps: 1e-06
2025-05-06,11:15:55 | INFO |   force_custom_text: False
2025-05-06,11:15:55 | INFO |   force_image_size: 224
2025-05-06,11:15:55 | INFO |   force_patch_dropout: None
2025-05-06,11:15:55 | INFO |   force_quick_gelu: False
2025-05-06,11:15:55 | INFO |   gather_with_grad: True
2025-05-06,11:15:55 | INFO |   global_batch_size: 16384
2025-05-06,11:15:55 | INFO |   grad_checkpointing: True
2025-05-06,11:15:55 | INFO |   grad_clip_norm: None
2025-05-06,11:15:55 | INFO |   horovod: False
2025-05-06,11:15:55 | INFO |   image_interpolation: None
2025-05-06,11:15:55 | INFO |   image_mean: None
2025-05-06,11:15:55 | INFO |   image_resize_mode: None
2025-05-06,11:15:55 | INFO |   image_std: None
2025-05-06,11:15:55 | INFO |   imagenet_v2: None
2025-05-06,11:15:55 | INFO |   imagenet_val: /mnt/bn/zilongdata-hl/dataset/imagenet/val
2025-05-06,11:15:55 | INFO |   is_cls_token: True
2025-05-06,11:15:55 | INFO |   local_loss: True
2025-05-06,11:15:55 | INFO |   local_rank: 0
2025-05-06,11:15:55 | INFO |   lock_image: False
2025-05-06,11:15:55 | INFO |   lock_image_freeze_bn_stats: False
2025-05-06,11:15:55 | INFO |   lock_image_unlocked_groups: 0
2025-05-06,11:15:55 | INFO |   lock_text: False
2025-05-06,11:15:55 | INFO |   lock_text_freeze_layer_norm: False
2025-05-06,11:15:55 | INFO |   lock_text_unlocked_layers: 0
2025-05-06,11:15:55 | INFO |   log_every_n_steps: 128
2025-05-06,11:15:55 | INFO |   log_level: 20
2025-05-06,11:15:55 | INFO |   log_local: False
2025-05-06,11:15:55 | INFO |   log_path: ./logs-lr1e-3-datacomp/clip_vit_b16_s512m_bs16k_mix0_8/out.log
2025-05-06,11:15:55 | INFO |   logs: ./logs-lr1e-3-datacomp
2025-05-06,11:15:55 | INFO |   lr: 0.001
2025-05-06,11:15:55 | INFO |   lr_cooldown_end: 0.0
2025-05-06,11:15:55 | INFO |   lr_cooldown_power: 1.0
2025-05-06,11:15:55 | INFO |   lr_scheduler: cosine
2025-05-06,11:15:55 | INFO |   max_seq_len: 15000
2025-05-06,11:15:55 | INFO |   model: ViT-B-16
2025-05-06,11:15:55 | INFO |   name: clip_vit_b16_s512m_bs16k_mix0_8
2025-05-06,11:15:55 | INFO |   native_dynamic_resolution: False
2025-05-06,11:15:55 | INFO |   no_set_device_rank: False
2025-05-06,11:15:55 | INFO |   only_packing: False
2025-05-06,11:15:55 | INFO |   precision: amp
2025-05-06,11:15:55 | INFO |   pretrained: 
2025-05-06,11:15:55 | INFO |   pretrained_image: 
2025-05-06,11:15:55 | INFO |   pretrained_text: 
2025-05-06,11:15:55 | INFO |   rank: 0
2025-05-06,11:15:55 | INFO |   remote_sync: None
2025-05-06,11:15:55 | INFO |   remote_sync_frequency: 300
2025-05-06,11:15:55 | INFO |   remote_sync_protocol: s3
2025-05-06,11:15:55 | INFO |   report_to: wandb
2025-05-06,11:15:55 | INFO |   resume: None
2025-05-06,11:15:55 | INFO |   rope_attn_num_heads: 12
2025-05-06,11:15:55 | INFO |   rope_model_width: 768
2025-05-06,11:15:55 | INFO |   save_every_n_steps: 6104
2025-05-06,11:15:55 | INFO |   save_frequency: 1
2025-05-06,11:15:55 | INFO |   save_most_recent: False
2025-05-06,11:15:55 | INFO |   seed: 0
2025-05-06,11:15:55 | INFO |   siglip: False
2025-05-06,11:15:55 | INFO |   skip_scheduler: False
2025-05-06,11:15:55 | INFO |   tensorboard: False
2025-05-06,11:15:55 | INFO |   tensorboard_path: 
2025-05-06,11:15:55 | INFO |   torchcompile: False
2025-05-06,11:15:55 | INFO |   torchscript: False
2025-05-06,11:15:55 | INFO |   trace: False
2025-05-06,11:15:55 | INFO |   train_data: /mnt/bn/zilongdata-hl/dataset/Recap-DataComp-1B-Dataset/{000000..140146}.tar
2025-05-06,11:15:55 | INFO |   train_data_upsampling_factors: None
2025-05-06,11:15:55 | INFO |   train_num_samples: 128000000
2025-05-06,11:15:55 | INFO |   use_bn_sync: False
2025-05-06,11:15:55 | INFO |   use_bnb_linear: None
2025-05-06,11:15:55 | INFO |   val_data: None
2025-05-06,11:15:55 | INFO |   val_frequency: 1
2025-05-06,11:15:55 | INFO |   val_num_samples: None
2025-05-06,11:15:55 | INFO |   val_steps: 0
2025-05-06,11:15:55 | INFO |   wandb: True
2025-05-06,11:15:55 | INFO |   wandb_notes: 
2025-05-06,11:15:55 | INFO |   wandb_project_name: cls-clip-NDR
2025-05-06,11:15:55 | INFO |   warmup: 500
2025-05-06,11:15:55 | INFO |   wd: 0.2
2025-05-06,11:15:55 | INFO |   workers: 1
2025-05-06,11:15:55 | INFO |   world_size: 16
2025-05-06,11:15:55 | INFO |   zeroshot_frequency: 4
2025-05-06,11:15:55 | INFO |   zeroshot_steps: 0
2025-05-06,11:16:11 | INFO | Start epoch 0
2025-05-06,11:16:26 | INFO | Train Epoch: 0 [    16384/128008192 (0%)] Data (t): 7.820 Batch (t): 14.995, 1092.60/s, 68.2874/s/gpu LR: 0.000002 Logit Scale: 14.286 Contrastive_loss: 9.8625 (9.8625) Loss: 9.8625 (9.8625)
2025-05-06,11:18:12 | WARNING | Handling webdataset error (OSError('image file is truncated (44 bytes not processed)')). Ignoring.
2025-05-06,11:22:42 | WARNING | Handling webdataset error (OSError('image file is truncated (47 bytes not processed)')). Ignoring.
2025-05-06,11:28:13 | INFO | Train Epoch: 0 [  2113536/128008192 (2%)] Data (t): 0.359 Batch (t): 5.527, 2971.75/s, 185.735/s/gpu LR: 0.000258 Logit Scale: 14.262 Contrastive_loss: 9.1416 (9.5020) Loss: 9.1416 (9.5020)
2025-05-06,11:32:39 | WARNING | Handling webdataset error (OSError('image file is truncated (68 bytes not processed)')). Ignoring.
2025-05-06,11:40:17 | INFO | Train Epoch: 0 [  4210688/128008192 (3%)] Data (t): 0.308 Batch (t): 5.653, 2750.86/s, 171.929/s/gpu LR: 0.000514 Logit Scale: 14.289 Contrastive_loss: 8.6437 (9.2159) Loss: 8.6437 (9.2159)
2025-05-06,11:42:53 | WARNING | Handling webdataset error (OSError('image file is truncated (25 bytes not processed)')). Ignoring.
2025-05-06,11:52:19 | INFO | Train Epoch: 0 [  6307840/128008192 (5%)] Data (t): 0.362 Batch (t): 5.643, 2882.29/s, 180.143/s/gpu LR: 0.000770 Logit Scale: 14.467 Contrastive_loss: 8.5072 (9.0387) Loss: 8.5072 (9.0387)
2025-05-06,12:04:14 | INFO | Train Epoch: 0 [  8404992/128008192 (7%)] Data (t): 0.369 Batch (t): 5.580, 2950.79/s, 184.424/s/gpu LR: 0.001000 Logit Scale: 15.028 Contrastive_loss: 7.9668 (8.8244) Loss: 7.9668 (8.8244)
2025-05-06,12:16:11 | INFO | Train Epoch: 0 [ 10502144/128008192 (8%)] Data (t): 0.373 Batch (t): 5.603, 2736.42/s, 171.026/s/gpu LR: 0.001000 Logit Scale: 16.438 Contrastive_loss: 7.5371 (8.6098) Loss: 7.5371 (8.6098)
2025-05-06,12:19:09 | WARNING | Handling webdataset error (OSError('image file is truncated (104 bytes not processed)')). Ignoring.
2025-05-06,12:20:09 | WARNING | Handling webdataset error (OSError('image file is truncated (21 bytes not processed)')). Ignoring.
2025-05-06,12:28:16 | INFO | Train Epoch: 0 [ 12599296/128008192 (10%)] Data (t): 0.328 Batch (t): 5.666, 2938.99/s, 183.687/s/gpu LR: 0.001000 Logit Scale: 18.184 Contrastive_loss: 7.1079 (8.3953) Loss: 7.1079 (8.3953)
2025-05-06,12:40:10 | INFO | Train Epoch: 0 [ 14696448/128008192 (11%)] Data (t): 0.363 Batch (t): 5.579, 2995.88/s, 187.242/s/gpu LR: 0.001000 Logit Scale: 20.349 Contrastive_loss: 6.7090 (8.1845) Loss: 6.7090 (8.1845)
2025-05-06,12:43:04 | WARNING | Handling webdataset error (OSError('image file is truncated (32 bytes not processed)')). Ignoring.
2025-05-06,12:49:38 | WARNING | Handling webdataset error (OSError('image file is truncated (23 bytes not processed)')). Ignoring.
2025-05-06,12:52:12 | INFO | Train Epoch: 0 [ 16793600/128008192 (13%)] Data (t): 0.355 Batch (t): 5.636, 2950.08/s, 184.380/s/gpu LR: 0.000999 Logit Scale: 22.680 Contrastive_loss: 6.1430 (7.9577) Loss: 6.1430 (7.9577)
2025-05-06,13:04:15 | INFO | Train Epoch: 0 [ 18890752/128008192 (15%)] Data (t): 0.460 Batch (t): 5.650, 3004.99/s, 187.812/s/gpu LR: 0.000999 Logit Scale: 25.027 Contrastive_loss: 3.6600 (7.5279) Loss: 3.6600 (7.5279)
2025-05-06,13:16:12 | INFO | Train Epoch: 0 [ 20987904/128008192 (16%)] Data (t): 0.366 Batch (t): 5.606, 2970.51/s, 185.657/s/gpu LR: 0.000998 Logit Scale: 27.521 Contrastive_loss: 5.6158 (7.3541) Loss: 5.6158 (7.3541)
2025-05-06,13:28:16 | INFO | Train Epoch: 0 [ 23085056/128008192 (18%)] Data (t): 0.379 Batch (t): 5.656, 2945.80/s, 184.113/s/gpu LR: 0.000998 Logit Scale: 30.048 Contrastive_loss: 5.2808 (7.1813) Loss: 5.2808 (7.1813)
2025-05-06,13:40:14 | INFO | Train Epoch: 0 [ 25182208/128008192 (20%)] Data (t): 0.372 Batch (t): 5.609, 2939.78/s, 183.736/s/gpu LR: 0.000997 Logit Scale: 32.361 Contrastive_loss: 5.7801 (7.0735) Loss: 5.7801 (7.0735)
2025-05-06,13:52:15 | INFO | Train Epoch: 0 [ 27279360/128008192 (21%)] Data (t): 0.454 Batch (t): 5.633, 2949.86/s, 184.366/s/gpu LR: 0.000996 Logit Scale: 34.083 Contrastive_loss: 5.0629 (6.9299) Loss: 5.0629 (6.9299)
2025-05-06,13:55:58 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
2025-05-06,14:04:25 | INFO | Train Epoch: 0 [ 29376512/128008192 (23%)] Data (t): 0.334 Batch (t): 5.703, 2931.14/s, 183.196/s/gpu LR: 0.000996 Logit Scale: 36.342 Contrastive_loss: 4.7965 (6.7877) Loss: 4.7965 (6.7877)
2025-05-06,14:06:52 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
2025-05-06,14:16:28 | INFO | Train Epoch: 0 [ 31473664/128008192 (25%)] Data (t): 0.371 Batch (t): 5.649, 2802.83/s, 175.177/s/gpu LR: 0.000995 Logit Scale: 38.258 Contrastive_loss: 4.5733 (6.6493) Loss: 4.5733 (6.6493)
2025-05-06,14:28:29 | INFO | Train Epoch: 0 [ 33570816/128008192 (26%)] Data (t): 0.370 Batch (t): 5.630, 2780.05/s, 173.753/s/gpu LR: 0.000994 Logit Scale: 40.039 Contrastive_loss: 4.5594 (6.5263) Loss: 4.5594 (6.5263)
2025-05-06,14:32:16 | WARNING | Handling webdataset error (OSError('image file is truncated (0 bytes not processed)')). Ignoring.
2025-05-06,14:40:36 | INFO | Train Epoch: 0 [ 35667968/128008192 (28%)] Data (t): 0.369 Batch (t): 5.679, 2787.70/s, 174.231/s/gpu LR: 0.000993 Logit Scale: 41.968 Contrastive_loss: 4.3889 (6.4076) Loss: 4.3889 (6.4076)
2025-05-06,14:51:55 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
2025-05-06,14:52:38 | INFO | Train Epoch: 0 [ 37765120/128008192 (30%)] Data (t): 0.364 Batch (t): 5.644, 2932.95/s, 183.310/s/gpu LR: 0.000992 Logit Scale: 43.302 Contrastive_loss: 4.1874 (6.2907) Loss: 4.1874 (6.2907)
2025-05-06,14:53:03 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
2025-05-06,15:03:15 | WARNING | Handling webdataset error (OSError('image file is truncated (84 bytes not processed)')). Ignoring.
2025-05-06,15:04:34 | INFO | Train Epoch: 0 [ 39862272/128008192 (31%)] Data (t): 0.403 Batch (t): 5.595, 2943.46/s, 183.967/s/gpu LR: 0.000990 Logit Scale: 44.770 Contrastive_loss: 2.1872 (6.0856) Loss: 2.1872 (6.0856)
2025-05-06,15:16:31 | INFO | Train Epoch: 0 [ 41959424/128008192 (33%)] Data (t): 0.375 Batch (t): 5.595, 2966.64/s, 185.415/s/gpu LR: 0.000989 Logit Scale: 45.999 Contrastive_loss: 1.9974 (5.8909) Loss: 1.9974 (5.8909)
2025-05-06,15:28:30 | INFO | Train Epoch: 0 [ 44056576/128008192 (34%)] Data (t): 0.368 Batch (t): 5.616, 2816.97/s, 176.061/s/gpu LR: 0.000988 Logit Scale: 47.398 Contrastive_loss: 1.8645 (5.7079) Loss: 1.8645 (5.7079)
2025-05-06,15:31:37 | WARNING | Handling webdataset error (OSError('image file is truncated (49 bytes not processed)')). Ignoring.
2025-05-06,15:40:34 | INFO | Train Epoch: 0 [ 46153728/128008192 (36%)] Data (t): 0.370 Batch (t): 5.658, 2996.94/s, 187.309/s/gpu LR: 0.000986 Logit Scale: 48.395 Contrastive_loss: 3.6444 (5.6182) Loss: 3.6444 (5.6182)
2025-05-06,15:52:35 | INFO | Train Epoch: 0 [ 48250880/128008192 (38%)] Data (t): 0.337 Batch (t): 5.638, 2700.96/s, 168.810/s/gpu LR: 0.000984 Logit Scale: 49.322 Contrastive_loss: 1.7958 (5.4589) Loss: 1.7958 (5.4589)
2025-05-06,16:04:41 | INFO | Train Epoch: 0 [ 50348032/128008192 (39%)] Data (t): 0.358 Batch (t): 5.673, 2938.82/s, 183.676/s/gpu LR: 0.000983 Logit Scale: 50.212 Contrastive_loss: 3.6379 (5.3861) Loss: 3.6379 (5.3861)
2025-05-06,16:16:44 | INFO | Train Epoch: 0 [ 52445184/128008192 (41%)] Data (t): 0.371 Batch (t): 5.648, 2734.32/s, 170.895/s/gpu LR: 0.000981 Logit Scale: 51.162 Contrastive_loss: 3.4426 (5.3113) Loss: 3.4426 (5.3113)
2025-05-06,16:28:44 | INFO | Train Epoch: 0 [ 54542336/128008192 (43%)] Data (t): 0.333 Batch (t): 5.620, 2961.32/s, 185.083/s/gpu LR: 0.000979 Logit Scale: 52.000 Contrastive_loss: 3.4624 (5.2428) Loss: 3.4624 (5.2428)
2025-05-06,16:40:40 | INFO | Train Epoch: 0 [ 56639488/128008192 (44%)] Data (t): 0.329 Batch (t): 5.597, 2905.17/s, 181.573/s/gpu LR: 0.000977 Logit Scale: 52.822 Contrastive_loss: 3.2615 (5.1721) Loss: 3.2615 (5.1721)
2025-05-06,16:52:39 | INFO | Train Epoch: 0 [ 58736640/128008192 (46%)] Data (t): 0.327 Batch (t): 5.612, 2931.77/s, 183.235/s/gpu LR: 0.000975 Logit Scale: 53.305 Contrastive_loss: 3.2917 (5.1072) Loss: 3.2917 (5.1072)
2025-05-06,16:53:04 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
2025-05-06,16:53:05 | WARNING | Handling webdataset error (OSError('image file is truncated (107 bytes not processed)')). Ignoring.
2025-05-06,16:54:40 | WARNING | Handling webdataset error (OSError('image file is truncated (73 bytes not processed)')). Ignoring.
2025-05-06,17:04:38 | INFO | Train Epoch: 0 [ 60833792/128008192 (48%)] Data (t): 0.322 Batch (t): 5.618, 2938.24/s, 183.640/s/gpu LR: 0.000973 Logit Scale: 54.192 Contrastive_loss: 3.3259 (5.0478) Loss: 3.3259 (5.0478)
2025-05-06,17:16:41 | INFO | Train Epoch: 0 [ 62930944/128008192 (49%)] Data (t): 0.436 Batch (t): 5.650, 2948.11/s, 184.257/s/gpu LR: 0.000971 Logit Scale: 54.741 Contrastive_loss: 3.1910 (4.9879) Loss: 3.1910 (4.9879)
2025-05-06,17:17:33 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
2025-05-06,17:28:43 | INFO | Train Epoch: 0 [ 65028096/128008192 (51%)] Data (t): 0.775 Batch (t): 5.639, 2745.05/s, 171.566/s/gpu LR: 0.000969 Logit Scale: 55.792 Contrastive_loss: 3.3818 (4.9378) Loss: 3.3818 (4.9378)
2025-05-06,17:29:38 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
2025-05-06,17:40:45 | INFO | Train Epoch: 0 [ 67125248/128008192 (52%)] Data (t): 0.661 Batch (t): 5.645, 2951.04/s, 184.440/s/gpu LR: 0.000967 Logit Scale: 56.196 Contrastive_loss: 3.2736 (4.8873) Loss: 3.2736 (4.8873)
2025-05-06,17:48:25 | WARNING | Handling webdataset error (OSError('image file is truncated (46 bytes not processed)')). Ignoring.
2025-05-06,17:51:06 | WARNING | Handling webdataset error (OSError('image file is truncated (87 bytes not processed)')). Ignoring.
2025-05-06,17:52:51 | INFO | Train Epoch: 0 [ 69222400/128008192 (54%)] Data (t): 0.376 Batch (t): 5.670, 2913.01/s, 182.063/s/gpu LR: 0.000964 Logit Scale: 56.975 Contrastive_loss: 1.5908 (4.7904) Loss: 1.5908 (4.7904)
2025-05-06,17:56:51 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
2025-05-06,18:04:51 | INFO | Train Epoch: 0 [ 71319552/128008192 (56%)] Data (t): 0.372 Batch (t): 5.623, 2942.26/s, 183.891/s/gpu LR: 0.000962 Logit Scale: 57.494 Contrastive_loss: 1.7209 (4.7027) Loss: 1.7209 (4.7027)
2025-05-06,18:16:59 | INFO | Train Epoch: 0 [ 73416704/128008192 (57%)] Data (t): 0.331 Batch (t): 5.690, 2904.88/s, 181.555/s/gpu LR: 0.000959 Logit Scale: 57.956 Contrastive_loss: 2.9529 (4.6541) Loss: 2.9529 (4.6541)
2025-05-06,18:29:03 | INFO | Train Epoch: 0 [ 75513856/128008192 (59%)] Data (t): 0.381 Batch (t): 5.657, 2872.98/s, 179.561/s/gpu LR: 0.000957 Logit Scale: 58.590 Contrastive_loss: 3.1925 (4.6146) Loss: 3.1925 (4.6146)
2025-05-06,18:37:20 | WARNING | Handling webdataset error (OSError('image file is truncated (64 bytes not processed)')). Ignoring.
2025-05-06,18:41:05 | INFO | Train Epoch: 0 [ 77611008/128008192 (61%)] Data (t): 0.365 Batch (t): 5.641, 2867.43/s, 179.214/s/gpu LR: 0.000954 Logit Scale: 59.061 Contrastive_loss: 3.0052 (4.5722) Loss: 3.0052 (4.5722)
2025-05-06,18:53:05 | INFO | Train Epoch: 0 [ 79708160/128008192 (62%)] Data (t): 0.367 Batch (t): 5.627, 2894.94/s, 180.933/s/gpu LR: 0.000951 Logit Scale: 59.597 Contrastive_loss: 2.9692 (4.5311) Loss: 2.9692 (4.5311)
2025-05-06,19:05:09 | INFO | Train Epoch: 0 [ 81805312/128008192 (64%)] Data (t): 0.361 Batch (t): 5.650, 2959.29/s, 184.956/s/gpu LR: 0.000948 Logit Scale: 60.057 Contrastive_loss: 3.0224 (4.4934) Loss: 3.0224 (4.4934)
2025-05-06,19:17:15 | INFO | Train Epoch: 0 [ 83902464/128008192 (66%)] Data (t): 0.356 Batch (t): 5.677, 2930.57/s, 183.160/s/gpu LR: 0.000945 Logit Scale: 60.575 Contrastive_loss: 2.8710 (4.4538) Loss: 2.8710 (4.4538)
2025-05-06,19:29:22 | INFO | Train Epoch: 0 [ 85999616/128008192 (67%)] Data (t): 0.366 Batch (t): 5.678, 2739.51/s, 171.219/s/gpu LR: 0.000942 Logit Scale: 61.143 Contrastive_loss: 2.8311 (4.4152) Loss: 2.8311 (4.4152)
2025-05-06,19:30:57 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
2025-05-06,19:32:27 | WARNING | Handling webdataset error (OSError('image file is truncated (72 bytes not processed)')). Ignoring.
2025-05-06,19:41:27 | INFO | Train Epoch: 0 [ 88096768/128008192 (69%)] Data (t): 0.367 Batch (t): 5.660, 2910.16/s, 181.885/s/gpu LR: 0.000939 Logit Scale: 61.564 Contrastive_loss: 2.7854 (4.3773) Loss: 2.7854 (4.3773)
2025-05-06,19:53:29 | INFO | Train Epoch: 0 [ 90193920/128008192 (70%)] Data (t): 0.380 Batch (t): 5.644, 2960.01/s, 185.000/s/gpu LR: 0.000936 Logit Scale: 62.039 Contrastive_loss: 2.6949 (4.3390) Loss: 2.6949 (4.3390)
2025-05-06,20:05:31 | INFO | Train Epoch: 0 [ 92291072/128008192 (72%)] Data (t): 0.359 Batch (t): 5.641, 2922.71/s, 182.669/s/gpu LR: 0.000933 Logit Scale: 62.545 Contrastive_loss: 2.6536 (4.3016) Loss: 2.6536 (4.3016)
2025-05-06,20:17:37 | INFO | Train Epoch: 0 [ 94388224/128008192 (74%)] Data (t): 0.363 Batch (t): 5.675, 2926.47/s, 182.904/s/gpu LR: 0.000930 Logit Scale: 62.956 Contrastive_loss: 2.7740 (4.2684) Loss: 2.7740 (4.2684)
2025-05-06,20:29:36 | INFO | Train Epoch: 0 [ 96485376/128008192 (75%)] Data (t): 0.372 Batch (t): 5.610, 2967.65/s, 185.478/s/gpu LR: 0.000926 Logit Scale: 63.332 Contrastive_loss: 2.6079 (4.2330) Loss: 2.6079 (4.2330)
2025-05-06,20:41:35 | INFO | Train Epoch: 0 [ 98582528/128008192 (77%)] Data (t): 0.374 Batch (t): 5.624, 2969.13/s, 185.571/s/gpu LR: 0.000923 Logit Scale: 63.708 Contrastive_loss: 2.7005 (4.2011) Loss: 2.7005 (4.2011)
2025-05-06,20:53:37 | INFO | Train Epoch: 0 [100679680/128008192 (79%)] Data (t): 0.403 Batch (t): 5.637, 2900.15/s, 181.260/s/gpu LR: 0.000919 Logit Scale: 64.110 Contrastive_loss: 2.5642 (4.1677) Loss: 2.5642 (4.1677)
2025-05-06,21:05:39 | INFO | Train Epoch: 0 [102776832/128008192 (80%)] Data (t): 0.366 Batch (t): 5.642, 2945.40/s, 184.087/s/gpu LR: 0.000916 Logit Scale: 64.332 Contrastive_loss: 2.4129 (4.1326) Loss: 2.4129 (4.1326)
2025-05-06,21:11:07 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
2025-05-06,21:17:46 | INFO | Train Epoch: 0 [104873984/128008192 (82%)] Data (t): 0.357 Batch (t): 5.679, 2885.83/s, 180.365/s/gpu LR: 0.000912 Logit Scale: 64.817 Contrastive_loss: 2.6817 (4.1042) Loss: 2.6817 (4.1042)
2025-05-06,21:29:47 | INFO | Train Epoch: 0 [106971136/128008192 (84%)] Data (t): 0.365 Batch (t): 5.631, 2893.45/s, 180.841/s/gpu LR: 0.000908 Logit Scale: 65.060 Contrastive_loss: 2.2398 (4.0683) Loss: 2.2398 (4.0683)
2025-05-06,21:41:49 | INFO | Train Epoch: 0 [109068288/128008192 (85%)] Data (t): 0.367 Batch (t): 5.640, 2882.60/s, 180.163/s/gpu LR: 0.000904 Logit Scale: 65.432 Contrastive_loss: 1.2447 (4.0150) Loss: 1.2447 (4.0150)
2025-05-06,21:43:35 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
2025-05-06,21:53:52 | INFO | Train Epoch: 0 [111165440/128008192 (87%)] Data (t): 0.371 Batch (t): 5.648, 2967.65/s, 185.478/s/gpu LR: 0.000900 Logit Scale: 65.804 Contrastive_loss: 2.4425 (3.9859) Loss: 2.4425 (3.9859)
2025-05-06,22:05:50 | INFO | Train Epoch: 0 [113262592/128008192 (88%)] Data (t): 0.379 Batch (t): 5.615, 2930.93/s, 183.183/s/gpu LR: 0.000897 Logit Scale: 66.192 Contrastive_loss: 2.6200 (3.9611) Loss: 2.6200 (3.9611)
2025-05-06,22:16:03 | WARNING | Handling webdataset error (OSError('image file is truncated (101 bytes not processed)')). Ignoring.
2025-05-06,22:17:52 | INFO | Train Epoch: 0 [115359744/128008192 (90%)] Data (t): 0.340 Batch (t): 5.634, 2891.57/s, 180.723/s/gpu LR: 0.000892 Logit Scale: 66.476 Contrastive_loss: 2.4153 (3.9335) Loss: 2.4153 (3.9335)
2025-05-06,22:29:50 | INFO | Train Epoch: 0 [117456896/128008192 (92%)] Data (t): 0.350 Batch (t): 5.613, 2825.58/s, 176.599/s/gpu LR: 0.000888 Logit Scale: 66.937 Contrastive_loss: 2.3317 (3.9054) Loss: 2.3317 (3.9054)
2025-05-06,22:41:48 | INFO | Train Epoch: 0 [119554048/128008192 (93%)] Data (t): 0.359 Batch (t): 5.606, 2873.59/s, 179.600/s/gpu LR: 0.000884 Logit Scale: 67.221 Contrastive_loss: 2.3377 (3.8783) Loss: 2.3377 (3.8783)
2025-05-06,22:46:22 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
2025-05-06,22:51:44 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
2025-05-06,22:53:51 | INFO | Train Epoch: 0 [121651200/128008192 (95%)] Data (t): 0.359 Batch (t): 5.650, 2977.78/s, 186.111/s/gpu LR: 0.000880 Logit Scale: 67.640 Contrastive_loss: 2.2995 (3.8516) Loss: 2.2995 (3.8516)
2025-05-06,22:55:37 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
2025-05-06,23:00:27 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
2025-05-06,23:05:53 | INFO | Train Epoch: 0 [123748352/128008192 (97%)] Data (t): 0.312 Batch (t): 5.639, 2969.79/s, 185.612/s/gpu LR: 0.000876 Logit Scale: 67.915 Contrastive_loss: 2.3092 (3.8259) Loss: 2.3092 (3.8259)
2025-05-06,23:17:52 | INFO | Train Epoch: 0 [125845504/128008192 (98%)] Data (t): 0.371 Batch (t): 5.624, 2970.50/s, 185.657/s/gpu LR: 0.000871 Logit Scale: 68.247 Contrastive_loss: 1.2282 (3.7833) Loss: 1.2282 (3.7833)
2025-05-06,23:29:56 | INFO | Train Epoch: 0 [127942656/128008192 (100%)] Data (t): 0.375 Batch (t): 5.653, 2944.31/s, 184.019/s/gpu LR: 0.000867 Logit Scale: 68.496 Contrastive_loss: 2.3140 (3.7596) Loss: 2.3140 (3.7596)
2025-05-06,23:30:19 | INFO | Train Epoch: 0 [128008192/128008192 (100%)] Data (t): 0.378 Batch (t): 5.613, 3045.60/s, 190.350/s/gpu LR: 0.000867 Logit Scale: 68.521 Contrastive_loss: 1.2767 (3.7202) Loss: 1.2767 (3.7202)
2025-05-06,23:30:27 | INFO | Start epoch 1
2025-05-06,23:30:39 | INFO | Train Epoch: 1 [    16384/128008192 (0%)] Data (t): 7.340 Batch (t): 11.680, 1402.76/s, 87.6724/s/gpu LR: 0.000867 Logit Scale: 68.523 Contrastive_loss: 2.2564 (2.2564) Loss: 2.2564 (2.2564)
2025-05-06,23:42:44 | INFO | Train Epoch: 1 [  2113536/128008192 (2%)] Data (t): 0.386 Batch (t): 5.667, 2926.47/s, 182.905/s/gpu LR: 0.000862 Logit Scale: 68.460 Contrastive_loss: 2.1479 (2.2022) Loss: 2.1479 (2.2022)
2025-05-06,23:45:53 | WARNING | Handling webdataset error (OSError('image file is truncated (53 bytes not processed)')). Ignoring.
2025-05-06,23:46:46 | WARNING | Handling webdataset error (OSError('image file is truncated (50 bytes not processed)')). Ignoring.
2025-05-06,23:54:48 | INFO | Train Epoch: 1 [  4210688/128008192 (3%)] Data (t): 0.364 Batch (t): 5.657, 2946.86/s, 184.179/s/gpu LR: 0.000858 Logit Scale: 69.136 Contrastive_loss: 2.1735 (2.1926) Loss: 2.1735 (2.1926)
2025-05-07,00:06:49 | INFO | Train Epoch: 1 [  6307840/128008192 (5%)] Data (t): 0.368 Batch (t): 5.632, 2903.47/s, 181.467/s/gpu LR: 0.000853 Logit Scale: 69.571 Contrastive_loss: 1.2332 (1.9528) Loss: 1.2332 (1.9528)
2025-05-07,00:18:55 | INFO | Train Epoch: 1 [  8404992/128008192 (7%)] Data (t): 0.356 Batch (t): 5.666, 2887.57/s, 180.473/s/gpu LR: 0.000849 Logit Scale: 69.777 Contrastive_loss: 2.1708 (1.9964) Loss: 2.1708 (1.9964)
2025-05-07,00:30:54 | INFO | Train Epoch: 1 [ 10502144/128008192 (8%)] Data (t): 0.345 Batch (t): 5.617, 2880.49/s, 180.031/s/gpu LR: 0.000844 Logit Scale: 70.145 Contrastive_loss: 2.1428 (2.0208) Loss: 2.1428 (2.0208)
2025-05-07,00:42:51 | INFO | Train Epoch: 1 [ 12599296/128008192 (10%)] Data (t): 0.360 Batch (t): 5.604, 2841.89/s, 177.618/s/gpu LR: 0.000839 Logit Scale: 70.491 Contrastive_loss: 2.2344 (2.0513) Loss: 2.2344 (2.0513)
2025-05-07,00:54:54 | INFO | Train Epoch: 1 [ 14696448/128008192 (11%)] Data (t): 0.381 Batch (t): 5.650, 2922.66/s, 182.667/s/gpu LR: 0.000834 Logit Scale: 70.821 Contrastive_loss: 2.0008 (2.0450) Loss: 2.0008 (2.0450)
2025-05-07,01:06:51 | INFO | Train Epoch: 1 [ 16793600/128008192 (13%)] Data (t): 0.373 Batch (t): 5.601, 2918.17/s, 182.386/s/gpu LR: 0.000829 Logit Scale: 71.105 Contrastive_loss: 1.9557 (2.0351) Loss: 1.9557 (2.0351)
2025-05-07,01:19:03 | INFO | Train Epoch: 1 [ 18890752/128008192 (15%)] Data (t): 0.364 Batch (t): 5.722, 2957.49/s, 184.843/s/gpu LR: 0.000824 Logit Scale: 71.453 Contrastive_loss: 2.0794 (2.0395) Loss: 2.0794 (2.0395)
2025-05-07,01:23:19 | WARNING | Handling webdataset error (OSError('image file is truncated (6 bytes not processed)')). Ignoring.
2025-05-07,01:31:01 | INFO | Train Epoch: 1 [ 20987904/128008192 (16%)] Data (t): 0.363 Batch (t): 5.608, 2938.28/s, 183.643/s/gpu LR: 0.000819 Logit Scale: 71.808 Contrastive_loss: 2.0115 (2.0370) Loss: 2.0115 (2.0370)
2025-05-07,01:33:02 | WARNING | Handling webdataset error (OSError('image file is truncated (32 bytes not processed)')). Ignoring.
2025-05-07,01:34:45 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
2025-05-07,01:42:58 | INFO | Train Epoch: 1 [ 23085056/128008192 (18%)] Data (t): 0.368 Batch (t): 5.599, 2922.59/s, 182.662/s/gpu LR: 0.000814 Logit Scale: 72.115 Contrastive_loss: 2.0285 (2.0362) Loss: 2.0285 (2.0362)
2025-05-07,01:53:54 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
2025-05-07,01:55:02 | INFO | Train Epoch: 1 [ 25182208/128008192 (20%)] Data (t): 0.326 Batch (t): 5.654, 2798.75/s, 174.922/s/gpu LR: 0.000809 Logit Scale: 72.311 Contrastive_loss: 1.9601 (2.0304) Loss: 1.9601 (2.0304)
2025-05-07,02:03:29 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
2025-05-07,02:05:49 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
2025-05-07,02:07:00 | INFO | Train Epoch: 1 [ 27279360/128008192 (21%)] Data (t): 0.367 Batch (t): 5.616, 2941.64/s, 183.852/s/gpu LR: 0.000804 Logit Scale: 72.576 Contrastive_loss: 2.0316 (2.0305) Loss: 2.0316 (2.0305)
2025-05-07,02:10:41 | WARNING | Handling webdataset error (OSError('image file is truncated (131 bytes not processed)')). Ignoring.
2025-05-07,02:18:59 | INFO | Train Epoch: 1 [ 29376512/128008192 (23%)] Data (t): 0.370 Batch (t): 5.618, 2779.56/s, 173.723/s/gpu LR: 0.000799 Logit Scale: 72.832 Contrastive_loss: 1.9496 (2.0251) Loss: 1.9496 (2.0251)
2025-05-07,02:20:53 | WARNING | Handling webdataset error (OSError('image file is truncated (15 bytes not processed)')). Ignoring.
2025-05-07,02:31:05 | INFO | Train Epoch: 1 [ 31473664/128008192 (25%)] Data (t): 0.358 Batch (t): 5.667, 2928.95/s, 183.059/s/gpu LR: 0.000794 Logit Scale: 73.059 Contrastive_loss: 2.0487 (2.0266) Loss: 2.0487 (2.0266)
2025-05-07,02:43:19 | INFO | Train Epoch: 1 [ 33570816/128008192 (26%)] Data (t): 0.374 Batch (t): 5.732, 2852.07/s, 178.254/s/gpu LR: 0.000788 Logit Scale: 73.303 Contrastive_loss: 2.0640 (2.0288) Loss: 2.0640 (2.0288)
2025-05-07,02:49:33 | WARNING | Handling webdataset error (OSError('image file is truncated (24 bytes not processed)')). Ignoring.
2025-05-07,02:55:16 | INFO | Train Epoch: 1 [ 35667968/128008192 (28%)] Data (t): 0.372 Batch (t): 5.602, 2955.47/s, 184.717/s/gpu LR: 0.000783 Logit Scale: 73.686 Contrastive_loss: 1.0964 (1.9770) Loss: 1.0964 (1.9770)
2025-05-07,03:06:18 | WARNING | Handling webdataset error (OSError('image file is truncated (186 bytes not processed)')). Ignoring.
2025-05-07,03:07:12 | INFO | Train Epoch: 1 [ 37765120/128008192 (30%)] Data (t): 0.369 Batch (t): 5.599, 2914.02/s, 182.126/s/gpu LR: 0.000777 Logit Scale: 73.844 Contrastive_loss: 1.9792 (1.9771) Loss: 1.9792 (1.9771)
2025-05-07,03:19:12 | INFO | Train Epoch: 1 [ 39862272/128008192 (31%)] Data (t): 0.373 Batch (t): 5.621, 2712.48/s, 169.530/s/gpu LR: 0.000772 Logit Scale: 74.111 Contrastive_loss: 1.9005 (1.9732) Loss: 1.9005 (1.9732)
2025-05-07,03:31:13 | INFO | Train Epoch: 1 [ 41959424/128008192 (33%)] Data (t): 0.325 Batch (t): 5.637, 2922.11/s, 182.632/s/gpu LR: 0.000767 Logit Scale: 74.389 Contrastive_loss: 2.0176 (1.9754) Loss: 2.0176 (1.9754)
2025-05-07,03:43:12 | INFO | Train Epoch: 1 [ 44056576/128008192 (34%)] Data (t): 0.362 Batch (t): 5.610, 2897.40/s, 181.087/s/gpu LR: 0.000761 Logit Scale: 74.533 Contrastive_loss: 1.1601 (1.9383) Loss: 1.1601 (1.9383)
2025-05-07,03:55:11 | INFO | Train Epoch: 1 [ 46153728/128008192 (36%)] Data (t): 0.371 Batch (t): 5.618, 2948.97/s, 184.311/s/gpu LR: 0.000755 Logit Scale: 74.779 Contrastive_loss: 1.8618 (1.9350) Loss: 1.8618 (1.9350)
2025-05-07,04:07:13 | INFO | Train Epoch: 1 [ 48250880/128008192 (38%)] Data (t): 0.364 Batch (t): 5.640, 2922.70/s, 182.669/s/gpu LR: 0.000750 Logit Scale: 74.973 Contrastive_loss: 1.8896 (1.9331) Loss: 1.8896 (1.9331)
2025-05-07,04:19:09 | INFO | Train Epoch: 1 [ 50348032/128008192 (39%)] Data (t): 0.370 Batch (t): 5.596, 2964.66/s, 185.291/s/gpu LR: 0.000744 Logit Scale: 75.155 Contrastive_loss: 1.7545 (1.9259) Loss: 1.7545 (1.9259)
2025-05-07,04:21:52 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
2025-05-07,04:22:59 | WARNING | Handling webdataset error (OSError('image file is truncated (14 bytes not processed)')). Ignoring.
2025-05-07,04:31:02 | INFO | Train Epoch: 1 [ 52445184/128008192 (41%)] Data (t): 0.346 Batch (t): 5.575, 2951.84/s, 184.490/s/gpu LR: 0.000738 Logit Scale: 75.450 Contrastive_loss: 1.0872 (1.8937) Loss: 1.0872 (1.8937)
2025-05-07,04:43:01 | INFO | Train Epoch: 1 [ 54542336/128008192 (43%)] Data (t): 0.368 Batch (t): 5.613, 2977.84/s, 186.115/s/gpu LR: 0.000733 Logit Scale: 75.772 Contrastive_loss: 1.8309 (1.8914) Loss: 1.8309 (1.8914)
2025-05-07,04:54:59 | INFO | Train Epoch: 1 [ 56639488/128008192 (44%)] Data (t): 0.361 Batch (t): 5.610, 2819.99/s, 176.249/s/gpu LR: 0.000727 Logit Scale: 75.975 Contrastive_loss: 1.7066 (1.8848) Loss: 1.7066 (1.8848)
2025-05-07,04:58:44 | WARNING | Handling webdataset error (OSError('image file is truncated (76 bytes not processed)')). Ignoring.
2025-05-07,04:59:32 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
2025-05-07,05:06:59 | INFO | Train Epoch: 1 [ 58736640/128008192 (46%)] Data (t): 0.529 Batch (t): 5.627, 2907.55/s, 181.722/s/gpu LR: 0.000721 Logit Scale: 76.197 Contrastive_loss: 1.7846 (1.8813) Loss: 1.7846 (1.8813)
2025-05-07,05:08:42 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
2025-05-07,05:19:02 | INFO | Train Epoch: 1 [ 60833792/128008192 (48%)] Data (t): 0.374 Batch (t): 5.651, 2805.53/s, 175.346/s/gpu LR: 0.000715 Logit Scale: 76.367 Contrastive_loss: 1.7702 (1.8776) Loss: 1.7702 (1.8776)
2025-05-07,05:31:12 | INFO | Train Epoch: 1 [ 62930944/128008192 (49%)] Data (t): 0.365 Batch (t): 5.702, 2902.54/s, 181.409/s/gpu LR: 0.000709 Logit Scale: 76.704 Contrastive_loss: 1.8091 (1.8754) Loss: 1.8091 (1.8754)
2025-05-07,05:40:30 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
2025-05-07,05:40:42 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
2025-05-07,05:41:39 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
2025-05-07,05:43:24 | INFO | Train Epoch: 1 [ 65028096/128008192 (51%)] Data (t): 0.326 Batch (t): 5.720, 2861.81/s, 178.863/s/gpu LR: 0.000703 Logit Scale: 76.839 Contrastive_loss: 1.9369 (1.8773) Loss: 1.9369 (1.8773)
2025-05-07,05:48:52 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
2025-05-07,05:54:25 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
2025-05-07,05:55:23 | INFO | Train Epoch: 1 [ 67125248/128008192 (52%)] Data (t): 0.320 Batch (t): 5.611, 2968.05/s, 185.503/s/gpu LR: 0.000697 Logit Scale: 77.012 Contrastive_loss: 1.8594 (1.8768) Loss: 1.8594 (1.8768)
2025-05-07,06:07:25 | INFO | Train Epoch: 1 [ 69222400/128008192 (54%)] Data (t): 0.362 Batch (t): 5.645, 2202.57/s, 137.660/s/gpu LR: 0.000691 Logit Scale: 77.211 Contrastive_loss: 1.0003 (1.8510) Loss: 1.0003 (1.8510)
2025-05-07,06:19:28 | INFO | Train Epoch: 1 [ 71319552/128008192 (56%)] Data (t): 0.362 Batch (t): 5.649, 2899.81/s, 181.238/s/gpu LR: 0.000685 Logit Scale: 77.492 Contrastive_loss: 1.7459 (1.8480) Loss: 1.7459 (1.8480)
2025-05-07,06:31:30 | INFO | Train Epoch: 1 [ 73416704/128008192 (57%)] Data (t): 0.720 Batch (t): 5.635, 2894.92/s, 180.933/s/gpu LR: 0.000679 Logit Scale: 77.717 Contrastive_loss: 1.3131 (1.8331) Loss: 1.3131 (1.8331)
2025-05-07,06:39:37 | WARNING | Handling webdataset error (OSError('image file is truncated (151 bytes not processed)')). Ignoring.
2025-05-07,06:43:27 | INFO | Train Epoch: 1 [ 75513856/128008192 (59%)] Data (t): 0.359 Batch (t): 5.602, 2938.57/s, 183.661/s/gpu LR: 0.000673 Logit Scale: 77.788 Contrastive_loss: 0.93351 (1.8088) Loss: 0.93351 (1.8088)
2025-05-07,06:55:26 | INFO | Train Epoch: 1 [ 77611008/128008192 (61%)] Data (t): 0.362 Batch (t): 5.622, 3039.94/s, 189.996/s/gpu LR: 0.000667 Logit Scale: 78.041 Contrastive_loss: 1.5853 (1.8029) Loss: 1.5853 (1.8029)
2025-05-07,07:07:23 | INFO | Train Epoch: 1 [ 79708160/128008192 (62%)] Data (t): 0.359 Batch (t): 5.604, 2886.94/s, 180.434/s/gpu LR: 0.000661 Logit Scale: 78.338 Contrastive_loss: 1.8795 (1.8049) Loss: 1.8795 (1.8049)
2025-05-07,07:19:21 | INFO | Train Epoch: 1 [ 81805312/128008192 (64%)] Data (t): 0.366 Batch (t): 5.607, 2926.36/s, 182.897/s/gpu LR: 0.000654 Logit Scale: 78.432 Contrastive_loss: 1.6660 (1.8014) Loss: 1.6660 (1.8014)
2025-05-07,07:31:26 | INFO | Train Epoch: 1 [ 83902464/128008192 (66%)] Data (t): 0.367 Batch (t): 5.664, 2890.90/s, 180.681/s/gpu LR: 0.000648 Logit Scale: 78.677 Contrastive_loss: 1.7602 (1.8004) Loss: 1.7602 (1.8004)
2025-05-07,07:43:25 | INFO | Train Epoch: 1 [ 85999616/128008192 (67%)] Data (t): 0.376 Batch (t): 5.616, 2969.24/s, 185.577/s/gpu LR: 0.000642 Logit Scale: 78.888 Contrastive_loss: 1.0844 (1.7834) Loss: 1.0844 (1.7834)
2025-05-07,07:55:27 | INFO | Train Epoch: 1 [ 88096768/128008192 (69%)] Data (t): 0.350 Batch (t): 5.639, 2998.76/s, 187.423/s/gpu LR: 0.000636 Logit Scale: 79.049 Contrastive_loss: 1.8032 (1.7838) Loss: 1.8032 (1.7838)
2025-05-07,08:07:26 | INFO | Train Epoch: 1 [ 90193920/128008192 (70%)] Data (t): 0.372 Batch (t): 5.621, 2920.46/s, 182.529/s/gpu LR: 0.000629 Logit Scale: 79.261 Contrastive_loss: 1.5168 (1.7778) Loss: 1.5168 (1.7778)
2025-05-07,08:19:25 | INFO | Train Epoch: 1 [ 92291072/128008192 (72%)] Data (t): 0.377 Batch (t): 5.613, 2931.48/s, 183.217/s/gpu LR: 0.000623 Logit Scale: 79.416 Contrastive_loss: 1.0427 (1.7614) Loss: 1.0427 (1.7614)
2025-05-07,08:23:50 | WARNING | Handling webdataset error (OSError('image file is truncated (99 bytes not processed)')). Ignoring.
2025-05-07,08:28:24 | WARNING | Handling webdataset error (OSError('image file is truncated (45 bytes not processed)')). Ignoring.
2025-05-07,08:28:36 | WARNING | Handling webdataset error (OSError('image file is truncated (55 bytes not processed)')). Ignoring.
2025-05-07,08:31:26 | INFO | Train Epoch: 1 [ 94388224/128008192 (74%)] Data (t): 0.613 Batch (t): 5.639, 2884.26/s, 180.266/s/gpu LR: 0.000617 Logit Scale: 79.486 Contrastive_loss: 1.6426 (1.7588) Loss: 1.6426 (1.7588)
2025-05-07,08:43:30 | INFO | Train Epoch: 1 [ 96485376/128008192 (75%)] Data (t): 0.350 Batch (t): 5.656, 2964.51/s, 185.282/s/gpu LR: 0.000610 Logit Scale: 79.670 Contrastive_loss: 1.1184 (1.7452) Loss: 1.1184 (1.7452)
2025-05-07,08:43:46 | WARNING | Handling webdataset error (OSError('image file is truncated (33 bytes not processed)')). Ignoring.
2025-05-07,08:55:25 | INFO | Train Epoch: 1 [ 98582528/128008192 (77%)] Data (t): 0.347 Batch (t): 5.588, 2903.82/s, 181.489/s/gpu LR: 0.000604 Logit Scale: 79.946 Contrastive_loss: 1.5715 (1.7416) Loss: 1.5715 (1.7416)
2025-05-07,09:07:32 | INFO | Train Epoch: 1 [100679680/128008192 (79%)] Data (t): 0.351 Batch (t): 5.676, 2970.38/s, 185.649/s/gpu LR: 0.000597 Logit Scale: 80.047 Contrastive_loss: 1.5171 (1.7370) Loss: 1.5171 (1.7370)
2025-05-07,09:10:46 | WARNING | Handling webdataset error (OSError('image file is truncated (0 bytes not processed)')). Ignoring.
2025-05-07,09:14:56 | WARNING | Handling webdataset error (OSError('image file is truncated (108 bytes not processed)')). Ignoring.
2025-05-07,09:19:28 | INFO | Train Epoch: 1 [102776832/128008192 (80%)] Data (t): 0.343 Batch (t): 5.594, 2909.82/s, 181.864/s/gpu LR: 0.000591 Logit Scale: 80.331 Contrastive_loss: 1.5939 (1.7342) Loss: 1.5939 (1.7342)
2025-05-07,09:22:26 | WARNING | Handling webdataset error (OSError('image file is truncated (85 bytes not processed)')). Ignoring.
2025-05-07,09:28:35 | WARNING | Handling webdataset error (OSError('image file is truncated (25 bytes not processed)')). Ignoring.
2025-05-07,09:31:27 | INFO | Train Epoch: 1 [104873984/128008192 (82%)] Data (t): 0.380 Batch (t): 5.614, 2946.33/s, 184.145/s/gpu LR: 0.000585 Logit Scale: 80.441 Contrastive_loss: 1.4932 (1.7294) Loss: 1.4932 (1.7294)
2025-05-07,09:43:28 | INFO | Train Epoch: 1 [106971136/128008192 (84%)] Data (t): 0.361 Batch (t): 5.633, 2903.55/s, 181.472/s/gpu LR: 0.000578 Logit Scale: 80.593 Contrastive_loss: 1.5514 (1.7260) Loss: 1.5514 (1.7260)
2025-05-07,09:52:39 | WARNING | Handling webdataset error (OSError('image file is truncated (67 bytes not processed)')). Ignoring.
2025-05-07,09:55:33 | INFO | Train Epoch: 1 [109068288/128008192 (85%)] Data (t): 0.354 Batch (t): 5.668, 2733.20/s, 170.825/s/gpu LR: 0.000572 Logit Scale: 80.807 Contrastive_loss: 1.1183 (1.7145) Loss: 1.1183 (1.7145)
2025-05-07,10:07:32 | INFO | Train Epoch: 1 [111165440/128008192 (87%)] Data (t): 0.330 Batch (t): 5.620, 2978.04/s, 186.128/s/gpu LR: 0.000565 Logit Scale: 81.061 Contrastive_loss: 1.5260 (1.7110) Loss: 1.5260 (1.7110)
2025-05-07,10:14:13 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
2025-05-07,10:15:22 | WARNING | Handling webdataset error (OSError('image file is truncated (21 bytes not processed)')). Ignoring.
2025-05-07,10:19:31 | INFO | Train Epoch: 1 [113262592/128008192 (88%)] Data (t): 0.364 Batch (t): 5.617, 2935.77/s, 183.486/s/gpu LR: 0.000559 Logit Scale: 81.182 Contrastive_loss: 1.5539 (1.7082) Loss: 1.5539 (1.7082)
2025-05-07,10:31:30 | INFO | Train Epoch: 1 [115359744/128008192 (90%)] Data (t): 0.352 Batch (t): 5.610, 2929.15/s, 183.072/s/gpu LR: 0.000552 Logit Scale: 81.423 Contrastive_loss: 1.6106 (1.7064) Loss: 1.6106 (1.7064)
2025-05-07,10:40:18 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
2025-05-07,10:43:27 | INFO | Train Epoch: 1 [117456896/128008192 (92%)] Data (t): 0.362 Batch (t): 5.604, 2805.89/s, 175.368/s/gpu LR: 0.000546 Logit Scale: 81.447 Contrastive_loss: 1.4178 (1.7014) Loss: 1.4178 (1.7014)
2025-05-07,10:53:40 | WARNING | Handling webdataset error (OSError('image file is truncated (38 bytes not processed)')). Ignoring.
2025-05-07,10:55:28 | INFO | Train Epoch: 1 [119554048/128008192 (93%)] Data (t): 0.367 Batch (t): 5.633, 2775.17/s, 173.448/s/gpu LR: 0.000539 Logit Scale: 81.697 Contrastive_loss: 1.4122 (1.6964) Loss: 1.4122 (1.6964)
2025-05-07,11:05:08 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
2025-05-07,11:07:22 | INFO | Train Epoch: 1 [121651200/128008192 (95%)] Data (t): 0.326 Batch (t): 5.583, 2808.75/s, 175.547/s/gpu LR: 0.000533 Logit Scale: 81.912 Contrastive_loss: 1.5345 (1.6937) Loss: 1.5345 (1.6937)
2025-05-07,11:13:08 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
2025-05-07,11:14:21 | WARNING | Handling webdataset error (OSError('image file is truncated (89 bytes not processed)')). Ignoring.
2025-05-07,11:19:20 | INFO | Train Epoch: 1 [123748352/128008192 (97%)] Data (t): 0.326 Batch (t): 5.606, 2967.95/s, 185.497/s/gpu LR: 0.000526 Logit Scale: 82.072 Contrastive_loss: 1.3680 (1.6882) Loss: 1.3680 (1.6882)
2025-05-07,11:31:35 | INFO | Train Epoch: 1 [125845504/128008192 (98%)] Data (t): 0.349 Batch (t): 5.741, 2755.96/s, 172.247/s/gpu LR: 0.000520 Logit Scale: 82.376 Contrastive_loss: 1.5748 (1.6864) Loss: 1.5748 (1.6864)
2025-05-07,11:39:37 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
2025-05-07,11:43:37 | INFO | Train Epoch: 1 [127942656/128008192 (100%)] Data (t): 0.371 Batch (t): 5.640, 2894.23/s, 180.889/s/gpu LR: 0.000513 Logit Scale: 82.542 Contrastive_loss: 1.5707 (1.6845) Loss: 1.5707 (1.6845)
2025-05-07,11:43:59 | INFO | Train Epoch: 1 [128008192/128008192 (100%)] Data (t): 0.371 Batch (t): 5.480, 3091.16/s, 193.197/s/gpu LR: 0.000513 Logit Scale: 82.566 Contrastive_loss: 1.0490 (1.6744) Loss: 1.0490 (1.6744)
2025-05-07,11:44:06 | INFO | Start epoch 2
2025-05-07,11:44:18 | INFO | Train Epoch: 2 [    16384/128008192 (0%)] Data (t): 7.758 Batch (t): 12.105, 1353.44/s, 84.5901/s/gpu LR: 0.000513 Logit Scale: 82.568 Contrastive_loss: 1.3768 (1.3768) Loss: 1.3768 (1.3768)
2025-05-07,11:48:43 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
2025-05-07,11:56:13 | INFO | Train Epoch: 2 [  2113536/128008192 (2%)] Data (t): 0.423 Batch (t): 5.579, 2888.97/s, 180.561/s/gpu LR: 0.000506 Logit Scale: 82.792 Contrastive_loss: 1.4711 (1.4240) Loss: 1.4711 (1.4240)
2025-05-07,12:02:55 | WARNING | Handling webdataset error (OSError('image file is truncated (101 bytes not processed)')). Ignoring.
2025-05-07,12:08:10 | INFO | Train Epoch: 2 [  4210688/128008192 (3%)] Data (t): 0.357 Batch (t): 5.607, 2966.83/s, 185.427/s/gpu LR: 0.000500 Logit Scale: 82.939 Contrastive_loss: 0.88065 (1.2429) Loss: 0.88065 (1.2429)
2025-05-07,12:18:10 | WARNING | Handling webdataset error (OSError('image file is truncated (46 bytes not processed)')). Ignoring.
2025-05-07,12:20:07 | INFO | Train Epoch: 2 [  6307840/128008192 (5%)] Data (t): 0.368 Batch (t): 5.601, 2912.71/s, 182.044/s/gpu LR: 0.000493 Logit Scale: 83.052 Contrastive_loss: 1.4302 (1.2897) Loss: 1.4302 (1.2897)
2025-05-07,12:25:37 | WARNING | Handling webdataset error (OSError('image file is truncated (3 bytes not processed)')). Ignoring.
2025-05-07,12:26:02 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
2025-05-07,12:31:15 | WARNING | Handling webdataset error (OSError('image file is truncated (61 bytes not processed)')). Ignoring.
2025-05-07,12:32:13 | INFO | Train Epoch: 2 [  8404992/128008192 (7%)] Data (t): 0.369 Batch (t): 5.672, 2945.98/s, 184.124/s/gpu LR: 0.000487 Logit Scale: 83.280 Contrastive_loss: 1.3575 (1.3033) Loss: 1.3575 (1.3033)
2025-05-07,12:44:09 | INFO | Train Epoch: 2 [ 10502144/128008192 (8%)] Data (t): 0.379 Batch (t): 5.592, 2904.28/s, 181.517/s/gpu LR: 0.000480 Logit Scale: 83.503 Contrastive_loss: 1.3899 (1.3177) Loss: 1.3899 (1.3177)
2025-05-07,12:56:11 | INFO | Train Epoch: 2 [ 12599296/128008192 (10%)] Data (t): 0.357 Batch (t): 5.637, 2795.81/s, 174.738/s/gpu LR: 0.000474 Logit Scale: 83.668 Contrastive_loss: 1.4585 (1.3378) Loss: 1.4585 (1.3378)
2025-05-07,13:08:08 | INFO | Train Epoch: 2 [ 14696448/128008192 (11%)] Data (t): 0.322 Batch (t): 5.603, 2907.26/s, 181.704/s/gpu LR: 0.000467 Logit Scale: 83.892 Contrastive_loss: 1.3279 (1.3366) Loss: 1.3279 (1.3366)
2025-05-07,13:20:08 | INFO | Train Epoch: 2 [ 16793600/128008192 (13%)] Data (t): 0.374 Batch (t): 5.625, 2889.58/s, 180.599/s/gpu LR: 0.000461 Logit Scale: 84.020 Contrastive_loss: 1.3775 (1.3411) Loss: 1.3775 (1.3411)
2025-05-07,13:24:10 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
2025-05-07,13:27:24 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
2025-05-07,13:32:07 | INFO | Train Epoch: 2 [ 18890752/128008192 (15%)] Data (t): 0.355 Batch (t): 5.615, 2903.24/s, 181.453/s/gpu LR: 0.000454 Logit Scale: 84.226 Contrastive_loss: 1.2731 (1.3343) Loss: 1.2731 (1.3343)
2025-05-07,13:33:13 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
2025-05-07,13:44:01 | INFO | Train Epoch: 2 [ 20987904/128008192 (16%)] Data (t): 0.333 Batch (t): 5.582, 2908.03/s, 181.752/s/gpu LR: 0.000447 Logit Scale: 84.394 Contrastive_loss: 1.4364 (1.3436) Loss: 1.4364 (1.3436)
2025-05-07,13:50:37 | WARNING | Handling webdataset error (OSError('image file is truncated (47 bytes not processed)')). Ignoring.
2025-05-07,13:56:07 | INFO | Train Epoch: 2 [ 23085056/128008192 (18%)] Data (t): 0.366 Batch (t): 5.668, 2921.16/s, 182.573/s/gpu LR: 0.000441 Logit Scale: 84.633 Contrastive_loss: 1.0712 (1.3209) Loss: 1.0712 (1.3209)
2025-05-07,13:58:37 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
2025-05-07,14:08:01 | INFO | Train Epoch: 2 [ 25182208/128008192 (20%)] Data (t): 0.318 Batch (t): 5.581, 2964.23/s, 185.265/s/gpu LR: 0.000435 Logit Scale: 84.808 Contrastive_loss: 1.2264 (1.3136) Loss: 1.2264 (1.3136)
2025-05-07,14:20:00 | INFO | Train Epoch: 2 [ 27279360/128008192 (21%)] Data (t): 0.330 Batch (t): 5.618, 2962.89/s, 185.180/s/gpu LR: 0.000428 Logit Scale: 84.964 Contrastive_loss: 0.99704 (1.2910) Loss: 0.99704 (1.2910)
2025-05-07,14:31:56 | INFO | Train Epoch: 2 [ 29376512/128008192 (23%)] Data (t): 0.383 Batch (t): 5.590, 2926.31/s, 182.894/s/gpu LR: 0.000422 Logit Scale: 85.129 Contrastive_loss: 1.3815 (1.2970) Loss: 1.3815 (1.2970)
2025-05-07,14:43:52 | INFO | Train Epoch: 2 [ 31473664/128008192 (25%)] Data (t): 0.377 Batch (t): 5.593, 2909.77/s, 181.860/s/gpu LR: 0.000415 Logit Scale: 85.211 Contrastive_loss: 1.1837 (1.2900) Loss: 1.1837 (1.2900)
2025-05-07,14:55:52 | INFO | Train Epoch: 2 [ 33570816/128008192 (26%)] Data (t): 0.379 Batch (t): 5.626, 3002.02/s, 187.626/s/gpu LR: 0.000409 Logit Scale: 85.379 Contrastive_loss: 1.0019 (1.2730) Loss: 1.0019 (1.2730)
2025-05-07,15:07:50 | INFO | Train Epoch: 2 [ 35667968/128008192 (28%)] Data (t): 0.376 Batch (t): 5.612, 2921.54/s, 182.597/s/gpu LR: 0.000402 Logit Scale: 85.569 Contrastive_loss: 0.97724 (1.2566) Loss: 0.97724 (1.2566)
2025-05-07,15:19:48 | INFO | Train Epoch: 2 [ 37765120/128008192 (30%)] Data (t): 0.375 Batch (t): 5.610, 2921.18/s, 182.574/s/gpu LR: 0.000396 Logit Scale: 85.819 Contrastive_loss: 1.1814 (1.2526) Loss: 1.1814 (1.2526)
2025-05-07,15:31:48 | INFO | Train Epoch: 2 [ 39862272/128008192 (31%)] Data (t): 0.375 Batch (t): 5.628, 2982.89/s, 186.430/s/gpu LR: 0.000389 Logit Scale: 85.933 Contrastive_loss: 1.3243 (1.2562) Loss: 1.3243 (1.2562)
2025-05-07,15:33:32 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
2025-05-07,15:36:25 | WARNING | Handling webdataset error (OSError('image file is truncated (76 bytes not processed)')). Ignoring.
2025-05-07,15:37:34 | WARNING | Handling webdataset error (OSError('image file is truncated (230 bytes not processed)')). Ignoring.
2025-05-07,15:44:05 | INFO | Train Epoch: 2 [ 41959424/128008192 (33%)] Data (t): 0.356 Batch (t): 5.753, 2908.45/s, 181.778/s/gpu LR: 0.000383 Logit Scale: 86.170 Contrastive_loss: 1.3186 (1.2592) Loss: 1.3186 (1.2592)
2025-05-07,15:46:58 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
2025-05-07,15:56:15 | INFO | Train Epoch: 2 [ 44056576/128008192 (34%)] Data (t): 0.384 Batch (t): 5.708, 3440.17/s, 215.011/s/gpu LR: 0.000377 Logit Scale: 86.381 Contrastive_loss: 1.3185 (1.2619) Loss: 1.3185 (1.2619)
2025-05-07,16:08:09 | INFO | Train Epoch: 2 [ 46153728/128008192 (36%)] Data (t): 0.508 Batch (t): 5.578, 2839.18/s, 177.449/s/gpu LR: 0.000370 Logit Scale: 86.581 Contrastive_loss: 0.91212 (1.2467) Loss: 0.91212 (1.2467)
2025-05-07,16:09:59 | WARNING | Handling webdataset error (OSError('image file is truncated (19 bytes not processed)')). Ignoring.
2025-05-07,16:12:05 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
2025-05-07,16:20:11 | INFO | Train Epoch: 2 [ 48250880/128008192 (38%)] Data (t): 0.512 Batch (t): 5.639, 2933.58/s, 183.349/s/gpu LR: 0.000364 Logit Scale: 86.738 Contrastive_loss: 1.2363 (1.2462) Loss: 1.2363 (1.2462)
2025-05-07,16:32:12 | INFO | Train Epoch: 2 [ 50348032/128008192 (39%)] Data (t): 0.531 Batch (t): 5.634, 2902.81/s, 181.426/s/gpu LR: 0.000358 Logit Scale: 86.891 Contrastive_loss: 1.0084 (1.2367) Loss: 1.0084 (1.2367)
2025-05-07,16:35:12 | WARNING | Handling webdataset error (OSError('image file is truncated (3 bytes not processed)')). Ignoring.
2025-05-07,16:35:23 | WARNING | Handling webdataset error (OSError('image file is truncated (43 bytes not processed)')). Ignoring.
2025-05-07,16:44:16 | INFO | Train Epoch: 2 [ 52445184/128008192 (41%)] Data (t): 0.525 Batch (t): 5.657, 2894.95/s, 180.934/s/gpu LR: 0.000352 Logit Scale: 87.060 Contrastive_loss: 1.3024 (1.2393) Loss: 1.3024 (1.2393)
2025-05-07,16:56:21 | INFO | Train Epoch: 2 [ 54542336/128008192 (43%)] Data (t): 0.457 Batch (t): 5.659, 2749.78/s, 171.861/s/gpu LR: 0.000345 Logit Scale: 87.341 Contrastive_loss: 1.2838 (1.2409) Loss: 1.2838 (1.2409)
2025-05-07,17:08:17 | INFO | Train Epoch: 2 [ 56639488/128008192 (44%)] Data (t): 0.314 Batch (t): 5.598, 2908.17/s, 181.760/s/gpu LR: 0.000339 Logit Scale: 87.473 Contrastive_loss: 0.94573 (1.2304) Loss: 0.94573 (1.2304)
2025-05-07,17:20:18 | INFO | Train Epoch: 2 [ 58736640/128008192 (46%)] Data (t): 0.379 Batch (t): 5.629, 2919.91/s, 182.494/s/gpu LR: 0.000333 Logit Scale: 87.619 Contrastive_loss: 1.1648 (1.2281) Loss: 1.1648 (1.2281)
2025-05-07,17:32:16 | INFO | Train Epoch: 2 [ 60833792/128008192 (48%)] Data (t): 0.363 Batch (t): 5.615, 2905.06/s, 181.566/s/gpu LR: 0.000327 Logit Scale: 87.872 Contrastive_loss: 0.86235 (1.2159) Loss: 0.86235 (1.2159)
2025-05-07,17:37:39 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
2025-05-07,17:44:17 | INFO | Train Epoch: 2 [ 62930944/128008192 (49%)] Data (t): 0.347 Batch (t): 5.630, 2940.91/s, 183.807/s/gpu LR: 0.000321 Logit Scale: 88.094 Contrastive_loss: 1.2874 (1.2182) Loss: 1.2874 (1.2182)
2025-05-07,17:48:19 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
2025-05-07,17:56:13 | INFO | Train Epoch: 2 [ 65028096/128008192 (51%)] Data (t): 0.381 Batch (t): 5.597, 2952.90/s, 184.556/s/gpu LR: 0.000315 Logit Scale: 88.220 Contrastive_loss: 1.2260 (1.2185) Loss: 1.2260 (1.2185)
2025-05-07,18:00:34 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
2025-05-07,18:02:53 | WARNING | Handling webdataset error (OSError('image file is truncated (22 bytes not processed)')). Ignoring.
2025-05-07,18:08:09 | INFO | Train Epoch: 2 [ 67125248/128008192 (52%)] Data (t): 0.375 Batch (t): 5.594, 2979.01/s, 186.188/s/gpu LR: 0.000309 Logit Scale: 88.450 Contrastive_loss: 1.1228 (1.2156) Loss: 1.1228 (1.2156)
2025-05-07,18:20:12 | INFO | Train Epoch: 2 [ 69222400/128008192 (54%)] Data (t): 0.368 Batch (t): 5.647, 2923.06/s, 182.691/s/gpu LR: 0.000303 Logit Scale: 88.607 Contrastive_loss: 0.87884 (1.2057) Loss: 0.87884 (1.2057)
2025-05-07,18:32:13 | INFO | Train Epoch: 2 [ 71319552/128008192 (56%)] Data (t): 0.376 Batch (t): 5.630, 2892.54/s, 180.784/s/gpu LR: 0.000297 Logit Scale: 88.736 Contrastive_loss: 0.94438 (1.1982) Loss: 0.94438 (1.1982)
2025-05-07,18:44:10 | INFO | Train Epoch: 2 [ 73416704/128008192 (57%)] Data (t): 0.378 Batch (t): 5.605, 2859.94/s, 178.746/s/gpu LR: 0.000291 Logit Scale: 88.990 Contrastive_loss: 1.1171 (1.1959) Loss: 1.1171 (1.1959)
2025-05-07,18:50:47 | WARNING | Handling webdataset error (OSError('image file is truncated (34 bytes not processed)')). Ignoring.
2025-05-07,18:56:17 | INFO | Train Epoch: 2 [ 75513856/128008192 (59%)] Data (t): 0.374 Batch (t): 5.679, 2964.50/s, 185.281/s/gpu LR: 0.000285 Logit Scale: 89.219 Contrastive_loss: 1.1197 (1.1939) Loss: 1.1197 (1.1939)
2025-05-07,19:02:16 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
2025-05-07,19:08:13 | INFO | Train Epoch: 2 [ 77611008/128008192 (61%)] Data (t): 0.334 Batch (t): 5.595, 2902.16/s, 181.385/s/gpu LR: 0.000279 Logit Scale: 89.378 Contrastive_loss: 1.2041 (1.1941) Loss: 1.2041 (1.1941)
2025-05-07,19:08:16 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
2025-05-07,19:13:15 | WARNING | Handling webdataset error (OSError('image file is truncated (152 bytes not processed)')). Ignoring.
2025-05-07,19:14:36 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
2025-05-07,19:16:15 | WARNING | Handling webdataset error (OSError('image file is truncated (16 bytes not processed)')). Ignoring.
2025-05-07,19:20:11 | INFO | Train Epoch: 2 [ 79708160/128008192 (62%)] Data (t): 0.337 Batch (t): 5.608, 2974.28/s, 185.893/s/gpu LR: 0.000273 Logit Scale: 89.629 Contrastive_loss: 1.1077 (1.1919) Loss: 1.1077 (1.1919)
2025-05-07,19:32:10 | INFO | Train Epoch: 2 [ 81805312/128008192 (64%)] Data (t): 0.379 Batch (t): 5.614, 2874.74/s, 179.671/s/gpu LR: 0.000267 Logit Scale: 89.834 Contrastive_loss: 1.1738 (1.1915) Loss: 1.1738 (1.1915)
2025-05-07,19:32:10 | WARNING | Handling webdataset error (OSError('image file is truncated (60 bytes not processed)')). Ignoring.
2025-05-07,19:44:08 | INFO | Train Epoch: 2 [ 83902464/128008192 (66%)] Data (t): 0.388 Batch (t): 5.614, 2983.10/s, 186.444/s/gpu LR: 0.000261 Logit Scale: 89.958 Contrastive_loss: 1.2044 (1.1918) Loss: 1.2044 (1.1918)
2025-05-07,19:56:08 | INFO | Train Epoch: 2 [ 85999616/128008192 (67%)] Data (t): 0.354 Batch (t): 5.620, 2946.98/s, 184.186/s/gpu LR: 0.000256 Logit Scale: 90.127 Contrastive_loss: 1.0898 (1.1894) Loss: 1.0898 (1.1894)
2025-05-07,20:03:09 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
2025-05-07,20:08:07 | INFO | Train Epoch: 2 [ 88096768/128008192 (69%)] Data (t): 0.352 Batch (t): 5.620, 2684.36/s, 167.772/s/gpu LR: 0.000250 Logit Scale: 90.309 Contrastive_loss: 0.77449 (1.1797) Loss: 0.77449 (1.1797)
2025-05-07,20:20:08 | INFO | Train Epoch: 2 [ 90193920/128008192 (70%)] Data (t): 0.377 Batch (t): 5.631, 2986.93/s, 186.683/s/gpu LR: 0.000244 Logit Scale: 90.521 Contrastive_loss: 0.87286 (1.1727) Loss: 0.87286 (1.1727)
2025-05-07,20:23:59 | WARNING | Handling webdataset error (OSError('image file is truncated (33 bytes not processed)')). Ignoring.
2025-05-07,20:24:52 | WARNING | Handling webdataset error (OSError('image file is truncated (16 bytes not processed)')). Ignoring.
2025-05-07,20:32:03 | INFO | Train Epoch: 2 [ 92291072/128008192 (72%)] Data (t): 0.374 Batch (t): 5.585, 2920.65/s, 182.540/s/gpu LR: 0.000239 Logit Scale: 90.756 Contrastive_loss: 1.2122 (1.1736) Loss: 1.2122 (1.1736)
2025-05-07,20:44:04 | INFO | Train Epoch: 2 [ 94388224/128008192 (74%)] Data (t): 0.373 Batch (t): 5.636, 2901.20/s, 181.325/s/gpu LR: 0.000233 Logit Scale: 90.975 Contrastive_loss: 1.1097 (1.1722) Loss: 1.1097 (1.1722)
2025-05-07,20:56:04 | INFO | Train Epoch: 2 [ 96485376/128008192 (75%)] Data (t): 0.378 Batch (t): 5.627, 2937.51/s, 183.594/s/gpu LR: 0.000228 Logit Scale: 91.126 Contrastive_loss: 1.0784 (1.1702) Loss: 1.0784 (1.1702)
2025-05-07,21:06:33 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
2025-05-07,21:07:59 | INFO | Train Epoch: 2 [ 98582528/128008192 (77%)] Data (t): 0.345 Batch (t): 5.583, 2934.45/s, 183.403/s/gpu LR: 0.000222 Logit Scale: 91.303 Contrastive_loss: 1.0321 (1.1674) Loss: 1.0321 (1.1674)
2025-05-07,21:19:56 | INFO | Train Epoch: 2 [100679680/128008192 (79%)] Data (t): 0.348 Batch (t): 5.601, 2958.14/s, 184.883/s/gpu LR: 0.000217 Logit Scale: 91.554 Contrastive_loss: 0.77563 (1.1594) Loss: 0.77563 (1.1594)
2025-05-07,21:32:10 | INFO | Train Epoch: 2 [102776832/128008192 (80%)] Data (t): 0.362 Batch (t): 5.733, 2892.05/s, 180.753/s/gpu LR: 0.000211 Logit Scale: 91.747 Contrastive_loss: 0.93499 (1.1549) Loss: 0.93499 (1.1549)
2025-05-07,21:44:08 | INFO | Train Epoch: 2 [104873984/128008192 (82%)] Data (t): 0.370 Batch (t): 5.614, 2937.42/s, 183.589/s/gpu LR: 0.000206 Logit Scale: 92.009 Contrastive_loss: 1.0118 (1.1521) Loss: 1.0118 (1.1521)
2025-05-07,21:56:07 | INFO | Train Epoch: 2 [106971136/128008192 (84%)] Data (t): 0.361 Batch (t): 5.618, 2921.42/s, 182.589/s/gpu LR: 0.000201 Logit Scale: 92.195 Contrastive_loss: 1.0296 (1.1497) Loss: 1.0296 (1.1497)
2025-05-07,21:59:11 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
2025-05-07,22:08:11 | INFO | Train Epoch: 2 [109068288/128008192 (85%)] Data (t): 0.716 Batch (t): 5.650, 2780.27/s, 173.767/s/gpu LR: 0.000196 Logit Scale: 92.340 Contrastive_loss: 1.0085 (1.1470) Loss: 1.0085 (1.1470)
2025-05-07,22:12:45 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
2025-05-07,22:14:46 | WARNING | Handling webdataset error (OSError('image file is truncated (29 bytes not processed)')). Ignoring.
2025-05-07,22:17:52 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
2025-05-07,22:18:34 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
2025-05-07,22:20:17 | INFO | Train Epoch: 2 [111165440/128008192 (87%)] Data (t): 0.384 Batch (t): 5.678, 2927.97/s, 182.998/s/gpu LR: 0.000190 Logit Scale: 92.503 Contrastive_loss: 1.1436 (1.1470) Loss: 1.1436 (1.1470)
2025-05-07,22:26:57 | WARNING | Handling webdataset error (OSError('image file is truncated (67 bytes not processed)')). Ignoring.
2025-05-07,22:29:42 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
2025-05-07,22:29:43 | WARNING | Handling webdataset error (OSError('image file is truncated (19 bytes not processed)')). Ignoring.
2025-05-07,22:32:28 | INFO | Train Epoch: 2 [113262592/128008192 (88%)] Data (t): 0.359 Batch (t): 5.705, 2995.11/s, 187.194/s/gpu LR: 0.000185 Logit Scale: 92.666 Contrastive_loss: 1.0610 (1.1454) Loss: 1.0610 (1.1454)
2025-05-07,22:34:32 | WARNING | Handling webdataset error (OSError('image file is truncated (93 bytes not processed)')). Ignoring.
2025-05-07,22:44:24 | INFO | Train Epoch: 2 [115359744/128008192 (90%)] Data (t): 0.315 Batch (t): 5.597, 2935.38/s, 183.461/s/gpu LR: 0.000180 Logit Scale: 92.865 Contrastive_loss: 1.0644 (1.1440) Loss: 1.0644 (1.1440)
2025-05-07,22:49:47 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
2025-05-07,22:56:24 | INFO | Train Epoch: 2 [117456896/128008192 (92%)] Data (t): 0.325 Batch (t): 5.622, 2952.99/s, 184.562/s/gpu LR: 0.000175 Logit Scale: 93.075 Contrastive_loss: 1.0572 (1.1424) Loss: 1.0572 (1.1424)
2025-05-07,23:08:15 | INFO | Train Epoch: 2 [119554048/128008192 (93%)] Data (t): 0.327 Batch (t): 5.559, 2920.54/s, 182.534/s/gpu LR: 0.000170 Logit Scale: 93.244 Contrastive_loss: 0.96107 (1.1393) Loss: 0.96107 (1.1393)
2025-05-07,23:20:12 | INFO | Train Epoch: 2 [121651200/128008192 (95%)] Data (t): 0.368 Batch (t): 5.598, 2942.47/s, 183.904/s/gpu LR: 0.000165 Logit Scale: 93.405 Contrastive_loss: 0.98694 (1.1367) Loss: 0.98694 (1.1367)
2025-05-07,23:28:50 | WARNING | Handling webdataset error (OSError('image file is truncated (96 bytes not processed)')). Ignoring.
2025-05-07,23:32:20 | INFO | Train Epoch: 2 [123748352/128008192 (97%)] Data (t): 0.364 Batch (t): 5.687, 2714.48/s, 169.655/s/gpu LR: 0.000161 Logit Scale: 93.621 Contrastive_loss: 1.1067 (1.1362) Loss: 1.1067 (1.1362)
2025-05-07,23:44:21 | INFO | Train Epoch: 2 [125845504/128008192 (98%)] Data (t): 0.653 Batch (t): 5.636, 2957.64/s, 184.852/s/gpu LR: 0.000156 Logit Scale: 93.880 Contrastive_loss: 1.0097 (1.1342) Loss: 1.0097 (1.1342)
2025-05-07,23:49:27 | WARNING | Handling webdataset error (OSError('image file is truncated (29 bytes not processed)')). Ignoring.
2025-05-07,23:56:28 | INFO | Train Epoch: 2 [127942656/128008192 (100%)] Data (t): 0.459 Batch (t): 5.680, 2983.01/s, 186.438/s/gpu LR: 0.000151 Logit Scale: 94.001 Contrastive_loss: 0.99418 (1.1319) Loss: 0.99418 (1.1319)
2025-05-07,23:56:50 | INFO | Train Epoch: 2 [128008192/128008192 (100%)] Data (t): 0.485 Batch (t): 5.502, 3070.17/s, 191.886/s/gpu LR: 0.000151 Logit Scale: 93.998 Contrastive_loss: 0.89453 (1.1281) Loss: 0.89453 (1.1281)
2025-05-07,23:56:58 | INFO | Start epoch 3
2025-05-07,23:57:10 | INFO | Train Epoch: 3 [    16384/128008192 (0%)] Data (t): 7.491 Batch (t): 11.841, 1383.71/s, 86.4821/s/gpu LR: 0.000151 Logit Scale: 93.998 Contrastive_loss: 0.92352 (0.92352) Loss: 0.92352 (0.92352)
2025-05-08,00:06:30 | WARNING | Handling webdataset error (OSError('image file is truncated (50 bytes not processed)')). Ignoring.
2025-05-08,00:09:23 | INFO | Train Epoch: 3 [  2113536/128008192 (2%)] Data (t): 0.586 Batch (t): 5.725, 2851.88/s, 178.242/s/gpu LR: 0.000146 Logit Scale: 94.254 Contrastive_loss: 0.96123 (0.94238) Loss: 0.96123 (0.94238)
2025-05-08,00:21:36 | INFO | Train Epoch: 3 [  4210688/128008192 (3%)] Data (t): 0.556 Batch (t): 5.726, 2974.96/s, 185.935/s/gpu LR: 0.000142 Logit Scale: 94.436 Contrastive_loss: 0.91698 (0.93391) Loss: 0.91698 (0.93391)
2025-05-08,00:33:42 | INFO | Train Epoch: 3 [  6307840/128008192 (5%)] Data (t): 0.520 Batch (t): 5.674, 2751.83/s, 171.990/s/gpu LR: 0.000137 Logit Scale: 94.659 Contrastive_loss: 0.85714 (0.91472) Loss: 0.85714 (0.91472)
2025-05-08,00:44:23 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
2025-05-08,00:45:43 | INFO | Train Epoch: 3 [  8404992/128008192 (7%)] Data (t): 0.368 Batch (t): 5.634, 2935.48/s, 183.468/s/gpu LR: 0.000133 Logit Scale: 94.866 Contrastive_loss: 0.93602 (0.91898) Loss: 0.93602 (0.91898)
2025-05-08,00:57:45 | INFO | Train Epoch: 3 [ 10502144/128008192 (8%)] Data (t): 0.353 Batch (t): 5.637, 2914.81/s, 182.175/s/gpu LR: 0.000128 Logit Scale: 95.026 Contrastive_loss: 0.87356 (0.91141) Loss: 0.87356 (0.91141)
2025-05-08,01:03:32 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
2025-05-08,01:09:42 | INFO | Train Epoch: 3 [ 12599296/128008192 (10%)] Data (t): 0.366 Batch (t): 5.608, 2941.16/s, 183.823/s/gpu LR: 0.000124 Logit Scale: 95.184 Contrastive_loss: 0.72369 (0.88459) Loss: 0.72369 (0.88459)
2025-05-08,01:18:08 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
2025-05-08,01:21:44 | INFO | Train Epoch: 3 [ 14696448/128008192 (11%)] Data (t): 0.370 Batch (t): 5.640, 3017.60/s, 188.600/s/gpu LR: 0.000120 Logit Scale: 95.390 Contrastive_loss: 0.84045 (0.87907) Loss: 0.84045 (0.87907)
2025-05-08,01:33:44 | INFO | Train Epoch: 3 [ 16793600/128008192 (13%)] Data (t): 0.421 Batch (t): 5.624, 2887.76/s, 180.485/s/gpu LR: 0.000116 Logit Scale: 95.556 Contrastive_loss: 0.81983 (0.87249) Loss: 0.81983 (0.87249)
2025-05-08,01:43:07 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
2025-05-08,01:45:45 | INFO | Train Epoch: 3 [ 18890752/128008192 (15%)] Data (t): 0.368 Batch (t): 5.632, 2882.59/s, 180.162/s/gpu LR: 0.000111 Logit Scale: 95.749 Contrastive_loss: 0.88202 (0.87344) Loss: 0.88202 (0.87344)
2025-05-08,01:57:40 | INFO | Train Epoch: 3 [ 20987904/128008192 (16%)] Data (t): 0.374 Batch (t): 5.588, 2866.69/s, 179.168/s/gpu LR: 0.000107 Logit Scale: 95.921 Contrastive_loss: 0.90958 (0.87673) Loss: 0.90958 (0.87673)
2025-05-08,02:09:39 | INFO | Train Epoch: 3 [ 23085056/128008192 (18%)] Data (t): 0.377 Batch (t): 5.617, 2921.81/s, 182.613/s/gpu LR: 0.000103 Logit Scale: 96.101 Contrastive_loss: 0.83029 (0.87286) Loss: 0.83029 (0.87286)
2025-05-08,02:21:39 | INFO | Train Epoch: 3 [ 25182208/128008192 (20%)] Data (t): 0.376 Batch (t): 5.625, 2678.30/s, 167.394/s/gpu LR: 0.000099 Logit Scale: 96.281 Contrastive_loss: 0.71671 (0.86085) Loss: 0.71671 (0.86085)
2025-05-08,02:27:29 | WARNING | Handling webdataset error (OSError('image file is truncated (10 bytes not processed)')). Ignoring.
2025-05-08,02:33:41 | INFO | Train Epoch: 3 [ 27279360/128008192 (21%)] Data (t): 0.372 Batch (t): 5.641, 2993.89/s, 187.118/s/gpu LR: 0.000095 Logit Scale: 96.435 Contrastive_loss: 0.93564 (0.86619) Loss: 0.93564 (0.86619)
2025-05-08,02:43:28 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
2025-05-08,02:45:37 | INFO | Train Epoch: 3 [ 29376512/128008192 (23%)] Data (t): 0.355 Batch (t): 5.593, 2906.38/s, 181.649/s/gpu LR: 0.000092 Logit Scale: 96.582 Contrastive_loss: 0.82735 (0.86360) Loss: 0.82735 (0.86360)
2025-05-08,02:50:07 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
2025-05-08,02:56:55 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
2025-05-08,02:57:32 | INFO | Train Epoch: 3 [ 31473664/128008192 (25%)] Data (t): 0.364 Batch (t): 5.582, 2941.26/s, 183.829/s/gpu LR: 0.000088 Logit Scale: 96.733 Contrastive_loss: 0.73455 (0.85554) Loss: 0.73455 (0.85554)
2025-05-08,02:58:01 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
2025-05-08,03:09:28 | INFO | Train Epoch: 3 [ 33570816/128008192 (26%)] Data (t): 0.375 Batch (t): 5.599, 2937.90/s, 183.619/s/gpu LR: 0.000084 Logit Scale: 96.898 Contrastive_loss: 0.86666 (0.85619) Loss: 0.86666 (0.85619)
2025-05-08,03:14:41 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
2025-05-08,03:21:27 | INFO | Train Epoch: 3 [ 35667968/128008192 (28%)] Data (t): 0.370 Batch (t): 5.615, 2926.80/s, 182.925/s/gpu LR: 0.000081 Logit Scale: 97.086 Contrastive_loss: 0.87349 (0.85715) Loss: 0.87349 (0.85715)
2025-05-08,03:33:25 | INFO | Train Epoch: 3 [ 37765120/128008192 (30%)] Data (t): 0.383 Batch (t): 5.606, 2994.55/s, 187.160/s/gpu LR: 0.000077 Logit Scale: 97.266 Contrastive_loss: 0.80121 (0.85421) Loss: 0.80121 (0.85421)
2025-05-08,03:45:21 | INFO | Train Epoch: 3 [ 39862272/128008192 (31%)] Data (t): 0.363 Batch (t): 5.599, 2927.06/s, 182.941/s/gpu LR: 0.000074 Logit Scale: 97.356 Contrastive_loss: 0.88279 (0.85564) Loss: 0.88279 (0.85564)
2025-05-08,03:54:41 | WARNING | Handling webdataset error (OSError('image file is truncated (20 bytes not processed)')). Ignoring.
2025-05-08,03:57:21 | INFO | Train Epoch: 3 [ 41959424/128008192 (33%)] Data (t): 0.375 Batch (t): 5.622, 2925.52/s, 182.845/s/gpu LR: 0.000070 Logit Scale: 97.509 Contrastive_loss: 0.81097 (0.85351) Loss: 0.81097 (0.85351)
2025-05-08,03:58:59 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
2025-05-08,04:09:13 | INFO | Train Epoch: 3 [ 44056576/128008192 (34%)] Data (t): 0.377 Batch (t): 5.563, 2995.04/s, 187.190/s/gpu LR: 0.000067 Logit Scale: 97.659 Contrastive_loss: 0.82750 (0.85233) Loss: 0.82750 (0.85233)
2025-05-08,04:20:32 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
2025-05-08,04:20:37 | WARNING | Handling webdataset error (OSError('image file is truncated (48 bytes not processed)')). Ignoring.
2025-05-08,04:21:07 | INFO | Train Epoch: 3 [ 46153728/128008192 (36%)] Data (t): 0.379 Batch (t): 5.582, 2974.08/s, 185.880/s/gpu LR: 0.000064 Logit Scale: 97.809 Contrastive_loss: 0.79372 (0.84978) Loss: 0.79372 (0.84978)
2025-05-08,04:33:05 | INFO | Train Epoch: 3 [ 48250880/128008192 (38%)] Data (t): 0.370 Batch (t): 5.604, 2904.15/s, 181.509/s/gpu LR: 0.000061 Logit Scale: 97.937 Contrastive_loss: 0.65580 (0.84170) Loss: 0.65580 (0.84170)
2025-05-08,04:44:13 | WARNING | Handling webdataset error (OSError('image file is truncated (152 bytes not processed)')). Ignoring.
2025-05-08,04:45:00 | INFO | Train Epoch: 3 [ 50348032/128008192 (39%)] Data (t): 0.379 Batch (t): 5.588, 2952.38/s, 184.524/s/gpu LR: 0.000058 Logit Scale: 98.086 Contrastive_loss: 0.80200 (0.84011) Loss: 0.80200 (0.84011)
2025-05-08,04:54:56 | WARNING | Handling webdataset error (OSError('image file is truncated (22 bytes not processed)')). Ignoring.
2025-05-08,04:57:04 | INFO | Train Epoch: 3 [ 52445184/128008192 (41%)] Data (t): 0.363 Batch (t): 5.658, 2918.82/s, 182.426/s/gpu LR: 0.000055 Logit Scale: 98.196 Contrastive_loss: 0.76640 (0.83727) Loss: 0.76640 (0.83727)
2025-05-08,05:09:08 | INFO | Train Epoch: 3 [ 54542336/128008192 (43%)] Data (t): 0.372 Batch (t): 5.656, 2924.29/s, 182.768/s/gpu LR: 0.000052 Logit Scale: 98.306 Contrastive_loss: 0.82000 (0.83663) Loss: 0.82000 (0.83663)
2025-05-08,05:15:45 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
2025-05-08,05:21:05 | INFO | Train Epoch: 3 [ 56639488/128008192 (44%)] Data (t): 0.340 Batch (t): 5.600, 2996.36/s, 187.273/s/gpu LR: 0.000049 Logit Scale: 98.427 Contrastive_loss: 0.79141 (0.83502) Loss: 0.79141 (0.83502)
2025-05-08,05:33:07 | INFO | Train Epoch: 3 [ 58736640/128008192 (46%)] Data (t): 0.484 Batch (t): 5.642, 2680.17/s, 167.511/s/gpu LR: 0.000046 Logit Scale: 98.539 Contrastive_loss: 0.79817 (0.83375) Loss: 0.79817 (0.83375)
2025-05-08,05:33:32 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
2025-05-08,05:38:56 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
2025-05-08,05:45:09 | INFO | Train Epoch: 3 [ 60833792/128008192 (48%)] Data (t): 0.365 Batch (t): 5.640, 2900.16/s, 181.260/s/gpu LR: 0.000043 Logit Scale: 98.648 Contrastive_loss: 0.71987 (0.82995) Loss: 0.71987 (0.82995)
2025-05-08,05:57:19 | INFO | Train Epoch: 3 [ 62930944/128008192 (49%)] Data (t): 0.364 Batch (t): 5.698, 2765.56/s, 172.848/s/gpu LR: 0.000041 Logit Scale: 98.762 Contrastive_loss: 0.74987 (0.82737) Loss: 0.74987 (0.82737)
2025-05-08,06:09:18 | INFO | Train Epoch: 3 [ 65028096/128008192 (51%)] Data (t): 0.338 Batch (t): 5.618, 2880.07/s, 180.004/s/gpu LR: 0.000038 Logit Scale: 98.839 Contrastive_loss: 0.69782 (0.82332) Loss: 0.69782 (0.82332)
2025-05-08,06:13:48 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
2025-05-08,06:21:26 | INFO | Train Epoch: 3 [ 67125248/128008192 (52%)] Data (t): 0.351 Batch (t): 5.687, 2767.77/s, 172.985/s/gpu LR: 0.000036 Logit Scale: 98.950 Contrastive_loss: 0.71998 (0.82019) Loss: 0.71998 (0.82019)
2025-05-08,06:24:18 | WARNING | Handling webdataset error (OSError('image file is truncated (13 bytes not processed)')). Ignoring.
2025-05-08,06:33:24 | INFO | Train Epoch: 3 [ 69222400/128008192 (54%)] Data (t): 0.368 Batch (t): 5.615, 2892.94/s, 180.809/s/gpu LR: 0.000033 Logit Scale: 99.051 Contrastive_loss: 0.74257 (0.81791) Loss: 0.74257 (0.81791)
2025-05-08,06:36:35 | WARNING | Handling webdataset error (OSError('image file is truncated (123 bytes not processed)')). Ignoring.
2025-05-08,06:45:28 | INFO | Train Epoch: 3 [ 71319552/128008192 (56%)] Data (t): 0.765 Batch (t): 5.649, 2935.34/s, 183.459/s/gpu LR: 0.000031 Logit Scale: 99.162 Contrastive_loss: 0.69823 (0.81449) Loss: 0.69823 (0.81449)
2025-05-08,06:57:28 | INFO | Train Epoch: 3 [ 73416704/128008192 (57%)] Data (t): 0.371 Batch (t): 5.625, 2998.09/s, 187.381/s/gpu LR: 0.000029 Logit Scale: 99.260 Contrastive_loss: 0.75956 (0.81296) Loss: 0.75956 (0.81296)
2025-05-08,07:09:31 | INFO | Train Epoch: 3 [ 75513856/128008192 (59%)] Data (t): 0.359 Batch (t): 5.650, 2970.19/s, 185.637/s/gpu LR: 0.000027 Logit Scale: 99.340 Contrastive_loss: 0.69145 (0.80968) Loss: 0.69145 (0.80968)
2025-05-08,07:15:26 | WARNING | Handling webdataset error (OSError('image file is truncated (27 bytes not processed)')). Ignoring.
2025-05-08,07:21:37 | INFO | Train Epoch: 3 [ 77611008/128008192 (61%)] Data (t): 0.359 Batch (t): 5.671, 2932.14/s, 183.259/s/gpu LR: 0.000025 Logit Scale: 99.398 Contrastive_loss: 0.75933 (0.80835) Loss: 0.75933 (0.80835)
2025-05-08,07:30:40 | WARNING | Handling webdataset error (OSError('image file is truncated (74 bytes not processed)')). Ignoring.
2025-05-08,07:33:37 | INFO | Train Epoch: 3 [ 79708160/128008192 (62%)] Data (t): 0.374 Batch (t): 5.631, 2693.93/s, 168.371/s/gpu LR: 0.000023 Logit Scale: 99.470 Contrastive_loss: 0.61913 (0.80350) Loss: 0.61913 (0.80350)
2025-05-08,07:45:34 | INFO | Train Epoch: 3 [ 81805312/128008192 (64%)] Data (t): 0.373 Batch (t): 5.600, 2861.15/s, 178.822/s/gpu LR: 0.000021 Logit Scale: 99.544 Contrastive_loss: 0.72858 (0.80163) Loss: 0.72858 (0.80163)
2025-05-08,07:49:30 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
2025-05-08,07:56:06 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
2025-05-08,07:57:27 | INFO | Train Epoch: 3 [ 83902464/128008192 (66%)] Data (t): 0.350 Batch (t): 5.567, 2898.39/s, 181.149/s/gpu LR: 0.000019 Logit Scale: 99.597 Contrastive_loss: 0.77588 (0.80100) Loss: 0.77588 (0.80100)
2025-05-08,08:06:38 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
2025-05-08,08:09:28 | INFO | Train Epoch: 3 [ 85999616/128008192 (67%)] Data (t): 0.497 Batch (t): 5.638, 2792.88/s, 174.555/s/gpu LR: 0.000017 Logit Scale: 99.646 Contrastive_loss: 0.65114 (0.79743) Loss: 0.65114 (0.79743)
2025-05-08,08:21:26 | INFO | Train Epoch: 3 [ 88096768/128008192 (69%)] Data (t): 0.792 Batch (t): 5.610, 2952.94/s, 184.559/s/gpu LR: 0.000015 Logit Scale: 99.706 Contrastive_loss: 0.69861 (0.79513) Loss: 0.69861 (0.79513)
2025-05-08,08:23:58 | WARNING | Handling webdataset error (OSError('image file is truncated (121 bytes not processed)')). Ignoring.
2025-05-08,08:27:53 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
2025-05-08,08:29:49 | WARNING | Handling webdataset error (OSError('image file is truncated (58 bytes not processed)')). Ignoring.
2025-05-08,08:33:27 | INFO | Train Epoch: 3 [ 90193920/128008192 (70%)] Data (t): 0.452 Batch (t): 5.626, 2940.24/s, 183.765/s/gpu LR: 0.000014 Logit Scale: 99.745 Contrastive_loss: 0.77609 (0.79470) Loss: 0.77609 (0.79470)
2025-05-08,08:33:39 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
2025-05-08,08:45:34 | INFO | Train Epoch: 3 [ 92291072/128008192 (72%)] Data (t): 0.359 Batch (t): 5.680, 2894.32/s, 180.895/s/gpu LR: 0.000012 Logit Scale: 99.795 Contrastive_loss: 0.66336 (0.79178) Loss: 0.66336 (0.79178)
2025-05-08,08:57:32 | INFO | Train Epoch: 3 [ 94388224/128008192 (74%)] Data (t): 0.369 Batch (t): 5.615, 2939.65/s, 183.728/s/gpu LR: 0.000011 Logit Scale: 99.821 Contrastive_loss: 0.81662 (0.79232) Loss: 0.81662 (0.79232)
2025-05-08,09:02:33 | WARNING | Handling webdataset error (OSError('image file is truncated (24 bytes not processed)')). Ignoring.
2025-05-08,09:09:34 | INFO | Train Epoch: 3 [ 96485376/128008192 (75%)] Data (t): 0.350 Batch (t): 5.639, 2825.22/s, 176.576/s/gpu LR: 0.000010 Logit Scale: 99.850 Contrastive_loss: 0.68474 (0.79003) Loss: 0.68474 (0.79003)
2025-05-08,09:10:59 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
2025-05-08,09:21:33 | INFO | Train Epoch: 3 [ 98582528/128008192 (77%)] Data (t): 0.346 Batch (t): 5.613, 2958.79/s, 184.924/s/gpu LR: 0.000008 Logit Scale: 99.879 Contrastive_loss: 0.65538 (0.78723) Loss: 0.65538 (0.78723)
2025-05-08,09:33:31 | INFO | Train Epoch: 3 [100679680/128008192 (79%)] Data (t): 0.352 Batch (t): 5.617, 2898.83/s, 181.177/s/gpu LR: 0.000007 Logit Scale: 99.902 Contrastive_loss: 0.66111 (0.78465) Loss: 0.66111 (0.78465)
2025-05-08,09:39:50 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
2025-05-08,09:45:27 | INFO | Train Epoch: 3 [102776832/128008192 (80%)] Data (t): 0.373 Batch (t): 5.593, 2776.26/s, 173.516/s/gpu LR: 0.000006 Logit Scale: 99.922 Contrastive_loss: 0.66706 (0.78230) Loss: 0.66706 (0.78230)
2025-05-08,09:47:22 | WARNING | Handling webdataset error (OSError('image file is truncated (97 bytes not processed)')). Ignoring.
2025-05-08,09:57:26 | INFO | Train Epoch: 3 [104873984/128008192 (82%)] Data (t): 0.375 Batch (t): 5.617, 2941.01/s, 183.813/s/gpu LR: 0.000005 Logit Scale: 99.950 Contrastive_loss: 0.60179 (0.77876) Loss: 0.60179 (0.77876)
2025-05-08,10:09:31 | INFO | Train Epoch: 3 [106971136/128008192 (84%)] Data (t): 0.362 Batch (t): 5.665, 2954.08/s, 184.630/s/gpu LR: 0.000004 Logit Scale: 99.959 Contrastive_loss: 0.87652 (0.78064) Loss: 0.87652 (0.78064)
2025-05-08,10:10:02 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
2025-05-08,10:21:28 | INFO | Train Epoch: 3 [109068288/128008192 (85%)] Data (t): 0.374 Batch (t): 5.601, 2917.53/s, 182.346/s/gpu LR: 0.000003 Logit Scale: 99.973 Contrastive_loss: 0.71769 (0.77945) Loss: 0.71769 (0.77945)
2025-05-08,10:25:51 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
2025-05-08,10:33:32 | INFO | Train Epoch: 3 [111165440/128008192 (87%)] Data (t): 0.357 Batch (t): 5.655, 2963.85/s, 185.241/s/gpu LR: 0.000003 Logit Scale: 99.981 Contrastive_loss: 0.72722 (0.77849) Loss: 0.72722 (0.77849)
2025-05-08,10:36:18 | WARNING | Handling webdataset error (OSError('image file is truncated (58 bytes not processed)')). Ignoring.
2025-05-08,10:45:27 | INFO | Train Epoch: 3 [113262592/128008192 (88%)] Data (t): 0.311 Batch (t): 5.584, 2899.14/s, 181.196/s/gpu LR: 0.000002 Logit Scale: 99.989 Contrastive_loss: 0.62307 (0.77566) Loss: 0.62307 (0.77566)
2025-05-08,10:49:38 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
2025-05-08,10:57:29 | INFO | Train Epoch: 3 [115359744/128008192 (90%)] Data (t): 0.324 Batch (t): 5.640, 2876.68/s, 179.793/s/gpu LR: 0.000002 Logit Scale: 99.991 Contrastive_loss: 0.81450 (0.77636) Loss: 0.81450 (0.77636)
2025-05-08,11:02:03 | WARNING | Handling webdataset error (OSError('image file is truncated (6 bytes not processed)')). Ignoring.
2025-05-08,11:09:29 | INFO | Train Epoch: 3 [117456896/128008192 (92%)] Data (t): 0.371 Batch (t): 5.627, 2964.98/s, 185.311/s/gpu LR: 0.000001 Logit Scale: 99.993 Contrastive_loss: 0.80150 (0.77680) Loss: 0.80150 (0.77680)
2025-05-08,11:17:53 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
2025-05-08,11:21:30 | INFO | Train Epoch: 3 [119554048/128008192 (93%)] Data (t): 0.342 Batch (t): 5.633, 2890.31/s, 180.644/s/gpu LR: 0.000001 Logit Scale: 99.994 Contrastive_loss: 0.78408 (0.77692) Loss: 0.78408 (0.77692)
2025-05-08,11:22:19 | WARNING | Handling webdataset error (OSError('image file is truncated (70 bytes not processed)')). Ignoring.
2025-05-08,11:33:31 | INFO | Train Epoch: 3 [121651200/128008192 (95%)] Data (t): 0.320 Batch (t): 5.632, 2722.51/s, 170.157/s/gpu LR: 0.000000 Logit Scale: 99.993 Contrastive_loss: 0.68627 (0.77539) Loss: 0.68627 (0.77539)
2025-05-08,11:45:28 | INFO | Train Epoch: 3 [123748352/128008192 (97%)] Data (t): 0.364 Batch (t): 5.600, 2924.32/s, 182.770/s/gpu LR: 0.000000 Logit Scale: 99.993 Contrastive_loss: 0.63552 (0.77305) Loss: 0.63552 (0.77305)
2025-05-08,11:47:23 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
2025-05-08,11:57:28 | INFO | Train Epoch: 3 [125845504/128008192 (98%)] Data (t): 0.344 Batch (t): 5.629, 2963.38/s, 185.211/s/gpu LR: 0.000000 Logit Scale: 99.993 Contrastive_loss: 0.65867 (0.77118) Loss: 0.65867 (0.77118)
2025-05-08,12:09:34 | INFO | Train Epoch: 3 [127942656/128008192 (100%)] Data (t): 0.356 Batch (t): 5.673, 2939.31/s, 183.707/s/gpu LR: 0.000000 Logit Scale: 99.993 Contrastive_loss: 0.79569 (0.77157) Loss: 0.79569 (0.77157)
2025-05-08,12:09:56 | INFO | Train Epoch: 3 [128008192/128008192 (100%)] Data (t): 0.311 Batch (t): 5.481, 3102.16/s, 193.885/s/gpu LR: 0.000000 Logit Scale: 99.993 Contrastive_loss: 0.76234 (0.77143) Loss: 0.76234 (0.77143)
2025-05-08,12:10:04 | INFO | Starting zero-shot imagenet.
2025-05-08,12:10:04 | INFO | Building zero-shot classifier
2025-05-08,12:10:22 | INFO | Using classifier